Does this fit the bill for you ? http://stackoverflow.com/questions/8295100/how-to-re-run-failed-junit-tests-immediately
Kristian 2014-05-28 9:14 GMT+02:00 Alexander Kriegisch <[email protected]>: > > > -- > Alexander Kriegisch > http://scrum-master.de > > Am 28.05.2014 um 05:13 schrieb Benson Margulies <[email protected]>: > >> On Tue, May 27, 2014 at 1:14 PM, Qingzhou Luo <[email protected]>wrote: >> >>> I am an intern at Google. The first step of my internship project is to add >>> the ability to Maven to automatically rerun failing tests a few times, to >>> see if they ever pass in any of the reruns. It is useful because in many >>> cases a test fails because it is flaky, not because there is a bug in the >>> new source code change. >> >> I entirely disagree. 99.9999% of programs are completely deterministic. If >> you feed them the same inputs, you get the same output. > > Well, in my previous project (I am an Agile Coach) my teams were working on a > distributed, multi-threaded system and around 10% of all unit tests and next > to all integration tests were affected by the exception you mentioned: > >> The exceptions are tests that use threads, and it's quite hard to do a good >> job writing them. > > One can argue that several of the unit tests were not real unit tests because > they mocked some, but not all of the infrastructure they needed for running. > Anyway, they were running in Surefire (the integration tests in Failsafe), > and some lf them were flaky. > >> Just 'rerunning a few times' does not lead to useful >> information in my view. > > Only the information that indeed they are flaky and a rough percentage of > cases in which they fail, which would be rather dependend on the test > environment though. Better than nothing information-wise. So far, so good. > > Having said all that, I still agree to Benson that a failing test should just > fail, period. I do not want developers to think, "it is only failing only in > 5% of all runs, so it is fine / low priority to fix". OTOH I understand > Qingzhou from a perspective of wishing to "test the test" or rather wishing > to harden a flaky test into a stable one, either by fixing the test or the > threading bug(s) it exposes. Benson's suggestion below looks promising > though, so thanks for that. > >> if you want random testing, there's a fine alternative at >> http://labs.carrotsearch.com/randomizedtesting.html, and it has a fine >> maven plugin. It can already 'rerun your tests a few times' very easily. >> >> I wonder what the more involved surefire maintainers think, but as a PMC >> member I'd be -1 on this proposal. >> >>> We think the right way to achieve this is to modify surefire plugin of >>> maven. We want to add it as a part of the configuration of surefire, so >>> users can decide whether they want to enable this feature, and how many >>> times they want to rerun failing tests. We plan to open-source our >>> contribution, and hopefully can merge our code into surefire master branch >>> in the end. Therefore, we are wondering do you have any >>> comments/suggestions/opinions regarding this? We appreciate any input. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
