I have not tried, Kristian, but it looks promising. Both your link and the randomised testing one I have forwarded to my old Scrum team (I am no longer coaching them, thus unable to access their code base and try for myself).
Thank you. :-) -- Alexander Kriegisch > Am 28.05.2014 um 09:50 schrieb Kristian Rosenvold > <[email protected]>: > > Does this fit the bill for you ? > > http://stackoverflow.com/questions/8295100/how-to-re-run-failed-junit-tests-immediately > > Kristian > > > 2014-05-28 9:14 GMT+02:00 Alexander Kriegisch <[email protected]>: >> >> >> -- >> Alexander Kriegisch >> http://scrum-master.de >> >>> Am 28.05.2014 um 05:13 schrieb Benson Margulies <[email protected]>: >>> >>> On Tue, May 27, 2014 at 1:14 PM, Qingzhou Luo <[email protected]>wrote: >>> >>>> I am an intern at Google. The first step of my internship project is to add >>>> the ability to Maven to automatically rerun failing tests a few times, to >>>> see if they ever pass in any of the reruns. It is useful because in many >>>> cases a test fails because it is flaky, not because there is a bug in the >>>> new source code change. >>> >>> I entirely disagree. 99.9999% of programs are completely deterministic. If >>> you feed them the same inputs, you get the same output. >> >> Well, in my previous project (I am an Agile Coach) my teams were working on >> a distributed, multi-threaded system and around 10% of all unit tests and >> next to all integration tests were affected by the exception you mentioned: >> >>> The exceptions are tests that use threads, and it's quite hard to do a good >>> job writing them. >> >> One can argue that several of the unit tests were not real unit tests >> because they mocked some, but not all of the infrastructure they needed for >> running. Anyway, they were running in Surefire (the integration tests in >> Failsafe), and some lf them were flaky. >> >>> Just 'rerunning a few times' does not lead to useful >>> information in my view. >> >> Only the information that indeed they are flaky and a rough percentage of >> cases in which they fail, which would be rather dependend on the test >> environment though. Better than nothing information-wise. So far, so good. >> >> Having said all that, I still agree to Benson that a failing test should >> just fail, period. I do not want developers to think, "it is only failing >> only in 5% of all runs, so it is fine / low priority to fix". OTOH I >> understand Qingzhou from a perspective of wishing to "test the test" or >> rather wishing to harden a flaky test into a stable one, either by fixing >> the test or the threading bug(s) it exposes. Benson's suggestion below looks >> promising though, so thanks for that. >> >>> if you want random testing, there's a fine alternative at >>> http://labs.carrotsearch.com/randomizedtesting.html, and it has a fine >>> maven plugin. It can already 'rerun your tests a few times' very easily. >>> >>> I wonder what the more involved surefire maintainers think, but as a PMC >>> member I'd be -1 on this proposal. >>> >>>> We think the right way to achieve this is to modify surefire plugin of >>>> maven. We want to add it as a part of the configuration of surefire, so >>>> users can decide whether they want to enable this feature, and how many >>>> times they want to rerun failing tests. We plan to open-source our >>>> contribution, and hopefully can merge our code into surefire master branch >>>> in the end. Therefore, we are wondering do you have any >>>> comments/suggestions/opinions regarding this? We appreciate any input. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
