I would really like to see and understand properly how this would collaborate with JUnit rules for retries. I suspect there might be some interesting issues regarding reporting (i.e. can the existing logic handle reporting of n different executions of a single test). How does this explode i parallel junit is selected :) ?
Technically you can just pass a system property from your pom and into the surefire fork with the number of retries, you can read this proprety in the junit rule. I really think you should start with this approach and flesh out the details from there. It might turn out this is all you need, or that some other entirely unforseen issues pop up. As for the actual property in the plugin, I really think we should see where this leads before deciding on how to do this. On a similar note, I know some people with large corporate clouds sometimes run a test on 10000 different nodes in parallel, to investigate if there is any kind of 1/10000 type failures, just to be able to summarize in 20 seconds instead of 10000 linear runs. This might also be another one of those cases where we could consider picking up user-written code from the test classpath; but I think I need another year thinking about that :) Kristian 2014-05-28 19:58 GMT+02:00 Jason van Zyl <[email protected]>: > I personally don't see an issue with those feature additions, but Kristian > does the lions share of the work on the Surefire plugin so it would be his > call. I would definitely support the additions of those features and would > help test. > > On May 28, 2014, at 1:46 PM, Qingzhou Luo <[email protected]> wrote: > >> Hi Jason, >> >> Thanks for your reply. We plan to first extend surefire, so users can put a >> boolean parameter rerunFailintTests and an int parameter >> rerunFailintTestsCount, to tell surefire whether to rerun failing tests >> immediately after they fail. If a test passes in any of those reruns if will >> be marked as pass, but all the failing attempts will still be recorded and >> may be presented to users in a different way. >> >> We plan to finish everything in next 1-2 months. So we wonder would you be >> willing to accept our contribution in the end, and patch our code into >> surefire? >> >> Thanks, >> >> Qingzhou >> >> >> On Wed, May 28, 2014 at 5:41 AM, Jason van Zyl <[email protected]> wrote: >> >> On May 27, 2014, at 1:14 PM, Qingzhou Luo <[email protected]> wrote: >> >>> Hi, >>> >>> I am an intern at Google. The first step of my internship project is to add >>> the ability to Maven to automatically rerun failing tests a few times, to >>> see if they ever pass in any of the reruns. It is useful because in many >>> cases a test fails because it is flaky, not because there is a bug in the >>> new source code change. >>> >> >> I think generally being able to re-run failed tests is useful. I use this >> feature in the IDE all the time in a fail/fix/run/repeat scenario. If you're >> trying to identify flaky tests I think that's useful especially if you can >> collect statistics. While In the long term you may not want to rely on the >> re-running of failed tests that later succeed without alteration as the >> basis for good testing, but it's a good identification tool, but a useful >> feature in its own right. >> >>> We think the right way to achieve this is to modify surefire plugin of >>> maven. We want to add it as a part of the configuration of surefire, so >>> users can decide whether they want to enable this feature, and how many >>> times they want to rerun failing tests. We plan to open-source our >>> contribution, and hopefully can merge our code into surefire master branch >>> in the end. Therefore, we are wondering do you have any >>> comments/suggestions/opinions regarding this? We appreciate any input. >>> >> >> Maybe a simple mechanism where you store the failed tests in a file and then >> have a flag to run only the failed tests. I think this would be very useful. >> Then possible a small, configurable loop around the main execution of tests >> if you wanted to specify how many times to run the tests. >> >>> Thank you very much! >>> >>> Regards, >>> >>> Qingzhou >> >> Thanks, >> >> Jason >> >> ---------------------------------------------------------- >> Jason van Zyl >> Founder, Apache Maven >> http://twitter.com/jvanzyl >> http://twitter.com/takari_io >> --------------------------------------------------------- >> >> The modern conservative is engaged in one of man's oldest exercises in moral >> philosophy; that is, >> the search for a superior moral justification for selfishness. >> >> -- John Kenneth Galbraith >> >> >> >> >> >> >> >> >> >> > > Thanks, > > Jason > > ---------------------------------------------------------- > Jason van Zyl > Founder, Apache Maven > http://twitter.com/jvanzyl > http://twitter.com/takari_io > --------------------------------------------------------- > > Selfish deeds are the shortest path to self destruction. > > -- The Seven Samuari, Akira Kurosawa > > > > > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
