Hi Jason, Thanks for your reply. We plan to first extend surefire, so users can put a boolean parameter rerunFailintTests and an int parameter rerunFailintTestsCount, to tell surefire whether to rerun failing tests immediately after they fail. If a test passes in any of those reruns if will be marked as pass, but all the failing attempts will still be recorded and may be presented to users in a different way.
We plan to finish everything in next 1-2 months. So we wonder would you be willing to accept our contribution in the end, and patch our code into surefire? Thanks, Qingzhou On Wed, May 28, 2014 at 5:41 AM, Jason van Zyl <[email protected]> wrote: > > On May 27, 2014, at 1:14 PM, Qingzhou Luo <[email protected]> wrote: > > Hi, > > I am an intern at Google. The first step of my internship project is to add > the ability to Maven to automatically rerun failing tests a few times, to > see if they ever pass in any of the reruns. It is useful because in many > cases a test fails because it is flaky, not because there is a bug in the > new source code change. > > > I think generally being able to re-run failed tests is useful. I use this > feature in the IDE all the time in a fail/fix/run/repeat scenario. If > you're trying to identify flaky tests I think that's useful especially if > you can collect statistics. While In the long term you may not want to rely > on the re-running of failed tests that later succeed without alteration as > the basis for good testing, but it's a good identification tool, but a > useful feature in its own right. > > We think the right way to achieve this is to modify surefire plugin of > maven. We want to add it as a part of the configuration of surefire, so > users can decide whether they want to enable this feature, and how many > times they want to rerun failing tests. We plan to open-source our > contribution, and hopefully can merge our code into surefire master branch > in the end. Therefore, we are wondering do you have any > comments/suggestions/opinions regarding this? We appreciate any input. > > > Maybe a simple mechanism where you store the failed tests in a file and > then have a flag to run only the failed tests. I think this would be very > useful. Then possible a small, configurable loop around the main execution > of tests if you wanted to specify how many times to run the tests. > > Thank you very much! > > Regards, > > Qingzhou > > > Thanks, > > Jason > > ---------------------------------------------------------- > Jason van Zyl > Founder, Apache Maven > http://twitter.com/jvanzyl > http://twitter.com/takari_io > --------------------------------------------------------- > > > The modern conservative is engaged in one of man's oldest exercises in moral > philosophy; that is, > the search for a superior moral justification for selfishness. > > -- John Kenneth Galbraith > > > > > > > > > >
