On May 27, 2014, at 1:14 PM, Qingzhou Luo <[email protected]> wrote:
> Hi, > > I am an intern at Google. The first step of my internship project is to add > the ability to Maven to automatically rerun failing tests a few times, to > see if they ever pass in any of the reruns. It is useful because in many > cases a test fails because it is flaky, not because there is a bug in the > new source code change. > I think generally being able to re-run failed tests is useful. I use this feature in the IDE all the time in a fail/fix/run/repeat scenario. If you're trying to identify flaky tests I think that's useful especially if you can collect statistics. While In the long term you may not want to rely on the re-running of failed tests that later succeed without alteration as the basis for good testing, but it's a good identification tool, but a useful feature in its own right. > We think the right way to achieve this is to modify surefire plugin of > maven. We want to add it as a part of the configuration of surefire, so > users can decide whether they want to enable this feature, and how many > times they want to rerun failing tests. We plan to open-source our > contribution, and hopefully can merge our code into surefire master branch > in the end. Therefore, we are wondering do you have any > comments/suggestions/opinions regarding this? We appreciate any input. > Maybe a simple mechanism where you store the failed tests in a file and then have a flag to run only the failed tests. I think this would be very useful. Then possible a small, configurable loop around the main execution of tests if you wanted to specify how many times to run the tests. > Thank you very much! > > Regards, > > Qingzhou Thanks, Jason ---------------------------------------------------------- Jason van Zyl Founder, Apache Maven http://twitter.com/jvanzyl http://twitter.com/takari_io --------------------------------------------------------- The modern conservative is engaged in one of man's oldest exercises in moral philosophy; that is, the search for a superior moral justification for selfishness. -- John Kenneth Galbraith
