Hi there, We had a pull request for such a feature and it was merged. It will be part of our next release: https://jira.codehaus.org/browse/SUREFIRE <https://jira.codehaus.org/browse/SUREFIRE-1087>-1087
I think the feature makes expecially sense for tracking flaky tests over a longer period of time, as those flaky tests show up more nicely on the surefire tester port xml files and can thus be tracked specifically in the CI system. Surefire 2.18 should be out in the next couple of weeks. Hth, Andreas Am Samstag, 20. September 2014 schrieb Kevin Burton : > The loop idea is a good one.. I’d have to implement that on my own but it’s > not the end of the world. > > And I’d love to live in a world you present above where all test cases work > and can be isolated :) > > … but it’s not practical. Google has a big problem with this in their unit > testing. They have millions of unit tests and they can’t fail a whole > build because 1/5000 of their tests are flakey. At 1M tests you’re going > to end up with a lot failing. > > In my situations I have weird race conditions with ActiveMQ having a JMX > bug where there’s a race from the GC time of the queue and the time I read > the JMX value. > > I have about 2-3 of these.. I CAN NOT reproduce these at all.. only during > testing. It’s amazingly annoying :-( > > On Fri, Sep 19, 2014 at 10:18 AM, Martin Todorov <[email protected] > <javascript:;>> > wrote: > > > I agree with Curtis about the possible approach of doing a for loop. I'm > > also aware of the fact that every corner case is a case of it's own and > > that you sometimes need to be able to do things quick and dirty. > > > > I had a manager who insisted I should simply ignore one of the test cases > > he'd written, which was failing regularly only during releasing on our CI > > server. He refused to fix the test for about a year and a half, as it was > > "a flake". Eventually, it started hitting people big time, especially > when > > we moved to the cloud, and it turned out to be something which shouldn't > > have been ignored at all... > > > > In the end, it's all up to you, but I'd recommend playing it as safe as > > possible! :-) > > > > On Fri, Sep 19, 2014 at 6:08 PM, Curtis Rueden <[email protected] > <javascript:;>> wrote: > > > > > Hi Kevin, > > > > > > > Is there a way to retry a flakey test? > > > > > > In general I agree with Martin Todorov that tests should be small and > > > atomic, and flakiness is a sign of larger problems. However, I also > agree > > > with you that sometimes flaky tests are a reality: my group has run > into > > > this with behavior of the JVM garbage collector differing on Windows, > for > > > example, in a rather nondeterministic way. In that case, retrying tests > > > ~5-10 times was a good solution for us. > > > > > > But rather than configuring Surefire or Failsafe to do this, I'd > suggest > > > (if possible) to write it into the test itself. Put the test in a for > > loop, > > > and then assert at the end that numSuccesses > 0 or whatever. You then > > also > > > issue a warning if numSuccesses != numAttempts, as you requested. In > > short: > > > it gives you the flexibility you need. > > > > > > Regards, > > > Curtis > > > > > > On Fri, Sep 19, 2014 at 11:24 AM, Kevin Burton <[email protected] > <javascript:;>> > > wrote: > > > > > > > Is there a way to retry a flakey test? > > > > > > > > I’d basically like to have a flag that retries a failing test 2 or 3 > > > times. > > > > > > > > Flakey and non deterministic tests are a fact of life in more > > complicated > > > > systems. > > > > > > > > The main problem being that they are impossible to setup again > because > > > > they’re usually race conditions. > > > > > > > > If I could just retry the test a second time , that would solve this > > > issue. > > > > > > > > Would be nice to have the build warned that the test is failing > > though.. > > > > > > > > -- > > > > > > > > Founder/CEO Spinn3r.com > > > > Location: *San Francisco, CA* > > > > blog: http://burtonator.wordpress.com > > > > … or check out my Google+ profile > > > > <https://plus.google.com/102718274791889610666/posts> > > > > <http://spinn3r.com> > > > > > > > > > > > > > -- > > Founder/CEO Spinn3r.com > Location: *San Francisco, CA* > blog: http://burtonator.wordpress.com > … or check out my Google+ profile > <https://plus.google.com/102718274791889610666/posts> > <http://spinn3r.com> >
