The loop idea is a good one.. I’d have to implement that on my own but it’s not the end of the world.
And I’d love to live in a world you present above where all test cases work and can be isolated :) … but it’s not practical. Google has a big problem with this in their unit testing. They have millions of unit tests and they can’t fail a whole build because 1/5000 of their tests are flakey. At 1M tests you’re going to end up with a lot failing. In my situations I have weird race conditions with ActiveMQ having a JMX bug where there’s a race from the GC time of the queue and the time I read the JMX value. I have about 2-3 of these.. I CAN NOT reproduce these at all.. only during testing. It’s amazingly annoying :-( On Fri, Sep 19, 2014 at 10:18 AM, Martin Todorov <carlspr...@gmail.com> wrote: > I agree with Curtis about the possible approach of doing a for loop. I'm > also aware of the fact that every corner case is a case of it's own and > that you sometimes need to be able to do things quick and dirty. > > I had a manager who insisted I should simply ignore one of the test cases > he'd written, which was failing regularly only during releasing on our CI > server. He refused to fix the test for about a year and a half, as it was > "a flake". Eventually, it started hitting people big time, especially when > we moved to the cloud, and it turned out to be something which shouldn't > have been ignored at all... > > In the end, it's all up to you, but I'd recommend playing it as safe as > possible! :-) > > On Fri, Sep 19, 2014 at 6:08 PM, Curtis Rueden <ctrue...@wisc.edu> wrote: > > > Hi Kevin, > > > > > Is there a way to retry a flakey test? > > > > In general I agree with Martin Todorov that tests should be small and > > atomic, and flakiness is a sign of larger problems. However, I also agree > > with you that sometimes flaky tests are a reality: my group has run into > > this with behavior of the JVM garbage collector differing on Windows, for > > example, in a rather nondeterministic way. In that case, retrying tests > > ~5-10 times was a good solution for us. > > > > But rather than configuring Surefire or Failsafe to do this, I'd suggest > > (if possible) to write it into the test itself. Put the test in a for > loop, > > and then assert at the end that numSuccesses > 0 or whatever. You then > also > > issue a warning if numSuccesses != numAttempts, as you requested. In > short: > > it gives you the flexibility you need. > > > > Regards, > > Curtis > > > > On Fri, Sep 19, 2014 at 11:24 AM, Kevin Burton <bur...@spinn3r.com> > wrote: > > > > > Is there a way to retry a flakey test? > > > > > > I’d basically like to have a flag that retries a failing test 2 or 3 > > times. > > > > > > Flakey and non deterministic tests are a fact of life in more > complicated > > > systems. > > > > > > The main problem being that they are impossible to setup again because > > > they’re usually race conditions. > > > > > > If I could just retry the test a second time , that would solve this > > issue. > > > > > > Would be nice to have the build warned that the test is failing > though.. > > > > > > -- > > > > > > Founder/CEO Spinn3r.com > > > Location: *San Francisco, CA* > > > blog: http://burtonator.wordpress.com > > > … or check out my Google+ profile > > > <https://plus.google.com/102718274791889610666/posts> > > > <http://spinn3r.com> > > > > > > -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile <https://plus.google.com/102718274791889610666/posts> <http://spinn3r.com>
