I agree with Curtis about the possible approach of doing a for loop. I'm also aware of the fact that every corner case is a case of it's own and that you sometimes need to be able to do things quick and dirty.
I had a manager who insisted I should simply ignore one of the test cases he'd written, which was failing regularly only during releasing on our CI server. He refused to fix the test for about a year and a half, as it was "a flake". Eventually, it started hitting people big time, especially when we moved to the cloud, and it turned out to be something which shouldn't have been ignored at all... In the end, it's all up to you, but I'd recommend playing it as safe as possible! :-) On Fri, Sep 19, 2014 at 6:08 PM, Curtis Rueden <ctrue...@wisc.edu> wrote: > Hi Kevin, > > > Is there a way to retry a flakey test? > > In general I agree with Martin Todorov that tests should be small and > atomic, and flakiness is a sign of larger problems. However, I also agree > with you that sometimes flaky tests are a reality: my group has run into > this with behavior of the JVM garbage collector differing on Windows, for > example, in a rather nondeterministic way. In that case, retrying tests > ~5-10 times was a good solution for us. > > But rather than configuring Surefire or Failsafe to do this, I'd suggest > (if possible) to write it into the test itself. Put the test in a for loop, > and then assert at the end that numSuccesses > 0 or whatever. You then also > issue a warning if numSuccesses != numAttempts, as you requested. In short: > it gives you the flexibility you need. > > Regards, > Curtis > > On Fri, Sep 19, 2014 at 11:24 AM, Kevin Burton <bur...@spinn3r.com> wrote: > > > Is there a way to retry a flakey test? > > > > I’d basically like to have a flag that retries a failing test 2 or 3 > times. > > > > Flakey and non deterministic tests are a fact of life in more complicated > > systems. > > > > The main problem being that they are impossible to setup again because > > they’re usually race conditions. > > > > If I could just retry the test a second time , that would solve this > issue. > > > > Would be nice to have the build warned that the test is failing though.. > > > > -- > > > > Founder/CEO Spinn3r.com > > Location: *San Francisco, CA* > > blog: http://burtonator.wordpress.com > > … or check out my Google+ profile > > <https://plus.google.com/102718274791889610666/posts> > > <http://spinn3r.com> > > >
