NICE!!! This is awesome. I’m definitely going to be using this :) On Sat, Sep 20, 2014 at 8:44 AM, Andreas Gudian <andreas.gud...@gmail.com> wrote:
> Hi there, > > We had a pull request for such a feature and it was merged. It will be part > of our next release: https://jira.codehaus.org/browse/SUREFIRE > <https://jira.codehaus.org/browse/SUREFIRE-1087>-1087 > > I think the feature makes expecially sense for tracking flaky tests over a > longer period of time, as those flaky tests show up more nicely on the > surefire tester port xml files and can thus be tracked specifically in the > CI system. > > Surefire 2.18 should be out in the next couple of weeks. > > Hth, > Andreas > > Am Samstag, 20. September 2014 schrieb Kevin Burton : > > > The loop idea is a good one.. I’d have to implement that on my own but > it’s > > not the end of the world. > > > > And I’d love to live in a world you present above where all test cases > work > > and can be isolated :) > > > > … but it’s not practical. Google has a big problem with this in their > unit > > testing. They have millions of unit tests and they can’t fail a whole > > build because 1/5000 of their tests are flakey. At 1M tests you’re going > > to end up with a lot failing. > > > > In my situations I have weird race conditions with ActiveMQ having a JMX > > bug where there’s a race from the GC time of the queue and the time I > read > > the JMX value. > > > > I have about 2-3 of these.. I CAN NOT reproduce these at all.. only > during > > testing. It’s amazingly annoying :-( > > > > On Fri, Sep 19, 2014 at 10:18 AM, Martin Todorov <carlspr...@gmail.com > > <javascript:;>> > > wrote: > > > > > I agree with Curtis about the possible approach of doing a for loop. > I'm > > > also aware of the fact that every corner case is a case of it's own and > > > that you sometimes need to be able to do things quick and dirty. > > > > > > I had a manager who insisted I should simply ignore one of the test > cases > > > he'd written, which was failing regularly only during releasing on our > CI > > > server. He refused to fix the test for about a year and a half, as it > was > > > "a flake". Eventually, it started hitting people big time, especially > > when > > > we moved to the cloud, and it turned out to be something which > shouldn't > > > have been ignored at all... > > > > > > In the end, it's all up to you, but I'd recommend playing it as safe as > > > possible! :-) > > > > > > On Fri, Sep 19, 2014 at 6:08 PM, Curtis Rueden <ctrue...@wisc.edu > > <javascript:;>> wrote: > > > > > > > Hi Kevin, > > > > > > > > > Is there a way to retry a flakey test? > > > > > > > > In general I agree with Martin Todorov that tests should be small and > > > > atomic, and flakiness is a sign of larger problems. However, I also > > agree > > > > with you that sometimes flaky tests are a reality: my group has run > > into > > > > this with behavior of the JVM garbage collector differing on Windows, > > for > > > > example, in a rather nondeterministic way. In that case, retrying > tests > > > > ~5-10 times was a good solution for us. > > > > > > > > But rather than configuring Surefire or Failsafe to do this, I'd > > suggest > > > > (if possible) to write it into the test itself. Put the test in a for > > > loop, > > > > and then assert at the end that numSuccesses > 0 or whatever. You > then > > > also > > > > issue a warning if numSuccesses != numAttempts, as you requested. In > > > short: > > > > it gives you the flexibility you need. > > > > > > > > Regards, > > > > Curtis > > > > > > > > On Fri, Sep 19, 2014 at 11:24 AM, Kevin Burton <bur...@spinn3r.com > > <javascript:;>> > > > wrote: > > > > > > > > > Is there a way to retry a flakey test? > > > > > > > > > > I’d basically like to have a flag that retries a failing test 2 or > 3 > > > > times. > > > > > > > > > > Flakey and non deterministic tests are a fact of life in more > > > complicated > > > > > systems. > > > > > > > > > > The main problem being that they are impossible to setup again > > because > > > > > they’re usually race conditions. > > > > > > > > > > If I could just retry the test a second time , that would solve > this > > > > issue. > > > > > > > > > > Would be nice to have the build warned that the test is failing > > > though.. > > > > > > > > > > -- > > > > > > > > > > Founder/CEO Spinn3r.com > > > > > Location: *San Francisco, CA* > > > > > blog: http://burtonator.wordpress.com > > > > > … or check out my Google+ profile > > > > > <https://plus.google.com/102718274791889610666/posts> > > > > > <http://spinn3r.com> > > > > > > > > > > > > > > > > > > > > -- > > > > Founder/CEO Spinn3r.com > > Location: *San Francisco, CA* > > blog: http://burtonator.wordpress.com > > … or check out my Google+ profile > > <https://plus.google.com/102718274791889610666/posts> > > <http://spinn3r.com> > > > -- Founder/CEO Spinn3r.com Location: *San Francisco, CA* blog: http://burtonator.wordpress.com … or check out my Google+ profile <https://plus.google.com/102718274791889610666/posts> <http://spinn3r.com>
