Re: retrying flakey tests?

2014-09-20 Thread Andreas Gudian
Hi there, We had a pull request for such a feature and it was merged. It will be part of our next release: https://jira.codehaus.org/browse/SUREFIRE https://jira.codehaus.org/browse/SUREFIRE-1087-1087 I think the feature makes expecially sense for tracking flaky tests over a longer period of

Re: retrying flakey tests?

2014-09-20 Thread Kevin Burton
NICE!!! This is awesome. I’m definitely going to be using this :) On Sat, Sep 20, 2014 at 8:44 AM, Andreas Gudian andreas.gud...@gmail.com wrote: Hi there, We had a pull request for such a feature and it was merged. It will be part of our next release:

Re: retrying flakey tests?

2014-09-19 Thread Martin Todorov
Hi Kevin, I can't see an option in the maven-surefire-plugin, but even if there was one, I would seriously not recommend it. If a test case fails non-deterministically, this means it's design is flawed. Test cases should either produce their own resources, or use such that haven't been

Re: retrying flakey tests?

2014-09-19 Thread Curtis Rueden
Hi Kevin, Is there a way to retry a flakey test? In general I agree with Martin Todorov that tests should be small and atomic, and flakiness is a sign of larger problems. However, I also agree with you that sometimes flaky tests are a reality: my group has run into this with behavior of the JVM

Re: retrying flakey tests?

2014-09-19 Thread Martin Todorov
I agree with Curtis about the possible approach of doing a for loop. I'm also aware of the fact that every corner case is a case of it's own and that you sometimes need to be able to do things quick and dirty. I had a manager who insisted I should simply ignore one of the test cases he'd written,

Re: retrying flakey tests?

2014-09-19 Thread Kevin Burton
The loop idea is a good one.. I’d have to implement that on my own but it’s not the end of the world. And I’d love to live in a world you present above where all test cases work and can be isolated :) … but it’s not practical. Google has a big problem with this in their unit testing. They have