[
https://issues.apache.org/jira/browse/SUREFIRE-1202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041337#comment-15041337
]
Tibor Digana commented on SUREFIRE-1202:
----------------------------------------
>>Are you saying you did try to build an IT for this?
reuseForks=false would not work in 2.19 feature.
Initially skipAfterFailureCount was not count because it was boolean and
therefore rerunFailingTestsCount did not make sense.
Now it may have sense.
>>What counts would you consider realistic?
Maybe 5 test classes or more with Thread.sleep() 3 seconds, forkCount=2,
rerunFailingTestsCount = 3 and skipAfterFailureCount=2. There should be two
tests. One should work with #failures < rerunFailingTestsCount (all 5 test
classes passed with flaky tests), and second test in IT where #failures >
rerunFailingTestsCount (totally maybe only 3 or 4 test classes run and two
failed). Each class has one method.
The classes are ordered on alphabetical order and forked.
That's my idea but you can improve it of course and the system properties can
be used in order to force tests to fail e.g.
{code}
@Test test() {
if (sys-prop) Assert.fail();
}
{code}
Let's see how long sleep() should be in the tests..
>>I'm still assuming failureCount is meant to count failed tests, not bad
runs. You didn't answer my question about that (and the docs don't say),
but I do think it is better to count tests, as long as it is documented.
If the rerun is enabled, the flaky tests (I mean those bad tests <
rerunFailingTestsCount) should not be a subject to the feature with
skipAfterFailureCount. I don't remember the code in detail. We need the IT to
confirm both features may work together.
>>As I said, there should be no race condition if failureCount counts
(completely) failed tests, assuming the runs for each test are serialised.
Race condition is between forks. In extremely short test method race condition
may happen and the failure count may grow over expected skipAfterFailureCount .
> Allow rerunFailingTestsCount, skipAfterFailureCount together
> ------------------------------------------------------------
>
> Key: SUREFIRE-1202
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1202
> Project: Maven Surefire
> Issue Type: Improvement
> Components: Maven Failsafe Plugin
> Affects Versions: 2.19
> Reporter: Sean Flanigan
> Assignee: Tibor Digana
>
> I couldn't find this limitation documented anywhere, but if I try to use the
> options {{rerunFailingTestsCount}} and {{skipAfterFailureCount}} together, I
> get this error message:
> {noformat}
> org.apache.maven.plugin.MojoFailureException:
> Parameters ["rerunFailingTestsCount", "skipAfterFailureCount"]
> should not be enabled together
> {noformat}
> I have a use case which requires both options: some integration tests are
> flaky, so I want to use {{rerunFailingTestsCount}} to repeat them, but if all
> tests are failing (eg due to a configuration problem or a really bad change),
> I don't want to wait for the entire test suite to fail multiple times, so I
> would like to abort if say 10 tests have failed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
