Github user Tibor17 commented on the pull request:
https://github.com/apache/maven-surefire/pull/108#issuecomment-161945817
@seanf
>>Are you saying you did try to build an IT for this?
reuseForks=false would not work in 2.19 feature.
Initially skipAfterFailureCount was not count because it was boolean and
therefore rerunFailingTestsCount did not make sense.
Now it may have sense.
>>What counts would you consider realistic?
Maybe 5 test classes or more with Thread.sleep() 3 seconds, forkCount=2,
rerunFailingTestsCount = 3 and skipAfterFailureCount=2. There should be two
tests. One should work with #failures < rerunFailingTestsCount (all 5 test
classes passed with flaky tests), and second test in IT where #failures >
rerunFailingTestsCount (totally maybe only 3 or 4 test classes run and two
failed). Each class has one method.
The classes are ordered on alphabetical order and forked.
That's my idea but you can improve it of course and the system properties
can be used in order to force tests to fail e.g.
```
@Test test() {
if (sys-prop) Assert.fail();
}
```
Let's see how long sleep() should be in the tests..
>>I'm still assuming failureCount is meant to count failed tests, not bad
runs. You didn't answer my question about that (and the docs don't say),
but I do think it is better to count tests, as long as it is documented.
If the rerun is enabled, the flaky tests (I mean those bad tests <
rerunFailingTestsCount) should not be a subject to the feature with
skipAfterFailureCount. I don't remember the code in detail. We need the IT to
confirm both features may work together.
>>As I said, there should be no race condition if failureCount counts
(completely) failed tests, assuming the runs for each test are serialised.
Race condition is between forks. In extremely short test method race
condition may happen and the failure count may grow over expected
skipAfterFailureCount .
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
