[
https://issues.apache.org/jira/browse/SUREFIRE-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15113676#comment-15113676
]
Tibor Digana commented on SUREFIRE-1219:
----------------------------------------
[~sflanigan]
[~seanf]
As a short answer, the feature works with skipAfterFailureCount but simply
ignores rerunFailingTestsCount.
This means the failed test immediately incremented skipAfterFailureCount
counter but it is not incremented when rerun phase is taken into account. This
is very important because that's the only way how to make both JUnit providers
surefire-junit4 and surefire-junit47 working the same way and skipping the test
phase eagerly.
I know what you want to achieve, that the test is passed in next re-run but
does not increment the counter but that's not possible for surefire-junit47 and
not for forkCount=0 or 1 and forkCount=2C. Forking the tests is changing the
behavior but we want to keep it the same for all scenarios. The point of
skipAfterFailureCount is to skip the test-set eagerly and fail the build asap
without waiting for whole run of whole test-set. The users do not want to wait
for long running tests only due to they fail. Maybe they have common root
cause, due to last commit in SVN, thus no reason to wait for next 100 test
until they fail. If we placed the function of skipAfterFailureCount at the end
of test phase then this feature would not make much sense because it just did
not stop the build eagerly. Yes it is a kind of limit in rerun feature on how
far the rerun may go. That's the reason why you can see both failures/errors
and skipped tests.
> skipAfterFailureCount should not count flaky tests as failures when using
> rerunFailingTestsCount
> ------------------------------------------------------------------------------------------------
>
> Key: SUREFIRE-1219
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1219
> Project: Maven Surefire
> Issue Type: Bug
> Affects Versions: 2.19.1
> Reporter: Sean Flanigan
> Attachments: SUREFIRE-1219-tests.zip
>
>
> According to
> https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html
> "failed tests within re-run phase are not included in
> skipAfterFailureCount". For instance, if skipAfterFailureCount is 1, but
> every test passes on the second attempt, the tests are "flaky" but not
> "failures". This should be reflected in summaries, in XML result files and
> in fail fast (skipAfterFailureCount), but this is not the case.
> If I have 10 flaky-but-not-failing tests, set rerunFailingTestsCount=1 and
> skipAfterFailureCount=5:
> 1. I would expect something like this to be printed on the console:
> Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Flakes: 10
> 2. The XML report should list <flakyFailure>s but no <failures>.
> 3. All tests should be run, not just 5 of them, in other words
> skipAfterFailureCount should not count the flaky tests as part of the failure
> count.
> I have some tests demonstrating the problem, which I will attach to this
> issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)