[jira] [Assigned] (SUREFIRE-1219) skipAfterFailureCount should not count flaky tests as failures when using rerunFailingTestsCount
[ https://issues.apache.org/jira/browse/SUREFIRE-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sylwester Lachiewicz reassigned SUREFIRE-1219: -- Assignee: (was: Tibor Digana) > skipAfterFailureCount should not count flaky tests as failures when using > rerunFailingTestsCount > > > Key: SUREFIRE-1219 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1219 > Project: Maven Surefire > Issue Type: Bug >Affects Versions: 2.19.1 >Reporter: Sean Flanigan >Priority: Minor > Attachments: SUREFIRE-1219-tests.zip > > > According to > https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html > "failed tests within re-run phase are not included in > skipAfterFailureCount". For instance, if skipAfterFailureCount is 1, but > every test passes on the second attempt, the tests are "flaky" but not > "failures". This should be reflected in summaries, in XML result files and > in fail fast (skipAfterFailureCount), but this is not the case. > If I have 10 flaky-but-not-failing tests, set rerunFailingTestsCount=1 and > skipAfterFailureCount=5: > 1. I would expect something like this to be printed on the console: > Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Flakes: 10 > 2. The XML report should list s but no . > 3. All tests should be run, not just 5 of them, in other words > skipAfterFailureCount should not count the flaky tests as part of the failure > count. > I have some tests demonstrating the problem, which I will attach to this > issue. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (SUREFIRE-1219) skipAfterFailureCount should not count flaky tests as failures when using rerunFailingTestsCount
[ https://issues.apache.org/jira/browse/SUREFIRE-1219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tibor Digana reassigned SUREFIRE-1219: -- Assignee: Tibor Digana > skipAfterFailureCount should not count flaky tests as failures when using > rerunFailingTestsCount > > > Key: SUREFIRE-1219 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1219 > Project: Maven Surefire > Issue Type: Bug >Affects Versions: 2.19.1 >Reporter: Sean Flanigan >Assignee: Tibor Digana > Attachments: SUREFIRE-1219-tests.zip > > > According to > https://maven.apache.org/surefire/maven-surefire-plugin/examples/rerun-failing-tests.html > "failed tests within re-run phase are not included in > skipAfterFailureCount". For instance, if skipAfterFailureCount is 1, but > every test passes on the second attempt, the tests are "flaky" but not > "failures". This should be reflected in summaries, in XML result files and > in fail fast (skipAfterFailureCount), but this is not the case. > If I have 10 flaky-but-not-failing tests, set rerunFailingTestsCount=1 and > skipAfterFailureCount=5: > 1. I would expect something like this to be printed on the console: > Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Flakes: 10 > 2. The XML report should list s but no . > 3. All tests should be run, not just 5 of them, in other words > skipAfterFailureCount should not count the flaky tests as part of the failure > count. > I have some tests demonstrating the problem, which I will attach to this > issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
