[ https://issues.apache.org/jira/browse/SUREFIRE-1470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16416834#comment-16416834 ]
Tibor Digana commented on SUREFIRE-1470: ---------------------------------------- [~alexey.subach] Every CI should read and rely on {{XML}} and not the {{TXT}} file. > Unexpected report result when using rerunFailingTestsCount with groups filter > in Surefire > ----------------------------------------------------------------------------------------- > > Key: SUREFIRE-1470 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1470 > Project: Maven Surefire > Issue Type: Bug > Components: Maven Surefire Plugin > Reporter: Alexey Subach > Priority: Critical > Attachments: test.zip > > > When the following plugin configuration is used: > {code:java} > <plugin> > <groupId>org.apache.maven.plugins</groupId> > <artifactId>maven-surefire-plugin</artifactId> > <version>${surefire.version}</version> > <configuration> > <includes> > <include>**/*.java</include> > </includes> > <groups>TestGroup</groups> > <rerunFailingTestsCount>1</rerunFailingTestsCount> > </configuration> > </plugin>{code} > And there is a test that fails first time it's run, the resultant *txt* > report is unexpected and only contains the result of last rerun. > Suppose we have the following test class: > {code:java} > @Category(TestGroup.class) > public class CustomTest { > @Test > public void alwaysPassingTest() { > Assert.assertEquals(0, 0); > } > private static volatile int x = 0; > @Test > public synchronized void secondAttemptPassingTest() throws > InterruptedException { > Thread.sleep(5000); > if (x++ == 0) { > Assert.fail(); > } > } > } > {code} > The resultant xml report is all right, but the txt one is > {code} > ------------------------------------------------------------------------------- > Test set: CustomTest > ------------------------------------------------------------------------------- > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.063 s - in > CustomTest > {code} > Which is basically incorrect. The *txt* file is regenerated several times > during the test run. > The issue is present when groups filter (either exclusion or inclusion) is > present. When there is no group filter and only rerunFailingTestsCount is > configured, then the resultant report is the following one: > {code} > ------------------------------------------------------------------------------- > Test set: CustomTest > ------------------------------------------------------------------------------- > Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.065 s <<< > FAILURE! - in CustomTest > secondAttemptPassingTest(CustomTest) Time elapsed: 0.009 s <<< FAILURE! > java.lang.AssertionError > at CustomTest.secondAttemptPassingTest(CustomTest.java:18) > {code} > When no group filter is configured, Junit4Provider instead of JunitCore > provider is picked up and they seem to handle those cases differently. > The situation is worsened by the fact that TeamCity is not able to parse the > results correctly when the groups are configured along with > rerunFailingTestsCount and it marks the tests that passed after rerun as > failed, although the overall Maven result is SUCCESS in both cases. > The simple Maven project to reproduce the issue is attached. -- This message was sent by Atlassian JIRA (v7.6.3#76005)