[ https://issues.apache.org/jira/browse/SUREFIRE-1470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16418905#comment-16418905 ]
Alexey Subach commented on SUREFIRE-1470: ----------------------------------------- Before trying to fix the issue I wanted to get feedback from your side, which I had to wait for for quite long time :) The "correct" result also seems a but strange - if test passes on second run, why the total number of tests run is reported to be 3? Regarding using surefire-junit47 provider, how do I set it? If I follow the guidance at http://maven.apache.org/surefire/maven-surefire-plugin/examples/providers.html and set it like this: {code} <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-surefire-plugin</artifactId> <version>${surefire.version}</version> <configuration> <includes> <include>**/*.java</include> </includes> <groups>TestGroup</groups> <rerunFailingTestsCount>1</rerunFailingTestsCount> </configuration> <dependencies> <dependency> <groupId>org.apache.maven.surefire</groupId> <artifactId>surefire-junit47</artifactId> <version>${surefire.version}</version> </dependency> </dependencies> </plugin> </plugins> </build> {code} The resultant TXT report is still incorrect and the same as it was without explicit dependency specification. Am I doing something incorrectly, or is it another bug? > Unexpected report result when using rerunFailingTestsCount with groups filter > in Surefire > ----------------------------------------------------------------------------------------- > > Key: SUREFIRE-1470 > URL: https://issues.apache.org/jira/browse/SUREFIRE-1470 > Project: Maven Surefire > Issue Type: Bug > Components: Junit 4.x support, Maven Surefire Plugin > Reporter: Alexey Subach > Priority: Major > Attachments: test.zip > > > When the following plugin configuration is used: > {code:java} > <plugin> > <groupId>org.apache.maven.plugins</groupId> > <artifactId>maven-surefire-plugin</artifactId> > <version>${surefire.version}</version> > <configuration> > <includes> > <include>**/*.java</include> > </includes> > <groups>TestGroup</groups> > <rerunFailingTestsCount>1</rerunFailingTestsCount> > </configuration> > </plugin>{code} > And there is a test that fails first time it's run, the resultant *txt* > report is unexpected and only contains the result of last rerun. > Suppose we have the following test class: > {code:java} > @Category(TestGroup.class) > public class CustomTest { > @Test > public void alwaysPassingTest() { > Assert.assertEquals(0, 0); > } > private static volatile int x = 0; > @Test > public synchronized void secondAttemptPassingTest() throws > InterruptedException { > Thread.sleep(5000); > if (x++ == 0) { > Assert.fail(); > } > } > } > {code} > The resultant xml report is all right, but the txt one is > {code} > ------------------------------------------------------------------------------- > Test set: CustomTest > ------------------------------------------------------------------------------- > Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.063 s - in > CustomTest > {code} > Which is basically incorrect. The *txt* file is regenerated several times > during the test run. > The issue is present when groups filter (either exclusion or inclusion) is > present. When there is no group filter and only rerunFailingTestsCount is > configured, then the resultant report is the following one: > {code} > ------------------------------------------------------------------------------- > Test set: CustomTest > ------------------------------------------------------------------------------- > Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.065 s <<< > FAILURE! - in CustomTest > secondAttemptPassingTest(CustomTest) Time elapsed: 0.009 s <<< FAILURE! > java.lang.AssertionError > at CustomTest.secondAttemptPassingTest(CustomTest.java:18) > {code} > When no group filter is configured, Junit4Provider instead of JunitCore > provider is picked up and they seem to handle those cases differently. > The situation is worsened by the fact that TeamCity is not able to parse the > results correctly when the groups are configured along with > rerunFailingTestsCount and it marks the tests that passed after rerun as > failed, although the overall Maven result is SUCCESS in both cases. > The simple Maven project to reproduce the issue is attached. -- This message was sent by Atlassian JIRA (v7.6.3#76005)