[ 
https://issues.apache.org/jira/browse/SUREFIRE-1470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexey Subach updated SUREFIRE-1470:
------------------------------------
    Attachment: test.zip

> Unexpected report result when using rerunFailingTestsCount with groups filter 
> in Surefire
> -----------------------------------------------------------------------------------------
>
>                 Key: SUREFIRE-1470
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1470
>             Project: Maven Surefire
>          Issue Type: Bug
>          Components: Maven Surefire Plugin
>            Reporter: Alexey Subach
>            Priority: Critical
>         Attachments: test.zip
>
>
> When the following plugin configuration is used:
> {code:java}
> <plugin>
>     <groupId>org.apache.maven.plugins</groupId>
>     <artifactId>maven-surefire-plugin</artifactId>
>     <version>${surefire.version}</version>
>     <configuration>
>         <includes>
>             <include>**/*.java</include>
>         </includes>
>         <groups>TestGroup</groups>
>         <rerunFailingTestsCount>1</rerunFailingTestsCount>
>     </configuration>
> </plugin>{code}
> And there is a test that fails first time it's run, the resultant *txt* 
> report is unexpected and only contains the result of last rerun.
> Suppose we have the following test class:
> {code:java}
> @Category(TestGroup.class)
> public class CustomTest {
>     @Test
>     public void alwaysPassingTest() {
>         Assert.assertEquals(0, 0);
>     }
>     private static volatile int x = 0;
>     @Test
>     public synchronized void secondAttemptPassingTest() throws 
> InterruptedException {
>         Thread.sleep(5000);
>         if (x++ == 0) {
>             Assert.fail();
>         }
>     }
> }
> {code}
> The resultant xml report is all right, but the txt one is
> {code}
> -------------------------------------------------------------------------------
> Test set: CustomTest
> -------------------------------------------------------------------------------
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.063 s - in 
> CustomTest
> {code}
> Which is basically incorrect. The *txt* file is regenerated several times 
> during the test run.
> The issue is present when groups filter (either exclusion or inclusion) is 
> present. When there is no group filter and only rerunFailingTestsCount is 
> configured, then the resultant report is the following one:
> {code}
> -------------------------------------------------------------------------------
> Test set: CustomTest
> -------------------------------------------------------------------------------
> Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.065 s <<< 
> FAILURE! - in CustomTest
> secondAttemptPassingTest(CustomTest)  Time elapsed: 0.009 s  <<< FAILURE!
> java.lang.AssertionError
>       at CustomTest.secondAttemptPassingTest(CustomTest.java:18)
> {code}
> When no group filter is configured, Junit4Provider instead of JunitCore 
> provider is picked up and they seem to handle those cases differently.
> The situation is worsened by the fact that TeamCity is not able to parse the 
> results correctly when the groups are configured along with 
> rerunFailingTestsCount and it marks the tests that passed after rerun as 
> failed, although the overall Maven result is SUCCESS in both cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Reply via email to