Alexey Subach created SUREFIRE-1470:
---------------------------------------
Summary: Unexpected report result when using
rerunFailingTestsCount with groups filter in Surefire
Key: SUREFIRE-1470
URL: https://issues.apache.org/jira/browse/SUREFIRE-1470
Project: Maven Surefire
Issue Type: Bug
Components: Maven Surefire Plugin
Reporter: Alexey Subach
When the following plugin configuration is used:
{code:java}
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>${surefire.version}</version>
<configuration>
<includes>
<include>**/*.java</include>
</includes>
<groups>TestGroup</groups>
<rerunFailingTestsCount>1</rerunFailingTestsCount>
</configuration>
</plugin>{code}
And there is a test that fails first time it's run, the resultant *txt* report
is unexpected and only contains the result of last rerun.
Suppose we have the following test class:
{code:java}
@Category(TestGroup.class)
public class CustomTest {
@Test
public void alwaysPassingTest() {
Assert.assertEquals(0, 0);
}
private static volatile int x = 0;
@Test
public synchronized void secondAttemptPassingTest() throws
InterruptedException {
Thread.sleep(5000);
if (x++ == 0) {
Assert.fail();
}
}
}
{code}
The resultant xml report is all right, but the txt one is
{code}
-------------------------------------------------------------------------------
Test set: CustomTest
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.063 s - in
CustomTest
{code}
Which is basically incorrect. The *txt* file is regenerated several times
during the test run.
The issue is present when groups filter (either exclusion or inclusion) is
present. When there is no group filter and only rerunFailingTestsCount is
configured, then the resultant report is the following one:
{code}
-------------------------------------------------------------------------------
Test set: CustomTest
-------------------------------------------------------------------------------
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.065 s <<<
FAILURE! - in CustomTest
secondAttemptPassingTest(CustomTest) Time elapsed: 0.009 s <<< FAILURE!
java.lang.AssertionError
at CustomTest.secondAttemptPassingTest(CustomTest.java:18)
{code}
When no group filter is configured, Junit4Provider instead of JunitCore
provider is picked up and they seem to handle those cases differently.
The situation is worsened by the fact that TeamCity is not able to parse the
results correctly when the groups are configured along with
rerunFailingTestsCount and it marks the tests that passed after rerun as
failed, although the overall Maven result is SUCCESS in both cases.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
