[
https://issues.apache.org/jira/browse/SUREFIRE-1470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16419013#comment-16419013
]
Tibor Digana commented on SUREFIRE-1470:
----------------------------------------
I was testing your project with {{mvn test -X}} and surefire-junit47 is
activated which is ok.
Our integration tests do not parser TXT file, because we parse XML report and
we assert the values in our tests.
The TXT result gives you:
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.043 s - in
CustomTest
and the console output shows exactly the same line but it is only one event
after second re-run of one failed method:
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running CustomTest
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.005 s
<<< FAILURE! - in CustomTest
[ERROR] secondAttemptPassingTest(CustomTest) Time elapsed: 0.002 s <<<
FAILURE!
java.lang.AssertionError
at CustomTest.secondAttemptPassingTest(CustomTest.java:18)
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.043 s
- in CustomTest
[INFO]
[INFO] Results:
[INFO]
[WARNING] Flakes:
[WARNING] CustomTest.secondAttemptPassingTest(CustomTest)
[ERROR] Run 1: CustomTest.secondAttemptPassingTest:18
[INFO] Run 2: PASS
[INFO]
[INFO]
[WARNING] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
So you can see the TXT handled event result of particular test class and not
the summary result of test set.
I think you see latest event in TXT and not the same like it is in XML.
So yes, you are right the TXT should be fixed.
Maybe as a hint, you may want to debug the class {{DefaultReporterFactory}} to
get experiences but not sure if this class is buggy. Here is re-run events
processed.
> Unexpected report result when using rerunFailingTestsCount with groups filter
> in Surefire
> -----------------------------------------------------------------------------------------
>
> Key: SUREFIRE-1470
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1470
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.x support, Maven Surefire Plugin
> Reporter: Alexey Subach
> Priority: Major
> Attachments: test.zip
>
>
> When the following plugin configuration is used:
> {code:java}
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> <version>${surefire.version}</version>
> <configuration>
> <includes>
> <include>**/*.java</include>
> </includes>
> <groups>TestGroup</groups>
> <rerunFailingTestsCount>1</rerunFailingTestsCount>
> </configuration>
> </plugin>{code}
> And there is a test that fails first time it's run, the resultant *txt*
> report is unexpected and only contains the result of last rerun.
> Suppose we have the following test class:
> {code:java}
> @Category(TestGroup.class)
> public class CustomTest {
> @Test
> public void alwaysPassingTest() {
> Assert.assertEquals(0, 0);
> }
> private static volatile int x = 0;
> @Test
> public synchronized void secondAttemptPassingTest() throws
> InterruptedException {
> Thread.sleep(5000);
> if (x++ == 0) {
> Assert.fail();
> }
> }
> }
> {code}
> The resultant xml report is all right, but the txt one is
> {code}
> -------------------------------------------------------------------------------
> Test set: CustomTest
> -------------------------------------------------------------------------------
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.063 s - in
> CustomTest
> {code}
> Which is basically incorrect. The *txt* file is regenerated several times
> during the test run.
> The issue is present when groups filter (either exclusion or inclusion) is
> present. When there is no group filter and only rerunFailingTestsCount is
> configured, then the resultant report is the following one:
> {code}
> -------------------------------------------------------------------------------
> Test set: CustomTest
> -------------------------------------------------------------------------------
> Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.065 s <<<
> FAILURE! - in CustomTest
> secondAttemptPassingTest(CustomTest) Time elapsed: 0.009 s <<< FAILURE!
> java.lang.AssertionError
> at CustomTest.secondAttemptPassingTest(CustomTest.java:18)
> {code}
> When no group filter is configured, Junit4Provider instead of JunitCore
> provider is picked up and they seem to handle those cases differently.
> The situation is worsened by the fact that TeamCity is not able to parse the
> results correctly when the groups are configured along with
> rerunFailingTestsCount and it marks the tests that passed after rerun as
> failed, although the overall Maven result is SUCCESS in both cases.
> The simple Maven project to reproduce the issue is attached.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
