[
https://issues.apache.org/jira/browse/SUREFIRE-1470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16427043#comment-16427043
]
Alexey Subach commented on SUREFIRE-1470:
-----------------------------------------
[~tibor17] Hi again! I know you are busy with other stuff, but maybe you'll
find time to answer this. Actually the stuff I referred to about TXT is also
relevant for XML. And the XML reports are different depending on running with
or without groups.
I have two questions.
1) Is it expected that when both `groups` and `rerunFailingTestsCount` are set,
the resultant XML report gets re-generated and overwritten?
To be more verbose, on the first attempt to run the test, the resultant report
contains this:
{code}
<testcase name="secondAttemptPassingTest" classname="CustomTest" time="9.991">
<failure type="java.lang.AssertionError">java.lang.AssertionError
at CustomTest.secondAttemptPassingTest(CustomTest.java:19)
</failure>
</testcase>
{code}
which is eventually replaced by this:
{code}
<testcase name="secondAttemptPassingTest" classname="CustomTest" time="9.96">
<flakyFailure type="java.lang.AssertionError">java.lang.AssertionError
at CustomTest.secondAttemptPassingTest(CustomTest.java:19)
</flakyFailure>
</testcase>
{code}
i.e. <failure> gets replaced by <flakyFailure>.
2) For the attached project, after running with <groups>, I get the following
attributes of the <testsuite> element: `tests="1" errors="0" skipped="0"
failures="0"`
However, when no groups are defined, the attributes are `tests="3" errors="0"
skipped="0" failures="1"`.
The inner content of the <testsuite> is identical.
So which one is correct? Or neither? Please note that I actually have **two**
tests:
{code}
<testcase name="secondAttemptPassingTest" classname="CustomTest" time="10">
<flakyFailure type="java.lang.AssertionError">java.lang.AssertionError
at CustomTest.secondAttemptPassingTest(CustomTest.java:19)
</flakyFailure>
</testcase>
<testcase name="alwaysPassingTest" classname="CustomTest" time="0"/>
{code}
XML is not like TXT and should report the overall statistics, right? But it has
all the problems I reported above. Does it raise the priority of the issue?
> Unexpected report result when using rerunFailingTestsCount with groups filter
> in Surefire
> -----------------------------------------------------------------------------------------
>
> Key: SUREFIRE-1470
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1470
> Project: Maven Surefire
> Issue Type: Bug
> Components: Junit 4.x support, Maven Surefire Plugin
> Reporter: Alexey Subach
> Priority: Major
> Attachments: test.zip
>
>
> When the following plugin configuration is used:
> {code:java}
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> <version>${surefire.version}</version>
> <configuration>
> <includes>
> <include>**/*.java</include>
> </includes>
> <groups>TestGroup</groups>
> <rerunFailingTestsCount>1</rerunFailingTestsCount>
> </configuration>
> </plugin>{code}
> And there is a test that fails first time it's run, the resultant *txt*
> report is unexpected and only contains the result of last rerun.
> Suppose we have the following test class:
> {code:java}
> @Category(TestGroup.class)
> public class CustomTest {
> @Test
> public void alwaysPassingTest() {
> Assert.assertEquals(0, 0);
> }
> private static volatile int x = 0;
> @Test
> public synchronized void secondAttemptPassingTest() throws
> InterruptedException {
> Thread.sleep(5000);
> if (x++ == 0) {
> Assert.fail();
> }
> }
> }
> {code}
> The resultant xml report is all right, but the txt one is
> {code}
> -------------------------------------------------------------------------------
> Test set: CustomTest
> -------------------------------------------------------------------------------
> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.063 s - in
> CustomTest
> {code}
> Which is basically incorrect. The *txt* file is regenerated several times
> during the test run.
> The issue is present when groups filter (either exclusion or inclusion) is
> present. When there is no group filter and only rerunFailingTestsCount is
> configured, then the resultant report is the following one:
> {code}
> -------------------------------------------------------------------------------
> Test set: CustomTest
> -------------------------------------------------------------------------------
> Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.065 s <<<
> FAILURE! - in CustomTest
> secondAttemptPassingTest(CustomTest) Time elapsed: 0.009 s <<< FAILURE!
> java.lang.AssertionError
> at CustomTest.secondAttemptPassingTest(CustomTest.java:18)
> {code}
> When no group filter is configured, Junit4Provider instead of JunitCore
> provider is picked up and they seem to handle those cases differently.
> The situation is worsened by the fact that TeamCity is not able to parse the
> results correctly when the groups are configured along with
> rerunFailingTestsCount and it marks the tests that passed after rerun as
> failed, although the overall Maven result is SUCCESS in both cases.
> The simple Maven project to reproduce the issue is attached.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
