[jira] [Commented] (SUREFIRE-1751) Surefire report shows flaky tests as failures
[
https://issues.apache.org/jira/browse/SUREFIRE-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17051337#comment-17051337
]
Anu commented on SUREFIRE-1751:
---
I was out sick for two weeks. Will try this again today and keep you posted.
> Surefire report shows flaky tests as failures
> -
>
> Key: SUREFIRE-1751
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1751
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M4
>Reporter: Anu
>Priority: Critical
> Attachments: TEST-com.test.Flakes.xml, Test.zip,
> image-2020-02-13-11-41-03-906.png
>
>
> We have been using maven-surefire-plugin in 2.22.1 and Junit4.13 for our
> testing. Now we are trying to upgrade junit to junit5 and saw that the
> r{color:#6a8759}erunFailingTestsCount parameter isn't working with surefire
> plugin version. I upgraded surefire to 3.0.0.M4 and verified that the failing
> tests are now being retried. But the issue is flaky tests show up in the
> report as Failures.
> {color}
>
> {color:#6a8759}ie. the log shows
> {color}
> [WARNING] Tests run: 10455, Failures: 0, Errors: 0, Skipped: 993, Flakes: 10
>
> But the actual surefire report shows like below
> !image-2020-02-13-11-41-03-906.png!
>
> This is blocking our junit5 upgrade task. Can you please provide a fix for
> this at the earliest? Please let me know if you need anything else from me to
> help with the investigation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1751) Surefire report shows flaky tests as failures
[
https://issues.apache.org/jira/browse/SUREFIRE-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038718#comment-17038718
]
Tibor Digana commented on SUREFIRE-1751:
In your real test you have got:
Errors: 1, Failures: 7
Where these numbers come from? They should be zeros and they were the post
results of flakes?
So the XML provided looks too simple to break this test with
TestSuiteXmlParser. The XML should be more realistic in order to provocate the
bug as we can see it in the HTML report.
> Surefire report shows flaky tests as failures
> -
>
> Key: SUREFIRE-1751
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1751
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M4
>Reporter: Anu
>Priority: Critical
> Attachments: TEST-com.test.Flakes.xml, Test.zip,
> image-2020-02-13-11-41-03-906.png
>
>
> We have been using maven-surefire-plugin in 2.22.1 and Junit4.13 for our
> testing. Now we are trying to upgrade junit to junit5 and saw that the
> r{color:#6a8759}erunFailingTestsCount parameter isn't working with surefire
> plugin version. I upgraded surefire to 3.0.0.M4 and verified that the failing
> tests are now being retried. But the issue is flaky tests show up in the
> report as Failures.
> {color}
>
> {color:#6a8759}ie. the log shows
> {color}
> [WARNING] Tests run: 10455, Failures: 0, Errors: 0, Skipped: 993, Flakes: 10
>
> But the actual surefire report shows like below
> !image-2020-02-13-11-41-03-906.png!
>
> This is blocking our junit5 upgrade task. Can you please provide a fix for
> this at the earliest? Please let me know if you need anything else from me to
> help with the investigation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1751) Surefire report shows flaky tests as failures
[
https://issues.apache.org/jira/browse/SUREFIRE-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17038582#comment-17038582
]
Anu commented on SUREFIRE-1751:
---
Thanks [~tibordigana]. I tried creating a test method in the class like below
and ran the test with [^TEST-com.test.Flakes.xml] . The test passed (showed
correct number of tests, failures, flakes etc)
{code:java}
@Test
public void successfulSurefireTestReportWithFlakes() throws Exception {
TestSuiteXmlParser parser = new TestSuiteXmlParser( consoleLogger );
File surefireReport = new File(
"src/test/resources/junit-pathWithÜmlaut/TEST-com.test.Flakes.xml" );
assumeTrue( surefireReport.isFile() );
Collection suites = parser.parse(
surefireReport.getCanonicalPath() );
assertNotNull( suites );
assertEquals( 1, suites.size() );
ReportTestSuite suite = suites.iterator().next();
assertThat( suite.getNumberOfTests(), is( 1 ) );
assertEquals( 1, suite.getNumberOfTests() );
assertEquals( 1, suite.getNumberOfFlakes() );
assertEquals( 0, suite.getNumberOfFailures() );
assertEquals( 0, suite.getNumberOfErrors() );
assertEquals( 0, suite.getNumberOfSkipped() );
assertThat( suite.getFullClassName(), is( "com.test.Flakes" ) );
assertThat( suite.getPackageName(), is( "com.test" ) );
assertThat( suite.getName(), is( "Flakes" ) );
ReportTestCase test = suite.getTestCases().iterator().next();
assertTrue( test.isSuccessful() );
assertNull( test.getFailureDetail() );
assertNull( test.getFailureErrorLine() );
assertNull( test.getFailureType() );
assertThat( test.getFullClassName(), is( "com.test.Flakes" ) );
assertThat( test.getClassName(), is( "Flakes" ) );
assertThat( test.getName(), is( "run" ) );
assertThat( test.getFullName(), is( "com.test.Flakes.run" ) );
} {code}
I am new to the github world, so I am not sure how to commit this test to the
repository.
can you please help on how to proceed?
> Surefire report shows flaky tests as failures
> -
>
> Key: SUREFIRE-1751
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1751
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M4
>Reporter: Anu
>Priority: Critical
> Attachments: TEST-com.test.Flakes.xml, Test.zip,
> image-2020-02-13-11-41-03-906.png
>
>
> We have been using maven-surefire-plugin in 2.22.1 and Junit4.13 for our
> testing. Now we are trying to upgrade junit to junit5 and saw that the
> r{color:#6a8759}erunFailingTestsCount parameter isn't working with surefire
> plugin version. I upgraded surefire to 3.0.0.M4 and verified that the failing
> tests are now being retried. But the issue is flaky tests show up in the
> report as Failures.
> {color}
>
> {color:#6a8759}ie. the log shows
> {color}
> [WARNING] Tests run: 10455, Failures: 0, Errors: 0, Skipped: 993, Flakes: 10
>
> But the actual surefire report shows like below
> !image-2020-02-13-11-41-03-906.png!
>
> This is blocking our junit5 upgrade task. Can you please provide a fix for
> this at the earliest? Please let me know if you need anything else from me to
> help with the investigation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1751) Surefire report shows flaky tests as failures
[
https://issues.apache.org/jira/browse/SUREFIRE-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17037304#comment-17037304
]
Tibor Digana commented on SUREFIRE-1751:
[~anujose]
Thank you for the link.
I meant something else. Normally we want the users to make the fix. So
therefore I asked you whether you wanted to help us making the fix.
The developers open a pullrequest on https://github.com/apache/maven-surefire
and the fix is usually discussed and reviewed.
No worries if you are not convinced about the fix. We will help you with the
work.
I am almost totally convinced that the bug is in
[TestSuiteXmlParser|https://github.com/apache/maven-surefire/blob/master/surefire-report-parser/src/main/java/org/apache/maven/plugins/surefire/report/TestSuiteXmlParser.java]
It would be fine for you to write a new test method in
[TestSuiteXmlParserTest|https://github.com/apache/maven-surefire/blob/master/surefire-report-parser/src/test/java/org/apache/maven/plugins/surefire/report/TestSuiteXmlParserTest.java]
and you can add you XML report in {{src/test/resources}} - just see the test
method *successfulSurefireTestReport*.
Your test will prove the bug is in
[here|https://github.com/apache/maven-surefire/blob/master/surefire-report-parser/src/main/java/org/apache/maven/plugins/surefire/report/TestSuiteXmlParser.java#L122]
because the result of the testcase cannot be determined by parsing case by
case in the switch. It should real all cases and then the result should go to a
missing logic which specifically looks for flakes. This logic does not exist
and this should be the fix.
> Surefire report shows flaky tests as failures
> -
>
> Key: SUREFIRE-1751
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1751
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M4
>Reporter: Anu
>Priority: Critical
> Attachments: Test.zip, image-2020-02-13-11-41-03-906.png
>
>
> We have been using maven-surefire-plugin in 2.22.1 and Junit4.13 for our
> testing. Now we are trying to upgrade junit to junit5 and saw that the
> r{color:#6a8759}erunFailingTestsCount parameter isn't working with surefire
> plugin version. I upgraded surefire to 3.0.0.M4 and verified that the failing
> tests are now being retried. But the issue is flaky tests show up in the
> report as Failures.
> {color}
>
> {color:#6a8759}ie. the log shows
> {color}
> [WARNING] Tests run: 10455, Failures: 0, Errors: 0, Skipped: 993, Flakes: 10
>
> But the actual surefire report shows like below
> !image-2020-02-13-11-41-03-906.png!
>
> This is blocking our junit5 upgrade task. Can you please provide a fix for
> this at the earliest? Please let me know if you need anything else from me to
> help with the investigation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1751) Surefire report shows flaky tests as failures
[
https://issues.apache.org/jira/browse/SUREFIRE-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17037195#comment-17037195
]
Anu commented on SUREFIRE-1751:
---
I forgot to take out surefire-report-parse from the pom before sending it you.
And sorry. I misunderstood.This is my first time creating a ticket with third
party. I added the code to github like below. Is that what you were looking for?
[https://github.com/anujose-c/AnuTest]
> Surefire report shows flaky tests as failures
> -
>
> Key: SUREFIRE-1751
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1751
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M4
>Reporter: Anu
>Priority: Critical
> Attachments: Test.zip, image-2020-02-13-11-41-03-906.png
>
>
> We have been using maven-surefire-plugin in 2.22.1 and Junit4.13 for our
> testing. Now we are trying to upgrade junit to junit5 and saw that the
> r{color:#6a8759}erunFailingTestsCount parameter isn't working with surefire
> plugin version. I upgraded surefire to 3.0.0.M4 and verified that the failing
> tests are now being retried. But the issue is flaky tests show up in the
> report as Failures.
> {color}
>
> {color:#6a8759}ie. the log shows
> {color}
> [WARNING] Tests run: 10455, Failures: 0, Errors: 0, Skipped: 993, Flakes: 10
>
> But the actual surefire report shows like below
> !image-2020-02-13-11-41-03-906.png!
>
> This is blocking our junit5 upgrade task. Can you please provide a fix for
> this at the earliest? Please let me know if you need anything else from me to
> help with the investigation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1751) Surefire report shows flaky tests as failures
[
https://issues.apache.org/jira/browse/SUREFIRE-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036616#comment-17036616
]
Tibor Digana commented on SUREFIRE-1751:
Why you used {{surefire-report-parser}} in you POM?
I meant a contribution on Github ;-) So i can guide you through the fix in our
code.
> Surefire report shows flaky tests as failures
> -
>
> Key: SUREFIRE-1751
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1751
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M4
>Reporter: Anu
>Priority: Critical
> Attachments: Test.zip, image-2020-02-13-11-41-03-906.png
>
>
> We have been using maven-surefire-plugin in 2.22.1 and Junit4.13 for our
> testing. Now we are trying to upgrade junit to junit5 and saw that the
> r{color:#6a8759}erunFailingTestsCount parameter isn't working with surefire
> plugin version. I upgraded surefire to 3.0.0.M4 and verified that the failing
> tests are now being retried. But the issue is flaky tests show up in the
> report as Failures.
> {color}
>
> {color:#6a8759}ie. the log shows
> {color}
> [WARNING] Tests run: 10455, Failures: 0, Errors: 0, Skipped: 993, Flakes: 10
>
> But the actual surefire report shows like below
> !image-2020-02-13-11-41-03-906.png!
>
> This is blocking our junit5 upgrade task. Can you please provide a fix for
> this at the earliest? Please let me know if you need anything else from me to
> help with the investigation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1751) Surefire report shows flaky tests as failures
[
https://issues.apache.org/jira/browse/SUREFIRE-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036573#comment-17036573
]
Anu commented on SUREFIRE-1751:
---
Thanks [~tibordigana] for the response. I have attached Test.zip with the
ticket. Can you check if that helps?
The command I used to run the tests was
{code:java}
mvn surefire-report:report -Dsurefire.rerunFailingTestsCount=2
-Dtest=Flakes,Pass{code}
> Surefire report shows flaky tests as failures
> -
>
> Key: SUREFIRE-1751
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1751
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M4
>Reporter: Anu
>Priority: Critical
> Attachments: Test.zip, image-2020-02-13-11-41-03-906.png
>
>
> We have been using maven-surefire-plugin in 2.22.1 and Junit4.13 for our
> testing. Now we are trying to upgrade junit to junit5 and saw that the
> r{color:#6a8759}erunFailingTestsCount parameter isn't working with surefire
> plugin version. I upgraded surefire to 3.0.0.M4 and verified that the failing
> tests are now being retried. But the issue is flaky tests show up in the
> report as Failures.
> {color}
>
> {color:#6a8759}ie. the log shows
> {color}
> [WARNING] Tests run: 10455, Failures: 0, Errors: 0, Skipped: 993, Flakes: 10
>
> But the actual surefire report shows like below
> !image-2020-02-13-11-41-03-906.png!
>
> This is blocking our junit5 upgrade task. Can you please provide a fix for
> this at the earliest? Please let me know if you need anything else from me to
> help with the investigation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1751) Surefire report shows flaky tests as failures
[
https://issues.apache.org/jira/browse/SUREFIRE-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036515#comment-17036515
]
Tibor Digana commented on SUREFIRE-1751:
Alternatively, use your XML report, write a unit test passing that XML report
to the {{SurefireReportParser}} and build the test on expected assertions
statements. Call the method {{SurefireReportParser.parseXMLReportFiles()}} and
you should see the same result in the chart.
> Surefire report shows flaky tests as failures
> -
>
> Key: SUREFIRE-1751
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1751
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M4
>Reporter: Anu
>Priority: Critical
> Attachments: image-2020-02-13-11-41-03-906.png
>
>
> We have been using maven-surefire-plugin in 2.22.1 and Junit4.13 for our
> testing. Now we are trying to upgrade junit to junit5 and saw that the
> r{color:#6a8759}erunFailingTestsCount parameter isn't working with surefire
> plugin version. I upgraded surefire to 3.0.0.M4 and verified that the failing
> tests are now being retried. But the issue is flaky tests show up in the
> report as Failures.
> {color}
>
> {color:#6a8759}ie. the log shows
> {color}
> [WARNING] Tests run: 10455, Failures: 0, Errors: 0, Skipped: 993, Flakes: 10
>
> But the actual surefire report shows like below
> !image-2020-02-13-11-41-03-906.png!
>
> This is blocking our junit5 upgrade task. Can you please provide a fix for
> this at the earliest? Please let me know if you need anything else from me to
> help with the investigation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (SUREFIRE-1751) Surefire report shows flaky tests as failures
[
https://issues.apache.org/jira/browse/SUREFIRE-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17036498#comment-17036498
]
Tibor Digana commented on SUREFIRE-1751:
[~anujose]
Can you attach a minimum project with necessary sources in a zip file?
> Surefire report shows flaky tests as failures
> -
>
> Key: SUREFIRE-1751
> URL: https://issues.apache.org/jira/browse/SUREFIRE-1751
> Project: Maven Surefire
> Issue Type: Bug
>Affects Versions: 3.0.0-M4
>Reporter: Anu
>Priority: Critical
> Attachments: image-2020-02-13-11-41-03-906.png
>
>
> We have been using maven-surefire-plugin in 2.22.1 and Junit4.13 for our
> testing. Now we are trying to upgrade junit to junit5 and saw that the
> r{color:#6a8759}erunFailingTestsCount parameter isn't working with surefire
> plugin version. I upgraded surefire to 3.0.0.M4 and verified that the failing
> tests are now being retried. But the issue is flaky tests show up in the
> report as Failures.
> {color}
>
> {color:#6a8759}ie. the log shows
> {color}
> [WARNING] Tests run: 10455, Failures: 0, Errors: 0, Skipped: 993, Flakes: 10
>
> But the actual surefire report shows like below
> !image-2020-02-13-11-41-03-906.png!
>
> This is blocking our junit5 upgrade task. Can you please provide a fix for
> this at the earliest? Please let me know if you need anything else from me to
> help with the investigation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
