[ 
https://issues.apache.org/jira/browse/SUREFIRE-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037304#comment-17037304
 ] 

Tibor Digana commented on SUREFIRE-1751:
----------------------------------------

[~anujose]
Thank you for the link.
I meant something else. Normally we want the users to make the fix. So 
therefore I asked you whether you wanted to help us making the fix.
The developers open a pullrequest on https://github.com/apache/maven-surefire 
and the fix is usually discussed and reviewed.
No worries if you are not convinced about the fix. We will help you with the 
work.
I am almost totally convinced that the bug is in 
[TestSuiteXmlParser|https://github.com/apache/maven-surefire/blob/master/surefire-report-parser/src/main/java/org/apache/maven/plugins/surefire/report/TestSuiteXmlParser.java]
It would be fine for you to write a new test method in 
[TestSuiteXmlParserTest|https://github.com/apache/maven-surefire/blob/master/surefire-report-parser/src/test/java/org/apache/maven/plugins/surefire/report/TestSuiteXmlParserTest.java]
 and you can add you XML report in {{src/test/resources}} - just see the test 
method *successfulSurefireTestReport*.
Your test will prove the bug is in 
[here|https://github.com/apache/maven-surefire/blob/master/surefire-report-parser/src/main/java/org/apache/maven/plugins/surefire/report/TestSuiteXmlParser.java#L122]
 because the result of the testcase cannot be determined by parsing case by 
case in the switch. It should real all cases and then the result should go to a 
missing logic which specifically looks for flakes. This logic does not exist 
and this should be the fix.

> Surefire report shows flaky tests as failures
> ---------------------------------------------
>
>                 Key: SUREFIRE-1751
>                 URL: https://issues.apache.org/jira/browse/SUREFIRE-1751
>             Project: Maven Surefire
>          Issue Type: Bug
>    Affects Versions: 3.0.0-M4
>            Reporter: Anu
>            Priority: Critical
>         Attachments: Test.zip, image-2020-02-13-11-41-03-906.png
>
>
> We have been using maven-surefire-plugin in 2.22.1 and Junit4.13 for our 
> testing. Now we are trying to upgrade junit to junit5 and saw that the 
> r{color:#6a8759}erunFailingTestsCount parameter isn't working with surefire 
> plugin version. I upgraded surefire to 3.0.0.M4 and verified that the failing 
> tests are now being retried. But the issue is flaky tests show up in the 
> report as Failures.
> {color}
>  
> {color:#6a8759}ie. the log shows
> {color}
> [WARNING] Tests run: 10455, Failures: 0, Errors: 0, Skipped: 993, Flakes: 10
>  
> But the actual surefire report shows like below
> !image-2020-02-13-11-41-03-906.png!
>  
> This is blocking our junit5 upgrade task. Can you please provide a fix for 
> this at the earliest? Please let me know if you need anything else from me to 
> help with the investigation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to