Jeff Vincent created SUREFIRE-1336:
--------------------------------------
Summary: Sure-Fire test run totals don't match TestNG report on
failure.
Key: SUREFIRE-1336
URL: https://issues.apache.org/jira/browse/SUREFIRE-1336
Project: Maven Surefire
Issue Type: Bug
Components: TestNG support
Affects Versions: 2.19.1
Environment: Oracle JDK 1.8.0_121, Maven 3.3.9, TestNG 6.10
Reporter: Jeff Vincent
Priority: Minor
Attachments: testng-console.png, testng-reportnghtml.png
One of my TestNG test classes has 2 multi-threaded test methods that use
DataProviders to feed in unique parameter values. In total, I have a test run
that runs a total of 530,000 unique test instances between the two test
methods.
In all test output files (XML, HTML, etc.), the test totals are correct
(530,000 total, 22 failed, 529,978 passed.) The first test method runs 10,000
times, the 2nd test method runs 520,000 times. If all 520,000 pass, it reports
520,000 test cases succeed, but if any one of the iterations fail (or even
multiple failures), then it only reports the single test method as a failure
which is not accurate.
It seems that it isn't counting failures the same as it counts successes.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
