Marc Philipp created SUREFIRE-1866: -------------------------------------- Summary: Nested tests are reported ambiguously Key: SUREFIRE-1866 URL: https://issues.apache.org/jira/browse/SUREFIRE-1866 Project: Maven Surefire Issue Type: Bug Components: Maven Failsafe Plugin, Maven Surefire Plugin Affects Versions: 3.0.0-M5, 3.0.0-M4 Reporter: Marc Philipp
In 3.0.0-M4 the way nested tests and test suites are reported changed. I've prepared a small reproducer project at https://github.com/marcphilipp/gradle-sandbox/tree/master/maven-surefire-nested-test-reproducer h2. Expected behavior (with 3.0.0-M3) Run {{mvn clean test -Dmaven.surefire.version=3.0.0-M3}}. h3. Console output {code} [INFO] Running org.example.FailingTests [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.024 s <<< FAILURE! - in org.example.FailingTests [ERROR] test Time elapsed: 0.021 s <<< FAILURE! org.opentest4j.AssertionFailedError: expected: <true> but was: <false> at org.example.AbstractTests$Inner.test(AbstractTests.java:14) [INFO] Running org.example.SuccessfulTests [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 s - in org.example.SuccessfulTests {code} h3. XML Reports h4. TEST-org.example.FailingTests.xml {code:xml} <?xml version="1.0" encoding="UTF-8"?> <testsuite version="3.0" name="org.example.FailingTests" time="0.024" tests="1" errors="0" skipped="0" failures="1"> <properties> [...] </properties> <testcase name="test" classname="org.example.AbstractTests$Inner" time="0.021"> <failure message="expected: <true> but was: <false>" type="org.opentest4j.AssertionFailedError"><![CDATA[org.opentest4j.AssertionFailedError: expected: <true> but was: <false> at org.example.AbstractTests$Inner.test(AbstractTests.java:14) ]]></failure> </testcase> </testsuite> {code} h4. TEST-org.example.SuccessfulTests.xml {code:xml} <?xml version="1.0" encoding="UTF-8"?> <testsuite version="3.0" name="org.example.SuccessfulTests" time="0" tests="1" errors="0" skipped="0" failures="0"> <properties> [...] </properties> <testcase name="test" classname="org.example.AbstractTests$Inner" time="0"/> </testsuite> {code} h2. Actual behavior (with 3.0.0-M4 and 3.0.0-M5) Run {{mvn clean test}}. h3. Console output {code} [INFO] Running org.example.FailingTests [INFO] Running org.example.AbstractTests$Inner [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.037 s <<< FAILURE! - in org.example.AbstractTests$Inner [ERROR] org.example.AbstractTests$Inner.test Time elapsed: 0.02 s <<< FAILURE! org.opentest4j.AssertionFailedError: expected: <true> but was: <false> at org.example.AbstractTests$Inner.test(AbstractTests.java:14) [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.046 s - in org.example.FailingTests [INFO] Running org.example.SuccessfulTests [INFO] Running org.example.AbstractTests$Inner [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s - in org.example.AbstractTests$Inner [INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 s - in org.example.SuccessfulTests {code} h3. XML Reports h4. TEST-org.example.FailingTests.xml {code:xml} <testsuite version="3.0" name="org.example.FailingTests" time="0.042" tests="0" errors="0" skipped="0" failures="0"> <properties> [...] </properties> </testsuite> {code} h4. TEST-org.example.SuccessfulTests.xml {code:xml} <?xml version="1.0" encoding="UTF-8"?> <testsuite version="3.0" name="org.example.SuccessfulTests" time="0.005" tests="0" errors="0" skipped="0" failures="0"> <properties> [...] </properties> </testsuite> {code} h4. TEST-org.example.AbstractTests$Inner.xml {code:xml} <?xml version="1.0" encoding="UTF-8"?> <testsuite version="3.0" name="org.example.AbstractTests$Inner" time="0.002" tests="1" errors="0" skipped="0" failures="0"> <properties> [...] </properties> <testcase name="test" classname="org.example.AbstractTests$Inner" time="0.018"> <failure type="org.opentest4j.AssertionFailedError"><![CDATA[org.opentest4j.AssertionFailedError: expected: <true> but was: <false> at org.example.AbstractTests$Inner.test(AbstractTests.java:14) ]]></failure> </testcase> <testcase name="test" classname="org.example.AbstractTests$Inner" time="0.001"/> </testsuite> {code} h2. Summary The main issue is that the reporting is now ambiguous, i.e. there's no way to tell which test failed. -- This message was sent by Atlassian Jira (v8.3.4#803005)