XN137 commented on PR #3338:
URL: https://github.com/apache/maven-surefire/pull/3338#issuecomment-4190810411
also i ran the ITs against older releases of surefire to compare current
behavior to previous versions.
(note that i had to delete the
`.assertContainsText("name=\"initializationError\"")` as it only exists on
master)
when going to 3.5.5 for junit4:
```
[INFO] Running org.apache.maven.surefire.its.JUnit4FailingAfterClassIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.402
s -- in org.apache.maven.surefire.its.JUnit4FailingAfterClassIT
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
```
but we have to change junit4 assertion to:
```
outputValidator.assertThatLogLine(containsString("testOne passed"), is(1));
```
when going to 3.5.5 for junit5:
```
[INFO] Running org.apache.maven.surefire.its.JUnit5FailingAfterAllIT
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.617
s -- in org.apache.maven.surefire.its.JUnit5FailingAfterAllIT
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
```
when going to 3.5.4 for junit4:
```
[INFO] Running junit4.AlwaysFailingAfterClassTest
testOne passed
testTwo passed
[ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
0.001 s <<< FAILURE! -- in junit4.AlwaysFailingAfterClassTest
[ERROR] junit4.AlwaysFailingAfterClassTest -- Time elapsed: 0.001 s <<<
ERROR!
java.lang.IllegalStateException: AfterClass always fails
at
junit4.AlwaysFailingAfterClassTest.tearDown(AlwaysFailingAfterClassTest.java:15)
```
when going to 3.5.4 for junit5:
```
[INFO] Running junit5.AlwaysFailingAfterAllTest
testOne passed
testTwo passed
[ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
0.003 s <<< FAILURE! -- in junit5.AlwaysFailingAfterAllTest
[ERROR] junit5.AlwaysFailingAfterAllTest -- Time elapsed: 0.003 s <<< ERROR!
java.lang.IllegalStateException: AfterAll always fails
at
junit5.AlwaysFailingAfterAllTest.tearDown(AlwaysFailingAfterAllTest.java:15)
```
so as far as i can tell 3.5.4 reported class teardown exceptions as errors
and 3.5.5 stopped doing that.
this behavior change isnt really mentioned in the release notes:
https://github.com/apache/maven-surefire/releases/tag/surefire-3.5.5
but b4a53de9b3a9ce02f306f02b064f4f58070678f4 caught my eye and i confirmed
locally that if i check out 3.5.5 and revert that changes, the reporting of
teardown issues goes back to "normal".
also only junit5 used to rerun test methods for `rerunFailingTestsCount`
when class teardown fails - junit4 did not do that but does so now on master (i
guess this is expected since master uses the common platform for execution?).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
