surefire exit code should explicitly distinguish between crashed and failed
unit test(s)
----------------------------------------------------------------------------------------
Key: SUREFIRE-611
URL: http://jira.codehaus.org/browse/SUREFIRE-611
Project: Maven Surefire
Issue Type: Improvement
Reporter: Tycho Lamerigts
Attachments: tc-junit-crash.tar.gz
- If a unit test crashes, surefire skips all remaining tests, prints a message
"[ERROR] There are test failures. ..." to the console, and does NOT print a
test summary report.
- If one or more unit tests fail, maven completes remaining tests, prints a
message "[ERROR] There are test failures. ..." to the console, and DOES print
a test summary report.
The only distinction between crash and fail is the presence (or absence) of a
summary report.
This is a problem for some tools (like TeamCity), because they typically don't
pick up on the fact that many testcases were not run at all in case of a
crashed unit test. Instead in case of a crashed test they simply report, for
example, 100 tests succeeded, 0 failed. They do not report "50 tests skipped
due to a crash after test #100", and it is quite hard for those tools to
recognize this situation. If surefire would report a crash more explicitly,
other tools can act accordingly.
I have attached a small maven project to reproduce the problem. By commenting
and uncommenting the marked line in file ATest.java, you can reproduce both
fail and crash scenarios. In the crash scenario, Maven will correctly report a
failed build (unless you use the -Dmaven.test.failure.ignore flag, in which
case it even won't do that), but it fails to indicate there was a crash which
caused tests (in this case 3) to be skipped.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://jira.codehaus.org/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
