Mark Miller created SUREFIRE-1945:
-------------------------------------
Summary: unit tests with large logging output does not produce
surefire report
Key: SUREFIRE-1945
URL: https://issues.apache.org/jira/browse/SUREFIRE-1945
Project: Maven Surefire
Issue Type: Bug
Components: Maven Surefire Plugin
Affects Versions: 3.0.0-M5, 3.0.0-M4, 3.0.0-M3, 3.0.0-M1, 3.0.0-M2, 2.22.2,
2.22.1, 2.22.0, 2.21.0, 2.20.1, 2.20
Environment: Have reproduced with both Java 8 and 11:
openjdk version "1.8.0_292"
openjdk version "11.0.11" 2021-04-20
and maven
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Reporter: Mark Miller
Attachments: surefire_bug.zip
Upon upgrading our maven-surefire-plugin from version 2.15 to version 2.21.0.
We noticed that the number of tests being run by our jenkins test job changed.
Upon further investigation we found a test class with 7 test cases that were
not being reported at all (not as passed, failed, or skipped).
When we ran just that test class from the command line it failed with
{quote}
[ERROR] Crashed tests:
[ERROR] com.sensus.dataextraction.reportcomparetool.TestFlowPreHHFReports
[ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException:
ExecutionException The forked VM terminated without properly saying goodbye. VM
crash or System.exit called?
[ERROR] Command was cmd.exe /X /C "C:\tools\java8\jre\bin\java
org.apache.maven.surefire.booter.ForkedBooter
C:\tools\cygwin\tmp\surefire7735045605128719483 2021-09-17T10-56-30_089-jvmRun1
surefire4447346130857727490tmp surefire_04489252332082939918tmp"
[ERROR] Process Exit Code: 0
[ERROR] Crashed tests:
{quote}
but did not produce any output in the target/surefire-reports directory
explaining why the failures were not reported in the jobs test report. Trying
different maven-surefire-plugin versions the tests pass successfully with
version 2.19.1, but fail with every version tried between 2.20 and 3.0.0-M5.
Note we run our tests with these options:
<reuseForks>false</reuseForks>
<forkCount>1</forkCount>
<reportFormat>xml</reportFormat>
I also noticed that the tests in question did produce a lot of logging output.
I found that the tests would pass if I added the configuration option:
<redirectTestOutputToFile>true</redirectTestOutputToFile>
I have reproduced the problem with a simpler test module that is attached.
Upon playing with the output in this module the issue appears to happen when
the output is about 1.2MB or higher.
Not sure what changed between version 2.19.1 and 2.20 with how output from the
forked jvm is collected but it appear to have a problem if there is too much
output. If the tests were being reported as failed then this issue would not
be a huge problem, but having tests just not be reported because their output
grows is a critical issue for us as we may not notice that the test has
effectively been dropped from our test suite, unless we closely audit the
~30000 test cases we run every run to detect missing tests which is hard to do
as new tests are constantly being added.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
