nkeywal created SUREFIRE-813:
--------------------------------
Summary: Randomly wrong tests count and empty summary files with
JUnitProvider47 when forking the tests.
Key: SUREFIRE-813
URL: https://jira.codehaus.org/browse/SUREFIRE-813
Project: Maven Surefire
Issue Type: Bug
Components: Junit 4.7+ (parallel) support
Affects Versions: 2.11, 2.10, 2.12
Environment: test case checked on windows 7 64 bits; issues on Linux.
jdk1.6.0_24
Reporter: nkeywal
Attachments: pom.xml, Test6.java
When running the test suite, with forkMode set and junit provider 47, we can
have some test results missing. The test class is executed, but:
- the summary file is empty (exists, but as a zero length file)
- the global test count is not increased.
It's random, and does not occurs all the time. The occurrence seems increased
when the machine is loaded, but it's more an impression. In our case, with ~250
test classes, it occurs on average once per run (so it's a real problem). It
seems to occur more if we activate 'perthread' parallelization.
I added some log lines, and it seems that ForkClient does not receive all the
test events when the problem occurs. It receives:
- ForkingRunListener.BOOTERCODE_TESTSET_STARTING
- ForkingRunListener.BOOTERCODE_TEST_STARTING
But not the final:
- ForkingRunListener.BOOTERCODE_TEST_SUCCEEDED
- ForkingRunListener.BOOTERCODE_TESTSET_COMPLETED
The original issue is on Linux, java 1.6, with a lot of tests running. I wrote
a test case, hopefully reproducing the real issue with surefire 2.10 and 2.11.
It's still random, but fails 50% of the time on my machine (laptop 2 cores,
Windows 7, 64 bits, jdk1.6.0_24).
In this example, we create a thread writing 1000 lines of
'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA" in stdout in the tear down method.
That's an example of success with surefire 2.11. You see that exactly two lines
of 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' has been written to the console. On some
other successes it can be zero lines or more.
{noformat}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Concurrency config is parallel='classes', perCoreThreadCount=true,
threadCount=2, useUnlimitedThreads=false
Running Test6
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec
658:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
659:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.724s
[INFO] Finished at: Tue Dec 27 11:46:03 CET 2011
[INFO] Final Memory: 10M/109M
[INFO] ------------------------------------------------------------------------
{noformat}
Error case, with surefire 2.11. In this one there is no
'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' in the output, but you can have errors with
multiple such lines.
You see that the number of tests run is zero.
{noformat}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Concurrency config is parallel='classes', perCoreThreadCount=true,
threadCount=2, useUnlimitedThreads=false
Running Test6
Results :
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.820s
[INFO] Finished at: Tue Dec 27 11:48:02 CET 2011
[INFO] Final Memory: 10M/109M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.11:test (default-test) on
project surefireparametizedbug: No tests were executed! (Set
-DfailIfNoTests=false to ignore this error.) -> [Help 1]
[ERROR]
{noformat}
With junit4 instead of junit47 it seems it never fails, but you can still have
some AAA lines in the final output:
{noformat}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running Test6
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.07 sec
18:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
19:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
20:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
21:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
22:AAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.667s
[INFO] Finished at: Tue Dec 27 11:54:16 CET 2011
[INFO] Final Memory: 10M/114M
[INFO] ------------------------------------------------------------------------
{noformat}
It can fail as well (but less often it seems) if redirectTestOutputToFile is
set to false.
It fails as well with forkMode=once (on surefire 2.11 at least).
parallel is set to "classes"; but if fails as well if it's not set (still on
surefire 2.11 at least).
Note that I write 1000 lines in the test case, it can fail with much less as
well. It's just a simplification of the initially complex issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
