[
https://issues.apache.org/jira/browse/LUCENE-3785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13208826#comment-13208826
]
Dawid Weiss commented on LUCENE-3785:
-------------------------------------
An example improvement over trunk. I've designed a "super-failing" set of
classes that fail at various points (initializations, rules, hooks). The output
from the current trunk is interleaved and hard to read (compare trunk1.txt and
trunk2.txt -- they're executions with the same seed; it's hard to tell which
line of output is associated with which test). The output from junit4 is also
reordered (because parallel tests execution is effectively a race
condition...), but the OUTPUT passed to listeners (in this case the console
output) is always fully consistent.
Compare junit4-1.txt and junit4-2.txt. Every suite block is atomic. Like this
one:
{noformat}
[junit4] Running
org.apache.lucene.util.junitcompat.failures.TestFailOn07After
[junit4] ERROR 0.09s S3 | TestFailOn07After.testMethod
[junit4] > Throwable #1: java.lang.RuntimeException: Failure on @After.
[junit4] > at
org.apache.lucene.util.junitcompat.failures.TestFailOn07After.after(TestFailOn07After.java:13)
[junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
[junit4] > at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4] > at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] > at java.lang.reflect.Method.invoke(Method.java:601)
[junit4] > at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
[junit4] > at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
[junit4] > at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
[junit4] > at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:36)
[junit4] > at
org.apache.lucene.util.LuceneTestCase$SubclassSetupTeardownRule$1.evaluate(LuceneTestCase.java:700)
[junit4] > at
org.apache.lucene.util.LuceneTestCase$InternalSetupTeardownRule$1.evaluate(LuceneTestCase.java:599)
[junit4] > at
org.apache.lucene.util.LuceneTestCase$TestResultInterceptorRule$1.evaluate(LuceneTestCase.java:504)
[junit4] > at
org.apache.lucene.util.LuceneTestCase$RememberThreadRule$1.evaluate(LuceneTestCase.java:562)
[junit4] > at org.junit.rules.RunRules.evaluate(RunRules.java:18)
[junit4] > at
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
[junit4] > at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
[junit4] > at
org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:165)
[junit4] > at
org.apache.lucene.util.LuceneTestCaseRunner.runChild(LuceneTestCaseRunner.java:57)
[junit4] > at
org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
[junit4] > at
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
[junit4] > at
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
[junit4] > at
org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
[junit4] > at
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
[junit4] > at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
[junit4] > at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
[junit4] > at
org.junit.runners.ParentRunner.run(ParentRunner.java:300)
[junit4] > at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
[junit4] > at
com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:124)
[junit4] > at
com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:186)
[junit4] > at
com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:12)
[junit4] >
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestFailOn07After
-Dtestmethod=testMethod
-Dtests.seed=-737472ec6455983d:77a4c0da1ce834dd:1a2ea2ed4eba1655
-Dargs="-Dfile.encoding=Cp1252"
[junit4] 2>
[junit4] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time:
0.25s <<< FAILURES!
{noformat}
A block like this is composed of sections. It begins with the suite name,
followed by tests:
{noformat}
[junit4] Running
org.apache.lucene.util.junitcompat.failures.TestFailOn07After
{noformat}
If anything fails, it is reported as a consistent block within a suite. For
example here, an @After hook failed on testMethod:
{noformat}
[junit4] ERROR 0.09s S3 | TestFailOn07After.testMethod
[junit4] > Throwable #1: java.lang.RuntimeException: Failure on @After.
[junit4] > at
org.apache.lucene.util.junitcompat.failures.TestFailOn07After.after(TestFailOn07After.java:13)
[junit4] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
...
[junit4] > at
com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:12)
[junit4] >
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestFailOn07After
-Dtestmethod=testMethod
-Dtests.seed=-737472ec6455983d:77a4c0da1ce834dd:1a2ea2ed4eba1655
-Dargs="-Dfile.encoding=Cp1252"
[junit4] 2>
{noformat}
An output section like this also follows a pattern. The first line contains a
test case's status, time, slave JVM identifier (S3) and cause description:
{noformat}
[junit4] ERROR 0.09s S3 | TestFailOn07After.testMethod
{noformat}
then follows (">"-indented) throwable cause and stack:
{noformat}
[junit4] > Throwable #1: java.lang.RuntimeException: Failure on @After.
[junit4] > at
org.apache.lucene.util.junitcompat.failures.TestFailOn07After.after(TestFailOn07After.java:13)
...
{noformat}
finally, standards output and standard error streams come at the end. They are
also indented and prefixed with "1>" (stdout" or "2>" (stderr). stderr and
stdout calls come in the order they actually happened on the slave (they're
synchronized). In this case, only stderr was used:
{noformat}
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=TestFailOn07After
-Dtestmethod=testMethod
-Dtests.seed=-737472ec6455983d:77a4c0da1ce834dd:1a2ea2ed4eba1655
-Dargs="-Dfile.encoding=Cp1252"
[junit4] 2>
{noformat}
Note that even when when LuceneTestCase fails to provide reproduce string
(because the exception happened outside of @Rule boundaries), the output
exception cause is still nicely formatted and provided. As in this suite class,
which even fails to load properly:
{noformat}
[junit4] Running
org.apache.lucene.util.junitcompat.failures.TestFailOn01ClassLoad
[junit4] ERROR 0.00s S0 | TestFailOn01ClassLoad (suite)
[junit4] > Throwable #1: java.lang.ExceptionInInitializerError
[junit4] > at java.lang.Class.forName0(Native Method)
[junit4] > at java.lang.Class.forName(Class.java:186)
[junit4] > at
com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.instantiate(SlaveMain.java:138)
[junit4] > at
com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:116)
[junit4] > at
com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:186)
[junit4] > at
com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:12)
[junit4] > Caused by: java.lang.ArithmeticException: / by zero
[junit4] > at
org.apache.lucene.util.junitcompat.failures.TestFailOn01ClassLoad.<clinit>(TestFailOn01ClassLoad.java:11)
[junit4] > ... 6 more
[junit4] >
[junit4] Tests run: 0, Failures: 0, Errors: 1, Skipped: 0, Time:
0.00s <<< FAILURES!
{noformat}
The summary line contains "Tests run: 0" which may seem incorrect, but in fact
no test even began -- the error comes from the suite loader (the status line
contains no method name, just the suite name).
> Replace ant macros for running concurrent tests with ant-junit4.
> ----------------------------------------------------------------
>
> Key: LUCENE-3785
> URL: https://issues.apache.org/jira/browse/LUCENE-3785
> Project: Lucene - Java
> Issue Type: Sub-task
> Components: general/build, general/test
> Reporter: Dawid Weiss
> Assignee: Dawid Weiss
> Priority: Minor
> Fix For: 4.0
>
> Attachments: failure-cases.patch, junit4-1.txt, junit4-2.txt,
> trunk1.txt, trunk2.txt
>
>
> ant-junit4 is an ANT task for running tests in parallel (on slave JVMs). Its
> advantages over the current macros:
> - dynamic load balancing based on historical test runs and current execution
> (job-stealing),
> - jvm-crash resilience (I wrote tests that actually crash a slave jvms to
> make sure such an event is reported appropriately),
> - nicer console reporting (no need for syserrs on LuceneTestCase level).
> More documentation and info will follow as I roll out a patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
