Hi Chris! Right. All you say is true here. JUnit 4.x wasn't clear on the specification of how such multiple failures should be handled in the first place. Some things were wrapped in "multiple-failure" exception, which didn't seem right to me. When I wrote the dedicated runner there was a lot of guessing involved, including XML reports (this bit not even part of JUnit, but Apache Ant and Apache Maven).
> Junit even logs a warning about this when it sees us doing it... It's actually the listener that I wrote that emits this log. Which leads me to say that if it's more helpful we could emit a different type of report from the runner (in addition to what Jenkins consumes). Internally, listeners receive each and every exception separately so they can serialize them in a more palatable manner. The XML listener is just one implementation: https://github.com/randomizedtesting/randomizedtesting/blob/master/junit4-ant/src/main/java/com/carrotsearch/ant/tasks/junit4/listeners/antxml/AntXmlReport.java Ideally, the new JUnit (5) provides a more modern and structured framework to build on top of... lots of hacks wouldn't be needed if we switched over. I just don't have the time to take care of it right now, sorry about that. > Reading the logs (as a person), we can see that during the initial "ant > test" BasicZkTest's lone test method PASSED, but there were 4 suite level > errors reported... > [...] > below showing what the test report says about BasicZkTest -- note there is > a single "suite" block for this class, regardless of being run multiple > times. That's because tools assume each class can be initialized once. The "tests.dups" isn't even implemented as part of the runner (because of this kind of problems); it was added to Ant build script later on. The XMLs written from the runner will have separate section for each repeated suite (I believe... didn't check and it was quite a while ago); the aggregation of these individual XMLs down the chain in jenkins is screwed up somewhere. D. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
