: bq. if the jenkins build re-runs a TestCase 5 times, there's still : only one section for that suite : : I don't know about Jenkins XML, but this behavior is quite all right. : Suite exceptions are either from static class init blocks or from : "outside of test" scopes (like rule initialization, cleanup, : Before/AfterClass hooks, etc.). These are performed once per class,
They are performed once each time a test class is "run", but: 1) there may be multiple "suite level" failures reported for a single "run" of a suite -- which was the first time of confusing/problematic situation (even before the "repro" logic was added to some jenkins builds) that made it possible to see a suite level test failure rate > 100% from a single run of a test -- because there might be multiple suite level problems (IIRC one eample is an exception thrown during an @AfterClass *and* leaked threads are reported). 2) Add to that the fact that single jenkins job might run the same suite multiple times (because it's trying to repro a seed) and you can get even more "suite level" failures from a single test class in a single jenkins job -- but the test report doesn't contain any info about any many times the suite was "run" ... all of the method results & suite level failures for a single class *name* are aggregated into a single "suite" entry in the test report regardless of which run the failure came from. Junit even logs a warning about this when it sees us doing it... [junit4] Duplicate suite name used with XML reports: org.apache.solr.cloud.BasicZkTest. This may confuse tools that process XML reports. Set 'ignoreDuplicateSuites' to true to skip this message. Here's just one example of why the suite level failure rates can be wacky... Looking at this jenkins job.. https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/23269/ Reading the logs (as a person), we can see that during the initial "ant test" BasicZkTest's lone test method PASSED, but there were 4 suite level errors reported... [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=BasicZkTest -Dtests.seed=130E39C045A2E030 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sr -Dtests.timezone=Afr ica/El_Aaiun -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1 [junit4] ERROR 0.00s J0 | BasicZkTest (suite) <<< ... [junit4] Completed [317/837 (2!)] on J0 in 86.92s, 1 test, 4 errors <<< FAILURES! Later in that same jenkins job, that repro line was retried - and the test ran and passed w/o any suite level failures -- but note the xml snippet below showing what the test report says about BasicZkTest -- note there is a single "suite" block for this class, regardless of being run multiple times. Ideally I'd like to report that there were 4 total "runs" of the class, and *1* of them had suite level failureS ... but the report doesn't make it clear that the suite ran 4 times, let alone that these failures all came from a single run. So for now -- the summary reports I generated can only make silly assumptions, leading to things like 200% suite failure rates. https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/23269/testReport/api/xml?tree=suites%5Bcases%5BclassName,name,status%5D%5D ... <suite> <case> <className>org.apache.solr.cloud.BasicZkTest</className> <name>testBasic</name> <status>PASSED</status> </case> <case> <className>org.apache.solr.cloud.BasicZkTest</className> <name>testBasic</name> <status>PASSED</status> </case> <case> <className>org.apache.solr.cloud.BasicZkTest</className> <name>testBasic</name> <status>PASSED</status> </case> <case> <className>org.apache.solr.cloud.BasicZkTest</className> <name>testBasic</name> <status>PASSED</status> </case> <case> <className>junit.framework.TestSuite</className> <name>org.apache.solr.cloud.BasicZkTest</name> <status>FAILED</status> </case> <case> <className>junit.framework.TestSuite</className> <name>org.apache.solr.cloud.BasicZkTest</name> <status>FAILED</status> </case> <case> <className>junit.framework.TestSuite</className> <name>org.apache.solr.cloud.BasicZkTest</name> <status>FAILED</status> </case> <case> <className>junit.framework.TestSuite</className> <name>org.apache.solr.cloud.BasicZkTest</name> <status>FAILED</status> </case> </suite> -Hoss http://www.lucidworks.com/ --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
