Re: Jenkins test report improvements: Suspicious (Recent) Failure Rates
> from the runner, something would still need to be able to aggregate those > reports across multiple runs (ie: "ant test" being run in multiple dirs, That's correct. > Thinking about it purely from the point of view of the failure rate > reports i've been generating, it would be nice if *every* TestClass > invocation consistently included a single psuedo-method entry [...] The thing is this would probably break things elsewhere. Those XMLs reports, although not specified anywhere, are accepted by a range of tools and my xml listener just tries to mimic closely what Apache Ant and Apache Maven emit in similar circumstances. I wonder what happens if you had a project with a regular maven/ ant test output and executions of multiple instances of the same class... I probably cross-checked it when I was trying to mimic their output, but I bet it wasn't anticipated as a possible scenario... Another possibility would be to try to correct those aggregations of individual XML reports in Jenkins so that the output is easier to post-process... Don't know, really. It is a mess. D. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Jenkins test report improvements: Suspicious (Recent) Failure Rates
: first place. Some things were wrapped in "multiple-failure" exception, : which didn't seem right to me. When I wrote the dedicated runner there : was a lot of guessing involved, including XML reports (this bit not : even part of JUnit, but Apache Ant and Apache Maven). ... : It's actually the listener that I wrote that emits this log. Which : leads me to say that if it's more helpful we could emit a different : type of report from the runner (in addition to what Jenkins consumes). : Internally, listeners receive each and every exception separately so : they can serialize them in a more palatable manner. The XML listener : is just one implementation: Interesting point ... but if we did generate a "new" structured report from the runner, something would still need to be able to aggregate those reports across multiple runs (ie: "ant test" being run in multiple dirs, the repro logic re-running "ant test" later in the same jenkins job, etc...) for it to be useful. As long as we stick w/the constraints of the JUnit XML report we can leverage the ((ant|maven)+jenkins) testReport. Thinking about it purely from the point of view of the failure rate reports i've been generating, it would be nice if *every* TestClass invocation consistently included a single psuedo-method entry indicating either a PASS if there were no suite level initialization or tear down problems, and a FAIL if there were -- something akin to the existing "junit.framework.TestSuite" psuedo test method entries, but exactly one per "suite instance" regardless of wether it succeeds or not (unlike the way there can currently be multiple "junit.framework.TestSuite" instances in a single class if it has multiple problems, but none if it doesn't) ... that way the number of runs is just the exact number of entries with this special name, and the number of suite instances that had 1 or more problems is just the number of psuedo-mehtods that "FAILed" It would make the xml files bigger in the "no problem" situation, but not by much. -Hoss http://www.lucidworks.com/ - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Jenkins test report improvements: Suspicious (Recent) Failure Rates
Hi Chris! Right. All you say is true here. JUnit 4.x wasn't clear on the specification of how such multiple failures should be handled in the first place. Some things were wrapped in "multiple-failure" exception, which didn't seem right to me. When I wrote the dedicated runner there was a lot of guessing involved, including XML reports (this bit not even part of JUnit, but Apache Ant and Apache Maven). > Junit even logs a warning about this when it sees us doing it... It's actually the listener that I wrote that emits this log. Which leads me to say that if it's more helpful we could emit a different type of report from the runner (in addition to what Jenkins consumes). Internally, listeners receive each and every exception separately so they can serialize them in a more palatable manner. The XML listener is just one implementation: https://github.com/randomizedtesting/randomizedtesting/blob/master/junit4-ant/src/main/java/com/carrotsearch/ant/tasks/junit4/listeners/antxml/AntXmlReport.java Ideally, the new JUnit (5) provides a more modern and structured framework to build on top of... lots of hacks wouldn't be needed if we switched over. I just don't have the time to take care of it right now, sorry about that. > Reading the logs (as a person), we can see that during the initial "ant > test" BasicZkTest's lone test method PASSED, but there were 4 suite level > errors reported... > [...] > below showing what the test report says about BasicZkTest -- note there is > a single "suite" block for this class, regardless of being run multiple > times. That's because tools assume each class can be initialized once. The "tests.dups" isn't even implemented as part of the runner (because of this kind of problems); it was added to Ant build script later on. The XMLs written from the runner will have separate section for each repeated suite (I believe... didn't check and it was quite a while ago); the aggregation of these individual XMLs down the chain in jenkins is screwed up somewhere. D. - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Jenkins test report improvements: Suspicious (Recent) Failure Rates
: bq. if the jenkins build re-runs a TestCase 5 times, there's still : only one section for that suite : : I don't know about Jenkins XML, but this behavior is quite all right. : Suite exceptions are either from static class init blocks or from : "outside of test" scopes (like rule initialization, cleanup, : Before/AfterClass hooks, etc.). These are performed once per class, They are performed once each time a test class is "run", but: 1) there may be multiple "suite level" failures reported for a single "run" of a suite -- which was the first time of confusing/problematic situation (even before the "repro" logic was added to some jenkins builds) that made it possible to see a suite level test failure rate > 100% from a single run of a test -- because there might be multiple suite level problems (IIRC one eample is an exception thrown during an @AfterClass *and* leaked threads are reported). 2) Add to that the fact that single jenkins job might run the same suite multiple times (because it's trying to repro a seed) and you can get even more "suite level" failures from a single test class in a single jenkins job -- but the test report doesn't contain any info about any many times the suite was "run" ... all of the method results & suite level failures for a single class *name* are aggregated into a single "suite" entry in the test report regardless of which run the failure came from. Junit even logs a warning about this when it sees us doing it... [junit4] Duplicate suite name used with XML reports: org.apache.solr.cloud.BasicZkTest. This may confuse tools that process XML reports. Set 'ignoreDuplicateSuites' to true to skip this message. Here's just one example of why the suite level failure rates can be wacky... Looking at this jenkins job.. https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/23269/ Reading the logs (as a person), we can see that during the initial "ant test" BasicZkTest's lone test method PASSED, but there were 4 suite level errors reported... [junit4] 2> NOTE: reproduce with: ant test -Dtestcase=BasicZkTest -Dtests.seed=130E39C045A2E030 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=sr -Dtests.timezone=Afr ica/El_Aaiun -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1 [junit4] ERROR 0.00s J0 | BasicZkTest (suite) <<< ... [junit4] Completed [317/837 (2!)] on J0 in 86.92s, 1 test, 4 errors <<< FAILURES! Later in that same jenkins job, that repro line was retried - and the test ran and passed w/o any suite level failures -- but note the xml snippet below showing what the test report says about BasicZkTest -- note there is a single "suite" block for this class, regardless of being run multiple times. Ideally I'd like to report that there were 4 total "runs" of the class, and *1* of them had suite level failureS ... but the report doesn't make it clear that the suite ran 4 times, let alone that these failures all came from a single run. So for now -- the summary reports I generated can only make silly assumptions, leading to things like 200% suite failure rates. https://jenkins.thetaphi.de/view/Lucene-Solr/job/Lucene-Solr-master-Linux/23269/testReport/api/xml?tree=suites%5Bcases%5BclassName,name,status%5D%5D ... org.apache.solr.cloud.BasicZkTest testBasic PASSED org.apache.solr.cloud.BasicZkTest testBasic PASSED org.apache.solr.cloud.BasicZkTest testBasic PASSED org.apache.solr.cloud.BasicZkTest testBasic PASSED junit.framework.TestSuite org.apache.solr.cloud.BasicZkTest FAILED junit.framework.TestSuite org.apache.solr.cloud.BasicZkTest FAILED junit.framework.TestSuite org.apache.solr.cloud.BasicZkTest FAILED junit.framework.TestSuite org.apache.solr.cloud.BasicZkTest FAILED -Hoss http://www.lucidworks.com/ - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Jenkins test report improvements: Suspicious (Recent) Failure Rates
Hi Chris. bq. if the jenkins build re-runs a TestCase 5 times, there's still only one section for that suite I don't know about Jenkins XML, but this behavior is quite all right. Suite exceptions are either from static class init blocks or from "outside of test" scopes (like rule initialization, cleanup, Before/AfterClass hooks, etc.). These are performed once per class, regardless of how many tests a given class has associated to it. Don't know if this helps anyhow. Dawid - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Jenkins test report improvements: Suspicious (Recent) Failure Rates
FYI: I've made some improvements to my Jenkins Test reports recently... http://fucit.org/solr-jenkins-reports/ Most significantly: I've added a new summary report called "Suspicious Test Failure Rate"... http://fucit.org/solr-jenkins-reports/suspicious-failure-report.html This summarizes the "Recent" failure rate data (past 7 days, rolling) as compared to the "Historic" failure rate data (previous 3 calendar weeks) along with the "Delta" (Recent Rate - Historic Rage) for any test where the Delta is positive (ie: the "Recent" failure rate is higher then the "Historic" rate) Less interesting is that I cleaned up some of the data processing, so test methods that report themselves with paramaterized names (ie: using @Repeat, or @ParametersFactory) are now aggregated correctly -- this will make the (Suspicious) Failure Rate reports more accurate/useful to be to judge how often a give method is failure compared to other methods. I did this about a week ago, so the data is already cleaner -- but for things like the new Suspicious Test Failure Rate report it will take another ~2weeks for the "Historic" values to be reliable. (I also tried to "fix" the suite failure rate reporting, but ultimately this is just not possible because of the limitations in how the Jenkins/JUnit report formats thigns -- see the README.txt if you're curious) https://github.com/hossman/jenkins-reports/ -Hoss http://www.lucidworks.com/ - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
