[ https://issues.apache.org/jira/browse/SOLR-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16373292#comment-16373292 ]
Yonik Seeley commented on SOLR-12016: ------------------------------------- bq. Also, I expect to add the annotation to some more tests for a few days as infrequent failures occur. Per the email discussion, this should happen *after* the default for "ant test" is changed (the one that committers run manually before committing) so that regressions will be caught before they are committed. > Reduce noise from flakey tests > ------------------------------ > > Key: SOLR-12016 > URL: https://issues.apache.org/jira/browse/SOLR-12016 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: Tests > Affects Versions: 7.2, master (8.0) > Reporter: Erick Erickson > Assignee: Erick Erickson > Priority: Major > > We had a discussion of this topic on the dev list, look for a thread titled: > "Test failures are out of control.....". I'll try to summarize that > discussion here and we can move this JIRA forward. This may become an > umbrella issue. > Current situation concerns: > > There is so much noise from flakey tests (particularly Solr tests) that > > they are difficult to use. > > The number of tests that regularly fail is increasing > > Failures are being ignored > > The number of failing tests makes releasing more difficult. > > The number of failing tests make it harder to determine whether recent > > changes actually caused problems. Running the tests again until they > > succeed is used commonly at present, which is not robust. > > e-mail notifications of failing tests are largely being ignored. > Propsal: > > Mark all currently "flakey" tests as BadApple or AwaitsFix > > Run Jenkins jobs with BadApple (and/or AwaitsFix) enabled and disabled. > > Frequency TBD, depends partly on whether we can label emails from these > > runs for easy filtering of the two flavors. > >> Label these runs with something suitable in the subject line (wish list) > > Weekly reports on the tests labeled BadApple or AwaitsFix > >> Perhaps this could be incorporated in the reports linked below (wish list) > > Committers should enable BadApple (or AwaitsFix) regularly as a sanity > > check. Leave these as defaults. > > We start getting _much_ more aggressive about not allowing _new_ flakey > > tests. > NOTE: It's perfectly acceptable to have failing flakey tests as long as > someone is activey working on _fixing_ them. > Concerns with solution > > Decreases test coverage > > Decreases visibility of flakey tests, making fixing them less likely. > > Some tools (see below) that report on bad tests will not see tests that are > > annotated with BadApple or AwaitsFix. > > Running unit tests and reporting errors are being conflated > To be decided: > > Can we label e-mails with failing tests with something in the subject line > > identifying whether they were run with BadApple/Awaits fix enabled or > > disabled? Can someone volunteer? > > Is there any difference between BadApple and AwaitsFix? If not should we > > deprecate one? I propose we just use AwaitsFix and deprecate BadApple. > > Can the automated reports (see below) be enhanced to also report tests > > labeled BadApple or AwaitsFix? > Useful tools: > > Steve Rowe's work on a Jenkins job to reproduce test failures (LUCENE-8106) > > Hoss has worked on aggregating all test failures from the 3 Jenkins systems > > (ASF, Policeman, and Steve's), downloading the test results & logs, and > > running some reports/stats on failures. > >> http://fucit.org/solr-jenkins-reports/ > >> https://github.com/hossman/jenkins-reports/ > >> http://fucit.org/solr-jenkins-reports/failure-report.html > I've assigned this JIRA to myslef, but all volunteers welcome, especially > anything that changes the build system..... > I've decided to make this a SOLR jira on the theory that most of the > offending tests are in the Solr hive, any sub-tasks for touching the build > system can go under LUCENE if wanted. > Also, I expect to add the annotation to some more tests for a few days as > infrequent failures occur. Once we have stability (defined by there being > little noise) that'll stop. > 3 BadApple 23 AwaitsFix annotations are currently in the code, linked to > these issues: > HADOOP-14044 > HADOOP-9893 > LUCENE-3869 > LUCENE-5575") > LUCENE-5595 > LUCENE-5737 > LUCENE-6709 > LUCENE-7161 > SOLR-2715 > SOLR-6213 > SOLR-6443 > SOLR-6944 > SOLR-7736 > SOLR-9036 > SOLR-10071 > SOLR-10107 > SOLR-10136 > SOLR-10734 > SOLR-10191 > SOLR-11134 > SOLR-11458 > SOLR-11714 > SOLR-11974 > Solr JIRAS about bad tests > SOLR-2175 > SOLR-4147 > SOLR-5880 > SOLR-6423 > SOLR-6944 > SOLR-6961 > SOLR-6974 > SOLR-8122 > SOLR-8182 > SOLR-9869 > SOLR-10053 > SOLR-10070 > SOLR-10071 > SOLR-10139 > SOLR-10287 > SOLR-10815 > SOLR-11911 -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org