[
https://issues.apache.org/jira/browse/SOLR-12016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uwe Schindler updated SOLR-12016:
---------------------------------
Attachment: SOLR-12016-buildsystem.patch
> Reduce noise from flakey tests
> ------------------------------
>
> Key: SOLR-12016
> URL: https://issues.apache.org/jira/browse/SOLR-12016
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Tests
> Affects Versions: 7.2, master (8.0)
> Reporter: Erick Erickson
> Assignee: Erick Erickson
> Priority: Major
> Attachments: SOLR-12016-buildsystem.patch,
> SOLR-12016-buildsystem.patch, SOLR-12016-buildsystem.patch
>
>
> We had a discussion of this topic on the dev list, look for a thread titled:
> "Test failures are out of control.....". I'll try to summarize that
> discussion here and we can move this JIRA forward. This may become an
> umbrella issue.
> Current situation concerns:
> > There is so much noise from flakey tests (particularly Solr tests) that
> > they are difficult to use.
> > The number of tests that regularly fail is increasing
> > Failures are being ignored
> > The number of failing tests makes releasing more difficult.
> > The number of failing tests make it harder to determine whether recent
> > changes actually caused problems. Running the tests again until they
> > succeed is used commonly at present, which is not robust.
> > e-mail notifications of failing tests are largely being ignored.
> Propsal:
> > Mark all currently "flakey" tests as BadApple or AwaitsFix
> > Run Jenkins jobs with BadApple (and/or AwaitsFix) enabled and disabled.
> > Frequency TBD, depends partly on whether we can label emails from these
> > runs for easy filtering of the two flavors.
> >> Label these runs with something suitable in the subject line (wish list)
> > Weekly reports on the tests labeled BadApple or AwaitsFix
> >> Perhaps this could be incorporated in the reports linked below (wish list)
> > Committers should enable BadApple (or AwaitsFix) regularly as a sanity
> > check. Leave these as defaults.
> > We start getting _much_ more aggressive about not allowing _new_ flakey
> > tests.
> NOTE: It's perfectly acceptable to have failing flakey tests as long as
> someone is activey working on _fixing_ them.
> Concerns with solution
> > Decreases test coverage
> > Decreases visibility of flakey tests, making fixing them less likely.
> > Some tools (see below) that report on bad tests will not see tests that are
> > annotated with BadApple or AwaitsFix.
> > Running unit tests and reporting errors are being conflated
> To be decided:
> > Can we label e-mails with failing tests with something in the subject line
> > identifying whether they were run with BadApple/Awaits fix enabled or
> > disabled? Can someone volunteer?
> > Is there any difference between BadApple and AwaitsFix? If not should we
> > deprecate one? I propose we just use AwaitsFix and deprecate BadApple.
> > Can the automated reports (see below) be enhanced to also report tests
> > labeled BadApple or AwaitsFix?
> Useful tools:
> > Steve Rowe's work on a Jenkins job to reproduce test failures (LUCENE-8106)
> > Hoss has worked on aggregating all test failures from the 3 Jenkins systems
> > (ASF, Policeman, and Steve's), downloading the test results & logs, and
> > running some reports/stats on failures.
> >> http://fucit.org/solr-jenkins-reports/
> >> https://github.com/hossman/jenkins-reports/
> >> http://fucit.org/solr-jenkins-reports/failure-report.html
> I've assigned this JIRA to myslef, but all volunteers welcome, especially
> anything that changes the build system.....
> I've decided to make this a SOLR jira on the theory that most of the
> offending tests are in the Solr hive, any sub-tasks for touching the build
> system can go under LUCENE if wanted.
> Also, I expect to add the annotation to some more tests for a few days as
> infrequent failures occur. Once we have stability (defined by there being
> little noise) that'll stop.
> 3 BadApple 23 AwaitsFix annotations are currently in the code, linked to
> these issues:
> HADOOP-14044
> HADOOP-9893
> LUCENE-3869
> LUCENE-5575")
> LUCENE-5595
> LUCENE-5737
> LUCENE-6709
> LUCENE-7161
> SOLR-2715
> SOLR-6213
> SOLR-6443
> SOLR-6944
> SOLR-7736
> SOLR-9036
> SOLR-10071
> SOLR-10107
> SOLR-10136
> SOLR-10734
> SOLR-10191
> SOLR-11134
> SOLR-11458
> SOLR-11714
> SOLR-11974
> Solr JIRAS about bad tests
> SOLR-2175
> SOLR-4147
> SOLR-5880
> SOLR-6423
> SOLR-6944
> SOLR-6961
> SOLR-6974
> SOLR-8122
> SOLR-8182
> SOLR-9869
> SOLR-10053
> SOLR-10070
> SOLR-10071
> SOLR-10139
> SOLR-10287
> SOLR-10815
> SOLR-11911
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
