Erick Erickson created SOLR-12028:
-------------------------------------
Summary: BadApple and AwaitsFix annotations usage
Key: SOLR-12028
URL: https://issues.apache.org/jira/browse/SOLR-12028
Project: Solr
Issue Type: Task
Security Level: Public (Default Security Level. Issues are Public)
Components: Tests
Reporter: Erick Erickson
There's a long discussion of this topic at SOLR-12016. Here's a summary:
- BadApple annotations are used for tests that intermittently fail, say < 30%
of the time. Tests that fail more often shold be moved to AwaitsFix. This is,
of course, a judgement call
- AwaitsFix annotations are used for tests that, for some reason, the problem
can't be fixed immediately. Likely reasons are third-party dependencies,
extreme difficulty tracking down, dependency on another JIRA etc.
Jenkins jobs will typically run with BadApple disabled to cut down on noise.
Periodically Jenkins jobs will be run with BadApples enabled so BadApple tests
won't be lost and reports can be generated. Tests that run with BadApples
disabled that fail require _immediate_ attention.
The default for developers is that BadApple is enabled.
If you are working on one of these tests and cannot get the test to fail
locally, it is perfectly acceptable to comment the annotation out. You should
let the dev list know that this is deliberate.
This JIRA is a placeholder for BadApple tests to point to between the times
they're identified as BadApple and they're either fixed or changed to AwaitsFix
or assigned their own JIRA.
I've assigned this to myself to track so I don't lose track of it. No one
person will fix all of these issues, this will be an ongoing technical debt
cleanup effort.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
