Erick Erickson commented on SOLR-12028:

Well, the process is evolving. I saw no use in keeping a separate open JIRA 
that simply said "this test doesn't succeed very often". I thought that 
pointing people at the uber-JIRA would provide more context. Whether that's the 
right call or not is an open question.

SOLR-7736: Right, I linked it under "related to" rather than "duplicates", we 
can add a duplicates link in if you'd like. I put in both JIRAs to allow a 
complete record, if it's too confusing we can remove, say, the 7736 reference.

As for AwaitsFix, the theory (which I may not have adhered to completely, feel 
free to correct) is that when something is actually known to be broken, i.e. 
code we can point to and say "this code is broken" doesn't need to be run 
until, well, the code is fixed. This would include dependent libs, code where 
the root cause is known but not fixed and the like. AwaitsFix aren't run 
periodically so disappear from sight.

BadApple, OTOH, is "this test doesn't run reliably, we have no clue why". These 
are also run periodically for the various reports to be able to see. That way 
they don't get lost.

It looked to me like AwaitsFix and BadApple were applied somewhat 
interchangeably so this is an attempt to bring regularity to the annotations.

I reconstructed SOLR-12016 under SOLR-12037 BTW.

bq: so now this test is running for all developers

As it should IMO unless and until we do one of the following
1> decide it's just a crappy test and remove it
2> identify the root problem is and move it to AwaitsFix and link it to a JIRA 
with the RCA.
3> fix it and remove the annotation altogether.

Note that for <3>, I'm starting by commenting out the annotation and providing 
the date it was commented out on the theory that if the test starts failing 
again it would be good information to have. OTOH if we look at this in 6 months 
and it hasn't failed in that time, the commented-out annotation can be removed.

My two main goals are:
1> surface any tests that fail regularly with no RCA for reports etc.
2> get clean runs with BadApple=false to catch regressions

I don't particularly care what the mechanics are. You'll see, every Sunday, a 
report to the dev list for the current BadApple and AwaitsFix annotations, file 
and method. This should complement your reports (which are way cool BTW).

So that's all the theory, I expect a few things to shift around before it all 
settles out...

> BadApple and AwaitsFix annotations usage
> ----------------------------------------
>                 Key: SOLR-12028
>                 URL: https://issues.apache.org/jira/browse/SOLR-12028
>             Project: Solr
>          Issue Type: Task
>      Security Level: Public(Default Security Level. Issues are Public) 
>          Components: Tests
>            Reporter: Erick Erickson
>            Assignee: Erick Erickson
>            Priority: Major
>         Attachments: SOLR-12016-buildsystem.patch, 
> SOLR-12028-sysprops-reproduce.patch, SOLR-12028.patch, SOLR-12028.patch
> There's a long discussion of this topic at SOLR-12016. Here's a summary:
> - BadApple annotations are used for tests that intermittently fail, say < 30% 
> of the time. Tests that fail more often shold be moved to AwaitsFix. This is, 
> of course, a judgement call
> - AwaitsFix annotations are used for tests that, for some reason, the problem 
> can't be fixed immediately. Likely reasons are third-party dependencies, 
> extreme difficulty tracking down, dependency on another JIRA etc.
> Jenkins jobs will typically run with BadApple disabled to cut down on noise. 
> Periodically Jenkins jobs will be run with BadApples enabled so BadApple 
> tests won't be lost and reports can be generated. Tests that run with 
> BadApples disabled that fail require _immediate_ attention.
> The default for developers is that BadApple is enabled.
> If you are working on one of these tests and cannot get the test to fail 
> locally, it is perfectly acceptable to comment the annotation out. You should 
> let the dev list know that this is deliberate.
> This JIRA is a placeholder for BadApple tests to point to between the times 
> they're identified as BadApple and they're either fixed or changed to 
> AwaitsFix or assigned their own JIRA.
> I've assigned this to myself to track so I don't lose track of it. No one 
> person will fix all of these issues, this will be an ongoing technical debt 
> cleanup effort.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Reply via email to