[
https://issues.apache.org/jira/browse/SOLR-12028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384219#comment-16384219
]
Hoss Man commented on SOLR-12028:
---------------------------------
bq. As for AwaitsFix, the theory (which I may not have adhered to completely,
feel free to correct) is that when something is actually known to be broken,
i.e. code we can point to and say "this code is broken" doesn't need to be run
until, well, the code is fixed. ...
isn't that exactly the situation SOLR-7736 was in before you started this? the
test was known to be broken and reliably fails 100% of the time, and had an
open isssue to track fixing it in the future?
{quote}
bq: so now this test is running for all developers
As it should IMO unless and until we do one of the following
...
2> identify the root problem is and move it to AwaitsFix and link it to a JIRA
with the RCA.
...
{quote}
As i said: SOLR-7736 already had a jira -- the description was sparse, but
FWIW: it did link back to another jira that explained the problem with the test
... AFAICT it already met the "end goal" you were aiming for.
----
Bottom line...
* I understood the initial motivation to: identify flaky tests; mark them as
BAdApple to get them off the regular jenkins builds; maybe mark them as
AwaitsFix.
* I also understand the value in saying "AwaitsFix pointed at a closed issue is
useless" -- let's assess those, maybe try them as BadApple, open new issues /
re-AwaitsFix if needed
* I do not understand what happened with this test...
** why should *ANY* test that was _already_ marked AwaitsFix, pointing to an
*open* jira have been changed to BadApple?
** why should any _open_ jira saying "this test is broken" have been resolved
just to redirect here / open a new issue later?
if the concern was that some issues didn't do a "good enough" job of explaining
why the test was broken, then that should have been raised in those jiras. it
just doesn't make much sense to me at all to resolve them (to just open new
ones later). Like wise it seems completely bizare to me to take a test that's
already marked AwaitsFix, which can easily be demonstrated to fail 100% of the
time and change it to "BadApple" so that it starts getting in the way and
failing for all devs.
> BadApple and AwaitsFix annotations usage
> ----------------------------------------
>
> Key: SOLR-12028
> URL: https://issues.apache.org/jira/browse/SOLR-12028
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Tests
> Reporter: Erick Erickson
> Assignee: Erick Erickson
> Priority: Major
> Attachments: SOLR-12016-buildsystem.patch,
> SOLR-12028-sysprops-reproduce.patch, SOLR-12028.patch, SOLR-12028.patch
>
>
> There's a long discussion of this topic at SOLR-12016. Here's a summary:
> - BadApple annotations are used for tests that intermittently fail, say < 30%
> of the time. Tests that fail more often shold be moved to AwaitsFix. This is,
> of course, a judgement call
> - AwaitsFix annotations are used for tests that, for some reason, the problem
> can't be fixed immediately. Likely reasons are third-party dependencies,
> extreme difficulty tracking down, dependency on another JIRA etc.
> Jenkins jobs will typically run with BadApple disabled to cut down on noise.
> Periodically Jenkins jobs will be run with BadApples enabled so BadApple
> tests won't be lost and reports can be generated. Tests that run with
> BadApples disabled that fail require _immediate_ attention.
> The default for developers is that BadApple is enabled.
> If you are working on one of these tests and cannot get the test to fail
> locally, it is perfectly acceptable to comment the annotation out. You should
> let the dev list know that this is deliberate.
> This JIRA is a placeholder for BadApple tests to point to between the times
> they're identified as BadApple and they're either fixed or changed to
> AwaitsFix or assigned their own JIRA.
> I've assigned this to myself to track so I don't lose track of it. No one
> person will fix all of these issues, this will be an ongoing technical debt
> cleanup effort.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
