+1 for disabling flakies Or a flaky hunt? I would pay a beer for every fixed flaky test for the contributor, and another 2 if it was actually a bug... :) (3 years ago when we were actively fixing flakies 1 in every 3 were real bugs :( )
Zoltan Haindrich <[email protected]> ezt írta (időpont: 2019. dec. 11., Sze 10:17): > Hey all! > > Recently the number of unstable tests have rised above the "usual" level... > These are usually hard to fix - and in most cases need a deep dive in the > area where the test operates. > Because of that I tend to just reattach the patch to the jira to get > another run in a day or so... > > The downside of the above approach is that reattaching on unrelated > failures has a positive hivqqa queuesize coefficient. > There is another downside which might not be obvious first: it reduces the > trust in the system and as a result there were cases when I did reattach > the patch; but it was a > genuine failure...it seemed unrelated; but actually it was. > > Instead of continuing to reattach patches every day; I would like to > propose a way to handle them: > > * check that the falling test has nothing to do with the actual patch > * it's important to be able to run test on our machines - but the most > important is to maintain that HiveQA is able to run them successfully; for > this reason I think > having 2 HiveQA runs for the same changeset where in one of them the > unstable test fails is the best > * you can search the jira for the testcase and look if other patches > have also bumped into it > * ? > * add a comment about that you are about the disable the test in > HIVE-22621 and commit it > * I think it would be ok to skip the regular code change process > * create a new subtask under HIVE-22619 with the details you know about > the falling testcase > * (resubmit your patch) > > What do you think? > > cheers, > Zoltan >
