In the last couple weeks the number of broken tests have started to go
up...and even tho I run bisect/etc from time to time ; sometimes people
don’t react to my comments/tickets/etc.
Because keeping this many failing tests makes it easier for a new one to
slip in...I think reverting the patch introducing the test failures
would also help in some case.
I think it would help a lot to prevent further test breaks to revert the
patch if any of the following conditions is met:
C1) if the notification/comment about the fact that the patch indeed
broken a test somehow have been unanswered for at least 24 hours.
C2) if the patch is in for 7 days; but the test failure is still not
addressed (note that in this case there might be a conversation about
fixing it...but in this case ; to enable other people to work in a
cleaner environment is more important than a single patch - and if it
can't be fixed in 7 days...well it might not get fixed in a month).
I would like to also note that I've seen a few tickets which have been
picked up by people who were not involved in creating the original
change - and although the intention was good, they might miss the
context of the original patch and may "fix" the tests in the wrong way:
accept a q.out which is inappropriate or ignore the test...
would it be ok to implement this from now on? because it makes my
efforts practically useless if people are not reacting…
note: just to be on the same page - this is only about running a single
test which falls on its own - I feel that flaky tests are an entirely
- reverting test-breaking changes Zoltan Haindrich