On Wed, Mar 8, 2017 at 4:01 PM, <chris.ryan.pea...@gmail.com> wrote: > In the past I have not always been made aware when my tests were disabled, > which has lead to me feeling jaded.
We have a process (in theory) that ensures the relevant people get notified of tests. The process involves these steps: 1) There is a moz.build file somewhere in the tree which covers the tests in question and specifies a BUG_COMPONENT for it 2) If a test starts failing intermittently, a bug is filed in the aforementioned component 3) The component is monitored/triaged regularly by the module owner or team 4) Whoever triages the bug notifies the individual owner if they're not already on the bug So if for some reason anybody feels like they were not notified when their tests are being disabled, they should find the link the above chain where things broke down (e.g. no BUG_COMPONENT for a test, bug filed in a component other than BUG_COMPONENT, nobody triaging new bugs, etc.) and do something about it. > >> and it would be reasonable to expect a fix. > > I think it's unreasonable to assume that developers can drop whatever they're > doing and turn around a fix in a two weeks, given how long these things often > take to fix, and given that developers often have a pre-existing list of > other high priority stuff to work on. > In my experience it's not so much that a fix is needed in two weeks, it's that you need to put in a good-faith effort to respond and start investigation. Oftentimes it legitimately takes longer than two weeks to fix intermittents, but I've never had a scenario where I asked for more time and was denied that. > > I think: > * Acceptable failure rates as expressed as an absolute number aren't > meaningful; we should be expressing acceptable rates as a percentage. On the one hand, I agree with your rationale here. On the other hand, the people who have to deal with the problem (sheriffs) have to deal with it linearly. i.e. 60 failures is twice as much work for them than 30 failures, regardless of the number of pushes. So from their perspective an absolute number does make sense. I don't have a strong opinion on this either way, just pointing out the other side. Cheers, kats _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
