On Monday, March 4, 2013 5:17:29 PM UTC-8, Gregory Szorc wrote: > On 3/4/13 5:09 PM, Dave Mandelin wrote: > > >> We already don't back back out changes for regressing a benchmark like > >> we back them out for regressing tests. I think this is at least > >> partially because a general sentiment that not all of our benchmarks > >> correlate strongly to what they're trying to measure. > > > I know this has been a hot topic lately. I think getting more clarity on > > this would be great, *if* of course we could have an answer that was both > > operationally beneficial and clear, which seems to be incredibly difficult. > > > But this thread gives me a new idea. If each test run in automation had an > > owner (as I suggested elsewhere), how about also making the owners > > responsible for informing the sheriffs about what to do in case of > > regression? If the owners know the test is reliable and measures something > > important, they can ask for monitoring and presumptive backout. If not, > > they can ask sheriffs to ignore the test, inform and coordinate with the > > owning team, inform the landing person only, or some other action. > > This should be annotated in the tests themselves, IMO. We could even > have said annotation influence the color on TBPL.
I like it. We would need to make sure the annotations reflect active consideration by the test owners, but I suppose failures are likely to self-correct. > IMO we should be focusing on lessening the burden on the > sheriffs and leaving them to focus on real problems. Absolutely. Dave _______________________________________________ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
