For the last 6 months the code sheriffs have had a NEW flag on
autoland/mozilla-central to aid in their sheriffing. As of April 8th, this
feature is also available on try.
What is this "NEW flag"?
- when clicking on a failed job there is a list of failure lines- there
will be an orange background NEW word if the failure is new [1]. This
means we haven't seen this failure line in the last 3 weeks on
autoland/central.
- NOTE: if the failure is slightly different than an existing failure
it will show as NEW, or if the previous failure is 22 days old it will show
as NEW
- NOTE: with this change we will not track failures from try server,
only from autoland/central.
How effective is this on Autoland [2]?
90% of regressions are identified with NEW (top edge cases here: regression
seen originally on m-c, backed out multiple times, intermittent turns perma
fail)
96% of new bugs filed by the sheriffs have the NEW flag (edge cases:
creating new bugs related to root cause, existing errors on other tasks but
in a longer list and we only annotate the first failure)
15% of failures marked as existing intermittent had a NEW flag - this would
be NOISE
How will this help on Try?
If you click through the failures and see a NEW flag in the error summary,
it is probably worth looking at, retrigger, etc.
If you do not see a NEW flag, the chance of introducing a regression by
ignoring this is pretty low.
There are some ideas we have to filter out failures without the NEW flag as
well as making try server help find more regressions and be less
confusing. If you want to provide feedback or communicate more on this,
please fill out this survey:
https://forms.gle/edae2f1EASqUEwqs6
Happy try pushing!
[1]
https://treeherder.mozilla.org/jobs?repo=mozilla-central&revision=ba377571e5cd05957790422128fd6657ac65601d&searchStr=web-platform&selectedTaskRun=Tx-T2HheQSms-58hCccmrw.0
[2]
https://docs.google.com/document/d/1mrtr620rp-X1clU1z6NF1fl7RfKYaIbNqN3d89KaHu0/edit?usp=sharing
--
You received this message because you are subscribed to the Google Groups
"[email protected]" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/06525400-bfef-4a02-9029-6dbd28113893n%40mozilla.org.
