This is super awesome and sure to be a huge help to evaluating my try
pushes in conjunction with the also very awesome Push Health mechanism!
Many thanks to you and anyone/everyone involved!
Andrew
On 4/10/23 1:17 PM, Joel Maher wrote:
For the last 6 months the code sheriffs have had a NEW flag on
autoland/mozilla-central to aid in their sheriffing. As of April 8th,
this feature is also available on try.
What is this "NEW flag"?
- when clicking on a failed job there is a list of failure lines-
there will be an orange background NEW word if the failure is new
[1]. This means we haven't seen this failure line in the last 3
weeks on autoland/central.
- NOTE: if the failure is slightly different than an existing
failure it will show as NEW, or if the previous failure is 22 days old
it will show as NEW
- NOTE: with this change we will not track failures from try
server, only from autoland/central.
How effective is this on Autoland [2]?
90% of regressions are identified with NEW (top edge cases here:
regression seen originally on m-c, backed out multiple times,
intermittent turns perma fail)
96% of new bugs filed by the sheriffs have the NEW flag (edge cases:
creating new bugs related to root cause, existing errors on other
tasks but in a longer list and we only annotate the first failure)
15% of failures marked as existing intermittent had a NEW flag - this
would be NOISE
How will this help on Try?
If you click through the failures and see a NEW flag in the error
summary, it is probably worth looking at, retrigger, etc.
If you do not see a NEW flag, the chance of introducing a regression
by ignoring this is pretty low.
There are some ideas we have to filter out failures without the NEW
flag as well as making try server help find more regressions and be
less confusing. If you want to provide feedback or communicate more
on this, please fill out this survey:
https://forms.gle/edae2f1EASqUEwqs6
Happy try pushing!
[1]
https://treeherder.mozilla.org/jobs?repo=mozilla-central&revision=ba377571e5cd05957790422128fd6657ac65601d&searchStr=web-platform&selectedTaskRun=Tx-T2HheQSms-58hCccmrw.0
[2]
https://docs.google.com/document/d/1mrtr620rp-X1clU1z6NF1fl7RfKYaIbNqN3d89KaHu0/edit?usp=sharing
--
You received this message because you are subscribed to the Google
Groups "[email protected]" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to [email protected].
To view this discussion on the web visit
https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/06525400-bfef-4a02-9029-6dbd28113893n%40mozilla.org
<https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/06525400-bfef-4a02-9029-6dbd28113893n%40mozilla.org?utm_medium=email&utm_source=footer>.
--
You received this message because you are subscribed to the Google Groups
"[email protected]" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/a/mozilla.org/d/msgid/dev-platform/5184f2a4-4836-0715-d7cf-ad9840aae487%40asutherland.org.
