Autoclassification of (a subset of) intermittent failures is now running
on treeherder. You may have spotted that some jobs are now annotated
with a hollow star symbol; this means that the autoclassifier matched
all the error lines in that job with previously observed intermittents.
The star will become filled once a sheriff has verified the matches made
by the autoclassifier.
In the short term this feature should not have a big impact on most
developers; the main benefit is that the unfilled star will indicate a
try orange that is entirely due to known intermittents.
The UI used to interact with the autoclassifier is hidden behind a URL
parameter; add &autoclassify to your treeherder URL to see it. This UI
is mainly for sheriffing, but can also be used to see if a job was
partially classified. Note that saving classifications in this UI will
affect what is classified in the future, across trees, so avoid saving
anything that may not be an intermittent.
The current classification is very basic; it only matches test failures,
and only where the test/status/message exactly match a previous
classification. It also relies on structured logging. As a result, many
relatively-common failures are not supported:
* Test failures with a variable message
* Crashes
* Suites that don't use structured logging at all or use it in a way
that doesn't maintain the required invariants (notably reftests)
* Logging messages
Work to improve the range of supported scenarios is in progress. Once we
are happy with the way that the system is working, it will be possible
to integrate with other engineering productivity tools such as autoland.
_______________________________________________
dev-platform mailing list
[email protected]
https://lists.mozilla.org/listinfo/dev-platform
