Hello everyone,
A handful of us have been hard at work on an initiative to become smarter
about which tests we schedule for a given push. We are approaching our
first major milestone, a try selector that can automatically pick which
tests are most relevant to your push on your behalf. You can use it today
by pulling the latest central, then running:
$ ./mach try auto
Running this command will feed your patch(es) through a machine learning
algorithm that will guess which tests are most likely to be relevant. While
early results look promising, the tool is considered very experimental. So
we'd love to collect feedback from a wider developer audience before
recommending that people start using it by default.
Feedback we are looking for includes:
1. Pushes where |mach try auto| failed to detect a regression (you either
got backed out or detected it yourself via some other means).
2. Sentiment on resource usage. Does |mach try auto| schedule too much or
too little? How often are you willing to tolerate a backout?
3. Bug reports, especially decision task failures.
4. Feature requests. What would you find most useful when using this tool?
For example, a flag to increase/decrease the number of tasks that get run.
Or maybe a way to explicitly choose which platforms/configurations to run
on.
If you have any questions or would like to provide feedback, please reach
out to us in #test-selection:mozilla.org
<https://matrix.to/#/!ahqVzULkEEXTziqCfU:mozilla.org?via=mozilla.org&via=t2bot.io>
on Matrix. You can also file bug and feature reports in Firefox Build
System :: Try
<https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox%20Build%20System&component=Try>
.
Ongoing improvements to the algorithm combined with your feedback and an
ever growing dataset to train the ML, means that |mach try auto| will
continuously improve over time. Eventually we'll recommend everyone use it
by default as well as use the same algorithm on autoland.
Thanks in advance for your help in getting us there!
-Andrew
p.s If you've previously tried |mach try auto| and dismissed it for running
too many tasks or frequent decision task failures, please pull and give it
another shot!
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
