Re: Feedback requested: mach try auto

[Andrew Halberstadt]

On Wed, May 6, 2020 at 4:04 PM Steve Fink  wrote:

> Would it work to make a `mach try fuzzy --auto` or similar entrypoint,
> where it brings up the fzf selector with the automatic set of tests as a
> starting point?
>

Yes, this is something I've thought about as well. We could remove chunk
numbers from the fzf interface so you are basically selecting
configurations rather than tasks. Then ML would supply the tests to run and
the taskgraph would ensure only the cross section of tests + configurations
end up being scheduled. The same approach could apply to `./mach try
chooser`.

I filed bug 1635921 
to keep track of this idea. Thanks!



On 5/6/20 6:55 AM, Andrew Halberstadt wrote:
> >   Hello everyone,
> >
> > A handful of us have been hard at work on an initiative to become smarter
> > about which tests we schedule for a given push. We are approaching our
> > first major milestone, a try selector that can automatically pick which
> > tests are most relevant to your push on your behalf. You can use it today
> > by pulling the latest central, then running:
> >
> >  $ ./mach try auto
> >
> > Running this command will feed your patch(es) through a machine learning
> > algorithm that will guess which tests are most likely to be relevant.
> While
> > early results look promising, the tool is considered very experimental.
> So
> > we'd love to collect feedback from a wider developer audience before
> > recommending that people start using it by default.
> >
> > Feedback we are looking for includes:
> >
> > 1. Pushes where |mach try auto| failed to detect a regression (you either
> > got backed out or detected it yourself via some other means).
> > 2. Sentiment on resource usage. Does |mach try auto| schedule too much or
> > too little? How often are you willing to tolerate a backout?
> > 3. Bug reports, especially decision task failures.
> > 4. Feature requests. What would you find most useful when using this
> tool?
> > For example, a flag to increase/decrease the number of tasks that get
> run.
> > Or maybe a way to explicitly choose which platforms/configurations to run
> > on.
> >
> > If you have any questions or would like to provide feedback, please reach
> > out to us in #test-selection:mozilla.org
> > <
> https://matrix.to/#/!ahqVzULkEEXTziqCfU:mozilla.org?via=mozilla.org=t2bot.io
> >
> > on Matrix. You can also file bug and feature reports in Firefox Build
> > System :: Try
> > <
> https://bugzilla.mozilla.org/enter_bug.cgi?product=Firefox%20Build%20System=Try
> >
> > .
> >
> > Ongoing improvements to the algorithm combined with your feedback and an
> > ever growing dataset to train the ML, means that |mach try auto| will
> > continuously improve over time. Eventually we'll recommend everyone use
> it
> > by default as well as use the same algorithm on autoland.
> >
> > Thanks in advance for your help in getting us there!
> > -Andrew
> >
> > p.s If you've previously tried |mach try auto| and dismissed it for
> running
> > too many tasks or frequent decision task failures, please pull and give
> it
> > another shot!
> > ___
> > dev-platform mailing list
> > dev-platform@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-platform
>
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: New and improved stack-fixing

[Gabriele Svelto]

On 06/05/20 17:01, Markus Stange wrote:
> I see. What about crashes during regular optimized builds? I'd hope
> those print stacks.

You mean local builds? Unless you have ac_add_options --enable-debug in
your mozconfig you won't see stack traces. What you can do in that case
is build with the crash reporter enabled and point the
MINIDUMP_STACKWALK env variable to a build of the stackwalker. When
you'll hit an assertion a minidump will be generated, the test harness
will detect it and print out a stack-trace by feeding it into the stack
walker. Not optimal but better than nothing.

All of the above already happens on automation when running tests for
optimized builds: you won't have inline stack traces for crashes but at
the end of the test you'll see the crash dump with the complete stack trace.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: Feedback requested: mach try auto

[Steve Fink]

Would it work to make a `mach try fuzzy --auto` or similar entrypoint, 
where it brings up the fzf selector with the automatic set of tests as a 
starting point?


On 5/6/20 6:55 AM, Andrew Halberstadt wrote:

  Hello everyone,

A handful of us have been hard at work on an initiative to become smarter
about which tests we schedule for a given push. We are approaching our
first major milestone, a try selector that can automatically pick which
tests are most relevant to your push on your behalf. You can use it today
by pulling the latest central, then running:

 $ ./mach try auto

Running this command will feed your patch(es) through a machine learning
algorithm that will guess which tests are most likely to be relevant. While
early results look promising, the tool is considered very experimental. So
we'd love to collect feedback from a wider developer audience before
recommending that people start using it by default.

Feedback we are looking for includes:

1. Pushes where |mach try auto| failed to detect a regression (you either
got backed out or detected it yourself via some other means).
2. Sentiment on resource usage. Does |mach try auto| schedule too much or
too little? How often are you willing to tolerate a backout?
3. Bug reports, especially decision task failures.
4. Feature requests. What would you find most useful when using this tool?
For example, a flag to increase/decrease the number of tasks that get run.
Or maybe a way to explicitly choose which platforms/configurations to run
on.

If you have any questions or would like to provide feedback, please reach
out to us in #test-selection:mozilla.org

on Matrix. You can also file bug and feature reports in Firefox Build
System :: Try

.

Ongoing improvements to the algorithm combined with your feedback and an
ever growing dataset to train the ML, means that |mach try auto| will
continuously improve over time. Eventually we'll recommend everyone use it
by default as well as use the same algorithm on autoland.

Thanks in advance for your help in getting us there!
-Andrew

p.s If you've previously tried |mach try auto| and dismissed it for running
too many tasks or frequent decision task failures, please pull and give it
another shot!
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: New and improved stack-fixing

[Gabriele Svelto]

On 05/05/20 23:38, Nicholas Nethercote wrote:
> As for why that check is there... do opt builds produce any stack traces in
> tests? Normal assertions aren't enabled on opt builds, but
> diagnostic/release assertions are. I can't remember off the top of my head
> if they produce stacks traces in opt builds, and likewise with crashes.

Stack traces are printed out only on debug builds:

https://searchfox.org/mozilla-central/rev/7908ce29657cfd623993046bd8e38664e1c0b28e/mfbt/Assertions.h#451-454

So optimized debug builds will print assertions, but regular optimized
builds we have on automation will not.

 Gabriele



signature.asc
Description: OpenPGP digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Feedback requested: mach try auto

[Andrew Halberstadt]

 Hello everyone,

A handful of us have been hard at work on an initiative to become smarter
about which tests we schedule for a given push. We are approaching our
first major milestone, a try selector that can automatically pick which
tests are most relevant to your push on your behalf. You can use it today
by pulling the latest central, then running:

$ ./mach try auto

Running this command will feed your patch(es) through a machine learning
algorithm that will guess which tests are most likely to be relevant. While
early results look promising, the tool is considered very experimental. So
we'd love to collect feedback from a wider developer audience before
recommending that people start using it by default.

Feedback we are looking for includes:

1. Pushes where |mach try auto| failed to detect a regression (you either
got backed out or detected it yourself via some other means).
2. Sentiment on resource usage. Does |mach try auto| schedule too much or
too little? How often are you willing to tolerate a backout?
3. Bug reports, especially decision task failures.
4. Feature requests. What would you find most useful when using this tool?
For example, a flag to increase/decrease the number of tasks that get run.
Or maybe a way to explicitly choose which platforms/configurations to run
on.

If you have any questions or would like to provide feedback, please reach
out to us in #test-selection:mozilla.org

on Matrix. You can also file bug and feature reports in Firefox Build
System :: Try

.

Ongoing improvements to the algorithm combined with your feedback and an
ever growing dataset to train the ML, means that |mach try auto| will
continuously improve over time. Eventually we'll recommend everyone use it
by default as well as use the same algorithm on autoland.

Thanks in advance for your help in getting us there!
-Andrew

p.s If you've previously tried |mach try auto| and dismissed it for running
too many tasks or frequent decision task failures, please pull and give it
another shot!
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: New and improved stack-fixing

[Markus Stange]

On Wed, May 6, 2020 at 6:44 AM Gabriele Svelto  wrote:

> On 05/05/20 23:38, Nicholas Nethercote wrote:
> > As for why that check is there... do opt builds produce any stack traces
> in
> > tests? Normal assertions aren't enabled on opt builds, but
> > diagnostic/release assertions are. I can't remember off the top of my
> head
> > if they produce stacks traces in opt builds, and likewise with crashes.
>
> Stack traces are printed out only on debug builds:
>
>
> https://searchfox.org/mozilla-central/rev/7908ce29657cfd623993046bd8e38664e1c0b28e/mfbt/Assertions.h#451-454
>
> So optimized debug builds will print assertions, but regular optimized
> builds we have on automation will not.
>

I see. What about crashes during regular optimized builds? I'd hope those
print stacks.

Markus
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform