Re: rr chaos mode update

[Andreas Tolfsen]

On 16 February 2016 at 16:38, Nick Fitzgerald  wrote:
> It seems like try/tbpl could automatically detect new test files and run
> them N times. That way, the developer doesn't have to do it manually, so it
> is less "intimidating" and also less likely to be skipped by accident or
> forgotten.

We employed this strategy with some success at Opera.

New tests would be put into quarantine and run N times so that their
stability could be determined.  Test stability was value from 0.0 to
1.0, where 1.0 indicated absolute stableness.

A quarantined test would be run alongside other stable tests, but
would be excluded from the regression report.  The regression report
was roughly the equivalent of a green try run indicating that it’s
okay to check in your code to inbound; except it showed that your
change caused no _additional_ test failures compared to before.

Once the test had reached a certain threshold it would either be
deemed stable, and included in regression reports, or unstable, and
disabled permanently.

As I understand it, one of the hurdles to implementing a similar
system at Mozilla is that we do not have a concept of test uniqueness
and a central database collecting tests and their results, but perhaps
this can be retrospectively mitigated using ActiveData
(https://wiki.mozilla.org/Auto-tools/Projects/ActiveData).
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: rr chaos mode update

[Nick Fitzgerald]

It seems like try/tbpl could automatically detect new test files and run
them N times. That way, the developer doesn't have to do it manually, so it
is less "intimidating" and also less likely to be skipped by accident or
forgotten.

Running under rr would be nice, but even without rr this seems like it
would be a nice addition to our testing infrastructure.

On Mon, Feb 15, 2016 at 11:59 PM, Axel Hecht  wrote:

> On 16/02/16 03:15, Kyle Huey wrote:
>
>> Seems like a good thing to expect developers to do locally today.
>>
>
> Two concerns:
>
> What's the successs criteria here?
>
> Also, speaking as an occasional code contributor, newcomers and folks like
> me will probably give up on contributing patches earlier.
>
> Axel
>
>
>> - Kyle
>>
>> On Mon, Feb 15, 2016 at 6:08 PM, Justin Dolske 
>> wrote:
>>
>> On 2/14/16 9:25 PM, Bobby Holley wrote:
>>>
>>> How far are we from being able to use cloud (rather than local) machine
>>>
 time to produce a trace of an intermittently-failing bug? Some one-click
 procedure to produce a trace from a failure on treeherder seems like it
 would lower the activation energy significantly.


>>> And with that... At some point, what about having all *new* tests be
>>> battle-tested by X runs of rr-chaos testing?  If it passes, it's allowed
>>> to
>>> run in the usual CI automation. If it fails, it's not (and you have a
>>> handy
>>> recording to debug).
>>>
>>> Justin
>>>
>>> ___
>>> dev-platform mailing list
>>> dev-platform@lists.mozilla.org
>>> https://lists.mozilla.org/listinfo/dev-platform
>>>
>>>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: rr chaos mode update

[Axel Hecht]

On 16/02/16 03:15, Kyle Huey wrote:

Seems like a good thing to expect developers to do locally today.


Two concerns:

What's the successs criteria here?

Also, speaking as an occasional code contributor, newcomers and folks 
like me will probably give up on contributing patches earlier.


Axel



- Kyle

On Mon, Feb 15, 2016 at 6:08 PM, Justin Dolske  wrote:


On 2/14/16 9:25 PM, Bobby Holley wrote:

How far are we from being able to use cloud (rather than local) machine

time to produce a trace of an intermittently-failing bug? Some one-click
procedure to produce a trace from a failure on treeherder seems like it
would lower the activation energy significantly.



And with that... At some point, what about having all *new* tests be
battle-tested by X runs of rr-chaos testing?  If it passes, it's allowed to
run in the usual CI automation. If it fails, it's not (and you have a handy
recording to debug).

Justin

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform



___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: rr chaos mode update

[Bobby Holley]

On Mon, Feb 15, 2016 at 6:26 PM, Kyle Huey  wrote:

> rr runs fine under VMWare ;)
>

Yes, but asking developers to fire up a VM and sync their changes to that
VM when adding a test adds a pretty high impediment to adding a test. (Your
post might have tongue-in-cheek on this point, but I can't quite tell).

But anyway, this is all to say that we should be handling this in
automation.


>
> - Kyle
>
> On Mon, Feb 15, 2016 at 6:18 PM, Bobby Holley 
> wrote:
>
>> On Mon, Feb 15, 2016 at 6:15 PM, Kyle Huey  wrote:
>>
>>> Seems like a good thing to expect developers to do locally today.
>>>
>>
>> I don't think that's realistic for developers who aren't on Linux.
>>
>>
>>>
>>> - Kyle
>>>
>>> On Mon, Feb 15, 2016 at 6:08 PM, Justin Dolske 
>>> wrote:
>>>
>>> > On 2/14/16 9:25 PM, Bobby Holley wrote:
>>> >
>>> > How far are we from being able to use cloud (rather than local) machine
>>> >> time to produce a trace of an intermittently-failing bug? Some
>>> one-click
>>> >> procedure to produce a trace from a failure on treeherder seems like
>>> it
>>> >> would lower the activation energy significantly.
>>> >>
>>> >
>>> > And with that... At some point, what about having all *new* tests be
>>> > battle-tested by X runs of rr-chaos testing?  If it passes, it's
>>> allowed to
>>> > run in the usual CI automation. If it fails, it's not (and you have a
>>> handy
>>> > recording to debug).
>>> >
>>> > Justin
>>> >
>>> > ___
>>> > dev-platform mailing list
>>> > dev-platform@lists.mozilla.org
>>> > https://lists.mozilla.org/listinfo/dev-platform
>>> >
>>> ___
>>> dev-platform mailing list
>>> dev-platform@lists.mozilla.org
>>> https://lists.mozilla.org/listinfo/dev-platform
>>>
>>
>>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: rr chaos mode update

[Bobby Holley]

On Mon, Feb 15, 2016 at 6:15 PM, Kyle Huey  wrote:

> Seems like a good thing to expect developers to do locally today.
>

I don't think that's realistic for developers who aren't on Linux.


>
> - Kyle
>
> On Mon, Feb 15, 2016 at 6:08 PM, Justin Dolske  wrote:
>
> > On 2/14/16 9:25 PM, Bobby Holley wrote:
> >
> > How far are we from being able to use cloud (rather than local) machine
> >> time to produce a trace of an intermittently-failing bug? Some one-click
> >> procedure to produce a trace from a failure on treeherder seems like it
> >> would lower the activation energy significantly.
> >>
> >
> > And with that... At some point, what about having all *new* tests be
> > battle-tested by X runs of rr-chaos testing?  If it passes, it's allowed
> to
> > run in the usual CI automation. If it fails, it's not (and you have a
> handy
> > recording to debug).
> >
> > Justin
> >
> > ___
> > dev-platform mailing list
> > dev-platform@lists.mozilla.org
> > https://lists.mozilla.org/listinfo/dev-platform
> >
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: rr chaos mode update

[Kyle Huey]

rr runs fine under VMWare ;)

- Kyle

On Mon, Feb 15, 2016 at 6:18 PM, Bobby Holley  wrote:

> On Mon, Feb 15, 2016 at 6:15 PM, Kyle Huey  wrote:
>
>> Seems like a good thing to expect developers to do locally today.
>>
>
> I don't think that's realistic for developers who aren't on Linux.
>
>
>>
>> - Kyle
>>
>> On Mon, Feb 15, 2016 at 6:08 PM, Justin Dolske 
>> wrote:
>>
>> > On 2/14/16 9:25 PM, Bobby Holley wrote:
>> >
>> > How far are we from being able to use cloud (rather than local) machine
>> >> time to produce a trace of an intermittently-failing bug? Some
>> one-click
>> >> procedure to produce a trace from a failure on treeherder seems like it
>> >> would lower the activation energy significantly.
>> >>
>> >
>> > And with that... At some point, what about having all *new* tests be
>> > battle-tested by X runs of rr-chaos testing?  If it passes, it's
>> allowed to
>> > run in the usual CI automation. If it fails, it's not (and you have a
>> handy
>> > recording to debug).
>> >
>> > Justin
>> >
>> > ___
>> > dev-platform mailing list
>> > dev-platform@lists.mozilla.org
>> > https://lists.mozilla.org/listinfo/dev-platform
>> >
>> ___
>> dev-platform mailing list
>> dev-platform@lists.mozilla.org
>> https://lists.mozilla.org/listinfo/dev-platform
>>
>
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: rr chaos mode update

[Kyle Huey]

Seems like a good thing to expect developers to do locally today.

- Kyle

On Mon, Feb 15, 2016 at 6:08 PM, Justin Dolske  wrote:

> On 2/14/16 9:25 PM, Bobby Holley wrote:
>
> How far are we from being able to use cloud (rather than local) machine
>> time to produce a trace of an intermittently-failing bug? Some one-click
>> procedure to produce a trace from a failure on treeherder seems like it
>> would lower the activation energy significantly.
>>
>
> And with that... At some point, what about having all *new* tests be
> battle-tested by X runs of rr-chaos testing?  If it passes, it's allowed to
> run in the usual CI automation. If it fails, it's not (and you have a handy
> recording to debug).
>
> Justin
>
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: rr chaos mode update

[Justin Dolske]

On 2/14/16 9:25 PM, Bobby Holley wrote:


How far are we from being able to use cloud (rather than local) machine
time to produce a trace of an intermittently-failing bug? Some one-click
procedure to produce a trace from a failure on treeherder seems like it
would lower the activation energy significantly.


And with that... At some point, what about having all *new* tests be 
battle-tested by X runs of rr-chaos testing?  If it passes, it's allowed 
to run in the usual CI automation. If it fails, it's not (and you have a 
handy recording to debug).


Justin
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: rr chaos mode update

[Chris Peterson]

On 2/14/16 9:16 PM, Robert O'Callahan wrote:

Lots of tests have been disabled for intermittency over the years. Now we
have the ability to fix (at least some of) them without much pain, it may
be worth revisiting them, though i don't know how to prioritize that.

We might want to revisit our workflow. If we had the ability to mark tests
as disabled-for-intermittency explicitly, maybe we could automatically
disable intermittent tests as they show up and dedicate a pool of machines
to reproducing them with rr.


While not rr chaos mode, Kats added support to enable Firefox's chaos 
mode for individual reftests. (This was bug 1164218.) Developers should 
consider enabling this option when writing new reftests or fixing issues 
in existing reftests.

___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: rr chaos mode update

[Robert O'Callahan]

On Mon, Feb 15, 2016 at 10:58 PM, Gijs Kruitbosch 
wrote:

> On 15/02/2016 05:16, Robert O'Callahan wrote:
>
>> At this point the limiting factor is getting developers to actually debug
>> and fix recorded test failures.
>>
>
> Well, and platform (OS) support for rr, right? And rr also doesn't
> effectively support debugging frontend JS tests, AIUI? Have either of those
> changed recently? For example, one of the most frequent intermittents in
> "my area" is: https://bugzilla.mozilla.org/show_bug.cgi?id=1130411 . But
> that's Windows only and pretty much all frontend JS code.


Good points. Just because a test only shows up on one platform in
automation doesn't mean rr can't find it on Linux. It could be a
cross-platform bug that just happens to hit the right conditions on one
platform; we've already seen a couple of examples of this.

Front-end JS code isn't easy to debug with rr but it is possible with JS
engine gdb helpers if you're desperate. Setting breakpoints in Gecko code
to catch JS touching native objects or calling JS callbacks, plus liberal
use of ::DumpJSStack, helps.

Rob
-- 
lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf
toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea
lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr
esn
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: rr chaos mode update

[Gijs Kruitbosch]

On 15/02/2016 05:16, Robert O'Callahan wrote:

At this point the limiting factor is getting developers to actually debug
and fix recorded test failures.


Well, and platform (OS) support for rr, right? And rr also doesn't 
effectively support debugging frontend JS tests, AIUI? Have either of 
those changed recently? For example, one of the most frequent 
intermittents in "my area" is: 
https://bugzilla.mozilla.org/show_bug.cgi?id=1130411 . But that's 
Windows only and pretty much all frontend JS code.


~ Gijs
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: rr chaos mode update

[Robert O'Callahan]

On Mon, Feb 15, 2016 at 7:21 PM, Mike Hommey  wrote:

> Relatedly, roc, is it possible to replay, on a different host, with
> possibly a different CPU, a record that would have been taken on the
> cloud? Does using a VM make it possible? If yes, having "the cloud" (or
> a set of developers) try to reproduce intermittents, and then have
> developers download the records and corresponding VM would be very
> useful. If not, we'd need a system like we have for build/test slave
> loaners.
>

Currently that isn't supported. It could be implemented, possibly with VM
and/or kernel changes. In the short term it's easier to provide ssh access
to the machine with the recording.

Rob
-- 
lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf
toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea
lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr
esn
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: rr chaos mode update

[Robert O'Callahan]

On Mon, Feb 15, 2016 at 6:26 PM, Kyle Huey  wrote:

>
> FWIW, every failure that I've debugged to completion so far has been a bug
> in the test (although I have two fatal assertion bugs I'm working through
> that will obviously be flaws in Gecko).  I think one of the things we
> really want to get a feeling for is how often we find actual bugs in the
> product.
>

Yes. So far I've found three Gecko bugs, but we'll find many bugs in tests.

Rob
-- 
lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf
toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea
lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr
esn
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: rr chaos mode update

[Benoit Girard]

I've got RR working under digital oceans and it works great there.

We've built a harness for generating replays. Once a replay is generated I
match the replay with the bug and comment in the bug looking for developers
to investigate. When they respond they can investigate by ssh'ing. Example:
https://bugzilla.mozilla.org/show_bug.cgi?id=1223249#c12

If we can we should prefer to have an ssh end point running rather than
ship a large VM image. It's also my understanding that while RR works
inside a VM, the trace will not work if the VM has changed host.

However right now we've decided that it's really overkill for the time
being. Producing interesting RR replays is really trivial at the moment.
Finding enough engineers to analyze them is not.

On Mon, Feb 15, 2016 at 1:21 AM, Mike Hommey  wrote:

> On Sun, Feb 14, 2016 at 09:25:58PM -0800, Bobby Holley wrote:
> > This is so. Damn. Exciting. Thank you roc for having the vision and
> > persistence to bring this dream to reality.
> >
> > How far are we from being able to use cloud (rather than local) machine
> > time to produce a trace of an intermittently-failing bug? Some one-click
> > procedure to produce a trace from a failure on treeherder seems like it
> > would lower the activation energy significantly.
>
> One limiting factor is the CPU features required, that are not
> virtualized on AWS (they are on digital ocean, and that's about the only
> cloud provider where they are ttbomk).
>
> Relatedly, roc, is it possible to replay, on a different host, with
> possibly a different CPU, a record that would have been taken on the
> cloud? Does using a VM make it possible? If yes, having "the cloud" (or
> a set of developers) try to reproduce intermittents, and then have
> developers download the records and corresponding VM would be very
> useful. If not, we'd need a system like we have for build/test slave
> loaners.
>
> Mike
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: rr chaos mode update

[Mike Hommey]

On Sun, Feb 14, 2016 at 09:25:58PM -0800, Bobby Holley wrote:
> This is so. Damn. Exciting. Thank you roc for having the vision and
> persistence to bring this dream to reality.
> 
> How far are we from being able to use cloud (rather than local) machine
> time to produce a trace of an intermittently-failing bug? Some one-click
> procedure to produce a trace from a failure on treeherder seems like it
> would lower the activation energy significantly.

One limiting factor is the CPU features required, that are not
virtualized on AWS (they are on digital ocean, and that's about the only
cloud provider where they are ttbomk).

Relatedly, roc, is it possible to replay, on a different host, with
possibly a different CPU, a record that would have been taken on the
cloud? Does using a VM make it possible? If yes, having "the cloud" (or
a set of developers) try to reproduce intermittents, and then have
developers download the records and corresponding VM would be very
useful. If not, we'd need a system like we have for build/test slave
loaners.

Mike
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: rr chaos mode update

[Kyle Huey]

On Sun, Feb 14, 2016 at 9:37 PM, L. David Baron  wrote:

> On Sunday 2016-02-14 21:26 -0800, Kyle Huey wrote:
> > On Sun, Feb 14, 2016 at 9:16 PM, Robert O'Callahan 
> wrote:
> > > Over the last few days we have had a lot of positive experiences
> > > reproducing bugs with rr chaos mode. Kyle tells me that, in fact, he's
> been
> > > able to reproduce every single bug he tried with enough machine time
> thrown
> > > at it.
> >
> > Of five or so, but yes.
>
> How many of those were intermittents that were never actually
> reported on Linux on our test infrastructure (i.e., reported only on
> other platforms), but that you were able to reproduce in rr's chaos
> mode on Linux?
>

At least one, bug 1150737, had only appeared in any great quantity on 10.6,
and may never have appeared on non-Mac tests in automation.  Chaos mode
reproduced it in a minute or two on Linux.

- Kyle
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: rr chaos mode update

[L. David Baron]

On Sunday 2016-02-14 21:26 -0800, Kyle Huey wrote:
> On Sun, Feb 14, 2016 at 9:16 PM, Robert O'Callahan  
> wrote:
> > Over the last few days we have had a lot of positive experiences
> > reproducing bugs with rr chaos mode. Kyle tells me that, in fact, he's been
> > able to reproduce every single bug he tried with enough machine time thrown
> > at it.
> 
> Of five or so, but yes.

How many of those were intermittents that were never actually
reported on Linux on our test infrastructure (i.e., reported only on
other platforms), but that you were able to reproduce in rr's chaos
mode on Linux?

-David

-- 
턞   L. David Baron http://dbaron.org/   턂
턢   Mozilla  https://www.mozilla.org/   턂
 Before I built a wall I'd ask to know
 What I was walling in or walling out,
 And to whom I was like to give offense.
   - Robert Frost, Mending Wall (1914)


signature.asc
Description: Digital signature
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: rr chaos mode update

[Bobby Holley]

This is so. Damn. Exciting. Thank you roc for having the vision and
persistence to bring this dream to reality.

How far are we from being able to use cloud (rather than local) machine
time to produce a trace of an intermittently-failing bug? Some one-click
procedure to produce a trace from a failure on treeherder seems like it
would lower the activation energy significantly.

On Sun, Feb 14, 2016 at 9:16 PM, Robert O'Callahan 
wrote:

> Over the last few days we have had a lot of positive experiences
> reproducing bugs with rr chaos mode. Kyle tells me that, in fact, he's been
> able to reproduce every single bug he tried with enough machine time thrown
> at it.
>
> At this point the limiting factor is getting developers to actually debug
> and fix recorded test failures. Anyone should be able to set up a VM on
> their local machine, build Firefox, record some failures and fix them. For
> best results, run just one test that's known intermittent, or possibly the
> whole directory of tests if there might be inter-test dependencies. Use
> --shuffle and --run-until-failure. The most convenient way to run rr with
> chaos mode is probably to create a script rr-chaos that prepends the
> --chaos option, and use --debugger rr-chaos.
>
> Lots of tests have been disabled for intermittency over the years. Now we
> have the ability to fix (at least some of) them without much pain, it may
> be worth revisiting them, though i don't know how to prioritize that.
>
> We might want to revisit our workflow. If we had the ability to mark tests
> as disabled-for-intermittency explicitly, maybe we could automatically
> disable intermittent tests as they show up and dedicate a pool of machines
> to reproducing them with rr.
>
> Rob
> --
> lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf
> toD
> selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
> rdsme,aoreseoouoto
> o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea
> lurpr
> .a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr
> esn
> ___
> dev-platform mailing list
> dev-platform@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-platform
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: rr chaos mode update

[Kyle Huey]

On Sun, Feb 14, 2016 at 9:16 PM, Robert O'Callahan 
wrote:

> Over the last few days we have had a lot of positive experiences
> reproducing bugs with rr chaos mode. Kyle tells me that, in fact, he's been
> able to reproduce every single bug he tried with enough machine time thrown
> at it.
>

Of five or so, but yes.

At this point the limiting factor is getting developers to actually debug
> and fix recorded test failures. Anyone should be able to set up a VM on
> their local machine, build Firefox, record some failures and fix them. For
> best results, run just one test that's known intermittent, or possibly the
> whole directory of tests if there might be inter-test dependencies. Use
> --shuffle and --run-until-failure. The most convenient way to run rr with
> chaos mode is probably to create a script rr-chaos that prepends the
> --chaos option, and use --debugger rr-chaos.
>

You can generally pass --debugger=/path/to/rr --debugger-args="record -h"
to mach to get things working.

Lots of tests have been disabled for intermittency over the years. Now we
> have the ability to fix (at least some of) them without much pain, it may
> be worth revisiting them, though i don't know how to prioritize that.
>

FWIW, every failure that I've debugged to completion so far has been a bug
in the test (although I have two fatal assertion bugs I'm working through
that will obviously be flaws in Gecko).  I think one of the things we
really want to get a feeling for is how often we find actual bugs in the
product.

- Kyle
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

rr chaos mode update

[Robert O'Callahan]

Over the last few days we have had a lot of positive experiences
reproducing bugs with rr chaos mode. Kyle tells me that, in fact, he's been
able to reproduce every single bug he tried with enough machine time thrown
at it.

At this point the limiting factor is getting developers to actually debug
and fix recorded test failures. Anyone should be able to set up a VM on
their local machine, build Firefox, record some failures and fix them. For
best results, run just one test that's known intermittent, or possibly the
whole directory of tests if there might be inter-test dependencies. Use
--shuffle and --run-until-failure. The most convenient way to run rr with
chaos mode is probably to create a script rr-chaos that prepends the
--chaos option, and use --debugger rr-chaos.

Lots of tests have been disabled for intermittency over the years. Now we
have the ability to fix (at least some of) them without much pain, it may
be worth revisiting them, though i don't know how to prioritize that.

We might want to revisit our workflow. If we had the ability to mark tests
as disabled-for-intermittency explicitly, maybe we could automatically
disable intermittent tests as they show up and dedicate a pool of machines
to reproducing them with rr.

Rob
-- 
lbir ye,ea yer.tnietoehr  rdn rdsme,anea lurpr  edna e hnysnenh hhe uresyf
toD
selthor  stor  edna  siewaoeodm  or v sstvr  esBa  kbvted,t
rdsme,aoreseoouoto
o l euetiuruewFa  kbn e hnystoivateweh uresyf tulsa rehr  rdm  or rnea
lurpr
.a war hsrer holsa rodvted,t  nenh hneireseoouot.tniesiewaoeivatewt sstvr
esn
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform