On 6/4/15 11:32 AM, kgu...@mozilla.com wrote:
I just landed bug 1164218 on inbound, which adds the ability to run individual
mochitests and reftests in chaos mode. (For those unfamiliar with chaos mode,
it's a feature added by roc a while back that makes already-random things more
random; see [1] or bug 955888 for details).
The idea with making it available per-test is that new tests should be written
and tested locally/on try with chaos mode enabled, to flush out possible
intermittent failures faster. Ideally we should also land them with chaos mode
enabled. At this time we're still not certain if this will provide a lot of
value (i.e. if chaos-mode-triggered failures are representative of real bugs)
so it's not mandatory to make your tests run in chaos mode, but please do let
me know if you try enabling it on your test and are either successful or not.
We need to collect more data on the usefulness of this to see where we should
take it. If it does turn out to be valuable, my hope is that we can start
making pre-existing tests chaos-mode enabled as well, and eventually reduce the
intermittent failure rate.
Will chaos mode enabled tests run on Try and release branches?
We don't know if chaos mode test failures are representative of real
bugs, but could chaos mode hide bugs that only reveal themselves when
users run without chaos mode?
See [2] for an example of how to enable chaos mode in your tests. Basically you
can add chaos-mode to the reftest.list file for reftests, or call
SimpleTest.testInChaosMode() for mochitests.
If you do run into intermittent failures, the best way to debug them is usually
to grab a recording of the failure using rr [3] and then debug the recording to
see what was going on. This only works on Linux (and has some hardware
requirements as well) but it's a really great tool to have.
Cheers,
kats
[1] http://robert.ocallahan.org/2014/03/introducing-chaos-mode.html
[2] https://hg.mozilla.org/integration/mozilla-inbound/rev/89ac61464a45
[3] http://rr-project.org/ or https://github.com/mozilla/rr/
_______________________________________________
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform
