I just landed bug 1164218 on inbound, which adds the ability to run individual mochitests and reftests in chaos mode. (For those unfamiliar with chaos mode, it's a feature added by roc a while back that makes already-random things more random; see [1] or bug 955888 for details).
The idea with making it available per-test is that new tests should be written and tested locally/on try with chaos mode enabled, to flush out possible intermittent failures faster. Ideally we should also land them with chaos mode enabled. At this time we're still not certain if this will provide a lot of value (i.e. if chaos-mode-triggered failures are representative of real bugs) so it's not mandatory to make your tests run in chaos mode, but please do let me know if you try enabling it on your test and are either successful or not. We need to collect more data on the usefulness of this to see where we should take it. If it does turn out to be valuable, my hope is that we can start making pre-existing tests chaos-mode enabled as well, and eventually reduce the intermittent failure rate. See [2] for an example of how to enable chaos mode in your tests. Basically you can add chaos-mode to the reftest.list file for reftests, or call SimpleTest.testInChaosMode() for mochitests. If you do run into intermittent failures, the best way to debug them is usually to grab a recording of the failure using rr [3] and then debug the recording to see what was going on. This only works on Linux (and has some hardware requirements as well) but it's a really great tool to have. Cheers, kats [1] http://robert.ocallahan.org/2014/03/introducing-chaos-mode.html [2] https://hg.mozilla.org/integration/mozilla-inbound/rev/89ac61464a45 [3] http://rr-project.org/ or https://github.com/mozilla/rr/ _______________________________________________ dev-platform mailing list [email protected] https://lists.mozilla.org/listinfo/dev-platform
