Re: Per-test chaos mode now available, use it to help win the war on orange!
I've filed a bug for enabled Chaos Mode without recompiling: https://bugzilla.mozilla.org/show_bug.cgi?id=1182516 On Mon, Jun 8, 2015 at 9:12 AM, kgu...@mozilla.com wrote: On Thursday, June 4, 2015 at 6:15:35 PM UTC-4, Chris AtLee wrote: Very interesting, thank you! Would there be a way to add an environment variable or harness flag to run all tests in chaos mode? There isn't at the moment but one can definitely be added if there is a use for it. At the moment the easiest way to enable chaos mode globally is to change the value of sChaosFeatures in ChaosMode.h and recompile. kats ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Per-test chaos mode now available, use it to help win the war on orange!
On Thursday, June 4, 2015 at 5:30:32 PM UTC-4, Chris Peterson wrote: Will chaos mode enabled tests run on Try and release branches? If a test has chaos mode enabled, then it will have chaos mode enabled on all branches, including Try and release branches. We don't know if chaos mode test failures are representative of real bugs, but could chaos mode hide bugs that only reveal themselves when users run without chaos mode? It's possible but I think this is pretty unlikely. In all of the callsites I checked, chaos mode is used to push us off the beaten path with respect to code execution flow, but it does so randomly and so in some cases we should take the same path as if chaos mode were disabled. kats ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Per-test chaos mode now available, use it to help win the war on orange!
On Thursday, June 4, 2015 at 6:15:35 PM UTC-4, Chris AtLee wrote: Very interesting, thank you! Would there be a way to add an environment variable or harness flag to run all tests in chaos mode? There isn't at the moment but one can definitely be added if there is a use for it. At the moment the easiest way to enable chaos mode globally is to change the value of sChaosFeatures in ChaosMode.h and recompile. kats ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Per-test chaos mode now available, use it to help win the war on orange!
On 6/4/15 11:32 AM, kgu...@mozilla.com wrote: I just landed bug 1164218 on inbound, which adds the ability to run individual mochitests and reftests in chaos mode. (For those unfamiliar with chaos mode, it's a feature added by roc a while back that makes already-random things more random; see [1] or bug 955888 for details). The idea with making it available per-test is that new tests should be written and tested locally/on try with chaos mode enabled, to flush out possible intermittent failures faster. Ideally we should also land them with chaos mode enabled. At this time we're still not certain if this will provide a lot of value (i.e. if chaos-mode-triggered failures are representative of real bugs) so it's not mandatory to make your tests run in chaos mode, but please do let me know if you try enabling it on your test and are either successful or not. We need to collect more data on the usefulness of this to see where we should take it. If it does turn out to be valuable, my hope is that we can start making pre-existing tests chaos-mode enabled as well, and eventually reduce the intermittent failure rate. Will chaos mode enabled tests run on Try and release branches? We don't know if chaos mode test failures are representative of real bugs, but could chaos mode hide bugs that only reveal themselves when users run without chaos mode? See [2] for an example of how to enable chaos mode in your tests. Basically you can add chaos-mode to the reftest.list file for reftests, or call SimpleTest.testInChaosMode() for mochitests. If you do run into intermittent failures, the best way to debug them is usually to grab a recording of the failure using rr [3] and then debug the recording to see what was going on. This only works on Linux (and has some hardware requirements as well) but it's a really great tool to have. Cheers, kats [1] http://robert.ocallahan.org/2014/03/introducing-chaos-mode.html [2] https://hg.mozilla.org/integration/mozilla-inbound/rev/89ac61464a45 [3] http://rr-project.org/ or https://github.com/mozilla/rr/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Re: Per-test chaos mode now available, use it to help win the war on orange!
Very interesting, thank you! Would there be a way to add an environment variable or harness flag to run all tests in chaos mode? On Thu, Jun 4, 2015 at 5:31 PM, Chris Peterson cpeter...@mozilla.com wrote: On 6/4/15 11:32 AM, kgu...@mozilla.com wrote: I just landed bug 1164218 on inbound, which adds the ability to run individual mochitests and reftests in chaos mode. (For those unfamiliar with chaos mode, it's a feature added by roc a while back that makes already-random things more random; see [1] or bug 955888 for details). The idea with making it available per-test is that new tests should be written and tested locally/on try with chaos mode enabled, to flush out possible intermittent failures faster. Ideally we should also land them with chaos mode enabled. At this time we're still not certain if this will provide a lot of value (i.e. if chaos-mode-triggered failures are representative of real bugs) so it's not mandatory to make your tests run in chaos mode, but please do let me know if you try enabling it on your test and are either successful or not. We need to collect more data on the usefulness of this to see where we should take it. If it does turn out to be valuable, my hope is that we can start making pre-existing tests chaos-mode enabled as well, and eventually reduce the intermittent failure rate. Will chaos mode enabled tests run on Try and release branches? We don't know if chaos mode test failures are representative of real bugs, but could chaos mode hide bugs that only reveal themselves when users run without chaos mode? See [2] for an example of how to enable chaos mode in your tests. Basically you can add chaos-mode to the reftest.list file for reftests, or call SimpleTest.testInChaosMode() for mochitests. If you do run into intermittent failures, the best way to debug them is usually to grab a recording of the failure using rr [3] and then debug the recording to see what was going on. This only works on Linux (and has some hardware requirements as well) but it's a really great tool to have. Cheers, kats [1] http://robert.ocallahan.org/2014/03/introducing-chaos-mode.html [2] https://hg.mozilla.org/integration/mozilla-inbound/rev/89ac61464a45 [3] http://rr-project.org/ or https://github.com/mozilla/rr/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
Per-test chaos mode now available, use it to help win the war on orange!
I just landed bug 1164218 on inbound, which adds the ability to run individual mochitests and reftests in chaos mode. (For those unfamiliar with chaos mode, it's a feature added by roc a while back that makes already-random things more random; see [1] or bug 955888 for details). The idea with making it available per-test is that new tests should be written and tested locally/on try with chaos mode enabled, to flush out possible intermittent failures faster. Ideally we should also land them with chaos mode enabled. At this time we're still not certain if this will provide a lot of value (i.e. if chaos-mode-triggered failures are representative of real bugs) so it's not mandatory to make your tests run in chaos mode, but please do let me know if you try enabling it on your test and are either successful or not. We need to collect more data on the usefulness of this to see where we should take it. If it does turn out to be valuable, my hope is that we can start making pre-existing tests chaos-mode enabled as well, and eventually reduce the intermittent failure rate. See [2] for an example of how to enable chaos mode in your tests. Basically you can add chaos-mode to the reftest.list file for reftests, or call SimpleTest.testInChaosMode() for mochitests. If you do run into intermittent failures, the best way to debug them is usually to grab a recording of the failure using rr [3] and then debug the recording to see what was going on. This only works on Linux (and has some hardware requirements as well) but it's a really great tool to have. Cheers, kats [1] http://robert.ocallahan.org/2014/03/introducing-chaos-mode.html [2] https://hg.mozilla.org/integration/mozilla-inbound/rev/89ac61464a45 [3] http://rr-project.org/ or https://github.com/mozilla/rr/ ___ dev-platform mailing list dev-platform@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-platform
