I've got no idea how the test runners work, but can we manage the too-many-retries allowed problem by turning off retries by default but adding a whitelist of tests that should be retried on failure? That would give us data about which tests are sound, but would allow sheriffs to quiet the bad intermittents.
Maybe if a test filename begins with "racy_" then it gets retried, otherwise it doesn't? That would make it very clear to test maintainers which ones need work! David On Wed, Nov 4, 2015 at 7:39 AM, Michael Henretty <[email protected]> wrote: > Hi Gaia Folk, > > If you've been doing Gaia core work for any length of time, you are > probably aware that we have *many* intermittent Gij test failures on > Treeherder [1]. But the problem is even worse than you may know! You see, > each Gij test is run 5 times within a test chunk (g. Gij4) before it is > marked as failing. Then that chunk itself is retried up to 5 times before > the whole thing is marked as failing. This means that for a test to be > marked as "passing," it only has to run successfully once in *25* times. > I'm not kidding. Our retry logic, especially those inside the test chunk, > make it hard to know which intermittent tests are our worst offenders. This > is bad. > > My suggestion is to stop doing the retries inside the chunks. That way, > the failures will at least surface on Treeherder, which means we can star > more test, which means we'll have a lot more visibility on the bad > intermittents. Sheriffs will complain a lot, so we have to be ready to act > on these bugs. But the alternative is that we continue to write tests with > a low "raciness" bar which, IMO, have a much lower chance of catching > regressions. The longer we wait, the worse this problem becomes. > > Thoughts? > > Thanks, > Michael > > 1.) > https://bugzilla.mozilla.org/buglist.cgi?keywords=intermittent-failure&keywords_type=allwords&list_id=12657856&resolution=---&query_format=advanced&product=Firefox%20OS > > _______________________________________________ > dev-fxos mailing list > [email protected] > https://lists.mozilla.org/listinfo/dev-fxos > >
_______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
