On Wed, Nov 4, 2015 at 10:27 AM, Gareth Aye <[email protected]> wrote: > On Wed, Nov 4, 2015 at 10:39 AM, Michael Henretty <[email protected]> > wrote: >> >> Hi Gaia Folk, >> >> If you've been doing Gaia core work for any length of time, you are >> probably aware that we have *many* intermittent Gij test failures on >> Treeherder [1]. But the problem is even worse than you may know! You see, >> each Gij test is run 5 times within a test chunk (g. Gij4) before it is >> marked as failing. Then that chunk itself is retried up to 5 times before >> the whole thing is marked as failing. This means that for a test to be >> marked as "passing," it only has to run successfully once in 25 times. I'm >> not kidding. Our retry logic, especially those inside the test chunk, make >> it hard to know which intermittent tests are our worst offenders. This is >> bad. > > > I'm not sure that it is so bad. From my own experience, regressions rarely > cause intermittent failures. They mostly pop up as permareds. I think it > would make sense to demonstrate that we are, in fact, masking a lot of real > broken functionality before making our intermittents noisier for sheriffs.
I couldn't disagree more. A decade+ of Firefox and Gecko test automation has mountains of evidence that intermittent failures are caused by regressions or exposed by seemingly unrelated changes. - jst _______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
