Just to be clear, I meant to ask questions and you can neither agree nor disagree with a question. The assertion here is that the oranges are masking real issues. My intention was really to ask to what extent we know that oranges are masking real issues. I only added my own experience that many regressions have resulted in permareds rather than oranges to support the idea that we might look into quantifying the badness of the situation before creating more noise for sheriffs. That part is falsifiable and it would make more sense to argue (if you're intent on disagreeing with me : ) that it's not worth quantifying the extent to which oranges mask real issues for reasons x, y, z, etc.
On Thu, Nov 5, 2015 at 11:07 AM, Johnny Stenback <[email protected]> wrote: > On Wed, Nov 4, 2015 at 10:27 AM, Gareth Aye <[email protected]> wrote: > > On Wed, Nov 4, 2015 at 10:39 AM, Michael Henretty <[email protected] > > > > wrote: > >> > >> Hi Gaia Folk, > >> > >> If you've been doing Gaia core work for any length of time, you are > >> probably aware that we have *many* intermittent Gij test failures on > >> Treeherder [1]. But the problem is even worse than you may know! You > see, > >> each Gij test is run 5 times within a test chunk (g. Gij4) before it is > >> marked as failing. Then that chunk itself is retried up to 5 times > before > >> the whole thing is marked as failing. This means that for a test to be > >> marked as "passing," it only has to run successfully once in 25 times. > I'm > >> not kidding. Our retry logic, especially those inside the test chunk, > make > >> it hard to know which intermittent tests are our worst offenders. This > is > >> bad. > > > > > > I'm not sure that it is so bad. From my own experience, regressions > rarely > > cause intermittent failures. They mostly pop up as permareds. I think it > > would make sense to demonstrate that we are, in fact, masking a lot of > real > > broken functionality before making our intermittents noisier for > sheriffs. > > I couldn't disagree more. A decade+ of Firefox and Gecko test > automation has mountains of evidence that intermittent failures are > caused by regressions or exposed by seemingly unrelated changes. > > - jst >
_______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
