As someone on this thread pointed out, the way that Gij is currently run actually hides many of the intermittents from Treeherder, and thus from our ability to track them. So the picture we see is very incomplete, and not an accurate representation of the stability of that suite.
If people want to get serious about fixing the intermittents, we should stop hiding them from Treeherder so we can track them and see what the worst offenders are. This means both stopping (or at least reducing) the internal reruns and stopping the automatic TaskCluster retries (or at least marking them as retriggers instead of retries), although in order for that not to turn into an explosion of orange on Treeherder, we'd probably only want to do one change at a time. Jonathan On Thu, Nov 5, 2015 at 8:43 AM, Johnny Stenback <[email protected]> wrote: > Fair enough. I personally don't think it's worth any more time trying > to prove this one way or another as we've seen intermittent issues > arise time and time again by seemingly unrelated changes. The A-team > at Mozilla has tons of data on this from years of tracking oranges on > tbpl and now tree herder, jgriffin can point you to that if needed. > > My point is simply that if we're care at all about quality then we > need a test harness that brings intermittent issues to light as > opposed to tries to hide them. From the op here it sounds like we have > the latter. > > - jst > > > On Thu, Nov 5, 2015 at 8:36 AM, Gareth Aye <[email protected]> wrote: > > Just to be clear, I meant to ask questions and you can neither agree nor > > disagree with a question. The assertion here is that the oranges are > masking > > real issues. My intention was really to ask to what extent we know that > > oranges are masking real issues. I only added my own experience that many > > regressions have resulted in permareds rather than oranges to support the > > idea that we might look into quantifying the badness of the situation > before > > creating more noise for sheriffs. That part is falsifiable and it would > make > > more sense to argue (if you're intent on disagreeing with me : ) that > it's > > not worth quantifying the extent to which oranges mask real issues for > > reasons x, y, z, etc. > > > > On Thu, Nov 5, 2015 at 11:07 AM, Johnny Stenback <[email protected]> > wrote: > >> > >> On Wed, Nov 4, 2015 at 10:27 AM, Gareth Aye <[email protected]> > wrote: > >> > On Wed, Nov 4, 2015 at 10:39 AM, Michael Henretty > >> > <[email protected]> > >> > wrote: > >> >> > >> >> Hi Gaia Folk, > >> >> > >> >> If you've been doing Gaia core work for any length of time, you are > >> >> probably aware that we have *many* intermittent Gij test failures on > >> >> Treeherder [1]. But the problem is even worse than you may know! You > >> >> see, > >> >> each Gij test is run 5 times within a test chunk (g. Gij4) before it > is > >> >> marked as failing. Then that chunk itself is retried up to 5 times > >> >> before > >> >> the whole thing is marked as failing. This means that for a test to > be > >> >> marked as "passing," it only has to run successfully once in 25 > times. > >> >> I'm > >> >> not kidding. Our retry logic, especially those inside the test chunk, > >> >> make > >> >> it hard to know which intermittent tests are our worst offenders. > This > >> >> is > >> >> bad. > >> > > >> > > >> > I'm not sure that it is so bad. From my own experience, regressions > >> > rarely > >> > cause intermittent failures. They mostly pop up as permareds. I think > it > >> > would make sense to demonstrate that we are, in fact, masking a lot of > >> > real > >> > broken functionality before making our intermittents noisier for > >> > sheriffs. > >> > >> I couldn't disagree more. A decade+ of Firefox and Gecko test > >> automation has mountains of evidence that intermittent failures are > >> caused by regressions or exposed by seemingly unrelated changes. > >> > >> - jst > > > > > _______________________________________________ > dev-fxos mailing list > [email protected] > https://lists.mozilla.org/listinfo/dev-fxos >
_______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
