This sounds like the perfect use-case for a project repository. Lets disable the retry logic and fix/disable tests there.
-Gregor On Wednesday, November 4, 2015 at 4:46:02 PM UTC+1, Fabrice Desré wrote: > Can we *right now* identify the worst offenders by looking at the tests > results/re-runs? You know that sheriffs will very quickly hide and > ignore tests that are really flaky. > > Fabrice > > On 11/04/2015 07:39 AM, Michael Henretty wrote: > > Hi Gaia Folk, > > > > If you've been doing Gaia core work for any length of time, you are > > probably aware that we have *many* intermittent Gij test failures on > > Treeherder [1]. But the problem is even worse than you may know! You > > see, each Gij test is run 5 times within a test chunk (g. Gij4) before > > it is marked as failing. Then that chunk itself is retried up to 5 times > > before the whole thing is marked as failing. This means that for a test > > to be marked as "passing," it only has to run successfully once in *25* > > times. I'm not kidding. Our retry logic, especially those inside the > > test chunk, make it hard to know which intermittent tests are our worst > > offenders. This is bad. > > > > My suggestion is to stop doing the retries inside the chunks. That way, > > the failures will at least surface on Treeherder, which means we can > > star more test, which means we'll have a lot more visibility on the bad > > intermittents. Sheriffs will complain a lot, so we have to be ready to > > act on these bugs. But the alternative is that we continue to write > > tests with a low "raciness" bar which, IMO, have a much lower chance of > > catching regressions. The longer we wait, the worse this problem becomes. > > > > Thoughts? > > > > Thanks, > > Michael > > > > 1.) > > https://bugzilla.mozilla.org/buglist.cgi?keywords=intermittent-failure&keywords_type=allwords&list_id=12657856&resolution=---&query_format=advanced&product=Firefox%20OS > > > > > > _______________________________________________ > > dev-fxos mailing list > > [email protected] > > https://lists.mozilla.org/listinfo/dev-fxos > > > > > -- > Fabrice Desré > b2g team > Mozilla Corporation _______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
