Most of the intermittent from what I seen comes from race conditions, so it's hard to get a constant reproduction rate.
Adding logging statements might make the issue disappear due to the timing becoming different from writing the log. I think there's a few tools to help show the concurrent processes. top, b2g-ps, b2g-info... I guess in some sense we'll need to try profiling some too? More over is there anything QA can do to help narrow these things faster? Regression ranges are hard to find on intermittent. On Thu, Nov 5, 2015 at 8:07 AM, Johnny Stenback <[email protected]> wrote: > On Wed, Nov 4, 2015 at 10:27 AM, Gareth Aye <[email protected]> wrote: > > On Wed, Nov 4, 2015 at 10:39 AM, Michael Henretty <[email protected] > > > > wrote: > >> > >> Hi Gaia Folk, > >> > >> If you've been doing Gaia core work for any length of time, you are > >> probably aware that we have *many* intermittent Gij test failures on > >> Treeherder [1]. But the problem is even worse than you may know! You > see, > >> each Gij test is run 5 times within a test chunk (g. Gij4) before it is > >> marked as failing. Then that chunk itself is retried up to 5 times > before > >> the whole thing is marked as failing. This means that for a test to be > >> marked as "passing," it only has to run successfully once in 25 times. > I'm > >> not kidding. Our retry logic, especially those inside the test chunk, > make > >> it hard to know which intermittent tests are our worst offenders. This > is > >> bad. > > > > > > I'm not sure that it is so bad. From my own experience, regressions > rarely > > cause intermittent failures. They mostly pop up as permareds. I think it > > would make sense to demonstrate that we are, in fact, masking a lot of > real > > broken functionality before making our intermittents noisier for > sheriffs. > > I couldn't disagree more. A decade+ of Firefox and Gecko test > automation has mountains of evidence that intermittent failures are > caused by regressions or exposed by seemingly unrelated changes. > > - jst > _______________________________________________ > dev-fxos mailing list > [email protected] > https://lists.mozilla.org/listinfo/dev-fxos >
_______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
