Both dbaron's and Gregor's suggestions sound good to me. Of the two, I like Gregor's approach since we can use the Treeherder page of the project repo to have a nice view of the oranges. But dbaron's approach probably requires less manpower to set up. Not sure.
Regardless, to do something about this we would need serious commitment to fixing the situation. *We put this retry logic there in the first place because there are Heisenbugs somewhere in the marionette runner, client, or server stacks*. Ie, having just one or two people on the B2G side look at this wasn't enough. We would need a task force that includes Gij experts, Taskcluster experts, and Marionette exports. I'm not sure how much effort this would require, but IMO it's worth it. So I guess the real question is, is it worth that kind of resource commitment to fix? On Wed, Nov 4, 2015 at 5:13 PM, L. David Baron <[email protected]> wrote: > On Wednesday 2015-11-04 16:48 +0100, Michael Henretty wrote: > > On Wed, Nov 4, 2015 at 4:45 PM, Fabrice Desré <[email protected]> > wrote: > > > Can we *right now* identify the worst offenders by looking at the tests > > > results/re-runs? You know that sheriffs will very quickly hide and > > > ignore tests that are really flaky. > > > > Yes, that's an important point. The problem is that you have to actually > > look at the logs of an individual chunk to see which tests failed. If a > > certain Gij test passes at least 1 out of it's 5 given runs, it will not > > surface to Treeherder, which means we can't start it. Looking through > each > > chunk log file (of which we have 40 per run) is doable, but more time > > consuming and error prone. > > Can you write a script to gather the data? Seems like you should be > able to get to it through various JSON files exposed for treeherder. > > (The code in > https://hg.mozilla.org/users/dbaron_mozilla.com/buildbot-json-tools/ > might not work anymore (although it might), but might provide some > useful hints on how to find the right logs. Treeherder source > should provide better hints, but is larger.) > > -David > > -- > 𝄞 L. David Baron http://dbaron.org/ 𝄂 > 𝄢 Mozilla https://www.mozilla.org/ 𝄂 > Before I built a wall I'd ask to know > What I was walling in or walling out, > And to whom I was like to give offense. > - Robert Frost, Mending Wall (1914) >
_______________________________________________ dev-fxos mailing list [email protected] https://lists.mozilla.org/listinfo/dev-fxos
