Just for fun, I broke time up into 15 minute intervals and counted how many machines were showing red on HEAD at each sample point (lateral join for last tick interpolation of data I collect from the BF), and plotted that over time. See attached.
I excluded seawasp (it tells us about *future* breakage against LLVM which unfortunately we don't always act on immediately, more on that shortly) and lorikeet (it is known to be broken, not our fault, though FWIW it hasn't crashed in > 3 months so I think we did actually work around it successfully). It looks a bit like the CI system has cut out the high spikes (= trashing the whole farm) and generally reduced the area of red.
