They seem like the right priorities. We have a work around to report it manually for now. Just wanted to get the idea on the list while I thought of it. cheers -ben
On Sun, Aug 23, 2015 at 6:15 PM, stepharo <[email protected]> wrote: > HI ben > > you are right. We are all running like mad. > We are focusing on the bootstrap because guille will leave to brussels end > of the month. > Esteban is fighting with FFI to get migrating to Spur. > But yes this issue is a real one. > Stef > > Le 23/8/15 04:08, Ben Coman a écrit : > >> Is there somewhere that statistics of failures from the CI validation >> report are gathered? >> >> We've all experienced transient CI-monkey validation failures >> unrelated to our submitted fix, where just resubmitting to the monkey >> without change works fine. The problem is: >> >> * this may have been introduced several builds ago, but the transient >> failure passed the monkey at the time >> >> * it can be hard to identify which build introduced a transient error >> >> * doubly hard if the failure is due to some CI infrastructure change, >> or one particular build server >> >> * its hard to get an overview of how many issues get the same failure >> - is this a "once in six months" failure or is it happening 50% of the >> time in every issue. >> >> * it takes longer for an individual to observes a false-failure is >> common enough to worry about, than if reports were merged. Since >> different slices should produce different errors, failures common >> across all slices should stand out in statistics. >> >> If such statistics are not currently available, in cases where a >> simple resubmission passed the monkey, should we: >> >> * have an issue for each such false-failure, to get some early >> indication from manually track occurrences >> >> * where such an issue hasn't yet been created, first report comes to the >> list >> >> ------ >> So, I just got the following false-failures in separate runs for: >> https://pharo.fogbugz.com/f/cases/16265 >> * ProcessTerminateBug>>#testTerminationDuringUnwind >> * a CISmallIntRule: Timeout (after 0:00:40:00) occured while loading... >> >> Anyone else? >> >> cheers -ben >> >> P.S. I understand some errors are filtered. Can this list be output as >> part of the validation report (or is it already) ? >> >> > >
