I started the list of fleaky tests at the end of this document https://docs.google.com/document/d/13Zv-za9V54aurmRwHSgObqGecN9PY8rc-t8U_oI_s-w/edit
On Tue, Aug 22, 2017 at 7:12 PM, Stephane Ducasse <[email protected]> wrote: > Hi denis > > you can have a look at > https://ci.inria.fr/pharo-ci-jenkins2/job/Test%20pending%20pull%20request%20and%20branch%20Pipeline/job/development/ > > I started to crawl this list and log the sporadic failing tests. > But I do it when integrating. > > Stef > > On Tue, Aug 22, 2017 at 6:41 PM, Denis Kudriashov <[email protected]> > wrote: >> Hi. >> >>> >>> 1) we have some sporadic test failures (sometimes network, sometimes >>> some mutex or delay issues). They do not happen all the time and >>> that's why sometimes there is one or two failing tests. We should >>> detect these tests and fix them to be more robust >> >> >> I thing I am responsible for Mutex problems because I wrote a bit naive >> waiting logic in some tests (with the hope that with green threads it will >> not fail). >> So if you have list of failing Mutex tests send it to me. I will fix them. >> >> 2017-08-22 17:51 GMT+02:00 Guillermo Polito <[email protected]>: >>> >>> Hi Marcus, >>> >>> Please, do not relaunch the build if there is a test failure. That >>> simply does not fix the problem, it just adds noise. There are two >>> problems: >>> >>> 1) we have some sporadic test failures (sometimes network, sometimes >>> some mutex or delay issues). They do not happen all the time and >>> that's why sometimes there is one or two failing tests. We should >>> detect these tests and fix them to be more robust >>> >>> >> Note, is there some brave soul that would like to take action here? >>> >>> 2) sometimes the vm crashes while running the tests. We should detect >>> the bug and fix it. >>> >>> >> Note, is there some brave soul that would like to take action here? >>> >>> Now, if we are integrating a fix, this means that the PR was **already >>> validated as green**. That means that the integrated commit can be >>> bootstrapped and tested (and all tests are green). >>> >>> Moreover, running the tests after integration (and its build color) >>> does not change the integration in itself: the commit was already >>> merged and pushed into the main branch, so even if it fails always, >>> it's too late. So, relaunching the build does not really change >>> anything (besides the impression we may have about the build). >>> >>> A complementary solution (that should go together with fixing the >>> tests) is to add a retry policy while running tests in the CI. >>> >>> >> Note, is there some brave soul that would like to take action here? >>> >>> Guille, on holidays from the phone. >>> >>> On 8/22/17, Marcus Denker <[email protected]> wrote: >>> > [ Pharo 70 ] Build 52 PR 205 >>> > index-inst-var-should-move-from-Slot-to-IndexedSlot >>> > https://github.com/pharo-project/pharo/pull/205 >>> > https://pharo.fogbugz.com/f/cases/20191 >>> > >>> > Build 53: failed, redone as 54 >>> > >>> > [ Pharo 70 ] Build 54 PR 193 Integrate WebBrowser-Core package to be >>> > able to >>> > open a browser on a URL >>> > https://pharo.fogbugz.com/f/cases/20304 >>> > https://github.com/pharo-project/pharo/pull/193 >>> > >>> > Build 55: failed, redone as 56 (I do not like to use this build number… >>> > with builds failing, we will always get the >>> > question “and what was in build XX?). >>> > >>> > [ Pharo 70 ] Build 56 PR 174 >>> > 20260-FastTable-not-support-cell-morphs-with-stepping-animation #174 >>> > https://github.com/pharo-project/pharo/pull/174 >>> > https://pharo.fogbugz.com/f/cases/20260/ >>> > >>> > >>> > >>> >>> >>> -- >>> >>> >>> >>> >>> Guille Polito >>> >>> >>> Research Engineer >>> >>> French National Center for Scientific Research - *http://www.cnrs.fr* >>> <http://www.cnrs.fr> >>> >>> >>> >>> *Web:* *http://guillep.github.io* <http://guillep.github.io> >>> >>> *Phone: *+33 06 52 70 66 13 >>> >>
