> On 7 Jan 2018, at 10:30, Alistair Grant <[email protected]> wrote: > > Hi Sven. > > On 6 January 2018 at 22:55, Sven Van Caekenberghe <[email protected]> wrote: >> >> >>> On 6 Jan 2018, at 20:08, Alistair Grant <[email protected]> wrote: >>> >>> Hi Ben, >>> >>> On 6 January 2018 at 12:09, Ben Coman <[email protected]> wrote: >>>> It great having a CI system to check all our tests a passing, >>>> but I thought I'd go old school and run them from the TestRunner GUI >>>> and found in a fresh latest Pharo 7 there are "128 failures, 63 errors" >>>> >>>> This one is interesting... >>>> ArrayTest>>test0FixtureAsSetForIdentityMultiplinessTest >>>> | anElement res | >>>> self elementsCopyNonIdenticalWithoutEqualElements. >>>> anElement := self elementsCopyNonIdenticalWithoutEqualElements anyOne. >>>> self deny: anElement copy == anElement. "<<<<FAILS" >>>> >>>> where anElement ==> SmallFloat64 (1.5) >>>> so the deny fails. >>> >>> Which platform and image? >>> >>> Pharo 7.0 >>> Build information: >>> Pharo-7.0+alpha.build.410.sha.aa40385cd6fee69bb51d4a3afcacfb5f91eed206 >>> (32 Bit) >>> Ubuntu 16.04 >>> >>> 14915 run, 14859 passes, 31 skipped, 50 expected failures, 5 failures, >>> 1 errors, 0 unexpected passes >>> Failures: >>> ReleaseTest>>#testMethodsWithUnboundGlobals >>> TonelWriterTest>>#testWriteSnapshot >>> ReleaseTest>>#testUnpackagedClasses >>> ReleaseTest>>#testNoEmptyPackages >>> ReleaseTest>>#testThatAllMethodsArePackaged >>> >>> Errors: >>> ZnClientTests>>#testRedirect >> >> Hmm, strange. Does it failed each time ? >> >> It does not fail for me (macOS). Furthermore, all Zinc tests are green >> https://ci.inria.fr/pharo-contribution/job/ZincHTTPComponents/ > > It's failing quite rarely. I think we just got (un)lucky this time. > #testRedirect connects to http://zn.stfx.eu, so it could well be an > intermittent internet connection issue. > > What would we lose by restructuring the tests to use an internal > server? My initial reaction is that an increase in test reliability > would outweigh any loss of testing. And we could create manually run > tests for anything that can't be covered by the internal server tests.
I know, but most (functional) tests *ARE* internal (are running against a server created just for the test). The remaining are there because HTTP *IS* a network communication protocol where it is important to test against servers 'out there in the wild'. It would be too limiting to exclude the networking (with its connectivity and timing issues) and the fact that there are hundreds of server and TLS/HTTPS implementations out there. Splitting the tests in two group might be a first step. > Cheers, > Alistair > > > > > >>> Cheers, >>> Alistair >>> >>> >>> >>>> It seems that the test setUp is wrong since >>>> #elementsCopyNonIdenticalWithoutEqualElements >>>> does not return a collection (of elements for which copy is not identical ) >>>> without equal elements:" >>>> >>>> This seems like a permanent error rather than a transient one. >>>> How does this slip through CI testing? >>>> and what can be done about cleaning this up? >>>> >>>> cheers -ben >>>> >>>> P.S. With the loss of method version info is impossible to guess when this >>>> may have been introduced, to guess which Issue it was related to, to guess >>>> how I might approach fixing this myself.
