IMO, we should remove/disable this test unless it can be “really” fixed. It is unacceptable to have a test that we know injects noise into our CI systems. Having to second guess whether failures are “real” or not undermines their value, and we should continue to stomp out any nondeterminism from the build and test cycle.
-Chris > On Mar 16, 2016, at 11:34 AM, Ben Langmuir via swift-dev > <swift-dev@swift.org> wrote: > > I was tired of seeing and writing these emails: > > commit 24f685c1d29cd926b1674e099bfcca6ed8673491 > Author: Ben Langmuir <blangm...@apple.com <mailto:blangm...@apple.com>> > Date: Wed Mar 16 10:36:02 2016 > > Decrease the probability that stdlib/Hashing.swift will fail > > Bump the number of trials from 10 to 14, which makes the test orders of > magnitude less likely to fail. For a range of size 10, doing 10 trials > meant that a single call to checkRange would fail ~0.03% of the time; > with 14 trials the spurious failure rate is ~0.0003%. We have 10 calls > to checkRange in this test with ranges of size <= 10. > > While this test didn't fail that often before, we have a very large > number of automated builds and it has been a constant low-level source > of friction that this test fails and we have to look at and then ignore > the results. > > >> On Mar 16, 2016, at 10:01 AM, Jordan Rose via swift-dev <swift-dev@swift.org >> <mailto:swift-dev@swift.org>> wrote: >> >> Unrelated and harmless: this test is probabilistic. >> >>> On Mar 16, 2016, at 10:00 , no-re...@swift.org <mailto:no-re...@swift.org> >>> wrote: >>> >>> [FAILURE] oss-swift-incremental-RA-osx [#2761] >>> >>> Build URL: https://ci.swift.org/job/oss-swift-incremental-RA-osx/2761/ >>> <https://ci.swift.org/job/oss-swift-incremental-RA-osx/2761/> >>> Project: oss-swift-incremental-RA-osx >>> Date of build: Wed, 16 Mar 2016 09:31:51 -0700 >>> Build duration: 29 min >>> Identified problems: >>> >>> Regression test failed: This build failed because a regression test in the >>> test suite FAILed. Below is a list of all errors: >>> Indication 1 >>> <https://ci.swift.org//job/oss-swift-incremental-RA-osx/2761/consoleFull#-1004795036fca400bf-2f4a-462e-b517-e058d770b2d7> >>> Tests: >>> >>> Name: Swift >>> Failed: 1 test(s), Passed: 7772 test(s), Total: 7773 test(s) >>> Failed: Swift.stdlib.Hashing.swift >>> <https://ci.swift.org/job/oss-swift-incremental-RA-osx/2761/testReport/junit/Swift/stdlib/Hashing_swift> >>> Name: Swift-Unit >>> Failed: 0 test(s), Passed: 278 test(s), Total: 278 test(s) >>> >>> Changes >>> >>> Commit c43943d7a884f7b43a6ac06877c29656d5214653 by jordan_rose: >>> [build-script] Remove -X option for "edit-only" Xcode projects. >>> >>> edit: utils/build-script-impl >>> edit: README.md >>> edit: CMakeLists.txt >>> edit: cmake/modules/SwiftAddCustomCommandTarget.cmake >>> edit: utils/build-script >> >> _______________________________________________ >> swift-dev mailing list >> swift-dev@swift.org <mailto:swift-dev@swift.org> >> https://lists.swift.org/mailman/listinfo/swift-dev > > _______________________________________________ > swift-dev mailing list > swift-dev@swift.org > https://lists.swift.org/mailman/listinfo/swift-dev
_______________________________________________ swift-dev mailing list swift-dev@swift.org https://lists.swift.org/mailman/listinfo/swift-dev
