Hey all, as you might know, we've set up an internal CI system that is running `make check` on a variety of different platforms and configurations, 16 in total.
As we've experienced more and more pain maintaining a green master, I've compiled some statistics about which tests are most flaky. I thought other people might also be interested to have a look at that data: Last Week: # CI Statistics since 2018-10-05 14:22:35.422882 for branches containing 'asf/master' Total: 41 failing tests, 28 unique. (avg 0.142361111111 failing tests per build) Top 5 failing tests: 6x: [empty] 4x: ResourceStatistics 2x: CreateDestroyDiskRecovery 2x: INTERNET_CURL_InvokeFetchByName 2x: RecoverNestedContainer Last Month: # CI Statistics since 2018-09-12 14:23:36.272031 for branches containing 'asf/master' Total: 320 failing tests, 75 unique. (avg 0.285714285714 failing tests per build) Top 5 failing tests: 57x: Used 32x: LongLivedDefaultExecutorRestart 27x: PythonFramework 23x: ROOT_CGROUPS_LaunchNestedContainerSessionsInParallel 22x: ResourceStatistics Last year: # CI Statistics since 2017-10-12 14:24:31.639792 for branches containing 'asf/master' Total: 3045 failing tests, 225 unique. (avg 0.184054642166 failing tests per build) Top 5 failing tests: 292x: [empty] 272x: ROOT_LOGROTATE_UNPRIVILEGED_USER_RotateWithSwitchUserTrueOrFalse 136x: LOGROTATE_RotateInSandbox 136x: LOGROTATE_CustomRotateOptions 131x: ResourceStatistics I don't really have a point with all of this, but some observations: - [empty] means that the `mesos-tests` binary crashed - The data also includes "real", i.e. non-flaky test failures, but they should not appear in the top 5 lists because we would hopefully either revert or fix them before they can accumulate dozens of failures - Over the whole year, we seem to be pretty good at fixing the nastiest flakes, with only one of the top 5 still appearing in this weeks test results - Sadly, the fail percentage isn't as different between now and then as we might have hoped. Hope this was interesting, and best regards, -- Benno Evers Software Engineer, Mesosphere