We did this on my last project to deal with flaky test infrastructure. It worked well in that test failures were pretty much guaranteed to be real (we ran tests 3 times and only reported failure if a test failed all 3 times), but it did definitely make us stop caring about flaky tests.
Idealizing this some more, what if it could auto-add flaky tests to test_expectations, or at least generate a daily report that someone could then manually add? This would make us keep track of the flakiness and would only have the slowdown in test run times until a flaky test is added to test_expectations. On Tue, Oct 13, 2009 at 12:02 PM, Ojan Vafai <[email protected]> wrote: > We could rerun any unexpected fail/crash/timeout tests. If they pass the > second time around, then the tree turns orange instead of red. This has come > up many times, but we've never agreed on whether it's a good idea. > > Pros: > -Easier to distinguish between flakiness and new failures > -Tree will be much greener > -Try bots will be much more reliable > -Increase in overall team sanity > > Cons: > -Almost guaranteed to sweep some new flakiness under the rug > -If enough flakiness accumulates it will affect cycle time (each > timeout/crash test we rerun takes seconds) > > Shouldn't be too hard to implement. > > Ojan > > On Tue, Oct 13, 2009 at 11:32 AM, Peter Kasting <[email protected]>wrote: > >> On Tue, Oct 13, 2009 at 11:17 AM, Nicolas Sylvain >> <[email protected]>wrote: >> >>> It's because the sheriff don't notice the new failing tests, because most >>> of the time the gardener does a good job of updating the list at the same >>> time as the merge, so the tree stays mostly green. >>> See http://src.chromium.org/viewvc/chrome?view=rev&revision=28727 >>> >> >> In my case I was watching jparent's commits, so I stand by my assertion. >> >> PK >> >> >> > > > > --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
