We could rerun any unexpected fail/crash/timeout tests. If they pass the
second time around, then the tree turns orange instead of red. This has come
up many times, but we've never agreed on whether it's a good idea.
Pros:
-Easier to distinguish between flakiness and new failures
-Tree will be
We did this on my last project to deal with flaky test infrastructure. It
worked well in that test failures were pretty much guaranteed to be real (we
ran tests 3 times and only reported failure if a test failed all 3 times),
but it did definitely make us stop caring about flaky tests.
On Tue, Oct 13, 2009 at 1:00 PM, Julie Parent jpar...@chromium.org wrote:
We did this on my last project to deal with flaky test infrastructure. It
worked well in that test failures were pretty much guaranteed to be real (we
ran tests 3 times and only reported failure if a test failed all 3
