If a test sometimes crashes or hangs, it'll still be disabled, right?-darin
On Tue, Oct 6, 2009 at 5:02 PM, Nicolas Sylvain <nsylv...@chromium.org>wrote:
> Hello,
> We currently have more than 50 unit tests that are disabled. Most of them
> because they were flaky.
>
> Disabling tests is bad because we lose complete coverage on them, so I
> implemented a way to mark
> tests as "flaky".
>
> The same way you disable a test with DISABLED_ at the beginning of its
> name, you can now mark
> is as flaky with FLAKY_. The behavior is exactly the same as any other
> running tests. You will still
> be able to see when it fails (and why). The only difference is that if
> only FLAKY_ tests failed, the
> buildbot/trybots won't consider it as a failure. On the waterfall, it will
> show the box as orange with the
> list of all flaky tests that failed (pending one more buildbot restart). On
> the console view it will stay
> green.
>
> But.. this is not a toy. Flaky tests are bad. We should mark tests flaky
> only if we really have to, and
> if you do, please make sure to file a P1 bug. Set the owner of the bug to
> whoever regressed the test.
> If you can't find who regressed the test, assign it to the person who
> originally wrote the test.
>
> Once we start tagging the flaky tests, we will monitor the flakiness
> dashboard and make sure
> that a test that is no longer flaky has its FLAKY_ tag removed.
>
> Let me know if you have questions.
>
> Thanks
>
> Nicolas
>
>
> >
>
--~--~---------~--~----~------------~-------~--~----~
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
-~----------~----~----~----~------~----~------~--~---
