I think the question is most interesting in the abstract as a question of general policy about flakey tests. I think handling the circumstances of each bug is best left for discussion in the bug themselves. That said, I've attached a list of recently filed bugs about flakey tests, per your request.
-eric https://bugs.webkit.org/show_bug.cgi?id=29322 (mentioned in the original mail) https://bugs.webkit.org/show_bug.cgi?id=29505 (same root bug, I suspect) https://bugs.webkit.org/show_bug.cgi?id=29620 (being worked on, skipping not yet proposed) https://bugs.webkit.org/show_bug.cgi?id=28845 (OS bug, resulted in 2 skips so far) https://bugs.webkit.org/show_bug.cgi?id=28624 (same OS bug, not yet skipped) https://bugs.webkit.org/show_bug.cgi?id=29035 (same OS bug, not yet skipped) https://bugs.webkit.org/show_bug.cgi?id=29154 (still being investigated, skipping not yet proposed) On Wed, Sep 23, 2009 at 11:23 PM, Maciej Stachowiak <m...@apple.com> wrote: > > On Sep 23, 2009, at 11:09 PM, Eric Seidel wrote: > > Alexey and I have been discussing if WebKit should add flakey tests to > Skipped lists: > https://bugs.webkit.org/show_bug.cgi?id=29322 > > Alexey asked that I bring the discussion to a larger audience. > > Pros: > - Buildbots stay green. > - Red bots/tests means your change caused an error. > > Cons: > - Skipped tests may be more likely to be forgotten (and thus never fixed). > > What does WebKit think? Should we skip flakey tests which can't be > resolved in a timely manner (and instead track issues via bugs) as > WebKit policy? Or should we reserve the skipped list for other > conditions (like bugs in the OS)? > > Thoughts? > > I'm a little concerned about sweeping failures under the carpet, but on the > other hand the buildbots are much less valuable if they are red too much. > It's hard to think about this in the abstract. Do you have any concrete > examples? > - Maciej > _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
