On Fri, Sep 25, 2009 at 1:59 PM, Darin Adler <da...@apple.com> wrote: > For tests that give intermittent and inconsistent results, the best we can > currently do is to skip the test. I think it would make sense to instead > allow multiple expected results. I gather that one of the tools used in the > Chromium project has this concept and I think there’s no real reason not to > add the concept to run-webkit-tests as long as we are conscientious about > not using it when it’s not needed.
Not to derail the discussion, but to provide context for Darin's reply: Yes, Chromium's version of run-webkit-tests (called run_webkit_tests.py) has multiple-expected-results support (along with running tests in parallel and other goodness). But it's missing a bunch of the nifty flags WebKit's run-webkit-tests has. My hope is to eventually unify them, which we've filed some bugs on: http://code.google.com/p/chromium/issues/detail?id=23099 https://bugs.webkit.org/show_bug.cgi?id=10906 -eric On Fri, Sep 25, 2009 at 1:59 PM, Darin Adler <da...@apple.com> wrote: > Green buildbots have a lot of value. > > I think it’s worthwhile finding a way to have them even when there are test > failures. > > For predictable failures, the best approach is to land the expected failure > as an expected result, and use a bug to track the fact that it’s wrong. To > me this does seem a bit like “sweeping something under the rug”, a bug > report is much easier to overlook than a red buildbot. We don’t have a great > system for keeping track of the most important bugs. > > For tests that give intermittent and inconsistent results, the best we can > currently do is to skip the test. I think it would make sense to instead > allow multiple expected results. I gather that one of the tools used in the > Chromium project has this concept and I think there’s no real reason not to > add the concept to run-webkit-tests as long as we are conscientious about > not using it when it’s not needed. And use a bug to track the fact that the > test gives insufficient results. This has the same downsides as landing the > expected failure results. > > For tests that have an adverse effect on other tests, the best we can > currently do is to skip the test. > > I think we are overusing the Skipped machinery at the moment for platform > differences. I think in many cases it would be better to instead land an > expected failure result. On the other hand, one really great thing about the > Skipped file is that there’s a complete list in the file, allowing everyone > to see the list. It makes a good to do list, probably better than just a > list of bugs. This made Darin Fisher’s recent “why are so many tests > skipped, lets fix it” message possible. > > -- Darin > > _______________________________________________ > webkit-dev mailing list > webkit-dev@lists.webkit.org > http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev > _______________________________________________ webkit-dev mailing list webkit-dev@lists.webkit.org http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
