Green buildbots have a lot of value.
I think it’s worthwhile finding a way to have them even when there are
test failures.
For predictable failures, the best approach is to land the expected
failure as an expected result, and use a bug to track the fact that
it’s wrong. To me this does seem a bit like “sweeping something under
the rug”, a bug report is much easier to overlook than a red buildbot.
We don’t have a great system for keeping track of the most important
bugs.
For tests that give intermittent and inconsistent results, the best we
can currently do is to skip the test. I think it would make sense to
instead allow multiple expected results. I gather that one of the
tools used in the Chromium project has this concept and I think
there’s no real reason not to add the concept to run-webkit-tests as
long as we are conscientious about not using it when it’s not needed.
And use a bug to track the fact that the test gives insufficient
results. This has the same downsides as landing the expected failure
results.
For tests that have an adverse effect on other tests, the best we can
currently do is to skip the test.
I think we are overusing the Skipped machinery at the moment for
platform differences. I think in many cases it would be better to
instead land an expected failure result. On the other hand, one really
great thing about the Skipped file is that there’s a complete list in
the file, allowing everyone to see the list. It makes a good to do
list, probably better than just a list of bugs. This made Darin
Fisher’s recent “why are so many tests skipped, lets fix it” message
possible.
-- Darin
_______________________________________________
webkit-dev mailing list
webkit-dev@lists.webkit.org
http://lists.webkit.org/mailman/listinfo.cgi/webkit-dev
