OK, thanks for the clarification. I'll starting whinging at people who disable tests without a comment pointing at an associated bug.
-atw On Mon, Dec 14, 2009 at 10:52 AM, Darin Fisher <da...@chromium.org> wrote: > On Mon, Dec 14, 2009 at 10:42 AM, Drew Wilson <atwil...@chromium.org>wrote: > >> They'll sometimes get disabled due to webkit updates, other times they'll >> get disabled due to other things (for example, we changed the valgrind bots >> to fail noisily if individual tests fail, regardless of whether they >> actually generate valgrind errors - this meant that previously-silent worker >> test failures suddenly started causing redness, leading to sheriffs >> disabling them). >> >> But, yeah, it'd be nice to have ui_tests run by the webkit.org bots, >> although in the case of flaky tests I'm not sure whether that'd help (not >> sure if the gardener would pick up on a 25% failure on the FYI bots). >> > > We have to start somewhere. That we have no coverage for worker related > things in the layout tests means that the gardener is flying blind w.r.t. > workers. I think it should be a priority for the canary bots to provide > coverage for all webkit features. > > > >> >> At this point, I'm just trying to figure out what people are *supposed* to >> do when disabling tests - should they always log a bug and add a comment? >> > > Yes. That has always been the rule. > -Darin > > > >> I'd say yes, that if you have time to babysit a CL through the build bot >> process, you have time to log a bug/add a comment, even if you don't know >> the correct owner. >> >> -atw >> >> >> On Mon, Dec 14, 2009 at 10:28 AM, Darin Fisher <da...@chromium.org>wrote: >> >>> I presume it is frequently the case that these tests get disabled after a >>> webkit update? >>> >>> Only the "Linux Perf (webkit.org)" bot appears to run the ui_tests. >>> Perhaps that is not sufficient? >>> >>> -Darin >>> >>> >>> >>> On Mon, Dec 14, 2009 at 8:54 AM, Drew Wilson <atwil...@chromium.org>wrote: >>> >>>> I spent a few hours last week and this weekend trying to untangle the >>>> mess that was the worker ui_tests. The problem is that the tests have been >>>> sporadically flakey due to various causes, and so various sheriffs/good >>>> samaritans have disabled them to keep the trees green. Some of the tests >>>> were disabled due to failing under valgrind, but now that we have a way to >>>> disable tests specifically for valgrind and some of the worker bugs have >>>> been fixed upstream I figured it was a good time to clean house a bit and >>>> re-enable some of the tests. >>>> >>>> I found when I was going through the worker_uitest file that it was hard >>>> to figure out why a given test was disabled, so it was unclear whether it >>>> was safe to re-enable them - some of the tests had comments pointing at a >>>> tracking bug, but some of them didn't, and it was a pain to track the root >>>> cause especially since the specific lines of code had sometimes been >>>> touched >>>> multiple times (adding new platforms to disable, etc). >>>> >>>> Anyhow, what's our best practices for disabling tests? I think ideally >>>> we'd always log a tracking bug and add a comment, akin to what we do in the >>>> test_expectations file for layout tests. This might be too much of a burden >>>> on sheriffs, so an alternative is for people who are working on various >>>> areas (like workers) track checkins to the associated files and add some >>>> documentation after the fact. Or we can just live with the SVN logs, in >>>> which case I need to get better about figuring out how to track through the >>>> SVN/git history of the various files :) >>>> >>>> -atw >>>> >>>> -- >>>> Chromium Developers mailing list: chromium-dev@googlegroups.com >>>> View archives, change email options, or unsubscribe: >>>> http://groups.google.com/group/chromium-dev >>>> >>> >>> -- >>> Chromium Developers mailing list: chromium-dev@googlegroups.com >>> View archives, change email options, or unsubscribe: >>> http://groups.google.com/group/chromium-dev >>> >> >> > -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
