I personally prefer having a bug assigned for the layout tests that we want to be fixed soon... It makes for a more consistent way of following up on our progress... Even if the end result is just a re-baseline, we also gain the link to the bug from the committed change list (and vice versa). And if we want some sort of dashboard for this, we could add a page on the chromium-status appEngine that would read from the latest version of the test list file, and maybe some details (e.g., owner) from the issue tracker... Maybe... ;-)
BYE MAD On Tue, Mar 24, 2009 at 2:57 PM, Ojan Vafai <[email protected]> wrote: > I'm going about adding support to the webkit test list for BUGxxxxx > metadata and replacing DEFER with P0/P1/P2/P3. I've come to the conclusion > that we need to better understand our desired workflow for dealing with > failing layout tests. > *TEST OWNERSHIP:* > Firstly, can we move away from using the spreadsheet to take ownership of > layout tests and just put our names directly in the test list? This seems > way more intuitive to me and avoids needing to look at multiple locations > for the state of the layout tests. I like having a read-only dashboard that > present this information in a useful way like the spreadsheet currently > does, but there should only be one place we modify. > > BUGS/PRIORITY TRIAGING > *Option 1* > Every P0/P1/P2 test is required to have a bug id associated with it. New > failures get the UNTRIAGED property (does not require a bug id perhaps?). > The people who triage bugs also triage the UNTRIAGED layout tests and assign > them a priority (and file a bug?). The bugs are then fixed via our normal > bug triage process. People own fixing a given layout test by becoming the > owner of the bug. > > Pros: > -Works with our current bug triage process (kind of) > -Makes for one easy place that people need to look for their todo list (the > google code issue tracker) > Cons: > -Overhead of filing and closing bugs when the common case is just a > rebaseline anyways > -Hard to triage layout tests without understanding what's wrong with them > > *Option 2* > The same as above, except that bug ids are not required. Bug ids are just > for cases where someone has looked into a test and needs to provide > information about why/how it's failing, but can't fix it immediately. People > become owners for a given layout test by putting their name as one of the > metadata bits for that test. Like Option 1, there needs to be a triage > process to assign priorities. > > Pros: > -Minimizes overhead of managing layout tests > Cons: > -Does not work with our current bug process > -People need to look in two places to see what issues they need to fix > -Hard to triage layout tests without understanding what's wrong with them > -Does not send people an email when they get assigned to a test (can be > fixed by a simple script though) > > I lean towards option 2. There is so much churn with layout tests that > adding overhead for each failure actually adds a good deal of unnecessary > work. In terms of bug triage, I think it needs to happens slightly > differently than the current bug triage process anyways since it first > requires an engineer (the webkit sheriff?) to find out why each test is > failing. > > *PRIORITIES* > P0 - Something is catostrophically wrong and should be fixed now. > P1 - Regresssions and tests that have known or likely impact on real sites. > Blocks next milestone release. > P2 - Tests that we should be passing or for smallish features that we > really should have implemented by now (e.g. input type=search) > P3 - Tests for large features that we have yet to turn on (e.g. workers) > > Finaly, it's not clear to me that we need an explicit UNTRIAGED property. > If a test lacks a priority, then it's clearly untriaged. > > Ojan > > > > --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
