OK. So, what I'm hearing is that every test should have a bug assigned to it, no matter the priority. In that case, there's a couple other options.
*Option 3* Get rid of DEFER and don't add priorities to the test list. Instead require that every test have an associated bug (multiple tests can share a bug) and rely on the bug priority/owner to figure out when the test needs fixing and who is responsible for fixing it. Pros: -Works with our current bug triage process (kind of) -Makes for one easy place that people need to look for their todo list (the google code issue tracker) Cons: -Overhead of filing and closing bugs when the common case is just a rebaseline anyways -Hard to triage layout tests without understanding what's wrong with them *Option 4* Same as option 3, except we have a script that monitors the test list and automatically files a bug whenever a new test appears. The subject of the test is just the path listed in the test list, so the test can be found by searching the issue tracker. Similarly, when a test is removed from the test list, the bug is automatically closed. This has the same pros and cons as option 3 except that it totally removes the overhead associated with having a bug for each test path. Also, this would require someone to write the script to do this. Ojan On Tue, Mar 24, 2009 at 12:31 PM, David Levin <[email protected]> wrote: > I like this proposal. I would also like a bugs for P3 which could explain > why it is a P3. If is it an unimplemented feature, then all tests for that > unimplemented feature could have the same bug. > (Since I do merges and took a while to layout test file bugs from that) I > like option 2. BUT I'm concerned that adding someone's email address to > test_fixable will not get any attention. Right now, when you file a bug, > people get email and the bugs seem to be followed up. > > BUGS/PRIORITY TRIAGING > Option 1 > > Addition Pro > > Email sent about new bug alerts people to the new issue -- I suppose one > could email people separately when adding their email address to > test_fixable (but this step could easily be missed). > > Dave > > > On Tue, Mar 24, 2009 at 12:14 PM, Marc-Andre Decoste <[email protected]> > wrote: >> >> I personally prefer having a bug assigned for the layout tests that we >> want to be fixed soon... It makes for a more consistent way of following up >> on our progress... Even if the end result is just a re-baseline, we also >> gain the link to the bug from the committed change list (and vice versa). >> And if we want some sort of dashboard for this, we could add a page on the >> chromium-status appEngine that would read from the latest version of the >> test list file, and maybe some details (e.g., owner) from the issue >> tracker... Maybe... ;-) >> >> BYE >> MAD >> On Tue, Mar 24, 2009 at 2:57 PM, Ojan Vafai <[email protected]> wrote: >>> >>> I'm going about adding support to the webkit test list for BUGxxxxx >>> metadata and replacing DEFER with P0/P1/P2/P3. I've come to the conclusion >>> that we need to better understand our desired workflow for dealing with >>> failing layout tests. >>> TEST OWNERSHIP: >>> Firstly, can we move away from using the spreadsheet to take ownership of >>> layout tests and just put our names directly in the test list? This seems >>> way more intuitive to me and avoids needing to look at multiple locations >>> for the state of the layout tests. I like having a read-only dashboard that >>> present this information in a useful way like the spreadsheet currently >>> does, but there should only be one place we modify. >>> BUGS/PRIORITY TRIAGING >>> Option 1 >>> Every P0/P1/P2 test is required to have a bug id associated with it. New >>> failures get the UNTRIAGED property (does not require a bug id perhaps?). >>> The people who triage bugs also triage the UNTRIAGED layout tests and assign >>> them a priority (and file a bug?). The bugs are then fixed via our normal >>> bug triage process. People own fixing a given layout test by becoming the >>> owner of the bug. >>> Pros: >>> -Works with our current bug triage process (kind of) >>> -Makes for one easy place that people need to look for their todo list >>> (the google code issue tracker) >>> Cons: >>> -Overhead of filing and closing bugs when the common case is just a >>> rebaseline anyways >>> -Hard to triage layout tests without understanding what's wrong with them >>> Option 2 >>> The same as above, except that bug ids are not required. Bug ids are just >>> for cases where someone has looked into a test and needs to provide >>> information about why/how it's failing, but can't fix it immediately. People >>> become owners for a given layout test by putting their name as one of the >>> metadata bits for that test. Like Option 1, there needs to be a triage >>> process to assign priorities. >>> Pros: >>> -Minimizes overhead of managing layout tests >>> Cons: >>> -Does not work with our current bug process >>> -People need to look in two places to see what issues they need to fix >>> -Hard to triage layout tests without understanding what's wrong with them >>> -Does not send people an email when they get assigned to a test (can be >>> fixed by a simple script though) >>> I lean towards option 2. There is so much churn with layout tests that >>> adding overhead for each failure actually adds a good deal of unnecessary >>> work. In terms of bug triage, I think it needs to happens slightly >>> differently than the current bug triage process anyways since it first >>> requires an engineer (the webkit sheriff?) to find out why each test is >>> failing. >>> PRIORITIES >>> P0 - Something is catostrophically wrong and should be fixed now. >>> P1 - Regresssions and tests that have known or likely impact on real >>> sites. Blocks next milestone release. >>> P2 - Tests that we should be passing or for smallish features that we >>> really should have implemented by now (e.g. input type=search) >>> P3 - Tests for large features that we have yet to turn on (e.g. workers) >>> Finaly, it's not clear to me that we need an explicit UNTRIAGED property. >>> If a test lacks a priority, then it's clearly untriaged. >>> Ojan >>> >> >> >> >> > > --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
