We could go with option 3 until someone is annoyed enough by the overhead to write a script. :)
On Tue, Mar 24, 2009 at 4:19 PM, Ojan Vafai <[email protected]> wrote: > OK. So, what I'm hearing is that every test should have a bug assigned to > it, no matter the priority. In that case, there's a couple other options. > > *Option 3* > Get rid of DEFER and don't add priorities to the test list. Instead require > that every test have an associated bug (multiple tests can share a bug) and > rely on the bug priority/owner to figure out when the test needs fixing and > who is responsible for fixing it. > Pros: > -Works with our current bug triage process (kind of) > -Makes for one easy place that people need to look for their todo list (the > google code issue tracker) > Cons: > -Overhead of filing and closing bugs when the common case is just a > rebaseline anyways > -Hard to triage layout tests without understanding what's wrong with them > > *Option 4* > Same as option 3, except we have a script that monitors the test list and > automatically files a bug whenever a new test appears. The subject of the > test is just the path listed in the test list, so the test can be found by > searching the issue tracker. Similarly, when a test is removed from the test > list, the bug is automatically closed. > > This has the same pros and cons as option 3 except that it totally removes > the overhead associated with having a bug for each test path. Also, this > would require someone to write the script to do this. > > Ojan > > On Tue, Mar 24, 2009 at 12:31 PM, David Levin <[email protected]> wrote: > > I like this proposal. I would also like a bugs for P3 which could > explain > > why it is a P3. If is it an unimplemented feature, then all tests for > that > > unimplemented feature could have the same bug. > > (Since I do merges and took a while to layout test file bugs from that) I > > like option 2. BUT I'm concerned that adding someone's email address to > > test_fixable will not get any attention. Right now, when you file a bug, > > people get email and the bugs seem to be followed up. > > > > BUGS/PRIORITY TRIAGING > > Option 1 > > > > Addition Pro > > > > Email sent about new bug alerts people to the new issue -- I suppose one > > could email people separately when adding their email address to > > test_fixable (but this step could easily be missed). > > > > Dave > > > > > > On Tue, Mar 24, 2009 at 12:14 PM, Marc-Andre Decoste <[email protected]> > > wrote: > >> > >> I personally prefer having a bug assigned for the layout tests that we > >> want to be fixed soon... It makes for a more consistent way of following > up > >> on our progress... Even if the end result is just a re-baseline, we also > >> gain the link to the bug from the committed change list (and vice > versa). > >> And if we want some sort of dashboard for this, we could add a page on > the > >> chromium-status appEngine that would read from the latest version of the > >> test list file, and maybe some details (e.g., owner) from the issue > >> tracker... Maybe... ;-) > >> > >> BYE > >> MAD > >> On Tue, Mar 24, 2009 at 2:57 PM, Ojan Vafai <[email protected]> wrote: > >>> > >>> I'm going about adding support to the webkit test list for BUGxxxxx > >>> metadata and replacing DEFER with P0/P1/P2/P3. I've come to the > conclusion > >>> that we need to better understand our desired workflow for dealing with > >>> failing layout tests. > >>> TEST OWNERSHIP: > >>> Firstly, can we move away from using the spreadsheet to take ownership > of > >>> layout tests and just put our names directly in the test list? This > seems > >>> way more intuitive to me and avoids needing to look at multiple > locations > >>> for the state of the layout tests. I like having a read-only dashboard > that > >>> present this information in a useful way like the spreadsheet currently > >>> does, but there should only be one place we modify. > >>> BUGS/PRIORITY TRIAGING > >>> Option 1 > >>> Every P0/P1/P2 test is required to have a bug id associated with it. > New > >>> failures get the UNTRIAGED property (does not require a bug id > perhaps?). > >>> The people who triage bugs also triage the UNTRIAGED layout tests and > assign > >>> them a priority (and file a bug?). The bugs are then fixed via our > normal > >>> bug triage process. People own fixing a given layout test by becoming > the > >>> owner of the bug. > >>> Pros: > >>> -Works with our current bug triage process (kind of) > >>> -Makes for one easy place that people need to look for their todo list > >>> (the google code issue tracker) > >>> Cons: > >>> -Overhead of filing and closing bugs when the common case is just a > >>> rebaseline anyways > >>> -Hard to triage layout tests without understanding what's wrong with > them > >>> Option 2 > >>> The same as above, except that bug ids are not required. Bug ids are > just > >>> for cases where someone has looked into a test and needs to provide > >>> information about why/how it's failing, but can't fix it immediately. > People > >>> become owners for a given layout test by putting their name as one of > the > >>> metadata bits for that test. Like Option 1, there needs to be a triage > >>> process to assign priorities. > >>> Pros: > >>> -Minimizes overhead of managing layout tests > >>> Cons: > >>> -Does not work with our current bug process > >>> -People need to look in two places to see what issues they need to fix > >>> -Hard to triage layout tests without understanding what's wrong with > them > >>> -Does not send people an email when they get assigned to a test (can be > >>> fixed by a simple script though) > >>> I lean towards option 2. There is so much churn with layout tests that > >>> adding overhead for each failure actually adds a good deal of > unnecessary > >>> work. In terms of bug triage, I think it needs to happens slightly > >>> differently than the current bug triage process anyways since it first > >>> requires an engineer (the webkit sheriff?) to find out why each test is > >>> failing. > >>> PRIORITIES > >>> P0 - Something is catostrophically wrong and should be fixed now. > >>> P1 - Regresssions and tests that have known or likely impact on real > >>> sites. Blocks next milestone release. > >>> P2 - Tests that we should be passing or for smallish features that we > >>> really should have implemented by now (e.g. input type=search) > >>> P3 - Tests for large features that we have yet to turn on (e.g. > workers) > >>> Finaly, it's not clear to me that we need an explicit UNTRIAGED > property. > >>> If a test lacks a priority, then it's clearly untriaged. > >>> Ojan > >>> > >> > >> > >> > >> > > > > > > --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
