Just want to make sure everyone sees this. Please voice yourself now if you care about layout test fixing process and about managing test list process.
I'll give another day. Unless I hear objections, I'll make run_webkit_tests do option 3. I'm not quite sure how we transition from the current world with no bug ids to a future world with a bug id for each test, but I'll figure that out. Ojan On Wed, Mar 25, 2009 at 3:51 PM, David Levin <[email protected]> wrote: > We could go with option 3 until someone is annoyed enough by the overhead > to write a script. :) > > > > On Tue, Mar 24, 2009 at 4:19 PM, Ojan Vafai <[email protected]> wrote: > >> OK. So, what I'm hearing is that every test should have a bug assigned to >> it, no matter the priority. In that case, there's a couple other options. >> >> *Option 3* >> Get rid of DEFER and don't add priorities to the test list. Instead >> require that every test have an associated bug (multiple tests can share a >> bug) and rely on the bug priority/owner to figure out when the test needs >> fixing and who is responsible for fixing it. >> Pros: >> -Works with our current bug triage process (kind of) >> -Makes for one easy place that people need to look for their todo list >> (the google code issue tracker) >> Cons: >> -Overhead of filing and closing bugs when the common case is just a >> rebaseline anyways >> -Hard to triage layout tests without understanding what's wrong with them >> >> *Option 4* >> Same as option 3, except we have a script that monitors the test list and >> automatically files a bug whenever a new test appears. The subject of the >> test is just the path listed in the test list, so the test can be found by >> searching the issue tracker. Similarly, when a test is removed from the test >> list, the bug is automatically closed. >> >> This has the same pros and cons as option 3 except that it totally removes >> the overhead associated with having a bug for each test path. Also, this >> would require someone to write the script to do this. >> >> Ojan >> >> On Tue, Mar 24, 2009 at 12:31 PM, David Levin <[email protected]> wrote: >> > I like this proposal. I would also like a bugs for P3 which could >> explain >> > why it is a P3. If is it an unimplemented feature, then all tests for >> that >> > unimplemented feature could have the same bug. >> > (Since I do merges and took a while to layout test file bugs from that) >> I >> > like option 2. BUT I'm concerned that adding someone's email address to >> > test_fixable will not get any attention. Right now, when you file a >> bug, >> > people get email and the bugs seem to be followed up. >> > >> > BUGS/PRIORITY TRIAGING >> > Option 1 >> > >> > Addition Pro >> > >> > Email sent about new bug alerts people to the new issue -- I suppose one >> > could email people separately when adding their email address to >> > test_fixable (but this step could easily be missed). >> > >> > Dave >> > >> > >> > On Tue, Mar 24, 2009 at 12:14 PM, Marc-Andre Decoste <[email protected]> >> > wrote: >> >> >> >> I personally prefer having a bug assigned for the layout tests that we >> >> want to be fixed soon... It makes for a more consistent way of >> following up >> >> on our progress... Even if the end result is just a re-baseline, we >> also >> >> gain the link to the bug from the committed change list (and vice >> versa). >> >> And if we want some sort of dashboard for this, we could add a page on >> the >> >> chromium-status appEngine that would read from the latest version of >> the >> >> test list file, and maybe some details (e.g., owner) from the issue >> >> tracker... Maybe... ;-) >> >> >> >> BYE >> >> MAD >> >> On Tue, Mar 24, 2009 at 2:57 PM, Ojan Vafai <[email protected]> wrote: >> >>> >> >>> I'm going about adding support to the webkit test list for BUGxxxxx >> >>> metadata and replacing DEFER with P0/P1/P2/P3. I've come to the >> conclusion >> >>> that we need to better understand our desired workflow for dealing >> with >> >>> failing layout tests. >> >>> TEST OWNERSHIP: >> >>> Firstly, can we move away from using the spreadsheet to take ownership >> of >> >>> layout tests and just put our names directly in the test list? This >> seems >> >>> way more intuitive to me and avoids needing to look at multiple >> locations >> >>> for the state of the layout tests. I like having a read-only dashboard >> that >> >>> present this information in a useful way like the spreadsheet >> currently >> >>> does, but there should only be one place we modify. >> >>> BUGS/PRIORITY TRIAGING >> >>> Option 1 >> >>> Every P0/P1/P2 test is required to have a bug id associated with it. >> New >> >>> failures get the UNTRIAGED property (does not require a bug id >> perhaps?). >> >>> The people who triage bugs also triage the UNTRIAGED layout tests and >> assign >> >>> them a priority (and file a bug?). The bugs are then fixed via our >> normal >> >>> bug triage process. People own fixing a given layout test by becoming >> the >> >>> owner of the bug. >> >>> Pros: >> >>> -Works with our current bug triage process (kind of) >> >>> -Makes for one easy place that people need to look for their todo list >> >>> (the google code issue tracker) >> >>> Cons: >> >>> -Overhead of filing and closing bugs when the common case is just a >> >>> rebaseline anyways >> >>> -Hard to triage layout tests without understanding what's wrong with >> them >> >>> Option 2 >> >>> The same as above, except that bug ids are not required. Bug ids are >> just >> >>> for cases where someone has looked into a test and needs to provide >> >>> information about why/how it's failing, but can't fix it immediately. >> People >> >>> become owners for a given layout test by putting their name as one of >> the >> >>> metadata bits for that test. Like Option 1, there needs to be a triage >> >>> process to assign priorities. >> >>> Pros: >> >>> -Minimizes overhead of managing layout tests >> >>> Cons: >> >>> -Does not work with our current bug process >> >>> -People need to look in two places to see what issues they need to fix >> >>> -Hard to triage layout tests without understanding what's wrong with >> them >> >>> -Does not send people an email when they get assigned to a test (can >> be >> >>> fixed by a simple script though) >> >>> I lean towards option 2. There is so much churn with layout tests that >> >>> adding overhead for each failure actually adds a good deal of >> unnecessary >> >>> work. In terms of bug triage, I think it needs to happens slightly >> >>> differently than the current bug triage process anyways since it first >> >>> requires an engineer (the webkit sheriff?) to find out why each test >> is >> >>> failing. >> >>> PRIORITIES >> >>> P0 - Something is catostrophically wrong and should be fixed now. >> >>> P1 - Regresssions and tests that have known or likely impact on real >> >>> sites. Blocks next milestone release. >> >>> P2 - Tests that we should be passing or for smallish features that we >> >>> really should have implemented by now (e.g. input type=search) >> >>> P3 - Tests for large features that we have yet to turn on (e.g. >> workers) >> >>> Finaly, it's not clear to me that we need an explicit UNTRIAGED >> property. >> >>> If a test lacks a priority, then it's clearly untriaged. >> >>> Ojan >> >>> >> >> >> >> >> >> >> >> >> > >> > >> >> > --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
