I'm going about adding support to the webkit test list for BUGxxxxx metadata and replacing DEFER with P0/P1/P2/P3. I've come to the conclusion that we need to better understand our desired workflow for dealing with failing layout tests. *TEST OWNERSHIP:* Firstly, can we move away from using the spreadsheet to take ownership of layout tests and just put our names directly in the test list? This seems way more intuitive to me and avoids needing to look at multiple locations for the state of the layout tests. I like having a read-only dashboard that present this information in a useful way like the spreadsheet currently does, but there should only be one place we modify.
BUGS/PRIORITY TRIAGING *Option 1* Every P0/P1/P2 test is required to have a bug id associated with it. New failures get the UNTRIAGED property (does not require a bug id perhaps?). The people who triage bugs also triage the UNTRIAGED layout tests and assign them a priority (and file a bug?). The bugs are then fixed via our normal bug triage process. People own fixing a given layout test by becoming the owner of the bug. Pros: -Works with our current bug triage process (kind of) -Makes for one easy place that people need to look for their todo list (the google code issue tracker) Cons: -Overhead of filing and closing bugs when the common case is just a rebaseline anyways -Hard to triage layout tests without understanding what's wrong with them *Option 2* The same as above, except that bug ids are not required. Bug ids are just for cases where someone has looked into a test and needs to provide information about why/how it's failing, but can't fix it immediately. People become owners for a given layout test by putting their name as one of the metadata bits for that test. Like Option 1, there needs to be a triage process to assign priorities. Pros: -Minimizes overhead of managing layout tests Cons: -Does not work with our current bug process -People need to look in two places to see what issues they need to fix -Hard to triage layout tests without understanding what's wrong with them -Does not send people an email when they get assigned to a test (can be fixed by a simple script though) I lean towards option 2. There is so much churn with layout tests that adding overhead for each failure actually adds a good deal of unnecessary work. In terms of bug triage, I think it needs to happens slightly differently than the current bug triage process anyways since it first requires an engineer (the webkit sheriff?) to find out why each test is failing. *PRIORITIES* P0 - Something is catostrophically wrong and should be fixed now. P1 - Regresssions and tests that have known or likely impact on real sites. Blocks next milestone release. P2 - Tests that we should be passing or for smallish features that we really should have implemented by now (e.g. input type=search) P3 - Tests for large features that we have yet to turn on (e.g. workers) Finaly, it's not clear to me that we need an explicit UNTRIAGED property. If a test lacks a priority, then it's clearly untriaged. Ojan --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
