I like this proposal. *I would also like a bugs for P3* which could explain
why it is a P3. If is it an unimplemented feature, then all tests for that
unimplemented feature could have the same bug.
(Since I do merges and took a while to layout test file bugs from that) I
like option 2. *BUT I'm concerned that adding someone's email address to
test_fixable will not get any attention. Right now, when you file a bug,
people get email and the bugs seem to be followed up.*
*
*
BUGS/PRIORITY TRIAGING
Option 1
Addition Pro
Email sent about new bug alerts people to the new issue -- I suppose one
could email people separately when adding their email address to
test_fixable (but this step could easily be missed).
Dave
On Tue, Mar 24, 2009 at 12:14 PM, Marc-Andre Decoste <m...@chromium.org>wrote:
> I personally prefer having a bug assigned for the layout tests that we want
> to be fixed soon... It makes for a more consistent way of following up on
> our progress... Even if the end result is just a re-baseline, we also gain
> the link to the bug from the committed change list (and vice versa).
> And if we want some sort of dashboard for this, we could add a page on the
> chromium-status appEngine that would read from the latest version of the
> test list file, and maybe some details (e.g., owner) from the issue
> tracker... Maybe... ;-)
>
> BYE
> MAD
>
> On Tue, Mar 24, 2009 at 2:57 PM, Ojan Vafai <o...@google.com> wrote:
>
>> I'm going about adding support to the webkit test list for BUGxxxxx
>> metadata and replacing DEFER with P0/P1/P2/P3. I've come to the conclusion
>> that we need to better understand our desired workflow for dealing with
>> failing layout tests.
>> *TEST OWNERSHIP:*
>> Firstly, can we move away from using the spreadsheet to take ownership of
>> layout tests and just put our names directly in the test list? This seems
>> way more intuitive to me and avoids needing to look at multiple locations
>> for the state of the layout tests. I like having a read-only dashboard that
>> present this information in a useful way like the spreadsheet currently
>> does, but there should only be one place we modify.
>>
>> BUGS/PRIORITY TRIAGING
>> *Option 1*
>> Every P0/P1/P2 test is required to have a bug id associated with it. New
>> failures get the UNTRIAGED property (does not require a bug id perhaps?).
>> The people who triage bugs also triage the UNTRIAGED layout tests and assign
>> them a priority (and file a bug?). The bugs are then fixed via our normal
>> bug triage process. People own fixing a given layout test by becoming the
>> owner of the bug.
>>
>> Pros:
>> -Works with our current bug triage process (kind of)
>> -Makes for one easy place that people need to look for their todo list
>> (the google code issue tracker)
>> Cons:
>> -Overhead of filing and closing bugs when the common case is just a
>> rebaseline anyways
>> -Hard to triage layout tests without understanding what's wrong with them
>>
>> *Option 2*
>> The same as above, except that bug ids are not required. Bug ids are just
>> for cases where someone has looked into a test and needs to provide
>> information about why/how it's failing, but can't fix it immediately. People
>> become owners for a given layout test by putting their name as one of the
>> metadata bits for that test. Like Option 1, there needs to be a triage
>> process to assign priorities.
>>
>> Pros:
>> -Minimizes overhead of managing layout tests
>> Cons:
>> -Does not work with our current bug process
>> -People need to look in two places to see what issues they need to fix
>> -Hard to triage layout tests without understanding what's wrong with them
>> -Does not send people an email when they get assigned to a test (can be
>> fixed by a simple script though)
>>
>> I lean towards option 2. There is so much churn with layout tests that
>> adding overhead for each failure actually adds a good deal of unnecessary
>> work. In terms of bug triage, I think it needs to happens slightly
>> differently than the current bug triage process anyways since it first
>> requires an engineer (the webkit sheriff?) to find out why each test is
>> failing.
>>
>> *PRIORITIES*
>> P0 - Something is catostrophically wrong and should be fixed now.
>> P1 - Regresssions and tests that have known or likely impact on real
>> sites. Blocks next milestone release.
>> P2 - Tests that we should be passing or for smallish features that we
>> really should have implemented by now (e.g. input type=search)
>> P3 - Tests for large features that we have yet to turn on (e.g. workers)
>>
>> Finaly, it's not clear to me that we need an explicit UNTRIAGED property.
>> If a test lacks a priority, then it's clearly untriaged.
>>
>> Ojan
>>
>>
>>
>
> >
>
--~--~---------~--~----~------------~-------~--~----~
Chromium Developers mailing list: chromium-dev@googlegroups.com
View archives, change email options, or unsubscribe:
http://groups.google.com/group/chromium-dev
-~----------~----~----~----~------~----~------~--~---
