This matches my understanding. As a transition plan, I'm thinking to replace DEFER with UNTRIAGED. That way there is a way we can keep from adding 400 "this test has no bug id" warnings until we add bugs for all the currently deferred tests. Ojan
On Thu, Mar 26, 2009 at 3:28 PM, Pam Greene <[email protected]> wrote: > Just to make sure I understand correctly, the model here is that each test > has a BUG12345 note, possibly shared with other tests. But it doesn't have a > name or priority, not even the (ambiguous) priority implied by DEFER, > directly in the list. Instead, we use the bug tracker to track all that, > that being what it as designed for, after all. > > Sounds good to me. I can think of several advantages to this plan not > listed in your original email, and no additional disadvantages. (And as for > the overhead of filing and closing bugs for a rebaseline, if it's clear a > priori that that's all that's needed, you can always choose to do the > rebaseline instead.) > > The merger, or whoever is adding a test to the list, should still be > responsible for a preliminary investigation. Bug reports should be more than > "test a/b/c.html is failing". At a minimum, "started failing in r34567"; > better, a sentence or two such as "started failing in r34567, which looks > like David's change to the zorkmid system", or "new test in r34567, will > fail until we implement FrobozzClient". (So any script that does this > should ask for a description, not just file a bare bug.) > > - Pam > > > On Wed, Mar 25, 2009 at 4:00 PM, Ojan Vafai <[email protected]> wrote: > >> Just want to make sure everyone sees this. Please voice yourself now if >> you care about layout test fixing process and about managing test list >> process. >> >> I'll give another day. Unless I hear objections, I'll make >> run_webkit_tests do option 3. I'm not quite sure how we transition from the >> current world with no bug ids to a future world with a bug id for each test, >> but I'll figure that out. >> >> Ojan >> >> On Wed, Mar 25, 2009 at 3:51 PM, David Levin <[email protected]> wrote: >> >>> We could go with option 3 until someone is annoyed enough by the overhead >>> to write a script. :) >>> >>> >>> >>> On Tue, Mar 24, 2009 at 4:19 PM, Ojan Vafai <[email protected]> wrote: >>> >>>> OK. So, what I'm hearing is that every test should have a bug assigned >>>> to it, no matter the priority. In that case, there's a couple other >>>> options. >>>> >>>> *Option 3* >>>> Get rid of DEFER and don't add priorities to the test list. Instead >>>> require that every test have an associated bug (multiple tests can share a >>>> bug) and rely on the bug priority/owner to figure out when the test needs >>>> fixing and who is responsible for fixing it. >>>> Pros: >>>> -Works with our current bug triage process (kind of) >>>> -Makes for one easy place that people need to look for their todo list >>>> (the google code issue tracker) >>>> Cons: >>>> -Overhead of filing and closing bugs when the common case is just a >>>> rebaseline anyways >>>> -Hard to triage layout tests without understanding what's wrong with >>>> them >>>> >>>> *Option 4* >>>> Same as option 3, except we have a script that monitors the test list >>>> and automatically files a bug whenever a new test appears. The subject of >>>> the test is just the path listed in the test list, so the test can be found >>>> by searching the issue tracker. Similarly, when a test is removed from the >>>> test list, the bug is automatically closed. >>>> >>>> This has the same pros and cons as option 3 except that it totally >>>> removes the overhead associated with having a bug for each test path. Also, >>>> this would require someone to write the script to do this. >>>> >>>> Ojan >>>> >>>> On Tue, Mar 24, 2009 at 12:31 PM, David Levin <[email protected]> wrote: >>>> > I like this proposal. I would also like a bugs for P3 which could >>>> explain >>>> > why it is a P3. If is it an unimplemented feature, then all tests for >>>> that >>>> > unimplemented feature could have the same bug. >>>> > (Since I do merges and took a while to layout test file bugs from >>>> that) I >>>> > like option 2. BUT I'm concerned that adding someone's email address >>>> to >>>> > test_fixable will not get any attention. Right now, when you file a >>>> bug, >>>> > people get email and the bugs seem to be followed up. >>>> > >>>> > BUGS/PRIORITY TRIAGING >>>> > Option 1 >>>> > >>>> > Addition Pro >>>> > >>>> > Email sent about new bug alerts people to the new issue -- I suppose >>>> one >>>> > could email people separately when adding their email address to >>>> > test_fixable (but this step could easily be missed). >>>> > >>>> > Dave >>>> > >>>> > >>>> > On Tue, Mar 24, 2009 at 12:14 PM, Marc-Andre Decoste < >>>> [email protected]> >>>> > wrote: >>>> >> >>>> >> I personally prefer having a bug assigned for the layout tests that >>>> we >>>> >> want to be fixed soon... It makes for a more consistent way of >>>> following up >>>> >> on our progress... Even if the end result is just a re-baseline, we >>>> also >>>> >> gain the link to the bug from the committed change list (and vice >>>> versa). >>>> >> And if we want some sort of dashboard for this, we could add a page >>>> on the >>>> >> chromium-status appEngine that would read from the latest version of >>>> the >>>> >> test list file, and maybe some details (e.g., owner) from the issue >>>> >> tracker... Maybe... ;-) >>>> >> >>>> >> BYE >>>> >> MAD >>>> >> On Tue, Mar 24, 2009 at 2:57 PM, Ojan Vafai <[email protected]> wrote: >>>> >>> >>>> >>> I'm going about adding support to the webkit test list for BUGxxxxx >>>> >>> metadata and replacing DEFER with P0/P1/P2/P3. I've come to the >>>> conclusion >>>> >>> that we need to better understand our desired workflow for dealing >>>> with >>>> >>> failing layout tests. >>>> >>> TEST OWNERSHIP: >>>> >>> Firstly, can we move away from using the spreadsheet to take >>>> ownership of >>>> >>> layout tests and just put our names directly in the test list? This >>>> seems >>>> >>> way more intuitive to me and avoids needing to look at multiple >>>> locations >>>> >>> for the state of the layout tests. I like having a read-only >>>> dashboard that >>>> >>> present this information in a useful way like the spreadsheet >>>> currently >>>> >>> does, but there should only be one place we modify. >>>> >>> BUGS/PRIORITY TRIAGING >>>> >>> Option 1 >>>> >>> Every P0/P1/P2 test is required to have a bug id associated with it. >>>> New >>>> >>> failures get the UNTRIAGED property (does not require a bug id >>>> perhaps?). >>>> >>> The people who triage bugs also triage the UNTRIAGED layout tests >>>> and assign >>>> >>> them a priority (and file a bug?). The bugs are then fixed via our >>>> normal >>>> >>> bug triage process. People own fixing a given layout test by >>>> becoming the >>>> >>> owner of the bug. >>>> >>> Pros: >>>> >>> -Works with our current bug triage process (kind of) >>>> >>> -Makes for one easy place that people need to look for their todo >>>> list >>>> >>> (the google code issue tracker) >>>> >>> Cons: >>>> >>> -Overhead of filing and closing bugs when the common case is just a >>>> >>> rebaseline anyways >>>> >>> -Hard to triage layout tests without understanding what's wrong with >>>> them >>>> >>> Option 2 >>>> >>> The same as above, except that bug ids are not required. Bug ids are >>>> just >>>> >>> for cases where someone has looked into a test and needs to provide >>>> >>> information about why/how it's failing, but can't fix it >>>> immediately. People >>>> >>> become owners for a given layout test by putting their name as one >>>> of the >>>> >>> metadata bits for that test. Like Option 1, there needs to be a >>>> triage >>>> >>> process to assign priorities. >>>> >>> Pros: >>>> >>> -Minimizes overhead of managing layout tests >>>> >>> Cons: >>>> >>> -Does not work with our current bug process >>>> >>> -People need to look in two places to see what issues they need to >>>> fix >>>> >>> -Hard to triage layout tests without understanding what's wrong with >>>> them >>>> >>> -Does not send people an email when they get assigned to a test (can >>>> be >>>> >>> fixed by a simple script though) >>>> >>> I lean towards option 2. There is so much churn with layout tests >>>> that >>>> >>> adding overhead for each failure actually adds a good deal of >>>> unnecessary >>>> >>> work. In terms of bug triage, I think it needs to happens slightly >>>> >>> differently than the current bug triage process anyways since it >>>> first >>>> >>> requires an engineer (the webkit sheriff?) to find out why each test >>>> is >>>> >>> failing. >>>> >>> PRIORITIES >>>> >>> P0 - Something is catostrophically wrong and should be fixed now. >>>> >>> P1 - Regresssions and tests that have known or likely impact on real >>>> >>> sites. Blocks next milestone release. >>>> >>> P2 - Tests that we should be passing or for smallish features that >>>> we >>>> >>> really should have implemented by now (e.g. input type=search) >>>> >>> P3 - Tests for large features that we have yet to turn on (e.g. >>>> workers) >>>> >>> Finaly, it's not clear to me that we need an explicit UNTRIAGED >>>> property. >>>> >>> If a test lacks a priority, then it's clearly untriaged. >>>> >>> Ojan >>>> >>> >>>> >> >>>> >> >>>> >> > >>>> > >>>> >>>> >>> >> >> >> >> > --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: [email protected] View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
