Excellent, thanks Stephen. I'm sure not all the slots will be filled. I'd say the best way to complete the list is lazily, in the software sense -- that is, when a test fails, if it has no owner, find the best person (or a random volunteer, if nobody has any knowledge about the topic) and then add them to the list.
- Pam On Wed, Oct 14, 2009 at 12:24 PM, Stephen White <senorbla...@chromium.org>wrote: > Ok, I have lousy docs-sharing-fu, but I gave it a shot. This should be > writeable by anyone @chromium.org: > > http://spreadsheets.google.com/a/chromium.org/ccc?key=0AmBv0BNymMh5dHlHZnJDSjR1X0ZzNnRYOFZTVWtvb0E&hl=en > > (To edit, you'll probably have to log into > http://docs.google.com/a/chromium.org first, then click the link above.). > > I basically split up the big directories, and the ones which obviously > contained more than one type of test. Feel free to split/merge at will. > > It's probably a good idea that each directory has more than one > expert/owner, so don't be shy. > > Stephen > > On Wed, Oct 14, 2009 at 2:43 PM, Dirk Pranke <dpra...@chromium.org> wrote: > >> +1 >> >> On Tue, Oct 13, 2009 at 10:20 PM, Pam Greene <p...@chromium.org> wrote: >> > If there are areas that nobody knows anything about, that's a lack >> that's >> > hobbling us. Suppose we take the entire list of directories, slap it >> into a >> > doc, and assign at least one owner to everything. For the areas that >> don't >> > yet have anyone knowledgeable, we take volunteers to >> become knowledgeable if >> > needed. It will be a valuable investment. >> > Tests often fail due to problems outside their nominal tested areas, but >> the >> > area owner would still be better than an arbitrary gardener at >> recognizing >> > that and reassigning the bug. >> > - Pam >> > >> > On Tue, Oct 13, 2009 at 10:09 PM, Dimitri Glazkov < >> dglaz...@chromium.org> >> > wrote: >> >> >> >> Ownership is a great concept. I started out planning LTTF as >> >> ownership-based. Unfortunately, the types of failures are scattered >> >> far and wide across the directories, some clustered, some not. After a >> >> few initial passes, I walked away thinking that it's not as simple as >> >> drawing the lines and basically gave up. That's how Finders/Fixers >> >> idea was born. >> >> >> >> :DG< >> >> >> >> On Tue, Oct 13, 2009 at 4:24 PM, Yaar Schnitman <y...@chromium.org> >> wrote: >> >> > I think ownership might actually help with flakiness. >> >> > Today, in order to distinguish flakiness from real bugs, the gardener >> >> > needs >> >> > to have intimate knowledge of the relevant part of the code base and >> its >> >> > history. That is beyond the capabilities of the average webkit >> gardener. >> >> > Now, imagine a world were every layout test has an owner who can >> decide >> >> > intelligently that the bug is flakey and advise the gardener what to >> do >> >> > with >> >> > it. Wouldn't it make gardening much easier? >> >> > [Flakiness dashboard is very helpful in making the decision, but >> >> > specialized >> >> > knowledge topples generic statistics, especially if a test just >> started >> >> > flaking] >> >> > On Tue, Oct 13, 2009 at 1:21 PM, Julie Parent <jpar...@chromium.org> >> >> > wrote: >> >> >> >> >> >> I like the idea of ownership of groups of layout tests. Maybe these >> >> >> directory "owners" could be more like the "finders"? An owner >> >> >> shouldn't >> >> >> have to necessarily fix everything in a group/directory, but they >> >> >> should be >> >> >> responsible for triaging and getting meaningful bugs filled for >> them, >> >> >> to >> >> >> keep things moving along. (I volunteer for editing/) >> >> >> Another complicating factor - >> >> >> The state of the main Chrome tree has a lot of effect on the >> gardener. >> >> >> If >> >> >> the tree is already filled with flakiness, then the webkit roll is >> >> >> likely to >> >> >> show failures, which may or may not have been there before the roll. >> >> >> This >> >> >> was largely the case in the situation pkasting was referring to, >> when >> >> >> he >> >> >> took over as sheriff, he inherited a tree with a lot of flakiness >> not >> >> >> reflected in test_expectations/disabled ui tests. I think very few >> (if >> >> >> any) >> >> >> of the tests he added to test_expectations had anything to >> >> >> do with the roll. >> >> >> Any policy we make needs to keep in mind that main tree sheriffs >> deal >> >> >> with >> >> >> flakiness differently; some cross their fingers and hope it goes >> away, >> >> >> and >> >> >> some do clean up. Maybe we need to get better at enforcing (or >> >> >> automating) >> >> >> adding flaky tests to expectations, so we at least have a clean >> slate >> >> >> for gardeners to start with. >> >> >> On Tue, Oct 13, 2009 at 11:53 AM, Stephen White >> >> >> <senorbla...@chromium.org> >> >> >> wrote: >> >> >>> >> >> >>> I agree with Dimitri that we're fighting a losing battle here. >> >> >>> In my last stint as gardener, I did informally what I proposed >> >> >>> formally >> >> >>> last time: I spent basically 1 full day just triaging failures >> from >> >> >>> my 2 >> >> >>> days gardening. Not fixing, but just running tests locally, >> >> >>> analyzing, >> >> >>> grouping, creating bugs, assigning to appropriate people (when I >> knew >> >> >>> who >> >> >>> they were, cc'ing poor dglazkov when I didn't). So at least I >> didn't >> >> >>> leave >> >> >>> a monster bug with "layout tests broken by merge #foo" but at least >> >> >>> grouped >> >> >>> by area. That was manageable, but I don't know if another day >> would >> >> >>> actually be enough for a meaningful amount of fixing. >> >> >>> I also agree with Drew that actively fixing all the broken tests is >> >> >>> usually beyond the skills of any one gardener. >> >> >>> Perhaps we should start taking ownership of particular groups of >> >> >>> layout >> >> >>> tests? And maybe automatically assign them (or least cc them), the >> >> >>> same way >> >> >>> Area-Foo causes automatic cc'ing in bugs.chromium.org (I think?) >> That >> >> >>> way, >> >> >>> the gardener wouldn't have to know who to assign to. >> >> >>> >> >> >>> I've basically taken responsibility for fixing all layout tests >> broken >> >> >>> by >> >> >>> Skia rolls, which can pretty heavy on its own, but I'm willing to >> take >> >> >>> ownership of a directory or two. >> >> >>> BTW, the layout test flakiness dashboard has become an invaluable >> tool >> >> >>> for analyzing failures: searching for a test by name is >> >> >>> lightning-fast, and >> >> >>> you can clearly see if a test has become flaky, on which platforms, >> >> >>> and >> >> >>> which WebKit merge was responsible, which can also help with >> grouping. >> >> >>> (Props to Ojan for that). >> >> >>> Also, it may be Gilbert-and-Sullivan-esque of me, but I think >> everyone >> >> >>> who contributes patches to WebKit for chromium should be on the >> WebKit >> >> >>> gardener rotation. >> >> >>> Stephen >> >> >>> >> >> >>> On Tue, Oct 13, 2009 at 1:53 PM, Drew Wilson < >> atwil...@chromium.org> >> >> >>> wrote: >> >> >>>> >> >> >>>> I've been thinking quite a bit about this - I agree with Dmitri >> that >> >> >>>> the >> >> >>>> current Sisyphean approach is unsustainable. >> >> >>>> I don't think the right path is to ask the sheriffs to do the >> cleanup >> >> >>>> themselves - for example, a webkit roll that breaks workers in >> some >> >> >>>> obscure >> >> >>>> way is almost certainly beyond the ability of any random gardener >> to >> >> >>>> fix in >> >> >>>> two days, especially when there may be multiple bugs. >> >> >>>> A better solution would be to have the sheriff (or someone from >> LTTF) >> >> >>>> assign the bugs to specific people, with a general rule that such >> >> >>>> bugs must >> >> >>>> be fixed within two days (make these bugs the top priority over >> other >> >> >>>> tasks). This allows for load balancing of bugs, and also makes >> sure >> >> >>>> that we >> >> >>>> have the right people working on any specific bug. >> >> >>>> -atw >> >> >>>> On Tue, Oct 13, 2009 at 10:40 AM, Pam Greene <p...@chromium.org> >> >> >>>> wrote: >> >> >>>>> >> >> >>>>> I don't think it's realistic to expect the gardener, or any one >> >> >>>>> person, >> >> >>>>> to be able to fix an arbitrary broken layout test in a reasonable >> >> >>>>> period of >> >> >>>>> time. That's certainly true for new tests, but even for >> regressions >> >> >>>>> I often >> >> >>>>> can't even tell for sure whether our results are correct, much >> less >> >> >>>>> what to >> >> >>>>> do if they're not. >> >> >>>>> It's far more efficient to have the "right" person fix a test. >> (Of >> >> >>>>> course, people should also strive to broaden their knowledge, but >> >> >>>>> there's a >> >> >>>>> limit to how much of that one can do in a week.) Never having >> broken >> >> >>>>> layout >> >> >>>>> tests is an excellent goal, but quite frankly I don't think it's >> one >> >> >>>>> we >> >> >>>>> should prioritize so high that we hobble other efforts and burn >> out >> >> >>>>> developers. >> >> >>>>> - Pam >> >> >>>>> On Tue, Oct 13, 2009 at 10:31 AM, Dimitri Glazkov >> >> >>>>> <dglaz...@chromium.org> wrote: >> >> >>>>>> >> >> >>>>>> I think we need to change something. I am not sure what -- I >> have >> >> >>>>>> ideas, but -- I would appreciate some collective thinking on >> this. >> >> >>>>>> >> >> >>>>>> PROBLEM: We accumulate more test failures via WebKit rolls than >> we >> >> >>>>>> fix >> >> >>>>>> with our LTTF effort. This ain't right. >> >> >>>>>> >> >> >>>>>> ANALYSIS: >> >> >>>>>> >> >> >>>>>> Ok, WebKit gardening is hard. So is fixing layout tests. You >> can't >> >> >>>>>> call it a successful WebKit roll if it breaks layout tests. But >> we >> >> >>>>>> don't revert WebKit rolls. It's a forward-only thing. And we >> want >> >> >>>>>> to >> >> >>>>>> roll quickly, so that we can react to next "big breaker" faster. >> So >> >> >>>>>> we're stuck with roll-now/clean-up-after deal. This sucks, >> because >> >> >>>>>> the >> >> >>>>>> "clean-up-after" is rarely fully completed. Which brings failing >> >> >>>>>> layout tests, which brings the suffering and spells asymptotic >> doom >> >> >>>>>> to >> >> >>>>>> the LTTF effort. >> >> >>>>>> >> >> >>>>>> POSSIBLE SOLUTIONS: >> >> >>>>>> >> >> >>>>>> * Extend WebKit gardener's duties to 4 days. First two days you >> >> >>>>>> roll. >> >> >>>>>> Next two days you fix layout tests. Not file bugs -- actually >> fix >> >> >>>>>> them. The net result of 4 days should be 0 (or less!) new layout >> >> >>>>>> test >> >> >>>>>> failures. This solution kind of expects the gardener to be part >> of >> >> >>>>>> LTTF, which is not always the case. So it may not seem totally >> >> >>>>>> fair. >> >> >>>>>> >> >> >>>>>> * Assign LTTF folks specifically for test clean-up every day. >> The >> >> >>>>>> idea >> >> >>>>>> here is to slant LTTF effort aggressively toward fixing newer >> >> >>>>>> failures. This seems nice for the gardeners, but appears to >> >> >>>>>> separate >> >> >>>>>> the action/responsibility dependency: no matter what you roll, >> the >> >> >>>>>> LTTF elves will fix it. >> >> >>>>>> >> >> >>>>>> * [ your idea goes here ] >> >> >>>>>> >> >> >>>>>> TIMELINE: >> >> >>>>>> >> >> >>>>>> I would like for us to agree on a solution and make the >> necessary >> >> >>>>>> changes to the process today. Tomorrow is a new day, full of >> >> >>>>>> surprising changes upstream. >> >> >>>>>> >> >> >>>>>> :DG< >> >> >>>>>> >> >> >>>>>> >> >> >>>>> >> >> >>>>> >> >> >>>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> -- >> >> >>> All truth passes through three stages. First, it is ridiculed. >> Second, >> >> >>> it >> >> >>> is violently opposed. Third, it is accepted as being self-evident. >> -- >> >> >>> Schopenhauer >> >> >>> >> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> > >> >> > >> > >> > >> > >> > >> > >> > > > > -- > All truth passes through three stages. First, it is ridiculed. Second, it > is violently opposed. Third, it is accepted as being self-evident. -- > Schopenhauer > --~--~---------~--~----~------------~-------~--~----~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~----------~----~----~----~------~----~------~--~---
