[chromium-dev] Re: Scalability
Hi The patch below should fix it (there are arguably other perhaps better ways to tackle the debugger dependency etc.). With that patch, the linux shared make build compiles and links all targets for me. --Craig diff --git a/chrome/chrome.gyp b/chrome/chrome.gyp index 3fa1905..c0caeb6 100755 --- a/chrome/chrome.gyp +++ b/chrome/chrome.gyp @@ -741,6 +741,7 @@ 'common', 'chrome_resources', 'chrome_strings', +'debugger', 'theme_resources', '../app/app.gyp:app_resources', '../app/app.gyp:app_strings', @@ -3024,6 +3025,7 @@ '../third_party/npapi/npapi.gyp:npapi', '../webkit/webkit.gyp:glue', '../native_client/src/trusted/plugin/plugin_chrome.gyp:npGoogleNaClPluginChrome', + '../native_client/src/trusted/service_runtime/service_runtime.gyp:expiration', '../native_client/src/trusted/service_runtime/service_runtime.gyp:sel', '../native_client/src/trusted/validator_x86/validator_x86.gyp:ncvalidate', '../native_client/src/trusted/platform_qualify/platform_qualify.gyp:platform_qual_lib', On Tue, Oct 13, 2009 at 3:35 AM, Jacob Mandelson ja...@mandelson.org wrote: On Mon, Oct 12, 2009 at 06:29:02PM -0700, Lei Zhang wrote: Maybe you need to clobber? The shared build bot on the FYI waterfall is working: http://build.chromium.org/buildbot/waterfall.fyi/builders/Chromium%20Linux%20Builder%20(dbg-shlib)/builds/1069 gclient runhooks regenerated a bunch of .mk's, but I'm still getting the same link error. -- Jacob --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: WebKit Hygiene: Please be kind. Test your patches. Warn about two-sided patches.
On Fri, Sep 25, 2009 at 10:21 AM, Jeremy Orlow jor...@chromium.org wrote: On Fri, Sep 25, 2009 at 10:01 AM, Amanda Walker ama...@chromium.orgwrote: On Tue, Sep 22, 2009 at 9:06 PM, Dimitri Glazkov dglaz...@google.comwrote: This all means that we have to be a bit more diligent. We shouldn't be paying these unnecessary costs. So, from now on, I propose a fairly simple set of new rules: 1) if you write a Chromium patch for WebKit, you must provide URLs of successful trybot runs with your submission. Chromium WebKit reviewers will not r+ your patch otherwise. If you can't provide the trybot URLs for some reason, please explain in detail why this patch could still land. 2) if the two-sided patch you authored broke the canary and this happened with no coordination with the WebKit gardener, you assume WebKit gardening responsibility for the next 24 hours. Hopefully, these amendments to our existing ways will bring a bit more peace and stability to the Chromium land. What do you think? I think they're a good start, but they are symptoms of a larger problem. Move fast and clean up messes when they happen worked great when we had one big mess a week. These days, we have multiple ones per day. And as good as the try bots and canaries are (kudos to M-A for setting up the try bots in the first place, and everyone who helps keep the ever-growing herd of bots running), they don't catch everything. They don't catch build time regressions because someone forgot svn:ignores when refactoring where projects get generated, or accidentally checks something into the wrong directory (not to pick on anyone in particular, these are just recent examples). I'd add a 3rd principle: 3) If you change how chromium is built, however innocuous your change seems (gyp changes, new libraries, etc.), take extra care and use more reviewers than usual (preferably including someone intimately acquainted with the bots, such as maruel, thomasvl, markmentovai, nsylvain, etc.). If you're reviewing such a change, don't just look at the diffs, try out the patch and flag anything that seems out of the ordinary. Build breakage means more than just doesn't compile; it can also mean overcompiles (which kills bot performance) or requires a clobber unnecessarily. I had to land a 2 sided patch yesterday. It blew up an important unit test in a very creative way, and we're still trying to figure out how to clean it up nicely. In the mean time, we have a dev channel release blocker. There has to be a better way... Here's a crazy idea: 4) Create a WebKit.next branch. Have full build bot coverage on it. All integrations would go through it. Other than that, only 2 sided patches would land on it. Whenever everything passes, we'd merge in with the main branch. We'd try very hard never to let it get more than a day or so out of sync. This would make 2 sided merges (which are often the reason for rushing a DEPS roll--don't want to keep the canary red for too long, otherwise it's very hard to sort things out) much less haphazard. We could even have it roll the DEPS automatically every time a new webkit.org patch lands, and thus eliminate our need for dedicated canaries. Yeah, it's crazybut I kind of like it. And yeah, when the WebKit API lands things should be better in terms of others breaking us, but this addresses 2 sided patches...which are not going away any time soon. J I think we should just finish the WebKit API :-) Here's the bug list: http://code.google.com/p/chromium/issues/list?q=label:WebKitAPI I'm looking for volunteers to help take on some of these tasks. -Darin --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Need your help
HI, I want to explore Chromium's auto-testing mechanism, include: auto- test, auto collect performance data. Can you provide me some document about it? thanks! --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Error when trying to build chromium with toolkit_views=1.
Hi, I tried to build chromium with toolkit_views=1 but got error when executing following commands: $ export GYP_DEFINES=toolkit_views=1 $ gclient runhooks --force it complained: src/third_party/cros/cros_api.gyp not found. There is no cros directory in my src/third_party, where can I get it? Regards James Su --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Buildbot upgrade
On Mon, Oct 12, 2009 at 7:44 AM, Nicolas Sylvain nsylv...@chromium.org wrote: On Sun, Oct 11, 2009 at 11:33 AM, Jeremy Orlow jor...@chromium.org wrote: Is there documentation anywhere for all the parameters you can feed into the buildbot webpage? If not, a cheat sheet would be really helpful. the help link at the bottom has most of them http://build.chromium.org/buildbot/waterfall/waterfall/help Note that the help link at the bottom is at the bottom of the console page. That had me confused this morning :-). -scott --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: WebKit Hygiene: Please be kind. Test your patches. Warn about two-sided patches.
The webkit api won't help if chromium folks (especially when you change v8 bindings) don't *run the layout tests *which is what happened yesterday and causes quite a few of our worst problems while gardening. If you're changing the v8 bindings, you're doing it to fix a problem in chromium, so you must have a build of chomium (and thus be able to run layout tests). dave On Mon, Oct 12, 2009 at 11:40 PM, Darin Fisher da...@chromium.org wrote: On Fri, Sep 25, 2009 at 10:21 AM, Jeremy Orlow jor...@chromium.orgwrote: On Fri, Sep 25, 2009 at 10:01 AM, Amanda Walker ama...@chromium.orgwrote: On Tue, Sep 22, 2009 at 9:06 PM, Dimitri Glazkov dglaz...@google.comwrote: This all means that we have to be a bit more diligent. We shouldn't be paying these unnecessary costs. So, from now on, I propose a fairly simple set of new rules: 1) if you write a Chromium patch for WebKit, you must provide URLs of successful trybot runs with your submission. Chromium WebKit reviewers will not r+ your patch otherwise. If you can't provide the trybot URLs for some reason, please explain in detail why this patch could still land. 2) if the two-sided patch you authored broke the canary and this happened with no coordination with the WebKit gardener, you assume WebKit gardening responsibility for the next 24 hours. Hopefully, these amendments to our existing ways will bring a bit more peace and stability to the Chromium land. What do you think? I think they're a good start, but they are symptoms of a larger problem. Move fast and clean up messes when they happen worked great when we had one big mess a week. These days, we have multiple ones per day. And as good as the try bots and canaries are (kudos to M-A for setting up the try bots in the first place, and everyone who helps keep the ever-growing herd of bots running), they don't catch everything. They don't catch build time regressions because someone forgot svn:ignores when refactoring where projects get generated, or accidentally checks something into the wrong directory (not to pick on anyone in particular, these are just recent examples). I'd add a 3rd principle: 3) If you change how chromium is built, however innocuous your change seems (gyp changes, new libraries, etc.), take extra care and use more reviewers than usual (preferably including someone intimately acquainted with the bots, such as maruel, thomasvl, markmentovai, nsylvain, etc.). If you're reviewing such a change, don't just look at the diffs, try out the patch and flag anything that seems out of the ordinary. Build breakage means more than just doesn't compile; it can also mean overcompiles (which kills bot performance) or requires a clobber unnecessarily. I had to land a 2 sided patch yesterday. It blew up an important unit test in a very creative way, and we're still trying to figure out how to clean it up nicely. In the mean time, we have a dev channel release blocker. There has to be a better way... Here's a crazy idea: 4) Create a WebKit.next branch. Have full build bot coverage on it. All integrations would go through it. Other than that, only 2 sided patches would land on it. Whenever everything passes, we'd merge in with the main branch. We'd try very hard never to let it get more than a day or so out of sync. This would make 2 sided merges (which are often the reason for rushing a DEPS roll--don't want to keep the canary red for too long, otherwise it's very hard to sort things out) much less haphazard. We could even have it roll the DEPS automatically every time a new webkit.org patch lands, and thus eliminate our need for dedicated canaries. Yeah, it's crazybut I kind of like it. And yeah, when the WebKit API lands things should be better in terms of others breaking us, but this addresses 2 sided patches...which are not going away any time soon. J I think we should just finish the WebKit API :-) Here's the bug list: http://code.google.com/p/chromium/issues/list?q=label:WebKitAPI I'm looking for volunteers to help take on some of these tasks. -Darin --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: WebKit Hygiene: Please be kind. Test your patches. Warn about two-sided patches.
On Tue, Oct 13, 2009 at 8:09 AM, David Levin le...@google.com wrote: The webkit api won't help if chromium folks (especially when you change v8 bindings) don't run the layout tests which is what happened yesterday and causes quite a few of our worst problems while gardening. I agree this is essential. If you're changing the v8 bindings, you're doing it to fix a problem in chromium, so you must have a build of chomium (and thus be able to run layout tests). Eric is going to add commit-queue coverage for the V8 bindings, which should help with this problem significantly. Adam --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: WebKit Hygiene: Please be kind. Test your patches. Warn about two-sided patches.
On Tue, Oct 13, 2009 at 8:21 AM, Adam Barth aba...@chromium.org wrote: On Tue, Oct 13, 2009 at 8:09 AM, David Levin le...@google.com wrote: The webkit api won't help if chromium folks (especially when you change v8 bindings) don't run the layout tests which is what happened yesterday and causes quite a few of our worst problems while gardening. I agree this is essential. If you're changing the v8 bindings, you're doing it to fix a problem in chromium, so you must have a build of chomium (and thus be able to run layout tests). Eric is going to add commit-queue coverage for the V8 bindings, which should help with this problem significantly. I think this is a great addition, but shouldn't the canary bots have had just as much of a chance of catching this issue? Why didn't they catch it? Is it b/c of flakiness? If so, then try bots won't help that much. What I think we need to do is beef up the canary bots so that they provide equivalent coverage to the mainline Chromium bots. This will be essential if we ever wish to roll WebKit less often (once per week say), which will only really be possible once we have the WebKit API. -Darin Adam --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: WebKit Hygiene: Please be kind. Test your patches. Warn about two-sided patches.
On Tue, Oct 13, 2009 at 9:26 AM, Darin Fisher da...@chromium.org wrote: On Tue, Oct 13, 2009 at 8:21 AM, Adam Barth aba...@chromium.org wrote: On Tue, Oct 13, 2009 at 8:09 AM, David Levin le...@google.com wrote: The webkit api won't help if chromium folks (especially when you change v8 bindings) don't run the layout tests which is what happened yesterday and causes quite a few of our worst problems while gardening. I agree this is essential. If you're changing the v8 bindings, you're doing it to fix a problem in chromium, so you must have a build of chomium (and thus be able to run layout tests). Eric is going to add commit-queue coverage for the V8 bindings, which should help with this problem significantly. I think this is a great addition, but shouldn't the canary bots have had just as much of a chance of catching this issue? Why didn't they catch it? Is it b/c of flakiness? If so, then try bots won't help that much. What I think we need to do is beef up the canary bots so that they provide equivalent coverage to the mainline Chromium bots. This will be essential if we ever wish to roll WebKit less often (once per week say), which will only really be possible once we have the WebKit API. -Darin The canaries did catch it. The gardener is also to blame for this. :DG --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] [LTTF][WebKit Gardening]: Keeping up with the weeds.
I think we need to change something. I am not sure what -- I have ideas, but -- I would appreciate some collective thinking on this. PROBLEM: We accumulate more test failures via WebKit rolls than we fix with our LTTF effort. This ain't right. ANALYSIS: Ok, WebKit gardening is hard. So is fixing layout tests. You can't call it a successful WebKit roll if it breaks layout tests. But we don't revert WebKit rolls. It's a forward-only thing. And we want to roll quickly, so that we can react to next big breaker faster. So we're stuck with roll-now/clean-up-after deal. This sucks, because the clean-up-after is rarely fully completed. Which brings failing layout tests, which brings the suffering and spells asymptotic doom to the LTTF effort. POSSIBLE SOLUTIONS: * Extend WebKit gardener's duties to 4 days. First two days you roll. Next two days you fix layout tests. Not file bugs -- actually fix them. The net result of 4 days should be 0 (or less!) new layout test failures. This solution kind of expects the gardener to be part of LTTF, which is not always the case. So it may not seem totally fair. * Assign LTTF folks specifically for test clean-up every day. The idea here is to slant LTTF effort aggressively toward fixing newer failures. This seems nice for the gardeners, but appears to separate the action/responsibility dependency: no matter what you roll, the LTTF elves will fix it. * [ your idea goes here ] TIMELINE: I would like for us to agree on a solution and make the necessary changes to the process today. Tomorrow is a new day, full of surprising changes upstream. :DG --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: [LTTF][WebKit Gardening]: Keeping up with the weeds.
When I'm sheriffing, the vast majority of issues I see are flaky tests in general rather than clear bustage from WebKit updates. Even if a particular update causes flakiness, it can be hard to determine and fix. In that sense, the highest priority IMO would be to eliminate flakiness, which would make failures clearer and easier to track down no matter who was responsible. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: [LTTF][WebKit Gardening]: Keeping up with the weeds.
I don't think it's realistic to expect the gardener, or any one person, to be able to fix an arbitrary broken layout test in a reasonable period of time. That's certainly true for new tests, but even for regressions I often can't even tell for sure whether our results are correct, much less what to do if they're not. It's far more efficient to have the right person fix a test. (Of course, people should also strive to broaden their knowledge, but there's a limit to how much of that one can do in a week.) Never having broken layout tests is an excellent goal, but quite frankly I don't think it's one we should prioritize so high that we hobble other efforts and burn out developers. - Pam On Tue, Oct 13, 2009 at 10:31 AM, Dimitri Glazkov dglaz...@chromium.orgwrote: I think we need to change something. I am not sure what -- I have ideas, but -- I would appreciate some collective thinking on this. PROBLEM: We accumulate more test failures via WebKit rolls than we fix with our LTTF effort. This ain't right. ANALYSIS: Ok, WebKit gardening is hard. So is fixing layout tests. You can't call it a successful WebKit roll if it breaks layout tests. But we don't revert WebKit rolls. It's a forward-only thing. And we want to roll quickly, so that we can react to next big breaker faster. So we're stuck with roll-now/clean-up-after deal. This sucks, because the clean-up-after is rarely fully completed. Which brings failing layout tests, which brings the suffering and spells asymptotic doom to the LTTF effort. POSSIBLE SOLUTIONS: * Extend WebKit gardener's duties to 4 days. First two days you roll. Next two days you fix layout tests. Not file bugs -- actually fix them. The net result of 4 days should be 0 (or less!) new layout test failures. This solution kind of expects the gardener to be part of LTTF, which is not always the case. So it may not seem totally fair. * Assign LTTF folks specifically for test clean-up every day. The idea here is to slant LTTF effort aggressively toward fixing newer failures. This seems nice for the gardeners, but appears to separate the action/responsibility dependency: no matter what you roll, the LTTF elves will fix it. * [ your idea goes here ] TIMELINE: I would like for us to agree on a solution and make the necessary changes to the process today. Tomorrow is a new day, full of surprising changes upstream. :DG --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: [LTTF][WebKit Gardening]: Keeping up with the weeds.
Let's not conflate the two. There are flakes, and there are clearly, consistently failing tests, arriving in chunks every day via WebKit rolls. :DG On Tue, Oct 13, 2009 at 10:36 AM, Peter Kasting pkast...@google.com wrote: When I'm sheriffing, the vast majority of issues I see are flaky tests in general rather than clear bustage from WebKit updates. Even if a particular update causes flakiness, it can be hard to determine and fix. In that sense, the highest priority IMO would be to eliminate flakiness, which would make failures clearer and easier to track down no matter who was responsible. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: [LTTF][WebKit Gardening]: Keeping up with the weeds.
On Tue, Oct 13, 2009 at 10:43 AM, Dimitri Glazkov dglaz...@chromium.orgwrote: Let's not conflate the two. There are flakes, and there are clearly, consistently failing tests, arriving in chunks every day via WebKit rolls. OK, I'm just saying that my observations are that the obvious, consistent failing tests that arrive are like 0.1% compared with the flaky tests. I added many dozens of flaky test lines in my last sheriffing stint and the number of additional failures from WebKit updates was like 3. So I don't agree with your assessment of the problem space. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: [LTTF][WebKit Gardening]: Keeping up with the weeds.
I've been thinking quite a bit about this - I agree with Dmitri that the current Sisyphean approach is unsustainable. I don't think the right path is to ask the sheriffs to do the cleanup themselves - for example, a webkit roll that breaks workers in some obscure way is almost certainly beyond the ability of any random gardener to fix in two days, especially when there may be multiple bugs. A better solution would be to have the sheriff (or someone from LTTF) assign the bugs to specific people, with a general rule that such bugs must be fixed within two days (make these bugs the top priority over other tasks). This allows for load balancing of bugs, and also makes sure that we have the right people working on any specific bug. -atw On Tue, Oct 13, 2009 at 10:40 AM, Pam Greene p...@chromium.org wrote: I don't think it's realistic to expect the gardener, or any one person, to be able to fix an arbitrary broken layout test in a reasonable period of time. That's certainly true for new tests, but even for regressions I often can't even tell for sure whether our results are correct, much less what to do if they're not. It's far more efficient to have the right person fix a test. (Of course, people should also strive to broaden their knowledge, but there's a limit to how much of that one can do in a week.) Never having broken layout tests is an excellent goal, but quite frankly I don't think it's one we should prioritize so high that we hobble other efforts and burn out developers. - Pam On Tue, Oct 13, 2009 at 10:31 AM, Dimitri Glazkov dglaz...@chromium.orgwrote: I think we need to change something. I am not sure what -- I have ideas, but -- I would appreciate some collective thinking on this. PROBLEM: We accumulate more test failures via WebKit rolls than we fix with our LTTF effort. This ain't right. ANALYSIS: Ok, WebKit gardening is hard. So is fixing layout tests. You can't call it a successful WebKit roll if it breaks layout tests. But we don't revert WebKit rolls. It's a forward-only thing. And we want to roll quickly, so that we can react to next big breaker faster. So we're stuck with roll-now/clean-up-after deal. This sucks, because the clean-up-after is rarely fully completed. Which brings failing layout tests, which brings the suffering and spells asymptotic doom to the LTTF effort. POSSIBLE SOLUTIONS: * Extend WebKit gardener's duties to 4 days. First two days you roll. Next two days you fix layout tests. Not file bugs -- actually fix them. The net result of 4 days should be 0 (or less!) new layout test failures. This solution kind of expects the gardener to be part of LTTF, which is not always the case. So it may not seem totally fair. * Assign LTTF folks specifically for test clean-up every day. The idea here is to slant LTTF effort aggressively toward fixing newer failures. This seems nice for the gardeners, but appears to separate the action/responsibility dependency: no matter what you roll, the LTTF elves will fix it. * [ your idea goes here ] TIMELINE: I would like for us to agree on a solution and make the necessary changes to the process today. Tomorrow is a new day, full of surprising changes upstream. :DG --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: [LTTF][WebKit Gardening]: Keeping up with the weeds.
On Tue, Oct 13, 2009 at 10:45 AM, Peter Kasting pkast...@google.com wrote: On Tue, Oct 13, 2009 at 10:43 AM, Dimitri Glazkov dglaz...@chromium.orgwrote: Let's not conflate the two. There are flakes, and there are clearly, consistently failing tests, arriving in chunks every day via WebKit rolls. OK, I'm just saying that my observations are that the obvious, consistent failing tests that arrive are like 0.1% compared with the flaky tests. I added many dozens of flaky test lines in my last sheriffing stint and the number of additional failures from WebKit updates was like 3. So I don't agree with your assessment of the problem space. It's because the sheriff don't notice the new failing tests, because most of the time the gardener does a good job of updating the list at the same time as the merge, so the tree stays mostly green. See http://src.chromium.org/viewvc/chrome?view=revrevision=28727 What sheriffs see is leftovers and flakiness introduced by the merge. Nicolas Nicolas PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: [LTTF][WebKit Gardening]: Keeping up with the weeds.
A better solution would be to have the sheriff (or someone from LTTF) assign the bugs to specific people So how do you figure out who to assign it to? with a general rule that such bugs must be fixed within two days (make these bugs the top priority over other tasks). Over security bugs? Over code reviews which other people need (some of which are large and involved)? Over interviews and feedback (if your company is doing this)? Over fixes for a release with a given deadline that may be looming? Over your next round of WebKit gardening? Over fixing a top crasher? :) This allows for load balancing of bugs, and also makes sure that we have the right people working on any specific bug. As long as one area isn't broken disproportionally or somehow one person isn't assigned bugs disproportionately. First two days you roll. Next two days you fix layout tests. The net result of 4 days should be 0 (or less!) new layout test failures. Not all WebKit gardening days are equivalent and some will take quite a bit of time to clean up. For example, what happens when someone add structured clones for JSC and we fail the associated layout test? Does the person on that rotation end up getting a new feature to implement? Or if we take Drew's approach, do we make Drew drop what he is doing to implement structured clones, since they are most closely associated with MessagePorts? I don't know the right answer but I don't feel like it is there yet Assign LTTF folks specifically for test clean-up every day. ... appears to separate the action/responsibility dependency. *There already is a separation of action from responsibility*. For example, in yesterday's tragic roll, it shouldn't have been done, but the real action was not done by hclam. It was done by the person who created the v8 patch, and there is no responsibility for the havoc and mess caused by that change. This happens frequently to the gardener. My vote: I lean closest to assigning something like the LTTF folks to clean up recent failures and have rotating duty on to do this for maybe a week or two at a time (even after LTTF is ended). With the ability to pull in/assign/hassle other folks to fix/investigate issues if it is beyond the knowledge of folks on that team. If you have something like the LTTF folks cleaning up, then it seems possible to get more people doing the WebKit gardening. Perhaps, if you are on what I'll call the LTTF rotations, then you don't have to do the gardening, but you are available to help gardener if they hit some build break that they don't know how to deal with. Dave On Tue, Oct 13, 2009 at 10:53 AM, Drew Wilson atwil...@chromium.org wrote: I've been thinking quite a bit about this - I agree with Dmitri that the current Sisyphean approach is unsustainable. I don't think the right path is to ask the sheriffs to do the cleanup themselves - for example, a webkit roll that breaks workers in some obscure way is almost certainly beyond the ability of any random gardener to fix in two days, especially when there may be multiple bugs. A better solution would be to have the sheriff (or someone from LTTF) assign the bugs to specific people, with a general rule that such bugs must be fixed within two days (make these bugs the top priority over other tasks). This allows for load balancing of bugs, and also makes sure that we have the right people working on any specific bug. -atw On Tue, Oct 13, 2009 at 10:40 AM, Pam Greene p...@chromium.org wrote: I don't think it's realistic to expect the gardener, or any one person, to be able to fix an arbitrary broken layout test in a reasonable period of time. That's certainly true for new tests, but even for regressions I often can't even tell for sure whether our results are correct, much less what to do if they're not. It's far more efficient to have the right person fix a test. (Of course, people should also strive to broaden their knowledge, but there's a limit to how much of that one can do in a week.) Never having broken layout tests is an excellent goal, but quite frankly I don't think it's one we should prioritize so high that we hobble other efforts and burn out developers. - Pam On Tue, Oct 13, 2009 at 10:31 AM, Dimitri Glazkov dglaz...@chromium.org wrote: I think we need to change something. I am not sure what -- I have ideas, but -- I would appreciate some collective thinking on this. PROBLEM: We accumulate more test failures via WebKit rolls than we fix with our LTTF effort. This ain't right. ANALYSIS: Ok, WebKit gardening is hard. So is fixing layout tests. You can't call it a successful WebKit roll if it breaks layout tests. But we don't revert WebKit rolls. It's a forward-only thing. And we want to roll quickly, so that we can react to next big breaker faster. So we're stuck with roll-now/clean-up-after deal. This sucks, because the clean-up-after is rarely fully completed. Which brings failing layout tests, which brings the
[chromium-dev] Re: [LTTF][WebKit Gardening]: Keeping up with the weeds.
On Tue, Oct 13, 2009 at 11:17 AM, Nicolas Sylvain nsylv...@chromium.orgwrote: It's because the sheriff don't notice the new failing tests, because most of the time the gardener does a good job of updating the list at the same time as the merge, so the tree stays mostly green. See http://src.chromium.org/viewvc/chrome?view=revrevision=28727 In my case I was watching jparent's commits, so I stand by my assertion. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: [LTTF][WebKit Gardening]: Keeping up with the weeds.
On Tue, Oct 13, 2009 at 10:31 AM, Dimitri Glazkov dglaz...@chromium.org wrote: I think we need to change something. I am not sure what -- I have ideas, but -- I would appreciate some collective thinking on this. PROBLEM: We accumulate more test failures via WebKit rolls than we fix with our LTTF effort. This ain't right. ANALYSIS: Ok, WebKit gardening is hard. So is fixing layout tests. You can't call it a successful WebKit roll if it breaks layout tests. But we don't revert WebKit rolls. It's a forward-only thing. And we want to roll quickly, so that we can react to next big breaker faster. So we're stuck with roll-now/clean-up-after deal. This sucks, because the clean-up-after is rarely fully completed. Which brings failing layout tests, which brings the suffering and spells asymptotic doom to the LTTF effort. POSSIBLE SOLUTIONS: * Extend WebKit gardener's duties to 4 days. First two days you roll. Next two days you fix layout tests. Not file bugs -- actually fix them. The net result of 4 days should be 0 (or less!) new layout test failures. This solution kind of expects the gardener to be part of LTTF, which is not always the case. So it may not seem totally fair. * Assign LTTF folks specifically for test clean-up every day. The idea here is to slant LTTF effort aggressively toward fixing newer failures. This seems nice for the gardeners, but appears to separate the action/responsibility dependency: no matter what you roll, the LTTF elves will fix it. * [ your idea goes here ] * Stop WebKit committers from checking in changes that break our code? Granted, I don't know how to do this, but it seems like this gets closer to the real problem. Do we need more stuff running at webkit.org? Do we need someone watching the webkit buildbots and asking them to revert changes that break our builds? Are there other things we can do here? -- Dirk --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: [LTTF][WebKit Gardening]: Keeping up with the weeds.
I agree with Dimitri that we're fighting a losing battle here. In my last stint as gardener, I did informally what I proposed formally last time: I spent basically 1 full day just triaging failures from my 2 days gardening. Not fixing, but just running tests locally, analyzing, grouping, creating bugs, assigning to appropriate people (when I knew who they were, cc'ing poor dglazkov when I didn't). So at least I didn't leave a monster bug with layout tests broken by merge #foo but at least grouped by area. That was manageable, but I don't know if another day would actually be enough for a meaningful amount of fixing. I also agree with Drew that actively fixing all the broken tests is usually beyond the skills of any one gardener. Perhaps we should start taking ownership of particular groups of layout tests? And maybe automatically assign them (or least cc them), the same way Area-Foo causes automatic cc'ing in bugs.chromium.org (I think?) That way, the gardener wouldn't have to know who to assign to. I've basically taken responsibility for fixing all layout tests broken by Skia rolls, which can pretty heavy on its own, but I'm willing to take ownership of a directory or two. BTW, the layout test flakiness dashboard has become an invaluable tool for analyzing failures: searching for a test by name is lightning-fast, and you can clearly see if a test has become flaky, on which platforms, and which WebKit merge was responsible, which can also help with grouping. (Props to Ojan for that). Also, it may be Gilbert-and-Sullivan-esque of me, but I think everyone who contributes patches to WebKit for chromium should be on the WebKit gardener rotation. Stephen On Tue, Oct 13, 2009 at 1:53 PM, Drew Wilson atwil...@chromium.org wrote: I've been thinking quite a bit about this - I agree with Dmitri that the current Sisyphean approach is unsustainable. I don't think the right path is to ask the sheriffs to do the cleanup themselves - for example, a webkit roll that breaks workers in some obscure way is almost certainly beyond the ability of any random gardener to fix in two days, especially when there may be multiple bugs. A better solution would be to have the sheriff (or someone from LTTF) assign the bugs to specific people, with a general rule that such bugs must be fixed within two days (make these bugs the top priority over other tasks). This allows for load balancing of bugs, and also makes sure that we have the right people working on any specific bug. -atw On Tue, Oct 13, 2009 at 10:40 AM, Pam Greene p...@chromium.org wrote: I don't think it's realistic to expect the gardener, or any one person, to be able to fix an arbitrary broken layout test in a reasonable period of time. That's certainly true for new tests, but even for regressions I often can't even tell for sure whether our results are correct, much less what to do if they're not. It's far more efficient to have the right person fix a test. (Of course, people should also strive to broaden their knowledge, but there's a limit to how much of that one can do in a week.) Never having broken layout tests is an excellent goal, but quite frankly I don't think it's one we should prioritize so high that we hobble other efforts and burn out developers. - Pam On Tue, Oct 13, 2009 at 10:31 AM, Dimitri Glazkov dglaz...@chromium.orgwrote: I think we need to change something. I am not sure what -- I have ideas, but -- I would appreciate some collective thinking on this. PROBLEM: We accumulate more test failures via WebKit rolls than we fix with our LTTF effort. This ain't right. ANALYSIS: Ok, WebKit gardening is hard. So is fixing layout tests. You can't call it a successful WebKit roll if it breaks layout tests. But we don't revert WebKit rolls. It's a forward-only thing. And we want to roll quickly, so that we can react to next big breaker faster. So we're stuck with roll-now/clean-up-after deal. This sucks, because the clean-up-after is rarely fully completed. Which brings failing layout tests, which brings the suffering and spells asymptotic doom to the LTTF effort. POSSIBLE SOLUTIONS: * Extend WebKit gardener's duties to 4 days. First two days you roll. Next two days you fix layout tests. Not file bugs -- actually fix them. The net result of 4 days should be 0 (or less!) new layout test failures. This solution kind of expects the gardener to be part of LTTF, which is not always the case. So it may not seem totally fair. * Assign LTTF folks specifically for test clean-up every day. The idea here is to slant LTTF effort aggressively toward fixing newer failures. This seems nice for the gardeners, but appears to separate the action/responsibility dependency: no matter what you roll, the LTTF elves will fix it. * [ your idea goes here ] TIMELINE: I would like for us to agree on a solution and make the necessary changes to the process today. Tomorrow is a new day, full of
Layout test flakiness WAS: [chromium-dev] Re: [LTTF][WebKit Gardening]: Keeping up with the weeds.
We could rerun any unexpected fail/crash/timeout tests. If they pass the second time around, then the tree turns orange instead of red. This has come up many times, but we've never agreed on whether it's a good idea. Pros: -Easier to distinguish between flakiness and new failures -Tree will be much greener -Try bots will be much more reliable -Increase in overall team sanity Cons: -Almost guaranteed to sweep some new flakiness under the rug -If enough flakiness accumulates it will affect cycle time (each timeout/crash test we rerun takes seconds) Shouldn't be too hard to implement. Ojan On Tue, Oct 13, 2009 at 11:32 AM, Peter Kasting pkast...@google.com wrote: On Tue, Oct 13, 2009 at 11:17 AM, Nicolas Sylvain nsylv...@chromium.orgwrote: It's because the sheriff don't notice the new failing tests, because most of the time the gardener does a good job of updating the list at the same time as the merge, so the tree stays mostly green. See http://src.chromium.org/viewvc/chrome?view=revrevision=28727 In my case I was watching jparent's commits, so I stand by my assertion. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
Re: Layout test flakiness WAS: [chromium-dev] Re: [LTTF][WebKit Gardening]: Keeping up with the weeds.
We did this on my last project to deal with flaky test infrastructure. It worked well in that test failures were pretty much guaranteed to be real (we ran tests 3 times and only reported failure if a test failed all 3 times), but it did definitely make us stop caring about flaky tests. Idealizing this some more, what if it could auto-add flaky tests to test_expectations, or at least generate a daily report that someone could then manually add? This would make us keep track of the flakiness and would only have the slowdown in test run times until a flaky test is added to test_expectations. On Tue, Oct 13, 2009 at 12:02 PM, Ojan Vafai o...@chromium.org wrote: We could rerun any unexpected fail/crash/timeout tests. If they pass the second time around, then the tree turns orange instead of red. This has come up many times, but we've never agreed on whether it's a good idea. Pros: -Easier to distinguish between flakiness and new failures -Tree will be much greener -Try bots will be much more reliable -Increase in overall team sanity Cons: -Almost guaranteed to sweep some new flakiness under the rug -If enough flakiness accumulates it will affect cycle time (each timeout/crash test we rerun takes seconds) Shouldn't be too hard to implement. Ojan On Tue, Oct 13, 2009 at 11:32 AM, Peter Kasting pkast...@google.comwrote: On Tue, Oct 13, 2009 at 11:17 AM, Nicolas Sylvain nsylv...@chromium.orgwrote: It's because the sheriff don't notice the new failing tests, because most of the time the gardener does a good job of updating the list at the same time as the merge, so the tree stays mostly green. See http://src.chromium.org/viewvc/chrome?view=revrevision=28727 In my case I was watching jparent's commits, so I stand by my assertion. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
Re: Layout test flakiness WAS: [chromium-dev] Re: [LTTF][WebKit Gardening]: Keeping up with the weeds.
On Tue, Oct 13, 2009 at 1:00 PM, Julie Parent jpar...@chromium.org wrote: We did this on my last project to deal with flaky test infrastructure. It worked well in that test failures were pretty much guaranteed to be real (we ran tests 3 times and only reported failure if a test failed all 3 times), but it did definitely make us stop caring about flaky tests. Idealizing this some more, what if it could auto-add flaky tests to test_expectations, or at least generate a daily report that someone could then manually add? This would make us keep track of the flakiness and would only have the slowdown in test run times until a flaky test is added to test_expectations. I agree that this would address the slowdown issue. It's more work though. :) Tony also pointed out that rerunning tests would not allow you to distinguish tests that consistently pass in isolation, but fail when run in the whole suite. I think as long as we don't automatically add to test_expectations though, this should be fine. Whoever goes to fix the test can check if it consistently fails/passes only in isolation. Another possibility would be to rerun unexpected failures, but continue to turn the tree red. This will provide more immediate feedback to people trying to green to tree as to whether tests are flaky or regressions. It would help with sheriffing/gardening without the other downsides. Ojan On Tue, Oct 13, 2009 at 12:02 PM, Ojan Vafai o...@chromium.org wrote: We could rerun any unexpected fail/crash/timeout tests. If they pass the second time around, then the tree turns orange instead of red. This has come up many times, but we've never agreed on whether it's a good idea. Pros: -Easier to distinguish between flakiness and new failures -Tree will be much greener -Try bots will be much more reliable -Increase in overall team sanity Cons: -Almost guaranteed to sweep some new flakiness under the rug -If enough flakiness accumulates it will affect cycle time (each timeout/crash test we rerun takes seconds) Shouldn't be too hard to implement. Ojan On Tue, Oct 13, 2009 at 11:32 AM, Peter Kasting pkast...@google.comwrote: On Tue, Oct 13, 2009 at 11:17 AM, Nicolas Sylvain nsylv...@chromium.org wrote: It's because the sheriff don't notice the new failing tests, because most of the time the gardener does a good job of updating the list at the same time as the merge, so the tree stays mostly green. See http://src.chromium.org/viewvc/chrome?view=revrevision=28727 In my case I was watching jparent's commits, so I stand by my assertion. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: [LTTF][WebKit Gardening]: Keeping up with the weeds.
I like the idea of ownership of groups of layout tests. Maybe these directory owners could be more like the finders? An owner shouldn't have to necessarily fix everything in a group/directory, but they should be responsible for triaging and getting meaningful bugs filled for them, to keep things moving along. (I volunteer for editing/) Another complicating factor - The state of the main Chrome tree has a lot of effect on the gardener. If the tree is already filled with flakiness, then the webkit roll is likely to show failures, which may or may not have been there before the roll. This was largely the case in the situation pkasting was referring to, when he took over as sheriff, he inherited a tree with a lot of flakiness not reflected in test_expectations/disabled ui tests. I think very few (if any) of the tests he added to test_expectations had anything to do with the roll. Any policy we make needs to keep in mind that main tree sheriffs deal with flakiness differently; some cross their fingers and hope it goes away, and some do clean up. Maybe we need to get better at enforcing (or automating) adding flaky tests to expectations, so we at least have a clean slate for gardeners to start with. On Tue, Oct 13, 2009 at 11:53 AM, Stephen White senorbla...@chromium.orgwrote: I agree with Dimitri that we're fighting a losing battle here. In my last stint as gardener, I did informally what I proposed formally last time: I spent basically 1 full day just triaging failures from my 2 days gardening. Not fixing, but just running tests locally, analyzing, grouping, creating bugs, assigning to appropriate people (when I knew who they were, cc'ing poor dglazkov when I didn't). So at least I didn't leave a monster bug with layout tests broken by merge #foo but at least grouped by area. That was manageable, but I don't know if another day would actually be enough for a meaningful amount of fixing. I also agree with Drew that actively fixing all the broken tests is usually beyond the skills of any one gardener. Perhaps we should start taking ownership of particular groups of layout tests? And maybe automatically assign them (or least cc them), the same way Area-Foo causes automatic cc'ing in bugs.chromium.org (I think?) That way, the gardener wouldn't have to know who to assign to. I've basically taken responsibility for fixing all layout tests broken by Skia rolls, which can pretty heavy on its own, but I'm willing to take ownership of a directory or two. BTW, the layout test flakiness dashboard has become an invaluable tool for analyzing failures: searching for a test by name is lightning-fast, and you can clearly see if a test has become flaky, on which platforms, and which WebKit merge was responsible, which can also help with grouping. (Props to Ojan for that). Also, it may be Gilbert-and-Sullivan-esque of me, but I think everyone who contributes patches to WebKit for chromium should be on the WebKit gardener rotation. Stephen On Tue, Oct 13, 2009 at 1:53 PM, Drew Wilson atwil...@chromium.orgwrote: I've been thinking quite a bit about this - I agree with Dmitri that the current Sisyphean approach is unsustainable. I don't think the right path is to ask the sheriffs to do the cleanup themselves - for example, a webkit roll that breaks workers in some obscure way is almost certainly beyond the ability of any random gardener to fix in two days, especially when there may be multiple bugs. A better solution would be to have the sheriff (or someone from LTTF) assign the bugs to specific people, with a general rule that such bugs must be fixed within two days (make these bugs the top priority over other tasks). This allows for load balancing of bugs, and also makes sure that we have the right people working on any specific bug. -atw On Tue, Oct 13, 2009 at 10:40 AM, Pam Greene p...@chromium.org wrote: I don't think it's realistic to expect the gardener, or any one person, to be able to fix an arbitrary broken layout test in a reasonable period of time. That's certainly true for new tests, but even for regressions I often can't even tell for sure whether our results are correct, much less what to do if they're not. It's far more efficient to have the right person fix a test. (Of course, people should also strive to broaden their knowledge, but there's a limit to how much of that one can do in a week.) Never having broken layout tests is an excellent goal, but quite frankly I don't think it's one we should prioritize so high that we hobble other efforts and burn out developers. - Pam On Tue, Oct 13, 2009 at 10:31 AM, Dimitri Glazkov dglaz...@chromium.org wrote: I think we need to change something. I am not sure what -- I have ideas, but -- I would appreciate some collective thinking on this. PROBLEM: We accumulate more test failures via WebKit rolls than we fix with our LTTF effort. This ain't right. ANALYSIS: Ok, WebKit
[chromium-dev] talk on Mozilla's Content Security Policy @ Stanford today @ 4:30
Hi all, Someone from Mozilla is talking about their proposed new security spec, CSP, today at Stanford. I'm planning to go; was anyone else from MTV aware of this and hoping to go? I can send out a summary of the talk afterwards if there's interest. https://wiki.mozilla.org/Security/CSP/Spec I have not heard of any discussion of this spec or if we plan to implement it. Anyone have any thoughts? -- Dirk Title: Shutting Down XSS with Content Security Policy Speaker: Sid Stamm, Mozilla Abstract: The last 3 years have seen a dramatic increase in both awareness and exploitation of Web Application Vulnerabilities. 2008 saw dozens of high-profile attacks against websites using Cross Site Scripting (XSS) and Cross Site Request Forgery (CSRF) for the purposes of information stealing, website defacement, malware planting, etc. While an ideal solution may be to develop web applications free from any exploitable vulnerabilities, real world security is usually provided in layers. We present Content Security Policy (CSP), which intends to be one such layer. CSP is a content restrictions policy language and enforcement system that allows site designers or server administrators specify how content interacts on their web sites. We also discuss the long road traveled to a useful policy definition and lessons learned along the way to an implementation in Firefox. 13 Oct (Tuesday) at 1630 hrs Gates 4B (opposite 490) --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: talk on Mozilla's Content Security Policy @ Stanford today @ 4:30
On Tue, Oct 13, 2009 at 1:31 PM, Dirk Pranke dpra...@chromium.org wrote: I have not heard of any discussion of this spec or if we plan to implement it. Anyone have any thoughts? We've mentioned implementing it in the past and Linus has been favorable. I filed https://bugs.webkit.org/show_bug.cgi?id=30081 for getting it in WebKit. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: [LTTF][WebKit Gardening]: Keeping up with the weeds.
On Tue, Oct 13, 2009 at 10:31 AM, Dimitri Glazkov dglaz...@chromium.org wrote: * [ your idea goes here ] I'm probably going to get things thrown at me, but ... I don't think the idea of a webkit sheriff really works. By definition the webkit sheriff is pulling in code that they don't know how to fix and aren't allowed to rollback. So creating bugs is the only option. I feel like the effort of keeping up with upstream (and making any changes required upstream to make that easier) should be centralized in a dedicated team. The fun of building new platform features comes hand-in-hand with the non-fun of keeping up with upstream. Sorry. - a --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: [LTTF][WebKit Gardening]: Keeping up with the weeds.
On Tue, Oct 13, 2009 at 1:36 PM, Aaron Boodman a...@chromium.org wrote: I'm probably going to get things thrown at me, but ... Balls, I just checked with Dimitri and we already do my proposal. Ignore. - a --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: talk on Mozilla's Content Security Policy @ Stanford today @ 4:30
On Tue, Oct 13, 2009 at 1:35 PM, Peter Kasting pkast...@google.com wrote: On Tue, Oct 13, 2009 at 1:31 PM, Dirk Pranke dpra...@chromium.org wrote: I have not heard of any discussion of this spec or if we plan to implement it. Anyone have any thoughts? We've mentioned implementing it in the past and Linus has been favorable. I filed https://bugs.webkit.org/show_bug.cgi?id=30081 for getting it in WebKit. I'll be at the talk. I started a prototype implementation last night in a local Git branch to see what it would look like in code. Adam --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: [LTTF][WebKit Gardening]: Keeping up with the weeds.
Based on the feedback, it sounds like we need to take the approach with LTTF team adding more resources on cleaning up the bottom of the test_expectations file (i.e. stuff recently added by the gardeners). It is still the gardener's responsibility to take care of the rebaselines, right? What should they do with the rest? File bugs? Mark as BUG_LTTF? :DG --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: [LTTF] Flaky tests and setTimeout
This might only apply to the media layout tests, but I'll give the heads up anyway... We found flakiness even when using load if the test is explicitly trying to test for stalled/progress/abort events before the load is completed: http://svn.webkit.org/repository/webkit/trunk/LayoutTests/media/video-error-abort.html What's happening is loadstart fires and the video reloads which should cause an abort event. For some reason load will occasionally fire after loadstart, ending the test. I know we can patch the test, but I've been digging through the event dispatching code to verify that the flakiness isn't limited to Chromium. Andrew On Fri, Oct 9, 2009 at 8:16 PM, Amanda Walker ama...@chromium.org wrote: Most awesome. Very good catch. On Fri, Oct 9, 2009 at 9:52 PM, Julie Parent jpar...@chromium.org wrote: For those of you looking into flaky tests - I've found a surprising number of tests that are flaky because they use a setTimeout to guarantee that a resource (usually an iframe) has loaded. This leads to slower running, flaky tests. To address this, change the tests upstream to use onload instead and make the world a better place :) I'm converting all of the editing tests now. Julie -- Portability is generally the result of advance planning rather than trench warfare involving #ifdef -- Henry Spencer (1992) --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: [LTTF] Flaky tests and setTimeout
On Tue, Oct 13, 2009 at 3:50 PM, Andrew Scherkus scher...@chromium.orgwrote: What's happening is loadstart fires and the video reloads which should cause an abort event. For some reason load will occasionally fire after loadstart, ending the test. I know we can patch the test, but I've been digging through the event dispatching code to verify that the flakiness isn't limited to Chromium. Even if it is just limited to Chromium, it is still a bug, right? Unless I'm not understanding, there's no case under which we should be patching the test in the above case. I don't know anything about video events, so I don't really know what event order guarantees there should be. Ojan --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] seeking webkit reviewer for chrome-specific patches
Both of these patches don't really have an obvious reviewer, but they're pretty simple. https://bugs.webkit.org/show_bug.cgi?id=30319 https://bugs.webkit.org/show_bug.cgi?id=30320 The latter one will require an epic amount of rebaselining, which I have volunteered to do. If anyone has advice on how to do that in a way that makes the webkit sheriffs happy, let me know. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: [LTTF][WebKit Gardening]: Keeping up with the weeds.
I think ownership might actually help with flakiness. Today, in order to distinguish flakiness from real bugs, the gardener needs to have intimate knowledge of the relevant part of the code base and its history. That is beyond the capabilities of the average webkit gardener. Now, imagine a world were every layout test has an owner who can decide intelligently that the bug is flakey and advise the gardener what to do with it. Wouldn't it make gardening much easier? [Flakiness dashboard is very helpful in making the decision, but specialized knowledge topples generic statistics, especially if a test just started flaking] On Tue, Oct 13, 2009 at 1:21 PM, Julie Parent jpar...@chromium.org wrote: I like the idea of ownership of groups of layout tests. Maybe these directory owners could be more like the finders? An owner shouldn't have to necessarily fix everything in a group/directory, but they should be responsible for triaging and getting meaningful bugs filled for them, to keep things moving along. (I volunteer for editing/) Another complicating factor - The state of the main Chrome tree has a lot of effect on the gardener. If the tree is already filled with flakiness, then the webkit roll is likely to show failures, which may or may not have been there before the roll. This was largely the case in the situation pkasting was referring to, when he took over as sheriff, he inherited a tree with a lot of flakiness not reflected in test_expectations/disabled ui tests. I think very few (if any) of the tests he added to test_expectations had anything to do with the roll. Any policy we make needs to keep in mind that main tree sheriffs deal with flakiness differently; some cross their fingers and hope it goes away, and some do clean up. Maybe we need to get better at enforcing (or automating) adding flaky tests to expectations, so we at least have a clean slate for gardeners to start with. On Tue, Oct 13, 2009 at 11:53 AM, Stephen White senorbla...@chromium.orgwrote: I agree with Dimitri that we're fighting a losing battle here. In my last stint as gardener, I did informally what I proposed formally last time: I spent basically 1 full day just triaging failures from my 2 days gardening. Not fixing, but just running tests locally, analyzing, grouping, creating bugs, assigning to appropriate people (when I knew who they were, cc'ing poor dglazkov when I didn't). So at least I didn't leave a monster bug with layout tests broken by merge #foo but at least grouped by area. That was manageable, but I don't know if another day would actually be enough for a meaningful amount of fixing. I also agree with Drew that actively fixing all the broken tests is usually beyond the skills of any one gardener. Perhaps we should start taking ownership of particular groups of layout tests? And maybe automatically assign them (or least cc them), the same way Area-Foo causes automatic cc'ing in bugs.chromium.org (I think?) That way, the gardener wouldn't have to know who to assign to. I've basically taken responsibility for fixing all layout tests broken by Skia rolls, which can pretty heavy on its own, but I'm willing to take ownership of a directory or two. BTW, the layout test flakiness dashboard has become an invaluable tool for analyzing failures: searching for a test by name is lightning-fast, and you can clearly see if a test has become flaky, on which platforms, and which WebKit merge was responsible, which can also help with grouping. (Props to Ojan for that). Also, it may be Gilbert-and-Sullivan-esque of me, but I think everyone who contributes patches to WebKit for chromium should be on the WebKit gardener rotation. Stephen On Tue, Oct 13, 2009 at 1:53 PM, Drew Wilson atwil...@chromium.orgwrote: I've been thinking quite a bit about this - I agree with Dmitri that the current Sisyphean approach is unsustainable. I don't think the right path is to ask the sheriffs to do the cleanup themselves - for example, a webkit roll that breaks workers in some obscure way is almost certainly beyond the ability of any random gardener to fix in two days, especially when there may be multiple bugs. A better solution would be to have the sheriff (or someone from LTTF) assign the bugs to specific people, with a general rule that such bugs must be fixed within two days (make these bugs the top priority over other tasks). This allows for load balancing of bugs, and also makes sure that we have the right people working on any specific bug. -atw On Tue, Oct 13, 2009 at 10:40 AM, Pam Greene p...@chromium.org wrote: I don't think it's realistic to expect the gardener, or any one person, to be able to fix an arbitrary broken layout test in a reasonable period of time. That's certainly true for new tests, but even for regressions I often can't even tell for sure whether our results are correct, much less what to do if they're not. It's far more
[chromium-dev] Re: seeking webkit reviewer for chrome-specific patches
Be sheriff that day :) Real advice: Once you have webkit patch R+'ed and chrome rebaselines ready, let the gardener know. Once the gardener is caught up, they can set commit-queue flag on your change, so it gets committed at a time when they are ready to deal with it and your follow up change will be there ready to fix everything. On Tue, Oct 13, 2009 at 4:06 PM, Evan Martin e...@chromium.org wrote: Both of these patches don't really have an obvious reviewer, but they're pretty simple. https://bugs.webkit.org/show_bug.cgi?id=30319 https://bugs.webkit.org/show_bug.cgi?id=30320 The latter one will require an epic amount of rebaselining, which I have volunteered to do. If anyone has advice on how to do that in a way that makes the webkit sheriffs happy, let me know. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: [LTTF][WebKit Gardening]: Keeping up with the weeds.
On Tue, Oct 13, 2009 at 3:46 PM, Dimitri Glazkov dglaz...@chromium.orgwrote: Based on the feedback, it sounds like we need to take the approach with LTTF team adding more resources on cleaning up the bottom of the test_expectations file (i.e. stuff recently added by the gardeners). It is still the gardener's responsibility to take care of the rebaselines, right? What should they do with the rest? File bugs? Mark as BUG_LTTF? I agree with the need to have some method of assigning failed tests and have someone fix them soon. I'm not sure the sheriff is the best person for the reasons given above. However one thing that I think we need to be very diligent in is assigning bugs to the correct person. The sheriff usually creates bugs, but often they may not have an owner, the owner might be left to the default which is themselves, etc. The person who owns that area is then unaware that tests in their area is failing. I wish that any bugs that are created, whether because of a failing layout test or a Chrome unit test, must have an owner. This shouldn't take much time by going through the svn log or asking on #chromium. :DG --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: seeking webkit reviewer for chrome-specific patches
I glanced at the other one. there were a few issues with it: 1. It does more than it says. It changes how the horizontal lines are drawn without explanation (nothing in the changelog and the bug only mentions the other direction). 2. It uses a lot of magic numbers and the same ones repeatedly. It would be nice to make constants out of some of them to help inform the reader where they come from. 3. It is unclear if there are any layout tests which cover this change. I would be happy to r- if you want :) but am uncomfortable with an r+ due to those reasons. On Tue, Oct 13, 2009 at 5:25 PM, Evan Martin e...@chromium.org wrote: dimich looked at one. Any reviewer for the other? It's like 8 obvious Linux-specific lines, no stress I promise. :) On Tue, Oct 13, 2009 at 4:06 PM, Evan Martin e...@chromium.org wrote: Both of these patches don't really have an obvious reviewer, but they're pretty simple. https://bugs.webkit.org/show_bug.cgi?id=30319 https://bugs.webkit.org/show_bug.cgi?id=30320 The latter one will require an epic amount of rebaselining, which I have volunteered to do. If anyone has advice on how to do that in a way that makes the webkit sheriffs happy, let me know. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: seeking webkit reviewer for chrome-specific patches
3. It is unclear if there are any layout tests which cover this change. I don't think layout tests can or should cover that change, except for pixel tests, which will just need to be rebaselined. --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] FYI: new tab cold slowdown
In r28924, I made a change to the new tab tests so it the data in the history database would be recent. Because of this, the test now shows 8 most recently visited thumbnails when it was previously showing the 2 stock thumbnails. This causes a slight regression in the new tab cold times, but it's actually a new baseline. It gives us something more realistic to try to optimize. A long time ago, this is how the test was originally designed, it is a regression that we stopped showing the thumbnails in the test. tony --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Querying and casting NPObject
In my implementation for PlainTextController (see http://codereview.chromium.org/243032) for the test shell I have run into some issues for which I'm not sure what's the best way to address them. Thusly I wanted to ask around if someone could give me some pointers: .) The implementation of PlainTextController::plainText() should check whether the passed-in argument is indeed a range. At the moment I have a test (not yet submitted) that looks for a range-specific method, using WebKit::WebBindings::hasMethod(setStartBefore) that seems to work. However, this is rather hackish. The WebKit (Mac) version of PlainTextController uses 'isKindOfClass'. Is there a way to emulate this, or another workaround that is more elegant than what I described above? .) To retrieve the result, my current implementation invokes toString. For parity with the WebKit implementation of PlainTextController however, PlainTextController::plainText() should rather use a special text() method on range that is not exposed in the IDL. As I don't think extending the IDL is an option, is there a way to cast a NPObject to a WebRange object so that I can call the function directly? Any pointers and suggestions would be very helpful. Cheers, Roland --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: [LTTF][WebKit Gardening]: Keeping up with the weeds.
Ownership is a great concept. I started out planning LTTF as ownership-based. Unfortunately, the types of failures are scattered far and wide across the directories, some clustered, some not. After a few initial passes, I walked away thinking that it's not as simple as drawing the lines and basically gave up. That's how Finders/Fixers idea was born. :DG On Tue, Oct 13, 2009 at 4:24 PM, Yaar Schnitman y...@chromium.org wrote: I think ownership might actually help with flakiness. Today, in order to distinguish flakiness from real bugs, the gardener needs to have intimate knowledge of the relevant part of the code base and its history. That is beyond the capabilities of the average webkit gardener. Now, imagine a world were every layout test has an owner who can decide intelligently that the bug is flakey and advise the gardener what to do with it. Wouldn't it make gardening much easier? [Flakiness dashboard is very helpful in making the decision, but specialized knowledge topples generic statistics, especially if a test just started flaking] On Tue, Oct 13, 2009 at 1:21 PM, Julie Parent jpar...@chromium.org wrote: I like the idea of ownership of groups of layout tests. Maybe these directory owners could be more like the finders? An owner shouldn't have to necessarily fix everything in a group/directory, but they should be responsible for triaging and getting meaningful bugs filled for them, to keep things moving along. (I volunteer for editing/) Another complicating factor - The state of the main Chrome tree has a lot of effect on the gardener. If the tree is already filled with flakiness, then the webkit roll is likely to show failures, which may or may not have been there before the roll. This was largely the case in the situation pkasting was referring to, when he took over as sheriff, he inherited a tree with a lot of flakiness not reflected in test_expectations/disabled ui tests. I think very few (if any) of the tests he added to test_expectations had anything to do with the roll. Any policy we make needs to keep in mind that main tree sheriffs deal with flakiness differently; some cross their fingers and hope it goes away, and some do clean up. Maybe we need to get better at enforcing (or automating) adding flaky tests to expectations, so we at least have a clean slate for gardeners to start with. On Tue, Oct 13, 2009 at 11:53 AM, Stephen White senorbla...@chromium.org wrote: I agree with Dimitri that we're fighting a losing battle here. In my last stint as gardener, I did informally what I proposed formally last time: I spent basically 1 full day just triaging failures from my 2 days gardening. Not fixing, but just running tests locally, analyzing, grouping, creating bugs, assigning to appropriate people (when I knew who they were, cc'ing poor dglazkov when I didn't). So at least I didn't leave a monster bug with layout tests broken by merge #foo but at least grouped by area. That was manageable, but I don't know if another day would actually be enough for a meaningful amount of fixing. I also agree with Drew that actively fixing all the broken tests is usually beyond the skills of any one gardener. Perhaps we should start taking ownership of particular groups of layout tests? And maybe automatically assign them (or least cc them), the same way Area-Foo causes automatic cc'ing in bugs.chromium.org (I think?) That way, the gardener wouldn't have to know who to assign to. I've basically taken responsibility for fixing all layout tests broken by Skia rolls, which can pretty heavy on its own, but I'm willing to take ownership of a directory or two. BTW, the layout test flakiness dashboard has become an invaluable tool for analyzing failures: searching for a test by name is lightning-fast, and you can clearly see if a test has become flaky, on which platforms, and which WebKit merge was responsible, which can also help with grouping. (Props to Ojan for that). Also, it may be Gilbert-and-Sullivan-esque of me, but I think everyone who contributes patches to WebKit for chromium should be on the WebKit gardener rotation. Stephen On Tue, Oct 13, 2009 at 1:53 PM, Drew Wilson atwil...@chromium.org wrote: I've been thinking quite a bit about this - I agree with Dmitri that the current Sisyphean approach is unsustainable. I don't think the right path is to ask the sheriffs to do the cleanup themselves - for example, a webkit roll that breaks workers in some obscure way is almost certainly beyond the ability of any random gardener to fix in two days, especially when there may be multiple bugs. A better solution would be to have the sheriff (or someone from LTTF) assign the bugs to specific people, with a general rule that such bugs must be fixed within two days (make these bugs the top priority over other tasks). This allows for load balancing of bugs, and also makes sure that we have the right people working on any
[chromium-dev] Re: [LTTF][WebKit Gardening]: Keeping up with the weeds.
If there are areas that nobody knows anything about, that's a lack that's hobbling us. Suppose we take the entire list of directories, slap it into a doc, and assign at least one owner to everything. For the areas that don't yet have anyone knowledgeable, we take volunteers to *become* knowledgeable if needed. It will be a valuable investment. Tests often fail due to problems outside their nominal tested areas, but the area owner would still be better than an arbitrary gardener at recognizing that and reassigning the bug. - Pam On Tue, Oct 13, 2009 at 10:09 PM, Dimitri Glazkov dglaz...@chromium.orgwrote: Ownership is a great concept. I started out planning LTTF as ownership-based. Unfortunately, the types of failures are scattered far and wide across the directories, some clustered, some not. After a few initial passes, I walked away thinking that it's not as simple as drawing the lines and basically gave up. That's how Finders/Fixers idea was born. :DG On Tue, Oct 13, 2009 at 4:24 PM, Yaar Schnitman y...@chromium.org wrote: I think ownership might actually help with flakiness. Today, in order to distinguish flakiness from real bugs, the gardener needs to have intimate knowledge of the relevant part of the code base and its history. That is beyond the capabilities of the average webkit gardener. Now, imagine a world were every layout test has an owner who can decide intelligently that the bug is flakey and advise the gardener what to do with it. Wouldn't it make gardening much easier? [Flakiness dashboard is very helpful in making the decision, but specialized knowledge topples generic statistics, especially if a test just started flaking] On Tue, Oct 13, 2009 at 1:21 PM, Julie Parent jpar...@chromium.org wrote: I like the idea of ownership of groups of layout tests. Maybe these directory owners could be more like the finders? An owner shouldn't have to necessarily fix everything in a group/directory, but they should be responsible for triaging and getting meaningful bugs filled for them, to keep things moving along. (I volunteer for editing/) Another complicating factor - The state of the main Chrome tree has a lot of effect on the gardener. If the tree is already filled with flakiness, then the webkit roll is likely to show failures, which may or may not have been there before the roll. This was largely the case in the situation pkasting was referring to, when he took over as sheriff, he inherited a tree with a lot of flakiness not reflected in test_expectations/disabled ui tests. I think very few (if any) of the tests he added to test_expectations had anything to do with the roll. Any policy we make needs to keep in mind that main tree sheriffs deal with flakiness differently; some cross their fingers and hope it goes away, and some do clean up. Maybe we need to get better at enforcing (or automating) adding flaky tests to expectations, so we at least have a clean slate for gardeners to start with. On Tue, Oct 13, 2009 at 11:53 AM, Stephen White senorbla...@chromium.org wrote: I agree with Dimitri that we're fighting a losing battle here. In my last stint as gardener, I did informally what I proposed formally last time: I spent basically 1 full day just triaging failures from my 2 days gardening. Not fixing, but just running tests locally, analyzing, grouping, creating bugs, assigning to appropriate people (when I knew who they were, cc'ing poor dglazkov when I didn't). So at least I didn't leave a monster bug with layout tests broken by merge #foo but at least grouped by area. That was manageable, but I don't know if another day would actually be enough for a meaningful amount of fixing. I also agree with Drew that actively fixing all the broken tests is usually beyond the skills of any one gardener. Perhaps we should start taking ownership of particular groups of layout tests? And maybe automatically assign them (or least cc them), the same way Area-Foo causes automatic cc'ing in bugs.chromium.org (I think?) That way, the gardener wouldn't have to know who to assign to. I've basically taken responsibility for fixing all layout tests broken by Skia rolls, which can pretty heavy on its own, but I'm willing to take ownership of a directory or two. BTW, the layout test flakiness dashboard has become an invaluable tool for analyzing failures: searching for a test by name is lightning-fast, and you can clearly see if a test has become flaky, on which platforms, and which WebKit merge was responsible, which can also help with grouping. (Props to Ojan for that). Also, it may be Gilbert-and-Sullivan-esque of me, but I think everyone who contributes patches to WebKit for chromium should be on the WebKit gardener rotation. Stephen On Tue, Oct 13, 2009 at 1:53 PM, Drew Wilson atwil...@chromium.org wrote: I've been thinking quite a
