[chromium-dev] Re: Webkit merges and tree closures
On Tue, Sep 22, 2009 at 8:40 PM, Nicolas Sylvain nsylv...@chromium.orgwrote: Oh, and, if your change turned the tree red for 3 hours, don't be mad at the sheriff when he pings you repeatedly about the status of the fix. His job is to keep the tree green and running. He does not care about your change. Thoughts? Generally when this happens it's not the webkit gardener's change that has broken everything--it's an upstream change (usually by someone working on chromium, though not always). Since we are committed to taking all upstream webkit changes, and webkit continues to roll along while we clean up the damage, the webkit gardener's only choice is roll now or roll later. If the canary is red, roll later just means that more breakage will pile up behind it. Jeremy and Dmitry have covered most of what I was going to say; I agree that the real solution is to keep pedaling hard to finish the webkit API. Until we do that, all we're doing is to rearranging the pain, not reducing it. --Amanda --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Webkit merges and tree closures
WebKit gardening occurs more often than sheriff duties. afaik all WebKit gardeners also have sheriff duties. This implies there are people on the team who don't have gardening duties. We need to fix that ASAP. Nobody should get special dispensation. -- Mike Pinkerton Mac Weenie pinker...@google.com --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Webkit merges and tree closures
On Wed, Sep 23, 2009 at 6:52 AM, Amanda Walker ama...@chromium.org wrote: ... I agree that the real solution is to keep pedaling hard to finish the webkit API. If anyone has cycles to help with the WebKit API, please let me know! :-) -Darin --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Webkit merges and tree closures
It is hard to be a WebKit gardener if you do not have WebKit commit access. Sometimes the gardener has to commit a quick bustage fix upstream or roll back a fellow Chromium committers change to WebKit. -Darin Correct, it is hard, but many/most of us who are webkit sheriffs/gardeners are not webkit committers (for example, the entire mac group). I don't understand your point. Are you saying that only webkit committers should be on the webkit sheriff rotation? -- Mike Pinkerton Mac Weenie pinker...@google.com --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Webkit merges and tree closures
I find that being a WebKit gardener is always dancing on a minefield regardless of my familiarity with the WebKit code base. Look at yesterday for an example. In addition, we have several gardeners who are not actively working on WebKit (amanda@ has 0 WebKit commits, pinkerton@ has a few all in 2008). Lastly, can we add anyone to the rotation who has ideas how WebKit gardeners should/can do their job better? (This will help them in getting more ideas.) Dave On Wed, Sep 23, 2009 at 7:53 AM, Dimitri Glazkov dglaz...@chromium.orgwrote: On Wed, Sep 23, 2009 at 7:46 AM, Mike Pinkerton pinker...@chromium.org wrote: It is hard to be a WebKit gardener if you do not have WebKit commit access. Sometimes the gardener has to commit a quick bustage fix upstream or roll back a fellow Chromium committers change to WebKit. -Darin Correct, it is hard, but many/most of us who are webkit sheriffs/gardeners are not webkit committers (for example, the entire mac group). I don't understand your point. Are you saying that only webkit committers should be on the webkit sheriff rotation? In my experience, better familiarity with WebKit code base is a huge advantage for a gardener. I am almost tempted to say that if you're not actively working on WebKit, being a gardener will be a foreign and dancing-on-a-minefield-type task. :DG --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Webkit merges and tree closures
I'll take the action item to remove anyone in the rotation who is not actively working on Webkit / is not a committer. In the past, I've floated the idea in the past of having a Webkit deputy who helps the gardener keep the canary green. I'm not sure if that would help, though. Regards, Glenn On Wed, Sep 23, 2009 at 8:40 AM, David Levin le...@chromium.org wrote: I find that being a WebKit gardener is always dancing on a minefield regardless of my familiarity with the WebKit code base. Look at yesterday for an example. In addition, we have several gardeners who are not actively working on WebKit (amanda@ has 0 WebKit commits, pinkerton@ has a few all in 2008). Lastly, can we add anyone to the rotation who has ideas how WebKit gardeners should/can do their job better? (This will help them in getting more ideas.) Dave On Wed, Sep 23, 2009 at 7:53 AM, Dimitri Glazkov dglaz...@chromium.orgwrote: On Wed, Sep 23, 2009 at 7:46 AM, Mike Pinkerton pinker...@chromium.org wrote: It is hard to be a WebKit gardener if you do not have WebKit commit access. Sometimes the gardener has to commit a quick bustage fix upstream or roll back a fellow Chromium committers change to WebKit. -Darin Correct, it is hard, but many/most of us who are webkit sheriffs/gardeners are not webkit committers (for example, the entire mac group). I don't understand your point. Are you saying that only webkit committers should be on the webkit sheriff rotation? In my experience, better familiarity with WebKit code base is a huge advantage for a gardener. I am almost tempted to say that if you're not actively working on WebKit, being a gardener will be a foreign and dancing-on-a-minefield-type task. :DG --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Webkit merges and tree closures
I think it would be a good idea to ensure that there is at least someone on call to help new gardeners if they have questions or find themselves in a mess. I know I just asked dglazkov whenever I wasn't sure what to do, but he'd probably appreciate not being the only designated knowledgeable person. :-) On Wed, Sep 23, 2009 at 8:53 AM, Glenn Wilson gwil...@chromium.org wrote: I'll take the action item to remove anyone in the rotation who is not actively working on Webkit / is not a committer. In the past, I've floated the idea in the past of having a Webkit deputy who helps the gardener keep the canary green. I'm not sure if that would help, though. Regards, Glenn On Wed, Sep 23, 2009 at 8:40 AM, David Levin le...@chromium.org wrote: I find that being a WebKit gardener is always dancing on a minefield regardless of my familiarity with the WebKit code base. Look at yesterday for an example. In addition, we have several gardeners who are not actively working on WebKit (amanda@ has 0 WebKit commits, pinkerton@ has a few all in 2008). Lastly, can we add anyone to the rotation who has ideas how WebKit gardeners should/can do their job better? (This will help them in getting more ideas.) Dave On Wed, Sep 23, 2009 at 7:53 AM, Dimitri Glazkov dglaz...@chromium.orgwrote: On Wed, Sep 23, 2009 at 7:46 AM, Mike Pinkerton pinker...@chromium.org wrote: It is hard to be a WebKit gardener if you do not have WebKit commit access. Sometimes the gardener has to commit a quick bustage fix upstream or roll back a fellow Chromium committers change to WebKit. -Darin Correct, it is hard, but many/most of us who are webkit sheriffs/gardeners are not webkit committers (for example, the entire mac group). I don't understand your point. Are you saying that only webkit committers should be on the webkit sheriff rotation? In my experience, better familiarity with WebKit code base is a huge advantage for a gardener. I am almost tempted to say that if you're not actively working on WebKit, being a gardener will be a foreign and dancing-on-a-minefield-type task. :DG --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Webkit merges and tree closures
On Wed, Sep 23, 2009 at 8:53 AM, Glenn Wilson gwil...@chromium.org wrote: I'll take the action item to remove anyone in the rotation who is not actively working on Webkit / is not a committer. Oh no... now it will happen even more often :) Should there also be the reverse adding WebKit committers who are not gardeners to the rotation? In the past, I've floated the idea in the past of having a Webkit deputy who helps the gardener keep the canary green. I'm not sure if that would help, though. fwiw, earlier in the thread I gave 5 things that would definitely help -- For those concerned about this issue, if you tackle these, it would make things go smoother. dave --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Webkit merges and tree closures
On Tue, Sep 22, 2009 at 10:47 PM, David Levin le...@chromium.org wrote: Actionable items for keeping the tree green (in addition to blaming the WebKit gardener for [insert action here]): - *Get people putting in chromium patches upstream to run their changes through trybots, etc*. imo, patches from @chromium folks cause well over 50% of the grief with WebKit rolls (and usually the worst issues like today). - As Adam suggested, *make changes to the WebKit commit queue to run items through the chromium try bots*. - *WebKit gardeners should be able to submit high priority jobs to the try bots*. These jobs should job to the front of any queues so that they run asap. Why? Time is critical when doing the gardening because if you find a breakage upstream, you need to check in a fix and then roll to that fix. The longer the delay, the more likely another breakage will be checked in. I haven't seen a queue on the try bots in weeks now. If you do see one, and your change is not tested on all platforms as soon as you upload it, please let me know and we'll allocate more resources. Nicolas - *Consider auto-rolling WebKit deps*. Have the something like a parallel buildbot that runs against tip of tree WebKit. If everything passes except layout tests, add any layout test failures to test_expectations.txt (if there are less than 15) and roll DEPS on passing. If things fail, then turn red. - *Make it easier/faster to disable tests and file bugs about them *(using the last person in the blame/annotate for the test as the initial assignee or auto-assign it to the sheriff so (s)he can assign it to the right person) * *because issues will slip from WebKit rolls even though the gardeners try to be thorough. Also, this should help with turning the tree green in other cases as well. The sheriff (and everyone on the chromium team) should care about the WebKit roll as this is critical to the success of this project. Frequent rolls, should isolate issues and hopefully help to keep the tree green. * * To help shed some light on why WebKit gardening is more painful than sheriffing: 1. WebKit gardeners are all alone in trying to deal with things. 2. When things go bad on the canary, no one shuts down the tree for you and any changes to help with merging are not given priority (if the tree is red and you have an innocuous change to fix the webkit merge, you won't get it in.) 3. When tests fail anywhere (on the canary, when committing the roll, etc.), you have to figure out why, typically for a lot of changes that you know nothing about. 4. Two days of gardening -- multiple days of clean up afterward. 5. WebKit gardening occurs more often than sheriff duties. 6. afaik all WebKit gardeners also have sheriff duties. Net: chromium sheriffs please be willing to give a little extra help to the WebKit gardener. Remember that their hair is turning white as they try to run in front of a locomotive. Dave --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Webkit merges and tree closures
On Tue, Sep 22, 2009 at 5:40 PM, Nicolas Sylvain nsylv...@chromium.orgwrote: If this is an issue, I am proposing that Webkit merges be done outside peak hours (11am-5pm pacific). If we implement this, it can cause problems for cases where we need to do a merge/land/merge pattern to coordinate landing a local and upstream fix without having the canary bot broken for hours (which is really bad). I think we'll have to allow merges to land during daytime hours. We should definitely document on the WebKit Merge wiki/dev site page the best practices for running the merge through various try servers, even if that info is already available elsewhere. PK --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Webkit merges and tree closures
On Tue, Sep 22, 2009 at 17:40, Nicolas Sylvain nsylv...@chromium.orgwrote: 3 PM : two failing ui tests are disabled by the webkit sheriff I was looking at the UI tests and it wasn't immediately obvious that a webkit update might break them. Can we run all the UI tests on the webkit canary bot? If that takes too long, we can selectively exclude the slow tests, based on http://build.chromium.org/buildbot/slowness-report/ By the way, the layout test failures... can we catch them on the webkit canary bot(s)? --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Webkit merges and tree closures
On Tue, Sep 22, 2009 at 5:40 PM, Nicolas Sylvain nsylv...@chromium.org wrote: If this is an issue, I am proposing that Webkit merges be done outside peak hours (11am-5pm pacific). This seems backwards. Don't we want to integrate more often so we can catch and fix these issue faster? Ideally, we'll be able to merge with every webkit.org commit once we get the WebKit API in place. Adam --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Webkit merges and tree closures
There are 2 major issues here (besides leaving things for the Sheriff to clean up): 1) a lot of the gardeners are inexperienced and drop the ball. This has bitten us many times. The last time we had a big string of problems related to this, I meant to send out an email giving people advice on what to expect and how to prepare. I will do it this time. Hopefully people listen. 2) we don't have enough tools for gardeners to do their jobs. As I mentioned in another thread you started the other day, we really need more try bots and/or canaries that run memory tests, our full suite of tests, etc. Without that, things are not going to get better. Just to be clear, on bad days, gardening is WAY harder than sheriffing by yourself. Like you mentioned, reverting a merge is pretty much not an option because you only get further behind, which makes things harder. If several hour tree closures once or twice a week are not an option, then we need more bots. J On Tue, Sep 22, 2009 at 5:50 PM, Paweł Hajdan Jr. phajdan...@chromium.orgwrote: On Tue, Sep 22, 2009 at 17:40, Nicolas Sylvain nsylv...@chromium.orgwrote: 3 PM : two failing ui tests are disabled by the webkit sheriff I was looking at the UI tests and it wasn't immediately obvious that a webkit update might break them. Can we run all the UI tests on the webkit canary bot? If that takes too long, we can selectively exclude the slow tests, based on http://build.chromium.org/buildbot/slowness-report/ By the way, the layout test failures... can we catch them on the webkit canary bot(s)? --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Webkit merges and tree closures
On Tue, Sep 22, 2009 at 6:08 PM, Adam Barth aba...@chromium.org wrote: On Tue, Sep 22, 2009 at 5:40 PM, Nicolas Sylvain nsylv...@chromium.org wrote: If this is an issue, I am proposing that Webkit merges be done outside peak hours (11am-5pm pacific). This seems backwards. Don't we want to integrate more often so we can catch and fix these issue faster? Ideally, we'll be able to merge with every webkit.org commit once we get the WebKit API in place. I really want to stress the first part of my sentence: if this is an issue . For small webkit merges, I really don't expect this to be an issue. Once the API is in place, I'm pretty sure we won't have this problem. Or at least not as often. Nicolas Adam --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Webkit merges and tree closures
On Tue, Sep 22, 2009 at 6:08 PM, Adam Barth aba...@chromium.org wrote: On Tue, Sep 22, 2009 at 5:40 PM, Nicolas Sylvain nsylv...@chromium.org wrote: If this is an issue, I am proposing that Webkit merges be done outside peak hours (11am-5pm pacific). This seems backwards. Don't we want to integrate more often so we can catch and fix these issue faster? Ideally, we'll be able to merge with every webkit.org commit once we get the WebKit API in place. Indeed. I would rather see small rolls throughout the day. Trying to accumulate them into larger spans will result in larger blasts. :DG --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Webkit merges and tree closures
On Tue, Sep 22, 2009 at 6:16 PM, Nicolas Sylvain nsylv...@chromium.orgwrote: On Tue, Sep 22, 2009 at 6:10 PM, Jeremy Orlow jor...@chromium.org wrote: There are 2 major issues here (besides leaving things for the Sheriff to clean up): 1) a lot of the gardeners are inexperienced and drop the ball. This has bitten us many times. The last time we had a big string of problems related to this, I meant to send out an email giving people advice on what to expect and how to prepare. I will do it this time. Hopefully people listen. 2) we don't have enough tools for gardeners to do their jobs. As I mentioned in another thread you started the other day, we really need more try bots and/or canaries that run memory tests, our full suite of tests, etc. Without that, things are not going to get better. Just to be clear, on bad days, gardening is WAY harder than sheriffing by yourself. Like you mentioned, reverting a merge is pretty much not an option because you only get further behind, which makes things harder. If several hour tree closures once or twice a week are not an option, then we need more bots. We are creating bots as fast as we can. We delivered almost (all?) of them you asked for. You have purify canary, valgrind canary, perf tests canary, and selenium tests canary. You also have the layout tests try slaves, and the valgrind try slave. I guess the one that remains in ui tests? We can work on that and make it happen. (which you have a try server for in the mean time). Sorry, I haven't gardened in a month or so, so I'm obviously behind the times. These will DEFINITELY help. :-) Nicolas J On Tue, Sep 22, 2009 at 5:50 PM, Paweł Hajdan Jr. phajdan...@chromium.org wrote: On Tue, Sep 22, 2009 at 17:40, Nicolas Sylvain nsylv...@chromium.orgwrote: 3 PM : two failing ui tests are disabled by the webkit sheriff I was looking at the UI tests and it wasn't immediately obvious that a webkit update might break them. Can we run all the UI tests on the webkit canary bot? If that takes too long, we can selectively exclude the slow tests, based on http://build.chromium.org/buildbot/slowness-report/ By the way, the layout test failures... can we catch them on the webkit canary bot(s)? --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
[chromium-dev] Re: Webkit merges and tree closures
Actionable items for keeping the tree green (in addition to blaming the WebKit gardener for [insert action here]): - *Get people putting in chromium patches upstream to run their changes through trybots, etc*. imo, patches from @chromium folks cause well over 50% of the grief with WebKit rolls (and usually the worst issues like today). - As Adam suggested, *make changes to the WebKit commit queue to run items through the chromium try bots*. - *WebKit gardeners should be able to submit high priority jobs to the try bots*. These jobs should job to the front of any queues so that they run asap. Why? Time is critical when doing the gardening because if you find a breakage upstream, you need to check in a fix and then roll to that fix. The longer the delay, the more likely another breakage will be checked in. - *Consider auto-rolling WebKit deps*. Have the something like a parallel buildbot that runs against tip of tree WebKit. If everything passes except layout tests, add any layout test failures to test_expectations.txt (if there are less than 15) and roll DEPS on passing. If things fail, then turn red. - *Make it easier/faster to disable tests and file bugs about them *(using the last person in the blame/annotate for the test as the initial assignee or auto-assign it to the sheriff so (s)he can assign it to the right person) * *because issues will slip from WebKit rolls even though the gardeners try to be thorough. Also, this should help with turning the tree green in other cases as well. The sheriff (and everyone on the chromium team) should care about the WebKit roll as this is critical to the success of this project. Frequent rolls, should isolate issues and hopefully help to keep the tree green. * * To help shed some light on why WebKit gardening is more painful than sheriffing: 1. WebKit gardeners are all alone in trying to deal with things. 2. When things go bad on the canary, no one shuts down the tree for you and any changes to help with merging are not given priority (if the tree is red and you have an innocuous change to fix the webkit merge, you won't get it in.) 3. When tests fail anywhere (on the canary, when committing the roll, etc.), you have to figure out why, typically for a lot of changes that you know nothing about. 4. Two days of gardening -- multiple days of clean up afterward. 5. WebKit gardening occurs more often than sheriff duties. 6. afaik all WebKit gardeners also have sheriff duties. Net: chromium sheriffs please be willing to give a little extra help to the WebKit gardener. Remember that their hair is turning white as they try to run in front of a locomotive. Dave --~--~-~--~~~---~--~~ Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -~--~~~~--~~--~--~---
