Re: [chromium-dev] More sheriffs?
On Fri, Nov 13, 2009 at 12:35 PM, Ben Goodger b...@google.com wrote: On Fri, Nov 13, 2009 at 12:31 PM, Peter Kasting pkast...@google.com wrote: * The team is large enough that on the current schedule, you go months between sheriffing, which is so long that you forget things like what tools help you do what. This info should be written down and kept up to date by sheriffs on a daily basis. See http://dev.chromium.org/developers/tree-sheriffs , which is linked off our main developer wiki page. PK -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
Re: [chromium-dev] More sheriffs?
On Fri, Nov 13, 2009 at 12:44 PM, Stuart Morgan stuartmor...@google.comwrote: If we end up actually having four at a time that seems likely to be worse than two: either four people are doing nothing but sheriffing, which there is probably not enough work for, or all four people are more likely to think that someone else is probably watching and they can do something else. I can only say that in my own sheriffing experience that this is utterly untrue, and having two people at once is amazingly helpful since we can track down different problem areas; one working on purify and valgrind errors while another works on layout tests. There has never been a time in such cases where we both did nothing because we thought the other person was working on it; we were always pinging each other and dividing work on the fly. I don't think Chromium team members are so irresponsible that they would not work out some system in such cases. And part of the point is that it would be nice to be able to get a _little_ bit of work done on the days you're sheriffing, or go to lunch, or whatever. PK -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
Re: [chromium-dev] More sheriffs?
For a while now, I've advocated having 2 pacific timezone sheriffs always on duty and having one or two in other time zones. I still advocate such an idea. So, to be clear, I think this is a good idea as long as the distribution of sheriffs (time zone wise) is deliberate. (I think this addresses Stuart's concern as well.) J On Fri, Nov 13, 2009 at 12:31 PM, Peter Kasting pkast...@google.com wrote: At lunch today, a few of us discussed the idea of moving from two sheriffs to four. There are several reasons we contemplated such a change: * The team is large enough that on the current schedule, you go months between sheriffing, which is so long that you forget things like what tools help you do what. * Sheriffing is a heavy burden, and getting moreso with more team members. * Either the two sheriffs are in different time zones, in which case you have effectively one sheriff on duty who has to do everything (bad due to point above), or they're not, in which case a chunk of the day is not covered at all. * New sheriffs could really use a mentor sheriff with them, which is pretty difficult to schedule. I think these are good reasons, so I propose we make this change. Comments? PK -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
Re: [chromium-dev] More sheriffs?
(resending to chromium-dev) Sheriffing the PST time zone is usually the worst. We could experiment with tweaking the scheduling algorithm to have two PST sheriffs and one non-PST sheriff per shift. Other than that -- fixing flaky tests would go a long way to making the job easier. Right now out of 12 failing bots, only 1 is a true failure. On Fri, Nov 13, 2009 at 12:48 PM, Peter Kasting pkast...@google.com wrote: On Fri, Nov 13, 2009 at 12:44 PM, Stuart Morgan stuartmor...@google.comwrote: If we end up actually having four at a time that seems likely to be worse than two: either four people are doing nothing but sheriffing, which there is probably not enough work for, or all four people are more likely to think that someone else is probably watching and they can do something else. I can only say that in my own sheriffing experience that this is utterly untrue, and having two people at once is amazingly helpful since we can track down different problem areas; one working on purify and valgrind errors while another works on layout tests. There has never been a time in such cases where we both did nothing because we thought the other person was working on it; we were always pinging each other and dividing work on the fly. I don't think Chromium team members are so irresponsible that they would not work out some system in such cases. And part of the point is that it would be nice to be able to get a _little_ bit of work done on the days you're sheriffing, or go to lunch, or whatever. PK -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
Re: [chromium-dev] More sheriffs?
On Fri, Nov 13, 2009 at 1:15 PM, Finnur Thorarinsson fin...@google.comwrote: If the sheriff load is too much for two people to devote 100% of their time to, then there is something wrong with the process. It's clearly too much, given that I hardly see any other sheriffs even attempt to maintain the rule of every bot green all the time, which is what you're supposed to do as sheriff. And when I maintain it, I need to keep the tree closed for long periods while I deal with the myriad of issues that come up. Solving the problem by having the tree open if things aren't too bad is not good enough. Right now I just checked and the purify and valgrind bots were red. As usual. No sign of anyone looking into them. Sheriffs are in theory supposed to watch all the perf bots too. Do you? I don't. I doubt very many people do. There is tons of information available to sheriffs and too few people to cover it. Someone watching perf, someone watching purify/valgrind, someone watching layout tests, and someone watching everything else would be really helpful. Especially if one of those people was experienced enough to help somebody else doing it for the first time. The team is growing fast enough that we have a _lot_ of first-time sheriffs. PK -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
Re: [chromium-dev] More sheriffs?
Big +1 for at least a third sheriff. With two sheriffs, if one is not in PST, then really we only have one sheriff. If that sheriff happens to be new, then we have 0 = num_sheriffs = 1. On Fri, Nov 13, 2009 at 12:31 PM, Peter Kasting pkast...@google.com wrote: At lunch today, a few of us discussed the idea of moving from two sheriffs to four. There are several reasons we contemplated such a change: * The team is large enough that on the current schedule, you go months between sheriffing, which is so long that you forget things like what tools help you do what. * Sheriffing is a heavy burden, and getting moreso with more team members. * Either the two sheriffs are in different time zones, in which case you have effectively one sheriff on duty who has to do everything (bad due to point above), or they're not, in which case a chunk of the day is not covered at all. * New sheriffs could really use a mentor sheriff with them, which is pretty difficult to schedule. I think these are good reasons, so I propose we make this change. Comments? PK -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
Re: [chromium-dev] More sheriffs?
On Fri, Nov 13, 2009 at 1:33 PM, Stuart Morgan stuartmor...@google.comwrote: On Fri, Nov 13, 2009 at 1:25 PM, Peter Kasting pkast...@google.com wrote: Sheriffs are in theory supposed to watch all the perf bots too. Do you? I don't. I doubt very many people do. That's probably mostly a function of the fact that there's essentially no mention of monitoring perf (the fact that they should, how to do it, how to handle regressions, etc.) on the page about what sheriffs should do, not a manpower issue. Given that our project lead didn't even know there _was_ such a page, I'm not convinced. I don't think most sheriffs exhaustively read and understand that page, and the tasks and best practices as sheriff change rapidly (I hadn't ever heard of drover last time I sheriffed), which is part of the motivation for speeding up the cycle. PK -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
Re: [chromium-dev] More sheriffs?
Having just come off sheriffing four days in the past two weeks ... On Fri, Nov 13, 2009 at 12:31 PM, Peter Kasting pkast...@google.com wrote: At lunch today, a few of us discussed the idea of moving from two sheriffs to four. There are several reasons we contemplated such a change: * The team is large enough that on the current schedule, you go months between sheriffing, which is so long that you forget things like what tools help you do what. This is perhaps true, but I think it's more an issue that people don't run more of the tests on their own machines (or, alternatively, are asked to sheriff for areas of the system they never touch). * Sheriffing is a heavy burden, and getting moreso with more team members. * Either the two sheriffs are in different time zones, in which case you have effectively one sheriff on duty who has to do everything (bad due to point above), or they're not, in which case a chunk of the day is not covered at all. I think two sheriffs in US/Pacific during US/Pacific work hours is plenty. I can't speak to how much an issue the lack of sheriffs are to people outside that window. * New sheriffs could really use a mentor sheriff with them, which is pretty difficult to schedule. Last week was actually my first time, and I didn't think it was a big deal, although I did ask a few people a few questions. I was pretty much full time on keeping the tree green and cleaning up flaky tests. Given that I'm otherwise full time on LTTF, this wasn't much of a change. I think it's unrealistic to expect to do anything real on a project while sheriffing, because you can't context-switch that fast to do a good job on either (at least, I can't). I also think the bots would've been green most of the time except that someone has clearly been ignoring the memory tests for a long time. If bots fails for a couple days straight, it's beyond a sheriff to try and fix it - I think someone needs to get assigned that problem specifically. So, I'd probably leave things mostly the way they are unless there's a desire to have better sheriffing outside of the MTV hours. I fully support always having two sheriffs during MTV hours. -- Dirk -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
Re: [chromium-dev] More sheriffs?
On Fri, Nov 13, 2009 at 2:56 PM, Dirk Pranke dpra...@chromium.org wrote: I think two sheriffs in US/Pacific during US/Pacific work hours is plenty. I was told at lunch that we already try to some degree to schedule PST with non-PST people (although obvioulsy there are far more of the former), which gives me the impression that there is a large percentage of time where we have one, rather than two, sheriffs. That is perhaps the most important thing I'm trying to rectify in this proposal. On Fri, Nov 13, 2009 at 2:58 PM, Nicolas Sylvain nsylv...@google.com wrote: As for http://dev.chromium.org/developers/tree-sheriffs, every sheriff receives it in the reminder email the day before they start their sheriff duty. I see calendar reminder mails and think of them as conveying a reminder of an event, so I'd never noticed that these mails also mention a web page I'm supposed to be reading. I know that is my own fault, but maybe there are others in the same boat. In any case, I still think Ben's suggestions would be useful. Overall I am surprised at how many people are skeptical of this proposal given how unilaterally positive the smaller lunchtime discussion was. I guess I perceive us as not having a very effective sheriff system right now--it's certainly been difficult for me--and am looking for ways to remedy that. It seems like those who aren't in favor of this generally wouldn't agree with that assessment, and thus perceive this as adding overhead and reducing effectiveness rather than combating a notable lack. If that is accurate, I'm not sure how to square the two worldviews. I guess I will leave this idea in the hands of the green tree task force to decide whether it would be helpful. PK -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
Re: [chromium-dev] More sheriffs?
On Fri, Nov 13, 2009 at 3:38 PM, Peter Kasting pkast...@google.com wrote: On Fri, Nov 13, 2009 at 2:56 PM, Dirk Pranke dpra...@chromium.org wrote: I think two sheriffs in US/Pacific during US/Pacific work hours is plenty. I was told at lunch that we already try to some degree to schedule PST with non-PST people (although obvioulsy there are far more of the former), which gives me the impression that there is a large percentage of time where we have one, rather than two, sheriffs. That is perhaps the most important thing I'm trying to rectify in this proposal. On Fri, Nov 13, 2009 at 2:58 PM, Nicolas Sylvain nsylv...@google.com wrote: As for http://dev.chromium.org/developers/tree-sheriffs, every sheriff receives it in the reminder email the day before they start their sheriff duty. I see calendar reminder mails and think of them as conveying a reminder of an event, so I'd never noticed that these mails also mention a web page I'm supposed to be reading. I know that is my own fault, but maybe there are others in the same boat. In any case, I still think Ben's suggestions would be useful. Overall I am surprised at how many people are skeptical of this proposal given how unilaterally positive the smaller lunchtime discussion was. I guess I perceive us as not having a very effective sheriff system right now--it's certainly been difficult for me--and am looking for ways to remedy that. It seems like those who aren't in favor of this generally wouldn't agree with that assessment, and thus perceive this as adding overhead and reducing effectiveness rather than combating a notable lack. If that is accurate, I'm not sure how to square the two worldviews. I guess I will leave this idea in the hands of the green tree task force to decide whether it would be helpful. It'd be interesting if others from lunch chimed in with why they think it's a good idea. Also, I think there was clear consensus in adding another sheriff so we always have 2 in the Americas (or maybe even PST). Do we know what the next steps are to implement this? -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
Re: [chromium-dev] More sheriffs?
Peter Kasting wrote: On Fri, Nov 13, 2009 at 12:44 PM, Stuart Morgan stuartmor...@google.com wrote: If we end up actually having four at a time that seems likely to be worse than two: either four people are doing nothing but sheriffing, which there is probably not enough work for, or all four people are more likely to think that someone else is probably watching and they can do something else. I didn’t see Stuart’s original message, so I don’t know if there was more context, but I agree with what he’s saying here. In my experience, sheriffing is a one-person job, except we want that one person to be able to take a break or have lunch or have someone to fall back on when there are compound problems. I think it’s actually pretty rare for there to be more than three things wrong at a time, and usually when there are that many wrong, they didn’t all go bad simultaneously. It’s a one-person job, but it’s more than a full-time job, so we schedule two. Recently, there have been a few cases where people on the schedule couldn’t sheriff and didn’t arrange for a replacement. Things have gotten really bad when this happened, and for that reason alone, I’d support going to three. I also agree that going three months between shifts means that you might lose touch with how to do it effectively. Maybe we’ve got enough people now that we don’t need to sheriff for two days at a time. Maybe we can move from two sheriffs for two days to three for one. I’m not terribly motivated by any of the time zone policies, because I haven't seen this as a significant source of problems. Mark -- Chromium Developers mailing list: chromium-dev@googlegroups.com View archives, change email options, or unsubscribe: http://groups.google.com/group/chromium-dev
