Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
I'm fairly certain the buzzing sound I can hear is a bee in my bonnet... so I suspect that I'm starting to sound like someone chasing a bee that only they can hear. I'm not sure if it's helpful to keep this discussion on this list - would there be a better forum somewhere else? On Fri, Aug 29, 2014 at 7:34 PM, Thierry Carrez thie...@openstack.org wrote: James Polley wrote: However, Thierry pointed to https://wiki.openstack.org/wiki/Governance/Foundation/Structure which still refers to Project Technical Leads and says explicitly that they lead individual projects, not programs. I actually have edit access to that page, so I could at least update that with a simple s/Project/Program/, if I was sure that was the right thing to do. Don't underestimate how stale wiki pages can become! Yes, fix it. I don't know if I've fixed it, but I've certainly replaced all users of the word Project with Program. Whether or not it now matches reality, I'm not sure. I alsp removed (what I assume is) a stale reference to the PPB and added a new heading for the TC. It looks correct to me, thanks! http://www.openstack.org/ has a link in the bottom nav that says Projects; it points to http://www.openstack.org/projects/ which redirects to http://www.openstack.org/software/ which has a list of things like Compute and Storage - which as far as I know are Programs, not Projects. I don't know how to update that link in the nav panel. That's because the same word (compute) is used for two different things: a program name (Compute) and an official OpenStack name for a project (OpenStack Compute a.k.a. Nova). Basically official OpenStack names reduce confusion for newcomers (What is Nova ?), but they confuse old-timers at some point (so the Compute program produces Nova a.k.a. OpenStack Compute ?). That's confusing to me. I had thought that part of the reason for the separation was to enable a level of indirection - if the Compute program team decide that a new project called (for example) SuperNova should be the main project, that just means that Openstack Compute is now a pointer to a different project, supported by the same program team. It sounds like that isn't the intent though? That's more of a side-effect than the intent, IMHO. The indirection we created is between teams and code repositories. I wasn't around when the original Programs/Projects discussion was happening - which, I suspect, has a lot to do with why I'm confused today - it seems as though people who were around at the time understand the difference, but people who have joined since then are relying on multiple conflicting verbal definitions. I believe, though, that http://lists.openstack.org/pipermail/openstack-dev/2013-June/010821.html was one of the earliest starting points of the discussion. That page points at https://wiki.openstack.org/wiki/Projects, which today contains a list of Programs. That page does have a definition of what a Program is, but doesn't explain what a Project is or how they relate to Programs. This page seems to be locked down, so I can't edit it. https://wiki.openstack.org/wiki/Projects was renamed to https://wiki.openstack.org/wiki/Programs with the wiki helpfully leaving a redirect behind. So the content you are seeing here is the Programs wiki page, which is why it doesn't define projects. We don't really use the word project that much anymore, we prefer to talk about code repositories. Programs are teams working on a set of code repositories. Some of those code repositories may appear in the integrated release. This explanation of the difference between projects and programs sounds like it would be useful to add to /Programs - but I can't edit that page. This page reflects the official list of programs, which is why it's protected. it's supposed to be replaced by an automatic publication from http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml which is the ultimate source of truth on that topic. I was going to ask about the reference to The process new projects can follow to become an Integrated project - is that intended to refer to a project or a program? But then I read https://review.openstack.org/#/c/116727/ and and http://git.openstack.org/cgit/openstack/governance/tree/reference/incubation-integration-requirements.rst, seem to make it clear that it's entirely possible that the Kitty program might have a mix of Integrated and non-Integrated projects. Is it safe to assume that the Governance repo is canonical and up-to-date, and rework the wiki pages based on the information in the Governance repo? [1]
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
James Polley wrote: However, Thierry pointed to https://wiki.openstack.org/wiki/Governance/Foundation/Structure which still refers to Project Technical Leads and says explicitly that they lead individual projects, not programs. I actually have edit access to that page, so I could at least update that with a simple s/Project/Program/, if I was sure that was the right thing to do. Don't underestimate how stale wiki pages can become! Yes, fix it. I don't know if I've fixed it, but I've certainly replaced all users of the word Project with Program. Whether or not it now matches reality, I'm not sure. I alsp removed (what I assume is) a stale reference to the PPB and added a new heading for the TC. It looks correct to me, thanks! http://www.openstack.org/ has a link in the bottom nav that says Projects; it points to http://www.openstack.org/projects/ which redirects to http://www.openstack.org/software/ which has a list of things like Compute and Storage - which as far as I know are Programs, not Projects. I don't know how to update that link in the nav panel. That's because the same word (compute) is used for two different things: a program name (Compute) and an official OpenStack name for a project (OpenStack Compute a.k.a. Nova). Basically official OpenStack names reduce confusion for newcomers (What is Nova ?), but they confuse old-timers at some point (so the Compute program produces Nova a.k.a. OpenStack Compute ?). That's confusing to me. I had thought that part of the reason for the separation was to enable a level of indirection - if the Compute program team decide that a new project called (for example) SuperNova should be the main project, that just means that Openstack Compute is now a pointer to a different project, supported by the same program team. It sounds like that isn't the intent though? That's more of a side-effect than the intent, IMHO. The indirection we created is between teams and code repositories. I wasn't around when the original Programs/Projects discussion was happening - which, I suspect, has a lot to do with why I'm confused today - it seems as though people who were around at the time understand the difference, but people who have joined since then are relying on multiple conflicting verbal definitions. I believe, though, that http://lists.openstack.org/pipermail/openstack-dev/2013-June/010821.html was one of the earliest starting points of the discussion. That page points at https://wiki.openstack.org/wiki/Projects, which today contains a list of Programs. That page does have a definition of what a Program is, but doesn't explain what a Project is or how they relate to Programs. This page seems to be locked down, so I can't edit it. https://wiki.openstack.org/wiki/Projects was renamed to https://wiki.openstack.org/wiki/Programs with the wiki helpfully leaving a redirect behind. So the content you are seeing here is the Programs wiki page, which is why it doesn't define projects. We don't really use the word project that much anymore, we prefer to talk about code repositories. Programs are teams working on a set of code repositories. Some of those code repositories may appear in the integrated release. This explanation of the difference between projects and programs sounds like it would be useful to add to /Programs - but I can't edit that page. This page reflects the official list of programs, which is why it's protected. it's supposed to be replaced by an automatic publication from http://git.openstack.org/cgit/openstack/governance/tree/reference/programs.yaml which is the ultimate source of truth on that topic. [1] https://wiki.openstack.org/wiki/ProjectTypes I *can* edit that page; I'd like to bring it up-to-date. It seems like a good basis for explaining the difference between Programs and Projects and the historical reasons for the split. I'll aim to take a stab at this next week. Please feel free to do so, however that page is really an artifact of the old way we were structured, and is therefore useful as an historic leftover :) It's not linked from anywhere those days. Maybe you should create a new page, like https://wiki.openstack.org/wiki/Projects_vs_Programs ? What you want to talk about is not really about Project Types anyway. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
James Polley wrote: Point of clarification: I've heard PTL=Project Technical Lead and PTL=Program Technical Lead. Which is it? It is kind of important as OpenStack grows, because the first is responsible for *a* project, and the second is responsible for all projects within a program. Now Program, formerly Project. I think this is worthy of more exploration. Our docs seem to be very inconsistent about what a PTL is - and more broadly, what the difference is between a Project and a Program. Just a few examples: https://wiki.openstack.org/wiki/PTLguide says Program Technical Lead. https://wiki.openstack.org/wiki/PTL_Elections_March/April_2014 simply says PTL - but does say that each PTL is elected by/for a Program. However, Thierry pointed to https://wiki.openstack.org/wiki/Governance/Foundation/Structure which still refers to Project Technical Leads and says explicitly that they lead individual projects, not programs. I actually have edit access to that page, so I could at least update that with a simple s/Project/Program/, if I was sure that was the right thing to do. Don't underestimate how stale wiki pages can become! Yes, fix it. http://www.openstack.org/ has a link in the bottom nav that says Projects; it points to http://www.openstack.org/projects/ which redirects to http://www.openstack.org/software/ which has a list of things like Compute and Storage - which as far as I know are Programs, not Projects. I don't know how to update that link in the nav panel. That's because the same word (compute) is used for two different things: a program name (Compute) and an official OpenStack name for a project (OpenStack Compute a.k.a. Nova). Basically official OpenStack names reduce confusion for newcomers (What is Nova ?), but they confuse old-timers at some point (so the Compute program produces Nova a.k.a. OpenStack Compute ?). I wasn't around when the original Programs/Projects discussion was happening - which, I suspect, has a lot to do with why I'm confused today - it seems as though people who were around at the time understand the difference, but people who have joined since then are relying on multiple conflicting verbal definitions. I believe, though, that http://lists.openstack.org/pipermail/openstack-dev/2013-June/010821.html was one of the earliest starting points of the discussion. That page points at https://wiki.openstack.org/wiki/Projects, which today contains a list of Programs. That page does have a definition of what a Program is, but doesn't explain what a Project is or how they relate to Programs. This page seems to be locked down, so I can't edit it. https://wiki.openstack.org/wiki/Projects was renamed to https://wiki.openstack.org/wiki/Programs with the wiki helpfully leaving a redirect behind. So the content you are seeing here is the Programs wiki page, which is why it doesn't define projects. We don't really use the word project that much anymore, we prefer to talk about code repositories. Programs are teams working on a set of code repositories. Some of those code repositories may appear in the integrated release. That page does mention projects, once. The context makes it read, to me, as though a program can follow one process to become part of OpenStack and then another process to become an Integrated project and part of the OpenStack coordinated release - when my understanding of reality is that the second process applies to Projects, not Programs. I've tried to find any other page that talks about what a Project is and how they relate to Programs, but I haven't been able to find anything. Perhaps there's some definition locked up in a mailing list thread or some TC minutes, but I haven't been able to find it. During the previous megathread, I got the feeling that at least some of the differing viewpoints we saw were possibly down to some people thinking of a PTL as responsible for just one project, while others think of a PTL as being responsible for any projects that might fit within a Program's scope. As we approach the next PTL elections, I think it would be helpful for us to recap the discussions that led to the Program/Project split and make sure our docs are consistent, so that people who weren't following the discussion this time last year can still be clear what they're voting for. Programs are just acknowledging that code repositories should be organized in the way that makes the most sense technically. They should not be artificially organized to match our governance structure. Before programs existed, it was difficult for teams to organize their code the way they wanted, because there was only one code repository (The Project), so everything had to be in it. Then we added an exception for the Python client projects, so the Nova team could work on the Nova project *and* the Python client for it. But then it made sense to organize the code differently,
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On Thu, Aug 28, 2014 at 10:40 PM, Thierry Carrez thie...@openstack.org wrote: James Polley wrote: Point of clarification: I've heard PTL=Project Technical Lead and PTL=Program Technical Lead. Which is it? It is kind of important as OpenStack grows, because the first is responsible for *a* project, and the second is responsible for all projects within a program. Now Program, formerly Project. I think this is worthy of more exploration. Our docs seem to be very inconsistent about what a PTL is - and more broadly, what the difference is between a Project and a Program. Just a few examples: https://wiki.openstack.org/wiki/PTLguide says Program Technical Lead. https://wiki.openstack.org/wiki/PTL_Elections_March/April_2014 simply says PTL - but does say that each PTL is elected by/for a Program. However, Thierry pointed to https://wiki.openstack.org/wiki/Governance/Foundation/Structure which still refers to Project Technical Leads and says explicitly that they lead individual projects, not programs. I actually have edit access to that page, so I could at least update that with a simple s/Project/Program/, if I was sure that was the right thing to do. Don't underestimate how stale wiki pages can become! Yes, fix it. I don't know if I've fixed it, but I've certainly replaced all users of the word Project with Program. Whether or not it now matches reality, I'm not sure. I alsp removed (what I assume is) a stale reference to the PPB and added a new heading for the TC. http://www.openstack.org/ has a link in the bottom nav that says Projects; it points to http://www.openstack.org/projects/ which redirects to http://www.openstack.org/software/ which has a list of things like Compute and Storage - which as far as I know are Programs, not Projects. I don't know how to update that link in the nav panel. That's because the same word (compute) is used for two different things: a program name (Compute) and an official OpenStack name for a project (OpenStack Compute a.k.a. Nova). Basically official OpenStack names reduce confusion for newcomers (What is Nova ?), but they confuse old-timers at some point (so the Compute program produces Nova a.k.a. OpenStack Compute ?). That's confusing to me. I had thought that part of the reason for the separation was to enable a level of indirection - if the Compute program team decide that a new project called (for example) SuperNova should be the main project, that just means that Openstack Compute is now a pointer to a different project, supported by the same program team. It sounds like that isn't the intent though? I wasn't around when the original Programs/Projects discussion was happening - which, I suspect, has a lot to do with why I'm confused today - it seems as though people who were around at the time understand the difference, but people who have joined since then are relying on multiple conflicting verbal definitions. I believe, though, that http://lists.openstack.org/pipermail/openstack-dev/2013-June/010821.html was one of the earliest starting points of the discussion. That page points at https://wiki.openstack.org/wiki/Projects, which today contains a list of Programs. That page does have a definition of what a Program is, but doesn't explain what a Project is or how they relate to Programs. This page seems to be locked down, so I can't edit it. https://wiki.openstack.org/wiki/Projects was renamed to https://wiki.openstack.org/wiki/Programs with the wiki helpfully leaving a redirect behind. So the content you are seeing here is the Programs wiki page, which is why it doesn't define projects. We don't really use the word project that much anymore, we prefer to talk about code repositories. Programs are teams working on a set of code repositories. Some of those code repositories may appear in the integrated release. This explanation of the difference between projects and programs sounds like it would be useful to add to /Programs - but I can't edit that page. That page does mention projects, once. The context makes it read, to me, as though a program can follow one process to become part of OpenStack and then another process to become an Integrated project and part of the OpenStack coordinated release - when my understanding of reality is that the second process applies to Projects, not Programs. I've tried to find any other page that talks about what a Project is and how they relate to Programs, but I haven't been able to find anything. Perhaps there's some definition locked up in a mailing list thread or some TC minutes, but I haven't been able to find it. During the previous megathread, I got the feeling that at least some of the differing viewpoints we saw were possibly down to some people thinking of a PTL as responsible for just one project, while others think of a PTL as being
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On Sat, Aug 23, 2014 at 11:02 AM, Anne Gentle a...@openstack.org wrote: On Fri, Aug 22, 2014 at 6:17 PM, Rochelle.RochelleGrober rochelle.gro...@huawei.com wrote: /flame-on Ok, this is funny to some of us in the community. The general populace of this community is so against the idea of management that they will use the term for a despotic dictator as a position name rather than manager. Sorry, but this needed to be said. /flame-off Specific comments in line: Thierry Carrez wrote: Hi everyone, We all know being a project PTL is an extremely busy job. That's because in our structure the PTL is responsible for almost everything in a project: - Release management contact - Work prioritization - Keeping bugs under control - Communicate about work being planned or done - Make sure the gate is not broken - Team logistics (run meetings, organize sprints) - ... Point of clarification: I've heard PTL=Project Technical Lead and PTL=Program Technical Lead. Which is it? It is kind of important as OpenStack grows, because the first is responsible for *a* project, and the second is responsible for all projects within a program. Now Program, formerly Project. I think this is worthy of more exploration. Our docs seem to be very inconsistent about what a PTL is - and more broadly, what the difference is between a Project and a Program. Just a few examples: https://wiki.openstack.org/wiki/PTLguide says Program Technical Lead. https://wiki.openstack.org/wiki/PTL_Elections_March/April_2014 simply says PTL - but does say that each PTL is elected by/for a Program. However, Thierry pointed to https://wiki.openstack.org/wiki/Governance/Foundation/Structure which still refers to Project Technical Leads and says explicitly that they lead individual projects, not programs. I actually have edit access to that page, so I could at least update that with a simple s/Project/Program/, if I was sure that was the right thing to do. http://www.openstack.org/ has a link in the bottom nav that says Projects; it points to http://www.openstack.org/projects/ which redirects to http://www.openstack.org/software/ which has a list of things like Compute and Storage - which as far as I know are Programs, not Projects. I don't know how to update that link in the nav panel. I wasn't around when the original Programs/Projects discussion was happening - which, I suspect, has a lot to do with why I'm confused today - it seems as though people who were around at the time understand the difference, but people who have joined since then are relying on multiple conflicting verbal definitions. I believe, though, that http://lists.openstack.org/pipermail/openstack-dev/2013-June/010821.html was one of the earliest starting points of the discussion. That page points at https://wiki.openstack.org/wiki/Projects, which today contains a list of Programs. That page does have a definition of what a Program is, but doesn't explain what a Project is or how they relate to Programs. This page seems to be locked down, so I can't edit it. That page does mention projects, once. The context makes it read, to me, as though a program can follow one process to become part of OpenStack and then another process to become an Integrated project and part of the OpenStack coordinated release - when my understanding of reality is that the second process applies to Projects, not Programs. I've tried to find any other page that talks about what a Project is and how they relate to Programs, but I haven't been able to find anything. Perhaps there's some definition locked up in a mailing list thread or some TC minutes, but I haven't been able to find it. During the previous megathread, I got the feeling that at least some of the differing viewpoints we saw were possibly down to some people thinking of a PTL as responsible for just one project, while others think of a PTL as being responsible for any projects that might fit within a Program's scope. As we approach the next PTL elections, I think it would be helpful for us to recap the discussions that led to the Program/Project split and make sure our docs are consistent, so that people who weren't following the discussion this time last year can still be clear what they're voting for. I'd also like to set out as an example of a Program that is growing to encompass multiple projects, the Neutron Program. Look at how it is expanding: Multiple sub-teams for: LBAAS, DNAAS, GBP, etc. This model could be extended such that: - the subteam is responsible for code reviews, including the first +2 for design, architecture and code of the sub-project, always also keeping an eye out that the sub-project code continues to both integrate well with the program, and that the program continues to provide the needed code bits, architecture modifications and improvements, etc. to support the sub-project. - the final +2/A would be from the Program reviewers to ensure that
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
OK, now that we have evacuated the terminology issue (we'll use liaison or janitor or secretary, not czar), and side-stepped the offtopic development (this is not about suppressing PTLs, just a framework to let them delegate along predetermined lines if they want to)... which of those unnamed roles do we need ? In the thread were mentioned: - Bugs janitor (keep reported bugs under control) - Oslo liaison (already in place) - Security mule (VMT first point of contact) - Release secretary (communication with integrated release management) - Infrastructure contact (for gate and other infra issues) - Docs lieutenant (docs point of contact) Anita mentioned the 3rd party space person, but I wonder if it would not be specific to some projects. Would it actually be separate from the Infra contact role ? Do we need someone to cover the QA space ? Anything else missing ? At first glance I don't think we need a role for logistics (chairing meetings and organizing meetups), design summit planning, roadmapping, user point of contact, or spokesperson -- as I expect the PTL to retain those roles anyway... -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On 08/26/2014 11:13 AM, Thierry Carrez wrote: OK, now that we have evacuated the terminology issue (we'll use liaison or janitor or secretary, not czar), and side-stepped the offtopic development (this is not about suppressing PTLs, just a framework to let them delegate along predetermined lines if they want to)... which of those unnamed roles do we need ? In the thread were mentioned: - Bugs janitor (keep reported bugs under control) - Oslo liaison (already in place) - Security mule (VMT first point of contact) - Release secretary (communication with integrated release management) - Infrastructure contact (for gate and other infra issues) - Docs lieutenant (docs point of contact) Sounds good to me. Anita mentioned the 3rd party space person, but I wonder if it would not be specific to some projects. Would it actually be separate from the Infra contact role ? Do we need someone to cover the QA space ? Anything else missing ? I think we do. We have someone assigned to QA in Zaqar and that has worked pretty well for us. Malini has been taking care of tempest, devstack patches and syncing with OpenStack's QA team. Her efforts there have been a key on getting Zaqar in the gate and making sure we're up-to-date with the latest changes happening in our QA. At first glance I don't think we need a role for logistics (chairing meetings and organizing meetups), design summit planning, roadmapping, user point of contact, or spokesperson -- as I expect the PTL to retain those roles anyway... +1 for letting the PTL take care of this. Flavio -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On Aug 26, 2014, at 5:13 AM, Thierry Carrez thie...@openstack.org wrote: OK, now that we have evacuated the terminology issue (we'll use liaison or janitor or secretary, not czar), and side-stepped the offtopic development (this is not about suppressing PTLs, just a framework to let them delegate along predetermined lines if they want to)... which of those unnamed roles do we need ? In the thread were mentioned: - Bugs janitor (keep reported bugs under control) - Oslo liaison (already in place) - Security mule (VMT first point of contact) - Release secretary (communication with integrated release management) - Infrastructure contact (for gate and other infra issues) - Docs lieutenant (docs point of contact) Anita mentioned the 3rd party space person, but I wonder if it would not be specific to some projects. Would it actually be separate from the Infra contact role ? Do we need someone to cover the QA space ? Anything else missing ? It seems the QA team is also feeling pressure due to the growing community, so it seems wise to ensure every team has someone designated to help with coordinating work on QA projects. Doug At first glance I don't think we need a role for logistics (chairing meetings and organizing meetups), design summit planning, roadmapping, user point of contact, or spokesperson -- as I expect the PTL to retain those roles anyway... -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On Fri, Aug 22, 2014 at 8:19 PM, John Dickinson m...@not.mn wrote: I think Anne makes some excellent points about the pattern being proposed being unlikely to be commonly implemented across all the programs (or, at best, very difficult). Let's not try to formalize another best practice that works many times and force it to work every time. Here's an alternate proposal: Let's let PTLs be PTLs and effectively coordinate and manage the activity in their respective projects. And let's get the PTLs together for one or two days every cycle to discuss project issues. Just PTLs, and let's focus on the project management stuff and some cross-project issues. Getting the PTLs together would allow them to discuss cross-project issues, share frustrations and solutions about what does and doesn't work. Basically, think of it as a mid-cycle meetup, but for PTLs. (Perhaps we could even ask the Foundation to sponsor it.) +100 I think John nails the key point here: PTLs are likely already doing a lot of what Thierry originally proposed here. I know that I work with many people in Neutron to help offload things like bug triaging, gate failure analysis, etc. Without that, Neutron wouldn't scale. In effect what's proposed here is something we're already doing a lot of. I don't think forcing this on each project is a good way of doing this, because each project has different challenges and needs. Thanks, Kyle --John On Aug 22, 2014, at 6:02 PM, Anne Gentle a...@openstack.org wrote: On Fri, Aug 22, 2014 at 6:17 PM, Rochelle.RochelleGrober rochelle.gro...@huawei.com wrote: /flame-on Ok, this is funny to some of us in the community. The general populace of this community is so against the idea of management that they will use the term for a despotic dictator as a position name rather than manager. Sorry, but this needed to be said. /flame-off Specific comments in line: Thierry Carrez wrote: Hi everyone, We all know being a project PTL is an extremely busy job. That's because in our structure the PTL is responsible for almost everything in a project: - Release management contact - Work prioritization - Keeping bugs under control - Communicate about work being planned or done - Make sure the gate is not broken - Team logistics (run meetings, organize sprints) - ... Point of clarification: I've heard PTL=Project Technical Lead and PTL=Program Technical Lead. Which is it? It is kind of important as OpenStack grows, because the first is responsible for *a* project, and the second is responsible for all projects within a program. Now Program, formerly Project. I'd also like to set out as an example of a Program that is growing to encompass multiple projects, the Neutron Program. Look at how it is expanding: Multiple sub-teams for: LBAAS, DNAAS, GBP, etc. This model could be extended such that: - the subteam is responsible for code reviews, including the first +2 for design, architecture and code of the sub-project, always also keeping an eye out that the sub-project code continues to both integrate well with the program, and that the program continues to provide the needed code bits, architecture modifications and improvements, etc. to support the sub-project. - the final +2/A would be from the Program reviewers to ensure that all integrate nicely together into a single, cohesive program. - This would allow sub-projects to have core reviewers, along with the program and be a good separation of duties. It would also help to increase the number of reviews moving to merged code. - Taken to a logical stepping stone, you would have project technical leads for each project, and they would make up a program council, with the program technical lead being the chair of the council. This is a way to offload a good chunk of PTL tactical responsibilities and help them focus more on the strategic. They end up being completely drowned in those day-to-day operational duties, miss the big picture, can't help in development that much anymore, get burnt out. Since you're either the PTL or not the PTL, you're very alone and succession planning is not working that great either. There have been a number of experiments to solve that problem. John Garbutt has done an incredible job at helping successive Nova PTLs handling the release management aspect. Tracy Jones took over Nova bug management. Doug Hellmann successfully introduced the concept of Oslo liaisons to get clear point of contacts for Oslo library adoption in projects. It may be time to generalize that solution. The issue is one of responsibility: the PTL is ultimately responsible for everything in a project. If we can more formally delegate that responsibility, we can avoid getting up to the PTL for everything, we can rely on a team of people rather than just one person. Enter the Czar system: each project should have a
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On Mon, Aug 25, 2014 at 4:45 AM, Alan Kavanagh alan.kavan...@ericsson.com wrote: That's a fair point Jay. The Czar does sound like a reasonable approach and what would be useful and helpful would be to appoint additional PTL's and not have the burden of everything falling on one individual which becomes over loading after a period of time. In this case, imho it would be useful to have 2 or more PTL's assigned per project to adjust the workload and have different views and assess the sticky points with different views. /Alan I disagree with this assessment. Having multiple PTLs defeated the purpose of a PTL. Also, the PTL should be about building consensus, not using a hammer as was noted in other parts of this thread. As developers in Open Source, you have to be able to build consensus before some ideas and concepts can move forward. The PTL, in my opinion, is about helping to establish that consensus and being able to say no when that consensus isn't built. I don't think having multiple PTLs would help here. Thanks, Kyle -Original Message- From: Jay Pipes [mailto:jaypi...@gmail.com] Sent: August-25-14 1:58 AM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs On 08/23/2014 06:35 PM, Clint Byrum wrote: I agree as well. PTL is a servant of the community, as any good leader is. If the PTL feels they have to drop the hammer, or if an impass is reached where they are asked to, it is because they have failed to get everyone communicating effectively, not because that's their job. The problem isn't really that teams are not communicating effectively, nor is the problem to do with some deficit of a PTL in either putting the hammer down or failing to figure out common ground. The issue in my opinion and my experience is that there are multiple valid ways of doing something (say, deployment or metering or making toast) and the TC and our governing structure has decided to pick winners in spaces instead of having a big tent and welcoming different solutions and projects into the OpenStack fold. We pick winners and by doing so, we are exclusionary, and this exclusivity does not benefit our user community, but rather just gives it fewer options. IMHO, the TC should become an advisory team that recommends to interested project teams ways in which they can design and architect their projects to integrate well with other projects in the OpenStack community, and design their projects for the scale, stability and requirements (such as multi-tenancy) that an open cloud software ecosystem demands. Just my two cents, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On Aug 26, 2014, at 10:10 AM, Kyle Mestery mest...@mestery.com wrote: On Fri, Aug 22, 2014 at 8:19 PM, John Dickinson m...@not.mn wrote: I think Anne makes some excellent points about the pattern being proposed being unlikely to be commonly implemented across all the programs (or, at best, very difficult). Let's not try to formalize another best practice that works many times and force it to work every time. Here's an alternate proposal: Let's let PTLs be PTLs and effectively coordinate and manage the activity in their respective projects. And let's get the PTLs together for one or two days every cycle to discuss project issues. Just PTLs, and let's focus on the project management stuff and some cross-project issues. Getting the PTLs together would allow them to discuss cross-project issues, share frustrations and solutions about what does and doesn't work. Basically, think of it as a mid-cycle meetup, but for PTLs. (Perhaps we could even ask the Foundation to sponsor it.) +100 I think John nails the key point here: PTLs are likely already doing a lot of what Thierry originally proposed here. I know that I work with many people in Neutron to help offload things like bug triaging, gate failure analysis, etc. Without that, Neutron wouldn't scale. In effect what's proposed here is something we're already doing a lot of. I don't think forcing this on each project is a good way of doing this, because each project has different challenges and needs. This proposal isn’t about how teams are organized internally; it’s about how they interface with the other OpenStack teams. The cross-project teams need more direct coordination and participation than we are getting from some projects, and so we want the projects and PTLs to recognize that the areas Thierry has listed are responsibilities that need to be met. Someone needs to do these basic coordination tasks in order for the overall project to be successful. It’s up to each team to decide who fills a given role, but by identifying these points of contact the other teams don’t have to know that you make those decisions by asking for volunteers, holding an election, or drawing straws. Doug Thanks, Kyle --John On Aug 22, 2014, at 6:02 PM, Anne Gentle a...@openstack.org wrote: On Fri, Aug 22, 2014 at 6:17 PM, Rochelle.RochelleGrober rochelle.gro...@huawei.com wrote: /flame-on Ok, this is funny to some of us in the community. The general populace of this community is so against the idea of management that they will use the term for a despotic dictator as a position name rather than manager. Sorry, but this needed to be said. /flame-off Specific comments in line: Thierry Carrez wrote: Hi everyone, We all know being a project PTL is an extremely busy job. That's because in our structure the PTL is responsible for almost everything in a project: - Release management contact - Work prioritization - Keeping bugs under control - Communicate about work being planned or done - Make sure the gate is not broken - Team logistics (run meetings, organize sprints) - ... Point of clarification: I've heard PTL=Project Technical Lead and PTL=Program Technical Lead. Which is it? It is kind of important as OpenStack grows, because the first is responsible for *a* project, and the second is responsible for all projects within a program. Now Program, formerly Project. I'd also like to set out as an example of a Program that is growing to encompass multiple projects, the Neutron Program. Look at how it is expanding: Multiple sub-teams for: LBAAS, DNAAS, GBP, etc. This model could be extended such that: - the subteam is responsible for code reviews, including the first +2 for design, architecture and code of the sub-project, always also keeping an eye out that the sub-project code continues to both integrate well with the program, and that the program continues to provide the needed code bits, architecture modifications and improvements, etc. to support the sub-project. - the final +2/A would be from the Program reviewers to ensure that all integrate nicely together into a single, cohesive program. - This would allow sub-projects to have core reviewers, along with the program and be a good separation of duties. It would also help to increase the number of reviews moving to merged code. - Taken to a logical stepping stone, you would have project technical leads for each project, and they would make up a program council, with the program technical lead being the chair of the council. This is a way to offload a good chunk of PTL tactical responsibilities and help them focus more on the strategic. They end up being completely drowned in those day-to-day operational duties, miss the big picture, can't help in development that much anymore, get burnt out. Since you're either the PTL or not the PTL, you're very
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On 08/26/2014 05:13 AM, Thierry Carrez wrote: OK, now that we have evacuated the terminology issue (we'll use liaison or janitor or secretary, not czar), and side-stepped the offtopic development (this is not about suppressing PTLs, just a framework to let them delegate along predetermined lines if they want to)... which of those unnamed roles do we need ? In the thread were mentioned: - Bugs janitor (keep reported bugs under control) - Oslo liaison (already in place) - Security mule (VMT first point of contact) - Release secretary (communication with integrated release management) - Infrastructure contact (for gate and other infra issues) - Docs lieutenant (docs point of contact) Anita mentioned the 3rd party space person, but I wonder if it would not be specific to some projects. Would it actually be separate from the Infra contact role ? I think your assessment of specific to some projects is accurate. For instance, for Neutron Kyle wants to continue to be responsible for this role. Fine by me as long as I have a name. For Cinder, Jay Bryant attends the third party meeting and gives updates, Ramy has put in the most work on CIs and is a great resource for answering questions and Duncan holds the hammer on what stays in master but the third party meetings are too late for him to attend most times, so he co-ordinates with Jay. Again as long as I know who to talk to. For Nova? Well right now Nova is kind of hit and miss so I would appreciate a name there. Other projects? If you have vendors or ci accounts testing your projects, please get in touch. Thanks, Anita. Do we need someone to cover the QA space ? Anything else missing ? At first glance I don't think we need a role for logistics (chairing meetings and organizing meetups), design summit planning, roadmapping, user point of contact, or spokesperson -- as I expect the PTL to retain those roles anyway... ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On 08/26/2014 10:04 AM, Doug Hellmann wrote: On Aug 26, 2014, at 5:13 AM, Thierry Carrez thie...@openstack.org wrote: OK, now that we have evacuated the terminology issue (we'll use liaison or janitor or secretary, not czar), and side-stepped the offtopic development (this is not about suppressing PTLs, just a framework to let them delegate along predetermined lines if they want to)... which of those unnamed roles do we need ? In the thread were mentioned: - Bugs janitor (keep reported bugs under control) - Oslo liaison (already in place) - Security mule (VMT first point of contact) - Release secretary (communication with integrated release management) - Infrastructure contact (for gate and other infra issues) - Docs lieutenant (docs point of contact) Anita mentioned the 3rd party space person, but I wonder if it would not be specific to some projects. Would it actually be separate from the Infra contact role ? Do we need someone to cover the QA space ? Anything else missing ? It seems the QA team is also feeling pressure due to the growing community, so it seems wise to ensure every team has someone designated to help with coordinating work on QA projects. Very much so, and having such a someone would help. But I also think that the moving of functional tests to be housed in-project will help even more. -David Doug At first glance I don't think we need a role for logistics (chairing meetings and organizing meetups), design summit planning, roadmapping, user point of contact, or spokesperson -- as I expect the PTL to retain those roles anyway... -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On Tue, Aug 26, 2014 at 10:04:41AM -0400, Doug Hellmann wrote: On Aug 26, 2014, at 5:13 AM, Thierry Carrez thie...@openstack.org wrote: OK, now that we have evacuated the terminology issue (we'll use liaison or janitor or secretary, not czar), and side-stepped the offtopic development (this is not about suppressing PTLs, just a framework to let them delegate along predetermined lines if they want to)... which of those unnamed roles do we need ? In the thread were mentioned: - Bugs janitor (keep reported bugs under control) - Oslo liaison (already in place) - Security mule (VMT first point of contact) - Release secretary (communication with integrated release management) - Infrastructure contact (for gate and other infra issues) - Docs lieutenant (docs point of contact) Anita mentioned the 3rd party space person, but I wonder if it would not be specific to some projects. Would it actually be separate from the Infra contact role ? Do we need someone to cover the QA space ? Anything else missing ? It seems the QA team is also feeling pressure due to the growing community, so it seems wise to ensure every team has someone designated to help with coordinating work on QA projects. Yes I agree, I was actually planning to start a liaison (I guess I'll have to come up with a different name...) system similar to oslo at some point soon. We discussed it during the QA meeting last week. [1] My plan was actually going to be starting a thread on that this week before Thierry beat me to it. -Matt Treinish [1] http://eavesdrop.openstack.org/meetings/qa/2014/qa.2014-08-21-22.01.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On Aug 26, 2014, at 11:28 AM, Matthew Treinish mtrein...@kortar.org wrote: On Tue, Aug 26, 2014 at 10:04:41AM -0400, Doug Hellmann wrote: On Aug 26, 2014, at 5:13 AM, Thierry Carrez thie...@openstack.org wrote: OK, now that we have evacuated the terminology issue (we'll use liaison or janitor or secretary, not czar), and side-stepped the offtopic development (this is not about suppressing PTLs, just a framework to let them delegate along predetermined lines if they want to)... which of those unnamed roles do we need ? In the thread were mentioned: - Bugs janitor (keep reported bugs under control) - Oslo liaison (already in place) - Security mule (VMT first point of contact) - Release secretary (communication with integrated release management) - Infrastructure contact (for gate and other infra issues) - Docs lieutenant (docs point of contact) Anita mentioned the 3rd party space person, but I wonder if it would not be specific to some projects. Would it actually be separate from the Infra contact role ? Do we need someone to cover the QA space ? Anything else missing ? It seems the QA team is also feeling pressure due to the growing community, so it seems wise to ensure every team has someone designated to help with coordinating work on QA projects. Yes I agree, I was actually planning to start a liaison (I guess I'll have to come up with a different name...) system similar to oslo at some point soon. We discussed it during the QA meeting last week. [1] My plan was actually going to be starting a thread on that this week before Thierry beat me to it. In the spirit of collaboration, the Oslo team is willing to share the term “liaison” with the QA team if you would like to use it. ;-) Doug -Matt Treinish [1] http://eavesdrop.openstack.org/meetings/qa/2014/qa.2014-08-21-22.01.html ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
That's a fair point Jay. The Czar does sound like a reasonable approach and what would be useful and helpful would be to appoint additional PTL's and not have the burden of everything falling on one individual which becomes over loading after a period of time. In this case, imho it would be useful to have 2 or more PTL's assigned per project to adjust the workload and have different views and assess the sticky points with different views. /Alan -Original Message- From: Jay Pipes [mailto:jaypi...@gmail.com] Sent: August-25-14 1:58 AM To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs On 08/23/2014 06:35 PM, Clint Byrum wrote: I agree as well. PTL is a servant of the community, as any good leader is. If the PTL feels they have to drop the hammer, or if an impass is reached where they are asked to, it is because they have failed to get everyone communicating effectively, not because that's their job. The problem isn't really that teams are not communicating effectively, nor is the problem to do with some deficit of a PTL in either putting the hammer down or failing to figure out common ground. The issue in my opinion and my experience is that there are multiple valid ways of doing something (say, deployment or metering or making toast) and the TC and our governing structure has decided to pick winners in spaces instead of having a big tent and welcoming different solutions and projects into the OpenStack fold. We pick winners and by doing so, we are exclusionary, and this exclusivity does not benefit our user community, but rather just gives it fewer options. IMHO, the TC should become an advisory team that recommends to interested project teams ways in which they can design and architect their projects to integrate well with other projects in the OpenStack community, and design their projects for the scale, stability and requirements (such as multi-tenancy) that an open cloud software ecosystem demands. Just my two cents, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
Anne Gentle wrote: Rochelle.RochelleGrober wrote: /flame-on Let's call spades, spades here. Czar is not only overkill, but the wrong metaphor. /flame-off I'm with Rocky on the anti-czar-as-a-word camp. We all like clever names to shed the corporate stigma but this word ain't it. Liaison or lead? Sure! I certainly didn't want to offend anyone with my choice of terms. Google lists two meanings for the word: 1. an emperor of Russia before 1917 (Tsar Nicholas II) 2. a person appointed by government to advise on and coordinate policy in a particular area (the former British drugs czar) I was referring to the latter meaning (like the US presidency has had cybersecurity czars in the past), but I'm fine with using liaison instead -- it's just slightly more boring. Also wanted to point to https://wiki.openstack.org/wiki/PTLguide as it's quite nice. I think PTLs tend to find help when they absolutely are ready to fall over, and I'm all for a plan that helps us not fall over. I've had people step up for bug triaging, gate work, tests, and oslo, sometimes one person did three or four at once. I certainly can't do it all for cross-project. Based on what I've seen, I doubt that we can add this much formality to this across 20+ programs. It's the bigger more integrated project vs. smaller more focused project difference that won't let us do a pattern here. We can certainly document the responsibilities and let programs know there are some best practices. In smaller programs, I would expect the PTL to keep filling most, if not all, of those roles. As long as the PTL is not overworked, the roles are well-known and everyone knows who to contact, it works. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
Zane Bitter wrote: On 22/08/14 12:45, Dolph Mathews wrote: I'm all for getting a final decision, but a 'final' decision that has been imposed from outside rather than internalised by the participants is... rarely final. The expectation of a PTL isn't to stomp around and make final decisions, it's to step in when necessary and help both sides find the best solution. To moderate. Oh sure, but that's not just the PTL's job. That's everyone's job. Don't you think? I did that before I was the PTL and will continue to do it after I'm no longer the PTL. And if anyone in the (especially) the core or wider Heat team sees an opportunity to step in and moderate a disagreement I certainly expect them to take it and not wait for me to step in. I'm not calling for no leadership here - I'm calling for leadership from _everyone_, not just from one person who holds a particular role. I guess the difference between you and me is that I don't see having a PTL as preventing that moderation and leadership from everyone. I really see it as a safety valve in case things ever go badly wrong. I prefer that safety valve to be built into the project leadership, rather than at the TC level. Could you explain how having a PTL is preventing that leadership from everyone ? Did it prevent it in Heat ? Did having the PTL safety valve hurt you ? I'm open to the alternative solution (which would be for programs which are not interested in having a PTL to just not have one). But then if things go badly wrong, you require the TC to step in with threats of removal of OpenStack and/or to force an election/vote in the middle of the crisis. I'm really failing to see how that would result, in those hypothetical crisis scenarios, in a better outcome. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
Tim Bell wrote: As part of the user feedback loop, we've found the PTL role extremely useful to channel feedback. The operator PTL discussions during the Atlanta summit helped to clarify a number of areas where the PTL can then take the points back to the design summit. It is not clear how czars would address the outward facing part of the PTL role which is clearly needed in view of the various discussions around program management and priorities. If we have lots of czars or major differences in how each project is structured, it is not clear how we channel user feedback to the project teams. Would there be a user czar on each project ? I agree that this external/user communication role is essential, and is likely to stay with the PTL, which is why I didn't have a User czar in my list. I have no problem with lots of czars to delegate activities across the projects but having a single accountable and elected PTL who can choose the level of delegation (and be assessed on that) seems to be a very good feature. Indeed, I see it more as a clear list of the duties that usually fall onto the PTL lap, but which could be delegated. If some programs want to keep it the way it is (with the PTL being responsible for all those duties), it's totally fine. It's a framework for clear delegation within the current system, not a whole new system :) -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On 08/25/2014 12:30 PM, Thierry Carrez wrote: Zane Bitter wrote: On 22/08/14 12:45, Dolph Mathews wrote: I'm all for getting a final decision, but a 'final' decision that has been imposed from outside rather than internalised by the participants is... rarely final. The expectation of a PTL isn't to stomp around and make final decisions, it's to step in when necessary and help both sides find the best solution. To moderate. Oh sure, but that's not just the PTL's job. That's everyone's job. Don't you think? I did that before I was the PTL and will continue to do it after I'm no longer the PTL. And if anyone in the (especially) the core or wider Heat team sees an opportunity to step in and moderate a disagreement I certainly expect them to take it and not wait for me to step in. I'm not calling for no leadership here - I'm calling for leadership from _everyone_, not just from one person who holds a particular role. I guess the difference between you and me is that I don't see having a PTL as preventing that moderation and leadership from everyone. I really see it as a safety valve in case things ever go badly wrong. I prefer that safety valve to be built into the project leadership, rather than at the TC level. Could you explain how having a PTL is preventing that leadership from everyone ? Did it prevent it in Heat ? Did having the PTL safety valve hurt you ? I'd like to see projects leadership as federated as possible. I think PTLs are actually responsible for spreading leadership throughout the team. From my point of view, and this is a very personal point of view, PTLs should worry for making the team grow as a team and as individuals. Spreading the responsibilities and making everyone part of the decision making process will help the team to mature and it'll also encourage members of that team to send their candidacy for future elections. This is why I think the Czar's (or whatever it'll be called) idea is important. There may be lots of volunteers or none. However, it's important to encourage people to participate and I believe this is also part of the PTLs responsibility. People could argue saying that we don't need a PTL to do that, everyone in the community should help with encouraging others and I would certainly agree with that too. Probably, the issue here is not about having leaders but about encouraging people to lead and volunteer as much as they can without burning out. Thing is, what's the easiest way to get there without slowing down development or hurting existing projects? For example: I've heard folks saying: The project is in this state because the PTL X, Y and Z This is wrong, the project is in that state because the *whole* team took it there or because no one cares. What I'm trying to say is that we need to change the way we talk/think about PTLs. Flavio I'm open to the alternative solution (which would be for programs which are not interested in having a PTL to just not have one). But then if things go badly wrong, you require the TC to step in with threats of removal of OpenStack and/or to force an election/vote in the middle of the crisis. I'm really failing to see how that would result, in those hypothetical crisis scenarios, in a better outcome. -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On 22/08/14 21:02, Anne Gentle wrote: I'm with Rocky on the anti-czar-as-a-word camp. We all like clever names to shed the corporate stigma but this word ain't it. Liaison or lead? +1. The only time you hear the word 'czar' in regular life (outside of references to pre-revolutionary Russia) it means that the government is looking for a cheap PR win that doesn't require actually doing/changing anything. Liaison or Contact would be fine choices IMHO. cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On Aug 23, 2014, at 6:35 PM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Dolph Mathews's message of 2014-08-22 09:45:37 -0700: On Fri, Aug 22, 2014 at 11:32 AM, Zane Bitter zbit...@redhat.com wrote: On 22/08/14 11:19, Thierry Carrez wrote: Zane Bitter wrote: On 22/08/14 08:33, Thierry Carrez wrote: We also still need someone to have the final say in case of deadlocked issues. -1 we really don't. I know we disagree on that :) No problem, you and I work in different programs so we can both get our way ;) People say we don't have that many deadlocks in OpenStack for which the PTL ultimate power is needed, so we could get rid of them. I'd argue that the main reason we don't have that many deadlocks in OpenStack is precisely *because* we have a system to break them if they arise. s/that many/any/ IME and I think that threatening to break a deadlock by fiat is just as bad as actually doing it. And by 'bad' I mean community-poisoningly, trust-destroyingly bad. I guess I've been active in too many dysfunctional free and open source software projects -- I put a very high value on the ability to make a final decision. Not being able to make a decision is about as community-poisoning, and also results in inability to make any significant change or decision. I'm all for getting a final decision, but a 'final' decision that has been imposed from outside rather than internalised by the participants is... rarely final. The expectation of a PTL isn't to stomp around and make final decisions, it's to step in when necessary and help both sides find the best solution. To moderate. Have we had many instances where a project's community divided into two camps and dug in to the point where they actually needed active moderation? And in those cases, was the PTL not already on one side of said argument? I'd prefer specific examples here. I have yet to see a deadlock in Heat that wasn't resolved by better communication. Moderation == bettering communication. I'm under the impression that you and Thierry are agreeing here, just from opposite ends of the same spectrum. I agree as well. PTL is a servant of the community, as any good leader is. If the PTL feels they have to drop the hammer, or if an impass is reached where they are asked to, it is because they have failed to get everyone communicating effectively, not because that's their job.” That’s certainly how I approach it. I consider myself responsible for helping the team coordinate and making sure we have everything covered. Sometimes that means asking for volunteers for a new task, and sometimes it means a gentle reminder of an previous commitment. That said, some responsibilities of the PTL role are outward-facing. Potentially anyone on the team could pick up those duties, just as any of the inward-facing duties, but having a single initial point of contact is especially useful when the priorities of two teams need to be aligned for a period of time to work on a joint task. Doug ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
Zane Bitter wrote: On 22/08/14 21:02, Anne Gentle wrote: I'm with Rocky on the anti-czar-as-a-word camp. We all like clever names to shed the corporate stigma but this word ain't it. Liaison or lead? +1. The only time you hear the word 'czar' in regular life (outside of references to pre-revolutionary Russia) it means that the government is looking for a cheap PR win that doesn't require actually doing/changing anything. Liaison or Contact would be fine choices IMHO. Or, how about Secretary? Such as part of a cabinet? Secretary of bugs is kinda cool in that they collect info and report, troubleshoot, etc., but final decisions are directed by PTL. --Rocky cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On 2014-08-25 19:39:30 + (+), Rochelle.RochelleGrober wrote: Or, how about Secretary? [...] While we're painting this particular bike shed, I have a preference for janitor, drudge, mule, valet, slogger or similar terms which make it apparent that there is nothing at all glamorous nor powerful about the appointment. -- Jeremy Stanley ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On Mon 25 Aug 2014 03:38:18 PM CDT, Zane Bitter wrote: I'd say we've done fairly well, but I would attribute that at least in part to the fact that we've treated the PTL as effectively the temporary release management contact more than the guy who will resolve disputes for us. In other words, despite rather than because of the requirement to have a PTL. I found this (despite rather than because) a non sequitur. Let's put this conversation to rest since it seems there is no disagreement on the fact that the PTL role is working and stop investigating why it's working. Imagine a scenario where the community is more or less evenly split and neither side is willing to back down even after seeking guidance from the TC, the PTL breaks the deadlock by fiat in lieu of consensus, followed by [...] You highlight an interesting scenario: let's file it in the 'risks' bucket. I think we have already ways to mitigate the risk of this unlikely event to happen, with Foundation intervening, Board and other mediation coming from *outside* of the dev community. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
Zane Bitter [August 25, 2014 1:38 PM] wrote: . . . snip I'd say we've done fairly well, but I would attribute that at least in part to the fact that we've treated the PTL as effectively the temporary release management contact more than the guy who will resolve disputes for us. In other words, despite rather than because of the requirement to have a PTL. I do think that having a PTL with no actual responsibilities runs the risk of having it be seen as a trophy rather than a service to the community. The good PTLs - which so far is all of them - are likely to respond by volunteering for just as many things as they were doing before, so that will tend to counteract the goal of preventing burnout. I don't think anyone is saying or even really believes that distributing the workload would result in a PTL no actual responsibilities. There is just so much to do as a PTL for the larger projects that even having a team focused on ensuring the tactical activities happen will still leave the PTL with a superhuman workload: planning, coordinating, correcting, driving, regrouping, focusing, liaising, etc. As I said in my previous mail (got lost in the conversation about what to call these team members), To keep growing quality, communications, contributions and the health of the projects and OpenStack as an ecosystem, the PTLs must not only be strategic thinkers, but strategic actors. And it's pretty darn hard to be strategic when you're down in the tactical, day-to-day weeds. All of it is important, but the only way to keep OpenStack growing *and* healthy is to start to specialize in organizational areas, not just code areas. Look at the organic growth of technical projects in general. When you start with just a few people, communication is easy. Everyone knows the whole project and it's easy to stay on the same page. New people come on board and now you need to document design, operation and organizational lore. More people come on board and you need to track bugs, maybe features, and maybe split into groups, which means a leader needs to arise in each group such that the multiple groups can stay in sync and integrate their components. And it continues to grow. Some OpenStack projects have reached the state where each of them are really multiple projects, each with a lead. Neutron and TripleO both address this situation, empowering the internal projects and project leads, with the PTLs becoming more strategic and more focused on the ecosystem of the subprojects. I bet Kyle Mestery, Jay Pipes and Robert Collins would be happy to dissuade you of the idea that they don't have any responsibilities ;-) --Rocky I'm open to the alternative solution (which would be for programs which are not interested in having a PTL to just not have one). But then if things go badly wrong, you require the TC to step in with threats of removal of OpenStack and/or to force an election/vote in the middle of the crisis. I'm really failing to see how that would result, in those hypothetical crisis scenarios, in a better outcome. I don't think there are any good scenarios if you get to that crisis point. Imagine a scenario where the community is more or less evenly split and neither side is willing to back down even after seeking guidance from the TC, the PTL breaks the deadlock by fiat in lieu of consensus, followed by an unusually high number of spelling mistakes corrections in the source code, a single-issue election, potentially a reversal of the decision and possibly a fork that will force the TC to step in and choose a side. (Note: not choosing is also a choice.) That's pretty much an unmitigated disaster too. Our goal must be to avoid reaching the crisis point, and it seems to me that it is actually helpful to make clear to projects that their choices are: Option A: reach consensus Option B: there is no Option B cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On 08/22/2014 08:19 PM, John Dickinson wrote: I think Anne makes some excellent points about the pattern being proposed being unlikely to be commonly implemented across all the programs (or, at best, very difficult). Let's not try to formalize another best practice that works many times and force it to work every time. I kinda agree with John here. I think the most important effect of this proposal is that it will add *transparency* to the development effort. One of the most common questions I have to answer is 'Who should I talk to to address this issue? and I can't blame them. Just to give you an example, the list of core reviewers of many project is not immediately available, sometimes even the name of the PTL is missing from their wiki pages (since the wiki pages don't follow a common template). Since we've been growing so much I'm not surprised nor afraid of becoming more of a bureaucratic organization (in Taylor/Weber's sense of an organizational structure, we're simply growing up --poor wikipedia page on this topic http://en.wikipedia.org/wiki/Organizational_structure#Bureaucratic_structures) and introduce more rules. I like the idea of having a template of roles that support PTLs in their job and I think we should let PTLs pick which of the roles to fill in, but not divert too much from the common list of roles. And we need to make sure that these roles and people are communicated properly. I think we outgrew the phase where any PTL could experiment and self-organize in full autonomy because snowflakes are too hard to manage and produce only entropy at our size. We have too many projects, programs and teams and if all of them operate very differently it becomes next to impossible for newcomers to join. It gets too hard even for community managers to advice newcomers. We're too big for that. Some level of standardization is inevitable :) And let's get the PTLs together for one or two days every cycle to discuss project issues. Just PTLs, and let's focus on the project management stuff and some cross-project issues. This is orthogonal to the PTLs supporting roles: we can sure do something to help in this aspect. On another note, bike shed 'czar' gets -1 from me. Liaison or any of fungi's suggestions sound better :) /stef ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On Fri, Aug 22, 2014 at 02:33:27PM +0200, Thierry Carrez wrote: [Snip some well articulated thoughts.] Enter the Czar system: each project should have a number of liaisons / official contacts / delegates that are fully responsible to cover one aspect of the project. We need to have Bugs czars, which are responsible for getting bugs under control. We need to have Oslo czars, which serve as liaisons for the Oslo program but also as active project-local oslo advocates. We need Security czars, which the VMT can go to to progress quickly on plugging vulnerabilities. We need release management czars, to handle the communication and process with that painful OpenStack release manager. We need Gate czars to serve as first-line-of-contact getting gate issues fixed... You get the idea. Just a gentle note on Gate: From my limited observation, I feel there's a chance that people might burn out early if they're only focused on Gate 100% of the time -- given the sheer number of moving parts and the cross-project coordination one has to be involved with. I recall Sean Dague mention elsewhere in a different thread[1] about feeling (understandably) frustrated trying to chase Gate issues and the lack of enough volunteers on that front. So, probably Gate czars might need to offset their work with some other activity, to retain a sense of sanity. Some people can be czars of multiple areas. This point should cover that. [1] http://lists.openstack.org/pipermail/openstack-dev/2014-June/037620.html -- /kashyap ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On 08/23/2014 06:35 PM, Clint Byrum wrote: I agree as well. PTL is a servant of the community, as any good leader is. If the PTL feels they have to drop the hammer, or if an impass is reached where they are asked to, it is because they have failed to get everyone communicating effectively, not because that's their job. The problem isn't really that teams are not communicating effectively, nor is the problem to do with some deficit of a PTL in either putting the hammer down or failing to figure out common ground. The issue in my opinion and my experience is that there are multiple valid ways of doing something (say, deployment or metering or making toast) and the TC and our governing structure has decided to pick winners in spaces instead of having a big tent and welcoming different solutions and projects into the OpenStack fold. We pick winners and by doing so, we are exclusionary, and this exclusivity does not benefit our user community, but rather just gives it fewer options. IMHO, the TC should become an advisory team that recommends to interested project teams ways in which they can design and architect their projects to integrate well with other projects in the OpenStack community, and design their projects for the scale, stability and requirements (such as multi-tenancy) that an open cloud software ecosystem demands. Just my two cents, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On 08/22/2014 09:02 PM, Anne Gentle wrote: /flame-on Let's call spades, spades here. Czar is not only overkill, but the wrong metaphor. /flame-off I'm with Rocky on the anti-czar-as-a-word camp. We all like clever names to shed the corporate stigma but this word ain't it. Liaison or lead? Also wanted to point to https://wiki.openstack.org/wiki/PTLguide as it's quite nice. In the Linux Kernel world they are called Lieutenants, but they are per sub system. I'm guessing that is more like the PTL role today. THe etymology is good, to, as a Lieutenant is one who stands in Lieu of the commander. I think that some projects probably need to divvy up the roles along different line. For example, in Keystone, we have KVS, SQL and LDAP backends. Morgan's been the KVS guy due to rewriting things for Dogpile, and I've been kind of the LDAP guy since I was originally responsible for that. I suspect the same is true of other projects. Many of the roles you listed (Bug, etc) are straight PTL responsibilities in my book. What is lacking is a structure over delegating out some subset of that responsibility: no regular bug triage meeting, etc. I'd rather not throw all bug-triage onto one persons shoulders, as it really should be a team effort. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
-Original Message- From: John Dickinson [mailto:m...@not.mn] Sent: 23 August 2014 03:20 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs I think Anne makes some excellent points about the pattern being proposed being unlikely to be commonly implemented across all the programs (or, at best, very difficult). Let's not try to formalize another best practice that works many times and force it to work every time. Here's an alternate proposal: Let's let PTLs be PTLs and effectively coordinate and manage the activity in their respective projects. And let's get the PTLs together for one or two days every cycle to discuss project issues. Just PTLs, and let's focus on the project management stuff and some cross-project issues. Getting the PTLs together would allow them to discuss cross-project issues, share frustrations and solutions about what does and doesn't work. Basically, think of it as a mid-cycle meetup, but for PTLs. (Perhaps we could even ask the Foundation to sponsor it.) --John As part of the user feedback loop, we've found the PTL role extremely useful to channel feedback. The operator PTL discussions during the Atlanta summit helped to clarify a number of areas where the PTL can then take the points back to the design summit. It is not clear how czars would address the outward facing part of the PTL role which is clearly needed in view of the various discussions around program management and priorities. If we have lots of czars or major differences in how each project is structured, it is not clear how we channel user feedback to the project teams. Would there be a user czar on each project ? I have no problem with lots of czars to delegate activities across the projects but having a single accountable and elected PTL who can choose the level of delegation (and be assessed on that) seems to be a very good feature. Tim ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
Excerpts from Dolph Mathews's message of 2014-08-22 09:45:37 -0700: On Fri, Aug 22, 2014 at 11:32 AM, Zane Bitter zbit...@redhat.com wrote: On 22/08/14 11:19, Thierry Carrez wrote: Zane Bitter wrote: On 22/08/14 08:33, Thierry Carrez wrote: We also still need someone to have the final say in case of deadlocked issues. -1 we really don't. I know we disagree on that :) No problem, you and I work in different programs so we can both get our way ;) People say we don't have that many deadlocks in OpenStack for which the PTL ultimate power is needed, so we could get rid of them. I'd argue that the main reason we don't have that many deadlocks in OpenStack is precisely *because* we have a system to break them if they arise. s/that many/any/ IME and I think that threatening to break a deadlock by fiat is just as bad as actually doing it. And by 'bad' I mean community-poisoningly, trust-destroyingly bad. I guess I've been active in too many dysfunctional free and open source software projects -- I put a very high value on the ability to make a final decision. Not being able to make a decision is about as community-poisoning, and also results in inability to make any significant change or decision. I'm all for getting a final decision, but a 'final' decision that has been imposed from outside rather than internalised by the participants is... rarely final. The expectation of a PTL isn't to stomp around and make final decisions, it's to step in when necessary and help both sides find the best solution. To moderate. Have we had many instances where a project's community divided into two camps and dug in to the point where they actually needed active moderation? And in those cases, was the PTL not already on one side of said argument? I'd prefer specific examples here. I have yet to see a deadlock in Heat that wasn't resolved by better communication. Moderation == bettering communication. I'm under the impression that you and Thierry are agreeing here, just from opposite ends of the same spectrum. I agree as well. PTL is a servant of the community, as any good leader is. If the PTL feels they have to drop the hammer, or if an impass is reached where they are asked to, it is because they have failed to get everyone communicating effectively, not because that's their job. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
Hi everyone, We all know being a project PTL is an extremely busy job. That's because in our structure the PTL is responsible for almost everything in a project: - Release management contact - Work prioritization - Keeping bugs under control - Communicate about work being planned or done - Make sure the gate is not broken - Team logistics (run meetings, organize sprints) - ... They end up being completely drowned in those day-to-day operational duties, miss the big picture, can't help in development that much anymore, get burnt out. Since you're either the PTL or not the PTL, you're very alone and succession planning is not working that great either. There have been a number of experiments to solve that problem. John Garbutt has done an incredible job at helping successive Nova PTLs handling the release management aspect. Tracy Jones took over Nova bug management. Doug Hellmann successfully introduced the concept of Oslo liaisons to get clear point of contacts for Oslo library adoption in projects. It may be time to generalize that solution. The issue is one of responsibility: the PTL is ultimately responsible for everything in a project. If we can more formally delegate that responsibility, we can avoid getting up to the PTL for everything, we can rely on a team of people rather than just one person. Enter the Czar system: each project should have a number of liaisons / official contacts / delegates that are fully responsible to cover one aspect of the project. We need to have Bugs czars, which are responsible for getting bugs under control. We need to have Oslo czars, which serve as liaisons for the Oslo program but also as active project-local oslo advocates. We need Security czars, which the VMT can go to to progress quickly on plugging vulnerabilities. We need release management czars, to handle the communication and process with that painful OpenStack release manager. We need Gate czars to serve as first-line-of-contact getting gate issues fixed... You get the idea. Some people can be czars of multiple areas. PTLs can retain some czar activity if they wish. Czars can collaborate with their equivalents in other projects to share best practices. We just need a clear list of areas/duties and make sure each project has a name assigned to each. Now, why czars ? Why not rely on informal activity ? Well, for that system to work we'll need a lot of people to step up and sign up for more responsibility. Making them czars makes sure that effort is recognized and gives them something back. Also if we don't formally designate people, we can't really delegate and the PTL will still be directly held responsible. The Release management czar should be able to sign off release SHAs without asking the PTL. The czars and the PTL should collectively be the new project drivers. At that point, why not also get rid of the PTL ? And replace him with a team of czars ? If the czar system is successful, the PTL should be freed from the day-to-day operational duties and will be able to focus on the project health again. We still need someone to keep an eye on the project-wide picture and coordinate the work of the czars. We need someone to pick czars, in the event multiple candidates sign up. We also still need someone to have the final say in case of deadlocked issues. People say we don't have that many deadlocks in OpenStack for which the PTL ultimate power is needed, so we could get rid of them. I'd argue that the main reason we don't have that many deadlocks in OpenStack is precisely *because* we have a system to break them if they arise. That encourages everyone to find a lazy consensus. That part of the PTL job works. Let's fix the part that doesn't work (scaling/burnout). -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On 08/22/2014 08:33 AM, Thierry Carrez wrote: Hi everyone, We all know being a project PTL is an extremely busy job. That's because in our structure the PTL is responsible for almost everything in a project: - Release management contact - Work prioritization - Keeping bugs under control - Communicate about work being planned or done - Make sure the gate is not broken - Team logistics (run meetings, organize sprints) - ... They end up being completely drowned in those day-to-day operational duties, miss the big picture, can't help in development that much anymore, get burnt out. Since you're either the PTL or not the PTL, you're very alone and succession planning is not working that great either. There have been a number of experiments to solve that problem. John Garbutt has done an incredible job at helping successive Nova PTLs handling the release management aspect. Tracy Jones took over Nova bug management. Doug Hellmann successfully introduced the concept of Oslo liaisons to get clear point of contacts for Oslo library adoption in projects. It may be time to generalize that solution. The issue is one of responsibility: the PTL is ultimately responsible for everything in a project. If we can more formally delegate that responsibility, we can avoid getting up to the PTL for everything, we can rely on a team of people rather than just one person. Enter the Czar system: each project should have a number of liaisons / official contacts / delegates that are fully responsible to cover one aspect of the project. We need to have Bugs czars, which are responsible for getting bugs under control. We need to have Oslo czars, which serve as liaisons for the Oslo program but also as active project-local oslo advocates. We need Security czars, which the VMT can go to to progress quickly on plugging vulnerabilities. We need release management czars, to handle the communication and process with that painful OpenStack release manager. We need Gate czars to serve as first-line-of-contact getting gate issues fixed... You get the idea. Some people can be czars of multiple areas. PTLs can retain some czar activity if they wish. Czars can collaborate with their equivalents in other projects to share best practices. We just need a clear list of areas/duties and make sure each project has a name assigned to each. Now, why czars ? Why not rely on informal activity ? Well, for that system to work we'll need a lot of people to step up and sign up for more responsibility. Making them czars makes sure that effort is recognized and gives them something back. Also if we don't formally designate people, we can't really delegate and the PTL will still be directly held responsible. The Release management czar should be able to sign off release SHAs without asking the PTL. The czars and the PTL should collectively be the new project drivers. At that point, why not also get rid of the PTL ? And replace him with a team of czars ? If the czar system is successful, the PTL should be freed from the day-to-day operational duties and will be able to focus on the project health again. We still need someone to keep an eye on the project-wide picture and coordinate the work of the czars. We need someone to pick czars, in the event multiple candidates sign up. We also still need someone to have the final say in case of deadlocked issues. People say we don't have that many deadlocks in OpenStack for which the PTL ultimate power is needed, so we could get rid of them. I'd argue that the main reason we don't have that many deadlocks in OpenStack is precisely *because* we have a system to break them if they arise. That encourages everyone to find a lazy consensus. That part of the PTL job works. Let's fix the part that doesn't work (scaling/burnout). +1 on czars. That's what was working best for me to start scaling things in Nova, especially through my 2nd term (Icehouse). John and Tracy were a big help as you mentioned as examples. There were others that were stepping up, too. I think it's been working well enough to formalize it. Another area worth calling out is a gate czar. Having someone who understands infra and QA quite well and is regularly on top of the status of the project in the gate is helpful and quite important. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On 08/22/2014 09:40 AM, Russell Bryant wrote: Another area worth calling out is a gate czar. Having someone who understands infra and QA quite well and is regularly on top of the status of the project in the gate is helpful and quite important. Oops, you said this one, too. Anyway, +1. -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On 08/22/2014 07:33 AM, Thierry Carrez wrote: Hi everyone, We all know being a project PTL is an extremely busy job. That's because in our structure the PTL is responsible for almost everything in a project: - Release management contact - Work prioritization - Keeping bugs under control - Communicate about work being planned or done - Make sure the gate is not broken - Team logistics (run meetings, organize sprints) - ... They end up being completely drowned in those day-to-day operational duties, miss the big picture, can't help in development that much anymore, get burnt out. Since you're either the PTL or not the PTL, you're very alone and succession planning is not working that great either. There have been a number of experiments to solve that problem. John Garbutt has done an incredible job at helping successive Nova PTLs handling the release management aspect. Tracy Jones took over Nova bug management. Doug Hellmann successfully introduced the concept of Oslo liaisons to get clear point of contacts for Oslo library adoption in projects. It may be time to generalize that solution. The issue is one of responsibility: the PTL is ultimately responsible for everything in a project. If we can more formally delegate that responsibility, we can avoid getting up to the PTL for everything, we can rely on a team of people rather than just one person. Enter the Czar system: each project should have a number of liaisons / official contacts / delegates that are fully responsible to cover one aspect of the project. We need to have Bugs czars, which are responsible for getting bugs under control. We need to have Oslo czars, which serve as liaisons for the Oslo program but also as active project-local oslo advocates. We need Security czars, which the VMT can go to to progress quickly on plugging vulnerabilities. We need release management czars, to handle the communication and process with that painful OpenStack release manager. We need Gate czars to serve as first-line-of-contact getting gate issues fixed... You get the idea. Some people can be czars of multiple areas. PTLs can retain some czar activity if they wish. Czars can collaborate with their equivalents in other projects to share best practices. We just need a clear list of areas/duties and make sure each project has a name assigned to each. Now, why czars ? Why not rely on informal activity ? Well, for that system to work we'll need a lot of people to step up and sign up for more responsibility. Making them czars makes sure that effort is recognized and gives them something back. Also if we don't formally designate people, we can't really delegate and the PTL will still be directly held responsible. The Release management czar should be able to sign off release SHAs without asking the PTL. The czars and the PTL should collectively be the new project drivers. At that point, why not also get rid of the PTL ? And replace him with a him or her (those French pronouns again!) team of czars ? If the czar system is successful, the PTL should be freed from the day-to-day operational duties and will be able to focus on the project health again. We still need someone to keep an eye on the project-wide picture and coordinate the work of the czars. We need someone to pick czars, in the event multiple candidates sign up. We also still need someone to have the final say in case of deadlocked issues. People say we don't have that many deadlocks in OpenStack for which the PTL ultimate power is needed, so we could get rid of them. I'd argue that the main reason we don't have that many deadlocks in OpenStack is precisely *because* we have a system to break them if they arise. That encourages everyone to find a lazy consensus. That part of the PTL job works. Let's fix the part that doesn't work (scaling/burnout). I would like to work with a single point of contact for any program engaged in the third party space. Thanks, Anita. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On Fri, Aug 22, 2014 at 02:33:27PM +0200, Thierry Carrez wrote: Hi everyone, We all know being a project PTL is an extremely busy job. That's because in our structure the PTL is responsible for almost everything in a project: - Release management contact - Work prioritization - Keeping bugs under control - Communicate about work being planned or done - Make sure the gate is not broken - Team logistics (run meetings, organize sprints) - ... This is a good list of responsbilities, but I feel like we're missing something that is possibly a little too fuzzy to describe a bullet point. In terms of our governance process, PTL is an elected role, so in that sense the PTL holds an implicit duty to represent the interests of the project constituents. I'd characterise this as setting the overall big picture direction, so that the core team (and/or czars) are focusing their efforts in the right directioon and generally ensuring that the project is operating in a healthy manner. Perhaps you could class that as 'team logistics' but I feel that the idea of 'representation the voters' is is worth calling out explicitly. Is there any formal writeup of the PTL role's responsibilities ? With Google so far I only found one paragraph of governance https://wiki.openstack.org/wiki/Governance/Foundation/Structure Project Technical Leads (PTLs) lead individual projects. A PTL is ultimately responsible for the direction for each project, makes tough calls when needed, organizes the work and teams in the project and determines if other forms of project leadership are needed. The PTL for a project is elected by the body of contributors to that particular project. Anyway, with the idea of elections representation in mind Enter the Czar system: each project should have a number of liaisons / official contacts / delegates that are fully responsible to cover one aspect of the project. We need to have Bugs czars, which are responsible for getting bugs under control. We need to have Oslo czars, which serve as liaisons for the Oslo program but also as active project-local oslo advocates. We need Security czars, which the VMT can go to to progress quickly on plugging vulnerabilities. We need release management czars, to handle the communication and process with that painful OpenStack release manager. We need Gate czars to serve as first-line-of-contact getting gate issues fixed... You get the idea. Some people can be czars of multiple areas. PTLs can retain some czar activity if they wish. Czars can collaborate with their equivalents in other projects to share best practices. We just need a clear list of areas/duties and make sure each project has a name assigned to each. Now, why czars ? Why not rely on informal activity ? Well, for that system to work we'll need a lot of people to step up and sign up for more responsibility. Making them czars makes sure that effort is recognized and gives them something back. Also if we don't formally designate people, we can't really delegate and the PTL will still be directly held responsible. The Release management czar should be able to sign off release SHAs without asking the PTL. The czars and the PTL should collectively be the new project drivers. At that point, why not also get rid of the PTL ? And replace him with a team of czars ? If the czar system is successful, the PTL should be freed from the day-to-day operational duties and will be able to focus on the project health again. We still need someone to keep an eye on the project-wide picture and coordinate the work of the czars. We need someone to pick czars, in the event multiple candidates sign up. We also still need someone to have the final say in case of deadlocked issues. I'm wondering how people come to be czars ? You don't say explicitly, but reading this it feels like the team of czars would be more or less self-selecting amongst the project contributors or nominated by the PTL ? Thus if we took the next step and got rid of the PTL, then we seem to have entirely removed the idea of democratically elected leadership from the individual projects. Is this interpretation correct, wrt the idea of a czar system with PTL role abolished ? People say we don't have that many deadlocks in OpenStack for which the PTL ultimate power is needed, so we could get rid of them. I'd argue that the main reason we don't have that many deadlocks in OpenStack is precisely *because* we have a system to break them if they arise. That encourages everyone to find a lazy consensus. That part of the PTL job works. Let's fix the part that doesn't work (scaling/burnout). Even if we didn't have deadlocks with a pure czar system, I fear that we would be loosing something important by no longer having the project members directly elect their leader. The elections serve to give the membership a sense of representation control over the direction of the project. Without that
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On 08/22/2014 02:33 PM, Thierry Carrez wrote: Hi everyone, We all know being a project PTL is an extremely busy job. That's because in our structure the PTL is responsible for almost everything in a project: - Release management contact - Work prioritization - Keeping bugs under control - Communicate about work being planned or done - Make sure the gate is not broken - Team logistics (run meetings, organize sprints) - ... They end up being completely drowned in those day-to-day operational duties, miss the big picture, can't help in development that much anymore, get burnt out. Since you're either the PTL or not the PTL, you're very alone and succession planning is not working that great either. There have been a number of experiments to solve that problem. John Garbutt has done an incredible job at helping successive Nova PTLs handling the release management aspect. Tracy Jones took over Nova bug management. Doug Hellmann successfully introduced the concept of Oslo liaisons to get clear point of contacts for Oslo library adoption in projects. It may be time to generalize that solution. The issue is one of responsibility: the PTL is ultimately responsible for everything in a project. If we can more formally delegate that responsibility, we can avoid getting up to the PTL for everything, we can rely on a team of people rather than just one person. Enter the Czar system: each project should have a number of liaisons / official contacts / delegates that are fully responsible to cover one aspect of the project. We need to have Bugs czars, which are responsible for getting bugs under control. We need to have Oslo czars, which serve as liaisons for the Oslo program but also as active project-local oslo advocates. We need Security czars, which the VMT can go to to progress quickly on plugging vulnerabilities. We need release management czars, to handle the communication and process with that painful OpenStack release manager. We need Gate czars to serve as first-line-of-contact getting gate issues fixed... You get the idea. Some people can be czars of multiple areas. PTLs can retain some czar activity if they wish. Czars can collaborate with their equivalents in other projects to share best practices. We just need a clear list of areas/duties and make sure each project has a name assigned to each. Now, why czars ? Why not rely on informal activity ? Well, for that system to work we'll need a lot of people to step up and sign up for more responsibility. Making them czars makes sure that effort is recognized and gives them something back. Also if we don't formally designate people, we can't really delegate and the PTL will still be directly held responsible. The Release management czar should be able to sign off release SHAs without asking the PTL. The czars and the PTL should collectively be the new project drivers. At that point, why not also get rid of the PTL ? And replace him with a team of czars ? If the czar system is successful, the PTL should be freed from the day-to-day operational duties and will be able to focus on the project health again. We still need someone to keep an eye on the project-wide picture and coordinate the work of the czars. We need someone to pick czars, in the event multiple candidates sign up. We also still need someone to have the final say in case of deadlocked issues. People say we don't have that many deadlocks in OpenStack for which the PTL ultimate power is needed, so we could get rid of them. I'd argue that the main reason we don't have that many deadlocks in OpenStack is precisely *because* we have a system to break them if they arise. That encourages everyone to find a lazy consensus. That part of the PTL job works. Let's fix the part that doesn't work (scaling/burnout). +1 FWIW, this is how things have worked in Zaqar since the very beginning. We've split our duties throughout the team, which helped clearing some of the PTL duties. So far, we have people responsible for: * Bugs for the server * Bugs for the client * Gate and Tests * Client releases The areas not listed there are still part of the PTLs responsibilities. Cheers, Flavio -- @flaper87 Flavio Percoco ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On 22/08/14 08:33, Thierry Carrez wrote: Hi everyone, We all know being a project PTL is an extremely busy job. That's because in our structure the PTL is responsible for almost everything in a project: - Release management contact - Work prioritization - Keeping bugs under control - Communicate about work being planned or done - Make sure the gate is not broken - Team logistics (run meetings, organize sprints) - ... They end up being completely drowned in those day-to-day operational duties, miss the big picture, can't help in development that much anymore, get burnt out. Since you're either the PTL or not the PTL, you're very alone and succession planning is not working that great either. Succession planning works as well as you want it to IMO. In Kilo, Heat will have its 5th PTL in 5 release cycles, and all of those successions except the first were planned. We have multiple potential candidates for the next election whom I think would be more than capable of doing a great job; the harder thing is finding people who are able to commit the time. There have been a number of experiments to solve that problem. John Garbutt has done an incredible job at helping successive Nova PTLs handling the release management aspect. Tracy Jones took over Nova bug management. Doug Hellmann successfully introduced the concept of Oslo liaisons to get clear point of contacts for Oslo library adoption in projects. It may be time to generalize that solution. +1 My goal as the Heat PTL has been to try to identify all of the places where the PTL can be a single point of failure and work toward eliminating them, as a first step toward eliminating PTLs altogether. BTW the big two remaining are scheduling sessions for the design summit and approving/targeting blueprints in Launchpad - the specs repos helped somewhat with the latter, but it won't be completely fixed until Launchpad goes away. The issue is one of responsibility: the PTL is ultimately responsible for everything in a project. If we can more formally delegate that responsibility, we can avoid getting up to the PTL for everything, we can rely on a team of people rather than just one person. First off, the PTL is not responsible for everything in a project. *Everyone* is responsible for everything in a project. The PTL is *accountable* for everything in a project. PTLs are the mechanism the TC uses to ensure that programs remain accountable to the wider community. Enter the Czar system: each project should have a number of liaisons / official contacts / delegates that are fully responsible to cover one aspect of the project. We need to have Bugs czars, which are responsible for getting bugs under control. We need to have Oslo czars, which serve as liaisons for the Oslo program but also as active project-local oslo advocates. We need Security czars, which the VMT can go to to progress quickly on plugging vulnerabilities. We need release management czars, to handle the communication and process with that painful OpenStack release manager. We need Gate czars to serve as first-line-of-contact getting gate issues fixed... You get the idea. +1 Rather than putting it all on one person, we should enumerate the ways in which we want projects to be accountable to the wider community, and allow the projects themselves to determine who is accountable for each particular function. Furthermore, we should allow them to do so on their own cadence, rather than only at 6-month intervals. Some people can be czars of multiple areas. PTLs can retain some czar activity if they wish. Czars can collaborate with their equivalents in other projects to share best practices. We just need a clear list of areas/duties and make sure each project has a name assigned to each. Exactly, maybe we'd have a wiki page or something listing the contact person for each area in each project. Now, why czars ? Why not rely on informal activity ? Well, for that system to work we'll need a lot of people to step up and sign up for more responsibility. Making them czars makes sure that effort is recognized and gives them something back. Also if we don't formally designate people, we can't really delegate and the PTL will still be directly held responsible. The Release management czar should be able to sign off release SHAs without asking the PTL. The czars and the PTL should collectively be the new project drivers. At that point, why not also get rid of the PTL ? +1 :) And replace him with a team of czars ? If the czar system is successful, the PTL should be freed from the day-to-day operational duties and will be able to focus on the project health again. I don't see that as something the wider OpenStack community needs to dictate. We have a heavyweight election process for PTLs once every cycle because that used to be the process for electing the TC. Now that it no longer serves this dual purpose, PTL elections have outlived their usefulness. If
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
Russell Bryant wrote: On 08/22/2014 09:40 AM, Russell Bryant wrote: Another area worth calling out is a gate czar. Having someone who understands infra and QA quite well and is regularly on top of the status of the project in the gate is helpful and quite important. Oops, you said this one, too. Anyway, +1. The one I forgot in my example list would be the Docs czar. I'm pretty sure Anne would appreciate a clear point contact for docs in every integrated project. Another interesting side-effect of having clear positions is that if nobody fills them (which is a possibility), it's clear and public that there is a gap. Today the gap just ends up on the PTL's list of other things to also do. -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
Daniel P. Berrange wrote: On Fri, Aug 22, 2014 at 02:33:27PM +0200, Thierry Carrez wrote: Hi everyone, We all know being a project PTL is an extremely busy job. That's because in our structure the PTL is responsible for almost everything in a project: - Release management contact - Work prioritization - Keeping bugs under control - Communicate about work being planned or done - Make sure the gate is not broken - Team logistics (run meetings, organize sprints) - ... This is a good list of responsbilities, but I feel like we're missing something that is possibly a little too fuzzy to describe a bullet point. In terms of our governance process, PTL is an elected role, so in that sense the PTL holds an implicit duty to represent the interests of the project constituents. I'd characterise this as setting the overall big picture direction, so that the core team (and/or czars) are focusing their efforts in the right directioon and generally ensuring that the project is operating in a healthy manner. Perhaps you could class that as 'team logistics' but I feel that the idea of 'representation the voters' is is worth calling out explicitly. Indeed. I touch on the keep an eye on the big picture aspect of the job later in the email, but I didn't call it out in that list. Is there any formal writeup of the PTL role's responsibilities ? With Google so far I only found one paragraph of governance https://wiki.openstack.org/wiki/Governance/Foundation/Structure Project Technical Leads (PTLs) lead individual projects. A PTL is ultimately responsible for the direction for each project, makes tough calls when needed, organizes the work and teams in the project and determines if other forms of project leadership are needed. The PTL for a project is elected by the body of contributors to that particular project. The reference would be: https://wiki.openstack.org/wiki/PTLguide Anyway, with the idea of elections representation in mind Enter the Czar system: each project should have a number of liaisons / official contacts / delegates that are fully responsible to cover one aspect of the project. We need to have Bugs czars, which are responsible for getting bugs under control. We need to have Oslo czars, which serve as liaisons for the Oslo program but also as active project-local oslo advocates. We need Security czars, which the VMT can go to to progress quickly on plugging vulnerabilities. We need release management czars, to handle the communication and process with that painful OpenStack release manager. We need Gate czars to serve as first-line-of-contact getting gate issues fixed... You get the idea. Some people can be czars of multiple areas. PTLs can retain some czar activity if they wish. Czars can collaborate with their equivalents in other projects to share best practices. We just need a clear list of areas/duties and make sure each project has a name assigned to each. Now, why czars ? Why not rely on informal activity ? Well, for that system to work we'll need a lot of people to step up and sign up for more responsibility. Making them czars makes sure that effort is recognized and gives them something back. Also if we don't formally designate people, we can't really delegate and the PTL will still be directly held responsible. The Release management czar should be able to sign off release SHAs without asking the PTL. The czars and the PTL should collectively be the new project drivers. At that point, why not also get rid of the PTL ? And replace him with a team of czars ? If the czar system is successful, the PTL should be freed from the day-to-day operational duties and will be able to focus on the project health again. We still need someone to keep an eye on the project-wide picture and coordinate the work of the czars. We need someone to pick czars, in the event multiple candidates sign up. We also still need someone to have the final say in case of deadlocked issues. I'm wondering how people come to be czars ? You don't say explicitly, but reading this it feels like the team of czars would be more or less self-selecting amongst the project contributors or nominated by the PTL ? I think you would have volunteers (and volunteered) people. In the unlikely case people fight to become the czar, I guess the PTL could have final say, as it's also good to ensure that all czars don't happen to come up from a single company, etc. Thus if we took the next step and got rid of the PTL, then we seem to have entirely removed the idea of democratically elected leadership from the individual projects. Is this interpretation correct, wrt the idea of a czar system with PTL role abolished ? If you also abolish the PTL, yes. People say we don't have that many deadlocks in OpenStack for which the PTL ultimate power is needed, so we could get rid of them. I'd argue that the main reason we don't have that many
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
Zane Bitter wrote: On 22/08/14 08:33, Thierry Carrez wrote: We also still need someone to have the final say in case of deadlocked issues. -1 we really don't. I know we disagree on that :) People say we don't have that many deadlocks in OpenStack for which the PTL ultimate power is needed, so we could get rid of them. I'd argue that the main reason we don't have that many deadlocks in OpenStack is precisely *because* we have a system to break them if they arise. s/that many/any/ IME and I think that threatening to break a deadlock by fiat is just as bad as actually doing it. And by 'bad' I mean community-poisoningly, trust-destroyingly bad. I guess I've been active in too many dysfunctional free and open source software projects -- I put a very high value on the ability to make a final decision. Not being able to make a decision is about as community-poisoning, and also results in inability to make any significant change or decision. That encourages everyone to find a lazy consensus. That part of the PTL job works. Let's fix the part that doesn't work (scaling/burnout). Let's allow projects to decide for themselves what works. Not every project is the same. The net effect of not having a PTL having the final call means the final call would reside at the Technical Committee level. I don't feel like the Technical Committee should have final say on a project-specific matter. It's way better that the local leader, chosen by all the contributors of THAT project every 6 months, makes that final decision. Or do you also want to get rid of the Technical Committee ? -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On Fri, 2014-08-22 at 11:01 -0400, Zane Bitter wrote: I don't see that as something the wider OpenStack community needs to dictate. We have a heavyweight election process for PTLs once every cycle because that used to be the process for electing the TC. Now that it no longer serves this dual purpose, PTL elections have outlived their usefulness. If projects want to have a designated tech lead, let them. If they want to have the lead elected in a form of representative democracy, let them. But there's no need to impose that process on every project. If they want to rotate the tech lead every week instead of every 6 months, why not let them? We'll soon see from experimentation which models work. Let a thousand flowers bloom, c. I like the idea of projects being free to experiment with their governance rather than the TC mandating detailed governance models from above. But I also like the way Thierry is taking a trend we're seeing work out well across multiple projects, and generalizing it. If individual projects are to adopt explicit PTL duty delegation, then all the better if those projects adopt it in similar ways. i.e. this should turn out to be an optional best practice model that projects can choose to adopt, in much the way the *-specs repo idea took hold. Mark. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On 08/22/2014 08:33 AM, Thierry Carrez wrote: Hi everyone, We all know being a project PTL is an extremely busy job. That's because in our structure the PTL is responsible for almost everything in a project: - Release management contact - Work prioritization - Keeping bugs under control - Communicate about work being planned or done - Make sure the gate is not broken - Team logistics (run meetings, organize sprints) - ... They end up being completely drowned in those day-to-day operational duties, miss the big picture, can't help in development that much anymore, get burnt out. Since you're either the PTL or not the PTL, you're very alone and succession planning is not working that great either. There have been a number of experiments to solve that problem. John Garbutt has done an incredible job at helping successive Nova PTLs handling the release management aspect. Tracy Jones took over Nova bug management. Doug Hellmann successfully introduced the concept of Oslo liaisons to get clear point of contacts for Oslo library adoption in projects. It may be time to generalize that solution. The issue is one of responsibility: the PTL is ultimately responsible for everything in a project. If we can more formally delegate that responsibility, we can avoid getting up to the PTL for everything, we can rely on a team of people rather than just one person. Enter the Czar system: each project should have a number of liaisons / official contacts / delegates that are fully responsible to cover one aspect of the project. We need to have Bugs czars, which are responsible for getting bugs under control. We need to have Oslo czars, which serve as liaisons for the Oslo program but also as active project-local oslo advocates. We need Security czars, which the VMT can go to to progress quickly on plugging vulnerabilities. We need release management czars, to handle the communication and process with that painful OpenStack release manager. We need Gate czars to serve as first-line-of-contact getting gate issues fixed... You get the idea. Some people can be czars of multiple areas. PTLs can retain some czar activity if they wish. Czars can collaborate with their equivalents in other projects to share best practices. We just need a clear list of areas/duties and make sure each project has a name assigned to each. Now, why czars ? Why not rely on informal activity ? Well, for that system to work we'll need a lot of people to step up and sign up for more responsibility. Making them czars makes sure that effort is recognized and gives them something back. Also if we don't formally designate people, we can't really delegate and the PTL will still be directly held responsible. The Release management czar should be able to sign off release SHAs without asking the PTL. The czars and the PTL should collectively be the new project drivers. At that point, why not also get rid of the PTL ? And replace him with a team of czars ? If the czar system is successful, the PTL should be freed from the day-to-day operational duties and will be able to focus on the project health again. We still need someone to keep an eye on the project-wide picture and coordinate the work of the czars. We need someone to pick czars, in the event multiple candidates sign up. We also still need someone to have the final say in case of deadlocked issues. People say we don't have that many deadlocks in OpenStack for which the PTL ultimate power is needed, so we could get rid of them. I'd argue that the main reason we don't have that many deadlocks in OpenStack is precisely *because* we have a system to break them if they arise. That encourages everyone to find a lazy consensus. That part of the PTL job works. Let's fix the part that doesn't work (scaling/burnout). I think the czars approach is sensible and seems to have worked pretty well in a couple projects so far. And, since I work for a software company with Russian origin, I support the term czar as well ;) On the topic of whether a PTL is still needed once a czar system is put in place, I think that should be left up to each individual project to decide. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On Fri, Aug 22, 2014 at 10:51 AM, Jay Pipes jaypi...@gmail.com wrote: On 08/22/2014 08:33 AM, Thierry Carrez wrote: Hi everyone, We all know being a project PTL is an extremely busy job. That's because in our structure the PTL is responsible for almost everything in a project: - Release management contact - Work prioritization - Keeping bugs under control - Communicate about work being planned or done - Make sure the gate is not broken - Team logistics (run meetings, organize sprints) - ... They end up being completely drowned in those day-to-day operational duties, miss the big picture, can't help in development that much anymore, get burnt out. Since you're either the PTL or not the PTL, you're very alone and succession planning is not working that great either. There have been a number of experiments to solve that problem. John Garbutt has done an incredible job at helping successive Nova PTLs handling the release management aspect. Tracy Jones took over Nova bug management. Doug Hellmann successfully introduced the concept of Oslo liaisons to get clear point of contacts for Oslo library adoption in projects. It may be time to generalize that solution. The issue is one of responsibility: the PTL is ultimately responsible for everything in a project. If we can more formally delegate that responsibility, we can avoid getting up to the PTL for everything, we can rely on a team of people rather than just one person. Enter the Czar system: each project should have a number of liaisons / official contacts / delegates that are fully responsible to cover one aspect of the project. We need to have Bugs czars, which are responsible for getting bugs under control. We need to have Oslo czars, which serve as liaisons for the Oslo program but also as active project-local oslo advocates. We need Security czars, which the VMT can go to to progress quickly on plugging vulnerabilities. We need release management czars, to handle the communication and process with that painful OpenStack release manager. We need Gate czars to serve as first-line-of-contact getting gate issues fixed... You get the idea. Some people can be czars of multiple areas. PTLs can retain some czar activity if they wish. Czars can collaborate with their equivalents in other projects to share best practices. We just need a clear list of areas/duties and make sure each project has a name assigned to each. Now, why czars ? Why not rely on informal activity ? Well, for that system to work we'll need a lot of people to step up and sign up for more responsibility. Making them czars makes sure that effort is recognized and gives them something back. Also if we don't formally designate people, we can't really delegate and the PTL will still be directly held responsible. The Release management czar should be able to sign off release SHAs without asking the PTL. The czars and the PTL should collectively be the new project drivers. At that point, why not also get rid of the PTL ? And replace him with a team of czars ? If the czar system is successful, the PTL should be freed from the day-to-day operational duties and will be able to focus on the project health again. We still need someone to keep an eye on the project-wide picture and coordinate the work of the czars. We need someone to pick czars, in the event multiple candidates sign up. We also still need someone to have the final say in case of deadlocked issues. People say we don't have that many deadlocks in OpenStack for which the PTL ultimate power is needed, so we could get rid of them. I'd argue that the main reason we don't have that many deadlocks in OpenStack is precisely *because* we have a system to break them if they arise. That encourages everyone to find a lazy consensus. That part of the PTL job works. Let's fix the part that doesn't work (scaling/burnout). I think the czars approach is sensible and seems to have worked pretty well in a couple projects so far. And, since I work for a software company with Russian origin, I support the term czar as well ;) On the topic of whether a PTL is still needed once a czar system is put in place, I think that should be left up to each individual project to decide. Best, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev +1 to czars +1 to considering the future and role of TC versus future of PTL ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On 22/08/14 12:45, Dolph Mathews wrote: I'm all for getting a final decision, but a 'final' decision that has been imposed from outside rather than internalised by the participants is... rarely final. The expectation of a PTL isn't to stomp around and make final decisions, it's to step in when necessary and help both sides find the best solution. To moderate. Oh sure, but that's not just the PTL's job. That's everyone's job. Don't you think? I did that before I was the PTL and will continue to do it after I'm no longer the PTL. And if anyone in the (especially) the core or wider Heat team sees an opportunity to step in and moderate a disagreement I certainly expect them to take it and not wait for me to step in. I'm not calling for no leadership here - I'm calling for leadership from _everyone_, not just from one person who holds a particular role. cheers, Zane. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
/flame-on Ok, this is funny to some of us in the community. The general populace of this community is so against the idea of management that they will use the term for a despotic dictator as a position name rather than manager. Sorry, but this needed to be said. /flame-off Specific comments in line: Thierry Carrez wrote: Hi everyone, We all know being a project PTL is an extremely busy job. That's because in our structure the PTL is responsible for almost everything in a project: - Release management contact - Work prioritization - Keeping bugs under control - Communicate about work being planned or done - Make sure the gate is not broken - Team logistics (run meetings, organize sprints) - ... Point of clarification: I've heard PTL=Project Technical Lead and PTL=Program Technical Lead. Which is it? It is kind of important as OpenStack grows, because the first is responsible for *a* project, and the second is responsible for all projects within a program. I'd also like to set out as an example of a Program that is growing to encompass multiple projects, the Neutron Program. Look at how it is expanding: Multiple sub-teams for: LBAAS, DNAAS, GBP, etc. This model could be extended such that: - the subteam is responsible for code reviews, including the first +2 for design, architecture and code of the sub-project, always also keeping an eye out that the sub-project code continues to both integrate well with the program, and that the program continues to provide the needed code bits, architecture modifications and improvements, etc. to support the sub-project. - the final +2/A would be from the Program reviewers to ensure that all integrate nicely together into a single, cohesive program. - This would allow sub-projects to have core reviewers, along with the program and be a good separation of duties. It would also help to increase the number of reviews moving to merged code. - Taken to a logical stepping stone, you would have project technical leads for each project, and they would make up a program council, with the program technical lead being the chair of the council. This is a way to offload a good chunk of PTL tactical responsibilities and help them focus more on the strategic. They end up being completely drowned in those day-to-day operational duties, miss the big picture, can't help in development that much anymore, get burnt out. Since you're either the PTL or not the PTL, you're very alone and succession planning is not working that great either. There have been a number of experiments to solve that problem. John Garbutt has done an incredible job at helping successive Nova PTLs handling the release management aspect. Tracy Jones took over Nova bug management. Doug Hellmann successfully introduced the concept of Oslo liaisons to get clear point of contacts for Oslo library adoption in projects. It may be time to generalize that solution. The issue is one of responsibility: the PTL is ultimately responsible for everything in a project. If we can more formally delegate that responsibility, we can avoid getting up to the PTL for everything, we can rely on a team of people rather than just one person. Enter the Czar system: each project should have a number of liaisons / official contacts / delegates that are fully responsible to cover one aspect of the project. We need to have Bugs czars, which are responsible for getting bugs under control. We need to have Oslo czars, which serve as liaisons for the Oslo program but also as active project-local oslo advocates. We need Security czars, which the VMT can go to to progress quickly on plugging vulnerabilities. We need release management czars, to handle the communication and process with that painful OpenStack release manager. We need Gate czars to serve as first-line-of-contact getting gate issues fixed... You get the idea. /flame-on Let's call spades, spades here. Czar is not only overkill, but the wrong metaphor. /flame-off Each position suggested here exists in corporate development projects: - Bug czar == bug manager/administrator/QA engineer/whatever - someone in charge of making sure bugs get triages, assigned and completed - Oslo czar == systems engineers/project managers who make sure that the project is in line with the rest of the projects that together make an integrated release. This position needs to stretch beyond just Oslo to encompass all the cross-project requirements and will likely be its own committee - Gate Czar == integration engineer(manager)/QA engineer(manager)/build-release engineer. This position would also likely be a liaison to Infra. - Security Czar == security guru (that name takes me back ;-) - Release management Czar == Project release manager - Doc Czar == tech editor - Tempest Czar == QA engineer(manager) Yes, programs are now mostly big enough to require coordination and management. The roles are long defined, so
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
On Fri, Aug 22, 2014 at 6:17 PM, Rochelle.RochelleGrober rochelle.gro...@huawei.com wrote: /flame-on Ok, this is funny to some of us in the community. The general populace of this community is so against the idea of management that they will use the term for a despotic dictator as a position name rather than manager. Sorry, but this needed to be said. /flame-off Specific comments in line: Thierry Carrez wrote: Hi everyone, We all know being a project PTL is an extremely busy job. That's because in our structure the PTL is responsible for almost everything in a project: - Release management contact - Work prioritization - Keeping bugs under control - Communicate about work being planned or done - Make sure the gate is not broken - Team logistics (run meetings, organize sprints) - ... Point of clarification: I've heard PTL=Project Technical Lead and PTL=Program Technical Lead. Which is it? It is kind of important as OpenStack grows, because the first is responsible for *a* project, and the second is responsible for all projects within a program. Now Program, formerly Project. I'd also like to set out as an example of a Program that is growing to encompass multiple projects, the Neutron Program. Look at how it is expanding: Multiple sub-teams for: LBAAS, DNAAS, GBP, etc. This model could be extended such that: - the subteam is responsible for code reviews, including the first +2 for design, architecture and code of the sub-project, always also keeping an eye out that the sub-project code continues to both integrate well with the program, and that the program continues to provide the needed code bits, architecture modifications and improvements, etc. to support the sub-project. - the final +2/A would be from the Program reviewers to ensure that all integrate nicely together into a single, cohesive program. - This would allow sub-projects to have core reviewers, along with the program and be a good separation of duties. It would also help to increase the number of reviews moving to merged code. - Taken to a logical stepping stone, you would have project technical leads for each project, and they would make up a program council, with the program technical lead being the chair of the council. This is a way to offload a good chunk of PTL tactical responsibilities and help them focus more on the strategic. They end up being completely drowned in those day-to-day operational duties, miss the big picture, can't help in development that much anymore, get burnt out. Since you're either the PTL or not the PTL, you're very alone and succession planning is not working that great either. There have been a number of experiments to solve that problem. John Garbutt has done an incredible job at helping successive Nova PTLs handling the release management aspect. Tracy Jones took over Nova bug management. Doug Hellmann successfully introduced the concept of Oslo liaisons to get clear point of contacts for Oslo library adoption in projects. It may be time to generalize that solution. The issue is one of responsibility: the PTL is ultimately responsible for everything in a project. If we can more formally delegate that responsibility, we can avoid getting up to the PTL for everything, we can rely on a team of people rather than just one person. Enter the Czar system: each project should have a number of liaisons / official contacts / delegates that are fully responsible to cover one aspect of the project. We need to have Bugs czars, which are responsible for getting bugs under control. We need to have Oslo czars, which serve as liaisons for the Oslo program but also as active project-local oslo advocates. We need Security czars, which the VMT can go to to progress quickly on plugging vulnerabilities. We need release management czars, to handle the communication and process with that painful OpenStack release manager. We need Gate czars to serve as first-line-of-contact getting gate issues fixed... You get the idea. /flame-on Let's call spades, spades here. Czar is not only overkill, but the wrong metaphor. /flame-off I'm with Rocky on the anti-czar-as-a-word camp. We all like clever names to shed the corporate stigma but this word ain't it. Liaison or lead? Also wanted to point to https://wiki.openstack.org/wiki/PTLguide as it's quite nice. I think PTLs tend to find help when they absolutely are ready to fall over, and I'm all for a plan that helps us not fall over. I've had people step up for bug triaging, gate work, tests, and oslo, sometimes one person did three or four at once. I certainly can't do it all for cross-project. Based on what I've seen, I doubt that we can add this much formality to this across 20+ programs. It's the bigger more integrated project vs. smaller more focused project difference that won't let us do a pattern here. We can certainly document the
Re: [openstack-dev] [all] [ptls] The Czar system, or how to scale PTLs
I think Anne makes some excellent points about the pattern being proposed being unlikely to be commonly implemented across all the programs (or, at best, very difficult). Let's not try to formalize another best practice that works many times and force it to work every time. Here's an alternate proposal: Let's let PTLs be PTLs and effectively coordinate and manage the activity in their respective projects. And let's get the PTLs together for one or two days every cycle to discuss project issues. Just PTLs, and let's focus on the project management stuff and some cross-project issues. Getting the PTLs together would allow them to discuss cross-project issues, share frustrations and solutions about what does and doesn't work. Basically, think of it as a mid-cycle meetup, but for PTLs. (Perhaps we could even ask the Foundation to sponsor it.) --John On Aug 22, 2014, at 6:02 PM, Anne Gentle a...@openstack.org wrote: On Fri, Aug 22, 2014 at 6:17 PM, Rochelle.RochelleGrober rochelle.gro...@huawei.com wrote: /flame-on Ok, this is funny to some of us in the community. The general populace of this community is so against the idea of management that they will use the term for a despotic dictator as a position name rather than manager. Sorry, but this needed to be said. /flame-off Specific comments in line: Thierry Carrez wrote: Hi everyone, We all know being a project PTL is an extremely busy job. That's because in our structure the PTL is responsible for almost everything in a project: - Release management contact - Work prioritization - Keeping bugs under control - Communicate about work being planned or done - Make sure the gate is not broken - Team logistics (run meetings, organize sprints) - ... Point of clarification: I've heard PTL=Project Technical Lead and PTL=Program Technical Lead. Which is it? It is kind of important as OpenStack grows, because the first is responsible for *a* project, and the second is responsible for all projects within a program. Now Program, formerly Project. I'd also like to set out as an example of a Program that is growing to encompass multiple projects, the Neutron Program. Look at how it is expanding: Multiple sub-teams for: LBAAS, DNAAS, GBP, etc. This model could be extended such that: - the subteam is responsible for code reviews, including the first +2 for design, architecture and code of the sub-project, always also keeping an eye out that the sub-project code continues to both integrate well with the program, and that the program continues to provide the needed code bits, architecture modifications and improvements, etc. to support the sub-project. - the final +2/A would be from the Program reviewers to ensure that all integrate nicely together into a single, cohesive program. - This would allow sub-projects to have core reviewers, along with the program and be a good separation of duties. It would also help to increase the number of reviews moving to merged code. - Taken to a logical stepping stone, you would have project technical leads for each project, and they would make up a program council, with the program technical lead being the chair of the council. This is a way to offload a good chunk of PTL tactical responsibilities and help them focus more on the strategic. They end up being completely drowned in those day-to-day operational duties, miss the big picture, can't help in development that much anymore, get burnt out. Since you're either the PTL or not the PTL, you're very alone and succession planning is not working that great either. There have been a number of experiments to solve that problem. John Garbutt has done an incredible job at helping successive Nova PTLs handling the release management aspect. Tracy Jones took over Nova bug management. Doug Hellmann successfully introduced the concept of Oslo liaisons to get clear point of contacts for Oslo library adoption in projects. It may be time to generalize that solution. The issue is one of responsibility: the PTL is ultimately responsible for everything in a project. If we can more formally delegate that responsibility, we can avoid getting up to the PTL for everything, we can rely on a team of people rather than just one person. Enter the Czar system: each project should have a number of liaisons / official contacts / delegates that are fully responsible to cover one aspect of the project. We need to have Bugs czars, which are responsible for getting bugs under control. We need to have Oslo czars, which serve as liaisons for the Oslo program but also as active project-local oslo advocates. We need Security czars, which the VMT can go to to progress quickly on plugging vulnerabilities. We need release management czars, to handle the communication and process with that painful OpenStack release manager. We need Gate
