On Fri, Aug 8, 2014 at 6:58 AM, Kyle Mestery <mest...@mestery.com> wrote:
> On Thu, Aug 7, 2014 at 1:26 PM, Joe Gordon <joe.gord...@gmail.com> wrote: > > > > > > > > On Tue, Aug 5, 2014 at 9:03 AM, Thierry Carrez <thie...@openstack.org> > > wrote: > >> > >> Hi everyone, > >> > >> With the incredible growth of OpenStack, our development community is > >> facing complex challenges. How we handle those might determine the > >> ultimate success or failure of OpenStack. > >> > >> With this cycle we hit new limits in our processes, tools and cultural > >> setup. This resulted in new limiting factors on our overall velocity, > >> which is frustrating for developers. This resulted in the burnout of key > >> firefighting resources. This resulted in tension between people who try > >> to get specific work done and people who try to keep a handle on the big > >> picture. > >> > >> It all boils down to an imbalance between strategic and tactical > >> contributions. At the beginning of this project, we had a strong inner > >> group of people dedicated to fixing all loose ends. Then a lot of > >> companies got interested in OpenStack and there was a surge in tactical, > >> short-term contributions. We put on a call for more resources to be > >> dedicated to strategic contributions like critical bugfixing, > >> vulnerability management, QA, infrastructure... and that call was > >> answered by a lot of companies that are now key members of the OpenStack > >> Foundation, and all was fine again. But OpenStack contributors kept on > >> growing, and we grew the narrowly-focused population way faster than the > >> cross-project population. > >> > >> > >> At the same time, we kept on adding new projects to incubation and to > >> the integrated release, which is great... but the new developers you get > >> on board with this are much more likely to be tactical than strategic > >> contributors. This also contributed to the imbalance. The penalty for > >> that imbalance is twofold: we don't have enough resources available to > >> solve old, known OpenStack-wide issues; but we also don't have enough > >> resources to identify and fix new issues. > >> > >> We have several efforts under way, like calling for new strategic > >> contributors, driving towards in-project functional testing, making > >> solving rare issues a more attractive endeavor, or hiring resources > >> directly at the Foundation level to help address those. But there is a > >> topic we haven't raised yet: should we concentrate on fixing what is > >> currently in the integrated release rather than adding new projects ? > > > > > > TL;DR: Our development model is having growing pains. until we sort out > the > > growing pains adding more projects spreads us too thin. > > > +100 > > > In addition to the issues mentioned above, with the scale of OpenStack > today > > we have many major cross project issues to address and no good place to > > discuss them. > > > We do have the ML, as well as the cross-project meeting every Tuesday > [1], but we as a project need to do a better job of actually bringing > up relevant issues here. > > [1] https://wiki.openstack.org/wiki/Meetings/ProjectMeeting > > >> > >> > >> We seem to be unable to address some key issues in the software we > >> produce, and part of it is due to strategic contributors (and core > >> reviewers) being overwhelmed just trying to stay afloat of what's > >> happening. For such projects, is it time for a pause ? Is it time to > >> define key cycle goals and defer everything else ? > > > > > > > > I really like this idea, as Michael and others alluded to in above, we > are > > attempting to set cycle goals for Kilo in Nova. but I think it is worth > > doing for all of OpenStack. We would like to make a list of key goals > before > > the summit so that we can plan our summit sessions around the goals. On a > > really high level one way to look at this is, in Kilo we need to pay down > > our technical debt. > > > > The slots/runway idea is somewhat separate from defining key cycle > goals; we > > can be approve blueprints based on key cycle goals without doing slots. > But > > with so many concurrent blueprints up for review at any given time, the > > review teams are doing a lot of multitasking and humans are not very > good at > > multitasking. Hopefully slots can help address this issue, and hopefully > > allow us to actually merge more blueprints in a given cycle. > > > I'm not 100% sold on what the slots idea buys us. What I've seen this > cycle in Neutron is that we have a LOT of BPs proposed. We approve > them after review. And then we hit one of two issues: Slow review > cycles, and slow code turnaround issues. I don't think slots would > help this, and in fact may cause more issues. If we approve a BP and > give it a slot for which the eventual result is slow review and/or > code review turnaround, we're right back where we started. Even worse, > we may have not picked a BP for which the code submitter would have > turned around reviews faster. So we've now doubly hurt ourselves. I > have no idea how to solve this issue, but by over subscribing the > slots (e.g. over approving), we allow for the submissions with faster > turnaround a chance to merge quicker. With slots, we've removed this > capability by limiting what is even allowed to be considered for > review. > Slow review: by limiting the number of blueprints up we hope to focus our efforts on fewer concurrent things slow code turn around: when a blueprint is given a slot (runway) we will first make sure the author/owner is available for fast code turnaround. If a blueprint review stalls out (slow code turnaround, stalemate in review discussions etc.) we will take the slot and give it to another blueprint. > > Thanks, > Kyle > > >> > >> > >> On the integrated release side, "more projects" means stretching our > >> limited strategic resources more. Is it time for the Technical Committee > >> to more aggressively define what is "in" and what is "out" ? If we go > >> through such a redefinition, shall we push currently-integrated projects > >> that fail to match that definition out of the "integrated release" inner > >> circle ? > >> > >> The TC discussion on what the integrated release should or should not > >> include has always been informally going on. Some people would like to > >> strictly limit to end-user-facing projects. Some others suggest that > >> "OpenStack" should just be about integrating/exposing/scaling smart > >> functionality that lives in specialized external projects, rather than > >> trying to outsmart those by writing our own implementation. Some others > >> are advocates of carefully moving up the stack, and to resist from > >> further addressing IaaS+ services until we "complete" the pure IaaS > >> space in a satisfactory manner. Some others would like to build a > >> roadmap based on AWS services. Some others would just add anything that > >> fits the incubation/integration requirements. > >> > >> > >> On one side this is a long-term discussion, but on the other we also > >> need to make quick decisions. With 4 incubated projects, and 2 new ones > >> currently being proposed, there are a lot of people knocking at the > door. > > > > > > > > I have a slightly different short term opinion on what 'OpenStack' should > > be: it should work really well. > > > > While we need to figure out howto increase our strategic resources, > > realistically I think that will still be the limiting factor in Kilo. So > in > > order to better allocate our cross project strategic resources, I think > we > > should do a project level triage ('identify projects that are likely to > die, > > regardless of what care they receive' to paraphrase the medical > definition > > of the term), and tighten our focus until we get our development process > > into a better state. > > > > > >> > >> > >> Thanks for reading this braindump this far. I hope this will trigger the > >> open discussions we need to have, as an open source project, to reach > >> the next level. > > > > > > Thank you for bringing this topic up. > > > >> > >> > >> Cheers, > >> > >> -- > >> Thierry Carrez (ttx) > >> > >> _______________________________________________ > >> OpenStack-dev mailing list > >> OpenStack-dev@lists.openstack.org > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > > > > _______________________________________________ > > OpenStack-dev mailing list > > OpenStack-dev@lists.openstack.org > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev