On Aug 12, 2014, at 11:08 AM, Doug Hellmann <d...@doughellmann.com> wrote:

> On Aug 12, 2014, at 1:44 PM, Dolph Mathews <dolph.math...@gmail.com> wrote:
>> On Tue, Aug 12, 2014 at 12:30 AM, Joe Gordon <joe.gord...@gmail.com> wrote:
>> On Fri, Aug 8, 2014 at 6:58 AM, Kyle Mestery <mest...@mestery.com> wrote:
>> On Thu, Aug 7, 2014 at 1:26 PM, Joe Gordon <joe.gord...@gmail.com> wrote:
>> >
>> >
>> >
>> > On Tue, Aug 5, 2014 at 9:03 AM, Thierry Carrez <thie...@openstack.org>
>> > wrote:
>> >>
>> >> Hi everyone,
>> >>
>> >> With the incredible growth of OpenStack, our development community is
>> >> facing complex challenges. How we handle those might determine the
>> >> ultimate success or failure of OpenStack.
>> >>
>> >> With this cycle we hit new limits in our processes, tools and cultural
>> >> setup. This resulted in new limiting factors on our overall velocity,
>> >> which is frustrating for developers. This resulted in the burnout of key
>> >> firefighting resources. This resulted in tension between people who try
>> >> to get specific work done and people who try to keep a handle on the big
>> >> picture.
>> >>
>> >> It all boils down to an imbalance between strategic and tactical
>> >> contributions. At the beginning of this project, we had a strong inner
>> >> group of people dedicated to fixing all loose ends. Then a lot of
>> >> companies got interested in OpenStack and there was a surge in tactical,
>> >> short-term contributions. We put on a call for more resources to be
>> >> dedicated to strategic contributions like critical bugfixing,
>> >> vulnerability management, QA, infrastructure... and that call was
>> >> answered by a lot of companies that are now key members of the OpenStack
>> >> Foundation, and all was fine again. But OpenStack contributors kept on
>> >> growing, and we grew the narrowly-focused population way faster than the
>> >> cross-project population.
>> >>
>> >>
>> >> At the same time, we kept on adding new projects to incubation and to
>> >> the integrated release, which is great... but the new developers you get
>> >> on board with this are much more likely to be tactical than strategic
>> >> contributors. This also contributed to the imbalance. The penalty for
>> >> that imbalance is twofold: we don't have enough resources available to
>> >> solve old, known OpenStack-wide issues; but we also don't have enough
>> >> resources to identify and fix new issues.
>> >>
>> >> We have several efforts under way, like calling for new strategic
>> >> contributors, driving towards in-project functional testing, making
>> >> solving rare issues a more attractive endeavor, or hiring resources
>> >> directly at the Foundation level to help address those. But there is a
>> >> topic we haven't raised yet: should we concentrate on fixing what is
>> >> currently in the integrated release rather than adding new projects ?
>> >
>> >
>> > TL;DR: Our development model is having growing pains. until we sort out the
>> > growing pains adding more projects spreads us too thin.
>> >
>> +100
>> > In addition to the issues mentioned above, with the scale of OpenStack 
>> > today
>> > we have many major cross project issues to address and no good place to
>> > discuss them.
>> >
>> We do have the ML, as well as the cross-project meeting every Tuesday
>> [1], but we as a project need to do a better job of actually bringing
>> up relevant issues here.
>> [1] https://wiki.openstack.org/wiki/Meetings/ProjectMeeting
>> >>
>> >>
>> >> We seem to be unable to address some key issues in the software we
>> >> produce, and part of it is due to strategic contributors (and core
>> >> reviewers) being overwhelmed just trying to stay afloat of what's
>> >> happening. For such projects, is it time for a pause ? Is it time to
>> >> define key cycle goals and defer everything else ?
>> >
>> >
>> >
>> > I really like this idea, as Michael and others alluded to in above, we are
>> > attempting to set cycle goals for Kilo in Nova. but I think it is worth
>> > doing for all of OpenStack. We would like to make a list of key goals 
>> > before
>> > the summit so that we can plan our summit sessions around the goals. On a
>> > really high level one way to look at this is, in Kilo we need to pay down
>> > our technical debt.
>> >
>> > The slots/runway idea is somewhat separate from defining key cycle goals; 
>> > we
>> > can be approve blueprints based on key cycle goals without doing slots.  
>> > But
>> > with so many concurrent blueprints up for review at any given time, the
>> > review teams are doing a lot of multitasking and humans are not very good 
>> > at
>> > multitasking. Hopefully slots can help address this issue, and hopefully
>> > allow us to actually merge more blueprints in a given cycle.
>> >
>> I'm not 100% sold on what the slots idea buys us. What I've seen this
>> cycle in Neutron is that we have a LOT of BPs proposed. We approve
>> them after review. And then we hit one of two issues: Slow review
>> cycles, and slow code turnaround issues. I don't think slots would
>> help this, and in fact may cause more issues. If we approve a BP and
>> give it a slot for which the eventual result is slow review and/or
>> code review turnaround, we're right back where we started. Even worse,
>> we may have not picked a BP for which the code submitter would have
>> turned around reviews faster. So we've now doubly hurt ourselves. I
>> have no idea how to solve this issue, but by over subscribing the
>> slots (e.g. over approving), we allow for the submissions with faster
>> turnaround a chance to merge quicker. With slots, we've removed this
>> capability by limiting what is even allowed to be considered for
>> review.
>> Slow review: by limiting the number of blueprints up we hope to focus our 
>> efforts on fewer concurrent things
>> slow code turn around: when a blueprint is given a slot (runway) we will 
>> first make sure the author/owner is available for fast code turnaround.
>> If a blueprint review stalls out (slow code turnaround, stalemate in review 
>> discussions etc.) we will take the slot and give it to another blueprint.
>> How is that more efficient than today's do-the-best-we-can approach? It just 
>> sounds like bureaucracy to me.
>> Reading between the lines throughout this thread, it sounds like what we're 
>> lacking is a reliable method to communicate review prioritization to core 
>> reviewers.
> It seems like this is exactly what the slots give us, though. The core review 
> team picks a number of slots indicating how much work they think they can 
> actually do (less than the available number of blueprints), and then 
> blueprints queue up to get a slot based on priorities and turnaround time and 
> other criteria that try to make slot allocation fair. By having the slots, 
> not only is the review priority communicated to the review team, it is also 
> communicated to anyone watching the project.
> Doug

FWIW, the way we've tracked and coordinated this within Swift is with a wiki 
page. Nothing fancy. I, as the PTL, keep an eye on proposed patches, bugs, etc 
and keep the wiki page up-to-date. Reviewers can look at that page to see what 
they should tackle first. This has worked well for us for quite a while now, 
and it was particularly useful while we were getting storage policies finished 



Attachment: signature.asc
Description: Message signed with OpenPGP using GPGMail

OpenStack-dev mailing list

Reply via email to