Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[James Bottomley]

On Wed, 2015-03-04 at 11:19 +0100, Thierry Carrez wrote:
> James Bottomley wrote:
> > On Tue, 2015-03-03 at 11:59 +0100, Thierry Carrez wrote:
> >> Second it's at a very different evolution/maturity point (20 years old
> >> vs. 0-4 years old for OpenStack projects).
> > 
> > Yes, but I thought I covered this in the email: you can see that at the
> > 4 year point in its lifecycle, the kernel was behaving very differently
> > (and in fact more similar to OpenStack).  The question I thought was
> > still valid is whether anything was learnable from the way the kernel
> > evolved later.  I think the key issue, which you seem to have in
> > OpenStack is that the separate develop/stabilise phases caused
> > frustration to build up in our system which (nine years later) led the
> > kernel to adopt the main branch stabilisation with overlapping subsystem
> > development cycle.
> 
> I agree with you: the evolution the kernel went through is almost a
> natural law, and I know we won't stay in the current model forever. I'm
> just not sure we have reached the level of general stability that makes
> it possible to change *just now*. I welcome brainstorming and discussion
> on future evolutions, though, and intend to lead a cross-project session
> discussion on that in Vancouver.

OK, I'll be in Vancouver, so happy to come and give input from
participating in the kernel process (for a bit longer than I care to
admit to ...).

One interesting thing might be to try and work out where roughly
OpenStack is on the project trajectory.  It's progressing much more
rapidly than the kernel (by 4 years in, the kernel didn't even have
source control), so the release crisis it took the kernel 12 years to
reach might be a bit closer than people imagine.

James


James



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Steve Gordon]

- Original Message -
> From: "Thierry Carrez" 
> To: "James Bottomley" 
 
> > It's certainly a lot less than you, but we have the entire system
> > call
> > man pages.  It's an official project of the kernel:
> > 
> > https://www.kernel.org/doc/man-pages/
> > 
> > And we maintain translations for it
> > 
> > https://www.kernel.org/doc/man-pages/translations.html
> 
> By translations I meant strings in the software itself, not doc
> translations. We don't translate docs upstream either :) I guess we
> could drop those (and/or downstream them in a way) if that was the
> last
> thing holding up adding more agility.

There is actually a group of contributors working on translation of 
documentation and translations in various stages of completeness are available 
at docs.openstack.org (hit the more releases and languages drop down to find 
them). The challenge for the translators of course is they are trying to hit a 
target that is moving not just until the release but often well beyond it as 
the documentation team themselves try to catch up.

-Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Clint Byrum]

Excerpts from Thierry Carrez's message of 2015-03-04 02:19:48 -0800:
> James Bottomley wrote:
> > On Tue, 2015-03-03 at 11:59 +0100, Thierry Carrez wrote:
> >> James Bottomley wrote:
> >>> Actually, this is possible: look at Linux, it freezes for 10 weeks of a
> >>> 12 month release cycle (or 6 weeks of an 8 week one).  More on this
> >>> below.
> >>
> >> I'd be careful with comparisons with the Linux kernel. First it's a
> >> single bit of software, not a collection of interconnected projects.
> > 
> > Well, we do have interconnection: the kernel on it's own doesn't do
> > anything without a userspace.  The theory was that we didn't have to be
> > like BSD (coupled user space and kernel) and we could rely on others
> > (principally the GNU project in the early days) to provide the userspace
> > and that we could decouple kernel development from the userspace
> > releases.  Threading models were, I think, the biggest challenges to
> > this assumption, but we survived.
> 
> Right. My point there is that you only release one thing. We release a
> lot more pieces. There is (was?) downstream value in coordinating those
> releases, which is a factor in our ability to do it more often than
> twice a year.
> 

I think the value of coordinated releases has been agreed upon for a
long time. This thread is more about the cost, don't you think?

> >> Second it's at a very different evolution/maturity point (20 years old
> >> vs. 0-4 years old for OpenStack projects).
> > 
> > Yes, but I thought I covered this in the email: you can see that at the
> > 4 year point in its lifecycle, the kernel was behaving very differently
> > (and in fact more similar to OpenStack).  The question I thought was
> > still valid is whether anything was learnable from the way the kernel
> > evolved later.  I think the key issue, which you seem to have in
> > OpenStack is that the separate develop/stabilise phases caused
> > frustration to build up in our system which (nine years later) led the
> > kernel to adopt the main branch stabilisation with overlapping subsystem
> > development cycle.
> 
> I agree with you: the evolution the kernel went through is almost a
> natural law, and I know we won't stay in the current model forever. I'm
> just not sure we have reached the level of general stability that makes
> it possible to change *just now*. I welcome brainstorming and discussion
> on future evolutions, though, and intend to lead a cross-project session
> discussion on that in Vancouver.
> 

I don't believe that the kernel reached maturity as a point of
eventuality. Just like humans aren't going to jump across the grand
canyon no matter how strong they get, it will take a concerted effort
that may put other goals on hold to build a bridge. With the kernel
there was a clear moment where leadership had tried a few things and
then just had to make it clear that all the code goes in one place, but
instability would not be tolerated. They crossed that chasm, and while
there have been chaotic branches and ruffled feathers, once everybody
got over the paradox, it's been business as usual since then with the
model James describes.

I think the less mature a project is, the wider that chasm is, but I
don't think it's ever going to be an easy thing to do. Since we don't
have a dictator to force us to cross the chasm, we should really think
about planning for the crossing ASAP.

> >>  Finally it sits at a
> >> different layer, so there is less need for documentation/translations to
> >> be shipped with the software release.
> > 
> > It's certainly a lot less than you, but we have the entire system call
> > man pages.  It's an official project of the kernel:
> > 
> > https://www.kernel.org/doc/man-pages/
> > 
> > And we maintain translations for it
> > 
> > https://www.kernel.org/doc/man-pages/translations.html
> 
> By translations I meant strings in the software itself, not doc
> translations. We don't translate docs upstream either :) I guess we
> could drop those (and/or downstream them in a way) if that was the last
> thing holding up adding more agility.
> 
> So in summary, yes we can (and do) learn from kernel history, but those
> projects are sufficiently different that the precise timeframes and
> numbers can't really be compared. Apples and oranges are both fruits
> which mature (and rot if left unchecked), but they evolve at different
> speeds :)
> 

I'm not super excited about being an apple or an orange, since neither
are sentient and thus cannot collaborate on a better existence than
rotting.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Thierry Carrez]

James Bottomley wrote:
> On Tue, 2015-03-03 at 11:59 +0100, Thierry Carrez wrote:
>> James Bottomley wrote:
>>> Actually, this is possible: look at Linux, it freezes for 10 weeks of a
>>> 12 month release cycle (or 6 weeks of an 8 week one).  More on this
>>> below.
>>
>> I'd be careful with comparisons with the Linux kernel. First it's a
>> single bit of software, not a collection of interconnected projects.
> 
> Well, we do have interconnection: the kernel on it's own doesn't do
> anything without a userspace.  The theory was that we didn't have to be
> like BSD (coupled user space and kernel) and we could rely on others
> (principally the GNU project in the early days) to provide the userspace
> and that we could decouple kernel development from the userspace
> releases.  Threading models were, I think, the biggest challenges to
> this assumption, but we survived.

Right. My point there is that you only release one thing. We release a
lot more pieces. There is (was?) downstream value in coordinating those
releases, which is a factor in our ability to do it more often than
twice a year.

>> Second it's at a very different evolution/maturity point (20 years old
>> vs. 0-4 years old for OpenStack projects).
> 
> Yes, but I thought I covered this in the email: you can see that at the
> 4 year point in its lifecycle, the kernel was behaving very differently
> (and in fact more similar to OpenStack).  The question I thought was
> still valid is whether anything was learnable from the way the kernel
> evolved later.  I think the key issue, which you seem to have in
> OpenStack is that the separate develop/stabilise phases caused
> frustration to build up in our system which (nine years later) led the
> kernel to adopt the main branch stabilisation with overlapping subsystem
> development cycle.

I agree with you: the evolution the kernel went through is almost a
natural law, and I know we won't stay in the current model forever. I'm
just not sure we have reached the level of general stability that makes
it possible to change *just now*. I welcome brainstorming and discussion
on future evolutions, though, and intend to lead a cross-project session
discussion on that in Vancouver.

>>  Finally it sits at a
>> different layer, so there is less need for documentation/translations to
>> be shipped with the software release.
> 
> It's certainly a lot less than you, but we have the entire system call
> man pages.  It's an official project of the kernel:
> 
> https://www.kernel.org/doc/man-pages/
> 
> And we maintain translations for it
> 
> https://www.kernel.org/doc/man-pages/translations.html

By translations I meant strings in the software itself, not doc
translations. We don't translate docs upstream either :) I guess we
could drop those (and/or downstream them in a way) if that was the last
thing holding up adding more agility.

So in summary, yes we can (and do) learn from kernel history, but those
projects are sufficiently different that the precise timeframes and
numbers can't really be compared. Apples and oranges are both fruits
which mature (and rot if left unchecked), but they evolve at different
speeds :)

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[James Bottomley]

On Tue, 2015-03-03 at 11:59 +0100, Thierry Carrez wrote:
> James Bottomley wrote:
> > Actually, this is possible: look at Linux, it freezes for 10 weeks of a
> > 12 month release cycle (or 6 weeks of an 8 week one).  More on this
> > below.
> 
> I'd be careful with comparisons with the Linux kernel. First it's a
> single bit of software, not a collection of interconnected projects.

Well, we do have interconnection: the kernel on it's own doesn't do
anything without a userspace.  The theory was that we didn't have to be
like BSD (coupled user space and kernel) and we could rely on others
(principally the GNU project in the early days) to provide the userspace
and that we could decouple kernel development from the userspace
releases.  Threading models were, I think, the biggest challenges to
this assumption, but we survived.

> Second it's at a very different evolution/maturity point (20 years old
> vs. 0-4 years old for OpenStack projects).

Yes, but I thought I covered this in the email: you can see that at the
4 year point in its lifecycle, the kernel was behaving very differently
(and in fact more similar to OpenStack).  The question I thought was
still valid is whether anything was learnable from the way the kernel
evolved later.  I think the key issue, which you seem to have in
OpenStack is that the separate develop/stabilise phases caused
frustration to build up in our system which (nine years later) led the
kernel to adopt the main branch stabilisation with overlapping subsystem
development cycle.

>  Finally it sits at a
> different layer, so there is less need for documentation/translations to
> be shipped with the software release.

It's certainly a lot less than you, but we have the entire system call
man pages.  It's an official project of the kernel:

https://www.kernel.org/doc/man-pages/

And we maintain translations for it

https://www.kernel.org/doc/man-pages/translations.html

The main difference is that our translation projects are not integrated
into our releases.  They're done mostly by the downstream (debian if you
look: the rest of the distros do invest in translations but they don't
make the upstream effort with them), so we don't specify what
translations we have, we allow interested parties to choose to help.

I would argue that the downstream is the best place to manage
translations because they decide which markets are important or
interesting, but if you still want to control this at the upstream end,
with Daniel's proposal, it would mean you only really need translations
for stable releases.

James

> The only comparable project in terms of evolution/maturity in the
> OpenStack world would be Swift, and it happily produces releases every
> ~2months with a 1-week stabilisation period.




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Chris Dent]

On Tue, 3 Mar 2015, Thierry Carrez wrote:


I'd be careful with comparisons with the Linux kernel. First it's a
single bit of software, not a collection of interconnected projects.
Second it's at a very different evolution/maturity point (20 years old
vs. 0-4 years old for OpenStack projects). Finally it sits at a
different layer, so there is less need for documentation/translations to
be shipped with the software release.


I think this is probably trampled ground, but is there some history on
why the OpenStack projects chose to own documentation, translation and
the concept of "shipping the release" instead of ceding that to the
"downstream"?

I'm not suggesting it should be (or should have been) ceded, just
wondering how it ended up that way.

There seems to be energy oriented toward centralization, marketing,
consolidation, arrogation etc. that might otherwise be spent on making
code that can later be refined by the people who want to make
money with it.

At the moment there is a strong coupling between the projects and the
product. Is that the best arrangement or are there others that might
be worth considering?

--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Thierry Carrez]

James Bottomley wrote:
> Actually, this is possible: look at Linux, it freezes for 10 weeks of a
> 12 month release cycle (or 6 weeks of an 8 week one).  More on this
> below.

I'd be careful with comparisons with the Linux kernel. First it's a
single bit of software, not a collection of interconnected projects.
Second it's at a very different evolution/maturity point (20 years old
vs. 0-4 years old for OpenStack projects). Finally it sits at a
different layer, so there is less need for documentation/translations to
be shipped with the software release.

The only comparable project in terms of evolution/maturity in the
OpenStack world would be Swift, and it happily produces releases every
~2months with a 1-week stabilisation period.

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Clint Byrum]

Excerpts from Angus Salkeld's message of 2015-03-02 17:08:15 -0800:
> On Tue, Mar 3, 2015 at 9:45 AM, James Bottomley <
> james.bottom...@hansenpartnership.com> wrote:
> 
> > On Tue, 2015-02-24 at 12:05 +0100, Thierry Carrez wrote:
> > > Daniel P. Berrange wrote:
> > > > [...]
> > > > The key observations
> > > > 
> > > >
> > > > The first key observation from the schedule is that although we have
> > > > a 6 month release cycle, we in fact make 4 releases in that six
> > > > months because there are 3 milestones releases approx 6-7 weeks apart
> > > > from each other, in addition to the final release. So one of the key
> > > > burdens of a more frequent release cycle is already being felt, to
> > > > some degree.
> > > >
> > > > The second observation is that thanks to the need to support a
> > > > continuous deployment models, the GIT master branches are generally
> > > > considered to be production ready at all times. The tree does not
> > > > typically go through periods of major instability that can be seen
> > > > in other projects, particular those which lack such comprehensive
> > > > testing infrastructure.
> > > >
> > > > The third observation is that due to the relatively long cycle, and
> > > > increasing amounts of process, the work accomplished during the
> > > > cycles is becoming increasingly bursty. This is in turn causing
> > > > unacceptably long delays for contributors when their work is unlucky
> > > > enough to not get accepted during certain critical short windows of
> > > > opportunity in the cycle.
> > > >
> > > > The first two observations strongly suggest that the choice of 6
> > > > months as a cycle length is a fairly arbitrary decision that can be
> > > > changed without unreasonable pain. The third observation suggests a
> > > > much shorter cycle length would smooth out the bumps and lead to a
> > > > more efficient & satisfying development process for all involved.
> > >
> > > I think you're judging the cycle from the perspective of developers
> > > only. 6 months was not an arbitrary decision. Translations and
> > > documentation teams basically need a month of feature/string freeze in
> > > order to complete their work. Since we can't reasonably freeze one month
> > > every 2 months, we picked 6 months.
> >
> > Actually, this is possible: look at Linux, it freezes for 10 weeks of a
> > 12 month release cycle (or 6 weeks of an 8 week one).  More on this
> > below.
> >
> > > It's also worth noting that we were on a 3-month cycle at the start of
> > > OpenStack. That was dropped after a cataclysmic release that managed the
> > > feat of (a) not having anything significant done, and (b) have out of
> > > date documentation and translations.
> > >
> > > While I agree that the packagers and stable teams can opt to skip a
> > > release, the docs, translations or security teams don't really have that
> > > luxury... Please go beyond the developers needs and consider the needs
> > > of the other teams.
> > >
> > > Random other comments below:
> > >
> > > > [...]
> > > > Release schedule
> > > > 
> > > >
> > > > First the releases would probably be best attached to a set of
> > > > pre-determined fixed dates that don't ever vary from year to year.
> > > > eg releses happen Feb 1st, Apr 1st, Jun 1st, Aug 1st, Oct 1st, and
> > > > Dec 1st. If a particular release slips, don't alter following release
> > > > dates, just shorten the length of the dev cycle, so it becomes fully
> > > > self-correcting. The even numbered months are suggested to avoid a
> > > > release landing in xmas/new year :-)
> > >
> > > The Feb 1 release would probably be pretty empty :)
> > >
> > > > [...]
> > > > Stable branches
> > > > ---
> > > >
> > > > The consequences of a 2 month release cycle appear fairly severe for
> > > > the stable branch maint teams at first sight. This is not, however,
> > > > an insurmountable problem. The linux kernel shows an easy way forward
> > > > with their approach of only maintaining stable branches for a subset
> > > > of major releases, based around user / vendor demand. So it is still
> > > > entirely conceivable that the stable team only provide stable branch
> > > > releases for 2 out of the 6 yearly releases. ie no additional burden
> > > > over what they face today. Of course they might decide they want to
> > > > do more stable branches, but maintain each for a shorter time. So I
> > > > could equally see them choosing todo 3 or 4 stable branches a year.
> > > > Whatever is most effective for those involved and those consuming
> > > > them is fine.
> > >
> > > Stable branches may have the luxury of skipping releases and designate a
> > > "stable" one from time to time (I reject the Linux comparison because
> > > the kernel is at a very different moment in software lifecycle). The
> > > trick being, making one release "special" is sure to recreate the peak
> > > issues you're trying to solve.
> >
> > I don't disagree with

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Angus Salkeld]

On Tue, Mar 3, 2015 at 9:45 AM, James Bottomley <
james.bottom...@hansenpartnership.com> wrote:

> On Tue, 2015-02-24 at 12:05 +0100, Thierry Carrez wrote:
> > Daniel P. Berrange wrote:
> > > [...]
> > > The key observations
> > > 
> > >
> > > The first key observation from the schedule is that although we have
> > > a 6 month release cycle, we in fact make 4 releases in that six
> > > months because there are 3 milestones releases approx 6-7 weeks apart
> > > from each other, in addition to the final release. So one of the key
> > > burdens of a more frequent release cycle is already being felt, to
> > > some degree.
> > >
> > > The second observation is that thanks to the need to support a
> > > continuous deployment models, the GIT master branches are generally
> > > considered to be production ready at all times. The tree does not
> > > typically go through periods of major instability that can be seen
> > > in other projects, particular those which lack such comprehensive
> > > testing infrastructure.
> > >
> > > The third observation is that due to the relatively long cycle, and
> > > increasing amounts of process, the work accomplished during the
> > > cycles is becoming increasingly bursty. This is in turn causing
> > > unacceptably long delays for contributors when their work is unlucky
> > > enough to not get accepted during certain critical short windows of
> > > opportunity in the cycle.
> > >
> > > The first two observations strongly suggest that the choice of 6
> > > months as a cycle length is a fairly arbitrary decision that can be
> > > changed without unreasonable pain. The third observation suggests a
> > > much shorter cycle length would smooth out the bumps and lead to a
> > > more efficient & satisfying development process for all involved.
> >
> > I think you're judging the cycle from the perspective of developers
> > only. 6 months was not an arbitrary decision. Translations and
> > documentation teams basically need a month of feature/string freeze in
> > order to complete their work. Since we can't reasonably freeze one month
> > every 2 months, we picked 6 months.
>
> Actually, this is possible: look at Linux, it freezes for 10 weeks of a
> 12 month release cycle (or 6 weeks of an 8 week one).  More on this
> below.
>
> > It's also worth noting that we were on a 3-month cycle at the start of
> > OpenStack. That was dropped after a cataclysmic release that managed the
> > feat of (a) not having anything significant done, and (b) have out of
> > date documentation and translations.
> >
> > While I agree that the packagers and stable teams can opt to skip a
> > release, the docs, translations or security teams don't really have that
> > luxury... Please go beyond the developers needs and consider the needs
> > of the other teams.
> >
> > Random other comments below:
> >
> > > [...]
> > > Release schedule
> > > 
> > >
> > > First the releases would probably be best attached to a set of
> > > pre-determined fixed dates that don't ever vary from year to year.
> > > eg releses happen Feb 1st, Apr 1st, Jun 1st, Aug 1st, Oct 1st, and
> > > Dec 1st. If a particular release slips, don't alter following release
> > > dates, just shorten the length of the dev cycle, so it becomes fully
> > > self-correcting. The even numbered months are suggested to avoid a
> > > release landing in xmas/new year :-)
> >
> > The Feb 1 release would probably be pretty empty :)
> >
> > > [...]
> > > Stable branches
> > > ---
> > >
> > > The consequences of a 2 month release cycle appear fairly severe for
> > > the stable branch maint teams at first sight. This is not, however,
> > > an insurmountable problem. The linux kernel shows an easy way forward
> > > with their approach of only maintaining stable branches for a subset
> > > of major releases, based around user / vendor demand. So it is still
> > > entirely conceivable that the stable team only provide stable branch
> > > releases for 2 out of the 6 yearly releases. ie no additional burden
> > > over what they face today. Of course they might decide they want to
> > > do more stable branches, but maintain each for a shorter time. So I
> > > could equally see them choosing todo 3 or 4 stable branches a year.
> > > Whatever is most effective for those involved and those consuming
> > > them is fine.
> >
> > Stable branches may have the luxury of skipping releases and designate a
> > "stable" one from time to time (I reject the Linux comparison because
> > the kernel is at a very different moment in software lifecycle). The
> > trick being, making one release "special" is sure to recreate the peak
> > issues you're trying to solve.
>
> I don't disagree with the observation about different points in the
> lifecycle, but perhaps it might be instructive to ask if the linux
> kernel ever had a period in its development history that looks somewhat
> like OpenStack does now.  I would claim it did: before 2.6, we had t

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[James Bottomley]

On Tue, 2015-02-24 at 12:05 +0100, Thierry Carrez wrote:
> Daniel P. Berrange wrote:
> > [...]
> > The key observations
> > 
> > 
> > The first key observation from the schedule is that although we have
> > a 6 month release cycle, we in fact make 4 releases in that six
> > months because there are 3 milestones releases approx 6-7 weeks apart
> > from each other, in addition to the final release. So one of the key
> > burdens of a more frequent release cycle is already being felt, to
> > some degree.
> > 
> > The second observation is that thanks to the need to support a
> > continuous deployment models, the GIT master branches are generally
> > considered to be production ready at all times. The tree does not
> > typically go through periods of major instability that can be seen
> > in other projects, particular those which lack such comprehensive
> > testing infrastructure.
> > 
> > The third observation is that due to the relatively long cycle, and
> > increasing amounts of process, the work accomplished during the
> > cycles is becoming increasingly bursty. This is in turn causing
> > unacceptably long delays for contributors when their work is unlucky
> > enough to not get accepted during certain critical short windows of
> > opportunity in the cycle.
> > 
> > The first two observations strongly suggest that the choice of 6
> > months as a cycle length is a fairly arbitrary decision that can be
> > changed without unreasonable pain. The third observation suggests a
> > much shorter cycle length would smooth out the bumps and lead to a
> > more efficient & satisfying development process for all involved.
> 
> I think you're judging the cycle from the perspective of developers
> only. 6 months was not an arbitrary decision. Translations and
> documentation teams basically need a month of feature/string freeze in
> order to complete their work. Since we can't reasonably freeze one month
> every 2 months, we picked 6 months.

Actually, this is possible: look at Linux, it freezes for 10 weeks of a
12 month release cycle (or 6 weeks of an 8 week one).  More on this
below.

> It's also worth noting that we were on a 3-month cycle at the start of
> OpenStack. That was dropped after a cataclysmic release that managed the
> feat of (a) not having anything significant done, and (b) have out of
> date documentation and translations.
> 
> While I agree that the packagers and stable teams can opt to skip a
> release, the docs, translations or security teams don't really have that
> luxury... Please go beyond the developers needs and consider the needs
> of the other teams.
> 
> Random other comments below:
> 
> > [...]
> > Release schedule
> > 
> > 
> > First the releases would probably be best attached to a set of
> > pre-determined fixed dates that don't ever vary from year to year.
> > eg releses happen Feb 1st, Apr 1st, Jun 1st, Aug 1st, Oct 1st, and
> > Dec 1st. If a particular release slips, don't alter following release
> > dates, just shorten the length of the dev cycle, so it becomes fully
> > self-correcting. The even numbered months are suggested to avoid a
> > release landing in xmas/new year :-)
> 
> The Feb 1 release would probably be pretty empty :)
> 
> > [...]
> > Stable branches
> > ---
> > 
> > The consequences of a 2 month release cycle appear fairly severe for
> > the stable branch maint teams at first sight. This is not, however,
> > an insurmountable problem. The linux kernel shows an easy way forward
> > with their approach of only maintaining stable branches for a subset
> > of major releases, based around user / vendor demand. So it is still
> > entirely conceivable that the stable team only provide stable branch
> > releases for 2 out of the 6 yearly releases. ie no additional burden
> > over what they face today. Of course they might decide they want to
> > do more stable branches, but maintain each for a shorter time. So I
> > could equally see them choosing todo 3 or 4 stable branches a year.
> > Whatever is most effective for those involved and those consuming
> > them is fine.
> 
> Stable branches may have the luxury of skipping releases and designate a
> "stable" one from time to time (I reject the Linux comparison because
> the kernel is at a very different moment in software lifecycle). The
> trick being, making one release "special" is sure to recreate the peak
> issues you're trying to solve.

I don't disagree with the observation about different points in the
lifecycle, but perhaps it might be instructive to ask if the linux
kernel ever had a period in its development history that looks somewhat
like OpenStack does now.  I would claim it did: before 2.6, we had the
odd/even develop/stabilise cycle.  The theory driving it was that we
needed a time for everyone to develop then a time for everyone to help
make stable.  You yourself said this in the other thread:

> Joe Gordon wrote:
> > [...]
> > I think a lot of the frustration with

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Kyle Mestery]

On Mon, Mar 2, 2015 at 3:38 PM, Joe Gordon  wrote:

>
>
> On Mon, Mar 2, 2015 at 9:59 AM, Kyle Mestery  wrote:
>
>> On Mon, Mar 2, 2015 at 9:57 AM, Ihar Hrachyshka 
>> wrote:
>>
>>> -BEGIN PGP SIGNED MESSAGE-
>>> Hash: SHA1
>>>
>>> Hi Daniel,
>>>
>>> thanks for a clear write-up of the matter and food for thought.
>>>
>>> I think the idea of having more smooth development mode that would not
>>> make people to wait for 6+ months to release a new feature is great.
>>>
>>> ++
>>
>>
>>> It's insane to expect that feature priorities won't ever slightly
>>> shift in the next 6 months. Having a limited list of features targeted
>>> for the next 6 months is prone to mistakes, since people behind some
>>>
>>
> * Sure, we have had a few things that popped up, nova EC2 split for
> example. But this is fairly rare.
>
>
>> of approved features may need to postpone the effort for any type of
>>> reasons (personal, job rush, bad resource allocation, ...), and we
>>
>> then end up with approved specs with no actual code drops, using
>>> review 'slots' that would better be spent for other features that were
>>> not that lucky to get a rubber stamp during spec phase. Prior resource
>>>
>>
> * As stated below specs approval is very much not rubber stamping
> * As stated below this doesn't even make sense, we are *not* using review
> slots.
>
>
>> allocation would probably work somehow if we were working for the same
>>> company that would define priorities to us, but it's not the case.
>>
>>
>>> It should be noted that even though Nova is using slots for reviews,
>> Neutron is not. I've found that it's hard to try and "slot" people in to
>> review specific things. During Juno I tried this for Neutron, and it failed
>> miserably. For Kilo in Neutron, we're not using slots but instead I've
>> tried to highlight the approved specs of "Essential" and "High" priority
>> for review by all reviews, core and non-core included. It's gone ok, but
>> the reality is you can't force people to review things. There are steps
>> submitters can take to try and get timely review (lots of small, easy to
>> digest patches, quick turnaround of comments, engagement in IRC and ML,
>> etc.).
>>
>
> So this is a big fat lie, one that others believe as well. Nova is *not*
> using slots for reviews. We discussed using slots for reviews but did not
> adopt them.
>
>
"But I read it on the internet, it must be true."

As I said in a prior email, I'm sorry for that. I recalled reading about
nova's use of slots.



>
>>
>>> Anecdotally, in neutron, we have two Essential blueprints for Kilo,
>>> and there are no code drops or patches in review for any of those, so
>>> I would expect them to fail to merge. At the same time, I will need to
>>> wait for the end of Kilo to consider adding support for guru reports
>>> to the project. Or in oslo world, I will need to wait for Liberty to
>>> introduce features in oslo.policy that are needed by neutron to switch
>>> to it, etc.
>>>
>>> To be fair, there are many reasons those to "Essential" BPs do not have
>> code. I still expect the Pecan focused to have code, but I already moved
>> the Plugin one out of Kilo at this point because there was no chance the
>> code would land.
>>
>> But I get your point here. I think this thread has highlighted the fact
>> that the BP/spec process worked to some extent, but for small things, the
>> core reviewer team should have the ability to say "Yes, we can easily merge
>> that, lets approve that spec" even if it's late in the cycle.
>>
>>
>>> Another problem is that currently milestones are used merely for
>>> targeting bugs and features, but no one really cares about whether the
>>> actual milestone shipment actually works. Again, a story from neutron
>>> world: Kilo-1 was shipped in the middle of advanced services split,
>>> with some essential patches around distutils setup missing (no proper
>>> migration plan applied, conflicting config files in neutron and *aas
>>> repos, etc.)
>>>
>>> This is true, the milestone release matter but are not given enough
>> focus and they release (for the most part) irregardless of items in them,
>> given they are not long-lived, etc.
>>
>> So I'm all for reforms around processes we apply.
>>>
>>> "If there's one thing OpenStack is good at, it's change."
>>
>>
>>> That said, I don't believe the real problem here is that we don't
>>> generate project tarballs frequently enough.
>>>
>>> Major problems I see as critical to tackle in our dev process are:
>>>
>>> - - enforced spec/dev mode. Solution: allow to propose (and approve) a
>>> reasonable spec at any moment in the cycle; allow code drops for
>>> approved specs at any moment in the cycle (except pre-release
>>> stabilization time); stop targeting specs: if it's sane, it's probably
>>> sane N+2 cycle from now too.
>>>
>>> I'd say this is fine for specs that are small and people agree can
>> easily be merged. I'd say this is not the case for large features near the
>> end of the rele

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Joe Gordon]

On Mon, Mar 2, 2015 at 9:59 AM, Kyle Mestery  wrote:

> On Mon, Mar 2, 2015 at 9:57 AM, Ihar Hrachyshka 
> wrote:
>
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA1
>>
>> Hi Daniel,
>>
>> thanks for a clear write-up of the matter and food for thought.
>>
>> I think the idea of having more smooth development mode that would not
>> make people to wait for 6+ months to release a new feature is great.
>>
>> ++
>
>
>> It's insane to expect that feature priorities won't ever slightly
>> shift in the next 6 months. Having a limited list of features targeted
>> for the next 6 months is prone to mistakes, since people behind some
>>
>
* Sure, we have had a few things that popped up, nova EC2 split for
example. But this is fairly rare.


> of approved features may need to postpone the effort for any type of
>> reasons (personal, job rush, bad resource allocation, ...), and we
>
> then end up with approved specs with no actual code drops, using
>> review 'slots' that would better be spent for other features that were
>> not that lucky to get a rubber stamp during spec phase. Prior resource
>>
>
* As stated below specs approval is very much not rubber stamping
* As stated below this doesn't even make sense, we are *not* using review
slots.


> allocation would probably work somehow if we were working for the same
>> company that would define priorities to us, but it's not the case.
>
>
>> It should be noted that even though Nova is using slots for reviews,
> Neutron is not. I've found that it's hard to try and "slot" people in to
> review specific things. During Juno I tried this for Neutron, and it failed
> miserably. For Kilo in Neutron, we're not using slots but instead I've
> tried to highlight the approved specs of "Essential" and "High" priority
> for review by all reviews, core and non-core included. It's gone ok, but
> the reality is you can't force people to review things. There are steps
> submitters can take to try and get timely review (lots of small, easy to
> digest patches, quick turnaround of comments, engagement in IRC and ML,
> etc.).
>

So this is a big fat lie, one that others believe as well. Nova is *not*
using slots for reviews. We discussed using slots for reviews but did not
adopt them.


>
>
>> Anecdotally, in neutron, we have two Essential blueprints for Kilo,
>> and there are no code drops or patches in review for any of those, so
>> I would expect them to fail to merge. At the same time, I will need to
>> wait for the end of Kilo to consider adding support for guru reports
>> to the project. Or in oslo world, I will need to wait for Liberty to
>> introduce features in oslo.policy that are needed by neutron to switch
>> to it, etc.
>>
>> To be fair, there are many reasons those to "Essential" BPs do not have
> code. I still expect the Pecan focused to have code, but I already moved
> the Plugin one out of Kilo at this point because there was no chance the
> code would land.
>
> But I get your point here. I think this thread has highlighted the fact
> that the BP/spec process worked to some extent, but for small things, the
> core reviewer team should have the ability to say "Yes, we can easily merge
> that, lets approve that spec" even if it's late in the cycle.
>
>
>> Another problem is that currently milestones are used merely for
>> targeting bugs and features, but no one really cares about whether the
>> actual milestone shipment actually works. Again, a story from neutron
>> world: Kilo-1 was shipped in the middle of advanced services split,
>> with some essential patches around distutils setup missing (no proper
>> migration plan applied, conflicting config files in neutron and *aas
>> repos, etc.)
>>
>> This is true, the milestone release matter but are not given enough focus
> and they release (for the most part) irregardless of items in them, given
> they are not long-lived, etc.
>
> So I'm all for reforms around processes we apply.
>>
>> "If there's one thing OpenStack is good at, it's change."
>
>
>> That said, I don't believe the real problem here is that we don't
>> generate project tarballs frequently enough.
>>
>> Major problems I see as critical to tackle in our dev process are:
>>
>> - - enforced spec/dev mode. Solution: allow to propose (and approve) a
>> reasonable spec at any moment in the cycle; allow code drops for
>> approved specs at any moment in the cycle (except pre-release
>> stabilization time); stop targeting specs: if it's sane, it's probably
>> sane N+2 cycle from now too.
>>
>> I'd say this is fine for specs that are small and people agree can easily
> be merged. I'd say this is not the case for large features near the end of
> the release which are unlikely to gather enough review momentum to actually
> merge.
>
>
>> - - core team rubber stamping a random set of specs and putting -2 on
>> all other specs due to "project priorities". Solution: stop pretending
>
> core team (or core drivers) can reasonably estimate review and coding
>> resources for the ne

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Kyle Mestery]

On Mon, Mar 2, 2015 at 11:59 AM, Kyle Mestery  wrote:

> On Mon, Mar 2, 2015 at 9:57 AM, Ihar Hrachyshka 
> wrote:
>
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA1
>>
>> Hi Daniel,
>>
>> thanks for a clear write-up of the matter and food for thought.
>>
>> I think the idea of having more smooth development mode that would not
>> make people to wait for 6+ months to release a new feature is great.
>>
>> ++
>
>
>> It's insane to expect that feature priorities won't ever slightly
>> shift in the next 6 months. Having a limited list of features targeted
>> for the next 6 months is prone to mistakes, since people behind some
>> of approved features may need to postpone the effort for any type of
>> reasons (personal, job rush, bad resource allocation, ...), and we
>> then end up with approved specs with no actual code drops, using
>> review 'slots' that would better be spent for other features that were
>> not that lucky to get a rubber stamp during spec phase. Prior resource
>> allocation would probably work somehow if we were working for the same
>> company that would define priorities to us, but it's not the case.
>>
>> It should be noted that even though Nova is using slots for reviews,
> Neutron is not. I've found that it's hard to try and "slot" people in to
> review specific things. During Juno I tried this for Neutron, and it failed
> miserably. For Kilo in Neutron, we're not using slots but instead I've
> tried to highlight the approved specs of "Essential" and "High" priority
> for review by all reviews, core and non-core included. It's gone ok, but
> the reality is you can't force people to review things. There are steps
> submitters can take to try and get timely review (lots of small, easy to
> digest patches, quick turnaround of comments, engagement in IRC and ML,
> etc.).
>
>
It was pointed out to me that nova is NOT using slots. Apologies for my
misunderstanding here.

Clearly, this thread has elicited a lot of strong thoughts and emotions. I
hope we can all use this energy to figure out a good solution and a way
forward for the issues presented here.



> Anecdotally, in neutron, we have two Essential blueprints for Kilo,
>> and there are no code drops or patches in review for any of those, so
>> I would expect them to fail to merge. At the same time, I will need to
>> wait for the end of Kilo to consider adding support for guru reports
>> to the project. Or in oslo world, I will need to wait for Liberty to
>> introduce features in oslo.policy that are needed by neutron to switch
>> to it, etc.
>>
>> To be fair, there are many reasons those to "Essential" BPs do not have
> code. I still expect the Pecan focused to have code, but I already moved
> the Plugin one out of Kilo at this point because there was no chance the
> code would land.
>
> But I get your point here. I think this thread has highlighted the fact
> that the BP/spec process worked to some extent, but for small things, the
> core reviewer team should have the ability to say "Yes, we can easily merge
> that, lets approve that spec" even if it's late in the cycle.
>
>
>> Another problem is that currently milestones are used merely for
>> targeting bugs and features, but no one really cares about whether the
>> actual milestone shipment actually works. Again, a story from neutron
>> world: Kilo-1 was shipped in the middle of advanced services split,
>> with some essential patches around distutils setup missing (no proper
>> migration plan applied, conflicting config files in neutron and *aas
>> repos, etc.)
>>
>> This is true, the milestone release matter but are not given enough focus
> and they release (for the most part) irregardless of items in them, given
> they are not long-lived, etc.
>
> So I'm all for reforms around processes we apply.
>>
>> "If there's one thing OpenStack is good at, it's change."
>
>
>> That said, I don't believe the real problem here is that we don't
>> generate project tarballs frequently enough.
>>
>> Major problems I see as critical to tackle in our dev process are:
>>
>> - - enforced spec/dev mode. Solution: allow to propose (and approve) a
>> reasonable spec at any moment in the cycle; allow code drops for
>> approved specs at any moment in the cycle (except pre-release
>> stabilization time); stop targeting specs: if it's sane, it's probably
>> sane N+2 cycle from now too.
>>
>> I'd say this is fine for specs that are small and people agree can easily
> be merged. I'd say this is not the case for large features near the end of
> the release which are unlikely to gather enough review momentum to actually
> merge.
>
>
>> - - core team rubber stamping a random set of specs and putting -2 on
>> all other specs due to "project priorities". Solution: stop pretending
>> core team (or core drivers) can reasonably estimate review and coding
>> resources for the next cycle. Instead allows community to decide
>> what's worth the effort by approving all technically reasonable specs
>> and allowing everyone 

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Doug Hellmann]

On Mon, Mar 2, 2015, at 10:57 AM, Ihar Hrachyshka wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> Hi Daniel,
> 
> thanks for a clear write-up of the matter and food for thought.
> 
> I think the idea of having more smooth development mode that would not
> make people to wait for 6+ months to release a new feature is great.
> 
> It's insane to expect that feature priorities won't ever slightly
> shift in the next 6 months. Having a limited list of features targeted
> for the next 6 months is prone to mistakes, since people behind some
> of approved features may need to postpone the effort for any type of
> reasons (personal, job rush, bad resource allocation, ...), and we
> then end up with approved specs with no actual code drops, using
> review 'slots' that would better be spent for other features that were
> not that lucky to get a rubber stamp during spec phase. Prior resource
> allocation would probably work somehow if we were working for the same
> company that would define priorities to us, but it's not the case.
> 
> Anecdotally, in neutron, we have two Essential blueprints for Kilo,
> and there are no code drops or patches in review for any of those, so
> I would expect them to fail to merge. At the same time, I will need to
> wait for the end of Kilo to consider adding support for guru reports
> to the project. Or in oslo world, I will need to wait for Liberty to
> introduce features in oslo.policy that are needed by neutron to switch
> to it, etc.

The Oslo policy is a bit more lenient than some of the others I've heard
described. We don't follow the feature proposal freeze. Instead, we only
have a hard freeze for the master branches of libraries until the
release candidates are done. Any features proposed at any other time are
candidates, with the usual caveats for review team time and attention.

Doug

> 
> Another problem is that currently milestones are used merely for
> targeting bugs and features, but no one really cares about whether the
> actual milestone shipment actually works. Again, a story from neutron
> world: Kilo-1 was shipped in the middle of advanced services split,
> with some essential patches around distutils setup missing (no proper
> migration plan applied, conflicting config files in neutron and *aas
> repos, etc.)
> 
> So I'm all for reforms around processes we apply.
> 
> That said, I don't believe the real problem here is that we don't
> generate project tarballs frequently enough.
> 
> Major problems I see as critical to tackle in our dev process are:
> 
> - - enforced spec/dev mode. Solution: allow to propose (and approve) a
> reasonable spec at any moment in the cycle; allow code drops for
> approved specs at any moment in the cycle (except pre-release
> stabilization time); stop targeting specs: if it's sane, it's probably
> sane N+2 cycle from now too.
> 
> - - core team rubber stamping a random set of specs and putting -2 on
> all other specs due to "project priorities". Solution: stop pretending
> core team (or core drivers) can reasonably estimate review and coding
> resources for the next cycle. Instead allows community to decide
> what's worth the effort by approving all technically reasonable specs
> and allowing everyone to invest time and effort in specs (s)he seems
> worth it.
> 
> - - no due stabilization process before dev milestones. Solution:
> introduce one in your team workflow, be more strict in what goes in
> during pre milestone stabilization time.
> 
> If all above is properly applied, we would get into position similar
> to your proposal. The difference though is that upstream project would
> not call milestone tarballs 'Releases'. Still, if there are brave
> vendors to ship milestones, fine, they would have the same freedom as
> in your proposal.
> 
> Note: all the steps mentioned above can be applied on *per team* basis
> without breaking existing release workflow.
> 
> Some more comments from stable-maint/distribution side below.
> 
> On 02/24/2015 10:53 AM, Daniel P. Berrange wrote:
> [...skip...]
> > The modest proposal ===
> [...skip...]
> > 
> > Stable branches ---
> > 
> > The consequences of a 2 month release cycle appear fairly severe
> > for the stable branch maint teams at first sight. This is not,
> > however, an insurmountable problem. The linux kernel shows an easy
> > way forward with their approach of only maintaining stable branches
> > for a subset of major releases, based around user / vendor demand.
> > So it is still entirely conceivable that the stable team only
> > provide stable branch releases for 2 out of the 6 yearly releases.
> > ie no additional burden over what they face today. Of course they
> > might decide they want to do more stable branches, but maintain
> > each for a shorter time. So I could equally see them choosing todo
> > 3 or 4 stable branches a year. Whatever is most effective for those
> > involved and those consuming them is fine.
> > 
> 
> Since it's not o

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Kyle Mestery]

On Mon, Mar 2, 2015 at 9:57 AM, Ihar Hrachyshka  wrote:

> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Hi Daniel,
>
> thanks for a clear write-up of the matter and food for thought.
>
> I think the idea of having more smooth development mode that would not
> make people to wait for 6+ months to release a new feature is great.
>
> ++


> It's insane to expect that feature priorities won't ever slightly
> shift in the next 6 months. Having a limited list of features targeted
> for the next 6 months is prone to mistakes, since people behind some
> of approved features may need to postpone the effort for any type of
> reasons (personal, job rush, bad resource allocation, ...), and we
> then end up with approved specs with no actual code drops, using
> review 'slots' that would better be spent for other features that were
> not that lucky to get a rubber stamp during spec phase. Prior resource
> allocation would probably work somehow if we were working for the same
> company that would define priorities to us, but it's not the case.
>
> It should be noted that even though Nova is using slots for reviews,
Neutron is not. I've found that it's hard to try and "slot" people in to
review specific things. During Juno I tried this for Neutron, and it failed
miserably. For Kilo in Neutron, we're not using slots but instead I've
tried to highlight the approved specs of "Essential" and "High" priority
for review by all reviews, core and non-core included. It's gone ok, but
the reality is you can't force people to review things. There are steps
submitters can take to try and get timely review (lots of small, easy to
digest patches, quick turnaround of comments, engagement in IRC and ML,
etc.).


> Anecdotally, in neutron, we have two Essential blueprints for Kilo,
> and there are no code drops or patches in review for any of those, so
> I would expect them to fail to merge. At the same time, I will need to
> wait for the end of Kilo to consider adding support for guru reports
> to the project. Or in oslo world, I will need to wait for Liberty to
> introduce features in oslo.policy that are needed by neutron to switch
> to it, etc.
>
> To be fair, there are many reasons those to "Essential" BPs do not have
code. I still expect the Pecan focused to have code, but I already moved
the Plugin one out of Kilo at this point because there was no chance the
code would land.

But I get your point here. I think this thread has highlighted the fact
that the BP/spec process worked to some extent, but for small things, the
core reviewer team should have the ability to say "Yes, we can easily merge
that, lets approve that spec" even if it's late in the cycle.


> Another problem is that currently milestones are used merely for
> targeting bugs and features, but no one really cares about whether the
> actual milestone shipment actually works. Again, a story from neutron
> world: Kilo-1 was shipped in the middle of advanced services split,
> with some essential patches around distutils setup missing (no proper
> migration plan applied, conflicting config files in neutron and *aas
> repos, etc.)
>
> This is true, the milestone release matter but are not given enough focus
and they release (for the most part) irregardless of items in them, given
they are not long-lived, etc.

So I'm all for reforms around processes we apply.
>
> "If there's one thing OpenStack is good at, it's change."


> That said, I don't believe the real problem here is that we don't
> generate project tarballs frequently enough.
>
> Major problems I see as critical to tackle in our dev process are:
>
> - - enforced spec/dev mode. Solution: allow to propose (and approve) a
> reasonable spec at any moment in the cycle; allow code drops for
> approved specs at any moment in the cycle (except pre-release
> stabilization time); stop targeting specs: if it's sane, it's probably
> sane N+2 cycle from now too.
>
> I'd say this is fine for specs that are small and people agree can easily
be merged. I'd say this is not the case for large features near the end of
the release which are unlikely to gather enough review momentum to actually
merge.


> - - core team rubber stamping a random set of specs and putting -2 on
> all other specs due to "project priorities". Solution: stop pretending
> core team (or core drivers) can reasonably estimate review and coding
> resources for the next cycle. Instead allows community to decide
> what's worth the effort by approving all technically reasonable specs
> and allowing everyone to invest time and effort in specs (s)he seems
> worth it.
>
> If you're referring to Neutron here, I think you fail to estimate the
amount of time the neutron-drivers team (along with a handful of other
folks) spent reviewing specs and trying to allocate them for this release.
We're not just rubber stamping things, we're reviewing, providing comment,
and ensuring things fit in a consistent roadmap for the next release. In
the past, we've had this sort of "

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Ihar Hrachyshka]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 02/26/2015 01:06 AM, Thomas Goirand wrote:
> On 02/24/2015 12:27 PM, Daniel P. Berrange wrote:
>> I'm actually trying to judge it from the POV of users, not just 
>> developers. I find it pretty untenable that in the fast moving 
>> world of cloud, users have to wait as long as 6 months for a 
>> feature to get into a openstack release, often much longer.
> 
> If you were trying to judge from the POV of users, then you would 
> consider that basically, they don't really care the brand new
> shiny feature which just appear. They care having a long time
> support for whatever version of OpenStack they have installed,
> without having the head-aches of upgrading which is famously
> painful with OpenStack. This shows clearly on our user surveys
> which are presented on every summit: users are lagging behind, with
> a majority still running with OpenStack releases which are already
> EOL.
> 

As was already said in the thread, they care about long support AND
their specific subset of shiny new features they need (and they don't
care about all other features that are not in their subjective list of
shiny ones).

> In fact, if you want to judge from the POV of our users, we should
> *SLOW DOWN* our release cycles, and probably move to something like
> one release every year or 2. We should also try to have longer
> periods of support for our stable releases, which would (with my
> Debian package maintainer hat on!) help distributions to do such
> security support.
> 
> Debian Jessie will be released in a few month from now, just
> before Icehouse (which it ships) will be EOL. RedHat, Canonical,
> IBM, and so many more are also on the same (sinking) ship.
> 

That won't come up magically. If no people from Debian or IBM are
actively involved in solving issues for stable branches, long term
will be just a wish and not a reality. At the moment we have several
people from Red Hat (me, Alan) and Ubuntu (Chuck, Adam) side active in
the team. That would be great to see more diversity, and more hands
dirty with stable branches on daily basis.

> As for my employer side of things, we've seen numerous cases with
> our customer requesting for LTS, which we have to provide by
> ourselves, since it's not supported upstream.
> 

Every downstream vendor would be happy to see upstream freely support
their product. :) But you get what you pay to upstream horizontal
initiatives like stable maintenance.

I've heard IBM still supports releases starting from C release!..

>> I think the majority of translation work can be done in parallel
>> with dev work and the freeze time just needs to tie up the small
>> remaining bits.
> 
> It'd be nice indeed, but I've never seen any project (open source
> or not) working this way for translations.

It's actually not true. I've coordinated multiple translation teams
for my language (Belarusian), and for what I can tell, the best
practice is to work on translations while development is ongoing. Yes,
it means some effort wasted, but it also means spreading the whole
effort throughout the year instead of doing all the work in
pre-release freeze time.

Freeze time is anyway good to make sure that no last minute patches
break translations, and I don't suggest we drop them completely.

> 
>> Documentation is again something I'd expect to be done more or
>> less in parallel with dev.
> 
> Let's go back to reality: the Juno install-guide is still not
> finished, and the doc team is lagging behind.
> 
>> It would be reasonable for the vulnerability team to take the
>> decision that they'll support fixes for master, and any branches
>> that the stable team decide to support. ie they would not
>> neccessarily have to commit to shipping fixes for every single
>> release made.
> 
> I've been crying for this type of decision. IE: drop Juno support
> early, and continue to maintain Icehouse for longer. I wish this
> happens, but the release team always complained that nobody works
> on maintaining the gate for the stable branches. Unless this
> changes, I don't see hope... :(
> 

- From Red Hat perspective, we put our effort in what we're interested
in. We're more interested in keeping the branch for the latest release
working, not about sticking to an old release that e.g. Debian chose
to put into their next official release, so unless someone from Debian
come to the team and start to invest into Icehouse, it will eventually
be dropped. That's life.

>> I really not trying to focus on the developers woes. I'm trying
>> to focus on making OpenStack better serve our users. My main
>> motiviation here is that I think we're doing a pretty terrible
>> job at getting work done that is important to our users in a
>> timely manner. This is caused by a workflow & release cycle that
>> is negatively impacting the developers.
> 
> Our workflow and the release cycle are 2 separate things. From my
> POV, it'd be a mistake to believe switching to a different release
> cycl

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Ihar Hrachyshka]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi Daniel,

thanks for a clear write-up of the matter and food for thought.

I think the idea of having more smooth development mode that would not
make people to wait for 6+ months to release a new feature is great.

It's insane to expect that feature priorities won't ever slightly
shift in the next 6 months. Having a limited list of features targeted
for the next 6 months is prone to mistakes, since people behind some
of approved features may need to postpone the effort for any type of
reasons (personal, job rush, bad resource allocation, ...), and we
then end up with approved specs with no actual code drops, using
review 'slots' that would better be spent for other features that were
not that lucky to get a rubber stamp during spec phase. Prior resource
allocation would probably work somehow if we were working for the same
company that would define priorities to us, but it's not the case.

Anecdotally, in neutron, we have two Essential blueprints for Kilo,
and there are no code drops or patches in review for any of those, so
I would expect them to fail to merge. At the same time, I will need to
wait for the end of Kilo to consider adding support for guru reports
to the project. Or in oslo world, I will need to wait for Liberty to
introduce features in oslo.policy that are needed by neutron to switch
to it, etc.

Another problem is that currently milestones are used merely for
targeting bugs and features, but no one really cares about whether the
actual milestone shipment actually works. Again, a story from neutron
world: Kilo-1 was shipped in the middle of advanced services split,
with some essential patches around distutils setup missing (no proper
migration plan applied, conflicting config files in neutron and *aas
repos, etc.)

So I'm all for reforms around processes we apply.

That said, I don't believe the real problem here is that we don't
generate project tarballs frequently enough.

Major problems I see as critical to tackle in our dev process are:

- - enforced spec/dev mode. Solution: allow to propose (and approve) a
reasonable spec at any moment in the cycle; allow code drops for
approved specs at any moment in the cycle (except pre-release
stabilization time); stop targeting specs: if it's sane, it's probably
sane N+2 cycle from now too.

- - core team rubber stamping a random set of specs and putting -2 on
all other specs due to "project priorities". Solution: stop pretending
core team (or core drivers) can reasonably estimate review and coding
resources for the next cycle. Instead allows community to decide
what's worth the effort by approving all technically reasonable specs
and allowing everyone to invest time and effort in specs (s)he seems
worth it.

- - no due stabilization process before dev milestones. Solution:
introduce one in your team workflow, be more strict in what goes in
during pre milestone stabilization time.

If all above is properly applied, we would get into position similar
to your proposal. The difference though is that upstream project would
not call milestone tarballs 'Releases'. Still, if there are brave
vendors to ship milestones, fine, they would have the same freedom as
in your proposal.

Note: all the steps mentioned above can be applied on *per team* basis
without breaking existing release workflow.

Some more comments from stable-maint/distribution side below.

On 02/24/2015 10:53 AM, Daniel P. Berrange wrote:
[...skip...]
> The modest proposal ===
[...skip...]
> 
> Stable branches ---
> 
> The consequences of a 2 month release cycle appear fairly severe
> for the stable branch maint teams at first sight. This is not,
> however, an insurmountable problem. The linux kernel shows an easy
> way forward with their approach of only maintaining stable branches
> for a subset of major releases, based around user / vendor demand.
> So it is still entirely conceivable that the stable team only
> provide stable branch releases for 2 out of the 6 yearly releases.
> ie no additional burden over what they face today. Of course they
> might decide they want to do more stable branches, but maintain
> each for a shorter time. So I could equally see them choosing todo
> 3 or 4 stable branches a year. Whatever is most effective for those
> involved and those consuming them is fine.
> 

Since it's not only stable branches that are affected (translators,
documentation writers, VMT were already mentioned), those affected
will probably need to come up with some synchronized decision.

Let's say we still decide to support two out of six releases (same
scheme as is now). In that case no process that we usually attach to
releases will be running after release dates. This makes me wonder how
it's different from milestones we already have.

Do you think any downstream vendors will actually ship and support
upstream releases that upstream drops any guarantees for (no VMT, no
stable branches, no gate runs, ...) ri

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Ed Leafe]

On Feb 26, 2015, at 9:49 AM, Ritesh Raj Sarraf  wrote:

>> I think you've nailed where the disconnect is between the two sides of this 
>> issue: what exactly do we see OpenStack being? You brought up several Linux 
>> vendors who ship on a longish cycle, and who provide LTS for their releases. 
>> But Linux itself is on no such cycle, nor does it provide long term anything.
> 
> But Linux is one monolith project.

No, not really. The kernel may be, but if you want a distro that is useful to 
people, you need to collect a set of tools and apps that allow users to have a 
full computing experience. If Linux were a monolith, then there really wouldn't 
be any difference between Ubuntu and Slackware.

> I am fairly new to OpenStack, but from what I've ascertained so far,
> there are, now, individual sub-releases of individual projects. That
> could be a difficult task for any Linux vendor, to distribute.

I don't think that anyone is implying that it is easy to do, no matter where it 
is done. It does seem that there are significant disadvantages on the 
development side of things to have one group doing it all.


-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Ritesh Raj Sarraf]

On 02/26/2015 07:06 PM, Ed Leafe wrote:
> I think you've nailed where the disconnect is between the two sides of this 
> issue: what exactly do we see OpenStack being? You brought up several Linux 
> vendors who ship on a longish cycle, and who provide LTS for their releases. 
> But Linux itself is on no such cycle, nor does it provide long term anything.
>

But Linux is one monolith project.

> OpenStack can't be all things to all people. Following the Linux analogy, we 
> need a few companies who want to become OpenStack distributors, packagers, 
> and supporters, in the manner of RedHat, Canonical, etc., are for Linux. As a 
> development project, we need to be able to move fluidly, and the release 
> cycle deadlines and freezes get in the way of that. As a packager and 
> distributor, the release cycle scheduler *helps* immeasurably. We can't be 
> both.

I am fairly new to OpenStack, but from what I've ascertained so far,
there are, now, individual sub-releases of individual projects. That
could be a difficult task for any Linux vendor, to distribute.

There's one project, Calibre - EBook Management Software, that does
weekly releases. But again, it is easy for them, because they are a
single controlled project.

For something big as OpenStack, IMO, close co-ordination is needed.

-- 
Given the large number of mailing lists I follow, I request you to CC me
in replies for quicker response


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Chris Dent]

On Thu, 26 Feb 2015, Ed Leafe wrote:


OpenStack can't be all things to all people. Following the Linux
analogy, we need a few companies who want to become OpenStack
distributors, packagers, and supporters, in the manner of RedHat,
Canonical, etc., are for Linux. As a development project, we need to
be able to move fluidly, and the release cycle deadlines and freezes
get in the way of that. As a packager and distributor, the release
cycle scheduler *helps* immeasurably. We can't be both.


Yes.

The special nature of OpenStack makes the influence of companies
quite strong and has many benefits (e.g. look ma, being paid to do open
source) but it also means that boundaries that are found in other
somewhat similar environments are not as strong.

Elsewhere in the thread there was discussion about the importance of the
release cycle for marketing. Marketing for who and by whom? Surely
marketing is^wought to be in the domain of the people who are making
money selling aggregations of OpenStack?

--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Ed Leafe]

On Feb 25, 2015, at 6:06 PM, Thomas Goirand  wrote:

> In fact, if you want to judge from the POV of our users, we should *SLOW
> DOWN* our release cycles, and probably move to something like one
> release every year or 2. We should also try to have longer periods of
> support for our stable releases, which would (with my Debian package
> maintainer hat on!) help distributions to do such security support.
> 
> Debian Jessie will be released in a few month from now, just before
> Icehouse (which it ships) will be EOL. RedHat, Canonical, IBM, and so
> many more are also on the same (sinking) ship.
> 
> As for my employer side of things, we've seen numerous cases with our
> customer requesting for LTS, which we have to provide by ourselves,
> since it's not supported upstream.

I think you've nailed where the disconnect is between the two sides of this 
issue: what exactly do we see OpenStack being? You brought up several Linux 
vendors who ship on a longish cycle, and who provide LTS for their releases. 
But Linux itself is on no such cycle, nor does it provide long term anything.

OpenStack can't be all things to all people. Following the Linux analogy, we 
need a few companies who want to become OpenStack distributors, packagers, and 
supporters, in the manner of RedHat, Canonical, etc., are for Linux. As a 
development project, we need to be able to move fluidly, and the release cycle 
deadlines and freezes get in the way of that. As a packager and distributor, 
the release cycle scheduler *helps* immeasurably. We can't be both.


-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Daniel P. Berrange]

On Thu, Feb 26, 2015 at 01:06:14AM +0100, Thomas Goirand wrote:
> On 02/24/2015 12:27 PM, Daniel P. Berrange wrote:
> > I'm actually trying to judge it from the POV of users, not just
> > developers. I find it pretty untenable that in the fast moving
> > world of cloud, users have to wait as long as 6 months for a
> > feature to get into a openstack release, often much longer.
> 
> If you were trying to judge from the POV of users, then you would
> consider that basically, they don't really care the brand new shiny
> feature which just appear. They care having a long time support for
> whatever version of OpenStack they have installed, without having the
> head-aches of upgrading which is famously painful with OpenStack. This
> shows clearly on our user surveys which are presented on every summit:
> users are lagging behind, with a majority still running with OpenStack
> releases which are already EOL.
> 
> In fact, if you want to judge from the POV of our users, we should *SLOW
> DOWN* our release cycles, and probably move to something like one
> release every year or 2. We should also try to have longer periods of
> support for our stable releases, which would (with my Debian package
> maintainer hat on!) help distributions to do such security support.

It is not actually that simple IME. As I describe elsewhere in this
thread, users basically want no new features at all ever, except for
the new features that are specifically relevant to what they want,
and of course every users desired feature is different. It is essentially
impossible to satisfy the requests in this way because they are inherantly
contradictory & conflicting. In reality what they asking is not that we
reduce the number of releases or frequency of releases, but rather that
they be able to stay running on a chosen release for longer.

IOW The frequency of releases doesn't matter as long as they are not being
forced to upgrade to the newer releases unreasonably quickly. From that
POV I can certainly see how OpenStack is not satisfying them well,
because if they do choose to skip 2-3 releases, they'll find themselves
EOLd and also they'll be forced to deploy each intermediate release in
order to apply upgrades, before finally getting to their target release.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Thomas Goirand]

On 02/24/2015 12:27 PM, Daniel P. Berrange wrote:
> I'm actually trying to judge it from the POV of users, not just
> developers. I find it pretty untenable that in the fast moving
> world of cloud, users have to wait as long as 6 months for a
> feature to get into a openstack release, often much longer.

If you were trying to judge from the POV of users, then you would
consider that basically, they don't really care the brand new shiny
feature which just appear. They care having a long time support for
whatever version of OpenStack they have installed, without having the
head-aches of upgrading which is famously painful with OpenStack. This
shows clearly on our user surveys which are presented on every summit:
users are lagging behind, with a majority still running with OpenStack
releases which are already EOL.

In fact, if you want to judge from the POV of our users, we should *SLOW
DOWN* our release cycles, and probably move to something like one
release every year or 2. We should also try to have longer periods of
support for our stable releases, which would (with my Debian package
maintainer hat on!) help distributions to do such security support.

Debian Jessie will be released in a few month from now, just before
Icehouse (which it ships) will be EOL. RedHat, Canonical, IBM, and so
many more are also on the same (sinking) ship.

As for my employer side of things, we've seen numerous cases with our
customer requesting for LTS, which we have to provide by ourselves,
since it's not supported upstream.

> I think the majority of
> translation work can be done in parallel with dev work and the freeze
> time just needs to tie up the small remaining bits.

It'd be nice indeed, but I've never seen any project (open source or
not) working this way for translations.

> Documentation is again something I'd expect to be done more or less
> in parallel with dev.

Let's go back to reality: the Juno install-guide is still not finished,
and the doc team is lagging behind.

> It would be reasonable for the vulnerability team to take the decision
> that they'll support fixes for master, and any branches that the stable
> team decide to support. ie they would not neccessarily have to commit
> to shipping fixes for every single release made.

I've been crying for this type of decision. IE: drop Juno support early,
and continue to maintain Icehouse for longer. I wish this happens, but
the release team always complained that nobody works on maintaining the
gate for the stable branches. Unless this changes, I don't see hope... :(

> I really not trying to focus on the developers woes. I'm trying to focus on
> making OpenStack better serve our users. My main motiviation here is that I
> think we're doing a pretty terrible job at getting work done that is important
> to our users in a timely manner. This is caused by a workflow & release cycle
> that is negatively impacting the developers.

Our workflow and the release cycle are 2 separate things. From my POV,
it'd be a mistake to believe switching to a different release cycle will
fix our workflow.

Also, I'd like to point out something: it's been 2 years that I do
release each and every beta release we do. But either they are bug free
(you bet... :)), or nobody uses them (more likely), because I've *never
ever* received some bug reports about them. Reasonably, there's no
consumer for beta releases.

Hoping my point of view is helpful,

Cheers,

Thomas Goirand (zigo)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Alex Glikson]

Tom Fifield  wrote on 25/02/2015 06:46:13 AM:

> On 24/02/15 19:27, Daniel P. Berrange wrote:
> > On Tue, Feb 24, 2015 at 12:05:17PM +0100, Thierry Carrez wrote:
> >> Daniel P. Berrange wrote:
> >>> [...]
> 
> > I'm not familiar with how the translations works, but if they are
> > waiting until the freeze before starting translation work I'd
> > say that is a mistaken approach. Obviously during active dev part
> > of the cycle, some translated strings are in flux, so if translation
> > was taking place in parallel there could be some wasted effort, but
> > I'd expect that to be the minority case. I think the majority of
> > translation work can be done in parallel with dev work and the freeze
> > time just needs to tie up the small remaining bits.
> 
> 
> So, two points:
> 
> 1) We wouldn't be talking about throwing just a couple of percent of
> their work away.
> 
> As an example, even without looking at the introduction of new strings
> or deleting others, you may not be aware that changing a single word in
> a string in the code means that entire string needs to be re-translated.
> Even with the extensive translation memory systems we have making
> suggestions as best they can, we're talking about very, very significant
> amounts of "wasted effort".

How difficult would it be to try quantifying this "wasted effort"? For 
example, if someone could write a script that extracts the data for a 
histogram showing the amount of strings (e.g., in Nova) that have been 
changed/overridden in consequent patches up to 1 week apart, between 1 and 
2 weeks apart, and so on up to, say, 52 weeks.

Regards,
Alex


> Regards,
> 
> 
> Tom
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Daniel P. Berrange]

On Wed, Feb 25, 2015 at 10:31:58AM +0100, Thierry Carrez wrote:
> Robert Collins wrote:
> >> It's also worth noting that we were on a 3-month cycle at the start of
> >> OpenStack. That was dropped after a cataclysmic release that managed the
> >> feat of (a) not having anything significant done, and (b) have out of
> >> date documentation and translations.
> > 
> > Oh! 
> > https://wiki.openstack.org/w/index.php?title=Obsolete:3MonthReleaseCycle&direction=prev&oldid=14016
> > appears to be the page about that, but my google-fu isn't turning up a
> > post-mortem of the C release which prompted the change. Perhaps some
> > old-hand like you can fill us in on the details :).
> 
> After the Austin release (a code drop, really), we were on a three-month
> time-based cycle with no intermediary milestone. The first cycle (Bexar)
> worked relatively well, the second one (Cactus) not so well. There were
> multiple reasons for that:
> 
> - Software was growing up in Bexar and we introduced more testing, but
> we were unable to detect and fix all the release blockers in time for
> the release date, resulting in a broken Bexar release. To address that
> in Cactus, we increased the duration of the frozen period, but then that
> didn't leave enough time for development, hence nothing getting done.

I wasn't involved in OpenStack back then, but is it not true that
OpenStack has changed almost beyond recognition since those early
days. The number of projects has certainly increased massively, as
has the number of contributors, both individual & corporate. There
is a hell of alot of testing now and people doing continuous deployment
of trunk, so I find it hard to believe that trunk is so unstable that
we can't do a release at any time we choose, nor that we have to have
such long freezes that we don't do get time for dev.

> - Not starting a release cycle with a global F2F gathering (no
> intermediary Design Summit) actually proved quite disastrous: no reset
> of community, no alignment on goal, no discussion on what to work on.
> That resulted in a limbo period after the Bexar release. That is why I'm
> so insistent on aligning our development cycles with our Design Summits.
> I've seen where we go when we don't.

I don't know about other projects so much, but I don't really see the
design summit as a positive thing wrt planning the next release. For
a start the design summit is atbout 5 weeks after the trunk opens for
development, so if people wait for the summit do do planning they have
thrown away half of the first milestones' development time. It is also
not inclusive as a decision making forum because we cannot assume every
one is able to make it to the summit in person, and even if they are
present, people often can't get involved in all sessions they wish to
due to conflicting schedules. If release planning were done via email
primarily, with IRC for cases where realtime planning is needed, it
would be more effective IMHO.  IOW i think the summit would be better
off if were explicitly not associated with releases, but rather be a
general forum for collaboration were we talk through ideas or do code
sprints, and more.

> > [...]
> >> I may appear defensive on my answers, but it's not my goal to defend the
> >> current system: it's just that most of those proposals generally ignore
> >> the diversity of the needs of the teams that make OpenStack possible, to
> >> focus on a particular set of contributors' woes. I'm trying to bring
> >> that wider perspective in -- the current system is a trade-off and the
> >> result of years of evolution, not an arbitrary historic choice that we
> >> can just change at will.
> > 
> > I agree that its not arbitrary and that changing it requires some
> > appropriate wide-spread consultation; OTOH the benefits of rolling and
> > higher frequency releases are really substantial: but we have to
> > backup the change with the appropriate engineering (such as reducing
> > our frictions that cause teams practicing CD to be weeks behind tip)
> > to make it feasible. My greatest concern about the proposals happening
> > now is that we may bite off more than we can chew. OTGH the reality is
> > that all the negative things multiply out, so we probably need to just
> > start somewhere and /do/ to give us the space to fix other things.
> 
> As I said, I'd very much like us to "release" a lot more often. I just
> don't see how we can do it without:
> 
> - abandoning the idea of translating the software
> - admit that docs will always lag code
> - dropping stable branch maintenance (and only fix vulnerabilities in
> master)

I don't really buy that. If we have 6 month cycles with 1 month freeze
for docs & i18n, vs 2 months cycles with 2 weeks freeze for docs & i18n
the latter is a win in terms of dev time vs stablization time, giving
docs & i18n 50% more time in aggregate.

> That would all make my life (and the life of most feature-focused
> developers) a *lot* easier. I'm not convinced that's th

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Kashyap Chamarthy]

On Tue, Feb 24, 2015 at 10:02:36PM -0800, Mark Atwood wrote:
> On Tue, Feb 24, 2015, at 04:28, Kashyap Chamarthy wrote:
> > 
> > Along with the below, if push comes to shove, OpenStack Foundation could
> > probably try a milder variant (obviously, not all activities can be
> > categorized as 'critical path') of Linux Foundation's "Critical
> > Infrastructure Protection Initiative"[1] to fund certain project
> > activities in need.
> 
> Speaking as a person who sits on the LF CII board meetings,
> and helps turn the crank on that particular sausage mill,
> no, we really don't want to go down that path at this point in
> time.

I didn't imply to do an _exact_ version of LF's CII. Given so much of
interest in OpenStack, vendors/interested parties must (yes) maintain a
balance between giving vs taking from the community. And, they should be
provided ample information on _where_ they can target people/engineers
(Dan Berrange noted this too in one of his responses in this thread).

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Thierry Carrez]

Robert Collins wrote:
>> It's also worth noting that we were on a 3-month cycle at the start of
>> OpenStack. That was dropped after a cataclysmic release that managed the
>> feat of (a) not having anything significant done, and (b) have out of
>> date documentation and translations.
> 
> Oh! 
> https://wiki.openstack.org/w/index.php?title=Obsolete:3MonthReleaseCycle&direction=prev&oldid=14016
> appears to be the page about that, but my google-fu isn't turning up a
> post-mortem of the C release which prompted the change. Perhaps some
> old-hand like you can fill us in on the details :).

After the Austin release (a code drop, really), we were on a three-month
time-based cycle with no intermediary milestone. The first cycle (Bexar)
worked relatively well, the second one (Cactus) not so well. There were
multiple reasons for that:

- Software was growing up in Bexar and we introduced more testing, but
we were unable to detect and fix all the release blockers in time for
the release date, resulting in a broken Bexar release. To address that
in Cactus, we increased the duration of the frozen period, but then that
didn't leave enough time for development, hence nothing getting done.

- Not starting a release cycle with a global F2F gathering (no
intermediary Design Summit) actually proved quite disastrous: no reset
of community, no alignment on goal, no discussion on what to work on.
That resulted in a limbo period after the Bexar release. That is why I'm
so insistent on aligning our development cycles with our Design Summits.
I've seen where we go when we don't.

The post-mortem (at the Diablo summit) recognized our inability as a
group (including QA, docs, stable branch, vulnerability management) to
do final release work more often than every 6 months. Milestones were
introduced so that we can communicate features faster to our users
(those wanting fresh stuff could consume milestones), and then focus QA
/ docs / stable support on the final release only.

Now, the funny thing is... this solution is very similar to what is
proposed in this thread. Intermediary releases every two months to get
code out really fast, and final releases which trigger stable
maintenance. So what happened ?

What happened then is what will happen if we reboot the exact same
solution now: developers ignoring milestones (or intermediary releases)
to focus only on the "real", stable-supported one. And now milestones
are just dates in a calendar.

So when I say that won't solve anything and devs will keep on focusing
on the "real" release, it's not just a gut feeling. We've been there. I
like to learn from history and not repeat the same mistakes.

> [...]
>> I may appear defensive on my answers, but it's not my goal to defend the
>> current system: it's just that most of those proposals generally ignore
>> the diversity of the needs of the teams that make OpenStack possible, to
>> focus on a particular set of contributors' woes. I'm trying to bring
>> that wider perspective in -- the current system is a trade-off and the
>> result of years of evolution, not an arbitrary historic choice that we
>> can just change at will.
> 
> I agree that its not arbitrary and that changing it requires some
> appropriate wide-spread consultation; OTOH the benefits of rolling and
> higher frequency releases are really substantial: but we have to
> backup the change with the appropriate engineering (such as reducing
> our frictions that cause teams practicing CD to be weeks behind tip)
> to make it feasible. My greatest concern about the proposals happening
> now is that we may bite off more than we can chew. OTGH the reality is
> that all the negative things multiply out, so we probably need to just
> start somewhere and /do/ to give us the space to fix other things.

As I said, I'd very much like us to "release" a lot more often. I just
don't see how we can do it without:

- abandoning the idea of translating the software
- admit that docs will always lag code
- dropping stable branch maintenance (and only fix vulnerabilities in
master)

That would all make my life (and the life of most feature-focused
developers) a *lot* easier. I'm not convinced that's the best solution
for our downstream users, though. I think they like docs,
translations(?), stable branches and backported security fixes.

PS: as a data point, at the last summit, in the stable branch session,
there was a whole group of downstream users crashing the session to ask
for releasing *less* often ("we can't keep up"). I think that's crazy
too, but that shows 6 months is really a trade-off.

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Mark Atwood]

On Tue, Feb 24, 2015, at 04:28, Kashyap Chamarthy wrote:
> 
> Along with the below, if push comes to shove, OpenStack Foundation could
> probably try a milder variant (obviously, not all activities can be
> categorized as 'critical path') of Linux Foundation's "Critical
> Infrastructure Protection Initiative"[1] to fund certain project
> activities in need.

Speaking as a person who sits on the LF CII board meetings,
and helps turn the crank on that particular sausage mill,
no, we really don't want to go down that path at this point in
time.

-- 
Mark Atwood, Director of Open Source Engagement, HP

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Tom Fifield]

On 24/02/15 19:27, Daniel P. Berrange wrote:
> On Tue, Feb 24, 2015 at 12:05:17PM +0100, Thierry Carrez wrote:
>> Daniel P. Berrange wrote:
>>> [...]

First, Daniel, thank you for the well-written and thought-through post.
I have some comments on translation specifically which I hope can shed
some light on this particular horizontal effort.

With the idea as stated below implemented, I think it would basically
demoralise our translation teams to the point they'd give up. We already
get complaints about people not respecting string freeze as it is :D


> I'm not familiar with how the translations works, but if they are
> waiting until the freeze before starting translation work I'd
> say that is a mistaken approach. Obviously during active dev part
> of the cycle, some translated strings are in flux, so if translation
> was taking place in parallel there could be some wasted effort, but
> I'd expect that to be the minority case. I think the majority of
> translation work can be done in parallel with dev work and the freeze
> time just needs to tie up the small remaining bits.


So, two points:

1) We wouldn't be talking about throwing just a couple of percent of
their work away.

As an example, even without looking at the introduction of new strings
or deleting others, you may not be aware that changing a single word in
a string in the code means that entire string needs to be re-translated.
Even with the extensive translation memory systems we have making
suggestions as best they can, we're talking about very, very significant
amounts of "wasted effort". Something as simple as adding "ing" on a
verb to fix an English grammar mistake means a completely different
sentence in languages I'm familiar with.


2) The overwhelming majority of our [hundreds of] translators are
volunteers.

Unlike many of those writing the software, they are not paid to do what
they do, and do it in their spare time. Make it less fun, and they
simply walk away.



To try and put this in a way that may be more understandable for a
non-translator ... your (impressive!) original email in this thread was
around 3000 words. Translatable strings in horizon is around five times
that at the moment. So imagine that, when writing an email five times
longer than the one you wrote, unpaid, someone you don't really know
that well decided that the section on "the key observations" (230 words
- about 1% of the text of our 'horizon') you just wrote should be
re-arranged - the order of the observations changed, one of them removed
and replaced with another slightly different one, and the conclusion
paragraph wording should be amended to suit.

It would be an understatement to say that such behaviour would be
'annoying' if it happened constantly as you were writing your email.
Consider then if it applied to every email you sought to write :)

Now, the amount of string changes within a release can be left for
someone to work out, but it's certainly a great deal more than a single
percent. Multiply that poor experience by the reality of string change
across all the projects we want to translate. Then multiply it by the
number of languages we want. Finally, multiply it by the number of
people we need to translate a single language. That's a really bad time
for a whole lot of people.


At the moment we're fixing this with string freeze, and that's generally
going pretty well. Right now I don't have a good solution if the strings
in code never stop changing for any period of time, but what has been
proposed above while well-meaning is unfortunately not workable.

We really need to keep our translators as happy as we can. People who
are literate in multiple languages are difficult to find, moreso those
who have technical vocabulary experience in both, and even moreso those
who'll give up their time to help us reach our mission goal of being the
ubiquitous Open Source Cloud Computing platform. We do need them to
achieve it.


Regards,


Tom

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Robert Collins]

On 25 February 2015 at 13:13, Jeremy Stanley  wrote:
> On 2015-02-24 11:27:05 + (+), Daniel P. Berrange wrote:
> [...]
>> It would be reasonable for the vulnerability team to take the decision
>> that they'll support fixes for master, and any branches that the stable
>> team decide to support.
> [...]
>
> Well, it's worth noting that the VMT doesn't even "support" (i.e.
> issue advisories for bugs in) master branches now, the exception
> being branchless projects where the bug appears in master prior to
> an existing release tag.
>
> But I think Thierry's earlier point is that as soon as you start
> marking _some_ releases as special (supported by VMT, stable maint,
> docs, translators...) then those become your new actual releases and
> the other interim releases become your milestones, and we're back to
> the current model again.

I don't think thats true actually. We'd still have a major smoothing
effect on work, which means less high peaks at release time and less
troughs at 'review direction' time and so on.

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Jeremy Stanley]

On 2015-02-24 11:27:05 + (+), Daniel P. Berrange wrote:
[...]
> It would be reasonable for the vulnerability team to take the decision
> that they'll support fixes for master, and any branches that the stable
> team decide to support.
[...]

Well, it's worth noting that the VMT doesn't even "support" (i.e.
issue advisories for bugs in) master branches now, the exception
being branchless projects where the bug appears in master prior to
an existing release tag.

But I think Thierry's earlier point is that as soon as you start
marking _some_ releases as special (supported by VMT, stable maint,
docs, translators...) then those become your new actual releases and
the other interim releases become your milestones, and we're back to
the current model again.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Robert Collins]

On 25 February 2015 at 00:05, Thierry Carrez  wrote:
> Daniel P. Berrange wrote:
>> [...]

>
> I think you're judging the cycle from the perspective of developers
> only.

You said that in the other thread, and I think its false. There is a
very good case to be made that the release cycle is directly
responsible for:
 - increased feature latency [when users get new shiny] - already
explained here.
 - decreased stability due to feature crush
 - decreased quality for the same reason - we don't put things in and
bake them before exposing, we get them in and since there is a long
gap before they can be consumed they're generally just enabled
immediately. Keystone's steps to address that are excellent, but made
harder by the gap between releases: there is social pressure to make
things graduate to support within each release.

>  6 months was not an arbitrary decision. Translations and
> documentation teams basically need a month of feature/string freeze in
> order to complete their work. Since we can't reasonably freeze one month
> every 2 months, we picked 6 months.

Do they need a month because 6 months worth of changes have been
aggregated? E.g. if they have 2 months of changes to deal with, do
they need 1.5 weeks?

Perhaps we could branch the code release in advance of docs and
translations, and announce the release when those artifacts are ready?

> It's also worth noting that we were on a 3-month cycle at the start of
> OpenStack. That was dropped after a cataclysmic release that managed the
> feat of (a) not having anything significant done, and (b) have out of
> date documentation and translations.

Oh! 
https://wiki.openstack.org/w/index.php?title=Obsolete:3MonthReleaseCycle&direction=prev&oldid=14016
appears to be the page about that, but my google-fu isn't turning up a
post-mortem of the C release which prompted the change. Perhaps some
old-hand like you can fill us in on the details :).

> Stable branches may have the luxury of skipping releases and designate a
> "stable" one from time to time (I reject the Linux comparison because
> the kernel is at a very different moment in software lifecycle). The
> trick being, making one release "special" is sure to recreate the peak
> issues you're trying to solve.
>
> I'm actually more concerned about the teams that don't have the luxury
> of skipping a release (like the vulnerability management team). Docs and
> translations, as mentioned above, will also have a hard time producing
> quality docs and translations every 2 months with a very short freeze
> period.

I agree that the vulnerability team will be rather more impacted by
this than regular project teams - I made a nod to that in the other
thread. More thought needed still :).


> I may appear defensive on my answers, but it's not my goal to defend the
> current system: it's just that most of those proposals generally ignore
> the diversity of the needs of the teams that make OpenStack possible, to
> focus on a particular set of contributors' woes. I'm trying to bring
> that wider perspective in -- the current system is a trade-off and the
> result of years of evolution, not an arbitrary historic choice that we
> can just change at will.

I agree that its not arbitrary and that changing it requires some
appropriate wide-spread consultation; OTOH the benefits of rolling and
higher frequency releases are really substantial: but we have to
backup the change with the appropriate engineering (such as reducing
our frictions that cause teams practicing CD to be weeks behind tip)
to make it feasible. My greatest concern about the proposals happening
now is that we may bite off more than we can chew. OTGH the reality is
that all the negative things multiply out, so we probably need to just
start somewhere and /do/ to give us the space to fix other things.

-Rob

-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Robert Collins]

On 24 February 2015 at 22:53, Daniel P. Berrange  wrote:
> I was writing this mail for the past few days, but the nova thread
> today prompted me to finish it off & send it :-)

++


> The first two observations strongly suggest that the choice of 6
> months as a cycle length is a fairly arbitrary decision that can be
> changed without unreasonable pain. The third observation suggests a
> much shorter cycle length would smooth out the bumps and lead to a
> more efficient & satisfying development process for all involved.

I'm very glad to see this being discussed (again :)). Any proposal
that reduces our cycle time is going to get my enthusiastic support.

...
> Upgrades & deprecation
> --
>
> It is common right now for projects to say upgrades are only
> supported between releases N-1 and N. ie to go from Icehouse
> to Kilo, you need to first deploy Juno. This is passable when
> you're talking 6 month gaps between cycles, but when there are
> 2 month gaps it is not reasonable to expect everyone to be
> moving fast enough to keep up with every release. If an
> organization's beurocracy means they can't deploy more often
> than every 12 months, forcing them to deploy the 5 intermediate
> releases to run upgrade scripts is quite unpleasant. We would
> likely have to look at officially allowing upgrades between
> any (N-3, N-2, N-1) to N. From a database POV this should not
> be hard, since the DB migration scripts don't have any built
> in assumptions about this. Similarly the versioned objects used
> by Nova are quite flexible in this regard too, as long as the
> compat code isn't deleted too soon.
>
> Deprecation warnings would need similar consideration. It would
> not be sufficient to deprecate in one release and delete in the
> next. We'd likely want to say that depecations last for a given
> time period rather than number of releases, eg 6 months. This
> could be easily handled by simply including the date of initial
> deprecation in the deprecation message. It would thus be clear
> when the feature will be removed to all involved.

I think a useful thing here is to consider online upgrades vs offline
upgrades. If we care to we could say that online upgrades are only
supported release to release, with offline upgrades being supported by
releases up to 8 months apart. The benefit of this would be to reduce
our test matrix: online upgrades are subject to much greater
interactions between concurrent processes, and its hard enough to
validate that N -> N+1 works with any deep confidence vs also checking
that N->N+2, N->N+3 also work: for a 6 month sliding window to match
the current thing, we need to allow upgrades from Feb through August:
a Feb->Apr
b Feb->Jun
c Feb->Aug
d Apr-> Jun
e Apr->Aug
f Jun->Aug

We'd need to be testing all the combinations leading to the branch a
patch is for, so changes to Aug would need c, e and f all tested.


Thats 3 times the overhead of supporting:
Feb->Apr and then
Apr->Jun and then
Jun->Aug
serially where we'd only be testing one combination at a time.

-Rob


-- 
Robert Collins 
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Sean Dague]

On 02/24/2015 03:21 PM, Joe Gordon wrote:
> 
> 
> On Tue, Feb 24, 2015 at 6:57 AM, Daniel P. Berrange  > wrote:
> 
> On Tue, Feb 24, 2015 at 08:50:45AM -0500, Sean Dague wrote:
> > On 02/24/2015 07:48 AM, Russell Bryant wrote:
> > > On 02/24/2015 12:54 PM, Daniel P. Berrange wrote:
> > >> On Tue, Feb 24, 2015 at 11:48:29AM +, Chris Dent wrote:
> > >>> On Tue, 24 Feb 2015, Daniel P. Berrange wrote:
> > >>>
> >  need to do more work. If this is so, then I don't think this
> is a blocker,
> >  it is just a sign that the project needs to focus on
> providing more resources
> >  to the teams impacted in that way.
> > >>>
> > >>> What are the mechanisms whereby the project provides more
> resources
> > >>> to teams?
> > >>
> > >> The technical committee and / or foundation board can highlight
> the need
> > >> for investment of resources in critical areas of the project,
> to either
> > >> the community members or vendors involved. As an example, this
> was done
> > >> successfully recently to increase involvement in maintaining
> the EC2
> > >> API support.  There are plenty of vendors involved in OpenStack
> which
> > >> have the ability to target resources, if they can learn where those
> > >> resources are best spent.
> > >
> > > Indeed ... and if horizontal teams are the ones hit the most by the
> > > extra work, each project should help with that burden.  For example,
> > > projects may need to take their responsibility for documentation
> more
> > > seriously and require documentation with features (content at
> least, not
> > > necessarily integration into the proper documentation deliverables)
> > > instead of assuming it magically gets written later.
> >
> > Right, and I think this actually hits at the most important part
> of the
> > discussion. The question of:
> >
> > 1) what would we need to do to make different release cadences viable?
> > 2) are those good things to do regardless of release cadence?
> >
> > The horizontal teams really can't function at different cadences. It
> > completely breaks any flow and planning at turns them even further
> into
> > firefighting because now everyone has crunch time at different times,
> > and the horizontal efforts are required to just play catch up. I know
> > what that future looks like, the horizontal teams dry up because
> no one
> > wants that job.
> >
> > Ok, so that being said, what we'd need to do is have horizontal teams
> > move to more of a self supporting model. So that the relevant content
> > for a project (docs, full stack tests, requirements, etc) all live
> > within that project itself, and aren't centrally synchronized.
> > Installation of projects needs to be fully isolated from each other so
> > that upgrading project A can be done independent of project B, as
> their
> > release cadences might all bit disparate. Basically, ever OpenStack
> > project needs to reabsorb the cross project efforts they've
> externalized.
> >
> > Then if project A decided to move off the coupled release, it's impact
> > to the rest would be minimal. These are robust components that
> stand on
> > their own, and work well with robust other components.
> >
> > Which... is basically the point of the big tent / new project
> governance
> > model. Decompose OpenStack from a giant blob of goo into Robust
> elements
> > that are more loosely coupled (so independently robust, and robust in
> > their interaction with others). Move the horizontal teams into
> > infrastructure vs. content roles, have projects own more of this
> content
> > themselves.
> >
> > But it is a long hard process. Devstack external plugins was implement
> > to support this kind of model, but having walked a bunch of different
> > teams through this (at all skill levels) there ends up being a lot of
> > work to get this right, and a lot of rethinking by teams that assumed
> > their interaction with full stack testing is something they'd get to
> > contribute once and have someone else maintain (instead of something
> > they now need dedicated watchful eye on).
> >
> > The amount of full stack configurations immediately goes beyond
> anywhere
> > near testable, so it requires more robust project testing to ensure
> > every exposed interface is more robust (i.e. the testing in pyramids
> > from https://review.openstack.org/#/c/150653/).
> >
> > And, I think the answer to #2 is: yes, this just makes it all better.
> >
> > So, honestly, I'm massively supportive of the end game. I've been
> > carving out the bits of this I can for the last six months. But I
> think
>  

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Joe Gordon]

On Tue, Feb 24, 2015 at 6:57 AM, Daniel P. Berrange 
wrote:

> On Tue, Feb 24, 2015 at 08:50:45AM -0500, Sean Dague wrote:
> > On 02/24/2015 07:48 AM, Russell Bryant wrote:
> > > On 02/24/2015 12:54 PM, Daniel P. Berrange wrote:
> > >> On Tue, Feb 24, 2015 at 11:48:29AM +, Chris Dent wrote:
> > >>> On Tue, 24 Feb 2015, Daniel P. Berrange wrote:
> > >>>
> >  need to do more work. If this is so, then I don't think this is a
> blocker,
> >  it is just a sign that the project needs to focus on providing more
> resources
> >  to the teams impacted in that way.
> > >>>
> > >>> What are the mechanisms whereby the project provides more resources
> > >>> to teams?
> > >>
> > >> The technical committee and / or foundation board can highlight the
> need
> > >> for investment of resources in critical areas of the project, to
> either
> > >> the community members or vendors involved. As an example, this was
> done
> > >> successfully recently to increase involvement in maintaining the EC2
> > >> API support.  There are plenty of vendors involved in OpenStack which
> > >> have the ability to target resources, if they can learn where those
> > >> resources are best spent.
> > >
> > > Indeed ... and if horizontal teams are the ones hit the most by the
> > > extra work, each project should help with that burden.  For example,
> > > projects may need to take their responsibility for documentation more
> > > seriously and require documentation with features (content at least,
> not
> > > necessarily integration into the proper documentation deliverables)
> > > instead of assuming it magically gets written later.
> >
> > Right, and I think this actually hits at the most important part of the
> > discussion. The question of:
> >
> > 1) what would we need to do to make different release cadences viable?
> > 2) are those good things to do regardless of release cadence?
> >
> > The horizontal teams really can't function at different cadences. It
> > completely breaks any flow and planning at turns them even further into
> > firefighting because now everyone has crunch time at different times,
> > and the horizontal efforts are required to just play catch up. I know
> > what that future looks like, the horizontal teams dry up because no one
> > wants that job.
> >
> > Ok, so that being said, what we'd need to do is have horizontal teams
> > move to more of a self supporting model. So that the relevant content
> > for a project (docs, full stack tests, requirements, etc) all live
> > within that project itself, and aren't centrally synchronized.
> > Installation of projects needs to be fully isolated from each other so
> > that upgrading project A can be done independent of project B, as their
> > release cadences might all bit disparate. Basically, ever OpenStack
> > project needs to reabsorb the cross project efforts they've externalized.
> >
> > Then if project A decided to move off the coupled release, it's impact
> > to the rest would be minimal. These are robust components that stand on
> > their own, and work well with robust other components.
> >
> > Which... is basically the point of the big tent / new project governance
> > model. Decompose OpenStack from a giant blob of goo into Robust elements
> > that are more loosely coupled (so independently robust, and robust in
> > their interaction with others). Move the horizontal teams into
> > infrastructure vs. content roles, have projects own more of this content
> > themselves.
> >
> > But it is a long hard process. Devstack external plugins was implement
> > to support this kind of model, but having walked a bunch of different
> > teams through this (at all skill levels) there ends up being a lot of
> > work to get this right, and a lot of rethinking by teams that assumed
> > their interaction with full stack testing is something they'd get to
> > contribute once and have someone else maintain (instead of something
> > they now need dedicated watchful eye on).
> >
> > The amount of full stack configurations immediately goes beyond anywhere
> > near testable, so it requires more robust project testing to ensure
> > every exposed interface is more robust (i.e. the testing in pyramids
> > from https://review.openstack.org/#/c/150653/).
> >
> > And, I think the answer to #2 is: yes, this just makes it all better.
> >
> > So, honestly, I'm massively supportive of the end game. I've been
> > carving out the bits of this I can for the last six months. But I think
> > the way we get there is to actually get the refactoring of the
> > horizontal efforts first.
>
> I pretty much fully agree that refactoring the horizontal efforts to
> distribute responsbility across the individual projects is the way
> forward. I don't think it needs to be a pre-requisite step for changing
> the release cycle. We can do both in parallel if we put our minds to
> it.
>
> My biggest fear is that we just keeping debating problems and alternatives,
> but continue to be too afraid of theore

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Sean Dague]

On 02/24/2015 12:33 PM, John Griffith wrote:
> 
> 
> On Tue, Feb 24, 2015 at 10:04 AM, David Kranz  > wrote:
> 
> On 02/24/2015 09:37 AM, Chris Dent wrote:
> 
> On Tue, 24 Feb 2015, Sean Dague wrote:
> 
> That also provides a very concrete answer to "will people
> show up".
> Because if they do, and we get this horizontal refactoring
> happening,
> then we get to the point of being able to change release
> cadences
> faster. If they don't, we remain with the existing system.
> Vs changing
> the system and hoping someone is going to run in and
> backfill the breaks.
> 
> 
> Isn't this the way of the world? People only put halon in the
> machine room after the fire.
> 
> I agree that "people showing up" is a real concern, but I also think
> that we shy away too much from the productive energy of stuff
> breaking. It's the breakage that shows where stuff isn't good
> enough.
> 
> [Flavio said]:
> 
> To this I'd also add that bug fixing is way easier when you have
> aligned releases for projects that are expected to be deployed
> together. It's easier to know what the impact of a change/bug is
> throughout the infrastructure.
> 
> 
> Can't this be interpreted as an excuse for making software which
> does not have a low surface area and a good API?
> 
> (Note I'm taking a relatively unrealistic position for sake of
> conversation.)
> 
> I'm not so sure about that. IMO, much of this goes back to the
> question of whether OpenStack services are APIs or implementations.
> This was debated with much heat at the Diablo summit (Hi Jay). I
> frequently have conversations where there is an issue about release
> X vs Y when it is really about api versions. Even if we say that we
> are about implementations as well as apis, we can start to organize
> our processes and code as if we were just apis. If each service had
> a well-defined, versioned, discoverable, well-tested api, then
> projects could follow their own release schedule, relying on distros
> or integrators to put the pieces together and verify the quality of
> the whole stack to the users. Such entities could still collaborate
> on that task, and still identify longer release cycles, using
> "stable branches". The upstream project could still test the latest
> released versions together. Some of these steps are now being taken
> to resolve gate issues and horizontal resource issues. Doing this
> would vastly increase agility but with some costs:
> 
> 1. The upstream project would likely have to give up on the worthy
> goal of providing an actual deployable stack that could be used as
> an alternative to AWS, etc. That saddens me, but for various
> reasons, including that we do no scale/performance testing on the
> upstream code, we are not achieving that goal anyway. The big tent
> proposals are also a move away from that goal.
> 
> 2. We would have to give up on incompatible api changes. But with
> the replacement of nova v3 with microversions we are already doing
> that. Massive adoption with release agility is simply incompatible
> with allowing incompatible api changes.
> 
> Most of this is just echoing what Jay said. I think this is the way
> any SOA would be designed. If we did this, and projects released
> frequently, would there be a reason for any one to be chasing master?
> 
>  -David
> 
> 
> 
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request@lists.__openstack.org?subject:__unsubscribe
> 
> http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev 
> 
> 
> 
> 
> ​Seems like some of the proposals around frequency (increasing in
> particular) just sort of move the bottlenecks around.  Honestly, I
> thought we were already on a path with some of the ideas that Sean and
> David (and indirectly Jay) proposed.  Get rid of the whole "coordinated
> release" altogether.  I think there should still be some sort of tagging
> or something at some interval that just says "here's a point in time
> collection that we call X".
> 
> Another proposal I think I've talked to some folks about is a true
> CI/Train model.  Cut out some of the artificial milestone deadlines etc,
> just keep rolling and what's ready at the release point is what's ready;
> you make a cut of what's there at that time and roll on.  Basically
> eliminate the feature freeze and other com

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[John Griffith]

On Tue, Feb 24, 2015 at 10:04 AM, David Kranz  wrote:

> On 02/24/2015 09:37 AM, Chris Dent wrote:
>
>> On Tue, 24 Feb 2015, Sean Dague wrote:
>>
>>  That also provides a very concrete answer to "will people show up".
>>> Because if they do, and we get this horizontal refactoring happening,
>>> then we get to the point of being able to change release cadences
>>> faster. If they don't, we remain with the existing system. Vs changing
>>> the system and hoping someone is going to run in and backfill the breaks.
>>>
>>
>> Isn't this the way of the world? People only put halon in the
>> machine room after the fire.
>>
>> I agree that "people showing up" is a real concern, but I also think
>> that we shy away too much from the productive energy of stuff
>> breaking. It's the breakage that shows where stuff isn't good
>> enough.
>>
>> [Flavio said]:
>>
>>> To this I'd also add that bug fixing is way easier when you have
>>> aligned releases for projects that are expected to be deployed
>>> together. It's easier to know what the impact of a change/bug is
>>> throughout the infrastructure.
>>>
>>
>> Can't this be interpreted as an excuse for making software which
>> does not have a low surface area and a good API?
>>
>> (Note I'm taking a relatively unrealistic position for sake of
>> conversation.)
>>
> I'm not so sure about that. IMO, much of this goes back to the question of
> whether OpenStack services are APIs or implementations. This was debated
> with much heat at the Diablo summit (Hi Jay). I frequently have
> conversations where there is an issue about release X vs Y when it is
> really about api versions. Even if we say that we are about implementations
> as well as apis, we can start to organize our processes and code as if we
> were just apis. If each service had a well-defined, versioned,
> discoverable, well-tested api, then projects could follow their own release
> schedule, relying on distros or integrators to put the pieces together and
> verify the quality of the whole stack to the users. Such entities could
> still collaborate on that task, and still identify longer release cycles,
> using "stable branches". The upstream project could still test the latest
> released versions together. Some of these steps are now being taken to
> resolve gate issues and horizontal resource issues. Doing this would vastly
> increase agility but with some costs:
>
> 1. The upstream project would likely have to give up on the worthy goal of
> providing an actual deployable stack that could be used as an alternative
> to AWS, etc. That saddens me, but for various reasons, including that we do
> no scale/performance testing on the upstream code, we are not achieving
> that goal anyway. The big tent proposals are also a move away from that
> goal.
>
> 2. We would have to give up on incompatible api changes. But with the
> replacement of nova v3 with microversions we are already doing that.
> Massive adoption with release agility is simply incompatible with allowing
> incompatible api changes.
>
> Most of this is just echoing what Jay said. I think this is the way any
> SOA would be designed. If we did this, and projects released frequently,
> would there be a reason for any one to be chasing master?
>
>  -David
>
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


​Seems like some of the proposals around frequency (increasing in
particular) just sort of move the bottlenecks around.  Honestly, I thought
we were already on a path with some of the ideas that Sean and David (and
indirectly Jay) proposed.  Get rid of the whole "coordinated release"
altogether.  I think there should still be some sort of tagging or
something at some interval that just says "here's a point in time
collection that we call X".

Another proposal I think I've talked to some folks about is a true CI/Train
model.  Cut out some of the artificial milestone deadlines etc, just keep
rolling and what's ready at the release point is what's ready; you make a
cut of what's there at that time and roll on.  Basically eliminate the
feature freeze and other components and "hopefully" keep feature commit
distributed.  There are certainly all sorts of gotchas here, but I don't
think it's very interesting to most so I won't go into a bunch of theory on
it.

Regardless, I do think that no matter the direction we all seem to be of
the opinion that we need to move more towards projects being responsible
for more of the horizontal functions (as Sean put it).  Personally I think
this is a great direction for a number of reasons, and I also think that it
might force us to be better about our API's and requirements than we have
been in the past.
__
OpenStack Dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[David Kranz]

On 02/24/2015 09:37 AM, Chris Dent wrote:

On Tue, 24 Feb 2015, Sean Dague wrote:


That also provides a very concrete answer to "will people show up".
Because if they do, and we get this horizontal refactoring happening,
then we get to the point of being able to change release cadences
faster. If they don't, we remain with the existing system. Vs changing
the system and hoping someone is going to run in and backfill the 
breaks.


Isn't this the way of the world? People only put halon in the
machine room after the fire.

I agree that "people showing up" is a real concern, but I also think
that we shy away too much from the productive energy of stuff
breaking. It's the breakage that shows where stuff isn't good
enough.

[Flavio said]:

To this I'd also add that bug fixing is way easier when you have
aligned releases for projects that are expected to be deployed
together. It's easier to know what the impact of a change/bug is
throughout the infrastructure.


Can't this be interpreted as an excuse for making software which
does not have a low surface area and a good API?

(Note I'm taking a relatively unrealistic position for sake of
conversation.)
I'm not so sure about that. IMO, much of this goes back to the question 
of whether OpenStack services are APIs or implementations. This was 
debated with much heat at the Diablo summit (Hi Jay). I frequently have 
conversations where there is an issue about release X vs Y when it is 
really about api versions. Even if we say that we are about 
implementations as well as apis, we can start to organize our processes 
and code as if we were just apis. If each service had a well-defined, 
versioned, discoverable, well-tested api, then projects could follow 
their own release schedule, relying on distros or integrators to put the 
pieces together and verify the quality of the whole stack to the users. 
Such entities could still collaborate on that task, and still identify 
longer release cycles, using "stable branches". The upstream project 
could still test the latest released versions together. Some of these 
steps are now being taken to resolve gate issues and horizontal resource 
issues. Doing this would vastly increase agility but with some costs:


1. The upstream project would likely have to give up on the worthy goal 
of providing an actual deployable stack that could be used as an 
alternative to AWS, etc. That saddens me, but for various reasons, 
including that we do no scale/performance testing on the upstream code, 
we are not achieving that goal anyway. The big tent proposals are also a 
move away from that goal.


2. We would have to give up on incompatible api changes. But with the 
replacement of nova v3 with microversions we are already doing that. 
Massive adoption with release agility is simply incompatible with 
allowing incompatible api changes.


Most of this is just echoing what Jay said. I think this is the way any 
SOA would be designed. If we did this, and projects released frequently, 
would there be a reason for any one to be chasing master?


 -David


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Daniel P. Berrange]

On Tue, Feb 24, 2015 at 08:50:45AM -0500, Sean Dague wrote:
> On 02/24/2015 07:48 AM, Russell Bryant wrote:
> > On 02/24/2015 12:54 PM, Daniel P. Berrange wrote:
> >> On Tue, Feb 24, 2015 at 11:48:29AM +, Chris Dent wrote:
> >>> On Tue, 24 Feb 2015, Daniel P. Berrange wrote:
> >>>
>  need to do more work. If this is so, then I don't think this is a 
>  blocker,
>  it is just a sign that the project needs to focus on providing more 
>  resources
>  to the teams impacted in that way.
> >>>
> >>> What are the mechanisms whereby the project provides more resources
> >>> to teams?
> >>
> >> The technical committee and / or foundation board can highlight the need
> >> for investment of resources in critical areas of the project, to either
> >> the community members or vendors involved. As an example, this was done
> >> successfully recently to increase involvement in maintaining the EC2
> >> API support.  There are plenty of vendors involved in OpenStack which
> >> have the ability to target resources, if they can learn where those
> >> resources are best spent.
> > 
> > Indeed ... and if horizontal teams are the ones hit the most by the
> > extra work, each project should help with that burden.  For example,
> > projects may need to take their responsibility for documentation more
> > seriously and require documentation with features (content at least, not
> > necessarily integration into the proper documentation deliverables)
> > instead of assuming it magically gets written later.
> 
> Right, and I think this actually hits at the most important part of the
> discussion. The question of:
> 
> 1) what would we need to do to make different release cadences viable?
> 2) are those good things to do regardless of release cadence?
> 
> The horizontal teams really can't function at different cadences. It
> completely breaks any flow and planning at turns them even further into
> firefighting because now everyone has crunch time at different times,
> and the horizontal efforts are required to just play catch up. I know
> what that future looks like, the horizontal teams dry up because no one
> wants that job.
> 
> Ok, so that being said, what we'd need to do is have horizontal teams
> move to more of a self supporting model. So that the relevant content
> for a project (docs, full stack tests, requirements, etc) all live
> within that project itself, and aren't centrally synchronized.
> Installation of projects needs to be fully isolated from each other so
> that upgrading project A can be done independent of project B, as their
> release cadences might all bit disparate. Basically, ever OpenStack
> project needs to reabsorb the cross project efforts they've externalized.
> 
> Then if project A decided to move off the coupled release, it's impact
> to the rest would be minimal. These are robust components that stand on
> their own, and work well with robust other components.
> 
> Which... is basically the point of the big tent / new project governance
> model. Decompose OpenStack from a giant blob of goo into Robust elements
> that are more loosely coupled (so independently robust, and robust in
> their interaction with others). Move the horizontal teams into
> infrastructure vs. content roles, have projects own more of this content
> themselves.
> 
> But it is a long hard process. Devstack external plugins was implement
> to support this kind of model, but having walked a bunch of different
> teams through this (at all skill levels) there ends up being a lot of
> work to get this right, and a lot of rethinking by teams that assumed
> their interaction with full stack testing is something they'd get to
> contribute once and have someone else maintain (instead of something
> they now need dedicated watchful eye on).
> 
> The amount of full stack configurations immediately goes beyond anywhere
> near testable, so it requires more robust project testing to ensure
> every exposed interface is more robust (i.e. the testing in pyramids
> from https://review.openstack.org/#/c/150653/).
> 
> And, I think the answer to #2 is: yes, this just makes it all better.
> 
> So, honestly, I'm massively supportive of the end game. I've been
> carving out the bits of this I can for the last six months. But I think
> the way we get there is to actually get the refactoring of the
> horizontal efforts first.

I pretty much fully agree that refactoring the horizontal efforts to
distribute responsbility across the individual projects is the way
forward. I don't think it needs to be a pre-requisite step for changing
the release cycle. We can do both in parallel if we put our minds to
it.

My biggest fear is that we just keeping debating problems and alternatives,
but continue to be too afraid of theoretical problems, to actually take the
risk of effecting meaningful improvemnts in the operation of the project.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://li

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Chris Dent]

On Tue, 24 Feb 2015, Sean Dague wrote:


That also provides a very concrete answer to "will people show up".
Because if they do, and we get this horizontal refactoring happening,
then we get to the point of being able to change release cadences
faster. If they don't, we remain with the existing system. Vs changing
the system and hoping someone is going to run in and backfill the breaks.


Isn't this the way of the world? People only put halon in the
machine room after the fire.

I agree that "people showing up" is a real concern, but I also think
that we shy away too much from the productive energy of stuff
breaking. It's the breakage that shows where stuff isn't good
enough.

[Flavio said]:

To this I'd also add that bug fixing is way easier when you have
aligned releases for projects that are expected to be deployed
together. It's easier to know what the impact of a change/bug is
throughout the infrastructure.


Can't this be interpreted as an excuse for making software which
does not have a low surface area and a good API?

(Note I'm taking a relatively unrealistic position for sake of
conversation.)

--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Flavio Percoco]

On 24/02/15 11:02 +, Daniel P. Berrange wrote:

On Tue, Feb 24, 2015 at 10:44:57AM +, Chris Dent wrote:

On Tue, 24 Feb 2015, Daniel P. Berrange wrote:

>I was writing this mail for the past few days, but the nova thread
>today prompted me to finish it off & send it :-)

Thanks for doing this. I think you're probably right that the current
release cycle has many negative impacts on the development process and
deserve at least some critical thinking if not outright changing.

Thanks especially for listing some expected questions (with answers).

One additional question I'd like to see answered in some depth is:
Why have unified release schedules? That is, why should Nova and Glance
or anything else release on the same schedule as any other
OpenStack-related project?

Disentangling the release cycles might lead to stronger encapsulation
of, and stronger contracts between, the projects. It might also lead to
a total mess.


For peripheral projects I don't think co-ordinate release cycle is needed,
but for the core projects I think it is helpful in general to have some
co-ordination of releases. It allows marketing to more effectively promote
the new project releases, it helps when landing features that span across
projects to know they'll be available to users at the same time in general
and minimize burden on devs & users to remember many different dates. It
is hard enough remembering the dates for our coordinated release cycle,
let alone dates for 30 different project cycles. IME predictable dates is
a really important & useful thing to have for a planning POV. This is why
I suggested, we do a 2 month cycle, with a strict date of 1st of the month
in Feb, Apr, Jun, Aug, Oct, Dec.


To this I'd also add that bug fixing is way easier when you have
aligned releases for projects that are expected to be deployed
together. It's easier to know what the impact of a change/bug is
throughout the infrastructure.

Flavio



Regards,
Daniel
--
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
@flaper87
Flavio Percoco


pgpFaVXWQm9Ii.pgp
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Russell Bryant]

On 02/24/2015 12:54 PM, Daniel P. Berrange wrote:
> On Tue, Feb 24, 2015 at 11:48:29AM +, Chris Dent wrote:
>> On Tue, 24 Feb 2015, Daniel P. Berrange wrote:
>>
>>> need to do more work. If this is so, then I don't think this is a blocker,
>>> it is just a sign that the project needs to focus on providing more 
>>> resources
>>> to the teams impacted in that way.
>>
>> What are the mechanisms whereby the project provides more resources
>> to teams?
> 
> The technical committee and / or foundation board can highlight the need
> for investment of resources in critical areas of the project, to either
> the community members or vendors involved. As an example, this was done
> successfully recently to increase involvement in maintaining the EC2
> API support.  There are plenty of vendors involved in OpenStack which
> have the ability to target resources, if they can learn where those
> resources are best spent.

Indeed ... and if horizontal teams are the ones hit the most by the
extra work, each project should help with that burden.  For example,
projects may need to take their responsibility for documentation more
seriously and require documentation with features (content at least, not
necessarily integration into the proper documentation deliverables)
instead of assuming it magically gets written later.

-- 
Russell Bryant

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Russell Bryant]

On 02/24/2015 01:28 PM, Kashyap Chamarthy wrote:
> On Tue, Feb 24, 2015 at 11:54:31AM +, Daniel P. Berrange wrote:
>> On Tue, Feb 24, 2015 at 11:48:29AM +, Chris Dent wrote:
>>> On Tue, 24 Feb 2015, Daniel P. Berrange wrote:
>>>
 need to do more work. If this is so, then I don't think this is a blocker,
 it is just a sign that the project needs to focus on providing more 
 resources
 to the teams impacted in that way.
>>>
>>> What are the mechanisms whereby the project provides more resources
>>> to teams?
> 
> Along with the below, if push comes to shove, OpenStack Foundation could
> probably try a milder variant (obviously, not all activities can be
> categorized as 'critical path') of Linux Foundation's "Critical
> Infrastructure Protection Initiative"[1] to fund certain project
> activities in need.

The OpenStack Foundation effectively already does this.  In particular,
the Foundation is helping fund critical horizontal efforts like release
management, infrastructure, and community management.

-- 
Russell Bryant

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Kashyap Chamarthy]

On Tue, Feb 24, 2015 at 11:54:31AM +, Daniel P. Berrange wrote:
> On Tue, Feb 24, 2015 at 11:48:29AM +, Chris Dent wrote:
> > On Tue, 24 Feb 2015, Daniel P. Berrange wrote:
> > 
> > >need to do more work. If this is so, then I don't think this is a blocker,
> > >it is just a sign that the project needs to focus on providing more 
> > >resources
> > >to the teams impacted in that way.
> > 
> > What are the mechanisms whereby the project provides more resources
> > to teams?

Along with the below, if push comes to shove, OpenStack Foundation could
probably try a milder variant (obviously, not all activities can be
categorized as 'critical path') of Linux Foundation's "Critical
Infrastructure Protection Initiative"[1] to fund certain project
activities in need.
 
> The technical committee and / or foundation board can highlight the
> need for investment of resources in critical areas of the project, to
> either the community members or vendors involved. As an example, this
> was done successfully recently to increase involvement in maintaining
> the EC2 API support.  There are plenty of vendors involved in
> OpenStack which have the ability to target resources, if they can
> learn where those resources are best spent.
 

[1] http://www.linuxfoundation.org/programs/core-infrastructure-initiative

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Daniel P. Berrange]

On Tue, Feb 24, 2015 at 11:48:29AM +, Chris Dent wrote:
> On Tue, 24 Feb 2015, Daniel P. Berrange wrote:
> 
> >need to do more work. If this is so, then I don't think this is a blocker,
> >it is just a sign that the project needs to focus on providing more resources
> >to the teams impacted in that way.
> 
> What are the mechanisms whereby the project provides more resources
> to teams?

The technical committee and / or foundation board can highlight the need
for investment of resources in critical areas of the project, to either
the community members or vendors involved. As an example, this was done
successfully recently to increase involvement in maintaining the EC2
API support.  There are plenty of vendors involved in OpenStack which
have the ability to target resources, if they can learn where those
resources are best spent.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Chris Dent]

On Tue, 24 Feb 2015, Daniel P. Berrange wrote:


need to do more work. If this is so, then I don't think this is a blocker,
it is just a sign that the project needs to focus on providing more resources
to the teams impacted in that way.


What are the mechanisms whereby the project provides more resources
to teams?

--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

[Chris Dent]

On Tue, 24 Feb 2015, Daniel P. Berrange wrote:


I was writing this mail for the past few days, but the nova thread
today prompted me to finish it off & send it :-)


Thanks for doing this. I think you're probably right that the current
release cycle has many negative impacts on the development process and
deserve at least some critical thinking if not outright changing.

Thanks especially for listing some expected questions (with answers).

One additional question I'd like to see answered in some depth is:
Why have unified release schedules? That is, why should Nova and Glance
or anything else release on the same schedule as any other
OpenStack-related project?

Disentangling the release cycles might lead to stronger encapsulation
of, and stronger contracts between, the projects. It might also lead to
a total mess.

Thanks.
--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev