Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-03-04 Thread Clint Byrum
Excerpts from Thierry Carrez's message of 2015-03-04 02:19:48 -0800:
 James Bottomley wrote:
  On Tue, 2015-03-03 at 11:59 +0100, Thierry Carrez wrote:
  James Bottomley wrote:
  Actually, this is possible: look at Linux, it freezes for 10 weeks of a
  12 month release cycle (or 6 weeks of an 8 week one).  More on this
  below.
 
  I'd be careful with comparisons with the Linux kernel. First it's a
  single bit of software, not a collection of interconnected projects.
  
  Well, we do have interconnection: the kernel on it's own doesn't do
  anything without a userspace.  The theory was that we didn't have to be
  like BSD (coupled user space and kernel) and we could rely on others
  (principally the GNU project in the early days) to provide the userspace
  and that we could decouple kernel development from the userspace
  releases.  Threading models were, I think, the biggest challenges to
  this assumption, but we survived.
 
 Right. My point there is that you only release one thing. We release a
 lot more pieces. There is (was?) downstream value in coordinating those
 releases, which is a factor in our ability to do it more often than
 twice a year.
 

I think the value of coordinated releases has been agreed upon for a
long time. This thread is more about the cost, don't you think?

  Second it's at a very different evolution/maturity point (20 years old
  vs. 0-4 years old for OpenStack projects).
  
  Yes, but I thought I covered this in the email: you can see that at the
  4 year point in its lifecycle, the kernel was behaving very differently
  (and in fact more similar to OpenStack).  The question I thought was
  still valid is whether anything was learnable from the way the kernel
  evolved later.  I think the key issue, which you seem to have in
  OpenStack is that the separate develop/stabilise phases caused
  frustration to build up in our system which (nine years later) led the
  kernel to adopt the main branch stabilisation with overlapping subsystem
  development cycle.
 
 I agree with you: the evolution the kernel went through is almost a
 natural law, and I know we won't stay in the current model forever. I'm
 just not sure we have reached the level of general stability that makes
 it possible to change *just now*. I welcome brainstorming and discussion
 on future evolutions, though, and intend to lead a cross-project session
 discussion on that in Vancouver.
 

I don't believe that the kernel reached maturity as a point of
eventuality. Just like humans aren't going to jump across the grand
canyon no matter how strong they get, it will take a concerted effort
that may put other goals on hold to build a bridge. With the kernel
there was a clear moment where leadership had tried a few things and
then just had to make it clear that all the code goes in one place, but
instability would not be tolerated. They crossed that chasm, and while
there have been chaotic branches and ruffled feathers, once everybody
got over the paradox, it's been business as usual since then with the
model James describes.

I think the less mature a project is, the wider that chasm is, but I
don't think it's ever going to be an easy thing to do. Since we don't
have a dictator to force us to cross the chasm, we should really think
about planning for the crossing ASAP.

   Finally it sits at a
  different layer, so there is less need for documentation/translations to
  be shipped with the software release.
  
  It's certainly a lot less than you, but we have the entire system call
  man pages.  It's an official project of the kernel:
  
  https://www.kernel.org/doc/man-pages/
  
  And we maintain translations for it
  
  https://www.kernel.org/doc/man-pages/translations.html
 
 By translations I meant strings in the software itself, not doc
 translations. We don't translate docs upstream either :) I guess we
 could drop those (and/or downstream them in a way) if that was the last
 thing holding up adding more agility.
 
 So in summary, yes we can (and do) learn from kernel history, but those
 projects are sufficiently different that the precise timeframes and
 numbers can't really be compared. Apples and oranges are both fruits
 which mature (and rot if left unchecked), but they evolve at different
 speeds :)
 

I'm not super excited about being an apple or an orange, since neither
are sentient and thus cannot collaborate on a better existence than
rotting.

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-03-04 Thread Steve Gordon
- Original Message -
 From: Thierry Carrez thie...@openstack.org
 To: James Bottomley james.bottom...@hansenpartnership.com
 
  It's certainly a lot less than you, but we have the entire system
  call
  man pages.  It's an official project of the kernel:
  
  https://www.kernel.org/doc/man-pages/
  
  And we maintain translations for it
  
  https://www.kernel.org/doc/man-pages/translations.html
 
 By translations I meant strings in the software itself, not doc
 translations. We don't translate docs upstream either :) I guess we
 could drop those (and/or downstream them in a way) if that was the
 last
 thing holding up adding more agility.

There is actually a group of contributors working on translation of 
documentation and translations in various stages of completeness are available 
at docs.openstack.org (hit the more releases and languages drop down to find 
them). The challenge for the translators of course is they are trying to hit a 
target that is moving not just until the release but often well beyond it as 
the documentation team themselves try to catch up.

-Steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-03-04 Thread James Bottomley
On Wed, 2015-03-04 at 11:19 +0100, Thierry Carrez wrote:
 James Bottomley wrote:
  On Tue, 2015-03-03 at 11:59 +0100, Thierry Carrez wrote:
  Second it's at a very different evolution/maturity point (20 years old
  vs. 0-4 years old for OpenStack projects).
  
  Yes, but I thought I covered this in the email: you can see that at the
  4 year point in its lifecycle, the kernel was behaving very differently
  (and in fact more similar to OpenStack).  The question I thought was
  still valid is whether anything was learnable from the way the kernel
  evolved later.  I think the key issue, which you seem to have in
  OpenStack is that the separate develop/stabilise phases caused
  frustration to build up in our system which (nine years later) led the
  kernel to adopt the main branch stabilisation with overlapping subsystem
  development cycle.
 
 I agree with you: the evolution the kernel went through is almost a
 natural law, and I know we won't stay in the current model forever. I'm
 just not sure we have reached the level of general stability that makes
 it possible to change *just now*. I welcome brainstorming and discussion
 on future evolutions, though, and intend to lead a cross-project session
 discussion on that in Vancouver.

OK, I'll be in Vancouver, so happy to come and give input from
participating in the kernel process (for a bit longer than I care to
admit to ...).

One interesting thing might be to try and work out where roughly
OpenStack is on the project trajectory.  It's progressing much more
rapidly than the kernel (by 4 years in, the kernel didn't even have
source control), so the release crisis it took the kernel 12 years to
reach might be a bit closer than people imagine.

James


James



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-03-04 Thread Thierry Carrez
James Bottomley wrote:
 On Tue, 2015-03-03 at 11:59 +0100, Thierry Carrez wrote:
 James Bottomley wrote:
 Actually, this is possible: look at Linux, it freezes for 10 weeks of a
 12 month release cycle (or 6 weeks of an 8 week one).  More on this
 below.

 I'd be careful with comparisons with the Linux kernel. First it's a
 single bit of software, not a collection of interconnected projects.
 
 Well, we do have interconnection: the kernel on it's own doesn't do
 anything without a userspace.  The theory was that we didn't have to be
 like BSD (coupled user space and kernel) and we could rely on others
 (principally the GNU project in the early days) to provide the userspace
 and that we could decouple kernel development from the userspace
 releases.  Threading models were, I think, the biggest challenges to
 this assumption, but we survived.

Right. My point there is that you only release one thing. We release a
lot more pieces. There is (was?) downstream value in coordinating those
releases, which is a factor in our ability to do it more often than
twice a year.

 Second it's at a very different evolution/maturity point (20 years old
 vs. 0-4 years old for OpenStack projects).
 
 Yes, but I thought I covered this in the email: you can see that at the
 4 year point in its lifecycle, the kernel was behaving very differently
 (and in fact more similar to OpenStack).  The question I thought was
 still valid is whether anything was learnable from the way the kernel
 evolved later.  I think the key issue, which you seem to have in
 OpenStack is that the separate develop/stabilise phases caused
 frustration to build up in our system which (nine years later) led the
 kernel to adopt the main branch stabilisation with overlapping subsystem
 development cycle.

I agree with you: the evolution the kernel went through is almost a
natural law, and I know we won't stay in the current model forever. I'm
just not sure we have reached the level of general stability that makes
it possible to change *just now*. I welcome brainstorming and discussion
on future evolutions, though, and intend to lead a cross-project session
discussion on that in Vancouver.

  Finally it sits at a
 different layer, so there is less need for documentation/translations to
 be shipped with the software release.
 
 It's certainly a lot less than you, but we have the entire system call
 man pages.  It's an official project of the kernel:
 
 https://www.kernel.org/doc/man-pages/
 
 And we maintain translations for it
 
 https://www.kernel.org/doc/man-pages/translations.html

By translations I meant strings in the software itself, not doc
translations. We don't translate docs upstream either :) I guess we
could drop those (and/or downstream them in a way) if that was the last
thing holding up adding more agility.

So in summary, yes we can (and do) learn from kernel history, but those
projects are sufficiently different that the precise timeframes and
numbers can't really be compared. Apples and oranges are both fruits
which mature (and rot if left unchecked), but they evolve at different
speeds :)

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-03-03 Thread Thierry Carrez
James Bottomley wrote:
 Actually, this is possible: look at Linux, it freezes for 10 weeks of a
 12 month release cycle (or 6 weeks of an 8 week one).  More on this
 below.

I'd be careful with comparisons with the Linux kernel. First it's a
single bit of software, not a collection of interconnected projects.
Second it's at a very different evolution/maturity point (20 years old
vs. 0-4 years old for OpenStack projects). Finally it sits at a
different layer, so there is less need for documentation/translations to
be shipped with the software release.

The only comparable project in terms of evolution/maturity in the
OpenStack world would be Swift, and it happily produces releases every
~2months with a 1-week stabilisation period.

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-03-03 Thread Chris Dent

On Tue, 3 Mar 2015, Thierry Carrez wrote:


I'd be careful with comparisons with the Linux kernel. First it's a
single bit of software, not a collection of interconnected projects.
Second it's at a very different evolution/maturity point (20 years old
vs. 0-4 years old for OpenStack projects). Finally it sits at a
different layer, so there is less need for documentation/translations to
be shipped with the software release.


I think this is probably trampled ground, but is there some history on
why the OpenStack projects chose to own documentation, translation and
the concept of shipping the release instead of ceding that to the
downstream?

I'm not suggesting it should be (or should have been) ceded, just
wondering how it ended up that way.

There seems to be energy oriented toward centralization, marketing,
consolidation, arrogation etc. that might otherwise be spent on making
code that can later be refined by the people who want to make
money with it.

At the moment there is a strong coupling between the projects and the
product. Is that the best arrangement or are there others that might
be worth considering?

--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-03-03 Thread James Bottomley
On Tue, 2015-03-03 at 11:59 +0100, Thierry Carrez wrote:
 James Bottomley wrote:
  Actually, this is possible: look at Linux, it freezes for 10 weeks of a
  12 month release cycle (or 6 weeks of an 8 week one).  More on this
  below.
 
 I'd be careful with comparisons with the Linux kernel. First it's a
 single bit of software, not a collection of interconnected projects.

Well, we do have interconnection: the kernel on it's own doesn't do
anything without a userspace.  The theory was that we didn't have to be
like BSD (coupled user space and kernel) and we could rely on others
(principally the GNU project in the early days) to provide the userspace
and that we could decouple kernel development from the userspace
releases.  Threading models were, I think, the biggest challenges to
this assumption, but we survived.

 Second it's at a very different evolution/maturity point (20 years old
 vs. 0-4 years old for OpenStack projects).

Yes, but I thought I covered this in the email: you can see that at the
4 year point in its lifecycle, the kernel was behaving very differently
(and in fact more similar to OpenStack).  The question I thought was
still valid is whether anything was learnable from the way the kernel
evolved later.  I think the key issue, which you seem to have in
OpenStack is that the separate develop/stabilise phases caused
frustration to build up in our system which (nine years later) led the
kernel to adopt the main branch stabilisation with overlapping subsystem
development cycle.

  Finally it sits at a
 different layer, so there is less need for documentation/translations to
 be shipped with the software release.

It's certainly a lot less than you, but we have the entire system call
man pages.  It's an official project of the kernel:

https://www.kernel.org/doc/man-pages/

And we maintain translations for it

https://www.kernel.org/doc/man-pages/translations.html

The main difference is that our translation projects are not integrated
into our releases.  They're done mostly by the downstream (debian if you
look: the rest of the distros do invest in translations but they don't
make the upstream effort with them), so we don't specify what
translations we have, we allow interested parties to choose to help.

I would argue that the downstream is the best place to manage
translations because they decide which markets are important or
interesting, but if you still want to control this at the upstream end,
with Daniel's proposal, it would mean you only really need translations
for stable releases.

James

 The only comparable project in terms of evolution/maturity in the
 OpenStack world would be Swift, and it happily produces releases every
 ~2months with a 1-week stabilisation period.




__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-03-02 Thread Kyle Mestery
On Mon, Mar 2, 2015 at 11:59 AM, Kyle Mestery mest...@mestery.com wrote:

 On Mon, Mar 2, 2015 at 9:57 AM, Ihar Hrachyshka ihrac...@redhat.com
 wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 Hi Daniel,

 thanks for a clear write-up of the matter and food for thought.

 I think the idea of having more smooth development mode that would not
 make people to wait for 6+ months to release a new feature is great.

 ++


 It's insane to expect that feature priorities won't ever slightly
 shift in the next 6 months. Having a limited list of features targeted
 for the next 6 months is prone to mistakes, since people behind some
 of approved features may need to postpone the effort for any type of
 reasons (personal, job rush, bad resource allocation, ...), and we
 then end up with approved specs with no actual code drops, using
 review 'slots' that would better be spent for other features that were
 not that lucky to get a rubber stamp during spec phase. Prior resource
 allocation would probably work somehow if we were working for the same
 company that would define priorities to us, but it's not the case.

 It should be noted that even though Nova is using slots for reviews,
 Neutron is not. I've found that it's hard to try and slot people in to
 review specific things. During Juno I tried this for Neutron, and it failed
 miserably. For Kilo in Neutron, we're not using slots but instead I've
 tried to highlight the approved specs of Essential and High priority
 for review by all reviews, core and non-core included. It's gone ok, but
 the reality is you can't force people to review things. There are steps
 submitters can take to try and get timely review (lots of small, easy to
 digest patches, quick turnaround of comments, engagement in IRC and ML,
 etc.).


It was pointed out to me that nova is NOT using slots. Apologies for my
misunderstanding here.

Clearly, this thread has elicited a lot of strong thoughts and emotions. I
hope we can all use this energy to figure out a good solution and a way
forward for the issues presented here.



 Anecdotally, in neutron, we have two Essential blueprints for Kilo,
 and there are no code drops or patches in review for any of those, so
 I would expect them to fail to merge. At the same time, I will need to
 wait for the end of Kilo to consider adding support for guru reports
 to the project. Or in oslo world, I will need to wait for Liberty to
 introduce features in oslo.policy that are needed by neutron to switch
 to it, etc.

 To be fair, there are many reasons those to Essential BPs do not have
 code. I still expect the Pecan focused to have code, but I already moved
 the Plugin one out of Kilo at this point because there was no chance the
 code would land.

 But I get your point here. I think this thread has highlighted the fact
 that the BP/spec process worked to some extent, but for small things, the
 core reviewer team should have the ability to say Yes, we can easily merge
 that, lets approve that spec even if it's late in the cycle.


 Another problem is that currently milestones are used merely for
 targeting bugs and features, but no one really cares about whether the
 actual milestone shipment actually works. Again, a story from neutron
 world: Kilo-1 was shipped in the middle of advanced services split,
 with some essential patches around distutils setup missing (no proper
 migration plan applied, conflicting config files in neutron and *aas
 repos, etc.)

 This is true, the milestone release matter but are not given enough focus
 and they release (for the most part) irregardless of items in them, given
 they are not long-lived, etc.

 So I'm all for reforms around processes we apply.

 If there's one thing OpenStack is good at, it's change.


 That said, I don't believe the real problem here is that we don't
 generate project tarballs frequently enough.

 Major problems I see as critical to tackle in our dev process are:

 - - enforced spec/dev mode. Solution: allow to propose (and approve) a
 reasonable spec at any moment in the cycle; allow code drops for
 approved specs at any moment in the cycle (except pre-release
 stabilization time); stop targeting specs: if it's sane, it's probably
 sane N+2 cycle from now too.

 I'd say this is fine for specs that are small and people agree can easily
 be merged. I'd say this is not the case for large features near the end of
 the release which are unlikely to gather enough review momentum to actually
 merge.


 - - core team rubber stamping a random set of specs and putting -2 on
 all other specs due to project priorities. Solution: stop pretending
 core team (or core drivers) can reasonably estimate review and coding
 resources for the next cycle. Instead allows community to decide
 what's worth the effort by approving all technically reasonable specs
 and allowing everyone to invest time and effort in specs (s)he seems
 worth it.

 If you're referring to Neutron here, I think you fail to estimate the
 

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-03-02 Thread Joe Gordon
On Mon, Mar 2, 2015 at 9:59 AM, Kyle Mestery mest...@mestery.com wrote:

 On Mon, Mar 2, 2015 at 9:57 AM, Ihar Hrachyshka ihrac...@redhat.com
 wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 Hi Daniel,

 thanks for a clear write-up of the matter and food for thought.

 I think the idea of having more smooth development mode that would not
 make people to wait for 6+ months to release a new feature is great.

 ++


 It's insane to expect that feature priorities won't ever slightly
 shift in the next 6 months. Having a limited list of features targeted
 for the next 6 months is prone to mistakes, since people behind some


* Sure, we have had a few things that popped up, nova EC2 split for
example. But this is fairly rare.


 of approved features may need to postpone the effort for any type of
 reasons (personal, job rush, bad resource allocation, ...), and we

 then end up with approved specs with no actual code drops, using
 review 'slots' that would better be spent for other features that were
 not that lucky to get a rubber stamp during spec phase. Prior resource


* As stated below specs approval is very much not rubber stamping
* As stated below this doesn't even make sense, we are *not* using review
slots.


 allocation would probably work somehow if we were working for the same
 company that would define priorities to us, but it's not the case.


 It should be noted that even though Nova is using slots for reviews,
 Neutron is not. I've found that it's hard to try and slot people in to
 review specific things. During Juno I tried this for Neutron, and it failed
 miserably. For Kilo in Neutron, we're not using slots but instead I've
 tried to highlight the approved specs of Essential and High priority
 for review by all reviews, core and non-core included. It's gone ok, but
 the reality is you can't force people to review things. There are steps
 submitters can take to try and get timely review (lots of small, easy to
 digest patches, quick turnaround of comments, engagement in IRC and ML,
 etc.).


So this is a big fat lie, one that others believe as well. Nova is *not*
using slots for reviews. We discussed using slots for reviews but did not
adopt them.




 Anecdotally, in neutron, we have two Essential blueprints for Kilo,
 and there are no code drops or patches in review for any of those, so
 I would expect them to fail to merge. At the same time, I will need to
 wait for the end of Kilo to consider adding support for guru reports
 to the project. Or in oslo world, I will need to wait for Liberty to
 introduce features in oslo.policy that are needed by neutron to switch
 to it, etc.

 To be fair, there are many reasons those to Essential BPs do not have
 code. I still expect the Pecan focused to have code, but I already moved
 the Plugin one out of Kilo at this point because there was no chance the
 code would land.

 But I get your point here. I think this thread has highlighted the fact
 that the BP/spec process worked to some extent, but for small things, the
 core reviewer team should have the ability to say Yes, we can easily merge
 that, lets approve that spec even if it's late in the cycle.


 Another problem is that currently milestones are used merely for
 targeting bugs and features, but no one really cares about whether the
 actual milestone shipment actually works. Again, a story from neutron
 world: Kilo-1 was shipped in the middle of advanced services split,
 with some essential patches around distutils setup missing (no proper
 migration plan applied, conflicting config files in neutron and *aas
 repos, etc.)

 This is true, the milestone release matter but are not given enough focus
 and they release (for the most part) irregardless of items in them, given
 they are not long-lived, etc.

 So I'm all for reforms around processes we apply.

 If there's one thing OpenStack is good at, it's change.


 That said, I don't believe the real problem here is that we don't
 generate project tarballs frequently enough.

 Major problems I see as critical to tackle in our dev process are:

 - - enforced spec/dev mode. Solution: allow to propose (and approve) a
 reasonable spec at any moment in the cycle; allow code drops for
 approved specs at any moment in the cycle (except pre-release
 stabilization time); stop targeting specs: if it's sane, it's probably
 sane N+2 cycle from now too.

 I'd say this is fine for specs that are small and people agree can easily
 be merged. I'd say this is not the case for large features near the end of
 the release which are unlikely to gather enough review momentum to actually
 merge.


 - - core team rubber stamping a random set of specs and putting -2 on
 all other specs due to project priorities. Solution: stop pretending

 core team (or core drivers) can reasonably estimate review and coding
 resources for the next cycle. Instead allows community to decide
 what's worth the effort by approving all technically reasonable specs
 and allowing 

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-03-02 Thread Kyle Mestery
On Mon, Mar 2, 2015 at 3:38 PM, Joe Gordon joe.gord...@gmail.com wrote:



 On Mon, Mar 2, 2015 at 9:59 AM, Kyle Mestery mest...@mestery.com wrote:

 On Mon, Mar 2, 2015 at 9:57 AM, Ihar Hrachyshka ihrac...@redhat.com
 wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 Hi Daniel,

 thanks for a clear write-up of the matter and food for thought.

 I think the idea of having more smooth development mode that would not
 make people to wait for 6+ months to release a new feature is great.

 ++


 It's insane to expect that feature priorities won't ever slightly
 shift in the next 6 months. Having a limited list of features targeted
 for the next 6 months is prone to mistakes, since people behind some


 * Sure, we have had a few things that popped up, nova EC2 split for
 example. But this is fairly rare.


 of approved features may need to postpone the effort for any type of
 reasons (personal, job rush, bad resource allocation, ...), and we

 then end up with approved specs with no actual code drops, using
 review 'slots' that would better be spent for other features that were
 not that lucky to get a rubber stamp during spec phase. Prior resource


 * As stated below specs approval is very much not rubber stamping
 * As stated below this doesn't even make sense, we are *not* using review
 slots.


 allocation would probably work somehow if we were working for the same
 company that would define priorities to us, but it's not the case.


 It should be noted that even though Nova is using slots for reviews,
 Neutron is not. I've found that it's hard to try and slot people in to
 review specific things. During Juno I tried this for Neutron, and it failed
 miserably. For Kilo in Neutron, we're not using slots but instead I've
 tried to highlight the approved specs of Essential and High priority
 for review by all reviews, core and non-core included. It's gone ok, but
 the reality is you can't force people to review things. There are steps
 submitters can take to try and get timely review (lots of small, easy to
 digest patches, quick turnaround of comments, engagement in IRC and ML,
 etc.).


 So this is a big fat lie, one that others believe as well. Nova is *not*
 using slots for reviews. We discussed using slots for reviews but did not
 adopt them.


But I read it on the internet, it must be true.

As I said in a prior email, I'm sorry for that. I recalled reading about
nova's use of slots.





 Anecdotally, in neutron, we have two Essential blueprints for Kilo,
 and there are no code drops or patches in review for any of those, so
 I would expect them to fail to merge. At the same time, I will need to
 wait for the end of Kilo to consider adding support for guru reports
 to the project. Or in oslo world, I will need to wait for Liberty to
 introduce features in oslo.policy that are needed by neutron to switch
 to it, etc.

 To be fair, there are many reasons those to Essential BPs do not have
 code. I still expect the Pecan focused to have code, but I already moved
 the Plugin one out of Kilo at this point because there was no chance the
 code would land.

 But I get your point here. I think this thread has highlighted the fact
 that the BP/spec process worked to some extent, but for small things, the
 core reviewer team should have the ability to say Yes, we can easily merge
 that, lets approve that spec even if it's late in the cycle.


 Another problem is that currently milestones are used merely for
 targeting bugs and features, but no one really cares about whether the
 actual milestone shipment actually works. Again, a story from neutron
 world: Kilo-1 was shipped in the middle of advanced services split,
 with some essential patches around distutils setup missing (no proper
 migration plan applied, conflicting config files in neutron and *aas
 repos, etc.)

 This is true, the milestone release matter but are not given enough
 focus and they release (for the most part) irregardless of items in them,
 given they are not long-lived, etc.

 So I'm all for reforms around processes we apply.

 If there's one thing OpenStack is good at, it's change.


 That said, I don't believe the real problem here is that we don't
 generate project tarballs frequently enough.

 Major problems I see as critical to tackle in our dev process are:

 - - enforced spec/dev mode. Solution: allow to propose (and approve) a
 reasonable spec at any moment in the cycle; allow code drops for
 approved specs at any moment in the cycle (except pre-release
 stabilization time); stop targeting specs: if it's sane, it's probably
 sane N+2 cycle from now too.

 I'd say this is fine for specs that are small and people agree can
 easily be merged. I'd say this is not the case for large features near the
 end of the release which are unlikely to gather enough review momentum to
 actually merge.


 - - core team rubber stamping a random set of specs and putting -2 on
 all other specs due to project priorities. Solution: stop 

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-03-02 Thread Clint Byrum
Excerpts from Angus Salkeld's message of 2015-03-02 17:08:15 -0800:
 On Tue, Mar 3, 2015 at 9:45 AM, James Bottomley 
 james.bottom...@hansenpartnership.com wrote:
 
  On Tue, 2015-02-24 at 12:05 +0100, Thierry Carrez wrote:
   Daniel P. Berrange wrote:
[...]
The key observations

   
The first key observation from the schedule is that although we have
a 6 month release cycle, we in fact make 4 releases in that six
months because there are 3 milestones releases approx 6-7 weeks apart
from each other, in addition to the final release. So one of the key
burdens of a more frequent release cycle is already being felt, to
some degree.
   
The second observation is that thanks to the need to support a
continuous deployment models, the GIT master branches are generally
considered to be production ready at all times. The tree does not
typically go through periods of major instability that can be seen
in other projects, particular those which lack such comprehensive
testing infrastructure.
   
The third observation is that due to the relatively long cycle, and
increasing amounts of process, the work accomplished during the
cycles is becoming increasingly bursty. This is in turn causing
unacceptably long delays for contributors when their work is unlucky
enough to not get accepted during certain critical short windows of
opportunity in the cycle.
   
The first two observations strongly suggest that the choice of 6
months as a cycle length is a fairly arbitrary decision that can be
changed without unreasonable pain. The third observation suggests a
much shorter cycle length would smooth out the bumps and lead to a
more efficient  satisfying development process for all involved.
  
   I think you're judging the cycle from the perspective of developers
   only. 6 months was not an arbitrary decision. Translations and
   documentation teams basically need a month of feature/string freeze in
   order to complete their work. Since we can't reasonably freeze one month
   every 2 months, we picked 6 months.
 
  Actually, this is possible: look at Linux, it freezes for 10 weeks of a
  12 month release cycle (or 6 weeks of an 8 week one).  More on this
  below.
 
   It's also worth noting that we were on a 3-month cycle at the start of
   OpenStack. That was dropped after a cataclysmic release that managed the
   feat of (a) not having anything significant done, and (b) have out of
   date documentation and translations.
  
   While I agree that the packagers and stable teams can opt to skip a
   release, the docs, translations or security teams don't really have that
   luxury... Please go beyond the developers needs and consider the needs
   of the other teams.
  
   Random other comments below:
  
[...]
Release schedule

   
First the releases would probably be best attached to a set of
pre-determined fixed dates that don't ever vary from year to year.
eg releses happen Feb 1st, Apr 1st, Jun 1st, Aug 1st, Oct 1st, and
Dec 1st. If a particular release slips, don't alter following release
dates, just shorten the length of the dev cycle, so it becomes fully
self-correcting. The even numbered months are suggested to avoid a
release landing in xmas/new year :-)
  
   The Feb 1 release would probably be pretty empty :)
  
[...]
Stable branches
---
   
The consequences of a 2 month release cycle appear fairly severe for
the stable branch maint teams at first sight. This is not, however,
an insurmountable problem. The linux kernel shows an easy way forward
with their approach of only maintaining stable branches for a subset
of major releases, based around user / vendor demand. So it is still
entirely conceivable that the stable team only provide stable branch
releases for 2 out of the 6 yearly releases. ie no additional burden
over what they face today. Of course they might decide they want to
do more stable branches, but maintain each for a shorter time. So I
could equally see them choosing todo 3 or 4 stable branches a year.
Whatever is most effective for those involved and those consuming
them is fine.
  
   Stable branches may have the luxury of skipping releases and designate a
   stable one from time to time (I reject the Linux comparison because
   the kernel is at a very different moment in software lifecycle). The
   trick being, making one release special is sure to recreate the peak
   issues you're trying to solve.
 
  I don't disagree with the observation about different points in the
  lifecycle, but perhaps it might be instructive to ask if the linux
  kernel ever had a period in its development history that looks somewhat
  like OpenStack does now.  I would claim it did: before 2.6, we had the
  odd/even develop/stabilise cycle.  The theory driving it was that we
  

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-03-02 Thread James Bottomley
On Tue, 2015-02-24 at 12:05 +0100, Thierry Carrez wrote:
 Daniel P. Berrange wrote:
  [...]
  The key observations
  
  
  The first key observation from the schedule is that although we have
  a 6 month release cycle, we in fact make 4 releases in that six
  months because there are 3 milestones releases approx 6-7 weeks apart
  from each other, in addition to the final release. So one of the key
  burdens of a more frequent release cycle is already being felt, to
  some degree.
  
  The second observation is that thanks to the need to support a
  continuous deployment models, the GIT master branches are generally
  considered to be production ready at all times. The tree does not
  typically go through periods of major instability that can be seen
  in other projects, particular those which lack such comprehensive
  testing infrastructure.
  
  The third observation is that due to the relatively long cycle, and
  increasing amounts of process, the work accomplished during the
  cycles is becoming increasingly bursty. This is in turn causing
  unacceptably long delays for contributors when their work is unlucky
  enough to not get accepted during certain critical short windows of
  opportunity in the cycle.
  
  The first two observations strongly suggest that the choice of 6
  months as a cycle length is a fairly arbitrary decision that can be
  changed without unreasonable pain. The third observation suggests a
  much shorter cycle length would smooth out the bumps and lead to a
  more efficient  satisfying development process for all involved.
 
 I think you're judging the cycle from the perspective of developers
 only. 6 months was not an arbitrary decision. Translations and
 documentation teams basically need a month of feature/string freeze in
 order to complete their work. Since we can't reasonably freeze one month
 every 2 months, we picked 6 months.

Actually, this is possible: look at Linux, it freezes for 10 weeks of a
12 month release cycle (or 6 weeks of an 8 week one).  More on this
below.

 It's also worth noting that we were on a 3-month cycle at the start of
 OpenStack. That was dropped after a cataclysmic release that managed the
 feat of (a) not having anything significant done, and (b) have out of
 date documentation and translations.
 
 While I agree that the packagers and stable teams can opt to skip a
 release, the docs, translations or security teams don't really have that
 luxury... Please go beyond the developers needs and consider the needs
 of the other teams.
 
 Random other comments below:
 
  [...]
  Release schedule
  
  
  First the releases would probably be best attached to a set of
  pre-determined fixed dates that don't ever vary from year to year.
  eg releses happen Feb 1st, Apr 1st, Jun 1st, Aug 1st, Oct 1st, and
  Dec 1st. If a particular release slips, don't alter following release
  dates, just shorten the length of the dev cycle, so it becomes fully
  self-correcting. The even numbered months are suggested to avoid a
  release landing in xmas/new year :-)
 
 The Feb 1 release would probably be pretty empty :)
 
  [...]
  Stable branches
  ---
  
  The consequences of a 2 month release cycle appear fairly severe for
  the stable branch maint teams at first sight. This is not, however,
  an insurmountable problem. The linux kernel shows an easy way forward
  with their approach of only maintaining stable branches for a subset
  of major releases, based around user / vendor demand. So it is still
  entirely conceivable that the stable team only provide stable branch
  releases for 2 out of the 6 yearly releases. ie no additional burden
  over what they face today. Of course they might decide they want to
  do more stable branches, but maintain each for a shorter time. So I
  could equally see them choosing todo 3 or 4 stable branches a year.
  Whatever is most effective for those involved and those consuming
  them is fine.
 
 Stable branches may have the luxury of skipping releases and designate a
 stable one from time to time (I reject the Linux comparison because
 the kernel is at a very different moment in software lifecycle). The
 trick being, making one release special is sure to recreate the peak
 issues you're trying to solve.

I don't disagree with the observation about different points in the
lifecycle, but perhaps it might be instructive to ask if the linux
kernel ever had a period in its development history that looks somewhat
like OpenStack does now.  I would claim it did: before 2.6, we had the
odd/even develop/stabilise cycle.  The theory driving it was that we
needed a time for everyone to develop then a time for everyone to help
make stable.  You yourself said this in the other thread:

 Joe Gordon wrote:
  [...]
  I think a lot of the frustration with our current cadence comes out of
  the big stop everything (development, planning etc.), and stabilize the
  release process. Which in turn isn't 

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-03-02 Thread Angus Salkeld
On Tue, Mar 3, 2015 at 9:45 AM, James Bottomley 
james.bottom...@hansenpartnership.com wrote:

 On Tue, 2015-02-24 at 12:05 +0100, Thierry Carrez wrote:
  Daniel P. Berrange wrote:
   [...]
   The key observations
   
  
   The first key observation from the schedule is that although we have
   a 6 month release cycle, we in fact make 4 releases in that six
   months because there are 3 milestones releases approx 6-7 weeks apart
   from each other, in addition to the final release. So one of the key
   burdens of a more frequent release cycle is already being felt, to
   some degree.
  
   The second observation is that thanks to the need to support a
   continuous deployment models, the GIT master branches are generally
   considered to be production ready at all times. The tree does not
   typically go through periods of major instability that can be seen
   in other projects, particular those which lack such comprehensive
   testing infrastructure.
  
   The third observation is that due to the relatively long cycle, and
   increasing amounts of process, the work accomplished during the
   cycles is becoming increasingly bursty. This is in turn causing
   unacceptably long delays for contributors when their work is unlucky
   enough to not get accepted during certain critical short windows of
   opportunity in the cycle.
  
   The first two observations strongly suggest that the choice of 6
   months as a cycle length is a fairly arbitrary decision that can be
   changed without unreasonable pain. The third observation suggests a
   much shorter cycle length would smooth out the bumps and lead to a
   more efficient  satisfying development process for all involved.
 
  I think you're judging the cycle from the perspective of developers
  only. 6 months was not an arbitrary decision. Translations and
  documentation teams basically need a month of feature/string freeze in
  order to complete their work. Since we can't reasonably freeze one month
  every 2 months, we picked 6 months.

 Actually, this is possible: look at Linux, it freezes for 10 weeks of a
 12 month release cycle (or 6 weeks of an 8 week one).  More on this
 below.

  It's also worth noting that we were on a 3-month cycle at the start of
  OpenStack. That was dropped after a cataclysmic release that managed the
  feat of (a) not having anything significant done, and (b) have out of
  date documentation and translations.
 
  While I agree that the packagers and stable teams can opt to skip a
  release, the docs, translations or security teams don't really have that
  luxury... Please go beyond the developers needs and consider the needs
  of the other teams.
 
  Random other comments below:
 
   [...]
   Release schedule
   
  
   First the releases would probably be best attached to a set of
   pre-determined fixed dates that don't ever vary from year to year.
   eg releses happen Feb 1st, Apr 1st, Jun 1st, Aug 1st, Oct 1st, and
   Dec 1st. If a particular release slips, don't alter following release
   dates, just shorten the length of the dev cycle, so it becomes fully
   self-correcting. The even numbered months are suggested to avoid a
   release landing in xmas/new year :-)
 
  The Feb 1 release would probably be pretty empty :)
 
   [...]
   Stable branches
   ---
  
   The consequences of a 2 month release cycle appear fairly severe for
   the stable branch maint teams at first sight. This is not, however,
   an insurmountable problem. The linux kernel shows an easy way forward
   with their approach of only maintaining stable branches for a subset
   of major releases, based around user / vendor demand. So it is still
   entirely conceivable that the stable team only provide stable branch
   releases for 2 out of the 6 yearly releases. ie no additional burden
   over what they face today. Of course they might decide they want to
   do more stable branches, but maintain each for a shorter time. So I
   could equally see them choosing todo 3 or 4 stable branches a year.
   Whatever is most effective for those involved and those consuming
   them is fine.
 
  Stable branches may have the luxury of skipping releases and designate a
  stable one from time to time (I reject the Linux comparison because
  the kernel is at a very different moment in software lifecycle). The
  trick being, making one release special is sure to recreate the peak
  issues you're trying to solve.

 I don't disagree with the observation about different points in the
 lifecycle, but perhaps it might be instructive to ask if the linux
 kernel ever had a period in its development history that looks somewhat
 like OpenStack does now.  I would claim it did: before 2.6, we had the
 odd/even develop/stabilise cycle.  The theory driving it was that we
 needed a time for everyone to develop then a time for everyone to help
 make stable.  You yourself said this in the other thread:

  Joe Gordon wrote:
   [...]
   I think 

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-03-02 Thread Doug Hellmann


On Mon, Mar 2, 2015, at 10:57 AM, Ihar Hrachyshka wrote:
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1
 
 Hi Daniel,
 
 thanks for a clear write-up of the matter and food for thought.
 
 I think the idea of having more smooth development mode that would not
 make people to wait for 6+ months to release a new feature is great.
 
 It's insane to expect that feature priorities won't ever slightly
 shift in the next 6 months. Having a limited list of features targeted
 for the next 6 months is prone to mistakes, since people behind some
 of approved features may need to postpone the effort for any type of
 reasons (personal, job rush, bad resource allocation, ...), and we
 then end up with approved specs with no actual code drops, using
 review 'slots' that would better be spent for other features that were
 not that lucky to get a rubber stamp during spec phase. Prior resource
 allocation would probably work somehow if we were working for the same
 company that would define priorities to us, but it's not the case.
 
 Anecdotally, in neutron, we have two Essential blueprints for Kilo,
 and there are no code drops or patches in review for any of those, so
 I would expect them to fail to merge. At the same time, I will need to
 wait for the end of Kilo to consider adding support for guru reports
 to the project. Or in oslo world, I will need to wait for Liberty to
 introduce features in oslo.policy that are needed by neutron to switch
 to it, etc.

The Oslo policy is a bit more lenient than some of the others I've heard
described. We don't follow the feature proposal freeze. Instead, we only
have a hard freeze for the master branches of libraries until the
release candidates are done. Any features proposed at any other time are
candidates, with the usual caveats for review team time and attention.

Doug

 
 Another problem is that currently milestones are used merely for
 targeting bugs and features, but no one really cares about whether the
 actual milestone shipment actually works. Again, a story from neutron
 world: Kilo-1 was shipped in the middle of advanced services split,
 with some essential patches around distutils setup missing (no proper
 migration plan applied, conflicting config files in neutron and *aas
 repos, etc.)
 
 So I'm all for reforms around processes we apply.
 
 That said, I don't believe the real problem here is that we don't
 generate project tarballs frequently enough.
 
 Major problems I see as critical to tackle in our dev process are:
 
 - - enforced spec/dev mode. Solution: allow to propose (and approve) a
 reasonable spec at any moment in the cycle; allow code drops for
 approved specs at any moment in the cycle (except pre-release
 stabilization time); stop targeting specs: if it's sane, it's probably
 sane N+2 cycle from now too.
 
 - - core team rubber stamping a random set of specs and putting -2 on
 all other specs due to project priorities. Solution: stop pretending
 core team (or core drivers) can reasonably estimate review and coding
 resources for the next cycle. Instead allows community to decide
 what's worth the effort by approving all technically reasonable specs
 and allowing everyone to invest time and effort in specs (s)he seems
 worth it.
 
 - - no due stabilization process before dev milestones. Solution:
 introduce one in your team workflow, be more strict in what goes in
 during pre milestone stabilization time.
 
 If all above is properly applied, we would get into position similar
 to your proposal. The difference though is that upstream project would
 not call milestone tarballs 'Releases'. Still, if there are brave
 vendors to ship milestones, fine, they would have the same freedom as
 in your proposal.
 
 Note: all the steps mentioned above can be applied on *per team* basis
 without breaking existing release workflow.
 
 Some more comments from stable-maint/distribution side below.
 
 On 02/24/2015 10:53 AM, Daniel P. Berrange wrote:
 [...skip...]
  The modest proposal ===
 [...skip...]
  
  Stable branches ---
  
  The consequences of a 2 month release cycle appear fairly severe
  for the stable branch maint teams at first sight. This is not,
  however, an insurmountable problem. The linux kernel shows an easy
  way forward with their approach of only maintaining stable branches
  for a subset of major releases, based around user / vendor demand.
  So it is still entirely conceivable that the stable team only
  provide stable branch releases for 2 out of the 6 yearly releases.
  ie no additional burden over what they face today. Of course they
  might decide they want to do more stable branches, but maintain
  each for a shorter time. So I could equally see them choosing todo
  3 or 4 stable branches a year. Whatever is most effective for those
  involved and those consuming them is fine.
  
 
 Since it's not only stable branches that are affected (translators,
 documentation writers, VMT were already mentioned), those 

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-03-02 Thread Ihar Hrachyshka
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hi Daniel,

thanks for a clear write-up of the matter and food for thought.

I think the idea of having more smooth development mode that would not
make people to wait for 6+ months to release a new feature is great.

It's insane to expect that feature priorities won't ever slightly
shift in the next 6 months. Having a limited list of features targeted
for the next 6 months is prone to mistakes, since people behind some
of approved features may need to postpone the effort for any type of
reasons (personal, job rush, bad resource allocation, ...), and we
then end up with approved specs with no actual code drops, using
review 'slots' that would better be spent for other features that were
not that lucky to get a rubber stamp during spec phase. Prior resource
allocation would probably work somehow if we were working for the same
company that would define priorities to us, but it's not the case.

Anecdotally, in neutron, we have two Essential blueprints for Kilo,
and there are no code drops or patches in review for any of those, so
I would expect them to fail to merge. At the same time, I will need to
wait for the end of Kilo to consider adding support for guru reports
to the project. Or in oslo world, I will need to wait for Liberty to
introduce features in oslo.policy that are needed by neutron to switch
to it, etc.

Another problem is that currently milestones are used merely for
targeting bugs and features, but no one really cares about whether the
actual milestone shipment actually works. Again, a story from neutron
world: Kilo-1 was shipped in the middle of advanced services split,
with some essential patches around distutils setup missing (no proper
migration plan applied, conflicting config files in neutron and *aas
repos, etc.)

So I'm all for reforms around processes we apply.

That said, I don't believe the real problem here is that we don't
generate project tarballs frequently enough.

Major problems I see as critical to tackle in our dev process are:

- - enforced spec/dev mode. Solution: allow to propose (and approve) a
reasonable spec at any moment in the cycle; allow code drops for
approved specs at any moment in the cycle (except pre-release
stabilization time); stop targeting specs: if it's sane, it's probably
sane N+2 cycle from now too.

- - core team rubber stamping a random set of specs and putting -2 on
all other specs due to project priorities. Solution: stop pretending
core team (or core drivers) can reasonably estimate review and coding
resources for the next cycle. Instead allows community to decide
what's worth the effort by approving all technically reasonable specs
and allowing everyone to invest time and effort in specs (s)he seems
worth it.

- - no due stabilization process before dev milestones. Solution:
introduce one in your team workflow, be more strict in what goes in
during pre milestone stabilization time.

If all above is properly applied, we would get into position similar
to your proposal. The difference though is that upstream project would
not call milestone tarballs 'Releases'. Still, if there are brave
vendors to ship milestones, fine, they would have the same freedom as
in your proposal.

Note: all the steps mentioned above can be applied on *per team* basis
without breaking existing release workflow.

Some more comments from stable-maint/distribution side below.

On 02/24/2015 10:53 AM, Daniel P. Berrange wrote:
[...skip...]
 The modest proposal ===
[...skip...]
 
 Stable branches ---
 
 The consequences of a 2 month release cycle appear fairly severe
 for the stable branch maint teams at first sight. This is not,
 however, an insurmountable problem. The linux kernel shows an easy
 way forward with their approach of only maintaining stable branches
 for a subset of major releases, based around user / vendor demand.
 So it is still entirely conceivable that the stable team only
 provide stable branch releases for 2 out of the 6 yearly releases.
 ie no additional burden over what they face today. Of course they
 might decide they want to do more stable branches, but maintain
 each for a shorter time. So I could equally see them choosing todo
 3 or 4 stable branches a year. Whatever is most effective for those
 involved and those consuming them is fine.
 

Since it's not only stable branches that are affected (translators,
documentation writers, VMT were already mentioned), those affected
will probably need to come up with some synchronized decision.

Let's say we still decide to support two out of six releases (same
scheme as is now). In that case no process that we usually attach to
releases will be running after release dates. This makes me wonder how
it's different from milestones we already have.

Do you think any downstream vendors will actually ship and support
upstream releases that upstream drops any guarantees for (no VMT, no
stable branches, no gate runs, ...) right after the 

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-03-02 Thread Ihar Hrachyshka
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 02/26/2015 01:06 AM, Thomas Goirand wrote:
 On 02/24/2015 12:27 PM, Daniel P. Berrange wrote:
 I'm actually trying to judge it from the POV of users, not just 
 developers. I find it pretty untenable that in the fast moving 
 world of cloud, users have to wait as long as 6 months for a 
 feature to get into a openstack release, often much longer.
 
 If you were trying to judge from the POV of users, then you would 
 consider that basically, they don't really care the brand new
 shiny feature which just appear. They care having a long time
 support for whatever version of OpenStack they have installed,
 without having the head-aches of upgrading which is famously
 painful with OpenStack. This shows clearly on our user surveys
 which are presented on every summit: users are lagging behind, with
 a majority still running with OpenStack releases which are already
 EOL.
 

As was already said in the thread, they care about long support AND
their specific subset of shiny new features they need (and they don't
care about all other features that are not in their subjective list of
shiny ones).

 In fact, if you want to judge from the POV of our users, we should
 *SLOW DOWN* our release cycles, and probably move to something like
 one release every year or 2. We should also try to have longer
 periods of support for our stable releases, which would (with my
 Debian package maintainer hat on!) help distributions to do such
 security support.
 
 Debian Jessie will be released in a few month from now, just
 before Icehouse (which it ships) will be EOL. RedHat, Canonical,
 IBM, and so many more are also on the same (sinking) ship.
 

That won't come up magically. If no people from Debian or IBM are
actively involved in solving issues for stable branches, long term
will be just a wish and not a reality. At the moment we have several
people from Red Hat (me, Alan) and Ubuntu (Chuck, Adam) side active in
the team. That would be great to see more diversity, and more hands
dirty with stable branches on daily basis.

 As for my employer side of things, we've seen numerous cases with
 our customer requesting for LTS, which we have to provide by
 ourselves, since it's not supported upstream.
 

Every downstream vendor would be happy to see upstream freely support
their product. :) But you get what you pay to upstream horizontal
initiatives like stable maintenance.

I've heard IBM still supports releases starting from C release!..

 I think the majority of translation work can be done in parallel
 with dev work and the freeze time just needs to tie up the small
 remaining bits.
 
 It'd be nice indeed, but I've never seen any project (open source
 or not) working this way for translations.

It's actually not true. I've coordinated multiple translation teams
for my language (Belarusian), and for what I can tell, the best
practice is to work on translations while development is ongoing. Yes,
it means some effort wasted, but it also means spreading the whole
effort throughout the year instead of doing all the work in
pre-release freeze time.

Freeze time is anyway good to make sure that no last minute patches
break translations, and I don't suggest we drop them completely.

 
 Documentation is again something I'd expect to be done more or
 less in parallel with dev.
 
 Let's go back to reality: the Juno install-guide is still not
 finished, and the doc team is lagging behind.
 
 It would be reasonable for the vulnerability team to take the
 decision that they'll support fixes for master, and any branches
 that the stable team decide to support. ie they would not
 neccessarily have to commit to shipping fixes for every single
 release made.
 
 I've been crying for this type of decision. IE: drop Juno support
 early, and continue to maintain Icehouse for longer. I wish this
 happens, but the release team always complained that nobody works
 on maintaining the gate for the stable branches. Unless this
 changes, I don't see hope... :(
 

- From Red Hat perspective, we put our effort in what we're interested
in. We're more interested in keeping the branch for the latest release
working, not about sticking to an old release that e.g. Debian chose
to put into their next official release, so unless someone from Debian
come to the team and start to invest into Icehouse, it will eventually
be dropped. That's life.

 I really not trying to focus on the developers woes. I'm trying
 to focus on making OpenStack better serve our users. My main
 motiviation here is that I think we're doing a pretty terrible
 job at getting work done that is important to our users in a
 timely manner. This is caused by a workflow  release cycle that
 is negatively impacting the developers.
 
 Our workflow and the release cycle are 2 separate things. From my
 POV, it'd be a mistake to believe switching to a different release
 cycle will fix our workflow.

Here, I actually agree. There are lots of enhancements in 

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-03-02 Thread Kyle Mestery
On Mon, Mar 2, 2015 at 9:57 AM, Ihar Hrachyshka ihrac...@redhat.com wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA1

 Hi Daniel,

 thanks for a clear write-up of the matter and food for thought.

 I think the idea of having more smooth development mode that would not
 make people to wait for 6+ months to release a new feature is great.

 ++


 It's insane to expect that feature priorities won't ever slightly
 shift in the next 6 months. Having a limited list of features targeted
 for the next 6 months is prone to mistakes, since people behind some
 of approved features may need to postpone the effort for any type of
 reasons (personal, job rush, bad resource allocation, ...), and we
 then end up with approved specs with no actual code drops, using
 review 'slots' that would better be spent for other features that were
 not that lucky to get a rubber stamp during spec phase. Prior resource
 allocation would probably work somehow if we were working for the same
 company that would define priorities to us, but it's not the case.

 It should be noted that even though Nova is using slots for reviews,
Neutron is not. I've found that it's hard to try and slot people in to
review specific things. During Juno I tried this for Neutron, and it failed
miserably. For Kilo in Neutron, we're not using slots but instead I've
tried to highlight the approved specs of Essential and High priority
for review by all reviews, core and non-core included. It's gone ok, but
the reality is you can't force people to review things. There are steps
submitters can take to try and get timely review (lots of small, easy to
digest patches, quick turnaround of comments, engagement in IRC and ML,
etc.).


 Anecdotally, in neutron, we have two Essential blueprints for Kilo,
 and there are no code drops or patches in review for any of those, so
 I would expect them to fail to merge. At the same time, I will need to
 wait for the end of Kilo to consider adding support for guru reports
 to the project. Or in oslo world, I will need to wait for Liberty to
 introduce features in oslo.policy that are needed by neutron to switch
 to it, etc.

 To be fair, there are many reasons those to Essential BPs do not have
code. I still expect the Pecan focused to have code, but I already moved
the Plugin one out of Kilo at this point because there was no chance the
code would land.

But I get your point here. I think this thread has highlighted the fact
that the BP/spec process worked to some extent, but for small things, the
core reviewer team should have the ability to say Yes, we can easily merge
that, lets approve that spec even if it's late in the cycle.


 Another problem is that currently milestones are used merely for
 targeting bugs and features, but no one really cares about whether the
 actual milestone shipment actually works. Again, a story from neutron
 world: Kilo-1 was shipped in the middle of advanced services split,
 with some essential patches around distutils setup missing (no proper
 migration plan applied, conflicting config files in neutron and *aas
 repos, etc.)

 This is true, the milestone release matter but are not given enough focus
and they release (for the most part) irregardless of items in them, given
they are not long-lived, etc.

So I'm all for reforms around processes we apply.

 If there's one thing OpenStack is good at, it's change.


 That said, I don't believe the real problem here is that we don't
 generate project tarballs frequently enough.

 Major problems I see as critical to tackle in our dev process are:

 - - enforced spec/dev mode. Solution: allow to propose (and approve) a
 reasonable spec at any moment in the cycle; allow code drops for
 approved specs at any moment in the cycle (except pre-release
 stabilization time); stop targeting specs: if it's sane, it's probably
 sane N+2 cycle from now too.

 I'd say this is fine for specs that are small and people agree can easily
be merged. I'd say this is not the case for large features near the end of
the release which are unlikely to gather enough review momentum to actually
merge.


 - - core team rubber stamping a random set of specs and putting -2 on
 all other specs due to project priorities. Solution: stop pretending
 core team (or core drivers) can reasonably estimate review and coding
 resources for the next cycle. Instead allows community to decide
 what's worth the effort by approving all technically reasonable specs
 and allowing everyone to invest time and effort in specs (s)he seems
 worth it.

 If you're referring to Neutron here, I think you fail to estimate the
amount of time the neutron-drivers team (along with a handful of other
folks) spent reviewing specs and trying to allocate them for this release.
We're not just rubber stamping things, we're reviewing, providing comment,
and ensuring things fit in a consistent roadmap for the next release. In
the past, we've had this sort of wild west where all specs are approved,
everyone focuses on 

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-26 Thread Ritesh Raj Sarraf


On 02/26/2015 07:06 PM, Ed Leafe wrote:
 I think you've nailed where the disconnect is between the two sides of this 
 issue: what exactly do we see OpenStack being? You brought up several Linux 
 vendors who ship on a longish cycle, and who provide LTS for their releases. 
 But Linux itself is on no such cycle, nor does it provide long term anything.


But Linux is one monolith project.

 OpenStack can't be all things to all people. Following the Linux analogy, we 
 need a few companies who want to become OpenStack distributors, packagers, 
 and supporters, in the manner of RedHat, Canonical, etc., are for Linux. As a 
 development project, we need to be able to move fluidly, and the release 
 cycle deadlines and freezes get in the way of that. As a packager and 
 distributor, the release cycle scheduler *helps* immeasurably. We can't be 
 both.

I am fairly new to OpenStack, but from what I've ascertained so far,
there are, now, individual sub-releases of individual projects. That
could be a difficult task for any Linux vendor, to distribute.

There's one project, Calibre - EBook Management Software, that does
weekly releases. But again, it is easy for them, because they are a
single controlled project.

For something big as OpenStack, IMO, close co-ordination is needed.

-- 
Given the large number of mailing lists I follow, I request you to CC me
in replies for quicker response


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-26 Thread Ed Leafe
On Feb 25, 2015, at 6:06 PM, Thomas Goirand z...@debian.org wrote:

 In fact, if you want to judge from the POV of our users, we should *SLOW
 DOWN* our release cycles, and probably move to something like one
 release every year or 2. We should also try to have longer periods of
 support for our stable releases, which would (with my Debian package
 maintainer hat on!) help distributions to do such security support.
 
 Debian Jessie will be released in a few month from now, just before
 Icehouse (which it ships) will be EOL. RedHat, Canonical, IBM, and so
 many more are also on the same (sinking) ship.
 
 As for my employer side of things, we've seen numerous cases with our
 customer requesting for LTS, which we have to provide by ourselves,
 since it's not supported upstream.

I think you've nailed where the disconnect is between the two sides of this 
issue: what exactly do we see OpenStack being? You brought up several Linux 
vendors who ship on a longish cycle, and who provide LTS for their releases. 
But Linux itself is on no such cycle, nor does it provide long term anything.

OpenStack can't be all things to all people. Following the Linux analogy, we 
need a few companies who want to become OpenStack distributors, packagers, and 
supporters, in the manner of RedHat, Canonical, etc., are for Linux. As a 
development project, we need to be able to move fluidly, and the release cycle 
deadlines and freezes get in the way of that. As a packager and distributor, 
the release cycle scheduler *helps* immeasurably. We can't be both.


-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-26 Thread Chris Dent

On Thu, 26 Feb 2015, Ed Leafe wrote:


OpenStack can't be all things to all people. Following the Linux
analogy, we need a few companies who want to become OpenStack
distributors, packagers, and supporters, in the manner of RedHat,
Canonical, etc., are for Linux. As a development project, we need to
be able to move fluidly, and the release cycle deadlines and freezes
get in the way of that. As a packager and distributor, the release
cycle scheduler *helps* immeasurably. We can't be both.


Yes.

The special nature of OpenStack makes the influence of companies
quite strong and has many benefits (e.g. look ma, being paid to do open
source) but it also means that boundaries that are found in other
somewhat similar environments are not as strong.

Elsewhere in the thread there was discussion about the importance of the
release cycle for marketing. Marketing for who and by whom? Surely
marketing is^wought to be in the domain of the people who are making
money selling aggregations of OpenStack?

--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-26 Thread Ed Leafe
On Feb 26, 2015, at 9:49 AM, Ritesh Raj Sarraf r...@researchut.com wrote:

 I think you've nailed where the disconnect is between the two sides of this 
 issue: what exactly do we see OpenStack being? You brought up several Linux 
 vendors who ship on a longish cycle, and who provide LTS for their releases. 
 But Linux itself is on no such cycle, nor does it provide long term anything.
 
 But Linux is one monolith project.

No, not really. The kernel may be, but if you want a distro that is useful to 
people, you need to collect a set of tools and apps that allow users to have a 
full computing experience. If Linux were a monolith, then there really wouldn't 
be any difference between Ubuntu and Slackware.

 I am fairly new to OpenStack, but from what I've ascertained so far,
 there are, now, individual sub-releases of individual projects. That
 could be a difficult task for any Linux vendor, to distribute.

I don't think that anyone is implying that it is easy to do, no matter where it 
is done. It does seem that there are significant disadvantages on the 
development side of things to have one group doing it all.


-- Ed Leafe







signature.asc
Description: Message signed with OpenPGP using GPGMail
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-26 Thread Daniel P. Berrange
On Thu, Feb 26, 2015 at 01:06:14AM +0100, Thomas Goirand wrote:
 On 02/24/2015 12:27 PM, Daniel P. Berrange wrote:
  I'm actually trying to judge it from the POV of users, not just
  developers. I find it pretty untenable that in the fast moving
  world of cloud, users have to wait as long as 6 months for a
  feature to get into a openstack release, often much longer.
 
 If you were trying to judge from the POV of users, then you would
 consider that basically, they don't really care the brand new shiny
 feature which just appear. They care having a long time support for
 whatever version of OpenStack they have installed, without having the
 head-aches of upgrading which is famously painful with OpenStack. This
 shows clearly on our user surveys which are presented on every summit:
 users are lagging behind, with a majority still running with OpenStack
 releases which are already EOL.
 
 In fact, if you want to judge from the POV of our users, we should *SLOW
 DOWN* our release cycles, and probably move to something like one
 release every year or 2. We should also try to have longer periods of
 support for our stable releases, which would (with my Debian package
 maintainer hat on!) help distributions to do such security support.

It is not actually that simple IME. As I describe elsewhere in this
thread, users basically want no new features at all ever, except for
the new features that are specifically relevant to what they want,
and of course every users desired feature is different. It is essentially
impossible to satisfy the requests in this way because they are inherantly
contradictory  conflicting. In reality what they asking is not that we
reduce the number of releases or frequency of releases, but rather that
they be able to stay running on a chosen release for longer.

IOW The frequency of releases doesn't matter as long as they are not being
forced to upgrade to the newer releases unreasonably quickly. From that
POV I can certainly see how OpenStack is not satisfying them well,
because if they do choose to skip 2-3 releases, they'll find themselves
EOLd and also they'll be forced to deploy each intermediate release in
order to apply upgrades, before finally getting to their target release.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-25 Thread Alex Glikson
Tom Fifield t...@openstack.org wrote on 25/02/2015 06:46:13 AM:

 On 24/02/15 19:27, Daniel P. Berrange wrote:
  On Tue, Feb 24, 2015 at 12:05:17PM +0100, Thierry Carrez wrote:
  Daniel P. Berrange wrote:
  [...]
 
  I'm not familiar with how the translations works, but if they are
  waiting until the freeze before starting translation work I'd
  say that is a mistaken approach. Obviously during active dev part
  of the cycle, some translated strings are in flux, so if translation
  was taking place in parallel there could be some wasted effort, but
  I'd expect that to be the minority case. I think the majority of
  translation work can be done in parallel with dev work and the freeze
  time just needs to tie up the small remaining bits.
 
 
 So, two points:
 
 1) We wouldn't be talking about throwing just a couple of percent of
 their work away.
 
 As an example, even without looking at the introduction of new strings
 or deleting others, you may not be aware that changing a single word in
 a string in the code means that entire string needs to be re-translated.
 Even with the extensive translation memory systems we have making
 suggestions as best they can, we're talking about very, very significant
 amounts of wasted effort.

How difficult would it be to try quantifying this wasted effort? For 
example, if someone could write a script that extracts the data for a 
histogram showing the amount of strings (e.g., in Nova) that have been 
changed/overridden in consequent patches up to 1 week apart, between 1 and 
2 weeks apart, and so on up to, say, 52 weeks.

Regards,
Alex


 Regards,
 
 
 Tom
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-25 Thread Thomas Goirand
On 02/24/2015 12:27 PM, Daniel P. Berrange wrote:
 I'm actually trying to judge it from the POV of users, not just
 developers. I find it pretty untenable that in the fast moving
 world of cloud, users have to wait as long as 6 months for a
 feature to get into a openstack release, often much longer.

If you were trying to judge from the POV of users, then you would
consider that basically, they don't really care the brand new shiny
feature which just appear. They care having a long time support for
whatever version of OpenStack they have installed, without having the
head-aches of upgrading which is famously painful with OpenStack. This
shows clearly on our user surveys which are presented on every summit:
users are lagging behind, with a majority still running with OpenStack
releases which are already EOL.

In fact, if you want to judge from the POV of our users, we should *SLOW
DOWN* our release cycles, and probably move to something like one
release every year or 2. We should also try to have longer periods of
support for our stable releases, which would (with my Debian package
maintainer hat on!) help distributions to do such security support.

Debian Jessie will be released in a few month from now, just before
Icehouse (which it ships) will be EOL. RedHat, Canonical, IBM, and so
many more are also on the same (sinking) ship.

As for my employer side of things, we've seen numerous cases with our
customer requesting for LTS, which we have to provide by ourselves,
since it's not supported upstream.

 I think the majority of
 translation work can be done in parallel with dev work and the freeze
 time just needs to tie up the small remaining bits.

It'd be nice indeed, but I've never seen any project (open source or
not) working this way for translations.

 Documentation is again something I'd expect to be done more or less
 in parallel with dev.

Let's go back to reality: the Juno install-guide is still not finished,
and the doc team is lagging behind.

 It would be reasonable for the vulnerability team to take the decision
 that they'll support fixes for master, and any branches that the stable
 team decide to support. ie they would not neccessarily have to commit
 to shipping fixes for every single release made.

I've been crying for this type of decision. IE: drop Juno support early,
and continue to maintain Icehouse for longer. I wish this happens, but
the release team always complained that nobody works on maintaining the
gate for the stable branches. Unless this changes, I don't see hope... :(

 I really not trying to focus on the developers woes. I'm trying to focus on
 making OpenStack better serve our users. My main motiviation here is that I
 think we're doing a pretty terrible job at getting work done that is important
 to our users in a timely manner. This is caused by a workflow  release cycle
 that is negatively impacting the developers.

Our workflow and the release cycle are 2 separate things. From my POV,
it'd be a mistake to believe switching to a different release cycle will
fix our workflow.

Also, I'd like to point out something: it's been 2 years that I do
release each and every beta release we do. But either they are bug free
(you bet... :)), or nobody uses them (more likely), because I've *never
ever* received some bug reports about them. Reasonably, there's no
consumer for beta releases.

Hoping my point of view is helpful,

Cheers,

Thomas Goirand (zigo)


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-25 Thread Kashyap Chamarthy
On Tue, Feb 24, 2015 at 10:02:36PM -0800, Mark Atwood wrote:
 On Tue, Feb 24, 2015, at 04:28, Kashyap Chamarthy wrote:
  
  Along with the below, if push comes to shove, OpenStack Foundation could
  probably try a milder variant (obviously, not all activities can be
  categorized as 'critical path') of Linux Foundation's Critical
  Infrastructure Protection Initiative[1] to fund certain project
  activities in need.
 
 Speaking as a person who sits on the LF CII board meetings,
 and helps turn the crank on that particular sausage mill,
 no, we really don't want to go down that path at this point in
 time.

I didn't imply to do an _exact_ version of LF's CII. Given so much of
interest in OpenStack, vendors/interested parties must (yes) maintain a
balance between giving vs taking from the community. And, they should be
provided ample information on _where_ they can target people/engineers
(Dan Berrange noted this too in one of his responses in this thread).

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-25 Thread Daniel P. Berrange
On Wed, Feb 25, 2015 at 10:31:58AM +0100, Thierry Carrez wrote:
 Robert Collins wrote:
  It's also worth noting that we were on a 3-month cycle at the start of
  OpenStack. That was dropped after a cataclysmic release that managed the
  feat of (a) not having anything significant done, and (b) have out of
  date documentation and translations.
  
  Oh! 
  https://wiki.openstack.org/w/index.php?title=Obsolete:3MonthReleaseCycledirection=prevoldid=14016
  appears to be the page about that, but my google-fu isn't turning up a
  post-mortem of the C release which prompted the change. Perhaps some
  old-hand like you can fill us in on the details :).
 
 After the Austin release (a code drop, really), we were on a three-month
 time-based cycle with no intermediary milestone. The first cycle (Bexar)
 worked relatively well, the second one (Cactus) not so well. There were
 multiple reasons for that:
 
 - Software was growing up in Bexar and we introduced more testing, but
 we were unable to detect and fix all the release blockers in time for
 the release date, resulting in a broken Bexar release. To address that
 in Cactus, we increased the duration of the frozen period, but then that
 didn't leave enough time for development, hence nothing getting done.

I wasn't involved in OpenStack back then, but is it not true that
OpenStack has changed almost beyond recognition since those early
days. The number of projects has certainly increased massively, as
has the number of contributors, both individual  corporate. There
is a hell of alot of testing now and people doing continuous deployment
of trunk, so I find it hard to believe that trunk is so unstable that
we can't do a release at any time we choose, nor that we have to have
such long freezes that we don't do get time for dev.

 - Not starting a release cycle with a global F2F gathering (no
 intermediary Design Summit) actually proved quite disastrous: no reset
 of community, no alignment on goal, no discussion on what to work on.
 That resulted in a limbo period after the Bexar release. That is why I'm
 so insistent on aligning our development cycles with our Design Summits.
 I've seen where we go when we don't.

I don't know about other projects so much, but I don't really see the
design summit as a positive thing wrt planning the next release. For
a start the design summit is atbout 5 weeks after the trunk opens for
development, so if people wait for the summit do do planning they have
thrown away half of the first milestones' development time. It is also
not inclusive as a decision making forum because we cannot assume every
one is able to make it to the summit in person, and even if they are
present, people often can't get involved in all sessions they wish to
due to conflicting schedules. If release planning were done via email
primarily, with IRC for cases where realtime planning is needed, it
would be more effective IMHO.  IOW i think the summit would be better
off if were explicitly not associated with releases, but rather be a
general forum for collaboration were we talk through ideas or do code
sprints, and more.

  [...]
  I may appear defensive on my answers, but it's not my goal to defend the
  current system: it's just that most of those proposals generally ignore
  the diversity of the needs of the teams that make OpenStack possible, to
  focus on a particular set of contributors' woes. I'm trying to bring
  that wider perspective in -- the current system is a trade-off and the
  result of years of evolution, not an arbitrary historic choice that we
  can just change at will.
  
  I agree that its not arbitrary and that changing it requires some
  appropriate wide-spread consultation; OTOH the benefits of rolling and
  higher frequency releases are really substantial: but we have to
  backup the change with the appropriate engineering (such as reducing
  our frictions that cause teams practicing CD to be weeks behind tip)
  to make it feasible. My greatest concern about the proposals happening
  now is that we may bite off more than we can chew. OTGH the reality is
  that all the negative things multiply out, so we probably need to just
  start somewhere and /do/ to give us the space to fix other things.
 
 As I said, I'd very much like us to release a lot more often. I just
 don't see how we can do it without:
 
 - abandoning the idea of translating the software
 - admit that docs will always lag code
 - dropping stable branch maintenance (and only fix vulnerabilities in
 master)

I don't really buy that. If we have 6 month cycles with 1 month freeze
for docs  i18n, vs 2 months cycles with 2 weeks freeze for docs  i18n
the latter is a win in terms of dev time vs stablization time, giving
docs  i18n 50% more time in aggregate.

 That would all make my life (and the life of most feature-focused
 developers) a *lot* easier. I'm not convinced that's the best solution
 for our downstream users, though. I think they like docs,
 translations(?), stable 

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-25 Thread Thierry Carrez
Robert Collins wrote:
 It's also worth noting that we were on a 3-month cycle at the start of
 OpenStack. That was dropped after a cataclysmic release that managed the
 feat of (a) not having anything significant done, and (b) have out of
 date documentation and translations.
 
 Oh! 
 https://wiki.openstack.org/w/index.php?title=Obsolete:3MonthReleaseCycledirection=prevoldid=14016
 appears to be the page about that, but my google-fu isn't turning up a
 post-mortem of the C release which prompted the change. Perhaps some
 old-hand like you can fill us in on the details :).

After the Austin release (a code drop, really), we were on a three-month
time-based cycle with no intermediary milestone. The first cycle (Bexar)
worked relatively well, the second one (Cactus) not so well. There were
multiple reasons for that:

- Software was growing up in Bexar and we introduced more testing, but
we were unable to detect and fix all the release blockers in time for
the release date, resulting in a broken Bexar release. To address that
in Cactus, we increased the duration of the frozen period, but then that
didn't leave enough time for development, hence nothing getting done.

- Not starting a release cycle with a global F2F gathering (no
intermediary Design Summit) actually proved quite disastrous: no reset
of community, no alignment on goal, no discussion on what to work on.
That resulted in a limbo period after the Bexar release. That is why I'm
so insistent on aligning our development cycles with our Design Summits.
I've seen where we go when we don't.

The post-mortem (at the Diablo summit) recognized our inability as a
group (including QA, docs, stable branch, vulnerability management) to
do final release work more often than every 6 months. Milestones were
introduced so that we can communicate features faster to our users
(those wanting fresh stuff could consume milestones), and then focus QA
/ docs / stable support on the final release only.

Now, the funny thing is... this solution is very similar to what is
proposed in this thread. Intermediary releases every two months to get
code out really fast, and final releases which trigger stable
maintenance. So what happened ?

What happened then is what will happen if we reboot the exact same
solution now: developers ignoring milestones (or intermediary releases)
to focus only on the real, stable-supported one. And now milestones
are just dates in a calendar.

So when I say that won't solve anything and devs will keep on focusing
on the real release, it's not just a gut feeling. We've been there. I
like to learn from history and not repeat the same mistakes.

 [...]
 I may appear defensive on my answers, but it's not my goal to defend the
 current system: it's just that most of those proposals generally ignore
 the diversity of the needs of the teams that make OpenStack possible, to
 focus on a particular set of contributors' woes. I'm trying to bring
 that wider perspective in -- the current system is a trade-off and the
 result of years of evolution, not an arbitrary historic choice that we
 can just change at will.
 
 I agree that its not arbitrary and that changing it requires some
 appropriate wide-spread consultation; OTOH the benefits of rolling and
 higher frequency releases are really substantial: but we have to
 backup the change with the appropriate engineering (such as reducing
 our frictions that cause teams practicing CD to be weeks behind tip)
 to make it feasible. My greatest concern about the proposals happening
 now is that we may bite off more than we can chew. OTGH the reality is
 that all the negative things multiply out, so we probably need to just
 start somewhere and /do/ to give us the space to fix other things.

As I said, I'd very much like us to release a lot more often. I just
don't see how we can do it without:

- abandoning the idea of translating the software
- admit that docs will always lag code
- dropping stable branch maintenance (and only fix vulnerabilities in
master)

That would all make my life (and the life of most feature-focused
developers) a *lot* easier. I'm not convinced that's the best solution
for our downstream users, though. I think they like docs,
translations(?), stable branches and backported security fixes.

PS: as a data point, at the last summit, in the stable branch session,
there was a whole group of downstream users crashing the session to ask
for releasing *less* often (we can't keep up). I think that's crazy
too, but that shows 6 months is really a trade-off.

-- 
Thierry Carrez (ttx)

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-24 Thread Sean Dague
On 02/24/2015 03:21 PM, Joe Gordon wrote:
 
 
 On Tue, Feb 24, 2015 at 6:57 AM, Daniel P. Berrange berra...@redhat.com
 mailto:berra...@redhat.com wrote:
 
 On Tue, Feb 24, 2015 at 08:50:45AM -0500, Sean Dague wrote:
  On 02/24/2015 07:48 AM, Russell Bryant wrote:
   On 02/24/2015 12:54 PM, Daniel P. Berrange wrote:
   On Tue, Feb 24, 2015 at 11:48:29AM +, Chris Dent wrote:
   On Tue, 24 Feb 2015, Daniel P. Berrange wrote:
  
   need to do more work. If this is so, then I don't think this
 is a blocker,
   it is just a sign that the project needs to focus on
 providing more resources
   to the teams impacted in that way.
  
   What are the mechanisms whereby the project provides more
 resources
   to teams?
  
   The technical committee and / or foundation board can highlight
 the need
   for investment of resources in critical areas of the project,
 to either
   the community members or vendors involved. As an example, this
 was done
   successfully recently to increase involvement in maintaining
 the EC2
   API support.  There are plenty of vendors involved in OpenStack
 which
   have the ability to target resources, if they can learn where those
   resources are best spent.
  
   Indeed ... and if horizontal teams are the ones hit the most by the
   extra work, each project should help with that burden.  For example,
   projects may need to take their responsibility for documentation
 more
   seriously and require documentation with features (content at
 least, not
   necessarily integration into the proper documentation deliverables)
   instead of assuming it magically gets written later.
 
  Right, and I think this actually hits at the most important part
 of the
  discussion. The question of:
 
  1) what would we need to do to make different release cadences viable?
  2) are those good things to do regardless of release cadence?
 
  The horizontal teams really can't function at different cadences. It
  completely breaks any flow and planning at turns them even further
 into
  firefighting because now everyone has crunch time at different times,
  and the horizontal efforts are required to just play catch up. I know
  what that future looks like, the horizontal teams dry up because
 no one
  wants that job.
 
  Ok, so that being said, what we'd need to do is have horizontal teams
  move to more of a self supporting model. So that the relevant content
  for a project (docs, full stack tests, requirements, etc) all live
  within that project itself, and aren't centrally synchronized.
  Installation of projects needs to be fully isolated from each other so
  that upgrading project A can be done independent of project B, as
 their
  release cadences might all bit disparate. Basically, ever OpenStack
  project needs to reabsorb the cross project efforts they've
 externalized.
 
  Then if project A decided to move off the coupled release, it's impact
  to the rest would be minimal. These are robust components that
 stand on
  their own, and work well with robust other components.
 
  Which... is basically the point of the big tent / new project
 governance
  model. Decompose OpenStack from a giant blob of goo into Robust
 elements
  that are more loosely coupled (so independently robust, and robust in
  their interaction with others). Move the horizontal teams into
  infrastructure vs. content roles, have projects own more of this
 content
  themselves.
 
  But it is a long hard process. Devstack external plugins was implement
  to support this kind of model, but having walked a bunch of different
  teams through this (at all skill levels) there ends up being a lot of
  work to get this right, and a lot of rethinking by teams that assumed
  their interaction with full stack testing is something they'd get to
  contribute once and have someone else maintain (instead of something
  they now need dedicated watchful eye on).
 
  The amount of full stack configurations immediately goes beyond
 anywhere
  near testable, so it requires more robust project testing to ensure
  every exposed interface is more robust (i.e. the testing in pyramids
  from https://review.openstack.org/#/c/150653/).
 
  And, I think the answer to #2 is: yes, this just makes it all better.
 
  So, honestly, I'm massively supportive of the end game. I've been
  carving out the bits of this I can for the last six months. But I
 think
  the way we get there is to actually get the refactoring of the
  horizontal efforts first.
 
 I pretty much fully agree that refactoring the horizontal efforts to
 distribute responsbility across the 

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-24 Thread Joe Gordon
On Tue, Feb 24, 2015 at 6:57 AM, Daniel P. Berrange berra...@redhat.com
wrote:

 On Tue, Feb 24, 2015 at 08:50:45AM -0500, Sean Dague wrote:
  On 02/24/2015 07:48 AM, Russell Bryant wrote:
   On 02/24/2015 12:54 PM, Daniel P. Berrange wrote:
   On Tue, Feb 24, 2015 at 11:48:29AM +, Chris Dent wrote:
   On Tue, 24 Feb 2015, Daniel P. Berrange wrote:
  
   need to do more work. If this is so, then I don't think this is a
 blocker,
   it is just a sign that the project needs to focus on providing more
 resources
   to the teams impacted in that way.
  
   What are the mechanisms whereby the project provides more resources
   to teams?
  
   The technical committee and / or foundation board can highlight the
 need
   for investment of resources in critical areas of the project, to
 either
   the community members or vendors involved. As an example, this was
 done
   successfully recently to increase involvement in maintaining the EC2
   API support.  There are plenty of vendors involved in OpenStack which
   have the ability to target resources, if they can learn where those
   resources are best spent.
  
   Indeed ... and if horizontal teams are the ones hit the most by the
   extra work, each project should help with that burden.  For example,
   projects may need to take their responsibility for documentation more
   seriously and require documentation with features (content at least,
 not
   necessarily integration into the proper documentation deliverables)
   instead of assuming it magically gets written later.
 
  Right, and I think this actually hits at the most important part of the
  discussion. The question of:
 
  1) what would we need to do to make different release cadences viable?
  2) are those good things to do regardless of release cadence?
 
  The horizontal teams really can't function at different cadences. It
  completely breaks any flow and planning at turns them even further into
  firefighting because now everyone has crunch time at different times,
  and the horizontal efforts are required to just play catch up. I know
  what that future looks like, the horizontal teams dry up because no one
  wants that job.
 
  Ok, so that being said, what we'd need to do is have horizontal teams
  move to more of a self supporting model. So that the relevant content
  for a project (docs, full stack tests, requirements, etc) all live
  within that project itself, and aren't centrally synchronized.
  Installation of projects needs to be fully isolated from each other so
  that upgrading project A can be done independent of project B, as their
  release cadences might all bit disparate. Basically, ever OpenStack
  project needs to reabsorb the cross project efforts they've externalized.
 
  Then if project A decided to move off the coupled release, it's impact
  to the rest would be minimal. These are robust components that stand on
  their own, and work well with robust other components.
 
  Which... is basically the point of the big tent / new project governance
  model. Decompose OpenStack from a giant blob of goo into Robust elements
  that are more loosely coupled (so independently robust, and robust in
  their interaction with others). Move the horizontal teams into
  infrastructure vs. content roles, have projects own more of this content
  themselves.
 
  But it is a long hard process. Devstack external plugins was implement
  to support this kind of model, but having walked a bunch of different
  teams through this (at all skill levels) there ends up being a lot of
  work to get this right, and a lot of rethinking by teams that assumed
  their interaction with full stack testing is something they'd get to
  contribute once and have someone else maintain (instead of something
  they now need dedicated watchful eye on).
 
  The amount of full stack configurations immediately goes beyond anywhere
  near testable, so it requires more robust project testing to ensure
  every exposed interface is more robust (i.e. the testing in pyramids
  from https://review.openstack.org/#/c/150653/).
 
  And, I think the answer to #2 is: yes, this just makes it all better.
 
  So, honestly, I'm massively supportive of the end game. I've been
  carving out the bits of this I can for the last six months. But I think
  the way we get there is to actually get the refactoring of the
  horizontal efforts first.

 I pretty much fully agree that refactoring the horizontal efforts to
 distribute responsbility across the individual projects is the way
 forward. I don't think it needs to be a pre-requisite step for changing
 the release cycle. We can do both in parallel if we put our minds to
 it.

 My biggest fear is that we just keeping debating problems and alternatives,
 but continue to be too afraid of theoretical problems, to actually take the
 risk of effecting meaningful improvemnts in the operation of the project.


I tend to agree with you on this.  I don't think completing the refactoring
of the horizontal 

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-24 Thread Russell Bryant
On 02/24/2015 01:28 PM, Kashyap Chamarthy wrote:
 On Tue, Feb 24, 2015 at 11:54:31AM +, Daniel P. Berrange wrote:
 On Tue, Feb 24, 2015 at 11:48:29AM +, Chris Dent wrote:
 On Tue, 24 Feb 2015, Daniel P. Berrange wrote:

 need to do more work. If this is so, then I don't think this is a blocker,
 it is just a sign that the project needs to focus on providing more 
 resources
 to the teams impacted in that way.

 What are the mechanisms whereby the project provides more resources
 to teams?
 
 Along with the below, if push comes to shove, OpenStack Foundation could
 probably try a milder variant (obviously, not all activities can be
 categorized as 'critical path') of Linux Foundation's Critical
 Infrastructure Protection Initiative[1] to fund certain project
 activities in need.

The OpenStack Foundation effectively already does this.  In particular,
the Foundation is helping fund critical horizontal efforts like release
management, infrastructure, and community management.

-- 
Russell Bryant

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-24 Thread Kashyap Chamarthy
On Tue, Feb 24, 2015 at 11:54:31AM +, Daniel P. Berrange wrote:
 On Tue, Feb 24, 2015 at 11:48:29AM +, Chris Dent wrote:
  On Tue, 24 Feb 2015, Daniel P. Berrange wrote:
  
  need to do more work. If this is so, then I don't think this is a blocker,
  it is just a sign that the project needs to focus on providing more 
  resources
  to the teams impacted in that way.
  
  What are the mechanisms whereby the project provides more resources
  to teams?

Along with the below, if push comes to shove, OpenStack Foundation could
probably try a milder variant (obviously, not all activities can be
categorized as 'critical path') of Linux Foundation's Critical
Infrastructure Protection Initiative[1] to fund certain project
activities in need.
 
 The technical committee and / or foundation board can highlight the
 need for investment of resources in critical areas of the project, to
 either the community members or vendors involved. As an example, this
 was done successfully recently to increase involvement in maintaining
 the EC2 API support.  There are plenty of vendors involved in
 OpenStack which have the ability to target resources, if they can
 learn where those resources are best spent.
 

[1] http://www.linuxfoundation.org/programs/core-infrastructure-initiative

-- 
/kashyap

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-24 Thread Flavio Percoco

On 24/02/15 11:02 +, Daniel P. Berrange wrote:

On Tue, Feb 24, 2015 at 10:44:57AM +, Chris Dent wrote:

On Tue, 24 Feb 2015, Daniel P. Berrange wrote:

I was writing this mail for the past few days, but the nova thread
today prompted me to finish it off  send it :-)

Thanks for doing this. I think you're probably right that the current
release cycle has many negative impacts on the development process and
deserve at least some critical thinking if not outright changing.

Thanks especially for listing some expected questions (with answers).

One additional question I'd like to see answered in some depth is:
Why have unified release schedules? That is, why should Nova and Glance
or anything else release on the same schedule as any other
OpenStack-related project?

Disentangling the release cycles might lead to stronger encapsulation
of, and stronger contracts between, the projects. It might also lead to
a total mess.


For peripheral projects I don't think co-ordinate release cycle is needed,
but for the core projects I think it is helpful in general to have some
co-ordination of releases. It allows marketing to more effectively promote
the new project releases, it helps when landing features that span across
projects to know they'll be available to users at the same time in general
and minimize burden on devs  users to remember many different dates. It
is hard enough remembering the dates for our coordinated release cycle,
let alone dates for 30 different project cycles. IME predictable dates is
a really important  useful thing to have for a planning POV. This is why
I suggested, we do a 2 month cycle, with a strict date of 1st of the month
in Feb, Apr, Jun, Aug, Oct, Dec.


To this I'd also add that bug fixing is way easier when you have
aligned releases for projects that are expected to be deployed
together. It's easier to know what the impact of a change/bug is
throughout the infrastructure.

Flavio



Regards,
Daniel
--
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


--
@flaper87
Flavio Percoco


pgpFaVXWQm9Ii.pgp
Description: PGP signature
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-24 Thread Daniel P. Berrange
On Tue, Feb 24, 2015 at 11:48:29AM +, Chris Dent wrote:
 On Tue, 24 Feb 2015, Daniel P. Berrange wrote:
 
 need to do more work. If this is so, then I don't think this is a blocker,
 it is just a sign that the project needs to focus on providing more resources
 to the teams impacted in that way.
 
 What are the mechanisms whereby the project provides more resources
 to teams?

The technical committee and / or foundation board can highlight the need
for investment of resources in critical areas of the project, to either
the community members or vendors involved. As an example, this was done
successfully recently to increase involvement in maintaining the EC2
API support.  There are plenty of vendors involved in OpenStack which
have the ability to target resources, if they can learn where those
resources are best spent.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-24 Thread Chris Dent

On Tue, 24 Feb 2015, Daniel P. Berrange wrote:


I was writing this mail for the past few days, but the nova thread
today prompted me to finish it off  send it :-)


Thanks for doing this. I think you're probably right that the current
release cycle has many negative impacts on the development process and
deserve at least some critical thinking if not outright changing.

Thanks especially for listing some expected questions (with answers).

One additional question I'd like to see answered in some depth is:
Why have unified release schedules? That is, why should Nova and Glance
or anything else release on the same schedule as any other
OpenStack-related project?

Disentangling the release cycles might lead to stronger encapsulation
of, and stronger contracts between, the projects. It might also lead to
a total mess.

Thanks.
--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-24 Thread Chris Dent

On Tue, 24 Feb 2015, Daniel P. Berrange wrote:


need to do more work. If this is so, then I don't think this is a blocker,
it is just a sign that the project needs to focus on providing more resources
to the teams impacted in that way.


What are the mechanisms whereby the project provides more resources
to teams?

--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-24 Thread Russell Bryant
On 02/24/2015 12:54 PM, Daniel P. Berrange wrote:
 On Tue, Feb 24, 2015 at 11:48:29AM +, Chris Dent wrote:
 On Tue, 24 Feb 2015, Daniel P. Berrange wrote:

 need to do more work. If this is so, then I don't think this is a blocker,
 it is just a sign that the project needs to focus on providing more 
 resources
 to the teams impacted in that way.

 What are the mechanisms whereby the project provides more resources
 to teams?
 
 The technical committee and / or foundation board can highlight the need
 for investment of resources in critical areas of the project, to either
 the community members or vendors involved. As an example, this was done
 successfully recently to increase involvement in maintaining the EC2
 API support.  There are plenty of vendors involved in OpenStack which
 have the ability to target resources, if they can learn where those
 resources are best spent.

Indeed ... and if horizontal teams are the ones hit the most by the
extra work, each project should help with that burden.  For example,
projects may need to take their responsibility for documentation more
seriously and require documentation with features (content at least, not
necessarily integration into the proper documentation deliverables)
instead of assuming it magically gets written later.

-- 
Russell Bryant

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-24 Thread Daniel P. Berrange
On Tue, Feb 24, 2015 at 08:50:45AM -0500, Sean Dague wrote:
 On 02/24/2015 07:48 AM, Russell Bryant wrote:
  On 02/24/2015 12:54 PM, Daniel P. Berrange wrote:
  On Tue, Feb 24, 2015 at 11:48:29AM +, Chris Dent wrote:
  On Tue, 24 Feb 2015, Daniel P. Berrange wrote:
 
  need to do more work. If this is so, then I don't think this is a 
  blocker,
  it is just a sign that the project needs to focus on providing more 
  resources
  to the teams impacted in that way.
 
  What are the mechanisms whereby the project provides more resources
  to teams?
 
  The technical committee and / or foundation board can highlight the need
  for investment of resources in critical areas of the project, to either
  the community members or vendors involved. As an example, this was done
  successfully recently to increase involvement in maintaining the EC2
  API support.  There are plenty of vendors involved in OpenStack which
  have the ability to target resources, if they can learn where those
  resources are best spent.
  
  Indeed ... and if horizontal teams are the ones hit the most by the
  extra work, each project should help with that burden.  For example,
  projects may need to take their responsibility for documentation more
  seriously and require documentation with features (content at least, not
  necessarily integration into the proper documentation deliverables)
  instead of assuming it magically gets written later.
 
 Right, and I think this actually hits at the most important part of the
 discussion. The question of:
 
 1) what would we need to do to make different release cadences viable?
 2) are those good things to do regardless of release cadence?
 
 The horizontal teams really can't function at different cadences. It
 completely breaks any flow and planning at turns them even further into
 firefighting because now everyone has crunch time at different times,
 and the horizontal efforts are required to just play catch up. I know
 what that future looks like, the horizontal teams dry up because no one
 wants that job.
 
 Ok, so that being said, what we'd need to do is have horizontal teams
 move to more of a self supporting model. So that the relevant content
 for a project (docs, full stack tests, requirements, etc) all live
 within that project itself, and aren't centrally synchronized.
 Installation of projects needs to be fully isolated from each other so
 that upgrading project A can be done independent of project B, as their
 release cadences might all bit disparate. Basically, ever OpenStack
 project needs to reabsorb the cross project efforts they've externalized.
 
 Then if project A decided to move off the coupled release, it's impact
 to the rest would be minimal. These are robust components that stand on
 their own, and work well with robust other components.
 
 Which... is basically the point of the big tent / new project governance
 model. Decompose OpenStack from a giant blob of goo into Robust elements
 that are more loosely coupled (so independently robust, and robust in
 their interaction with others). Move the horizontal teams into
 infrastructure vs. content roles, have projects own more of this content
 themselves.
 
 But it is a long hard process. Devstack external plugins was implement
 to support this kind of model, but having walked a bunch of different
 teams through this (at all skill levels) there ends up being a lot of
 work to get this right, and a lot of rethinking by teams that assumed
 their interaction with full stack testing is something they'd get to
 contribute once and have someone else maintain (instead of something
 they now need dedicated watchful eye on).
 
 The amount of full stack configurations immediately goes beyond anywhere
 near testable, so it requires more robust project testing to ensure
 every exposed interface is more robust (i.e. the testing in pyramids
 from https://review.openstack.org/#/c/150653/).
 
 And, I think the answer to #2 is: yes, this just makes it all better.
 
 So, honestly, I'm massively supportive of the end game. I've been
 carving out the bits of this I can for the last six months. But I think
 the way we get there is to actually get the refactoring of the
 horizontal efforts first.

I pretty much fully agree that refactoring the horizontal efforts to
distribute responsbility across the individual projects is the way
forward. I don't think it needs to be a pre-requisite step for changing
the release cycle. We can do both in parallel if we put our minds to
it.

My biggest fear is that we just keeping debating problems and alternatives,
but continue to be too afraid of theoretical problems, to actually take the
risk of effecting meaningful improvemnts in the operation of the project.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ 

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-24 Thread Chris Dent

On Tue, 24 Feb 2015, Sean Dague wrote:


That also provides a very concrete answer to will people show up.
Because if they do, and we get this horizontal refactoring happening,
then we get to the point of being able to change release cadences
faster. If they don't, we remain with the existing system. Vs changing
the system and hoping someone is going to run in and backfill the breaks.


Isn't this the way of the world? People only put halon in the
machine room after the fire.

I agree that people showing up is a real concern, but I also think
that we shy away too much from the productive energy of stuff
breaking. It's the breakage that shows where stuff isn't good
enough.

[Flavio said]:

To this I'd also add that bug fixing is way easier when you have
aligned releases for projects that are expected to be deployed
together. It's easier to know what the impact of a change/bug is
throughout the infrastructure.


Can't this be interpreted as an excuse for making software which
does not have a low surface area and a good API?

(Note I'm taking a relatively unrealistic position for sake of
conversation.)

--
Chris Dent tw:@anticdent freenode:cdent
https://tank.peermore.com/tanks/cdent

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-24 Thread David Kranz

On 02/24/2015 09:37 AM, Chris Dent wrote:

On Tue, 24 Feb 2015, Sean Dague wrote:


That also provides a very concrete answer to will people show up.
Because if they do, and we get this horizontal refactoring happening,
then we get to the point of being able to change release cadences
faster. If they don't, we remain with the existing system. Vs changing
the system and hoping someone is going to run in and backfill the 
breaks.


Isn't this the way of the world? People only put halon in the
machine room after the fire.

I agree that people showing up is a real concern, but I also think
that we shy away too much from the productive energy of stuff
breaking. It's the breakage that shows where stuff isn't good
enough.

[Flavio said]:

To this I'd also add that bug fixing is way easier when you have
aligned releases for projects that are expected to be deployed
together. It's easier to know what the impact of a change/bug is
throughout the infrastructure.


Can't this be interpreted as an excuse for making software which
does not have a low surface area and a good API?

(Note I'm taking a relatively unrealistic position for sake of
conversation.)
I'm not so sure about that. IMO, much of this goes back to the question 
of whether OpenStack services are APIs or implementations. This was 
debated with much heat at the Diablo summit (Hi Jay). I frequently have 
conversations where there is an issue about release X vs Y when it is 
really about api versions. Even if we say that we are about 
implementations as well as apis, we can start to organize our processes 
and code as if we were just apis. If each service had a well-defined, 
versioned, discoverable, well-tested api, then projects could follow 
their own release schedule, relying on distros or integrators to put the 
pieces together and verify the quality of the whole stack to the users. 
Such entities could still collaborate on that task, and still identify 
longer release cycles, using stable branches. The upstream project 
could still test the latest released versions together. Some of these 
steps are now being taken to resolve gate issues and horizontal resource 
issues. Doing this would vastly increase agility but with some costs:


1. The upstream project would likely have to give up on the worthy goal 
of providing an actual deployable stack that could be used as an 
alternative to AWS, etc. That saddens me, but for various reasons, 
including that we do no scale/performance testing on the upstream code, 
we are not achieving that goal anyway. The big tent proposals are also a 
move away from that goal.


2. We would have to give up on incompatible api changes. But with the 
replacement of nova v3 with microversions we are already doing that. 
Massive adoption with release agility is simply incompatible with 
allowing incompatible api changes.


Most of this is just echoing what Jay said. I think this is the way any 
SOA would be designed. If we did this, and projects released frequently, 
would there be a reason for any one to be chasing master?


 -David


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-24 Thread John Griffith
On Tue, Feb 24, 2015 at 10:04 AM, David Kranz dkr...@redhat.com wrote:

 On 02/24/2015 09:37 AM, Chris Dent wrote:

 On Tue, 24 Feb 2015, Sean Dague wrote:

  That also provides a very concrete answer to will people show up.
 Because if they do, and we get this horizontal refactoring happening,
 then we get to the point of being able to change release cadences
 faster. If they don't, we remain with the existing system. Vs changing
 the system and hoping someone is going to run in and backfill the breaks.


 Isn't this the way of the world? People only put halon in the
 machine room after the fire.

 I agree that people showing up is a real concern, but I also think
 that we shy away too much from the productive energy of stuff
 breaking. It's the breakage that shows where stuff isn't good
 enough.

 [Flavio said]:

 To this I'd also add that bug fixing is way easier when you have
 aligned releases for projects that are expected to be deployed
 together. It's easier to know what the impact of a change/bug is
 throughout the infrastructure.


 Can't this be interpreted as an excuse for making software which
 does not have a low surface area and a good API?

 (Note I'm taking a relatively unrealistic position for sake of
 conversation.)

 I'm not so sure about that. IMO, much of this goes back to the question of
 whether OpenStack services are APIs or implementations. This was debated
 with much heat at the Diablo summit (Hi Jay). I frequently have
 conversations where there is an issue about release X vs Y when it is
 really about api versions. Even if we say that we are about implementations
 as well as apis, we can start to organize our processes and code as if we
 were just apis. If each service had a well-defined, versioned,
 discoverable, well-tested api, then projects could follow their own release
 schedule, relying on distros or integrators to put the pieces together and
 verify the quality of the whole stack to the users. Such entities could
 still collaborate on that task, and still identify longer release cycles,
 using stable branches. The upstream project could still test the latest
 released versions together. Some of these steps are now being taken to
 resolve gate issues and horizontal resource issues. Doing this would vastly
 increase agility but with some costs:

 1. The upstream project would likely have to give up on the worthy goal of
 providing an actual deployable stack that could be used as an alternative
 to AWS, etc. That saddens me, but for various reasons, including that we do
 no scale/performance testing on the upstream code, we are not achieving
 that goal anyway. The big tent proposals are also a move away from that
 goal.

 2. We would have to give up on incompatible api changes. But with the
 replacement of nova v3 with microversions we are already doing that.
 Massive adoption with release agility is simply incompatible with allowing
 incompatible api changes.

 Most of this is just echoing what Jay said. I think this is the way any
 SOA would be designed. If we did this, and projects released frequently,
 would there be a reason for any one to be chasing master?

  -David



 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



​Seems like some of the proposals around frequency (increasing in
particular) just sort of move the bottlenecks around.  Honestly, I thought
we were already on a path with some of the ideas that Sean and David (and
indirectly Jay) proposed.  Get rid of the whole coordinated release
altogether.  I think there should still be some sort of tagging or
something at some interval that just says here's a point in time
collection that we call X.

Another proposal I think I've talked to some folks about is a true CI/Train
model.  Cut out some of the artificial milestone deadlines etc, just keep
rolling and what's ready at the release point is what's ready; you make a
cut of what's there at that time and roll on.  Basically eliminate the
feature freeze and other components and hopefully keep feature commit
distributed.  There are certainly all sorts of gotchas here, but I don't
think it's very interesting to most so I won't go into a bunch of theory on
it.

Regardless, I do think that no matter the direction we all seem to be of
the opinion that we need to move more towards projects being responsible
for more of the horizontal functions (as Sean put it).  Personally I think
this is a great direction for a number of reasons, and I also think that it
might force us to be better about our API's and requirements than we have
been in the past.
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: 

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-24 Thread Sean Dague
On 02/24/2015 12:33 PM, John Griffith wrote:
 
 
 On Tue, Feb 24, 2015 at 10:04 AM, David Kranz dkr...@redhat.com
 mailto:dkr...@redhat.com wrote:
 
 On 02/24/2015 09:37 AM, Chris Dent wrote:
 
 On Tue, 24 Feb 2015, Sean Dague wrote:
 
 That also provides a very concrete answer to will people
 show up.
 Because if they do, and we get this horizontal refactoring
 happening,
 then we get to the point of being able to change release
 cadences
 faster. If they don't, we remain with the existing system.
 Vs changing
 the system and hoping someone is going to run in and
 backfill the breaks.
 
 
 Isn't this the way of the world? People only put halon in the
 machine room after the fire.
 
 I agree that people showing up is a real concern, but I also think
 that we shy away too much from the productive energy of stuff
 breaking. It's the breakage that shows where stuff isn't good
 enough.
 
 [Flavio said]:
 
 To this I'd also add that bug fixing is way easier when you have
 aligned releases for projects that are expected to be deployed
 together. It's easier to know what the impact of a change/bug is
 throughout the infrastructure.
 
 
 Can't this be interpreted as an excuse for making software which
 does not have a low surface area and a good API?
 
 (Note I'm taking a relatively unrealistic position for sake of
 conversation.)
 
 I'm not so sure about that. IMO, much of this goes back to the
 question of whether OpenStack services are APIs or implementations.
 This was debated with much heat at the Diablo summit (Hi Jay). I
 frequently have conversations where there is an issue about release
 X vs Y when it is really about api versions. Even if we say that we
 are about implementations as well as apis, we can start to organize
 our processes and code as if we were just apis. If each service had
 a well-defined, versioned, discoverable, well-tested api, then
 projects could follow their own release schedule, relying on distros
 or integrators to put the pieces together and verify the quality of
 the whole stack to the users. Such entities could still collaborate
 on that task, and still identify longer release cycles, using
 stable branches. The upstream project could still test the latest
 released versions together. Some of these steps are now being taken
 to resolve gate issues and horizontal resource issues. Doing this
 would vastly increase agility but with some costs:
 
 1. The upstream project would likely have to give up on the worthy
 goal of providing an actual deployable stack that could be used as
 an alternative to AWS, etc. That saddens me, but for various
 reasons, including that we do no scale/performance testing on the
 upstream code, we are not achieving that goal anyway. The big tent
 proposals are also a move away from that goal.
 
 2. We would have to give up on incompatible api changes. But with
 the replacement of nova v3 with microversions we are already doing
 that. Massive adoption with release agility is simply incompatible
 with allowing incompatible api changes.
 
 Most of this is just echoing what Jay said. I think this is the way
 any SOA would be designed. If we did this, and projects released
 frequently, would there be a reason for any one to be chasing master?
 
  -David
 
 
 
 
 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe:
 OpenStack-dev-request@lists.__openstack.org?subject:__unsubscribe
 http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev 
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
 
 
 
 ​Seems like some of the proposals around frequency (increasing in
 particular) just sort of move the bottlenecks around.  Honestly, I
 thought we were already on a path with some of the ideas that Sean and
 David (and indirectly Jay) proposed.  Get rid of the whole coordinated
 release altogether.  I think there should still be some sort of tagging
 or something at some interval that just says here's a point in time
 collection that we call X.
 
 Another proposal I think I've talked to some folks about is a true
 CI/Train model.  Cut out some of the artificial milestone deadlines etc,
 just keep rolling and what's ready at the release point is what's ready;
 you make a cut of what's there at that time and roll on.  Basically
 eliminate the feature freeze and other components and hopefully keep
 feature commit distributed.  There are certainly all sorts of gotchas
 

Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-24 Thread Mark Atwood
On Tue, Feb 24, 2015, at 04:28, Kashyap Chamarthy wrote:
 
 Along with the below, if push comes to shove, OpenStack Foundation could
 probably try a milder variant (obviously, not all activities can be
 categorized as 'critical path') of Linux Foundation's Critical
 Infrastructure Protection Initiative[1] to fund certain project
 activities in need.

Speaking as a person who sits on the LF CII board meetings,
and helps turn the crank on that particular sausage mill,
no, we really don't want to go down that path at this point in
time.

-- 
Mark Atwood, Director of Open Source Engagement, HP

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-24 Thread Robert Collins
On 25 February 2015 at 13:13, Jeremy Stanley fu...@yuggoth.org wrote:
 On 2015-02-24 11:27:05 + (+), Daniel P. Berrange wrote:
 [...]
 It would be reasonable for the vulnerability team to take the decision
 that they'll support fixes for master, and any branches that the stable
 team decide to support.
 [...]

 Well, it's worth noting that the VMT doesn't even support (i.e.
 issue advisories for bugs in) master branches now, the exception
 being branchless projects where the bug appears in master prior to
 an existing release tag.

 But I think Thierry's earlier point is that as soon as you start
 marking _some_ releases as special (supported by VMT, stable maint,
 docs, translators...) then those become your new actual releases and
 the other interim releases become your milestones, and we're back to
 the current model again.

I don't think thats true actually. We'd still have a major smoothing
effect on work, which means less high peaks at release time and less
troughs at 'review direction' time and so on.

-Rob


-- 
Robert Collins rbtcoll...@hp.com
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-24 Thread Jeremy Stanley
On 2015-02-24 11:27:05 + (+), Daniel P. Berrange wrote:
[...]
 It would be reasonable for the vulnerability team to take the decision
 that they'll support fixes for master, and any branches that the stable
 team decide to support.
[...]

Well, it's worth noting that the VMT doesn't even support (i.e.
issue advisories for bugs in) master branches now, the exception
being branchless projects where the bug appears in master prior to
an existing release tag.

But I think Thierry's earlier point is that as soon as you start
marking _some_ releases as special (supported by VMT, stable maint,
docs, translators...) then those become your new actual releases and
the other interim releases become your milestones, and we're back to
the current model again.
-- 
Jeremy Stanley

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-24 Thread Robert Collins
On 24 February 2015 at 22:53, Daniel P. Berrange berra...@redhat.com wrote:
 I was writing this mail for the past few days, but the nova thread
 today prompted me to finish it off  send it :-)

++


 The first two observations strongly suggest that the choice of 6
 months as a cycle length is a fairly arbitrary decision that can be
 changed without unreasonable pain. The third observation suggests a
 much shorter cycle length would smooth out the bumps and lead to a
 more efficient  satisfying development process for all involved.

I'm very glad to see this being discussed (again :)). Any proposal
that reduces our cycle time is going to get my enthusiastic support.

...
 Upgrades  deprecation
 --

 It is common right now for projects to say upgrades are only
 supported between releases N-1 and N. ie to go from Icehouse
 to Kilo, you need to first deploy Juno. This is passable when
 you're talking 6 month gaps between cycles, but when there are
 2 month gaps it is not reasonable to expect everyone to be
 moving fast enough to keep up with every release. If an
 organization's beurocracy means they can't deploy more often
 than every 12 months, forcing them to deploy the 5 intermediate
 releases to run upgrade scripts is quite unpleasant. We would
 likely have to look at officially allowing upgrades between
 any (N-3, N-2, N-1) to N. From a database POV this should not
 be hard, since the DB migration scripts don't have any built
 in assumptions about this. Similarly the versioned objects used
 by Nova are quite flexible in this regard too, as long as the
 compat code isn't deleted too soon.

 Deprecation warnings would need similar consideration. It would
 not be sufficient to deprecate in one release and delete in the
 next. We'd likely want to say that depecations last for a given
 time period rather than number of releases, eg 6 months. This
 could be easily handled by simply including the date of initial
 deprecation in the deprecation message. It would thus be clear
 when the feature will be removed to all involved.

I think a useful thing here is to consider online upgrades vs offline
upgrades. If we care to we could say that online upgrades are only
supported release to release, with offline upgrades being supported by
releases up to 8 months apart. The benefit of this would be to reduce
our test matrix: online upgrades are subject to much greater
interactions between concurrent processes, and its hard enough to
validate that N - N+1 works with any deep confidence vs also checking
that N-N+2, N-N+3 also work: for a 6 month sliding window to match
the current thing, we need to allow upgrades from Feb through August:
a Feb-Apr
b Feb-Jun
c Feb-Aug
d Apr- Jun
e Apr-Aug
f Jun-Aug

We'd need to be testing all the combinations leading to the branch a
patch is for, so changes to Aug would need c, e and f all tested.


Thats 3 times the overhead of supporting:
Feb-Apr and then
Apr-Jun and then
Jun-Aug
serially where we'd only be testing one combination at a time.

-Rob


-- 
Robert Collins rbtcoll...@hp.com
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-24 Thread Tom Fifield
On 24/02/15 19:27, Daniel P. Berrange wrote:
 On Tue, Feb 24, 2015 at 12:05:17PM +0100, Thierry Carrez wrote:
 Daniel P. Berrange wrote:
 [...]

First, Daniel, thank you for the well-written and thought-through post.
I have some comments on translation specifically which I hope can shed
some light on this particular horizontal effort.

With the idea as stated below implemented, I think it would basically
demoralise our translation teams to the point they'd give up. We already
get complaints about people not respecting string freeze as it is :D


 I'm not familiar with how the translations works, but if they are
 waiting until the freeze before starting translation work I'd
 say that is a mistaken approach. Obviously during active dev part
 of the cycle, some translated strings are in flux, so if translation
 was taking place in parallel there could be some wasted effort, but
 I'd expect that to be the minority case. I think the majority of
 translation work can be done in parallel with dev work and the freeze
 time just needs to tie up the small remaining bits.


So, two points:

1) We wouldn't be talking about throwing just a couple of percent of
their work away.

As an example, even without looking at the introduction of new strings
or deleting others, you may not be aware that changing a single word in
a string in the code means that entire string needs to be re-translated.
Even with the extensive translation memory systems we have making
suggestions as best they can, we're talking about very, very significant
amounts of wasted effort. Something as simple as adding ing on a
verb to fix an English grammar mistake means a completely different
sentence in languages I'm familiar with.


2) The overwhelming majority of our [hundreds of] translators are
volunteers.

Unlike many of those writing the software, they are not paid to do what
they do, and do it in their spare time. Make it less fun, and they
simply walk away.



To try and put this in a way that may be more understandable for a
non-translator ... your (impressive!) original email in this thread was
around 3000 words. Translatable strings in horizon is around five times
that at the moment. So imagine that, when writing an email five times
longer than the one you wrote, unpaid, someone you don't really know
that well decided that the section on the key observations (230 words
- about 1% of the text of our 'horizon') you just wrote should be
re-arranged - the order of the observations changed, one of them removed
and replaced with another slightly different one, and the conclusion
paragraph wording should be amended to suit.

It would be an understatement to say that such behaviour would be
'annoying' if it happened constantly as you were writing your email.
Consider then if it applied to every email you sought to write :)

Now, the amount of string changes within a release can be left for
someone to work out, but it's certainly a great deal more than a single
percent. Multiply that poor experience by the reality of string change
across all the projects we want to translate. Then multiply it by the
number of languages we want. Finally, multiply it by the number of
people we need to translate a single language. That's a really bad time
for a whole lot of people.


At the moment we're fixing this with string freeze, and that's generally
going pretty well. Right now I don't have a good solution if the strings
in code never stop changing for any period of time, but what has been
proposed above while well-meaning is unfortunately not workable.

We really need to keep our translators as happy as we can. People who
are literate in multiple languages are difficult to find, moreso those
who have technical vocabulary experience in both, and even moreso those
who'll give up their time to help us reach our mission goal of being the
ubiquitous Open Source Cloud Computing platform. We do need them to
achieve it.


Regards,


Tom

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle

2015-02-24 Thread Robert Collins
On 25 February 2015 at 00:05, Thierry Carrez thie...@openstack.org wrote:
 Daniel P. Berrange wrote:
 [...]


 I think you're judging the cycle from the perspective of developers
 only.

You said that in the other thread, and I think its false. There is a
very good case to be made that the release cycle is directly
responsible for:
 - increased feature latency [when users get new shiny] - already
explained here.
 - decreased stability due to feature crush
 - decreased quality for the same reason - we don't put things in and
bake them before exposing, we get them in and since there is a long
gap before they can be consumed they're generally just enabled
immediately. Keystone's steps to address that are excellent, but made
harder by the gap between releases: there is social pressure to make
things graduate to support within each release.

  6 months was not an arbitrary decision. Translations and
 documentation teams basically need a month of feature/string freeze in
 order to complete their work. Since we can't reasonably freeze one month
 every 2 months, we picked 6 months.

Do they need a month because 6 months worth of changes have been
aggregated? E.g. if they have 2 months of changes to deal with, do
they need 1.5 weeks?

Perhaps we could branch the code release in advance of docs and
translations, and announce the release when those artifacts are ready?

 It's also worth noting that we were on a 3-month cycle at the start of
 OpenStack. That was dropped after a cataclysmic release that managed the
 feat of (a) not having anything significant done, and (b) have out of
 date documentation and translations.

Oh! 
https://wiki.openstack.org/w/index.php?title=Obsolete:3MonthReleaseCycledirection=prevoldid=14016
appears to be the page about that, but my google-fu isn't turning up a
post-mortem of the C release which prompted the change. Perhaps some
old-hand like you can fill us in on the details :).

 Stable branches may have the luxury of skipping releases and designate a
 stable one from time to time (I reject the Linux comparison because
 the kernel is at a very different moment in software lifecycle). The
 trick being, making one release special is sure to recreate the peak
 issues you're trying to solve.

 I'm actually more concerned about the teams that don't have the luxury
 of skipping a release (like the vulnerability management team). Docs and
 translations, as mentioned above, will also have a hard time producing
 quality docs and translations every 2 months with a very short freeze
 period.

I agree that the vulnerability team will be rather more impacted by
this than regular project teams - I made a nod to that in the other
thread. More thought needed still :).


 I may appear defensive on my answers, but it's not my goal to defend the
 current system: it's just that most of those proposals generally ignore
 the diversity of the needs of the teams that make OpenStack possible, to
 focus on a particular set of contributors' woes. I'm trying to bring
 that wider perspective in -- the current system is a trade-off and the
 result of years of evolution, not an arbitrary historic choice that we
 can just change at will.

I agree that its not arbitrary and that changing it requires some
appropriate wide-spread consultation; OTOH the benefits of rolling and
higher frequency releases are really substantial: but we have to
backup the change with the appropriate engineering (such as reducing
our frictions that cause teams practicing CD to be weeks behind tip)
to make it feasible. My greatest concern about the proposals happening
now is that we may bite off more than we can chew. OTGH the reality is
that all the negative things multiply out, so we probably need to just
start somewhere and /do/ to give us the space to fix other things.

-Rob

-- 
Robert Collins rbtcoll...@hp.com
Distinguished Technologist
HP Converged Cloud

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev