Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
Excerpts from Thierry Carrez's message of 2015-03-04 02:19:48 -0800: James Bottomley wrote: On Tue, 2015-03-03 at 11:59 +0100, Thierry Carrez wrote: James Bottomley wrote: Actually, this is possible: look at Linux, it freezes for 10 weeks of a 12 month release cycle (or 6 weeks of an 8 week one). More on this below. I'd be careful with comparisons with the Linux kernel. First it's a single bit of software, not a collection of interconnected projects. Well, we do have interconnection: the kernel on it's own doesn't do anything without a userspace. The theory was that we didn't have to be like BSD (coupled user space and kernel) and we could rely on others (principally the GNU project in the early days) to provide the userspace and that we could decouple kernel development from the userspace releases. Threading models were, I think, the biggest challenges to this assumption, but we survived. Right. My point there is that you only release one thing. We release a lot more pieces. There is (was?) downstream value in coordinating those releases, which is a factor in our ability to do it more often than twice a year. I think the value of coordinated releases has been agreed upon for a long time. This thread is more about the cost, don't you think? Second it's at a very different evolution/maturity point (20 years old vs. 0-4 years old for OpenStack projects). Yes, but I thought I covered this in the email: you can see that at the 4 year point in its lifecycle, the kernel was behaving very differently (and in fact more similar to OpenStack). The question I thought was still valid is whether anything was learnable from the way the kernel evolved later. I think the key issue, which you seem to have in OpenStack is that the separate develop/stabilise phases caused frustration to build up in our system which (nine years later) led the kernel to adopt the main branch stabilisation with overlapping subsystem development cycle. I agree with you: the evolution the kernel went through is almost a natural law, and I know we won't stay in the current model forever. I'm just not sure we have reached the level of general stability that makes it possible to change *just now*. I welcome brainstorming and discussion on future evolutions, though, and intend to lead a cross-project session discussion on that in Vancouver. I don't believe that the kernel reached maturity as a point of eventuality. Just like humans aren't going to jump across the grand canyon no matter how strong they get, it will take a concerted effort that may put other goals on hold to build a bridge. With the kernel there was a clear moment where leadership had tried a few things and then just had to make it clear that all the code goes in one place, but instability would not be tolerated. They crossed that chasm, and while there have been chaotic branches and ruffled feathers, once everybody got over the paradox, it's been business as usual since then with the model James describes. I think the less mature a project is, the wider that chasm is, but I don't think it's ever going to be an easy thing to do. Since we don't have a dictator to force us to cross the chasm, we should really think about planning for the crossing ASAP. Finally it sits at a different layer, so there is less need for documentation/translations to be shipped with the software release. It's certainly a lot less than you, but we have the entire system call man pages. It's an official project of the kernel: https://www.kernel.org/doc/man-pages/ And we maintain translations for it https://www.kernel.org/doc/man-pages/translations.html By translations I meant strings in the software itself, not doc translations. We don't translate docs upstream either :) I guess we could drop those (and/or downstream them in a way) if that was the last thing holding up adding more agility. So in summary, yes we can (and do) learn from kernel history, but those projects are sufficiently different that the precise timeframes and numbers can't really be compared. Apples and oranges are both fruits which mature (and rot if left unchecked), but they evolve at different speeds :) I'm not super excited about being an apple or an orange, since neither are sentient and thus cannot collaborate on a better existence than rotting. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
- Original Message - From: Thierry Carrez thie...@openstack.org To: James Bottomley james.bottom...@hansenpartnership.com It's certainly a lot less than you, but we have the entire system call man pages. It's an official project of the kernel: https://www.kernel.org/doc/man-pages/ And we maintain translations for it https://www.kernel.org/doc/man-pages/translations.html By translations I meant strings in the software itself, not doc translations. We don't translate docs upstream either :) I guess we could drop those (and/or downstream them in a way) if that was the last thing holding up adding more agility. There is actually a group of contributors working on translation of documentation and translations in various stages of completeness are available at docs.openstack.org (hit the more releases and languages drop down to find them). The challenge for the translators of course is they are trying to hit a target that is moving not just until the release but often well beyond it as the documentation team themselves try to catch up. -Steve __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On Wed, 2015-03-04 at 11:19 +0100, Thierry Carrez wrote: James Bottomley wrote: On Tue, 2015-03-03 at 11:59 +0100, Thierry Carrez wrote: Second it's at a very different evolution/maturity point (20 years old vs. 0-4 years old for OpenStack projects). Yes, but I thought I covered this in the email: you can see that at the 4 year point in its lifecycle, the kernel was behaving very differently (and in fact more similar to OpenStack). The question I thought was still valid is whether anything was learnable from the way the kernel evolved later. I think the key issue, which you seem to have in OpenStack is that the separate develop/stabilise phases caused frustration to build up in our system which (nine years later) led the kernel to adopt the main branch stabilisation with overlapping subsystem development cycle. I agree with you: the evolution the kernel went through is almost a natural law, and I know we won't stay in the current model forever. I'm just not sure we have reached the level of general stability that makes it possible to change *just now*. I welcome brainstorming and discussion on future evolutions, though, and intend to lead a cross-project session discussion on that in Vancouver. OK, I'll be in Vancouver, so happy to come and give input from participating in the kernel process (for a bit longer than I care to admit to ...). One interesting thing might be to try and work out where roughly OpenStack is on the project trajectory. It's progressing much more rapidly than the kernel (by 4 years in, the kernel didn't even have source control), so the release crisis it took the kernel 12 years to reach might be a bit closer than people imagine. James James __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
James Bottomley wrote: On Tue, 2015-03-03 at 11:59 +0100, Thierry Carrez wrote: James Bottomley wrote: Actually, this is possible: look at Linux, it freezes for 10 weeks of a 12 month release cycle (or 6 weeks of an 8 week one). More on this below. I'd be careful with comparisons with the Linux kernel. First it's a single bit of software, not a collection of interconnected projects. Well, we do have interconnection: the kernel on it's own doesn't do anything without a userspace. The theory was that we didn't have to be like BSD (coupled user space and kernel) and we could rely on others (principally the GNU project in the early days) to provide the userspace and that we could decouple kernel development from the userspace releases. Threading models were, I think, the biggest challenges to this assumption, but we survived. Right. My point there is that you only release one thing. We release a lot more pieces. There is (was?) downstream value in coordinating those releases, which is a factor in our ability to do it more often than twice a year. Second it's at a very different evolution/maturity point (20 years old vs. 0-4 years old for OpenStack projects). Yes, but I thought I covered this in the email: you can see that at the 4 year point in its lifecycle, the kernel was behaving very differently (and in fact more similar to OpenStack). The question I thought was still valid is whether anything was learnable from the way the kernel evolved later. I think the key issue, which you seem to have in OpenStack is that the separate develop/stabilise phases caused frustration to build up in our system which (nine years later) led the kernel to adopt the main branch stabilisation with overlapping subsystem development cycle. I agree with you: the evolution the kernel went through is almost a natural law, and I know we won't stay in the current model forever. I'm just not sure we have reached the level of general stability that makes it possible to change *just now*. I welcome brainstorming and discussion on future evolutions, though, and intend to lead a cross-project session discussion on that in Vancouver. Finally it sits at a different layer, so there is less need for documentation/translations to be shipped with the software release. It's certainly a lot less than you, but we have the entire system call man pages. It's an official project of the kernel: https://www.kernel.org/doc/man-pages/ And we maintain translations for it https://www.kernel.org/doc/man-pages/translations.html By translations I meant strings in the software itself, not doc translations. We don't translate docs upstream either :) I guess we could drop those (and/or downstream them in a way) if that was the last thing holding up adding more agility. So in summary, yes we can (and do) learn from kernel history, but those projects are sufficiently different that the precise timeframes and numbers can't really be compared. Apples and oranges are both fruits which mature (and rot if left unchecked), but they evolve at different speeds :) -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
James Bottomley wrote: Actually, this is possible: look at Linux, it freezes for 10 weeks of a 12 month release cycle (or 6 weeks of an 8 week one). More on this below. I'd be careful with comparisons with the Linux kernel. First it's a single bit of software, not a collection of interconnected projects. Second it's at a very different evolution/maturity point (20 years old vs. 0-4 years old for OpenStack projects). Finally it sits at a different layer, so there is less need for documentation/translations to be shipped with the software release. The only comparable project in terms of evolution/maturity in the OpenStack world would be Swift, and it happily produces releases every ~2months with a 1-week stabilisation period. -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On Mon, Mar 2, 2015 at 11:59 AM, Kyle Mestery mest...@mestery.com wrote: On Mon, Mar 2, 2015 at 9:57 AM, Ihar Hrachyshka ihrac...@redhat.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Daniel, thanks for a clear write-up of the matter and food for thought. I think the idea of having more smooth development mode that would not make people to wait for 6+ months to release a new feature is great. ++ It's insane to expect that feature priorities won't ever slightly shift in the next 6 months. Having a limited list of features targeted for the next 6 months is prone to mistakes, since people behind some of approved features may need to postpone the effort for any type of reasons (personal, job rush, bad resource allocation, ...), and we then end up with approved specs with no actual code drops, using review 'slots' that would better be spent for other features that were not that lucky to get a rubber stamp during spec phase. Prior resource allocation would probably work somehow if we were working for the same company that would define priorities to us, but it's not the case. It should be noted that even though Nova is using slots for reviews, Neutron is not. I've found that it's hard to try and slot people in to review specific things. During Juno I tried this for Neutron, and it failed miserably. For Kilo in Neutron, we're not using slots but instead I've tried to highlight the approved specs of Essential and High priority for review by all reviews, core and non-core included. It's gone ok, but the reality is you can't force people to review things. There are steps submitters can take to try and get timely review (lots of small, easy to digest patches, quick turnaround of comments, engagement in IRC and ML, etc.). It was pointed out to me that nova is NOT using slots. Apologies for my misunderstanding here. Clearly, this thread has elicited a lot of strong thoughts and emotions. I hope we can all use this energy to figure out a good solution and a way forward for the issues presented here. Anecdotally, in neutron, we have two Essential blueprints for Kilo, and there are no code drops or patches in review for any of those, so I would expect them to fail to merge. At the same time, I will need to wait for the end of Kilo to consider adding support for guru reports to the project. Or in oslo world, I will need to wait for Liberty to introduce features in oslo.policy that are needed by neutron to switch to it, etc. To be fair, there are many reasons those to Essential BPs do not have code. I still expect the Pecan focused to have code, but I already moved the Plugin one out of Kilo at this point because there was no chance the code would land. But I get your point here. I think this thread has highlighted the fact that the BP/spec process worked to some extent, but for small things, the core reviewer team should have the ability to say Yes, we can easily merge that, lets approve that spec even if it's late in the cycle. Another problem is that currently milestones are used merely for targeting bugs and features, but no one really cares about whether the actual milestone shipment actually works. Again, a story from neutron world: Kilo-1 was shipped in the middle of advanced services split, with some essential patches around distutils setup missing (no proper migration plan applied, conflicting config files in neutron and *aas repos, etc.) This is true, the milestone release matter but are not given enough focus and they release (for the most part) irregardless of items in them, given they are not long-lived, etc. So I'm all for reforms around processes we apply. If there's one thing OpenStack is good at, it's change. That said, I don't believe the real problem here is that we don't generate project tarballs frequently enough. Major problems I see as critical to tackle in our dev process are: - - enforced spec/dev mode. Solution: allow to propose (and approve) a reasonable spec at any moment in the cycle; allow code drops for approved specs at any moment in the cycle (except pre-release stabilization time); stop targeting specs: if it's sane, it's probably sane N+2 cycle from now too. I'd say this is fine for specs that are small and people agree can easily be merged. I'd say this is not the case for large features near the end of the release which are unlikely to gather enough review momentum to actually merge. - - core team rubber stamping a random set of specs and putting -2 on all other specs due to project priorities. Solution: stop pretending core team (or core drivers) can reasonably estimate review and coding resources for the next cycle. Instead allows community to decide what's worth the effort by approving all technically reasonable specs and allowing everyone to invest time and effort in specs (s)he seems worth it. If you're referring to Neutron here, I think you fail to estimate the
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On Mon, Mar 2, 2015 at 9:59 AM, Kyle Mestery mest...@mestery.com wrote: On Mon, Mar 2, 2015 at 9:57 AM, Ihar Hrachyshka ihrac...@redhat.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Daniel, thanks for a clear write-up of the matter and food for thought. I think the idea of having more smooth development mode that would not make people to wait for 6+ months to release a new feature is great. ++ It's insane to expect that feature priorities won't ever slightly shift in the next 6 months. Having a limited list of features targeted for the next 6 months is prone to mistakes, since people behind some * Sure, we have had a few things that popped up, nova EC2 split for example. But this is fairly rare. of approved features may need to postpone the effort for any type of reasons (personal, job rush, bad resource allocation, ...), and we then end up with approved specs with no actual code drops, using review 'slots' that would better be spent for other features that were not that lucky to get a rubber stamp during spec phase. Prior resource * As stated below specs approval is very much not rubber stamping * As stated below this doesn't even make sense, we are *not* using review slots. allocation would probably work somehow if we were working for the same company that would define priorities to us, but it's not the case. It should be noted that even though Nova is using slots for reviews, Neutron is not. I've found that it's hard to try and slot people in to review specific things. During Juno I tried this for Neutron, and it failed miserably. For Kilo in Neutron, we're not using slots but instead I've tried to highlight the approved specs of Essential and High priority for review by all reviews, core and non-core included. It's gone ok, but the reality is you can't force people to review things. There are steps submitters can take to try and get timely review (lots of small, easy to digest patches, quick turnaround of comments, engagement in IRC and ML, etc.). So this is a big fat lie, one that others believe as well. Nova is *not* using slots for reviews. We discussed using slots for reviews but did not adopt them. Anecdotally, in neutron, we have two Essential blueprints for Kilo, and there are no code drops or patches in review for any of those, so I would expect them to fail to merge. At the same time, I will need to wait for the end of Kilo to consider adding support for guru reports to the project. Or in oslo world, I will need to wait for Liberty to introduce features in oslo.policy that are needed by neutron to switch to it, etc. To be fair, there are many reasons those to Essential BPs do not have code. I still expect the Pecan focused to have code, but I already moved the Plugin one out of Kilo at this point because there was no chance the code would land. But I get your point here. I think this thread has highlighted the fact that the BP/spec process worked to some extent, but for small things, the core reviewer team should have the ability to say Yes, we can easily merge that, lets approve that spec even if it's late in the cycle. Another problem is that currently milestones are used merely for targeting bugs and features, but no one really cares about whether the actual milestone shipment actually works. Again, a story from neutron world: Kilo-1 was shipped in the middle of advanced services split, with some essential patches around distutils setup missing (no proper migration plan applied, conflicting config files in neutron and *aas repos, etc.) This is true, the milestone release matter but are not given enough focus and they release (for the most part) irregardless of items in them, given they are not long-lived, etc. So I'm all for reforms around processes we apply. If there's one thing OpenStack is good at, it's change. That said, I don't believe the real problem here is that we don't generate project tarballs frequently enough. Major problems I see as critical to tackle in our dev process are: - - enforced spec/dev mode. Solution: allow to propose (and approve) a reasonable spec at any moment in the cycle; allow code drops for approved specs at any moment in the cycle (except pre-release stabilization time); stop targeting specs: if it's sane, it's probably sane N+2 cycle from now too. I'd say this is fine for specs that are small and people agree can easily be merged. I'd say this is not the case for large features near the end of the release which are unlikely to gather enough review momentum to actually merge. - - core team rubber stamping a random set of specs and putting -2 on all other specs due to project priorities. Solution: stop pretending core team (or core drivers) can reasonably estimate review and coding resources for the next cycle. Instead allows community to decide what's worth the effort by approving all technically reasonable specs and allowing
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On Mon, Mar 2, 2015 at 3:38 PM, Joe Gordon joe.gord...@gmail.com wrote: On Mon, Mar 2, 2015 at 9:59 AM, Kyle Mestery mest...@mestery.com wrote: On Mon, Mar 2, 2015 at 9:57 AM, Ihar Hrachyshka ihrac...@redhat.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Daniel, thanks for a clear write-up of the matter and food for thought. I think the idea of having more smooth development mode that would not make people to wait for 6+ months to release a new feature is great. ++ It's insane to expect that feature priorities won't ever slightly shift in the next 6 months. Having a limited list of features targeted for the next 6 months is prone to mistakes, since people behind some * Sure, we have had a few things that popped up, nova EC2 split for example. But this is fairly rare. of approved features may need to postpone the effort for any type of reasons (personal, job rush, bad resource allocation, ...), and we then end up with approved specs with no actual code drops, using review 'slots' that would better be spent for other features that were not that lucky to get a rubber stamp during spec phase. Prior resource * As stated below specs approval is very much not rubber stamping * As stated below this doesn't even make sense, we are *not* using review slots. allocation would probably work somehow if we were working for the same company that would define priorities to us, but it's not the case. It should be noted that even though Nova is using slots for reviews, Neutron is not. I've found that it's hard to try and slot people in to review specific things. During Juno I tried this for Neutron, and it failed miserably. For Kilo in Neutron, we're not using slots but instead I've tried to highlight the approved specs of Essential and High priority for review by all reviews, core and non-core included. It's gone ok, but the reality is you can't force people to review things. There are steps submitters can take to try and get timely review (lots of small, easy to digest patches, quick turnaround of comments, engagement in IRC and ML, etc.). So this is a big fat lie, one that others believe as well. Nova is *not* using slots for reviews. We discussed using slots for reviews but did not adopt them. But I read it on the internet, it must be true. As I said in a prior email, I'm sorry for that. I recalled reading about nova's use of slots. Anecdotally, in neutron, we have two Essential blueprints for Kilo, and there are no code drops or patches in review for any of those, so I would expect them to fail to merge. At the same time, I will need to wait for the end of Kilo to consider adding support for guru reports to the project. Or in oslo world, I will need to wait for Liberty to introduce features in oslo.policy that are needed by neutron to switch to it, etc. To be fair, there are many reasons those to Essential BPs do not have code. I still expect the Pecan focused to have code, but I already moved the Plugin one out of Kilo at this point because there was no chance the code would land. But I get your point here. I think this thread has highlighted the fact that the BP/spec process worked to some extent, but for small things, the core reviewer team should have the ability to say Yes, we can easily merge that, lets approve that spec even if it's late in the cycle. Another problem is that currently milestones are used merely for targeting bugs and features, but no one really cares about whether the actual milestone shipment actually works. Again, a story from neutron world: Kilo-1 was shipped in the middle of advanced services split, with some essential patches around distutils setup missing (no proper migration plan applied, conflicting config files in neutron and *aas repos, etc.) This is true, the milestone release matter but are not given enough focus and they release (for the most part) irregardless of items in them, given they are not long-lived, etc. So I'm all for reforms around processes we apply. If there's one thing OpenStack is good at, it's change. That said, I don't believe the real problem here is that we don't generate project tarballs frequently enough. Major problems I see as critical to tackle in our dev process are: - - enforced spec/dev mode. Solution: allow to propose (and approve) a reasonable spec at any moment in the cycle; allow code drops for approved specs at any moment in the cycle (except pre-release stabilization time); stop targeting specs: if it's sane, it's probably sane N+2 cycle from now too. I'd say this is fine for specs that are small and people agree can easily be merged. I'd say this is not the case for large features near the end of the release which are unlikely to gather enough review momentum to actually merge. - - core team rubber stamping a random set of specs and putting -2 on all other specs due to project priorities. Solution: stop
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
Excerpts from Angus Salkeld's message of 2015-03-02 17:08:15 -0800: On Tue, Mar 3, 2015 at 9:45 AM, James Bottomley james.bottom...@hansenpartnership.com wrote: On Tue, 2015-02-24 at 12:05 +0100, Thierry Carrez wrote: Daniel P. Berrange wrote: [...] The key observations The first key observation from the schedule is that although we have a 6 month release cycle, we in fact make 4 releases in that six months because there are 3 milestones releases approx 6-7 weeks apart from each other, in addition to the final release. So one of the key burdens of a more frequent release cycle is already being felt, to some degree. The second observation is that thanks to the need to support a continuous deployment models, the GIT master branches are generally considered to be production ready at all times. The tree does not typically go through periods of major instability that can be seen in other projects, particular those which lack such comprehensive testing infrastructure. The third observation is that due to the relatively long cycle, and increasing amounts of process, the work accomplished during the cycles is becoming increasingly bursty. This is in turn causing unacceptably long delays for contributors when their work is unlucky enough to not get accepted during certain critical short windows of opportunity in the cycle. The first two observations strongly suggest that the choice of 6 months as a cycle length is a fairly arbitrary decision that can be changed without unreasonable pain. The third observation suggests a much shorter cycle length would smooth out the bumps and lead to a more efficient satisfying development process for all involved. I think you're judging the cycle from the perspective of developers only. 6 months was not an arbitrary decision. Translations and documentation teams basically need a month of feature/string freeze in order to complete their work. Since we can't reasonably freeze one month every 2 months, we picked 6 months. Actually, this is possible: look at Linux, it freezes for 10 weeks of a 12 month release cycle (or 6 weeks of an 8 week one). More on this below. It's also worth noting that we were on a 3-month cycle at the start of OpenStack. That was dropped after a cataclysmic release that managed the feat of (a) not having anything significant done, and (b) have out of date documentation and translations. While I agree that the packagers and stable teams can opt to skip a release, the docs, translations or security teams don't really have that luxury... Please go beyond the developers needs and consider the needs of the other teams. Random other comments below: [...] Release schedule First the releases would probably be best attached to a set of pre-determined fixed dates that don't ever vary from year to year. eg releses happen Feb 1st, Apr 1st, Jun 1st, Aug 1st, Oct 1st, and Dec 1st. If a particular release slips, don't alter following release dates, just shorten the length of the dev cycle, so it becomes fully self-correcting. The even numbered months are suggested to avoid a release landing in xmas/new year :-) The Feb 1 release would probably be pretty empty :) [...] Stable branches --- The consequences of a 2 month release cycle appear fairly severe for the stable branch maint teams at first sight. This is not, however, an insurmountable problem. The linux kernel shows an easy way forward with their approach of only maintaining stable branches for a subset of major releases, based around user / vendor demand. So it is still entirely conceivable that the stable team only provide stable branch releases for 2 out of the 6 yearly releases. ie no additional burden over what they face today. Of course they might decide they want to do more stable branches, but maintain each for a shorter time. So I could equally see them choosing todo 3 or 4 stable branches a year. Whatever is most effective for those involved and those consuming them is fine. Stable branches may have the luxury of skipping releases and designate a stable one from time to time (I reject the Linux comparison because the kernel is at a very different moment in software lifecycle). The trick being, making one release special is sure to recreate the peak issues you're trying to solve. I don't disagree with the observation about different points in the lifecycle, but perhaps it might be instructive to ask if the linux kernel ever had a period in its development history that looks somewhat like OpenStack does now. I would claim it did: before 2.6, we had the odd/even develop/stabilise cycle. The theory driving it was that we
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On Tue, 2015-02-24 at 12:05 +0100, Thierry Carrez wrote: Daniel P. Berrange wrote: [...] The key observations The first key observation from the schedule is that although we have a 6 month release cycle, we in fact make 4 releases in that six months because there are 3 milestones releases approx 6-7 weeks apart from each other, in addition to the final release. So one of the key burdens of a more frequent release cycle is already being felt, to some degree. The second observation is that thanks to the need to support a continuous deployment models, the GIT master branches are generally considered to be production ready at all times. The tree does not typically go through periods of major instability that can be seen in other projects, particular those which lack such comprehensive testing infrastructure. The third observation is that due to the relatively long cycle, and increasing amounts of process, the work accomplished during the cycles is becoming increasingly bursty. This is in turn causing unacceptably long delays for contributors when their work is unlucky enough to not get accepted during certain critical short windows of opportunity in the cycle. The first two observations strongly suggest that the choice of 6 months as a cycle length is a fairly arbitrary decision that can be changed without unreasonable pain. The third observation suggests a much shorter cycle length would smooth out the bumps and lead to a more efficient satisfying development process for all involved. I think you're judging the cycle from the perspective of developers only. 6 months was not an arbitrary decision. Translations and documentation teams basically need a month of feature/string freeze in order to complete their work. Since we can't reasonably freeze one month every 2 months, we picked 6 months. Actually, this is possible: look at Linux, it freezes for 10 weeks of a 12 month release cycle (or 6 weeks of an 8 week one). More on this below. It's also worth noting that we were on a 3-month cycle at the start of OpenStack. That was dropped after a cataclysmic release that managed the feat of (a) not having anything significant done, and (b) have out of date documentation and translations. While I agree that the packagers and stable teams can opt to skip a release, the docs, translations or security teams don't really have that luxury... Please go beyond the developers needs and consider the needs of the other teams. Random other comments below: [...] Release schedule First the releases would probably be best attached to a set of pre-determined fixed dates that don't ever vary from year to year. eg releses happen Feb 1st, Apr 1st, Jun 1st, Aug 1st, Oct 1st, and Dec 1st. If a particular release slips, don't alter following release dates, just shorten the length of the dev cycle, so it becomes fully self-correcting. The even numbered months are suggested to avoid a release landing in xmas/new year :-) The Feb 1 release would probably be pretty empty :) [...] Stable branches --- The consequences of a 2 month release cycle appear fairly severe for the stable branch maint teams at first sight. This is not, however, an insurmountable problem. The linux kernel shows an easy way forward with their approach of only maintaining stable branches for a subset of major releases, based around user / vendor demand. So it is still entirely conceivable that the stable team only provide stable branch releases for 2 out of the 6 yearly releases. ie no additional burden over what they face today. Of course they might decide they want to do more stable branches, but maintain each for a shorter time. So I could equally see them choosing todo 3 or 4 stable branches a year. Whatever is most effective for those involved and those consuming them is fine. Stable branches may have the luxury of skipping releases and designate a stable one from time to time (I reject the Linux comparison because the kernel is at a very different moment in software lifecycle). The trick being, making one release special is sure to recreate the peak issues you're trying to solve. I don't disagree with the observation about different points in the lifecycle, but perhaps it might be instructive to ask if the linux kernel ever had a period in its development history that looks somewhat like OpenStack does now. I would claim it did: before 2.6, we had the odd/even develop/stabilise cycle. The theory driving it was that we needed a time for everyone to develop then a time for everyone to help make stable. You yourself said this in the other thread: Joe Gordon wrote: [...] I think a lot of the frustration with our current cadence comes out of the big stop everything (development, planning etc.), and stabilize the release process. Which in turn isn't
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On Tue, Mar 3, 2015 at 9:45 AM, James Bottomley james.bottom...@hansenpartnership.com wrote: On Tue, 2015-02-24 at 12:05 +0100, Thierry Carrez wrote: Daniel P. Berrange wrote: [...] The key observations The first key observation from the schedule is that although we have a 6 month release cycle, we in fact make 4 releases in that six months because there are 3 milestones releases approx 6-7 weeks apart from each other, in addition to the final release. So one of the key burdens of a more frequent release cycle is already being felt, to some degree. The second observation is that thanks to the need to support a continuous deployment models, the GIT master branches are generally considered to be production ready at all times. The tree does not typically go through periods of major instability that can be seen in other projects, particular those which lack such comprehensive testing infrastructure. The third observation is that due to the relatively long cycle, and increasing amounts of process, the work accomplished during the cycles is becoming increasingly bursty. This is in turn causing unacceptably long delays for contributors when their work is unlucky enough to not get accepted during certain critical short windows of opportunity in the cycle. The first two observations strongly suggest that the choice of 6 months as a cycle length is a fairly arbitrary decision that can be changed without unreasonable pain. The third observation suggests a much shorter cycle length would smooth out the bumps and lead to a more efficient satisfying development process for all involved. I think you're judging the cycle from the perspective of developers only. 6 months was not an arbitrary decision. Translations and documentation teams basically need a month of feature/string freeze in order to complete their work. Since we can't reasonably freeze one month every 2 months, we picked 6 months. Actually, this is possible: look at Linux, it freezes for 10 weeks of a 12 month release cycle (or 6 weeks of an 8 week one). More on this below. It's also worth noting that we were on a 3-month cycle at the start of OpenStack. That was dropped after a cataclysmic release that managed the feat of (a) not having anything significant done, and (b) have out of date documentation and translations. While I agree that the packagers and stable teams can opt to skip a release, the docs, translations or security teams don't really have that luxury... Please go beyond the developers needs and consider the needs of the other teams. Random other comments below: [...] Release schedule First the releases would probably be best attached to a set of pre-determined fixed dates that don't ever vary from year to year. eg releses happen Feb 1st, Apr 1st, Jun 1st, Aug 1st, Oct 1st, and Dec 1st. If a particular release slips, don't alter following release dates, just shorten the length of the dev cycle, so it becomes fully self-correcting. The even numbered months are suggested to avoid a release landing in xmas/new year :-) The Feb 1 release would probably be pretty empty :) [...] Stable branches --- The consequences of a 2 month release cycle appear fairly severe for the stable branch maint teams at first sight. This is not, however, an insurmountable problem. The linux kernel shows an easy way forward with their approach of only maintaining stable branches for a subset of major releases, based around user / vendor demand. So it is still entirely conceivable that the stable team only provide stable branch releases for 2 out of the 6 yearly releases. ie no additional burden over what they face today. Of course they might decide they want to do more stable branches, but maintain each for a shorter time. So I could equally see them choosing todo 3 or 4 stable branches a year. Whatever is most effective for those involved and those consuming them is fine. Stable branches may have the luxury of skipping releases and designate a stable one from time to time (I reject the Linux comparison because the kernel is at a very different moment in software lifecycle). The trick being, making one release special is sure to recreate the peak issues you're trying to solve. I don't disagree with the observation about different points in the lifecycle, but perhaps it might be instructive to ask if the linux kernel ever had a period in its development history that looks somewhat like OpenStack does now. I would claim it did: before 2.6, we had the odd/even develop/stabilise cycle. The theory driving it was that we needed a time for everyone to develop then a time for everyone to help make stable. You yourself said this in the other thread: Joe Gordon wrote: [...] I think
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On Mon, Mar 2, 2015, at 10:57 AM, Ihar Hrachyshka wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Daniel, thanks for a clear write-up of the matter and food for thought. I think the idea of having more smooth development mode that would not make people to wait for 6+ months to release a new feature is great. It's insane to expect that feature priorities won't ever slightly shift in the next 6 months. Having a limited list of features targeted for the next 6 months is prone to mistakes, since people behind some of approved features may need to postpone the effort for any type of reasons (personal, job rush, bad resource allocation, ...), and we then end up with approved specs with no actual code drops, using review 'slots' that would better be spent for other features that were not that lucky to get a rubber stamp during spec phase. Prior resource allocation would probably work somehow if we were working for the same company that would define priorities to us, but it's not the case. Anecdotally, in neutron, we have two Essential blueprints for Kilo, and there are no code drops or patches in review for any of those, so I would expect them to fail to merge. At the same time, I will need to wait for the end of Kilo to consider adding support for guru reports to the project. Or in oslo world, I will need to wait for Liberty to introduce features in oslo.policy that are needed by neutron to switch to it, etc. The Oslo policy is a bit more lenient than some of the others I've heard described. We don't follow the feature proposal freeze. Instead, we only have a hard freeze for the master branches of libraries until the release candidates are done. Any features proposed at any other time are candidates, with the usual caveats for review team time and attention. Doug Another problem is that currently milestones are used merely for targeting bugs and features, but no one really cares about whether the actual milestone shipment actually works. Again, a story from neutron world: Kilo-1 was shipped in the middle of advanced services split, with some essential patches around distutils setup missing (no proper migration plan applied, conflicting config files in neutron and *aas repos, etc.) So I'm all for reforms around processes we apply. That said, I don't believe the real problem here is that we don't generate project tarballs frequently enough. Major problems I see as critical to tackle in our dev process are: - - enforced spec/dev mode. Solution: allow to propose (and approve) a reasonable spec at any moment in the cycle; allow code drops for approved specs at any moment in the cycle (except pre-release stabilization time); stop targeting specs: if it's sane, it's probably sane N+2 cycle from now too. - - core team rubber stamping a random set of specs and putting -2 on all other specs due to project priorities. Solution: stop pretending core team (or core drivers) can reasonably estimate review and coding resources for the next cycle. Instead allows community to decide what's worth the effort by approving all technically reasonable specs and allowing everyone to invest time and effort in specs (s)he seems worth it. - - no due stabilization process before dev milestones. Solution: introduce one in your team workflow, be more strict in what goes in during pre milestone stabilization time. If all above is properly applied, we would get into position similar to your proposal. The difference though is that upstream project would not call milestone tarballs 'Releases'. Still, if there are brave vendors to ship milestones, fine, they would have the same freedom as in your proposal. Note: all the steps mentioned above can be applied on *per team* basis without breaking existing release workflow. Some more comments from stable-maint/distribution side below. On 02/24/2015 10:53 AM, Daniel P. Berrange wrote: [...skip...] The modest proposal === [...skip...] Stable branches --- The consequences of a 2 month release cycle appear fairly severe for the stable branch maint teams at first sight. This is not, however, an insurmountable problem. The linux kernel shows an easy way forward with their approach of only maintaining stable branches for a subset of major releases, based around user / vendor demand. So it is still entirely conceivable that the stable team only provide stable branch releases for 2 out of the 6 yearly releases. ie no additional burden over what they face today. Of course they might decide they want to do more stable branches, but maintain each for a shorter time. So I could equally see them choosing todo 3 or 4 stable branches a year. Whatever is most effective for those involved and those consuming them is fine. Since it's not only stable branches that are affected (translators, documentation writers, VMT were already mentioned), those
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Daniel, thanks for a clear write-up of the matter and food for thought. I think the idea of having more smooth development mode that would not make people to wait for 6+ months to release a new feature is great. It's insane to expect that feature priorities won't ever slightly shift in the next 6 months. Having a limited list of features targeted for the next 6 months is prone to mistakes, since people behind some of approved features may need to postpone the effort for any type of reasons (personal, job rush, bad resource allocation, ...), and we then end up with approved specs with no actual code drops, using review 'slots' that would better be spent for other features that were not that lucky to get a rubber stamp during spec phase. Prior resource allocation would probably work somehow if we were working for the same company that would define priorities to us, but it's not the case. Anecdotally, in neutron, we have two Essential blueprints for Kilo, and there are no code drops or patches in review for any of those, so I would expect them to fail to merge. At the same time, I will need to wait for the end of Kilo to consider adding support for guru reports to the project. Or in oslo world, I will need to wait for Liberty to introduce features in oslo.policy that are needed by neutron to switch to it, etc. Another problem is that currently milestones are used merely for targeting bugs and features, but no one really cares about whether the actual milestone shipment actually works. Again, a story from neutron world: Kilo-1 was shipped in the middle of advanced services split, with some essential patches around distutils setup missing (no proper migration plan applied, conflicting config files in neutron and *aas repos, etc.) So I'm all for reforms around processes we apply. That said, I don't believe the real problem here is that we don't generate project tarballs frequently enough. Major problems I see as critical to tackle in our dev process are: - - enforced spec/dev mode. Solution: allow to propose (and approve) a reasonable spec at any moment in the cycle; allow code drops for approved specs at any moment in the cycle (except pre-release stabilization time); stop targeting specs: if it's sane, it's probably sane N+2 cycle from now too. - - core team rubber stamping a random set of specs and putting -2 on all other specs due to project priorities. Solution: stop pretending core team (or core drivers) can reasonably estimate review and coding resources for the next cycle. Instead allows community to decide what's worth the effort by approving all technically reasonable specs and allowing everyone to invest time and effort in specs (s)he seems worth it. - - no due stabilization process before dev milestones. Solution: introduce one in your team workflow, be more strict in what goes in during pre milestone stabilization time. If all above is properly applied, we would get into position similar to your proposal. The difference though is that upstream project would not call milestone tarballs 'Releases'. Still, if there are brave vendors to ship milestones, fine, they would have the same freedom as in your proposal. Note: all the steps mentioned above can be applied on *per team* basis without breaking existing release workflow. Some more comments from stable-maint/distribution side below. On 02/24/2015 10:53 AM, Daniel P. Berrange wrote: [...skip...] The modest proposal === [...skip...] Stable branches --- The consequences of a 2 month release cycle appear fairly severe for the stable branch maint teams at first sight. This is not, however, an insurmountable problem. The linux kernel shows an easy way forward with their approach of only maintaining stable branches for a subset of major releases, based around user / vendor demand. So it is still entirely conceivable that the stable team only provide stable branch releases for 2 out of the 6 yearly releases. ie no additional burden over what they face today. Of course they might decide they want to do more stable branches, but maintain each for a shorter time. So I could equally see them choosing todo 3 or 4 stable branches a year. Whatever is most effective for those involved and those consuming them is fine. Since it's not only stable branches that are affected (translators, documentation writers, VMT were already mentioned), those affected will probably need to come up with some synchronized decision. Let's say we still decide to support two out of six releases (same scheme as is now). In that case no process that we usually attach to releases will be running after release dates. This makes me wonder how it's different from milestones we already have. Do you think any downstream vendors will actually ship and support upstream releases that upstream drops any guarantees for (no VMT, no stable branches, no gate runs, ...) right after the
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/26/2015 01:06 AM, Thomas Goirand wrote: On 02/24/2015 12:27 PM, Daniel P. Berrange wrote: I'm actually trying to judge it from the POV of users, not just developers. I find it pretty untenable that in the fast moving world of cloud, users have to wait as long as 6 months for a feature to get into a openstack release, often much longer. If you were trying to judge from the POV of users, then you would consider that basically, they don't really care the brand new shiny feature which just appear. They care having a long time support for whatever version of OpenStack they have installed, without having the head-aches of upgrading which is famously painful with OpenStack. This shows clearly on our user surveys which are presented on every summit: users are lagging behind, with a majority still running with OpenStack releases which are already EOL. As was already said in the thread, they care about long support AND their specific subset of shiny new features they need (and they don't care about all other features that are not in their subjective list of shiny ones). In fact, if you want to judge from the POV of our users, we should *SLOW DOWN* our release cycles, and probably move to something like one release every year or 2. We should also try to have longer periods of support for our stable releases, which would (with my Debian package maintainer hat on!) help distributions to do such security support. Debian Jessie will be released in a few month from now, just before Icehouse (which it ships) will be EOL. RedHat, Canonical, IBM, and so many more are also on the same (sinking) ship. That won't come up magically. If no people from Debian or IBM are actively involved in solving issues for stable branches, long term will be just a wish and not a reality. At the moment we have several people from Red Hat (me, Alan) and Ubuntu (Chuck, Adam) side active in the team. That would be great to see more diversity, and more hands dirty with stable branches on daily basis. As for my employer side of things, we've seen numerous cases with our customer requesting for LTS, which we have to provide by ourselves, since it's not supported upstream. Every downstream vendor would be happy to see upstream freely support their product. :) But you get what you pay to upstream horizontal initiatives like stable maintenance. I've heard IBM still supports releases starting from C release!.. I think the majority of translation work can be done in parallel with dev work and the freeze time just needs to tie up the small remaining bits. It'd be nice indeed, but I've never seen any project (open source or not) working this way for translations. It's actually not true. I've coordinated multiple translation teams for my language (Belarusian), and for what I can tell, the best practice is to work on translations while development is ongoing. Yes, it means some effort wasted, but it also means spreading the whole effort throughout the year instead of doing all the work in pre-release freeze time. Freeze time is anyway good to make sure that no last minute patches break translations, and I don't suggest we drop them completely. Documentation is again something I'd expect to be done more or less in parallel with dev. Let's go back to reality: the Juno install-guide is still not finished, and the doc team is lagging behind. It would be reasonable for the vulnerability team to take the decision that they'll support fixes for master, and any branches that the stable team decide to support. ie they would not neccessarily have to commit to shipping fixes for every single release made. I've been crying for this type of decision. IE: drop Juno support early, and continue to maintain Icehouse for longer. I wish this happens, but the release team always complained that nobody works on maintaining the gate for the stable branches. Unless this changes, I don't see hope... :( - From Red Hat perspective, we put our effort in what we're interested in. We're more interested in keeping the branch for the latest release working, not about sticking to an old release that e.g. Debian chose to put into their next official release, so unless someone from Debian come to the team and start to invest into Icehouse, it will eventually be dropped. That's life. I really not trying to focus on the developers woes. I'm trying to focus on making OpenStack better serve our users. My main motiviation here is that I think we're doing a pretty terrible job at getting work done that is important to our users in a timely manner. This is caused by a workflow release cycle that is negatively impacting the developers. Our workflow and the release cycle are 2 separate things. From my POV, it'd be a mistake to believe switching to a different release cycle will fix our workflow. Here, I actually agree. There are lots of enhancements in
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On Mon, Mar 2, 2015 at 9:57 AM, Ihar Hrachyshka ihrac...@redhat.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hi Daniel, thanks for a clear write-up of the matter and food for thought. I think the idea of having more smooth development mode that would not make people to wait for 6+ months to release a new feature is great. ++ It's insane to expect that feature priorities won't ever slightly shift in the next 6 months. Having a limited list of features targeted for the next 6 months is prone to mistakes, since people behind some of approved features may need to postpone the effort for any type of reasons (personal, job rush, bad resource allocation, ...), and we then end up with approved specs with no actual code drops, using review 'slots' that would better be spent for other features that were not that lucky to get a rubber stamp during spec phase. Prior resource allocation would probably work somehow if we were working for the same company that would define priorities to us, but it's not the case. It should be noted that even though Nova is using slots for reviews, Neutron is not. I've found that it's hard to try and slot people in to review specific things. During Juno I tried this for Neutron, and it failed miserably. For Kilo in Neutron, we're not using slots but instead I've tried to highlight the approved specs of Essential and High priority for review by all reviews, core and non-core included. It's gone ok, but the reality is you can't force people to review things. There are steps submitters can take to try and get timely review (lots of small, easy to digest patches, quick turnaround of comments, engagement in IRC and ML, etc.). Anecdotally, in neutron, we have two Essential blueprints for Kilo, and there are no code drops or patches in review for any of those, so I would expect them to fail to merge. At the same time, I will need to wait for the end of Kilo to consider adding support for guru reports to the project. Or in oslo world, I will need to wait for Liberty to introduce features in oslo.policy that are needed by neutron to switch to it, etc. To be fair, there are many reasons those to Essential BPs do not have code. I still expect the Pecan focused to have code, but I already moved the Plugin one out of Kilo at this point because there was no chance the code would land. But I get your point here. I think this thread has highlighted the fact that the BP/spec process worked to some extent, but for small things, the core reviewer team should have the ability to say Yes, we can easily merge that, lets approve that spec even if it's late in the cycle. Another problem is that currently milestones are used merely for targeting bugs and features, but no one really cares about whether the actual milestone shipment actually works. Again, a story from neutron world: Kilo-1 was shipped in the middle of advanced services split, with some essential patches around distutils setup missing (no proper migration plan applied, conflicting config files in neutron and *aas repos, etc.) This is true, the milestone release matter but are not given enough focus and they release (for the most part) irregardless of items in them, given they are not long-lived, etc. So I'm all for reforms around processes we apply. If there's one thing OpenStack is good at, it's change. That said, I don't believe the real problem here is that we don't generate project tarballs frequently enough. Major problems I see as critical to tackle in our dev process are: - - enforced spec/dev mode. Solution: allow to propose (and approve) a reasonable spec at any moment in the cycle; allow code drops for approved specs at any moment in the cycle (except pre-release stabilization time); stop targeting specs: if it's sane, it's probably sane N+2 cycle from now too. I'd say this is fine for specs that are small and people agree can easily be merged. I'd say this is not the case for large features near the end of the release which are unlikely to gather enough review momentum to actually merge. - - core team rubber stamping a random set of specs and putting -2 on all other specs due to project priorities. Solution: stop pretending core team (or core drivers) can reasonably estimate review and coding resources for the next cycle. Instead allows community to decide what's worth the effort by approving all technically reasonable specs and allowing everyone to invest time and effort in specs (s)he seems worth it. If you're referring to Neutron here, I think you fail to estimate the amount of time the neutron-drivers team (along with a handful of other folks) spent reviewing specs and trying to allocate them for this release. We're not just rubber stamping things, we're reviewing, providing comment, and ensuring things fit in a consistent roadmap for the next release. In the past, we've had this sort of wild west where all specs are approved, everyone focuses on
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On 02/26/2015 07:06 PM, Ed Leafe wrote: I think you've nailed where the disconnect is between the two sides of this issue: what exactly do we see OpenStack being? You brought up several Linux vendors who ship on a longish cycle, and who provide LTS for their releases. But Linux itself is on no such cycle, nor does it provide long term anything. But Linux is one monolith project. OpenStack can't be all things to all people. Following the Linux analogy, we need a few companies who want to become OpenStack distributors, packagers, and supporters, in the manner of RedHat, Canonical, etc., are for Linux. As a development project, we need to be able to move fluidly, and the release cycle deadlines and freezes get in the way of that. As a packager and distributor, the release cycle scheduler *helps* immeasurably. We can't be both. I am fairly new to OpenStack, but from what I've ascertained so far, there are, now, individual sub-releases of individual projects. That could be a difficult task for any Linux vendor, to distribute. There's one project, Calibre - EBook Management Software, that does weekly releases. But again, it is easy for them, because they are a single controlled project. For something big as OpenStack, IMO, close co-ordination is needed. -- Given the large number of mailing lists I follow, I request you to CC me in replies for quicker response __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On Feb 25, 2015, at 6:06 PM, Thomas Goirand z...@debian.org wrote: In fact, if you want to judge from the POV of our users, we should *SLOW DOWN* our release cycles, and probably move to something like one release every year or 2. We should also try to have longer periods of support for our stable releases, which would (with my Debian package maintainer hat on!) help distributions to do such security support. Debian Jessie will be released in a few month from now, just before Icehouse (which it ships) will be EOL. RedHat, Canonical, IBM, and so many more are also on the same (sinking) ship. As for my employer side of things, we've seen numerous cases with our customer requesting for LTS, which we have to provide by ourselves, since it's not supported upstream. I think you've nailed where the disconnect is between the two sides of this issue: what exactly do we see OpenStack being? You brought up several Linux vendors who ship on a longish cycle, and who provide LTS for their releases. But Linux itself is on no such cycle, nor does it provide long term anything. OpenStack can't be all things to all people. Following the Linux analogy, we need a few companies who want to become OpenStack distributors, packagers, and supporters, in the manner of RedHat, Canonical, etc., are for Linux. As a development project, we need to be able to move fluidly, and the release cycle deadlines and freezes get in the way of that. As a packager and distributor, the release cycle scheduler *helps* immeasurably. We can't be both. -- Ed Leafe signature.asc Description: Message signed with OpenPGP using GPGMail __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On Thu, 26 Feb 2015, Ed Leafe wrote: OpenStack can't be all things to all people. Following the Linux analogy, we need a few companies who want to become OpenStack distributors, packagers, and supporters, in the manner of RedHat, Canonical, etc., are for Linux. As a development project, we need to be able to move fluidly, and the release cycle deadlines and freezes get in the way of that. As a packager and distributor, the release cycle scheduler *helps* immeasurably. We can't be both. Yes. The special nature of OpenStack makes the influence of companies quite strong and has many benefits (e.g. look ma, being paid to do open source) but it also means that boundaries that are found in other somewhat similar environments are not as strong. Elsewhere in the thread there was discussion about the importance of the release cycle for marketing. Marketing for who and by whom? Surely marketing is^wought to be in the domain of the people who are making money selling aggregations of OpenStack? -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On Feb 26, 2015, at 9:49 AM, Ritesh Raj Sarraf r...@researchut.com wrote: I think you've nailed where the disconnect is between the two sides of this issue: what exactly do we see OpenStack being? You brought up several Linux vendors who ship on a longish cycle, and who provide LTS for their releases. But Linux itself is on no such cycle, nor does it provide long term anything. But Linux is one monolith project. No, not really. The kernel may be, but if you want a distro that is useful to people, you need to collect a set of tools and apps that allow users to have a full computing experience. If Linux were a monolith, then there really wouldn't be any difference between Ubuntu and Slackware. I am fairly new to OpenStack, but from what I've ascertained so far, there are, now, individual sub-releases of individual projects. That could be a difficult task for any Linux vendor, to distribute. I don't think that anyone is implying that it is easy to do, no matter where it is done. It does seem that there are significant disadvantages on the development side of things to have one group doing it all. -- Ed Leafe signature.asc Description: Message signed with OpenPGP using GPGMail __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On Thu, Feb 26, 2015 at 01:06:14AM +0100, Thomas Goirand wrote: On 02/24/2015 12:27 PM, Daniel P. Berrange wrote: I'm actually trying to judge it from the POV of users, not just developers. I find it pretty untenable that in the fast moving world of cloud, users have to wait as long as 6 months for a feature to get into a openstack release, often much longer. If you were trying to judge from the POV of users, then you would consider that basically, they don't really care the brand new shiny feature which just appear. They care having a long time support for whatever version of OpenStack they have installed, without having the head-aches of upgrading which is famously painful with OpenStack. This shows clearly on our user surveys which are presented on every summit: users are lagging behind, with a majority still running with OpenStack releases which are already EOL. In fact, if you want to judge from the POV of our users, we should *SLOW DOWN* our release cycles, and probably move to something like one release every year or 2. We should also try to have longer periods of support for our stable releases, which would (with my Debian package maintainer hat on!) help distributions to do such security support. It is not actually that simple IME. As I describe elsewhere in this thread, users basically want no new features at all ever, except for the new features that are specifically relevant to what they want, and of course every users desired feature is different. It is essentially impossible to satisfy the requests in this way because they are inherantly contradictory conflicting. In reality what they asking is not that we reduce the number of releases or frequency of releases, but rather that they be able to stay running on a chosen release for longer. IOW The frequency of releases doesn't matter as long as they are not being forced to upgrade to the newer releases unreasonably quickly. From that POV I can certainly see how OpenStack is not satisfying them well, because if they do choose to skip 2-3 releases, they'll find themselves EOLd and also they'll be forced to deploy each intermediate release in order to apply upgrades, before finally getting to their target release. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
Tom Fifield t...@openstack.org wrote on 25/02/2015 06:46:13 AM: On 24/02/15 19:27, Daniel P. Berrange wrote: On Tue, Feb 24, 2015 at 12:05:17PM +0100, Thierry Carrez wrote: Daniel P. Berrange wrote: [...] I'm not familiar with how the translations works, but if they are waiting until the freeze before starting translation work I'd say that is a mistaken approach. Obviously during active dev part of the cycle, some translated strings are in flux, so if translation was taking place in parallel there could be some wasted effort, but I'd expect that to be the minority case. I think the majority of translation work can be done in parallel with dev work and the freeze time just needs to tie up the small remaining bits. So, two points: 1) We wouldn't be talking about throwing just a couple of percent of their work away. As an example, even without looking at the introduction of new strings or deleting others, you may not be aware that changing a single word in a string in the code means that entire string needs to be re-translated. Even with the extensive translation memory systems we have making suggestions as best they can, we're talking about very, very significant amounts of wasted effort. How difficult would it be to try quantifying this wasted effort? For example, if someone could write a script that extracts the data for a histogram showing the amount of strings (e.g., in Nova) that have been changed/overridden in consequent patches up to 1 week apart, between 1 and 2 weeks apart, and so on up to, say, 52 weeks. Regards, Alex Regards, Tom __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On 02/24/2015 12:27 PM, Daniel P. Berrange wrote: I'm actually trying to judge it from the POV of users, not just developers. I find it pretty untenable that in the fast moving world of cloud, users have to wait as long as 6 months for a feature to get into a openstack release, often much longer. If you were trying to judge from the POV of users, then you would consider that basically, they don't really care the brand new shiny feature which just appear. They care having a long time support for whatever version of OpenStack they have installed, without having the head-aches of upgrading which is famously painful with OpenStack. This shows clearly on our user surveys which are presented on every summit: users are lagging behind, with a majority still running with OpenStack releases which are already EOL. In fact, if you want to judge from the POV of our users, we should *SLOW DOWN* our release cycles, and probably move to something like one release every year or 2. We should also try to have longer periods of support for our stable releases, which would (with my Debian package maintainer hat on!) help distributions to do such security support. Debian Jessie will be released in a few month from now, just before Icehouse (which it ships) will be EOL. RedHat, Canonical, IBM, and so many more are also on the same (sinking) ship. As for my employer side of things, we've seen numerous cases with our customer requesting for LTS, which we have to provide by ourselves, since it's not supported upstream. I think the majority of translation work can be done in parallel with dev work and the freeze time just needs to tie up the small remaining bits. It'd be nice indeed, but I've never seen any project (open source or not) working this way for translations. Documentation is again something I'd expect to be done more or less in parallel with dev. Let's go back to reality: the Juno install-guide is still not finished, and the doc team is lagging behind. It would be reasonable for the vulnerability team to take the decision that they'll support fixes for master, and any branches that the stable team decide to support. ie they would not neccessarily have to commit to shipping fixes for every single release made. I've been crying for this type of decision. IE: drop Juno support early, and continue to maintain Icehouse for longer. I wish this happens, but the release team always complained that nobody works on maintaining the gate for the stable branches. Unless this changes, I don't see hope... :( I really not trying to focus on the developers woes. I'm trying to focus on making OpenStack better serve our users. My main motiviation here is that I think we're doing a pretty terrible job at getting work done that is important to our users in a timely manner. This is caused by a workflow release cycle that is negatively impacting the developers. Our workflow and the release cycle are 2 separate things. From my POV, it'd be a mistake to believe switching to a different release cycle will fix our workflow. Also, I'd like to point out something: it's been 2 years that I do release each and every beta release we do. But either they are bug free (you bet... :)), or nobody uses them (more likely), because I've *never ever* received some bug reports about them. Reasonably, there's no consumer for beta releases. Hoping my point of view is helpful, Cheers, Thomas Goirand (zigo) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On Tue, Feb 24, 2015 at 10:02:36PM -0800, Mark Atwood wrote: On Tue, Feb 24, 2015, at 04:28, Kashyap Chamarthy wrote: Along with the below, if push comes to shove, OpenStack Foundation could probably try a milder variant (obviously, not all activities can be categorized as 'critical path') of Linux Foundation's Critical Infrastructure Protection Initiative[1] to fund certain project activities in need. Speaking as a person who sits on the LF CII board meetings, and helps turn the crank on that particular sausage mill, no, we really don't want to go down that path at this point in time. I didn't imply to do an _exact_ version of LF's CII. Given so much of interest in OpenStack, vendors/interested parties must (yes) maintain a balance between giving vs taking from the community. And, they should be provided ample information on _where_ they can target people/engineers (Dan Berrange noted this too in one of his responses in this thread). -- /kashyap __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On Wed, Feb 25, 2015 at 10:31:58AM +0100, Thierry Carrez wrote: Robert Collins wrote: It's also worth noting that we were on a 3-month cycle at the start of OpenStack. That was dropped after a cataclysmic release that managed the feat of (a) not having anything significant done, and (b) have out of date documentation and translations. Oh! https://wiki.openstack.org/w/index.php?title=Obsolete:3MonthReleaseCycledirection=prevoldid=14016 appears to be the page about that, but my google-fu isn't turning up a post-mortem of the C release which prompted the change. Perhaps some old-hand like you can fill us in on the details :). After the Austin release (a code drop, really), we were on a three-month time-based cycle with no intermediary milestone. The first cycle (Bexar) worked relatively well, the second one (Cactus) not so well. There were multiple reasons for that: - Software was growing up in Bexar and we introduced more testing, but we were unable to detect and fix all the release blockers in time for the release date, resulting in a broken Bexar release. To address that in Cactus, we increased the duration of the frozen period, but then that didn't leave enough time for development, hence nothing getting done. I wasn't involved in OpenStack back then, but is it not true that OpenStack has changed almost beyond recognition since those early days. The number of projects has certainly increased massively, as has the number of contributors, both individual corporate. There is a hell of alot of testing now and people doing continuous deployment of trunk, so I find it hard to believe that trunk is so unstable that we can't do a release at any time we choose, nor that we have to have such long freezes that we don't do get time for dev. - Not starting a release cycle with a global F2F gathering (no intermediary Design Summit) actually proved quite disastrous: no reset of community, no alignment on goal, no discussion on what to work on. That resulted in a limbo period after the Bexar release. That is why I'm so insistent on aligning our development cycles with our Design Summits. I've seen where we go when we don't. I don't know about other projects so much, but I don't really see the design summit as a positive thing wrt planning the next release. For a start the design summit is atbout 5 weeks after the trunk opens for development, so if people wait for the summit do do planning they have thrown away half of the first milestones' development time. It is also not inclusive as a decision making forum because we cannot assume every one is able to make it to the summit in person, and even if they are present, people often can't get involved in all sessions they wish to due to conflicting schedules. If release planning were done via email primarily, with IRC for cases where realtime planning is needed, it would be more effective IMHO. IOW i think the summit would be better off if were explicitly not associated with releases, but rather be a general forum for collaboration were we talk through ideas or do code sprints, and more. [...] I may appear defensive on my answers, but it's not my goal to defend the current system: it's just that most of those proposals generally ignore the diversity of the needs of the teams that make OpenStack possible, to focus on a particular set of contributors' woes. I'm trying to bring that wider perspective in -- the current system is a trade-off and the result of years of evolution, not an arbitrary historic choice that we can just change at will. I agree that its not arbitrary and that changing it requires some appropriate wide-spread consultation; OTOH the benefits of rolling and higher frequency releases are really substantial: but we have to backup the change with the appropriate engineering (such as reducing our frictions that cause teams practicing CD to be weeks behind tip) to make it feasible. My greatest concern about the proposals happening now is that we may bite off more than we can chew. OTGH the reality is that all the negative things multiply out, so we probably need to just start somewhere and /do/ to give us the space to fix other things. As I said, I'd very much like us to release a lot more often. I just don't see how we can do it without: - abandoning the idea of translating the software - admit that docs will always lag code - dropping stable branch maintenance (and only fix vulnerabilities in master) I don't really buy that. If we have 6 month cycles with 1 month freeze for docs i18n, vs 2 months cycles with 2 weeks freeze for docs i18n the latter is a win in terms of dev time vs stablization time, giving docs i18n 50% more time in aggregate. That would all make my life (and the life of most feature-focused developers) a *lot* easier. I'm not convinced that's the best solution for our downstream users, though. I think they like docs, translations(?), stable
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
Robert Collins wrote: It's also worth noting that we were on a 3-month cycle at the start of OpenStack. That was dropped after a cataclysmic release that managed the feat of (a) not having anything significant done, and (b) have out of date documentation and translations. Oh! https://wiki.openstack.org/w/index.php?title=Obsolete:3MonthReleaseCycledirection=prevoldid=14016 appears to be the page about that, but my google-fu isn't turning up a post-mortem of the C release which prompted the change. Perhaps some old-hand like you can fill us in on the details :). After the Austin release (a code drop, really), we were on a three-month time-based cycle with no intermediary milestone. The first cycle (Bexar) worked relatively well, the second one (Cactus) not so well. There were multiple reasons for that: - Software was growing up in Bexar and we introduced more testing, but we were unable to detect and fix all the release blockers in time for the release date, resulting in a broken Bexar release. To address that in Cactus, we increased the duration of the frozen period, but then that didn't leave enough time for development, hence nothing getting done. - Not starting a release cycle with a global F2F gathering (no intermediary Design Summit) actually proved quite disastrous: no reset of community, no alignment on goal, no discussion on what to work on. That resulted in a limbo period after the Bexar release. That is why I'm so insistent on aligning our development cycles with our Design Summits. I've seen where we go when we don't. The post-mortem (at the Diablo summit) recognized our inability as a group (including QA, docs, stable branch, vulnerability management) to do final release work more often than every 6 months. Milestones were introduced so that we can communicate features faster to our users (those wanting fresh stuff could consume milestones), and then focus QA / docs / stable support on the final release only. Now, the funny thing is... this solution is very similar to what is proposed in this thread. Intermediary releases every two months to get code out really fast, and final releases which trigger stable maintenance. So what happened ? What happened then is what will happen if we reboot the exact same solution now: developers ignoring milestones (or intermediary releases) to focus only on the real, stable-supported one. And now milestones are just dates in a calendar. So when I say that won't solve anything and devs will keep on focusing on the real release, it's not just a gut feeling. We've been there. I like to learn from history and not repeat the same mistakes. [...] I may appear defensive on my answers, but it's not my goal to defend the current system: it's just that most of those proposals generally ignore the diversity of the needs of the teams that make OpenStack possible, to focus on a particular set of contributors' woes. I'm trying to bring that wider perspective in -- the current system is a trade-off and the result of years of evolution, not an arbitrary historic choice that we can just change at will. I agree that its not arbitrary and that changing it requires some appropriate wide-spread consultation; OTOH the benefits of rolling and higher frequency releases are really substantial: but we have to backup the change with the appropriate engineering (such as reducing our frictions that cause teams practicing CD to be weeks behind tip) to make it feasible. My greatest concern about the proposals happening now is that we may bite off more than we can chew. OTGH the reality is that all the negative things multiply out, so we probably need to just start somewhere and /do/ to give us the space to fix other things. As I said, I'd very much like us to release a lot more often. I just don't see how we can do it without: - abandoning the idea of translating the software - admit that docs will always lag code - dropping stable branch maintenance (and only fix vulnerabilities in master) That would all make my life (and the life of most feature-focused developers) a *lot* easier. I'm not convinced that's the best solution for our downstream users, though. I think they like docs, translations(?), stable branches and backported security fixes. PS: as a data point, at the last summit, in the stable branch session, there was a whole group of downstream users crashing the session to ask for releasing *less* often (we can't keep up). I think that's crazy too, but that shows 6 months is really a trade-off. -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On 02/24/2015 03:21 PM, Joe Gordon wrote: On Tue, Feb 24, 2015 at 6:57 AM, Daniel P. Berrange berra...@redhat.com mailto:berra...@redhat.com wrote: On Tue, Feb 24, 2015 at 08:50:45AM -0500, Sean Dague wrote: On 02/24/2015 07:48 AM, Russell Bryant wrote: On 02/24/2015 12:54 PM, Daniel P. Berrange wrote: On Tue, Feb 24, 2015 at 11:48:29AM +, Chris Dent wrote: On Tue, 24 Feb 2015, Daniel P. Berrange wrote: need to do more work. If this is so, then I don't think this is a blocker, it is just a sign that the project needs to focus on providing more resources to the teams impacted in that way. What are the mechanisms whereby the project provides more resources to teams? The technical committee and / or foundation board can highlight the need for investment of resources in critical areas of the project, to either the community members or vendors involved. As an example, this was done successfully recently to increase involvement in maintaining the EC2 API support. There are plenty of vendors involved in OpenStack which have the ability to target resources, if they can learn where those resources are best spent. Indeed ... and if horizontal teams are the ones hit the most by the extra work, each project should help with that burden. For example, projects may need to take their responsibility for documentation more seriously and require documentation with features (content at least, not necessarily integration into the proper documentation deliverables) instead of assuming it magically gets written later. Right, and I think this actually hits at the most important part of the discussion. The question of: 1) what would we need to do to make different release cadences viable? 2) are those good things to do regardless of release cadence? The horizontal teams really can't function at different cadences. It completely breaks any flow and planning at turns them even further into firefighting because now everyone has crunch time at different times, and the horizontal efforts are required to just play catch up. I know what that future looks like, the horizontal teams dry up because no one wants that job. Ok, so that being said, what we'd need to do is have horizontal teams move to more of a self supporting model. So that the relevant content for a project (docs, full stack tests, requirements, etc) all live within that project itself, and aren't centrally synchronized. Installation of projects needs to be fully isolated from each other so that upgrading project A can be done independent of project B, as their release cadences might all bit disparate. Basically, ever OpenStack project needs to reabsorb the cross project efforts they've externalized. Then if project A decided to move off the coupled release, it's impact to the rest would be minimal. These are robust components that stand on their own, and work well with robust other components. Which... is basically the point of the big tent / new project governance model. Decompose OpenStack from a giant blob of goo into Robust elements that are more loosely coupled (so independently robust, and robust in their interaction with others). Move the horizontal teams into infrastructure vs. content roles, have projects own more of this content themselves. But it is a long hard process. Devstack external plugins was implement to support this kind of model, but having walked a bunch of different teams through this (at all skill levels) there ends up being a lot of work to get this right, and a lot of rethinking by teams that assumed their interaction with full stack testing is something they'd get to contribute once and have someone else maintain (instead of something they now need dedicated watchful eye on). The amount of full stack configurations immediately goes beyond anywhere near testable, so it requires more robust project testing to ensure every exposed interface is more robust (i.e. the testing in pyramids from https://review.openstack.org/#/c/150653/). And, I think the answer to #2 is: yes, this just makes it all better. So, honestly, I'm massively supportive of the end game. I've been carving out the bits of this I can for the last six months. But I think the way we get there is to actually get the refactoring of the horizontal efforts first. I pretty much fully agree that refactoring the horizontal efforts to distribute responsbility across the
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On Tue, Feb 24, 2015 at 6:57 AM, Daniel P. Berrange berra...@redhat.com wrote: On Tue, Feb 24, 2015 at 08:50:45AM -0500, Sean Dague wrote: On 02/24/2015 07:48 AM, Russell Bryant wrote: On 02/24/2015 12:54 PM, Daniel P. Berrange wrote: On Tue, Feb 24, 2015 at 11:48:29AM +, Chris Dent wrote: On Tue, 24 Feb 2015, Daniel P. Berrange wrote: need to do more work. If this is so, then I don't think this is a blocker, it is just a sign that the project needs to focus on providing more resources to the teams impacted in that way. What are the mechanisms whereby the project provides more resources to teams? The technical committee and / or foundation board can highlight the need for investment of resources in critical areas of the project, to either the community members or vendors involved. As an example, this was done successfully recently to increase involvement in maintaining the EC2 API support. There are plenty of vendors involved in OpenStack which have the ability to target resources, if they can learn where those resources are best spent. Indeed ... and if horizontal teams are the ones hit the most by the extra work, each project should help with that burden. For example, projects may need to take their responsibility for documentation more seriously and require documentation with features (content at least, not necessarily integration into the proper documentation deliverables) instead of assuming it magically gets written later. Right, and I think this actually hits at the most important part of the discussion. The question of: 1) what would we need to do to make different release cadences viable? 2) are those good things to do regardless of release cadence? The horizontal teams really can't function at different cadences. It completely breaks any flow and planning at turns them even further into firefighting because now everyone has crunch time at different times, and the horizontal efforts are required to just play catch up. I know what that future looks like, the horizontal teams dry up because no one wants that job. Ok, so that being said, what we'd need to do is have horizontal teams move to more of a self supporting model. So that the relevant content for a project (docs, full stack tests, requirements, etc) all live within that project itself, and aren't centrally synchronized. Installation of projects needs to be fully isolated from each other so that upgrading project A can be done independent of project B, as their release cadences might all bit disparate. Basically, ever OpenStack project needs to reabsorb the cross project efforts they've externalized. Then if project A decided to move off the coupled release, it's impact to the rest would be minimal. These are robust components that stand on their own, and work well with robust other components. Which... is basically the point of the big tent / new project governance model. Decompose OpenStack from a giant blob of goo into Robust elements that are more loosely coupled (so independently robust, and robust in their interaction with others). Move the horizontal teams into infrastructure vs. content roles, have projects own more of this content themselves. But it is a long hard process. Devstack external plugins was implement to support this kind of model, but having walked a bunch of different teams through this (at all skill levels) there ends up being a lot of work to get this right, and a lot of rethinking by teams that assumed their interaction with full stack testing is something they'd get to contribute once and have someone else maintain (instead of something they now need dedicated watchful eye on). The amount of full stack configurations immediately goes beyond anywhere near testable, so it requires more robust project testing to ensure every exposed interface is more robust (i.e. the testing in pyramids from https://review.openstack.org/#/c/150653/). And, I think the answer to #2 is: yes, this just makes it all better. So, honestly, I'm massively supportive of the end game. I've been carving out the bits of this I can for the last six months. But I think the way we get there is to actually get the refactoring of the horizontal efforts first. I pretty much fully agree that refactoring the horizontal efforts to distribute responsbility across the individual projects is the way forward. I don't think it needs to be a pre-requisite step for changing the release cycle. We can do both in parallel if we put our minds to it. My biggest fear is that we just keeping debating problems and alternatives, but continue to be too afraid of theoretical problems, to actually take the risk of effecting meaningful improvemnts in the operation of the project. I tend to agree with you on this. I don't think completing the refactoring of the horizontal
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On 02/24/2015 01:28 PM, Kashyap Chamarthy wrote: On Tue, Feb 24, 2015 at 11:54:31AM +, Daniel P. Berrange wrote: On Tue, Feb 24, 2015 at 11:48:29AM +, Chris Dent wrote: On Tue, 24 Feb 2015, Daniel P. Berrange wrote: need to do more work. If this is so, then I don't think this is a blocker, it is just a sign that the project needs to focus on providing more resources to the teams impacted in that way. What are the mechanisms whereby the project provides more resources to teams? Along with the below, if push comes to shove, OpenStack Foundation could probably try a milder variant (obviously, not all activities can be categorized as 'critical path') of Linux Foundation's Critical Infrastructure Protection Initiative[1] to fund certain project activities in need. The OpenStack Foundation effectively already does this. In particular, the Foundation is helping fund critical horizontal efforts like release management, infrastructure, and community management. -- Russell Bryant __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On Tue, Feb 24, 2015 at 11:54:31AM +, Daniel P. Berrange wrote: On Tue, Feb 24, 2015 at 11:48:29AM +, Chris Dent wrote: On Tue, 24 Feb 2015, Daniel P. Berrange wrote: need to do more work. If this is so, then I don't think this is a blocker, it is just a sign that the project needs to focus on providing more resources to the teams impacted in that way. What are the mechanisms whereby the project provides more resources to teams? Along with the below, if push comes to shove, OpenStack Foundation could probably try a milder variant (obviously, not all activities can be categorized as 'critical path') of Linux Foundation's Critical Infrastructure Protection Initiative[1] to fund certain project activities in need. The technical committee and / or foundation board can highlight the need for investment of resources in critical areas of the project, to either the community members or vendors involved. As an example, this was done successfully recently to increase involvement in maintaining the EC2 API support. There are plenty of vendors involved in OpenStack which have the ability to target resources, if they can learn where those resources are best spent. [1] http://www.linuxfoundation.org/programs/core-infrastructure-initiative -- /kashyap __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On 24/02/15 11:02 +, Daniel P. Berrange wrote: On Tue, Feb 24, 2015 at 10:44:57AM +, Chris Dent wrote: On Tue, 24 Feb 2015, Daniel P. Berrange wrote: I was writing this mail for the past few days, but the nova thread today prompted me to finish it off send it :-) Thanks for doing this. I think you're probably right that the current release cycle has many negative impacts on the development process and deserve at least some critical thinking if not outright changing. Thanks especially for listing some expected questions (with answers). One additional question I'd like to see answered in some depth is: Why have unified release schedules? That is, why should Nova and Glance or anything else release on the same schedule as any other OpenStack-related project? Disentangling the release cycles might lead to stronger encapsulation of, and stronger contracts between, the projects. It might also lead to a total mess. For peripheral projects I don't think co-ordinate release cycle is needed, but for the core projects I think it is helpful in general to have some co-ordination of releases. It allows marketing to more effectively promote the new project releases, it helps when landing features that span across projects to know they'll be available to users at the same time in general and minimize burden on devs users to remember many different dates. It is hard enough remembering the dates for our coordinated release cycle, let alone dates for 30 different project cycles. IME predictable dates is a really important useful thing to have for a planning POV. This is why I suggested, we do a 2 month cycle, with a strict date of 1st of the month in Feb, Apr, Jun, Aug, Oct, Dec. To this I'd also add that bug fixing is way easier when you have aligned releases for projects that are expected to be deployed together. It's easier to know what the impact of a change/bug is throughout the infrastructure. Flavio Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco pgpFaVXWQm9Ii.pgp Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On Tue, Feb 24, 2015 at 11:48:29AM +, Chris Dent wrote: On Tue, 24 Feb 2015, Daniel P. Berrange wrote: need to do more work. If this is so, then I don't think this is a blocker, it is just a sign that the project needs to focus on providing more resources to the teams impacted in that way. What are the mechanisms whereby the project provides more resources to teams? The technical committee and / or foundation board can highlight the need for investment of resources in critical areas of the project, to either the community members or vendors involved. As an example, this was done successfully recently to increase involvement in maintaining the EC2 API support. There are plenty of vendors involved in OpenStack which have the ability to target resources, if they can learn where those resources are best spent. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On Tue, 24 Feb 2015, Daniel P. Berrange wrote: I was writing this mail for the past few days, but the nova thread today prompted me to finish it off send it :-) Thanks for doing this. I think you're probably right that the current release cycle has many negative impacts on the development process and deserve at least some critical thinking if not outright changing. Thanks especially for listing some expected questions (with answers). One additional question I'd like to see answered in some depth is: Why have unified release schedules? That is, why should Nova and Glance or anything else release on the same schedule as any other OpenStack-related project? Disentangling the release cycles might lead to stronger encapsulation of, and stronger contracts between, the projects. It might also lead to a total mess. Thanks. -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On Tue, 24 Feb 2015, Daniel P. Berrange wrote: need to do more work. If this is so, then I don't think this is a blocker, it is just a sign that the project needs to focus on providing more resources to the teams impacted in that way. What are the mechanisms whereby the project provides more resources to teams? -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On 02/24/2015 12:54 PM, Daniel P. Berrange wrote: On Tue, Feb 24, 2015 at 11:48:29AM +, Chris Dent wrote: On Tue, 24 Feb 2015, Daniel P. Berrange wrote: need to do more work. If this is so, then I don't think this is a blocker, it is just a sign that the project needs to focus on providing more resources to the teams impacted in that way. What are the mechanisms whereby the project provides more resources to teams? The technical committee and / or foundation board can highlight the need for investment of resources in critical areas of the project, to either the community members or vendors involved. As an example, this was done successfully recently to increase involvement in maintaining the EC2 API support. There are plenty of vendors involved in OpenStack which have the ability to target resources, if they can learn where those resources are best spent. Indeed ... and if horizontal teams are the ones hit the most by the extra work, each project should help with that burden. For example, projects may need to take their responsibility for documentation more seriously and require documentation with features (content at least, not necessarily integration into the proper documentation deliverables) instead of assuming it magically gets written later. -- Russell Bryant __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On Tue, Feb 24, 2015 at 08:50:45AM -0500, Sean Dague wrote: On 02/24/2015 07:48 AM, Russell Bryant wrote: On 02/24/2015 12:54 PM, Daniel P. Berrange wrote: On Tue, Feb 24, 2015 at 11:48:29AM +, Chris Dent wrote: On Tue, 24 Feb 2015, Daniel P. Berrange wrote: need to do more work. If this is so, then I don't think this is a blocker, it is just a sign that the project needs to focus on providing more resources to the teams impacted in that way. What are the mechanisms whereby the project provides more resources to teams? The technical committee and / or foundation board can highlight the need for investment of resources in critical areas of the project, to either the community members or vendors involved. As an example, this was done successfully recently to increase involvement in maintaining the EC2 API support. There are plenty of vendors involved in OpenStack which have the ability to target resources, if they can learn where those resources are best spent. Indeed ... and if horizontal teams are the ones hit the most by the extra work, each project should help with that burden. For example, projects may need to take their responsibility for documentation more seriously and require documentation with features (content at least, not necessarily integration into the proper documentation deliverables) instead of assuming it magically gets written later. Right, and I think this actually hits at the most important part of the discussion. The question of: 1) what would we need to do to make different release cadences viable? 2) are those good things to do regardless of release cadence? The horizontal teams really can't function at different cadences. It completely breaks any flow and planning at turns them even further into firefighting because now everyone has crunch time at different times, and the horizontal efforts are required to just play catch up. I know what that future looks like, the horizontal teams dry up because no one wants that job. Ok, so that being said, what we'd need to do is have horizontal teams move to more of a self supporting model. So that the relevant content for a project (docs, full stack tests, requirements, etc) all live within that project itself, and aren't centrally synchronized. Installation of projects needs to be fully isolated from each other so that upgrading project A can be done independent of project B, as their release cadences might all bit disparate. Basically, ever OpenStack project needs to reabsorb the cross project efforts they've externalized. Then if project A decided to move off the coupled release, it's impact to the rest would be minimal. These are robust components that stand on their own, and work well with robust other components. Which... is basically the point of the big tent / new project governance model. Decompose OpenStack from a giant blob of goo into Robust elements that are more loosely coupled (so independently robust, and robust in their interaction with others). Move the horizontal teams into infrastructure vs. content roles, have projects own more of this content themselves. But it is a long hard process. Devstack external plugins was implement to support this kind of model, but having walked a bunch of different teams through this (at all skill levels) there ends up being a lot of work to get this right, and a lot of rethinking by teams that assumed their interaction with full stack testing is something they'd get to contribute once and have someone else maintain (instead of something they now need dedicated watchful eye on). The amount of full stack configurations immediately goes beyond anywhere near testable, so it requires more robust project testing to ensure every exposed interface is more robust (i.e. the testing in pyramids from https://review.openstack.org/#/c/150653/). And, I think the answer to #2 is: yes, this just makes it all better. So, honestly, I'm massively supportive of the end game. I've been carving out the bits of this I can for the last six months. But I think the way we get there is to actually get the refactoring of the horizontal efforts first. I pretty much fully agree that refactoring the horizontal efforts to distribute responsbility across the individual projects is the way forward. I don't think it needs to be a pre-requisite step for changing the release cycle. We can do both in parallel if we put our minds to it. My biggest fear is that we just keeping debating problems and alternatives, but continue to be too afraid of theoretical problems, to actually take the risk of effecting meaningful improvemnts in the operation of the project. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On Tue, 24 Feb 2015, Sean Dague wrote: That also provides a very concrete answer to will people show up. Because if they do, and we get this horizontal refactoring happening, then we get to the point of being able to change release cadences faster. If they don't, we remain with the existing system. Vs changing the system and hoping someone is going to run in and backfill the breaks. Isn't this the way of the world? People only put halon in the machine room after the fire. I agree that people showing up is a real concern, but I also think that we shy away too much from the productive energy of stuff breaking. It's the breakage that shows where stuff isn't good enough. [Flavio said]: To this I'd also add that bug fixing is way easier when you have aligned releases for projects that are expected to be deployed together. It's easier to know what the impact of a change/bug is throughout the infrastructure. Can't this be interpreted as an excuse for making software which does not have a low surface area and a good API? (Note I'm taking a relatively unrealistic position for sake of conversation.) -- Chris Dent tw:@anticdent freenode:cdent https://tank.peermore.com/tanks/cdent __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On 02/24/2015 09:37 AM, Chris Dent wrote: On Tue, 24 Feb 2015, Sean Dague wrote: That also provides a very concrete answer to will people show up. Because if they do, and we get this horizontal refactoring happening, then we get to the point of being able to change release cadences faster. If they don't, we remain with the existing system. Vs changing the system and hoping someone is going to run in and backfill the breaks. Isn't this the way of the world? People only put halon in the machine room after the fire. I agree that people showing up is a real concern, but I also think that we shy away too much from the productive energy of stuff breaking. It's the breakage that shows where stuff isn't good enough. [Flavio said]: To this I'd also add that bug fixing is way easier when you have aligned releases for projects that are expected to be deployed together. It's easier to know what the impact of a change/bug is throughout the infrastructure. Can't this be interpreted as an excuse for making software which does not have a low surface area and a good API? (Note I'm taking a relatively unrealistic position for sake of conversation.) I'm not so sure about that. IMO, much of this goes back to the question of whether OpenStack services are APIs or implementations. This was debated with much heat at the Diablo summit (Hi Jay). I frequently have conversations where there is an issue about release X vs Y when it is really about api versions. Even if we say that we are about implementations as well as apis, we can start to organize our processes and code as if we were just apis. If each service had a well-defined, versioned, discoverable, well-tested api, then projects could follow their own release schedule, relying on distros or integrators to put the pieces together and verify the quality of the whole stack to the users. Such entities could still collaborate on that task, and still identify longer release cycles, using stable branches. The upstream project could still test the latest released versions together. Some of these steps are now being taken to resolve gate issues and horizontal resource issues. Doing this would vastly increase agility but with some costs: 1. The upstream project would likely have to give up on the worthy goal of providing an actual deployable stack that could be used as an alternative to AWS, etc. That saddens me, but for various reasons, including that we do no scale/performance testing on the upstream code, we are not achieving that goal anyway. The big tent proposals are also a move away from that goal. 2. We would have to give up on incompatible api changes. But with the replacement of nova v3 with microversions we are already doing that. Massive adoption with release agility is simply incompatible with allowing incompatible api changes. Most of this is just echoing what Jay said. I think this is the way any SOA would be designed. If we did this, and projects released frequently, would there be a reason for any one to be chasing master? -David __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On Tue, Feb 24, 2015 at 10:04 AM, David Kranz dkr...@redhat.com wrote: On 02/24/2015 09:37 AM, Chris Dent wrote: On Tue, 24 Feb 2015, Sean Dague wrote: That also provides a very concrete answer to will people show up. Because if they do, and we get this horizontal refactoring happening, then we get to the point of being able to change release cadences faster. If they don't, we remain with the existing system. Vs changing the system and hoping someone is going to run in and backfill the breaks. Isn't this the way of the world? People only put halon in the machine room after the fire. I agree that people showing up is a real concern, but I also think that we shy away too much from the productive energy of stuff breaking. It's the breakage that shows where stuff isn't good enough. [Flavio said]: To this I'd also add that bug fixing is way easier when you have aligned releases for projects that are expected to be deployed together. It's easier to know what the impact of a change/bug is throughout the infrastructure. Can't this be interpreted as an excuse for making software which does not have a low surface area and a good API? (Note I'm taking a relatively unrealistic position for sake of conversation.) I'm not so sure about that. IMO, much of this goes back to the question of whether OpenStack services are APIs or implementations. This was debated with much heat at the Diablo summit (Hi Jay). I frequently have conversations where there is an issue about release X vs Y when it is really about api versions. Even if we say that we are about implementations as well as apis, we can start to organize our processes and code as if we were just apis. If each service had a well-defined, versioned, discoverable, well-tested api, then projects could follow their own release schedule, relying on distros or integrators to put the pieces together and verify the quality of the whole stack to the users. Such entities could still collaborate on that task, and still identify longer release cycles, using stable branches. The upstream project could still test the latest released versions together. Some of these steps are now being taken to resolve gate issues and horizontal resource issues. Doing this would vastly increase agility but with some costs: 1. The upstream project would likely have to give up on the worthy goal of providing an actual deployable stack that could be used as an alternative to AWS, etc. That saddens me, but for various reasons, including that we do no scale/performance testing on the upstream code, we are not achieving that goal anyway. The big tent proposals are also a move away from that goal. 2. We would have to give up on incompatible api changes. But with the replacement of nova v3 with microversions we are already doing that. Massive adoption with release agility is simply incompatible with allowing incompatible api changes. Most of this is just echoing what Jay said. I think this is the way any SOA would be designed. If we did this, and projects released frequently, would there be a reason for any one to be chasing master? -David __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Seems like some of the proposals around frequency (increasing in particular) just sort of move the bottlenecks around. Honestly, I thought we were already on a path with some of the ideas that Sean and David (and indirectly Jay) proposed. Get rid of the whole coordinated release altogether. I think there should still be some sort of tagging or something at some interval that just says here's a point in time collection that we call X. Another proposal I think I've talked to some folks about is a true CI/Train model. Cut out some of the artificial milestone deadlines etc, just keep rolling and what's ready at the release point is what's ready; you make a cut of what's there at that time and roll on. Basically eliminate the feature freeze and other components and hopefully keep feature commit distributed. There are certainly all sorts of gotchas here, but I don't think it's very interesting to most so I won't go into a bunch of theory on it. Regardless, I do think that no matter the direction we all seem to be of the opinion that we need to move more towards projects being responsible for more of the horizontal functions (as Sean put it). Personally I think this is a great direction for a number of reasons, and I also think that it might force us to be better about our API's and requirements than we have been in the past. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe:
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On 02/24/2015 12:33 PM, John Griffith wrote: On Tue, Feb 24, 2015 at 10:04 AM, David Kranz dkr...@redhat.com mailto:dkr...@redhat.com wrote: On 02/24/2015 09:37 AM, Chris Dent wrote: On Tue, 24 Feb 2015, Sean Dague wrote: That also provides a very concrete answer to will people show up. Because if they do, and we get this horizontal refactoring happening, then we get to the point of being able to change release cadences faster. If they don't, we remain with the existing system. Vs changing the system and hoping someone is going to run in and backfill the breaks. Isn't this the way of the world? People only put halon in the machine room after the fire. I agree that people showing up is a real concern, but I also think that we shy away too much from the productive energy of stuff breaking. It's the breakage that shows where stuff isn't good enough. [Flavio said]: To this I'd also add that bug fixing is way easier when you have aligned releases for projects that are expected to be deployed together. It's easier to know what the impact of a change/bug is throughout the infrastructure. Can't this be interpreted as an excuse for making software which does not have a low surface area and a good API? (Note I'm taking a relatively unrealistic position for sake of conversation.) I'm not so sure about that. IMO, much of this goes back to the question of whether OpenStack services are APIs or implementations. This was debated with much heat at the Diablo summit (Hi Jay). I frequently have conversations where there is an issue about release X vs Y when it is really about api versions. Even if we say that we are about implementations as well as apis, we can start to organize our processes and code as if we were just apis. If each service had a well-defined, versioned, discoverable, well-tested api, then projects could follow their own release schedule, relying on distros or integrators to put the pieces together and verify the quality of the whole stack to the users. Such entities could still collaborate on that task, and still identify longer release cycles, using stable branches. The upstream project could still test the latest released versions together. Some of these steps are now being taken to resolve gate issues and horizontal resource issues. Doing this would vastly increase agility but with some costs: 1. The upstream project would likely have to give up on the worthy goal of providing an actual deployable stack that could be used as an alternative to AWS, etc. That saddens me, but for various reasons, including that we do no scale/performance testing on the upstream code, we are not achieving that goal anyway. The big tent proposals are also a move away from that goal. 2. We would have to give up on incompatible api changes. But with the replacement of nova v3 with microversions we are already doing that. Massive adoption with release agility is simply incompatible with allowing incompatible api changes. Most of this is just echoing what Jay said. I think this is the way any SOA would be designed. If we did this, and projects released frequently, would there be a reason for any one to be chasing master? -David __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: OpenStack-dev-request@lists.__openstack.org?subject:__unsubscribe http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Seems like some of the proposals around frequency (increasing in particular) just sort of move the bottlenecks around. Honestly, I thought we were already on a path with some of the ideas that Sean and David (and indirectly Jay) proposed. Get rid of the whole coordinated release altogether. I think there should still be some sort of tagging or something at some interval that just says here's a point in time collection that we call X. Another proposal I think I've talked to some folks about is a true CI/Train model. Cut out some of the artificial milestone deadlines etc, just keep rolling and what's ready at the release point is what's ready; you make a cut of what's there at that time and roll on. Basically eliminate the feature freeze and other components and hopefully keep feature commit distributed. There are certainly all sorts of gotchas
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On Tue, Feb 24, 2015, at 04:28, Kashyap Chamarthy wrote: Along with the below, if push comes to shove, OpenStack Foundation could probably try a milder variant (obviously, not all activities can be categorized as 'critical path') of Linux Foundation's Critical Infrastructure Protection Initiative[1] to fund certain project activities in need. Speaking as a person who sits on the LF CII board meetings, and helps turn the crank on that particular sausage mill, no, we really don't want to go down that path at this point in time. -- Mark Atwood, Director of Open Source Engagement, HP __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On 25 February 2015 at 13:13, Jeremy Stanley fu...@yuggoth.org wrote: On 2015-02-24 11:27:05 + (+), Daniel P. Berrange wrote: [...] It would be reasonable for the vulnerability team to take the decision that they'll support fixes for master, and any branches that the stable team decide to support. [...] Well, it's worth noting that the VMT doesn't even support (i.e. issue advisories for bugs in) master branches now, the exception being branchless projects where the bug appears in master prior to an existing release tag. But I think Thierry's earlier point is that as soon as you start marking _some_ releases as special (supported by VMT, stable maint, docs, translators...) then those become your new actual releases and the other interim releases become your milestones, and we're back to the current model again. I don't think thats true actually. We'd still have a major smoothing effect on work, which means less high peaks at release time and less troughs at 'review direction' time and so on. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On 2015-02-24 11:27:05 + (+), Daniel P. Berrange wrote: [...] It would be reasonable for the vulnerability team to take the decision that they'll support fixes for master, and any branches that the stable team decide to support. [...] Well, it's worth noting that the VMT doesn't even support (i.e. issue advisories for bugs in) master branches now, the exception being branchless projects where the bug appears in master prior to an existing release tag. But I think Thierry's earlier point is that as soon as you start marking _some_ releases as special (supported by VMT, stable maint, docs, translators...) then those become your new actual releases and the other interim releases become your milestones, and we're back to the current model again. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On 24 February 2015 at 22:53, Daniel P. Berrange berra...@redhat.com wrote: I was writing this mail for the past few days, but the nova thread today prompted me to finish it off send it :-) ++ The first two observations strongly suggest that the choice of 6 months as a cycle length is a fairly arbitrary decision that can be changed without unreasonable pain. The third observation suggests a much shorter cycle length would smooth out the bumps and lead to a more efficient satisfying development process for all involved. I'm very glad to see this being discussed (again :)). Any proposal that reduces our cycle time is going to get my enthusiastic support. ... Upgrades deprecation -- It is common right now for projects to say upgrades are only supported between releases N-1 and N. ie to go from Icehouse to Kilo, you need to first deploy Juno. This is passable when you're talking 6 month gaps between cycles, but when there are 2 month gaps it is not reasonable to expect everyone to be moving fast enough to keep up with every release. If an organization's beurocracy means they can't deploy more often than every 12 months, forcing them to deploy the 5 intermediate releases to run upgrade scripts is quite unpleasant. We would likely have to look at officially allowing upgrades between any (N-3, N-2, N-1) to N. From a database POV this should not be hard, since the DB migration scripts don't have any built in assumptions about this. Similarly the versioned objects used by Nova are quite flexible in this regard too, as long as the compat code isn't deleted too soon. Deprecation warnings would need similar consideration. It would not be sufficient to deprecate in one release and delete in the next. We'd likely want to say that depecations last for a given time period rather than number of releases, eg 6 months. This could be easily handled by simply including the date of initial deprecation in the deprecation message. It would thus be clear when the feature will be removed to all involved. I think a useful thing here is to consider online upgrades vs offline upgrades. If we care to we could say that online upgrades are only supported release to release, with offline upgrades being supported by releases up to 8 months apart. The benefit of this would be to reduce our test matrix: online upgrades are subject to much greater interactions between concurrent processes, and its hard enough to validate that N - N+1 works with any deep confidence vs also checking that N-N+2, N-N+3 also work: for a 6 month sliding window to match the current thing, we need to allow upgrades from Feb through August: a Feb-Apr b Feb-Jun c Feb-Aug d Apr- Jun e Apr-Aug f Jun-Aug We'd need to be testing all the combinations leading to the branch a patch is for, so changes to Aug would need c, e and f all tested. Thats 3 times the overhead of supporting: Feb-Apr and then Apr-Jun and then Jun-Aug serially where we'd only be testing one combination at a time. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On 24/02/15 19:27, Daniel P. Berrange wrote: On Tue, Feb 24, 2015 at 12:05:17PM +0100, Thierry Carrez wrote: Daniel P. Berrange wrote: [...] First, Daniel, thank you for the well-written and thought-through post. I have some comments on translation specifically which I hope can shed some light on this particular horizontal effort. With the idea as stated below implemented, I think it would basically demoralise our translation teams to the point they'd give up. We already get complaints about people not respecting string freeze as it is :D I'm not familiar with how the translations works, but if they are waiting until the freeze before starting translation work I'd say that is a mistaken approach. Obviously during active dev part of the cycle, some translated strings are in flux, so if translation was taking place in parallel there could be some wasted effort, but I'd expect that to be the minority case. I think the majority of translation work can be done in parallel with dev work and the freeze time just needs to tie up the small remaining bits. So, two points: 1) We wouldn't be talking about throwing just a couple of percent of their work away. As an example, even without looking at the introduction of new strings or deleting others, you may not be aware that changing a single word in a string in the code means that entire string needs to be re-translated. Even with the extensive translation memory systems we have making suggestions as best they can, we're talking about very, very significant amounts of wasted effort. Something as simple as adding ing on a verb to fix an English grammar mistake means a completely different sentence in languages I'm familiar with. 2) The overwhelming majority of our [hundreds of] translators are volunteers. Unlike many of those writing the software, they are not paid to do what they do, and do it in their spare time. Make it less fun, and they simply walk away. To try and put this in a way that may be more understandable for a non-translator ... your (impressive!) original email in this thread was around 3000 words. Translatable strings in horizon is around five times that at the moment. So imagine that, when writing an email five times longer than the one you wrote, unpaid, someone you don't really know that well decided that the section on the key observations (230 words - about 1% of the text of our 'horizon') you just wrote should be re-arranged - the order of the observations changed, one of them removed and replaced with another slightly different one, and the conclusion paragraph wording should be amended to suit. It would be an understatement to say that such behaviour would be 'annoying' if it happened constantly as you were writing your email. Consider then if it applied to every email you sought to write :) Now, the amount of string changes within a release can be left for someone to work out, but it's certainly a great deal more than a single percent. Multiply that poor experience by the reality of string change across all the projects we want to translate. Then multiply it by the number of languages we want. Finally, multiply it by the number of people we need to translate a single language. That's a really bad time for a whole lot of people. At the moment we're fixing this with string freeze, and that's generally going pretty well. Right now I don't have a good solution if the strings in code never stop changing for any period of time, but what has been proposed above while well-meaning is unfortunately not workable. We really need to keep our translators as happy as we can. People who are literate in multiple languages are difficult to find, moreso those who have technical vocabulary experience in both, and even moreso those who'll give up their time to help us reach our mission goal of being the ubiquitous Open Source Cloud Computing platform. We do need them to achieve it. Regards, Tom __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle
On 25 February 2015 at 00:05, Thierry Carrez thie...@openstack.org wrote: Daniel P. Berrange wrote: [...] I think you're judging the cycle from the perspective of developers only. You said that in the other thread, and I think its false. There is a very good case to be made that the release cycle is directly responsible for: - increased feature latency [when users get new shiny] - already explained here. - decreased stability due to feature crush - decreased quality for the same reason - we don't put things in and bake them before exposing, we get them in and since there is a long gap before they can be consumed they're generally just enabled immediately. Keystone's steps to address that are excellent, but made harder by the gap between releases: there is social pressure to make things graduate to support within each release. 6 months was not an arbitrary decision. Translations and documentation teams basically need a month of feature/string freeze in order to complete their work. Since we can't reasonably freeze one month every 2 months, we picked 6 months. Do they need a month because 6 months worth of changes have been aggregated? E.g. if they have 2 months of changes to deal with, do they need 1.5 weeks? Perhaps we could branch the code release in advance of docs and translations, and announce the release when those artifacts are ready? It's also worth noting that we were on a 3-month cycle at the start of OpenStack. That was dropped after a cataclysmic release that managed the feat of (a) not having anything significant done, and (b) have out of date documentation and translations. Oh! https://wiki.openstack.org/w/index.php?title=Obsolete:3MonthReleaseCycledirection=prevoldid=14016 appears to be the page about that, but my google-fu isn't turning up a post-mortem of the C release which prompted the change. Perhaps some old-hand like you can fill us in on the details :). Stable branches may have the luxury of skipping releases and designate a stable one from time to time (I reject the Linux comparison because the kernel is at a very different moment in software lifecycle). The trick being, making one release special is sure to recreate the peak issues you're trying to solve. I'm actually more concerned about the teams that don't have the luxury of skipping a release (like the vulnerability management team). Docs and translations, as mentioned above, will also have a hard time producing quality docs and translations every 2 months with a very short freeze period. I agree that the vulnerability team will be rather more impacted by this than regular project teams - I made a nod to that in the other thread. More thought needed still :). I may appear defensive on my answers, but it's not my goal to defend the current system: it's just that most of those proposals generally ignore the diversity of the needs of the teams that make OpenStack possible, to focus on a particular set of contributors' woes. I'm trying to bring that wider perspective in -- the current system is a trade-off and the result of years of evolution, not an arbitrary historic choice that we can just change at will. I agree that its not arbitrary and that changing it requires some appropriate wide-spread consultation; OTOH the benefits of rolling and higher frequency releases are really substantial: but we have to backup the change with the appropriate engineering (such as reducing our frictions that cause teams practicing CD to be weeks behind tip) to make it feasible. My greatest concern about the proposals happening now is that we may bite off more than we can chew. OTGH the reality is that all the negative things multiply out, so we probably need to just start somewhere and /do/ to give us the space to fix other things. -Rob -- Robert Collins rbtcoll...@hp.com Distinguished Technologist HP Converged Cloud __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
