Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle [metrics]
Stefano Maffulli writes: > In any case, since Sean said that nova (and other projects) already > remove unmergeable changesets regularly, I think the data are already > "clean enough" to give us food for thoughts. I am asking you to please independently remove changes that you don't think should be considered from your metrics. If you rely on Sean or others to abandon changes, then you are, in essence, relying on core reviewers abandoning changes for the purposes of providing "clean" (as you put it) input to a metrics system. I think abandoning changes so that the metrics look the way we want is a terrible experience for contributors. Especially as it appears some projects, such as nova, are in a position where they are actually leaving -2 votes on changes which will not be lifted for 2 or 3 months. That means that if someone runs a script like Sean's, these changes will be abandoned, yet there is nothing that the submitter can do to progress the change in the mean time. Abandoning such a review is making an already bad experience for contributors even worse. -Jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle [metrics]
On Thu, 2015-02-26 at 16:44 -0800, James E. Blair wrote: > It is good to recognize the impact of this, however, I would suggest > that if having open changes that are not "actively being worked" is a > problem for statistics, I don't think it's a problem for the statistics per se. The reports are only a tool to analyze complex phenomenons and translate them into manageable items. In fact, we keep adding more stats as we go because every chart and table leaves us with more questions. > let's change the statistics calculation. Please do not abandon the > work of contributors to improve the appearance of > these metrics. Instead, simply decide what criteria you think should > apply and exclude those changes from your calculations. I'm currently thinking that it would be informative to plot the distribution of the efficiency metrics, instead of simply come up with a filter to ignore long standing changes with slow/null activity over some arbitrary amount of time. I think it would be more interesting to see how many 'inactive' vs 'active' there are at a given time. In any case, since Sean said that nova (and other projects) already remove unmergeable changesets regularly, I think the data are already "clean enough" to give us food for thoughts. Why owners seem to be getting slower and slower to provide new patches, despite the fact that the number of patches per changeset is fairly stable? I'll look into the data more carefully with Daniel Izquierdo as I think there are huge outliers skewing the data (the diff between median and average is huge). /stef __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle [metrics]
On Fri, Feb 27, 2015 at 4:02 AM, Daniel P. Berrange wrote: > On Fri, Feb 27, 2015 at 09:51:34AM +1100, Michael Still wrote: > > On Fri, Feb 27, 2015 at 9:41 AM, Stefano Maffulli > wrote: > > > > > Does it make sense to purge old stuff regularly so we have a better > > > overview? Or maybe we should chart a distribution of age of proposed > > > changesets, too in order to get a better understanding of where the > > > outliers are? > > > > Given the abandon of a review isn't binding (a proposer can easily > > unabandon), I do think we should abandon more than we do now. The > > problem at the moment being that its a manual process which isn't much > > fun for the person doing the work. > > > > Another factor to consider here is that abandoned patches against bugs > > make the bug look like someone is working on a fix, which probably > > isn't the case. > > > > Nova has been trying some very specific things to try and address > > these issues, and I think we're improving. Those things are: > > > > * specs > > * priority features > > This increased level of process in Nova has actually made the negative > effects of the 6 month cycle noticably worse on balance. If you aren't > able to propose your feature in the right window of the dev cycle your > chances of getting stuff merged has gone down significantly and the time > before users are likely to see your feature has correspondingly gone up. > Previously people could come along with simple features at the end of > the cycle and we had the flexibility to be pragmmatic and review and > approve them. Now we're lacking them that ability even if we have the > spare review cycles to consider it. The processes adopted have merely > made us more efficient at disappointing contributors earlier in the > cycle. There's been no changes made that would solve the bigger problem > of the fact that Nova is far too large vs the size of the core review > team, so we have a ongoing major bottleneck in our development. That, > bottleneck combined with the length of the 6 month cycle is an ongoing > disaster for our contributors. > > This is part of the reason we have moved to split Neutron into smaller, bite-sized chunk repositories with sometimes overlapping core reviewer teams. It's also why we're spinning out the backend logic from in-tree drivers and plugins to allow faster iteration for the maintainers. Early evidence indicates this has been succesful, we'll see how it looks once we get into the Liberty development cycle. For a bit more context, you can see the blog I wrote on this [1]. Thanks, Kyle [1] http://www.siliconloons.com/posts/2015-02-26-scaling-openstack-neutron-development/ Regards, > Daniel > -- > |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ > :| > |: http://libvirt.org -o- http://virt-manager.org > :| > |: http://autobuild.org -o- http://search.cpan.org/~danberr/ > :| > |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc > :| > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle [metrics]
On 02/26/2015 05:41 PM, Stefano Maffulli wrote: > On Thu, 2015-02-26 at 15:58 -0600, Kevin L. Mitchell wrote: >> One thing that comes to mind is that there are a lot of reviews that >> appear to have been abandoned; I just cleared several from the >> novaclient review queue (or commented on them to see if they were still >> alive). I also know of a few novaclient changes that are waiting for >> corresponding nova changes before they can be merged. Could these be >> introducing a skew factor? > > Maybe, depending on how many they are and how old are we talking about. > How much cruft is there? Maybe the fact that we don't autoabandon > anymore is a relevant factor? > > Looking at Nova time to merge (not the client, since clients are not > analyzed individually), the median is over 10 days (the mean wait is > 29). But if you look at the trends of time to way for reviewers, they've > been trending down for 3 quarters in a row (both, average and median) > while time to wait for submitter is trending up. > > http://git.openstack.org/cgit/openstack-infra/activity-board/plain/reports/2014-q4/pdf/projects/nova.pdf > > Does it make sense to purge old stuff regularly so we have a better > overview? Or maybe we should chart a distribution of age of proposed > changesets, too in order to get a better understanding of where the > outliers are? We already purge old stuff that's unmergable (no activity in > 4 weeks with either a core -2 or Jenkins -1). The last purge was about 4 weeks ago. So effectively abandoned code isn't in the system. The merge conflict detector will also mean that all patches eventually get a Jenkins -1 if they aren't maintained. So you should consider everything in the system active for some definition. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle [metrics]
On Fri, Feb 27, 2015 at 09:51:34AM +1100, Michael Still wrote: > On Fri, Feb 27, 2015 at 9:41 AM, Stefano Maffulli > wrote: > > > Does it make sense to purge old stuff regularly so we have a better > > overview? Or maybe we should chart a distribution of age of proposed > > changesets, too in order to get a better understanding of where the > > outliers are? > > Given the abandon of a review isn't binding (a proposer can easily > unabandon), I do think we should abandon more than we do now. The > problem at the moment being that its a manual process which isn't much > fun for the person doing the work. > > Another factor to consider here is that abandoned patches against bugs > make the bug look like someone is working on a fix, which probably > isn't the case. > > Nova has been trying some very specific things to try and address > these issues, and I think we're improving. Those things are: > > * specs > * priority features This increased level of process in Nova has actually made the negative effects of the 6 month cycle noticably worse on balance. If you aren't able to propose your feature in the right window of the dev cycle your chances of getting stuff merged has gone down significantly and the time before users are likely to see your feature has correspondingly gone up. Previously people could come along with simple features at the end of the cycle and we had the flexibility to be pragmmatic and review and approve them. Now we're lacking them that ability even if we have the spare review cycles to consider it. The processes adopted have merely made us more efficient at disappointing contributors earlier in the cycle. There's been no changes made that would solve the bigger problem of the fact that Nova is far too large vs the size of the core review team, so we have a ongoing major bottleneck in our development. That, bottleneck combined with the length of the 6 month cycle is an ongoing disaster for our contributors. Regards, Daniel -- |: http://berrange.com -o-http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle [metrics]
Stefano Maffulli writes: > On Thu, 2015-02-26 at 15:58 -0600, Kevin L. Mitchell wrote: >> One thing that comes to mind is that there are a lot of reviews that >> appear to have been abandoned; I just cleared several from the >> novaclient review queue (or commented on them to see if they were still >> alive). I also know of a few novaclient changes that are waiting for >> corresponding nova changes before they can be merged. Could these be >> introducing a skew factor? > > Maybe, depending on how many they are and how old are we talking about. > How much cruft is there? Maybe the fact that we don't autoabandon > anymore is a relevant factor? > > Looking at Nova time to merge (not the client, since clients are not > analyzed individually), the median is over 10 days (the mean wait is > 29). But if you look at the trends of time to way for reviewers, they've > been trending down for 3 quarters in a row (both, average and median) > while time to wait for submitter is trending up. > > http://git.openstack.org/cgit/openstack-infra/activity-board/plain/reports/2014-q4/pdf/projects/nova.pdf > > Does it make sense to purge old stuff regularly so we have a better > overview? Or maybe we should chart a distribution of age of proposed > changesets, too in order to get a better understanding of where the > outliers are? It is good to recognize the impact of this, however, I would suggest that if having open changes that are not "actively being worked" is a problem for statistics, let's change the statistics calculation. Please do not abandon the work of contributors to improve the appearance of these metrics. Instead, simply decide what criteria you think should apply and exclude those changes from your calculations. -Jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle [metrics]
On Fri, Feb 27, 2015 at 9:41 AM, Stefano Maffulli wrote: > Does it make sense to purge old stuff regularly so we have a better > overview? Or maybe we should chart a distribution of age of proposed > changesets, too in order to get a better understanding of where the > outliers are? Given the abandon of a review isn't binding (a proposer can easily unabandon), I do think we should abandon more than we do now. The problem at the moment being that its a manual process which isn't much fun for the person doing the work. Another factor to consider here is that abandoned patches against bugs make the bug look like someone is working on a fix, which probably isn't the case. Nova has been trying some very specific things to try and address these issues, and I think we're improving. Those things are: * specs * priority features * trivial patch monkeying We will continue to seek further ways to improve our throughput. Michael -- Rackspace Australia __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle [metrics]
On Thu, 2015-02-26 at 14:18 -0600, Anne Gentle wrote: > Do the features listed in the Release Notes each have appropriate > documentation? So far we just link to the specifications for nova, for > example. [1] So to me, it could be a focus on the specification > acceptance means less time/energy for the actual user-facing docs. > Great question. I have no idea of how we can correlate blueprints/specs completed and the existence of accompanying doc. The tool we have now scans git repositories and launchpad (will soon scan storyboard, too): what data is in there that this tool can use to guess documentation coverage? Happy to brainstorm this with you and Bitergia's Jesus and Daniel as they may have ideas. /stef __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle [metrics]
On Thu, 2015-02-26 at 15:58 -0600, Kevin L. Mitchell wrote: > One thing that comes to mind is that there are a lot of reviews that > appear to have been abandoned; I just cleared several from the > novaclient review queue (or commented on them to see if they were still > alive). I also know of a few novaclient changes that are waiting for > corresponding nova changes before they can be merged. Could these be > introducing a skew factor? Maybe, depending on how many they are and how old are we talking about. How much cruft is there? Maybe the fact that we don't autoabandon anymore is a relevant factor? Looking at Nova time to merge (not the client, since clients are not analyzed individually), the median is over 10 days (the mean wait is 29). But if you look at the trends of time to way for reviewers, they've been trending down for 3 quarters in a row (both, average and median) while time to wait for submitter is trending up. http://git.openstack.org/cgit/openstack-infra/activity-board/plain/reports/2014-q4/pdf/projects/nova.pdf Does it make sense to purge old stuff regularly so we have a better overview? Or maybe we should chart a distribution of age of proposed changesets, too in order to get a better understanding of where the outliers are? /stef __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle [metrics]
On Thu, 2015-02-26 at 11:45 -0800, Stefano Maffulli wrote: > The interesting bit of those charts is that overall for OpenStack > projects, it seems that the reviews (comments to patchsets) are arriving > quite quickly but the new patchsets take a lot more to be submitted. > > Too much debating and commenting over each patch? Or are the > authors/owners of the changeset slow to respond with new patches? I > don't have an answer. I'd be happy to look at the data with other > people. One thing that comes to mind is that there are a lot of reviews that appear to have been abandoned; I just cleared several from the novaclient review queue (or commented on them to see if they were still alive). I also know of a few novaclient changes that are waiting for corresponding nova changes before they can be merged. Could these be introducing a skew factor? -- Kevin L. Mitchell Rackspace __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle [metrics]
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 02/26/2015 01:45 PM, Stefano Maffulli wrote: > The interesting bit of those charts is that overall for OpenStack > projects, it seems that the reviews (comments to patchsets) are arriving > quite quickly but the new patchsets take a lot more to be submitted. > > Too much debating and commenting over each patch? Or are the > authors/owners of the changeset slow to respond with new patches? I > don't have an answer. I'd be happy to look at the data with other > people. I would think that the expansion of the spec requirements is a major contributor to this trend. Much of the debate that used to happen with patch sets now happens on the spec, which delays the first submission of a patch, and shortens the time for review, since reviewers are already familiar with the changes being proposed. - -- Ed Leafe -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.14 (GNU/Linux) iQIcBAEBAgAGBQJU74gVAAoJEKMgtcocwZqLKqMP/iW63bOpu64iCV+cTwwUbO1f IK2Jou+W8zFXjKaK2Q6q0sU5nN0RlgJOXVuKxdZk1VQZknkUm3S53/JxCJPWEanp RLavJZUNnWmmxYoFcmSIx3gCi1idunEZUSHIKy/QQ1P9BCiej1szSskqP2BxTxBo tqLCCVtRKcmH+U4zvjyuNhJhRoZiPyOiPtxg1vS9VqEK3hyxTtt4QPI0TfxL3cGC yZPhE15R/rnWErgJx4mD9Orz8eUlUrqctDQxd/+XXobj9WiyUxjqBtW3/vOVJTcU AtwgLvsiXKmHPBGXST09Ow3Ab5wWjxq3GvAXTKLtiu7xjkWuKK2KuLxJeS3r0yOs HC6QdoZO6tytbhLRFEwIn98rfm63i5wWuL0GBRPxBW+HOnEcBm87L1Se8gPl9paD clT4OIwabzoZNyyIeTaSWukxEeVA9EEhssyctlFFQlqoRRmyWdfUwC8I/v/z0UF8 q4T7XpRyuCVuqEO23LSRW5U74sfiaffJ3YhMNhVUfY9ECOSRpzMHt6GJbvnvzS8c ug4VbNl7y06BXpPznukD42S7E2+uUjkbxPViN3BLDfmrKPb8G236fQY+6I+YTeyP rWl6ZW0ArZOrYNcNyZ3zBwV3cjvJcxQdBkJwr4zkq4jg/fCcrNeYZKjxJbWYh3tz HTBhA6Eaoim9n7bjba4M =6Psn -END PGP SIGNATURE- __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle [metrics]
On Thu, Feb 26, 2015 at 1:45 PM, Stefano Maffulli wrote: > On Wed, 2015-02-25 at 21:15 +, Ian Cordasco wrote: > > I read it the same was as Doug. I don’t think Jeremy was trying to > > imply your reviews would move through more quickly if you reviewed > > other people’s work. Just that, as with most open source projects, > > there’s always at least 2 distinct groups: people who push code more > > often and people who review code more often. > > this conversation reminded me that the median time to merge new code has > been increasing every quarter in the past year, but dropped for the > first time during last quarter (table and chart on page 23 of Q4 > quarterly report [1]). The mean number of iterations (patchsets) per > proposed change has also decreased for the first time in Q4 2014. > > The interesting bit of those charts is that overall for OpenStack > projects, it seems that the reviews (comments to patchsets) are arriving > quite quickly but the new patchsets take a lot more to be submitted. > > Too much debating and commenting over each patch? Or are the > authors/owners of the changeset slow to respond with new patches? I > don't have an answer. I'd be happy to look at the data with other > people. > > I think more analysis is needed before we can identify and remove the > problem. > > /Stef > > [1] > > http://git.openstack.org/cgit/openstack-infra/activity-board/plain/reports/2014-q4/pdf/2014-q4_OpenStack_report.pdf > The analysis doesn't count the *-specs repositories, only code and docs. > > Thanks for the analysis Stef. One additional analysis I would like to see is this: Do the features listed in the Release Notes each have appropriate documentation? So far we just link to the specifications for nova, for example. [1] So to me, it could be a focus on the specification acceptance means less time/energy for the actual user-facing docs. What could I do to analyze and correlate the feature completeness to doc completeness? A desire to release docs with code is great but we don't seem to be doing so in the way that I define docs being done. I've worked with Agile teams in the past, and you have to put your definition of done for docs in with the code. Often times, it's "release notes only" in the definition of done. So I think we're working at that level of "done" for docs, instead of "end user docs complete" or "API docs complete" or "config docs complete" or "administration documented" as the marker of complete for release. We've seen with the python-clients that the docs are very much complained about. The release cadence doesn't really give docs a chance to do anything but scrape the help text to automate a collection of information about each command in the CLI Reference. [2] That's one solution but one that serves speed rather than quality. Users tell me they'd prefer to have commands for use cases and scenarios in the docs, which is closer to our End User Guide. [3] But even the End User Guide could be improved by showing examples of what's returned for each command. We've only got about 10% coverage there. We're doing everything we can to make it easier to contribute to docs by migrating to RST,[4] so let's please consider an energy transfer from specs acceptance to end-user docs, especially during feature freeze time. Kept meaning to note the difficulties for docs on the other thread, but Stef's email reminds me we need more analysis as well. Thanks, Anne 1. https://wiki.openstack.org/wiki/ReleaseNotes/Juno#OpenStack_Compute_.28Nova.29 2. http://docs.openstack.org/cli-reference/content/ 3. http://docs.openstack.org/user-guide/content/ 4. http://justwriteclick.com/2015/02/23/state-of-the-migration-to-sphinxrst/ > PS The analysis of the individual projects are in their own pdf > > http://git.openstack.org/cgit/openstack-infra/activity-board/tree/reports/2014-q4/pdf/projects > > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Anne Gentle annegen...@justwriteclick.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle [metrics]
On Wed, 2015-02-25 at 21:15 +, Ian Cordasco wrote: > I read it the same was as Doug. I don’t think Jeremy was trying to > imply your reviews would move through more quickly if you reviewed > other people’s work. Just that, as with most open source projects, > there’s always at least 2 distinct groups: people who push code more > often and people who review code more often. this conversation reminded me that the median time to merge new code has been increasing every quarter in the past year, but dropped for the first time during last quarter (table and chart on page 23 of Q4 quarterly report [1]). The mean number of iterations (patchsets) per proposed change has also decreased for the first time in Q4 2014. The interesting bit of those charts is that overall for OpenStack projects, it seems that the reviews (comments to patchsets) are arriving quite quickly but the new patchsets take a lot more to be submitted. Too much debating and commenting over each patch? Or are the authors/owners of the changeset slow to respond with new patches? I don't have an answer. I'd be happy to look at the data with other people. I think more analysis is needed before we can identify and remove the problem. /Stef [1] http://git.openstack.org/cgit/openstack-infra/activity-board/plain/reports/2014-q4/pdf/2014-q4_OpenStack_report.pdf The analysis doesn't count the *-specs repositories, only code and docs. PS The analysis of the individual projects are in their own pdf http://git.openstack.org/cgit/openstack-infra/activity-board/tree/reports/2014-q4/pdf/projects __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle
On 2/25/15, 14:41, "Doug Hellmann" wrote: > > >On Wed, Feb 25, 2015, at 12:35 PM, Johannes Erdfelt wrote: >> On Tue, Feb 24, 2015, Jeremy Stanley wrote: >> > On 2015-02-24 10:00:51 -0800 (-0800), Johannes Erdfelt wrote: >> > [...] >> > > Recently, I have spent a lot more time waiting on reviews than I >> > > have spent writing the actual code. >> > >> > That's awesome, assuming what you mean here is that you've spent >> > more time reviewing submitted code than writing more. That's where >> > we're all falling down as a project and should be doing better, so I >> > applaud your efforts in this area. >> >> I think I understand what you're trying to do here, but to be clear, are >> you saying that I only have myself to blame for how long it takes to >> get code merged nowadays? > >I read that as a reminder that we are all collaborators, and that >working together is more effective and less frustrating than not working >together. So while you wait, look at some other contributions and >provide feedback. Others will do the same for your patches. Reviewed >patches improve and land faster. We all win. > >Doug I read it the same was as Doug. I don’t think Jeremy was trying to imply your reviews would move through more quickly if you reviewed other people’s work. Just that, as with most open source projects, there’s always at least 2 distinct groups: people who push code more often and people who review code more often. I think Jeremy took your comments to mean that you were reviewing code more often than you were pushing it and was thanking you for helping review outstanding changes. Reviews in general are hard to come by on some projects, really good reviews even harder. All reviews help make the project better. Cheers, Ian __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle
On Wed, Feb 25, 2015, at 12:35 PM, Johannes Erdfelt wrote: > On Tue, Feb 24, 2015, Jeremy Stanley wrote: > > On 2015-02-24 10:00:51 -0800 (-0800), Johannes Erdfelt wrote: > > [...] > > > Recently, I have spent a lot more time waiting on reviews than I > > > have spent writing the actual code. > > > > That's awesome, assuming what you mean here is that you've spent > > more time reviewing submitted code than writing more. That's where > > we're all falling down as a project and should be doing better, so I > > applaud your efforts in this area. > > I think I understand what you're trying to do here, but to be clear, are > you saying that I only have myself to blame for how long it takes to > get code merged nowadays? I read that as a reminder that we are all collaborators, and that working together is more effective and less frustrating than not working together. So while you wait, look at some other contributions and provide feedback. Others will do the same for your patches. Reviewed patches improve and land faster. We all win. Doug > > JE > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: > openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle
On Tue, Feb 24, 2015, Jeremy Stanley wrote: > On 2015-02-24 10:00:51 -0800 (-0800), Johannes Erdfelt wrote: > [...] > > Recently, I have spent a lot more time waiting on reviews than I > > have spent writing the actual code. > > That's awesome, assuming what you mean here is that you've spent > more time reviewing submitted code than writing more. That's where > we're all falling down as a project and should be doing better, so I > applaud your efforts in this area. I think I understand what you're trying to do here, but to be clear, are you saying that I only have myself to blame for how long it takes to get code merged nowadays? JE __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle
On 2015-02-24 22:06:32 -0600 (-0600), Chris Friesen wrote: [...] > Taking it as a dig, sometimes people just have their one itch to > scratch and the rest of the project is "good enough" for them, and > I think that's actually okay. Not intended as a dig at anyone... just a subtle reminder that there are other things that can be done besides "waiting for someone else to review contributions," and some of those things can even actively improve the situation. Otherwise it's sort of like going on vacation and complaining there are too many tourists. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle
On 02/24/2015 05:46 PM, Jeremy Stanley wrote: On 2015-02-24 10:00:51 -0800 (-0800), Johannes Erdfelt wrote: [...] Recently, I have spent a lot more time waiting on reviews than I have spent writing the actual code. That's awesome, assuming what you mean here is that you've spent more time reviewing submitted code than writing more. That's where we're all falling down as a project and should be doing better, so I applaud your efforts in this area. I can't tell if your statement should be taken at face value, or is a subtle dig indicating that everyone should be reviewing other people's code rather than writing more of their own. Taking it at face value, I interpret him to mean that he has some features that he would like to submit, and they're blocked waiting for core reviewers. I've gone through that myself--it can take a long time to get meaningful feedback, much longer than I've experienced in other open-source projects. Taking it as a dig, sometimes people just have their one itch to scratch and the rest of the project is "good enough" for them, and I think that's actually okay. Chris __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle
On 2015-02-24 10:00:51 -0800 (-0800), Johannes Erdfelt wrote: [...] > Recently, I have spent a lot more time waiting on reviews than I > have spent writing the actual code. That's awesome, assuming what you mean here is that you've spent more time reviewing submitted code than writing more. That's where we're all falling down as a project and should be doing better, so I applaud your efforts in this area. -- Jeremy Stanley __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle
On 23/02/15 19:14, Joe Gordon wrote: Was: http://lists.openstack.org/pipermail/openstack-dev/2015-February/057578.html There has been frustration with our current 6 month development cadence. This is an attempt to explain those frustrations and propose a very rough outline of a possible alternative. Currently we follow a 6 month release cadence, with 3 intermediate milestones (6 weeks apart), as explained here: https://wiki.openstack.org/wiki/Kilo_Release_Schedule Current issues * 3 weeks of feature freeze for all projects at the end of each cycle (3 out of the 26 feature blocked) * 3 weeks of release candidates. Once a candidate is cut development is open for next release. While this is good in theory, not much work actually starts on next release. * some projects have non priority feature freezes and at Milestone 2 (so 9 out of 26 weeks restricted in those projects) * vicious development circle o vicious circle: + big push right to land lots of features right before the release + a lot of effort is spent getting the release ready + after the release people are a little burnt out and take it easy until the next summit + Blueprints have to be re-discussed and re-approved for the next cycle I'm sure there's a good reason for this one that I'm not aware of, but it certainly sounds a lot like declaring a war of attrition against a project's own contributors. If core teams think that a design that they previously approved needs to change because of other changes _they_ approved in the previous release cycle or a decision _they_ made during the design summit, the onus should be on _them_ to actively make the change (even if that's as simple as proposing a patch deleting *a* previously-approved spec - not the whole lot en masse). To delete everything by default and force the contributor to get it reapproved is tantamount to handing them a shovel and forcing them to shift the goalposts themselves. It's flirting with the line between poor policy and outright evil. + makes it hard to land blueprints early in the cycle casing the bug rush at the end of the cycle, see step 1 o Makes it hard to plan things across a release o This actually destabilizes things right as we go into the stabilization period (We actually have great data on this too) o Means postponing blueprints that miss a deadline several months I'm yet to be convinced by the solution offered here, but I do think this is a fairly accurate description of the problem. I always tell folks that if you want big features to land in the next release, you have to start working on them as soon as you can finish up on any release blockers and well before Summit. Mostly that month feels like dead time, though. Sometimes I daydream about what it would be like if we had the design summit a few weeks _before_ the release instead of after. Usually I snap out of it when I remember the downsides, like having developers under pressure to fix bugs at the same time as the design summit, or not having a shiny new release to trumpet at the conference, or everyone getting excited about new feature discussions and not working on urgent bugs when they get back. Still, I keep thinking about it... cheers, Zane. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle
On Feb 24, 2015, at 1:50 PM, Joe Gordon wrote: >> I think the release candidate >> period is the only thing that makes your code drops actually usable. >> It's the only moment in the cycle where integrators test. It's the only >> moment in the cycle where developers work on bugs they did not file >> themselves, but focus on a project-wide priority list of release >> blockers. If you remove that period, then nobody will ever work on >> release blockers that do not directly affect them. Shorten that period >> to one week, and no integrator will have the time to run a proper QA >> program to detect those release blockers. > > I still think the 3 week RC candidate cycle needs to happen, the difference > is it would be done by stable maintenance. And I agree, the RC candidate > period is one of the few moments where developers work on bugs they did not > file themselves. So I am not sure how this would actually work. Perhaps the > answer is we have deeper issues if we don't want to fix bugs until the last > minute. I like the notion that there isn't an overall release that all development is tied to, but that at some more or less regular interval, a new stable "release" is cut, and intense integration testing is done on that, with bug fixes as needed. But how to get developers who are intent on coding their new features to hold off and work on fixing bugs identified by the stable testing is a big question mark to me. -- Ed Leafe signature.asc Description: Message signed with OpenPGP using GPGMail __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle
On Tue, Feb 24, 2015 at 2:59 AM, Daniel P. Berrange wrote: > On Mon, Feb 23, 2015 at 04:14:36PM -0800, Joe Gordon wrote: > > Was: > > > http://lists.openstack.org/pipermail/openstack-dev/2015-February/057578.html > > > > There has been frustration with our current 6 month development cadence. > > This is an attempt to explain those frustrations and propose a very rough > > outline of a possible alternative. > > > > > > Currently we follow a 6 month release cadence, with 3 intermediate > > milestones (6 weeks apart), as explained here: > > https://wiki.openstack.org/wiki/Kilo_Release_Schedule > > > > > > Current issues > > > >- 3 weeks of feature freeze for all projects at the end of each cycle > (3 > >out of the 26 feature blocked) > >- 3 weeks of release candidates. Once a candidate is cut development > is > >open for next release. While this is good in theory, not much work > actually > >starts on next release. > >- some projects have non priority feature freezes and at Milestone 2 > (so > >9 out of 26 weeks restricted in those projects) > >- vicious development circle > > - vicious circle: > > - big push right to land lots of features right before the > release > > - a lot of effort is spent getting the release ready > > - after the release people are a little burnt out and take it > easy > > until the next summit > > - Blueprints have to be re-discussed and re-approved for the > next > > cycle > > - makes it hard to land blueprints early in the cycle casing the > > bug rush at the end of the cycle, see step 1 > > - Makes it hard to plan things across a release > > - This actually destabilizes things right as we go into the > > stabilization period (We actually have great data on this too) > > - Means postponing blueprints that miss a deadline several months > > > > > > Requirements for a new model > > > >- Keep 6 month release cadence. Not everyone is willing to deploy from > >trunk > >- Keep stable branches for at least 6 months. In order to test > upgrades > >from the last stable branch, we need a stable branch to test against > >- Keep supporting continuous deployment. Some people really want to > >deploy from trunk > > > > > > What We can drop > > > >- While we need to keep releasing a stable branch every 6 months, we > >don't have to do all of our development planning around it. We can > treat it > >as just another milestone > > > > > > I think a lot of the frustration with our current cadence comes out of > the > > big stop everything (development, planning etc.), and stabilize the > release > > process. Which in turn isn't really making things more stable. So I > propose > > we keep the 6 month release cycle, but change the development cycle from > a > > 6 month one with 3 intermediate milestones to a 6 week one with a > milestone > > at the end of it. > > You're solving some issues around developer experiance by letting > developers > iterate on a faster cycle which is something I agree with, but by keeping > the 6 month release cycle I think you're missing the bigger opportunity > here. > Namely, the chance to get the features to the users faster, which is > ultimtely > the reason why contributors currently push us so hard towards the end of > the > release. I think we have to be more ambitious here and actually make the > release > cycle itself faster, putting it on a 2 month cycle. More detail about why > I think > this is needed is here: > > > http://lists.openstack.org/pipermail/openstack-dev/2015-February/057614.html > > Nothing like having to concurrent threads on the same thing with very similar subjects. [openstack-dev] [all] Re-evaluating the suitability of the 6 month release cycle vs [openstack-dev][stable][all] Revisiting the 6 month release cycle I'll respond to your proposal, on the other thread. > > What this actually means: > > > >- Stop approving blueprints for specific stable releases, instead just > >approve them and target them to milestones. > > - Milestones stop becoming Kilo-1, Kilo-2, Kilo-3 etc. and just > > become 1,2,3,4,5,6,7,8,9 etc. > > - If something misses what was previously known as Kilo-3 it has to > > wait a week for what for milestone 4. > >- Development focuses on milestones only. So 6 week cyc
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle
On Tue, Feb 24, 2015 at 10:00 AM, Johannes Erdfelt wrote: > On Tue, Feb 24, 2015, Thierry Carrez wrote: > > Agree on the pain of maintaining milestone plans though, which is why I > > propose we get rid of most of it in Liberty. That will actually be > > discussed at the cross-project meeting today: > > > > > https://wiki.openstack.org/wiki/Release_Cycle_Management/Liberty_Tracking > > I'm happy to see this. > ++ > > "Assignees may target their blueprint to a future milestone, as an > indication of when they intend to land it (not mandatory)" > > That seems useless to me. I have no control over when things land. I can > only control when my code is put up for review. > > Recently, I have spent a lot more time waiting on reviews than I have > spent writing the actual code. > I think this is a symptom of a much deeper problem. And adjusting the release cadence won't make a big impact on this. > > JE > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle
On Tue, Feb 24, 2015 at 2:38 AM, Thierry Carrez wrote: > Joe Gordon wrote: > > [...] > > I think a lot of the frustration with our current cadence comes out of > > the big stop everything (development, planning etc.), and stabilize the > > release process. Which in turn isn't really making things more stable. > > I guess I have a different position. I think the release candidate > period is the only thing that makes your code drops actually usable. > It's the only moment in the cycle where integrators test. It's the only > moment in the cycle where developers work on bugs they did not file > themselves, but focus on a project-wide priority list of release > blockers. If you remove that period, then nobody will ever work on > release blockers that do not directly affect them. Shorten that period > to one week, and no integrator will have the time to run a proper QA > program to detect those release blockers. > I still think the 3 week RC candidate cycle needs to happen, the difference is it would be done by stable maintenance. And I agree, the RC candidate period is one of the few moments where developers work on bugs they did not file themselves. So I am not sure how this would actually work. Perhaps the answer is we have deeper issues if we don't want to fix bugs until the last minute. > > I understand that from your developer perspective it's annoying to have > to work on project-wide priorities rather than your own, and therefore > you'd like to get rid of those -- but the resulting drop in quality is > just something we can't afford. > > > So I propose we keep the 6 month release cycle, but change the > > development cycle from a 6 month one with 3 intermediate milestones to a > > 6 week one with a milestone at the end of it. > > > > What this actually means: > > > > * Stop approving blueprints for specific stable releases, instead just > > approve them and target them to milestones. > > o Milestones stop becoming Kilo-1, Kilo-2, Kilo-3 etc. and just > > become 1,2,3,4,5,6,7,8,9 etc. > > o If something misses what was previously known as Kilo-3 it has > > to wait a week for what for milestone 4. > > * Development focuses on milestones only. So 6 week cycle with say 1 > > week of stabilization, finish things up before each milestone > > * Process of cutting a stable branch (planning, making the branch, > > doing release candidates, testing etc.) is done by a dedicated > > stable branch team. And it should be done based on a specific > milestone > > * Goal: Change the default development planning mode to ignore stable > > branches, and allow for us to think of things in terms of the number > > of milestones needed, not will it make the stable branch or not > > I don't think that would solve any of the issues you mentioned: > > Current issues > > * 3 weeks of feature freeze for all projects at the end of each cycle > > (3 out of the 26 feature blocked) > > So you'll have 3 x 1 week of feature freeze for all projects, instead of > 1 x 3 weeks. That will be less efficient (integrators need a >1week > feature freeze period to actually start testing a non-moving target), > more painful (have to organize it 3 times instead of 1), and likely > inefficient (takes generally more than one week to find critical bugs, > develop the fix, and get it reviewed). And in the end, it's still 3 out > of the 26 feature blocked. > As said before, I don't envision integrator consuming every milestone. Just the standard 3 week RC cycle for stable branch candidates. > > > * 3 weeks of release candidates. Once a candidate is cut development > > is open for next release. While this is good in theory, not much > > work actually starts on next release. > > That is not really what I observe. People start landing their feature in > the master branch starting the day after RC1. I actually observe the > opposite: too many people switching to master development, and not > enough people working on RC2+ bugs. > Unfortunately I think we are both right. To many people move on and don't work on RC2 bugs, but development still slows down. > > > * some projects have non priority feature freezes and at Milestone 2 > > (so 9 out of 26 weeks restricted in those projects) > > That was their own choice. I for one was really surprised that they > would freeze earlier -- I think 3 weeks is the right balance. > > > * vicious development circle > > o vicious circle: > > + big push right to land lots of features right before the > release > > I think you'll have exactly the same push before the "stable" milestone > (or the one that will be adopted by $DISTRO). > I am hoping the push would be smaller, but I don't think we can remove it completely. > > >> + a lot of effort is spent getting the release ready > >> + after the release people are a little burnt out and take it > >> easy until the next summit > > No
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle
On Tue, Feb 24, 2015, Thierry Carrez wrote: > Agree on the pain of maintaining milestone plans though, which is why I > propose we get rid of most of it in Liberty. That will actually be > discussed at the cross-project meeting today: > > https://wiki.openstack.org/wiki/Release_Cycle_Management/Liberty_Tracking I'm happy to see this. "Assignees may target their blueprint to a future milestone, as an indication of when they intend to land it (not mandatory)" That seems useless to me. I have no control over when things land. I can only control when my code is put up for review. Recently, I have spent a lot more time waiting on reviews than I have spent writing the actual code. JE __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle
Johannes Erdfelt wrote: > On Mon, Feb 23, 2015, Joe Gordon wrote: >> What this actually means: >> >>- Stop approving blueprints for specific stable releases, instead just >>approve them and target them to milestones. >> - Milestones stop becoming Kilo-1, Kilo-2, Kilo-3 etc. and just >> become 1,2,3,4,5,6,7,8,9 etc. >> - If something misses what was previously known as Kilo-3 it has to >> wait a week for what for milestone 4. >>- Development focuses on milestones only. So 6 week cycle with say 1 >>week of stabilization, finish things up before each milestone > > What is the motiviation for having milestones at all? > > At least in the Nova world, it seems like milestones mean nothing at > all. It's just something John Garbutt spends a lot of his time updating > that doesn't appear to provide any value to anyone. It has *some* value in cross-project coordination. It's a way to describe common points in time. Saying "it should be done by kilo-1" is easier than using random dates that vary across projects. Another value is to exercise the release automation more regularly. Agree on the pain of maintaining milestone plans though, which is why I propose we get rid of most of it in Liberty. That will actually be discussed at the cross-project meeting today: https://wiki.openstack.org/wiki/Release_Cycle_Management/Liberty_Tracking Cheers, -- Thierry Carrez (ttx) __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle
On Tue, Feb 24, 2015, Thierry Carrez wrote: > Joe Gordon wrote: > > [...] > > I think a lot of the frustration with our current cadence comes out of > > the big stop everything (development, planning etc.), and stabilize the > > release process. Which in turn isn't really making things more stable. > > I guess I have a different position. I think the release candidate > period is the only thing that makes your code drops actually usable. > It's the only moment in the cycle where integrators test. It's the only > moment in the cycle where developers work on bugs they did not file > themselves, but focus on a project-wide priority list of release > blockers. If you remove that period, then nobody will ever work on > release blockers that do not directly affect them. Shorten that period > to one week, and no integrator will have the time to run a proper QA > program to detect those release blockers. > > I understand that from your developer perspective it's annoying to have > to work on project-wide priorities rather than your own, and therefore > you'd like to get rid of those -- but the resulting drop in quality is > just something we can't afford. Speaking as an operator that deploys from trunk, code quality isn't our biggest problem right now. One of our biggest problems is merging our local branch with upstream. There are a handful of contributing factors to this, but a major part is the patches we need to carry locally until they are merged upstream. If we can get patches/features we develop merged upstream faster, that would be the biggest help for us. JE __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle
On Mon, Feb 23, 2015, Joe Gordon wrote: > What this actually means: > >- Stop approving blueprints for specific stable releases, instead just >approve them and target them to milestones. > - Milestones stop becoming Kilo-1, Kilo-2, Kilo-3 etc. and just > become 1,2,3,4,5,6,7,8,9 etc. > - If something misses what was previously known as Kilo-3 it has to > wait a week for what for milestone 4. >- Development focuses on milestones only. So 6 week cycle with say 1 >week of stabilization, finish things up before each milestone What is the motiviation for having milestones at all? At least in the Nova world, it seems like milestones mean nothing at all. It's just something John Garbutt spends a lot of his time updating that doesn't appear to provide any value to anyone. JE __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [stable][all] Revisiting the 6 month release cycle
On Mon, Feb 23, 2015 at 04:14:36PM -0800, Joe Gordon wrote: > Was: > http://lists.openstack.org/pipermail/openstack-dev/2015-February/057578.html > > There has been frustration with our current 6 month development cadence. > This is an attempt to explain those frustrations and propose a very rough > outline of a possible alternative. > > > Currently we follow a 6 month release cadence, with 3 intermediate > milestones (6 weeks apart), as explained here: > https://wiki.openstack.org/wiki/Kilo_Release_Schedule > > > Current issues > >- 3 weeks of feature freeze for all projects at the end of each cycle (3 >out of the 26 feature blocked) >- 3 weeks of release candidates. Once a candidate is cut development is >open for next release. While this is good in theory, not much work actually >starts on next release. >- some projects have non priority feature freezes and at Milestone 2 (so >9 out of 26 weeks restricted in those projects) >- vicious development circle > - vicious circle: > - big push right to land lots of features right before the release > - a lot of effort is spent getting the release ready > - after the release people are a little burnt out and take it easy > until the next summit > - Blueprints have to be re-discussed and re-approved for the next > cycle > - makes it hard to land blueprints early in the cycle casing the > bug rush at the end of the cycle, see step 1 > - Makes it hard to plan things across a release > - This actually destabilizes things right as we go into the > stabilization period (We actually have great data on this too) > - Means postponing blueprints that miss a deadline several months > > > Requirements for a new model > >- Keep 6 month release cadence. Not everyone is willing to deploy from >trunk >- Keep stable branches for at least 6 months. In order to test upgrades >from the last stable branch, we need a stable branch to test against >- Keep supporting continuous deployment. Some people really want to >deploy from trunk > > > What We can drop > >- While we need to keep releasing a stable branch every 6 months, we >don't have to do all of our development planning around it. We can treat it >as just another milestone > > > I think a lot of the frustration with our current cadence comes out of the > big stop everything (development, planning etc.), and stabilize the release > process. Which in turn isn't really making things more stable. So I propose > we keep the 6 month release cycle, but change the development cycle from a > 6 month one with 3 intermediate milestones to a 6 week one with a milestone > at the end of it. You're solving some issues around developer experiance by letting developers iterate on a faster cycle which is something I agree with, but by keeping the 6 month release cycle I think you're missing the bigger opportunity here. Namely, the chance to get the features to the users faster, which is ultimtely the reason why contributors currently push us so hard towards the end of the release. I think we have to be more ambitious here and actually make the release cycle itself faster, putting it on a 2 month cycle. More detail about why I think this is needed is here: http://lists.openstack.org/pipermail/openstack-dev/2015-February/057614.html > What this actually means: > >- Stop approving blueprints for specific stable releases, instead just >approve them and target them to milestones. > - Milestones stop becoming Kilo-1, Kilo-2, Kilo-3 etc. and just > become 1,2,3,4,5,6,7,8,9 etc. > - If something misses what was previously known as Kilo-3 it has to > wait a week for what for milestone 4. >- Development focuses on milestones only. So 6 week cycle with say 1 >week of stabilization, finish things up before each milestone >- Process of cutting a stable branch (planning, making the branch, doing >release candidates, testing etc.) is done by a dedicated stable branch >team. And it should be done based on a specific milestone >- Goal: Change the default development planning mode to ignore stable >branches, and allow for us to think of things in terms of the number of >milestones needed, not will it make the stable branch or not > > > Gotchas, questions etc: > >- Some developers will still care about getting a feature into a >specific stable release, so we may still get a small rush for the milestone >before each stable branch >- Requires significantly more work from the stable maintenance team I think we can increase the release cycle to 2 months without impacting the stable team to any great extent. We simply don't have to provide stabel branches for every single release - compare with Linux, only a subset of major releases get stable branches & that generally works pretty well. >- Nami
[openstack-dev] [stable][all] Revisiting the 6 month release cycle
Was: http://lists.openstack.org/pipermail/openstack-dev/2015-February/057578.html There has been frustration with our current 6 month development cadence. This is an attempt to explain those frustrations and propose a very rough outline of a possible alternative. Currently we follow a 6 month release cadence, with 3 intermediate milestones (6 weeks apart), as explained here: https://wiki.openstack.org/wiki/Kilo_Release_Schedule Current issues - 3 weeks of feature freeze for all projects at the end of each cycle (3 out of the 26 feature blocked) - 3 weeks of release candidates. Once a candidate is cut development is open for next release. While this is good in theory, not much work actually starts on next release. - some projects have non priority feature freezes and at Milestone 2 (so 9 out of 26 weeks restricted in those projects) - vicious development circle - vicious circle: - big push right to land lots of features right before the release - a lot of effort is spent getting the release ready - after the release people are a little burnt out and take it easy until the next summit - Blueprints have to be re-discussed and re-approved for the next cycle - makes it hard to land blueprints early in the cycle casing the bug rush at the end of the cycle, see step 1 - Makes it hard to plan things across a release - This actually destabilizes things right as we go into the stabilization period (We actually have great data on this too) - Means postponing blueprints that miss a deadline several months Requirements for a new model - Keep 6 month release cadence. Not everyone is willing to deploy from trunk - Keep stable branches for at least 6 months. In order to test upgrades from the last stable branch, we need a stable branch to test against - Keep supporting continuous deployment. Some people really want to deploy from trunk What We can drop - While we need to keep releasing a stable branch every 6 months, we don't have to do all of our development planning around it. We can treat it as just another milestone I think a lot of the frustration with our current cadence comes out of the big stop everything (development, planning etc.), and stabilize the release process. Which in turn isn't really making things more stable. So I propose we keep the 6 month release cycle, but change the development cycle from a 6 month one with 3 intermediate milestones to a 6 week one with a milestone at the end of it. What this actually means: - Stop approving blueprints for specific stable releases, instead just approve them and target them to milestones. - Milestones stop becoming Kilo-1, Kilo-2, Kilo-3 etc. and just become 1,2,3,4,5,6,7,8,9 etc. - If something misses what was previously known as Kilo-3 it has to wait a week for what for milestone 4. - Development focuses on milestones only. So 6 week cycle with say 1 week of stabilization, finish things up before each milestone - Process of cutting a stable branch (planning, making the branch, doing release candidates, testing etc.) is done by a dedicated stable branch team. And it should be done based on a specific milestone - Goal: Change the default development planning mode to ignore stable branches, and allow for us to think of things in terms of the number of milestones needed, not will it make the stable branch or not Gotchas, questions etc: - Some developers will still care about getting a feature into a specific stable release, so we may still get a small rush for the milestone before each stable branch - Requires significantly more work from the stable maintenance team - Naming the stable branches becomes less fun, as we refer to the stable branches less - Since planning is continuous 6 month cycle for summits becomes a little strange. This is still an open ended question to me. Anyway, that has been rattling around my head since Paris and I needed to write it down somewhere. Thanks for reading. Thoughts, comments, concerns welcome. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
