Re: [openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree
Hi Aaron: There is a certain fear of another cascading chain of emails so with hesitation i send this email out. :-) 1) I could not agree with u more on the issue with the logs and the pain with debugging issues here. Yes for sure bugs do keep popping up but often times, (speaking for fwaas) - given the L3 agent interactions - there are a multitude of reasons for a failure. An L3Agent crash or a router issue - also manifests itself as an fwaas issue - i think ur first paste is along those lines (perhaps i could be wrong without much context here). The L3Agent - service coexistence is far from ideal - but this experience has lead to two proposals - a vendor proposal [1] that actually tries to address such agent limitations and collaboration on another community proposal[2] to enable the L3 Agent to be more suited to hosting services. Hopefully [2] will get picked up in K and should help provide the necessary infrastructure to clean up the reference implementation. 2) Regarding ur point on the FWaaS API - the intent of the feature by design was to keep the service abstraction separate from how it is inserted in the network - to keep this vendor/technology neutral. The first priority, post Havana was to address service insertion to get away from the all routers model [3] but did not get the blessings needed. Now with a redrafted proposal for Juno[4] again an effort is being made to address this now for the second time in the 2 releases post H. In general, I would make a request that before we decide to go ahead and start moving things out into an incubator area - more discussion is needed. We don’t want to land up in a situation in K-3 where we find out that this model does not quite work for whatever reason. Also i wonder about the hoops to get out from incubation. As vendors who try to align with the community and upstream our work - we don’t want to land up waiting more cycles - there is quite a bit of frustration that is felt here too. Lets also think about the impacts on feature velocity, somehow that keeps popping into my head every time i buy a book from a certain online retailer. :-) [1] https://review.openstack.org/#/c/90729/ [2] https://review.openstack.org/#/c/91532/ [3] https://review.openstack.org/#/c/62599/ [4] https://review.openstack.org/#/c/93128/ Thanks Sridar From: Aaron Rosen aaronoro...@gmail.commailto:aaronoro...@gmail.com Reply-To: OpenStack List openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Wednesday, August 13, 2014 at 3:56 PM To: OpenStack List openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree Hi, I've been thinking a good bit on this on the right way to move forward with this and in general the right way new services should be added. Yesterday I was working on a bug that was causing some problems in the openstack infra. We tracked down the issue then I uploaded a patch for it. A little while later after jenkins voted back a -1 so I started looking through the logs to see what the source of the failure was (which was actually unrelated to my patch). The random failure in the fwaas/vpn/l3-agent code which all outputs to the same log file that contains many traces for every run even successful ones. In one skim of this log file I was able to spot 4 [1]bugs which shows these new experimental services that we've added to neutron have underlying problems (even though they've been in the tree for 2 releases+ now). This puts a huge strain on the whole openstack development community as they are always recheck no bug'ing due to neutron failures. If you look at the fwaas work that was done. This merged over two releases ago and still does not have a complete API as there is no concept of where enforcement should be done. Today, enforcement is done across all of a tenant's routers making it more or less useless imho and we're just carrying it along in the tree (and it's causing us problems)! I think Mark's idea of neutron-incubator[2] is a great step forward to improving neutron. We can easily move these things out of the neutron source tree and we can plug these things in here: https://github.com/openstack/neutron/blob/master/etc/neutron.conf#L52 https://github.com/openstack/neutron/blob/master/etc/neutron.conf#L72 (GASP: We have seen shipped our own API's here to customers before we were able to upstream them). This allows us to decouple these experimental things from the neutron core and allows us to release these components on their own making things more modular/maintainable and stable (I think these things might even be better long term living out of the tree). Most importantly though it doesn't put a burden on everyone else. Best, Aaron [1] http://paste.openstack.org/show/94664/ http://paste.openstack.org/show/94670/ http://paste.openstack.org/show
Re: [openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree
FWaas can't seamlessly work with DVR yet. A BP [1] has been submitted, but it can only handle NS traffic, leaving W-E untouched. If we implement the WE firewall in DVR, the iptable might be applied at a per port basis, so there are some overlapping with SG (Can we image a packet run into iptable hook twice between VM and the wire, for both ingress and egress directions?). Maybe the overall service plugins (including service extension in ML2) needs some cleaning up, It seems that Neutron is just built from separate single blocks. [1] http://git.openstack.org/cgit/openstack/neutron-specs/tree/specs/juno/neutron-dvr-fwaas.rst From: Sridar Kandaswamy (skandasw) [skand...@cisco.com] Sent: Thursday, August 14, 2014 3:07 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree Hi Aaron: There is a certain fear of another cascading chain of emails so with hesitation i send this email out. :-) 1) I could not agree with u more on the issue with the logs and the pain with debugging issues here. Yes for sure bugs do keep popping up but often times, (speaking for fwaas) - given the L3 agent interactions - there are a multitude of reasons for a failure. An L3Agent crash or a router issue - also manifests itself as an fwaas issue - i think ur first paste is along those lines (perhaps i could be wrong without much context here). The L3Agent - service coexistence is far from ideal - but this experience has lead to two proposals - a vendor proposal [1] that actually tries to address such agent limitations and collaboration on another community proposal[2] to enable the L3 Agent to be more suited to hosting services. Hopefully [2] will get picked up in K and should help provide the necessary infrastructure to clean up the reference implementation. 2) Regarding ur point on the FWaaS API - the intent of the feature by design was to keep the service abstraction separate from how it is inserted in the network - to keep this vendor/technology neutral. The first priority, post Havana was to address service insertion to get away from the all routers model [3] but did not get the blessings needed. Now with a redrafted proposal for Juno[4] again an effort is being made to address this now for the second time in the 2 releases post H. In general, I would make a request that before we decide to go ahead and start moving things out into an incubator area - more discussion is needed. We don’t want to land up in a situation in K-3 where we find out that this model does not quite work for whatever reason. Also i wonder about the hoops to get out from incubation. As vendors who try to align with the community and upstream our work - we don’t want to land up waiting more cycles - there is quite a bit of frustration that is felt here too. Lets also think about the impacts on feature velocity, somehow that keeps popping into my head every time i buy a book from a certain online retailer. :-) [1] https://review.openstack.org/#/c/90729/ [2] https://review.openstack.org/#/c/91532/ [3] https://review.openstack.org/#/c/62599/ [4] https://review.openstack.org/#/c/93128/ Thanks Sridar From: Aaron Rosen aaronoro...@gmail.commailto:aaronoro...@gmail.com Reply-To: OpenStack List openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Wednesday, August 13, 2014 at 3:56 PM To: OpenStack List openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree Hi, I've been thinking a good bit on this on the right way to move forward with this and in general the right way new services should be added. Yesterday I was working on a bug that was causing some problems in the openstack infra. We tracked down the issue then I uploaded a patch for it. A little while later after jenkins voted back a -1 so I started looking through the logs to see what the source of the failure was (which was actually unrelated to my patch). The random failure in the fwaas/vpn/l3-agent code which all outputs to the same log file that contains many traces for every run even successful ones. In one skim of this log file I was able to spot 4 [1]bugs which shows these new experimental services that we've added to neutron have underlying problems (even though they've been in the tree for 2 releases+ now). This puts a huge strain on the whole openstack development community as they are always recheck no bug'ing due to neutron failures. If you look at the fwaas work that was done. This merged over two releases ago and still does not have a complete API as there is no concept of where enforcement should be done. Today, enforcement is done across all of a tenant's routers making it more or less useless imho and we're just carrying it along in the tree
Re: [openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree
Hi Sridar, Yes I know this is only for phase 1, while I'm also thinking about how it should be in next phase. At least, zone concept should be introduced, we may use it to replace SG, to eliminate potential conflicts of defining ACL in two different places. From: Sridar Kandaswamy (skandasw) [skand...@cisco.com] Sent: Thursday, August 14, 2014 10:12 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree Hi Wuhongning: Yes u are correct – this is phase 1 to at least get basic perimeter firewall support working with DVR before looking for an optimal way to address E – W traffic. Thanks Sridar From: Wuhongning wuhongn...@huawei.commailto:wuhongn...@huawei.com Reply-To: OpenStack List openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Date: Thursday, August 14, 2014 at 1:05 AM To: OpenStack List openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree FWaas can't seamlessly work with DVR yet. A BP [1] has been submitted, but it can only handle NS traffic, leaving W-E untouched. If we implement the WE firewall in DVR, the iptable might be applied at a per port basis, so there are some overlapping with SG (Can we image a packet run into iptable hook twice between VM and the wire, for both ingress and egress directions?). Maybe the overall service plugins (including service extension in ML2) needs some cleaning up, It seems that Neutron is just built from separate single blocks. [1] http://git.openstack.org/cgit/openstack/neutron-specs/tree/specs/juno/neutron-dvr-fwaas.rst ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree
Hi, I've been thinking a good bit on this on the right way to move forward with this and in general the right way new services should be added. Yesterday I was working on a bug that was causing some problems in the openstack infra. We tracked down the issue then I uploaded a patch for it. A little while later after jenkins voted back a -1 so I started looking through the logs to see what the source of the failure was (which was actually unrelated to my patch). The random failure in the fwaas/vpn/l3-agent code which all outputs to the same log file that contains many traces for every run even successful ones. In one skim of this log file I was able to spot 4 [1]bugs which shows these new experimental services that we've added to neutron have underlying problems (even though they've been in the tree for 2 releases+ now). This puts a huge strain on the whole openstack development community as they are always recheck no bug'ing due to neutron failures. If you look at the fwaas work that was done. This merged over two releases ago and still does not have a complete API as there is no concept of where enforcement should be done. Today, enforcement is done across all of a tenant's routers making it more or less useless imho and we're just carrying it along in the tree (and it's causing us problems)! I think Mark's idea of neutron-incubator[2] is a great step forward to improving neutron. We can easily move these things out of the neutron source tree and we can plug these things in here: https://github.com/openstack/neutron/blob/master/etc/neutron.conf#L52 https://github.com/openstack/neutron/blob/master/etc/neutron.conf#L72 (GASP: We have seen shipped our own API's here to customers before we were able to upstream them). This allows us to decouple these experimental things from the neutron core and allows us to release these components on their own making things more modular/maintainable and stable (I think these things might even be better long term living out of the tree). Most importantly though it doesn't put a burden on everyone else. Best, Aaron [1] http://paste.openstack.org/show/94664/ http://paste.openstack.org/show/94670/ http://paste.openstack.org/show/94663/ http://paste.openstack.org/show/94662/ [2] - https://etherpad.openstack.org/p/neutron-incubator On Mon, Aug 11, 2014 at 9:58 AM, Robert Kukura kuk...@noironetworks.com wrote: On 8/8/14, 6:28 PM, Salvatore Orlando wrote: If we want to keep everything the way it is, we have to change everything [1] This is pretty much how I felt after reading this proposal, and I felt that this quote, which Ivar will probably appreciate, was apt to the situation. Recent events have spurred a discussion about the need for a change in process. It is not uncommon indeed to believe that by fixing the process, things will inevitably change for better. While no-one argues that flaws in processes need to be fixed, no process change will ever change anything, in my opinion, unless it is aimed at spurring change in people as well. From what I understand, this proposal starts with the assumption that any new feature which is committed to Neutron (ie: has a blueprint approved), and is not a required neutron component should be considered as a preview. This is not different from the process, which, albeit more informally, has been adopted so far. Load Balancing, Firewall, VPN, have all been explicitly documented as experimental in their first release; I would argue that even if not experimental anymore, they may not be considered stable until their stability was proven by upstream QA with API and scenario tests - but this is not sanctioned anywhere currently, I think. Correct, this proposal is not so much a new process or change in process as a formalization of what we've already been doing, and a suggested adaptation to clarify the current expectations around stability of new APIs. According to this proposal, for preview features: - all the code would be moved to a preview package Yes. - Options will be marked as preview Yes. - URIs should be prefixed with preview That's what I suggested, but, as several people have pointed out, this does seem like its worth the cost of breaking the API compatibility just at the point when it is being declared stable. I'd like to withdraw this item. - CLIs will note the features are preview in their help strings Yes. - Documentation will explicitly state this feature is preview (I think we already mark them as experimental, frankly I don't think there are a lot of differences in terminology here) Yes. Again to me, failure is one likely outcome of an experiment. The term preview is intended to imply more of a commitment to quickly reach stability. - Database migrations will be in the main alembic path as usual Right. - CLI, Devstack and Heat support will be available Right, as appropriate for the feature. - Can be used by non-preview neutron
Re: [openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree
gustavo panizzo (gfa) wrote: only one think i didn't like it why all url,api, etc has to include the word 'preview'? i imagine that i would be consuming the new feature using heat, puppet, local scripts, custom horizon, whatever. Why do you make me to change all them when the feature moves out of preview? it could be a lot of rework (for consumers) without gain. I would totally support other big fat warnings everywhere (logs, documentation, startup log of neutron-server) but don't change the API if is not necessary I see two issues with this proposal: the first one is what Gustavo just said: the use of the preview package/configoption/API creates friction when the feature needs to go mainstream (package renaming, change in configuration for deployers, change in API calls for users...). My understanding is that the goal is to make it easy for people to try the preview feature, and keeping the experimental feature in-tree is seen as simpler to experiment with. But the pain from this friction imho outweighs the pain of deploying an out-of-tree plugin for deployers. The second issue is that once the feature is in preview in tree, it moves the responsibility/burden of making it official (or removed) to the core developers (as Salvatore mentioned). I kind of like the approach where experimental features are developed in faster iterations out-of-tree and when they are celebrated by experimenters and are ready to be stable-supported, they are moved in tree. Regards, -- Thierry Carrez (ttx) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree
On 8/11/14, 4:52 AM, Thierry Carrez wrote: gustavo panizzo (gfa) wrote: only one think i didn't like it why all url,api, etc has to include the word 'preview'? i imagine that i would be consuming the new feature using heat, puppet, local scripts, custom horizon, whatever. Why do you make me to change all them when the feature moves out of preview? it could be a lot of rework (for consumers) without gain. I would totally support other big fat warnings everywhere (logs, documentation, startup log of neutron-server) but don't change the API if is not necessary I see two issues with this proposal: the first one is what Gustavo just said: the use of the preview package/configoption/API creates friction when the feature needs to go mainstream (package renaming, change in configuration for deployers, change in API calls for users...). Hi Thierry, I completely agree with you and with Gustavo that mangling the REST URIs to include preview may have more cost (i.e. friction when the API becomes stable) than benefit. I'm happy to drop that particular part of the proposal. The email was intended to kick off discussion of these sorts of details. My understanding is that the goal is to make it easy for people to try the preview feature, and keeping the experimental feature in-tree is seen as simpler to experiment with. But the pain from this friction imho outweighs the pain of deploying an out-of-tree plugin for deployers. I agree out-of-tree is a better option for truly experimental features. This in-tree stabilization is intended for a beta release, as opposed to a prototype. The second issue is that once the feature is in preview in tree, it moves the responsibility/burden of making it official (or removed) to the core developers (as Salvatore mentioned). I kind of like the approach where experimental features are developed in faster iterations out-of-tree and when they are celebrated by experimenters and are ready to be stable-supported, they are moved in tree. I don't think we are really disagreeing here. There are clearly situations where rapid iteration out-of-tree, without the burden of the core review process, is most appropriate. But this proposal is intended for features that are on the cusp of being declared stable, rather than for experimentation. The intent is absolutely to have all changes to the code go through the regular core review process during this stabilization phase. This enables the feature to be fully reviewed and integrated (also including CLIs, Horizon and Heat support, documentation, etc.) at the point when the decision is made that no further incompatible API changes will be needed. Once the feature is declared stable, from the end-user perspective, its just a matter of removing the preview label. Moving the feature's code from the preview subtree to its normal locations in the tree will not effect most users or operators. Note that the GBP team had implemented a proof-of-concept prior to the start of the Juno cycle out-of-tree. Our initial plan was to get this PoC code reviewed and merged at the beginning of Juno, and then iteratively improve it throughout the cycle. But we got a lot of resistance to the idea of merging a large body of code that had been developed outside the Neutron development and review process. We've instead had to break it into multiple pieces, and make sure each of those is production ready, to have any chance of getting through the review process during Juno. Its not really clear that something significant developed externally can ever be moved in tree, at least without major upheaval, including incompatible API changes, as it goes through the review/merge process. Finally, consider that many interesting potential features for Neutron involve integrations with external back-ends, such as ODL or vendor-specific devices or controllers, along with a reference implementation that doesn't depend on external systems. To really validate that the API, the model, and the driver framework code for a new feature are all stable, it is necessary to implement and deploy several of these back-end integrations along with the reference implementation. But vendors may not be willing to invest substantially in this integration effort without the assurances about the quality and relative stability of the interfaces involved that comes from the core review process, and without the clear path to market that comes with in-tree development of an approved blueprint targeted at a specific Neutron release. -Bob Regards, ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree
On 8/8/14, 6:28 PM, Salvatore Orlando wrote: If we want to keep everything the way it is, we have to change everything [1] This is pretty much how I felt after reading this proposal, and I felt that this quote, which Ivar will probably appreciate, was apt to the situation. Recent events have spurred a discussion about the need for a change in process. It is not uncommon indeed to believe that by fixing the process, things will inevitably change for better. While no-one argues that flaws in processes need to be fixed, no process change will ever change anything, in my opinion, unless it is aimed at spurring change in people as well. From what I understand, this proposal starts with the assumption that any new feature which is committed to Neutron (ie: has a blueprint approved), and is not a required neutron component should be considered as a preview. This is not different from the process, which, albeit more informally, has been adopted so far. Load Balancing, Firewall, VPN, have all been explicitly documented as experimental in their first release; I would argue that even if not experimental anymore, they may not be considered stable until their stability was proven by upstream QA with API and scenario tests - but this is not sanctioned anywhere currently, I think. Correct, this proposal is not so much a new process or change in process as a formalization of what we've already been doing, and a suggested adaptation to clarify the current expectations around stability of new APIs. According to this proposal, for preview features: - all the code would be moved to a preview package Yes. - Options will be marked as preview Yes. - URIs should be prefixed with preview That's what I suggested, but, as several people have pointed out, this does seem like its worth the cost of breaking the API compatibility just at the point when it is being declared stable. I'd like to withdraw this item. - CLIs will note the features are preview in their help strings Yes. - Documentation will explicitly state this feature is preview (I think we already mark them as experimental, frankly I don't think there are a lot of differences in terminology here) Yes. Again to me, failure is one likely outcome of an experiment. The term preview is intended to imply more of a commitment to quickly reach stability. - Database migrations will be in the main alembic path as usual Right. - CLI, Devstack and Heat support will be available Right, as appropriate for the feature. - Can be used by non-preview neutron code No, I suggested No non-preview Neutron code should import code from anywhere under the neutron.preview module, - Will undergo the usual review process Right. This is key for the code to not have to jump through a new major upheaval at right as it becomes stable. - QA will be desirable, but will done either with WIP tempest patches or merging the relevant scenario tests in the preview feature iself More than desirable. We need a way to maintain and run the tempest-like API and scenario tests during the stabilization process, but to let then evolve with the feature. - The feature might be promoted or removed, but the process for this is not yet defined. Any suggestions? I did try to address preventing long-term stagnation of preview features. As a starting point, reviewing and merging a patch that moves the code from the preview sub-tree to its intended location could be a lightweight promotion process. I don't think this change in process will actually encourage better behaviour both by contributors and core reviewers. Encouraging better behavior might be necessary, but wasn't the main intent of this proposal. This proposal was intended to clarify and formalize the stability expectations around the initial releases of new features. It was specifically intended to address the conundrum currently faced by reviewers regarding patches that meet all applicable quality standards, but may not yet have (somehow, miraculously) achieved the maturity associated with stable APIs and features fully supported for widespread deployment. I reckon that better behaviour might be encouraged by forcing developer and reviewers to merge in the neutron source code tree only code which meets the highest quality standards. A change in process should enforce this - and when I think about the criteria, I think at the same kind of criteria we're being imposed to declare parity with nova. Proven reliability, and scalability should be a must. Proven usability should be a requirement for all new APIs. I agree regarding the quality standards for merging of code, and am not suggesting relaxing those one bit. But proving all of the desirable system-level attributes of a complex new feature before merging anything precludes any kind of iterative development process. I think we should consider enforcing things like proven reliability, scalability, and usability at the point where the
[openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree
[Note - I understand there are ongoing discussion that may lead to a proposal for an out-of-tree incubation process for new Neutron features. This is a complementary proposal that describes how our existing development process can be used to stabilize new features in-tree over the time frame of a release cycle or two. We should fully consider both proposals, and where each might apply. I hope something like the approach I propose here will allow the implementations of Neutron BPs with non-trivial APIs that have been targeted for the Juno release to be included in that release, used by early adopters, and stabilized as quickly as possible for general consumption.] According to our existing development process, once a blueprint and associated specification for a new Neutron feature have been reviewed, approved, and targeted to a release, development proceeds, resulting in a series of patches to be reviewed and merged to the Neutron source tree. This source tree is then the basis for milestone releases and the final release for the cycle. Ideally, this development process would conclude successfully, well in advance of the cycle's final release, and the resulting feature and its API would be considered fully stable in that release. Stable features are ready for widespread general deployment. Going forward, any further modifications to a stable API must be backwards-compatible with previously released versions. Upgrades must not lose any persistent state associated with stable features. Upgrade processes and their impact on a deployments (downtime, etc.) should be consistent for all stable features. In reality, we developers are not perfect, and minor (or more significant) changes may be identified as necessary or highly desirable once early adopters of the new feature have had a chance to use it. These changes may be difficult or impossible to do in a way that honors the guarantees associated with stable features. For new features that effect the core Neutron API and therefore impact all Neutron deployments, the stability requirement is strict. But for features that do not effect the core API, such as services whose deployment is optional, the stability requirement can be relaxed initially, allowing time for feedback from early adopters to be incorporated before declaring these APIs stable. The key in doing this is to manage the expectations of developers, packagers, operators, and end users regarding these new optional features while they stabilize. I therefore propose that we manage these expectations, while new optional features in the source tree stabilize, by clearly labeling these features with the term preview until they are declared stable, and sufficiently isolating them so that they are not confused with stable features. The proposed guidelines would apply during development as the patches implementing the feature are first merged, in the initial release containing the feature, and in any subsequent releases that are necessary to fully stabilize the feature. Here are my initial not-fully-baked ideas for how our current process can be adapted with a preview feature concept supporting in-tree stabilization of optional features: * Preview features are implementations of blueprints that have been reviewed, approved, and targeted for a Neutron release. The process is intended for features for which there is a commitment to add the feature to Neutron, not for experimentation where failing fast is an acceptable outcome. * Preview features must be optional to deploy, such as by configuring a service plugin or some set of drivers. Blueprint implementations whose deployment is not optional are not eligible to be treated as preview features. * Patches implementing a preview feature are merged to the the master branch of the Neutron source tree. This makes them immediately available to all direct consumers of the source tree, such as developers, trunk-chasing operators, packagers, and evaluators or end-users that use DevStack, maximizing the opportunity to get the feedback that is essential to quickly stabilize the feature. * The process for reviewing, approving and merging patches implementing preview features is exactly the same as for all other Neutron patches. The patches must meet HACKING standards, be production-quality code, have adequate test coverage, have DB migration scripts, etc., and require two +2s and a +A from Neutron core developers to merge. * DB migrations for preview features are treated similarly to other DB migrations, forming a single ordered list that results in the current overall DB schema, including the schema for the preview feature. But DB migrations for a preview feature are not yet required to preserve existing persistent state in a deployment, as would be required for a stable feature. * All code that is part of a preview feature is located under neutron/preview/feature/. Associated unit
Re: [openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree
Hi Robert, I think this is a great proposal. Easy to understand and (at a first glance) easy to implement. Also, it seems the perfect compromise for those who wanted GBP (as a representative for this kind of extension) in tree, and those who wanted it to be more mature first. Fwiw, You have my support on this. Ivar. On Aug 8, 2014 10:27 PM, Robert Kukura kuk...@noironetworks.com wrote: [Note - I understand there are ongoing discussion that may lead to a proposal for an out-of-tree incubation process for new Neutron features. This is a complementary proposal that describes how our existing development process can be used to stabilize new features in-tree over the time frame of a release cycle or two. We should fully consider both proposals, and where each might apply. I hope something like the approach I propose here will allow the implementations of Neutron BPs with non-trivial APIs that have been targeted for the Juno release to be included in that release, used by early adopters, and stabilized as quickly as possible for general consumption.] According to our existing development process, once a blueprint and associated specification for a new Neutron feature have been reviewed, approved, and targeted to a release, development proceeds, resulting in a series of patches to be reviewed and merged to the Neutron source tree. This source tree is then the basis for milestone releases and the final release for the cycle. Ideally, this development process would conclude successfully, well in advance of the cycle's final release, and the resulting feature and its API would be considered fully stable in that release. Stable features are ready for widespread general deployment. Going forward, any further modifications to a stable API must be backwards-compatible with previously released versions. Upgrades must not lose any persistent state associated with stable features. Upgrade processes and their impact on a deployments (downtime, etc.) should be consistent for all stable features. In reality, we developers are not perfect, and minor (or more significant) changes may be identified as necessary or highly desirable once early adopters of the new feature have had a chance to use it. These changes may be difficult or impossible to do in a way that honors the guarantees associated with stable features. For new features that effect the core Neutron API and therefore impact all Neutron deployments, the stability requirement is strict. But for features that do not effect the core API, such as services whose deployment is optional, the stability requirement can be relaxed initially, allowing time for feedback from early adopters to be incorporated before declaring these APIs stable. The key in doing this is to manage the expectations of developers, packagers, operators, and end users regarding these new optional features while they stabilize. I therefore propose that we manage these expectations, while new optional features in the source tree stabilize, by clearly labeling these features with the term preview until they are declared stable, and sufficiently isolating them so that they are not confused with stable features. The proposed guidelines would apply during development as the patches implementing the feature are first merged, in the initial release containing the feature, and in any subsequent releases that are necessary to fully stabilize the feature. Here are my initial not-fully-baked ideas for how our current process can be adapted with a preview feature concept supporting in-tree stabilization of optional features: * Preview features are implementations of blueprints that have been reviewed, approved, and targeted for a Neutron release. The process is intended for features for which there is a commitment to add the feature to Neutron, not for experimentation where failing fast is an acceptable outcome. * Preview features must be optional to deploy, such as by configuring a service plugin or some set of drivers. Blueprint implementations whose deployment is not optional are not eligible to be treated as preview features. * Patches implementing a preview feature are merged to the the master branch of the Neutron source tree. This makes them immediately available to all direct consumers of the source tree, such as developers, trunk-chasing operators, packagers, and evaluators or end-users that use DevStack, maximizing the opportunity to get the feedback that is essential to quickly stabilize the feature. * The process for reviewing, approving and merging patches implementing preview features is exactly the same as for all other Neutron patches. The patches must meet HACKING standards, be production-quality code, have adequate test coverage, have DB migration scripts, etc., and require two +2s and a +A from Neutron core developers to merge. * DB migrations for preview features are treated similarly to other DB migrations, forming a single ordered list that results in the current overall DB
Re: [openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree
On Fri, Aug 8, 2014 at 1:21 PM, Robert Kukura kuk...@noironetworks.com wrote: [Note - I understand there are ongoing discussion that may lead to a proposal for an out-of-tree incubation process for new Neutron features. This is a complementary proposal that describes how our existing development process can be used to stabilize new features in-tree over the time frame of a release cycle or two. We should fully consider both proposals, and where each might apply. I hope something like the approach I propose here will allow the implementations of Neutron BPs with non-trivial APIs that have been targeted for the Juno release to be included in that release, used by early adopters, and stabilized as quickly as possible for general consumption.] +1. I think this proposal is simple to understand, has limited process and operational overhead while achieving the desired benefit of handing preview features to early adopters with the right set of expectations. According to our existing development process, once a blueprint and associated specification for a new Neutron feature have been reviewed, approved, and targeted to a release, development proceeds, resulting in a series of patches to be reviewed and merged to the Neutron source tree. This source tree is then the basis for milestone releases and the final release for the cycle. Ideally, this development process would conclude successfully, well in advance of the cycle's final release, and the resulting feature and its API would be considered fully stable in that release. Stable features are ready for widespread general deployment. Going forward, any further modifications to a stable API must be backwards-compatible with previously released versions. Upgrades must not lose any persistent state associated with stable features. Upgrade processes and their impact on a deployments (downtime, etc.) should be consistent for all stable features. In reality, we developers are not perfect, and minor (or more significant) changes may be identified as necessary or highly desirable once early adopters of the new feature have had a chance to use it. These changes may be difficult or impossible to do in a way that honors the guarantees associated with stable features. For new features that effect the core Neutron API and therefore impact all Neutron deployments, the stability requirement is strict. But for features that do not effect the core API, such as services whose deployment is optional, the stability requirement can be relaxed initially, allowing time for feedback from early adopters to be incorporated before declaring these APIs stable. The key in doing this is to manage the expectations of developers, packagers, operators, and end users regarding these new optional features while they stabilize. I therefore propose that we manage these expectations, while new optional features in the source tree stabilize, by clearly labeling these features with the term preview until they are declared stable, and sufficiently isolating them so that they are not confused with stable features. The proposed guidelines would apply during development as the patches implementing the feature are first merged, in the initial release containing the feature, and in any subsequent releases that are necessary to fully stabilize the feature. Here are my initial not-fully-baked ideas for how our current process can be adapted with a preview feature concept supporting in-tree stabilization of optional features: * Preview features are implementations of blueprints that have been reviewed, approved, and targeted for a Neutron release. The process is intended for features for which there is a commitment to add the feature to Neutron, not for experimentation where failing fast is an acceptable outcome. * Preview features must be optional to deploy, such as by configuring a service plugin or some set of drivers. Blueprint implementations whose deployment is not optional are not eligible to be treated as preview features. * Patches implementing a preview feature are merged to the the master branch of the Neutron source tree. This makes them immediately available to all direct consumers of the source tree, such as developers, trunk-chasing operators, packagers, and evaluators or end-users that use DevStack, maximizing the opportunity to get the feedback that is essential to quickly stabilize the feature. * The process for reviewing, approving and merging patches implementing preview features is exactly the same as for all other Neutron patches. The patches must meet HACKING standards, be production-quality code, have adequate test coverage, have DB migration scripts, etc., and require two +2s and a +A from Neutron core developers to merge. * DB migrations for preview features are treated similarly to other DB migrations, forming a single ordered list that results in the current overall DB schema, including the schema for
Re: [openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree
If we want to keep everything the way it is, we have to change everything [1] This is pretty much how I felt after reading this proposal, and I felt that this quote, which Ivar will probably appreciate, was apt to the situation. Recent events have spurred a discussion about the need for a change in process. It is not uncommon indeed to believe that by fixing the process, things will inevitably change for better. While no-one argues that flaws in processes need to be fixed, no process change will ever change anything, in my opinion, unless it is aimed at spurring change in people as well. From what I understand, this proposal starts with the assumption that any new feature which is committed to Neutron (ie: has a blueprint approved), and is not a required neutron component should be considered as a preview. This is not different from the process, which, albeit more informally, has been adopted so far. Load Balancing, Firewall, VPN, have all been explicitly documented as experimental in their first release; I would argue that even if not experimental anymore, they may not be considered stable until their stability was proven by upstream QA with API and scenario tests - but this is not sanctioned anywhere currently, I think. According to this proposal, for preview features: - all the code would be moved to a preview package - Options will be marked as preview - URIs should be prefixed with preview - CLIs will note the features are preview in their help strings - Documentation will explicitly state this feature is preview (I think we already mark them as experimental, frankly I don't think there are a lot of differences in terminology here) - Database migrations will be in the main alembic path as usual - CLI, Devstack and Heat support will be available - Can be used by non-preview neutron code - Will undergo the usual review process - QA will be desirable, but will done either with WIP tempest patches or merging the relevant scenario tests in the preview feature iself - The feature might be promoted or removed, but the process for this is not yet defined. I don't think this change in process will actually encourage better behaviour both by contributors and core reviewers. I reckon that better behaviour might be encouraged by forcing developer and reviewers to merge in the neutron source code tree only code which meets the highest quality standards. A change in process should enforce this - and when I think about the criteria, I think at the same kind of criteria we're being imposed to declare parity with nova. Proven reliability, and scalability should be a must. Proven usability should be a requirement for all new APIs. On the other hand we also need to avoid to over bureaucratise Neutron - nobody loves that - and therefore ensure this process is enforced only when really needed. Looking at this proposal I see a few thing I'm not comfortable with: - having no clear criterion for exclusion a feature might imply that will be silently bit-rotting code in the preview package. Which what would happen for instance if we end up with a badly maintained feature , but since one or two core devs care about it, they'll keep vetoing the removal - using the normal review process will still not solve the problem of slow review cycles, pointless downvotes for reviewers which actually just do not understand the subject matter, and other pains associated with lack of interest from small or large parts of the core team. For instance, I think there is a line of pretty annoyed contributors as we did not even bother reviewing their specs. - The current provision about QA seems to state that it's ok to keep code in the main repo that does not adhere to appropriate quality standards. Which is the mistake we did with lbaas and other features, and I would like to avoid. And to me it is not sufficient that the code is buried in the 'preview' package. - Mostly important, this process provides a justification for contributors to push features which do not meet the same standards as other neutron parts and expect them to be merged and eventually promoted, and on the other hand provides the core team with the entitlement for merging them - therefore my main concern that it does not encourages better behaviour in people, which should be the ultimate aim of a process change. If you managed to read through all of this, and tolerated my dorky literature references, I really appreciate your patience, and would like to conclude that here we're discussing proposals for a process change, whereas I expect to discuss in the next neutron meeting the following: - whether is acceptable to change the process now - what did go wrong in our spec review process, as we ended up with at least an approved spec which is actually fiercely opposed by other core team members. Have a good weekend, Salvatore [1] Quote from Il Gattopardo by Giuseppe Tomasi di Lampedusa (english name: The Leopard) On 8 August 2014 22:21, Robert Kukura
Re: [openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree
Hi, Robert Kukura's proposal does address the following: 1. Make it explicit to the user that an API is in preview until it's moved out of the preview directories 2. One of the criteria to accept a BP for preview is for the functionality to be optional via configuration. This will not impact the stability of neutron in the general case 3. Allows for new API/functionality to be built leveraging all the neutron infrastructure available 4. Allows for users to adopt new API, increasing the footprint of neutron users +1 on this proposal from my side. Regards, -hemanth On Fri, Aug 8, 2014 at 3:28 PM, Salvatore Orlando sorla...@nicira.com wrote: If we want to keep everything the way it is, we have to change everything [1] This is pretty much how I felt after reading this proposal, and I felt that this quote, which Ivar will probably appreciate, was apt to the situation. Recent events have spurred a discussion about the need for a change in process. It is not uncommon indeed to believe that by fixing the process, things will inevitably change for better. While no-one argues that flaws in processes need to be fixed, no process change will ever change anything, in my opinion, unless it is aimed at spurring change in people as well. From what I understand, this proposal starts with the assumption that any new feature which is committed to Neutron (ie: has a blueprint approved), and is not a required neutron component should be considered as a preview. This is not different from the process, which, albeit more informally, has been adopted so far. Load Balancing, Firewall, VPN, have all been explicitly documented as experimental in their first release; I would argue that even if not experimental anymore, they may not be considered stable until their stability was proven by upstream QA with API and scenario tests - but this is not sanctioned anywhere currently, I think. According to this proposal, for preview features: - all the code would be moved to a preview package - Options will be marked as preview - URIs should be prefixed with preview - CLIs will note the features are preview in their help strings - Documentation will explicitly state this feature is preview (I think we already mark them as experimental, frankly I don't think there are a lot of differences in terminology here) - Database migrations will be in the main alembic path as usual - CLI, Devstack and Heat support will be available - Can be used by non-preview neutron code - Will undergo the usual review process - QA will be desirable, but will done either with WIP tempest patches or merging the relevant scenario tests in the preview feature iself - The feature might be promoted or removed, but the process for this is not yet defined. I don't think this change in process will actually encourage better behaviour both by contributors and core reviewers. I reckon that better behaviour might be encouraged by forcing developer and reviewers to merge in the neutron source code tree only code which meets the highest quality standards. A change in process should enforce this - and when I think about the criteria, I think at the same kind of criteria we're being imposed to declare parity with nova. Proven reliability, and scalability should be a must. Proven usability should be a requirement for all new APIs. On the other hand we also need to avoid to over bureaucratise Neutron - nobody loves that - and therefore ensure this process is enforced only when really needed. Looking at this proposal I see a few thing I'm not comfortable with: - having no clear criterion for exclusion a feature might imply that will be silently bit-rotting code in the preview package. Which what would happen for instance if we end up with a badly maintained feature , but since one or two core devs care about it, they'll keep vetoing the removal - using the normal review process will still not solve the problem of slow review cycles, pointless downvotes for reviewers which actually just do not understand the subject matter, and other pains associated with lack of interest from small or large parts of the core team. For instance, I think there is a line of pretty annoyed contributors as we did not even bother reviewing their specs. - The current provision about QA seems to state that it's ok to keep code in the main repo that does not adhere to appropriate quality standards. Which is the mistake we did with lbaas and other features, and I would like to avoid. And to me it is not sufficient that the code is buried in the 'preview' package. - Mostly important, this process provides a justification for contributors to push features which do not meet the same standards as other neutron parts and expect them to be merged and eventually promoted, and on the other hand provides the core team with the entitlement for merging them - therefore my main concern that it does not encourages better behaviour in people,
Re: [openstack-dev] [Neutron] Simple proposal for stabilizing new features in-tree
i like your idea, as an operator, it gives me new features while keep my core running fine. only one think i didn't like it why all url,api, etc has to include the word 'preview'? i imagine that i would be consuming the new feature using heat, puppet, local scripts, custom horizon, whatever. Why do you make me to change all them when the feature moves out of preview? it could be a lot of rework (for consumers) without gain. I would totally support other big fat warnings everywhere (logs, documentation, startup log of neutron-server) but don't change the API if is not necessary thanks On 08/08/2014 05:21 PM, Robert Kukura wrote: [Note - I understand there are ongoing discussion that may lead to a proposal for an out-of-tree incubation process for new Neutron features. This is a complementary proposal that describes how our existing development process can be used to stabilize new features in-tree over the time frame of a release cycle or two. We should fully consider both proposals, and where each might apply. I hope something like the approach I propose here will allow the implementations of Neutron BPs with non-trivial APIs that have been targeted for the Juno release to be included in that release, used by early adopters, and stabilized as quickly as possible for general consumption.] According to our existing development process, once a blueprint and associated specification for a new Neutron feature have been reviewed, approved, and targeted to a release, development proceeds, resulting in a series of patches to be reviewed and merged to the Neutron source tree. This source tree is then the basis for milestone releases and the final release for the cycle. Ideally, this development process would conclude successfully, well in advance of the cycle's final release, and the resulting feature and its API would be considered fully stable in that release. Stable features are ready for widespread general deployment. Going forward, any further modifications to a stable API must be backwards-compatible with previously released versions. Upgrades must not lose any persistent state associated with stable features. Upgrade processes and their impact on a deployments (downtime, etc.) should be consistent for all stable features. In reality, we developers are not perfect, and minor (or more significant) changes may be identified as necessary or highly desirable once early adopters of the new feature have had a chance to use it. These changes may be difficult or impossible to do in a way that honors the guarantees associated with stable features. For new features that effect the core Neutron API and therefore impact all Neutron deployments, the stability requirement is strict. But for features that do not effect the core API, such as services whose deployment is optional, the stability requirement can be relaxed initially, allowing time for feedback from early adopters to be incorporated before declaring these APIs stable. The key in doing this is to manage the expectations of developers, packagers, operators, and end users regarding these new optional features while they stabilize. I therefore propose that we manage these expectations, while new optional features in the source tree stabilize, by clearly labeling these features with the term preview until they are declared stable, and sufficiently isolating them so that they are not confused with stable features. The proposed guidelines would apply during development as the patches implementing the feature are first merged, in the initial release containing the feature, and in any subsequent releases that are necessary to fully stabilize the feature. Here are my initial not-fully-baked ideas for how our current process can be adapted with a preview feature concept supporting in-tree stabilization of optional features: * Preview features are implementations of blueprints that have been reviewed, approved, and targeted for a Neutron release. The process is intended for features for which there is a commitment to add the feature to Neutron, not for experimentation where failing fast is an acceptable outcome. * Preview features must be optional to deploy, such as by configuring a service plugin or some set of drivers. Blueprint implementations whose deployment is not optional are not eligible to be treated as preview features. * Patches implementing a preview feature are merged to the the master branch of the Neutron source tree. This makes them immediately available to all direct consumers of the source tree, such as developers, trunk-chasing operators, packagers, and evaluators or end-users that use DevStack, maximizing the opportunity to get the feedback that is essential to quickly stabilize the feature. * The process for reviewing, approving and merging patches implementing preview features is exactly the same as for all other Neutron patches. The patches must meet HACKING
