Re: [openstack-dev] [nova] Thought exercise for a V2 only world
On Mon, 3 Mar 2014 21:31:23 -0800 Morgan Fainberg m...@metacloud.com wrote: I think that in a V2 only world a 2 cycle deprecation model would be sufficient for any extension. I don’t foresee any complaints on that front, especially if there is work to supersede or obsolete the need for the extension. Given the context of feedback saying we're not able to deprecate the V2 API as-it-is for a very long time I don't see how a 2 cycle deprecation model for an extension is sufficient. Perhaps it comes down to how do we really know its unused? If it hasn't ever worked (we had one of those!) or accidentally hadn't worked for a couple of cycles and no one noticed then perhaps deprecating it then is ok. But otherwise whilst we can get input from public cloud providers fairly easily there's going to be a lot of small private deployments as well with custom bits of API using code which we won't hear from. And we'll be forcing them off the API which people say is exactly what we don't want to do. Deprecating functionality and still calling it V2 is I think nearly always a bad thing. Because it is very different from what people consider to be major version API stability - eg you may get new features, but old ones stay. Its for similar reasons I'm pretty hesitant about using microversioning for backwards incompatible changes in addition to backwards compatible ones. Because we end up with a concept of major version stability which is quite different from what people expect. I don't think we should be seeing versioning as a magic bullet to get out of API mistakes (except perhaps under really exceptional circumstances) we've made because it really just shifts the pain to the users. Do it enough and people lose an understanding of what it means to have version X.Y of an API available versus X.(Y+n) and whether they can expect the software to still work. Cheers, Morgan — Morgan Fainberg Principal Software Engineer Core Developer, Keystone m...@metacloud.com On March 3, 2014 at 21:29:43, Joe Gordon (joe.gord...@gmail.com) wrote: Hi All, here's a case worth exploring in a v2 only world ... what about some extension we really think is dead and should go away? can we ever remove it? In the past we have said backwards compatibility means no we cannot remove any extensions, if we adopt the v2 only notion of backwards compatibility is this still true? best, Joe ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Thought exercise for a V2 only world
I think I missed the emphasis on the efforts towards superseding and/or making the call obsolete in my previous email (I was aiming to communicate far more than a the few lines managed to convey). I honestly think that if we are sticking with V2 because of the fact that Nova is such a large surface area, the only option is to treat each extension as it’s (almost) own isolated API in the context of deprecation, obsolescence, etc. I think the issue here is that we may be looking at the version of the API like something fixed in stone. Clearly, with the scope and surface area of Nova (this point has been made clear), there is no possible way we can handle a whole-sale change. As a deployer, and supporter of a number of OpenStack based clouds, as long as the API is stable (yes, I’ll give it a full year from when it is determined a change is needed), I don’t see my customers complaining excessively; maybe we make it a 4 cycle deprecation? It is silly to say “because we called it X we can’t ever take anything away”. I am all for not breaking the contract, but define the contract beyond “the spec”. This holds true especially when it comes down to continued growth and possibly moving the data from one place to somewhere better / more suited. Perhaps part of the real issue is the whole extension model. A well documented, interoperable (across deployments) API doesn’t have huge swaths of functionality that is optional (or more to the point what is OpenStack Compute’s API?). If you are adding a core-feature it should it be an “Extension”? Lets add one more step in, ask deployments if they are using the “extensions”? Make it part of the summit / development cycle. Get real information (send surveys?) and get to know the community running the software. That in itself will help to direct if an extension is used. I think the crux is that we do not have a clear (and have no way of getting the information) understanding of what is being used and what isn’t. Perhaps the best thing we can do is make this an exercise on understanding what people are using and how we can quantify that information before we worry about “how do we remove functionality”. Complete removal of functionality is probably going to be rare. It is far more likely that locations will shift and / or things will be rolled into more logical areas. At the speed that we are moving (not slow, but not as fast as other things), it should be totally possible to support a 2+ cycle deprecation if something is being moved / shuffled / absorbed. But more importantly, knowing use is far more important that knowing how to remove, the former will direct the latter. So I propose changing the topic of this thread slightly: In a V2 only world, how do we know if something is used? How do we understand how it is used and when? Lets answer that instead. — Morgan Fainberg Principal Software Engineer Core Developer, Keystone m...@metacloud.com On March 4, 2014 at 02:09:51, Christopher Yeoh (cbky...@gmail.com) wrote: On Mon, 3 Mar 2014 21:31:23 -0800 Morgan Fainberg m...@metacloud.com wrote: I think that in a V2 only world a 2 cycle deprecation model would be sufficient for any extension. I don’t foresee any complaints on that front, especially if there is work to supersede or obsolete the need for the extension. Given the context of feedback saying we're not able to deprecate the V2 API as-it-is for a very long time I don't see how a 2 cycle deprecation model for an extension is sufficient. Perhaps it comes down to how do we really know its unused? If it hasn't ever worked (we had one of those!) or accidentally hadn't worked for a couple of cycles and no one noticed then perhaps deprecating it then is ok. But otherwise whilst we can get input from public cloud providers fairly easily there's going to be a lot of small private deployments as well with custom bits of API using code which we won't hear from. And we'll be forcing them off the API which people say is exactly what we don't want to do. Deprecating functionality and still calling it V2 is I think nearly always a bad thing. Because it is very different from what people consider to be major version API stability - eg you may get new features, but old ones stay. Its for similar reasons I'm pretty hesitant about using microversioning for backwards incompatible changes in addition to backwards compatible ones. Because we end up with a concept of major version stability which is quite different from what people expect. I don't think we should be seeing versioning as a magic bullet to get out of API mistakes (except perhaps under really exceptional circumstances) we've made because it really just shifts the pain to the users. Do it enough and people lose an understanding of what it means to have version X.Y of an API available versus X.(Y+n) and whether they can expect the software to still work. Cheers, Morgan — Morgan Fainberg Principal Software Engineer
Re: [openstack-dev] [nova] Thought exercise for a V2 only world
On Mon, 3 Mar 2014 22:14:02 -0800 Chris Behrens cbehr...@codestud.com wrote: I don’t think I have an answer, but I’m going to throw out some of my random thoughts about extensions in general. They might influence a longer term decision. But I’m also curious if I’m the only one that feels this way: I tend to feel like extensions should start outside of nova and any other code needed to support the extension should be implemented by using hooks in nova. So at the moment we have this conflict between getting new api features in NOW and having to balance that with the pain of having to support any mistakes forever. Especially at this stage of cycle I feel really bad about -1'ing API code because it might mean that they have to wait another cycle and often the API code is just the tip of a really big chunk of work. But at the same time once we merge these API features we have to live with them forever. So there's been the odd discussion going on about how we raise the bar for making API changes, and I think what you have suggested would fit in really well. It would allow us to continue to innovate quickly, but extensions outside of nova could been seen as experimental and potentially subject to backwards incompatible API changes so we're not stuck with mistakes indefinitely. The modules implementing the hook code should be shipped with the extension. If hooks don’t exist where needed, they should be created in trunk. I like hooks. Of course, there’s probably such a thing as too many hooks, so… hmm… :) Anyway, this addresses another annoyance of mine whereby code for extensions is mixed in all over the place. Is it really an extension if all of the supporting code is in ‘core nova’? So the good news is that the V3 API framework already supports this sort of model really well. Under the hood it is based on hooks and allowing different parts of the API to offer hook points for other parts of the API to extend. It's one of the techniques we used to clean up the mess that is the V2 API servers code which is core but littered with extension specific code. Separating it all logically makes it much easier to read/maintain. There's also some nice extra controls so you can if you wish through whitelist/blacklists ensure that the environment doesn't start loading new API features unexpectedly and complains loudly when ones you think should be there aren't. That said, I then think that the only extensions shipped with nova are really ones we deem “optional core API components”. “optional” and “core” are probably oxymorons in this context, but I’m just going to go with it. There would be some sort of process by which we let extensions “graduate” into nova. Like I said, this is not really an answer. But if we had such a model, I wonder if it turns “deprecating extensions” into something more like “deprecating part of the API”… something less likely to happen. Extensions that aren’t used would more likely just never graduate into nova. So for want of a better term, incubating potential API features for a cycle or two before graduating them into the Nova tree would I think be very valuable. It avoids where its hard to really use the API until its implemented, but you learn a lot from using it but by then its too late. It allows us to have graduation type criteria like - is anyone using it?, is all of the interface used, or only part of it, how does this fit into the overall Nova API, etc. We can also fix all the various little bugs in the API design before it gets locked in. And we'll avoid situations like quota_classes which got merged and shipped for a while but as far as we can tell never really did anything useful because the rest of it didn't get finished. So a big +1 from me for this. But I think we'd want to be able to symmetrically gate on this code, otherwise it becomes a bit of nightmare for people maintaining code in this external repository. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Thought exercise for a V2 only world
On 03/04/2014 01:14 AM, Chris Behrens wrote: On Mar 3, 2014, at 9:23 PM, Joe Gordon joe.gord...@gmail.com wrote: Hi All, here's a case worth exploring in a v2 only world ... what about some extension we really think is dead and should go away? can we ever remove it? In the past we have said backwards compatibility means no we cannot remove any extensions, if we adopt the v2 only notion of backwards compatibility is this still true? I don’t think I have an answer, but I’m going to throw out some of my random thoughts about extensions in general. They might influence a longer term decision. But I’m also curious if I’m the only one that feels this way: I tend to feel like extensions should start outside of nova and any other code needed to support the extension should be implemented by using hooks in nova. The modules implementing the hook code should be shipped with the extension. If hooks don’t exist where needed, they should be created in trunk. I like hooks. Of course, there’s probably such a thing as too many hooks, so… hmm… :) Anyway, this addresses another annoyance of mine whereby code for extensions is mixed in all over the place. Is it really an extension if all of the supporting code is in ‘core nova’? That said, I then think that the only extensions shipped with nova are really ones we deem “optional core API components”. “optional” and “core” are probably oxymorons in this context, but I’m just going to go with it. There would be some sort of process by which we let extensions “graduate” into nova. Like I said, this is not really an answer. But if we had such a model, I wonder if it turns “deprecating extensions” into something more like “deprecating part of the API”… something less likely to happen. Extensions that aren’t used would more likely just never graduate into nova. So this approach actually really concerns me, because what it says is that we should be optimizing Nova for out of tree changes to the API which are vendor specific. Which I think is completely the wrong direction. Because in that world you'll never be able to move between Nova installations. What's worse is you'll get multiple people implementing the same feature out of tree, slightly differently. I 100% agree the current extensions approach is problematic. It's used as a way to circumvent the idea of a stable API (mostly with oh, it's an extension, we need this feature right now, and it's not part of core so we don't need to give the same guaruntees.) So realistically I want to march us towards a place where we stop doing that. Nova out of the box should have all the knobs that anyone needs to build these kinds of features on top of. If not, we should fix that. It shouldn't be optional. If that means we need to have more folks spending time on coming together on that interface, so be it, but this world where Nova is a bucket of parts isn't pro-user. A really great instance is the events extension that Dan is working on. Without it, Neutron can't work reliably. But it's an extension. So you can turn it off. So we just made it optional to work with the rest of OpenStack. Also the fact that it was an admin API was used as justification on that. But if reliable integration with Neutron isn't considered a core feature of Nova... well, I'm not sure what would be. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Thought exercise for a V2 only world
On Mar 4, 2014, at 4:09 AM, Sean Dague s...@dague.net wrote: On 03/04/2014 01:14 AM, Chris Behrens wrote: […] I don’t think I have an answer, but I’m going to throw out some of my random thoughts about extensions in general. They might influence a longer term decision. But I’m also curious if I’m the only one that feels this way: I tend to feel like extensions should start outside of nova and any other code needed to support the extension should be implemented by using hooks in nova. The modules implementing the hook code should be shipped with the extension. If hooks don’t exist where needed, they should be created in trunk. I like hooks. Of course, there’s probably such a thing as too many hooks, so… hmm… :) Anyway, this addresses another annoyance of mine whereby code for extensions is mixed in all over the place. Is it really an extension if all of the supporting code is in ‘core nova’? That said, I then think that the only extensions shipped with nova are really ones we deem “optional core API components”. “optional” and “core” are probably oxymorons in this context, but I’m just going to go with it. There would be some sort of process by which we let extensions “graduate” into nova. Like I said, this is not really an answer. But if we had such a model, I wonder if it turns “deprecating extensions” into something more like “deprecating part of the API”… something less likely to happen. Extensions that aren’t used would more likely just never graduate into nova. So this approach actually really concerns me, because what it says is that we should be optimizing Nova for out of tree changes to the API which are vendor specific. Which I think is completely the wrong direction. Because in that world you'll never be able to move between Nova installations. What's worse is you'll get multiple people implementing the same feature out of tree, slightly differently. Right. And I have an internal conflict because I also tend to agree with what you’re saying. :) But I think that if we have API extensions at all, we have your issue of “never being able to move”. Well, maybe not “never”, because at least they’d be easy to “turn on” if they are in nova. But I think for the random API extension that only 1 person ever wants to enable, there’s your same problem. This is somewhat off-topic, but I just don’t want a ton of bloat in nova for something few people use. I 100% agree the current extensions approach is problematic. It's used as a way to circumvent the idea of a stable API (mostly with oh, it's an extension, we need this feature right now, and it's not part of core so we don't need to give the same guaruntees.) Yeah, totally.. that’s bad. So realistically I want to march us towards a place where we stop doing that. Nova out of the box should have all the knobs that anyone needs to build these kinds of features on top of. If not, we should fix that. It shouldn't be optional. Agree, although I’m not sure if I’m reading this correctly as it sounds like you want the knobs that you said above concern you. I want some sort of balance. There’s extensions I think absolutely should be part of nova as optional features… but I don’t want everything. :) - Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Thought exercise for a V2 only world
On 03/04/2014 02:03 PM, Chris Behrens wrote: On Mar 4, 2014, at 4:09 AM, Sean Dague s...@dague.net mailto:s...@dague.net wrote: On 03/04/2014 01:14 AM, Chris Behrens wrote: […] I don’t think I have an answer, but I’m going to throw out some of my random thoughts about extensions in general. They might influence a longer term decision. But I’m also curious if I’m the only one that feels this way: I tend to feel like extensions should start outside of nova and any other code needed to support the extension should be implemented by using hooks in nova. The modules implementing the hook code should be shipped with the extension. If hooks don’t exist where needed, they should be created in trunk. I like hooks. Of course, there’s probably such a thing as too many hooks, so… hmm… :) Anyway, this addresses another annoyance of mine whereby code for extensions is mixed in all over the place. Is it really an extension if all of the supporting code is in ‘core nova’? That said, I then think that the only extensions shipped with nova are really ones we deem “optional core API components”. “optional” and “core” are probably oxymorons in this context, but I’m just going to go with it. There would be some sort of process by which we let extensions “graduate” into nova. Like I said, this is not really an answer. But if we had such a model, I wonder if it turns “deprecating extensions” into something more like “deprecating part of the API”… something less likely to happen. Extensions that aren’t used would more likely just never graduate into nova. So this approach actually really concerns me, because what it says is that we should be optimizing Nova for out of tree changes to the API which are vendor specific. Which I think is completely the wrong direction. Because in that world you'll never be able to move between Nova installations. What's worse is you'll get multiple people implementing the same feature out of tree, slightly differently. Right. And I have an internal conflict because I also tend to agree with what you’re saying. :) But I think that if we have API extensions at all, we have your issue of “never being able to move”. Well, maybe not “never”, because at least they’d be easy to “turn on” if they are in nova. But I think for the random API extension that only 1 person ever wants to enable, there’s your same problem. This is somewhat off-topic, but I just don’t want a ton of bloat in nova for something few people use. I 100% agree the current extensions approach is problematic. It's used as a way to circumvent the idea of a stable API (mostly with oh, it's an extension, we need this feature right now, and it's not part of core so we don't need to give the same guaruntees.) Yeah, totally.. that’s bad. So realistically I want to march us towards a place where we stop doing that. Nova out of the box should have all the knobs that anyone needs to build these kinds of features on top of. If not, we should fix that. It shouldn't be optional. Agree, although I’m not sure if I’m reading this correctly as it sounds like you want the knobs that you said above concern you. I want some sort of balance. There’s extensions I think absolutely should be part of nova as optional features… but I don’t want everything. :) I want to give the knobs to the users. If we thought it was important enough to review and test in Nova, then we made a judgement call that people should have access to it. -Sean -- Sean Dague Samsung Research America s...@dague.net / sean.da...@samsung.com http://dague.net signature.asc Description: OpenPGP digital signature ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Thought exercise for a V2 only world
On Mar 4, 2014, at 11:14 AM, Sean Dague s...@dague.net wrote: I want to give the knobs to the users. If we thought it was important enough to review and test in Nova, then we made a judgement call that people should have access to it. Oh, I see. But, I don’t agree, certainly not for every single knob. It’s less of an issue in the private cloud world, but when you start offering this as a service, not everything is appropriate to enable. - Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Thought exercise for a V2 only world
Hi Joe, I think that in a V2 only world a 2 cycle deprecation model would be sufficient for any extension. I don’t foresee any complaints on that front, especially if there is work to supersede or obsolete the need for the extension. Cheers, Morgan — Morgan Fainberg Principal Software Engineer Core Developer, Keystone m...@metacloud.com On March 3, 2014 at 21:29:43, Joe Gordon (joe.gord...@gmail.com) wrote: Hi All, here's a case worth exploring in a v2 only world ... what about some extension we really think is dead and should go away? can we ever remove it? In the past we have said backwards compatibility means no we cannot remove any extensions, if we adopt the v2 only notion of backwards compatibility is this still true? best, Joe ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Thought exercise for a V2 only world
On Mar 3, 2014, at 9:23 PM, Joe Gordon joe.gord...@gmail.com wrote: Hi All, here's a case worth exploring in a v2 only world ... what about some extension we really think is dead and should go away? can we ever remove it? In the past we have said backwards compatibility means no we cannot remove any extensions, if we adopt the v2 only notion of backwards compatibility is this still true? I don’t think I have an answer, but I’m going to throw out some of my random thoughts about extensions in general. They might influence a longer term decision. But I’m also curious if I’m the only one that feels this way: I tend to feel like extensions should start outside of nova and any other code needed to support the extension should be implemented by using hooks in nova. The modules implementing the hook code should be shipped with the extension. If hooks don’t exist where needed, they should be created in trunk. I like hooks. Of course, there’s probably such a thing as too many hooks, so… hmm… :) Anyway, this addresses another annoyance of mine whereby code for extensions is mixed in all over the place. Is it really an extension if all of the supporting code is in ‘core nova’? That said, I then think that the only extensions shipped with nova are really ones we deem “optional core API components”. “optional” and “core” are probably oxymorons in this context, but I’m just going to go with it. There would be some sort of process by which we let extensions “graduate” into nova. Like I said, this is not really an answer. But if we had such a model, I wonder if it turns “deprecating extensions” into something more like “deprecating part of the API”… something less likely to happen. Extensions that aren’t used would more likely just never graduate into nova. - Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
