Re: [openstack-dev] [nova] Proposal: remove the server groups feature
The topic is Scheduler hints for VM life cyclehttp://junodesignsummit.sched.org/event/77801877aa42b595f14ae8b020cd1999#. Thanks. 2014-05-04 10:06 GMT+08:00 Qiming Teng teng...@linux.vnet.ibm.com: On Thu, May 01, 2014 at 08:49:11PM +0800, Jay Lau wrote: Jay Pipes and all, I'm planning to merge this topic to http://junodesignsummit.sched.org/event/77801877aa42b595f14ae8b020cd1999after some discussion in this week's Gantt IRC meeting, hope it is OK. Thanks! The link above didn't work. How about telling us the name of the topic? 2014-05-01 19:56 GMT+08:00 Day, Phil philip@hp.com: In the original API there was a way to remove members from the group. This didn't make it into the code that was submitted. Well, it didn't make it in because it was broken. If you add an instance to a group after it's running, a migration may need to take place in order to keep the semantics of the group. That means that for a while the policy will be being violated, and if we can't migrate the instance somewhere to satisfy the policy then we need to either drop it back out, or be in violation. Either some additional states (such as being queued for inclusion in a group, etc) may be required, or some additional footnotes on what it means to be in a group might have to be made. It was for the above reasons, IIRC, that we decided to leave that bit out since the semantics and consequences clearly hadn't been fully thought-out. Obviously they can be addressed, but I fear the result will be ... ugly. I think there's a definite possibility that leaving out those dynamic functions will look more desirable than an actual implementation. If we look at a server group as a general contained or servers, that may have an attribute that expresses scheduling policy, then it doesn't seem to ugly to restrict the conditions on which an add is allowed to only those that don't break the (optional) policy.Wouldn't even have to go to the scheduler to work this out. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Thanks, Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
On Thu, May 01, 2014 at 08:49:11PM +0800, Jay Lau wrote: Jay Pipes and all, I'm planning to merge this topic to http://junodesignsummit.sched.org/event/77801877aa42b595f14ae8b020cd1999after some discussion in this week's Gantt IRC meeting, hope it is OK. Thanks! The link above didn't work. How about telling us the name of the topic? 2014-05-01 19:56 GMT+08:00 Day, Phil philip@hp.com: In the original API there was a way to remove members from the group. This didn't make it into the code that was submitted. Well, it didn't make it in because it was broken. If you add an instance to a group after it's running, a migration may need to take place in order to keep the semantics of the group. That means that for a while the policy will be being violated, and if we can't migrate the instance somewhere to satisfy the policy then we need to either drop it back out, or be in violation. Either some additional states (such as being queued for inclusion in a group, etc) may be required, or some additional footnotes on what it means to be in a group might have to be made. It was for the above reasons, IIRC, that we decided to leave that bit out since the semantics and consequences clearly hadn't been fully thought-out. Obviously they can be addressed, but I fear the result will be ... ugly. I think there's a definite possibility that leaving out those dynamic functions will look more desirable than an actual implementation. If we look at a server group as a general contained or servers, that may have an attribute that expresses scheduling policy, then it doesn't seem to ugly to restrict the conditions on which an add is allowed to only those that don't break the (optional) policy.Wouldn't even have to go to the scheduler to work this out. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
In the original API there was a way to remove members from the group. This didn't make it into the code that was submitted. Well, it didn't make it in because it was broken. If you add an instance to a group after it's running, a migration may need to take place in order to keep the semantics of the group. That means that for a while the policy will be being violated, and if we can't migrate the instance somewhere to satisfy the policy then we need to either drop it back out, or be in violation. Either some additional states (such as being queued for inclusion in a group, etc) may be required, or some additional footnotes on what it means to be in a group might have to be made. It was for the above reasons, IIRC, that we decided to leave that bit out since the semantics and consequences clearly hadn't been fully thought-out. Obviously they can be addressed, but I fear the result will be ... ugly. I think there's a definite possibility that leaving out those dynamic functions will look more desirable than an actual implementation. If we look at a server group as a general contained or servers, that may have an attribute that expresses scheduling policy, then it doesn't seem to ugly to restrict the conditions on which an add is allowed to only those that don't break the (optional) policy.Wouldn't even have to go to the scheduler to work this out. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
Jay Pipes and all, I'm planning to merge this topic to http://junodesignsummit.sched.org/event/77801877aa42b595f14ae8b020cd1999after some discussion in this week's Gantt IRC meeting, hope it is OK. Thanks! 2014-05-01 19:56 GMT+08:00 Day, Phil philip@hp.com: In the original API there was a way to remove members from the group. This didn't make it into the code that was submitted. Well, it didn't make it in because it was broken. If you add an instance to a group after it's running, a migration may need to take place in order to keep the semantics of the group. That means that for a while the policy will be being violated, and if we can't migrate the instance somewhere to satisfy the policy then we need to either drop it back out, or be in violation. Either some additional states (such as being queued for inclusion in a group, etc) may be required, or some additional footnotes on what it means to be in a group might have to be made. It was for the above reasons, IIRC, that we decided to leave that bit out since the semantics and consequences clearly hadn't been fully thought-out. Obviously they can be addressed, but I fear the result will be ... ugly. I think there's a definite possibility that leaving out those dynamic functions will look more desirable than an actual implementation. If we look at a server group as a general contained or servers, that may have an attribute that expresses scheduling policy, then it doesn't seem to ugly to restrict the conditions on which an add is allowed to only those that don't break the (optional) policy.Wouldn't even have to go to the scheduler to work this out. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Thanks, Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
On Thu, 2014-05-01 at 20:49 +0800, Jay Lau wrote: Jay Pipes and all, I'm planning to merge this topic to http://junodesignsummit.sched.org/event/77801877aa42b595f14ae8b020cd1999 after some discussion in this week's Gantt IRC meeting, hope it is OK. I'll be there :) Thanks! -jay 2014-05-01 19:56 GMT+08:00 Day, Phil philip@hp.com: In the original API there was a way to remove members from the group. This didn't make it into the code that was submitted. Well, it didn't make it in because it was broken. If you add an instance to a group after it's running, a migration may need to take place in order to keep the semantics of the group. That means that for a while the policy will be being violated, and if we can't migrate the instance somewhere to satisfy the policy then we need to either drop it back out, or be in violation. Either some additional states (such as being queued for inclusion in a group, etc) may be required, or some additional footnotes on what it means to be in a group might have to be made. It was for the above reasons, IIRC, that we decided to leave that bit out since the semantics and consequences clearly hadn't been fully thought-out. Obviously they can be addressed, but I fear the result will be ... ugly. I think there's a definite possibility that leaving out those dynamic functions will look more desirable than an actual implementation. If we look at a server group as a general contained or servers, that may have an attribute that expresses scheduling policy, then it doesn't seem to ugly to restrict the conditions on which an add is allowed to only those that don't break the (optional) policy.Wouldn't even have to go to the scheduler to work this out. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Thanks, Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
On Mon, Apr 28, 2014 at 10:58:50AM -0600, Chris Friesen wrote: On 04/25/2014 03:15 PM, Jay Pipes wrote: There are myriad problems with the above user experience and implementation. Let me explain them. 1. The user isn't creating a server group when they issue a nova server-group-create call. They are creating a policy and calling it a group. Cognitive dissonance results from this mismatch. I actually don't think this is true. From my perspective they are actually creating a group, and then when booting servers they can be added into the group. The group happens to have a policy, it is not only a policy. I tend to agree to this. Server groups today are just a starting point. A server group can be used for different purposes like load-balancing, auto-scaling, high-availability or even a combination of them, just name a few. With that said, I would also like to see the policies separated from the group itself. One or more policies can be associated with a group (just like a 'floating-ip'), but the policies are managed separately. 2. There's no way to add an existing server to this group. In the original API there was a way to add existing servers to the group. This didn't make it into the code that was submitted. It is however supported by the instance group db API in nova. To disallow putting an apple into a basket of oranges, we will need to come up with some admission control/validation. It is complicated, but doable, IMO. 3. There's no way to remove members from the group In the original API there was a way to remove members from the group. This didn't make it into the code that was submitted. Just like 'adding new members', 'removing members' can be done if certain conditions are met. We can safely assume the users do understand what they are doing. I mean, we can leave the high level policy decisions to users, with nova only providing dumb mechanisms. 4. There's no way to manually add members to the server group Isn't this the same as item 2? 5. The act of telling the scheduler to place instances near or away from some other instances has been hidden behind the server group API, which means that users doing a nova help boot will see a --group option that doesn't make much sense, as it doesn't describe the scheduling policy activity. The confusion here is understandable, because we want to consider both the policies and the mechanisms at the same time. Maybe we can leave the policies out of this discussion. There are many things hidden away that affect server booting...metadata matching between host aggregates and flavor extra specs, for instance. As I understand it, originally the concept of server groups was more broad. They supported multiple policies, arbitrary group metadata, etc. The scheduler policy was only one of the things that could be associated with a group. This is why the underlying database structure is more complicated than necessary for the current set of supported operations. What we have currently is sort of a dumbed-down version but now that we have the basic support we can start adding in additional functionality as desired. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
-Original Message- From: Jay Pipes [mailto:jaypi...@gmail.com] Sent: 25 April 2014 23:29 To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [nova] Proposal: remove the server groups feature On Fri, 2014-04-25 at 22:00 +, Day, Phil wrote: Hi Jay, I'm going to disagree with you on this one, because: No worries, Phil, I expected some dissention and I completely appreciate your feedback and perspective :) Always happy to meet your expectations ;-) Seem that where we mainly disagree is on the usability of the current Server Group API - and maybe we just need a wider range of views to see where the majority feeling is on that. The folks we've got who have been looking for affinity/anti-affinity scheduling (we've been holding off from the previous filters because they include DB lookups) don't seem to find the Server Groups schematic confusing - you know you want to do something with a group of servers so you create a group, define the properties for that group, and add servers to it as you create them. I agree there are a number of things needed to round this out (such as add/remove server, and some form of quota on the maximum size of a group), but I just don't see the basic approach as broken in the way that you do - and I am worried that we end up spinning on much needed functionality if we start to rework it now. The tagging approach (if I understand it correctly) seems like it would start to introduce system schematics to values that are currently just user defined free text - which I think might lead to more confusion / name-space clashes around which tags are now in effect reserved names and which are still user defined. I think I prefer the clearer separation. Phil i) This is a feature that was discussed in at least one if not two Design Summits and went through a long review period, it wasn't one of those changes that merged in 24 hours before people could take a good look at it. Completely understood. That still doesn't mean we can't propose to get rid of it early instead of letting it sit around when an alternate implementation would be better for the user of OpenStack. Whatever you feel about the implementation, it is now in the API and we should assume that people have started coding against it. Sure, maybe. AFAIK, it's only in the v2 API, though, not in the v3 API (sorry, I made a mistake about that in my original email). Is there a reason it wasn't added to the v3 API? I don't think it gives any credibility to Openstack as a platform if we yank features back out just after they've landed. Perhaps not, though I think we have less credibility if we don't recognize when a feature isn't implemented with users in mind and leave it in the code base to the detriment and confusion of users. We absolutely must, IMO, as a community, be able to say this isn't right and have a path for changing or removing something. If that path is deprecation vs outright removal, so be it, I'd be cool with that. I'd just like to nip this anti-feature in the bud early so that it doesn't become the next feature like file-injection to persist in Nova well after its time has come and passed. ii) Sever Group - It's a way of defining a group of servers, and the initial thing (only thing right now) you can define for such a group is the affinity or anti-affinity for scheduling. We already had ways of defining groups of servers. This new feature doesn't actually define a group of servers. It defines a policy, which is not particularly useful, as it's something that is better specified at the time of launching. Maybe in time we'll add other group properties or operations - like delete all the servers in a group (I know some QA folks that would love to have that feature). We already have the ability to define a group of servers using key=value tags. Deleting all servers in a group is a three-line bash script that loops over the results of a nova list command and calls nova delete. Trust me, I've done group deletes in this way many times. I don't see why it shouldn't be possible to have a server group that doesn't have a scheduling policy associated to it. I don't think the grouping of servers should have *anything* to do with scheduling :) That's the point of my proposal. Servers can and should be grouped using simple tags or key=value pair tags. The grouping of servers together doesn't have anything of substance to do with scheduling policies. I don't see any Cognitive dissonance here - I think your just assuming that the only reason for being able to group servers is for scheduling. Again, I don't think scheduling and grouping of servers has anything to do with each other, thus my proposal to remove the relationship between groups of servers and scheduling policies, which is what the existing server group API and implementation does
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
- Original Message - Hi Stackers, 8--8--8--8--8--8-- Proposal I propose to scrap the server groups API entirely and replace it with a simpler way to accomplish the same basic thing. Create two new options to nova boot: --near-tag TAG and --not-near-tag TAG The first would tell the scheduler to place the new VM near other VMs having a particular tag. The latter would tell the scheduler to place the new VM *not* near other VMs with a particular tag. Would we continue to grow this set of arguments in response to the addition of new policies, how much do we expect this to grow? The two most likely additions I can think of are soft/best effort versions of the current two, are there any other proposals/ideas out there - I know we're a creative bunch ;)? What is a tag? Well, currently, since the Compute API doesn't have a concept of a single string tag, the tag could be a key=value pair that would be matched against the server extra properties. Once a real user-controlled simple string tags system is added to the Compute API, a tag would be just that, a simple string that may be attached or detached from some object (in this case, a server object). How does this solve all the issues highlighted above? In order, it solves the issues like so: 1. There's no need to have any server group object any more. Servers have a set of tags (key/value pairs in v2/v3 API) that may be used to identify a type of server. The activity of launching an instance would now have options for the user to indicate their affinity preference, which removes the cognitive dissonance that happens due to the user needing to know what a server group is (a policy, not a group). Would the user's affinity preference stay with the instance for consideration in future operations post-boot (either now or in a future extension of this functionality)? 2. Since there is no more need to maintain a separate server group object, if a user launched 3 instances and then wanted to make sure that 3 new instances don't end up on the same hosts, all the user needs to do is tag the existing instances with a tag, and issue a call to: nova boot --not-near-tag $TAG ... and the affinity policy is applied properly. The fact that membership can't be changed, at least in the initial implementation, is explicitly called out in the design wiki for the server group api [1]. My understanding is that this was not because implementing an add/remove that works the way you suggest would have been particularly problematic but because user expectations when the group membership is modified are not just that the new instances booted into the group subsequently are placed with affinity/anti-affinity but that the existing instances that were added to the group are also evaluated and moved as necessary to ensure *all* members of the group meet the policy. So in the example this would mean ensuring that all 6 VMs have anti-affinity, not just the latest 3 that are being booted do (or perhaps I am misreading what you are proposing?). Similarly there is an expectation that it's possible to look at a group and easily determine whether it was placed with a policy, and if so what that policy was (not saying that could not be implemented on top of your proposal, just recording for completeness). Whether solutions to meet these expectations belongs in Nova or somewhere else is probably another matter but when dealing with users who use this functionality and they talk about modifying group membership *this* is what they expect. On the other hand this proposal does seem to offer more flexibility for some more complex use cases, for example where users want to place pairs of instances in their own instance groups with affinity, but have them placed with anti-affinity for other pairs. Perhaps I am missing something in the server groups design (or proposed extensions of it) on this point though. I think everyone agrees that the server groups API as it stands is not the final solution here, but I think it's important to refer back to the previous design summit discussions [2][3] on this functionality and ensure any replacement caters not just to reimplementing the current state of server groups but also ensuring that it's easily extended to cover future needs (particularly those already discussed/considered in framing the current functionality). Thanks, Steve [1] https://wiki.openstack.org/wiki/GroupApiExtension [2] https://etherpad.openstack.org/p/group-scheduling [3] https://etherpad.openstack.org/p/NovaIcehouse-Instance-Group-API ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
- Original Message - On Fri, 25 Apr 2014 18:28:38 -0400 Jay Pipes jaypi...@gmail.com wrote: Sure, maybe. AFAIK, it's only in the v2 API, though, not in the v3 API (sorry, I made a mistake about that in my original email). Is there a reason it wasn't added to the v3 API? We did have a pretty strong rule for most of the Icehouse development cycle to only merge new API features if the change was added either first to the V3 API or at the same time as the V2 API. However this (almost unintentionally) ended up getting relaxed whilst all the V2 vs V3 API discussions were occurring. As a result there are some features that were merged into V2 that we definitely need to now add to the V3 API in Juno. Since the V3 API is still experimental we have some flexibility, but transition pain for those moving from V2 to V3 is still going to be a factor in terms of what we want to support. Chris Yeah, looks like this fell by the wayside during those discussions, the v3 patchset linked from the blueprint is here: https://review.openstack.org/#/c/70533/ -Steve ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
On 04/25/2014 03:15 PM, Jay Pipes wrote: There are myriad problems with the above user experience and implementation. Let me explain them. 1. The user isn't creating a server group when they issue a nova server-group-create call. They are creating a policy and calling it a group. Cognitive dissonance results from this mismatch. I actually don't think this is true. From my perspective they are actually creating a group, and then when booting servers they can be added into the group. The group happens to have a policy, it is not only a policy. 2. There's no way to add an existing server to this group. In the original API there was a way to add existing servers to the group. This didn't make it into the code that was submitted. It is however supported by the instance group db API in nova. 3. There's no way to remove members from the group In the original API there was a way to remove members from the group. This didn't make it into the code that was submitted. 4. There's no way to manually add members to the server group Isn't this the same as item 2? 5. The act of telling the scheduler to place instances near or away from some other instances has been hidden behind the server group API, which means that users doing a nova help boot will see a --group option that doesn't make much sense, as it doesn't describe the scheduling policy activity. There are many things hidden away that affect server booting...metadata matching between host aggregates and flavor extra specs, for instance. As I understand it, originally the concept of server groups was more broad. They supported multiple policies, arbitrary group metadata, etc. The scheduler policy was only one of the things that could be associated with a group. This is why the underlying database structure is more complicated than necessary for the current set of supported operations. What we have currently is sort of a dumbed-down version but now that we have the basic support we can start adding in additional functionality as desired. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
On 04/28/2014 06:58 AM, Steve Gordon wrote: - Original Message - Create two new options to nova boot: --near-tag TAG and --not-near-tag TAG The first would tell the scheduler to place the new VM near other VMs having a particular tag. The latter would tell the scheduler to place the new VM *not* near other VMs with a particular tag. Would we continue to grow this set of arguments in response to the addition of new policies, how much do we expect this to grow? The two most likely additions I can think of are soft/best effort versions of the current two, are there any other proposals/ideas out there - I know we're a creative bunch ;)? One logical extension that came up previously is a max group size, maybe expressed as a quota or something. 1. There's no need to have any server group object any more. Servers have a set of tags (key/value pairs in v2/v3 API) that may be used to identify a type of server. The activity of launching an instance would now have options for the user to indicate their affinity preference, which removes the cognitive dissonance that happens due to the user needing to know what a server group is (a policy, not a group). Would the user's affinity preference stay with the instance for consideration in future operations post-boot (either now or in a future extension of this functionality)? Whichever way it's implemented, we need to preserve the boot time scheduler constraints so that any time we reschedule (migration, evacuation, resize, etc.) the constraints will be re-evaluated. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
2. There's no way to add an existing server to this group. In the original API there was a way to add existing servers to the group. This didn't make it into the code that was submitted. It is however supported by the instance group db API in nova. 3. There's no way to remove members from the group In the original API there was a way to remove members from the group. This didn't make it into the code that was submitted. Well, it didn't make it in because it was broken. If you add an instance to a group after it's running, a migration may need to take place in order to keep the semantics of the group. That means that for a while the policy will be being violated, and if we can't migrate the instance somewhere to satisfy the policy then we need to either drop it back out, or be in violation. Either some additional states (such as being queued for inclusion in a group, etc) may be required, or some additional footnotes on what it means to be in a group might have to be made. It was for the above reasons, IIRC, that we decided to leave that bit out since the semantics and consequences clearly hadn't been fully thought-out. Obviously they can be addressed, but I fear the result will be ... ugly. I think there's a definite possibility that leaving out those dynamic functions will look more desirable than an actual implementation. --Dan ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
On 04/28/2014 11:22 AM, Dan Smith wrote: 2. There's no way to add an existing server to this group. In the original API there was a way to add existing servers to the group. This didn't make it into the code that was submitted. It is however supported by the instance group db API in nova. 3. There's no way to remove members from the group In the original API there was a way to remove members from the group. This didn't make it into the code that was submitted. Well, it didn't make it in because it was broken. If you add an instance to a group after it's running, a migration may need to take place in order to keep the semantics of the group. That means that for a while the policy will be being violated, and if we can't migrate the instance somewhere to satisfy the policy then we need to either drop it back out, or be in violation. Either some additional states (such as being queued for inclusion in a group, etc) may be required, or some additional footnotes on what it means to be in a group might have to be made. I think your comment actually applies to adding existing instances to a group. There's no good reason not to allow removing instances from a group. As for the case of addition, we could start with something simple...if adding an instance to a group would violate the group scheduling policy, then raise an exception. It was for the above reasons, IIRC, that we decided to leave that bit out since the semantics and consequences clearly hadn't been fully thought-out. Obviously they can be addressed, but I fear the result will be ... ugly. I think there's a definite possibility that leaving out those dynamic functions will look more desirable than an actual implementation. Your idea of pending group membership doesn't sound too ugly. That said, I would expect adding existing instances to a group to be something that would be done under fairly well-controlled circumstances. In that case I think it would be reasonable to push the work of managing any migrations onto whoever is trying to create a group from existing instances. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
Steve Gordon sgor...@redhat.com wrote on 04/28/2014 08:58:35 AM: - Original Message - Hi Stackers, Proposal Create two new options to nova boot: --near-tag TAG and --not-near-tag TAG The first would tell the scheduler to place the new VM near other VMs having a particular tag. The latter would tell the scheduler to place the new VM *not* near other VMs with a particular tag. Would we continue to grow this set of arguments in response to the addition of new policies, how much do we expect this to grow? The two most likely additions I can think of are soft/best effort versions of the current two, are there any other proposals/ideas out there - I know we're a creative bunch ;)? I brought an extensive list to the last summit; see https://wiki.openstack.org/wiki/Heat/PolicyExtension (but concrete syntax proposals B and C, I only brought A at the time, and am editing B and C into existence now) and do not be distracted by the fact that it is a Heat proposal; the policy issues are prior issues for scheduling irrespective of heat involvement. Note that I not only add policy types but also endow some of them with parameters. Regards, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
On Fri, 25 Apr 2014 18:28:38 -0400 Jay Pipes jaypi...@gmail.com wrote: i) This is a feature that was discussed in at least one if not two Design Summits and went through a long review period, it wasn't one of those changes that merged in 24 hours before people could take a good look at it. Completely understood. That still doesn't mean we can't propose to get rid of it early instead of letting it sit around when an alternate implementation would be better for the user of OpenStack. Long term stability is also very important though - the balance between perfect and good enough. I think that raising the bar on what we allow in the first place is really the key here (and that as discussed previously may involve new features being considered experimental for a period of time). Whatever you feel about the implementation, it is now in the API and we should assume that people have started coding against it. Sure, maybe. AFAIK, it's only in the v2 API, though, not in the v3 API (sorry, I made a mistake about that in my original email). Is there a reason it wasn't added to the v3 API? We did have a pretty strong rule for most of the Icehouse development cycle to only merge new API features if the change was added either first to the V3 API or at the same time as the V2 API. However this (almost unintentionally) ended up getting relaxed whilst all the V2 vs V3 API discussions were occurring. As a result there are some features that were merged into V2 that we definitely need to now add to the V3 API in Juno. Since the V3 API is still experimental we have some flexibility, but transition pain for those moving from V2 to V3 is still going to be a factor in terms of what we want to support. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
On Sat, Apr 26, 2014 at 5:15 AM, Jay Pipes jaypi...@gmail.com wrote: Hi Stackers, When recently digging in to the new server group v3 API extension introduced in Icehouse, I was struck with a bit of cognitive dissonance that I can't seem to shake. While I understand and support the idea behind the feature (affinity and anti-affinity scheduling hints), I can't help but feel the implementation is half-baked and results in a very awkward user experience. The use case here is very simple: Alice wants to launch an instance and make sure that the instance does not land on a compute host that contains other instances of that type. The current user experience is that the user creates a server group like so: nova server-group-create $GROUP_NAME --policy=anti-affinity and then, when the user wishes to launch an instance and make sure it doesn't land on a host with another of that instance type, the user does the following: nova boot --group $GROUP_UUID ... There are myriad problems with the above user experience and implementation. Let me explain them. 1. The user isn't creating a server group when they issue a nova server-group-create call. They are creating a policy and calling it a group. Cognitive dissonance results from this mismatch. 2. There's no way to add an existing server to this group. What this means is that the user needs to effectively have pre-considered their environment and policy before ever launching a VM. To realize why this is a problem, consider the following: - User creates three VMs that consume high I/O utilization - User then wants to launch three more VMs of the same kind and make sure they don't end up on the same hosts as the others No can do, since the first three VMs weren't started using a --group scheduler hint. 3. There's no way to remove members from the group 4. There's no way to manually add members to the server group 5. The act of telling the scheduler to place instances near or away from some other instances has been hidden behind the server group API, which means that users doing a nova help boot will see a --group option that doesn't make much sense, as it doesn't describe the scheduling policy activity. Proposal I propose to scrap the server groups API entirely and replace it with a simpler way to accomplish the same basic thing. Create two new options to nova boot: --near-tag TAG and --not-near-tag TAG Hi jay, I have a little question. Whether it will support multiple tags for the server? Maybe we hope a server to near some servers and not near another servers currently. Best regards to you. Ricky The first would tell the scheduler to place the new VM near other VMs having a particular tag. The latter would tell the scheduler to place the new VM *not* near other VMs with a particular tag. What is a tag? Well, currently, since the Compute API doesn't have a concept of a single string tag, the tag could be a key=value pair that would be matched against the server extra properties. Once a real user-controlled simple string tags system is added to the Compute API, a tag would be just that, a simple string that may be attached or detached from some object (in this case, a server object). How does this solve all the issues highlighted above? In order, it solves the issues like so: 1. There's no need to have any server group object any more. Servers have a set of tags (key/value pairs in v2/v3 API) that may be used to identify a type of server. The activity of launching an instance would now have options for the user to indicate their affinity preference, which removes the cognitive dissonance that happens due to the user needing to know what a server group is (a policy, not a group). 2. Since there is no more need to maintain a separate server group object, if a user launched 3 instances and then wanted to make sure that 3 new instances don't end up on the same hosts, all the user needs to do is tag the existing instances with a tag, and issue a call to: nova boot --not-near-tag $TAG ... and the affinity policy is applied properly. 3. Removal of members of the server group is no longer an issue. Simply untag a server to remove it from the set of servers you wish to use in applying some affinity policy 4. Likewise, since there's no server group object, in order to relate an existing server to another is to simply place a tag on the server. 5. The act of applying affinity policies is now directly related to the act of launching instances, which is where it should be. I'll type up a real blueprint spec for this, but wanted to throw the idea out there, since it's something that struck me recently when I tried to explain the new server groups feature. Thoughts and feedback welcome, -jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
I think server group is an important feature especially when working with heat auto scaling group, there is already some discussion for this http://markmail.org/message/jl5wlx3nr3g53ko5 The current server group feature does support add/delete a VM instance to/from the server group but seems not able to manage existing VM instances, but this can be enhanced. The server group feature need two steps to create the VM instance: 1) Create a server group with policy 2) Create VMs for the server group What Jay Pipes proposed is using resource tags directly: 1) Create VMs with a resource tag to specify the policy. I think that those two directions are very similar, but what Jay Pipes proposed does not specify the resource group and seems the resource group was implicitly specified in resource tag. Just some of my understanding. Thanks! 2014-04-27 1:25 GMT+08:00 Vishvananda Ishaya vishvana...@gmail.com: On Apr 25, 2014, at 2:25 PM, Chris Behrens cbehr...@codestud.com wrote: On Apr 25, 2014, at 2:15 PM, Jay Pipes jaypi...@gmail.com wrote: Hi Stackers, When recently digging in to the new server group v3 API extension introduced in Icehouse, I was struck with a bit of cognitive dissonance that I can't seem to shake. While I understand and support the idea behind the feature (affinity and anti-affinity scheduling hints), I can't help but feel the implementation is half-baked and results in a very awkward user experience. I agree with all you said about this. Proposal I propose to scrap the server groups API entirely and replace it with a simpler way to accomplish the same basic thing. Create two new options to nova boot: --near-tag TAG and --not-near-tag TAG The first would tell the scheduler to place the new VM near other VMs having a particular tag. The latter would tell the scheduler to place the new VM *not* near other VMs with a particular tag. What is a tag? Well, currently, since the Compute API doesn't have a concept of a single string tag, the tag could be a key=value pair that would be matched against the server extra properties. You can actually already achieve this behavior… although with a little more work. There’s the Affinty filter which allows you to specify a same_host/different_host scheduler hint where you explicitly specify the instance uuids you want… (the extra work is having to know the instance uuids). It was my understanding from previous discussions that having the concept of a group was necessary for future schediuling decisions, especially involving live migration. The uuids you need to be far from at launch time won’t necessarily be the ones you need to be far from when a migration is performed. Server groups handle this case, although Jay’s proposal of near/far from tag would also solve this as long as the near-to/far-from data was saved in the instance record. My only concern here is removing an api we just added, so a smoother transition would be preferable. Vish But yeah, I think this makes more sense to me. - Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Thanks, Jay ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
Jay Lau jay.lau@gmail.com wrote on 04/27/2014 12:31:01 AM: I think server group is an important feature especially when working with heat auto scaling group, there is already some discussion for this http://markmail.org/message/jl5wlx3nr3g53ko5 The current server group feature does support add/delete a VM instance to/from the server group but seems not able to manage existing VM instances, but this can be enhanced. The server group feature need two steps to create the VM instance: 1) Create a server group with policy 2) Create VMs for the server group What Jay Pipes proposed is using resource tags directly: 1) Create VMs with a resource tag to specify the policy. No, the tag contributes to the grouping; the policy is identified by the choice of command line switch (--affinity vs. --anti-affinity). I think that those two directions are very similar, but what Jay Pipes proposed does not specify the resource group and seems the resource group was implicitly specified in resource tag. In short, the proposal from Jay Pipes *does* have groups but their membership is declared in a different way than in the current server groups feature. Jay's proposal is not really about *removing* server groups but rather it is a proposal to change their API. Regards, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
On Apr 25, 2014, at 2:15 PM, Jay Pipes jaypi...@gmail.com wrote: Hi Stackers, When recently digging in to the new server group v3 API extension introduced in Icehouse, I was struck with a bit of cognitive dissonance that I can't seem to shake. While I understand and support the idea behind the feature (affinity and anti-affinity scheduling hints), I can't help but feel the implementation is half-baked and results in a very awkward user experience. I agree with all you said about this. Proposal I propose to scrap the server groups API entirely and replace it with a simpler way to accomplish the same basic thing. Create two new options to nova boot: --near-tag TAG and --not-near-tag TAG The first would tell the scheduler to place the new VM near other VMs having a particular tag. The latter would tell the scheduler to place the new VM *not* near other VMs with a particular tag. What is a tag? Well, currently, since the Compute API doesn't have a concept of a single string tag, the tag could be a key=value pair that would be matched against the server extra properties. You can actually already achieve this behavior… although with a little more work. There’s the Affinty filter which allows you to specify a same_host/different_host scheduler hint where you explicitly specify the instance uuids you want… (the extra work is having to know the instance uuids). But yeah, I think this makes more sense to me. - Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
- Original Message - 5. The act of applying affinity policies is now directly related to the act of launching instances, which is where it should be. Well, that may be another discussion :). Something I have been pondering of late is the intersection of the server groups feature with the find host and evacuate instance proposal that allows evacuation without specifying a target host - instead re-scheduling the instance [1]. User expectations, I think, in this case would be that the group policies (however implemented) would stay with the VM throughout its lifecycle (or at least until the policy was explicitly removed/changed) and be taken into account when evacuating in this fashion. With the current model this expectation appears to be met for anti-affinity, but when it comes to affinity I would expect that the scheduler will place the evacuated instance right back where it came from as it doesn't have enough understanding to attempt to evacuate the entire group at once. -Steve [1] https://blueprints.launchpad.net/nova/+spec/find-host-and-evacuate-instance ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
Hi Jay, I'm going to disagree with you on this one, because: i) This is a feature that was discussed in at least one if not two Design Summits and went through a long review period, it wasn't one of those changes that merged in 24 hours before people could take a good look at it. Whatever you feel about the implementation, it is now in the API and we should assume that people have started coding against it. I don't think it gives any credibility to Openstack as a platform if we yank features back out just after they've landed. ii) Sever Group - It's a way of defining a group of servers, and the initial thing (only thing right now) you can define for such a group is the affinity or anti-affinity for scheduling. Maybe in time we'll add other group properties or operations - like delete all the servers in a group (I know some QA folks that would love to have that feature). I don't see why it shouldn't be possible to have a server group that doesn't have a scheduling policy associated to it. I don't see any Cognitive dissonance here - I think your just assuming that the only reason for being able to group servers is for scheduling. iii) If the issue is that you can't add or remove servers from a group, then why don't we add those operations to the API (you could add a server to a group providing doing so doesn't break any policy that might be associated with the group). Seems like a useful addition to me. iv) Since the user created the group, and chose a name for it that is presumably meaningful, then I don't understand why you think --group XXX isn't going to be meaningful to that same user ? So I think there are a bunch of API operations missing, but I don't see any advantage in throwing away what's now in place and replacing it with a tag mechanism that basically says everything with this tag is in a sort of group. Cheers, Phil PS: Congrats on the TC election -Original Message- From: Jay Pipes [mailto:jaypi...@gmail.com] Sent: 25 April 2014 22:16 To: OpenStack Development Mailing List Subject: [openstack-dev] [nova] Proposal: remove the server groups feature Hi Stackers, When recently digging in to the new server group v3 API extension introduced in Icehouse, I was struck with a bit of cognitive dissonance that I can't seem to shake. While I understand and support the idea behind the feature (affinity and anti-affinity scheduling hints), I can't help but feel the implementation is half-baked and results in a very awkward user experience. The use case here is very simple: Alice wants to launch an instance and make sure that the instance does not land on a compute host that contains other instances of that type. The current user experience is that the user creates a server group like so: nova server-group-create $GROUP_NAME --policy=anti-affinity and then, when the user wishes to launch an instance and make sure it doesn't land on a host with another of that instance type, the user does the following: nova boot --group $GROUP_UUID ... There are myriad problems with the above user experience and implementation. Let me explain them. 1. The user isn't creating a server group when they issue a nova server- group-create call. They are creating a policy and calling it a group. Cognitive dissonance results from this mismatch. 2. There's no way to add an existing server to this group. What this means is that the user needs to effectively have pre-considered their environment and policy before ever launching a VM. To realize why this is a problem, consider the following: - User creates three VMs that consume high I/O utilization - User then wants to launch three more VMs of the same kind and make sure they don't end up on the same hosts as the others No can do, since the first three VMs weren't started using a --group scheduler hint. 3. There's no way to remove members from the group 4. There's no way to manually add members to the server group 5. The act of telling the scheduler to place instances near or away from some other instances has been hidden behind the server group API, which means that users doing a nova help boot will see a --group option that doesn't make much sense, as it doesn't describe the scheduling policy activity. Proposal I propose to scrap the server groups API entirely and replace it with a simpler way to accomplish the same basic thing. Create two new options to nova boot: --near-tag TAG and --not-near-tag TAG The first would tell the scheduler to place the new VM near other VMs having a particular tag. The latter would tell the scheduler to place the new VM *not* near other VMs with a particular tag. What is a tag? Well, currently, since the Compute API doesn't have a concept of a single string tag, the tag could be a key=value pair that would be matched against the server extra properties. Once a real
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
On Fri, 2014-04-25 at 22:00 +, Day, Phil wrote: Hi Jay, I'm going to disagree with you on this one, because: No worries, Phil, I expected some dissention and I completely appreciate your feedback and perspective :) i) This is a feature that was discussed in at least one if not two Design Summits and went through a long review period, it wasn't one of those changes that merged in 24 hours before people could take a good look at it. Completely understood. That still doesn't mean we can't propose to get rid of it early instead of letting it sit around when an alternate implementation would be better for the user of OpenStack. Whatever you feel about the implementation, it is now in the API and we should assume that people have started coding against it. Sure, maybe. AFAIK, it's only in the v2 API, though, not in the v3 API (sorry, I made a mistake about that in my original email). Is there a reason it wasn't added to the v3 API? I don't think it gives any credibility to Openstack as a platform if we yank features back out just after they've landed. Perhaps not, though I think we have less credibility if we don't recognize when a feature isn't implemented with users in mind and leave it in the code base to the detriment and confusion of users. We absolutely must, IMO, as a community, be able to say this isn't right and have a path for changing or removing something. If that path is deprecation vs outright removal, so be it, I'd be cool with that. I'd just like to nip this anti-feature in the bud early so that it doesn't become the next feature like file-injection to persist in Nova well after its time has come and passed. ii) Sever Group - It's a way of defining a group of servers, and the initial thing (only thing right now) you can define for such a group is the affinity or anti-affinity for scheduling. We already had ways of defining groups of servers. This new feature doesn't actually define a group of servers. It defines a policy, which is not particularly useful, as it's something that is better specified at the time of launching. Maybe in time we'll add other group properties or operations - like delete all the servers in a group (I know some QA folks that would love to have that feature). We already have the ability to define a group of servers using key=value tags. Deleting all servers in a group is a three-line bash script that loops over the results of a nova list command and calls nova delete. Trust me, I've done group deletes in this way many times. I don't see why it shouldn't be possible to have a server group that doesn't have a scheduling policy associated to it. I don't think the grouping of servers should have *anything* to do with scheduling :) That's the point of my proposal. Servers can and should be grouped using simple tags or key=value pair tags. The grouping of servers together doesn't have anything of substance to do with scheduling policies. I don't see any Cognitive dissonance here - I think your just assuming that the only reason for being able to group servers is for scheduling. Again, I don't think scheduling and grouping of servers has anything to do with each other, thus my proposal to remove the relationship between groups of servers and scheduling policies, which is what the existing server group API and implementation does. iii) If the issue is that you can't add or remove servers from a group, then why don't we add those operations to the API (you could add a server to a group providing doing so doesn't break any policy that might be associated with the group). We already have this ability today, thus my proposal to get rid of server groups. Seems like a useful addition to me. It's an addition that isn't needed, as we already have this today. iv) Since the user created the group, and chose a name for it that is presumably meaningful, then I don't understand why you think --group XXX isn't going to be meaningful to that same user ? See point above about removing the unnecessary relationship between grouping of servers and scheduling policies. So I think there are a bunch of API operations missing, but I don't see any advantage in throwing away what's now in place and replacing it with a tag mechanism that basically says everything with this tag is in a sort of group. We already have the tag group mechanism in place, that's kind of what I've been saying... PS: Congrats on the TC election Cheers, appreciated. Looking forward to healthy debates and buying you a few beers at the summit :) best, -jay -Original Message- From: Jay Pipes [mailto:jaypi...@gmail.com] Sent: 25 April 2014 22:16 To: OpenStack Development Mailing List Subject: [openstack-dev] [nova] Proposal: remove the server groups feature Hi Stackers, When recently digging in to the new server group v3 API extension introduced in Icehouse, I was struck with a bit of cognitive
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
On 04/25/2014 05:15 PM, Jay Pipes wrote: Thoughts and feedback welcome, I'd love to talk about how to improve this functionality, while still allowing for future policy expansion. Can we punt this discussion to summit though? I don't think we need much time. Perhaps over a break or something. I'd like to be involved with this discussion and decision, but I'm out for the next month or so (but will be at the summit). -- Russell Bryant ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Proposal: remove the server groups feature
Jay Pipes jaypi...@gmail.com wrote on 04/25/2014 06:28:38 PM: On Fri, 2014-04-25 at 22:00 +, Day, Phil wrote: Hi Jay, I'm going to disagree with you on this one, because: No worries, Phil, I expected some dissention and I completely appreciate your feedback and perspective :) I myself sit between the two camps on this one. I share Jay's unhappiness with server groups, as they are today. However, I see an evolutionary path forward from today's server groups to something that makes much more sense to me and my colleagues. I do not see as clear a path forward from Jay's proposal, but am willing to think more about that. I will start by outlining where I want to go, and then address the specific points that have been raised in this email thread so far. I would like to see the OpenStack architecture have a place for what I have been calling holistic scheduling. That is making a simultaneous scheduling decision about a whole collection of virtual resources of various types (Nova VMs, Cinder storage volumes, network bandwidth, ...), taking into account a rich composite policy statement. This is not just a pipe dream, my group has been doing this for years. What we are struggling with is finding an evolutionary path to a place where it can be done in an OpenStack context. One part of the struggle is due to the fact that in our previous work the part that is analogous to Heat is not optional, while in OpenStack Heat is most definitely optional. Some of the things I have written in the past have not clearly separated scheduling and Heat and left Heat optional, but please rest assured that I am making no proposal now to violate those things. I see scheduling and orchestration as distinct functions; the potential for confusion arises because (a) holistic scheduling needs input that has some similarity to what you see in a Heat template today and (b) making the scheduling simultaneous requires moving it from its current place (downstream from orchestration) to an earlier place (upstream from orchestration). The OpenStack community has historically used the word scheduling to refer to placement problems, always in the time-invariant now-and-forseeable future, and I am following that usage here. Other communities consider scheduling to also include interesting variation over time, but I am not trying to bring that into this debate. (Nor am I denying its interest and value, I am just trying to keep this discussion focused.) The discussion in this email thread has recognized that scheduler hints are applied only at creation time today, but it has already been noted (e.g., in http://summit.openstack.org/cfp/details/99) that scheduling policy statements should be retained for the lifetime of the virtual resource. That is true regardless of whether the policy statements come in through today's server groups, the alternate proposal from Jay Pipes, or some other alternative or evolution. I agree with Jay that groups have no inherent connection to scheduling. My colleagues and I have found grouping to be a useful technique to make APIs and documents more concise, and we find a top-level group to be the natural scope for a simultaneous decision. We have been working example problems with a non-trivial size and amount of structure; when you get beyond small simple examples you see the usefulness of grouping more clearly. For a couple of examples, see a 3-tier web application in https://docs.google.com/drawings/d/1nridrUUwNaDrHQoGwSJ_KXYC7ik09wUuV3vXw1MyvlY and a deployment of an IBM product called Connections in https://docs.google.com/file/d/0BypF9OutGsW3ZUYwYkNjZGJFejQ (this latter example has been shorn of its networking policies, and is a literal abstract of something we did using software that could not cope with policies applied directly to virtual resources, so some of its groups are not well motivated --- but others *are*). The groups are handy for making it possible to draw pictures without too many lines, and write documents that are readably concise. But everything said with groups could be said without groups, if we allowed policy statements to be placed on virtual resources and on pairs of virtual resources --- it would just take a heck of a lot more policy statements. If you want to make a simultaneous decision about several virtual resources, you need a description of all those virtual resources up-front. So even in a totally Heat-free environment you find yourself wanting something that looks like a document or data structure describing multiple virtual resources --- and the policies that apply to them, and thus also the groups that allow for concise applications of policies; note also that the whole set of virtual resources involved is a group. When you have an example of non-trivial size and structure, you generally do not want to make a change by a collection of atomic edits, each individually scheduled.
