Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
I didn't see any command referring InstanceGroupMemberConnection. What is it exactly? Could you give an example? And how can we create an InstanceGroup? 1) Create an empty group 2) Add policy, metadata 3) Add group instances ... ? or in the InstanceGroup POST message there is already a description of all InstanceGroupMembers, Connections, etc ? An (raw) example would be really helpful to understand the proposition. Best regards, Toan - Original Message - From: "Mike Spreitzer" To: "Yathiraj Udupi (yudupi)" Cc: "OpenStack Development Mailing List" Sent: Wednesday, October 23, 2013 5:36:25 AM Subject: Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft "Yathiraj Udupi (yudupi)" wrote on 10/15/2013 03:08:32 AM: > I have made some edits to the document: https://docs.google.com/ > document/d/17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?pli=1# > ... One other minor thing to discuss in the modeling is metadata. I am not eager to totally gorp up the model, but shouldn't all sorts of things allow metadata? Thanks, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
"Yathiraj Udupi (yudupi)" wrote on 10/15/2013 03:08:32 AM: > I have made some edits to the document: https://docs.google.com/ > document/d/17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?pli=1# > ... One other minor thing to discuss in the modeling is metadata. I am not eager to totally gorp up the model, but shouldn't all sorts of things allow metadata? Thanks, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
I have made some edits to the document:
https://docs.google.com/document/d/17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?pli=1#
by updating the Instance Group Model and APIs based on the recent mailing list
discussion below and also about the Policy Model in another email thread. An
action item is to provide some examples of request and response to the REST
apis. But they will be based on enhancements and reuse of some examples
already provided here: https://review.openstack.org/#/c/30028/22
This API proposal is still under discussion and work-in-progress, and will
definitely be a good session topic to finalize this proposal.
Regards,
Yathi.
From: Yathiraj Udupi mailto:yud...@cisco.com>>
Date: Monday, October 14, 2013 10:17 AM
To: Mike Spreitzer mailto:mspre...@us.ibm.com>>, Debojyoti
Dutta mailto:ddu...@gmail.com>>, garyk
mailto:gkot...@vmware.com>>
Cc: OpenStack Development Mailing List
mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement -
Updated Instance Group Model and API extension model - WIP Draft
Hi Mike,
I read your email where you expressed concerns regarding create-time
dependencies, and I agree they are valid concerns to be addressed. But like we
all agree, as a starting point, we are just focusing on the APIs for now, and
will leave that aside as implementation details to be addressed later.
Thanks for sharing your suggestions on how we can simplify the APIs. I think
we are getting closer to finalizing this one.
Let us start at the model proposed here -
[1]
https://docs.google.com/document/d/17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?usp=sharing
(Ignore the white diamonds - they will be black, when I edit the doc)
The InstanceGroup represents all the information necessary to capture the group
- nodes, edges, policies, and metadata
InstanceGroupMember - is a reference to an Instance, which is saved separately,
using the existing Instance Model in Nova.
InstanceGroupMemberConnection - represents the edge
InstanceGroupPolicy is a reference to a Policy, which will also be saved
separately, (currently not existing in the model, but has to be created). Here
in the Policy model, I don't mind adding any number of additional fields, and
key-value pairs to be able to fully define a policy. I guess a Policy-metadata
dictionary is sufficient to capture all the required arguments.
The InstanceGroupPolicy will be associated to a group as a whole or an edge.
InstanceGroupMetadata - represents key-value dictionary for any additional
metadata for the instance group.
I think this should fully support what we care about - nodes, edges, policies
and metadata.
Do we all agree ?
Now going to the APIs,
Register GROUP API (from my doc [1]):
POST
/v3.0/{tenant_id}/groups
Register a group
I think the confusion is only about when the member (all nested members) and
policy about when they are saved in the DB (registered, but not CREATED
actually), such that we can associate a UUID. This led to my original thinking
that it is a 3-phase operation where we have to register (save in DB) the
nested members first, then register the group as a whole. But this is not
client friendly.
Like I had suggested earlier, as an implementation detail of the Group
registration API (CREATE part 1 in your terminology), we can support this: as
part of the group registration transaction, complete the registration of the
nested members, get their UUIDs, create the InstanceGroupMemberConnections,
and then complete saving the group - resulting in a UUID for the group, all in
a single transaction-scope. While you start the transaction, you can start
with a UUID for the group, so that you can add the group_id pointers to the
individual members, and then finally complete the transaction.
This means, you provide as an input to the REST API - the complete nested
tree, including all the details about the nested members and policies, and the
register API, will handle the saving of all the individual objects required.
But I think it does help to also add additional APIs to just register an
InstanceGroupMember and an InstanceGroupPolicy separately. This might help
the client while creating a group, rather than giving the entire nested tree.
(his makes it a 3-phase) This API will support adding members and policies to
an instance group that is created. (you can start with an empty group)
POST
/v3.0/{tenant_id}/groups/instance
Register an instance belonging to an instancegroup
POST
/v3.0/{tenant_id}/groups/policy
Register a policy belonging to an instance group
Are we okay with this ?
The next API - is the actual creation of the resources. (CREATE part 2 in
your terminology). This is was my create API in the doc-
POST
/v3.0/{tenant_id}/groups/{id}/create
Create and schedule an Instance group
This is just the API proposal, the und
Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
That came through beautifully formatted to me, but it looks much worse in
the archive. I'm going to use crude email tech here, so that I know it
won't lose anything in handling.
"Yathiraj Udupi (yudupi)" wrote on 10/14/2013 01:17:47
PM:
> I read your email where you expressed concerns regarding create-time
> dependencies, and I agree they are valid concerns to be addressed.
> But like we all agree, as a starting point, we are just focusing on
> the APIs for now, and will leave that aside as implementation
> details to be addressed later.
I am not sure I understand your language here. To me, design decisions
that affect what calls the clients make are not implementation details,
they are part of the API design.
> Thanks for sharing your suggestions on how we can simplify the APIs.
> I think we are getting closer to finalizing this one.
>
> Let us start at the model proposed here -
> [1] https://docs.google.com/document/d/
> 17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?usp=sharing
> (Ignore the white diamonds - they will be black, when I edit the doc)
>
> The InstanceGroup represents all the information necessary to
> capture the group - nodes, edges, policies, and metadata
>
> InstanceGroupMember - is a reference to an Instance, which is saved
> separately, using the existing Instance Model in Nova.
I think you mean this is a reference to either a group or an individual
Compute instance.
>
> InstanceGroupMemberConnection - represents the edge
>
> InstanceGroupPolicy is a reference to a Policy, which will also be
> saved separately, (currently not existing in the model, but has to
> be created). Here in the Policy model, I don't mind adding any
> number of additional fields, and key-value pairs to be able to fully
> define a policy. I guess a Policy-metadata dictionary is sufficient
> to capture all the required arguments.
> The InstanceGroupPolicy will be associated to a group as a whole or an
edge.
Like I said under separate cover, I think one of these is a policy *use*
rather than a policy *definition*. I go further and emphasize that the
interesting out-of-scope definitions are of policy *types*. A policy type
takes parameters. For example, policies of the anti-collocation (AKA
anti-affinity) type have a parameter that specifies the level in the
physical hierarchy where the location must differ (rack, host, ...). Each
policy type specifies a set of parameters, just like a procedure specifies
parameters; each use of a policy type supplies values for the parameters,
just like a procedure invocation supplies values for the procedure's
parameters. I suggest separating parameter values from metadata; the
former are described by the policy type, while the latter are unknown to
the policy type and are there for other needs of the client.
Yes, a use of a policy type is associated with a group or an edge. In my
own writing I have suggested a third possibility: that a policy use can be
directly associated with an individual resource. It just so happens that
the code my group already has been running also has your restriction: it
supports only policies associated with groups and relationships. But I
suggested allowing direct attachment to resources (as well as
relationships also being able to directly reference resources instead of
groups) because I think this restriction --- while it simplifies
implementation --- makes templates more verbose; I felt the latter was a
more important consideration than the former. If you want to roadmap this
--- restricted first, liberal later --- that's fine with me.
>
> InstanceGroupMetadata - represents key-value dictionary for any
> additional metadata for the instance group.
>
> I think this should fully support what we care about - nodes, edges,
> policies and metadata.
>
> Do we all agree ?
Yes, with exceptions noted above.
>
> Now going to the APIs,
>
> Register GROUP API (from my doc [1]):
>
> POST /v3.0/{tenant_id}/groups --- Register a group
In such specs it would be good to be explicit about the request parameters
and body. If I follow correctly,
https://review.openstack.org/#/c/30028/25/doc/api_samples/os-instance-groups/instance-groups-post-req.json
shows us that you intended (as of that patch) the body to carry a group
definition.
> I think the confusion is only about when the member (all nested
> members) and policy about when they are saved in the DB (registered,
> but not CREATED actually), such that we can associate a UUID. This
> led to my original thinking that it is a 3-phase operation where we
> have to register (save in DB) the nested members first, then
> register the group as a whole. But this is not client friendly.
>
> Like I had suggested earlier, as an implementation detail of the
> Group registration API (CREATE part 1 in your terminology), we can
> support this: as part of the group registration transaction,
> complete the registration of the nest
Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
Hi Mike,
I read your email where you expressed concerns regarding create-time
dependencies, and I agree they are valid concerns to be addressed. But like we
all agree, as a starting point, we are just focusing on the APIs for now, and
will leave that aside as implementation details to be addressed later.
Thanks for sharing your suggestions on how we can simplify the APIs. I think
we are getting closer to finalizing this one.
Let us start at the model proposed here -
[1]
https://docs.google.com/document/d/17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?usp=sharing
(Ignore the white diamonds - they will be black, when I edit the doc)
The InstanceGroup represents all the information necessary to capture the group
- nodes, edges, policies, and metadata
InstanceGroupMember - is a reference to an Instance, which is saved separately,
using the existing Instance Model in Nova.
InstanceGroupMemberConnection - represents the edge
InstanceGroupPolicy is a reference to a Policy, which will also be saved
separately, (currently not existing in the model, but has to be created). Here
in the Policy model, I don't mind adding any number of additional fields, and
key-value pairs to be able to fully define a policy. I guess a Policy-metadata
dictionary is sufficient to capture all the required arguments.
The InstanceGroupPolicy will be associated to a group as a whole or an edge.
InstanceGroupMetadata - represents key-value dictionary for any additional
metadata for the instance group.
I think this should fully support what we care about - nodes, edges, policies
and metadata.
Do we all agree ?
Now going to the APIs,
Register GROUP API (from my doc [1]):
POST
/v3.0/{tenant_id}/groups
Register a group
I think the confusion is only about when the member (all nested members) and
policy about when they are saved in the DB (registered, but not CREATED
actually), such that we can associate a UUID. This led to my original thinking
that it is a 3-phase operation where we have to register (save in DB) the
nested members first, then register the group as a whole. But this is not
client friendly.
Like I had suggested earlier, as an implementation detail of the Group
registration API (CREATE part 1 in your terminology), we can support this: as
part of the group registration transaction, complete the registration of the
nested members, get their UUIDs, create the InstanceGroupMemberConnections,
and then complete saving the group - resulting in a UUID for the group, all in
a single transaction-scope. While you start the transaction, you can start
with a UUID for the group, so that you can add the group_id pointers to the
individual members, and then finally complete the transaction.
This means, you provide as an input to the REST API - the complete nested
tree, including all the details about the nested members and policies, and the
register API, will handle the saving of all the individual objects required.
But I think it does help to also add additional APIs to just register an
InstanceGroupMember and an InstanceGroupPolicy separately. This might help
the client while creating a group, rather than giving the entire nested tree.
(his makes it a 3-phase) This API will support adding members and policies to
an instance group that is created. (you can start with an empty group)
POST
/v3.0/{tenant_id}/groups/instance
Register an instance belonging to an instancegroup
POST
/v3.0/{tenant_id}/groups/policy
Register a policy belonging to an instance group
Are we okay with this ?
The next API - is the actual creation of the resources. (CREATE part 2 in
your terminology). This is was my create API in the doc-
POST
/v3.0/{tenant_id}/groups/{id}/create
Create and schedule an Instance group
This is just the API proposal, the underlying implementation details will
involve all the required logic to ensure the creation of all the group members.
Here like you also suggest, as a starting point, we can try to first use
existing Nova mechanisms to create the members of this group. Eventually we
will need to get to the discussions for scheduling this entire group as a
whole, which covers cross-services support like I discuss in the unified
resource placement document -
https://docs.google.com/document/d/1IiPI0sfaWb1bdYiMWzAAx0HYR6UqzOan_Utgml5W1HI/edit?pli=1
Let me know your thoughts on this.
Thanks,
Yathi.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
I'll be at the summit too. Available Nov 4 if we want to do some prep then. It will be my first summit, I am not sure how overbooked my summit time will be. Regards, Mike From: Sylvain Bauza To: OpenStack Development Mailing List , Cc: Mike Spreitzer/Watson/IBM@IBMUS Date: 10/11/2013 08:19 AM Subject:Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft Long-story short, sounds like we do have the same concerns here in Climate. I'll be present at the Summit, any chance to do an unconference meeting in between all parties ? Thanks, -Sylvain Le 11/10/2013 08:25, Mike Spreitzer a écrit : Regarding Alex's question of which component does holistic infrastructure scheduling, I hesitate to simply answer "heat". Heat is about orchestration, and infrastructure scheduling is another matter. I have attempted to draw pictures to sort this out, see https://docs.google.com/drawings/d/1Y_yyIpql5_cdC8116XrBHzn6GfP_g0NHTTG_W4o0R9U and https://docs.google.com/drawings/d/1TCfNwzH_NBnx3bNz-GQQ1bRVgBpJdstpu0lH _TONw6g . In those you will see that I identify holistic infrastructure scheduling as separate functionality from infrastructure orchestration (the main job of today's heat engine) and also separate from software orchestration concerns. However, I also see a close relationship between holistic infrastructure scheduling and heat, as should be evident in those pictures too. Alex made a remark about the needed inputs, and I agree but would like to expand a little on the topic. One thing any scheduler needs is knowledge of the amount, structure, and capacity of the hosting thingies (I wish I could say "resources", but that would be confusing) onto which the workload is to be scheduled. Scheduling decisions are made against available capacity. I think the most practical way to determine available capacity is to separately track raw capacity and current (plus already planned!) allocations from that capacity, finally subtracting the latter from the former. In Nova, for example, sensing raw capacity is handled by the various nova-compute agents reporting that information. I think a holistic infrastructure scheduler should get that information from the various individual services (Nova, Cinder, etc) that it is concerned with (presumably they have it anyway). A holistic infrastructure scheduler can keep track of the allocations it has planned (regardless of whether they have been executed yet). However, there may also be allocations that did not originate in the holistic infrastructure scheduler. The individual underlying services should be able to report (to the holistic infrastructure scheduler, even if lowly users are not so authorized) all the allocations currently in effect. An accurate union of the current and planned allocations is what we want to subtract from raw capacity to get available capacity. If there is a long delay between planning and executing an allocation, there can be nasty surprises from competitors --- if there are any competitors. Actually, there can be nasty surprises anyway. Any scheduler should be prepared for nasty surprises, and react by some sensible retrying. If nasty surprises are rare, we are pretty much done. If nasty surprises due to the presence of competing managers are common, we may be able to combat the problem by changing the long delay to a short one --- by moving the allocation execution earlier into a stage that is only about locking in allocations, leaving all the other work involved in creating virtual resources to later (perhaps Climate will be good for this). If the delay between planning and executing an allocation is short and there are many nasty surprises due to competing managers, then you have too much competition between managers --- don't do that. Debo wants a simpler nova-centric story. OK, how about the following. This is for the first step in the roadmap, where scheduling decisions are still made independently for each VM instance. For the client/service interface, I think we can do this with a simple clean two-phase interface when traditional software orchestration is in play, a one-phase interface when slick new software orchestration is used. Let me outline the two-phase flow. We extend the Nova API with CRUD operations on VRTs (top-level groups). For example, the CREATE operation takes a definition of a top-level group and all its nested groups, definitions (excepting stuff like userdata) of all the resources (only VM instances, for now) contained in those groups, all the relationships among those groups/resources, and all the applications of policy to those groups, resources, and relationships. This is a rest-style interface; the CREATE operation takes a definition of the thing (a top-level group and all that it
Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
Long-story short, sounds like we do have the same concerns here in Climate. I'll be present at the Summit, any chance to do an unconference meeting in between all parties ? Thanks, -Sylvain Le 11/10/2013 08:25, Mike Spreitzer a écrit : Regarding Alex's question of which component does holistic infrastructure scheduling, I hesitate to simply answer "heat". Heat is about orchestration, and infrastructure scheduling is another matter. I have attempted to draw pictures to sort this out, see https://docs.google.com/drawings/d/1Y_yyIpql5_cdC8116XrBHzn6GfP_g0NHTTG_W4o0R9Uand https://docs.google.com/drawings/d/1TCfNwzH_NBnx3bNz-GQQ1bRVgBpJdstpu0lH_TONw6g. In those you will see that I identify holistic infrastructure scheduling as separate functionality from infrastructure orchestration (the main job of today's heat engine) and also separate from software orchestration concerns. However, I also see a close relationship between holistic infrastructure scheduling and heat, as should be evident in those pictures too. Alex made a remark about the needed inputs, and I agree but would like to expand a little on the topic. One thing any scheduler needs is knowledge of the amount, structure, and capacity of the hosting thingies (I wish I could say "resources", but that would be confusing) onto which the workload is to be scheduled. Scheduling decisions are made against available capacity. I think the most practical way to determine available capacity is to separately track raw capacity and current (plus already planned!) allocations from that capacity, finally subtracting the latter from the former. In Nova, for example, sensing raw capacity is handled by the various nova-compute agents reporting that information. I think a holistic infrastructure scheduler should get that information from the various individual services (Nova, Cinder, etc) that it is concerned with (presumably they have it anyway). A holistic infrastructure scheduler can keep track of the allocations it has planned (regardless of whether they have been executed yet). However, there may also be allocations that did not originate in the holistic infrastructure scheduler. The individual underlying services should be able to report (to the holistic infrastructure scheduler, even if lowly users are not so authorized) all the allocations currently in effect. An accurate union of the current and planned allocations is what we want to subtract from raw capacity to get available capacity. If there is a long delay between planning and executing an allocation, there can be nasty surprises from competitors --- if there are any competitors. Actually, there can be nasty surprises anyway. Any scheduler should be prepared for nasty surprises, and react by some sensible retrying. If nasty surprises are rare, we are pretty much done. If nasty surprises due to the presence of competing managers are common, we may be able to combat the problem by changing the long delay to a short one --- by moving the allocation execution earlier into a stage that is only about locking in allocations, leaving all the other work involved in creating virtual resources to later (perhaps Climate will be good for this). If the delay between planning and executing an allocation is short and there are many nasty surprises due to competing managers, then you have too much competition between managers --- don't do that. Debo wants a simpler nova-centric story. OK, how about the following. This is for the first step in the roadmap, where scheduling decisions are still made independently for each VM instance. For the client/service interface, I think we can do this with a simple clean two-phase interface when traditional software orchestration is in play, a one-phase interface when slick new software orchestration is used. Let me outline the two-phase flow. We extend the Nova API with CRUD operations on VRTs (top-level groups). For example, the CREATE operation takes a definition of a top-level group and all its nested groups, definitions (excepting stuff like userdata) of all the resources (only VM instances, for now) contained in those groups, all the relationships among those groups/resources, and all the applications of policy to those groups, resources, and relationships. This is a rest-style interface; the CREATE operation takes a definition of the thing (a top-level group and all that it contains) being created; the UPDATE operation takes a revised definition of the whole thing. Nova records the presented information; the familiar stuff is stored essentially as it is today (but marked as being in some new sort of tentative state), and the grouping, relationship, and policy stuff is stored according to a model like the one Debo&Yathi wrote. The CREATE operation returns a UUID for the newly created top-level group. The invocation of the top-level group CRUD is a single operation and it is t
Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
Regarding Alex's question of which component does holistic infrastructure scheduling, I hesitate to simply answer "heat". Heat is about orchestration, and infrastructure scheduling is another matter. I have attempted to draw pictures to sort this out, see https://docs.google.com/drawings/d/1Y_yyIpql5_cdC8116XrBHzn6GfP_g0NHTTG_W4o0R9U and https://docs.google.com/drawings/d/1TCfNwzH_NBnx3bNz-GQQ1bRVgBpJdstpu0lH_TONw6g . In those you will see that I identify holistic infrastructure scheduling as separate functionality from infrastructure orchestration (the main job of today's heat engine) and also separate from software orchestration concerns. However, I also see a close relationship between holistic infrastructure scheduling and heat, as should be evident in those pictures too. Alex made a remark about the needed inputs, and I agree but would like to expand a little on the topic. One thing any scheduler needs is knowledge of the amount, structure, and capacity of the hosting thingies (I wish I could say "resources", but that would be confusing) onto which the workload is to be scheduled. Scheduling decisions are made against available capacity. I think the most practical way to determine available capacity is to separately track raw capacity and current (plus already planned!) allocations from that capacity, finally subtracting the latter from the former. In Nova, for example, sensing raw capacity is handled by the various nova-compute agents reporting that information. I think a holistic infrastructure scheduler should get that information from the various individual services (Nova, Cinder, etc) that it is concerned with (presumably they have it anyway). A holistic infrastructure scheduler can keep track of the allocations it has planned (regardless of whether they have been executed yet). However, there may also be allocations that did not originate in the holistic infrastructure scheduler. The individual underlying services should be able to report (to the holistic infrastructure scheduler, even if lowly users are not so authorized) all the allocations currently in effect. An accurate union of the current and planned allocations is what we want to subtract from raw capacity to get available capacity. If there is a long delay between planning and executing an allocation, there can be nasty surprises from competitors --- if there are any competitors. Actually, there can be nasty surprises anyway. Any scheduler should be prepared for nasty surprises, and react by some sensible retrying. If nasty surprises are rare, we are pretty much done. If nasty surprises due to the presence of competing managers are common, we may be able to combat the problem by changing the long delay to a short one --- by moving the allocation execution earlier into a stage that is only about locking in allocations, leaving all the other work involved in creating virtual resources to later (perhaps Climate will be good for this). If the delay between planning and executing an allocation is short and there are many nasty surprises due to competing managers, then you have too much competition between managers --- don't do that. Debo wants a simpler nova-centric story. OK, how about the following. This is for the first step in the roadmap, where scheduling decisions are still made independently for each VM instance. For the client/service interface, I think we can do this with a simple clean two-phase interface when traditional software orchestration is in play, a one-phase interface when slick new software orchestration is used. Let me outline the two-phase flow. We extend the Nova API with CRUD operations on VRTs (top-level groups). For example, the CREATE operation takes a definition of a top-level group and all its nested groups, definitions (excepting stuff like userdata) of all the resources (only VM instances, for now) contained in those groups, all the relationships among those groups/resources, and all the applications of policy to those groups, resources, and relationships. This is a rest-style interface; the CREATE operation takes a definition of the thing (a top-level group and all that it contains) being created; the UPDATE operation takes a revised definition of the whole thing. Nova records the presented information; the familiar stuff is stored essentially as it is today (but marked as being in some new sort of tentative state), and the grouping, relationship, and policy stuff is stored according to a model like the one Debo&Yathi wrote. The CREATE operation returns a UUID for the newly created top-level group. The invocation of the top-level group CRUD is a single operation and it is the first of the two phases. In the second phase of a CREATE flow, the client creates individual resources with the same calls as are used today, except that each VM instance create call is augmented with a pointer into the policy information. T
Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
Alex, agree with your comments. I think we need to think of both 1.
and 2. as the eventual outcome and the destination. IF we decide to
improve upon scheduling/polices at the heat level, that should be a
very nice and independent endeavor and we can all learn from it. I
dont think we can design this all upfront.
IMO the simple enough road is to do
1. simple resource group extension and show how it can be used to
specify groups of resources - no matter what you do on top of this,
you will need to specify groups of resources (e.g. API proposal from
Yathi/Garyk/Mike/Me)
2. have policies which can be simple scheduler hints for now
3. have some notion of intelligent scheduling (a simple example is a
solver scheduler)
4. have some notion of fast state management (like Boris' proposal)
Ref:
[api]
https://docs.google.com/document/d/17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit
[overall]
https://docs.google.com/document/d/1IiPI0sfaWb1bdYiMWzAAx0HYR6UqzOan_Utgml5W1HI/edit
debo
Mike: I agree with most of your ideas of extensions to
heat/policies/ordering/dependencies etc but I wish we could start with
a simple API from the nova side that will grow into a cross services
thing while you could start from the Heat side and then eventually
come to teh same midpoint. I somehow feel we are almost there wrt the
1st cut of the API \cite{api}.
On Wed, Oct 9, 2013 at 11:11 PM, Alex Glikson wrote:
> Thanks for the pointer -- was not able to attend that meeting,
> unfortunately. Couple of observations, based on what I've heard till now.
> 1. I think it is important not to restrict the discussion to Nova resources.
> So, I like the general direction in [1] to target a generic mechanism and
> API. However, once we start following that path, it becomes more challenging
> to figure out which component should manage those cross-resource constructs
> (Heat sounds like a reasonable candidate -- which seems consistent with the
> proposal at [2]), and what should be the API between it and the services
> deciding on the actual placement of individual resources (nova, cinder,
> neutron).
> 2. Moreover, we should take into account that we may need to take into
> consideration multiple sources of topology -- physical (maybe provided by
> Ironic, affecting availability -- hosts, racks, etc), virtual-compute
> (provided by Nova, affecting resource isolation -- mainly hosts),
> virtual-network (affecting connectivity and bandwidth/latency.. think of SDN
> policies enforcing routing and QoS almost orthogonally to physical
> topology), virtual-storage (affecting VM-to-volume connectivity and
> bandwidth/latency.. think of FC network implying topology different than the
> physical one and the IP network one).
>
> I wonder whether we will be able to come up with a simple-enough initial
> approach & implementation, which would not limit the ability to extend &
> customize it going forward to cover all the above.
>
> Regards,
> Alex
>
> [1]
> https://docs.google.com/document/d/17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit
> [2] https://wiki.openstack.org/wiki/Heat/PolicyExtension
>
>
> Alex Glikson
> Manager, Cloud Operating System Technologies, IBM Haifa Research Lab
> http://w3.haifa.ibm.com/dept/stt/cloud_sys.html |
> http://www.research.ibm.com/haifa/dept/stt/cloud_sys.shtml
> Email: glik...@il.ibm.com | Phone: +972-4-8281085 | Mobile: +972-54-647
> | Fax: +972-4-8296112
>
>
>
>
> From: Mike Spreitzer
> To: OpenStack Development Mailing List
> ,
> Date: 10/10/2013 07:59 AM
> Subject:Re: [openstack-dev] [scheduler] APIs for Smart Resource
> Placement - Updated Instance Group Model and API extension model - WIP Draft
>
>
>
>
> Yes, there is more than the northbound API to discuss. Gary started us
> there in the Scheduler chat on Oct 1, when he broke the issues down like
> this:
>
> 11:12:22 AM garyk: 1. a user facing API
> 11:12:41 AM garyk: 2. understanding which resources need to be tracked
> 11:12:48 AM garyk: 3. backend implementation
>
> The full transcript is at
> http://eavesdrop.openstack.org/meetings/scheduling/2013/scheduling.2013-10-01-15.08.log.html
>
> Alex Glikson wrote on 10/09/2013 02:14:03 AM:
>>
>> Good summary. I would also add that in A1 the schedulers (e.g., in
>> Nova and Cinder) could talk to each other to coordinate. Besides
>> defining the policy, and the user-facing APIs, I think we should
>> also outline those cross-component APIs (need to think whether they
>> have to be user-visible, or can be admin).
>>
>> Regards,
>> Alex __
Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
Thanks for the pointer -- was not able to attend that meeting, unfortunately. Couple of observations, based on what I've heard till now. 1. I think it is important not to restrict the discussion to Nova resources. So, I like the general direction in [1] to target a generic mechanism and API. However, once we start following that path, it becomes more challenging to figure out which component should manage those cross-resource constructs (Heat sounds like a reasonable candidate -- which seems consistent with the proposal at [2]), and what should be the API between it and the services deciding on the actual placement of individual resources (nova, cinder, neutron). 2. Moreover, we should take into account that we may need to take into consideration multiple sources of topology -- physical (maybe provided by Ironic, affecting availability -- hosts, racks, etc), virtual-compute (provided by Nova, affecting resource isolation -- mainly hosts), virtual-network (affecting connectivity and bandwidth/latency.. think of SDN policies enforcing routing and QoS almost orthogonally to physical topology), virtual-storage (affecting VM-to-volume connectivity and bandwidth/latency.. think of FC network implying topology different than the physical one and the IP network one). I wonder whether we will be able to come up with a simple-enough initial approach & implementation, which would not limit the ability to extend & customize it going forward to cover all the above. Regards, Alex [1] https://docs.google.com/document/d/17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit [2] https://wiki.openstack.org/wiki/Heat/PolicyExtension Alex Glikson Manager, Cloud Operating System Technologies, IBM Haifa Research Lab http://w3.haifa.ibm.com/dept/stt/cloud_sys.html | http://www.research.ibm.com/haifa/dept/stt/cloud_sys.shtml Email: glik...@il.ibm.com | Phone: +972-4-8281085 | Mobile: +972-54-647 | Fax: +972-4-8296112 From: Mike Spreitzer To: OpenStack Development Mailing List , Date: 10/10/2013 07:59 AM Subject:Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft Yes, there is more than the northbound API to discuss. Gary started us there in the Scheduler chat on Oct 1, when he broke the issues down like this: 11:12:22 AM garyk: 1. a user facing API 11:12:41 AM garyk: 2. understanding which resources need to be tracked 11:12:48 AM garyk: 3. backend implementation The full transcript is at http://eavesdrop.openstack.org/meetings/scheduling/2013/scheduling.2013-10-01-15.08.log.html Alex Glikson wrote on 10/09/2013 02:14:03 AM: > > Good summary. I would also add that in A1 the schedulers (e.g., in > Nova and Cinder) could talk to each other to coordinate. Besides > defining the policy, and the user-facing APIs, I think we should > also outline those cross-component APIs (need to think whether they > have to be user-visible, or can be admin). > > Regards, > Alex ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
Debojyoti Dutta wrote on 10/09/2013 02:48:26 AM: > > Mike, I agree we could have a cleaner API but I am not sure how > cleanly it will integrate with current nova which IMO should be test > we should pass (assuming we do cross services later) I think the cleaner APIs integrate with Nova as well as the three phase API you suggested. Am I missing some obvious impediment? > ... > > To me the most frustrating aspect of this challenge is the need for the > > client to directly mediate the dependencies between resources; this is > > really what is driving us to do ugly things. As I mentioned before, I am > > coming from a setting that does not have this problem. So I am thinking > > about two alternatives: (A1) how clean can we make a system in which the > > client continues to directly mediate dependencies between resources, and > > (A2) how easily and cleanly can we make that problem go away. > > Am a little confused - How is the API dictating either A1 or A2? Isnt > that a function of the implementation of the API. For a moment let us > assume that the black box implementation will be awesome and address > your concerns. I am talking about how the client/service interface, it is not (just) a matter of service implementation. My complaint is that the software orchestration technique commonly used prevents us from having a one-phase API for holistic infrastructure scheduling. The commonly used software orchestration technique requires some serialization of the resource creation calls. For example, if one VM instance runs a database and another VM instance runs a web server that needs to be configured with the private IP address of the database, the common technique is for the client to first create the database VM instance, then take the private IP address from that VM instance and use it to compose the userdata that is passed in the Nova call that creates the web server VM instance. That client can not present all at once a fully concrete and literal specification of both VM instances, because the userdata for one is not knowable until the other has been created. The client has to be able to make create-like calls in some particular order rather than ask for all creation at once. If the client could ask for all creation at once then we could use a one-phase API: it simply takes a specification of the resources along with their policies and relationships. Of course, there is another way out. We do have in OpenStack a technology by which a client can present all at once a specification of many VM instances where the userdata of some depend on the results of creating others. If we were willing to use this technology, we could follow A2. The CREATE flow would go like this: (i) the client presents the specification of resources (including the computations that link some), with grouping, relationships, and policies, to our new API; (ii) our new service registers the new topology and (once we advance this far on the development roadmap) does holistic scheduling; (iii) our new service updates the resource specifications to include pointers into the policy data; (iv) our new service passes the enhanced resource specifications to that other service that can do the creation calls linked by the prescribed computations; (v) that other service does its thing, causing a series (maybe with some allowed parallelism) of creation calls, each augmented by the relevant pointer into the policy information; (vi) the service implementing a creation call gets what it normally does plus the policy pointer, which it follows to get the relevant policy information (at the first step in the development roadmap) or the scheduling decision (in the second step of the development roadmap). But I am getting ahead of myself here and discussing backend implementation; I think we are still working on the user-facing API. > The question is this - does the current API help > specify what we want assuming we will be able to extend the notion of > nodes, edges, policies and metadata? I am not sure I understand that remark. Of course the API you proposed is about enabling the client to express the policy information that we both advocate. I am not sure I understand why you add the qualifier of "assuming we will be able to extend the notion of ...". I do not think we (yet) have a policy type catalog set in stone, if that is the concern. I think there is an interesting discussion to have about defining that catalog. BTW, note that the class you called InstanceGroupPolicy is not just a reference to a policy, it also specifies one place where that policy is being applied. That is really the class of policy applications (or "uses"). I think some types of policies have parameters. A relationship policy about limiting the number of network hops takes a parameter that is the hop count limit. A policy about anti-collocation takes a physical hierarchy level as a parameter, to p
Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
Yes, there is more than the northbound API to discuss. Gary started us there in the Scheduler chat on Oct 1, when he broke the issues down like this: 11:12:22 AM garyk: 1. a user facing API 11:12:41 AM garyk: 2. understanding which resources need to be tracked 11:12:48 AM garyk: 3. backend implementation The full transcript is at http://eavesdrop.openstack.org/meetings/scheduling/2013/scheduling.2013-10-01-15.08.log.html Alex Glikson wrote on 10/09/2013 02:14:03 AM: > > Good summary. I would also add that in A1 the schedulers (e.g., in > Nova and Cinder) could talk to each other to coordinate. Besides > defining the policy, and the user-facing APIs, I think we should > also outline those cross-component APIs (need to think whether they > have to be user-visible, or can be admin). > > Regards, > Alex ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
Hi Yathi, Thanks for having taken time explaining your vision. Climate is about reservations, ie. preempting resources capacity and granting a user he will actually get exclusive access to a certain set of resources he asks for a certain period of time. The resource placement decisions are the core of the added-value of Climate, as historically we found that we need to do some efficiency on it. In other words, we will need to implement a Climate scheduler for picking up the right hosts best fitting the user requirements. In other words, provided an user (or a service) hits the Climate Host Reservation API asking for X hosts with these capabilities (and that could/should include network bandwidth or host architecture), Climate will create a host group (we call it "pcloud") on the lease creation with no hosts in it, and after a certain period of time (based on efficiency criterias - as of Climate v1 at lease start), Climate will take user requirements, elect the hosts and put them in the pcloud. That said, that's still a bit unclear to me but I would find two points where your efforts and our efforts could be joined : 1/ Climate could be seen as a broker for managing the states of the Instance Group by offering a backend system for implementing the need of a reservation system 2/ Climate could also see the Smart Resource Placement holder as an "scheduler" for helping to decide which hosts are the best opportunity in terms of efficiency What do you think about it ? -Sylvain Le 09/10/2013 01:51, Yathiraj Udupi (yudupi) a écrit : Hi Sylvain, Thanks for your comments. I can see that Climate is aiming to provide a reservation service for physical and now virtual resources also like you mention. The Instance-group [a][b] effort (proposed during the last summit, and good progress has been made so far) attempts to address the tenant facing API aspects in the bigger Smart Resource Placement puzzle [c]. The idea is to be able to represent an entire topology (a group of resources) that is requested by the tenant, that contains members or sub-groups , their connections, their associated policies and other metadata. The first part is to be able to persist this group, and use the group to create/schedule the resources together as a whole group, so that intelligent decisions can be made together considering all the requirements and constraints (policies). In the ongoing discussions in the Nova scheduler sub-team, we do agree that we need additional support to achieve the creation of the group as a whole. It will involve reservation too to achieve this. Once the Instance group is registered and persisted, we can trigger the creation/boot up of the instances, which will involve arriving at the resource placement decisions and then the actual creation. So one of the idea is to provide clear apis such an external component (such as climate, heat, or some other module) can take the placement decision results and do the actual creation of resource. As described in [c], we will also need the support of a global state repository to make all the resource states from across services available to smart placement decision engine. As part of the plan for [c], the first step is to tackle the representation and API for these InstanceGroups, and that is this ongoing effort within the Nova Scheduler sub-team. Our idea to separate the phases of this grand scale scheduling of resources, and keep the interfaces clean. If we have to interface with Climate for the final creation (I.e., once the smart placement decisions have been made), we should be able to do that, at least that is the vision. References [a]Instance Group Model and API extension doc - https://docs.google.com/document/d/17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?usp=sharing <https://docs.google.com/document/d/17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?usp=sharing> [b] Instance group blueprint - https://blueprints.launchpad.net/nova/+spec/instance-group-api-extension [c] Smart Resource Placement https://docs.google.com/document/d/1IiPI0sfaWb1bdYiMWzAAx0HYR6UqzOan_Utgml5W1HI/edit Thanks, Yathi. From: Sylvain Bauza <mailto:sylvain.ba...@bull.net>> Date: Tuesday, October 8, 2013 12:40 AM To: OpenStack Development Mailing List <mailto:openstack-dev@lists.openstack.org>> Cc: Yathiraj Udupi mailto:yud...@cisco.com>> Subject: Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft Hi Yathi, Le 08/10/2013 05:10, Yathiraj Udupi (yudupi) a écrit : Hi, Based on the discussions we have had in the past few scheduler sub-team meetings, I am sharing a document that proposes an updated Instance Group Model and API extension model. This is a work-in-progress draft version, but sharing it for early feedback. https://docs.google.com/docum
Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
Mike, I agree we could have a cleaner API but I am not sure how
cleanly it will integrate with current nova which IMO should be test
we should pass (assuming we do cross services later)
On Tue, Oct 8, 2013 at 10:39 PM, Mike Spreitzer wrote:
> Thanks for the clue about where the request/response bodies are documented.
> Is there any convenient way to view built documentation for Havana right
> now?
>
> You speak repeatedly of the desire for "clean" interfaces, and nobody could
> disagree with such words. I characterize my desire that way too. It might
> help me if you elaborate a little on what "clean" means to you. To me it is
> about minimizing the number of interactions between different modules/agents
> and the amount of information in those interactions. In short, it is about
> making narrow interfaces - a form of simplicity.
>
I think the word clean can be overloaded. For me a clean API is to use
minimal nouns and specify the policies, the resources we would like to
request and the extra metadata that we might want to pass. Thus the
three components.
> To me the most frustrating aspect of this challenge is the need for the
> client to directly mediate the dependencies between resources; this is
> really what is driving us to do ugly things. As I mentioned before, I am
> coming from a setting that does not have this problem. So I am thinking
> about two alternatives: (A1) how clean can we make a system in which the
> client continues to directly mediate dependencies between resources, and
> (A2) how easily and cleanly can we make that problem go away.
Am a little confused - How is the API dictating either A1 or A2? Isnt
that a function of the implementation of the API. For a moment let us
assume that the black box implementation will be awesome and address
your concerns. The question is this - does the current API help
specify what we want assuming we will be able to extend the notion of
nodes, edges, policies and metadata?
debo
>
> For A1, we need the client to make a distinct activation call for each
> resource. You have said that we should start the roadmap without joint
> scheduling; in this case, the scheduling can continue to be done
> independently for each resource and can be bundled with the activation call.
> That can be the call we know and love today, the one that creates a
> resource, except that it needs to be augmented to also carry some pointer
> that points into the policy data so that the relevant policy data can be
> taken into account when making the scheduling decision. Ergo, the client
> needs to know this pointer value for each resource. The simplest approach
> would be to let that pointer be the combination of (p1) a VRT's UUID and
> (p2) the local name for the resource within the VRT. Other alternatives are
> possible, but require more bookkeeping by the client.
>
> I think that at the first step of the roadmap for A1, the client/service
> interaction for CREATE can be in just two phases. In the first phase the
> client presents a topology (top-level InstanceGroup in your terminology),
> including resource definitions, to the new API for registration; the
> response is a UUID for that registered top-level group. In the second phase
> the client "creates" the resources as is done today, except that each
> creation call is augmented to carry the aforementioned pointer into the
> policy information. Each resource scheduler (just nova, at first) can use
> that pointer to access the relevant policy information and take it into
> account when scheduling. The client/service interaction for UPDATE would be
> in the same two phases: first update the policy&resource definitions at the
> new API, then do the individual resource updates in dependency order.
>
> I suppose the second step in the roadmap is to have Nova do joint
> scheduling. The client/service interaction pattern can stay the same. The
> only difference is that Nova makes the scheduling decisions in the first
> phase rather than the second. But that is not a detail exposed to the
> clients.
>
> Maybe the third step is to generalize beyond nova?
>
> For A2, the first question is how to remove "user-level" create-time
> dependencies between resources. We are only concerned with the "user-level"
> create-time dependencies here because it is only they that drive intimate
> client interactions. There are also create-time dependencies due to the
> nature of the resource APIs; for example, you can not attach a volume to a
> VM until after both have been created. But handling those kinds of
> create-time dependencies does not require intimate interactions with the
> client. I know of two software orchestration technologies developed in IBM,
> and both have the property that there are no "user-level" create-time
> dependencies between resources; rather, the startup code ("userdata") that
> each VM runs handles dependencies (using a library for cross-VM
> communication and synchronization). This can even
Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
Good summary. I would also add that in A1 the schedulers (e.g., in Nova
and Cinder) could talk to each other to coordinate. Besides defining the
policy, and the user-facing APIs, I think we should also outline those
cross-component APIs (need to think whether they have to be user-visible,
or can be admin).
Regards,
Alex
From: Mike Spreitzer
To: "Yathiraj Udupi (yudupi)" ,
Cc: OpenStack Development Mailing List
Date: 09/10/2013 08:46 AM
Subject: Re: [openstack-dev] [scheduler] APIs for Smart Resource
Placement - Updated Instance Group Model and API extension model - WIP
Draft
Thanks for the clue about where the request/response bodies are
documented. Is there any convenient way to view built documentation for
Havana right now?
You speak repeatedly of the desire for "clean" interfaces, and nobody
could disagree with such words. I characterize my desire that way too. It
might help me if you elaborate a little on what "clean" means to you. To
me it is about minimizing the number of interactions between different
modules/agents and the amount of information in those interactions. In
short, it is about making narrow interfaces - a form of simplicity.
To me the most frustrating aspect of this challenge is the need for the
client to directly mediate the dependencies between resources; this is
really what is driving us to do ugly things. As I mentioned before, I am
coming from a setting that does not have this problem. So I am thinking
about two alternatives: (A1) how clean can we make a system in which the
client continues to directly mediate dependencies between resources, and
(A2) how easily and cleanly can we make that problem go away.
For A1, we need the client to make a distinct activation call for each
resource. You have said that we should start the roadmap without joint
scheduling; in this case, the scheduling can continue to be done
independently for each resource and can be bundled with the activation
call. That can be the call we know and love today, the one that creates a
resource, except that it needs to be augmented to also carry some pointer
that points into the policy data so that the relevant policy data can be
taken into account when making the scheduling decision. Ergo, the client
needs to know this pointer value for each resource. The simplest approach
would be to let that pointer be the combination of (p1) a VRT's UUID and
(p2) the local name for the resource within the VRT. Other alternatives
are possible, but require more bookkeeping by the client.
I think that at the first step of the roadmap for A1, the client/service
interaction for CREATE can be in just two phases. In the first phase the
client presents a topology (top-level InstanceGroup in your terminology),
including resource definitions, to the new API for registration; the
response is a UUID for that registered top-level group. In the second
phase the client "creates" the resources as is done today, except that
each creation call is augmented to carry the aforementioned pointer into
the policy information. Each resource scheduler (just nova, at first) can
use that pointer to access the relevant policy information and take it
into account when scheduling. The client/service interaction for UPDATE
would be in the same two phases: first update the policy&resource
definitions at the new API, then do the individual resource updates in
dependency order.
I suppose the second step in the roadmap is to have Nova do joint
scheduling. The client/service interaction pattern can stay the same. The
only difference is that Nova makes the scheduling decisions in the first
phase rather than the second. But that is not a detail exposed to the
clients.
Maybe the third step is to generalize beyond nova?
For A2, the first question is how to remove "user-level" create-time
dependencies between resources. We are only concerned with the
"user-level" create-time dependencies here because it is only they that
drive intimate client interactions. There are also create-time
dependencies due to the nature of the resource APIs; for example, you can
not attach a volume to a VM until after both have been created. But
handling those kinds of create-time dependencies does not require intimate
interactions with the client. I know of two software orchestration
technologies developed in IBM, and both have the property that there are
no "user-level" create-time dependencies between resources; rather, the
startup code ("userdata") that each VM runs handles dependencies (using a
library for cross-VM communication and synchronization). This can even be
done in plain CFN, using wait conditions and handles (albeit somewhat
clunkily), right? So I think there are ways to get this nice property
already. The next question is how best to exploit it to make cleaner
Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
Thanks for the clue about where the request/response bodies are
documented. Is there any convenient way to view built documentation for
Havana right now?
You speak repeatedly of the desire for "clean" interfaces, and nobody
could disagree with such words. I characterize my desire that way too. It
might help me if you elaborate a little on what "clean" means to you. To
me it is about minimizing the number of interactions between different
modules/agents and the amount of information in those interactions. In
short, it is about making narrow interfaces - a form of simplicity.
To me the most frustrating aspect of this challenge is the need for the
client to directly mediate the dependencies between resources; this is
really what is driving us to do ugly things. As I mentioned before, I am
coming from a setting that does not have this problem. So I am thinking
about two alternatives: (A1) how clean can we make a system in which the
client continues to directly mediate dependencies between resources, and
(A2) how easily and cleanly can we make that problem go away.
For A1, we need the client to make a distinct activation call for each
resource. You have said that we should start the roadmap without joint
scheduling; in this case, the scheduling can continue to be done
independently for each resource and can be bundled with the activation
call. That can be the call we know and love today, the one that creates a
resource, except that it needs to be augmented to also carry some pointer
that points into the policy data so that the relevant policy data can be
taken into account when making the scheduling decision. Ergo, the client
needs to know this pointer value for each resource. The simplest approach
would be to let that pointer be the combination of (p1) a VRT's UUID and
(p2) the local name for the resource within the VRT. Other alternatives
are possible, but require more bookkeeping by the client.
I think that at the first step of the roadmap for A1, the client/service
interaction for CREATE can be in just two phases. In the first phase the
client presents a topology (top-level InstanceGroup in your terminology),
including resource definitions, to the new API for registration; the
response is a UUID for that registered top-level group. In the second
phase the client "creates" the resources as is done today, except that
each creation call is augmented to carry the aforementioned pointer into
the policy information. Each resource scheduler (just nova, at first) can
use that pointer to access the relevant policy information and take it
into account when scheduling. The client/service interaction for UPDATE
would be in the same two phases: first update the policy&resource
definitions at the new API, then do the individual resource updates in
dependency order.
I suppose the second step in the roadmap is to have Nova do joint
scheduling. The client/service interaction pattern can stay the same. The
only difference is that Nova makes the scheduling decisions in the first
phase rather than the second. But that is not a detail exposed to the
clients.
Maybe the third step is to generalize beyond nova?
For A2, the first question is how to remove "user-level" create-time
dependencies between resources. We are only concerned with the
"user-level" create-time dependencies here because it is only they that
drive intimate client interactions. There are also create-time
dependencies due to the nature of the resource APIs; for example, you can
not attach a volume to a VM until after both have been created. But
handling those kinds of create-time dependencies does not require intimate
interactions with the client. I know of two software orchestration
technologies developed in IBM, and both have the property that there are
no "user-level" create-time dependencies between resources; rather, the
startup code ("userdata") that each VM runs handles dependencies (using a
library for cross-VM communication and synchronization). This can even be
done in plain CFN, using wait conditions and handles (albeit somewhat
clunkily), right? So I think there are ways to get this nice property
already. The next question is how best to exploit it to make cleaner
APIs. I think we can have a one-step client/service interaction: the
client presents a top-level group (including leaf resource definitions) to
the new service, which registers it and proceeds to
create/schedule/activate the resources.
Regards,
Mike___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
Mike, Sylvain, I think some of Mike's questions were answered during today's scheduler team meeting. Mike: >> Thanks. I have a few questions. First, I am a bit stymied by the style of >> API documentation used in that document and many others: it shows the first >> line of an HTTP request but says nothing about all the other details. I am >> sure some of those requests must have interesting bodies, but I am >> not >> always sure which ones have a body at all, let alone what goes in it. I >> suspect there may be some headers that are important too. Am I missing >> something? Yathi: Do see some of the existing code written up for instance group here in this review, there are a few request/response examples of the older model - https://review.openstack.org/#/c/30028/ This code will be revived and the effort will be similar incorporating the newer model. Mike: >>That draft says the VMs are created before the group. Is there a way today >>to create a VM without scheduling it? Is there a way to activate a resource >>that has already been scheduled but not activated?By "activate" I mean, for a >>VM instance for example, to start running it. Sylvain: >> I can't answer for it, but as far as I can understand the draft, there is no >> clear understanding that we have to postpone the VM boot *after* creating >> the Groups. >>As stated in the document, there is a strong prereq which is that all the >>resources mapped to the Group must have their own uuids, but there is no >>clear outstanding that it should prevent the VMs to actually boot. >>At the moment, deferring a bootable state in Nova is not yet implemented and >>that's part of Climate folks to implement it, so I can't get your point. Yathi: I guess Gary Kotton can comment more here, but this is probably implementation detail. As long as the group is defined and registered, the actual activation can take care of assigning the created UUIDs for the instances.. But I do see there are internal DB APIs to save the instances and thereby creating uuids, but not actually activating them. my documentation assumes, that the instances need to be registered. The reason why we want to defer the instance boot, is because we want to get the complete big picture and do a unified scheduling taking all the parameters into consideration (read this as smart resource placement). This document does not yet mention anything about the actual process involved in the activation of the group. That will involve whole lot of work in terms of reservation, ordering of creation, etc, and it is here where we need to have clean interfaces to plug in external support to accomplish this. Thanks, Yathi. From: Sylvain Bauza mailto:sylvain.ba...@bull.net>> Date: Tuesday, October 8, 2013 12:08 AM To: OpenStack Development Mailing List mailto:openstack-dev@lists.openstack.org>> Cc: Mike Spreitzer mailto:mspre...@us.ibm.com>>, Yathiraj Udupi mailto:yud...@cisco.com>> Subject: Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft Hi Mike, Le 08/10/2013 08:51, Mike Spreitzer a écrit : On second thought, I should revise and extend my remarks. This message supersedes my previous two replies. Thanks. I have a few questions. First, I am a bit stymied by the style of API documentation used in that document and many others: it shows the first line of an HTTP request but says nothing about all the other details. I am sure some of those requests must have interesting bodies, but I am not always sure which ones have a body at all, let alone what goes in it. I suspect there may be some headers that are important too. Am I missing something? That draft says the VMs are created before the group. Is there a way today to create a VM without scheduling it? Is there a way to activate a resource that has already been scheduled but not activated?By "activate" I mean, for a VM instance for example, to start running it. As I understand your draft, it lays out a three phase process for a client to follow: create resources without scheduling or activating them, then present the groups and policies to the service for joint scheduling, then activate the resources. With regard to a given resource, things must happen in that order; between resources there is a little more flexibility. Activations are invoked by the client in an order that is consistent with (a) runtime dependencies that are mediated directly by the client (e.g., string slinging in the heat engine) and (b) the nature of the resources (for example, you can not attach a volume to a VM instance until after both have been created). Other than those considerations, the ordering and/or parallelism is a degree o
Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
Hi Sylvain, Thanks for your comments. I can see that Climate is aiming to provide a reservation service for physical and now virtual resources also like you mention. The Instance-group [a][b] effort (proposed during the last summit, and good progress has been made so far) attempts to address the tenant facing API aspects in the bigger Smart Resource Placement puzzle [c]. The idea is to be able to represent an entire topology (a group of resources) that is requested by the tenant, that contains members or sub-groups , their connections, their associated policies and other metadata. The first part is to be able to persist this group, and use the group to create/schedule the resources together as a whole group, so that intelligent decisions can be made together considering all the requirements and constraints (policies). In the ongoing discussions in the Nova scheduler sub-team, we do agree that we need additional support to achieve the creation of the group as a whole. It will involve reservation too to achieve this. Once the Instance group is registered and persisted, we can trigger the creation/boot up of the instances, which will involve arriving at the resource placement decisions and then the actual creation. So one of the idea is to provide clear apis such an external component (such as climate, heat, or some other module) can take the placement decision results and do the actual creation of resource. As described in [c], we will also need the support of a global state repository to make all the resource states from across services available to smart placement decision engine. As part of the plan for [c], the first step is to tackle the representation and API for these InstanceGroups, and that is this ongoing effort within the Nova Scheduler sub-team. Our idea to separate the phases of this grand scale scheduling of resources, and keep the interfaces clean. If we have to interface with Climate for the final creation (I.e., once the smart placement decisions have been made), we should be able to do that, at least that is the vision. References [a]Instance Group Model and API extension doc - https://docs.google.com/document/d/17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?usp=sharing<https://docs.google.com/document/d/17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?usp=sharing> [b] Instance group blueprint - https://blueprints.launchpad.net/nova/+spec/instance-group-api-extension [c] Smart Resource Placement https://docs.google.com/document/d/1IiPI0sfaWb1bdYiMWzAAx0HYR6UqzOan_Utgml5W1HI/edit Thanks, Yathi. From: Sylvain Bauza mailto:sylvain.ba...@bull.net>> Date: Tuesday, October 8, 2013 12:40 AM To: OpenStack Development Mailing List mailto:openstack-dev@lists.openstack.org>> Cc: Yathiraj Udupi mailto:yud...@cisco.com>> Subject: Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft Hi Yathi, Le 08/10/2013 05:10, Yathiraj Udupi (yudupi) a écrit : Hi, Based on the discussions we have had in the past few scheduler sub-team meetings, I am sharing a document that proposes an updated Instance Group Model and API extension model. This is a work-in-progress draft version, but sharing it for early feedback. https://docs.google.com/document/d/17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?usp=sharing This model support generic instance types, where an instance can represent a virtual node of any resource type. But in the context of Nova, an instance refers to the VM instance. This builds on the existing proposal for Instance Group Extension as documented here in this blueprint: https://blueprints.launchpad.net/nova/+spec/instance-group-api-extension Thanks, Yathi. Well, I actually read the design document, and I'm strongly interested in jumping to the project. We started a few months ago a Stackforge project, called Climate [0], aiming to reserve both physical and virtual resources. Initially, the project came from a blueprint targeting only physical reservations [1], and then Mirantis folks joined us having a new usecase for virtual reservations (potentially implementing deferred starts, as said above). Basically, the physical host reservation is not about deferred starts of instances, it's about grouping for a single tenant a list of hosts, in other words a whole host allocation (see [2]). We'll provide to end-users a Reservation API allowing to define policies for selecting hosts based on their capabilities [3] and then create host aggregates (or "Pclouds" if we implement [2]). Actually, we could define some policies in the Climate host aggregate for affinity and network-proximity policies, so that any VM to boot from one of these hosts would be applied these host aggregate policies. As you maybe see, there are some concerns which are close in between your BP [4] and our vision
Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
Hi Yathi, Le 08/10/2013 05:10, Yathiraj Udupi (yudupi) a écrit : Hi, Based on the discussions we have had in the past few scheduler sub-team meetings, I am sharing a document that proposes an updated Instance Group Model and API extension model. This is a work-in-progress draft version, but sharing it for early feedback. https://docs.google.com/document/d/17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?usp=sharing This model support generic instance types, where an instance can represent a virtual node of any resource type. But in the context of Nova, an instance refers to the VM instance. This builds on the existing proposal for Instance Group Extension as documented here in this blueprint: https://blueprints.launchpad.net/nova/+spec/instance-group-api-extension Thanks, Yathi. Well, I actually read the design document, and I'm strongly interested in jumping to the project. We started a few months ago a Stackforge project, called Climate [0], aiming to reserve both physical and virtual resources. Initially, the project came from a blueprint targeting only physical reservations [1], and then Mirantis folks joined us having a new usecase for virtual reservations (potentially implementing deferred starts, as said above). Basically, the physical host reservation is not about deferred starts of instances, it's about grouping for a single tenant a list of hosts, in other words a whole host allocation (see [2]). We'll provide to end-users a Reservation API allowing to define policies for selecting hosts based on their capabilities [3] and then create host aggregates (or "Pclouds" if we implement [2]). Actually, we could define some policies in the Climate host aggregate for affinity and network-proximity policies, so that any VM to boot from one of these hosts would be applied these host aggregate policies. As you maybe see, there are some concerns which are close in between your BP [4] and our vision of Climate. What are your thoughts about it ? [0] : https://github.com/stackforge/climate [1] : https://wiki.openstack.org/wiki/Blueprint-nova-planned-resource-reservation-api [2] : https://wiki.openstack.org/wiki/WholeHostAllocation [3] : https://docs.google.com/document/d/1U36k5wk0sOUyLl-4Cz8tmk8RQFQGWKO9dVhb87ZxPC8/edit#heading=h.ujapi6o0un65 [4] : https://blueprints.launchpad.net/nova/+spec/instance-group-api-extension ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
Hi Mike, Le 08/10/2013 08:51, Mike Spreitzer a écrit : On second thought, I should revise and extend my remarks. This message supersedes my previous two replies. Thanks. I have a few questions. First, I am a bit stymied by the style of API documentation used in that document and many others: it shows the first line of an HTTP request but says nothing about all the other details. I am sure some of those requests must have interesting bodies, but I am not always sure which ones have a body at all, let alone what goes in it. I suspect there may be some headers that are important too. Am I missing something? That draft says the VMs are created before the group. Is there a way today to create a VM without scheduling it? Is there a way to activate a resource that has already been scheduled but not activated?By "activate" I mean, for a VM instance for example, to start running it. As I understand your draft, it lays out a three phase process for a client to follow: create resources without scheduling or activating them, then present the groups and policies to the service for joint scheduling, then activate the resources. With regard to a given resource, things must happen in that order; between resources there is a little more flexibility. Activations are invoked by the client in an order that is consistent with (a) runtime dependencies that are mediated directly by the client (e.g., string slinging in the heat engine) and (b) the nature of the resources (for example, you can not attach a volume to a VM instance until after both have been created). Other than those considerations, the ordering and/or parallelism is a degree of freedom available to the client. Have I got this right? Couldn't we simplify this into a two phase process: create groups and resources with scheduling, then activate the resources in an acceptable order? I can't answer for it, but asfar as I can understand the draft, there is no clear understanding that we have topostpone the VM boot *after* creating the Groups. As stated in the document, there is a strong prereqwhich is that all the resources mapped to the Group must have their own uuids, but there is no clear outstanding that it should prevent the VMs to actually boot. At the moment, deferring a bootable state in Nova is not yet implemented and that's part of Climate folks to implement it, so I can't get your point. -Sylvain FYI: my group is using Weaver as the software orchestration technique, so there are no runtime dependencies that are mediated directly by the client. The client sees a very simple API: the client presents a definition of all the groups and resources, and the service first schedules it all then activates in an acceptable order. (We already have something in OpenStack that can do resource invocations in an acceptable order, right?) Weaver is not the only software orchestration technique with this property. The simplicity of this API is one reason I recommend software orchestration techniques that take dependency mediation out of the client's hands. I hope that with coming work on HOT we can get OpenStack to this level of API simplicity. But that struggle lies farther down the roadmap... I was wondering if you could explain why you included all those integer IDs; aren't the UUIDs sufficient? Do you intend that clients will see/manipulate the integer IDs? If I understand your UML correctly, an InstanceGroup owns its metadata but none of the other subsidiary objects introduced. Why not? If an InstanceGroup is deleted, shouldn't all those other subsidiary objects be deleted too? Thanks, Mike "Yathiraj Udupi (yudupi)" wrote on 10/07/2013 11:10:20 PM: > Hi, > > Based on the discussions we have had in the past few scheduler sub- > team meetings, I am sharing a document that proposes an updated > Instance Group Model and API extension model. > This is a work-in-progress draft version, but sharing it for early feedback. > https://docs.google.com/document/d/17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?usp=sharing > > This model support generic instance types, where an instance can > represent a virtual node of any resource type. But in the context > of Nova, an instance refers to the VM instance. > > This builds on the existing proposal for Instance Group Extension as > documented here in this blueprint: https:// > blueprints.launchpad.net/nova/+spec/instance-group-api-extension > > Thanks, > Yathi. > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
On second thought, I should revise and extend my remarks. This message supersedes my previous two replies. Thanks. I have a few questions. First, I am a bit stymied by the style of API documentation used in that document and many others: it shows the first line of an HTTP request but says nothing about all the other details. I am sure some of those requests must have interesting bodies, but I am not always sure which ones have a body at all, let alone what goes in it. I suspect there may be some headers that are important too. Am I missing something? That draft says the VMs are created before the group. Is there a way today to create a VM without scheduling it? Is there a way to activate a resource that has already been scheduled but not activated? By "activate" I mean, for a VM instance for example, to start running it. As I understand your draft, it lays out a three phase process for a client to follow: create resources without scheduling or activating them, then present the groups and policies to the service for joint scheduling, then activate the resources. With regard to a given resource, things must happen in that order; between resources there is a little more flexibility. Activations are invoked by the client in an order that is consistent with (a) runtime dependencies that are mediated directly by the client (e.g., string slinging in the heat engine) and (b) the nature of the resources (for example, you can not attach a volume to a VM instance until after both have been created). Other than those considerations, the ordering and/or parallelism is a degree of freedom available to the client. Have I got this right? Couldn't we simplify this into a two phase process: create groups and resources with scheduling, then activate the resources in an acceptable order? FYI: my group is using Weaver as the software orchestration technique, so there are no runtime dependencies that are mediated directly by the client. The client sees a very simple API: the client presents a definition of all the groups and resources, and the service first schedules it all then activates in an acceptable order. (We already have something in OpenStack that can do resource invocations in an acceptable order, right?) Weaver is not the only software orchestration technique with this property. The simplicity of this API is one reason I recommend software orchestration techniques that take dependency mediation out of the client's hands. I hope that with coming work on HOT we can get OpenStack to this level of API simplicity. But that struggle lies farther down the roadmap... I was wondering if you could explain why you included all those integer IDs; aren't the UUIDs sufficient? Do you intend that clients will see/manipulate the integer IDs? If I understand your UML correctly, an InstanceGroup owns its metadata but none of the other subsidiary objects introduced. Why not? If an InstanceGroup is deleted, shouldn't all those other subsidiary objects be deleted too? Thanks, Mike "Yathiraj Udupi (yudupi)" wrote on 10/07/2013 11:10:20 PM: > Hi, > > Based on the discussions we have had in the past few scheduler sub- > team meetings, I am sharing a document that proposes an updated > Instance Group Model and API extension model. > This is a work-in-progress draft version, but sharing it for early feedback. > https://docs.google.com/document/d/17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?usp=sharing > > This model support generic instance types, where an instance can > represent a virtual node of any resource type. But in the context > of Nova, an instance refers to the VM instance. > > This builds on the existing proposal for Instance Group Extension as > documented here in this blueprint: https:// > blueprints.launchpad.net/nova/+spec/instance-group-api-extension > > Thanks, > Yathi. > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
In addition to the other questions below, I was wondering if you could explain why you included all those integer IDs; aren't the UUIDs sufficient? Thanks, Mike From: Mike Spreitzer/Watson/IBM@IBMUS To: "Yathiraj Udupi (yudupi)" , Cc: OpenStack Development Mailing List Date: 10/08/2013 12:41 AM Subject: Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft Thanks. I have a few questions. First, I am a bit stymied by the style of API documentation used in that document and many others: it shows the first line of an HTTP request but says nothing about all the other details. I am sure some of those requests must have interesting bodies, but I am not always sure which ones have a body at all, let alone what goes in it. I suspect there may be some headers that are important too. Am I missing something? That draft says the VMs are created before the group. Is there a way today to create a VM without scheduling it? As I understand your draft, it lays out a three phase process for a client to follow: create resources without scheduling or activating them, then arrange them into groups, then schedule & activate them. By "activate" I mean, for a VM instance, to start running it. That ordering must hold independently for each resource. Activations are invoked by the client in an order that is consistent with (a) runtime dependencies that are mediated directly by the client (e.g., string slinging in the heat engine) and (b) the nature of the resources (for example, you can not attach a volume to a VM instance until after both have been created). Other than those considerations, the ordering and/or parallelism is a degree of freedom available to the client. Have I got this right? Couldn't we simplify this into a two phase process: create groups and resources with scheduling, then activate the resources in an acceptable order? FYI: my group is using Weaver as the software orchestration technique, so there are no runtime dependencies that are mediated directly by the client. The client sees a very simple API: the client presents a definition of all the groups and resources, and the service first schedules it all then activates in an acceptable order. (We already have something in OpenStack that can do activations in an acceptable order, right?) Weaver is not the only software orchestration technique with this property. The simplicity of this API is one reason I recommend software orchestration techniques that take dependency mediation out of the client's hands. I hope that with coming work on HOT we can get OpenStack to this level of API simplicity. But that struggle lies farther down the roadmap... Thanks, Mike "Yathiraj Udupi (yudupi)" wrote on 10/07/2013 11:10:20 PM: > > Hi, > > Based on the discussions we have had in the past few scheduler sub- > team meetings, I am sharing a document that proposes an updated > Instance Group Model and API extension model. > This is a work-in-progress draft version, but sharing it for early feedback. > https://docs.google.com/document/d/ > 17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?usp=sharing > > This model support generic instance types, where an instance can > represent a virtual node of any resource type. But in the context > of Nova, an instance refers to the VM instance. > > This builds on the existing proposal for Instance Group Extension as > documented here in this blueprint: https:// > blueprints.launchpad.net/nova/+spec/instance-group-api-extension > > Thanks, > Yathi. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
In addition to the other questions below, I was wondering if you could explain why you included all those integer IDs; aren't the UUIDs sufficient? Thanks, Mike From: Mike Spreitzer/Watson/IBM@IBMUS To: "Yathiraj Udupi (yudupi)" , Cc: OpenStack Development Mailing List Date: 10/08/2013 12:41 AM Subject: Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft Thanks. I have a few questions. First, I am a bit stymied by the style of API documentation used in that document and many others: it shows the first line of an HTTP request but says nothing about all the other details. I am sure some of those requests must have interesting bodies, but I am not always sure which ones have a body at all, let alone what goes in it. I suspect there may be some headers that are important too. Am I missing something? That draft says the VMs are created before the group. Is there a way today to create a VM without scheduling it? As I understand your draft, it lays out a three phase process for a client to follow: create resources without scheduling or activating them, then arrange them into groups, then schedule & activate them. By "activate" I mean, for a VM instance, to start running it. That ordering must hold independently for each resource. Activations are invoked by the client in an order that is consistent with (a) runtime dependencies that are mediated directly by the client (e.g., string slinging in the heat engine) and (b) the nature of the resources (for example, you can not attach a volume to a VM instance until after both have been created). Other than those considerations, the ordering and/or parallelism is a degree of freedom available to the client. Have I got this right? Couldn't we simplify this into a two phase process: create groups and resources with scheduling, then activate the resources in an acceptable order? FYI: my group is using Weaver as the software orchestration technique, so there are no runtime dependencies that are mediated directly by the client. The client sees a very simple API: the client presents a definition of all the groups and resources, and the service first schedules it all then activates in an acceptable order. (We already have something in OpenStack that can do activations in an acceptable order, right?) Weaver is not the only software orchestration technique with this property. The simplicity of this API is one reason I recommend software orchestration techniques that take dependency mediation out of the client's hands. I hope that with coming work on HOT we can get OpenStack to this level of API simplicity. But that struggle lies farther down the roadmap... Thanks, Mike "Yathiraj Udupi (yudupi)" wrote on 10/07/2013 11:10:20 PM: > > Hi, > > Based on the discussions we have had in the past few scheduler sub- > team meetings, I am sharing a document that proposes an updated > Instance Group Model and API extension model. > This is a work-in-progress draft version, but sharing it for early feedback. > https://docs.google.com/document/d/ > 17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?usp=sharing > > This model support generic instance types, where an instance can > represent a virtual node of any resource type. But in the context > of Nova, an instance refers to the VM instance. > > This builds on the existing proposal for Instance Group Extension as > documented here in this blueprint: https:// > blueprints.launchpad.net/nova/+spec/instance-group-api-extension > > Thanks, > Yathi. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
Thanks. I have a few questions. First, I am a bit stymied by the style of API documentation used in that document and many others: it shows the first line of an HTTP request but says nothing about all the other details. I am sure some of those requests must have interesting bodies, but I am not always sure which ones have a body at all, let alone what goes in it. I suspect there may be some headers that are important too. Am I missing something? That draft says the VMs are created before the group. Is there a way today to create a VM without scheduling it? As I understand your draft, it lays out a three phase process for a client to follow: create resources without scheduling or activating them, then arrange them into groups, then schedule & activate them. By "activate" I mean, for a VM instance, to start running it. That ordering must hold independently for each resource. Activations are invoked by the client in an order that is consistent with (a) runtime dependencies that are mediated directly by the client (e.g., string slinging in the heat engine) and (b) the nature of the resources (for example, you can not attach a volume to a VM instance until after both have been created). Other than those considerations, the ordering and/or parallelism is a degree of freedom available to the client. Have I got this right? Couldn't we simplify this into a two phase process: create groups and resources with scheduling, then activate the resources in an acceptable order? FYI: my group is using Weaver as the software orchestration technique, so there are no runtime dependencies that are mediated directly by the client. The client sees a very simple API: the client presents a definition of all the groups and resources, and the service first schedules it all then activates in an acceptable order. (We already have something in OpenStack that can do activations in an acceptable order, right?) Weaver is not the only software orchestration technique with this property. The simplicity of this API is one reason I recommend software orchestration techniques that take dependency mediation out of the client's hands. I hope that with coming work on HOT we can get OpenStack to this level of API simplicity. But that struggle lies farther down the roadmap... Thanks, Mike "Yathiraj Udupi (yudupi)" wrote on 10/07/2013 11:10:20 PM: > > Hi, > > Based on the discussions we have had in the past few scheduler sub- > team meetings, I am sharing a document that proposes an updated > Instance Group Model and API extension model. > This is a work-in-progress draft version, but sharing it for early feedback. > https://docs.google.com/document/d/ > 17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?usp=sharing > > This model support generic instance types, where an instance can > represent a virtual node of any resource type. But in the context > of Nova, an instance refers to the VM instance. > > This builds on the existing proposal for Instance Group Extension as > documented here in this blueprint: https:// > blueprints.launchpad.net/nova/+spec/instance-group-api-extension > > Thanks, > Yathi. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [scheduler] APIs for Smart Resource Placement - Updated Instance Group Model and API extension model - WIP Draft
Hi, Based on the discussions we have had in the past few scheduler sub-team meetings, I am sharing a document that proposes an updated Instance Group Model and API extension model. This is a work-in-progress draft version, but sharing it for early feedback. https://docs.google.com/document/d/17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA/edit?usp=sharing This model support generic instance types, where an instance can represent a virtual node of any resource type. But in the context of Nova, an instance refers to the VM instance. This builds on the existing proposal for Instance Group Extension as documented here in this blueprint: https://blueprints.launchpad.net/nova/+spec/instance-group-api-extension Thanks, Yathi. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
