Re: [openstack-dev] [Climate] Questions and comments
On 10/9/13 6:53 AM, Mike Spreitzer wrote: Yes, that helps. Please, guys, do not interpret my questions as hostility, I really am just trying to understand. I think there is some overlap between your concerns and mine, and I hope we can work together. No probs at all. Don't see a sign of hostility at all. Potential collaboration and understanding is really how we perceive your questions... Sticking to the physical reservations for the moment, let me ask for a little more explicit details. In your outline below, late in the game you write the actual reservation is performed by the lease manager plugin. Is that the point in time when something (the lease manager plugin, in fact) decides which hosts will be used to satisfy the reservation? Yes. The reservation service should return only a Pcloud uuid that is empty. The description of host capabilities and extra-specs is only defined as metadata of the Pcloud at this point. Or is that decided up-front when the reservation is made? I do not understand how the lease manager plugin can make this decision on its own, isn't the nova scheduler also deciding how to use hosts? Why isn't there a problem due to two independent allocators making allocations of the same resources (the system's hosts)? The way we are designing it excludes race conditions between Nova scheduler and the lease manager plugin for host reservations because the lease manager plugin will use a private pool of hosts for reservation (reservation pool) that is not shared with Nova scheduler. In our view, this is not a convenience design artifact but a purpose. It is because what we'd like to achieve really is energy efficiency management based on a reservation backlog and possibly dynamic management of host resources between the reservation pool and the multi-tenant pool. A Climate scheduler filter in Nova will do the triage filtering out those hosts that belong to the reservation pool and hosts that are reserved in an active lease. Another (longer term) goal behind this (was actually the primary justification for the reservation pool) is that the lease manager plugin could turn machines off to save electricity when the reservation backlog allows to do so and consequently turn them back on when a lease kicks in if that's needed. We anticipate that the resource management algorithms / heuristics behind that behavior is non-trivial but we believe that it would be hardly achievable without a reservation backlog and some form of capacity management capabilities left open to the provider. In particular, things become much trickier when it to comes decide what to do with the reserved hosts when a lease ends. We foresee few options: 1) Forcibly kill the instances running on reserved hosts and move them back to the reservation pool for the next lease to come 2) Keep the instances running on the reserved hosts and move them to an intermediary recycling pool until all the instances die at which point in time those hosts that are released from duty can return to the reservation pool. Case 1 and 2 could optionally be augmented by a grace period 3) Keep the instances running on the reserved hosts and move them to the multi-tenant pool. Then, it'll be up to the operator to repopulate the reservation pool using free hosts. Would require administrative tasks like disabling hosts, instance migrations, ... in other words certainly a pain if not fully automated. So, you noticed that all this relies very much on manipulating hosts aggregates, metadata and filtering behind the scene. That's one way of implementing the whole-host-reservation feature based on the tools we have at our disposable today. A substantial refactoring of Nova and scheduler could/should be a better way to go? Is it worth it? We don't know. We anyway have zero visibility on that. HTH, Patrick Thanks, Mike Patrick Petit patrick.pe...@bull.net wrote on 10/07/2013 07:02:36 AM: Hi Mike, There are actually more facets to this. Sorry if it's a little confusing :-( Climate's original blueprint https:// wiki.openstack.org/wiki/Blueprint-nova-planned-resource-reservation-api was about physical host reservation only. The typical use case being: I want to reserve x number of hosts that match the capabilities expressed in the reservation request. The lease is populated with reservations which at this point are only capacity descriptors. The reservation becomes active only when the lease starts at a specified time and for a specified duration. The lease manager plugin in charge of the physical reservation has a planning of reservations that allows Climate to grant a lease only if the requested capacity is available at that time. Once the lease becomes active, the user can request instances to be created on the reserved hosts using a lease handle as a Nova's scheduler hint. That's basically it. We do not assume or enforce how and by whom (Nova, Heat ,...) a resource
Re: [openstack-dev] [Climate] Questions and comments
Mike, I'll try to describe the reservation process for the virtual
reservations. I'll use Nova project as an example.
As I said, this Nova workflow is only the example that may and certainly
will be modified for other 'virtual' projects.
1) User goes to Nova via CLI/Dashboard and commits all usual actions like
he/she wants to boot instance. The only difference is that user passes
reservation-connected hints to Nova. In the CLI this request may look like
the following:
nova boot --flavor 1 --image bb3979c2-b2e1-4836-abbc-2ee510064718 --hint
reserved=True --hint lease_params='{name: lease1, start: now,
end: 2013-12-1 16:07}' vm1
If scheduling process went OK, we'll see the following by 'nova list'
command:
+--+--+--++-+--+
| ID | Name | Status | Task State |
Power State | Networks |
+--+--+--++-+--+
| a7ac3b2e-dca5-4d21-ab37-cd019a813636 | vm1 | RESERVED | None |
NOSTATE | private=10.0.0.3 |
+--+--+--++-+--+
2) Request passes up to the Compute Manager, where scheduling process is
already done. If Manager finds reservation related hints it uses Climate
client to create lease using passed to Nova params and id of the VM to be
reserved. Also Nova changes status of VM in its DB to 'RESERVED'. If there
are no reservation related hints filter properties, Nova just spawns
instance as usual.
3) Lease creation request goes to Climate Lease API via Climate Client.
Climate Lease API will be mostly used by other services (like Nova in this
example) and by admin users to manage leases as 'contracts'.
4) Climate Lease API passes lease creation request to Climate Manager
service via RPC. Climate Manager is the service that communicates with all
resource plugins and Climate DB. Climate Manager creates lease record in
DB, all reservation records (for the instance in this case) and all events
records. Even if user passes no additional events (like notifications in
future), at least two events for lease are created - 'start' and 'end'
events.
5) One more function that Manager does is periodical DB polling to find out
if there are any 'UNDONE' event to be processed. If there is such event
(for example, start event for the lease just saved in DB), manager begins
to process it. That means manager sets event status to 'IN_PROGRESS' and
for every reservation in lease commits 'on_start' actions for this
reservation. Now there is one-to-one relationship between lease and
reservation, but we suppose there may be cases for one-to-many
relationship. 'On_start' actions are defined in resource plugin responsible
for this resource type (virtual:instance) in this example. Plugins are
loaded using stevedore and needed ones are defined in climate.conf file.
6) virtual:instance plugin commits on_start actions. For VM it may be
'wake_up' action, that wakes reserved instance up through Nova API. This
may be implemented using Nova extensions mechanism. Wake up action really
spawns this instance.
7) If everything is ok, Manager sets event status to 'DONE' or 'COMPLETED'.
8) Almost the same process is done when Manager gets 'end' event for the
lease from DB.
Thank you for the attention.
Dina
On Wed, Oct 9, 2013 at 1:01 PM, Patrick Petit patrick.pe...@bull.netwrote:
On 10/9/13 6:53 AM, Mike Spreitzer wrote:
Yes, that helps. Please, guys, do not interpret my questions as
hostility, I really am just trying to understand. I think there is some
overlap between your concerns and mine, and I hope we can work together.
No probs at all. Don't see a sign of hostility at all. Potential
collaboration and understanding is really how we perceive your
questions...
Sticking to the physical reservations for the moment, let me ask for a
little more explicit details. In your outline below, late in the game you
write the actual reservation is performed by the lease manager plugin.
Is that the point in time when something (the lease manager plugin, in
fact) decides which hosts will be used to satisfy the reservation?
Yes. The reservation service should return only a Pcloud uuid that is
empty. The description of host capabilities and extra-specs is only
defined as metadata of the Pcloud at this point.
Or is that decided up-front when the reservation is made? I do not
understand how the lease manager plugin can make this decision on its own,
isn't the nova scheduler also deciding how to use hosts? Why isn't there a
problem due to two independent allocators making allocations of the same
resources (the system's hosts)?
The way we are designing it excludes race conditions between Nova
scheduler and the lease manager plugin for host reservations because the
lease manager plugin will use a private pool of
Re: [openstack-dev] [Climate] Questions and comments
Yes, that helps. Please, guys, do not interpret my questions as hostility, I really am just trying to understand. I think there is some overlap between your concerns and mine, and I hope we can work together. Sticking to the physical reservations for the moment, let me ask for a little more explicit details. In your outline below, late in the game you write the actual reservation is performed by the lease manager plugin. Is that the point in time when something (the lease manager plugin, in fact) decides which hosts will be used to satisfy the reservation? Or is that decided up-front when the reservation is made? I do not understand how the lease manager plugin can make this decision on its own, isn't the nova scheduler also deciding how to use hosts? Why isn't there a problem due to two independent allocators making allocations of the same resources (the system's hosts)? Thanks, Mike Patrick Petit patrick.pe...@bull.net wrote on 10/07/2013 07:02:36 AM: Hi Mike, There are actually more facets to this. Sorry if it's a little confusing :-( Climate's original blueprint https:// wiki.openstack.org/wiki/Blueprint-nova-planned-resource-reservation-api was about physical host reservation only. The typical use case being: I want to reserve x number of hosts that match the capabilities expressed in the reservation request. The lease is populated with reservations which at this point are only capacity descriptors. The reservation becomes active only when the lease starts at a specified time and for a specified duration. The lease manager plugin in charge of the physical reservation has a planning of reservations that allows Climate to grant a lease only if the requested capacity is available at that time. Once the lease becomes active, the user can request instances to be created on the reserved hosts using a lease handle as a Nova's scheduler hint. That's basically it. We do not assume or enforce how and by whom (Nova, Heat ,...) a resource instantiation is performed. In other words, a host reservation is like a whole host allocation https:// wiki.openstack.org/wiki/WholeHostAllocation that is reserved ahead of time by a tenant in anticipation of some workloads that is bound to happen in the future. Note that while we are primarily targeting hosts reservations the same service should be offered for storage. Now, Mirantis brought in a slew of new use cases that are targeted toward virtual resource reservation as explained earlier by Dina. While architecturally both reservation schemes (physical vs virtual) leverage common components, it is important to understand that they behave differently. For example, Climate exposes an API for the physical resource reservation that the virtual resource reservation doesn't. That's because virtual resources are supposed to be already reserved (through some yet to be created Nova, Heat, Cinder,... extensions) when the lease is created. Things work differently for the physical resource reservation in that the actual reservation is performed by the lease manager plugin not before the lease is created but when the lease becomes active (or some time before depending on the provisioning lead time) and released when the lease ends. HTH clarifying things. BR, Patrick ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Climate] Questions and comments
Sylvain: please do not interpret my questions as hostility. I am only trying to understand your proposal, but I am still confused. Can you please walk through a scenario involving Climate reservations on virtual resources? I mean from start to finish, outlining which party makes which decision when, based on what. I am trying to understand the relationship between the individual resource schedulers (such as nova, cinder) and climate --- they both seem to be about allocating the same resources. Thanks, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Climate] Questions and comments
Hi Mike, Dina and you outlined some differences in terms of seeing what is dependent on what. As Dina explained, Climate plans to be integrated into Nova and Heat logics, where Heat and Nova would request Climate API by asking for a lease and would tag on their own the resources as 'RESERVED'. On your point, and correct me if I'm wrong, you would rather see Climate on top of Heat and Nova, scheduling resources on its own, and only send creation requests to Heat and Nova. I'm happy to say both of you are right : Climate aims to be both called by Nova and *also* calling Nova. That's just matter of what Climate *is*. And here is the confusion. That's why Climate is not only one API endpoint. It actually have two distinct endpoints : one called the Lease API endpoint, and one called the Resource Reservation API endpoint. As a Climate developer working on physical hosts reservations (and not Heat stacks), my concern is to be able to guarantee to a REST client (either a user or another service) that if this user wants to provision X hosts on a specific timeframe in the future (immediate or in 10 years), Climate will be able to provision them. By meaning being able and guarantee, I do use strong words for stating that we engage ourselves to be able to plan what will be resources capacity state in the future. This decision-making process (ie. this Climate scheduler) will be implemented as RPC Service for the Reservation API, and thus will needs to keep its own persistence layer in Climate. Of course, it will request the Lease API for really creating the lease and managing lease start/end hooks, that's the Lease API job. Provided you would want to use the Reservation API for reserving Heat stacks, you would have to implement it tho. Thanks, -Sylvain Le 06/10/2013 20:41, Mike Spreitzer a écrit : Thanks, Dina. Yes, we do not understand each other; can I ask some more questions? You outlined a two-step reservation process (We assume the following reservation process for the OpenStack services...), and right after that talked about changing your mind to use Heat instead of individual services. So I am confused, I am not sure which of your remarks reflect your current thinking and which reflect old thinking. Can you just state your current thinking? On what basis would Climate decide to start or stop a lease? What sort of event notifications would Climate be sending, and when and why, and what would subscribers do upon receipt of such notifications? If the individual resource services continue to make independent scheduling decisions as they do today, what value does Climate add? Maybe a little more detailed outline of what happens in your current thinking, in support of an explicitly stated use case that shows the value, would help here. Thanks, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Climate] Questions and comments
Hi Mike, There are actually more facets to this. Sorry if it's a little confusing :-( Climate's original blueprint https://wiki.openstack.org/wiki/Blueprint-nova-planned-resource-reservation-api was about physical host reservation only. The typical use case being: I want to reserve x number of hosts that match the capabilities expressed in the reservation request. The lease is populated with reservations which at this point are only capacity descriptors. The reservation becomes active only when the lease starts at a specified time and for a specified duration. The lease manager plugin in charge of the physical reservation has a planning of reservations that allows Climate to grant a lease only if the requested capacity is available at that time. Once the lease becomes active, the user can request instances to be created on the reserved hosts using a lease handle as a Nova's scheduler hint. That's basically it. We do not assume or enforce how and by whom (Nova, Heat ,...) a resource instantiation is performed. In other words, a host reservation is like a whole host allocation https://wiki.openstack.org/wiki/WholeHostAllocation that is reserved ahead of time by a tenant in anticipation of some workloads that is bound to happen in the future. Note that while we are primarily targeting hosts reservations the same service should be offered for storage. Now, Mirantis brought in a slew of new use cases that are targeted toward virtual resource reservation as explained earlier by Dina. While architecturally both reservation schemes (physical vs virtual) leverage common components, it is important to understand that they behave differently. For example, Climate exposes an API for the physical resource reservation that the virtual resource reservation doesn't. That's because virtual resources are supposed to be already reserved (through some yet to be created Nova, Heat, Cinder,... extensions) when the lease is created. Things work differently for the physical resource reservation in that the actual reservation is performed by the lease manager plugin not before the lease is created but when the lease becomes active (or some time before depending on the provisioning lead time) and released when the lease ends. HTH clarifying things. BR, Patrick On 10/6/13 8:36 AM, Mike Spreitzer wrote: I looked at the blueprint (https://blueprints.launchpad.net/heat/+spec/stacks-reservation) and associated wiki page (https://wiki.openstack.org/wiki/Heat/Reservation), and I have a few comments/questions. The wiki page has some remarks that are Nova-centric, and some other remarks that emphasize that Climate is not just about Nova, and I do not understand the relationship between these remarks. Is this a roadmapping thought (start with just Nova, expand to other services later), or the inclusion of some details (related to Nova) and omission of other similar details (related to the other services), or what? Will Climate be an independent service, or part of Nova, or what? What will be the atomic operations? I presume the primary interesting operation will be something like reserving a bag of resources, where that bag is allowed to contain any mixture of resources from any services. Have I got that right? What exactly does reserving a resource mean? Does this atomic reservation operation include some atomic cooperation from the resources' services' schedulers (e.g., Nova scheduler, Cinder scheduler, etc)? Or is this reservation service logically independent of the resources' primary schedulers? Overall I am getting the suggestion that reservation is an independent service. The flow is something like first reserve a bag of resources, and then proceed to use them at your leisure. But I also suppose that the important thing about a reservation is that it includes the result of scheduling (placement) --- the point of a reservation is that it is holding capacity to host the reserved resources. You do not want an atomic operation to take a long time; do the scheduling decisions get made (tentatively, of course) before the real atomic section, with re-schedule and re-try on conflict detection, or is scheduling included in the atomic section? For how long does a reservation last? What sort of thing owns a reservation? I suppose one important use case is that a heat engine, or the heat service (in the multi-engine world), could own a reservation; in this use case, the reservation would last until heat releases it. Hopefully heat will persist its reservation information, so that a process crash will not cause a dangling reservation; how will you enable complete avoidance of timing splinters (e.g., heat engine crashes after making reservation but before persisting information about it)? Presumably other things besides heat could own reservations. Is this primarily about planning for the distant future, or focused on the immediate future? If a bag of
Re: [openstack-dev] [Climate] Questions and comments
Do not worry about what I want, right now I am just trying to understand the Climate proposal, wrt virtual resources (Patrick helped a lot on the physical side). Can you please walk through a scenario involving Climate reservations on virtual resources? I mean from start to finish, outlining which party makes which decision, based on what. Thanks, Mike From: Sylvain Bauza sylvain.ba...@bull.net To: OpenStack Development Mailing List openstack-dev@lists.openstack.org, Cc: Mike Spreitzer/Watson/IBM@IBMUS Date: 10/07/2013 05:07 AM Subject:Re: [openstack-dev] [Climate] Questions and comments Hi Mike, Dina and you outlined some differences in terms of seeing what is dependent on what. As Dina explained, Climate plans to be integrated into Nova and Heat logics, where Heat and Nova would request Climate API by asking for a lease and would tag on their own the resources as 'RESERVED'. On your point, and correct me if I'm wrong, you would rather see Climate on top of Heat and Nova, scheduling resources on its own, and only send creation requests to Heat and Nova. I'm happy to say both of you are right : Climate aims to be both called by Nova and *also* calling Nova. That's just matter of what Climate *is*. And here is the confusion. That's why Climate is not only one API endpoint. It actually have two distinct endpoints : one called the Lease API endpoint, and one called the Resource Reservation API endpoint. As a Climate developer working on physical hosts reservations (and not Heat stacks), my concern is to be able to guarantee to a REST client (either a user or another service) that if this user wants to provision X hosts on a specific timeframe in the future (immediate or in 10 years), Climate will be able to provision them. By meaning being able and guarantee, I do use strong words for stating that we engage ourselves to be able to plan what will be resources capacity state in the future. This decision-making process (ie. this Climate scheduler) will be implemented as RPC Service for the Reservation API, and thus will needs to keep its own persistence layer in Climate. Of course, it will request the Lease API for really creating the lease and managing lease start/end hooks, that's the Lease API job. Provided you would want to use the Reservation API for reserving Heat stacks, you would have to implement it tho. Thanks, -Sylvain Le 06/10/2013 20:41, Mike Spreitzer a écrit : Thanks, Dina. Yes, we do not understand each other; can I ask some more questions? You outlined a two-step reservation process (We assume the following reservation process for the OpenStack services...), and right after that talked about changing your mind to use Heat instead of individual services. So I am confused, I am not sure which of your remarks reflect your current thinking and which reflect old thinking. Can you just state your current thinking? On what basis would Climate decide to start or stop a lease? What sort of event notifications would Climate be sending, and when and why, and what would subscribers do upon receipt of such notifications? If the individual resource services continue to make independent scheduling decisions as they do today, what value does Climate add? Maybe a little more detailed outline of what happens in your current thinking, in support of an explicitly stated use case that shows the value, would help here. Thanks, Mike _ __ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Climate] Questions and comments
Hello, Mike! As for Nova connected thoughts, it was just a reference to the Roadmap, that includes implementing the reservations opportunity first of all for the solid virtual resources (VMs, volumes, …) and next for the complex ones (Heat stacks, Savanna clusters, …). To implement complex reservations we need implementation of the simple ones. Climate will be the independent service, that will the the underlaying layer for other OpenStack services wishing not just start some virtual resource, but reserve it for the future with different policies. We assume the following reservation process for the OpenStack services: 1) User goes to service and and does all actions as usual only passing his/her wish to reserve resource, not really start it. 2) Service does all the usual actions, but not really starts resource. Instead of that is uses Climate to create lease and then Climate decides what events should happen to the lease - start, end, send notification about some event and so on. We were thinking about leases of resources from different services for the first time, but here we found some logical problem - nobody really needs just VM, just volume and just floating IP. To work together, volume should be attached to VM and IP should be assigned too. But this is just the thing that Heat does. That's why now we are thinking of lease as reserving the resources from one service - to reserve VM+volume+IP using Heat will be the better idea I suppose. As for scheduling part, in our current scheme we'll use services' schedulers cause now they do their work in the best way. That's why Climate now looks as the underlaying service, not the upper one. That way provides the opportunity to use all capabilities of existing services and do not proxy all the resource-creation preparations and calls when user really ants to reserve something. Also I think we do not really understand each other in the question of how any OpenStack service and Climate may use each other. We were thinking that any service goes to Climate to create lease with already existing resources IDs in services' DB. In Nova case it looks like Nova schedules VM, creates in DB instance with the 'RESERVED' status and then sends Climate lease creation request. And then Climate uses its internal schedule to do all the actions with that resource defined by plugin for the lease start/end. In this view nobody except Climate really owns lease with all the reservations. But I see you have some other view on how that process may look like. Although it was not the idea we were thinking about for the Climate. As for we were thinking about service-Climate workflow, service already will mark 'RESERVED' resource in its DB when it'll send lease creation request to the Climate. That's why that backing capacity will be already used by service and mentioned in the further scheduling process. Now we'll reserve resource right at the moment of lease creation, that means it's more useful for the not so far future. But also we are thinking to implement the opportunity of not only the immediate resources reservation, but that is questionable. Thank you, Dina On Sun, Oct 6, 2013 at 10:36 AM, Mike Spreitzer mspre...@us.ibm.com wrote: I looked at the blueprint ( https://blueprints.launchpad.net/heat/+spec/stacks-reservation) and associated wiki page (https://wiki.openstack.org/wiki/Heat/Reservation), and I have a few comments/questions. The wiki page has some remarks that are Nova-centric, and some other remarks that emphasize that Climate is not just about Nova, and I do not understand the relationship between these remarks. Is this a roadmapping thought (start with just Nova, expand to other services later), or the inclusion of some details (related to Nova) and omission of other similar details (related to the other services), or what? Will Climate be an independent service, or part of Nova, or what? What will be the atomic operations? I presume the primary interesting operation will be something like reserving a bag of resources, where that bag is allowed to contain any mixture of resources from any services. Have I got that right? What exactly does reserving a resource mean? Does this atomic reservation operation include some atomic cooperation from the resources' services' schedulers (e.g., Nova scheduler, Cinder scheduler, etc)? Or is this reservation service logically independent of the resources' primary schedulers? Overall I am getting the suggestion that reservation is an independent service. The flow is something like first reserve a bag of resources, and then proceed to use them at your leisure. But I also suppose that the important thing about a reservation is that it includes the result of scheduling (placement) --- the point of a reservation is that it is holding capacity to host the reserved resources. You do not want an atomic operation to take a long time; do the scheduling decisions get made (tentatively, of
Re: [openstack-dev] [Climate] Questions and comments
Thanks, Dina. Yes, we do not understand each other; can I ask some more questions? You outlined a two-step reservation process (We assume the following reservation process for the OpenStack services...), and right after that talked about changing your mind to use Heat instead of individual services. So I am confused, I am not sure which of your remarks reflect your current thinking and which reflect old thinking. Can you just state your current thinking? On what basis would Climate decide to start or stop a lease? What sort of event notifications would Climate be sending, and when and why, and what would subscribers do upon receipt of such notifications? If the individual resource services continue to make independent scheduling decisions as they do today, what value does Climate add? Maybe a little more detailed outline of what happens in your current thinking, in support of an explicitly stated use case that shows the value, would help here. Thanks, Mike___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
