Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler
Second, there is nothing wrong with booting the instances (or instantiating other resources) as separate commands as long as we support some kind of reservation token. I'm not sure what reservation token would do, is it some kind of informing the scheduler that the resources would not be initiated until later ? Let's consider a following example: A user wants to create 2 VMs, a small one with 20 GB RAM, and a big one with 40 GB RAM in a datacenter consisted of 2 hosts: one with 50 GB RAM left, and another with 30 GB RAM left, using Filter Scheduler's default RamWeigher. If we pass the demand as two commands, there is a chance that the small VM arrives first. RamWeigher will put it in the 50 GB RAM host, which will be reduced to 30 GB RAM. Then, when the big VM request arrives, there will be no space left to host it. As a result, the whole demand is failed. Now if we can pass the two VMs in a command, SolverScheduler can put their constraints all together into one big LP as follow (x_uv = 1 if VM u is hosted in host v, 0 if not): 50GB RAM host constraint: 20 *x_11 + 40 * x_21 =50 30GB RAM host constraint: 20 *x_12 + 40 * x_22 =30 Small VM presence constraint:x_11 + x_12 = 1 Big VM presence constraint:x_21 + x_22 = 1 From these constraints there is only one root that is: x_11 = 0, x12 = 1; x_21 = 1; x_22 = 0; i.e, small VM hosted in 30 GB RAM host, and big VM hosted in 50 GB RAM host. As a conclusion, if we have VMs of multiple flavors to deal with, we cannot give the correct answer if we do not have all information. Therefore, if by reservation you mean that the scheduler would hold off the scheduling process and save the information until it receives all necessary information, then I'm agreed. But it just a workaround of passing the whole demand as a whole, which would better be handled by an API. That responses to your first point, too. If we don't mind that some VMs are placed and some are not (e.g. they belong to different apps), then it's OK to pass them to the scheduler without Instance Group. However, if the VMs are together (belong to an app), then we have to put them into an Instance Group. -Message d'origine- De : Chris Friesen [mailto:chris.frie...@windriver.com] Envoyé : lundi 10 février 2014 18:45 À : openstack-dev@lists.openstack.org Objet : Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler On 02/10/2014 10:54 AM, Khanh-Toan Tran wrote: Heat may orchestrate the provisioning process, but eventually the instances will be passed to Nova-scheduler (Gantt) as separated commands, which is exactly the problem Solver Scheduler wants to correct. Therefore the Instance Group API is needed, wherever it is used (nova-scheduler/Gantt). I'm not sure that this follows. First, the instance groups API is totally separate since we may want to schedule a number of instances simultaneously without them being part of an instance group. Certainly in the case of using instance groups that would be one input into the scheduler, but it's an optional input. Second, there is nothing wrong with booting the instances (or instantiating other resources) as separate commands as long as we support some kind of reservation token. In that model, we would pass a bunch of information about multiple resources to the solver scheduler, have it perform scheduling *and reserve the resources*, then return some kind of resource reservation tokens back to the caller for each resource. The caller could then allocate each resource, pass in the reservation token indicating both that the resources had already been reserved as well as what the specific resource that had been reserved (the compute-host in the case of an instance, for example). Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler
2014-02-10 18:45 GMT+01:00 Chris Friesen chris.frie...@windriver.com: In that model, we would pass a bunch of information about multiple resources to the solver scheduler, have it perform scheduling *and reserve the resources*, then return some kind of resource reservation tokens back to the caller for each resource. The caller could then allocate each resource, pass in the reservation token indicating both that the resources had already been reserved as well as what the specific resource that had been reserved (the compute-host in the case of an instance, for example). Chris Please be aware that Climate [1] already exists for managing resources reservations. That doesn't make sense and has been discussed during last summit that reservations should be managed by Nova, but rather by another service. -Sylvain [1] : https://launchpad.net/climate ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler
On 02/11/2014 03:21 AM, Khanh-Toan Tran wrote: Second, there is nothing wrong with booting the instances (or instantiating other resources) as separate commands as long as we support some kind of reservation token. I'm not sure what reservation token would do, is it some kind of informing the scheduler that the resources would not be initiated until later ? Like a restaurant reservation, it would claim the resources for use by someone at a later date. That way nobody else can use them. That way the scheduler would be responsible for determining where the resource should be allocated from, and getting a reservation for that resource. It would not have anything to do with actually instantiating the instance/volume/etc. Let's consider a following example: A user wants to create 2 VMs, a small one with 20 GB RAM, and a big one with 40 GB RAM in a datacenter consisted of 2 hosts: one with 50 GB RAM left, and another with 30 GB RAM left, using Filter Scheduler's default RamWeigher. If we pass the demand as two commands, there is a chance that the small VM arrives first. RamWeigher will put it in the 50 GB RAM host, which will be reduced to 30 GB RAM. Then, when the big VM request arrives, there will be no space left to host it. As a result, the whole demand is failed. Now if we can pass the two VMs in a command, SolverScheduler can put their constraints all together into one big LP as follow (x_uv = 1 if VM u is hosted in host v, 0 if not): Yes. So what I'm suggesting is that we schedule the two VMs as one call to the SolverScheduler. The scheduler then gets reservations for the necessary resources and returns them to the caller. This would be sort of like the existing Claim object in nova/compute/claims.py but generalized somewhat to other resources as well. The caller could then boot each instance separately (passing the appropriate reservation/claim along with the boot request). Because the caller has a reservation the core code would know it doesn't need to schedule or allocate resources, that's already been done. The advantage of this is that the scheduling and resource allocation is done separately from the instantiation. The instantiation API could remain basically as-is except for supporting an optional reservation token. That responses to your first point, too. If we don't mind that some VMs are placed and some are not (e.g. they belong to different apps), then it's OK to pass them to the scheduler without Instance Group. However, if the VMs are together (belong to an app), then we have to put them into an Instance Group. When I think of an Instance Group, I think of https://blueprints.launchpad.net/nova/+spec/instance-group-api-extension;. Fundamentally Instance Groups describes a runtime relationship between different instances. The scheduler doesn't necessarily care about a runtime relationship, it's just trying to allocate resources efficiently. In the above example, there is no need for those two instances to necessarily be part of an Instance Group--we just want to schedule them both at the same time to give the scheduler a better chance of fitting them both. More generally, the more instances I want to start up the more beneficial it can be to pass them all to the scheduler at once in order to give the scheduler more information. Those instances could be parts of completely independent Instance Groups, or not part of an Instance Group at all...the scheduler can still do a better job if it has more information to work with. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler
Le 11/02/2014 16:09, Chris Friesen a écrit : Yes. So what I'm suggesting is that we schedule the two VMs as one call to the SolverScheduler. The scheduler then gets reservations for the necessary resources and returns them to the caller. This would be sort of like the existing Claim object in nova/compute/claims.py but generalized somewhat to other resources as well. The caller could then boot each instance separately (passing the appropriate reservation/claim along with the boot request). Because the caller has a reservation the core code would know it doesn't need to schedule or allocate resources, that's already been done. The advantage of this is that the scheduling and resource allocation is done separately from the instantiation. The instantiation API could remain basically as-is except for supporting an optional reservation token. I think you really need to look at what Climate is, but I don't want to be boring... Climate can provide you some way for doing the reservation, and possibly you would only need to write a plugin for such a request. -Sylvain ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler
Like a restaurant reservation, it would claim the resources for use by someone at a later date. That way nobody else can use them. That way the scheduler would be responsible for determining where the resource should be allocated from, and getting a reservation for that resource. It would not have anything to do with actually instantiating the instance/volume/etc. Although I'm quite new to topic of Solver Scheduler, it seems to me that in that case you need to look on Climate project. It aims to provide resource reservation to OS clouds (and by resource I mean here instance/compute host/volume/etc.) And Climate logic is like: create lease - get resources from common pool - do smth with them when lease start time will come. I'll say one more time - I'm not really common with this discussion, but it looks like Climate might help here. Thanks Dina On Tue, Feb 11, 2014 at 7:09 PM, Chris Friesen chris.frie...@windriver.comwrote: On 02/11/2014 03:21 AM, Khanh-Toan Tran wrote: Second, there is nothing wrong with booting the instances (or instantiating other resources) as separate commands as long as we support some kind of reservation token. I'm not sure what reservation token would do, is it some kind of informing the scheduler that the resources would not be initiated until later ? Like a restaurant reservation, it would claim the resources for use by someone at a later date. That way nobody else can use them. That way the scheduler would be responsible for determining where the resource should be allocated from, and getting a reservation for that resource. It would not have anything to do with actually instantiating the instance/volume/etc. Let's consider a following example: A user wants to create 2 VMs, a small one with 20 GB RAM, and a big one with 40 GB RAM in a datacenter consisted of 2 hosts: one with 50 GB RAM left, and another with 30 GB RAM left, using Filter Scheduler's default RamWeigher. If we pass the demand as two commands, there is a chance that the small VM arrives first. RamWeigher will put it in the 50 GB RAM host, which will be reduced to 30 GB RAM. Then, when the big VM request arrives, there will be no space left to host it. As a result, the whole demand is failed. Now if we can pass the two VMs in a command, SolverScheduler can put their constraints all together into one big LP as follow (x_uv = 1 if VM u is hosted in host v, 0 if not): Yes. So what I'm suggesting is that we schedule the two VMs as one call to the SolverScheduler. The scheduler then gets reservations for the necessary resources and returns them to the caller. This would be sort of like the existing Claim object in nova/compute/claims.py but generalized somewhat to other resources as well. The caller could then boot each instance separately (passing the appropriate reservation/claim along with the boot request). Because the caller has a reservation the core code would know it doesn't need to schedule or allocate resources, that's already been done. The advantage of this is that the scheduling and resource allocation is done separately from the instantiation. The instantiation API could remain basically as-is except for supporting an optional reservation token. That responses to your first point, too. If we don't mind that some VMs are placed and some are not (e.g. they belong to different apps), then it's OK to pass them to the scheduler without Instance Group. However, if the VMs are together (belong to an app), then we have to put them into an Instance Group. When I think of an Instance Group, I think of https://blueprints.launchpad.net/nova/+spec/instance-group-api-extension;. Fundamentally Instance Groups describes a runtime relationship between different instances. The scheduler doesn't necessarily care about a runtime relationship, it's just trying to allocate resources efficiently. In the above example, there is no need for those two instances to necessarily be part of an Instance Group--we just want to schedule them both at the same time to give the scheduler a better chance of fitting them both. More generally, the more instances I want to start up the more beneficial it can be to pass them all to the scheduler at once in order to give the scheduler more information. Those instances could be parts of completely independent Instance Groups, or not part of an Instance Group at all...the scheduler can still do a better job if it has more information to work with. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Best regards, Dina Belova Software Engineer Mirantis Inc. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler
Thanks, I will look closely at it. De : Dina Belova [mailto:dbel...@mirantis.com] Envoyé : mardi 11 février 2014 16:45 À : OpenStack Development Mailing List (not for usage questions) Objet : Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler Like a restaurant reservation, it would claim the resources for use by someone at a later date. That way nobody else can use them. That way the scheduler would be responsible for determining where the resource should be allocated from, and getting a reservation for that resource. It would not have anything to do with actually instantiating the instance/volume/etc. Although I'm quite new to topic of Solver Scheduler, it seems to me that in that case you need to look on Climate project. It aims to provide resource reservation to OS clouds (and by resource I mean here instance/compute host/volume/etc.) And Climate logic is like: create lease - get resources from common pool - do smth with them when lease start time will come. I'll say one more time - I'm not really common with this discussion, but it looks like Climate might help here. Thanks Dina On Tue, Feb 11, 2014 at 7:09 PM, Chris Friesen chris.frie...@windriver.com wrote: On 02/11/2014 03:21 AM, Khanh-Toan Tran wrote: Second, there is nothing wrong with booting the instances (or instantiating other resources) as separate commands as long as we support some kind of reservation token. I'm not sure what reservation token would do, is it some kind of informing the scheduler that the resources would not be initiated until later ? Like a restaurant reservation, it would claim the resources for use by someone at a later date. That way nobody else can use them. That way the scheduler would be responsible for determining where the resource should be allocated from, and getting a reservation for that resource. It would not have anything to do with actually instantiating the instance/volume/etc. Let's consider a following example: A user wants to create 2 VMs, a small one with 20 GB RAM, and a big one with 40 GB RAM in a datacenter consisted of 2 hosts: one with 50 GB RAM left, and another with 30 GB RAM left, using Filter Scheduler's default RamWeigher. If we pass the demand as two commands, there is a chance that the small VM arrives first. RamWeigher will put it in the 50 GB RAM host, which will be reduced to 30 GB RAM. Then, when the big VM request arrives, there will be no space left to host it. As a result, the whole demand is failed. Now if we can pass the two VMs in a command, SolverScheduler can put their constraints all together into one big LP as follow (x_uv = 1 if VM u is hosted in host v, 0 if not): Yes. So what I'm suggesting is that we schedule the two VMs as one call to the SolverScheduler. The scheduler then gets reservations for the necessary resources and returns them to the caller. This would be sort of like the existing Claim object in nova/compute/claims.py but generalized somewhat to other resources as well. The caller could then boot each instance separately (passing the appropriate reservation/claim along with the boot request). Because the caller has a reservation the core code would know it doesn't need to schedule or allocate resources, that's already been done. The advantage of this is that the scheduling and resource allocation is done separately from the instantiation. The instantiation API could remain basically as-is except for supporting an optional reservation token. That responses to your first point, too. If we don't mind that some VMs are placed and some are not (e.g. they belong to different apps), then it's OK to pass them to the scheduler without Instance Group. However, if the VMs are together (belong to an app), then we have to put them into an Instance Group. When I think of an Instance Group, I think of https://blueprints.launchpad.net/nova/+spec/instance-group-api-extension; . Fundamentally Instance Groups describes a runtime relationship between different instances. The scheduler doesn't necessarily care about a runtime relationship, it's just trying to allocate resources efficiently. In the above example, there is no need for those two instances to necessarily be part of an Instance Group--we just want to schedule them both at the same time to give the scheduler a better chance of fitting them both. More generally, the more instances I want to start up the more beneficial it can be to pass them all to the scheduler at once in order to give the scheduler more information. Those instances could be parts of completely independent Instance Groups, or not part of an Instance Group at all...the scheduler can still do a better job if it has more information to work with. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Best regards, Dina Belova
Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler
I thought of adding some more points about the Solver Scheduler to this conversation. Think of SolverScheduler as a placement decision engine, which gives an optimal solution for the specified request based on the current information available at a specific time. The request could potentially be a set of instances of the same flavor or different flavors (assuming we eventually support scheduler APIs that can provide this). Once the optimal placement decision is known, then we need to allocate the resources. Currently Nova supports the final allocation of resources (resource provisioning) one at a time. I definitely agree there will be more success in allocating all the requested instances, if there is a support for reserving the spots as soon as a placement decision is taken by the SolverScheduler. This is something to explore to add in Nova, using a local service or external service (need to explore Climate). If the required set of instances is known, irrespective of whether it is part of one instance group, or multiple instance groups, you always get a more optimal solution, if all of this is requested in one shot to the solver scheduler, as the constraint solving happens in one shot, letting you know if the entire set of instances is feasible to be placed given the existing resource capacity. If it is okay to support partial instantiation of subset of instances, then it makes sense to provide support to retry one instance group at a time, when the entire request is not feasible. To add another point about instance group api implementation for icehouse, it was decided in the HongKong summit to initially support only flat instance groups without nesting. Hence if an application requires a big topology of instances, they can easily belong to multiple instance groups, and hence if you want the entire application requirement to be satisfied, the entire set of instances from multiple flat instance groups should be requested as a single shot to the solver scheduler. Also, there is additional work required to add new scheduler APIs to support requesting instance groups of multiple flavors. I think I have reiterated some of the points what Chris has mentioned below. But Yes, like I had stated earlier in this thread, we need to separate the decision making phase from the initial request making, and the final allocation provisioning (or orchestration). In these phases, a reservation phase, after the decision making, will add additional guarantees to allocate the placed instances. Thanks, Yathi. On 2/11/14, 7:09 AM, Chris Friesen chris.frie...@windriver.com wrote: On 02/11/2014 03:21 AM, Khanh-Toan Tran wrote: Second, there is nothing wrong with booting the instances (or instantiating other resources) as separate commands as long as we support some kind of reservation token. I'm not sure what reservation token would do, is it some kind of informing the scheduler that the resources would not be initiated until later ? Like a restaurant reservation, it would claim the resources for use by someone at a later date. That way nobody else can use them. That way the scheduler would be responsible for determining where the resource should be allocated from, and getting a reservation for that resource. It would not have anything to do with actually instantiating the instance/volume/etc. Let's consider a following example: A user wants to create 2 VMs, a small one with 20 GB RAM, and a big one with 40 GB RAM in a datacenter consisted of 2 hosts: one with 50 GB RAM left, and another with 30 GB RAM left, using Filter Scheduler's default RamWeigher. If we pass the demand as two commands, there is a chance that the small VM arrives first. RamWeigher will put it in the 50 GB RAM host, which will be reduced to 30 GB RAM. Then, when the big VM request arrives, there will be no space left to host it. As a result, the whole demand is failed. Now if we can pass the two VMs in a command, SolverScheduler can put their constraints all together into one big LP as follow (x_uv = 1 if VM u is hosted in host v, 0 if not): Yes. So what I'm suggesting is that we schedule the two VMs as one call to the SolverScheduler. The scheduler then gets reservations for the necessary resources and returns them to the caller. This would be sort of like the existing Claim object in nova/compute/claims.py but generalized somewhat to other resources as well. The caller could then boot each instance separately (passing the appropriate reservation/claim along with the boot request). Because the caller has a reservation the core code would know it doesn't need to schedule or allocate resources, that's already been done. The advantage of this is that the scheduling and resource allocation is done separately from the instantiation. The instantiation API could remain basically as-is except for supporting an optional reservation token. That responses to your first point, too. If we don't mind that some VMs are
Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler
Hi Dina, Thanks for note about Climate logic. This is something that will be very useful, when we will have to schedule from Nova multiple instances (of potentially different flavors) as a single request. If the Solver Scheduler, can make a request to the Climate service to reserve the resources soon after the placement decision has been made, then the nova provisioning logic can handle the resource provisioning using the climate reserved leases. Regarding Solver Scheduler for your reference, just sent another email about this with some pointers about it. Otherwise this is the blueprint - https://blueprints.launchpad.net/nova/+spec/solver-scheduler I guess this is something to explore more and see how Nova provisioning logic to work with Climate leases. Or this is something that already works. I need to find out more about Climate. Thanks, Yathi. On 2/11/14, 7:44 AM, Dina Belova dbel...@mirantis.commailto:dbel...@mirantis.com wrote: Like a restaurant reservation, it would claim the resources for use by someone at a later date. That way nobody else can use them. That way the scheduler would be responsible for determining where the resource should be allocated from, and getting a reservation for that resource. It would not have anything to do with actually instantiating the instance/volume/etc. Although I'm quite new to topic of Solver Scheduler, it seems to me that in that case you need to look on Climate project. It aims to provide resource reservation to OS clouds (and by resource I mean here instance/compute host/volume/etc.) And Climate logic is like: create lease - get resources from common pool - do smth with them when lease start time will come. I'll say one more time - I'm not really common with this discussion, but it looks like Climate might help here. Thanks Dina On Tue, Feb 11, 2014 at 7:09 PM, Chris Friesen chris.frie...@windriver.commailto:chris.frie...@windriver.com wrote: On 02/11/2014 03:21 AM, Khanh-Toan Tran wrote: Second, there is nothing wrong with booting the instances (or instantiating other resources) as separate commands as long as we support some kind of reservation token. I'm not sure what reservation token would do, is it some kind of informing the scheduler that the resources would not be initiated until later ? Like a restaurant reservation, it would claim the resources for use by someone at a later date. That way nobody else can use them. That way the scheduler would be responsible for determining where the resource should be allocated from, and getting a reservation for that resource. It would not have anything to do with actually instantiating the instance/volume/etc. Let's consider a following example: A user wants to create 2 VMs, a small one with 20 GB RAM, and a big one with 40 GB RAM in a datacenter consisted of 2 hosts: one with 50 GB RAM left, and another with 30 GB RAM left, using Filter Scheduler's default RamWeigher. If we pass the demand as two commands, there is a chance that the small VM arrives first. RamWeigher will put it in the 50 GB RAM host, which will be reduced to 30 GB RAM. Then, when the big VM request arrives, there will be no space left to host it. As a result, the whole demand is failed. Now if we can pass the two VMs in a command, SolverScheduler can put their constraints all together into one big LP as follow (x_uv = 1 if VM u is hosted in host v, 0 if not): Yes. So what I'm suggesting is that we schedule the two VMs as one call to the SolverScheduler. The scheduler then gets reservations for the necessary resources and returns them to the caller. This would be sort of like the existing Claim object in nova/compute/claims.py but generalized somewhat to other resources as well. The caller could then boot each instance separately (passing the appropriate reservation/claim along with the boot request). Because the caller has a reservation the core code would know it doesn't need to schedule or allocate resources, that's already been done. The advantage of this is that the scheduling and resource allocation is done separately from the instantiation. The instantiation API could remain basically as-is except for supporting an optional reservation token. That responses to your first point, too. If we don't mind that some VMs are placed and some are not (e.g. they belong to different apps), then it's OK to pass them to the scheduler without Instance Group. However, if the VMs are together (belong to an app), then we have to put them into an Instance Group. When I think of an Instance Group, I think of https://blueprints.launchpad.net/nova/+spec/instance-group-api-extension;. Fundamentally Instance Groups describes a runtime relationship between different instances. The scheduler doesn't necessarily care about a runtime relationship, it's just trying to allocate resources efficiently. In the above example, there is no need for those two instances to necessarily be part of an
Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler
Le 11/02/2014 17:23, Yathiraj Udupi (yudupi) a écrit : Hi Dina, Thanks for note about Climate logic. This is something that will be very useful, when we will have to schedule from Nova multiple instances (of potentially different flavors) as a single request. If the Solver Scheduler, can make a request to the Climate service to reserve the resources soon after the placement decision has been made, then the nova provisioning logic can handle the resource provisioning using the climate reserved leases. Regarding Solver Scheduler for your reference, just sent another email about this with some pointers about it. Otherwise this is the blueprint - https://blueprints.launchpad.net/nova/+spec/solver-scheduler I guess this is something to explore more and see how Nova provisioning logic to work with Climate leases. Or this is something that already works. I need to find out more about Climate. Thanks, Yathi. There are possibly 2 ways for creating a lease : either thru the CLI or by the python binding. We implemented these 2 possibilities within the current Climate 0.1 release : - a Nova extension plugin is responsible for creating the lease if a VM should be reserved (using the Climate pythonclient binding) - an user can request for reserving a compute host using the Climate python client directly Both logics (VM and compute host) are actually referring to 2 distinct plugins in the Climate manager, so the actions are completely different. Based on your use-case, you could imagine a call from the SolverScheduler to Climate for creating a lease containing multiple VM reservations, and either you would use the Climate VM plugin or you would use a dedicated plugin if your need is different. I don't think that's a huge volume of work, as Climate already defines and implements the main features that you need. -Sylvain ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler
This is something to explore to add in Nova, using a local service or external service (need to explore Climate). I need to find out more about Climate. Here is Climate Launchpad: https://launchpad.net/climate That's still really young project, but I believe it'll have great future speaking about resource reservation. So if you need some kind of reservation logic, I also believe that should be implemented in Climate (as it's proposed and currently implemented as Reservation-as-a-Service) Thanks On Tue, Feb 11, 2014 at 8:23 PM, Yathiraj Udupi (yudupi) yud...@cisco.comwrote: Hi Dina, Thanks for note about Climate logic. This is something that will be very useful, when we will have to schedule from Nova multiple instances (of potentially different flavors) as a single request. If the Solver Scheduler, can make a request to the Climate service to reserve the resources soon after the placement decision has been made, then the nova provisioning logic can handle the resource provisioning using the climate reserved leases. Regarding Solver Scheduler for your reference, just sent another email about this with some pointers about it. Otherwise this is the blueprint - https://blueprints.launchpad.net/nova/+spec/solver-scheduler I guess this is something to explore more and see how Nova provisioning logic to work with Climate leases. Or this is something that already works. I need to find out more about Climate. Thanks, Yathi. On 2/11/14, 7:44 AM, Dina Belova dbel...@mirantis.com wrote: Like a restaurant reservation, it would claim the resources for use by someone at a later date. That way nobody else can use them. That way the scheduler would be responsible for determining where the resource should be allocated from, and getting a reservation for that resource. It would not have anything to do with actually instantiating the instance/volume/etc. Although I'm quite new to topic of Solver Scheduler, it seems to me that in that case you need to look on Climate project. It aims to provide resource reservation to OS clouds (and by resource I mean here instance/compute host/volume/etc.) And Climate logic is like: create lease - get resources from common pool - do smth with them when lease start time will come. I'll say one more time - I'm not really common with this discussion, but it looks like Climate might help here. Thanks Dina On Tue, Feb 11, 2014 at 7:09 PM, Chris Friesen chris.frie...@windriver.com wrote: On 02/11/2014 03:21 AM, Khanh-Toan Tran wrote: Second, there is nothing wrong with booting the instances (or instantiating other resources) as separate commands as long as we support some kind of reservation token. I'm not sure what reservation token would do, is it some kind of informing the scheduler that the resources would not be initiated until later ? Like a restaurant reservation, it would claim the resources for use by someone at a later date. That way nobody else can use them. That way the scheduler would be responsible for determining where the resource should be allocated from, and getting a reservation for that resource. It would not have anything to do with actually instantiating the instance/volume/etc. Let's consider a following example: A user wants to create 2 VMs, a small one with 20 GB RAM, and a big one with 40 GB RAM in a datacenter consisted of 2 hosts: one with 50 GB RAM left, and another with 30 GB RAM left, using Filter Scheduler's default RamWeigher. If we pass the demand as two commands, there is a chance that the small VM arrives first. RamWeigher will put it in the 50 GB RAM host, which will be reduced to 30 GB RAM. Then, when the big VM request arrives, there will be no space left to host it. As a result, the whole demand is failed. Now if we can pass the two VMs in a command, SolverScheduler can put their constraints all together into one big LP as follow (x_uv = 1 if VM u is hosted in host v, 0 if not): Yes. So what I'm suggesting is that we schedule the two VMs as one call to the SolverScheduler. The scheduler then gets reservations for the necessary resources and returns them to the caller. This would be sort of like the existing Claim object in nova/compute/claims.py but generalized somewhat to other resources as well. The caller could then boot each instance separately (passing the appropriate reservation/claim along with the boot request). Because the caller has a reservation the core code would know it doesn't need to schedule or allocate resources, that's already been done. The advantage of this is that the scheduling and resource allocation is done separately from the instantiation. The instantiation API could remain basically as-is except for supporting an optional reservation token. That responses to your first point, too. If we don't mind that some VMs are placed and some are not (e.g. they belong to different apps),
Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler
hi Sylvain and Dina, Thanks for your pointers about Climate. I will take a closer look at it and try it out. So after a reservation lease for a VM is made by Climate, who acts on it to finally instantiate the VM ? Is it Climate or Nova should act on the lease to finally provision the VM. Thanks, Yathi. On 2/11/14, 8:42 AM, Sylvain Bauza sylvain.ba...@bull.netmailto:sylvain.ba...@bull.net wrote: Le 11/02/2014 17:23, Yathiraj Udupi (yudupi) a écrit : Hi Dina, Thanks for note about Climate logic. This is something that will be very useful, when we will have to schedule from Nova multiple instances (of potentially different flavors) as a single request. If the Solver Scheduler, can make a request to the Climate service to reserve the resources soon after the placement decision has been made, then the nova provisioning logic can handle the resource provisioning using the climate reserved leases. Regarding Solver Scheduler for your reference, just sent another email about this with some pointers about it. Otherwise this is the blueprint - https://blueprints.launchpad.net/nova/+spec/solver-scheduler I guess this is something to explore more and see how Nova provisioning logic to work with Climate leases. Or this is something that already works. I need to find out more about Climate. Thanks, Yathi. There are possibly 2 ways for creating a lease : either thru the CLI or by the python binding. We implemented these 2 possibilities within the current Climate 0.1 release : - a Nova extension plugin is responsible for creating the lease if a VM should be reserved (using the Climate pythonclient binding) - an user can request for reserving a compute host using the Climate python client directly Both logics (VM and compute host) are actually referring to 2 distinct plugins in the Climate manager, so the actions are completely different. Based on your use-case, you could imagine a call from the SolverScheduler to Climate for creating a lease containing multiple VM reservations, and either you would use the Climate VM plugin or you would use a dedicated plugin if your need is different. I don't think that's a huge volume of work, as Climate already defines and implements the main features that you need. -Sylvain ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler
2014-02-11 18:28 GMT+01:00 Yathiraj Udupi (yudupi) yud...@cisco.com: Thanks for your pointers about Climate. I will take a closer look at it and try it out. So after a reservation lease for a VM is made by Climate, who acts on it to finally instantiate the VM ? Is it Climate or Nova should act on the lease to finally provision the VM. That depends on the plugin mapped with the reservation type. The current plugin mapped for virtual instances does unshelve the instance at lease start (because Nova extension shelved it at lease creation). If you don't like this behaviour, that's quite easy to create another plugin for another reservation type. When the lease starts, it does run the lease_create() action defined by the plugin. So, if you don't like shelving/unshelving, that's not a problem, feel free to add a blueprint about your needs or create your own plugin. -Sylvain ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler
I'm not sure what you mean by whatever it takes to finally create the instances, but that sounds like what I had assumed everybody meant by orchestration (until I heard that there is no widespread agreement) --- and I think we need to take a properly open approach to that. I think the proper API for cross-service whole-pattern scheduling should primarily focus on conveying the placement problem to the thing that will make the joint decision. After the joint decision is made comes the time to create the individual resources. I think we can NOT mandate one particular agent or language for that. We will have to allow general clients to make calls on Nova, Cinder, etc. to do the individual resource creations (with some sort of reference to the decision that was already made). My original position was that we could use Heat for this, but I think we have gotten push-back saying it is NOT OK to *require* that. For example, note that some people do not want to use Heat at all, they prefer to make individual calls on Nova, Cinder, etc. Of course, we definitely want to support, among others, the people who *do* use Heat. I do not think Heat would be appropriate, either. Heat does not have the detailed knowledges on the infrastructure and it uses Nova (Gantt) API to pass the command, so if Nova (Gantt) API does not support multiple instances provisioning, Heat will not get the joint decision for all VMs as a whole. Heat may orchestrate the provisioning process, but eventually the instances will be passed to Nova-scheduler (Gantt) as separated commands, which is exactly the problem Solver Scheduler wants to correct. Therefore the Instance Group API is needed, wherever it is used (nova-scheduler/Gantt). I think Gantt should be the cross-service joint decision point. Heat still keeps orchestrating processes like it always does, but the provisioning decision has to be made all together as one atomic step in Heat's whole process. Here's a more detailed description of our thoughts on how such a protocol might look: https://wiki.openstack.org/wiki/Nova/PlacementAdvisorAndEngine We've concentrated on the Nova scheduler; Would be interesting to see if this protocol aligns with Yathiraj's thoughts on a global scheduler addressing compute+storage+network. Feedback is most welcome. Thank you for the link, I will give it a try. Best regards, Toan - Original Message - From: Mike Spreitzer mspre...@us.ibm.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Tuesday, February 4, 2014 9:05:22 AM Subject: Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler From: Khanh-Toan Tran khanh-toan.t...@cloudwatt.com ... There is an unexpected line break in the middle of the link, so I post it again: https://docs.google.com/document/d/1RfP7jRsw1mXMjd7in72ARjK0fTrsQv1bqolOri IQB2Y The mailing list software keeps inserting that line break. I re-constructed the URL and looked at the document. As you point out at the end, the way you attempt to formulate load balancing as a linear objective does not work. I think load-balancing is a non-linear thing. I also doubt that simple load balancing is what cloud providers want; I think cloud providers want to bunch up load, within limits, for example to keep some hosts idle so that they can be powered down to save on costs or left available for future exclusive use. From: Gil Rapaport g...@il.ibm.com ... As Alex Glikson hinted a couple of weekly meetings ago, our approach to this is to think of the driver's work as split between two entities: -- A Placement Advisor, that constructs placement problems for scheduling requests (filter-scheduler and policy-based-scheduler) -- A Placement Engine, that solves placement problems (HostManager in get_filtered_hosts() and solver-scheduler with its LP engine). Yes, I see the virtue in that separation. Let me egg it on a little. What Alex and KTT want is more structure in the Placement Advisor, where there is a multiplicity of plugins, each bound to some fraction of the whole system, and a protocol for combining the advice from the plugins. I would also like to remind you of another kind of structure: some of the placement desiderata come from the cloud users, and some from the cloud provider. From: Yathiraj Udupi (yudupi) yud...@cisco.com ... Like you point out, I do agree the two entities of placement advisor, and the placement engine, but I think there should be a third one – the provisioning engine, which should be responsible for whatever it takes to finally create the instances, after the placement decision has been taken. I'm not sure what you mean by whatever it takes to finally create the instances, but that sounds like what I had assumed everybody meant by orchestration (until I heard that there is no widespread agreement
Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler
Mike, exactly: we would like to allow flexibility complexity at the Advisor level without it affecting the placement computation. Advisors are expected to manifest complex behavior as suggested by these BPs and gather constraints from multiple sources (users and providers). The idea is indeed to define a protocol that can express placement requests without exposing the engine to complex/high-level/rapidly-changing/3rd-party semantics. I think BPs such as the group API and flexible-evacuation combined with the power of LP solvers Yathiraj mentioned do push the scheduler towards being a more generic placement oracle, so the protocol should probably not be limited to the current deploy one or more instances of the same kind request. Here's a more detailed description of our thoughts on how such a protocol might look: https://wiki.openstack.org/wiki/Nova/PlacementAdvisorAndEngine We've concentrated on the Nova scheduler; Would be interesting to see if this protocol aligns with Yathiraj's thoughts on a global scheduler addressing compute+storage+network. Feedback is most welcome. Regards, Gil From: Mike Spreitzer mspre...@us.ibm.com To: OpenStack Development Mailing List \(not for usage questions\) openstack-dev@lists.openstack.org, Date: 02/04/2014 10:10 AM Subject:Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler From: Khanh-Toan Tran khanh-toan.t...@cloudwatt.com ... There is an unexpected line break in the middle of the link, so I post it again: https://docs.google.com/document/d/1RfP7jRsw1mXMjd7in72ARjK0fTrsQv1bqolOri IQB2Y The mailing list software keeps inserting that line break. I re-constructed the URL and looked at the document. As you point out at the end, the way you attempt to formulate load balancing as a linear objective does not work. I think load-balancing is a non-linear thing. I also doubt that simple load balancing is what cloud providers want; I think cloud providers want to bunch up load, within limits, for example to keep some hosts idle so that they can be powered down to save on costs or left available for future exclusive use. From: Gil Rapaport g...@il.ibm.com ... As Alex Glikson hinted a couple of weekly meetings ago, our approach to this is to think of the driver's work as split between two entities: -- A Placement Advisor, that constructs placement problems for scheduling requests (filter-scheduler and policy-based-scheduler) -- A Placement Engine, that solves placement problems (HostManager in get_filtered_hosts() and solver-scheduler with its LP engine). Yes, I see the virtue in that separation. Let me egg it on a little. What Alex and KTT want is more structure in the Placement Advisor, where there is a multiplicity of plugins, each bound to some fraction of the whole system, and a protocol for combining the advice from the plugins. I would also like to remind you of another kind of structure: some of the placement desiderata come from the cloud users, and some from the cloud provider. From: Yathiraj Udupi (yudupi) yud...@cisco.com ... Like you point out, I do agree the two entities of placement advisor, and the placement engine, but I think there should be a third one – the provisioning engine, which should be responsible for whatever it takes to finally create the instances, after the placement decision has been taken. I'm not sure what you mean by whatever it takes to finally create the instances, but that sounds like what I had assumed everybody meant by orchestration (until I heard that there is no widespread agreement) --- and I think we need to take a properly open approach to that. I think the proper API for cross-service whole-pattern scheduling should primarily focus on conveying the placement problem to the thing that will make the joint decision. After the joint decision is made comes the time to create the individual resources. I think we can NOT mandate one particular agent or language for that. We will have to allow general clients to make calls on Nova, Cinder, etc. to do the individual resource creations (with some sort of reference to the decision that was already made). My original position was that we could use Heat for this, but I think we have gotten push-back saying it is NOT OK to *require* that. For example, note that some people do not want to use Heat at all, they prefer to make individual calls on Nova, Cinder, etc. Of course, we definitely want to support, among others, the people who *do* use Heat. From: Yathiraj Udupi (yudupi) yud...@cisco.com ... The solver-scheduler is designed to solve for an arbitrary list of instances of different flavors. We need to have some updated apis in the scheduler to be able to pass on such requests. Instance group api is an initial effort to specify such groups. I'll remind the other readers of our draft of such a thing, at https://docs.google.com
Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler
From: Khanh-Toan Tran khanh-toan.t...@cloudwatt.com ... There is an unexpected line break in the middle of the link, so I post it again: https://docs.google.com/document/d/1RfP7jRsw1mXMjd7in72ARjK0fTrsQv1bqolOri IQB2Y The mailing list software keeps inserting that line break. I re-constructed the URL and looked at the document. As you point out at the end, the way you attempt to formulate load balancing as a linear objective does not work. I think load-balancing is a non-linear thing. I also doubt that simple load balancing is what cloud providers want; I think cloud providers want to bunch up load, within limits, for example to keep some hosts idle so that they can be powered down to save on costs or left available for future exclusive use. From: Gil Rapaport g...@il.ibm.com ... As Alex Glikson hinted a couple of weekly meetings ago, our approach to this is to think of the driver's work as split between two entities: -- A Placement Advisor, that constructs placement problems for scheduling requests (filter-scheduler and policy-based-scheduler) -- A Placement Engine, that solves placement problems (HostManager in get_filtered_hosts() and solver-scheduler with its LP engine). Yes, I see the virtue in that separation. Let me egg it on a little. What Alex and KTT want is more structure in the Placement Advisor, where there is a multiplicity of plugins, each bound to some fraction of the whole system, and a protocol for combining the advice from the plugins. I would also like to remind you of another kind of structure: some of the placement desiderata come from the cloud users, and some from the cloud provider. From: Yathiraj Udupi (yudupi) yud...@cisco.com ... Like you point out, I do agree the two entities of placement advisor, and the placement engine, but I think there should be a third one – the provisioning engine, which should be responsible for whatever it takes to finally create the instances, after the placement decision has been taken. I'm not sure what you mean by whatever it takes to finally create the instances, but that sounds like what I had assumed everybody meant by orchestration (until I heard that there is no widespread agreement) --- and I think we need to take a properly open approach to that. I think the proper API for cross-service whole-pattern scheduling should primarily focus on conveying the placement problem to the thing that will make the joint decision. After the joint decision is made comes the time to create the individual resources. I think we can NOT mandate one particular agent or language for that. We will have to allow general clients to make calls on Nova, Cinder, etc. to do the individual resource creations (with some sort of reference to the decision that was already made). My original position was that we could use Heat for this, but I think we have gotten push-back saying it is NOT OK to *require* that. For example, note that some people do not want to use Heat at all, they prefer to make individual calls on Nova, Cinder, etc. Of course, we definitely want to support, among others, the people who *do* use Heat. From: Yathiraj Udupi (yudupi) yud...@cisco.com ... The solver-scheduler is designed to solve for an arbitrary list of instances of different flavors. We need to have some updated apis in the scheduler to be able to pass on such requests. Instance group api is an initial effort to specify such groups. I'll remind the other readers of our draft of such a thing, at https://docs.google.com/document/d/17OIiBoIavih-1y4zzK0oXyI66529f-7JTCVj-BcXURA (you will have to re-assemble the URL if the ML software broke it for you) My take-aways from the Icehouse summit's review of that are as follows. (1) do NOT put orchestration under the covers (as I said above), allow general clients to make the calls to create the individual resources. (2) The community was not convinced that this would scale as needed. (3) There were some remarks about too complicated but I am not clear on whether the issue(s) were: (a) there was not clear/compelling motivation for some of the expressiveness offered, (b) there is a simpler way to accomplish the same things, (c) it's all good but more than we can take on now, and/or (d) something else I did not get. Regards, Mike ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler
Nice idea. I also think that filters and weighers are FilterScheduler-specific. Thus, that is unnecessary for SolverScheduler to try to translate all filters/weighers into constraints. It would be easier for transition, though. Anyway, we just need some placement logics that will be written as constraints, as the they are currently represented as filters in FilterScheduler. So yes we will need a placement advisor here. For provisioning engine, isn't scheduler manager (or maybe nova-conductor) will be the one? I still don't figure out after we have gantt, how nova-conductor interact with gantt, or we put its logic into gantt, too. Another though would be the need for Instance Group API [1]. Currently users can only request multiple instances of the same flavors. These requests do not need LP to solve, just placing instances one by one is sufficient. Therefore we need this API so that users can request instances of different flavors, with some relations (constraints) among them. The advantage is that this logic and API will help us add Cinder volumes with ease (not sure how the Cinder-stackers think about it, though). Best regards, Toan [1] https://wiki.openstack.org/wiki/InstanceGroupApiExtension - Original Message - From: Yathiraj Udupi (yudupi) yud...@cisco.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: Thu, 30 Jan 2014 18:13:59 - (UTC) Subject: Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler It is really good we are reviving the conversation we started during the last summit in Hongkong during one of the scheduler sessions called “Smart resource placement”. This is the document we used to discuss during the session. Probably you may have seen this before: https://docs.google.com/document/d/1IiPI0sfaWb1bdYiMWzAAx0HYR6UqzOan_Utgml5W1HI/edit The idea is to separate out the logic for the placement decision engine from the actual request and the final provisioning phase. The placement engine itself can be pluggable, and as we show in the solver scheduler blueprint, we show how it fits inside of Nova. The discussions at the summit and in our weekly scheduler meetings led to us starting the “Smart resource placement” idea inside of Nova, and then take it to a unified global level spanning cross services such as cinder and neutron. Like you point out, I do agree the two entities of placement advisor, and the placement engine, but I think there should be a third one – the provisioning engine, which should be responsible for whatever it takes to finally create the instances, after the placement decision has been taken. It is good to take incremental approaches, hence we should try to get patches like these get accepted first within nova, and then slowly split up the logic into separate entities. Thanks, Yathi. On 1/30/14, 7:14 AM, Gil Rapaport g...@il.ibm.commailto:g...@il.ibm.com wrote: Hi all, Excellent definition of the issue at hand. The recent blueprints of policy-based-scheduler and solver-scheduler indeed highlight a possible weakness in the current design, as despite their completely independent contributions (i.e. which filters to apply per request vs. how to compute a valid placement) their implementation as drivers makes combining them non-trivial. As Alex Glikson hinted a couple of weekly meetings ago, our approach to this is to think of the driver's work as split between two entities: -- A Placement Advisor, that constructs placement problems for scheduling requests (filter-scheduler and policy-based-scheduler) -- A Placement Engine, that solves placement problems (HostManager in get_filtered_hosts() and solver-scheduler with its LP engine). Such modularity should allow developing independent mechanisms that can be combined seamlessly through a unified well-defined protocol based on constructing placement problem objects by the placement advisor and then passing them to the placement engine, which returns the solution. The protocol can be orchestrated by the scheduler manager. As can be seen at this point already, the policy-based-scheduler blueprint can now be positioned as an improvement of the placement advisor. Similarly, the solver-scheduler blueprint can be positioned as an improvement of the placement engine. I'm working on a wiki page that will get into the details. Would appreciate your initial thoughts on this approach. Regards, Gil From: Khanh-Toan Tran khanh-toan.t...@cloudwatt.commailto:khanh-toan.t...@cloudwatt.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org, Date: 01/30/2014 01:43 PM Subject: Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler Hi Sylvain, 1) Some Filters such as AggregateCoreFilter, AggregateRAMFilter can change its parameters
Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler
On 02/03/2014 12:28 PM, Khanh-Toan Tran wrote: Another though would be the need for Instance Group API [1]. Currently users can only request multiple instances of the same flavors. These requests do not need LP to solve, just placing instances one by one is sufficient. Therefore we need this API so that users can request instances of different flavors, with some relations (constraints) among them. The advantage is that this logic and API will help us add Cinder volumes with ease (not sure how the Cinder-stackers think about it, though). I don't think that the instance group API actually helps here. (I think it's a good idea, just not directly related to this.) I think what we really want is the ability to specify an arbitrary list of instances (or other things) that you want to schedule, each of which may have different image/flavor, each of which may be part of an instance group, a specific network, have metadata which associates with a host aggregate, desire specific PCI passthrough devices, etc. An immediate user of something like this would be heat, since it would let them pass the whole stack to the scheduler in one API call. The scheduler could then take a more holistic view, possibly doing a better fitting job than if the instances are scheduled one-at-a-time. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler
The solver-scheduler is designed to solve for an arbitrary list of instances of different flavors. We need to have some updated apis in the scheduler to be able to pass on such requests. Instance group api is an initial effort to specify such groups. Even now the existing solver scheduler patch, works for a group request, only that it is a group of a single flavor. It still solves once for the entire group based on the constraints on available capacity. With updates to the api that call the solver scheduler we can easily demonstrate how an arbitrary group of VM request can be satisfied and solved together in a single constraint solver run. (LP based solver for now in the current patch, But can be any constraint solver) Thanks, Yathi. -- Original message-- From: Chris Friesen Date: Mon, 2/3/2014 11:24 AM To: openstack-dev@lists.openstack.org; Subject:Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler On 02/03/2014 12:28 PM, Khanh-Toan Tran wrote: Another though would be the need for Instance Group API [1]. Currently users can only request multiple instances of the same flavors. These requests do not need LP to solve, just placing instances one by one is sufficient. Therefore we need this API so that users can request instances of different flavors, with some relations (constraints) among them. The advantage is that this logic and API will help us add Cinder volumes with ease (not sure how the Cinder-stackers think about it, though). I don't think that the instance group API actually helps here. (I think it's a good idea, just not directly related to this.) I think what we really want is the ability to specify an arbitrary list of instances (or other things) that you want to schedule, each of which may have different image/flavor, each of which may be part of an instance group, a specific network, have metadata which associates with a host aggregate, desire specific PCI passthrough devices, etc. An immediate user of something like this would be heat, since it would let them pass the whole stack to the scheduler in one API call. The scheduler could then take a more holistic view, possibly doing a better fitting job than if the instances are scheduled one-at-a-time. Chris ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler
There is an unexpected line break in the middle of the link, so I post it again: https://docs.google.com/document/d/1RfP7jRsw1mXMjd7in72ARjK0fTrsQv1bqolOri IQB2Y -Message d'origine- De : Khanh-Toan Tran [mailto:khanh-toan.t...@cloudwatt.com] Envoyé : mercredi 29 janvier 2014 13:25 À : 'OpenStack Development Mailing List (not for usage questions)' Objet : [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler Dear all, As promised in the Scheduler/Gantt meeting, here is our analysis on the connection between Policy Based Scheduler and Solver Scheduler: https://docs.google.com/document/d/1RfP7jRsw1mXMjd7in72ARjK0fTrsQv1bq olOri IQB2Y This document briefs the mechanism of the two schedulers and the possibility of cooperation. It is my personal point of view only. In a nutshell, Policy Based Scheduler allows admin to define policies for different physical resources (an aggregate, an availability-zone, or all infrastructure) or different (classes of) users. Admin can modify (add/remove/modify) any policy in runtime, and the modification effect is only in the target (e.g. the aggregate, the users) that the policy is defined to. Solver Scheduler solves the placement of groups of instances simultaneously by putting all the known information into a integer linear system and uses Integer Program solver to solve the latter. Thus relation between VMs and between VMs- computes are all accounted for. If working together, Policy Based Scheduler can supply the filters and weighers following the policies rules defined for different computes. These filters and weighers can be converted into constraints cost function for Solver Scheduler to solve. More detailed will be found in the doc. I look forward for comments and hope that we can work it out. Best regards, Khanh-Toan TRAN ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler
Hi Khanh-Toan, I only have one comment on your proposal : why are you proposing something new for overcommitments with aggregates while the AggregateCoreFilter [1] and AggregateRAMFilter [2]already exist, which AIUI provide same feature ? I'm also concerned about the scope of changes for scheduler, as Gantt is currently trying to replace it. Can we imagine such big changes to be committed on the Nova side, while it's planned to have a Scheduler service in the next future ? -Sylvain [1] https://github.com/openstack/nova/blob/master/nova/scheduler/filters/core_filter.py#L74 [2] https://github.com/openstack/nova/blob/master/nova/scheduler/filters/ram_filter.py#L75 2014-01-30 Khanh-Toan Tran khanh-toan.t...@cloudwatt.com There is an unexpected line break in the middle of the link, so I post it again: https://docs.google.com/document/d/1RfP7jRsw1mXMjd7in72ARjK0fTrsQv1bqolOri IQB2Yhttps://docs.google.com/document/d/1RfP7jRsw1mXMjd7in72ARjK0fTrsQv1bqolOriIQB2Y -Message d'origine- De : Khanh-Toan Tran [mailto:khanh-toan.t...@cloudwatt.com] Envoyé : mercredi 29 janvier 2014 13:25 À : 'OpenStack Development Mailing List (not for usage questions)' Objet : [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler Dear all, As promised in the Scheduler/Gantt meeting, here is our analysis on the connection between Policy Based Scheduler and Solver Scheduler: https://docs.google.com/document/d/1RfP7jRsw1mXMjd7in72ARjK0fTrsQv1bq olOri IQB2Y This document briefs the mechanism of the two schedulers and the possibility of cooperation. It is my personal point of view only. In a nutshell, Policy Based Scheduler allows admin to define policies for different physical resources (an aggregate, an availability-zone, or all infrastructure) or different (classes of) users. Admin can modify (add/remove/modify) any policy in runtime, and the modification effect is only in the target (e.g. the aggregate, the users) that the policy is defined to. Solver Scheduler solves the placement of groups of instances simultaneously by putting all the known information into a integer linear system and uses Integer Program solver to solve the latter. Thus relation between VMs and between VMs- computes are all accounted for. If working together, Policy Based Scheduler can supply the filters and weighers following the policies rules defined for different computes. These filters and weighers can be converted into constraints cost function for Solver Scheduler to solve. More detailed will be found in the doc. I look forward for comments and hope that we can work it out. Best regards, Khanh-Toan TRAN ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler
Hi Sylvain, 1) Some Filters such as AggregateCoreFilter, AggregateRAMFilter can change its parameters for aggregates. But what if admin wants to change for all hosts in an availability-zone? Does he have to rewrite all the parameters in all aggregates? Or should we create a new AvailabilityZoneCoreFilter? The Policy Based Scheduler (PBS) blueprint separates the effect (filter according to Core) from its target (all hosts in an aggregate, or in an availability-zone). It will benefit all filters, not just CoreFilter or RAMFilter, so that we can avoid creating for each filter XFilter the AggregateXFilter and AvailabilityZoneWFilter from now on. Beside, if admin wants to apply the a filter to some aggregates (or availability-zone) and not the other (dont call filters at all, not just modify parameters), he can do it. It help us avoid running all filters on all hosts. 2) In fact, we also prepare for a separated scheduler in which PBS is a very first step of it, thats why we purposely separate the Policy Based Scheduler from Policy Based Scheduling Module (PBSM) [1] which is the core of our architecture. If you look at our code, you will see that Policy_Based_Scheduler.py is only slightly different from Filter Scheduler. That is because we just want a link from Nova-scheduler to PBSM. Were trying to push some more management into scheduler without causing too much modification, as you can see in the patch . Thus Im very happy when Gantt is proposed. As I see it, Gantt is based on Nova-scheduler code, with the planning on replacing nova-scheduler in J. The separation from Nova will be complicated, but not on scheduling part. Thus integrating PBS and PBSM into Gantt would not be a problem. Best regards, [1] https://docs.google.com/document/d/1gr4Pb1ErXymxN9QXR4G_jVjLqNOg2ij9oA0JrL wMVRA Toan De : Sylvain Bauza [mailto:sylvain.ba...@gmail.com] Envoyé : jeudi 30 janvier 2014 11:16 À : OpenStack Development Mailing List (not for usage questions) Objet : Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler Hi Khanh-Toan, I only have one comment on your proposal : why are you proposing something new for overcommitments with aggregates while the AggregateCoreFilter [1] and AggregateRAMFilter [2]already exist, which AIUI provide same feature ? I'm also concerned about the scope of changes for scheduler, as Gantt is currently trying to replace it. Can we imagine such big changes to be committed on the Nova side, while it's planned to have a Scheduler service in the next future ? -Sylvain [1] https://github.com/openstack/nova/blob/master/nova/scheduler/filters/core_ filter.py#L74 [2] https://github.com/openstack/nova/blob/master/nova/scheduler/filters/ram_f ilter.py#L75 2014-01-30 Khanh-Toan Tran khanh-toan.t...@cloudwatt.com There is an unexpected line break in the middle of the link, so I post it again: https://docs.google.com/document/d/1RfP7jRsw1mXMjd7in72ARjK0fTrsQv1bqolOri https://docs.google.com/document/d/1RfP7jRsw1mXMjd7in72ARjK0fTrsQv1bqolOr iIQB2Y IQB2Y -Message d'origine- De : Khanh-Toan Tran [mailto:khanh-toan.t...@cloudwatt.com] Envoyé : mercredi 29 janvier 2014 13:25 À : 'OpenStack Development Mailing List (not for usage questions)' Objet : [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler Dear all, As promised in the Scheduler/Gantt meeting, here is our analysis on the connection between Policy Based Scheduler and Solver Scheduler: https://docs.google.com/document/d/1RfP7jRsw1mXMjd7in72ARjK0fTrsQv1bq olOri IQB2Y This document briefs the mechanism of the two schedulers and the possibility of cooperation. It is my personal point of view only. In a nutshell, Policy Based Scheduler allows admin to define policies for different physical resources (an aggregate, an availability-zone, or all infrastructure) or different (classes of) users. Admin can modify (add/remove/modify) any policy in runtime, and the modification effect is only in the target (e.g. the aggregate, the users) that the policy is defined to. Solver Scheduler solves the placement of groups of instances simultaneously by putting all the known information into a integer linear system and uses Integer Program solver to solve the latter. Thus relation between VMs and between VMs- computes are all accounted for. If working together, Policy Based Scheduler can supply the filters and weighers following the policies rules defined for different computes. These filters and weighers can be converted into constraints cost function for Solver Scheduler to solve. More detailed will be found in the doc. I look forward for comments and hope that we can work it out. Best regards, Khanh-Toan TRAN ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler
Hi all, Excellent definition of the issue at hand. The recent blueprints of policy-based-scheduler and solver-scheduler indeed highlight a possible weakness in the current design, as despite their completely independent contributions (i.e. which filters to apply per request vs. how to compute a valid placement) their implementation as drivers makes combining them non-trivial. As Alex Glikson hinted a couple of weekly meetings ago, our approach to this is to think of the driver's work as split between two entities: -- A Placement Advisor, that constructs placement problems for scheduling requests (filter-scheduler and policy-based-scheduler) -- A Placement Engine, that solves placement problems (HostManager in get_filtered_hosts() and solver-scheduler with its LP engine). Such modularity should allow developing independent mechanisms that can be combined seamlessly through a unified well-defined protocol based on constructing placement problem objects by the placement advisor and then passing them to the placement engine, which returns the solution. The protocol can be orchestrated by the scheduler manager. As can be seen at this point already, the policy-based-scheduler blueprint can now be positioned as an improvement of the placement advisor. Similarly, the solver-scheduler blueprint can be positioned as an improvement of the placement engine. I'm working on a wiki page that will get into the details. Would appreciate your initial thoughts on this approach. Regards, Gil From: Khanh-Toan Tran khanh-toan.t...@cloudwatt.com To: OpenStack Development Mailing List \(not for usage questions\) openstack-dev@lists.openstack.org, Date: 01/30/2014 01:43 PM Subject:Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler Hi Sylvain, 1) Some Filters such as AggregateCoreFilter, AggregateRAMFilter can change its parameters for aggregates. But what if admin wants to change for all hosts in an availability-zone? Does he have to rewrite all the parameters in all aggregates? Or should we create a new AvailabilityZoneCoreFilter? The Policy Based Scheduler (PBS) blueprint separates the effect (filter according to Core) from its target (all hosts in an aggregate, or in an availability-zone). It will benefit all filters, not just CoreFilter or RAMFilter, so that we can avoid creating for each filter XFilter the AggregateXFilter and AvailabilityZoneWFilter from now on. Beside, if admin wants to apply the a filter to some aggregates (or availability-zone) and not the other (don’t call filters at all, not just modify parameters), he can do it. It help us avoid running all filters on all hosts. 2) In fact, we also prepare for a separated scheduler in which PBS is a very first step of it, that’s why we purposely separate the Policy Based Scheduler from Policy Based Scheduling Module (PBSM) [1] which is the core of our architecture. If you look at our code, you will see that Policy_Based_Scheduler.py is only slightly different from Filter Scheduler. That is because we just want a link from Nova-scheduler to PBSM. We’re trying to push some more management into scheduler without causing too much modification, as you can see in the patch . Thus I’m very happy when Gantt is proposed. As I see it, Gantt is based on Nova-scheduler code, with the planning on replacing nova-scheduler in J. The separation from Nova will be complicated, but not on scheduling part. Thus integrating PBS and PBSM into Gantt would not be a problem. Best regards, [1] https://docs.google.com/document/d/1gr4Pb1ErXymxN9QXR4G_jVjLqNOg2ij9oA0JrLwMVRA Toan De : Sylvain Bauza [mailto:sylvain.ba...@gmail.com] Envoyé : jeudi 30 janvier 2014 11:16 À : OpenStack Development Mailing List (not for usage questions) Objet : Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler Hi Khanh-Toan, I only have one comment on your proposal : why are you proposing something new for overcommitments with aggregates while the AggregateCoreFilter [1] and AggregateRAMFilter [2]already exist, which AIUI provide same feature ? I'm also concerned about the scope of changes for scheduler, as Gantt is currently trying to replace it. Can we imagine such big changes to be committed on the Nova side, while it's planned to have a Scheduler service in the next future ? -Sylvain [1] https://github.com/openstack/nova/blob/master/nova/scheduler/filters/core_filter.py#L74 [2] https://github.com/openstack/nova/blob/master/nova/scheduler/filters/ram_filter.py#L75 2014-01-30 Khanh-Toan Tran khanh-toan.t...@cloudwatt.com There is an unexpected line break in the middle of the link, so I post it again: https://docs.google.com/document/d/1RfP7jRsw1mXMjd7in72ARjK0fTrsQv1bqolOri IQB2Y -Message d'origine- De : Khanh-Toan Tran [mailto:khanh-toan.t...@cloudwatt.com] Envoyé : mercredi 29 janvier 2014 13:25 À : 'OpenStack Development
Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler and Solver Scheduler
It is really good we are reviving the conversation we started during the last summit in Hongkong during one of the scheduler sessions called “Smart resource placement”. This is the document we used to discuss during the session. Probably you may have seen this before: https://docs.google.com/document/d/1IiPI0sfaWb1bdYiMWzAAx0HYR6UqzOan_Utgml5W1HI/edit The idea is to separate out the logic for the placement decision engine from the actual request and the final provisioning phase. The placement engine itself can be pluggable, and as we show in the solver scheduler blueprint, we show how it fits inside of Nova. The discussions at the summit and in our weekly scheduler meetings led to us starting the “Smart resource placement” idea inside of Nova, and then take it to a unified global level spanning cross services such as cinder and neutron. Like you point out, I do agree the two entities of placement advisor, and the placement engine, but I think there should be a third one – the provisioning engine, which should be responsible for whatever it takes to finally create the instances, after the placement decision has been taken. It is good to take incremental approaches, hence we should try to get patches like these get accepted first within nova, and then slowly split up the logic into separate entities. Thanks, Yathi. On 1/30/14, 7:14 AM, Gil Rapaport g...@il.ibm.commailto:g...@il.ibm.com wrote: Hi all, Excellent definition of the issue at hand. The recent blueprints of policy-based-scheduler and solver-scheduler indeed highlight a possible weakness in the current design, as despite their completely independent contributions (i.e. which filters to apply per request vs. how to compute a valid placement) their implementation as drivers makes combining them non-trivial. As Alex Glikson hinted a couple of weekly meetings ago, our approach to this is to think of the driver's work as split between two entities: -- A Placement Advisor, that constructs placement problems for scheduling requests (filter-scheduler and policy-based-scheduler) -- A Placement Engine, that solves placement problems (HostManager in get_filtered_hosts() and solver-scheduler with its LP engine). Such modularity should allow developing independent mechanisms that can be combined seamlessly through a unified well-defined protocol based on constructing placement problem objects by the placement advisor and then passing them to the placement engine, which returns the solution. The protocol can be orchestrated by the scheduler manager. As can be seen at this point already, the policy-based-scheduler blueprint can now be positioned as an improvement of the placement advisor. Similarly, the solver-scheduler blueprint can be positioned as an improvement of the placement engine. I'm working on a wiki page that will get into the details. Would appreciate your initial thoughts on this approach. Regards, Gil From:Khanh-Toan Tran khanh-toan.t...@cloudwatt.commailto:khanh-toan.t...@cloudwatt.com To:OpenStack Development Mailing List \(not for usage questions\) openstack-dev@lists.openstack.orgmailto:openstack-dev@lists.openstack.org, Date:01/30/2014 01:43 PM Subject:Re: [openstack-dev] [Nova][Scheduler] Policy Based Scheduler andSolver Scheduler Hi Sylvain, 1) Some Filters such as AggregateCoreFilter, AggregateRAMFilter can change its parameters for aggregates. But what if admin wants to change for all hosts in an availability-zone? Does he have to rewrite all the parameters in all aggregates? Or should we create a new AvailabilityZoneCoreFilter? The Policy Based Scheduler (PBS) blueprint separates the effect (filter according to Core) from its target (all hosts in an aggregate, or in an availability-zone). It will benefit all filters, not just CoreFilter or RAMFilter, so that we can avoid creating for each filter XFilter the AggregateXFilter and AvailabilityZoneWFilter from now on. Beside, if admin wants to apply the a filter to some aggregates (or availability-zone) and not the other (don’t call filters at all, not just modify parameters), he can do it. It help us avoid running all filters on all hosts. 2) In fact, we also prepare for a separated scheduler in which PBS is a very first step of it, that’s why we purposely separate the Policy Based Scheduler from Policy Based Scheduling Module (PBSM) [1] which is the core of our architecture. If you look at our code, you will see that Policy_Based_Scheduler.py is only slightly different from Filter Scheduler. That is because we just want a link from Nova-scheduler to PBSM. We’re trying to push some more management into scheduler without causing too much modification, as you can see in the patch . Thus I’m very happy when Gantt is proposed. As I see it, Gantt is based on Nova-scheduler code, with the planning on replacing nova-scheduler in J
