Re: [openstack-dev] [nova][scheduler] Proposal: FairShareScheduler.
On 06/30/2014 04:35 PM, Jesse Pretorius wrote: On 30 June 2014 16:05, Eric Frizziero eric.frizzi...@pd.infn.it mailto:eric.frizzi...@pd.infn.it wrote: In more detail, some features of the FairshareScheduler are: a) It assigns dynamically the proper priority to every new user requests; b) The priority of the queued requests will be recalculated periodically using the fairshare algorithm. This feature guarantees the usage of the cloud resources is distributed among users and groups by considering the portion of the cloud resources allocated to them (i.e. share) and the resources already consumed; c) all user requests will be inserted in a (persistent) priority queue and then processed asynchronously by the dedicated process (filtering + weighting phase) when compute resources are available; d) From the client point of view the queued requests remain in Scheduling state till the compute resources are available. No new states added: this prevents any possible interaction issue with the Openstack clients; e) User requests are dequeued by a pool of WorkerThreads (configurable), i.e. no sequential processing of the requests; f) The failed requests at filtering + weighting phase may be inserted again in the queue for n-times (configurable). We have integrated the FairShareScheduler in our Openstack installation (release HAVANA). We're now working to adapt the FairShareScheduler to the new release IceHouse. Does anyone have experiences in those issues found in our cloud scenario? Could the FairShareScheduler be useful for the Openstack community? In that case, we'll be happy to share our work. Sounds like an interesting option to have available. Thank you very much for your feedback! Cheers, Eric. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][scheduler] Proposal: FairShareScheduler.
Hi Belmiro, On 06/30/2014 11:42 PM, Belmiro Moreira wrote: Hi Eric, definitely... In my view a FairShareScheduler could be a very interesting option for private clouds that support scientific communities. Basically this is the model used by batch systems in order to fully use the available resources. Yes, it is. I'm very curious about the work that you are doing. You can find more info at the following link: https://agenda.infn.it/getFile.py/access?contribId=17sessionId=3resId=0materialId=slidesconfId=7915 Is it available in github? At slide 18, you can find the pointer to the FairShareScheduler. Cheers, Eric. Belmiro -- Belmiro Moreira CERN Email: belmiro.more...@cern.ch mailto:belmiro.more...@cern.ch IRC: belmoreira On Mon, Jun 30, 2014 at 4:05 PM, Eric Frizziero eric.frizzi...@pd.infn.it mailto:eric.frizzi...@pd.infn.it wrote: Hi All, we have analyzed the nova-scheduler component (FilterScheduler) in our Openstack installation used by some scientific teams. In our scenario, the cloud resources need to be distributed among the teams by considering the predefined share (e.g. quota) assigned to each team, the portion of the resources currently used and the resources they have already consumed. We have observed that: 1) User requests are sequentially processed (FIFO scheduling), i.e. FilterScheduler doesn't provide any dynamic priority algorithm; 2) User requests that cannot be satisfied (e.g. if resources are not available) fail and will be lost, i.e. on that scenario nova-scheduler doesn't provide any queuing of the requests; 3) OpenStack simply provides a static partitioning of resources among various projects / teams (use of quotas). If project/team 1 in a period is systematically underutilizing its quota and the project/team 2 instead is systematically saturating its quota, the only solution to give more resource to project/team 2 is a manual change (to be done by the admin) to the related quotas. The need to find a better approach to enable a more effective scheduling in Openstack becomes more and more evident when the number of the user requests to be handled increases significantly. This is a well known problem which has already been solved in the past for the Batch Systems. In order to solve those issues in our usage scenario of Openstack, we have developed a prototype of a pluggable scheduler, named FairShareScheduler, with the objective to extend the existing OpenStack scheduler (FilterScheduler) by integrating a (batch like) dynamic priority algorithm. The architecture of the FairShareScheduler is explicitly designed to provide a high scalability level. To all user requests will be assigned a priority value calculated by considering the share allocated to the user by the administrator and the evaluation of the effective resource usage consumed in the recent past. All requests will be inserted in a priority queue, and processed in parallel by a configurable pool of workers without interfering with the priority order. Moreover all significant information (e.g. priority queue) will be stored in a persistence layer in order to provide a fault tolerance mechanism while a proper logging system will annotate all relevant events, useful for auditing processing. In more detail, some features of the FairshareScheduler are: a) It assigns dynamically the proper priority to every new user requests; b) The priority of the queued requests will be recalculated periodically using the fairshare algorithm. This feature guarantees the usage of the cloud resources is distributed among users and groups by considering the portion of the cloud resources allocated to them (i.e. share) and the resources already consumed; c) all user requests will be inserted in a (persistent) priority queue and then processed asynchronously by the dedicated process (filtering + weighting phase) when compute resources are available; d) From the client point of view the queued requests remain in Scheduling state till the compute resources are available. No new states added: this prevents any possible interaction issue with the Openstack clients; e) User requests are dequeued by a pool of WorkerThreads (configurable), i.e. no sequential processing of the requests; f) The failed requests at filtering + weighting phase may be inserted again in the queue for n-times (configurable). We have integrated the FairShareScheduler in our Openstack installation (release HAVANA). We're now working to adapt the FairShareScheduler to the new release IceHouse. Does anyone have experiences in those issues found in our cloud scenario? Could the FairShareScheduler be useful for the Openstack community? In that case, we'll
Re: [openstack-dev] [nova][scheduler] Proposal: FairShareScheduler.
Hi Don, I and my colleague Lisa attend both IRC meetings (today and next week). In the meantime you can find more info about the FairShareScheduler at the following link: https://agenda.infn.it/getFile.py/access?contribId=17sessionId=3resId=0materialId=slidesconfId=7915 Thank you very much! Cheers, Eric. On 07/01/2014 01:48 AM, Dugger, Donald D wrote: Eric- We have a weekly scheduler sub-group (code name gantt) IRC meeting at 1500 UTC on Tuesdays. This would be an excellent topic to bring up at one of those meetings as a lot of people with interest in the scheduler will be there. It's a little short notice for tomorrow but do you think you could attend next week, 7/8, to talk about this? -- Don Dugger Censeo Toto nos in Kansa esse decisse. - D. Gale Ph: 303/443-3786 -Original Message- From: Eric Frizziero [mailto:eric.frizzi...@pd.infn.it] Sent: Monday, June 30, 2014 8:05 AM To: OpenStack Development Mailing List (not for usage questions) Subject: [openstack-dev] [nova][scheduler] Proposal: FairShareScheduler. Hi All, we have analyzed the nova-scheduler component (FilterScheduler) in our Openstack installation used by some scientific teams. In our scenario, the cloud resources need to be distributed among the teams by considering the predefined share (e.g. quota) assigned to each team, the portion of the resources currently used and the resources they have already consumed. We have observed that: 1) User requests are sequentially processed (FIFO scheduling), i.e. FilterScheduler doesn't provide any dynamic priority algorithm; 2) User requests that cannot be satisfied (e.g. if resources are not available) fail and will be lost, i.e. on that scenario nova-scheduler doesn't provide any queuing of the requests; 3) OpenStack simply provides a static partitioning of resources among various projects / teams (use of quotas). If project/team 1 in a period is systematically underutilizing its quota and the project/team 2 instead is systematically saturating its quota, the only solution to give more resource to project/team 2 is a manual change (to be done by the admin) to the related quotas. The need to find a better approach to enable a more effective scheduling in Openstack becomes more and more evident when the number of the user requests to be handled increases significantly. This is a well known problem which has already been solved in the past for the Batch Systems. In order to solve those issues in our usage scenario of Openstack, we have developed a prototype of a pluggable scheduler, named FairShareScheduler, with the objective to extend the existing OpenStack scheduler (FilterScheduler) by integrating a (batch like) dynamic priority algorithm. The architecture of the FairShareScheduler is explicitly designed to provide a high scalability level. To all user requests will be assigned a priority value calculated by considering the share allocated to the user by the administrator and the evaluation of the effective resource usage consumed in the recent past. All requests will be inserted in a priority queue, and processed in parallel by a configurable pool of workers without interfering with the priority order. Moreover all significant information (e.g. priority queue) will be stored in a persistence layer in order to provide a fault tolerance mechanism while a proper logging system will annotate all relevant events, useful for auditing processing. In more detail, some features of the FairshareScheduler are: a) It assigns dynamically the proper priority to every new user requests; b) The priority of the queued requests will be recalculated periodically using the fairshare algorithm. This feature guarantees the usage of the cloud resources is distributed among users and groups by considering the portion of the cloud resources allocated to them (i.e. share) and the resources already consumed; c) all user requests will be inserted in a (persistent) priority queue and then processed asynchronously by the dedicated process (filtering + weighting phase) when compute resources are available; d) From the client point of view the queued requests remain in Scheduling state till the compute resources are available. No new states added: this prevents any possible interaction issue with the Openstack clients; e) User requests are dequeued by a pool of WorkerThreads (configurable), i.e. no sequential processing of the requests; f) The failed requests at filtering + weighting phase may be inserted again in the queue for n-times (configurable). We have integrated the FairShareScheduler in our Openstack installation (release HAVANA). We're now working to adapt the FairShareScheduler to the new release IceHouse. Does anyone have experiences in those issues found in our cloud scenario? Could the FairShareScheduler be useful for the Openstack community? In that case, we'll be happy to share our work. Any feedback/comment is welcome! Cheers, Eric
[openstack-dev] [nova][scheduler] Proposal: FairShareScheduler.
Hi All, we have analyzed the nova-scheduler component (FilterScheduler) in our Openstack installation used by some scientific teams. In our scenario, the cloud resources need to be distributed among the teams by considering the predefined share (e.g. quota) assigned to each team, the portion of the resources currently used and the resources they have already consumed. We have observed that: 1) User requests are sequentially processed (FIFO scheduling), i.e. FilterScheduler doesn't provide any dynamic priority algorithm; 2) User requests that cannot be satisfied (e.g. if resources are not available) fail and will be lost, i.e. on that scenario nova-scheduler doesn't provide any queuing of the requests; 3) OpenStack simply provides a static partitioning of resources among various projects / teams (use of quotas). If project/team 1 in a period is systematically underutilizing its quota and the project/team 2 instead is systematically saturating its quota, the only solution to give more resource to project/team 2 is a manual change (to be done by the admin) to the related quotas. The need to find a better approach to enable a more effective scheduling in Openstack becomes more and more evident when the number of the user requests to be handled increases significantly. This is a well known problem which has already been solved in the past for the Batch Systems. In order to solve those issues in our usage scenario of Openstack, we have developed a prototype of a pluggable scheduler, named FairShareScheduler, with the objective to extend the existing OpenStack scheduler (FilterScheduler) by integrating a (batch like) dynamic priority algorithm. The architecture of the FairShareScheduler is explicitly designed to provide a high scalability level. To all user requests will be assigned a priority value calculated by considering the share allocated to the user by the administrator and the evaluation of the effective resource usage consumed in the recent past. All requests will be inserted in a priority queue, and processed in parallel by a configurable pool of workers without interfering with the priority order. Moreover all significant information (e.g. priority queue) will be stored in a persistence layer in order to provide a fault tolerance mechanism while a proper logging system will annotate all relevant events, useful for auditing processing. In more detail, some features of the FairshareScheduler are: a) It assigns dynamically the proper priority to every new user requests; b) The priority of the queued requests will be recalculated periodically using the fairshare algorithm. This feature guarantees the usage of the cloud resources is distributed among users and groups by considering the portion of the cloud resources allocated to them (i.e. share) and the resources already consumed; c) all user requests will be inserted in a (persistent) priority queue and then processed asynchronously by the dedicated process (filtering + weighting phase) when compute resources are available; d) From the client point of view the queued requests remain in “Scheduling” state till the compute resources are available. No new states added: this prevents any possible interaction issue with the Openstack clients; e) User requests are dequeued by a pool of WorkerThreads (configurable), i.e. no sequential processing of the requests; f) The failed requests at filtering + weighting phase may be inserted again in the queue for n-times (configurable). We have integrated the FairShareScheduler in our Openstack installation (release HAVANA). We're now working to adapt the FairShareScheduler to the new release IceHouse. Does anyone have experiences in those issues found in our cloud scenario? Could the FairShareScheduler be useful for the Openstack community? In that case, we'll be happy to share our work. Any feedback/comment is welcome! Cheers, Eric. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
