Re: [openstack-dev] Scheduler proposal

On Fri, Oct 16 2015, Joshua Harlow wrote: > Another idea is to use numpy and start representing filters as linear > equations, then use something like > https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.solve.html#numpy.linalg.solve > to solve linear equations given some data. > >

Re: [openstack-dev] Scheduler proposal

Clint Byrum wrote: Excerpts from Ed Leafe's message of 2015-10-15 11:56:24 -0700: Wow, I seem to have unleashed a bunch of pent-up frustration in the community! It's great to see everyone coming forward with their ideas and insights for improving the way Nova (and, by extension, all of OpenStack

Re: [openstack-dev] Scheduler proposal

On 10/15/15, 11:11 PM, "Clint Byrum" wrote: >Excerpts from Ed Leafe's message of 2015-10-15 11:56:24 -0700: >> Wow, I seem to have unleashed a bunch of pent-up frustration in the >> community! It's great to see everyone coming forward with their ideas and >> insights for improving the way

Re: [openstack-dev] Scheduler proposal

Excerpts from Ed Leafe's message of 2015-10-15 11:56:24 -0700: > Wow, I seem to have unleashed a bunch of pent-up frustration in the > community! It's great to see everyone coming forward with their ideas and > insights for improving the way Nova (and, by extension, all of OpenStack) can > poten

Re: [openstack-dev] Scheduler proposal

Ed Leafe wrote: Wow, I seem to have unleashed a bunch of pent-up frustration in the community! It's great to see everyone coming forward with their ideas and insights for improving the way Nova (and, by extension, all of OpenStack) can potentially scale. I do have a few comments on the discussio

Re: [openstack-dev] Scheduler proposal

Wow, I seem to have unleashed a bunch of pent-up frustration in the community! It's great to see everyone coming forward with their ideas and insights for improving the way Nova (and, by extension, all of OpenStack) can potentially scale. I do have a few comments on the discussion: 1) This isn

Re: [openstack-dev] Scheduler proposal

On 10/12/2015 07:10 PM, Monty Taylor wrote: > On 10/12/2015 12:43 PM, Clint Byrum wrote: >> Excerpts from Thomas Goirand's message of 2015-10-12 05:57:26 -0700: >>> On 10/11/2015 02:53 AM, Davanum Srinivas wrote: Thomas, i am curious as well. AFAIK, cassandra works well with OpenJDK.

Re: [openstack-dev] Scheduler proposal

On Tue, 2015-10-13 at 08:47 -0700, Joshua Harlow wrote: > Well great! > > When is that going to be accessible :-P > > Dulko, Michal wrote: > > On Mon, 2015-10-12 at 10:58 -0700, Joshua Harlow wrote: > >> Just a related thought/question. It really seems we (as a community) > >> need some kind of s

Re: [openstack-dev] Scheduler proposal

On 2015-10-13 10:17:26 -0700 (-0700), Joshua Harlow wrote: [...] > Interesting, doesn't the foundation have money? I was under the > assumption it does (but I'm not a finance person); seeing that the > membership fee to become a member afaik is not cheap, and there > seems to be quite a-lot of memb

Re: [openstack-dev] Scheduler proposal

Clint Byrum wrote: Excerpts from Ian Wells's message of 2015-10-13 09:24:42 -0700: On 12 October 2015 at 21:18, Clint Byrum wrote: We _would_ keep a local cache of the information in the schedulers. The centralized copy of it is to free the schedulers from the complexity of having to keep tra

Re: [openstack-dev] Scheduler proposal

Clint Byrum wrote: Excerpts from Jeremy Stanley's message of 2015-10-13 06:13:32 -0700: On 2015-10-12 20:49:44 -0700 (-0700), Joshua Harlow wrote: Does the openstack foundation have access to a scaling area that can be used by the community for this kind of experimental work? The OpenStack Fou

Re: [openstack-dev] Scheduler proposal

Excerpts from Ian Wells's message of 2015-10-13 09:24:42 -0700: > On 12 October 2015 at 21:18, Clint Byrum wrote: > > > We _would_ keep a local cache of the information in the schedulers. The > > centralized copy of it is to free the schedulers from the complexity of > > having to keep track of i

Re: [openstack-dev] Scheduler proposal

On 12 October 2015 at 21:18, Clint Byrum wrote: > We _would_ keep a local cache of the information in the schedulers. The > centralized copy of it is to free the schedulers from the complexity of > having to keep track of it as state, rather than as a cache. We also don't > have to provide a way

Re: [openstack-dev] Scheduler proposal

On 2015-10-13 09:15:02 -0700 (-0700), Clint Byrum wrote: > Excerpts from Jeremy Stanley's message of 2015-10-13 06:13:32 -0700: [...] > > it's not even within an order of magnitude of being 1k host > > scale (and at that, it's still a multi-cycle plan just to reach > > viability). > > Infra-cloud

Re: [openstack-dev] Scheduler proposal

Excerpts from Dulko, Michal's message of 2015-10-13 03:49:44 -0700: > On Mon, 2015-10-12 at 10:13 -0700, Clint Byrum wrote: > > Zookeeper sits in a very different space from Cassandra. I have had good > > success with it on OpenJDK as well. > > > > That said, we need to maybe go through some featu

Re: [openstack-dev] Scheduler proposal

Excerpts from Jeremy Stanley's message of 2015-10-13 06:13:32 -0700: > On 2015-10-12 20:49:44 -0700 (-0700), Joshua Harlow wrote: > > Does the openstack foundation have access to a scaling area that > > can be used by the community for this kind of experimental work? > > The OpenStack Foundation h

Re: [openstack-dev] Scheduler proposal

Well great! When is that going to be accessible :-P Dulko, Michal wrote: On Mon, 2015-10-12 at 10:58 -0700, Joshua Harlow wrote: Just a related thought/question. It really seems we (as a community) need some kind of scale testing ground. Internally at yahoo we were/are going to use a 200 hyper

Re: [openstack-dev] Scheduler proposal

Jeremy Stanley wrote: On 2015-10-12 20:49:44 -0700 (-0700), Joshua Harlow wrote: Does the openstack foundation have access to a scaling area that can be used by the community for this kind of experimental work? The OpenStack Foundation has a staff of fewer than 20 full-time employees, with a p

Re: [openstack-dev] Scheduler proposal

On 10/12/15, 12:05 PM, "Monty Taylor" wrote: >On 10/12/2015 02:45 PM, Joshua Harlow wrote: >> Alec Hothan (ahothan) wrote: >>> >>> >>> >>> > >I want to do 100k hypervisors. No, that's not hyperbole. > >Also, I do not think that ZK/consul/etcd are very costly for small >deployments. Given th

Re: [openstack-dev] Scheduler proposal

On 2015-10-12 20:49:44 -0700 (-0700), Joshua Harlow wrote: > Does the openstack foundation have access to a scaling area that > can be used by the community for this kind of experimental work? The OpenStack Foundation has a staff of fewer than 20 full-time employees, with a primary focus on event

Re: [openstack-dev] Scheduler proposal

On Mon, 2015-10-12 at 10:13 -0700, Clint Byrum wrote: > Zookeeper sits in a very different space from Cassandra. I have had good > success with it on OpenJDK as well. > > That said, we need to maybe go through some feature/risk matrices and > compare to etcd and Consul (this might be good to do as

Re: [openstack-dev] Scheduler proposal

On Mon, 2015-10-12 at 10:58 -0700, Joshua Harlow wrote: > Just a related thought/question. It really seems we (as a community) > need some kind of scale testing ground. Internally at yahoo we were/are > going to use a 200 hypervisor cluster for some of this and then expand > that into 200 * X by

Re: [openstack-dev] Scheduler proposal

Excerpts from Ian Wells's message of 2015-10-12 19:43:48 -0700: > On 11 October 2015 at 00:23, Clint Byrum wrote: > > > I'm in, except I think this gets simpler with an intermediary service > > like ZK/Consul to keep track of this 1GB of data and replace the need > > for 6, and changes the implem

Re: [openstack-dev] Scheduler proposal

Ian Wells wrote: On 10 October 2015 at 23:47, Clint Byrum mailto:cl...@fewbar.com>> wrote: > Per before, my suggestion was that every scheduler tries to maintain a copy > of the cloud's state in memory (in much the same way, per the previous > example, as every router on the i

Re: [openstack-dev] Scheduler proposal

On 11 October 2015 at 00:23, Clint Byrum wrote: > I'm in, except I think this gets simpler with an intermediary service > like ZK/Consul to keep track of this 1GB of data and replace the need > for 6, and changes the implementation of 5 to "updates its record and > signals its presence". > OK, s

Re: [openstack-dev] Scheduler proposal

On 10 October 2015 at 23:47, Clint Byrum wrote: > > Per before, my suggestion was that every scheduler tries to maintain a > copy > > of the cloud's state in memory (in much the same way, per the previous > > example, as every router on the internet tries to make a route table out > of > > what i

Re: [openstack-dev] Scheduler proposal

Alec Hothan (ahothan) wrote: On 10/12/15, 11:45 AM, "Joshua Harlow" wrote: Alec Hothan (ahothan) wrote: On 10/10/15, 11:35 PM, "Clint Byrum" wrote: Excerpts from Alec Hothan (ahothan)'s message of 2015-10-09 21:19:14 -0700: On 10/9/15, 6:29 PM, "Clint Byrum" wrote: Excerpts fr

Re: [openstack-dev] Scheduler proposal

On 10/12/15, 11:45 AM, "Joshua Harlow" wrote: >Alec Hothan (ahothan) wrote: >> >> >> >> >> On 10/10/15, 11:35 PM, "Clint Byrum" wrote: >> >>> Excerpts from Alec Hothan (ahothan)'s message of 2015-10-09 21:19:14 -0700: On 10/9/15, 6:29 PM, "Clint Byrum" wrote: > Excerpts from

Re: [openstack-dev] Scheduler proposal

On 10/12/2015 02:45 PM, Joshua Harlow wrote: Alec Hothan (ahothan) wrote: On 10/10/15, 11:35 PM, "Clint Byrum" wrote: Excerpts from Alec Hothan (ahothan)'s message of 2015-10-09 21:19:14 -0700: On 10/9/15, 6:29 PM, "Clint Byrum" wrote: Excerpts from Chris Friesen's message of 2015-10-

Re: [openstack-dev] Scheduler proposal

Alec Hothan (ahothan) wrote: On 10/10/15, 11:35 PM, "Clint Byrum" wrote: Excerpts from Alec Hothan (ahothan)'s message of 2015-10-09 21:19:14 -0700: On 10/9/15, 6:29 PM, "Clint Byrum" wrote: Excerpts from Chris Friesen's message of 2015-10-09 17:33:38 -0700: On 10/09/2015 03:36 PM, Ia

Re: [openstack-dev] Scheduler proposal

On 10/10/15, 11:35 PM, "Clint Byrum" wrote: >Excerpts from Alec Hothan (ahothan)'s message of 2015-10-09 21:19:14 -0700: >> >> On 10/9/15, 6:29 PM, "Clint Byrum" wrote: >> >> >Excerpts from Chris Friesen's message of 2015-10-09 17:33:38 -0700: >> >> On 10/09/2015 03:36 PM, Ian Wells wrote

Re: [openstack-dev] Scheduler proposal

Clint Byrum wrote: Excerpts from Boris Pavlovic's message of 2015-10-11 01:14:08 -0700: Clint, There are many PROS and CONS in both of approaches. Reinventing wheel (in this case it's quite simple task) and it gives more flexibility and doesn't require usage of ZK/Consul (which will simplify i

Re: [openstack-dev] Scheduler proposal

Excerpts from Boris Pavlovic's message of 2015-10-11 01:14:08 -0700: > Clint, > > There are many PROS and CONS in both of approaches. > > Reinventing wheel (in this case it's quite simple task) and it gives more > flexibility and doesn't require > usage of ZK/Consul (which will simplify integrati

Re: [openstack-dev] Scheduler proposal

Excerpts from Joshua Harlow's message of 2015-10-12 08:35:20 -0700: > Thierry Carrez wrote: > > Clint Byrum wrote: > >> Excerpts from Joshua Harlow's message of 2015-10-10 17:43:40 -0700: > >>> I'm curious is there any more detail about #1 below anywhere online? > >>> > >>> Does cassandra use some

Re: [openstack-dev] Scheduler proposal

On 10/12/2015 12:43 PM, Clint Byrum wrote: Excerpts from Thomas Goirand's message of 2015-10-12 05:57:26 -0700: On 10/11/2015 02:53 AM, Davanum Srinivas wrote: Thomas, i am curious as well. AFAIK, cassandra works well with OpenJDK. Can you please elaborate what you concerns are for #1? Thanks

Re: [openstack-dev] Scheduler proposal

Excerpts from Thomas Goirand's message of 2015-10-12 05:57:26 -0700: > On 10/11/2015 02:53 AM, Davanum Srinivas wrote: > > Thomas, > > > > i am curious as well. AFAIK, cassandra works well with OpenJDK. Can you > > please elaborate what you concerns are for #1? > > > > Thanks, > > Dims > > s/wor

Re: [openstack-dev] Scheduler proposal

Thierry Carrez wrote: Clint Byrum wrote: Excerpts from Joshua Harlow's message of 2015-10-10 17:43:40 -0700: I'm curious is there any more detail about #1 below anywhere online? Does cassandra use some features of the JVM that the openJDK version doesn't support? Something else? This about su

Re: [openstack-dev] Scheduler proposal

On 10/11/2015 02:53 AM, Davanum Srinivas wrote: > Thomas, > > i am curious as well. AFAIK, cassandra works well with OpenJDK. Can you > please elaborate what you concerns are for #1? > > Thanks, > Dims s/works well/works/ Upstream doesn't test against OpenJDK, and they close bugs without fixing

Re: [openstack-dev] Scheduler proposal

Hi everyone, What do you think about this proposal ? http://www.slideshare.net/viggates/openstack-india-meetupscheduler It seems they found a real solution for a scaling scheduler. The good idea is to move intelligence on compute. A synchronisation is only needed for anti-afinity and stuff like t

Re: [openstack-dev] Scheduler proposal

Clint Byrum wrote: > Excerpts from Joshua Harlow's message of 2015-10-10 17:43:40 -0700: >> I'm curious is there any more detail about #1 below anywhere online? >> >> Does cassandra use some features of the JVM that the openJDK version >> doesn't support? Something else? > > This about sums it up

Re: [openstack-dev] Scheduler proposal

Adam Lawson wrote: > I have a quick question: how is Amazon doing this? When choosing a next > path forward that reliably scales, would be interesting to know how this > is already being done. Well, those who know probably would be sued if they told. Since they have a limited set of instance type

Re: [openstack-dev] Scheduler proposal

-cassandra-support-openjdk From: Davanum Srinivas [mailto:dava...@gmail.com] Sent: Saturday, October 10, 2015 8:54 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] Scheduler proposal Not implying cassandra is the right option. Just curious about the

Re: [openstack-dev] Scheduler proposal

Clint Byrum wrote: Excerpts from Boris Pavlovic's message of 2015-10-11 00:02:39 -0700: 2Everybody, Just curios why we need such complexity. Let's take a look from other side: 1) Information about all hosts (even in case of 100k hosts) will be less then 1 GB 2) Usually servers that runs sched

Re: [openstack-dev] Scheduler proposal

Thanks Clint! On Sat, Oct 10, 2015 at 11:53 PM, Clint Byrum wrote: > Excerpts from Joshua Harlow's message of 2015-10-10 17:43:40 -0700: > > I'm curious is there any more detail about #1 below anywhere online? > > > > Does cassandra use some features of the JVM that the openJDK version > > doesn

Re: [openstack-dev] Scheduler proposal

I have a quick question: how is Amazon doing this? When choosing a next path forward that reliably scales, would be interesting to know how this is already being done. On Oct 9, 2015 10:12 AM, "Zane Bitter" wrote: > On 08/10/15 21:32, Ian Wells wrote: > >> >> > 2. if many hosts suit the 5 VMs

Re: [openstack-dev] Scheduler proposal

Clint, There are many PROS and CONS in both of approaches. Reinventing wheel (in this case it's quite simple task) and it gives more flexibility and doesn't require usage of ZK/Consul (which will simplify integration of it with current system) Using ZK/Consul for POC may save a lot of time and a

Re: [openstack-dev] Scheduler proposal

On 11/10/2015 6:25 PM, "Clint Byrum" wrote: > > Excerpts from Boris Pavlovic's message of 2015-10-11 00:02:39 -0700: > > 2Everybody, > > > > Just curios why we need such complexity. > > > > > > Let's take a look from other side: > > 1) Information about all hosts (even in case of 100k hosts) will

Re: [openstack-dev] Scheduler proposal

Excerpts from Boris Pavlovic's message of 2015-10-11 00:02:39 -0700: > 2Everybody, > > Just curios why we need such complexity. > > > Let's take a look from other side: > 1) Information about all hosts (even in case of 100k hosts) will be less > then 1 GB > 2) Usually servers that runs scheduler

Re: [openstack-dev] Scheduler proposal

2Everybody, Just curios why we need such complexity. Let's take a look from other side: 1) Information about all hosts (even in case of 100k hosts) will be less then 1 GB 2) Usually servers that runs scheduler service have at least 64GB RAM and more on the board 3) math.log(10) < 12 (binary

Re: [openstack-dev] Scheduler proposal

Excerpts from Joshua Harlow's message of 2015-10-10 17:43:40 -0700: > I'm curious is there any more detail about #1 below anywhere online? > > Does cassandra use some features of the JVM that the openJDK version > doesn't support? Something else? > This about sums it up: https://github.com/apa

Re: [openstack-dev] Scheduler proposal

Excerpts from Ian Wells's message of 2015-10-09 19:14:17 -0700: > On 9 October 2015 at 18:29, Clint Byrum wrote: > > > Instead of having the scheduler do all of the compute node inspection > > and querying though, you have the nodes push their stats into something > > like Zookeeper or consul, an

Re: [openstack-dev] Scheduler proposal

Excerpts from Alec Hothan (ahothan)'s message of 2015-10-09 21:19:14 -0700: > > On 10/9/15, 6:29 PM, "Clint Byrum" wrote: > > >Excerpts from Chris Friesen's message of 2015-10-09 17:33:38 -0700: > >> On 10/09/2015 03:36 PM, Ian Wells wrote: > >> > On 9 October 2015 at 12:50, Chris Friesen >> >

Re: [openstack-dev] Scheduler proposal

Excerpts from Chris Friesen's message of 2015-10-09 23:16:43 -0700: > On 10/09/2015 07:29 PM, Clint Byrum wrote: > > > Even if you figured out how to make the in-memory scheduler crazy fast, > > There's still value in concurrency for other reasons. No matter how > > fast you make the scheduler, yo

Re: [openstack-dev] Scheduler proposal

Not implying cassandra is the right option. Just curious about the assertion. -- Dims On Sat, Oct 10, 2015 at 5:53 PM, Davanum Srinivas wrote: > Thomas, > > i am curious as well. AFAIK, cassandra works well with OpenJDK. Can you > please elaborate what you concerns are for #1? > > Thanks, > Dim

Re: [openstack-dev] Scheduler proposal

Thomas, i am curious as well. AFAIK, cassandra works well with OpenJDK. Can you please elaborate what you concerns are for #1? Thanks, Dims On Sat, Oct 10, 2015 at 5:43 PM, Joshua Harlow wrote: > I'm curious is there any more detail about #1 below anywhere online? > > Does cassandra use some f

Re: [openstack-dev] Scheduler proposal

I'm curious is there any more detail about #1 below anywhere online? Does cassandra use some features of the JVM that the openJDK version doesn't support? Something else? -Josh Thomas Goirand wrote: On 10/07/2015 07:36 PM, Ed Leafe wrote: Several months ago I proposed an experiment [0] to s

Re: [openstack-dev] Scheduler proposal

On 10/07/2015 07:36 PM, Ed Leafe wrote: > Several months ago I proposed an experiment [0] to see if switching > the data model for the Nova scheduler to use Cassandra as the backend > would be a significant improvement as opposed to the current design This is probably right. I don't know, I'm not

Re: [openstack-dev] Scheduler proposal

On 10/09/2015 07:29 PM, Clint Byrum wrote: Even if you figured out how to make the in-memory scheduler crazy fast, There's still value in concurrency for other reasons. No matter how fast you make the scheduler, you'll be slave to the response time of a single scheduling request. If you take 1ms

Re: [openstack-dev] Scheduler proposal

On 10/9/15, 6:29 PM, "Clint Byrum" wrote: >Excerpts from Chris Friesen's message of 2015-10-09 17:33:38 -0700: >> On 10/09/2015 03:36 PM, Ian Wells wrote: >> > On 9 October 2015 at 12:50, Chris Friesen > > > wrote: >> > >> > Has anybody looked at why 1

Re: [openstack-dev] Scheduler proposal

On 9 October 2015 at 18:29, Clint Byrum wrote: > Instead of having the scheduler do all of the compute node inspection > and querying though, you have the nodes push their stats into something > like Zookeeper or consul, and then have schedulers watch those stats > for changes to keep their in-me

Re: [openstack-dev] Scheduler proposal

Excerpts from Chris Friesen's message of 2015-10-09 17:33:38 -0700: > On 10/09/2015 03:36 PM, Ian Wells wrote: > > On 9 October 2015 at 12:50, Chris Friesen > > wrote: > > > > Has anybody looked at why 1 instance is too slow and what it would take > > to >

Re: [openstack-dev] Scheduler proposal

On 10/09/2015 03:36 PM, Ian Wells wrote: On 9 October 2015 at 12:50, Chris Friesen mailto:chris.frie...@windriver.com>> wrote: Has anybody looked at why 1 instance is too slow and what it would take to make 1 scheduler instance work fast enough? This does not preclude the us

Re: [openstack-dev] Scheduler proposal

And one last reply with more code: http://paste.openstack.org/show/475941/ (a creator of services that dynamically creates services, and destroys them after a set amount of time is included in here, along with the prior resource watcher). Works locally, should work for u as well. Output from

Re: [openstack-dev] Scheduler proposal

wrong or imprecise language.) Regards, Neil Original Message From: Clint Byrum Sent: Friday, 9 October 2015 19:08 To: openstack-dev Reply To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] Scheduler proposal Excerpts from Chris Friesen's

Re: [openstack-dev] Scheduler proposal

Further example stuff, Get kazoo installed (http://kazoo.readthedocs.org/) Output from my local run (with no data) $ python test.py Kazoo client has changed to state: CONNECTED Got data: '' for new resource /node/compute_nodes/h1.hypervisor.yahoo.com Idling (ran for 0.00s). Known resources:

Re: [openstack-dev] Scheduler proposal

Gregory Haynes wrote: Excerpts from Joshua Harlow's message of 2015-10-08 15:24:18 +: On this point, and just thinking out loud. If we consider saving compute_node information into say a node in said DLM backend (for example a znode in zookeeper[1]); this information would be updated periodi

Re: [openstack-dev] Scheduler proposal

On 9 October 2015 at 12:50, Chris Friesen wrote: > Has anybody looked at why 1 instance is too slow and what it would take to > >> make 1 scheduler instance work fast enough? This does not preclude the >> use of >> concurrency for finer grain tasks in the background. >> > > Currently we pull data

Re: [openstack-dev] Scheduler proposal

There are several ways to make python code that deals with a lot of data faster, especially when it comes to operating on DB fields from SQL tables (and that is not limited to the nova scheduler). Pulling data from large SQL tables and operating on them through regular python code (using python

Re: [openstack-dev] Scheduler proposal

Excerpts from Joshua Harlow's message of 2015-10-08 15:24:18 +: > On this point, and just thinking out loud. If we consider saving > compute_node information into say a node in said DLM backend (for > example a znode in zookeeper[1]); this information would be updated > periodically by that com

Re: [openstack-dev] Scheduler proposal

Excerpts from Chris Friesen's message of 2015-10-09 19:36:03 +: > On 10/09/2015 12:55 PM, Gregory Haynes wrote: > > > There is a more generalized version of this algorithm for concurrent > > scheduling I've seen a few times - Pick N options at random, apply > > heuristic over that N to pick th

Re: [openstack-dev] Scheduler proposal

And also we should probably deprecate/not recommend: http://docs.openstack.org/developer/nova/api/nova.scheduler.filters.json_filter.html#nova.scheduler.filters.json_filter.JsonFilter That filter IMHO basically disallows optimizations like forming SQL statements for each filter (and then lettin

Re: [openstack-dev] Scheduler proposal

On 10/09/2015 12:25 PM, Alec Hothan (ahothan) wrote: Still the point from Chris is valid. I guess the main reason openstack is going with multiple concurrent schedulers is to scale out by distributing the load between multiple instances of schedulers because 1 instance is too slow. This discussi

Re: [openstack-dev] Scheduler proposal

On 10/09/2015 12:55 PM, Gregory Haynes wrote: There is a more generalized version of this algorithm for concurrent scheduling I've seen a few times - Pick N options at random, apply heuristic over that N to pick the best, attempt to schedule at your choice, retry on failure. As long as you have

Re: [openstack-dev] Scheduler proposal

Excerpts from Zane Bitter's message of 2015-10-09 17:09:46 +: > On 08/10/15 21:32, Ian Wells wrote: > > > > > 2. if many hosts suit the 5 VMs then this is *very* unlucky,because > > we should be choosing a host at random from the set of > > suitable hosts and that's a huge coincidence

Re: [openstack-dev] Scheduler proposal

Still the point from Chris is valid. I guess the main reason openstack is going with multiple concurrent schedulers is to scale out by distributing the load between multiple instances of schedulers because 1 instance is too slow. This discussion is about coordinating the many instances of schedu

Re: [openstack-dev] Scheduler proposal

Excerpts from Chris Friesen's message of 2015-10-09 10:54:36 -0700: > On 10/09/2015 11:09 AM, Zane Bitter wrote: > > > The optimal way to do this would be a weighted random selection, where the > > probability of any given host being selected is proportional to its > > weighting. > > (Obviously t

Re: [openstack-dev] Scheduler proposal

On 10/09/2015 11:09 AM, Zane Bitter wrote: The optimal way to do this would be a weighted random selection, where the probability of any given host being selected is proportional to its weighting. (Obviously this is limited by the accuracy of the weighting function in expressing your actual pref

Re: [openstack-dev] Scheduler proposal

On 08/10/15 21:32, Ian Wells wrote: > 2. if many hosts suit the 5 VMs then this is *very* unlucky,because we should be choosing a host at random from the set of suitable hosts and that's a huge coincidence - so this is a tiny corner case that we shouldn't be designing around He

Re: [openstack-dev] Scheduler proposal

On 8 October 2015 at 13:28, Ed Leafe wrote: > On Oct 8, 2015, at 1:38 PM, Ian Wells wrote: > > Truth be told, storing that data in MySQL is secondary to the correct > functioning of the scheduler. > > I have no problem with MySQL (well, I do, but that's not relevant to this > discussion). My iss

Re: [openstack-dev] Scheduler proposal

On Oct 8, 2015, at 1:38 PM, Ian Wells wrote: >> You've hit upon the problem with the current design: multiple, and >> potentially out-of-sync copies of the data. > > Arguably, this is the *intent* of the current design, not a problem with it. It may have been the intent, but that doesn't mean

Re: [openstack-dev] Scheduler proposal

On 8 October 2015 at 09:10, Ed Leafe wrote: > You've hit upon the problem with the current design: multiple, and > potentially out-of-sync copies of the data. Arguably, this is the *intent* of the current design, not a problem with it. The data can never be perfect (ever) so go with 'good enou

Re: [openstack-dev] Scheduler proposal

On 7 October 2015 at 22:17, Chris Friesen wrote: > On 10/07/2015 07:23 PM, Ian Wells wrote: > >> >> The whole process is inherently racy (and this is inevitable, and >> correct), >> >> > Why is it inevitable? > It's inevitable because everything takes time, and some things are unpredictable. Th

Re: [openstack-dev] Scheduler proposal

Clint Byrum wrote: Excerpts from Joshua Harlow's message of 2015-10-08 08:38:57 -0700: Joshua Harlow wrote: On Thu, 8 Oct 2015 10:43:01 -0400 Monty Taylor wrote: On 10/08/2015 09:01 AM, Thierry Carrez wrote: Maish Saidel-Keesing wrote: Operational overhead has a cost - maintaining 3 diffe

Re: [openstack-dev] Scheduler proposal

On Oct 8, 2015, at 8:01 AM, Thierry Carrez wrote: >> Operational overhead has a cost - maintaining 3 different database >> tools, backing them up, providing HA, etc. has operational cost. >> >> This is not to say that this cannot be overseen, but it should be taken >> into consideration. >> >>

Re: [openstack-dev] Scheduler proposal

On Oct 8, 2015, at 10:54 AM, Clint Byrum wrote: > ^^ THIS is the kind of architectural thinking I'd like to see us do more > of. Agreed. If nothing else, I'm glad that I was able to get people thinking about new approaches. > This isn't "hey I have a better database" it is "I have a way to red

Re: [openstack-dev] Scheduler proposal

On Oct 8, 2015, at 11:03 AM, Kevin L. Mitchell wrote: >> Theoretically if the DB knew about what resources were originally available >> and >> what resources have been consumed, then it should be able to allocate >> resources >> race-free (possibly with some retries involved if racing against

Re: [openstack-dev] Scheduler proposal

On Oct 8, 2015, at 10:24 AM, Joshua Harlow wrote: > Now if we imagine each/all schedulers having watches > on /nova/compute_nodes/ ([2] consul and etc.d have equivalent concepts > afaik) then when a compute_node updates that information a push > notification (the watch being triggered) will be

Re: [openstack-dev] Scheduler proposal

On Wed, 2015-10-07 at 23:17 -0600, Chris Friesen wrote: > Why is it inevitable? Well, I would say that this is probably a consequence of the CAP[1] theorem. > Theoretically if the DB knew about what resources were originally available > and > what resources have been consumed, then it should be

Re: [openstack-dev] Scheduler proposal

Excerpts from Joshua Harlow's message of 2015-10-08 08:38:57 -0700: > Joshua Harlow wrote: > > On Thu, 8 Oct 2015 10:43:01 -0400 > > Monty Taylor wrote: > > > >> On 10/08/2015 09:01 AM, Thierry Carrez wrote: > >>> Maish Saidel-Keesing wrote: > Operational overhead has a cost - maintaining 3 d

Re: [openstack-dev] Scheduler proposal

Joshua Harlow wrote: On Thu, 8 Oct 2015 10:43:01 -0400 Monty Taylor wrote: On 10/08/2015 09:01 AM, Thierry Carrez wrote: Maish Saidel-Keesing wrote: Operational overhead has a cost - maintaining 3 different database tools, backing them up, providing HA, etc. has operational cost. This is no

Re: [openstack-dev] Scheduler proposal

On Thu, 8 Oct 2015 10:43:01 -0400 Monty Taylor wrote: > On 10/08/2015 09:01 AM, Thierry Carrez wrote: > > Maish Saidel-Keesing wrote: > >> Operational overhead has a cost - maintaining 3 different database > >> tools, backing them up, providing HA, etc. has operational cost. > >> > >> This is not

Re: [openstack-dev] Scheduler proposal

On 10/08/2015 09:01 AM, Thierry Carrez wrote: Maish Saidel-Keesing wrote: Operational overhead has a cost - maintaining 3 different database tools, backing them up, providing HA, etc. has operational cost. This is not to say that this cannot be overseen, but it should be taken into consideratio

Re: [openstack-dev] Scheduler proposal

Maish Saidel-Keesing wrote: > Operational overhead has a cost - maintaining 3 different database > tools, backing them up, providing HA, etc. has operational cost. > > This is not to say that this cannot be overseen, but it should be taken > into consideration. > > And *if* they can be consolidat

Re: [openstack-dev] Scheduler proposal

Forgive the top-post. Cross-posting to openstack-operators for their feedback as well. Ed the work seems very promising, and I am interested to see how this evolves. With my operator hat on I have one piece of feedback. By adding in a new Database solution (Cassandra) we are now up to three

Re: [openstack-dev] Scheduler proposal

On 10/07/2015 07:23 PM, Ian Wells wrote: On 7 October 2015 at 16:00, Chris Friesen mailto:chris.frie...@windriver.com>> wrote: 1) Some resources (RAM) only require tracking amounts. Other resources (CPUs, PCI devices) require tracking allocation of specific individual host resources

Re: [openstack-dev] Scheduler proposal

On Oct 7, 2015, at 6:00 PM, Chris Friesen wrote: > I've wondered for a while (ever since I looked at the scheduler code, really) > why we couldn't implement more of the scheduler as database transactions. > > I haven't used Cassandra, so maybe you can clarify something about updates > across a

Re: [openstack-dev] Scheduler proposal

On 7 October 2015 at 16:00, Chris Friesen wrote: > 1) Some resources (RAM) only require tracking amounts. Other resources > (CPUs, PCI devices) require tracking allocation of specific individual host > resources (for CPU pinning, PCI device allocation, etc.). Presumably for > the latter we woul

Re: [openstack-dev] Scheduler proposal

PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] Scheduler proposal On Oct 7, 2015, at 2:28 PM, Zane Bitter wrote: > It seems to me (disclaimer: not a Nova dev) that which database to use is > completely irrelevant to your proposal, Wel

Re: [openstack-dev] Scheduler proposal

On Oct 7, 2015, at 2:28 PM, Zane Bitter wrote: > It seems to me (disclaimer: not a Nova dev) that which database to use is > completely irrelevant to your proposal, Well, not entirely. The difference is that what Cassandra offers that separates it from other DBs is exactly the feature that we

Re: [openstack-dev] Scheduler proposal

On 10/07/2015 11:36 AM, Ed Leafe wrote: I've finally gotten around to finishing writing up that proposal [1], and I'd like to hope that it would be the basis for future discussions about addressing some of the underlying issues that exist in OpenStack for historical reasons, and how we might ret

  1   2   >