Ah, so it only practically makes sense when the dedicated attribute is */something, but * would not make much sense. Seems reasonable to me.
On Wed, Mar 9, 2016 at 2:32 PM, Maxim Khutornenko <ma...@apache.org> wrote: > It's an *easy* way to get a virtual cluster with specific > requirements. One example: have a set of machines in a shared pool > with a different OS. This would let any existing or new customers try > their services for compliance. The alternative would be spinning off a > completely new physical cluster, which is a huge overhead on both > supply and demand sides. > > On Wed, Mar 9, 2016 at 2:26 PM, Bill Farner <wfar...@apache.org> wrote: > > What does it mean to have a 'dedicated' host that's free-for-all like > that? > > > > On Wed, Mar 9, 2016 at 2:16 PM, Maxim Khutornenko <ma...@apache.org> > wrote: > > > >> Reactivating this thread. I like Bill's suggestion to have scheduler > >> dedicated constraint management system. It will, however, require a > >> substantial effort to get done properly. Would anyone oppose adopting > >> Steve's patch in the meantime? The ROI is so high it would be a crime > >> NOT to take it :) > >> > >> On Wed, Jan 20, 2016 at 10:25 AM, Maxim Khutornenko <ma...@apache.org> > >> wrote: > >> > I should have looked closely, you are right! This indeed addresses > >> > both cases: a job with a named dedicated role is still allowed to get > >> > though if it's role matches the constraint and everything else > >> > (non-exclusive dedicated pool) is addressed with "*". > >> > > >> > What it does not solve though is the variety of non-exclusive > >> > dedicated pools (e.g. GPU, OS, high network bandwidth and etc.). For > >> > that we would need something similar to what Bill suggested. > >> > > >> > On Wed, Jan 20, 2016 at 10:03 AM, Steve Niemitz <sniem...@apache.org> > >> wrote: > >> >> An arbitrary job can't target a fully dedicated role with this > patch, it > >> >> will still get a "constraint not satisfied: dedicated" error. The > code > >> in > >> >> the scheduler that matches the constraints does a simple string > match, > >> so > >> >> "*/test" will not match "role1/test" when trying to place the task, > it > >> will > >> >> only match "*/test". > >> >> > >> >> On Wed, Jan 20, 2016 at 12:24 PM, Maxim Khutornenko < > ma...@apache.org> > >> >> wrote: > >> >> > >> >>> Thanks for the info, Steve! Yes, it would accomplish the same goal > but > >> >>> at the price of removing the exclusive dedicated constraint > >> >>> enforcement. With this patch any job could target a fully dedicated > >> >>> exclusive pool, which may be undesirable for dedicated pool owners. > >> >>> > >> >>> > >> >>> > >> >>> On Wed, Jan 20, 2016 at 7:13 AM, Steve Niemitz <sniem...@apache.org > > > >> >>> wrote: > >> >>> > We've been running a trivial patch [1] that does what I believe > >> you're > >> >>> > talking about for awhile now. It allows a * for the role name, > >> basically > >> >>> > allowing any role to match the constraint, so our constraints look > >> like > >> >>> > "*/secure" > >> >>> > > >> >>> > Our use case is we have a "secure" cluster of machines that is > >> >>> constrained > >> >>> > on what can run on it (via an external audit process) that > multiple > >> roles > >> >>> > run on. > >> >>> > > >> >>> > I believe I had talked to Bill about this a few months ago, but I > >> don't > >> >>> > remember where it ended up. > >> >>> > > >> >>> > [1] > >> >>> > > >> >>> > >> > https://github.com/tellapart/aurora/commit/76f978c76cc1377e19e602f7e0d050f7ce353562 > >> >>> > > >> >>> > On Tue, Jan 19, 2016 at 11:48 PM, Maxim Khutornenko < > >> ma...@apache.org> > >> >>> > wrote: > >> >>> > > >> >>> >> Oh, I didn't mean the memory GC pressure in the pure sense, > rather a > >> >>> >> logical garbage of orphaned hosts that never leave the scheduler. > >> It's > >> >>> >> not something to be concerned about from the performance > standpoint. > >> >>> >> It's, however, something operators need to be aware of when a > host > >> >>> >> from a dedicated pool gets dropped or replaced. > >> >>> >> > >> >>> >> On Tue, Jan 19, 2016 at 8:39 PM, Bill Farner <wfar...@apache.org > > > >> >>> wrote: > >> >>> >> > What do you mean by GC burden? What i'm proposing is > effectively > >> >>> >> > Map<String, String>. Even with an extremely forgetful operator > >> (even > >> >>> >> more > >> >>> >> > than Joe!), it would require a huge oversight to put a dent in > >> heap > >> >>> >> usage. > >> >>> >> > I'm sure there are ways we could even expose a useful stat to > flag > >> >>> such > >> >>> >> an > >> >>> >> > oversight. > >> >>> >> > > >> >>> >> > On Tue, Jan 19, 2016 at 8:31 PM, Maxim Khutornenko < > >> ma...@apache.org> > >> >>> >> wrote: > >> >>> >> > > >> >>> >> >> Right, that's what I thought. Yes, it sounds interesting. My > only > >> >>> >> >> concern is the GC burden of getting rid of hostnames that are > >> >>> obsolete > >> >>> >> >> and no longer exist. Relying on offers to update hostname > >> 'relevance' > >> >>> >> >> may not work as dedicated hosts may be fully packed and not > >> release > >> >>> >> >> any resources for a very long time. Let me explore this idea a > >> bit to > >> >>> >> >> see what it would take to implement. > >> >>> >> >> > >> >>> >> >> On Tue, Jan 19, 2016 at 8:22 PM, Bill Farner < > wfar...@apache.org > >> > > >> >>> >> wrote: > >> >>> >> >> > Not a host->attribute mapping (attribute in the mesos sense, > >> >>> anyway). > >> >>> >> >> Rather > >> >>> >> >> > an out-of-band API for marking machines as reserved. For > >> >>> task->offer > >> >>> >> >> > mapping it's just a matter of another data source. Does > that > >> make > >> >>> >> sense? > >> >>> >> >> > > >> >>> >> >> > On Tuesday, January 19, 2016, Maxim Khutornenko < > >> ma...@apache.org> > >> >>> >> >> wrote: > >> >>> >> >> > > >> >>> >> >> >> > > >> >>> >> >> >> > Can't this just be any old Constraint (not named > >> "dedicated"). > >> >>> In > >> >>> >> >> other > >> >>> >> >> >> > words, doesn't this code already deal with non-dedicated > >> >>> >> constraints?: > >> >>> >> >> >> > > >> >>> >> >> >> > > >> >>> >> >> >> > >> >>> >> >> > >> >>> >> > >> >>> > >> > https://github.com/apache/aurora/blob/master/src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java#L193-L197 > >> >>> >> >> >> > >> >>> >> >> >> > >> >>> >> >> >> Not really. There is a subtle difference here. A regular > >> >>> >> (non-dedicated) > >> >>> >> >> >> constraint does not prevent other tasks from landing on a > >> given > >> >>> >> machine > >> >>> >> >> set > >> >>> >> >> >> whereas dedicated keeps other tasks away by only allowing > >> those > >> >>> >> matching > >> >>> >> >> >> the dedicated attribute. What this proposal targets is > >> allowing > >> >>> >> >> exclusive > >> >>> >> >> >> machine pool matching any job that has this new constraint > >> while > >> >>> >> keeping > >> >>> >> >> >> all other tasks that don't have that attribute away. > >> >>> >> >> >> > >> >>> >> >> >> Following an example from my original post, imagine a GPU > >> machine > >> >>> >> pool. > >> >>> >> >> Any > >> >>> >> >> >> job (from any role) requiring GPU resource would be allowed > >> while > >> >>> all > >> >>> >> >> other > >> >>> >> >> >> jobs that don't have that constraint would be vetoed. > >> >>> >> >> >> > >> >>> >> >> >> Also, regarding dedicated constraints necessitating a slave > >> >>> restart - > >> >>> >> >> i've > >> >>> >> >> >> > pondered moving dedicated machine management to the > >> scheduler > >> >>> for > >> >>> >> >> similar > >> >>> >> >> >> > purposes. There's not really much forcing that behavior > to > >> be > >> >>> >> managed > >> >>> >> >> >> with > >> >>> >> >> >> > a slave attribute. > >> >>> >> >> >> > >> >>> >> >> >> > >> >>> >> >> >> Would you mind giving a few more hints on the mechanics > behind > >> >>> this? > >> >>> >> How > >> >>> >> >> >> would scheduler know about dedicated hw without the slave > >> >>> attributes > >> >>> >> >> set? > >> >>> >> >> >> Are you proposing storing hostname->attribute mapping in > the > >> >>> >> scheduler > >> >>> >> >> >> store? > >> >>> >> >> >> > >> >>> >> >> >> On Tue, Jan 19, 2016 at 7:53 PM, Bill Farner < > >> wfar...@apache.org > >> >>> >> >> >> <javascript:;>> wrote: > >> >>> >> >> >> > >> >>> >> >> >> > Joe - if you want to pursue this, I suggest you start > >> another > >> >>> >> thread > >> >>> >> >> to > >> >>> >> >> >> > keep this thread's discussion in tact. I will not be > able > >> to > >> >>> lead > >> >>> >> >> this > >> >>> >> >> >> > change, but can certainly shepherd! > >> >>> >> >> >> > > >> >>> >> >> >> > On Tuesday, January 19, 2016, Joe Smith < > >> yasumo...@gmail.com > >> >>> >> >> >> <javascript:;>> wrote: > >> >>> >> >> >> > > >> >>> >> >> >> > > As an operator, that'd be a relatively simple change in > >> >>> tooling, > >> >>> >> and > >> >>> >> >> >> the > >> >>> >> >> >> > > benefits of not forcing a slave restart would be > _huge_. > >> >>> >> >> >> > > > >> >>> >> >> >> > > Keeping the dedicated semantics (but adding > non-exclusive) > >> >>> would > >> >>> >> be > >> >>> >> >> >> ideal > >> >>> >> >> >> > > if possible. > >> >>> >> >> >> > > > >> >>> >> >> >> > > > On Jan 19, 2016, at 19:09, Bill Farner < > >> wfar...@apache.org > >> >>> >> >> >> <javascript:;> > >> >>> >> >> >> > > <javascript:;>> wrote: > >> >>> >> >> >> > > > > >> >>> >> >> >> > > > Also, regarding dedicated constraints necessitating a > >> slave > >> >>> >> >> restart - > >> >>> >> >> >> > > i've > >> >>> >> >> >> > > > pondered moving dedicated machine management to the > >> >>> scheduler > >> >>> >> for > >> >>> >> >> >> > similar > >> >>> >> >> >> > > > purposes. There's not really much forcing that > >> behavior to > >> >>> be > >> >>> >> >> >> managed > >> >>> >> >> >> > > with > >> >>> >> >> >> > > > a slave attribute. > >> >>> >> >> >> > > > > >> >>> >> >> >> > > > On Tue, Jan 19, 2016 at 7:05 PM, John Sirois < > >> >>> >> j...@conductant.com > >> >>> >> >> >> <javascript:;> > >> >>> >> >> >> > > <javascript:;>> wrote: > >> >>> >> >> >> > > > > >> >>> >> >> >> > > >> On Tue, Jan 19, 2016 at 7:22 PM, Maxim Khutornenko < > >> >>> >> >> >> ma...@apache.org <javascript:;> > >> >>> >> >> >> > > <javascript:;>> > >> >>> >> >> >> > > >> wrote: > >> >>> >> >> >> > > >> > >> >>> >> >> >> > > >>> Has anyone explored an idea of having a > non-exclusive > >> (wrt > >> >>> >> job > >> >>> >> >> >> role) > >> >>> >> >> >> > > >>> dedicated constraint in Aurora before? > >> >>> >> >> >> > > >> > >> >>> >> >> >> > > >> > >> >>> >> >> >> > > >>> We do have a dedicated constraint now but it > assumes > >> a 1:1 > >> >>> >> >> >> > > >>> relationship between a job role and a slave > attribute > >> [1]. > >> >>> >> For > >> >>> >> >> >> > > >>> example: a 'www-data/prod/hello' job with a > dedicated > >> >>> >> >> constraint of > >> >>> >> >> >> > > >>> 'dedicated': 'www-data/hello' may only be pinned > to a > >> >>> >> particular > >> >>> >> >> >> set > >> >>> >> >> >> > > >>> of slaves if all of them have 'www-data/hello' > >> attribute > >> >>> >> set. No > >> >>> >> >> >> > other > >> >>> >> >> >> > > >>> role tasks will be able to land on those slaves > unless > >> >>> their > >> >>> >> >> >> > > >>> 'role/name' pair is added into the slave attribute > >> set. > >> >>> >> >> >> > > >>> > >> >>> >> >> >> > > >>> The above is very limiting as it prevents carving > out > >> >>> subsets > >> >>> >> >> of a > >> >>> >> >> >> > > >>> shared pool cluster to be used by multiple roles at > >> the > >> >>> same > >> >>> >> >> time. > >> >>> >> >> >> > > >>> Would it make sense to have a free-form dedicated > >> >>> constraint > >> >>> >> not > >> >>> >> >> >> > bound > >> >>> >> >> >> > > >>> to a particular role? Multiple jobs could then use > >> this > >> >>> type > >> >>> >> of > >> >>> >> >> >> > > >>> constraint dynamically without modifying the slave > >> command > >> >>> >> line > >> >>> >> >> >> (and > >> >>> >> >> >> > > >>> requiring slave restart). > >> >>> >> >> >> > > >> > >> >>> >> >> >> > > >> Can't this just be any old Constraint (not named > >> >>> "dedicated"). > >> >>> >> >> In > >> >>> >> >> >> > other > >> >>> >> >> >> > > >> words, doesn't this code already deal with > >> non-dedicated > >> >>> >> >> >> constraints?: > >> >>> >> >> >> > > >> > >> >>> >> >> >> > > >> > >> >>> >> >> >> > > > >> >>> >> >> >> > > >> >>> >> >> >> > >> >>> >> >> > >> >>> >> > >> >>> > >> > https://github.com/apache/aurora/blob/master/src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java#L193-L197 > >> >>> >> >> >> > > >> > >> >>> >> >> >> > > >> > >> >>> >> >> >> > > >>> This could be quite useful for experimenting > purposes > >> >>> (e.g. > >> >>> >> >> >> different > >> >>> >> >> >> > > >>> host OS) or to target a different hardware offering > >> (e.g. > >> >>> >> >> GPUs). In > >> >>> >> >> >> > > >>> other words, only those jobs that explicitly > opt-in to > >> >>> >> >> participate > >> >>> >> >> >> in > >> >>> >> >> >> > > >>> an experiment or hw offering would be landing on > that > >> >>> slave > >> >>> >> set. > >> >>> >> >> >> > > >>> > >> >>> >> >> >> > > >>> Thanks, > >> >>> >> >> >> > > >>> Maxim > >> >>> >> >> >> > > >>> > >> >>> >> >> >> > > >>> [1]- > >> >>> >> >> >> > > >> > >> >>> >> >> >> > > > >> >>> >> >> >> > > >> >>> >> >> >> > >> >>> >> >> > >> >>> >> > >> >>> > >> > https://github.com/apache/aurora/blob/eec985d948f02f46637d87cd4d212eb2a70ef8d0/src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java#L272-L276 > >> >>> >> >> >> > > >> > >> >>> >> >> >> > > >> > >> >>> >> >> >> > > >> > >> >>> >> >> >> > > >> -- > >> >>> >> >> >> > > >> John Sirois > >> >>> >> >> >> > > >> 303-512-3301 > >> >>> >> >> >> > > >> > >> >>> >> >> >> > > > >> >>> >> >> >> > > >> >>> >> >> >> > >> >>> >> >> > >> >>> >> > >> >>> > >> >
