What does it mean to have a 'dedicated' host that's free-for-all like that?
On Wed, Mar 9, 2016 at 2:16 PM, Maxim Khutornenko <ma...@apache.org> wrote: > Reactivating this thread. I like Bill's suggestion to have scheduler > dedicated constraint management system. It will, however, require a > substantial effort to get done properly. Would anyone oppose adopting > Steve's patch in the meantime? The ROI is so high it would be a crime > NOT to take it :) > > On Wed, Jan 20, 2016 at 10:25 AM, Maxim Khutornenko <ma...@apache.org> > wrote: > > I should have looked closely, you are right! This indeed addresses > > both cases: a job with a named dedicated role is still allowed to get > > though if it's role matches the constraint and everything else > > (non-exclusive dedicated pool) is addressed with "*". > > > > What it does not solve though is the variety of non-exclusive > > dedicated pools (e.g. GPU, OS, high network bandwidth and etc.). For > > that we would need something similar to what Bill suggested. > > > > On Wed, Jan 20, 2016 at 10:03 AM, Steve Niemitz <sniem...@apache.org> > wrote: > >> An arbitrary job can't target a fully dedicated role with this patch, it > >> will still get a "constraint not satisfied: dedicated" error. The code > in > >> the scheduler that matches the constraints does a simple string match, > so > >> "*/test" will not match "role1/test" when trying to place the task, it > will > >> only match "*/test". > >> > >> On Wed, Jan 20, 2016 at 12:24 PM, Maxim Khutornenko <ma...@apache.org> > >> wrote: > >> > >>> Thanks for the info, Steve! Yes, it would accomplish the same goal but > >>> at the price of removing the exclusive dedicated constraint > >>> enforcement. With this patch any job could target a fully dedicated > >>> exclusive pool, which may be undesirable for dedicated pool owners. > >>> > >>> > >>> > >>> On Wed, Jan 20, 2016 at 7:13 AM, Steve Niemitz <sniem...@apache.org> > >>> wrote: > >>> > We've been running a trivial patch [1] that does what I believe > you're > >>> > talking about for awhile now. It allows a * for the role name, > basically > >>> > allowing any role to match the constraint, so our constraints look > like > >>> > "*/secure" > >>> > > >>> > Our use case is we have a "secure" cluster of machines that is > >>> constrained > >>> > on what can run on it (via an external audit process) that multiple > roles > >>> > run on. > >>> > > >>> > I believe I had talked to Bill about this a few months ago, but I > don't > >>> > remember where it ended up. > >>> > > >>> > [1] > >>> > > >>> > https://github.com/tellapart/aurora/commit/76f978c76cc1377e19e602f7e0d050f7ce353562 > >>> > > >>> > On Tue, Jan 19, 2016 at 11:48 PM, Maxim Khutornenko < > ma...@apache.org> > >>> > wrote: > >>> > > >>> >> Oh, I didn't mean the memory GC pressure in the pure sense, rather a > >>> >> logical garbage of orphaned hosts that never leave the scheduler. > It's > >>> >> not something to be concerned about from the performance standpoint. > >>> >> It's, however, something operators need to be aware of when a host > >>> >> from a dedicated pool gets dropped or replaced. > >>> >> > >>> >> On Tue, Jan 19, 2016 at 8:39 PM, Bill Farner <wfar...@apache.org> > >>> wrote: > >>> >> > What do you mean by GC burden? What i'm proposing is effectively > >>> >> > Map<String, String>. Even with an extremely forgetful operator > (even > >>> >> more > >>> >> > than Joe!), it would require a huge oversight to put a dent in > heap > >>> >> usage. > >>> >> > I'm sure there are ways we could even expose a useful stat to flag > >>> such > >>> >> an > >>> >> > oversight. > >>> >> > > >>> >> > On Tue, Jan 19, 2016 at 8:31 PM, Maxim Khutornenko < > ma...@apache.org> > >>> >> wrote: > >>> >> > > >>> >> >> Right, that's what I thought. Yes, it sounds interesting. My only > >>> >> >> concern is the GC burden of getting rid of hostnames that are > >>> obsolete > >>> >> >> and no longer exist. Relying on offers to update hostname > 'relevance' > >>> >> >> may not work as dedicated hosts may be fully packed and not > release > >>> >> >> any resources for a very long time. Let me explore this idea a > bit to > >>> >> >> see what it would take to implement. > >>> >> >> > >>> >> >> On Tue, Jan 19, 2016 at 8:22 PM, Bill Farner <wfar...@apache.org > > > >>> >> wrote: > >>> >> >> > Not a host->attribute mapping (attribute in the mesos sense, > >>> anyway). > >>> >> >> Rather > >>> >> >> > an out-of-band API for marking machines as reserved. For > >>> task->offer > >>> >> >> > mapping it's just a matter of another data source. Does that > make > >>> >> sense? > >>> >> >> > > >>> >> >> > On Tuesday, January 19, 2016, Maxim Khutornenko < > ma...@apache.org> > >>> >> >> wrote: > >>> >> >> > > >>> >> >> >> > > >>> >> >> >> > Can't this just be any old Constraint (not named > "dedicated"). > >>> In > >>> >> >> other > >>> >> >> >> > words, doesn't this code already deal with non-dedicated > >>> >> constraints?: > >>> >> >> >> > > >>> >> >> >> > > >>> >> >> >> > >>> >> >> > >>> >> > >>> > https://github.com/apache/aurora/blob/master/src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java#L193-L197 > >>> >> >> >> > >>> >> >> >> > >>> >> >> >> Not really. There is a subtle difference here. A regular > >>> >> (non-dedicated) > >>> >> >> >> constraint does not prevent other tasks from landing on a > given > >>> >> machine > >>> >> >> set > >>> >> >> >> whereas dedicated keeps other tasks away by only allowing > those > >>> >> matching > >>> >> >> >> the dedicated attribute. What this proposal targets is > allowing > >>> >> >> exclusive > >>> >> >> >> machine pool matching any job that has this new constraint > while > >>> >> keeping > >>> >> >> >> all other tasks that don't have that attribute away. > >>> >> >> >> > >>> >> >> >> Following an example from my original post, imagine a GPU > machine > >>> >> pool. > >>> >> >> Any > >>> >> >> >> job (from any role) requiring GPU resource would be allowed > while > >>> all > >>> >> >> other > >>> >> >> >> jobs that don't have that constraint would be vetoed. > >>> >> >> >> > >>> >> >> >> Also, regarding dedicated constraints necessitating a slave > >>> restart - > >>> >> >> i've > >>> >> >> >> > pondered moving dedicated machine management to the > scheduler > >>> for > >>> >> >> similar > >>> >> >> >> > purposes. There's not really much forcing that behavior to > be > >>> >> managed > >>> >> >> >> with > >>> >> >> >> > a slave attribute. > >>> >> >> >> > >>> >> >> >> > >>> >> >> >> Would you mind giving a few more hints on the mechanics behind > >>> this? > >>> >> How > >>> >> >> >> would scheduler know about dedicated hw without the slave > >>> attributes > >>> >> >> set? > >>> >> >> >> Are you proposing storing hostname->attribute mapping in the > >>> >> scheduler > >>> >> >> >> store? > >>> >> >> >> > >>> >> >> >> On Tue, Jan 19, 2016 at 7:53 PM, Bill Farner < > wfar...@apache.org > >>> >> >> >> <javascript:;>> wrote: > >>> >> >> >> > >>> >> >> >> > Joe - if you want to pursue this, I suggest you start > another > >>> >> thread > >>> >> >> to > >>> >> >> >> > keep this thread's discussion in tact. I will not be able > to > >>> lead > >>> >> >> this > >>> >> >> >> > change, but can certainly shepherd! > >>> >> >> >> > > >>> >> >> >> > On Tuesday, January 19, 2016, Joe Smith < > yasumo...@gmail.com > >>> >> >> >> <javascript:;>> wrote: > >>> >> >> >> > > >>> >> >> >> > > As an operator, that'd be a relatively simple change in > >>> tooling, > >>> >> and > >>> >> >> >> the > >>> >> >> >> > > benefits of not forcing a slave restart would be _huge_. > >>> >> >> >> > > > >>> >> >> >> > > Keeping the dedicated semantics (but adding non-exclusive) > >>> would > >>> >> be > >>> >> >> >> ideal > >>> >> >> >> > > if possible. > >>> >> >> >> > > > >>> >> >> >> > > > On Jan 19, 2016, at 19:09, Bill Farner < > wfar...@apache.org > >>> >> >> >> <javascript:;> > >>> >> >> >> > > <javascript:;>> wrote: > >>> >> >> >> > > > > >>> >> >> >> > > > Also, regarding dedicated constraints necessitating a > slave > >>> >> >> restart - > >>> >> >> >> > > i've > >>> >> >> >> > > > pondered moving dedicated machine management to the > >>> scheduler > >>> >> for > >>> >> >> >> > similar > >>> >> >> >> > > > purposes. There's not really much forcing that > behavior to > >>> be > >>> >> >> >> managed > >>> >> >> >> > > with > >>> >> >> >> > > > a slave attribute. > >>> >> >> >> > > > > >>> >> >> >> > > > On Tue, Jan 19, 2016 at 7:05 PM, John Sirois < > >>> >> j...@conductant.com > >>> >> >> >> <javascript:;> > >>> >> >> >> > > <javascript:;>> wrote: > >>> >> >> >> > > > > >>> >> >> >> > > >> On Tue, Jan 19, 2016 at 7:22 PM, Maxim Khutornenko < > >>> >> >> >> ma...@apache.org <javascript:;> > >>> >> >> >> > > <javascript:;>> > >>> >> >> >> > > >> wrote: > >>> >> >> >> > > >> > >>> >> >> >> > > >>> Has anyone explored an idea of having a non-exclusive > (wrt > >>> >> job > >>> >> >> >> role) > >>> >> >> >> > > >>> dedicated constraint in Aurora before? > >>> >> >> >> > > >> > >>> >> >> >> > > >> > >>> >> >> >> > > >>> We do have a dedicated constraint now but it assumes > a 1:1 > >>> >> >> >> > > >>> relationship between a job role and a slave attribute > [1]. > >>> >> For > >>> >> >> >> > > >>> example: a 'www-data/prod/hello' job with a dedicated > >>> >> >> constraint of > >>> >> >> >> > > >>> 'dedicated': 'www-data/hello' may only be pinned to a > >>> >> particular > >>> >> >> >> set > >>> >> >> >> > > >>> of slaves if all of them have 'www-data/hello' > attribute > >>> >> set. No > >>> >> >> >> > other > >>> >> >> >> > > >>> role tasks will be able to land on those slaves unless > >>> their > >>> >> >> >> > > >>> 'role/name' pair is added into the slave attribute > set. > >>> >> >> >> > > >>> > >>> >> >> >> > > >>> The above is very limiting as it prevents carving out > >>> subsets > >>> >> >> of a > >>> >> >> >> > > >>> shared pool cluster to be used by multiple roles at > the > >>> same > >>> >> >> time. > >>> >> >> >> > > >>> Would it make sense to have a free-form dedicated > >>> constraint > >>> >> not > >>> >> >> >> > bound > >>> >> >> >> > > >>> to a particular role? Multiple jobs could then use > this > >>> type > >>> >> of > >>> >> >> >> > > >>> constraint dynamically without modifying the slave > command > >>> >> line > >>> >> >> >> (and > >>> >> >> >> > > >>> requiring slave restart). > >>> >> >> >> > > >> > >>> >> >> >> > > >> Can't this just be any old Constraint (not named > >>> "dedicated"). > >>> >> >> In > >>> >> >> >> > other > >>> >> >> >> > > >> words, doesn't this code already deal with > non-dedicated > >>> >> >> >> constraints?: > >>> >> >> >> > > >> > >>> >> >> >> > > >> > >>> >> >> >> > > > >>> >> >> >> > > >>> >> >> >> > >>> >> >> > >>> >> > >>> > https://github.com/apache/aurora/blob/master/src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java#L193-L197 > >>> >> >> >> > > >> > >>> >> >> >> > > >> > >>> >> >> >> > > >>> This could be quite useful for experimenting purposes > >>> (e.g. > >>> >> >> >> different > >>> >> >> >> > > >>> host OS) or to target a different hardware offering > (e.g. > >>> >> >> GPUs). In > >>> >> >> >> > > >>> other words, only those jobs that explicitly opt-in to > >>> >> >> participate > >>> >> >> >> in > >>> >> >> >> > > >>> an experiment or hw offering would be landing on that > >>> slave > >>> >> set. > >>> >> >> >> > > >>> > >>> >> >> >> > > >>> Thanks, > >>> >> >> >> > > >>> Maxim > >>> >> >> >> > > >>> > >>> >> >> >> > > >>> [1]- > >>> >> >> >> > > >> > >>> >> >> >> > > > >>> >> >> >> > > >>> >> >> >> > >>> >> >> > >>> >> > >>> > https://github.com/apache/aurora/blob/eec985d948f02f46637d87cd4d212eb2a70ef8d0/src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java#L272-L276 > >>> >> >> >> > > >> > >>> >> >> >> > > >> > >>> >> >> >> > > >> > >>> >> >> >> > > >> -- > >>> >> >> >> > > >> John Sirois > >>> >> >> >> > > >> 303-512-3301 > >>> >> >> >> > > >> > >>> >> >> >> > > > >>> >> >> >> > > >>> >> >> >> > >>> >> >> > >>> >> > >>> >
