> On May 17, 2013, 7:12 p.m., Vinod Kone wrote: > > src/master/master.cpp, lines 1266-1272 > > <https://reviews.apache.org/r/11121/diff/2/?file=291857#file291857line1266> > > > > If we do decide to do this (I'm not completely convinced yet), we > > should do it in the allocator. > > Brenden Matthews wrote: > Care to elaborate? It looks like in > HierarchicalAllocatorProcess::allocate() the offers are in a hash map, so I'm > not sure how you could shuffle them there. > > Vinod Kone wrote: > Yes. You are right. The master gets a hashmap. > > That said, I'm still not convinced that master is the right place to do. > For example, can some frameworks benefit from getting the offers in the same > order? It seems to me that its pretty easy for frameworks to randomize, if > they choose to, but it would be hard for them to de-randomize randomized > offers. Makes sense? I was ok with doing in this in allocator because > allocators are pluggable.
I get what you're saying, but the problem is that AFAIK, most of the schedulers are 'dumb'. The schedulers assume the offers are ordered in a "use this one first" ordering (at least Hadoop and Chronos do this, Storm does something else). The hash map has kind of implicit ordering (intended or not), so you end up with poor distribution of tasks. The tasks are consistently assigned to the same slaves. I suppose I could just change the other schedulers, but this seems like a smaller change. I'm changing it in 1 place rather than N places. - Brenden ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11121/#review20716 ----------------------------------------------------------- On June 11, 2013, 9:54 p.m., Brenden Matthews wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/11121/ > ----------------------------------------------------------- > > (Updated June 11, 2013, 9:54 p.m.) > > > Review request for mesos. > > > Description > ------- > > Shuffle offers before sending to frameworks. > > This helps ensure fair allocation of resources. > > Review: https://reviews.apache.org/r/11121 > > > Diffs > ----- > > src/master/master.cpp 60c6d4f88f8024796c5e495d7cc6ddde2b754887 > > Diff: https://reviews.apache.org/r/11121/diff/ > > > Testing > ------- > > Used in production at airbnb. > > make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make > hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3 > > > Thanks, > > Brenden Matthews > >
