> On June 12, 2013, 7:06 p.m., Vinod Kone wrote: > > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, lines > > 471-488 > > <https://reviews.apache.org/r/11118/diff/4/?file=301280#file301280line471> > > > > How did you come up with the fair allocation and 60% spread policies? > > Is this what the default Hadoop scheduler does? > > > > It would be really awesome, if we can ask the underlying scheduler how > > to distribute the slots between maps and reduces, when we don't have > > sufficient resources. Have you looked into that? > > Brenden Matthews wrote: > I just chose the value arbitrarily. It has shown good results through > testing. > > I have not looked into that, but I'll investigate. Since jobtracker > schedulers are pluggable, it might be dependent on particular scheduler > implementation. I'm not sure there's a suitable interface for determining > how the underlying scheduler wants to allocate the slots. > > Brenden Matthews wrote: > I don't see anything about the interface that would allow this: > > > https://svn.apache.org/viewvc/hadoop/common/branches/branch-1.2/src/mapred/org/apache/hadoop/mapred/TaskScheduler.java?revision=1454892&view=markup > > Brenden Matthews wrote: > This still isn't perfect; I think it's much better than what we had > before, however.
Yea, too bad the scheduler doesn't expose anything of the sort. I still think the whole logic here is way too complicated. How about this? Since reduce tasks depend on map tasks, how about stuffing as many map tasks as possible into the offer. Any remaining resources in the offer can be used to fill up reduce tasks. - Vinod ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11118/#review21813 ----------------------------------------------------------- On June 26, 2013, 8:23 p.m., Brenden Matthews wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/11118/ > ----------------------------------------------------------- > > (Updated June 26, 2013, 8:23 p.m.) > > > Review request for mesos. > > > Repository: mesos > > > Description > ------- > > Be more intelligent about slot allocation. > > When we get a resource offer, we'll try to be clever about how we > allocate slots to the offers. Ideally we want to use as much of the > resources available when we have a lot of pending tasks, and we also > don't want to use more resources than we actually need. > > Review: https://reviews.apache.org/r/11118 > > > Diffs > ----- > > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java > c5a63bc7a172c44cc11d6bdd6b2eb4bc3df35cb7 > > Diff: https://reviews.apache.org/r/11118/diff/ > > > Testing > ------- > > Used in production at airbnb. > > > Thanks, > > Brenden Matthews > >
