> On June 12, 2013, 7:06 p.m., Vinod Kone wrote: > > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, lines > > 471-488 > > <https://reviews.apache.org/r/11118/diff/4/?file=301280#file301280line471> > > > > How did you come up with the fair allocation and 60% spread policies? > > Is this what the default Hadoop scheduler does? > > > > It would be really awesome, if we can ask the underlying scheduler how > > to distribute the slots between maps and reduces, when we don't have > > sufficient resources. Have you looked into that? > > Brenden Matthews wrote: > I just chose the value arbitrarily. It has shown good results through > testing. > > I have not looked into that, but I'll investigate. Since jobtracker > schedulers are pluggable, it might be dependent on particular scheduler > implementation. I'm not sure there's a suitable interface for determining > how the underlying scheduler wants to allocate the slots. > > Brenden Matthews wrote: > I don't see anything about the interface that would allow this: > > > https://svn.apache.org/viewvc/hadoop/common/branches/branch-1.2/src/mapred/org/apache/hadoop/mapred/TaskScheduler.java?revision=1454892&view=markup
This still isn't perfect; I think it's much better than what we had before, however. - Brenden ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11118/#review21813 ----------------------------------------------------------- On June 6, 2013, 2:12 a.m., Brenden Matthews wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/11118/ > ----------------------------------------------------------- > > (Updated June 6, 2013, 2:12 a.m.) > > > Review request for mesos. > > > Repository: mesos > > > Description > ------- > > Be more intelligent about slot allocation. > > When we get a resource offer, we'll try to be clever about how we > allocate slots to the offers. Ideally we want to use as much of the > resources available when we have a lot of pending tasks, and we also > don't want to use more resources than we actually need. > > Review: https://reviews.apache.org/r/11118 > > > Diffs > ----- > > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java > afe401f5265e3d9494af7eace42eec45943184a3 > > Diff: https://reviews.apache.org/r/11118/diff/ > > > Testing > ------- > > Used in production at airbnb. > > > Thanks, > > Brenden Matthews > >
