> On June 12, 2013, 7:06 p.m., Vinod Kone wrote: > > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, lines > > 471-488 > > <https://reviews.apache.org/r/11118/diff/4/?file=301280#file301280line471> > > > > How did you come up with the fair allocation and 60% spread policies? > > Is this what the default Hadoop scheduler does? > > > > It would be really awesome, if we can ask the underlying scheduler how > > to distribute the slots between maps and reduces, when we don't have > > sufficient resources. Have you looked into that? > > Brenden Matthews wrote: > I just chose the value arbitrarily. It has shown good results through > testing. > > I have not looked into that, but I'll investigate. Since jobtracker > schedulers are pluggable, it might be dependent on particular scheduler > implementation. I'm not sure there's a suitable interface for determining > how the underlying scheduler wants to allocate the slots. > > Brenden Matthews wrote: > I don't see anything about the interface that would allow this: > > > https://svn.apache.org/viewvc/hadoop/common/branches/branch-1.2/src/mapred/org/apache/hadoop/mapred/TaskScheduler.java?revision=1454892&view=markup > > Brenden Matthews wrote: > This still isn't perfect; I think it's much better than what we had > before, however. > > Vinod Kone wrote: > Yea, too bad the scheduler doesn't expose anything of the sort. I still > think the whole logic here is way too complicated. > > How about this? Since reduce tasks depend on map tasks, how about > stuffing as many map tasks as possible into the offer. Any remaining > resources in the offer can be used to fill up reduce tasks. > > > Brenden Matthews wrote: > This is a problem I've spent a lot of time working on. > > Handling slot starvation is tricky. Ideally you would only ever allocate > the exact number of slots you need for any given job, but that's not possible > when you have finite resources. > > Since the number of slots required varies with the flow of jobs, you can > do no better than trying to make a good guess about how many slots to > allocate. It's not great to have a fixed number of slots policy, because > most of the time you'll leave resources idle. > > Allocating many mappers and few reducers doesn't work very well; you'll > end up with jobs that take a very long time to complete because there aren't > enough reducers available. If you sacrifice some mappers in order to have > more reducers, the job will complete sooner.
I think we are on the same page regarding not using a fixed number of slots policy. I'm trying to figure out if there is a simpler or configurable way to do this. I found some interesting articles regarding this (you might know these already) : http://stackoverflow.com/questions/11672676/when-do-reduce-tasks-start-in-hadoop https://issues.apache.org/jira/browse/MAPREDUCE-1184 https://issues.apache.org/jira/browse/MAPREDUCE-1030 http://blog.cloudera.com/blog/2009/12/7-tips-for-improving-mapreduce-performance/ In particular, can we use "mapred.reduce.slowstart.completed.maps" parameter to drive the slot allocation decision? - Vinod ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11118/#review21813 ----------------------------------------------------------- On June 26, 2013, 8:23 p.m., Brenden Matthews wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/11118/ > ----------------------------------------------------------- > > (Updated June 26, 2013, 8:23 p.m.) > > > Review request for mesos. > > > Repository: mesos > > > Description > ------- > > Be more intelligent about slot allocation. > > When we get a resource offer, we'll try to be clever about how we > allocate slots to the offers. Ideally we want to use as much of the > resources available when we have a lot of pending tasks, and we also > don't want to use more resources than we actually need. > > Review: https://reviews.apache.org/r/11118 > > > Diffs > ----- > > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java > c5a63bc7a172c44cc11d6bdd6b2eb4bc3df35cb7 > > Diff: https://reviews.apache.org/r/11118/diff/ > > > Testing > ------- > > Used in production at airbnb. > > > Thanks, > > Brenden Matthews > >
