> On June 12, 2013, 7:06 p.m., Vinod Kone wrote: > > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, lines > > 471-488 > > <https://reviews.apache.org/r/11118/diff/4/?file=301280#file301280line471> > > > > How did you come up with the fair allocation and 60% spread policies? > > Is this what the default Hadoop scheduler does? > > > > It would be really awesome, if we can ask the underlying scheduler how > > to distribute the slots between maps and reduces, when we don't have > > sufficient resources. Have you looked into that? > > Brenden Matthews wrote: > I just chose the value arbitrarily. It has shown good results through > testing. > > I have not looked into that, but I'll investigate. Since jobtracker > schedulers are pluggable, it might be dependent on particular scheduler > implementation. I'm not sure there's a suitable interface for determining > how the underlying scheduler wants to allocate the slots. > > Brenden Matthews wrote: > I don't see anything about the interface that would allow this: > > > https://svn.apache.org/viewvc/hadoop/common/branches/branch-1.2/src/mapred/org/apache/hadoop/mapred/TaskScheduler.java?revision=1454892&view=markup > > Brenden Matthews wrote: > This still isn't perfect; I think it's much better than what we had > before, however. > > Vinod Kone wrote: > Yea, too bad the scheduler doesn't expose anything of the sort. I still > think the whole logic here is way too complicated. > > How about this? Since reduce tasks depend on map tasks, how about > stuffing as many map tasks as possible into the offer. Any remaining > resources in the offer can be used to fill up reduce tasks. > > > Brenden Matthews wrote: > This is a problem I've spent a lot of time working on. > > Handling slot starvation is tricky. Ideally you would only ever allocate > the exact number of slots you need for any given job, but that's not possible > when you have finite resources. > > Since the number of slots required varies with the flow of jobs, you can > do no better than trying to make a good guess about how many slots to > allocate. It's not great to have a fixed number of slots policy, because > most of the time you'll leave resources idle. > > Allocating many mappers and few reducers doesn't work very well; you'll > end up with jobs that take a very long time to complete because there aren't > enough reducers available. If you sacrifice some mappers in order to have > more reducers, the job will complete sooner. > > Vinod Kone wrote: > I think we are on the same page regarding not using a fixed number of > slots policy. I'm trying to figure out if there is a simpler or configurable > way to do this. > > I found some interesting articles regarding this (you might know these > already) : > > http://stackoverflow.com/questions/11672676/when-do-reduce-tasks-start-in-hadoop > https://issues.apache.org/jira/browse/MAPREDUCE-1184 > https://issues.apache.org/jira/browse/MAPREDUCE-1030 > > http://blog.cloudera.com/blog/2009/12/7-tips-for-improving-mapreduce-performance/ > > In particular, can we use "mapred.reduce.slowstart.completed.maps" > parameter to drive the slot allocation decision? > >
It doesn't change the number of mappers you need, it just delays the launch of the reduce tasks. I use 0.5 for this setting, which means it doesn't launch the reducers until half the maps are complete. In normal use, there are several jobs running simultaneously, so the slowstart helps to ensure reduce slots aren't occupied by reducers which are essentially idle waiting for maps to finish. Still, it doesn't help with the original problem of how to decide how many slots to allocate. - Brenden ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11118/#review21813 ----------------------------------------------------------- On June 26, 2013, 8:23 p.m., Brenden Matthews wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/11118/ > ----------------------------------------------------------- > > (Updated June 26, 2013, 8:23 p.m.) > > > Review request for mesos. > > > Repository: mesos > > > Description > ------- > > Be more intelligent about slot allocation. > > When we get a resource offer, we'll try to be clever about how we > allocate slots to the offers. Ideally we want to use as much of the > resources available when we have a lot of pending tasks, and we also > don't want to use more resources than we actually need. > > Review: https://reviews.apache.org/r/11118 > > > Diffs > ----- > > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java > c5a63bc7a172c44cc11d6bdd6b2eb4bc3df35cb7 > > Diff: https://reviews.apache.org/r/11118/diff/ > > > Testing > ------- > > Used in production at airbnb. > > > Thanks, > > Brenden Matthews > >
