> On June 12, 2013, 7:06 p.m., Vinod Kone wrote: > > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java, lines > > 471-488 > > <https://reviews.apache.org/r/11118/diff/4/?file=301280#file301280line471> > > > > How did you come up with the fair allocation and 60% spread policies? > > Is this what the default Hadoop scheduler does? > > > > It would be really awesome, if we can ask the underlying scheduler how > > to distribute the slots between maps and reduces, when we don't have > > sufficient resources. Have you looked into that? > > Brenden Matthews wrote: > I just chose the value arbitrarily. It has shown good results through > testing. > > I have not looked into that, but I'll investigate. Since jobtracker > schedulers are pluggable, it might be dependent on particular scheduler > implementation. I'm not sure there's a suitable interface for determining > how the underlying scheduler wants to allocate the slots. > > Brenden Matthews wrote: > I don't see anything about the interface that would allow this: > > > https://svn.apache.org/viewvc/hadoop/common/branches/branch-1.2/src/mapred/org/apache/hadoop/mapred/TaskScheduler.java?revision=1454892&view=markup > > Brenden Matthews wrote: > This still isn't perfect; I think it's much better than what we had > before, however. > > Vinod Kone wrote: > Yea, too bad the scheduler doesn't expose anything of the sort. I still > think the whole logic here is way too complicated. > > How about this? Since reduce tasks depend on map tasks, how about > stuffing as many map tasks as possible into the offer. Any remaining > resources in the offer can be used to fill up reduce tasks. >
This is a problem I've spent a lot of time working on. Handling slot starvation is tricky. Ideally you would only ever allocate the exact number of slots you need for any given job, but that's not possible when you have finite resources. Since the number of slots required varies with the flow of jobs, you can do no better than trying to make a good guess about how many slots to allocate. It's not great to have a fixed number of slots policy, because most of the time you'll leave resources idle. Allocating many mappers and few reducers doesn't work very well; you'll end up with jobs that take a very long time to complete because there aren't enough reducers available. If you sacrifice some mappers in order to have more reducers, the job will complete sooner. - Brenden ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11118/#review21813 ----------------------------------------------------------- On June 26, 2013, 8:23 p.m., Brenden Matthews wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/11118/ > ----------------------------------------------------------- > > (Updated June 26, 2013, 8:23 p.m.) > > > Review request for mesos. > > > Repository: mesos > > > Description > ------- > > Be more intelligent about slot allocation. > > When we get a resource offer, we'll try to be clever about how we > allocate slots to the offers. Ideally we want to use as much of the > resources available when we have a lot of pending tasks, and we also > don't want to use more resources than we actually need. > > Review: https://reviews.apache.org/r/11118 > > > Diffs > ----- > > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java > c5a63bc7a172c44cc11d6bdd6b2eb4bc3df35cb7 > > Diff: https://reviews.apache.org/r/11118/diff/ > > > Testing > ------- > > Used in production at airbnb. > > > Thanks, > > Brenden Matthews > >
