I understand your question now, Doug; thanks for clarifying. However, I don't think I can give you a great answer. I'll give it a shot, though: It does seem like having a single task configuration in theory would improve utilization, but it might also make things worse. For example, generally speaking, reducers take longer to execute. This means that it would be possible for some nodes to only perform reduce tasks for a given time period in a setup where each node had a dynamic amount of mappers and reducers. If a node was running all reducers, then that node would have lots of output data being written to it, hence not evenly distributing data well. Perhaps one could argue that over time data would still be distributed evenly, though.
That's the best I can do I think. Can others chime in? Alex On Tue, Oct 28, 2008 at 1:41 PM, Doug Balog <[EMAIL PROTECTED]> wrote: > Hi Alex, I'm sorry, I think you misunderstood my question. Let me explain > some more. > > I have a hadoop cluster of dual quad core machines. > I'm using hadoop-0.18.1 with Matei's fairscheduler patch > https://issues.apache.org/jira/browse/HADOOP-3746 running in FIFO mode. > I have about 5 different jobs running in a pipeline. The number of > map/reduce tasks per job > varies based on the input data. > I assign the various jobs different priorities, and Matei's FIFO scheduler > does almost exactly what I want. > (The default scheduler did a horrible job with our workload, because it > prefers map tasks.) > > I'm trying to tune the tasks per node to fully utilize my cluster, my goal > < 10% idle. > I'm pretty sure my jobs are cpu bound. I can control the number of tasks > per node by > setting mapred.tasktracker.map.tasks.maximum > and mapred.tasktracker.reduce.tasks.maximum in hadoop-site.xml. > > But I don't have a fixed number of maps and reduces that I run, so saying > 5+3 tends to leave > my nodes more idle than I want. I just want to say run 8 tasks per node, I > don't care what the mix between > map and reduce tasks per node. > > I've been wondering why there are separate task limits for map and reduce. >>> Why not a single generic task limit per node ? >>> >> >> > > > The only reason I can think of for having > separate map and reduce task limits, is the default scheduler. > It wants to schedule all map tasks first, so you really need to limit the > number of > them so that reduces have a chance to run. > > Thanks for any insight, > Doug > > > On Oct 27, 2008, at 6:26 PM, Alex Loddengaard wrote: > > In most jobs, map and reduce tasks are significantly differently, and >> their >> runtimes vary as well. The number of reducers also determines how many >> output files you have. So in the case when you would want one output >> file, >> having a single generic task limit would mean that you'd also have one >> mapper. This would be quite a limiting setup. >> Hope this helps. >> >> Alex >> >> On Mon, Oct 27, 2008 at 1:31 PM, Doug Balog <[EMAIL PROTECTED]> wrote: >> >> Hi, >>> I've been wondering why there are separate task limits for map and >>> reduce. >>> Why not a single generic task limit per node ? >>> >>> Thanks for any insight, >>> >>> Doug >>> >>> >>> >>> >
