Gianlorenzo Thione wrote:
Thanks for the answer. So far I am still trying to understand how each tasktracker gets multiple map or reduce tasks to be executed simultaneously. I have run a simple job with 53 map tasks on 5 nodes, and at all times each node was executing a single task. Each cluster node is a 4 core machine, so theoretically this was a 16-node cluster and I feel that the resources were actually underutilized. Am I missing something? Is there a parameter for a minimum number of tasks to be executed in parallel (I found a parameter for setting a maximum [which I set to 4])? If I run 4 TaskTrackers per node then each node gets a map task at the same time and execution seems overall much faster.
The task tracker can currently get starved for work when tasks complete too quickly. This is a bug that will hopefully be fixed soon. The problem is that the task tracker only polls for a new task once per heartbeat (10 seconds). Instead it should poll for new tasks as soon as tasks complete. As a short-term workaround you can decrease the heartbeat interval to one second in MRConstants.java. With smaller clusters (< 100 machines) that should not cause any problems.
Doug
