This can be easily done through HoD since it requires separate
configuration files for each tasktracker i.e node. As of now I dont think this can
be done in HADOOP. Anyways never seen such high values for max tasks. :)
Amar.
On Wed, 27 Feb 2008, Zhang,
Guibin wrote:
Hi,all
I find that when the sub-nodes' hardware configurations are
different, some nodes are strong(more cpus and more memory), others are
weak (leas cpus and less memory), when I run the job, the task are
almost evenly distributed to all the sub-nodes. This makes the weak
nodes pretty slow and a lot of tasks on the weak nodes are killed. This
may lead the whole job processing becoming slow, I am sure, because a
lot of tasks(more than 10 tasks) are processed twice.
Question: How can I configure the hadoop to distribute less
tasks to weak nodes and distribute more tasks to strong nodes?
I configure the strong nodes with
"mapred.tasktracker.map.tasks.maximum=75",
"mapred.map.tasks=60",
"mapred.tasktracker.reduce.tasks.maximum=18",
"mapred.reduce.tasks=15"
and the weak nodes with
"mapred.tasktracker.map.tasks.maximum=60",
"mapred.map.tasks=45",
"mapred.tasktracker.reduce.tasks.maximum=15",
"mapred.reduce.tasks=12"
I have 4 nodes totally. One for name node and job tracker, the others
are for sub-nodes.
Thanks.
Guibin zhang
