On Aug 12, 2008, at 11:21 AM, charles du wrote:
Hi:
Does hadoop always start a new process for each map task?
Yes. http://issues.apache.org/jira/browse/HADOOP-249 is open to
optimize that.
Till HADOOP-249 is fixed, you could try and launch fewer, fatter maps
by doing more work on each to get around your 'long-initialization'.
Please take a look at InputFormat.getSplits on how to do that.: http://hadoop.apache.org/core/docs/r0.17.1/api/org/apache/hadoop/mapred/InputFormat.html
Also, you could consider using the DistributedCache for distributing
read-only data if necessary for your maps/reduces:
http://hadoop.apache.org/core/docs/current/mapred_tutorial.html#DistributedCache
Arun
I have a 20s-machine cluster and configured each task tracker to run 2
concurrent tasks at most. So the cluster can run 40 task in
parallel. If I
start a hadoop job with 1000 tasks, will hadoop create 1000 map
processes
during the execution of the job, or it will start 40 processes at the
beginning, and process 1000 tasks one by one (of course, at any
particular
time, only 40 running)?
My map tasks have a long initialization time before they start
processing
data files. So it will be ideal if map processes could be reused among
different tasks, instead of creating a new process for each of them.
Is
there a way to do it?
Thanks.
--
tp