Thanks a lot for the information.

I used option '-file' provided by hadoop-streamiing to upload read-only
files for my map/reduce job, and read them as a local file in my perl
script. I am wondering if it is similar to what distributed cache does
performance wise? Thanks.

tp.

On Tue, Aug 12, 2008 at 5:07 PM, Arun C Murthy <[EMAIL PROTECTED]> wrote:

>
> On Aug 12, 2008, at 11:21 AM, charles du wrote:
>
>  Hi:
>>
>> Does hadoop always start a new process for each map task?
>>
>>
> Yes. http://issues.apache.org/jira/browse/HADOOP-249 is open to optimize
> that.
>
> Till HADOOP-249 is fixed, you could try and launch fewer, fatter maps by
> doing more work on each to get around your 'long-initialization'. Please
> take a look at InputFormat.getSplits on how to do that.:
> http://hadoop.apache.org/core/docs/r0.17.1/api/org/apache/hadoop/mapred/InputFormat.html
>
> Also, you could consider using the DistributedCache for distributing
> read-only data if necessary for your maps/reduces:
>
> http://hadoop.apache.org/core/docs/current/mapred_tutorial.html#DistributedCache
>
> Arun
>
>
>  I have a 20s-machine cluster and configured each task tracker to run 2
>> concurrent tasks at most. So the cluster can run 40 task in parallel. If I
>> start a hadoop job with 1000 tasks, will hadoop  create 1000 map processes
>> during the execution of the job, or it will start 40 processes at the
>> beginning, and process 1000 tasks one  by one (of course, at any
>> particular
>> time, only 40 running)?
>>
>> My map tasks have a long initialization time before they start processing
>> data files. So it will be ideal if map processes could be reused among
>> different tasks, instead of creating a new process for each of them. Is
>> there a way to do it?
>>
>> Thanks.
>>
>> --
>> tp
>>
>
>


-- 
tp

Reply via email to