[
https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12592816#action_12592816
]
Devaraj Das commented on HADOOP-3297:
-------------------------------------
An interesting observation regarding using the ramfs. I guess i should raise a
separate jira but let me put it here anyway -
I had a job (loadgen from hadoop-test) consisting of 2500 maps and 1 reducer.
The ramfs size was 300MB and io.sort.factor was 100. The cluster had 20 nodes.
Each map generated 5 MB of data. The amount of time it took to complete the job
was 45 minutes (with the above changes). The number of files that missed the
ramfs and ended up on disk was ~2000.
I ran the same job (with exactly the same config) with the reducer throttled -
if a ramfs merge is on, it would wait for that to complete before fetching
anything new. This basically results in all files ending up in the ramfs. The
job ran in 30 minutes.
So although I didn't notice any significant performance gain for this job with
the shuffle protocol changes as proposed in my last comment but in general it
looks like this is going to be true - for a given job, if we have a faster
shuffle, more files get created on the disk, and depending on the number/size
of map outputs for the job, this might adversely affects the final merge,
thereby affecting the overall runtime of the job.
I will see if the above behavior can be modelled.
> The way in which ReduceTask/TaskTracker gets completion events during shuffle
> can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
>
> Certain things like poll frequency, number of events fetched in one go, etc.
> can probably be improved to improve the shuffle performance. This would
> affect the task-->tasktracker and the tasktracker-->jobtracker shuffle
> related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
