[
https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12594808#action_12594808
]
Mahadev konar commented on HADOOP-3297:
---------------------------------------
the patch looks good... the only concern I have is that if we want to check and
see that it does not degrade perofrmace and lead to more problems at the
jobtracker or otherwise.
We could try running sort with this patch or some map reduce job where we have
a huge number of mappers say 100,000 and 500 reducers or something.
This is just to verify that the task tracker being agressive in fetching the
mapoutputs does not degrade performance. The maps are short lived, so its not
hard to imagine a situation that all the reduces start bombarding the
jobtracker with requests for maps at the same time asking for more. We should
check to see if the jobtracker can handle the load and the performance in such
a situation does not degrade.
> The way in which ReduceTask/TaskTracker gets completion events during shuffle
> can be improved
> ---------------------------------------------------------------------------------------------
>
> Key: HADOOP-3297
> URL: https://issues.apache.org/jira/browse/HADOOP-3297
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Devaraj Das
> Assignee: Devaraj Das
> Fix For: 0.18.0
>
> Attachments: 3297.patch, 3297.patch
>
>
> Certain things like poll frequency, number of events fetched in one go, etc.
> can probably be improved to improve the shuffle performance. This would
> affect the task-->tasktracker and the tasktracker-->jobtracker shuffle
> related RPCs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
