[
https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-5787:
-----------------------------------------------
Attachment: MAPREDUCE-5787-2.4.0-v6.patch
MAPREDUCE-5787-2.4.0-v5-v6-diff.patch
Your patch has some non-standard formatting and I had time to kill.
Here are two files attached - one which shows my additional changes
(formatting, refactoring and javadoc) and one with all the changes.
I think there is one more big issue to take care of. Because we are avoiding
hitting the disk for fetching the ShuffleHader multiple times, you are keeping
them in memory. There is a small bump in the memory foot-print after this
patch. Asssuming 500 reducers fetching outputs of 500 maps all at the same
time, this translates to 250KShuffleHeaders and assuming 40bytes per header,
the ShuffleHandler needs 10MB extra memory now. Which isn't significant, our
index-cache size itself is 10MB by default, but the problem is this increased
memory-foot-print is unbounded.
> Modify ShuffleHandler to support Keep-Alive
> -------------------------------------------
>
> Key: MAPREDUCE-5787
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: nodemanager
> Affects Versions: 2.4.0
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Priority: Critical
> Labels: ShuffleKeepalive
> Attachments: MAPREDUCE-5787-2.4.0-v2.patch,
> MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v4.patch,
> MAPREDUCE-5787-2.4.0-v5-v6-diff.patch, MAPREDUCE-5787-2.4.0-v5.patch,
> MAPREDUCE-5787-2.4.0-v6.patch, MAPREDUCE-5787-2.4.0.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
