[
https://issues.apache.org/jira/browse/MAPREDUCE-5787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-5787:
-----------------------------------------------
Fix Version/s: (was: 2.4.0)
Target Version/s: 2.4.0
Status: Open (was: Patch Available)
Looked at the patch, some comments:
Can we also change the MapReduce fetcher to use keep-alive depending on whether
it is enabled or not?
Suggestion for Configuration renames
- mapreduce.shuffle.enable.keep.alive ->
mapreduce.shuffle.connection-keep-alive.enable
- mapreduce.shuffle.enable.keep.alive.timeout ->
mapreduce.shuffle.connection-keep-alive.timeout
- Add both to the mapred-default.xml
ShuffleHandler.java
- SHUFFLE_KEEP_ALIVE -> SHUFFLE_CONNECTION_KEEP_ALIVE_ENABLED
- DEFAULT_SHUFFLE_KEEP_ALIVE -> DEFAULT_SHUFFLE_CONNECTION_KEEP_ALIVE_ENABLED
- SHUFFLE_KEEP_ALIVE_TIME_OUT -> SHUFFLE_CONNECTION_KEEP_ALIVE_TIME_OUT
- DEFAULT_SHUFFLE_KEEP_ALIVE_TIME_OUT ->
DEFAULT_SHUFFLE_CONNECTION_KEEP_ALIVE_TIME_OUT
- Variable keepAlive -> connectionKeepAliveEnabled
- Similarly keepAliveTimeOut -> connectionKeepAliveTimeOut
- LOG KeepAliveParam along with other things like jobId, mapId etc.
- populateHeaders
-- We are already parsing jobID, ApplicationId etc as part of
sendMapOutput. We should avoid doing the string parsing multiple times.
-- Is setting CONTENT_LENGTH important? Even so, for doing it, we are
reading the index-record two times - once here and once while sending the
output. This will have performance impact.
- Instead of re-defining new constants like CONNECTION_HEADER in
ShuffleHandler, can you use the standard constants in java (HttpHeaders)?
Finally, can you reuse code between the two tests? There seems to be quite a
bit of duplication.
> Modify ShuffleHandler to support Keep-Alive
> -------------------------------------------
>
> Key: MAPREDUCE-5787
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5787
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: nodemanager
> Affects Versions: 2.4.0
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Priority: Critical
> Labels: ShuffleKeepalive
> Attachments: MAPREDUCE-5787-2.4.0-v2.patch,
> MAPREDUCE-5787-2.4.0-v3.patch, MAPREDUCE-5787-2.4.0-v4.patch,
> MAPREDUCE-5787-2.4.0.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
