[
https://issues.apache.org/jira/browse/MAPREDUCE-5786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13926423#comment-13926423
]
Rajesh Balamohan commented on MAPREDUCE-5786:
---------------------------------------------
Thanks for comments Jason. We need to have
"mapreduce.shuffle.enable.keep.alive" to enable keep-alive in the
ShuffleHandler and "mapreduce.shuffle.enable.keep.alive.timeout" to determine
the time-out value for the persistent connection. E.g, "Keep-Alive:
timeout=60" header specifies that the connection will be kept alive for 60
seconds after which the connection will be closed. This will allow us to tune
persistent connection duration on large clusters with different job patterns.
> Support Keep-Alive connections in ShuffleHandler
> ------------------------------------------------
>
> Key: MAPREDUCE-5786
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5786
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.4.0
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Labels: shuffle
>
> Currently ShuffleHandler supports fetching map-outputs in batches from same
> host. But there are scenarios wherein, fetchers pull data aggressively (i.e
> start pulling the data as & when they are available). In this case, the
> number of mapIds that are pulled from same host remains at 1. This causes
> lots of connections to be established.
> Number of connections can be reduced a lot if ShuffleHandler supports
> Keep-Alive.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
