[
https://issues.apache.org/jira/browse/MAPREDUCE-5786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13925781#comment-13925781
]
Jason Lowe commented on MAPREDUCE-5786:
---------------------------------------
Couldn't this lead to file descriptor exhaustion on the NM side? Thinking of
cases where we're running many large jobs on the cluster with thousands of
reducers each and:
- reducer decides it wants to shuffle to memory but there isn't enough memory
yet so it waits for the memory merge to complete (which could take a while)
- reducer is waiting for a subsequent map to complete (which could take many
minutes or hours)
Seems like we could have a situation where reducers start piling up on the
shuffle handler and camping out.
> Support Keep-Alive connections in ShuffleHandler
> ------------------------------------------------
>
> Key: MAPREDUCE-5786
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5786
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.4.0
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Labels: shuffle
>
> Currently ShuffleHandler supports fetching map-outputs in batches from same
> host. But there are scenarios wherein, fetchers pull data aggressively (i.e
> start pulling the data as & when they are available). In this case, the
> number of mapIds that are pulled from same host remains at 1. This causes
> lots of connections to be established.
> Number of connections can be reduced a lot if ShuffleHandler supports
> Keep-Alive.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
