[
https://issues.apache.org/jira/browse/TEZ-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15876954#comment-15876954
]
Jonathan Eagles commented on TEZ-3633:
--------------------------------------
[~rajesh.balamohan], added a patch that I hope will clarify. To the best of my
knowledge, MR shuffle handler keep-alive hasn't been functional in since commit
in 2.4. Shuffle handler (both MR and tez) currently closes the connection
immediately after fulfilling a fetch request. The fetcher in tez (haven't
verified with MR) will try to reuse the connection and will send another fetch
request but it will reset and reconnect.
The issue I'm wondering is what impact of enabling keep-alive=true will be. If
you have a 1000 node cluster, you could end up with 1000x1000 open connections
(maybe even more since by default there are 30 fetcher threads that won't reuse
connections).
> Implement keep-alive timeout in tez shuffle handler
> ---------------------------------------------------
>
> Key: TEZ-3633
> URL: https://issues.apache.org/jira/browse/TEZ-3633
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Jonathan Eagles
> Assignee: Jonathan Eagles
> Attachments: TEZ-3633.1.patch
>
>
> MAPREDUCE-5787 which added keep-alive to mapreduce shuffle handler was not
> fully functional as despite advertising keep-alive option and adding the
> header to the response, all connections were closed immediately after write.
> This reduced the performance of certain fetches as now time is spent
> requesting a second get to the same serve, only for that server to reset the
> connection forcing the client to reestablish the connection on another port.
> The details of this is hidden behind HttpURLConnection and doesn't show in
> any log file at default logging level. However TCP sniffing does show errant
> behavior.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
