[
https://issues.apache.org/jira/browse/YARN-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14634513#comment-14634513
]
Siqi Li commented on YARN-3944:
-------------------------------
[~zxu] Thanks for your quick response, I think the reason we encounter this
problem is that we were setting IPC_CLIENT_CONNECT_MAX_RETRIES to 40 times
prior to 2.6, and when we merge 2.6 the value of
DEFAULT_CLIENT_NM_CONNECT_MAX_WAIT_MS is 15 * 60 * 1000. Therefore, the total
delay could end up with more than 1 hour.
So, I think the best solution could be backporting YARN-3554 instead, and maybe
dail down the number of IPC_CLIENT_CONNECT_MAX_RETRIES
> Connection refused to nodemanagers are retried at multiple levels
> -----------------------------------------------------------------
>
> Key: YARN-3944
> URL: https://issues.apache.org/jira/browse/YARN-3944
> Project: Hadoop YARN
> Issue Type: Bug
> Affects Versions: 2.6.0
> Reporter: Siqi Li
> Assignee: Siqi Li
> Priority: Blocker
> Attachments: YARN-3944.v1.patch
>
>
> This is related to YARN-3238. When NM is down, ipc client will get
> ConnectException.
> Caused by: java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
> at
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)
> at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
> at org.apache.hadoop.ipc.Client.call(Client.java:1438)
> However, retry happens at two layers(ipc retry 40 times and serverProxy
> retrying 91 times), this could end up with ~1 hour retry interval.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
