[ 
https://issues.apache.org/jira/browse/YARN-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14634416#comment-14634416
 ] 

zhihai xu commented on YARN-3944:
---------------------------------

Hi [~l201514], thanks for the information. I am not sure how you get 1 hour 
delay.
The following is my calculation based on the latest code which has the fix 
YARN-3554. Please correct me if I am wrong.
The IPC retry for ConnectException is handled at 
{{Client#handleConnectionFailure}}, which uses {{connectionRetryPolicy}}
{code}
        final int max = conf.getInt(
            CommonConfigurationKeysPublic.IPC_CLIENT_CONNECT_MAX_RETRIES_KEY,
            
CommonConfigurationKeysPublic.IPC_CLIENT_CONNECT_MAX_RETRIES_DEFAULT);
        final int retryInterval = conf.getInt(
            CommonConfigurationKeysPublic.IPC_CLIENT_CONNECT_RETRY_INTERVAL_KEY,
            CommonConfigurationKeysPublic
                .IPC_CLIENT_CONNECT_RETRY_INTERVAL_DEFAULT);
        connectionRetryPolicy = 
RetryPolicies.retryUpToMaximumCountWithFixedSleep(
            max, retryInterval, TimeUnit.MILLISECONDS);
  public static final int     IPC_CLIENT_CONNECT_MAX_RETRIES_DEFAULT = 10;
  public static final int     IPC_CLIENT_CONNECT_RETRY_INTERVAL_DEFAULT = 1000;
{code}
So based on the default value, it will retry 10 times with interval 1 seconds, 
the total is 10 seconds at IPC layer.
The second layer retry is at NMProxy:
{code}
        createRetryPolicy(conf,
          YarnConfiguration.CLIENT_NM_CONNECT_MAX_WAIT_MS,
          YarnConfiguration.DEFAULT_CLIENT_NM_CONNECT_MAX_WAIT_MS,
          YarnConfiguration.CLIENT_NM_CONNECT_RETRY_INTERVAL_MS,
          YarnConfiguration.DEFAULT_CLIENT_NM_CONNECT_RETRY_INTERVAL_MS);
  public static final long DEFAULT_CLIENT_NM_CONNECT_MAX_WAIT_MS =
      3 * 60 * 1000;
  public static final long DEFAULT_CLIENT_NM_CONNECT_RETRY_INTERVAL_MS
      = 10 * 1000;
{code}
It will retry 180,000/10,000 = 18 times at NMProxy.
So the total delay will be 18*(10 + 10) = 360 seconds, which is 6 minutes, 
one-tenth of the delay you got. 


> Connection refused to nodemanagers are retried at multiple levels
> -----------------------------------------------------------------
>
>                 Key: YARN-3944
>                 URL: https://issues.apache.org/jira/browse/YARN-3944
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.6.0
>            Reporter: Siqi Li
>            Assignee: Siqi Li
>            Priority: Blocker
>         Attachments: YARN-3944.v1.patch
>
>
> This is related to YARN-3238. When NM is down, ipc client will get 
> ConnectException.
> Caused by: java.net.ConnectException: Connection refused
>       at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>       at 
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
>       at 
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
>       at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
>       at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
>       at 
> org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:607)
>       at 
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:705)
>       at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:368)
>       at org.apache.hadoop.ipc.Client.getConnection(Client.java:1521)
>       at org.apache.hadoop.ipc.Client.call(Client.java:1438)
> However, retry happens at two layers(ipc retry 40 times and serverProxy 
> retrying 91 times), this could end up with ~1 hour retry interval.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to