Milan Potocnik commented on YARN-1994:

Hi [~cwelch], [~arpitagarwal],

I think some clarification is needed here. Initial reason we wanted to 
introduce _BIND_HOST options was to provide deterministic behaviors when 
clients try to connect to a service endpoint which is listening on all 
interfaces ( In short, _BIND_HOST is what services use to bind, 
_ADDRESS is what clients should use to connect. This way, everything is 

In Multi NIC environments, with default implementation, calls to 
conf.updateConnectAddress for address would eventually call 
InetSocketAddress.getHostName(). In MultiNIC environments, this can introduce 
non-deterministic behavior. Imagine you have DNS entries for each of the 
network interfaces and although you bind your service endpoint to all of them, 
you want users to use a specific one (for instance, InfiniBand for better 
performance). InetSocketAddress.getHostName() will return just the machine's 
hostname which will usually resolve to some random network interface of the 
service when the client resolves it. Although service binds to, some 
interfaces might be disabled by firewall.

This is why besides RPCUtil.getSocketAddress, we also need 
RPCUtil.updateConnectAddr to explicitly specify connect address which clients 
should use, i.e. a DNS entry pointing to a specific interface.

There are also two cases in the code, where current implementation does not 
work in MultiNIC environments which we fixed:
- MRClientService & TaskAttemptListenerImpl where we had to propagate NM 
hostname through context,
- which is set in ContainerManagerImpl via NodeId from 
YarnConfiguration.NM_ADDRESS (the logic Arpit mentioned in the comment)

Please have a look the the patch version 5, for easier understanding.

Hope this clarifies the initial idea.


> Expose YARN/MR endpoints on multiple interfaces
> -----------------------------------------------
>                 Key: YARN-1994
>                 URL: https://issues.apache.org/jira/browse/YARN-1994
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager, resourcemanager, webapp
>    Affects Versions: 2.4.0
>            Reporter: Arpit Agarwal
>            Assignee: Craig Welch
>         Attachments: YARN-1994.0.patch, YARN-1994.1.patch, 
> YARN-1994.11.patch, YARN-1994.11.patch, YARN-1994.12.patch, 
> YARN-1994.2.patch, YARN-1994.3.patch, YARN-1994.4.patch, YARN-1994.5.patch, 
> YARN-1994.6.patch, YARN-1994.7.patch
> YARN and MapReduce daemons currently do not support specifying a wildcard 
> address for the server endpoints. This prevents the endpoints from being 
> accessible from all interfaces on a multihomed machine.
> Note that if we do specify INADDR_ANY for any of the options, it will break 
> clients as they will attempt to connect to We need a solution that 
> allows specifying a hostname or IP-address for clients while requesting 
> wildcard bind for the servers.
> (List of endpoints is in a comment below)

This message was sent by Atlassian JIRA

Reply via email to