Milan Potocnik commented on YARN-1994:

I agree it is tricky to hunt down all service endpoints and make sure they 
support proper hostname. Also, when new endpoints are added, they have to be 
aware of this convention. I guess some logic could be added to RPC.Server in 
the long run.

Few notes, though.

We have experienced issues on Windows, when client and service are on the same 
machine. It turns out that in certain situations when client is resolving the 
connect address (which has actual ('main') hostname), it does not go to DNS 
Server, but rather performs it locally and in some cases might return an 
unwanted IP address (since the machine itself is aware of all of its network 
interfaces). If special hostname is used ('hostname-IB' in my earlier example), 
resolve will go to DNS Server and everything will work.

The proposed approach is not unprecedented, HDFS has similar functionality, you 
can specify custom hostnames for components (so that 
InetAddress.getLocalHost().getHostName() is never called). Please have a look 
 - fs.default.fs - you can specify custom hostname that namenode will use
 - dfs.namenode.rpc-bind-host - can be set to in that case
 - dfs.datanode.hostname - can be used to specify custom datanode hostname
 - dfs.datanode.address, dfs.datanode.ipc.address, etc... - can be set to in that case
 - dfs.client.use.datanode.hostname and dfs.datanode.use.datanode.hostname also 
need to be set

So I think it would make sense to have similar functionality available in 
YARN/MR as well.


> Expose YARN/MR endpoints on multiple interfaces
> -----------------------------------------------
>                 Key: YARN-1994
>                 URL: https://issues.apache.org/jira/browse/YARN-1994
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager, resourcemanager, webapp
>    Affects Versions: 2.4.0
>            Reporter: Arpit Agarwal
>            Assignee: Craig Welch
>         Attachments: YARN-1994.0.patch, YARN-1994.1.patch, 
> YARN-1994.11.patch, YARN-1994.11.patch, YARN-1994.12.patch, 
> YARN-1994.2.patch, YARN-1994.3.patch, YARN-1994.4.patch, YARN-1994.5.patch, 
> YARN-1994.6.patch, YARN-1994.7.patch
> YARN and MapReduce daemons currently do not support specifying a wildcard 
> address for the server endpoints. This prevents the endpoints from being 
> accessible from all interfaces on a multihomed machine.
> Note that if we do specify INADDR_ANY for any of the options, it will break 
> clients as they will attempt to connect to We need a solution that 
> allows specifying a hostname or IP-address for clients while requesting 
> wildcard bind for the servers.
> (List of endpoints is in a comment below)

This message was sent by Atlassian JIRA

Reply via email to