[
https://issues.apache.org/jira/browse/YARN-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14079104#comment-14079104
]
Milan Potocnik commented on YARN-1994:
--------------------------------------
I agree it is tricky to hunt down all service endpoints and make sure they
support proper hostname. Also, when new endpoints are added, they have to be
aware of this convention. I guess some logic could be added to RPC.Server in
the long run.
Few notes, though.
We have experienced issues on Windows, when client and service are on the same
machine. It turns out that in certain situations when client is resolving the
connect address (which has actual ('main') hostname), it does not go to DNS
Server, but rather performs it locally and in some cases might return an
unwanted IP address (since the machine itself is aware of all of its network
interfaces). If special hostname is used ('hostname-IB' in my earlier example),
resolve will go to DNS Server and everything will work.
The proposed approach is not unprecedented, HDFS has similar functionality, you
can specify custom hostnames for components (so that
InetAddress.getLocalHost().getHostName() is never called). Please have a look
at:
- fs.default.fs - you can specify custom hostname that namenode will use
- dfs.namenode.rpc-bind-host - can be set to 0.0.0.0 in that case
- dfs.datanode.hostname - can be used to specify custom datanode hostname
- dfs.datanode.address, dfs.datanode.ipc.address, etc... - can be set to
0.0.0.0 in that case
- dfs.client.use.datanode.hostname and dfs.datanode.use.datanode.hostname also
need to be set
So I think it would make sense to have similar functionality available in
YARN/MR as well.
Thanks,
Milan
> Expose YARN/MR endpoints on multiple interfaces
> -----------------------------------------------
>
> Key: YARN-1994
> URL: https://issues.apache.org/jira/browse/YARN-1994
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, resourcemanager, webapp
> Affects Versions: 2.4.0
> Reporter: Arpit Agarwal
> Assignee: Craig Welch
> Attachments: YARN-1994.0.patch, YARN-1994.1.patch,
> YARN-1994.11.patch, YARN-1994.11.patch, YARN-1994.12.patch,
> YARN-1994.2.patch, YARN-1994.3.patch, YARN-1994.4.patch, YARN-1994.5.patch,
> YARN-1994.6.patch, YARN-1994.7.patch
>
>
> YARN and MapReduce daemons currently do not support specifying a wildcard
> address for the server endpoints. This prevents the endpoints from being
> accessible from all interfaces on a multihomed machine.
> Note that if we do specify INADDR_ANY for any of the options, it will break
> clients as they will attempt to connect to 0.0.0.0. We need a solution that
> allows specifying a hostname or IP-address for clients while requesting
> wildcard bind for the servers.
> (List of endpoints is in a comment below)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
