[
https://issues.apache.org/jira/browse/YARN-8675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16587873#comment-16587873
]
Eric Yang commented on YARN-8675:
---------------------------------
Docker only prevents --hostname and --net=host flag to be combined in older
version of docker (1.13) . See [Docker
issue|https://github.com/moby/moby/pull/29144]. The following table illustrate
the possible combination when we should and should not support customized
hostname:
| YARN Registry DNS | YARN Service | Custom AM | Network Type | Custom Hostname
|
| Enabled | Yes | No | Host | Yes |
| Enabled | No | Yes | Host | No |
| Disabled | No | Yes | Host | No |
| Disabled | No | Yes | Bridge | N/A |
Today registryDNS and YARN service are coupled together, only YARN service
knows how to populate hostname information to registryDNS. If custom AM
creates it's own logic to generate custom hostname, it must have some way to
populate RegistryDNS to translate correct name. Without using YARN service,
there is no programmable API to customize hostname. This is the reason that
Spark on YARN cluster on docker mode fails with buzzard hostname composition.
For resolving spark issue, it is entirely possible to run spark using YARN
service API without making any code changes to spark standalone mode. To fix
this issue properly, it would be best to provide hint to docker runtime to
decide if custom hostname can be supported. Custom hostname is a new concept
that doesn't exist prior to docker container. Therefore, only new application
using YARN service should be supported in my view.
> Setting hostname of docker container breaks with "host" networking mode for
> Apps which do not run as a YARN service
> -------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-8675
> URL: https://issues.apache.org/jira/browse/YARN-8675
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Yesha Vora
> Assignee: Suma Shivaprasad
> Priority: Major
>
> Applications like the Spark AM currently do not run as a YARN service and
> setting hostname breaks driver/executor communication if docker version
> >=1.13.1 , especially with wire-encryption turned on.
> YARN-8027 sets the hostname if YARN DNS is enabled. But the cluster could
> have a mix of YARN service/native Applications.
> The proposal is to not set the hostname when "host" networking mode is
> enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
