[
https://issues.apache.org/jira/browse/YARN-10857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Suh updated YARN-10857:
-----------------------------
Description:
We have noticed that when the YarnClient is initialized and used, it is not
very resilient when dns or /etc/hosts is modified in the following scenario:
Take for instance the following (and reproducable) sequence of events that can
occur on a service that instantiates and uses YarnClient.
- Yarn has rm HA enabled (*yarn.resourcemanager.ha.enabled* is *true*) and
there are two rms (rm1 and rm2).
- *yarn.client.failover-proxy-provider* is set to
*org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider*
1) rm2 is currently the active rm
2) /etc/hosts (or dns) is missing host information for rm2
3) A service is started and it initializes the YarnClient at startup.
4) At some point in time after YarnClient is done initializing, /etc/hosts
is updated and contains host information for rm2
5) Yarn is queried, for instance calling *yarnclient.getApplications()*
6) All YarnClient attempts to communicate with rm2 fail with
UnknownHostExceptions, even though /etc/hosts now contains host information for
it.
was:
We have noticed that when the YarnClient is initialized and used, it is not
very resilient when dns or /etc/hosts is modified in the following scenario:
Take for instance the following (and reproducable) sequence of events that can
occur on a service that instantiates and uses YarnClient.
- Yarn has rm HA enabled (*yarn.resourcemanager.ha.enabled* is *true*) and
there are two rms (rm1 and rm2).
- *yarn.client.failover-proxy-provider* is set to
*org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider*
1) rm2 is currently the active rm
2) /etc/hosts (or dns) is missing host information for rm2
3) A service is started and it initializes the YarnClient at startup.
4) At some point in time after YarnClient is done initializing, /etc/hosts
is updated and contains host information for rm2
5) Yarn is queried using YarnClient, for instance calling
`.getApplications()`
6) All YarnClient attempts to communicate with rm2 fail with
UnknownHostExceptions, even though /etc/hosts now contains host information for
it.
> YarnClient Caching Addresses
> ----------------------------
>
> Key: YARN-10857
> URL: https://issues.apache.org/jira/browse/YARN-10857
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: client, yarn
> Reporter: Steve Suh
> Priority: Minor
>
> We have noticed that when the YarnClient is initialized and used, it is not
> very resilient when dns or /etc/hosts is modified in the following scenario:
> Take for instance the following (and reproducable) sequence of events that
> can occur on a service that instantiates and uses YarnClient.
> - Yarn has rm HA enabled (*yarn.resourcemanager.ha.enabled* is *true*) and
> there are two rms (rm1 and rm2).
> - *yarn.client.failover-proxy-provider* is set to
> *org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider*
> 1) rm2 is currently the active rm
> 2) /etc/hosts (or dns) is missing host information for rm2
> 3) A service is started and it initializes the YarnClient at startup.
> 4) At some point in time after YarnClient is done initializing, /etc/hosts
> is updated and contains host information for rm2
> 5) Yarn is queried, for instance calling *yarnclient.getApplications()*
> 6) All YarnClient attempts to communicate with rm2 fail with
> UnknownHostExceptions, even though /etc/hosts now contains host information
> for it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
