Xie YiFan created YARN-9811:
-------------------------------
Summary: FederationInterceptor fails to recover in Kerberos
environment
Key: YARN-9811
URL: https://issues.apache.org/jira/browse/YARN-9811
Project: Hadoop YARN
Issue Type: Bug
Components: amrmproxy
Reporter: Xie YiFan
Assignee: Xie YiFan
*scenario*:
Start up cluster in Kerberos environment with enable recover & AMRMProxy in
NM. Submit one application to cluster, and restart NM which has master
container. The NM will block in FederationInterceptor recover.
*LOG*
{code:java}
INFO org.apache.hadoop.yarn.server.nodemanager.amrmproxy.FederationInterceptor:
Recovering data for FederationInterceptor
INFO org.apache.hadoop.yarn.server.nodemanager.amrmproxy.FederationInterceptor:
Found 0 existing UAMs for application application_1561534175896_4102 in
NMStateStore
INFO org.apache.hadoop.yarn.server.utils.AMRMClientUtils: Creating RMProxy to
RM online-bx for protocol ApplicationClientProtocol for user recommend
(auth:SIMPLE)
INFO
org.apache.hadoop.yarn.server.federation.failover.FederationRMFailoverProxyProvider:
Initialized Federation proxy for user: recommend
INFO
org.apache.hadoop.yarn.server.federation.failover.FederationRMFailoverProxyProvider:
Failing over to the ResourceManager for SubClusterId: online-bx
INFO
org.apache.hadoop.yarn.server.federation.failover.FederationRMFailoverProxyProvider:
Connecting to /10.88.86.142:8032 subClusterId online-bx with protocol
ApplicationClientProtocol as user recommend (auth:SIMPLE)
WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to
the server : org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN, KERBEROS]
INFO
org.apache.hadoop.yarn.server.federation.failover.FederationRMFailoverProxyProvider:
Failing over to the ResourceManager for SubClusterId: online-bx
INFO org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade:
Flushing subClusters from cache and rehydrating from store, most likely on
account of RM failover.
INFO
org.apache.hadoop.yarn.server.federation.failover.FederationRMFailoverProxyProvider:
Connecting to /10.88.86.142:8032 subClusterId online-bx with protocol
ApplicationClientProtocol as user recommend (auth:SIMPLE)
WARN org.apache.hadoop.ipc.Client: Exception encountered while connecting to
the server : org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN, KERBEROS]
INFO org.apache.hadoop.io.retry.RetryInvocationHandler: java.io.IOException:
DestHost:destPort hadoop1684.bx.momo.com:8032 , LocalHost:localPort
hadoop999.bx.momo.com/10.88.64.186:0. Failed on local exception:
java.io.IOException: org.apache.hadoop.security.AccessControlException: Client
cannot authenticate via:[TOKEN, KERBEROS], while invoking
ApplicationClientProtocolPBClientImpl.getContainers over online-bx after 1
failover attempts. Trying to failover after sleeping for 3244ms.{code}
*Analysis*
rmclient.getContainers is called. But AuthMethod of appSubmitter is SIMPLE.We
should use createProxyUser instead of createRemoteUser in Security.
{code:java}
UserGroupInformation appSubmitter = UserGroupInformation
.createRemoteUser(getApplicationContext().getUser());
ApplicationClientProtocol rmClient =
createHomeRMProxy(getApplicationContext(),
ApplicationClientProtocol.class, appSubmitter);
GetContainersResponse response = rmClient
.getContainers(GetContainersRequest.newInstance(this.attemptId));
{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
