[
https://issues.apache.org/jira/browse/YARN-7765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328636#comment-16328636
]
Rohith Sharma K S commented on YARN-7765:
-----------------------------------------
Grepped log from NM is
{noformat}
2018-01-17 11:04:43,188 INFO containermanager.ContainerManagerImpl
(ContainerManagerImpl.java:startContainerInternal(1127)) - Creating a new
application reference for app application_1516182622885_0002
2018-01-17 11:04:43,206 INFO application.ApplicationImpl
(ApplicationImpl.java:handle(632)) - Application application_1516182622885_0002
transitioned from NEW to INITING
2018-01-17 11:04:43,333 INFO application.ApplicationImpl
(ApplicationImpl.java:transition(446)) - Adding
container_e07_1516182622885_0002_01_000001 to application
application_1516182622885_0002
2018-01-17 11:04:43,340 INFO application.ApplicationImpl
(ApplicationImpl.java:handle(632)) - Application application_1516182622885_0002
transitioned from INITING to RUNNING
2018-01-17 11:04:43,344 INFO container.ContainerImpl
(ContainerImpl.java:handle(2106)) - Container
container_e07_1516182622885_0002_01_000001 transitioned from NEW to LOCALIZING
2018-01-17 11:04:43,353 INFO containermanager.AuxServices
(AuxServices.java:handle(220)) - Got event CONTAINER_INIT for appId
application_1516182622885_0002
2018-01-17 11:04:43,359 INFO collector.TimelineCollectorManager
(TimelineCollectorManager.java:putIfAbsent(142)) - the collector for
application_1516182622885_0002 was added
2018-01-17 11:04:43,363 INFO collector.NodeTimelineCollectorManager
(NodeTimelineCollectorManager.java:updateTimelineCollectorContext(340)) - Get
timeline collector context for application_1516182622885_0002
2018-01-17 11:04:43,364 INFO collector.NodeTimelineCollectorManager
(NodeTimelineCollectorManager.java:getNMCollectorService(384)) -
nmCollectorServiceAddress: /0.0.0.0:8048
2018-01-17 11:04:43,415 INFO delegation.AbstractDelegationTokenSecretManager
(AbstractDelegationTokenSecretManager.java:createPassword(402)) - Creating
password for identifier: (TIMELINE_DELEGATION_TOKEN owner=ambari-qa,
renewer=yarn, realUser=, issueDate=1516187083415, maxDate=1516791883415,
sequenceNumber=1, masterKeyId=2), currentKey: 2
2018-01-17 11:04:43,419 INFO collector.NodeTimelineCollectorManager
(NodeTimelineCollectorManager.java:generateTokenAndSetTimer(228)) - Generated a
new token Kind: TIMELINE_DELEGATION_TOKEN, Service:
ctr-e137-1514896590304-21594-01-000009.hwx.site:36257, Ident:
(TIMELINE_DELEGATION_TOKEN owner=ambari-qa, renewer=yarn, realUser=,
issueDate=1516187083415, maxDate=1516791883415, sequenceNumber=1,
masterKeyId=2) for app application_1516182622885_0002
2018-01-17 11:04:43,427 INFO collector.NodeTimelineCollectorManager
(NodeTimelineCollectorManager.java:reportNewCollectorInfoToNM(330)) - Report a
new collector for application: application_1516182622885_0002 to the NM
Collector Service.
2018-01-17 11:04:43,435 INFO impl.TimelineV2ClientImpl
(TimelineV2ClientImpl.java:setTimelineCollectorInfo(172)) - Updated timeline
service address to ctr-e137-1514896590304-21594-01-000009.hwx.site:36257
2018-01-17 11:04:43,446 INFO localizer.ResourceLocalizationService
(ResourceLocalizationService.java:handle(791)) - Created localizer for
container_e07_1516182622885_0002_01_000001
2018-01-17 11:04:43,467 INFO localizer.ResourceLocalizationService
(ResourceLocalizationService.java:writeCredentials(1322)) - Writing credentials
to the nmPrivate file
/grid/0/hadoop/yarn/local/nmPrivate/container_e07_1516182622885_0002_01_000001.tokens
2018-01-17 11:04:45,879 WARN ipc.RpcClientImpl (RpcClientImpl.java:run(674)) -
Exception encountered while connecting to the server :
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos
tgt)]
2018-01-17 11:04:45,880 FATAL ipc.RpcClientImpl (RpcClientImpl.java:run(684)) -
SASL authentication failed. The most likely cause is missing or invalid
credentials. Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos
tgt)]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at
org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:617)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$700(RpcClientImpl.java:162)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:743)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:740)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:740)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:906)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1241)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.scan(ClientProtos.java:34094)
at
org.apache.hadoop.hbase.client.ClientSmallReversedScanner$SmallReversedScannerCallable.call(ClientSmallReversedScanner.java:298)
at
org.apache.hadoop.hbase.client.ClientSmallReversedScanner$SmallReversedScannerCallable.call(ClientSmallReversedScanner.java:276)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:210)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:364)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas$RetryingRPC.call(ScannerCallableWithReplicas.java:338)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:136)
at
org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture.run(ResultBoundedCompletionService.java:65)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed
to find any Kerberos tgt)
at
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
... 25 more
2018-01-17 11:04:46,230 INFO container.ContainerImpl
(ContainerImpl.java:handle(2106)) - Container
container_e07_1516182622885_0002_01_000001 transitioned from LOCALIZING to
SCHEDULED
2018-01-17 11:04:46,231 INFO scheduler.ContainerScheduler
(ContainerScheduler.java:startContainer(503)) - Starting container
[container_e07_1516182622885_0002_01_000001]
2018-01-17 11:04:46,268 INFO container.ContainerImpl
(ContainerImpl.java:handle(2106)) - Container
container_e07_1516182622885_0002_01_000001 transitioned from SCHEDULED to
RUNNING
2018-01-17 11:04:46,269 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:onStartMonitoringContainer(930)) - Starting
resource-monitoring for container_e07_1516182622885_0002_01_000001
{noformat}
> [Atsv2] App collector failed to authenticate with HBase in secure cluster
> -------------------------------------------------------------------------
>
> Key: YARN-7765
> URL: https://issues.apache.org/jira/browse/YARN-7765
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Rohith Sharma K S
> Priority: Critical
>
> Secure cluster is deployed and all YARN services are started successfully.
> When application is submitted, app collectors which is started as aux-service
> throwing below exception. But this exception is *NOT* observed from RM
> TimelineCollector.
> {noformat}
> 2018-01-17 11:04:48,017 FATAL ipc.RpcClientImpl (RpcClientImpl.java:run(684))
> - SASL authentication failed. The most likely cause is missing or invalid
> credentials. Consider 'kinit'.
> javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> {noformat}
> cc :/ [~vrushalic] [~varun_saxena]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
