[
https://issues.apache.org/jira/browse/HBASE-19883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16342994#comment-16342994
]
Rohith Sharma K S commented on HBASE-19883:
-------------------------------------------
After 24hours i.e when token lifetime expires, the RS and HM has below logs. At
this point of time, HDFS service was up and running.
{noformat}
2018-01-24 09:13:18,177 WARN [LeaseRenewer:yarn@mycluster]
security.UserGroupInformation: Not attempting to re-login since the last
re-login was attempted less than 600 seconds before.
2018-01-24 09:13:20,413 WARN [LeaseRenewer:yarn@mycluster]
security.UserGroupInformation: Not attempting to re-login since the last
re-login was attempted less than 600 seconds before.
2018-01-24 09:13:23,158 WARN [LeaseRenewer:yarn@mycluster]
security.UserGroupInformation: Not attempting to re-login since the last
re-login was attempted less than 600 seconds before.
2018-01-24 09:13:24,446 WARN [LeaseRenewer:yarn@mycluster]
security.UserGroupInformation: Not attempting to re-login since the last
re-login was attempted less than 600 seconds before.
2018-01-24 09:13:27,509 WARN [LeaseRenewer:yarn@mycluster] ipc.Client:
Couldn't setup connection for
yarn/[email protected] to
ctr-e137-1514896590304-33059-01-000003.hwx.site/172.27.12.21:8020
2018-01-24 09:13:27,510 INFO [LeaseRenewer:yarn@mycluster]
retry.RetryInvocationHandler: Exception while invoking renewLease of class
ClientNamenodeProtocolTranslatorPB over
ctr-e137-1514896590304-33059-01-000003.hwx.site/172.27.12.21:8020. Trying to
fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException: Couldn't
setup connection for
yarn/[email protected] to
ctr-e137-1514896590304-33059-01-000003.hwx.site/172.27.12.21:8020; Host Details
: local host is:
"ctr-e137-1514896590304-33059-01-000003.hwx.site/172.27.12.21"; destination
host is: "ctr-e137-1514896590304-33059-01-000003.hwx.site":8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
at org.apache.hadoop.ipc.Client.call(Client.java:1415)
at org.apache.hadoop.ipc.Client.call(Client.java:1364)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy15.renewLease(Unknown Source)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.renewLease(ClientNamenodeProtocolTranslatorPB.java:540)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy16.renewLease(Unknown Source)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:279)
at com.sun.proxy.$Proxy17.renewLease(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.renewLease(DFSClient.java:814)
at org.apache.hadoop.hdfs.LeaseRenewer.renew(LeaseRenewer.java:417)
at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:442)
at org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71)
at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Couldn't setup connection for
yarn/[email protected] to
ctr-e137-1514896590304-33059-01-000003.hwx.site/172.27.12.21:8020
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:671)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:642)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:725)
at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463)
at org.apache.hadoop.ipc.Client.call(Client.java:1382)
... 21 more
Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
at
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
at
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:552)
at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:367)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:717)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:713)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
... 24 more
Caused by: GSSException: No valid credentials provided (Mechanism level: Failed
to find any Kerberos tgt)
at
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
at
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)
at
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
at
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
... 33 more
{noformat}
> HM and RS are going down with Failed to find any Kerberos tgt after token
> lifetime expired
> ------------------------------------------------------------------------------------------
>
> Key: HBASE-19883
> URL: https://issues.apache.org/jira/browse/HBASE-19883
> Project: HBase
> Issue Type: Bug
> Affects Versions: 1.2.6
> Environment: Java Version : 1.8 Open JDK
> OS : centos-7 64-bit
> Reporter: Rohith Sharma K S
> Priority: Critical
>
> HBase non-ha secure cluster was installed and running in successfully with
> regular operations. HDFS service was HA and couple of time, NameNode switch
> happened back and forth first day.
> After 24 hours i.e when token lifetime expired, It is observed that HBase
> cluster daemons such as Hmaster and HRegionserver are shutting down with
> security exception No valid credentials provided (Mechanism level: Failed to
> find any Kerberos tgt)!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
