Hi Laxman, Have you obtained a Kerberos ticket before connecting to the cluster? Can you try the following from your client and then reconnect to the cluster? $ kinit testuser/your-client-hostname
-Yifeng On May 22, 2012, at 6:51 PM, Laxman wrote: > We got stuck with a problem while verifying client authentication in a secure > HBase cluster. > We are able to start a secure HBase cluster successfully. > > However, clients are not able to establish secure connection with HBase > server successfully. > > Other details: > HBase version: 0.94.0 > Hadoop version: 0.23.1 > Kerberos version: 1.10.1 > Java version: 1.6.0_31, 64 bit > Linux version: SuSE 11.1 [Kernel version : 2.6.32.12-0.7-default x86_64 > GNU/Linux] > > We had gone thru the solutions available @ > http://docs.oracle.com/javase/1.5.0/docs/guide/security/jgss/tutorials/Troubleshooting.html > https://ccp.cloudera.com/display/CDHDOC/Appendix+A+-+Troubleshooting#AppendixA-Troubleshooting-Problem2%3AJavaisunabletoreadtheKerberoscredentialscachecreatedbyversionsofMITKerberos1.8.1orhigher. > > But none of then seems to work. Any clue? > > There are no change in server logs as client is failing is failing even > before it communicates with server. > Exception we are hitting (Client side logs): > > 2012-05-22 09:42:22,627 WARN org.apache.hadoop.ipc.SecureClient: Exception > encountered while connecting to the server : > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)] > 2012-05-22 09:42:22,627 ERROR > org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException > as:testuser (auth:KERBEROS) cause:java.io.IOException: > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)] > 2012-05-22 09:42:22,630 DEBUG org.apache.hadoop.ipc.SecureClient: closing ipc > connection to HOST-10-18-40-19/10.18.40.19:60020: > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)] > java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt)] > at > org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:227) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37) > at org.apache.hadoop.hbase.security.User.call(User.java:586) > at org.apache.hadoop.hbase.security.User.access$700(User.java:50) > at > org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:440) > at > org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.handleSaslConnectionFailure(SecureClient.java:194) > at > org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:274) > at > org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.java:485) > at > org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.java:69) > at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:897) > at > org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:164) > at $Proxy6.getProtocolVersion(Unknown Source) > at > org.apache.hadoop.hbase.ipc.SecureRpcEngine.getProxy(SecureRpcEngine.java:208) > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:303) > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:280) > at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:332) > at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:236) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1284) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1240) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1227) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:936) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801) > at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174) > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133) > at hbase.test.Hbasetest.main(Hbasetest.java:37) > Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)] > at > com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194) > at > org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:138) > at > org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:176) > at > org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$500(SecureClient.java:84) > at > org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:267) > at > org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:264) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37) > at org.apache.hadoop.hbase.security.User.call(User.java:586) > at org.apache.hadoop.hbase.security.User.access$700(User.java:50) > at > org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:440) > at > org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:263) > ... 23 more > Caused by: GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt) > at > sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130) > at > sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106) > at > sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172) > at > sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209) > at > sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195) > at > sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162) > at > com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175) > ... 40 more > 2012-05-22 09:42:22,636 DEBUG org.apache.hadoop.ipc.SecureClient: IPC Client > (1778276127) connection to HOST-10-18-40-19/10.18.40.19:60020 from testuser: > closed > 2012-05-22 09:42:22,638 DEBUG > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > locateRegionInMeta parentTable=-ROOT-, > metaLocation={region=-ROOT-,,0.70236052, hostname=HOST-10-18-40-19, > port=60020}, attempt=0 of 120 failed; retrying after sleep of 1000 because: > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)] > 2012-05-22 09:42:22,640 DEBUG > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > Looked up root region location, > connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@6ecf829d; > serverName=HOST-10-18-40-19,60020,1337574445438 > 2012-05-22 09:42:23,641 DEBUG > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > Looked up root region location, > connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@6ecf829d; > serverName=HOST-10-18-40-19,60020,1337574445438 > 2012-05-22 09:42:23,642 DEBUG org.apache.hadoop.ipc.SecureClient: RPC Server > Kerberos principal name for > protocol=org.apache.hadoop.hbase.ipc.HRegionInterface is > hbase/[email protected] > > > -- > Regards, > Laxman >
