Hi All,
i’ve written a STORM topology with a HDFS and a HBase Bolt, which runs
absolutely fine with kerberos turned off in my hadoop cluster.
With activated security in the cluster, the HDFS Bolts works like a
charm, but I encounter authentication errors with the HBase Bolt.
Accessing HBase shell with a Kerberos ticket works also absolutely fine.
The java exception in the Storm topology log is as follows:
2015-03-15 12:21:11 o.a.h.h.i.RpcClient [WARN] Exception encountered
while connecting to the server : javax.security.sasl.SaslException: GSS
initiate failed [Caused by GSSExcept$
2015-03-15 12:21:11 o.a.h.h.i.RpcClient [ERROR] SASL authentication
failed. The most likely cause is missing or invalid credentials.
Consider 'kinit'.
javax.security.sasl.SaslException: GSS initiate failed
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
~[na:1.7.0_45]
at
org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:177)
~[stormjar.jar:na]
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupSaslConnection(RpcClient.java:815)
[stormjar.jar:na]
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.access$800(RpcClient.java:349)[stormjar.jar:na]
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:943)
~[stormjar.jar:na]
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection$2.run(RpcClient.java:940)
~[stormjar.jar:na]
at java.security.AccessController.doPrivileged(Native Method)
~[na:1.7.0_45]
at javax.security.auth.Subject.doAs(Subject.java:415)
~[na:1.7.0_45]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
~[hadoop-common-2.6.0.2.2.0.0-2041.jar:na]
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.setupIOstreams(RpcClient.java:940)
[stormjar.jar:na]
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.writeRequest(RpcClient.java:1094)
[stormjar.jar:na]
at
org.apache.hadoop.hbase.ipc.RpcClient$Connection.tracedWriteRequest(RpcClient.java:1061)
[stormjar.jar:na]
at
org.apache.hadoop.hbase.ipc.RpcClient.call(RpcClient.java:1516)
[stormjar.jar:na]
at
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1724)
[stormjar.jar:na]
at
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1777)
[stormjar.jar:na]
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:30373)
[stormjar.jar:na]
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRowOrBefore(ProtobufUtil.java:1604)
[stormjar.jar:na]
at
org.apache.hadoop.hbase.client.HTable$2.call(HTable.java:768)
[stormjar.jar:na]
at
org.apache.hadoop.hbase.client.HTable$2.call(HTable.java:766)
[stormjar.jar:na]
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126)
[stormjar.jar:na]
at
org.apache.hadoop.hbase.client.HTable.getRowOrBefore(HTable.java:772)
[stormjar.jar:na]
at
org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:160) [stormjar.jar:na]
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.prefetchRegionCache(ConnectionManager.java:1222)
[stormjar.jar:na]
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1286)
[stormjar.jar:na]
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1135)
[stormjar.jar:na]
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:362) [stormjar.jar:na]
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:313) [stormjar.jar:na]
at
org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:1066)
[stormjar.jar:na]
at
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1344)
[stormjar.jar:na]
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1001)
[stormjar.jar:na]
at
com.mhp.bigdata.storm.gt6.MHPHBaseBolt2.execute(MHPHBaseBolt2.java:159)
[stormjar.jar:na]
at
backtype.storm.daemon.executor$fn__5697$tuple_action_fn__5699.invoke(executor.clj:659)
[storm-core-0.9.3.2.2.0.0-2041.jar:0.9.3.2.2.0.0-2041]
at
backtype.storm.daemon.executor$mk_task_receiver$fn__5620.invoke(executor.clj:415)
[storm-core-0.9.3.2.2.0.0-2041.jar:0.9.3.2.2.0.0-2041]
at
backtype.storm.disruptor$clojure_handler$reify__1741.onEvent(disruptor.clj:58)
[storm-core-0.9.3.2.2.0.0-2041.jar:0.9.3.2.2.0.0-2041]
at
backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:120)
[storm-core-0.9.3.2.2.0.0-2041.jar:0.9.3.2.2.0.0-2041]
at
backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
[storm-core-0.9.3.2.2.0.0-2041.jar:0.9.3.2.2.0.0-2041]
at
backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
[storm-core-0.9.3.2.2.0.0-2041.jar:0.9.3.2.2.0.0-2041]
at
backtype.storm.daemon.executor$fn__5697$fn__5710$fn__5761.invoke(executor.clj:794)
[storm-core-0.9.3.2.2.0.0-2041.jar:0.9.3.2.2.0.0-2041]
at backtype.storm.util$async_loop$fn__452.invoke(util.clj:465)
[storm-core-0.9.3.2.2.0.0-2041.jar:0.9.3.2.2.0.0-2041]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
Caused by: org.ietf.jgss.GSSException: No valid credentials provided
(Mechanism level: Failed to find any Kerberos tgt)
at
sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
~[na:1.7.0_45]
at
sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
~[na:1.7.0_45]
at
sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
~[na:1.7.0_45]
at
sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
~[na:1.7.0_45]
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
~[na:1.7.0_45]
at
sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
~[na:1.7.0_45]
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
~[na:1.7.0_45]
... 40 common frames omitted
My Storm Version is: 0.9.3.2.2.0.0
My HBase Version is: 0.98.4.2.2.0.0
Java JCE is installed. Kerberos is working fine for all other components
of the cluster.
Is there any known bug? Or any chance to trace this issue down to its roots?
I’m looking forward to your responses.
Best regards
Frank
