Hi, I have a hdfs bolt in my storm topology, it is running perfectly fine for some time, but suddently it is failing with SASL security exceptions!!
I have followed the details outlined [Working with Secure HDFS - Using key tabs on all worker hosts] at link http://storm.apache.org/documentation/storm-hdfs.html as we are on Kerberos secured cluster. Shared below is the code i am using to configure hdfs bolt ! builder.setBolt("hdfs-bolt-1", HDFSBolt, 1) .setNumTasks(30) .shuffleGrouping("prev-bolt-1"); builder.setBolt("hdfs-bolt-2", invalidRecsHDFSBolt, 1) .setNumTasks(4) .shuffleGrouping("prev-bolt-2"); Below is the exception: java.io.IOException: IOException flush:java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host Details : local host is: "xxxxxxxxxxxx"; destination host is: "xxxxxxxx":8020; at org.apache.hadoop.hdfs.DFSOutputStream.flushOrSync(DFSOutputStream.java:2082) ~[stormjar.jar:na] at org.apache.hadoop.hdfs.DFSOutputStream.hsync(DFSOutputStream.java:1969) ~[stormjar.jar:na] at org.apache.hadoop.hdfs.client.HdfsDataOutputStream.hsync(HdfsDataOutputStream.java:95) ~[stormjar.jar:na] at org.apache.storm.hdfs.bolt.HdfsBolt.execute(HdfsBolt.java:100) ~[stormjar.jar:na] at backtype.storm.daemon.executor$fn__5265$tuple_action_fn__5267.invoke(executor.clj:659) [storm-core-0.9.3.2.2.4.12-1.jar:0.9.3.2.2.4.12-1] at backtype.storm.daemon.executor$mk_task_receiver$fn__5188.invoke(executor.clj:415) [storm-core-0.9.3.2.2.4.12-1.jar:0.9.3.2.2.4.12-1] at backtype.storm.disruptor$clojure_handler$reify__1064.onEvent(disruptor.clj:58) [storm-core-0.9.3.2.2.4.12-1.jar:0.9.3.2.2.4.12-1] at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) [storm-core-0.9.3.2.2.4.12-1.jar:0.9.3.2.2.4.12-1] at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) [storm-core-0.9.3.2.2.4.12-1.jar:0.9.3.2.2.4.12-1] at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) [storm-core-0.9.3.2.2.4.12-1.jar:0.9.3.2.2.4.12-1] at backtype.storm.daemon.executor$fn__5265$fn__5278$fn__5329.invoke(executor.clj:794) [storm-core-0.9.3.2.2.4.12-1.jar:0.9.3.2.2.4.12-1] at backtype.storm.util$async_loop$fn__551.invoke(util.clj:465) [storm-core-0.9.3.2.2.4.12-1.jar:0.9.3.2.2.4.12-1] at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na] at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45] Some where i read, storm hdfs bolt fails to authenticate, when multiple tasks/executors used!! Can someone pls help me fix the issue... Please share your thoughts.... to fix the issue. Regards, Raja Aravapalli
