Hi, Storm topology is created and deployed in a Kerberized environment. We have a bolt to write data in HBase & HDFS but it is unable to get HBase and HDFS credentials.I got the same error earlier but it got resolved by restarting the cluster. I doubt if it was right solution.
I also faced similar scenario where in the bolts were working fine but suddenly gave the exception after the cluster was unused for couple of days. Are these credentials cached in a temp directory? Please find below the worker logs for the same : 2016-03-23 08:38:19 b.s.s.a.AuthUtils [INFO] Got AutoCreds [com.config.setup.storm.security.AutoHBase@71f139b7, com.config.setup.storm.security.AutoHDFS@51f6292b] 2016-03-23 08:38:19 c.c.h.s.c.s.s.s.AutoHBase [ERROR] Could not obtain credentials from credentials map. java.io.StreamCorruptedException: invalid stream header: 64756D6D at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804) ~[na:1.7.0_67] at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) ~[na:1.7.0_67] at com.config.setup.storm.security.AutoHBase.getCredentials(AutoHBase.java:96) [config.jar:na] at com.config.setup.storm.security.AutoHBase.addCredentialToSubject(AutoHBase.java:129) [config.jar:na] at com.config.setup.storm.security.AutoHBase.populateSubject(AutoHBase.java:122) [config.jar:na] at backtype.storm.security.auth.AuthUtils.populateSubject(AuthUtils.java:189) [storm-core-0.9.3.2.2.9.0-3393.jar:0.9.3.2.2.9.0-3393] at backtype.storm.daemon.worker$fn__7701$exec_fn__1271__auto____7702.invoke(worker.clj:424) [storm-core-0.9.3.2.2.9.0-3393.jar:0.9.3.2.2.9.0-3393] at clojure.lang.AFn.applyToHelper(AFn.java:185) [clojure-1.5.1.jar:na] at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na] at clojure.core$apply.invoke(core.clj:617) [clojure-1.5.1.jar:na] at backtype.storm.daemon.worker$fn__7701$mk_worker__7778.doInvoke(worker.clj:408) [storm-core-0.9.3.2.2.9.0-3393.jar:0.9.3.2.2.9.0-3393] at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.5.1.jar:na] at backtype.storm.daemon.worker$_main.invoke(worker.clj:540) [storm-core-0.9.3.2.2.9.0-3393.jar:0.9.3.2.2.9.0-3393] at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.5.1.jar:na] at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na] at backtype.storm.daemon.worker.main(Unknown Source) [storm-core-0.9.3.2.2.9.0-3393.jar:0.9.3.2.2.9.0-3393] 2016-03-23 08:38:19 c.c.h.s.c.s.s.s.AutoHBase [INFO] No credential found in credentials map. 2016-03-23 08:38:19 c.c.h.s.c.s.s.s.AutoHDFS [ERROR] Could not obtain credentials from credentials map. java.io.StreamCorruptedException: invalid stream header: 64756D6D at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:804) ~[na:1.7.0_67] at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299) ~[na:1.7.0_67] at com.config.setup.storm.security.AutoHDFS.getCredentials(AutoHDFS.java:96) [sm-config.jar:na] at com.config.setup.storm.security.AutoHDFS.addCredentialToSubject(AutoHDFS.java:128) [sm-config.jar:na] at com.config.setup.storm.security.AutoHDFS.populateSubject(AutoHDFS.java:121) [sm-config.jar:na] at backtype.storm.daemon.worker$fn__7701$exec_fn__1271__auto____7702.invoke(worker.clj:424) [storm-core-0.9.3.2.2.9.0-3393.jar:0.9.3.2.2.9.0-3393] at clojure.lang.AFn.applyToHelper(AFn.java:185) [clojure-1.5.1.jar:na] at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na] at clojure.core$apply.invoke(core.clj:617) [clojure-1.5.1.jar:na] at backtype.storm.daemon.worker$fn__7701$mk_worker__7778.doInvoke(worker.clj:408) [storm-core-0.9.3.2.2.9.0-3393.jar:0.9.3.2.2.9.0-3393] at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.5.1.jar:na] at backtype.storm.daemon.worker$_main.invoke(worker.clj:540) [storm-core-0.9.3.2.2.9.0-3393.jar:0.9.3.2.2.9.0-3393] at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.5.1.jar:na] at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na] at backtype.storm.daemon.worker.main(Unknown Source) [storm-core-0.9.3.2.2.9.0-3393.jar:0.9.3.2.2.9.0-3393] 2016-03-23 08:38:19 c.c.h.s.c.s.s.s.AutoHDFS [INFO] No credential found in credentials Best Regards, Radhakrishnan Iyer
