storm 0.9.5
Workers are dying within 2 minutes of startup.
We even set up dedicated zookeeper on separate cloudera hadoop cluster.
Anyone seen this before? seems like zookeeper connections are failing.
In the worker log:
2015-09-02T07:18:25.271-0500 o.a.s.c.f.s.ConnectionStateManager [INFO]
State change: SUSPENDED
2015-09-02T07:18:25.272-0500 b.s.cluster [WARN] Received event
:disconnected::none: with disconnected Zookeeper.
2015-09-02T07:18:25.286-0500 b.s.d.worker [ERROR] Error when processing
event
java.lang.RuntimeException:
org.apache.storm.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for
/workerbeats/ClaimInfoProcessing-6-1441195936/1cb9375c-9b55-4811-a574-03d12ec9a20f-9700
at backtype.storm.util$wrap_in_runtime.invoke(util.clj:44)
~[storm-core-0.9.5.jar:0.9.5]
at backtype.storm.zookeeper$set_data.invoke(zookeeper.clj:173)
~[storm-core-0.9.5.jar:0.9.5]
at
backtype.storm.cluster$mk_distributed_cluster_state$reify__2073.set_data(cluster.clj:92)
~[storm-core-0.9.5.jar:0.9.5]
at
backtype.storm.cluster$mk_storm_cluster_state$reify__2530.worker_heartbeat_BANG_(cluster.clj:332)
~[storm-core-0.9.5.jar:0.9.5]
at sun.reflect.GeneratedMethodAccessor581.invoke(Unknown Source)
~[na:na]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[na:1.8.0_31]
at java.lang.reflect.Method.invoke(Method.java:483) ~[na:1.8.0_31]
at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
~[clojure-1.5.1.jar:na]
at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
~[clojure-1.5.1.jar:na]
at
backtype.storm.daemon.worker$do_executor_heartbeats.doInvoke(worker.clj:56)
~[storm-core-0.9.5.jar:0.9.5]
at clojure.lang.RestFn.invoke(RestFn.java:439)
~[clojure-1.5.1.jar:na]
at
backtype.storm.daemon.worker$fn__6959$exec_fn__1103__auto____6960$fn__6963.invoke(worker.clj:411)
~[storm-core-0.9.5.jar:0.9.5]
at
backtype.storm.timer$schedule_recurring$this__1807.invoke(timer.clj:99)
~[storm-core-0.9.5.jar:0.9.5]
at
backtype.storm.timer$mk_timer$fn__1790$fn__1791.invoke(timer.clj:50)
~[storm-core-0.9.5.jar:0.9.5]
at backtype.storm.timer$mk_timer$fn__1790.invoke(timer.clj:42)
[storm-core-0.9.5.jar:0.9.5]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_31]
Caused by:
org.apache.storm.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for
/workerbeats/ClaimInfoProcessing-6-1441195936/1cb9375c-9b55-4811-a574-03d12ec9a20f-9700
at
org.apache.storm.zookeeper.KeeperException.create(KeeperException.java:99)
~[storm-core-0.9.5.jar:0.9.5]
at
org.apache.storm.zookeeper.KeeperException.create(KeeperException.java:51)
~[storm-core-0.9.5.jar:0.9.5]
at
org.apache.storm.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270)
~[storm-core-0.9.5.jar:0.9.5]
at
org.apache.storm.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:260)
~[storm-core-0.9.5.jar:0.9.5]
at
org.apache.storm.curator.framework.imps.SetDataBuilderImpl$4.call(SetDataBuilderImpl.java:256)
~[storm-core-0.9.5.jar:0.9.5]
at
org.apache.storm.curator.RetryLoop.callWithRetry(RetryLoop.java:107)
~[storm-core-0.9.5.jar:0.9.5]
at
org.apache.storm.curator.framework.imps.SetDataBuilderImpl.pathInForeground(SetDataBuilderImpl.java:252)
~[storm-core-0.9.5.jar:0.9.5]
at
org.apache.storm.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:239)
~[storm-core-0.9.5.jar:0.9.5]
at
org.apache.storm.curator.framework.imps.SetDataBuilderImpl.forPath(SetDataBuilderImpl.java:39)
~[storm-core-0.9.5.jar:0.9.5]
at backtype.storm.zookeeper$set_data.invoke(zookeeper.clj:172)
~[storm-core-0.9.5.jar:0.9.5]
... 15 common frames omitted
2015-09-02T07:18:25.291-0500 b.s.util [ERROR] Halting process: ("Error when
processing an event")
java.lang.RuntimeException: ("Error when processing an event")
at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325)
[storm-core-0.9.5.jar:0.9.5]
at clojure.lang.RestFn.invoke(RestFn.java:423)
[clojure-1.5.1.jar:na]
at
backtype.storm.daemon.worker$mk_halting_timer$fn__6774.invoke(worker.clj:176)
[storm-core-0.9.5.jar:0.9.5]
at
backtype.storm.timer$mk_timer$fn__1790$fn__1791.invoke(timer.clj:68)
[storm-core-0.9.5.jar:0.9.5]
at backtype.storm.timer$mk_timer$fn__1790.invoke(timer.clj:42)
[storm-core-0.9.5.jar:0.9.5]
at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_31]
Any pointers where to look further? Zookeeper has 9 open connections... and
is pretty much idle.
