Hi I've submitted my project to the cluster after a while I've got the following output: Any idea?
2015-05-15 22:47:33 o.a.c.f.s.ConnectionStateManager [INFO] State change: SUSPENDED 2015-05-15 22:47:34 o.a.z.ClientCnxn [INFO] Opening socket connection to server user1-HVM-domU.local/213.233.170.200:2181. Will not attempt to authenticate using SASL (unknown error) 2015-05-15 22:47:34 o.a.z.ClientCnxn [INFO] Socket connection established to user1-HVM-domU.local/213.233.170.200:2181, initiating session 2015-05-15 22:47:38 o.a.c.f.s.ConnectionStateManager [WARN] There are no ConnectionStateListeners registered. 2015-05-15 22:47:49 b.s.cluster [WARN] Received event :disconnected::none: with disconnected Zookeeper. 2015-05-15 22:47:51 o.a.z.ClientCnxn [WARN] Session 0x14d572f6d9e08f7 for server user1-HVM-domU.local/213.233.170.200:2181, unexpected error, closing socket connection and attempting reconnect java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcher.write0(Native Method) ~[na:1.6.0_35] at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.6.0_35] at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:122) ~[na:1.6.0_35] at sun.nio.ch.IOUtil.write(IOUtil.java:93) ~[na:1.6.0_35] at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:352) ~[na:1.6.0_35] at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:117) ~[zookeeper-3.4.5.jar:3.4.5-1392090] at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355) ~[zookeeper-3.4.5.jar:3.4.5-1392090] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) ~[zookeeper-3.4.5.jar:3.4.5-1392090] 2015-05-15 22:47:55 o.a.z.ClientCnxn [INFO] Opening socket connection to server user1-HVM-domU.local/213.233.170.200:2181. Will not attempt to authenticate using SASL (unknown error) 2015-05-15 22:47:55 o.a.z.ClientCnxn [INFO] Socket connection established to user1-HVM-domU.local/213.233.170.200:2181, initiating session 2015-05-15 22:47:56 o.a.z.ClientCnxn [INFO] Unable to reconnect to ZooKeeper service, session 0x14d572f6d9e08f7 has expired, closing socket connection 2015-05-15 22:47:59 o.a.c.f.s.ConnectionStateManager [INFO] State change: LOST 2015-05-15 22:47:59 o.a.c.f.s.ConnectionStateManager [WARN] There are no ConnectionStateListeners registered. 2015-05-15 22:47:59 b.s.cluster [WARN] Received event :expired::none: with disconnected Zookeeper. 2015-05-15 22:47:59 o.a.c.ConnectionState [WARN] Session expired event received 2015-05-15 22:47:59 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=213.233.170.200:2181/storm sessionTimeout=20000 watcher=org.apache.curator.ConnectionState@4be4c643 2015-05-15 22:47:59 o.a.z.ClientCnxn [INFO] EventThread shut down 2015-05-15 22:47:59 o.a.z.ClientCnxn [INFO] Opening socket connection to server user1-HVM-domU.local/213.233.170.200:2181. Will not attempt to authenticate using SASL (unknown error) 2015-05-15 22:47:59 o.a.z.ClientCnxn [INFO] Socket connection established to user1-HVM-domU.local/213.233.170.200:2181, initiating session 2015-05-15 22:47:59 o.a.z.ClientCnxn [INFO] Session establishment complete on server user1-HVM-domU.local/213.233.170.200:2181, sessionid = 0x14d572f6d9e08fa, negotiated timeout = 20000 2015-05-15 22:47:59 o.a.c.f.s.ConnectionStateManager [INFO] State change: RECONNECTED 2015-05-15 22:47:59 o.a.c.f.s.ConnectionStateManager [WARN] There are no ConnectionStateListeners registered. 2015-05-15 22:48:56 o.a.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 47980ms for sessionid 0x14d572f6d9e08fa, closing socket connection and attempting reconnect 2015-05-15 22:48:59 o.a.c.f.s.ConnectionStateManager [INFO] State change: SUSPENDED 2015-05-15 22:48:59 o.a.c.f.s.ConnectionStateManager [WARN] There are no ConnectionStateListeners registered. 2015-05-15 22:48:59 b.s.cluster [WARN] Received event :disconnected::none: with disconnected Zookeeper. 2015-05-15 22:49:01 o.a.z.ClientCnxn [INFO] Opening socket connection to server user1-HVM-domU.local/213.233.170.200:2181. Will not attempt to authenticate using SASL (unknown error) 2015-05-15 22:49:01 o.a.z.ClientCnxn [INFO] Socket connection established to user1-HVM-domU.local/213.233.170.200:2181, initiating session 2015-05-15 22:49:01 o.a.c.f.s.ConnectionStateManager [INFO] State change: LOST 2015-05-15 22:49:01 o.a.c.f.s.ConnectionStateManager [WARN] There are no ConnectionStateListeners registered. 2015-05-15 22:49:01 b.s.cluster [WARN] Received event :expired::none: with disconnected Zookeeper. 2015-05-15 22:49:01 o.a.c.ConnectionState [WARN] Session expired event received 2015-05-15 22:49:01 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=213.233.170.200:2181/storm sessionTimeout=20000 watcher=org.apache.curator.ConnectionState@4be4c643 2015-05-15 22:49:01 o.a.z.ClientCnxn [INFO] Unable to reconnect to ZooKeeper service, session 0x14d572f6d9e08fa has expired, closing socket connection 2015-05-15 22:51:21 o.a.z.ClientCnxn [INFO] EventThread shut down 2015-05-15 22:52:01 o.a.z.ClientCnxn [INFO] Opening socket connection to server user1-HVM-domU.local/213.233.170.200:2181. Will not attempt to authenticate using SASL (unknown error) 2015-05-15 22:52:01 o.a.z.ClientCnxn [INFO] Socket connection established to user1-HVM-domU.local/213.233.170.200:2181, initiating session 2015-05-15 22:52:01 o.a.z.ClientCnxn [INFO] Session establishment complete on server user1-HVM-domU.local/213.233.170.200:2181, sessionid = 0x14d572f6d9e08fc, negotiated timeout = 20000 2015-05-15 22:52:01 o.a.c.f.s.ConnectionStateManager [INFO] State change: RECONNECTED 2015-05-15 22:52:01 o.a.c.f.s.ConnectionStateManager [WARN] There are no ConnectionStateListeners registered. 2015-05-15 22:52:03 o.a.c.ConnectionState [WARN] Connection attempt unsuccessful after 140089 (greater than max timeout of 20000). Resetting connection and trying again with a new connection. 2015-05-15 22:52:04 o.a.z.ZooKeeper [INFO] Session: 0x14d572f6d9e08fc closed 2015-05-15 22:52:04 o.a.z.ZooKeeper [INFO] Initiating client connection, connectString=213.233.170.200:2181/storm sessionTimeout=20000 watcher=org.apache.curator.ConnectionState@4be4c643 2015-05-15 22:52:04 o.a.z.ClientCnxn [INFO] EventThread shut down 2015-05-15 22:52:07 o.a.z.ClientCnxn [INFO] Opening socket connection to server user1-HVM-domU.local/213.233.170.200:2181. Will not attempt to authenticate using SASL (unknown error) 2015-05-15 22:52:08 o.a.z.ClientCnxn [INFO] Socket connection established to user1-HVM-domU.local/213.233.170.200:2181, initiating session 2015-05-15 22:52:08 o.a.z.ClientCnxn [INFO] Session establishment complete on server user1-HVM-domU.local/213.233.170.200:2181, sessionid = 0x14d572f6d9e08fd, negotiated timeout = 20000 2015-05-15 22:52:38 b.s.util [ERROR] Async loop died! java.lang.RuntimeException: java.io.IOException: Broken pipe at backtype.storm.spout.ShellSpout.querySubprocess(ShellSpout.java:119) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.spout.ShellSpout.nextTuple(ShellSpout.java:68) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.daemon.executor$fn__5573$fn__5588$fn__5617.invoke(executor.clj:563) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na] at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_35] Caused by: java.io.IOException: Broken pipe at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.6.0_35] at java.io.FileOutputStream.write(FileOutputStream.java:300) ~[na:1.6.0_35] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[na:1.6.0_35] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[na:1.6.0_35] at java.io.DataOutputStream.flush(DataOutputStream.java:123) ~[na:1.6.0_35] at com.yelp.pyleus.serializer.MessagePackSerializer.writeMessage(MessagePackSerializer.java:203) ~[stormjar.jar:na] at com.yelp.pyleus.serializer.MessagePackSerializer.writeTaskIds(MessagePackSerializer.java:194) ~[stormjar.jar:na] at backtype.storm.utils.ShellProcess.writeTaskIds(ShellProcess.java:116) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.spout.ShellSpout.querySubprocess(ShellSpout.java:109) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] ... 5 common frames omitted 2015-05-15 22:52:38 o.a.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 24784ms for sessionid 0x14d572f6d9e08fd, closing socket connection and attempting reconnect 2015-05-15 22:52:48 b.s.d.executor [ERROR] java.lang.RuntimeException: java.io.IOException: Broken pipe at backtype.storm.spout.ShellSpout.querySubprocess(ShellSpout.java:119) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.spout.ShellSpout.nextTuple(ShellSpout.java:68) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.daemon.executor$fn__5573$fn__5588$fn__5617.invoke(executor.clj:563) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na] at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_35] Caused by: java.io.IOException: Broken pipe at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.6.0_35] at java.io.FileOutputStream.write(FileOutputStream.java:300) ~[na:1.6.0_35] at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) ~[na:1.6.0_35] at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) ~[na:1.6.0_35] at java.io.DataOutputStream.flush(DataOutputStream.java:123) ~[na:1.6.0_35] at com.yelp.pyleus.serializer.MessagePackSerializer.writeMessage(MessagePackSerializer.java:203) ~[stormjar.jar:na] at com.yelp.pyleus.serializer.MessagePackSerializer.writeTaskIds(MessagePackSerializer.java:194) ~[stormjar.jar:na] at backtype.storm.utils.ShellProcess.writeTaskIds(ShellProcess.java:116) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.spout.ShellSpout.querySubprocess(ShellSpout.java:109) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] ... 5 common frames omitted 2015-05-15 22:52:57 o.a.c.f.s.ConnectionStateManager [INFO] State change: SUSPENDED 2015-05-15 22:52:57 o.a.c.f.s.ConnectionStateManager [WARN] There are no ConnectionStateListeners registered. 2015-05-15 22:52:57 b.s.util [ERROR] Async loop died! java.lang.RuntimeException: java.lang.RuntimeException: java.io.EOFException at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na] at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_35] Caused by: java.lang.RuntimeException: java.io.EOFException at backtype.storm.task.ShellBolt.execute(ShellBolt.java:157) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] ... 6 common frames omitted Caused by: java.io.EOFException: null at org.msgpack.io.StreamInput.readByte(StreamInput.java:60) ~[stormjar.jar:na] at org.msgpack.unpacker.MessagePackUnpacker.getHeadByte(MessagePackUnpacker.java:66) ~[stormjar.jar:na] at org.msgpack.unpacker.MessagePackUnpacker.trySkipNil(MessagePackUnpacker.java:396) ~[stormjar.jar:na] at org.msgpack.template.MapTemplate.read(MapTemplate.java:59) ~[stormjar.jar:na] at org.msgpack.template.MapTemplate.read(MapTemplate.java:27) ~[stormjar.jar:na] at org.msgpack.template.AbstractTemplate.read(AbstractTemplate.java:31) ~[stormjar.jar:na] at org.msgpack.MessagePack.read(MessagePack.java:527) ~[stormjar.jar:na] at org.msgpack.MessagePack.read(MessagePack.java:496) ~[stormjar.jar:na] at com.yelp.pyleus.serializer.MessagePackSerializer.readMessage(MessagePackSerializer.java:198) ~[stormjar.jar:na] at com.yelp.pyleus.serializer.MessagePackSerializer.readShellMsg(MessagePackSerializer.java:74) ~[stormjar.jar:na] at backtype.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:97) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.task.ShellBolt$1.run(ShellBolt.java:107) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] ... 1 common frames omitted 2015-05-15 22:52:57 b.s.d.executor [ERROR] java.lang.RuntimeException: java.lang.RuntimeException: java.io.EOFException at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.daemon.executor$fn__5641$fn__5653$fn__5700.invoke(executor.clj:746) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.util$async_loop$fn__457.invoke(util.clj:431) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na] at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_35] Caused by: java.lang.RuntimeException: java.io.EOFException at backtype.storm.task.ShellBolt.execute(ShellBolt.java:157) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.daemon.executor$fn__5641$tuple_action_fn__5643.invoke(executor.clj:631) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.daemon.executor$mk_task_receiver$fn__5564.invoke(executor.clj:399) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.disruptor$clojure_handler$reify__745.onEvent(disruptor.clj:58) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] ... 6 common frames omitted Caused by: java.io.EOFException: null at org.msgpack.io.StreamInput.readByte(StreamInput.java:60) ~[stormjar.jar:na] at org.msgpack.unpacker.MessagePackUnpacker.getHeadByte(MessagePackUnpacker.java:66) ~[stormjar.jar:na] at org.msgpack.unpacker.MessagePackUnpacker.trySkipNil(MessagePackUnpacker.java:396) ~[stormjar.jar:na] at org.msgpack.template.MapTemplate.read(MapTemplate.java:59) ~[stormjar.jar:na] at org.msgpack.template.MapTemplate.read(MapTemplate.java:27) ~[stormjar.jar:na] at org.msgpack.template.AbstractTemplate.read(AbstractTemplate.java:31) ~[stormjar.jar:na] at org.msgpack.MessagePack.read(MessagePack.java:527) ~[stormjar.jar:na] at org.msgpack.MessagePack.read(MessagePack.java:496) ~[stormjar.jar:na] at com.yelp.pyleus.serializer.MessagePackSerializer.readMessage(MessagePackSerializer.java:198) ~[stormjar.jar:na] at com.yelp.pyleus.serializer.MessagePackSerializer.readShellMsg(MessagePackSerializer.java:74) ~[stormjar.jar:na] at backtype.storm.utils.ShellProcess.readShellMsg(ShellProcess.java:97) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.task.ShellBolt$1.run(ShellBolt.java:107) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] ... 1 common frames omitted 2015-05-15 22:52:57 b.s.cluster [WARN] Received event :disconnected::none: with disconnected Zookeeper. 2015-05-15 22:53:04 o.a.z.ClientCnxn [INFO] Opening socket connection to server user1-HVM-domU.local/213.233.170.200:2181. Will not attempt to authenticate using SASL (unknown error) 2015-05-15 22:53:04 o.a.z.ClientCnxn [INFO] Socket connection established to user1-HVM-domU.local/213.233.170.200:2181, initiating session 2015-05-15 22:53:04 o.a.c.f.s.ConnectionStateManager [INFO] State change: LOST 2015-05-15 22:53:04 o.a.c.f.s.ConnectionStateManager [WARN] There are no ConnectionStateListeners registered. 2015-05-15 22:53:11 o.a.z.ClientCnxn [INFO] Unable to reconnect to ZooKeeper service, session 0x14d572f6d9e08fd has expired, closing socket connection 2015-05-15 22:53:27 o.a.c.ConnectionState [ERROR] Connection timed out for connection string (213.233.170.200:2181/storm) and timeout (15000) / elapsed (15086) org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:198) [curator-client-2.4.0.jar:na] at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88) [curator-client-2.4.0.jar:na] at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:113) ~[curator-client-2.4.0.jar:na] at org.apache.curator.framework.imps.CuratorFrameworkImpl.getZooKeeper(CuratorFrameworkImpl.java:457) ~[curator-framework-2.4.0.jar:na] at org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:172) ~[curator-framework-2.4.0.jar:na] at org.apache.curator.framework.imps.ExistsBuilderImpl$2.call(ExistsBuilderImpl.java:161) ~[curator-framework-2.4.0.jar:na] at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[curator-client-2.4.0.jar:na] at org.apache.curator.framework.imps.ExistsBuilderImpl.pathInForeground(ExistsBuilderImpl.java:157) ~[curator-framework-2.4.0.jar:na] at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:148) ~[curator-framework-2.4.0.jar:na] at org.apache.curator.framework.imps.ExistsBuilderImpl.forPath(ExistsBuilderImpl.java:36) ~[curator-framework-2.4.0.jar:na] at backtype.storm.zookeeper$exists_node_QMARK_$fn__1153.invoke(zookeeper.clj:101) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.zookeeper$exists_node_QMARK_.invoke(zookeeper.clj:98) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.zookeeper$mkdirs.invoke(zookeeper.clj:114) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.cluster$mk_distributed_cluster_state$reify__1865.mkdirs(cluster.clj:109) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.cluster$mk_storm_cluster_state$reify__2284.report_error(cluster.clj:368) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.daemon.executor$throttled_report_error_fn$fn__5421.invoke(executor.clj:178) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.daemon.executor$mk_executor_data$fn__5474$fn__5475.invoke(executor.clj:237) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at backtype.storm.util$async_loop$fn__457.invoke(util.clj:441) ~[storm-core-0.9.2-incubating.jar:0.9.2-incubating] at clojure.lang.AFn.run(AFn.java:24) ~[clojure-1.5.1.jar:na] at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_35] Thanks Hadi
