Hello all! I am having a trouble when trying to deploy a simple storm trident topology to the cluster.
I use BaseFunction to implement a bolt to execute a resemblance calculator. In order to achieve that, the bolt has to load some big graphs (from a file) on prepare function. Locally, the topology seems to run fine, even though it starts processing tuples late due to the graph loading, but when i submit it to the cluster it wont work the same way. I took a look at the logs and found out that the topology loads the first graph successfully, but then zookeeper experiences timeout (i guess) and tries to reconnect to the client that loads that specific bolt, as appears below. Then the bolt restarts. I tried to increase zk timeout and restarted zk and the storm cluster, but still get the same timeout (2000) on logs. *zoo.cgf:# The number of milliseconds of each ticktickTime=10000* Is there something i am missing here? Is there any way to achieve the bolt not to timeout until the graphs are loaded? Thank you in advance, *Log of worker that loads the specific bolt:* 2015-02-15T16:00:09.519+0200 g.n.u.Classifier [INFO] Loading graph #2... 2015-02-15T16:00:21.083+0200 o.a.s.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 15489ms for sessionid 0x14b8d7c14a00020, closing socket connection and attempting reconnect 2015-02-15T16:00:21.193+0200 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: SUSPENDED 2015-02-15T16:00:22.698+0200 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server IP/IP:2181. Will not attempt to authenticate using SASL (unknown error) 2015-02-15T16:00:22.700+0200 o.a.s.z.ClientCnxn [INFO] Socket connection established to IP/IP:2181, initiating session 2015-02-15T16:00:22.708+0200 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server IP/IP:2181, sessionid = 0x14b8d7c14a00020, negotiated timeout = 20000 2015-02-15T16:00:22.709+0200 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: RECONNECTED 2015-02-15T16:00:36.697+0200 o.a.s.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 13473ms for sessionid 0x14b8d7c14a00020, closing socket connection and attempting reconnect 2015-02-15T16:00:36.811+0200 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: SUSPENDED 2015-02-15T16:00:39.024+0200 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server IP/IP:2181. Will not attempt to authenticate using SASL (unknown error) 2015-02-15T16:00:39.027+0200 o.a.s.z.ClientCnxn [INFO] Socket connection established to IP/IP:2181, initiating session 2015-02-15T16:00:39.032+0200 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server IP/IP:2181, sessionid = 0x14b8d7c14a00020, negotiated timeout = 20000 2015-02-15T16:00:39.034+0200 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: RECONNECTED 2015-02-15T16:01:01.127+0200 o.a.s.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 14385ms for sessionid 0x14b8d7c14a0001e, closing socket connection and attempting reconnect 2015-02-15T16:01:01.127+0200 o.a.s.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 14459ms for sessionid 0x14b8d7c14a00020, closing socket connection and attempting reconnect 2015-02-15T16:01:01.229+0200 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: SUSPENDED 2015-02-15T16:01:01.231+0200 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: SUSPENDED 2015-02-15T16:01:01.232+0200 b.s.cluster [WARN] Received event :disconnected::none: with disconnected Zookeeper. 2015-02-15T16:01:02.417+0200 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server IP/IP:2181. Will not attempt to authenticate using SASL (unknown error) 2015-02-15T16:01:02.419+0200 o.a.s.z.ClientCnxn [INFO] Socket connection established to IP/IP:2181, initiating session 2015-02-15T16:01:02.427+0200 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server IP/IP:2181, sessionid = 0x14b8d7c14a00020, negotiated timeout = 20000 2015-02-15T16:01:02.429+0200 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: RECONNECTED 2015-02-15T16:01:03.115+0200 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server IP/IP:2181. Will not attempt to authenticate using SASL (unknown error) 2015-02-15T16:01:03.117+0200 o.a.s.z.ClientCnxn [INFO] Socket connection established to IP/IP:2181, initiating session 2015-02-15T16:01:03.130+0200 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server IP/IP:2181, sessionid = 0x14b8d7c14a0001e, negotiated timeout = 20000 2015-02-15T16:01:03.130+0200 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: RECONNECTED 2015-02-15T16:01:40.828+0200 o.a.s.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 13722ms for sessionid 0x14b8d7c14a00020, closing socket connection and attempting reconnect 2015-02-15T16:01:40.940+0200 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: SUSPENDED 2015-02-15T16:01:42.287+0200 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server IP/IP:2181. Will not attempt to authenticate using SASL (unknown error) 2015-02-15T16:01:42.290+0200 o.a.s.z.ClientCnxn [INFO] Socket connection established to IP/IP:2181, initiating session 2015-02-15T16:01:42.302+0200 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server IP/IP:2181, sessionid = 0x14b8d7c14a00020, negotiated timeout = 20000 2015-02-15T16:01:42.303+0200 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: RECONNECTED 2015-02-15T16:01:58.504+0200 o.a.s.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 14435ms for sessionid 0x14b8d7c14a00020, closing socket connection and attempting reconnect 2015-02-15T16:01:58.504+0200 o.a.s.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 14541ms for sessionid 0x14b8d7c14a0001e, closing socket connection and attempting reconnect 2015-02-15T16:01:58.605+0200 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: SUSPENDED 2015-02-15T16:01:58.621+0200 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: SUSPENDED 2015-02-15T16:01:58.623+0200 b.s.cluster [WARN] Received event :disconnected::none: with disconnected Zookeeper. 2015-02-15T16:02:00.001+0200 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server IP/IP:2181. Will not attempt to authenticate using SASL (unknown error) 2015-02-15T16:02:00.003+0200 o.a.s.z.ClientCnxn [INFO] Socket connection established to IP/IP:2181, initiating session 2015-02-15T16:02:00.011+0200 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server IP/IP:2181, sessionid = 0x14b8d7c14a00020, negotiated timeout = 20000 2015-02-15T16:02:00.012+0200 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: RECONNECTED 2015-02-15T16:02:00.261+0200 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server IP/IP:2181. Will not attempt to authenticate using SASL (unknown error) 2015-02-15T16:02:00.263+0200 o.a.s.z.ClientCnxn [INFO] Socket connection established to IP/IP:2181, initiating session 2015-02-15T16:02:00.271+0200 o.a.s.z.ClientCnxn [INFO] Session establishment complete on server IP/IP, sessionid = 0x14b8d7c14a0001e, negotiated timeout = 20000 2015-02-15T16:02:00.272+0200 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: RECONNECTED 2015-02-15T16:02:25.314+0200 o.a.s.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 13822ms for sessionid 0x14b8d7c14a0001e, closing socket connection and attempting reconnect 2015-02-15T16:02:25.315+0200 o.a.s.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 13891ms for sessionid 0x14b8d7c14a00020, closing socket connection and attempting reconnect 2015-02-15T16:02:25.416+0200 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: SUSPENDED 2015-02-15T16:02:25.417+0200 b.s.cluster [WARN] Received event :disconnected::none: with disconnected Zookeeper. 2015-02-15T16:02:25.419+0200 o.a.s.c.f.s.ConnectionStateManager [INFO] State change: SUSPENDED 2015-02-15T16:02:45.826+0200 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server IP/IP:2181. Will not attempt to authenticate using SASL (unknown error) 2015-02-15T16:02:45.827+0200 o.a.s.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 20409ms for sessionid 0x14b8d7c14a00020, closing socket connection and attempting reconnect 2015-02-15T16:02:45.827+0200 o.a.s.z.ClientCnxn [INFO] Opening socket connection to server IP/IP:2181. Will not attempt to authenticate using SASL (unknown error) 2015-02-15T16:02:45.831+0200 o.a.s.z.ClientCnxn [INFO] Client session timed out, have not heard from server in 20416ms for sessionid 0x14b8d7c14a0001e, closing socket connection and attempting reconnect 2015-02-15T16:02:55.058+0200 o.a.s.c.ConnectionState [WARN] Connection attempt unsuccessful after 29639 (greater than max timeout of 20000). Resetting connection and trying again with a new connection.
