We've occasional connection issues from client (Storm Supervisor) and
zk running in EC2 VPC, here is the log:
Storm Supervisor:
2014-05-11 00:09:52 o.a.z.ClientCnxn [INFO] Unable to read additional
data from server sessionid 0x244ec3718bb0089, likely server has closed
socket, closing socket connection and attempting reconnect
2014-05-11 00:09:52 c.n.c.f.s.ConnectionStateManager [INFO] State
change: SUSPENDED
2014-05-11 00:09:52 c.n.c.f.s.ConnectionStateManager [WARN] There are
no ConnectionStateListeners registered.
2014-05-11 00:09:52 b.s.cluster [WARN] Received event
:disconnected::none: with disconnected Zookeeper.
One of the zk (out of three):
2014-05-11 00:09:51,696 [myid:2] - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@197] -
Accepted socket connection from /10.0.13.21:58611
2014-05-11 00:09:51,697 [myid:2] - WARN
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@793] -
Connection request from old client /10.0.13.21:58611; will be dropped
if server is in r-o mode
2014-05-11 00:09:51,697 [myid:2] - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@832] -
Client attempting to renew session 0x244ec3718bb008f at
/10.0.13.21:58611
2014-05-11 00:09:51,698 [myid:2] - INFO
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@595] -
Established session 0x244ec3718bb008f with negotiated timeout 20000
for client /10.0.13.21:58611
2014-05-11 00:09:52,092 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Leader@490] - Shutting down
2014-05-11 00:09:52,109 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:Leader@496] - Shutdown called
java.lang.Exception: shutdown Leader! reason: Only 1 followers, need 1
at org.apache.zookeeper.server.quorum.Leader.shutdown(Leader.java:496)
at org.apache.zookeeper.server.quorum.Leader.lead(Leader.java:471)
at
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:753)
2014-05-11 00:09:52,121 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1001] - Closed
socket connection for client /10.0.13.61:39639 which had sessionid
0x344ec37d11b0071
2014-05-11 00:09:52,121 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1001] - Closed
socket connection for client /10.0.13.62:36706 which had sessionid
0x144ec37184100e0
2014-05-11 00:09:52,122 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1001] - Closed
socket connection for client /10.0.13.63:49909 which had sessionid
0x144ec37184100f0
2014-05-11 00:09:52,122 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1001] - Closed
socket connection for client /10.0.13.63:49240 which had sessionid
0x244ec3718bb0105
2014-05-11 00:09:52,122 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1001] - Closed
socket connection for client /10.0.13.61:39636 which had sessionid
0x144ec37184100f4
2014-05-11 00:09:52,123 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1001] - Closed
socket connection for client /10.0.13.62:47862 which had sessionid
0x244ec3718bb0089
2014-05-11 00:09:52,123 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1001] - Closed
socket connection for client /10.0.13.62:36709 which had sessionid
0x344ec37d11b00e8
2014-05-11 00:09:52,124 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1001] - Closed
socket connection for client /10.0.13.61:39633 which had sessionid
0x144ec37184100f5
2014-05-11 00:09:52,124 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1001] - Closed
socket connection for client /10.0.13.62:36710 which had sessionid
0x144ec37184100e3
2014-05-11 00:09:52,125 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1001] - Closed
socket connection for client /10.0.13.62:37290 which had sessionid
0x244ec3718bb00fc
2014-05-11 00:09:52,125 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1001] - Closed
socket connection for client /10.0.13.61:39640 which had sessionid
0x344ec37d11b00f2
2014-05-11 00:09:52,125 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1001] - Closed
socket connection for client /10.0.13.63:49910 which had sessionid
0x344ec37d11b00e0
2014-05-11 00:09:52,126 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1001] - Closed
socket connection for client /10.0.13.63:49906 which had sessionid
0x344ec37d11b0064
2014-05-11 00:09:52,126 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1001] - Closed
socket connection for client /10.0.13.61:39634 which had sessionid
0x344ec37d11b00e1
2014-05-11 00:09:52,126 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1001] - Closed
socket connection for client /10.0.13.63:49908 which had sessionid
0x144ec37184100ec
2014-05-11 00:09:52,127 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:NIOServerCnxn@1001] - Closed
socket connection for client /10.0.13.21:58611 which had sessionid
0x244ec3718bb008f
2014-05-11 00:09:52,127 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ZooKeeperServer@419] -
shutting down
2014-05-11 00:09:52,128 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:SessionTrackerImpl@225] -
Shutting down
2014-05-11 00:09:52,128 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:PrepRequestProcessor@743] -
Shutting down
2014-05-11 00:09:52,138 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:ProposalRequestProcessor@88]
- Shutting down
2014-05-11 00:09:52,138 [myid:2] - INFO
[QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181:CommitProcessor@181] -
Shutting down
Anyway I can troubleshoot the problem?
Thanks
