The topology of our kafka cluster is 1 producer,3 broker servers and 2 consumers The whole system runs well for several days,and yesterday we found that the producer kept connect and disconnect and then re-connect to the zookeeper.
below are the log that I thought might be relevant * * *producer(10.18.10.4):* [2012-06-11 17:25:55,315] INFO Socket connection established to zk-node-1/ 10.18.10.1:2181, initiating session (org.apache.zookeeper.ClientCnxn) [2012-06-11 17:25:55,339] INFO Session establishment complete on serverzk-node-1/ 10.18.10.1:2181, sessionid = 0x1379e52e0de154d, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn) [2012-06-11 17:25:55,339] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient) [2012-06-11 17:27:00,220] INFO Client session timed out, have not heard from server in 46827ms for sessionid 0x1379e52e0de154d, closing socket connection and attempting reconnect (org.apache.zookeeper. ClientCnxn) [2012-06-11 17:27:00,320] INFO zookeeper state changed (Disconnected) (org.I0Itec.zkclient.ZkClient) [2012-06-11 17:27:00,988] INFO Unable to reconnect to ZooKeeper service, session 0x1379e52e0de154d has expired, closing socket connection (org.apache.zookeeper.ClientCnxn) *zk node 1(10.18.10.1):* 25230 2012-06-11 17:25:55,365 - INFO [CommitProcessor:1:NIOServerCnxn@1580] - Established session 0x1379e52e0de154d with negotiated timeout 6000 for client /10.18.10.4:36134 // 10.18.10.4 is producer 25231 2012-06-11 17:26:22,004 - INFO [CommitProcessor:1:NIOServerCnxn@1435] - Closed socket connection for client /10.18.10.4:36134 which had sessionid 0x1379e52e0de154d *zk node 2:* 2012-06-11 17:26:22,001 - INFO [SessionTracker:ZooKeeperServer@316] - Expiring session 0x1379e52e0de154d, timeout of 6000ms exceeded 2012-06-11 17:26:22,001 - INFO [ProcessThread:-1:PrepRequestProcessor@399] - Processed session termination for sessionid: 0x1379e52e0de154d 2012-06-11 17:27:01,011 - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@251] - Accepted socket connection from /10.18.10.4:25923 2012-06-11 17:27:01,012 - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@770] - Client attempting to renew session 0x1379e52e0de154d at /10.18.10.4:25923 2012-06-11 17:27:01,012 - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1573] - Invalid session 0x1379e52e0de154dfor client /10.18.10.4:25923, probably expired 2012-06-11 17:27:01,013 - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435] - Closed socket connection for client /10.18.10.4:25923 which had sessionid 0x1379e52e0de154d *zk node 3:* (content has been rotated when I remembered to copy the content from log,while it is similar with zk node 1 and 2) -- Best Regards ---------------------- 刘明敏 | mmLiu
