Date: Tue, 10 May 2016 17:49:59 +0200
From: ra...@gruchalski.com
To: users@kafka.apache.org
Subject: Re: Zookeeper dies ... Kafka server unable to connect
Are you sure you’re getting the same IP address?
Regarding zookeeper connection being closed, is kubernetes doing a soft
shutdown of your container? If so, zookeeper is asked politely to stop.
–
Best regards,
Radek Gruchalski
radek@gruchalski.commailto:ra...@gruchalski.com
de.linkedin.com/in/radgruchalski
+4917685656526
Confidentiality:
This communication is intended for the above-named person and may be
confidential and/or legally privileged.
If it has come to you in error you must take no action based on it, nor must
you copy or show it to anyone; please delete/destroy and inform the sender
immediately.
On May 10, 2016 at 5:47:24 PM, Paolo Patierno (ppatie...@live.com) wrote:
Hi all,
experiencing with Kafka on Kubernetes I have the following error on Kafka server reconnection ...
A cluster with one zookeeper and two kafka server ... I turn off the zookeeper pod but kubernetes restart it and guaratees the same IP address for it but the kafka server starts to retry connection failing with following trace :
[2016-05-10 15:40:55,046] WARN Session 0x1549b308dd20002 for server 10.0.0.184/10.0.0.184:2181, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
[2016-05-10 15:40:55,149] INFO zookeeper state changed (Disconnected)
(org.I0Itec.zkclient.ZkClient)
[2016-05-10 15:40:57,093] INFO Opening socket connection to server
10.0.0.184/10.0.0.184:2181. Will not attempt to authenticate using SASL
(unknown error) (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:40:57,093] INFO Socket connection established to
10.0.0.184/10.0.0.184:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:40:57,158] INFO Unable to read additional data from server
sessionid 0x1549b308dd20002, likely server has closed socket, closing socket
connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:40:58,936] INFO Opening socket connection to server
10.0.0.184/10.0.0.184:2181. Will not attempt to authenticate using SASL
(unknown error) (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:40:58,936] INFO Socket connection established to
10.0.0.184/10.0.0.184:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:40:58,937] INFO Unable to read additional data from server
sessionid 0x1549b308dd20002, likely server has closed socket, closing socket
connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:00,845] INFO Opening socket connection to server
10.0.0.184/10.0.0.184:2181. Will not attempt to authenticate using SASL
(unknown error) (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:00,845] INFO Socket connection established to
10.0.0.184/10.0.0.184:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:00,846] INFO Unable to read additional data from server
sessionid 0x1549b308dd20002, likely server has closed socket, closing socket
connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:02,071] INFO Opening socket connection to server
10.0.0.184/10.0.0.184:2181. Will not attempt to authenticate using SASL
(unknown error) (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:02,071] INFO Socket connection established to
10.0.0.184/10.0.0.184:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:02,072] INFO Unable to read additional data from server
sessionid 0x1549b308dd20002, likely server has closed socket, closing socket
connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:03,336] INFO Opening socket connection to server
10.0.0.184/10.0.0.184:2181. Will not attempt to authenticate using SASL
(unknown error) (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:03,336] INFO Socket connection established to
10.0.0.184/10.0.0.184:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:03,337] INFO Unable to read additional data from server
sessionid 0x1549b308dd20002, likely server has closed socket, closing socket
connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:05,121] INFO Opening socket connection to server
10.0.0.184/10.0.0.184:2181. Will not attempt to authenticate using SASL
(unknown error) (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:05,121] INFO Socket connection established to
10.0.0.184/10.0.0.184:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2016-05-10 15:41:05,122] INFO Unable to read additional data from server
sessionid 0x1549b308dd20002, likely server has closed socket, closing socket
connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
You can see when the first zookeeper dies and connection is lost ... and all the retries by kafka server in order to connect to the new one (same IP, same port).
Why the zookeeper server closes the connection (I can see FIN ACK frames on Wireshark)
Thanks,
Paolo.
Paolo PatiernoSenior Software Engineer (IoT) @ Red Hat
Microsoft MVP on Windows Embedded & IoTMicrosoft Azure Advisor
Twitter : @ppatierno
Linkedin : paolopatierno
Blog : DevExperience
