Github user tophei commented on the issue:
https://github.com/apache/zookeeper/pull/150
We were recently run into this issue in cloud env.
Specifically, we have a kafka cluster setup in cloud, and it turns out to
be a failure when the zookeeper node is replaced, namely the ip changed while
the hostname remains same. We have patched this changes but it seems still not
works. While the new zk node ip can be re-resolved and kafka can establish
connection to zk node, but zk client seems not able to read content from the
connection. Anyone encounters similar issue?
Pasted the log from kafka server.
[2018-05-11 03:47:46,806] WARN SASL configuration failed:
javax.security.auth.login.LoginException: No JAAS configuration section named
'Client' was found in specified JAAS configuration file:
'/usr/share/kafka/config/kafka_server_jaas.
conf'. Will continue connection to Zookeeper server without SASL
authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)
[2018-05-11 03:47:46,806] ERROR [ZooKeeperClient] Auth failed.
(kafka.zookeeper.ZooKeeperClient)
[2018-05-11 03:47:46,806] ERROR [ZooKeeperClient] Auth failed.
(kafka.zookeeper.ZooKeeperClient)
[2018-05-11 03:47:46,806] INFO Opening socket connection to server
host.name/10.147.164.83:2181 (org.apache.zookeeper.ClientCnxn)
[2018-05-11 03:47:49,815] WARN Session 0x1634d42d88a001e for server null,
unexpected error, closing socket connection and attempting reconnect
(org.apache.zookeeper.ClientCnxn)
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[2018-05-11 03:47:50,667] INFO Resolving again hostname: host.name
(org.apache.zookeeper.client.StaticHostProvider)
[2018-05-11 03:47:50,667] WARN Cannot re-resolve server:
host.name/10.147.164.83:2181 (org.apache.zookeeper.client.StaticHostProvider)
java.net.UnknownHostException: host.name
at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:142)
at
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
[2018-05-11 03:47:51,668] WARN SASL configuration failed:
javax.security.auth.login.LoginException: No JAAS configuration section named
'Client' was found in specified JAAS configuration file:
'/usr/share/kafka/config/kafka_server_jaas.
conf'. Will continue connection to Zookeeper server without SASL
authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)
[2018-05-11 03:47:51,668] ERROR [ZooKeeperClient] Auth failed.
(kafka.zookeeper.ZooKeeperClient)
[2018-05-11 03:47:51,668] ERROR [ZooKeeperClient] Auth failed.
(kafka.zookeeper.ZooKeeperClient)
[2018-05-11 03:47:51,668] INFO Opening socket connection to server
host.name/10.147.164.83:2181 (org.apache.zookeeper.ClientCnxn)
[2018-05-11 03:47:52,822] WARN Session 0x1634d42d88a001e for server null,
unexpected error, closing socket connection and attempting reconnect
(org.apache.zookeeper.ClientCnxn)
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[2018-05-11 03:47:53,427] INFO Resolving again hostname: host.name
(org.apache.zookeeper.client.StaticHostProvider)
[2018-05-11 03:47:54,429] WARN SASL configuration failed:
javax.security.auth.login.LoginException: No JAAS configuration section named
'Client' was found in specified JAAS configuration file:
'/usr/share/kafka/config/kafka_server_jaas.
conf'. Will continue connection to Zookeeper server without SASL
authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)
[2018-05-11 03:47:54,429] ERROR [ZooKeeperClient] Auth failed.
(kafka.zookeeper.ZooKeeperClient)
[2018-05-11 03:47:54,429] ERROR [ZooKeeperClient] Auth failed.
(kafka.zookeeper.ZooKeeperClient)
[2018-05-11 03:47:54,429] INFO Opening socket connection to server
host.name/10.147.161.203:2181 (org.apache.zookeeper.ClientCnxn)
[2018-05-11 03:47:55,383] INFO Socket connection established to
host.name/10.147.161.203:2181, initiating session
(org.apache.zookeeper.ClientCnxn)
[2018-05-11 03:47:55,386] INFO Unable to read additional data from server
sessionid 0x1634d42d88a001e, likely server has closed socket, closing socket
connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
---
