Re: Help Needed: Leadership Issue upon ZK Restart (ZooKeeper 3.4.9)

[Prasanth Mathialagan]

Is 1.1.1.143:3888 reachable from the host in which you see this error?

On Fri, May 11, 2018 at 3:11 PM, Raghav  wrote:

> Hi
>
> We have a 3 node zk ensemble as well as 3 node Kafka Cluster. They both are
> hosted on the same 3 VMs.
>
> Before Restart
> 1. We were on Kafka 0.10.2.1
>
> After Restart
> 1. We moved to Kafka 1.1
>
> We observe that Kafkas report leadership issues, and for lot of partitions
> Leader is -1. I see some logs in ZK that mainly point towards some
> connectivity issue around restart time.
>
> *We are stuck on this one for a while now, and neither rolling restart of
> ZK is helping. Can you please help or point us how we can debug this.*
>
> *2018-05-11_17:20:49.00305 2018-05-11 17:20:49,002 [myid:1] - INFO
> [WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message
> format version), 1 (n.leader), 0x20112 (n.zxid), 0x1 (n.round), LOOKING
> (n.state), 1 (n.sid), 0x2 (n.peerEpoch) LOOKING (my
> state)2018-05-11_17:20:49.01201
> 2018-05-11 17:20:49,010 [myid:1] - WARN
> [WorkerSender[myid=1]:QuorumCnxManager@400] - Cannot open channel to 2 at
> election address /1.1.1.143:3888
> 
> 2018-05-11_17:20:49.01203 java.net.ConnectException: Connection
> refused
> 2018-05-11_17:20:49.01203   at
> java.net.PlainSocketImpl.socketConnect(Native
> Method)
> 2018-05-11_17:20:49.01203   at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:
> 345)
> 2018-05-11_17:20:49.01203   at
> java.net.AbstractPlainSocketImpl.connectToAddress(
> AbstractPlainSocketImpl.java:206)
> 2018-05-11_17:20:49.01204   at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> 2018-05-11_17:20:49.01204   at
> java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> 2018-05-11_17:20:49.01204   at
> java.net.Socket.connect(Socket.java:589)
> 2018-05-11_17:20:49.01204   at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.
> connectOne(QuorumCnxManager.java:381)
> 2018-05-11_17:20:49.01204   at
> org.apache.zookeeper.server.quorum.QuorumCnxManager.
> toSend(QuorumCnxManager.java:354)
> 2018-05-11_17:20:49.01205   at
> org.apache.zookeeper.server.quorum.FastLeaderElection$
> Messenger$WorkerSender.process(FastLeaderElection.java:452)
> 2018-05-11_17:20:49.01205   at
> org.apache.zookeeper.server.quorum.FastLeaderElection$
> Messenger$WorkerSender.run(FastLeaderElection.java:433)
> 2018-05-11_17:20:49.01206   at java.lang.Thread.run(Thread.java:745)*
>
>
> Raghav
>

Help Needed: Leadership Issue upon ZK Restart (ZooKeeper 3.4.9)

[Raghav]

Hi

We have a 3 node zk ensemble as well as 3 node Kafka Cluster. They both are
hosted on the same 3 VMs.

Before Restart
1. We were on Kafka 0.10.2.1

After Restart
1. We moved to Kafka 1.1

We observe that Kafkas report leadership issues, and for lot of partitions
Leader is -1. I see some logs in ZK that mainly point towards some
connectivity issue around restart time.

*We are stuck on this one for a while now, and neither rolling restart of
ZK is helping. Can you please help or point us how we can debug this.*

*2018-05-11_17:20:49.00305 2018-05-11 17:20:49,002 [myid:1] - INFO
[WorkerReceiver[myid=1]:FastLeaderElection@600] - Notification: 1 (message
format version), 1 (n.leader), 0x20112 (n.zxid), 0x1 (n.round), LOOKING
(n.state), 1 (n.sid), 0x2 (n.peerEpoch) LOOKING (my
state)2018-05-11_17:20:49.01201
2018-05-11 17:20:49,010 [myid:1] - WARN
[WorkerSender[myid=1]:QuorumCnxManager@400] - Cannot open channel to 2 at
election address /1.1.1.143:3888

2018-05-11_17:20:49.01203 java.net.ConnectException: Connection
refused
2018-05-11_17:20:49.01203   at
java.net.PlainSocketImpl.socketConnect(Native
Method)
2018-05-11_17:20:49.01203   at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
2018-05-11_17:20:49.01203   at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
2018-05-11_17:20:49.01204   at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
2018-05-11_17:20:49.01204   at
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
2018-05-11_17:20:49.01204   at
java.net.Socket.connect(Socket.java:589)
2018-05-11_17:20:49.01204   at
org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxManager.java:381)
2018-05-11_17:20:49.01204   at
org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager.java:354)
2018-05-11_17:20:49.01205   at
org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:452)
2018-05-11_17:20:49.01205   at
org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:433)
2018-05-11_17:20:49.01206   at java.lang.Thread.run(Thread.java:745)*


Raghav

Re: Apache ZooKeeper meetup May 9th in Palo Alto?

[Patrick Hunt]

Sorry, it wasn't recorded. Perhaps the presenters can post their slides.

Regards,

Patrick

On Fri, May 11, 2018 at 10:02 AM, Edward Ribeiro 
wrote:

> Oh, I missed the streaming. :(
>
> Was it recorded? The slides are available anywhere?
>
> On Thu, May 10, 2018 at 2:22 PM, Srikanth Viswanathan <
> srikant...@gmail.com>
> wrote:
>
> > Thanks for the presentations! Will the slides be posted somewhere?
> >
> > On Thu, May 10, 2018, 09:36 Prasanth Mathialagan <
> > prasanthmathiala...@gmail.com> wrote:
> >
> > > Gosh!! I missed it!! May be next time :)
> > >
> > > On Thu, May 10, 2018 at 8:24 AM, Andor Molnar 
> > wrote:
> > >
> > > > Thanks everybody for joining.
> > > > It was great to see you folks in person and talk.
> > > >
> > > > Regards,
> > > > Andor
> > > >
> > > >
> > > >
> > > > On Wed, May 9, 2018 at 9:44 PM, Patrick Hunt 
> wrote:
> > > >
> > > > > On Wed, May 9, 2018 at 9:24 PM, Jeff Widman 
> > > wrote:
> > > > >
> > > > > > Many thanks to Patrick, Andor and the rest of the Cloudera team
> for
> > > > > hosting
> > > > > > us tonight. I appreciated the chance to compare notes with other
> > > users
> > > > > and
> > > > > > also discuss some of where the future of Zookeeper is heading.
> > > > > >
> > > > > >
> > > > > Thanks everyone for attending. It was great to see some old as well
> > as
> > > > new
> > > > > faces.
> > > > >
> > > > >
> > > > > > Cheers,
> > > > > > Jeff
> > > > > >
> > > > > > PS: Apologies to the remote folks for the dead sound at the
> > end--the
> > > > > > hangout connection to the conference room sound system died, and
> > > nobody
> > > > > in
> > > > > > the room knew how to reconnect it.
> > > > > >
> > > > > >
> > > > > Yea, sorry about that. IT did some magic to connect the hangout
> into
> > > the
> > > > > internal audio system of the room and I wasn't able to figure out
> the
> > > > codes
> > > > > they used.
> > > > >
> > > > > Regards,
> > > > >
> > > > > Patrick
> > > > >
> > > > >
> > > > > >
> > > > > > On Fri, Apr 27, 2018 at 6:40 PM, Srikanth Viswanathan <
> > > > > > srikant...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Confirming attendance from Seattle as well. Looking forward to
> > the
> > > > > > > presentations. Particularly excited for the containers talk!
> > > > > > >
> > > > > > > On Thu, Apr 26, 2018, 10:29 Patrick Hunt 
> > wrote:
> > > > > > >
> > > > > > > > Ok, great. I have two speakers already lined up: one on ZK
> and
> > > > > > containers
> > > > > > > > and another on ZK failure modes and recovery. If you haven't
> > > > reached
> > > > > > out
> > > > > > > to
> > > > > > > > me yet and you have something to talk about please LMK asap.
> > > > > > > >
> > > > > > > > I believe we will have video conference available but I'm not
> > > > > entirely
> > > > > > > sure
> > > > > > > > yet - I will try.
> > > > > > > >
> > > > > > > > I'll finalize things and send out a more detailed agenda.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > Patrick
> > > > > > > >
> > > > > > > > On Thu, Apr 26, 2018 at 1:33 AM, Shivam Goel <
> > > shivam@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Count me in !!
> > > > > > > > >
> > > > > > > > > On Mon, Apr 23, 2018 at 10:08 AM Patrick Hunt <
> > > ph...@apache.org>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi folks. I am interested in hosting a ZooKeeper meetup
> May
> > > 9th
> > > > > in
> > > > > > > > > > Cloudera's Palo Alto offices. It's been a while since we
> > last
> > > > got
> > > > > > > > > together,
> > > > > > > > > > lots of recent changes and some big plans, new additions
> to
> > > the
> > > > > PMC
> > > > > > > and
> > > > > > > > > > committer lists, new contributors.
> > > > > > > > > >
> > > > > > > > > > I was hoping to use the mailing lists to gauge interest.
> > > Please
> > > > > > reply
> > > > > > > > if
> > > > > > > > > > you
> > > > > > > > > > think you would be able to attend or would prefer a
> > different
> > > > > date.
> > > > > > > > Also
> > > > > > > > > > let me know if there's something you would like to
> present
> > to
> > > > the
> > > > > > > > group.
> > > > > > > > > > Food
> > > > > > > > > > and beer will be provided.
> > > > > > > > > >
> > > > > > > > > > Looking forward to hearing from everyone.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > >
> > > > > > > > > > Patrick
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > *Jeff Widman*
> > > > > > jeffwidman.com  | 740-WIDMAN-J
> > > (943-6265)
> > > > > > <><
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache ZooKeeper meetup May 9th in Palo Alto?

[Edward Ribeiro]

Oh, I missed the streaming. :(

Was it recorded? The slides are available anywhere?

On Thu, May 10, 2018 at 2:22 PM, Srikanth Viswanathan 
wrote:

> Thanks for the presentations! Will the slides be posted somewhere?
>
> On Thu, May 10, 2018, 09:36 Prasanth Mathialagan <
> prasanthmathiala...@gmail.com> wrote:
>
> > Gosh!! I missed it!! May be next time :)
> >
> > On Thu, May 10, 2018 at 8:24 AM, Andor Molnar 
> wrote:
> >
> > > Thanks everybody for joining.
> > > It was great to see you folks in person and talk.
> > >
> > > Regards,
> > > Andor
> > >
> > >
> > >
> > > On Wed, May 9, 2018 at 9:44 PM, Patrick Hunt  wrote:
> > >
> > > > On Wed, May 9, 2018 at 9:24 PM, Jeff Widman 
> > wrote:
> > > >
> > > > > Many thanks to Patrick, Andor and the rest of the Cloudera team for
> > > > hosting
> > > > > us tonight. I appreciated the chance to compare notes with other
> > users
> > > > and
> > > > > also discuss some of where the future of Zookeeper is heading.
> > > > >
> > > > >
> > > > Thanks everyone for attending. It was great to see some old as well
> as
> > > new
> > > > faces.
> > > >
> > > >
> > > > > Cheers,
> > > > > Jeff
> > > > >
> > > > > PS: Apologies to the remote folks for the dead sound at the
> end--the
> > > > > hangout connection to the conference room sound system died, and
> > nobody
> > > > in
> > > > > the room knew how to reconnect it.
> > > > >
> > > > >
> > > > Yea, sorry about that. IT did some magic to connect the hangout into
> > the
> > > > internal audio system of the room and I wasn't able to figure out the
> > > codes
> > > > they used.
> > > >
> > > > Regards,
> > > >
> > > > Patrick
> > > >
> > > >
> > > > >
> > > > > On Fri, Apr 27, 2018 at 6:40 PM, Srikanth Viswanathan <
> > > > > srikant...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Confirming attendance from Seattle as well. Looking forward to
> the
> > > > > > presentations. Particularly excited for the containers talk!
> > > > > >
> > > > > > On Thu, Apr 26, 2018, 10:29 Patrick Hunt 
> wrote:
> > > > > >
> > > > > > > Ok, great. I have two speakers already lined up: one on ZK and
> > > > > containers
> > > > > > > and another on ZK failure modes and recovery. If you haven't
> > > reached
> > > > > out
> > > > > > to
> > > > > > > me yet and you have something to talk about please LMK asap.
> > > > > > >
> > > > > > > I believe we will have video conference available but I'm not
> > > > entirely
> > > > > > sure
> > > > > > > yet - I will try.
> > > > > > >
> > > > > > > I'll finalize things and send out a more detailed agenda.
> > > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > > Patrick
> > > > > > >
> > > > > > > On Thu, Apr 26, 2018 at 1:33 AM, Shivam Goel <
> > shivam@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Count me in !!
> > > > > > > >
> > > > > > > > On Mon, Apr 23, 2018 at 10:08 AM Patrick Hunt <
> > ph...@apache.org>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi folks. I am interested in hosting a ZooKeeper meetup May
> > 9th
> > > > in
> > > > > > > > > Cloudera's Palo Alto offices. It's been a while since we
> last
> > > got
> > > > > > > > together,
> > > > > > > > > lots of recent changes and some big plans, new additions to
> > the
> > > > PMC
> > > > > > and
> > > > > > > > > committer lists, new contributors.
> > > > > > > > >
> > > > > > > > > I was hoping to use the mailing lists to gauge interest.
> > Please
> > > > > reply
> > > > > > > if
> > > > > > > > > you
> > > > > > > > > think you would be able to attend or would prefer a
> different
> > > > date.
> > > > > > > Also
> > > > > > > > > let me know if there's something you would like to present
> to
> > > the
> > > > > > > group.
> > > > > > > > > Food
> > > > > > > > > and beer will be provided.
> > > > > > > > >
> > > > > > > > > Looking forward to hearing from everyone.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > >
> > > > > > > > > Patrick
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > *Jeff Widman*
> > > > > jeffwidman.com  | 740-WIDMAN-J
> > (943-6265)
> > > > > <><
> > > > >
> > > >
> > >
> >
>

Re: client cannot connect to zookeeper after node replacement

[Andor Molnar]

Hi Jeff,

There's another patch which is supposed to be the successor of #151 here:

https://github.com/apache/zookeeper/pull/451

Would you please try this one too and make sure it fixes the problem?

Thanks,
Andor




On Fri, May 11, 2018 at 5:13 AM, 徐朝晖  wrote:

> Hi there,
>
>
>
> We were recently running into an issue in cloud env.
> Specifically, we have a kafka cluster setup in cloud, and it turns out to
> be a failure when the zookeeper node is replaced, namely the ip changed
> while the hostname remains same. We have patched PR150 (
> https://github.com/apache/zookeeper/pull/150/files) but it seems still
> not works. While the new zk node ip can be re-resolved and zk client in
> kafka can establish connection to zk node, but the client seems not able to
> read content from the connection. Anyone encounters similar issue?
>
> Pasted the log from kafka server.
>
> [2018-05-11 03:47:46,806] WARN SASL configuration failed:
> javax.security.auth.login.LoginException: No JAAS configuration section
> named 'Client' was found in specified JAAS configuration file:
> '/usr/share/kafka/config/kafka_server_jaas.
> conf'. Will continue connection to Zookeeper server without SASL
> authentication, if Zookeeper server allows it. (org.apache.zookeeper.
> ClientCnxn)
> [2018-05-11 03:47:46,806] ERROR [ZooKeeperClient] Auth failed.
> (kafka.zookeeper.ZooKeeperClient)
> [2018-05-11 03:47:46,806] ERROR [ZooKeeperClient] Auth failed.
> (kafka.zookeeper.ZooKeeperClient)
> [2018-05-11 03:47:46,806] INFO Opening socket connection to server
> host.name/10.147.164.83:2181 (org.apache.zookeeper.ClientCnxn)
> [2018-05-11 03:47:49,815] WARN Session 0x1634d42d88a001e for server null,
> unexpected error, closing socket connection and attempting reconnect
> (org.apache.zookeeper.ClientCnxn)
> java.net.NoRouteToHostException: No route to host
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(
> ClientCnxnSocketNIO.java:361)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
> [2018-05-11 03:47:50,667] INFO Resolving again hostname: host.name
> (org.apache.zookeeper.client.StaticHostProvider)
> [2018-05-11 03:47:50,667] WARN Cannot re-resolve server:
> host.name/10.147.164.83:2181 (org.apache.zookeeper.client.
> StaticHostProvider)
> java.net.UnknownHostException: host.name
> at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
> at java.net.InetAddress.getAllByName(InetAddress.java:1192)
> at java.net.InetAddress.getAllByName(InetAddress.java:1126)
> at org.apache.zookeeper.client.StaticHostProvider.next(
> StaticHostProvider.java:142)
> at org.apache.zookeeper.ClientCnxn$SendThread.
> startConnect(ClientCnxn.java:997)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
> [2018-05-11 03:47:51,668] WARN SASL configuration failed:
> javax.security.auth.login.LoginException: No JAAS configuration section
> named 'Client' was found in specified JAAS configuration file:
> '/usr/share/kafka/config/kafka_server_jaas.
> conf'. Will continue connection to Zookeeper server without SASL
> authentication, if Zookeeper server allows it. (org.apache.zookeeper.
> ClientCnxn)
> [2018-05-11 03:47:51,668] ERROR [ZooKeeperClient] Auth failed.
> (kafka.zookeeper.ZooKeeperClient)
> [2018-05-11 03:47:51,668] ERROR [ZooKeeperClient] Auth failed.
> (kafka.zookeeper.ZooKeeperClient)
> [2018-05-11 03:47:51,668] INFO Opening socket connection to server
> host.name/10.147.164.83:2181 (org.apache.zookeeper.ClientCnxn)
> [2018-05-11 03:47:52,822] WARN Session 0x1634d42d88a001e for server null,
> unexpected error, closing socket connection and attempting reconnect
> (org.apache.zookeeper.ClientCnxn)
> java.net.NoRouteToHostException: No route to host
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(
> ClientCnxnSocketNIO.java:361)
> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
> [2018-05-11 03:47:53,427] INFO Resolving again hostname: host.name
> (org.apache.zookeeper.client.StaticHostProvider)
> [2018-05-11 03:47:54,429] WARN SASL configuration failed:
> javax.security.auth.login.LoginException: No JAAS configuration section
> named 'Client' was found in specified JAAS configuration file:
> '/usr/share/kafka/config/kafka_server_jaas.
> conf'. Will continue connection to Zookeeper server without SASL
> authentication, if Zookeeper server allows it. (org.apache.zookeeper.
> ClientCnxn)
> [2018-05-11 03:47:54,429] ERROR [ZooKeeperClient] Auth failed.
> (kafka.zookeeper.ZooKeeperClient)
> [2018-05-11 03:47:54,429] ERROR [ZooKeeperClient] Auth failed.
> (kafka.zookeeper.ZooKeeperClient)
> [2018-05-11 03:47:54,429] INFO Opening socket connection to server
> host.name/10.147.161.203:2181 (org.apache.

client cannot connect to zookeeper after node replacement

[徐朝晖]

Hi there,



We were recently running into an issue in cloud env.
Specifically, we have a kafka cluster setup in cloud, and it turns out to be a 
failure when the zookeeper node is replaced, namely the ip changed while the 
hostname remains same. We have patched PR150 
(https://github.com/apache/zookeeper/pull/150/files) but it seems still not 
works. While the new zk node ip can be re-resolved and zk client in kafka can 
establish connection to zk node, but the client seems not able to read content 
from the connection. Anyone encounters similar issue?

Pasted the log from kafka server.

[2018-05-11 03:47:46,806] WARN SASL configuration failed: 
javax.security.auth.login.LoginException: No JAAS configuration section named 
'Client' was found in specified JAAS configuration file: 
'/usr/share/kafka/config/kafka_server_jaas.
conf'. Will continue connection to Zookeeper server without SASL 
authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)
[2018-05-11 03:47:46,806] ERROR [ZooKeeperClient] Auth failed. 
(kafka.zookeeper.ZooKeeperClient)
[2018-05-11 03:47:46,806] ERROR [ZooKeeperClient] Auth failed. 
(kafka.zookeeper.ZooKeeperClient)
[2018-05-11 03:47:46,806] INFO Opening socket connection to server 
host.name/10.147.164.83:2181 (org.apache.zookeeper.ClientCnxn)
[2018-05-11 03:47:49,815] WARN Session 0x1634d42d88a001e for server null, 
unexpected error, closing socket connection and attempting reconnect 
(org.apache.zookeeper.ClientCnxn)
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[2018-05-11 03:47:50,667] INFO Resolving again hostname: host.name 
(org.apache.zookeeper.client.StaticHostProvider)
[2018-05-11 03:47:50,667] WARN Cannot re-resolve server: 
host.name/10.147.164.83:2181 (org.apache.zookeeper.client.StaticHostProvider)
java.net.UnknownHostException: host.name
at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:142)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:997)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1060)
[2018-05-11 03:47:51,668] WARN SASL configuration failed: 
javax.security.auth.login.LoginException: No JAAS configuration section named 
'Client' was found in specified JAAS configuration file: 
'/usr/share/kafka/config/kafka_server_jaas.
conf'. Will continue connection to Zookeeper server without SASL 
authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)
[2018-05-11 03:47:51,668] ERROR [ZooKeeperClient] Auth failed. 
(kafka.zookeeper.ZooKeeperClient)
[2018-05-11 03:47:51,668] ERROR [ZooKeeperClient] Auth failed. 
(kafka.zookeeper.ZooKeeperClient)
[2018-05-11 03:47:51,668] INFO Opening socket connection to server 
host.name/10.147.164.83:2181 (org.apache.zookeeper.ClientCnxn)
[2018-05-11 03:47:52,822] WARN Session 0x1634d42d88a001e for server null, 
unexpected error, closing socket connection and attempting reconnect 
(org.apache.zookeeper.ClientCnxn)
java.net.NoRouteToHostException: No route to host
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at 
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:361)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1141)
[2018-05-11 03:47:53,427] INFO Resolving again hostname: host.name 
(org.apache.zookeeper.client.StaticHostProvider)
[2018-05-11 03:47:54,429] WARN SASL configuration failed: 
javax.security.auth.login.LoginException: No JAAS configuration section named 
'Client' was found in specified JAAS configuration file: 
'/usr/share/kafka/config/kafka_server_jaas.
conf'. Will continue connection to Zookeeper server without SASL 
authentication, if Zookeeper server allows it. (org.apache.zookeeper.ClientCnxn)
[2018-05-11 03:47:54,429] ERROR [ZooKeeperClient] Auth failed. 
(kafka.zookeeper.ZooKeeperClient)
[2018-05-11 03:47:54,429] ERROR [ZooKeeperClient] Auth failed. 
(kafka.zookeeper.ZooKeeperClient)
[2018-05-11 03:47:54,429] INFO Opening socket connection to server 
host.name/10.147.161.203:2181 (org.apache.zookeeper.ClientCnxn)
[2018-05-11 03:47:55,383] INFO Socket connection established to 
host.name/10.147.161.203:2181, initiating session 
(org.apache.zookeeper.ClientCnxn)
[2018-05-11 03:47:55,386] INFO Unable to read additional data from server 
sessionid 0x1634d42d88a001e, likely server has closed socket, closing socket 
connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)




this ends