possible bug in zookeeper ?

2010-09-14 Thread Yatir Ben Shlomo
Hi I am using solrCloud which uses an ensemble of 3 zookeeper instances.

I am performing survivability  tests:
Taking one of the zookeeper instances down I would expect the client to use a 
different zookeeper server instance.

But as you can see in the below logs attached
Depending on which instance I choose to take down (in my case,  the last one in 
the list of zookeeper servers)
the client is constantly insisting on the same zookeeper server (Attempting 
connection to server zook3/192.168.252.78:2181)
and not switching to a different one
the problem seems to arrive from ClientCnxn.java
Any one has an idea on this ?

Solr cloud currently is using  zookeeper-3.2.2.jar
Is this a know bug that was fixed in later versions ?( 3.3.1)

Thanks in advance,
Yatir


Logs:

Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
WARNING: Ignoring exception during shutdown input
java.nio.channels.ClosedChannelException
at 
sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at 
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999)
at 
org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
WARNING: Ignoring exception during shutdown output
java.nio.channels.ClosedChannelException
at 
sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at 
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):1004)
at 
org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info
INFO: Attempting connection to server zook3/192.168.252.78:2181
Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
WARNING: Exception closing session 0x32b105244a20001 to 
sun.nio.ch.selectionkeyi...@3ca58cbf
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at 
org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933)
Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
WARNING: Ignoring exception during shutdown input
java.nio.channels.ClosedChannelException
at 
sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at 
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999)
at 
org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
WARNING: Ignoring exception during shutdown output
java.nio.channels.ClosedChannelException
at 
sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at 
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):1004)
at 
org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info
INFO: Attempting connection to server zook3/192.168.252.78:2181
Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
WARNING: Exception closing session 0x32b105244a2 to 
sun.nio.ch.selectionkeyi...@3960f81b
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at 
org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933)
Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
WARNING: Ignoring exception during shutdown input
java.nio.channels.ClosedChannelException
at 
sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at 
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999)
at 
org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
WARNING: Ignoring exception during shutdown output
java.nio.channels.ClosedChannelException
at 
sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at 
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):1004)
at 

Re: possible bug in zookeeper ?

2010-09-14 Thread Ted Dunning
What was the list of servers that was given originally to open the
connection to ZK?

On Tue, Sep 14, 2010 at 6:15 AM, Yatir Ben Shlomo yat...@outbrain.comwrote:

 Hi I am using solrCloud which uses an ensemble of 3 zookeeper instances.

 I am performing survivability  tests:
 Taking one of the zookeeper instances down I would expect the client to use
 a different zookeeper server instance.

 But as you can see in the below logs attached
 Depending on which instance I choose to take down (in my case,  the last
 one in the list of zookeeper servers)
 the client is constantly insisting on the same zookeeper server (Attempting
 connection to server zook3/192.168.252.78:2181)
 and not switching to a different one
 the problem seems to arrive from ClientCnxn.java
 Any one has an idea on this ?

 Solr cloud currently is using  zookeeper-3.2.2.jar
 Is this a know bug that was fixed in later versions ?( 3.3.1)

 Thanks in advance,
 Yatir


 Logs:

 Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown input
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
 Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown output
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):1004)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info
 INFO: Attempting connection to server zook3/192.168.252.78:2181
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Exception closing session 0x32b105244a20001 to
 sun.nio.ch.selectionkeyi...@3ca58cbf
 java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
at
 sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown input
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown output
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):1004)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info
 INFO: Attempting connection to server zook3/192.168.252.78:2181
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Exception closing session 0x32b105244a2 to
 sun.nio.ch.selectionkeyi...@3960f81b
 java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
at
 sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown input
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown output
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at 

RE: possible bug in zookeeper ?

2010-09-14 Thread Yatir Ben Shlomo
zook1:2181,zook2:2181,zook3:2181


-Original Message-
From: Ted Dunning [mailto:ted.dunn...@gmail.com] 
Sent: Tuesday, September 14, 2010 4:11 PM
To: zookeeper-user@hadoop.apache.org
Subject: Re: possible bug in zookeeper ?

What was the list of servers that was given originally to open the
connection to ZK?

On Tue, Sep 14, 2010 at 6:15 AM, Yatir Ben Shlomo yat...@outbrain.comwrote:

 Hi I am using solrCloud which uses an ensemble of 3 zookeeper instances.

 I am performing survivability  tests:
 Taking one of the zookeeper instances down I would expect the client to use
 a different zookeeper server instance.

 But as you can see in the below logs attached
 Depending on which instance I choose to take down (in my case,  the last
 one in the list of zookeeper servers)
 the client is constantly insisting on the same zookeeper server (Attempting
 connection to server zook3/192.168.252.78:2181)
 and not switching to a different one
 the problem seems to arrive from ClientCnxn.java
 Any one has an idea on this ?

 Solr cloud currently is using  zookeeper-3.2.2.jar
 Is this a know bug that was fixed in later versions ?( 3.3.1)

 Thanks in advance,
 Yatir


 Logs:

 Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown input
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
 Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown output
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):1004)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info
 INFO: Attempting connection to server zook3/192.168.252.78:2181
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Exception closing session 0x32b105244a20001 to
 sun.nio.ch.selectionkeyi...@3ca58cbf
 java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
at
 sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown input
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown output
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):1004)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info
 INFO: Attempting connection to server zook3/192.168.252.78:2181
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Exception closing session 0x32b105244a2 to
 sun.nio.ch.selectionkeyi...@3960f81b
 java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
at
 sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown input
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category 

Re: possible bug in zookeeper ?

2010-09-14 Thread Mahadev Konar
Hi yatir,
 Can you confirm that zook1 , zook2 can be nslookedup from the client
machine? 

We havent seen a bug like this. It would be great to nail this down.

Thanks
mahadev


On 9/14/10 8:44 AM, Yatir Ben Shlomo yat...@outbrain.com wrote:

 zook1:2181,zook2:2181,zook3:2181
 
 
 -Original Message-
 From: Ted Dunning [mailto:ted.dunn...@gmail.com]
 Sent: Tuesday, September 14, 2010 4:11 PM
 To: zookeeper-user@hadoop.apache.org
 Subject: Re: possible bug in zookeeper ?
 
 What was the list of servers that was given originally to open the
 connection to ZK?
 
 On Tue, Sep 14, 2010 at 6:15 AM, Yatir Ben Shlomo yat...@outbrain.comwrote:
 
 Hi I am using solrCloud which uses an ensemble of 3 zookeeper instances.
 
 I am performing survivability  tests:
 Taking one of the zookeeper instances down I would expect the client to use
 a different zookeeper server instance.
 
 But as you can see in the below logs attached
 Depending on which instance I choose to take down (in my case,  the last
 one in the list of zookeeper servers)
 the client is constantly insisting on the same zookeeper server (Attempting
 connection to server zook3/192.168.252.78:2181)
 and not switching to a different one
 the problem seems to arrive from ClientCnxn.java
 Any one has an idea on this ?
 
 Solr cloud currently is using  zookeeper-3.2.2.jar
 Is this a know bug that was fixed in later versions ?( 3.3.1)
 
 Thanks in advance,
 Yatir
 
 
 Logs:
 
 Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown input
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java)
 :999)
at
 
org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970
)
 Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown output
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java)
 :1004)
at
 
org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970
)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info
 INFO: Attempting connection to server zook3/192.168.252.78:2181
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Exception closing session 0x32b105244a20001 to
 sun.nio.ch.selectionkeyi...@3ca58cbf
 java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
at
 sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
 
org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933
)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown input
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java)
 :999)
at
 
org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970
)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown output
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java)
 :1004)
at
 
org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970
)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info
 INFO: Attempting connection to server zook3/192.168.252.78:2181
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Exception closing session 0x32b105244a2 to
 sun.nio.ch.selectionkeyi...@3960f81b
 java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
at
 sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
 
org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933
)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown input
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at 

Re: possible bug in zookeeper ?

2010-09-14 Thread Ted Dunning
And that you can connect from every client to every server?

On Tue, Sep 14, 2010 at 9:07 AM, Mahadev Konar maha...@yahoo-inc.comwrote:

 Hi yatir,
  Can you confirm that zook1 , zook2 can be nslookedup from the client
 machine?

 We havent seen a bug like this. It would be great to nail this down.

 Thanks
 mahadev


 On 9/14/10 8:44 AM, Yatir Ben Shlomo yat...@outbrain.com wrote:

  zook1:2181,zook2:2181,zook3:2181
 
 
  -Original Message-
  From: Ted Dunning [mailto:ted.dunn...@gmail.com]
  Sent: Tuesday, September 14, 2010 4:11 PM
  To: zookeeper-user@hadoop.apache.org
  Subject: Re: possible bug in zookeeper ?
 
  What was the list of servers that was given originally to open the
  connection to ZK?
 
  On Tue, Sep 14, 2010 at 6:15 AM, Yatir Ben Shlomo yat...@outbrain.com
 wrote:
 
  Hi I am using solrCloud which uses an ensemble of 3 zookeeper instances.
 
  I am performing survivability  tests:
  Taking one of the zookeeper instances down I would expect the client to
 use
  a different zookeeper server instance.
 
  But as you can see in the below logs attached
  Depending on which instance I choose to take down (in my case,  the last
  one in the list of zookeeper servers)
  the client is constantly insisting on the same zookeeper server
 (Attempting
  connection to server zook3/192.168.252.78:2181)
  and not switching to a different one
  the problem seems to arrive from ClientCnxn.java
  Any one has an idea on this ?
 
  Solr cloud currently is using  zookeeper-3.2.2.jar
  Is this a know bug that was fixed in later versions ?( 3.3.1)
 
  Thanks in advance,
  Yatir
 
 
  Logs:
 
  Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
  WARNING: Ignoring exception during shutdown input
  java.nio.channels.ClosedChannelException
 at
  sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
 at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java)
  :999)
 at
 

 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970
 )
  Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
  WARNING: Ignoring exception during shutdown output
  java.nio.channels.ClosedChannelException
 at
  sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
 at
 sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java)
  :1004)
 at
 

 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970
 )
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info
  INFO: Attempting connection to server zook3/192.168.252.78:2181
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
  WARNING: Exception closing session 0x32b105244a20001 to
  sun.nio.ch.selectionkeyi...@3ca58cbf
  java.net.ConnectException: Connection refused
 at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native
 Method)
 at
 sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
 at
  sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
 at
 

 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933
 )
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
  WARNING: Ignoring exception during shutdown input
  java.nio.channels.ClosedChannelException
 at
  sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
 at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java)
  :999)
 at
 

 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970
 )
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
  WARNING: Ignoring exception during shutdown output
  java.nio.channels.ClosedChannelException
 at
  sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
 at
 sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java)
  :1004)
 at
 

 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970
 )
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info
  INFO: Attempting connection to server zook3/192.168.252.78:2181
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
  WARNING: Exception closing session 0x32b105244a2 to
  sun.nio.ch.selectionkeyi...@3960f81b
  java.net.ConnectException: Connection refused
 at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native
 Method)
 at
 sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
 at
  sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
 at
 

 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933
 )

Re: possible bug in zookeeper ?

2010-09-14 Thread Patrick Hunt
That is unusual. I don't recall anyone reporting a similar issue, and
looking at the code I don't see any issues off hand. Can you try the
following?

1) on that particular zk client machine resolve the hosts zook1/zook2/zook3,
what ip addresses does this resolve to? (try dig)
2) try running the client using the 3.3.1 jar file (just replace the jar on
the client), it includes more log4j information, turn on DEBUG or TRACE
logging

Patrick

On Tue, Sep 14, 2010 at 8:44 AM, Yatir Ben Shlomo yat...@outbrain.comwrote:

 zook1:2181,zook2:2181,zook3:2181


 -Original Message-
 From: Ted Dunning [mailto:ted.dunn...@gmail.com]
 Sent: Tuesday, September 14, 2010 4:11 PM
 To: zookeeper-user@hadoop.apache.org
 Subject: Re: possible bug in zookeeper ?

 What was the list of servers that was given originally to open the
 connection to ZK?

 On Tue, Sep 14, 2010 at 6:15 AM, Yatir Ben Shlomo yat...@outbrain.com
 wrote:

  Hi I am using solrCloud which uses an ensemble of 3 zookeeper instances.
 
  I am performing survivability  tests:
  Taking one of the zookeeper instances down I would expect the client to
 use
  a different zookeeper server instance.
 
  But as you can see in the below logs attached
  Depending on which instance I choose to take down (in my case,  the last
  one in the list of zookeeper servers)
  the client is constantly insisting on the same zookeeper server
 (Attempting
  connection to server zook3/192.168.252.78:2181)
  and not switching to a different one
  the problem seems to arrive from ClientCnxn.java
  Any one has an idea on this ?
 
  Solr cloud currently is using  zookeeper-3.2.2.jar
  Is this a know bug that was fixed in later versions ?( 3.3.1)
 
  Thanks in advance,
  Yatir
 
 
  Logs:
 
  Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
  WARNING: Ignoring exception during shutdown input
  java.nio.channels.ClosedChannelException
 at
  sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
 at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
  Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
  WARNING: Ignoring exception during shutdown output
  java.nio.channels.ClosedChannelException
 at
  sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
 at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):1004)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info
  INFO: Attempting connection to server zook3/192.168.252.78:2181
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
  WARNING: Exception closing session 0x32b105244a20001 to
  sun.nio.ch.selectionkeyi...@3ca58cbf
  java.net.ConnectException: Connection refused
 at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method)
 at
 sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
 at
  sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933)
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
  WARNING: Ignoring exception during shutdown input
  java.nio.channels.ClosedChannelException
 at
  sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
 at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
  WARNING: Ignoring exception during shutdown output
  java.nio.channels.ClosedChannelException
 at
  sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
 at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):1004)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info
  INFO: Attempting connection to server zook3/192.168.252.78:2181
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
  WARNING: Exception closing session 0x32b105244a2 to
  sun.nio.ch.selectionkeyi...@3960f81b
  java.net.ConnectException: Connection refused
 at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method)
 at
 sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
 at
  sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)