Re: possible bug in zookeeper ?

2010-10-04 Thread Mahadev Konar
Hi Yatir,

  Any update on this? Are you still struggling with this problem?

Thanks
mahadev

On 9/15/10 12:56 AM, Yatir Ben Shlomo yat...@outbrain.com wrote:

 Thanks to all who replied, I appreciate your efforts:
 
 1. There is no connections problem from the client machine:
 (ob1078)(tom...@cass3:~)$ echo ruok | nc zook1 2181
 imok(ob1078)(tom...@cass3:~)$ echo ruok | nc zook2 2181
 imok(ob1078)(tom...@cass3:~)$ echo ruok | nc zook3 2181
 imok(ob1078)(tom...@cass3:~)$
 
 2. Unfortunately I have already tried to switch to the new jar but it does not
 seem to be backward compatible.
 It seems that the QuorumPeerConfig class does not have the following field
 protected int clientPort;
 It was replaced by InetSocketAddress clientPortAddress in the new jar
 So I am getting java.lang.NoSuchFieldError exception...
 
 3. I looked at the ClientCnxn.java code.
 It seems that the logic for iterating over the available servers
 (nextAddrToTry++ ) is used only inside the startConnect() function but not in
 the finishConnect() function, nor anywhere else.
 
 Possibly something along these lines is happening:
 some exception that happens inside the finishConnect() function is cauasing
 the cleanup() function which in turn causes another exception.
 Nowhere in this code path is the nextAddrToTry++ applied.
 Can this make sense to someone ?
 thanks
 
 
 
 
 
 
 -Original Message-
 From: Patrick Hunt [mailto:ph...@apache.org]
 Sent: Tuesday, September 14, 2010 6:20 PM
 To: zookeeper-user@hadoop.apache.org
 Subject: Re: possible bug in zookeeper ?
 
 That is unusual. I don't recall anyone reporting a similar issue, and
 looking at the code I don't see any issues off hand. Can you try the
 following?
 
 1) on that particular zk client machine resolve the hosts zook1/zook2/zook3,
 what ip addresses does this resolve to? (try dig)
 2) try running the client using the 3.3.1 jar file (just replace the jar on
 the client), it includes more log4j information, turn on DEBUG or TRACE
 logging
 
 Patrick
 
 On Tue, Sep 14, 2010 at 8:44 AM, Yatir Ben Shlomo yat...@outbrain.comwrote:
 
 zook1:2181,zook2:2181,zook3:2181
 
 
 -Original Message-
 From: Ted Dunning [mailto:ted.dunn...@gmail.com]
 Sent: Tuesday, September 14, 2010 4:11 PM
 To: zookeeper-user@hadoop.apache.org
 Subject: Re: possible bug in zookeeper ?
 
 What was the list of servers that was given originally to open the
 connection to ZK?
 
 On Tue, Sep 14, 2010 at 6:15 AM, Yatir Ben Shlomo yat...@outbrain.com
 wrote:
 
 Hi I am using solrCloud which uses an ensemble of 3 zookeeper instances.
 
 I am performing survivability  tests:
 Taking one of the zookeeper instances down I would expect the client to
 use
 a different zookeeper server instance.
 
 But as you can see in the below logs attached
 Depending on which instance I choose to take down (in my case,  the last
 one in the list of zookeeper servers)
 the client is constantly insisting on the same zookeeper server
 (Attempting
 connection to server zook3/192.168.252.78:2181)
 and not switching to a different one
 the problem seems to arrive from ClientCnxn.java
 Any one has an idea on this ?
 
 Solr cloud currently is using  zookeeper-3.2.2.jar
 Is this a know bug that was fixed in later versions ?( 3.3.1)
 
 Thanks in advance,
 Yatir
 
 
 Logs:
 
 Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown input
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
 
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java)
 :999)
at
 
 
org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970
)
 Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown output
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at
 
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java)
 :1004)
at
 
 
org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970
)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info
 INFO: Attempting connection to server zook3/192.168.252.78:2181
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Exception closing session 0x32b105244a20001 to
 sun.nio.ch.selectionkeyi...@3ca58cbf
 java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method)
at
 sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
at
 sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
 
 
org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933
)
 Sep 14, 2010 9:02:22 AM

RE: possible bug in zookeeper ?

2010-09-19 Thread Yatir Ben Shlomo
Ok Thanks for all the help!


-Original Message-
From: Patrick Hunt [mailto:phu...@gmail.com] 
Sent: Thursday, September 16, 2010 5:52 PM
To: zookeeper-user@hadoop.apache.org
Subject: Re: possible bug in zookeeper ?

Hi Yatir, sorry about that, from your report I took access to
QuorumPeerConfig to indicate problems on a server. This is due to the fact
that QPC should only be instantiated on a ZK server. Clients instantiate
ZooKeeper and ClientCnxn. That's very odd. Probably your best bet at this
point, if you haven't done so already, is to contact the Solr team and see
if they have any insight. Perhaps they can provide you with a build that
compiles against the 3.3.1 version of ZK.

Regards,

Patrick

On Thu, Sep 16, 2010 at 2:22 AM, Yatir Ben Shlomo yat...@outbrain.comwrote:

 Dear Patrick,
 I disagree with your conclusion that I presumably changed the jar all over.
 It's not true.
 I tried to replace it only in solr i.e. in the client.
 I don't even have write permissions to change the zookeeper service
 deployment.
 Because solr extends the class QuorumPeerConfig:

 ...class SolrZkServerProps extends QuorumPeerConfig {

 And because
 SolrZkServerProps reference the clientPort field in its super class -

 it cant compile once you change the jar and eliminate this field...


 yatir

 -Original Message-
 From: Patrick Hunt [mailto:ph...@apache.org]
 Sent: Wednesday, September 15, 2010 7:32 PM
 To: zookeeper-user@hadoop.apache.org
 Subject: Re: possible bug in zookeeper ?

 On Wed, Sep 15, 2010 at 12:56 AM, Yatir Ben Shlomo yat...@outbrain.com
 wrote:

  2. Unfortunately I have already tried to switch to the new jar but it
 does
  not seem to be backward compatible.
  It seems that the QuorumPeerConfig class does not have the following
 field
  protected int clientPort;
  It was replaced by InetSocketAddress clientPortAddress in the new jar
  So I am getting java.lang.NoSuchFieldError exception...
 
 
 That's why I suggested only changing it on the client, this error indicates
 that you changed the jar on the server as well. It seems that solr is using
 some of our lower layer apis.

 Just change the jar on the client, not everywhere, and retry. The logs
 should contain the exact connect string used. Also try increasing the
 logging level to debug or trace on the client, which might provide addl
 insight.

 Patrick



Re: possible bug in zookeeper ?

2010-09-14 Thread Ted Dunning
What was the list of servers that was given originally to open the
connection to ZK?

On Tue, Sep 14, 2010 at 6:15 AM, Yatir Ben Shlomo yat...@outbrain.comwrote:

 Hi I am using solrCloud which uses an ensemble of 3 zookeeper instances.

 I am performing survivability  tests:
 Taking one of the zookeeper instances down I would expect the client to use
 a different zookeeper server instance.

 But as you can see in the below logs attached
 Depending on which instance I choose to take down (in my case,  the last
 one in the list of zookeeper servers)
 the client is constantly insisting on the same zookeeper server (Attempting
 connection to server zook3/192.168.252.78:2181)
 and not switching to a different one
 the problem seems to arrive from ClientCnxn.java
 Any one has an idea on this ?

 Solr cloud currently is using  zookeeper-3.2.2.jar
 Is this a know bug that was fixed in later versions ?( 3.3.1)

 Thanks in advance,
 Yatir


 Logs:

 Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown input
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
 Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown output
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):1004)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info
 INFO: Attempting connection to server zook3/192.168.252.78:2181
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Exception closing session 0x32b105244a20001 to
 sun.nio.ch.selectionkeyi...@3ca58cbf
 java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
at
 sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown input
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown output
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):1004)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info
 INFO: Attempting connection to server zook3/192.168.252.78:2181
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Exception closing session 0x32b105244a2 to
 sun.nio.ch.selectionkeyi...@3960f81b
 java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
at
 sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown input
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown output
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at 

RE: possible bug in zookeeper ?

2010-09-14 Thread Yatir Ben Shlomo
zook1:2181,zook2:2181,zook3:2181


-Original Message-
From: Ted Dunning [mailto:ted.dunn...@gmail.com] 
Sent: Tuesday, September 14, 2010 4:11 PM
To: zookeeper-user@hadoop.apache.org
Subject: Re: possible bug in zookeeper ?

What was the list of servers that was given originally to open the
connection to ZK?

On Tue, Sep 14, 2010 at 6:15 AM, Yatir Ben Shlomo yat...@outbrain.comwrote:

 Hi I am using solrCloud which uses an ensemble of 3 zookeeper instances.

 I am performing survivability  tests:
 Taking one of the zookeeper instances down I would expect the client to use
 a different zookeeper server instance.

 But as you can see in the below logs attached
 Depending on which instance I choose to take down (in my case,  the last
 one in the list of zookeeper servers)
 the client is constantly insisting on the same zookeeper server (Attempting
 connection to server zook3/192.168.252.78:2181)
 and not switching to a different one
 the problem seems to arrive from ClientCnxn.java
 Any one has an idea on this ?

 Solr cloud currently is using  zookeeper-3.2.2.jar
 Is this a know bug that was fixed in later versions ?( 3.3.1)

 Thanks in advance,
 Yatir


 Logs:

 Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown input
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
 Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown output
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):1004)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info
 INFO: Attempting connection to server zook3/192.168.252.78:2181
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Exception closing session 0x32b105244a20001 to
 sun.nio.ch.selectionkeyi...@3ca58cbf
 java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
at
 sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown input
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown output
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):1004)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info
 INFO: Attempting connection to server zook3/192.168.252.78:2181
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Exception closing session 0x32b105244a2 to
 sun.nio.ch.selectionkeyi...@3960f81b
 java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
at
 sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown input
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999)
at
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category

Re: possible bug in zookeeper ?

2010-09-14 Thread Mahadev Konar
Hi yatir,
 Can you confirm that zook1 , zook2 can be nslookedup from the client
machine? 

We havent seen a bug like this. It would be great to nail this down.

Thanks
mahadev


On 9/14/10 8:44 AM, Yatir Ben Shlomo yat...@outbrain.com wrote:

 zook1:2181,zook2:2181,zook3:2181
 
 
 -Original Message-
 From: Ted Dunning [mailto:ted.dunn...@gmail.com]
 Sent: Tuesday, September 14, 2010 4:11 PM
 To: zookeeper-user@hadoop.apache.org
 Subject: Re: possible bug in zookeeper ?
 
 What was the list of servers that was given originally to open the
 connection to ZK?
 
 On Tue, Sep 14, 2010 at 6:15 AM, Yatir Ben Shlomo yat...@outbrain.comwrote:
 
 Hi I am using solrCloud which uses an ensemble of 3 zookeeper instances.
 
 I am performing survivability  tests:
 Taking one of the zookeeper instances down I would expect the client to use
 a different zookeeper server instance.
 
 But as you can see in the below logs attached
 Depending on which instance I choose to take down (in my case,  the last
 one in the list of zookeeper servers)
 the client is constantly insisting on the same zookeeper server (Attempting
 connection to server zook3/192.168.252.78:2181)
 and not switching to a different one
 the problem seems to arrive from ClientCnxn.java
 Any one has an idea on this ?
 
 Solr cloud currently is using  zookeeper-3.2.2.jar
 Is this a know bug that was fixed in later versions ?( 3.3.1)
 
 Thanks in advance,
 Yatir
 
 
 Logs:
 
 Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown input
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java)
 :999)
at
 
org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970
)
 Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown output
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java)
 :1004)
at
 
org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970
)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info
 INFO: Attempting connection to server zook3/192.168.252.78:2181
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Exception closing session 0x32b105244a20001 to
 sun.nio.ch.selectionkeyi...@3ca58cbf
 java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
at
 sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
 
org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933
)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown input
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java)
 :999)
at
 
org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970
)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown output
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java)
 :1004)
at
 
org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970
)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info
 INFO: Attempting connection to server zook3/192.168.252.78:2181
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Exception closing session 0x32b105244a2 to
 sun.nio.ch.selectionkeyi...@3960f81b
 java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
at
 sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
 
org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933
)
 Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
 WARNING: Ignoring exception during shutdown input
 java.nio.channels.ClosedChannelException
at
 sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638

Re: possible bug in zookeeper ?

2010-09-14 Thread Ted Dunning
And that you can connect from every client to every server?

On Tue, Sep 14, 2010 at 9:07 AM, Mahadev Konar maha...@yahoo-inc.comwrote:

 Hi yatir,
  Can you confirm that zook1 , zook2 can be nslookedup from the client
 machine?

 We havent seen a bug like this. It would be great to nail this down.

 Thanks
 mahadev


 On 9/14/10 8:44 AM, Yatir Ben Shlomo yat...@outbrain.com wrote:

  zook1:2181,zook2:2181,zook3:2181
 
 
  -Original Message-
  From: Ted Dunning [mailto:ted.dunn...@gmail.com]
  Sent: Tuesday, September 14, 2010 4:11 PM
  To: zookeeper-user@hadoop.apache.org
  Subject: Re: possible bug in zookeeper ?
 
  What was the list of servers that was given originally to open the
  connection to ZK?
 
  On Tue, Sep 14, 2010 at 6:15 AM, Yatir Ben Shlomo yat...@outbrain.com
 wrote:
 
  Hi I am using solrCloud which uses an ensemble of 3 zookeeper instances.
 
  I am performing survivability  tests:
  Taking one of the zookeeper instances down I would expect the client to
 use
  a different zookeeper server instance.
 
  But as you can see in the below logs attached
  Depending on which instance I choose to take down (in my case,  the last
  one in the list of zookeeper servers)
  the client is constantly insisting on the same zookeeper server
 (Attempting
  connection to server zook3/192.168.252.78:2181)
  and not switching to a different one
  the problem seems to arrive from ClientCnxn.java
  Any one has an idea on this ?
 
  Solr cloud currently is using  zookeeper-3.2.2.jar
  Is this a know bug that was fixed in later versions ?( 3.3.1)
 
  Thanks in advance,
  Yatir
 
 
  Logs:
 
  Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
  WARNING: Ignoring exception during shutdown input
  java.nio.channels.ClosedChannelException
 at
  sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
 at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java)
  :999)
 at
 

 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970
 )
  Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
  WARNING: Ignoring exception during shutdown output
  java.nio.channels.ClosedChannelException
 at
  sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
 at
 sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java)
  :1004)
 at
 

 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970
 )
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info
  INFO: Attempting connection to server zook3/192.168.252.78:2181
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
  WARNING: Exception closing session 0x32b105244a20001 to
  sun.nio.ch.selectionkeyi...@3ca58cbf
  java.net.ConnectException: Connection refused
 at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native
 Method)
 at
 sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
 at
  sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
 at
 

 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933
 )
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
  WARNING: Ignoring exception during shutdown input
  java.nio.channels.ClosedChannelException
 at
  sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
 at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java)
  :999)
 at
 

 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970
 )
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
  WARNING: Ignoring exception during shutdown output
  java.nio.channels.ClosedChannelException
 at
  sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
 at
 sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java)
  :1004)
 at
 

 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970
 )
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info
  INFO: Attempting connection to server zook3/192.168.252.78:2181
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
  WARNING: Exception closing session 0x32b105244a2 to
  sun.nio.ch.selectionkeyi...@3960f81b
  java.net.ConnectException: Connection refused
 at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native
 Method)
 at
 sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
 at
  sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
 at
 

 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933

Re: possible bug in zookeeper ?

2010-09-14 Thread Patrick Hunt
That is unusual. I don't recall anyone reporting a similar issue, and
looking at the code I don't see any issues off hand. Can you try the
following?

1) on that particular zk client machine resolve the hosts zook1/zook2/zook3,
what ip addresses does this resolve to? (try dig)
2) try running the client using the 3.3.1 jar file (just replace the jar on
the client), it includes more log4j information, turn on DEBUG or TRACE
logging

Patrick

On Tue, Sep 14, 2010 at 8:44 AM, Yatir Ben Shlomo yat...@outbrain.comwrote:

 zook1:2181,zook2:2181,zook3:2181


 -Original Message-
 From: Ted Dunning [mailto:ted.dunn...@gmail.com]
 Sent: Tuesday, September 14, 2010 4:11 PM
 To: zookeeper-user@hadoop.apache.org
 Subject: Re: possible bug in zookeeper ?

 What was the list of servers that was given originally to open the
 connection to ZK?

 On Tue, Sep 14, 2010 at 6:15 AM, Yatir Ben Shlomo yat...@outbrain.com
 wrote:

  Hi I am using solrCloud which uses an ensemble of 3 zookeeper instances.
 
  I am performing survivability  tests:
  Taking one of the zookeeper instances down I would expect the client to
 use
  a different zookeeper server instance.
 
  But as you can see in the below logs attached
  Depending on which instance I choose to take down (in my case,  the last
  one in the list of zookeeper servers)
  the client is constantly insisting on the same zookeeper server
 (Attempting
  connection to server zook3/192.168.252.78:2181)
  and not switching to a different one
  the problem seems to arrive from ClientCnxn.java
  Any one has an idea on this ?
 
  Solr cloud currently is using  zookeeper-3.2.2.jar
  Is this a know bug that was fixed in later versions ?( 3.3.1)
 
  Thanks in advance,
  Yatir
 
 
  Logs:
 
  Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
  WARNING: Ignoring exception during shutdown input
  java.nio.channels.ClosedChannelException
 at
  sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
 at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
  Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn
  WARNING: Ignoring exception during shutdown output
  java.nio.channels.ClosedChannelException
 at
  sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
 at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):1004)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info
  INFO: Attempting connection to server zook3/192.168.252.78:2181
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
  WARNING: Exception closing session 0x32b105244a20001 to
  sun.nio.ch.selectionkeyi...@3ca58cbf
  java.net.ConnectException: Connection refused
 at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method)
 at
 sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
 at
  sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933)
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
  WARNING: Ignoring exception during shutdown input
  java.nio.channels.ClosedChannelException
 at
  sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
 at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
  WARNING: Ignoring exception during shutdown output
  java.nio.channels.ClosedChannelException
 at
  sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
 at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):1004)
 at
 
 org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970)
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info
  INFO: Attempting connection to server zook3/192.168.252.78:2181
  Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn
  WARNING: Exception closing session 0x32b105244a2 to
  sun.nio.ch.selectionkeyi...@3960f81b
  java.net.ConnectException: Connection refused
 at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method)
 at
 sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java)
 at
  sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574