Re: possible bug in zookeeper ?
Hi Yatir, Any update on this? Are you still struggling with this problem? Thanks mahadev On 9/15/10 12:56 AM, "Yatir Ben Shlomo" wrote: > Thanks to all who replied, I appreciate your efforts: > > 1. There is no connections problem from the client machine: > (ob1078)(tom...@cass3:~)$ echo ruok | nc zook1 2181 > imok(ob1078)(tom...@cass3:~)$ echo ruok | nc zook2 2181 > imok(ob1078)(tom...@cass3:~)$ echo ruok | nc zook3 2181 > imok(ob1078)(tom...@cass3:~)$ > > 2. Unfortunately I have already tried to switch to the new jar but it does not > seem to be backward compatible. > It seems that the QuorumPeerConfig class does not have the following field > protected int clientPort; > It was replaced by InetSocketAddress clientPortAddress in the new jar > So I am getting java.lang.NoSuchFieldError exception... > > 3. I looked at the ClientCnxn.java code. > It seems that the logic for iterating over the available servers > (nextAddrToTry++ ) is used only inside the startConnect() function but not in > the finishConnect() function, nor anywhere else. > > Possibly something along these lines is happening: > some exception that happens inside the finishConnect() function is cauasing > the cleanup() function which in turn causes another exception. > Nowhere in this code path is the nextAddrToTry++ applied. > Can this make sense to someone ? > thanks > > > > > > > -Original Message- > From: Patrick Hunt [mailto:ph...@apache.org] > Sent: Tuesday, September 14, 2010 6:20 PM > To: zookeeper-user@hadoop.apache.org > Subject: Re: possible bug in zookeeper ? > > That is unusual. I don't recall anyone reporting a similar issue, and > looking at the code I don't see any issues off hand. Can you try the > following? > > 1) on that particular zk client machine resolve the hosts zook1/zook2/zook3, > what ip addresses does this resolve to? (try dig) > 2) try running the client using the 3.3.1 jar file (just replace the jar on > the client), it includes more log4j information, turn on DEBUG or TRACE > logging > > Patrick > > On Tue, Sep 14, 2010 at 8:44 AM, Yatir Ben Shlomo wrote: > >> zook1:2181,zook2:2181,zook3:2181 >> >> >> -Original Message- >> From: Ted Dunning [mailto:ted.dunn...@gmail.com] >> Sent: Tuesday, September 14, 2010 4:11 PM >> To: zookeeper-user@hadoop.apache.org >> Subject: Re: possible bug in zookeeper ? >> >> What was the list of servers that was given originally to open the >> connection to ZK? >> >> On Tue, Sep 14, 2010 at 6:15 AM, Yatir Ben Shlomo >> wrote: >> >>> Hi I am using solrCloud which uses an ensemble of 3 zookeeper instances. >>> >>> I am performing survivability tests: >>> Taking one of the zookeeper instances down I would expect the client to >> use >>> a different zookeeper server instance. >>> >>> But as you can see in the below logs attached >>> Depending on which instance I choose to take down (in my case, the last >>> one in the list of zookeeper servers) >>> the client is constantly insisting on the same zookeeper server >> (Attempting >>> connection to server zook3/192.168.252.78:2181) >>> and not switching to a different one >>> the problem seems to arrive from ClientCnxn.java >>> Any one has an idea on this ? >>> >>> Solr cloud currently is using zookeeper-3.2.2.jar >>> Is this a know bug that was fixed in later versions ?( 3.3.1) >>> >>> Thanks in advance, >>> Yatir >>> >>> >>> Logs: >>> >>> Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn >>> WARNING: Ignoring exception during shutdown input >>> java.nio.channels.ClosedChannelException >>>at >>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) >>>at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) >>>at >>> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java) >> :999) >>>at >>> >> org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970>> ) >>> Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn >>> WARNING: Ignoring exception during shutdown output >>> java.nio.channels.ClosedChannelException >>>at >>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) >>>at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) >>>at >>> >> org
RE: possible bug in zookeeper ?
Ok Thanks for all the help!
-Original Message-
From: Patrick Hunt [mailto:phu...@gmail.com]
Sent: Thursday, September 16, 2010 5:52 PM
To: zookeeper-user@hadoop.apache.org
Subject: Re: possible bug in zookeeper ?
Hi Yatir, sorry about that, from your report I took access to
"QuorumPeerConfig" to indicate problems on a server. This is due to the fact
that QPC should only be instantiated on a ZK server. Clients instantiate
ZooKeeper and ClientCnxn. That's very odd. Probably your best bet at this
point, if you haven't done so already, is to contact the Solr team and see
if they have any insight. Perhaps they can provide you with a build that
compiles against the 3.3.1 version of ZK.
Regards,
Patrick
On Thu, Sep 16, 2010 at 2:22 AM, Yatir Ben Shlomo wrote:
> Dear Patrick,
> I disagree with your conclusion that I presumably changed the jar all over.
> It's not true.
> I tried to replace it only in solr i.e. in the client.
> I don't even have write permissions to change the zookeeper service
> deployment.
> Because solr extends the class QuorumPeerConfig:
>
> "...class SolrZkServerProps extends QuorumPeerConfig {"
>
> And because
> SolrZkServerProps reference the clientPort field in its super class -
>
> it cant compile once you change the jar and eliminate this field...
>
>
> yatir
>
> -Original Message-
> From: Patrick Hunt [mailto:ph...@apache.org]
> Sent: Wednesday, September 15, 2010 7:32 PM
> To: zookeeper-user@hadoop.apache.org
> Subject: Re: possible bug in zookeeper ?
>
> On Wed, Sep 15, 2010 at 12:56 AM, Yatir Ben Shlomo >wrote:
>
> > 2. Unfortunately I have already tried to switch to the new jar but it
> does
> > not seem to be backward compatible.
> > It seems that the QuorumPeerConfig class does not have the following
> field
> > protected int clientPort;
> > It was replaced by InetSocketAddress clientPortAddress in the new jar
> > So I am getting java.lang.NoSuchFieldError exception...
> >
> >
> That's why I suggested only changing it on the client, this error indicates
> that you changed the jar on the server as well. It seems that solr is using
> some of our lower layer apis.
>
> Just change the jar on the client, not everywhere, and retry. The logs
> should contain the exact connect string used. Also try increasing the
> logging level to debug or trace on the client, which might provide addl
> insight.
>
> Patrick
>
Re: possible bug in zookeeper ?
Hi Yatir, sorry about that, from your report I took access to
"QuorumPeerConfig" to indicate problems on a server. This is due to the fact
that QPC should only be instantiated on a ZK server. Clients instantiate
ZooKeeper and ClientCnxn. That's very odd. Probably your best bet at this
point, if you haven't done so already, is to contact the Solr team and see
if they have any insight. Perhaps they can provide you with a build that
compiles against the 3.3.1 version of ZK.
Regards,
Patrick
On Thu, Sep 16, 2010 at 2:22 AM, Yatir Ben Shlomo wrote:
> Dear Patrick,
> I disagree with your conclusion that I presumably changed the jar all over.
> It's not true.
> I tried to replace it only in solr i.e. in the client.
> I don't even have write permissions to change the zookeeper service
> deployment.
> Because solr extends the class QuorumPeerConfig:
>
> "...class SolrZkServerProps extends QuorumPeerConfig {"
>
> And because
> SolrZkServerProps reference the clientPort field in its super class -
>
> it cant compile once you change the jar and eliminate this field...
>
>
> yatir
>
> -Original Message-
> From: Patrick Hunt [mailto:ph...@apache.org]
> Sent: Wednesday, September 15, 2010 7:32 PM
> To: zookeeper-user@hadoop.apache.org
> Subject: Re: possible bug in zookeeper ?
>
> On Wed, Sep 15, 2010 at 12:56 AM, Yatir Ben Shlomo >wrote:
>
> > 2. Unfortunately I have already tried to switch to the new jar but it
> does
> > not seem to be backward compatible.
> > It seems that the QuorumPeerConfig class does not have the following
> field
> > protected int clientPort;
> > It was replaced by InetSocketAddress clientPortAddress in the new jar
> > So I am getting java.lang.NoSuchFieldError exception...
> >
> >
> That's why I suggested only changing it on the client, this error indicates
> that you changed the jar on the server as well. It seems that solr is using
> some of our lower layer apis.
>
> Just change the jar on the client, not everywhere, and retry. The logs
> should contain the exact connect string used. Also try increasing the
> logging level to debug or trace on the client, which might provide addl
> insight.
>
> Patrick
>
Re: possible bug in zookeeper ?
Hi Yatir, sorry about that, from your report I took access to
"QuorumPeerConfig" to indicate problems on a server. This is due to the fact
that QPC should only be instantiated on a ZK server. Clients instantiate
ZooKeeper and ClientCnxn. That's very odd. Probably your best bet at this
point, if you haven't done so already, is to contact the Solr team and see
if they have any insight. Perhaps they can provide you with a build that
compiles against the 3.3.1 version of ZK.
Regards,
Patrick
On Thu, Sep 16, 2010 at 2:22 AM, Yatir Ben Shlomo wrote:
> Dear Patrick,
> I disagree with your conclusion that I presumably changed the jar all over.
> It's not true.
> I tried to replace it only in solr i.e. in the client.
> I don't even have write permissions to change the zookeeper service
> deployment.
> Because solr extends the class QuorumPeerConfig:
>
> "...class SolrZkServerProps extends QuorumPeerConfig {"
>
> And because
> SolrZkServerProps reference the clientPort field in its super class -
>
> it cant compile once you change the jar and eliminate this field...
>
>
> yatir
>
> -Original Message-
> From: Patrick Hunt [mailto:ph...@apache.org]
> Sent: Wednesday, September 15, 2010 7:32 PM
> To: zookeeper-user@hadoop.apache.org
> Subject: Re: possible bug in zookeeper ?
>
> On Wed, Sep 15, 2010 at 12:56 AM, Yatir Ben Shlomo >wrote:
>
> > 2. Unfortunately I have already tried to switch to the new jar but it
> does
> > not seem to be backward compatible.
> > It seems that the QuorumPeerConfig class does not have the following
> field
> > protected int clientPort;
> > It was replaced by InetSocketAddress clientPortAddress in the new jar
> > So I am getting java.lang.NoSuchFieldError exception...
> >
> >
> That's why I suggested only changing it on the client, this error indicates
> that you changed the jar on the server as well. It seems that solr is using
> some of our lower layer apis.
>
> Just change the jar on the client, not everywhere, and retry. The logs
> should contain the exact connect string used. Also try increasing the
> logging level to debug or trace on the client, which might provide addl
> insight.
>
> Patrick
>
RE: possible bug in zookeeper ?
Dear Patrick,
I disagree with your conclusion that I presumably changed the jar all over.
It's not true.
I tried to replace it only in solr i.e. in the client.
I don't even have write permissions to change the zookeeper service deployment.
Because solr extends the class QuorumPeerConfig:
"...class SolrZkServerProps extends QuorumPeerConfig {"
And because
SolrZkServerProps reference the clientPort field in its super class -
it cant compile once you change the jar and eliminate this field...
yatir
-Original Message-
From: Patrick Hunt [mailto:ph...@apache.org]
Sent: Wednesday, September 15, 2010 7:32 PM
To: zookeeper-user@hadoop.apache.org
Subject: Re: possible bug in zookeeper ?
On Wed, Sep 15, 2010 at 12:56 AM, Yatir Ben Shlomo wrote:
> 2. Unfortunately I have already tried to switch to the new jar but it does
> not seem to be backward compatible.
> It seems that the QuorumPeerConfig class does not have the following field
> protected int clientPort;
> It was replaced by InetSocketAddress clientPortAddress in the new jar
> So I am getting java.lang.NoSuchFieldError exception...
>
>
That's why I suggested only changing it on the client, this error indicates
that you changed the jar on the server as well. It seems that solr is using
some of our lower layer apis.
Just change the jar on the client, not everywhere, and retry. The logs
should contain the exact connect string used. Also try increasing the
logging level to debug or trace on the client, which might provide addl
insight.
Patrick
Re: possible bug in zookeeper ?
On Wed, Sep 15, 2010 at 12:56 AM, Yatir Ben Shlomo wrote: > 2. Unfortunately I have already tried to switch to the new jar but it does > not seem to be backward compatible. > It seems that the QuorumPeerConfig class does not have the following field > protected int clientPort; > It was replaced by InetSocketAddress clientPortAddress in the new jar > So I am getting java.lang.NoSuchFieldError exception... > > That's why I suggested only changing it on the client, this error indicates that you changed the jar on the server as well. It seems that solr is using some of our lower layer apis. Just change the jar on the client, not everywhere, and retry. The logs should contain the exact connect string used. Also try increasing the logging level to debug or trace on the client, which might provide addl insight. Patrick
Re: possible bug in zookeeper ?
Yatir, Can you try this out: From zook1, try running the zookeeper a simpole client library: http://hadoop.apache.org/zookeeper/docs/r3.2.1/zookeeperStarted.html bin/zkCli.sh -server zoo1:port,zoo2:port,zoo3:port And then try killing one of the servers and see if this client connects to the other servers. You can try this out for differetn zoookeeper versions by using different zookeeper jar releases. This way we can find out if anything is wrong with the release you are using or its a problem in general you are seeeing. Thanks mahadev On 9/15/10 12:56 AM, "Yatir Ben Shlomo" wrote: > Thanks to all who replied, I appreciate your efforts: > > 1. There is no connections problem from the client machine: > (ob1078)(tom...@cass3:~)$ echo ruok | nc zook1 2181 > imok(ob1078)(tom...@cass3:~)$ echo ruok | nc zook2 2181 > imok(ob1078)(tom...@cass3:~)$ echo ruok | nc zook3 2181 > imok(ob1078)(tom...@cass3:~)$ > > 2. Unfortunately I have already tried to switch to the new jar but it does not > seem to be backward compatible. > It seems that the QuorumPeerConfig class does not have the following field > protected int clientPort; > It was replaced by InetSocketAddress clientPortAddress in the new jar > So I am getting java.lang.NoSuchFieldError exception... > > 3. I looked at the ClientCnxn.java code. > It seems that the logic for iterating over the available servers > (nextAddrToTry++ ) is used only inside the startConnect() function but not in > the finishConnect() function, nor anywhere else. > > Possibly something along these lines is happening: > some exception that happens inside the finishConnect() function is cauasing > the cleanup() function which in turn causes another exception. > Nowhere in this code path is the nextAddrToTry++ applied. > Can this make sense to someone ? > thanks > > > > > > > -Original Message- > From: Patrick Hunt [mailto:ph...@apache.org] > Sent: Tuesday, September 14, 2010 6:20 PM > To: zookeeper-user@hadoop.apache.org > Subject: Re: possible bug in zookeeper ? > > That is unusual. I don't recall anyone reporting a similar issue, and > looking at the code I don't see any issues off hand. Can you try the > following? > > 1) on that particular zk client machine resolve the hosts zook1/zook2/zook3, > what ip addresses does this resolve to? (try dig) > 2) try running the client using the 3.3.1 jar file (just replace the jar on > the client), it includes more log4j information, turn on DEBUG or TRACE > logging > > Patrick > > On Tue, Sep 14, 2010 at 8:44 AM, Yatir Ben Shlomo wrote: > >> zook1:2181,zook2:2181,zook3:2181 >> >> >> -Original Message- >> From: Ted Dunning [mailto:ted.dunn...@gmail.com] >> Sent: Tuesday, September 14, 2010 4:11 PM >> To: zookeeper-user@hadoop.apache.org >> Subject: Re: possible bug in zookeeper ? >> >> What was the list of servers that was given originally to open the >> connection to ZK? >> >> On Tue, Sep 14, 2010 at 6:15 AM, Yatir Ben Shlomo >> wrote: >> >>> Hi I am using solrCloud which uses an ensemble of 3 zookeeper instances. >>> >>> I am performing survivability tests: >>> Taking one of the zookeeper instances down I would expect the client to >> use >>> a different zookeeper server instance. >>> >>> But as you can see in the below logs attached >>> Depending on which instance I choose to take down (in my case, the last >>> one in the list of zookeeper servers) >>> the client is constantly insisting on the same zookeeper server >> (Attempting >>> connection to server zook3/192.168.252.78:2181) >>> and not switching to a different one >>> the problem seems to arrive from ClientCnxn.java >>> Any one has an idea on this ? >>> >>> Solr cloud currently is using zookeeper-3.2.2.jar >>> Is this a know bug that was fixed in later versions ?( 3.3.1) >>> >>> Thanks in advance, >>> Yatir >>> >>> >>> Logs: >>> >>> Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn >>> WARNING: Ignoring exception during shutdown input >>> java.nio.channels.ClosedChannelException >>>at >>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) >>>at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) >>>at >>> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java) >> :999) >>>at >>> >> org.apache.zookeeper.ClientCnxn$SendThread.run(zo
RE: possible bug in zookeeper ?
Thanks to all who replied, I appreciate your efforts: 1. There is no connections problem from the client machine: (ob1078)(tom...@cass3:~)$ echo ruok | nc zook1 2181 imok(ob1078)(tom...@cass3:~)$ echo ruok | nc zook2 2181 imok(ob1078)(tom...@cass3:~)$ echo ruok | nc zook3 2181 imok(ob1078)(tom...@cass3:~)$ 2. Unfortunately I have already tried to switch to the new jar but it does not seem to be backward compatible. It seems that the QuorumPeerConfig class does not have the following field protected int clientPort; It was replaced by InetSocketAddress clientPortAddress in the new jar So I am getting java.lang.NoSuchFieldError exception... 3. I looked at the ClientCnxn.java code. It seems that the logic for iterating over the available servers (nextAddrToTry++ ) is used only inside the startConnect() function but not in the finishConnect() function, nor anywhere else. Possibly something along these lines is happening: some exception that happens inside the finishConnect() function is cauasing the cleanup() function which in turn causes another exception. Nowhere in this code path is the nextAddrToTry++ applied. Can this make sense to someone ? thanks -Original Message- From: Patrick Hunt [mailto:ph...@apache.org] Sent: Tuesday, September 14, 2010 6:20 PM To: zookeeper-user@hadoop.apache.org Subject: Re: possible bug in zookeeper ? That is unusual. I don't recall anyone reporting a similar issue, and looking at the code I don't see any issues off hand. Can you try the following? 1) on that particular zk client machine resolve the hosts zook1/zook2/zook3, what ip addresses does this resolve to? (try dig) 2) try running the client using the 3.3.1 jar file (just replace the jar on the client), it includes more log4j information, turn on DEBUG or TRACE logging Patrick On Tue, Sep 14, 2010 at 8:44 AM, Yatir Ben Shlomo wrote: > zook1:2181,zook2:2181,zook3:2181 > > > -Original Message- > From: Ted Dunning [mailto:ted.dunn...@gmail.com] > Sent: Tuesday, September 14, 2010 4:11 PM > To: zookeeper-user@hadoop.apache.org > Subject: Re: possible bug in zookeeper ? > > What was the list of servers that was given originally to open the > connection to ZK? > > On Tue, Sep 14, 2010 at 6:15 AM, Yatir Ben Shlomo >wrote: > > > Hi I am using solrCloud which uses an ensemble of 3 zookeeper instances. > > > > I am performing survivability tests: > > Taking one of the zookeeper instances down I would expect the client to > use > > a different zookeeper server instance. > > > > But as you can see in the below logs attached > > Depending on which instance I choose to take down (in my case, the last > > one in the list of zookeeper servers) > > the client is constantly insisting on the same zookeeper server > (Attempting > > connection to server zook3/192.168.252.78:2181) > > and not switching to a different one > > the problem seems to arrive from ClientCnxn.java > > Any one has an idea on this ? > > > > Solr cloud currently is using zookeeper-3.2.2.jar > > Is this a know bug that was fixed in later versions ?( 3.3.1) > > > > Thanks in advance, > > Yatir > > > > > > Logs: > > > > Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn > > WARNING: Ignoring exception during shutdown input > > java.nio.channels.ClosedChannelException > >at > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) > >at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) > >at > > > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999) > >at > > > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970) > > Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn > > WARNING: Ignoring exception during shutdown output > > java.nio.channels.ClosedChannelException > >at > > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) > >at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) > >at > > > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):1004) > >at > > > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970) > > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info > > INFO: Attempting connection to server zook3/192.168.252.78:2181 > > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn > > WARNING: Exception closing session 0x32b105244a20001 to > > sun.nio.ch.selectionkeyi...@3ca58cbf > > java.net.ConnectException: Connection refused > >at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Metho
Re: possible bug in zookeeper ?
Also try the four letter commands to each server. On Tue, Sep 14, 2010 at 9:20 AM, Patrick Hunt wrote: > That is unusual. I don't recall anyone reporting a similar issue, and > looking at the code I don't see any issues off hand. Can you try the > following? > > 1) on that particular zk client machine resolve the hosts > zook1/zook2/zook3, > what ip addresses does this resolve to? (try dig) > 2) try running the client using the 3.3.1 jar file (just replace the jar on > the client), it includes more log4j information, turn on DEBUG or TRACE > logging > > Patrick > > On Tue, Sep 14, 2010 at 8:44 AM, Yatir Ben Shlomo >wrote: > > > zook1:2181,zook2:2181,zook3:2181 > > > > > > -Original Message- > > From: Ted Dunning [mailto:ted.dunn...@gmail.com] > > Sent: Tuesday, September 14, 2010 4:11 PM > > To: zookeeper-user@hadoop.apache.org > > Subject: Re: possible bug in zookeeper ? > > > > What was the list of servers that was given originally to open the > > connection to ZK? > > > > On Tue, Sep 14, 2010 at 6:15 AM, Yatir Ben Shlomo > >wrote: > > > > > Hi I am using solrCloud which uses an ensemble of 3 zookeeper > instances. > > > > > > I am performing survivability tests: > > > Taking one of the zookeeper instances down I would expect the client to > > use > > > a different zookeeper server instance. > > > > > > But as you can see in the below logs attached > > > Depending on which instance I choose to take down (in my case, the > last > > > one in the list of zookeeper servers) > > > the client is constantly insisting on the same zookeeper server > > (Attempting > > > connection to server zook3/192.168.252.78:2181) > > > and not switching to a different one > > > the problem seems to arrive from ClientCnxn.java > > > Any one has an idea on this ? > > > > > > Solr cloud currently is using zookeeper-3.2.2.jar > > > Is this a know bug that was fixed in later versions ?( 3.3.1) > > > > > > Thanks in advance, > > > Yatir > > > > > > > > > Logs: > > > > > > Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn > > > WARNING: Ignoring exception during shutdown input > > > java.nio.channels.ClosedChannelException > > >at > > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) > > >at > sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) > > >at > > > > > > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999) > > >at > > > > > > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970) > > > Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn > > > WARNING: Ignoring exception during shutdown output > > > java.nio.channels.ClosedChannelException > > >at > > > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) > > >at > sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) > > >at > > > > > > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):1004) > > >at > > > > > > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970) > > > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info > > > INFO: Attempting connection to server zook3/192.168.252.78:2181 > > > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn > > > WARNING: Exception closing session 0x32b105244a20001 to > > > sun.nio.ch.selectionkeyi...@3ca58cbf > > > java.net.ConnectException: Connection refused > > >at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native > Method) > > >at > > sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java) > > >at > > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > > >at > > > > > > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933) > > > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn > > > WARNING: Ignoring exception during shutdown input > > > java.nio.channels.ClosedChannelException > > >at > > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) > > >at > sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) > > >at > > > > &
Re: possible bug in zookeeper ?
That is unusual. I don't recall anyone reporting a similar issue, and looking at the code I don't see any issues off hand. Can you try the following? 1) on that particular zk client machine resolve the hosts zook1/zook2/zook3, what ip addresses does this resolve to? (try dig) 2) try running the client using the 3.3.1 jar file (just replace the jar on the client), it includes more log4j information, turn on DEBUG or TRACE logging Patrick On Tue, Sep 14, 2010 at 8:44 AM, Yatir Ben Shlomo wrote: > zook1:2181,zook2:2181,zook3:2181 > > > -Original Message- > From: Ted Dunning [mailto:ted.dunn...@gmail.com] > Sent: Tuesday, September 14, 2010 4:11 PM > To: zookeeper-user@hadoop.apache.org > Subject: Re: possible bug in zookeeper ? > > What was the list of servers that was given originally to open the > connection to ZK? > > On Tue, Sep 14, 2010 at 6:15 AM, Yatir Ben Shlomo >wrote: > > > Hi I am using solrCloud which uses an ensemble of 3 zookeeper instances. > > > > I am performing survivability tests: > > Taking one of the zookeeper instances down I would expect the client to > use > > a different zookeeper server instance. > > > > But as you can see in the below logs attached > > Depending on which instance I choose to take down (in my case, the last > > one in the list of zookeeper servers) > > the client is constantly insisting on the same zookeeper server > (Attempting > > connection to server zook3/192.168.252.78:2181) > > and not switching to a different one > > the problem seems to arrive from ClientCnxn.java > > Any one has an idea on this ? > > > > Solr cloud currently is using zookeeper-3.2.2.jar > > Is this a know bug that was fixed in later versions ?( 3.3.1) > > > > Thanks in advance, > > Yatir > > > > > > Logs: > > > > Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn > > WARNING: Ignoring exception during shutdown input > > java.nio.channels.ClosedChannelException > >at > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) > >at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) > >at > > > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999) > >at > > > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970) > > Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn > > WARNING: Ignoring exception during shutdown output > > java.nio.channels.ClosedChannelException > >at > > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) > >at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) > >at > > > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):1004) > >at > > > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970) > > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info > > INFO: Attempting connection to server zook3/192.168.252.78:2181 > > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn > > WARNING: Exception closing session 0x32b105244a20001 to > > sun.nio.ch.selectionkeyi...@3ca58cbf > > java.net.ConnectException: Connection refused > >at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method) > >at > sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java) > >at > > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > >at > > > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933) > > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn > > WARNING: Ignoring exception during shutdown input > > java.nio.channels.ClosedChannelException > >at > > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) > >at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) > >at > > > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999) > >at > > > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970) > > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn > > WARNING: Ignoring exception during shutdown output > > java.nio.channels.ClosedChannelException > >at > > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) > >at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) > >at > > > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):1
Re: possible bug in zookeeper ?
And that you can connect from every client to every server? On Tue, Sep 14, 2010 at 9:07 AM, Mahadev Konar wrote: > Hi yatir, > Can you confirm that zook1 , zook2 can be nslookedup from the client > machine? > > We havent seen a bug like this. It would be great to nail this down. > > Thanks > mahadev > > > On 9/14/10 8:44 AM, "Yatir Ben Shlomo" wrote: > > > zook1:2181,zook2:2181,zook3:2181 > > > > > > -Original Message- > > From: Ted Dunning [mailto:ted.dunn...@gmail.com] > > Sent: Tuesday, September 14, 2010 4:11 PM > > To: zookeeper-user@hadoop.apache.org > > Subject: Re: possible bug in zookeeper ? > > > > What was the list of servers that was given originally to open the > > connection to ZK? > > > > On Tue, Sep 14, 2010 at 6:15 AM, Yatir Ben Shlomo >wrote: > > > >> Hi I am using solrCloud which uses an ensemble of 3 zookeeper instances. > >> > >> I am performing survivability tests: > >> Taking one of the zookeeper instances down I would expect the client to > use > >> a different zookeeper server instance. > >> > >> But as you can see in the below logs attached > >> Depending on which instance I choose to take down (in my case, the last > >> one in the list of zookeeper servers) > >> the client is constantly insisting on the same zookeeper server > (Attempting > >> connection to server zook3/192.168.252.78:2181) > >> and not switching to a different one > >> the problem seems to arrive from ClientCnxn.java > >> Any one has an idea on this ? > >> > >> Solr cloud currently is using zookeeper-3.2.2.jar > >> Is this a know bug that was fixed in later versions ?( 3.3.1) > >> > >> Thanks in advance, > >> Yatir > >> > >> > >> Logs: > >> > >> Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn > >> WARNING: Ignoring exception during shutdown input > >> java.nio.channels.ClosedChannelException > >>at > >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) > >>at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) > >>at > >> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java) > >> :999) > >>at > >> > > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970>> > ) > >> Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn > >> WARNING: Ignoring exception during shutdown output > >> java.nio.channels.ClosedChannelException > >>at > >> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) > >>at > sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) > >>at > >> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java) > >> :1004) > >>at > >> > > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970>> > ) > >> Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info > >> INFO: Attempting connection to server zook3/192.168.252.78:2181 > >> Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn > >> WARNING: Exception closing session 0x32b105244a20001 to > >> sun.nio.ch.selectionkeyi...@3ca58cbf > >> java.net.ConnectException: Connection refused > >>at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native > Method) > >>at > sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java) > >>at > >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > >>at > >> > > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933>> > ) > >> Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn > >> WARNING: Ignoring exception during shutdown input > >> java.nio.channels.ClosedChannelException > >>at > >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) > >>at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) > >>at > >> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java) > >> :999) > >>at > >> > > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970>> > ) > >> Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn > >> WARNING: Ignoring exceptio
Re: possible bug in zookeeper ?
Hi yatir, Can you confirm that zook1 , zook2 can be nslookedup from the client machine? We havent seen a bug like this. It would be great to nail this down. Thanks mahadev On 9/14/10 8:44 AM, "Yatir Ben Shlomo" wrote: > zook1:2181,zook2:2181,zook3:2181 > > > -Original Message- > From: Ted Dunning [mailto:ted.dunn...@gmail.com] > Sent: Tuesday, September 14, 2010 4:11 PM > To: zookeeper-user@hadoop.apache.org > Subject: Re: possible bug in zookeeper ? > > What was the list of servers that was given originally to open the > connection to ZK? > > On Tue, Sep 14, 2010 at 6:15 AM, Yatir Ben Shlomo wrote: > >> Hi I am using solrCloud which uses an ensemble of 3 zookeeper instances. >> >> I am performing survivability tests: >> Taking one of the zookeeper instances down I would expect the client to use >> a different zookeeper server instance. >> >> But as you can see in the below logs attached >> Depending on which instance I choose to take down (in my case, the last >> one in the list of zookeeper servers) >> the client is constantly insisting on the same zookeeper server (Attempting >> connection to server zook3/192.168.252.78:2181) >> and not switching to a different one >> the problem seems to arrive from ClientCnxn.java >> Any one has an idea on this ? >> >> Solr cloud currently is using zookeeper-3.2.2.jar >> Is this a know bug that was fixed in later versions ?( 3.3.1) >> >> Thanks in advance, >> Yatir >> >> >> Logs: >> >> Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn >> WARNING: Ignoring exception during shutdown input >> java.nio.channels.ClosedChannelException >>at >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) >>at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) >>at >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java) >> :999) >>at >> org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970>> ) >> Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn >> WARNING: Ignoring exception during shutdown output >> java.nio.channels.ClosedChannelException >>at >> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) >>at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) >>at >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java) >> :1004) >>at >> org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970>> ) >> Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info >> INFO: Attempting connection to server zook3/192.168.252.78:2181 >> Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn >> WARNING: Exception closing session 0x32b105244a20001 to >> sun.nio.ch.selectionkeyi...@3ca58cbf >> java.net.ConnectException: Connection refused >>at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method) >>at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java) >>at >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >>at >> org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933>> ) >> Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn >> WARNING: Ignoring exception during shutdown input >> java.nio.channels.ClosedChannelException >>at >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) >>at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) >>at >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java) >> :999) >>at >> org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970>> ) >> Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn >> WARNING: Ignoring exception during shutdown output >> java.nio.channels.ClosedChannelException >>at >> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) >>at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) >>at >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java) >> :1004) >>at >> org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970>> ) >> Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info >> INFO: Attempting connection to server zook3/192.168.252.78:2181 >> Sep 14, 2010 9:02:22 AM org.apache.log4j.Category war
RE: possible bug in zookeeper ?
zook1:2181,zook2:2181,zook3:2181 -Original Message- From: Ted Dunning [mailto:ted.dunn...@gmail.com] Sent: Tuesday, September 14, 2010 4:11 PM To: zookeeper-user@hadoop.apache.org Subject: Re: possible bug in zookeeper ? What was the list of servers that was given originally to open the connection to ZK? On Tue, Sep 14, 2010 at 6:15 AM, Yatir Ben Shlomo wrote: > Hi I am using solrCloud which uses an ensemble of 3 zookeeper instances. > > I am performing survivability tests: > Taking one of the zookeeper instances down I would expect the client to use > a different zookeeper server instance. > > But as you can see in the below logs attached > Depending on which instance I choose to take down (in my case, the last > one in the list of zookeeper servers) > the client is constantly insisting on the same zookeeper server (Attempting > connection to server zook3/192.168.252.78:2181) > and not switching to a different one > the problem seems to arrive from ClientCnxn.java > Any one has an idea on this ? > > Solr cloud currently is using zookeeper-3.2.2.jar > Is this a know bug that was fixed in later versions ?( 3.3.1) > > Thanks in advance, > Yatir > > > Logs: > > Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn > WARNING: Ignoring exception during shutdown input > java.nio.channels.ClosedChannelException >at > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) >at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) >at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999) >at > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970) > Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn > WARNING: Ignoring exception during shutdown output > java.nio.channels.ClosedChannelException >at > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) >at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) >at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):1004) >at > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970) > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info > INFO: Attempting connection to server zook3/192.168.252.78:2181 > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn > WARNING: Exception closing session 0x32b105244a20001 to > sun.nio.ch.selectionkeyi...@3ca58cbf > java.net.ConnectException: Connection refused >at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method) >at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java) >at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >at > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933) > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn > WARNING: Ignoring exception during shutdown input > java.nio.channels.ClosedChannelException >at > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) >at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) >at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999) >at > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970) > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn > WARNING: Ignoring exception during shutdown output > java.nio.channels.ClosedChannelException >at > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) >at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) >at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):1004) >at > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970) > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info > INFO: Attempting connection to server zook3/192.168.252.78:2181 > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn > WARNING: Exception closing session 0x32b105244a2 to > sun.nio.ch.selectionkeyi...@3960f81b > java.net.ConnectException: Connection refused >at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method) >at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java) >at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >at > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933) > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn > WARNING: Ignoring exception during shutdown input > java.nio.channels.ClosedChannelException >at > sun.nio.ch.SocketChannel
Re: possible bug in zookeeper ?
What was the list of servers that was given originally to open the connection to ZK? On Tue, Sep 14, 2010 at 6:15 AM, Yatir Ben Shlomo wrote: > Hi I am using solrCloud which uses an ensemble of 3 zookeeper instances. > > I am performing survivability tests: > Taking one of the zookeeper instances down I would expect the client to use > a different zookeeper server instance. > > But as you can see in the below logs attached > Depending on which instance I choose to take down (in my case, the last > one in the list of zookeeper servers) > the client is constantly insisting on the same zookeeper server (Attempting > connection to server zook3/192.168.252.78:2181) > and not switching to a different one > the problem seems to arrive from ClientCnxn.java > Any one has an idea on this ? > > Solr cloud currently is using zookeeper-3.2.2.jar > Is this a know bug that was fixed in later versions ?( 3.3.1) > > Thanks in advance, > Yatir > > > Logs: > > Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn > WARNING: Ignoring exception during shutdown input > java.nio.channels.ClosedChannelException >at > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) >at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) >at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999) >at > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970) > Sep 14, 2010 9:02:20 AM org.apache.log4j.Category warn > WARNING: Ignoring exception during shutdown output > java.nio.channels.ClosedChannelException >at > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) >at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) >at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):1004) >at > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970) > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info > INFO: Attempting connection to server zook3/192.168.252.78:2181 > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn > WARNING: Exception closing session 0x32b105244a20001 to > sun.nio.ch.selectionkeyi...@3ca58cbf > java.net.ConnectException: Connection refused >at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method) >at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java) >at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >at > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933) > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn > WARNING: Ignoring exception during shutdown input > java.nio.channels.ClosedChannelException >at > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) >at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) >at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999) >at > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970) > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn > WARNING: Ignoring exception during shutdown output > java.nio.channels.ClosedChannelException >at > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) >at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) >at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):1004) >at > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970) > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category info > INFO: Attempting connection to server zook3/192.168.252.78:2181 > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn > WARNING: Exception closing session 0x32b105244a2 to > sun.nio.ch.selectionkeyi...@3960f81b > java.net.ConnectException: Connection refused >at sun.nio.ch.SocketChannelImpl.$$YJP$$checkConnect(Native Method) >at sun.nio.ch.SocketChannelImpl.checkConnect(SocketChannelImpl.java) >at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >at > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):933) > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn > WARNING: Ignoring exception during shutdown input > java.nio.channels.ClosedChannelException >at > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) >at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) >at > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(zookeeper:ClientCnxn.java):999) >at > org.apache.zookeeper.ClientCnxn$SendThread.run(zookeeper:ClientCnxn.java):970) > Sep 14, 2010 9:02:22 AM org.apache.log4j.Category warn > WARNING: Ignoring exception during shutdown output > java.nio.channels.ClosedChannelException >at > sun.nio.ch.SocketChannelImpl.shutdownOutp
