Re: Spew after call to close

2010-09-08 Thread Patrick Hunt
No worries, let us know if something else pops up.

Patrick

On Tue, Sep 7, 2010 at 3:10 PM, Stack  wrote:

> Nevermind.  I figured it.  It was an hbase issue.  We were leaking a
> client reference.
> Sorry for the noise,
> St.Ack
>
>
> On Sat, Sep 4, 2010 at 10:58 AM, Stack  wrote:
> > Thats right -- client is shutdown first, then server... How do I stop
> > the client trying to come back from the dead?
> > Good on you Mahadev?
> > St.Ack
> >
> > On Fri, Sep 3, 2010 at 8:36 PM, Mahadev Konar 
> wrote:
> >>
> >> Hi Stack,
> >>  Looks like you are shutting down the server and shutting down the
> client at
> >> the same time? Is that the issue?
> >>
> >> Thanks
> >> mahadev
> >>
> >> On 9/3/10 4:47 PM, "Stack"  wrote:
> >>
> >>> Have you fellas seen this before? I call close on zookeeper but it
> insists
> >>> on doing the below exceptions.  Why is it doing this 'Session
> >>> 0x12ad9dccda30002
> >>> for server null, unexpected error, closing socket connection and
> >>> attempting reconnect'?   This would seem to come after the close has
> >>> been noticed and looking in code, i'd think we'd not do this since the
> >>> close flag should be set to true post call to close?
> >>>
> >>> Thanks lads  (The below looks ugly in our logs... this is zk 3.3.1),
> >>> St.Ack
> >>>
> >>> 2010-09-03 16:09:52,369 INFO
> >>> org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection
> >>> for client /fe80:0:0:0:0:0:0:1%1:56941 which had sessionid
> >>> 0x12ad9dccda30001
> >>> 2010-09-03 16:09:52,369 INFO
> >>> org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection
> >>> for client /127.0.0.1:56942 which had sessionid 0x12ad9dccda30002
> >>> 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable
> >>> to read additional data from server sessionid 0x12ad9dccda30001,
> >>> likely server has closed socket, closing socket connection and
> >>> attempting reconnect
> >>> 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable
> >>> to read additional data from server sessionid 0x12ad9dccda30002,
> >>> likely server has closed socket, closing socket connection and
> >>> attempting reconnect
> >>> 2010-09-03 16:09:52,370 INFO
> >>> org.apache.zookeeper.server.NIOServerCnxn: NIOServerCnxn factory
> >>> exited run method
> >>> 2010-09-03 16:09:52,370 INFO
> >>> org.apache.zookeeper.server.PrepRequestProcessor: PrepRequestProcessor
> >>> exited loop!
> >>> 2010-09-03 16:09:52,370 INFO
> >>> org.apache.zookeeper.server.SyncRequestProcessor: SyncRequestProcessor
> >>> exited!
> >>> 2010-09-03 16:09:52,370 INFO
> >>> org.apache.zookeeper.server.FinalRequestProcessor: shutdown of request
> >>> processor complete
> >>> 2010-09-03 16:09:52,470 DEBUG
> >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
> >>> Received ZooKeeper Event, type=None, state=Disconnected, path=null
> >>> 2010-09-03 16:09:52,470 INFO
> >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
> >>> Received Disconnected from ZooKeeper, ignoring
> >>> 2010-09-03 16:09:52,471 DEBUG
> >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
> >>> Received ZooKeeper Event, type=None, state=Disconnected, path=null
> >>> 2010-09-03 16:09:52,471 INFO
> >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
> >>> Received Disconnected from ZooKeeper, ignoring
> >>> 2010-09-03 16:09:52,857 INFO org.apache.zookeeper.ClientCnxn: Opening
> >>> socket connection to server localhost/0:0:0:0:0:0:0:1:2181
> >>> 2010-09-03 16:09:52,858 WARN org.apache.zookeeper.ClientCnxn: Session
> >>> 0x12ad9dccda30001 for server null, unexpected error, closing socket
> >>> connection and attempting reconnect
> >>> java.net.ConnectException: Connection refused
> >>>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>>at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> >>>at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
> >>> 2010-09-03 16:09:53,149 INFO org.apache.zookeeper.ClientCnxn: Opening
> >>> socket connection to server localhost/fe80:0:0:0:0:0:0:1%1:2181
> >>> 2010-09-03 16:09:53,150 WARN org.apache.zookeeper.ClientCnxn: Session
> >>> 0x12ad9dccda30002 for server null, unexpected error, closing socket
> >>> connection and attempting reconnect
> >>> java.net.ConnectException: Connection refused
> >>>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>>at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> >>>at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
> >>> 2010-09-03 16:09:53,576 INFO org.apache.zookeeper.ClientCnxn: Opening
> >>> socket connection to server localhost/127.0.0.1:2181
> >>> 2010-09-03 16:09:53,576 WARN org.apache.zookeeper.ClientCnxn: Session
> >>> 0x12ad9dccda30001 for server null, unexpected error, closing socket
> >>> connection and attempting reconnect
> >>> java.net.ConnectException: Connection refused
> >>>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Metho

Re: Spew after call to close

2010-09-08 Thread Patrick Hunt
No worries, let us know if something else pops up.

Patrick

On Tue, Sep 7, 2010 at 3:10 PM, Stack  wrote:

> Nevermind.  I figured it.  It was an hbase issue.  We were leaking a
> client reference.
> Sorry for the noise,
> St.Ack
>
>
> On Sat, Sep 4, 2010 at 10:58 AM, Stack  wrote:
> > Thats right -- client is shutdown first, then server... How do I stop
> > the client trying to come back from the dead?
> > Good on you Mahadev?
> > St.Ack
> >
> > On Fri, Sep 3, 2010 at 8:36 PM, Mahadev Konar 
> wrote:
> >>
> >> Hi Stack,
> >>  Looks like you are shutting down the server and shutting down the
> client at
> >> the same time? Is that the issue?
> >>
> >> Thanks
> >> mahadev
> >>
> >> On 9/3/10 4:47 PM, "Stack"  wrote:
> >>
> >>> Have you fellas seen this before? I call close on zookeeper but it
> insists
> >>> on doing the below exceptions.  Why is it doing this 'Session
> >>> 0x12ad9dccda30002
> >>> for server null, unexpected error, closing socket connection and
> >>> attempting reconnect'?   This would seem to come after the close has
> >>> been noticed and looking in code, i'd think we'd not do this since the
> >>> close flag should be set to true post call to close?
> >>>
> >>> Thanks lads  (The below looks ugly in our logs... this is zk 3.3.1),
> >>> St.Ack
> >>>
> >>> 2010-09-03 16:09:52,369 INFO
> >>> org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection
> >>> for client /fe80:0:0:0:0:0:0:1%1:56941 which had sessionid
> >>> 0x12ad9dccda30001
> >>> 2010-09-03 16:09:52,369 INFO
> >>> org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection
> >>> for client /127.0.0.1:56942 which had sessionid 0x12ad9dccda30002
> >>> 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable
> >>> to read additional data from server sessionid 0x12ad9dccda30001,
> >>> likely server has closed socket, closing socket connection and
> >>> attempting reconnect
> >>> 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable
> >>> to read additional data from server sessionid 0x12ad9dccda30002,
> >>> likely server has closed socket, closing socket connection and
> >>> attempting reconnect
> >>> 2010-09-03 16:09:52,370 INFO
> >>> org.apache.zookeeper.server.NIOServerCnxn: NIOServerCnxn factory
> >>> exited run method
> >>> 2010-09-03 16:09:52,370 INFO
> >>> org.apache.zookeeper.server.PrepRequestProcessor: PrepRequestProcessor
> >>> exited loop!
> >>> 2010-09-03 16:09:52,370 INFO
> >>> org.apache.zookeeper.server.SyncRequestProcessor: SyncRequestProcessor
> >>> exited!
> >>> 2010-09-03 16:09:52,370 INFO
> >>> org.apache.zookeeper.server.FinalRequestProcessor: shutdown of request
> >>> processor complete
> >>> 2010-09-03 16:09:52,470 DEBUG
> >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
> >>> Received ZooKeeper Event, type=None, state=Disconnected, path=null
> >>> 2010-09-03 16:09:52,470 INFO
> >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
> >>> Received Disconnected from ZooKeeper, ignoring
> >>> 2010-09-03 16:09:52,471 DEBUG
> >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
> >>> Received ZooKeeper Event, type=None, state=Disconnected, path=null
> >>> 2010-09-03 16:09:52,471 INFO
> >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
> >>> Received Disconnected from ZooKeeper, ignoring
> >>> 2010-09-03 16:09:52,857 INFO org.apache.zookeeper.ClientCnxn: Opening
> >>> socket connection to server localhost/0:0:0:0:0:0:0:1:2181
> >>> 2010-09-03 16:09:52,858 WARN org.apache.zookeeper.ClientCnxn: Session
> >>> 0x12ad9dccda30001 for server null, unexpected error, closing socket
> >>> connection and attempting reconnect
> >>> java.net.ConnectException: Connection refused
> >>>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>>at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> >>>at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
> >>> 2010-09-03 16:09:53,149 INFO org.apache.zookeeper.ClientCnxn: Opening
> >>> socket connection to server localhost/fe80:0:0:0:0:0:0:1%1:2181
> >>> 2010-09-03 16:09:53,150 WARN org.apache.zookeeper.ClientCnxn: Session
> >>> 0x12ad9dccda30002 for server null, unexpected error, closing socket
> >>> connection and attempting reconnect
> >>> java.net.ConnectException: Connection refused
> >>>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >>>at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
> >>>at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
> >>> 2010-09-03 16:09:53,576 INFO org.apache.zookeeper.ClientCnxn: Opening
> >>> socket connection to server localhost/127.0.0.1:2181
> >>> 2010-09-03 16:09:53,576 WARN org.apache.zookeeper.ClientCnxn: Session
> >>> 0x12ad9dccda30001 for server null, unexpected error, closing socket
> >>> connection and attempting reconnect
> >>> java.net.ConnectException: Connection refused
> >>>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Metho

Re: Spew after call to close

2010-09-07 Thread Stack
Nevermind.  I figured it.  It was an hbase issue.  We were leaking a
client reference.
Sorry for the noise,
St.Ack


On Sat, Sep 4, 2010 at 10:58 AM, Stack  wrote:
> Thats right -- client is shutdown first, then server... How do I stop
> the client trying to come back from the dead?
> Good on you Mahadev?
> St.Ack
>
> On Fri, Sep 3, 2010 at 8:36 PM, Mahadev Konar  wrote:
>>
>> Hi Stack,
>>  Looks like you are shutting down the server and shutting down the client at
>> the same time? Is that the issue?
>>
>> Thanks
>> mahadev
>>
>> On 9/3/10 4:47 PM, "Stack"  wrote:
>>
>>> Have you fellas seen this before? I call close on zookeeper but it insists
>>> on doing the below exceptions.  Why is it doing this 'Session
>>> 0x12ad9dccda30002
>>> for server null, unexpected error, closing socket connection and
>>> attempting reconnect'?   This would seem to come after the close has
>>> been noticed and looking in code, i'd think we'd not do this since the
>>> close flag should be set to true post call to close?
>>>
>>> Thanks lads  (The below looks ugly in our logs... this is zk 3.3.1),
>>> St.Ack
>>>
>>> 2010-09-03 16:09:52,369 INFO
>>> org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection
>>> for client /fe80:0:0:0:0:0:0:1%1:56941 which had sessionid
>>> 0x12ad9dccda30001
>>> 2010-09-03 16:09:52,369 INFO
>>> org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection
>>> for client /127.0.0.1:56942 which had sessionid 0x12ad9dccda30002
>>> 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable
>>> to read additional data from server sessionid 0x12ad9dccda30001,
>>> likely server has closed socket, closing socket connection and
>>> attempting reconnect
>>> 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable
>>> to read additional data from server sessionid 0x12ad9dccda30002,
>>> likely server has closed socket, closing socket connection and
>>> attempting reconnect
>>> 2010-09-03 16:09:52,370 INFO
>>> org.apache.zookeeper.server.NIOServerCnxn: NIOServerCnxn factory
>>> exited run method
>>> 2010-09-03 16:09:52,370 INFO
>>> org.apache.zookeeper.server.PrepRequestProcessor: PrepRequestProcessor
>>> exited loop!
>>> 2010-09-03 16:09:52,370 INFO
>>> org.apache.zookeeper.server.SyncRequestProcessor: SyncRequestProcessor
>>> exited!
>>> 2010-09-03 16:09:52,370 INFO
>>> org.apache.zookeeper.server.FinalRequestProcessor: shutdown of request
>>> processor complete
>>> 2010-09-03 16:09:52,470 DEBUG
>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
>>> Received ZooKeeper Event, type=None, state=Disconnected, path=null
>>> 2010-09-03 16:09:52,470 INFO
>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
>>> Received Disconnected from ZooKeeper, ignoring
>>> 2010-09-03 16:09:52,471 DEBUG
>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
>>> Received ZooKeeper Event, type=None, state=Disconnected, path=null
>>> 2010-09-03 16:09:52,471 INFO
>>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
>>> Received Disconnected from ZooKeeper, ignoring
>>> 2010-09-03 16:09:52,857 INFO org.apache.zookeeper.ClientCnxn: Opening
>>> socket connection to server localhost/0:0:0:0:0:0:0:1:2181
>>> 2010-09-03 16:09:52,858 WARN org.apache.zookeeper.ClientCnxn: Session
>>> 0x12ad9dccda30001 for server null, unexpected error, closing socket
>>> connection and attempting reconnect
>>> java.net.ConnectException: Connection refused
>>>    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>>>    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
>>> 2010-09-03 16:09:53,149 INFO org.apache.zookeeper.ClientCnxn: Opening
>>> socket connection to server localhost/fe80:0:0:0:0:0:0:1%1:2181
>>> 2010-09-03 16:09:53,150 WARN org.apache.zookeeper.ClientCnxn: Session
>>> 0x12ad9dccda30002 for server null, unexpected error, closing socket
>>> connection and attempting reconnect
>>> java.net.ConnectException: Connection refused
>>>    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>>>    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
>>> 2010-09-03 16:09:53,576 INFO org.apache.zookeeper.ClientCnxn: Opening
>>> socket connection to server localhost/127.0.0.1:2181
>>> 2010-09-03 16:09:53,576 WARN org.apache.zookeeper.ClientCnxn: Session
>>> 0x12ad9dccda30001 for server null, unexpected error, closing socket
>>> connection and attempting reconnect
>>> java.net.ConnectException: Connection refused
>>>    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>>>    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
>>> 2010-09-03 16:09:54,000 INFO
>>> org.apache.zookeeper.server.SessionTrackerImpl: SessionTrackerImpl
>>> exited loop!
>>> 2010-09-03 16:09:54,002 DEBU

Re: Spew after call to close

2010-09-03 Thread Mahadev Konar

Hi Stack,
 Looks like you are shutting down the server and shutting down the client at
the same time? Is that the issue?

Thanks
mahadev

On 9/3/10 4:47 PM, "Stack"  wrote:

> Have you fellas seen this before? I call close on zookeeper but it insists
> on doing the below exceptions.  Why is it doing this 'Session
> 0x12ad9dccda30002
> for server null, unexpected error, closing socket connection and
> attempting reconnect'?   This would seem to come after the close has
> been noticed and looking in code, i'd think we'd not do this since the
> close flag should be set to true post call to close?
> 
> Thanks lads  (The below looks ugly in our logs... this is zk 3.3.1),
> St.Ack
> 
> 2010-09-03 16:09:52,369 INFO
> org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection
> for client /fe80:0:0:0:0:0:0:1%1:56941 which had sessionid
> 0x12ad9dccda30001
> 2010-09-03 16:09:52,369 INFO
> org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection
> for client /127.0.0.1:56942 which had sessionid 0x12ad9dccda30002
> 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable
> to read additional data from server sessionid 0x12ad9dccda30001,
> likely server has closed socket, closing socket connection and
> attempting reconnect
> 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable
> to read additional data from server sessionid 0x12ad9dccda30002,
> likely server has closed socket, closing socket connection and
> attempting reconnect
> 2010-09-03 16:09:52,370 INFO
> org.apache.zookeeper.server.NIOServerCnxn: NIOServerCnxn factory
> exited run method
> 2010-09-03 16:09:52,370 INFO
> org.apache.zookeeper.server.PrepRequestProcessor: PrepRequestProcessor
> exited loop!
> 2010-09-03 16:09:52,370 INFO
> org.apache.zookeeper.server.SyncRequestProcessor: SyncRequestProcessor
> exited!
> 2010-09-03 16:09:52,370 INFO
> org.apache.zookeeper.server.FinalRequestProcessor: shutdown of request
> processor complete
> 2010-09-03 16:09:52,470 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
> Received ZooKeeper Event, type=None, state=Disconnected, path=null
> 2010-09-03 16:09:52,470 INFO
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
> Received Disconnected from ZooKeeper, ignoring
> 2010-09-03 16:09:52,471 DEBUG
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
> Received ZooKeeper Event, type=None, state=Disconnected, path=null
> 2010-09-03 16:09:52,471 INFO
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
> Received Disconnected from ZooKeeper, ignoring
> 2010-09-03 16:09:52,857 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server localhost/0:0:0:0:0:0:0:1:2181
> 2010-09-03 16:09:52,858 WARN org.apache.zookeeper.ClientCnxn: Session
> 0x12ad9dccda30001 for server null, unexpected error, closing socket
> connection and attempting reconnect
> java.net.ConnectException: Connection refused
>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
> 2010-09-03 16:09:53,149 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server localhost/fe80:0:0:0:0:0:0:1%1:2181
> 2010-09-03 16:09:53,150 WARN org.apache.zookeeper.ClientCnxn: Session
> 0x12ad9dccda30002 for server null, unexpected error, closing socket
> connection and attempting reconnect
> java.net.ConnectException: Connection refused
>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
> 2010-09-03 16:09:53,576 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server localhost/127.0.0.1:2181
> 2010-09-03 16:09:53,576 WARN org.apache.zookeeper.ClientCnxn: Session
> 0x12ad9dccda30001 for server null, unexpected error, closing socket
> connection and attempting reconnect
> java.net.ConnectException: Connection refused
>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
>at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
> 2010-09-03 16:09:54,000 INFO
> org.apache.zookeeper.server.SessionTrackerImpl: SessionTrackerImpl
> exited loop!
> 2010-09-03 16:09:54,002 DEBUG
> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Closed
> zookeeper sessionid=0x12ad9dccda30001
> 2010-09-03 16:09:54,129 INFO org.apache.zookeeper.ClientCnxn: Opening
> socket connection to server localhost/0:0:0:0:0:0:0:1:2181
> 2010-09-03 16:09:54,130 WARN org.apache.zookeeper.ClientCnxn: Session
> 0x12ad9dccda30002 for server null, unexpected error, closing socket
> connection and attempting reconnect
> java.net.ConnectException: Connection refused
>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>at sun.nio.ch.

Spew after call to close

2010-09-03 Thread Stack
Have you fellas seen this before? I call close on zookeeper but it insists
on doing the below exceptions.  Why is it doing this 'Session 0x12ad9dccda30002
for server null, unexpected error, closing socket connection and
attempting reconnect'?   This would seem to come after the close has
been noticed and looking in code, i'd think we'd not do this since the
close flag should be set to true post call to close?

Thanks lads  (The below looks ugly in our logs... this is zk 3.3.1),
St.Ack

2010-09-03 16:09:52,369 INFO
org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection
for client /fe80:0:0:0:0:0:0:1%1:56941 which had sessionid
0x12ad9dccda30001
2010-09-03 16:09:52,369 INFO
org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection
for client /127.0.0.1:56942 which had sessionid 0x12ad9dccda30002
2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable
to read additional data from server sessionid 0x12ad9dccda30001,
likely server has closed socket, closing socket connection and
attempting reconnect
2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable
to read additional data from server sessionid 0x12ad9dccda30002,
likely server has closed socket, closing socket connection and
attempting reconnect
2010-09-03 16:09:52,370 INFO
org.apache.zookeeper.server.NIOServerCnxn: NIOServerCnxn factory
exited run method
2010-09-03 16:09:52,370 INFO
org.apache.zookeeper.server.PrepRequestProcessor: PrepRequestProcessor
exited loop!
2010-09-03 16:09:52,370 INFO
org.apache.zookeeper.server.SyncRequestProcessor: SyncRequestProcessor
exited!
2010-09-03 16:09:52,370 INFO
org.apache.zookeeper.server.FinalRequestProcessor: shutdown of request
processor complete
2010-09-03 16:09:52,470 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
Received ZooKeeper Event, type=None, state=Disconnected, path=null
2010-09-03 16:09:52,470 INFO
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
Received Disconnected from ZooKeeper, ignoring
2010-09-03 16:09:52,471 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
Received ZooKeeper Event, type=None, state=Disconnected, path=null
2010-09-03 16:09:52,471 INFO
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: 
Received Disconnected from ZooKeeper, ignoring
2010-09-03 16:09:52,857 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server localhost/0:0:0:0:0:0:0:1:2181
2010-09-03 16:09:52,858 WARN org.apache.zookeeper.ClientCnxn: Session
0x12ad9dccda30001 for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
   at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
   at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
2010-09-03 16:09:53,149 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server localhost/fe80:0:0:0:0:0:0:1%1:2181
2010-09-03 16:09:53,150 WARN org.apache.zookeeper.ClientCnxn: Session
0x12ad9dccda30002 for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
   at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
   at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
2010-09-03 16:09:53,576 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server localhost/127.0.0.1:2181
2010-09-03 16:09:53,576 WARN org.apache.zookeeper.ClientCnxn: Session
0x12ad9dccda30001 for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
   at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
   at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
2010-09-03 16:09:54,000 INFO
org.apache.zookeeper.server.SessionTrackerImpl: SessionTrackerImpl
exited loop!
2010-09-03 16:09:54,002 DEBUG
org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Closed
zookeeper sessionid=0x12ad9dccda30001
2010-09-03 16:09:54,129 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server localhost/0:0:0:0:0:0:0:1:2181
2010-09-03 16:09:54,130 WARN org.apache.zookeeper.ClientCnxn: Session
0x12ad9dccda30002 for server null, unexpected error, closing socket
connection and attempting reconnect
java.net.ConnectException: Connection refused
   at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
   at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
   at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078)
2010-09-03 16:09:54,680 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server localhost/fe80:0:0:0:0:0:0:1%1:2181
2010-09-03 16:09:54,782 INFO org.apache.zookeeper.ZooKeeper: Session:
0x12ad9dccda30001