Nevermind. I figured it. It was an hbase issue. We were leaking a client reference. Sorry for the noise, St.Ack
On Sat, Sep 4, 2010 at 10:58 AM, Stack <st...@duboce.net> wrote: > Thats right -- client is shutdown first, then server... How do I stop > the client trying to come back from the dead? > Good on you Mahadev? > St.Ack > > On Fri, Sep 3, 2010 at 8:36 PM, Mahadev Konar <maha...@yahoo-inc.com> wrote: >> >> Hi Stack, >> Looks like you are shutting down the server and shutting down the client at >> the same time? Is that the issue? >> >> Thanks >> mahadev >> >> On 9/3/10 4:47 PM, "Stack" <st...@duboce.net> wrote: >> >>> Have you fellas seen this before? I call close on zookeeper but it insists >>> on doing the below exceptions. Why is it doing this 'Session >>> 0x12ad9dccda30002 >>> for server null, unexpected error, closing socket connection and >>> attempting reconnect'? This would seem to come after the close has >>> been noticed and looking in code, i'd think we'd not do this since the >>> close flag should be set to true post call to close? >>> >>> Thanks lads (The below looks ugly in our logs... this is zk 3.3.1), >>> St.Ack >>> >>> 2010-09-03 16:09:52,369 INFO >>> org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection >>> for client /fe80:0:0:0:0:0:0:1%1:56941 which had sessionid >>> 0x12ad9dccda30001 >>> 2010-09-03 16:09:52,369 INFO >>> org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection >>> for client /127.0.0.1:56942 which had sessionid 0x12ad9dccda30002 >>> 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable >>> to read additional data from server sessionid 0x12ad9dccda30001, >>> likely server has closed socket, closing socket connection and >>> attempting reconnect >>> 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable >>> to read additional data from server sessionid 0x12ad9dccda30002, >>> likely server has closed socket, closing socket connection and >>> attempting reconnect >>> 2010-09-03 16:09:52,370 INFO >>> org.apache.zookeeper.server.NIOServerCnxn: NIOServerCnxn factory >>> exited run method >>> 2010-09-03 16:09:52,370 INFO >>> org.apache.zookeeper.server.PrepRequestProcessor: PrepRequestProcessor >>> exited loop! >>> 2010-09-03 16:09:52,370 INFO >>> org.apache.zookeeper.server.SyncRequestProcessor: SyncRequestProcessor >>> exited! >>> 2010-09-03 16:09:52,370 INFO >>> org.apache.zookeeper.server.FinalRequestProcessor: shutdown of request >>> processor complete >>> 2010-09-03 16:09:52,470 DEBUG >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: <localhost:/hbase> >>> Received ZooKeeper Event, type=None, state=Disconnected, path=null >>> 2010-09-03 16:09:52,470 INFO >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: <localhost:/hbase> >>> Received Disconnected from ZooKeeper, ignoring >>> 2010-09-03 16:09:52,471 DEBUG >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: <localhost:/hbase> >>> Received ZooKeeper Event, type=None, state=Disconnected, path=null >>> 2010-09-03 16:09:52,471 INFO >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: <localhost:/hbase> >>> Received Disconnected from ZooKeeper, ignoring >>> 2010-09-03 16:09:52,857 INFO org.apache.zookeeper.ClientCnxn: Opening >>> socket connection to server localhost/0:0:0:0:0:0:0:1:2181 >>> 2010-09-03 16:09:52,858 WARN org.apache.zookeeper.ClientCnxn: Session >>> 0x12ad9dccda30001 for server null, unexpected error, closing socket >>> connection and attempting reconnect >>> java.net.ConnectException: Connection refused >>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) >>> 2010-09-03 16:09:53,149 INFO org.apache.zookeeper.ClientCnxn: Opening >>> socket connection to server localhost/fe80:0:0:0:0:0:0:1%1:2181 >>> 2010-09-03 16:09:53,150 WARN org.apache.zookeeper.ClientCnxn: Session >>> 0x12ad9dccda30002 for server null, unexpected error, closing socket >>> connection and attempting reconnect >>> java.net.ConnectException: Connection refused >>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) >>> 2010-09-03 16:09:53,576 INFO org.apache.zookeeper.ClientCnxn: Opening >>> socket connection to server localhost/127.0.0.1:2181 >>> 2010-09-03 16:09:53,576 WARN org.apache.zookeeper.ClientCnxn: Session >>> 0x12ad9dccda30001 for server null, unexpected error, closing socket >>> connection and attempting reconnect >>> java.net.ConnectException: Connection refused >>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) >>> 2010-09-03 16:09:54,000 INFO >>> org.apache.zookeeper.server.SessionTrackerImpl: SessionTrackerImpl >>> exited loop! >>> 2010-09-03 16:09:54,002 DEBUG >>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Closed >>> zookeeper sessionid=0x12ad9dccda30001 >>> 2010-09-03 16:09:54,129 INFO org.apache.zookeeper.ClientCnxn: Opening >>> socket connection to server localhost/0:0:0:0:0:0:0:1:2181 >>> 2010-09-03 16:09:54,130 WARN org.apache.zookeeper.ClientCnxn: Session >>> 0x12ad9dccda30002 for server null, unexpected error, closing socket >>> connection and attempting reconnect >>> java.net.ConnectException: Connection refused >>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) >>> 2010-09-03 16:09:54,680 INFO org.apache.zookeeper.ClientCnxn: Opening >>> socket connection to server localhost/fe80:0:0:0:0:0:0:1%1:2181 >>> 2010-09-03 16:09:54,782 INFO org.apache.zookeeper.ZooKeeper: Session: >>> 0x12ad9dccda30001 closed >>> 2010-09-03 16:09:54,883 DEBUG >>> org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Closed >>> zookeeper sessionid=0x12ad9dccda30002 >>> 2010-09-03 16:09:55,420 INFO org.apache.zookeeper.ClientCnxn: Opening >>> socket connection to server localhost/127.0.0.1:2181 >>> 2010-09-03 16:09:55,522 INFO org.apache.zookeeper.ZooKeeper: Session: >>> 0x12ad9dccda30002 closed >>> >> >> >
