Re: Spew after call to close
No worries, let us know if something else pops up. Patrick On Tue, Sep 7, 2010 at 3:10 PM, Stack wrote: > Nevermind. I figured it. It was an hbase issue. We were leaking a > client reference. > Sorry for the noise, > St.Ack > > > On Sat, Sep 4, 2010 at 10:58 AM, Stack wrote: > > Thats right -- client is shutdown first, then server... How do I stop > > the client trying to come back from the dead? > > Good on you Mahadev? > > St.Ack > > > > On Fri, Sep 3, 2010 at 8:36 PM, Mahadev Konar > wrote: > >> > >> Hi Stack, > >> Looks like you are shutting down the server and shutting down the > client at > >> the same time? Is that the issue? > >> > >> Thanks > >> mahadev > >> > >> On 9/3/10 4:47 PM, "Stack" wrote: > >> > >>> Have you fellas seen this before? I call close on zookeeper but it > insists > >>> on doing the below exceptions. Why is it doing this 'Session > >>> 0x12ad9dccda30002 > >>> for server null, unexpected error, closing socket connection and > >>> attempting reconnect'? This would seem to come after the close has > >>> been noticed and looking in code, i'd think we'd not do this since the > >>> close flag should be set to true post call to close? > >>> > >>> Thanks lads (The below looks ugly in our logs... this is zk 3.3.1), > >>> St.Ack > >>> > >>> 2010-09-03 16:09:52,369 INFO > >>> org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection > >>> for client /fe80:0:0:0:0:0:0:1%1:56941 which had sessionid > >>> 0x12ad9dccda30001 > >>> 2010-09-03 16:09:52,369 INFO > >>> org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection > >>> for client /127.0.0.1:56942 which had sessionid 0x12ad9dccda30002 > >>> 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable > >>> to read additional data from server sessionid 0x12ad9dccda30001, > >>> likely server has closed socket, closing socket connection and > >>> attempting reconnect > >>> 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable > >>> to read additional data from server sessionid 0x12ad9dccda30002, > >>> likely server has closed socket, closing socket connection and > >>> attempting reconnect > >>> 2010-09-03 16:09:52,370 INFO > >>> org.apache.zookeeper.server.NIOServerCnxn: NIOServerCnxn factory > >>> exited run method > >>> 2010-09-03 16:09:52,370 INFO > >>> org.apache.zookeeper.server.PrepRequestProcessor: PrepRequestProcessor > >>> exited loop! > >>> 2010-09-03 16:09:52,370 INFO > >>> org.apache.zookeeper.server.SyncRequestProcessor: SyncRequestProcessor > >>> exited! > >>> 2010-09-03 16:09:52,370 INFO > >>> org.apache.zookeeper.server.FinalRequestProcessor: shutdown of request > >>> processor complete > >>> 2010-09-03 16:09:52,470 DEBUG > >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: > >>> Received ZooKeeper Event, type=None, state=Disconnected, path=null > >>> 2010-09-03 16:09:52,470 INFO > >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: > >>> Received Disconnected from ZooKeeper, ignoring > >>> 2010-09-03 16:09:52,471 DEBUG > >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: > >>> Received ZooKeeper Event, type=None, state=Disconnected, path=null > >>> 2010-09-03 16:09:52,471 INFO > >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: > >>> Received Disconnected from ZooKeeper, ignoring > >>> 2010-09-03 16:09:52,857 INFO org.apache.zookeeper.ClientCnxn: Opening > >>> socket connection to server localhost/0:0:0:0:0:0:0:1:2181 > >>> 2010-09-03 16:09:52,858 WARN org.apache.zookeeper.ClientCnxn: Session > >>> 0x12ad9dccda30001 for server null, unexpected error, closing socket > >>> connection and attempting reconnect > >>> java.net.ConnectException: Connection refused > >>>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > >>>at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > >>>at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) > >>> 2010-09-03 16:09:53,149 INFO org.apache.zookeeper.ClientCnxn: Opening > >>> socket connection to server localhost/fe80:0:0:0:0:0:0:1%1:2181 > >>> 2010-09-03 16:09:53,150 WARN org.apache.zookeeper.ClientCnxn: Session > >>> 0x12ad9dccda30002 for server null, unexpected error, closing socket > >>> connection and attempting reconnect > >>> java.net.ConnectException: Connection refused > >>>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > >>>at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > >>>at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) > >>> 2010-09-03 16:09:53,576 INFO org.apache.zookeeper.ClientCnxn: Opening > >>> socket connection to server localhost/127.0.0.1:2181 > >>> 2010-09-03 16:09:53,576 WARN org.apache.zookeeper.ClientCnxn: Session > >>> 0x12ad9dccda30001 for server null, unexpected error, closing socket > >>> connection and attempting reconnect > >>> java.net.ConnectException: Connection refused > >>>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Metho
Re: Spew after call to close
No worries, let us know if something else pops up. Patrick On Tue, Sep 7, 2010 at 3:10 PM, Stack wrote: > Nevermind. I figured it. It was an hbase issue. We were leaking a > client reference. > Sorry for the noise, > St.Ack > > > On Sat, Sep 4, 2010 at 10:58 AM, Stack wrote: > > Thats right -- client is shutdown first, then server... How do I stop > > the client trying to come back from the dead? > > Good on you Mahadev? > > St.Ack > > > > On Fri, Sep 3, 2010 at 8:36 PM, Mahadev Konar > wrote: > >> > >> Hi Stack, > >> Looks like you are shutting down the server and shutting down the > client at > >> the same time? Is that the issue? > >> > >> Thanks > >> mahadev > >> > >> On 9/3/10 4:47 PM, "Stack" wrote: > >> > >>> Have you fellas seen this before? I call close on zookeeper but it > insists > >>> on doing the below exceptions. Why is it doing this 'Session > >>> 0x12ad9dccda30002 > >>> for server null, unexpected error, closing socket connection and > >>> attempting reconnect'? This would seem to come after the close has > >>> been noticed and looking in code, i'd think we'd not do this since the > >>> close flag should be set to true post call to close? > >>> > >>> Thanks lads (The below looks ugly in our logs... this is zk 3.3.1), > >>> St.Ack > >>> > >>> 2010-09-03 16:09:52,369 INFO > >>> org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection > >>> for client /fe80:0:0:0:0:0:0:1%1:56941 which had sessionid > >>> 0x12ad9dccda30001 > >>> 2010-09-03 16:09:52,369 INFO > >>> org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection > >>> for client /127.0.0.1:56942 which had sessionid 0x12ad9dccda30002 > >>> 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable > >>> to read additional data from server sessionid 0x12ad9dccda30001, > >>> likely server has closed socket, closing socket connection and > >>> attempting reconnect > >>> 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable > >>> to read additional data from server sessionid 0x12ad9dccda30002, > >>> likely server has closed socket, closing socket connection and > >>> attempting reconnect > >>> 2010-09-03 16:09:52,370 INFO > >>> org.apache.zookeeper.server.NIOServerCnxn: NIOServerCnxn factory > >>> exited run method > >>> 2010-09-03 16:09:52,370 INFO > >>> org.apache.zookeeper.server.PrepRequestProcessor: PrepRequestProcessor > >>> exited loop! > >>> 2010-09-03 16:09:52,370 INFO > >>> org.apache.zookeeper.server.SyncRequestProcessor: SyncRequestProcessor > >>> exited! > >>> 2010-09-03 16:09:52,370 INFO > >>> org.apache.zookeeper.server.FinalRequestProcessor: shutdown of request > >>> processor complete > >>> 2010-09-03 16:09:52,470 DEBUG > >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: > >>> Received ZooKeeper Event, type=None, state=Disconnected, path=null > >>> 2010-09-03 16:09:52,470 INFO > >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: > >>> Received Disconnected from ZooKeeper, ignoring > >>> 2010-09-03 16:09:52,471 DEBUG > >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: > >>> Received ZooKeeper Event, type=None, state=Disconnected, path=null > >>> 2010-09-03 16:09:52,471 INFO > >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: > >>> Received Disconnected from ZooKeeper, ignoring > >>> 2010-09-03 16:09:52,857 INFO org.apache.zookeeper.ClientCnxn: Opening > >>> socket connection to server localhost/0:0:0:0:0:0:0:1:2181 > >>> 2010-09-03 16:09:52,858 WARN org.apache.zookeeper.ClientCnxn: Session > >>> 0x12ad9dccda30001 for server null, unexpected error, closing socket > >>> connection and attempting reconnect > >>> java.net.ConnectException: Connection refused > >>>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > >>>at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > >>>at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) > >>> 2010-09-03 16:09:53,149 INFO org.apache.zookeeper.ClientCnxn: Opening > >>> socket connection to server localhost/fe80:0:0:0:0:0:0:1%1:2181 > >>> 2010-09-03 16:09:53,150 WARN org.apache.zookeeper.ClientCnxn: Session > >>> 0x12ad9dccda30002 for server null, unexpected error, closing socket > >>> connection and attempting reconnect > >>> java.net.ConnectException: Connection refused > >>>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > >>>at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) > >>>at > org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) > >>> 2010-09-03 16:09:53,576 INFO org.apache.zookeeper.ClientCnxn: Opening > >>> socket connection to server localhost/127.0.0.1:2181 > >>> 2010-09-03 16:09:53,576 WARN org.apache.zookeeper.ClientCnxn: Session > >>> 0x12ad9dccda30001 for server null, unexpected error, closing socket > >>> connection and attempting reconnect > >>> java.net.ConnectException: Connection refused > >>>at sun.nio.ch.SocketChannelImpl.checkConnect(Native Metho
Re: Spew after call to close
Nevermind. I figured it. It was an hbase issue. We were leaking a client reference. Sorry for the noise, St.Ack On Sat, Sep 4, 2010 at 10:58 AM, Stack wrote: > Thats right -- client is shutdown first, then server... How do I stop > the client trying to come back from the dead? > Good on you Mahadev? > St.Ack > > On Fri, Sep 3, 2010 at 8:36 PM, Mahadev Konar wrote: >> >> Hi Stack, >> Looks like you are shutting down the server and shutting down the client at >> the same time? Is that the issue? >> >> Thanks >> mahadev >> >> On 9/3/10 4:47 PM, "Stack" wrote: >> >>> Have you fellas seen this before? I call close on zookeeper but it insists >>> on doing the below exceptions. Why is it doing this 'Session >>> 0x12ad9dccda30002 >>> for server null, unexpected error, closing socket connection and >>> attempting reconnect'? This would seem to come after the close has >>> been noticed and looking in code, i'd think we'd not do this since the >>> close flag should be set to true post call to close? >>> >>> Thanks lads (The below looks ugly in our logs... this is zk 3.3.1), >>> St.Ack >>> >>> 2010-09-03 16:09:52,369 INFO >>> org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection >>> for client /fe80:0:0:0:0:0:0:1%1:56941 which had sessionid >>> 0x12ad9dccda30001 >>> 2010-09-03 16:09:52,369 INFO >>> org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection >>> for client /127.0.0.1:56942 which had sessionid 0x12ad9dccda30002 >>> 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable >>> to read additional data from server sessionid 0x12ad9dccda30001, >>> likely server has closed socket, closing socket connection and >>> attempting reconnect >>> 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable >>> to read additional data from server sessionid 0x12ad9dccda30002, >>> likely server has closed socket, closing socket connection and >>> attempting reconnect >>> 2010-09-03 16:09:52,370 INFO >>> org.apache.zookeeper.server.NIOServerCnxn: NIOServerCnxn factory >>> exited run method >>> 2010-09-03 16:09:52,370 INFO >>> org.apache.zookeeper.server.PrepRequestProcessor: PrepRequestProcessor >>> exited loop! >>> 2010-09-03 16:09:52,370 INFO >>> org.apache.zookeeper.server.SyncRequestProcessor: SyncRequestProcessor >>> exited! >>> 2010-09-03 16:09:52,370 INFO >>> org.apache.zookeeper.server.FinalRequestProcessor: shutdown of request >>> processor complete >>> 2010-09-03 16:09:52,470 DEBUG >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: >>> Received ZooKeeper Event, type=None, state=Disconnected, path=null >>> 2010-09-03 16:09:52,470 INFO >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: >>> Received Disconnected from ZooKeeper, ignoring >>> 2010-09-03 16:09:52,471 DEBUG >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: >>> Received ZooKeeper Event, type=None, state=Disconnected, path=null >>> 2010-09-03 16:09:52,471 INFO >>> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: >>> Received Disconnected from ZooKeeper, ignoring >>> 2010-09-03 16:09:52,857 INFO org.apache.zookeeper.ClientCnxn: Opening >>> socket connection to server localhost/0:0:0:0:0:0:0:1:2181 >>> 2010-09-03 16:09:52,858 WARN org.apache.zookeeper.ClientCnxn: Session >>> 0x12ad9dccda30001 for server null, unexpected error, closing socket >>> connection and attempting reconnect >>> java.net.ConnectException: Connection refused >>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) >>> 2010-09-03 16:09:53,149 INFO org.apache.zookeeper.ClientCnxn: Opening >>> socket connection to server localhost/fe80:0:0:0:0:0:0:1%1:2181 >>> 2010-09-03 16:09:53,150 WARN org.apache.zookeeper.ClientCnxn: Session >>> 0x12ad9dccda30002 for server null, unexpected error, closing socket >>> connection and attempting reconnect >>> java.net.ConnectException: Connection refused >>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) >>> 2010-09-03 16:09:53,576 INFO org.apache.zookeeper.ClientCnxn: Opening >>> socket connection to server localhost/127.0.0.1:2181 >>> 2010-09-03 16:09:53,576 WARN org.apache.zookeeper.ClientCnxn: Session >>> 0x12ad9dccda30001 for server null, unexpected error, closing socket >>> connection and attempting reconnect >>> java.net.ConnectException: Connection refused >>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >>> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >>> at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) >>> 2010-09-03 16:09:54,000 INFO >>> org.apache.zookeeper.server.SessionTrackerImpl: SessionTrackerImpl >>> exited loop! >>> 2010-09-03 16:09:54,002 DEBU
Re: Spew after call to close
Hi Stack, Looks like you are shutting down the server and shutting down the client at the same time? Is that the issue? Thanks mahadev On 9/3/10 4:47 PM, "Stack" wrote: > Have you fellas seen this before? I call close on zookeeper but it insists > on doing the below exceptions. Why is it doing this 'Session > 0x12ad9dccda30002 > for server null, unexpected error, closing socket connection and > attempting reconnect'? This would seem to come after the close has > been noticed and looking in code, i'd think we'd not do this since the > close flag should be set to true post call to close? > > Thanks lads (The below looks ugly in our logs... this is zk 3.3.1), > St.Ack > > 2010-09-03 16:09:52,369 INFO > org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection > for client /fe80:0:0:0:0:0:0:1%1:56941 which had sessionid > 0x12ad9dccda30001 > 2010-09-03 16:09:52,369 INFO > org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection > for client /127.0.0.1:56942 which had sessionid 0x12ad9dccda30002 > 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable > to read additional data from server sessionid 0x12ad9dccda30001, > likely server has closed socket, closing socket connection and > attempting reconnect > 2010-09-03 16:09:52,370 INFO org.apache.zookeeper.ClientCnxn: Unable > to read additional data from server sessionid 0x12ad9dccda30002, > likely server has closed socket, closing socket connection and > attempting reconnect > 2010-09-03 16:09:52,370 INFO > org.apache.zookeeper.server.NIOServerCnxn: NIOServerCnxn factory > exited run method > 2010-09-03 16:09:52,370 INFO > org.apache.zookeeper.server.PrepRequestProcessor: PrepRequestProcessor > exited loop! > 2010-09-03 16:09:52,370 INFO > org.apache.zookeeper.server.SyncRequestProcessor: SyncRequestProcessor > exited! > 2010-09-03 16:09:52,370 INFO > org.apache.zookeeper.server.FinalRequestProcessor: shutdown of request > processor complete > 2010-09-03 16:09:52,470 DEBUG > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: > Received ZooKeeper Event, type=None, state=Disconnected, path=null > 2010-09-03 16:09:52,470 INFO > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: > Received Disconnected from ZooKeeper, ignoring > 2010-09-03 16:09:52,471 DEBUG > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: > Received ZooKeeper Event, type=None, state=Disconnected, path=null > 2010-09-03 16:09:52,471 INFO > org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher: > Received Disconnected from ZooKeeper, ignoring > 2010-09-03 16:09:52,857 INFO org.apache.zookeeper.ClientCnxn: Opening > socket connection to server localhost/0:0:0:0:0:0:0:1:2181 > 2010-09-03 16:09:52,858 WARN org.apache.zookeeper.ClientCnxn: Session > 0x12ad9dccda30001 for server null, unexpected error, closing socket > connection and attempting reconnect > java.net.ConnectException: Connection refused >at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) > 2010-09-03 16:09:53,149 INFO org.apache.zookeeper.ClientCnxn: Opening > socket connection to server localhost/fe80:0:0:0:0:0:0:1%1:2181 > 2010-09-03 16:09:53,150 WARN org.apache.zookeeper.ClientCnxn: Session > 0x12ad9dccda30002 for server null, unexpected error, closing socket > connection and attempting reconnect > java.net.ConnectException: Connection refused >at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) > 2010-09-03 16:09:53,576 INFO org.apache.zookeeper.ClientCnxn: Opening > socket connection to server localhost/127.0.0.1:2181 > 2010-09-03 16:09:53,576 WARN org.apache.zookeeper.ClientCnxn: Session > 0x12ad9dccda30001 for server null, unexpected error, closing socket > connection and attempting reconnect > java.net.ConnectException: Connection refused >at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) >at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1078) > 2010-09-03 16:09:54,000 INFO > org.apache.zookeeper.server.SessionTrackerImpl: SessionTrackerImpl > exited loop! > 2010-09-03 16:09:54,002 DEBUG > org.apache.hadoop.hbase.client.HConnectionManager$TableServers: Closed > zookeeper sessionid=0x12ad9dccda30001 > 2010-09-03 16:09:54,129 INFO org.apache.zookeeper.ClientCnxn: Opening > socket connection to server localhost/0:0:0:0:0:0:0:1:2181 > 2010-09-03 16:09:54,130 WARN org.apache.zookeeper.ClientCnxn: Session > 0x12ad9dccda30002 for server null, unexpected error, closing socket > connection and attempting reconnect > java.net.ConnectException: Connection refused >at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >at sun.nio.ch.
