Please disregard - Re: Exception on close of connection (WAS - Re: c client on win32)
Please disregard. Sorry for the noise (Patrick, of note, I am seeing this session timeout on a cluster other than Zhenyus). St.Ack On Fri, Nov 20, 2009 at 4:24 PM, stack st...@duboce.net wrote: Sorry, I had a bad subject on the below question. St.Ack On Fri, Nov 20, 2009 at 4:22 PM, stack st...@duboce.net wrote: Below an excerpt from a single node zk quorum that was at heart of a small hbase cluster. Unfortunately the log is not at DEBUG level (I've asked the gentleman to up the log level meantime). What it seems to be reporting is that an exception while closing a session caused it to timeout all connected sessions. Here is the line that mentions the exception on close of session. There is no stack trace: 2009-11-20 03:41:04,766 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x124bc250d700790 due to java.io.IOException: Read error Is it correct that an error at this stage throws out all connected sessions? Thanks, St.Ack 2009-11-20 00:00:04,948 INFO org.apache.zookeeper.server.NIOServerCnxn: Connected to /10.1.20.101:50716 lastZxid 0 2009-11-20 00:00:04,982 INFO org.apache.zookeeper.server.NIOServerCnxn: Creating new session 0x1250f26319f0016 2009-11-20 00:00:05,051 INFO org.apache.zookeeper.server.NIOServerCnxn: Finished init of 0x1250f26319f0016 valid:true 2009-11-20 00:00:05,051 WARN org.apache.zookeeper.server.PrepRequestProcessor: Got exception when processing sessionid:0x1250f26319f0016 type:create c xid:0x1 zxid:0xfffe txntype:unknown n/a org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists at org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:245) at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:114) 2009-11-20 00:00:40,150 WARN org.apache.zookeeper.server.PrepRequestProcessor: Got exception when processing sessionid:0x1250f26319f0016 type:create c xid:0x4 zxid:0xfffe txntype:unknown n/a org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists at org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:245) at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:114) 2009-11-20 00:00:50,428 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x1250f26319f0016 due to java.io.IOExceptio n: Read error 2009-11-20 00:00:50,429 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x1250f26319f0016 NIOServerCnxn: java.nio.channels.SocketChann el[connected local=/10.1.20.101:2181 remote=/10.1.20.101:50716] 2009-11-20 00:01:22,002 INFO org.apache.zookeeper.server.SessionTrackerImpl: Expiring session 0x1250f26319f0016 2009-11-20 00:01:22,002 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x1250f26319f0016 2009-11-20 00:01:22,002 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination request for id: 0x1250f26319f0016 2009-11-20 03:41:04,766 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x124bc250d700790 due to java.io.IOExceptio n: Read error 2009-11-20 03:41:04,864 INFO org.apache.zookeeper.server.SessionTrackerImpl: Expiring session 0x1250f26319f 2009-11-20 03:41:04,927 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x1250f26319f 2009-11-20 03:41:04,927 INFO org.apache.zookeeper.server.SessionTrackerImpl: Expiring session 0x124bc250d7007a2 2009-11-20 03:41:04,927 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x124bc250d7007a2 2009-11-20 03:41:04,927 INFO org.apache.zookeeper.server.SessionTrackerImpl: Expiring session 0x124bc250d700794 2009-11-20 03:41:04,927 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x124bc250d700794
Re: Please disregard - Re: Exception on close of connection (WAS - Re: c client on win32)
Oops too late. ;-) I'm perplexed as to why you see all these expirations though. Are you killing your clients, ie not cleaning up the ZK session gracefully via close()? Patrick stack wrote: Please disregard. Sorry for the noise (Patrick, of note, I am seeing this session timeout on a cluster other than Zhenyus). St.Ack On Fri, Nov 20, 2009 at 4:24 PM, stack st...@duboce.net wrote: Sorry, I had a bad subject on the below question. St.Ack On Fri, Nov 20, 2009 at 4:22 PM, stack st...@duboce.net wrote: Below an excerpt from a single node zk quorum that was at heart of a small hbase cluster. Unfortunately the log is not at DEBUG level (I've asked the gentleman to up the log level meantime). What it seems to be reporting is that an exception while closing a session caused it to timeout all connected sessions. Here is the line that mentions the exception on close of session. There is no stack trace: 2009-11-20 03:41:04,766 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x124bc250d700790 due to java.io.IOException: Read error Is it correct that an error at this stage throws out all connected sessions? Thanks, St.Ack 2009-11-20 00:00:04,948 INFO org.apache.zookeeper.server.NIOServerCnxn: Connected to /10.1.20.101:50716 lastZxid 0 2009-11-20 00:00:04,982 INFO org.apache.zookeeper.server.NIOServerCnxn: Creating new session 0x1250f26319f0016 2009-11-20 00:00:05,051 INFO org.apache.zookeeper.server.NIOServerCnxn: Finished init of 0x1250f26319f0016 valid:true 2009-11-20 00:00:05,051 WARN org.apache.zookeeper.server.PrepRequestProcessor: Got exception when processing sessionid:0x1250f26319f0016 type:create c xid:0x1 zxid:0xfffe txntype:unknown n/a org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists at org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:245) at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:114) 2009-11-20 00:00:40,150 WARN org.apache.zookeeper.server.PrepRequestProcessor: Got exception when processing sessionid:0x1250f26319f0016 type:create c xid:0x4 zxid:0xfffe txntype:unknown n/a org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists at org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:245) at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:114) 2009-11-20 00:00:50,428 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x1250f26319f0016 due to java.io.IOExceptio n: Read error 2009-11-20 00:00:50,429 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x1250f26319f0016 NIOServerCnxn: java.nio.channels.SocketChann el[connected local=/10.1.20.101:2181 remote=/10.1.20.101:50716] 2009-11-20 00:01:22,002 INFO org.apache.zookeeper.server.SessionTrackerImpl: Expiring session 0x1250f26319f0016 2009-11-20 00:01:22,002 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x1250f26319f0016 2009-11-20 00:01:22,002 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination request for id: 0x1250f26319f0016 2009-11-20 03:41:04,766 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x124bc250d700790 due to java.io.IOExceptio n: Read error 2009-11-20 03:41:04,864 INFO org.apache.zookeeper.server.SessionTrackerImpl: Expiring session 0x1250f26319f 2009-11-20 03:41:04,927 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x1250f26319f 2009-11-20 03:41:04,927 INFO org.apache.zookeeper.server.SessionTrackerImpl: Expiring session 0x124bc250d7007a2 2009-11-20 03:41:04,927 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x124bc250d7007a2 2009-11-20 03:41:04,927 INFO org.apache.zookeeper.server.SessionTrackerImpl: Expiring session 0x124bc250d700794 2009-11-20 03:41:04,927 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x124bc250d700794
Re: Please disregard - Re: Exception on close of connection (WAS - Re: c client on win32)
I think now I can explain the session expirations; hbase cilents especially up in a map/reduce task can exit without closing the zk session. Will fix. St.Ack On Fri, Nov 20, 2009 at 4:45 PM, Patrick Hunt ph...@apache.org wrote: Yes, right, that's what I meant to say - what is causing the client to die, throwing read error on the server side, and then later you end up with the session expiration because the client was not closed gracefully. (thanks mahadev) Patrick Mahadev Konar wrote: That should be the case since the server gets an exception reading from the socket - meaning the client went away (not gracefully) and that leads the server to expire the session in 30 seconds. mahadev On 11/20/09 4:35 PM, Patrick Hunt ph...@apache.org wrote: Oops too late. ;-) I'm perplexed as to why you see all these expirations though. Are you killing your clients, ie not cleaning up the ZK session gracefully via close()? Patrick stack wrote: Please disregard. Sorry for the noise (Patrick, of note, I am seeing this session timeout on a cluster other than Zhenyus). St.Ack On Fri, Nov 20, 2009 at 4:24 PM, stack st...@duboce.net wrote: Sorry, I had a bad subject on the below question. St.Ack On Fri, Nov 20, 2009 at 4:22 PM, stack st...@duboce.net wrote: Below an excerpt from a single node zk quorum that was at heart of a small hbase cluster. Unfortunately the log is not at DEBUG level (I've asked the gentleman to up the log level meantime). What it seems to be reporting is that an exception while closing a session caused it to timeout all connected sessions. Here is the line that mentions the exception on close of session. There is no stack trace: 2009-11-20 03:41:04,766 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x124bc250d700790 due to java.io.IOException: Read error Is it correct that an error at this stage throws out all connected sessions? Thanks, St.Ack 2009-11-20 00:00:04,948 INFO org.apache.zookeeper.server.NIOServerCnxn: Connected to /10.1.20.101:50716 lastZxid 0 2009-11-20 00:00:04,982 INFO org.apache.zookeeper.server.NIOServerCnxn: Creating new session 0x1250f26319f0016 2009-11-20 00:00:05,051 INFO org.apache.zookeeper.server.NIOServerCnxn: Finished init of 0x1250f26319f0016 valid:true 2009-11-20 00:00:05,051 WARN org.apache.zookeeper.server.PrepRequestProcessor: Got exception when processing sessionid:0x1250f26319f0016 type:create c xid:0x1 zxid:0xfffe txntype:unknown n/a org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists at org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces sor.java:245) at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j ava:114) 2009-11-20 00:00:40,150 WARN org.apache.zookeeper.server.PrepRequestProcessor: Got exception when processing sessionid:0x1250f26319f0016 type:create c xid:0x4 zxid:0xfffe txntype:unknown n/a org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists at org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces sor.java:245) at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j ava:114) 2009-11-20 00:00:50,428 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x1250f26319f0016 due to java.io.IOExceptio n: Read error 2009-11-20 00:00:50,429 INFO org.apache.zookeeper.server.NIOServerCnxn: closing session:0x1250f26319f0016 NIOServerCnxn: java.nio.channels.SocketChann el[connected local=/10.1.20.101:2181 remote=/10.1.20.101:50716] 2009-11-20 00:01:22,002 INFO org.apache.zookeeper.server.SessionTrackerImpl: Expiring session 0x1250f26319f0016 2009-11-20 00:01:22,002 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x1250f26319f0016 2009-11-20 00:01:22,002 INFO org.apache.zookeeper.server.PrepRequestProcessor: Processed session termination request for id: 0x1250f26319f0016 2009-11-20 03:41:04,766 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x124bc250d700790 due to java.io.IOExceptio n: Read error 2009-11-20 03:41:04,864 INFO org.apache.zookeeper.server.SessionTrackerImpl: Expiring session 0x1250f26319f 2009-11-20 03:41:04,927 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x1250f26319f 2009-11-20 03:41:04,927 INFO org.apache.zookeeper.server.SessionTrackerImpl: Expiring session 0x124bc250d7007a2 2009-11-20 03:41:04,927 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x124bc250d7007a2 2009-11-20 03:41:04,927 INFO org.apache.zookeeper.server.SessionTrackerImpl: Expiring session 0x124bc250d700794 2009-11-20 03:41:04,927 INFO org.apache.zookeeper.server.ZooKeeperServer: Expiring session 0x124bc250d700794
