I think now I can explain the session expirations; hbase cilents especially up in a map/reduce task can exit without closing the zk session. Will fix. St.Ack
On Fri, Nov 20, 2009 at 4:45 PM, Patrick Hunt <ph...@apache.org> wrote: > Yes, right, that's what I meant to say - what is causing the client to > "die", throwing read error on the server side, and then later you end up > with the session expiration because the client was not closed gracefully. > > (thanks mahadev) > > Patrick > > > Mahadev Konar wrote: > >> That should be the case since the server gets an exception reading from >> the >> socket - meaning the client went away (not gracefully) and that leads the >> server to expire the session in 30 seconds. >> >> mahadev >> >> >> On 11/20/09 4:35 PM, "Patrick Hunt" <ph...@apache.org> wrote: >> >> Oops too late. ;-) >>> >>> I'm perplexed as to why you see all these expirations though. Are you >>> killing your clients, ie not cleaning up the ZK session gracefully via >>> close()? >>> >>> Patrick >>> >>> stack wrote: >>> >>>> Please disregard. Sorry for the noise (Patrick, of note, I am seeing >>>> this >>>> session timeout on a cluster other than Zhenyus). >>>> St.Ack >>>> >>>> On Fri, Nov 20, 2009 at 4:24 PM, stack <st...@duboce.net> wrote: >>>> >>>> Sorry, I had a bad subject on the below question. >>>>> St.Ack >>>>> >>>>> On Fri, Nov 20, 2009 at 4:22 PM, stack <st...@duboce.net> wrote: >>>>> >>>>> Below an excerpt from a single node zk quorum that was at heart of a >>>>>> small >>>>>> hbase cluster. Unfortunately the log is not at DEBUG level (I've >>>>>> asked the >>>>>> gentleman to up the log level meantime). What it seems to be >>>>>> reporting is >>>>>> that an exception while closing a session caused it to timeout all >>>>>> connected >>>>>> sessions. >>>>>> >>>>>> Here is the line that mentions the exception on close of session. >>>>>> There >>>>>> is no stack trace: >>>>>> >>>>>> 2009-11-20 03:41:04,766 WARN >>>>>> org.apache.zookeeper.server.NIOServerCnxn: >>>>>> Exception causing close of session 0x124bc250d700790 due to >>>>>> java.io.IOException: Read error >>>>>> >>>>>> Is it correct that an error at this stage throws out all connected >>>>>> sessions? >>>>>> >>>>>> Thanks, >>>>>> St.Ack >>>>>> >>>>>> >>>>>> 2009-11-20 00:00:04,948 INFO >>>>>> org.apache.zookeeper.server.NIOServerCnxn: >>>>>> Connected to /10.1.20.101:50716 lastZxid 0 >>>>>> 2009-11-20 00:00:04,982 INFO >>>>>> org.apache.zookeeper.server.NIOServerCnxn: >>>>>> Creating new session 0x1250f26319f0016 >>>>>> 2009-11-20 00:00:05,051 INFO >>>>>> org.apache.zookeeper.server.NIOServerCnxn: >>>>>> Finished init of 0x1250f26319f0016 valid:true >>>>>> 2009-11-20 00:00:05,051 WARN >>>>>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when >>>>>> processing sessionid:0x1250f26319f0016 type:create c >>>>>> xid:0x1 zxid:0xfffffffffffffffe txntype:unknown n/a >>>>>> org.apache.zookeeper.KeeperException$NodeExistsException: >>>>>> KeeperErrorCode >>>>>> = NodeExists >>>>>> at >>>>>> >>>>>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces >>>>>> sor.java:245) >>>>>> at >>>>>> >>>>>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j >>>>>> ava:114) >>>>>> 2009-11-20 00:00:40,150 WARN >>>>>> org.apache.zookeeper.server.PrepRequestProcessor: Got exception when >>>>>> processing sessionid:0x1250f26319f0016 type:create c >>>>>> xid:0x4 zxid:0xfffffffffffffffe txntype:unknown n/a >>>>>> org.apache.zookeeper.KeeperException$NodeExistsException: >>>>>> KeeperErrorCode >>>>>> = NodeExists >>>>>> at >>>>>> >>>>>> org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProces >>>>>> sor.java:245) >>>>>> at >>>>>> >>>>>> org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.j >>>>>> ava:114) >>>>>> 2009-11-20 00:00:50,428 WARN >>>>>> org.apache.zookeeper.server.NIOServerCnxn: >>>>>> Exception causing close of session 0x1250f26319f0016 due to >>>>>> java.io.IOExceptio >>>>>> n: Read error >>>>>> 2009-11-20 00:00:50,429 INFO >>>>>> org.apache.zookeeper.server.NIOServerCnxn: >>>>>> closing session:0x1250f26319f0016 NIOServerCnxn: >>>>>> java.nio.channels.SocketChann >>>>>> el[connected local=/10.1.20.101:2181 remote=/10.1.20.101:50716] >>>>>> 2009-11-20 00:01:22,002 INFO >>>>>> org.apache.zookeeper.server.SessionTrackerImpl: Expiring session >>>>>> 0x1250f26319f0016 >>>>>> 2009-11-20 00:01:22,002 INFO >>>>>> org.apache.zookeeper.server.ZooKeeperServer: >>>>>> Expiring session 0x1250f26319f0016 >>>>>> 2009-11-20 00:01:22,002 INFO >>>>>> org.apache.zookeeper.server.PrepRequestProcessor: Processed session >>>>>> termination request for id: 0x1250f26319f0016 >>>>>> 2009-11-20 03:41:04,766 WARN >>>>>> org.apache.zookeeper.server.NIOServerCnxn: >>>>>> Exception causing close of session 0x124bc250d700790 due to >>>>>> java.io.IOExceptio >>>>>> n: Read error >>>>>> 2009-11-20 03:41:04,864 INFO >>>>>> org.apache.zookeeper.server.SessionTrackerImpl: Expiring session >>>>>> 0x1250f26319f0000 >>>>>> 2009-11-20 03:41:04,927 INFO >>>>>> org.apache.zookeeper.server.ZooKeeperServer: >>>>>> Expiring session 0x1250f26319f0000 >>>>>> 2009-11-20 03:41:04,927 INFO >>>>>> org.apache.zookeeper.server.SessionTrackerImpl: Expiring session >>>>>> 0x124bc250d7007a2 >>>>>> 2009-11-20 03:41:04,927 INFO >>>>>> org.apache.zookeeper.server.ZooKeeperServer: >>>>>> Expiring session 0x124bc250d7007a2 >>>>>> 2009-11-20 03:41:04,927 INFO >>>>>> org.apache.zookeeper.server.SessionTrackerImpl: Expiring session >>>>>> 0x124bc250d700794 >>>>>> 2009-11-20 03:41:04,927 INFO >>>>>> org.apache.zookeeper.server.ZooKeeperServer: >>>>>> Expiring session 0x124bc250d700794 >>>>>> >>>>>> >>>>>> >>>>>> >>
