Hi,
I'm executing HBase 0.20.4 over Hadoop 0.20.2 on Amazon EC2 and it had been
running smooth for more than a month. Today it crashed apparently because of
some Zookeeper exception. It's a one-machine-only configuration (that will
add more nodes in the future) and I'm conscious that it makes the the whole
structure more susceptible to errors.
Some facts:
(1) The HBase master log still reports that my server is up even after the
error: 2010-05-27 09:53:40,812 INFO
org.apache.hadoop.hbase.master.ServerManager: 1 region servers, 0 dead,
average load 75.0
(2) I have plenty of available memory.
(3) Time out configurations were relaxed.
(3) The region server log contains no errors.
Shouldn't Zookeeper recovery nicely? How can I prevent such error from
happening in the future?
Thanks in advance,
Lucas
MASTER LOG
2010-05-27 08:40:40,676 INFO org.apache.hadoop.hbase.master.ServerManager: 1
region servers, 0 dead, average load 75.0
2010-05-27 08:40:44,252 ERROR org.apache.hadoop.hbase.master.BaseScanner:
Closing scanner
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:308)
at
org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:843)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:715)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333)
at $Proxy1.close(Unknown Source)
at
org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:215)
at
org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
at
org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
at
org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:153)
at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
2010-05-27 08:40:44,259 WARN org.apache.hadoop.hbase.master.BaseScanner:
Scan ROOT region
java.io.IOException: Call to /10.251.158.224:60020 failed on local
exception: java.io.IOException: Connection reset by peer
at
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:769)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:739)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333)
at $Proxy1.get(Unknown Source)
at
org.apache.hadoop.hbase.master.BaseScanner.checkAssigned(BaseScanner.java:543)
at
org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:192)
at
org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
at
org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
at
org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:153)
at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
Caused by: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(Unknown Source)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
at sun.nio.ch.IOUtil.read(Unknown Source)
at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
at
org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.FilterInputStream.read(Unknown Source)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:279)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at java.io.DataInputStream.readInt(Unknown Source)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:506)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:450)
2010-05-27 08:40:44,253 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x128c8b303040000 to sun.nio.ch.selectionkeyi...@470b9279
java.io.IOException: TIMED OUT
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:906)
2010-05-27 08:40:44,253 ERROR org.apache.hadoop.hbase.master.BaseScanner:
Closing scanner
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:308)
at
org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:843)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:715)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333)
at $Proxy1.close(Unknown Source)
at
org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:215)
at
org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
at
org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
at
org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:153)
at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
2010-05-27 08:40:44,270 WARN org.apache.hadoop.hbase.master.BaseScanner:
Scan one META region: {server: 10.251.158.224:60020, regionname: .META.,,1,
startKey: <>}
java.io.IOException: Call to /10.251.158.224:60020 failed on local
exception: java.io.IOException: Connection reset by peer
at
org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:769)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:739)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333)
at $Proxy1.next(Unknown Source)
at
org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:179)
at
org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
at
org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
at
org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:153)
at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
Caused by: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(Unknown Source)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(Unknown Source)
at sun.nio.ch.IOUtil.read(Unknown Source)
at sun.nio.ch.SocketChannelImpl.read(Unknown Source)
at
org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.FilterInputStream.read(Unknown Source)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection$PingInputStream.read(HBaseClient.java:279)
at java.io.BufferedInputStream.fill(Unknown Source)
at java.io.BufferedInputStream.read(Unknown Source)
at java.io.DataInputStream.readInt(Unknown Source)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:506)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:450)
2010-05-27 08:40:44,269 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {server: 10.251.158.224:60020,
regionname: -ROOT-,,0, startKey: <>}
2010-05-27 08:40:44,272 WARN org.apache.hadoop.hbase.master.BaseScanner:
Scan ROOT region
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:308)
at
org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:843)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:715)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333)
at $Proxy1.openScanner(Unknown Source)
at
org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:177)
at
org.apache.hadoop.hbase.master.RootScanner.scanRoot(RootScanner.java:54)
at
org.apache.hadoop.hbase.master.RootScanner.maintenanceScan(RootScanner.java:79)
at
org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:153)
at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
2010-05-27 08:40:45,238 INFO org.apache.hadoop.hbase.master.BaseScanner: All
1 .META. region(s) scanned
2010-05-27 08:40:45,238 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.metaScanner scanning meta region {server: 10.251.158.224:60020,
regionname: .META.,,1, startKey: <>}
2010-05-27 08:40:45,238 WARN org.apache.hadoop.hbase.master.BaseScanner:
Scan one META region: {server: 10.251.158.224:60020, regionname: .META.,,1,
startKey: <>}
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at
org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:308)
at
org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:843)
at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:715)
at
org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:333)
at $Proxy1.openScanner(Unknown Source)
at
org.apache.hadoop.hbase.master.BaseScanner.scanRegion(BaseScanner.java:177)
at
org.apache.hadoop.hbase.master.MetaScanner.scanOneMetaRegion(MetaScanner.java:73)
at
org.apache.hadoop.hbase.master.MetaScanner.maintenanceScan(MetaScanner.java:129)
at
org.apache.hadoop.hbase.master.BaseScanner.chore(BaseScanner.java:153)
at org.apache.hadoop.hbase.Chore.run(Chore.java:68)
2010-05-27 08:40:45,241 INFO org.apache.hadoop.hbase.master.BaseScanner: All
1 .META. region(s) scanned
2010-05-27 08:40:45,550 DEBUG
org.apache.hadoop.hbase.master.ZKMasterAddressWatcher: Got event None with
path null
2010-05-27 08:40:45,711 DEBUG org.apache.hadoop.hbase.master.HMaster: Got
event None with path null
2010-05-27 08:40:46,628 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ip-10-251-158-224.ec2.internal/10.251.158.224:2181
2010-05-27 08:40:46,629 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
10.251.158.224:39413 remote=ip-10-251-158-224.ec2.internal/
10.251.158.224:2181]
2010-05-27 08:40:46,629 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2010-05-27 08:40:46,630 WARN org.apache.zookeeper.ClientCnxn: Exception
closing session 0x128c8b303040000 to sun.nio.ch.selectionkeyi...@744e022c
java.io.IOException: Session Expired
at
org.apache.zookeeper.ClientCnxn$SendThread.readConnectResult(ClientCnxn.java:589)
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:709)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:945)
2010-05-27 08:40:46,631 DEBUG
org.apache.hadoop.hbase.master.ZKMasterAddressWatcher: Got event None with
path null
2010-05-27 08:40:46,631 DEBUG org.apache.hadoop.hbase.master.HMaster: Got
event None with path null
2010-05-27 08:40:46,631 INFO org.apache.hadoop.hbase.master.HMaster: Master
lost its znode, trying to get a new one
2010-05-27 08:40:46,631 INFO org.apache.zookeeper.ZooKeeper: Closing
session: 0x128c8b303040000
2010-05-27 08:40:46,631 INFO org.apache.zookeeper.ClientCnxn: Closing
ClientCnxn for session: 0x128c8b303040000
2010-05-27 08:40:46,651 INFO org.apache.zookeeper.ClientCnxn: Disconnecting
ClientCnxn for session: 0x128c8b303040000
2010-05-27 08:40:46,651 INFO org.apache.zookeeper.ZooKeeper: Session:
0x128c8b303040000 closed
2010-05-27 08:40:46,651 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Closed connection with
ZooKeeper
2010-05-27 08:40:46,684 INFO org.apache.zookeeper.ZooKeeper: Initiating
client connection, connectString=ip-10-251-158-224.ec2.internal:2181
sessionTimeout=600000 watcher=Thread[HMaster,5,main]
2010-05-27 08:40:46,686 INFO org.apache.zookeeper.ClientCnxn: Attempting
connection to server ip-10-251-158-224.ec2.internal/10.251.158.224:2181
2010-05-27 08:40:47,097 INFO org.apache.zookeeper.ClientCnxn: Priming
connection to java.nio.channels.SocketChannel[connected local=/
10.251.158.224:39414 remote=ip-10-251-158-224.ec2.internal/
10.251.158.224:2181]
2010-05-27 08:40:47,097 INFO org.apache.zookeeper.ClientCnxn: Server
connection successful
2010-05-27 08:40:47,115 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to read:
org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /hbase/master
2010-05-27 08:40:47,117 DEBUG org.apache.hadoop.hbase.master.HMaster: Got
event None with path null
2010-05-27 08:40:47,121 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Wrote master address
10.251.158.224:60000 to ZooKeeper
2010-05-27 08:40:47,127 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: State node exists.
2010-05-27 08:40:47,129 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/master
got 10.251.158.224:60000
2010-05-27 08:40:47,129 DEBUG org.apache.hadoop.hbase.master.HMaster:
Checking cluster state...
2010-05-27 08:40:47,130 DEBUG
org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode
/hbase/root-region-server got 10.251.158.224:60020
2010-05-27 08:40:47,131 DEBUG org.apache.hadoop.hbase.master.HMaster: This
is a fresh start, proceeding with normal startup
2010-05-27 08:40:47,134 INFO org.apache.hadoop.hbase.master.HMaster: Found
log folder : ip-10-251-158-224.ec2.internal,60020,1274677721442
2010-05-27 08:40:47,134 INFO org.apache.hadoop.hbase.master.HMaster: Log
folder belongs to an existing region server
2010-05-27 08:40:47,134 INFO org.apache.zookeeper.ClientCnxn: EventThread
shut down
2010-05-27 08:41:40,676 INFO org.apache.hadoop.hbase.master.ServerManager: 1
region servers, 0 dead, average load 75.0
2010-05-27 08:41:44,270 INFO org.apache.hadoop.hbase.master.BaseScanner:
RegionManager.rootScanner scanning meta region {server: 10.251.158.224:60020,
regionname: -ROOT-,,0, startKey: <>}
2010-05-27 08:41:44,271 WARN org.apache.hadoop.hbase.master.BaseScanner:
Scan ROOT region
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
ZOOKEEPER LOG
2010-05-27 08:40:44,230 INFO org.apache.zookeeper.server.ZooKeeperServer:
Expiring session 0x128c8b303040002
2010-05-27 08:40:44,232 INFO org.apache.zookeeper.server.NIOServerCnxn:
closing session:0x128c8b303040001 NIOServerCnxn:
java.nio.channels.SocketChannel[connected local=/10.251.158.224:2181remote=/
10.251.158.224:52406]
2010-05-27 08:40:44,233 INFO
org.apache.zookeeper.server.PrepRequestProcessor: Processed session
termination request for id: 0x128c8b303040582
2010-05-27 08:40:44,236 INFO
org.apache.zookeeper.server.PrepRequestProcessor: Processed session
termination request for id: 0x128c8b303040000
2010-05-27 08:40:44,236 INFO
org.apache.zookeeper.server.PrepRequestProcessor: Processed session
termination request for id: 0x128c8b303040002
2010-05-27 08:40:44,237 WARN org.apache.zookeeper.server.ZooKeeperServer:
Ignoring unexpected exception
java.io.IOException: Missing session 0x128c8b303040002
at
org.apache.zookeeper.server.ZooKeeperServer.touch(ZooKeeperServer.java:337)
at
org.apache.zookeeper.server.ZooKeeperServer.submitRequest(ZooKeeperServer.java:570)
at
org.apache.zookeeper.server.NIOServerCnxn.readRequest(NIOServerCnxn.java:569)
at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:411)
at
org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:239)
2010-05-27 08:40:44,243 WARN org.apache.zookeeper.server.NIOServerCnxn:
Exception causing close of session 0x128c8b303040002 due to
java.io.IOException: Read error
2010-05-27 08:40:44,244 INFO org.apache.zookeeper.server.NIOServerCnxn:
closing session:0x128c8b303040002 NIOServerCnxn:
java.nio.channels.SocketChannel[connected local=/10.251.158.224:2181remote=/
10.251.158.224:39520]
2010-05-27 08:40:44,246 WARN org.apache.zookeeper.server.NIOServerCnxn:
Exception causing close of session 0x128c8b303040582 due to
java.io.IOException: Read error
2010-05-27 08:40:44,246 INFO org.apache.zookeeper.server.NIOServerCnxn:
closing session:0x128c8b303040582 NIOServerCnxn:
java.nio.channels.SocketChannel[connected local=/10.251.158.224:2181remote=/
10.251.158.224:51069]
2010-05-27 08:40:44,247 INFO org.apache.zookeeper.server.NIOServerCnxn:
Connected to /10.251.158.224:39398 lastZxid 233730
2010-05-27 08:40:44,249 INFO org.apache.zookeeper.server.NIOServerCnxn:
Finished init of 0x128c8b303040002 valid:false
2010-05-27 08:40:44,249 INFO org.apache.zookeeper.server.NIOServerCnxn:
Renewing session 0x128c8b303040002
2010-05-27 08:40:44,249 WARN org.apache.zookeeper.server.ZooKeeperServer:
Ignoring unexpected exception
java.io.IOException: Missing session 0x128c8b303040002
at
org.apache.zookeeper.server.ZooKeeperServer.touch(ZooKeeperServer.java:337)
at
org.apache.zookeeper.server.ZooKeeperServer.submitRequest(ZooKeeperServer.java:570)
at
org.apache.zookeeper.server.NIOServerCnxn.readRequest(NIOServerCnxn.java:569)
at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:411)
at
org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:239)
2010-05-27 08:40:44,253 WARN org.apache.zookeeper.server.NIOServerCnxn:
Exception causing close of session 0x128c8b303040002 due to
java.io.IOException: Read error
2010-05-27 08:40:44,254 INFO org.apache.zookeeper.server.NIOServerCnxn:
closing session:0x128c8b303040002 NIOServerCnxn:
java.nio.channels.SocketChannel[connected local=/10.251.158.224:2181remote=/
10.251.158.224:39398]
2010-05-27 08:40:45,239 WARN org.apache.zookeeper.server.NIOServerCnxn:
Exception causing close of session 0x128c8b303040000 due to
java.io.IOException: Read error
2010-05-27 08:40:45,239 INFO org.apache.zookeeper.server.NIOServerCnxn:
closing session:0x128c8b303040000 NIOServerCnxn:
java.nio.channels.SocketChannel[connected local=/10.251.158.224:2181remote=/
10.251.158.224:52405]
2010-05-27 08:40:46,629 INFO org.apache.zookeeper.server.NIOServerCnxn:
Connected to /10.251.158.224:39413 lastZxid 233729
2010-05-27 08:40:46,630 INFO org.apache.zookeeper.server.NIOServerCnxn:
Finished init of 0x128c8b303040000 valid:false
2010-05-27 08:40:46,630 INFO org.apache.zookeeper.server.NIOServerCnxn:
Renewing session 0x128c8b303040000
2010-05-27 08:40:46,631 WARN org.apache.zookeeper.server.ZooKeeperServer:
Ignoring unexpected exception
java.io.IOException: Missing session 0x128c8b303040000
at
org.apache.zookeeper.server.ZooKeeperServer.touch(ZooKeeperServer.java:337)
at
org.apache.zookeeper.server.ZooKeeperServer.submitRequest(ZooKeeperServer.java:570)
at
org.apache.zookeeper.server.NIOServerCnxn.readRequest(NIOServerCnxn.java:569)
at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:411)
at
org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:239)
2010-05-27 08:40:46,631 WARN org.apache.zookeeper.server.NIOServerCnxn:
Exception causing close of session 0x128c8b303040000 due to
java.io.IOException: Read error
2010-05-27 08:40:46,631 INFO org.apache.zookeeper.server.NIOServerCnxn:
closing session:0x128c8b303040000 NIOServerCnxn:
java.nio.channels.SocketChannel[connected local=/10.251.158.224:2181remote=/
10.251.158.224:39413]
2010-05-27 08:40:47,098 INFO org.apache.zookeeper.server.NIOServerCnxn:
Connected to /10.251.158.224:39414 lastZxid 0
2010-05-27 08:40:47,098 INFO org.apache.zookeeper.server.NIOServerCnxn:
Creating new session 0x128c8b303040f9a
2010-05-27 08:40:47,116 INFO org.apache.zookeeper.server.NIOServerCnxn:
Finished init of 0x128c8b303040f9a valid:true
2010-05-27 08:40:47,124 WARN
org.apache.zookeeper.server.PrepRequestProcessor: Got exception when
processing sessionid:0x128c8b303040f9a type:create cxid:0x4
zxid:0xfffffffffffffffe txntype:unknown n/a
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
NodeExists
at
org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:245)
at
org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:114)
2010-05-27 08:40:56,782 INFO org.apache.zookeeper.server.NIOServerCnxn:
Connected to /10.251.158.224:39444 lastZxid 0
2010-05-27 08:40:56,782 INFO org.apache.zookeeper.server.NIOServerCnxn:
Creating new session 0x128c8b303040f9b
2010-05-27 08:40:56,858 INFO org.apache.zookeeper.server.NIOServerCnxn:
Finished init of 0x128c8b303040f9b valid:true
2010-05-27 08:41:12,483 INFO org.apache.zookeeper.server.NIOServerCnxn:
Connected to /10.251.158.224:39463 lastZxid 0
2010-05-27 08:41:12,484 INFO org.apache.zookeeper.server.NIOServerCnxn:
Creating new session 0x128c8b303040f9c
2010-05-27 08:41:12,513 INFO org.apache.zookeeper.server.NIOServerCnxn:
Finished init of 0x128c8b303040f9c valid:true
2010-05-27 08:41:15,396 INFO org.apache.zookeeper.server.NIOServerCnxn:
Connected to /10.251.158.224:39468 lastZxid 0
2010-05-27 08:41:15,396 INFO org.apache.zookeeper.server.NIOServerCnxn:
Creating new session 0x128c8b303040f9d
2010-05-27 08:41:15,463 INFO org.apache.zookeeper.server.NIOServerCnxn:
Finished init of 0x128c8b303040f9d valid:true
