I searched around and found HBASE-1645 "HBase could be a little more dainty when zk is pulled out from under it"
I am wondering whether my problem is related to this HBase-1645. Or probably has nothing to do with it? thanks On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <[email protected]>wrote: > Hi, > > I have been monitoring the system resources such as memory Heap usage for > Regionserver, load average, network bandwidth, etc. > > I experienced a disconnection last night. When I checked the memory Heap > for Regionserver, it was 2336 which is way below the max Heap 4096. I don't > think it is Regionserver problem. The requests that the regionserver got was > about 126 which doesn't seem to be high. The load average is kind of high > >100. (but I saw disconnections while the load average is around 30). The > bandwidth is not a problem. > > However, I found error messages from the zookeeper. Please see the > following error messages. There are some > BufferUnderflowException error, IOException error,etc. > May I ask what might be the causes that generate these error messages? > > Many thanks > zhenyu > > > 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring > exception during toString > java.nio.BufferUnderflowException > at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127) > at java.nio.ByteBuffer.get(ByteBuffer.java:675) > at org.apache.zookeeper.server.Request.toString(Request.java:199) > at java.lang.String.valueOf(String.java:2827) > at java.lang.StringBuilder.append(StringBuilder.java:115) > at > org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167) > at > org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68) > 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.NIOServerCnxn: > Exception causing close of session 0x424dfd908c50009 > due to java.io.IOException: Read error > 2009-11-11 04:29:00,810 DEBUG org.apache.zookeeper.server.NIOServerCnxn: > IOException stack trace > java.io.IOException: Read error > at > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372) > at > org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221) > 2009-11-11 04:29:00,810 DEBUG > org.apache.zookeeper.server.quorum.CommitProcessor: Processing request:: > sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8 > zxid:0xfffffffffffffffe txntype:unknown n/a > 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring > exception during toString > java.nio.BufferUnderflowException > at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127) > at java.nio.ByteBuffer.get(ByteBuffer.java:675) > at org.apache.zookeeper.server.Request.toString(Request.java:199) > at java.lang.String.valueOf(String.java:2827) > at java.lang.StringBuilder.append(StringBuilder.java:115) > at > org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74) > at > org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73) > 2009-11-11 04:29:00,810 DEBUG > org.apache.zookeeper.server.FinalRequestProcessor: Processing request:: > sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8 > zxid:0xfffffffffffffffe txntype:unknown n/a > 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring > exception during toString > java.nio.BufferUnderflowException > at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127) > at java.nio.ByteBuffer.get(ByteBuffer.java:675) > at org.apache.zookeeper.server.Request.toString(Request.java:199) > at > org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36) > at > org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80) > at > org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362) > at > org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403) > at > org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65) > at org.apache.log4j.PatternLayout.format(PatternLayout.java:502) > at > org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302) > at > org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359) > at org.apache.log4j.WriterAppender.append(WriterAppender.java:160) > at > org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251) > at > org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66) > at org.apache.log4j.Category.callAppenders(Category.java:206) > at org.apache.log4j.Category.forcedLog(Category.java:391) > at org.apache.log4j.Category.debug(Category.java:260) > at > org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147) > at > org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73) > 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring > exception during toString > sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8 > zxid:0xfffffffffffffffe txntype:unknown n/a > 2009-11-11 04:29:00,810 INFO org.apache.zookeeper.server.NIOServerCnxn: > closing session:0x424dfd908c50009 NIOServerCnxn: > java.nio.channels.SocketChannel[connected local=/192.168.100.131:2181remote=/ > 192.168.100.131:35500] > 2009-11-11 04:29:00,810 DEBUG org.apache.zookeeper.server.NIOServerCnxn: > ignoring exception during input shutdown > java.net.SocketException: Transport endpoint is not connected > at sun.nio.ch.SocketChannelImpl.shutdown(Native Method) > at > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640) > at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) > at > org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848) > at > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498) > at > org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221) >
