I'm really not sure exactly what the situation is with the java.nio.BufferUnderflowException, but what do you mean by "regionserver disconnection"? Could you give more details?
Thx, J-D On Wed, Nov 11, 2009 at 10:34 AM, Zhenyu Zhong <[email protected]> wrote: > I searched around and found HBASE-1645 "HBase could be a little more dainty > when zk is pulled out from under it" > > I am wondering whether my problem is related to this HBase-1645. Or probably > has nothing to do with it? > > thanks > > > On Wed, Nov 11, 2009 at 12:02 PM, Zhenyu Zhong <[email protected]>wrote: > >> Hi, >> >> I have been monitoring the system resources such as memory Heap usage for >> Regionserver, load average, network bandwidth, etc. >> >> I experienced a disconnection last night. When I checked the memory Heap >> for Regionserver, it was 2336 which is way below the max Heap 4096. I don't >> think it is Regionserver problem. The requests that the regionserver got was >> about 126 which doesn't seem to be high. The load average is kind of high >> >100. (but I saw disconnections while the load average is around 30). The >> bandwidth is not a problem. >> >> However, I found error messages from the zookeeper. Please see the >> following error messages. There are some >> BufferUnderflowException error, IOException error,etc. >> May I ask what might be the causes that generate these error messages? >> >> Many thanks >> zhenyu >> >> >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring >> exception during toString >> java.nio.BufferUnderflowException >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127) >> at java.nio.ByteBuffer.get(ByteBuffer.java:675) >> at org.apache.zookeeper.server.Request.toString(Request.java:199) >> at java.lang.String.valueOf(String.java:2827) >> at java.lang.StringBuilder.append(StringBuilder.java:115) >> at >> org.apache.zookeeper.server.quorum.CommitProcessor.processRequest(CommitProcessor.java:167) >> at >> org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:68) >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.NIOServerCnxn: >> Exception causing close of session 0x424dfd908c50009 >> due to java.io.IOException: Read error >> 2009-11-11 04:29:00,810 DEBUG org.apache.zookeeper.server.NIOServerCnxn: >> IOException stack trace >> java.io.IOException: Read error >> at >> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:372) >> at >> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221) >> 2009-11-11 04:29:00,810 DEBUG >> org.apache.zookeeper.server.quorum.CommitProcessor: Processing request:: >> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8 >> zxid:0xfffffffffffffffe txntype:unknown n/a >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring >> exception during toString >> java.nio.BufferUnderflowException >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127) >> at java.nio.ByteBuffer.get(ByteBuffer.java:675) >> at org.apache.zookeeper.server.Request.toString(Request.java:199) >> at java.lang.String.valueOf(String.java:2827) >> at java.lang.StringBuilder.append(StringBuilder.java:115) >> at >> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:74) >> at >> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73) >> 2009-11-11 04:29:00,810 DEBUG >> org.apache.zookeeper.server.FinalRequestProcessor: Processing request:: >> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8 >> zxid:0xfffffffffffffffe txntype:unknown n/a >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring >> exception during toString >> java.nio.BufferUnderflowException >> at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:127) >> at java.nio.ByteBuffer.get(ByteBuffer.java:675) >> at org.apache.zookeeper.server.Request.toString(Request.java:199) >> at >> org.apache.log4j.or.DefaultRenderer.doRender(DefaultRenderer.java:36) >> at >> org.apache.log4j.or.RendererMap.findAndRender(RendererMap.java:80) >> at >> org.apache.log4j.spi.LoggingEvent.getRenderedMessage(LoggingEvent.java:362) >> at >> org.apache.log4j.helpers.PatternParser$BasicPatternConverter.convert(PatternParser.java:403) >> at >> org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65) >> at org.apache.log4j.PatternLayout.format(PatternLayout.java:502) >> at >> org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:302) >> at >> org.apache.log4j.DailyRollingFileAppender.subAppend(DailyRollingFileAppender.java:359) >> at org.apache.log4j.WriterAppender.append(WriterAppender.java:160) >> at >> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251) >> at >> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66) >> at org.apache.log4j.Category.callAppenders(Category.java:206) >> at org.apache.log4j.Category.forcedLog(Category.java:391) >> at org.apache.log4j.Category.debug(Category.java:260) >> at >> org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:147) >> at >> org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73) >> 2009-11-11 04:29:00,810 WARN org.apache.zookeeper.server.Request: Ignoring >> exception during toString >> sessionid:0x424dfd908c50009 type:setWatches cxid:0xfffffffffffffff8 >> zxid:0xfffffffffffffffe txntype:unknown n/a >> 2009-11-11 04:29:00,810 INFO org.apache.zookeeper.server.NIOServerCnxn: >> closing session:0x424dfd908c50009 NIOServerCnxn: >> java.nio.channels.SocketChannel[connected local=/192.168.100.131:2181remote=/ >> 192.168.100.131:35500] >> 2009-11-11 04:29:00,810 DEBUG org.apache.zookeeper.server.NIOServerCnxn: >> ignoring exception during input shutdown >> java.net.SocketException: Transport endpoint is not connected >> at sun.nio.ch.SocketChannelImpl.shutdown(Native Method) >> at >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640) >> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) >> at >> org.apache.zookeeper.server.NIOServerCnxn.close(NIOServerCnxn.java:848) >> at >> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:498) >> at >> org.apache.zookeeper.server.NIOServerCnxn$Factory.run(NIOServerCnxn.java:221) >> >
