not sure if you could also try using different connections for your client program. see the related issue HBASE-11306. I guess sharing connection is a contributing factor for this issue.
On Mon, Jul 14, 2014 at 7:19 PM, Rural Hunter <[email protected]> wrote: > Hi Tian Qiang, > > Good to hear that. It seems the jira is fixed. I will seek time to build > the latest 0.96 branch and test it. > > 于 2014/7/14 18:06, Qiang Tian 写道: > >> Hi YuMing, :) >> yes. several iterations of jstack on the problem regionserver could help >> identify the problem >> >> Rural, >> you probably hit hbase11277(and probably YuMin as well) - the reader 14 >> loops again and again in >> below stack(high cpu usage) and listener 12 is blocked and cannot >> accept new connections. >> >> >> >> 1. Thread 12 (RpcServer.listener,port=60020): >> 2. State: BLOCKED >> 3. Blocked count: 123264191 >> 4. Waited count: 0 >> 5. Blocked on >> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader@77f87716 >> 6. Blocked by 14 (RpcServer.reader=1,port=60020) >> 7. Stack: >> 8. >> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader. >> registerChannel(RpcServer.java:598) >> 9. >> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doAccept( >> RpcServer.java:755) >> 10. >> org.apache.hadoop.hbase.ipc.RpcServer$Listener.run( >> RpcServer.java:673) >> 11. Thread 24 (RpcServer.responder): >> >> >> >> 1. Thread 14 (RpcServer.reader=1,port=60020): >> 2. State: RUNNABLE >> 3. Blocked count: 12510492 >> 4. Waited count: 12826560 >> 5. Stack: >> 6. sun.nio.ch.FileDispatcher.read0(Native Method) >> 7. sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) >> 8. sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:251) >> 9. sun.nio.ch.IOUtil.read(IOUtil.java:224) >> 10. sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:254) >> 11. >> org.apache.hadoop.hbase.ipc.RpcServer.channelIO(RpcServer.java:2438) >> 12. >> org.apache.hadoop.hbase.ipc.RpcServer.channelRead( >> RpcServer.java:2404) >> 13. >> org.apache.hadoop.hbase.ipc.RpcServer$Connection. >> readAndProcess(RpcServer.java:1498) >> 14. >> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead( >> RpcServer.java:780) >> 15. >> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader. >> doRunLoop(RpcServer.java:568) >> 16. >> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run( >> RpcServer.java:543) >> 17. >> java.util.concurrent.ThreadPoolExecutor.runWorker( >> ThreadPoolExecutor.java:1146) >> 18. >> java.util.concurrent.ThreadPoolExecutor$Worker.run( >> ThreadPoolExecutor.java:615) >> 19. java.lang.Thread.run(Thread.java:701) >> 20. Thread 13 (RpcServer.reader=0,port=60020): >> 21. >> >> >> >> 1. 2014-07-10 14:13:49,614 WARN [RpcServer.reader=7,port=60020] >> >> ipc.RpcServer: RpcServer.listener,port=60020: count of bytes read: 0 >> 2. java.io.IOException: Connection reset by peer >> 3. at sun.nio.ch.FileDispatcher.read0(Native Method) >> 4. at sun.nio.ch.SocketDispatcher. >> read(SocketDispatcher.java:39) >> 5. at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:251) >> 6. at sun.nio.ch.IOUtil.read(IOUtil.java:224) >> 7. at >> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:254) >> 8. at >> org.apache.hadoop.hbase.ipc.RpcServer.channelRead( >> RpcServer.java:2404) >> 9. at >> org.apache.hadoop.hbase.ipc.RpcServer$Connection. >> readAndProcess(RpcServer.java:1425) >> 10. at >> org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead( >> RpcServer.java:780) >> 11. at >> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader. >> doRunLoop(RpcServer.java:568) >> 12. at >> org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run( >> RpcServer.java:543) >> 13. at >> java.util.concurrent.ThreadPoolExecutor.runWorker( >> ThreadPoolExecutor.java:1146) >> 14. at >> java.util.concurrent.ThreadPoolExecutor$Worker.run( >> ThreadPoolExecutor.java:615) >> 15. at java.lang.Thread.run(Thread.java:701) >> >> >> >> >
