[
https://issues.apache.org/jira/browse/HBASE-16635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15493041#comment-15493041
]
ramkrishna.s.vasudevan commented on HBASE-16635:
------------------------------------------------
Thanks [~Apache9]
I think your guess i right. The two connection ByteBufs aren't released I
believe.
{code}
2016-09-15 22:05:12,617 ERROR [Default-IPC-NioEventLoopGroup-1-26]
util.ResourceLeakDetector: LEAK: ByteBuf.release() was not called before it's
garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for
more information.
WARNING: 2 leak records were discarded because the leak record count is limited
to 4. Use system property io.netty.leakDetection.maxRecords to increase the
limit.
Recent access records: 5
#5:
io.netty.buffer.AdvancedLeakAwareByteBuf.release(AdvancedLeakAwareByteBuf.java:955)
io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:84)
io.netty.util.ReferenceCountUtil.safeRelease(ReferenceCountUtil.java:109)
io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:265)
io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:346)
io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:426)
io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:844)
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:360)
io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:811)
io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1276)
io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:762)
io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:754)
io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:735)
io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117)
io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:762)
io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:788)
io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:800)
io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:780)
io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:817)
io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1011)
io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:289)
org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:228)
org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:218)
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:518)
io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:511)
io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:490)
io.netty.util.concurrent.DefaultPromise.notifyListenersWithStackOverFlowProtection(DefaultPromise.java:431)
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:108)
io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:304)
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:580)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:504)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:418)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:390)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742)
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145)
java.lang.Thread.run(Thread.java:745)
#4:
io.netty.buffer.AdvancedLeakAwareByteBuf.internalNioBuffer(AdvancedLeakAwareByteBuf.java:735)
io.netty.channel.ChannelOutboundBuffer.nioBuffers(ChannelOutboundBuffer.java:420)
io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:379)
io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:844)
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:360)
io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:811)
io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1276)
io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:762)
io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:754)
io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:735)
io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117)
io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:762)
io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:788)
io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:800)
io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:780)
io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:817)
io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1011)
io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:289)
org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:228)
org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:218)
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:518)
io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:511)
io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:490)
io.netty.util.concurrent.DefaultPromise.notifyListenersWithStackOverFlowProtection(DefaultPromise.java:431)
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:108)
io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:304)
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:580)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:504)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:418)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:390)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742)
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145)
java.lang.Thread.run(Thread.java:745)
#3:
io.netty.buffer.AdvancedLeakAwareByteBuf.nioBufferCount(AdvancedLeakAwareByteBuf.java:705)
io.netty.channel.ChannelOutboundBuffer.nioBuffers(ChannelOutboundBuffer.java:408)
io.netty.channel.socket.nio.NioSocketChannel.doWrite(NioSocketChannel.java:379)
io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:844)
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:360)
io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:811)
io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1276)
io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:762)
io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:754)
io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:735)
io.netty.channel.ChannelDuplexHandler.flush(ChannelDuplexHandler.java:117)
io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:762)
io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:788)
io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:800)
io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:780)
io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:817)
io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1011)
io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:289)
org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:228)
org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:218)
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:518)
io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:511)
io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:490)
io.netty.util.concurrent.DefaultPromise.notifyListenersWithStackOverFlowProtection(DefaultPromise.java:431)
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:108)
io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:304)
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:580)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:504)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:418)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:390)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742)
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145)
java.lang.Thread.run(Thread.java:745)
#2:
Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message
from this point.
io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107)
io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:796)
io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:709)
org.apache.hadoop.hbase.ipc.BufferCallBeforeInitHandler.write(BufferCallBeforeInitHandler.java:73)
io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:724)
io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:787)
io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:800)
io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:780)
io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:817)
io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1011)
io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:289)
org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:228)
org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:218)
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:518)
io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:511)
io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:490)
io.netty.util.concurrent.DefaultPromise.notifyListenersWithStackOverFlowProtection(DefaultPromise.java:431)
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:108)
io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:304)
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:580)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:504)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:418)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:390)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742)
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145)
java.lang.Thread.run(Thread.java:745)
#1:
Hint: 'BufferCallBeforeInitHandler#0' will handle the message from this
point.
io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107)
io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:796)
io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:780)
io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:817)
io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1011)
io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:289)
org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:228)
org.apache.hadoop.hbase.ipc.NettyRpcConnection$3.operationComplete(NettyRpcConnection.java:218)
io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:518)
io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:511)
io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:490)
io.netty.util.concurrent.DefaultPromise.notifyListenersWithStackOverFlowProtection(DefaultPromise.java:431)
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:420)
io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:108)
io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:304)
io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339)
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:580)
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:504)
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:418)
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:390)
io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:742)
io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:145)
java.lang.Thread.run(Thread.java:745)
Created at:
io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:69)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:170)
io.netty.buffer.Unpooled.directBuffer(Unpooled.java:125)
org.apache.hadoop.hbase.ipc.NettyRpcConnection.<init>(NettyRpcConnection.java:90)
org.apache.hadoop.hbase.ipc.NettyRpcClient.createConnection(NettyRpcClient.java:73)
org.apache.hadoop.hbase.ipc.NettyRpcClient.createConnection(NettyRpcClient.java:39)
org.apache.hadoop.hbase.ipc.AbstractRpcClient.getConnection(AbstractRpcClient.java:343)
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:395)
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:309)
org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:92)
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:522)
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:38943)
org.apache.hadoop.hbase.client.HTable$1.rpcCall(HTable.java:433)
org.apache.hadoop.hbase.client.HTable$1.rpcCall(HTable.java:428)
org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:115)
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:102)
org.apache.hadoop.hbase.client.HTable.get(HTable.java:438)
org.apache.hadoop.hbase.client.HTable.get(HTable.java:411)
org.apache.hadoop.hbase.MetaTableAccessor.getTableState(MetaTableAccessor.java:1100)
org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:413)
org.apache.hadoop.hbase.client.HBaseAdmin$2.rpcCall(HBaseAdmin.java:318)
org.apache.hadoop.hbase.client.HBaseAdmin$2.rpcCall(HBaseAdmin.java:315)
org.apache.hadoop.hbase.client.RpcRetryingCallable.call(RpcRetryingCallable.java:58)
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:102)
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:2889)
org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:2881)
org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:315)
org.apache.hadoop.hbase.PerformanceEvaluation.checkTable(PerformanceEvaluation.java:313)
org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:1799)
org.apache.hadoop.hbase.PerformanceEvaluation.run(PerformanceEvaluation.java:2213)
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:2234)
{code}
If you don't mind, can I work on this patch. Anyway it is good to learn netty
and its API usages. And pls do help me in reviews and fixes.
> RpcClient under heavy load leaks some netty bytebuf
> ---------------------------------------------------
>
> Key: HBASE-16635
> URL: https://issues.apache.org/jira/browse/HBASE-16635
> Project: HBase
> Issue Type: Bug
> Reporter: ramkrishna.s.vasudevan
> Assignee: ramkrishna.s.vasudevan
> Priority: Minor
> Fix For: 2.0.0
>
>
> Yet to analyse the actual root cause.
> But the case is that when we run a PE tool with 50 threads under heavy load
> when the writes are clogged I think we have some netty Bytebuf leak. Not sure
> if it is a serious issue but we get this log
> {code}
> 2016-09-14 19:37:09,767 ERROR [Default-IPC-NioEventLoopGroup-1-16]
> util.ResourceLeakDetector: LEAK: ByteBuf.release() was not called before it's
> garbage-collected. Enable advanced leak reporting to find out where the leak
> occurred. To enable advanced leak reporting, specify the JVM option
> '-Dio.netty.leakDetection.level=advanced' or call
> ResourceLeakDetector.setLevel() See
> http://netty.io/wiki/reference-counted-objects.html for more information.
> {code}
> So reading the given link it is because of some ByteBuf that was not released
> properly by the client and hence it gets GCed automatically. Netty provides
> tips and tricks to find the root cause. Will get back here.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
