[
https://issues.apache.org/jira/browse/ARTEMIS-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17123672#comment-17123672
]
Bas commented on ARTEMIS-2696:
------------------------------
We see this again on version 2.13
> Netty DirectBufferLeak noticed in the log and server not processing messages
> with out of direct memory errors after running for a day
> -------------------------------------------------------------------------------------------------------------------------------------
>
> Key: ARTEMIS-2696
> URL: https://issues.apache.org/jira/browse/ARTEMIS-2696
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker
> Affects Versions: 2.11.0, 2.13.0
> Environment: Ubuntu 18.04.1 LTS
> Netty 4.1.45
> Qpid 0.48
>
>
> Reporter: Bas
> Priority: Major
>
> Our servers started to crash daily once we were on the 2.11.0 version.
> The crashes were not killing the entire artemis server but will cause endless
> log statements of the following:
> Transport failed: io.netty.util.internal.OutOfDirectMemoryError: failed to
> allocate 16777216 byte(s) of direct memory (used: 637534215, max: 652738560)
> We launched an investigation and activated the netty advanced leak detection:
> -Dio.netty.leakDetectionLevel=advanced
> After which we indeed noticed leak detection log entries with following
> information:
> #1:
> io.netty.buffer.AdvancedLeakAwareByteBuf.readBytes(AdvancedLeakAwareByteBuf.java:484)
>
> org.apache.activemq.artemis.core.server.protocol.websocket.WebSocketFrameEncoder.writeContinuationFrame(WebSocketFrameEncoder.java:56)
>
> org.apache.activemq.artemis.core.server.protocol.websocket.WebSocketFrameEncoder.write(WebSocketFrameEncoder.java:45)
>
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:715)
>
> io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:762)
>
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:788)
>
> io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:756)
>
> io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1020)
> io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:299)
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnection.write(NettyConnection.java:382)
>
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnection.write(NettyConnection.java:286)
>
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnection.write(NettyConnection.java:281)
>
> org.apache.activemq.artemis.protocol.amqp.broker.AMQPConnectionCallback.onTransport(AMQPConnectionCallback.java:203)
>
> org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.pushBytes(AMQPConnectionContext.java:384)
>
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.actualFlush(ProtonHandler.java:210)
>
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
>
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
>
> #2: Hint: 'websocket-frame-encoder' will handle the message from this point.
> io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
>
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:784)
>
> io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:756)
>
> io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1020)
> io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:299)
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnection.write(NettyConnection.java:382)
>
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnection.write(NettyConnection.java:286)
>
> org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnection.write(NettyConnection.java:281)
>
> org.apache.activemq.artemis.protocol.amqp.broker.AMQPConnectionCallback.onTransport(AMQPConnectionCallback.java:203)
>
> org.apache.activemq.artemis.protocol.amqp.proton.AMQPConnectionContext.pushBytes(AMQPConnectionContext.java:384)
>
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.actualFlush(ProtonHandler.java:210)
>
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
>
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
>
> #3:
> io.netty.buffer.AdvancedLeakAwareByteBuf.writeBytes(AdvancedLeakAwareByteBuf.java:622)
>
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.actualFlush(ProtonHandler.java:207)
>
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
>
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
>
> Created at:
> io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:349)
>
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:187)
>
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:178)
>
> org.apache.activemq.artemis.protocol.amqp.proton.handler.ProtonHandler.actualFlush(ProtonHandler.java:206)
>
> io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
>
> io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
> io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
> io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
> io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
> org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
>
> We looked into the source code of some classes in the above stack traces to
> look for changes where a directbuffer.release() would be missing but did not
> find any. It is also though to grasp what is happening in these classes
> because I really need to look into the concepts here if I want to understand
> what is happening and what needs to happen. Which will take a lot of time.
> But something got my attention where I might be totally wrong in my
> estimation as being related to the issue.
>
> The only thing I have on this is the following information has a direct
> relation with a class in the leak report from netty:
> [https://github.com/apache/activemq-artemis/blob/master/artemis-server/src/main/java/org/apache/activemq/artemis/core/server/protocol/websocket/WebSocketFrameEncoder.java]
> This class is added in following commit doing something with a websocket
> continuation frame:
> [https://github.com/apache/activemq-artemis/commit/9fac4b866cf9cefbb6f7c13b820e07455b6649f5]
>
> This commit is not in 2.10:
> git tag --contains 9fac4b8
> 2.11.0
> So, we wanted to test if doing a rollback to 2.10 would fix the issue. The
> 2.10 version is now running for 3 days without having issues while with 2.11
> it would have crashed with out of memory errors 3 times already.
>
> The issue could be in a different commit/location than what is analysed above
> but 2.10 seems to work without issues.
> We also wiped our entine journal before going to 2.10 so I want to test the
> 2.11 again with a clean journal and see if wiping the data store could have
> caused 2.10 to function better. So more information will follow.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
