[
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364639#comment-16364639
]
Phillip Jenkins commented on ARTEMIS-1286:
------------------------------------------
[~jbertram], apologies for finally get back to you. Thank you for looking into
this. It's greatly appreciated!
I found the issue was reproduced with the sample code I provided last July on
the Artemis code at the time. Since you could not reproduce the issue I'm going
to jump up to v2.4 and try reproducing the issue. If I can i will pull your
branch and look into your example as you suggest. I'll respond back here with
my test results.
> Server stops responding and throws OutOfDirectMemoryError when sending &
> receiving lots of 2MB messages.
> --------------------------------------------------------------------------------------------------------
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Components: Broker, MQTT
> Affects Versions: 2.1.0
> Reporter: Phillip Jenkins
> Assignee: Martyn Taylor
> Priority: Blocker
> Fix For: unscheduled
>
> Attachments: broker.xml, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via
> Netty connector, the server throws the following OutOfDirectMemoryError
> exception. The server stops responding and will not (any longer) accept
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO [org.apache.activemq.artemis] AMQ241001: HTTP Server
> started at http://localhost:8161
> 11:24:07,434 INFO [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN [org.apache.activemq.artemis.core.server] AMQ222151:
> removing consumer which did not handle a message, consumer=ServerConsumerImpl
> [id=2229, filter=null, binding=LocalQueueBinding
> [address=SOAP.S.PRN.ACBCAF6238234680,
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680,
> postOffice=PostOfficeImpl
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d],
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress =
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))],
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680,
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
> timestamp=0,expiration=0, durable=false,
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
> io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
> at
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:68)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledByteBufAllocator.java:164)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
> [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
> [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
> at
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendMessage(MQTTPublishManager.java:121)
> [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
> at
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTSessionCallback.sendMessage(MQTTSessionCallback.java:52)
> [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.deliverStandardMessage(ServerConsumerImpl.java:1078)
> [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.activemq.artemis.core.server.impl.ServerConsumerImpl.proceedDeliver(ServerConsumerImpl.java:450)
> [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.proceedDeliver(QueueImpl.java:2688)
> [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.deliverDirect(QueueImpl.java:2673)
> [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.activemq.artemis.core.server.impl.QueueImpl.addTail(QueueImpl.java:645)
> [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.addReferences(PostOfficeImpl.java:1297)
> [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.access$000(PostOfficeImpl.java:93)
> [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl$1.done(PostOfficeImpl.java:1157)
> [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.executeOnCompletion(OperationContextImpl.java:181)
> [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.OperationContextImpl.executeOnCompletion(OperationContextImpl.java:130)
> [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.afterCompleteOperations(AbstractJournalStorageManager.java:303)
> [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.processRoute(PostOfficeImpl.java:1149)
> [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:787)
> [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.route(PostOfficeImpl.java:689)
> [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.doSend(ServerSessionImpl.java:1674)
> [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.send(ServerSessionImpl.java:1351)
> [artemis-server-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
> at
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendInternal(MQTTPublishManager.java:166)
> [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
> at
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.handleMessage(MQTTPublishManager.java:136)
> [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
> at
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.handlePublish(MQTTProtocolHandler.java:204)
> [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
> at
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.channelRead(MQTTProtocolHandler.java:118)
> [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:624)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:559)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:476)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:438)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_72]
> {code}
> Subsequent attempts to connect with the server will fail and causes an
> infinite loop throwing the following exception into the log until the server
> is shut down.
> {code:java}
> 12:10:31,004 WARN [io.netty.channel.DefaultChannelPipeline] An
> exceptionCaught() event was fired, and it reached at the tail of the
> pipeline. It usually means the last handler in the pipeline did not handle
> the exception.: io.netty.util.internal.OutOfDirectMemoryError: failed to
> allocate 16777216 byte(s) of direct memory (used: 1070952692, max: 1073741824)
> at
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:760)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:736)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at io.netty.buffer.PoolArena.allocate(PoolArena.java:214)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at io.netty.buffer.PoolArena.allocate(PoolArena.java:146)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:320)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:172)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:133)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:80)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:122)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:624)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:559)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:476)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:438)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
> at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_72]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
