subject:"\[jira\] \[Commented\] \(ARTEMIS\-1286\) Server stops responding and throws OutOfDirectMemoryError when sending \& receiving lots of 2MB messages."

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-03-14 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399436#comment-16399436
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~clebertsuconic] and [~jbertram]

I was doing my testing using Justin's branch and his example code which was 
based on my example code. Hence we were on the same page. :D

That's great news [~jbertram]! You have made me very happy! I was beginning to 
wonder if I was hallucinating the problem. When the time comes I will be VERY 
happy to assist testing. 

Gentlemen thank you very much for your patience and help.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: MQTTBasicPubSubExample2.zip, artemis log showing v2.4 
> still having trouble handling large messages.log, broker.xml, 
> image-2018-02-24-19-54-00-892.png, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:6

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-03-14 Thread Justin Bertram (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399420#comment-16399420
 ] 

Justin Bertram commented on ARTEMIS-1286:
-

I believe Clebert and I have identified a fix.  One of us will send a PR 
shortly which you can use to test in your own environment.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: MQTTBasicPubSubExample2.zip, artemis log showing v2.4 
> still having trouble handling large messages.log, broker.xml, 
> image-2018-02-24-19-54-00-892.png, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQ

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-03-14 Thread Justin Bertram (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399204#comment-16399204
 ] 

Justin Bertram commented on ARTEMIS-1286:
-

I increased the number of iterations the example runs to 500 and I've 
reproduced the OODME.  I'm continuing to investigate.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: MQTTBasicPubSubExample2.zip, artemis log showing v2.4 
> still having trouble handling large messages.log, broker.xml, 
> image-2018-02-24-19-54-00-892.png, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublis

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-03-14 Thread Justin Bertram (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399190#comment-16399190
 ] 

Justin Bertram commented on ARTEMIS-1286:
-

The example I created on [my 
branch|https://github.com/jbertram/activemq-artemis/tree/ARTEMIS-1286] to 
reproduce your issue has always used 
{{100Mb}}, but I haven't been able to 
reproduce the issue.  How is it you were using a different global-max-size if 
you were testing with the example from my branch?  If you weren't testing using 
the example from my branch, why not?

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: MQTTBasicPubSubExample2.zip, artemis log showing v2.4 
> still having trouble handling large messages.log, broker.xml, 
> image-2018-02-24-19-54-00-892.png, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.F

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-03-14 Thread clebert suconic (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399169#comment-16399169
 ] 

clebert suconic commented on ARTEMIS-1286:
--

Why you make a 2M max size BTW? 

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: MQTTBasicPubSubExample2.zip, artemis log showing v2.4 
> still having trouble handling large messages.log, broker.xml, 
> image-2018-02-24-19-54-00-892.png, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SN

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-03-14 Thread clebert suconic (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399155#comment-16399155
 ] 

clebert suconic commented on ARTEMIS-1286:
--

And did you try 2.5?

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: MQTTBasicPubSubExample2.zip, artemis log showing v2.4 
> still having trouble handling large messages.log, broker.xml, 
> image-2018-02-24-19-54-00-892.png, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-03-14 Thread clebert suconic (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399144#comment-16399144
 ] 

clebert suconic commented on ARTEMIS-1286:
--

[~pwjenkins] if you have lots of pages.. it means messages are not being Acked.

 

Can you attach a print-page

 

./artemis data print

 

 

if you are concerned with safety of your data, do 

 

./artemis data print --safe

 

(that will remove data specific from  your messages).

 

 

 

 

At this point.. it would be nice if you could reproduce the issue with some 
code... it could be environmental or a real bug.. but I have no way to guess 
without a reproducer. I take paging and journal seriously.. and I would work on 
it if you provide me a way to replicate the issue.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: MQTTBasicPubSubExample2.zip, artemis log showing v2.4 
> still having trouble handling large messages.log, broker.xml, 
> image-2018-02-24-19-54-00-892.png, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.A

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-03-14 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399134#comment-16399134
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~clebertsuconic] and [~jbertram],

I've reproduced the issue using 100Mb. 

I have the server in a separate console to see the error easily. This time I 
watched the java process that is running the server and noticed it growing 
during the test. As the messages were published during the test it continued to 
grow. That looks wrong and appears to show the root cause of the OODME error. 
Why does it grow? What sets the limit for that? The message QoS = 0 and I 
expected the messages to disappear as they are consumed. Also, the paging 
directory shows LOTS of PAGE files that appear to be my messages. Why doesn't 
that directory remove the consumed message? 

If [~jbertram] doesn't get the error then lets look at why my process grows and 
the paging directory isn't cleaned up. 

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: MQTTBasicPubSubExample2.zip, artemis log showing v2.4 
> still having trouble handling large messages.log, broker.xml, 
> image-2018-02-24-19-54-00-892.png, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-03-08 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391653#comment-16391653
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~clebertsuconic] and [~jbertram],

I was using the default global-max-size which comes out at around 500MB. I 
found that by increasing the memory I could delay the error. However, it was 
just a delay and eventually the error would appear. I can suggest increasing 
the loop count to 500 to see if it will appear. Something to run over lunch so 
it doesn't impact you too much. Until you reproduce you're stuck which means 
I'm stuck. 

It's a big deal here so I must brainstorm with others for possible next steps. 
Any suggestions from you would be most welcome indeed. 

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: MQTTBasicPubSubExample2.zip, artemis log showing v2.4 
> still having trouble handling large messages.log, broker.xml, 
> image-2018-02-24-19-54-00-892.png, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByt

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-03-07 Thread Justin Bertram (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390306#comment-16390306
 ] 

Justin Bertram commented on ARTEMIS-1286:
-

Let me clarify my previous comment:
* A reproducer is key.  I can't fix what I can't reproduce.
* Since the issue involves "direct" memory there may be non-obvious 
environmental related issues contributing to the problem (which may be why you 
can reproduce it and I can't).

Given you're receiving an {{OutOfDirectMemoryError}} I assume the the error is 
in fact related to a lack of memory - "direct" memory in particular.  Exactly 
how/why that's happening I can't say as I'm no expert on direct memory.

As I said previously, it's impossible for me to draw any firm conclusions at 
this point.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: MQTTBasicPubSubExample2.zip, artemis log showing v2.4 
> still having trouble handling large messages.log, broker.xml, 
> image-2018-02-24-19-54-00-892.png, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.ne

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-03-07 Thread clebert suconic (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390293#comment-16390293
 ] 

clebert suconic commented on ARTEMIS-1286:
--

You're still using the default global-max-size?

 

 

I will change the next version for a lower amount. but it's basically around 
the global-max-size.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: MQTTBasicPubSubExample2.zip, artemis log showing v2.4 
> still having trouble handling large messages.log, broker.xml, 
> image-2018-02-24-19-54-00-892.png, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemi

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-03-07 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390267#comment-16390267
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~jbertram], let me see if I understand you. My Windows workstation with 16G of 
memory is running out of memory? I have little understanding of netty and no 
understanding of what causes the OODME. Are you saying that because it's a 
"direct memory error" it must be an issue with how much available free memory I 
have? Despite having gigabytes of available RAM sending 3MB messages will cause 
my to consume available memory? Please clarify. 

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: MQTTBasicPubSubExample2.zip, artemis log showing v2.4 
> still having trouble handling large messages.log, broker.xml, 
> image-2018-02-24-19-54-00-892.png, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.rea

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-03-06 Thread Justin Bertram (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16388378#comment-16388378
 ] 

Justin Bertram commented on ARTEMIS-1286:
-

Aside from scheduling concerns I'm not sure how beneficial an interactive 
session would be.  At the end of the day, I need to be able to reproduce the 
issue in order to validate any fix and mitigate future regressions.  

Given the fact that the error here is related to "direct" memory (i.e. native 
memory off the Java heap) and the fact that you can reproduce it and I can't 
leads me to believe this is really an environmental issue.  I have 32GB of 
memory on my machine which is probably more than average.  Maybe that's causing 
the discrepancy in results.  Perhaps it's a tuning issue on your side.  At this 
point it's impossible to draw any firm conclusions.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: MQTTBasicPubSubExample2.zip, artemis log showing v2.4 
> still having trouble handling large messages.log, broker.xml, 
> image-2018-02-24-19-54-00-892.png, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocat

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-03-05 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387314#comment-16387314
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~jbertram], would it be possible to get together over webex or similar to walk 
thru a demo of the error?

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: MQTTBasicPubSubExample2.zip, artemis log showing v2.4 
> still having trouble handling large messages.log, broker.xml, 
> image-2018-02-24-19-54-00-892.png, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendSer

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-03-05 Thread Justin Bertram (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386028#comment-16386028
 ] 

Justin Bertram commented on ARTEMIS-1286:
-

[~pwjenkins], I let it run all the way through (i.e. to 200) both times.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: MQTTBasicPubSubExample2.zip, artemis log showing v2.4 
> still having trouble handling large messages.log, broker.xml, 
> image-2018-02-24-19-54-00-892.png, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:2

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-03-04 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16385619#comment-16385619
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~jbertram], that totally makes no sense. Hmm... I ran and repeated it twice. 
Running the server in a separate console makes the issue appear very clearly in 
the server console. At around message 100 it will throw the OODME exception. 
You did let it run that long didn't you? How long did you let it run for?

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: MQTTBasicPubSubExample2.zip, artemis log showing v2.4 
> still having trouble handling large messages.log, broker.xml, 
> image-2018-02-24-19-54-00-892.png, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-03-01 Thread Justin Bertram (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16382677#comment-16382677
 ] 

Justin Bertram commented on ARTEMIS-1286:
-

I update my branch with your attached code, but I still couldn't reproduce the 
problem.  I ran the example with both the broker and the client together (i.e. 
using {{mvn verify}}) and I also tried running the broker and client standalone 
(i.e. using {{mvn -PnoServer verify}}).

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: MQTTBasicPubSubExample2.zip, artemis log showing v2.4 
> still having trouble handling large messages.log, broker.xml, 
> image-2018-02-24-19-54-00-892.png, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(C

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-02-28 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16381504#comment-16381504
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~jbertram], I've attached an updated MQTTBasicPubSubExample2 class file that 
will cause the OODME in netty. Please try it at your earliest convenience.
Many thanks!

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: MQTTBasicPubSubExample2.zip, artemis log showing v2.4 
> still having trouble handling large messages.log, broker.xml, 
> image-2018-02-24-19-54-00-892.png, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.ac

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-02-26 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16377173#comment-16377173
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~jbertram], I set aside the issue in my last comment to focus only on getting 
the example code running in the Artemis 2.2.x source code from July 2017. I did 
not have these issues last July when I assembled the example code. So I'm 
struggling to get Artemis up and running now. This is about getting Artemis 
running not my code. I can see the SNAPSHOT in the m2 repository so this is 
quite frustrating. Why doesn't Maven see the jar is there and succeed? I'm 
grateful for any suggestions to get this working.

*Running:*
C:\Projects\activemq-artemis-master-July\examples\protocols\mqtt>mvn -Pexamples 
verify

*Results:*
[ERROR] Failed to execute goal 
org.apache.activemq:artemis-maven-plugin:2.2.0-SNAPSHOT:create (create) on 
project artemis-mqtt-publish-example: org/apache/activemq/artemis/boot/Artemis 
-> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.apache.activemq:artemis-maven-plugin:2.2.0-SNAPSHOT:create (create) on 
project artemis-mqtt-publish-example: org/apache/activemq/artemis/boot/Artemis
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:213)
at 
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
...

[~clebertsuconic]  [~martyntaylor]

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: artemis log showing v2.4 still having trouble handling 
> large messages.log, broker.xml, image-2018-02-24-19-54-00-892.png, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-02-24 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16375887#comment-16375887
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~jbertram], I've been trying to get Artemis up and running. You mentioned 
having trouble getting it up and running. I've reverted back to the code base 
of July to try to get this running. It shows the following failure I've yet to 
resolve. Suggestions would be most welcome.

!image-2018-02-24-19-54-00-892.png!

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: artemis log showing v2.4 still having trouble handling 
> large messages.log, broker.xml, image-2018-02-24-19-54-00-892.png, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBy

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-02-22 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16373427#comment-16373427
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~jbertram], I just ran our code against the v2.4 release. Sadly I got a 
different error message but the error still relates to large message. I've 
attached the log.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: artemis log showing v2.4 still having trouble handling 
> large messages.log, broker.xml, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMes

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-02-14 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364639#comment-16364639
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~jbertram], apologies for finally get back to you. Thank you for looking into 
this. It's greatly appreciated!

I found the issue was reproduced with the sample code I provided last July on 
the Artemis code at the time. Since you could not reproduce the issue I'm going 
to jump up to v2.4 and try reproducing the issue. If I can i will pull your 
branch and look into your example as you suggest. I'll respond back here with 
my test results.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: broker.xml, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBuf

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-01-18 Thread Justin Bertram (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16331232#comment-16331232
 ] 

Justin Bertram commented on ARTEMIS-1286:
-

I created a new branch on my GitHub repo - 
[https://github.com/jbertram/activemq-artemis/tree/ARTEMIS-1286|https://github.com/jbertram/activemq-artemis/tree/ARTEMIS-1286.].
  It has a new example called "basic-pubsub2" based on your attached example + 
broker.xml.  There are a few modifications from what you had so that it will 
actually compile and execute properly plus I removed all the dead code.  I 
recommend we use it as a baseline to create something that will reliably 
reproduce the OODME.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: broker.xml, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> o

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2018-01-18 Thread Justin Bertram (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16331171#comment-16331171
 ] 

Justin Bertram commented on ARTEMIS-1286:
-

I'm trying to reproduce the OODME with the example you provided in the attached 
mqtt.zip, but I have been unsuccessful so far.  Can you confirm that running 
the attached example reliably reproduced the OODME?

For what it's worth I had to actually modify the example in a few places as it 
wouldn't compile (due to instances access static fields).

I've been doing all my testing on the HEAD of the master branch.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: unscheduled
>
> Attachments: broker.xml, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBuff

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-08-09 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121055#comment-16121055
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~tabish121]  Please let me sincerely apologize for any rudeness on my part. 
Clebert indicated 2 weeks ago he might get to look at this. We need it and are 
trying to plan. I wish we could fix it. We have contributed to Artemis in the 
past but this issue is beyond us. I'm here to help in anyway I can.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: broker.xml, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemi

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-08-08 Thread Timothy Bish (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118349#comment-16118349
 ] 

Timothy Bish commented on ARTEMIS-1286:
---

[~pwjenkins] The folks working an the ActiveMQ project are in many cases 
working on issues on there own time or in coordination with other pressing 
matters in their day jobs.  Pressing people who are volunteering their time and 
energy to fix your problem is usually a good way of annoying people into 
ignoring you.  Artemis is an open source project, you can dive in and try and 
resolve issues as anyone else can, I'd recommend you take some time and try and 
see if you can fix your own issue.  

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: broker.xml, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.a

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-08-07 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117874#comment-16117874
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~martyntaylor], checking to see if there's been any work on this yet. Any 
progress? Any estimate on when it might get to resolution?

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: broker.xml, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.ar

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-08-06 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16115962#comment-16115962
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~martyntaylor], checking to see if there's been any work on this yet. Any 
progress? Any estimate on when it might get to resolution?

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: broker.xml, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.ar

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-08-03 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16113923#comment-16113923
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~martyntaylor], checking to see if there's been any work on this yet. Any 
progress? Any estimate on when it might get to resolution?

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Assignee: Martyn Taylor
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: broker.xml, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.ar

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-08-02 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16112140#comment-16112140
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~clebertsuconic], checking to see if there's been any work on this yet. Any 
progress? Any estimate on when it might get to resolution?

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: broker.xml, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublis

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-08-01 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110318#comment-16110318
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~clebertsuconic], checking to see if there's been any work on this yet. Any 
progress? Any estimate on when it might get to resolution? 

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: broker.xml, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPubli

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-07-31 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107799#comment-16107799
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~clebertsuconic], checking to see if this is in progress?  Any estimate of 
when you might get to resolution? 

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: broker.xml, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendMessage(MQTT

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-07-27 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16103366#comment-16103366
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

Just checking in to see if there was any review of this since last reply 
comment. 

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: broker.xml, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendMessage(MQTTPublishManager.java:121)
>

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-07-25 Thread clebert suconic (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100410#comment-16100410
 ] 

clebert suconic commented on ARTEMIS-1286:
--

I already cut 2.2.0 today... will have to look on this tomorrow...
we can do a 2.3.0 shortly.. .no need to wait this long again.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Priority: Blocker
> Fix For: 2.2.0
>
> Attachments: broker.xml, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-07-25 Thread clebert suconic (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100412#comment-16100412
 ] 

clebert suconic commented on ARTEMIS-1286:
--

I'm still not sure this is an issue.. it could be the fact that MQTT has 
message on hold due to the protocol.. I will have to check how you're doing 
this... will work on it tomorrow.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Priority: Blocker
> Fix For: 2.2.0
>
> Attachments: broker.xml, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apach

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-07-25 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100369#comment-16100369
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~clebertsuconic], just checking in to see if there was any review of this? Any 
discussion of a resolution release? 

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Priority: Blocker
> Fix For: 2.2.0
>
> Attachments: broker.xml, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendMessag

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-07-19 Thread clebert suconic (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093932#comment-16093932
 ] 

clebert suconic commented on ARTEMIS-1286:
--

Yes.. I won't have time to fix this myself... 

That wouldn't stop someone else to fix it by then... I will cut a release 
monday no matter what... we can have another one cut shortly after when this is 
done.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Priority: Blocker
> Fix For: 2.2.0
>
> Attachments: broker.xml, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-07-19 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093922#comment-16093922
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~clebertsuconic], I saw your message today that you are targeting Monday for 
v2.2 release. Presumably that's Monday 7/24. Is that correct? 

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Priority: Blocker
> Fix For: 2.2.0
>
> Attachments: broker.xml, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTP

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-07-19 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16093138#comment-16093138
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~clebertsuconic], have you tried using the attached project to reproduce the 
exception?
[~eduda], please consider fixing this issue as part of 2.2 release.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Priority: Blocker
> Fix For: 2.2.0
>
> Attachments: broker.xml, mqtt.zip
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.artemis.core.pro

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-07-17 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090676#comment-16090676
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~clebertsuconic], I have reproduced the exception. I started with the 
*ActiveMQ Artemis MQTT Examples* under 
*activemq-artemis-master\examples\protocols\mqtt* and modified from there.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Priority: Blocker
> Fix For: 2.2.0
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPu

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-07-14 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088102#comment-16088102
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

OK. Give me a bit to work that out. I'll get back ASAP.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Priority: Blocker
> Fix For: 2.2.0
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendMessage(MQTTPublishManager.java:121)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activ

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-07-14 Thread clebert suconic (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088074#comment-16088074
 ] 

clebert suconic commented on ARTEMIS-1286:
--

I would need something replicating the issue you're seeing. maybe you could 
deploy on top of the examples?

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0
>Reporter: Phillip Jenkins
>Priority: Blocker
> Fix For: 2.2.0
>
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendMessage(MQTTPublishManager.java:121)
>  [artemis-mqtt-protocol

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-07-14 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088041#comment-16088041
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~clebertsuconic], it looked good for a while but eventually failed. The 
settings are as they were before for v2.1 testing. 
global-max-size = 100Mb
max-size-bytes = 2Mb
page-size-bytes = 512000

{code:java}
15:38:35,200 WARN  [org.apache.activemq.artemis.core.server] AMQ222038: 
Starting paging on address 'SOAP.S.PRN.ADE0E203D58FBD00'; size is currently: 
4,194,817 bytes; max-size-bytes: 2,097,152
15:38:41,568 WARN  [org.apache.activemq.artemis.core.server] AMQ222038: 
Starting paging on address 'SOAP.S.PRN.ADE0E203D58FBD00'; size is currently: 
4,194,817 bytes; max-size-bytes: 2,097,152
15:38:41,583 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
removing consumer which did not handle a message, consumer=ServerConsumerImpl 
[id=2925, filter=null, binding=LocalQueueBinding 
[address=SOAP.S.PRN.ADE0E203D58FBD00, 
queue=QueueImpl[name=ADE0E203D58FBD00.SOAP.S.PRN.ADE0E203D58FBD00, 
postOffice=PostOfficeImpl 
[server=ActiveMQServerImpl::serverUUID=12799181-68d1-11e7-a7de-7c7a9188fa86], 
temp=false]@9145353, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
name=ADE0E203D58FBD00.SOAP.S.PRN.ADE0E203D58FBD00, 
clusterName=ADE0E203D58FBD00.SOAP.S.PRN.ADE0E203D58FBD0012799181-68d1-11e7-a7de-7c7a9188fa86]],
 message=PagedReferenceImpl [position=PagePositionImpl [pageNr=122, 
messageNr=0, recordID=-1], message=PagedMessageImpl [queueIDs=[2923], 
transactionID=-1, 
message=CoreMessage[messageID=4550,durable=false,userID=null,priority=0, 
timestamp=0,expiration=0, durable=false, 
address=SOAP.S.PRN.ADE0E203D58FBD00,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@220972713],
 deliveryTime=null, persistedCount=0, deliveryCount=1, 
subscription=PageSubscriptionImpl [cursorId=2923, 
queue=QueueImpl[name=ADE0E203D58FBD00.SOAP.S.PRN.ADE0E203D58FBD00, 
postOffice=PostOfficeImpl 
[server=ActiveMQServerImpl::serverUUID=12799181-68d1-11e7-a7de-7c7a9188fa86], 
temp=false]@9145353, filter = FilterImpl [sfilterString=NOT ((AMQAddress = 
'activemq.management') OR (AMQAddress = 'activemq.notifications'))]]]: 
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
byte(s) of direct memory (used: 1070032615, max: 1073741824)
at 
io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
 [netty-all-4.1.9.Final.jar:4.1.9.Final]
at 
io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
 [netty-all-4.1.9.Final.jar:4.1.9.Final]
at 
io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
 [netty-all-4.1.9.Final.jar:4.1.9.Final]
at 
io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
 [netty-all-4.1.9.Final.jar:4.1.9.Final]
at 
io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
 [netty-all-4.1.9.Final.jar:4.1.9.Final]
at 
io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
 [netty-all-4.1.9.Final.jar:4.1.9.Final]
at 
io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
 [netty-all-4.1.9.Final.jar:4.1.9.Final]
at 
io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
 [netty-all-4.1.9.Final.jar:4.1.9.Final]
at 
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
 [netty-all-4.1.9.Final.jar:4.1.9.Final]
at 
io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
 [netty-all-4.1.9.Final.jar:4.1.9.Final]
at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
[netty-all-4.1.9.Final.jar:4.1.9.Final]
at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
[netty-all-4.1.9.Final.jar:4.1.9.Final]
at 
org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
 [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at 
org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
 [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
at 
org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendMessage(MQTTPublishManager.java:121)
 [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
at 
org.apache.activemq.artemis.core.protocol.mqtt.MQTTSessionCallback.sendMessage(MQTTSessionCallback.java:52)
 [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
at 
org.apach

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-07-14 Thread clebert suconic (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087984#comment-16087984
 ] 

clebert suconic commented on ARTEMIS-1286:
--

I forgot about the latest MQTT fix when I suggested paging.. sorry about that. 
the native memory you mentioned seems pretty accurate with the other fix.

Let me know how it goes.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Phillip Jenkins
>Priority: Blocker
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendMessage(MQTTPu

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-07-14 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087962#comment-16087962
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

I'm working thru the test now. I'll get back ASAP.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Phillip Jenkins
>Priority: Blocker
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendMessage(MQTTPublishManager.java:121)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.artemis.core.protocol.mqt

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-07-14 Thread clebert suconic (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087939#comment-16087939
 ] 

clebert suconic commented on ARTEMIS-1286:
--

if you can try against master (next 2.2).. it would be helpful. yes!

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Phillip Jenkins
>Priority: Blocker
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendMessage(MQTTPublishManager.java:121)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.artemis

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-07-14 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087913#comment-16087913
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~clebertsuconic], since you pointed to a configuration issue I went back to 
v2.1. However, you raise a valid question. Should I run this against v2.2 so 
that I pickup the OODME fixes in master?

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Phillip Jenkins
>Priority: Blocker
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.s

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-07-14 Thread clebert suconic (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087899#comment-16087899
 ] 

clebert suconic commented on ARTEMIS-1286:
--

did you run against master? there was a recent fix recently, and I'm about to 
release.


other than that.. if you still see an isssue, please attach your test.. then we 
will know if it's a bug or something else.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Phillip Jenkins
>Priority: Blocker
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQT

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-07-14 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087873#comment-16087873
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

[~clebertsuconic], I've been trying various configurations. I'm not convinced. 
I can see paging is doing something. I see messages in the log and the paging 
folder has 29 bytes of data. Here's my broker.xml settings of interest.
global-max-size = 100Mb
max-size-bytes = 2Mb
page-size-bytes = 512000

The exception thrown is as follows:
14:27:03,090 WARN  [org.apache.activemq.artemis.core.server] AMQ222038: 
Starting paging on address 'SOAP.S.PRN.ADE0E203D58FBD00'; size is currently: 
4,194,817 bytes; max-size-bytes: 2,097,152
14:27:09,351 WARN  [org.apache.activemq.artemis.core.server] AMQ222038: 
Starting paging on address 'SOAP.S.PRN.ADE0E203D58FBD00'; size is currently: 
4,194,817 bytes; max-size-bytes: 2,097,152
14:27:09,366 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
removing consumer which did not handle a message, consumer=ServerConsumerImpl 
[id=15, filter=null, binding=LocalQueueBinding 
[address=SOAP.S.PRN.ADE0E203D58FBD00, 
queue=QueueImpl[name=ADE0E203D58FBD00.SOAP.S.PRN.ADE0E203D58FBD00, 
postOffice=PostOfficeImpl 
[server=ActiveMQServerImpl::serverUUID=cbf29555-68c8-11e7-a9f0-54ee7505882d], 
temp=false]@7b992a73, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
name=ADE0E203D58FBD00.SOAP.S.PRN.ADE0E203D58FBD00, 
clusterName=ADE0E203D58FBD00.SOAP.S.PRN.ADE0E203D58FBD00cbf29555-68c8-11e7-a9f0-54ee7505882d]],
 message=PagedReferenceImpl [position=PagePositionImpl [pageNr=72, messageNr=0, 
recordID=-1], message=PagedMessageImpl [queueIDs=[13], transactionID=-1, 
message=CoreMessage[messageID=2500,durable=false,userID=null,priority=0, 
timestamp=0,expiration=0, durable=false, 
address=SOAP.S.PRN.ADE0E203D58FBD00,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1737757493],
 deliveryTime=null, persistedCount=0, deliveryCount=1, 
subscription=PageSubscriptionImpl [cursorId=13, 
queue=QueueImpl[name=ADE0E203D58FBD00.SOAP.S.PRN.ADE0E203D58FBD00, 
postOffice=PostOfficeImpl 
[server=ActiveMQServerImpl::serverUUID=cbf29555-68c8-11e7-a9f0-54ee7505882d], 
temp=false]@7b992a73, filter = FilterImpl [sfilterString=NOT ((AMQAddress = 
'activemq.management') OR (AMQAddress = 'activemq.notifications'))]]]: 
io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
byte(s) of direct memory (used: 1073248181, max: 1073741824)
at 
io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
 [netty-all-4.1.9.Final.jar:4.1.9.Final]
at 
io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
 [netty-all-4.1.9.Final.jar:4.1.9.Final]
at 
io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
 [netty-all-4.1.9.Final.jar:4.1.9.Final]
at 
io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
 [netty-all-4.1.9.Final.jar:4.1.9.Final]
at 
io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
 [netty-all-4.1.9.Final.jar:4.1.9.Final]
at 
io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
 [netty-all-4.1.9.Final.jar:4.1.9.Final]
at 
io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
 [netty-all-4.1.9.Final.jar:4.1.9.Final]
at 
io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
 [netty-all-4.1.9.Final.jar:4.1.9.Final]
at 
io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
 [netty-all-4.1.9.Final.jar:4.1.9.Final]
at 
io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
 [netty-all-4.1.9.Final.jar:4.1.9.Final]
at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
[netty-all-4.1.9.Final.jar:4.1.9.Final]
at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
[netty-all-4.1.9.Final.jar:4.1.9.Final]
at 
org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
 [artemis-commons-2.1.0.jar:2.1.0]
at 
org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:274)
 [artemis-mqtt-protocol-2.1.0.jar:]
at 
org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendMessage(MQTTPublishManager.java:119)
 [artemis-mqtt-protocol-2.1.0.jar:]
...

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-07-13 Thread Phillip Jenkins (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16086188#comment-16086188
 ] 

Phillip Jenkins commented on ARTEMIS-1286:
--

Thanks for getting back. I will pursue that and get back.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Phillip Jenkins
>Priority: Blocker
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendMessage(MQTTPublishManager.java:121)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.artemis.core.proto

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-07-13 Thread clebert suconic (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16086178#comment-16086178
 ] 

clebert suconic commented on ARTEMIS-1286:
--

I will let you confirm this before closing.. or please close it yourself.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Phillip Jenkins
>Priority: Blocker
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendMessage(MQTTPublishManager.java:121)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.ar

[jira] [Commented] (ARTEMIS-1286) Server stops responding and throws OutOfDirectMemoryError when sending & receiving lots of 2MB messages.

2017-07-13 Thread clebert suconic (JIRA)


[ 
https://issues.apache.org/jira/browse/ARTEMIS-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16086176#comment-16086176
 ] 

clebert suconic commented on ARTEMIS-1286:
--

use a lower global-max-size, and make sure it's paging...

this is not a bug.. it's a configuration issue.

> Server stops responding and throws OutOfDirectMemoryError when sending & 
> receiving lots of 2MB messages.
> 
>
> Key: ARTEMIS-1286
> URL: https://issues.apache.org/jira/browse/ARTEMIS-1286
> Project: ActiveMQ Artemis
>  Issue Type: Bug
>  Components: Broker, MQTT
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Phillip Jenkins
>Priority: Blocker
>
> Originally seen in v2.1 and present in v2.2.
> If you send and receive a lot of 2MB messages in short time thru Artemis via 
> Netty connector, the server throws the following OutOfDirectMemoryError 
> exception. The server stops responding and will not (any longer) accept 
> connections without throwing exceptions in an infinite loop.
> {code:java}
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241001: HTTP Server 
> started at http://localhost:8161
> 11:24:07,434 INFO  [org.apache.activemq.artemis] AMQ241002: Artemis Jolokia 
> REST API available at http://localhost:8161/jolokia
> 11:58:35,991 WARN  [org.apache.activemq.artemis.core.server] AMQ222151: 
> removing consumer which did not handle a message, consumer=ServerConsumerImpl 
> [id=2229, filter=null, binding=LocalQueueBinding 
> [address=SOAP.S.PRN.ACBCAF6238234680, 
> queue=QueueImpl[name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> postOffice=PostOfficeImpl 
> [server=ActiveMQServerImpl::serverUUID=afa7de73-67e7-11e7-a231-54ee7505882d], 
> temp=false]@46adc2a5, filter=FilterImpl [sfilterString=NOT ((AMQAddress = 
> 'activemq.management') OR (AMQAddress = 'activemq.notifications'))], 
> name=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680, 
> clusterName=ACBCAF6238234680.SOAP.S.PRN.ACBCAF6238234680afa7de73-67e7-11e7-a231-54ee7505882d]],
>  
> message=Reference[3551]:NON-RELIABLE:CoreMessage[messageID=3551,durable=false,userID=null,priority=0,
>  timestamp=0,expiration=0, durable=false, 
> address=SOAP.S.PRN.ACBCAF6238234680,properties=TypedProperties[mqtt.message.retain=false,mqtt.qos.level=0]]@1305748199:
>  io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 3729415 
> byte(s) of direct memory (used: 1070952692, max: 1073741824)
>   at 
> io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:585)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:539)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledByteBufAllocator.java:169)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeDirectByteBuf.(UnpooledUnsafeDirectByteBuf.java:68)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator$InstrumentedUnpooledUnsafeNoCleanerDirectByteBuf.(UnpooledByteBufAllocator.java:164)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.UnpooledByteBufAllocator.newDirectBuffer(UnpooledByteBufAllocator.java:73)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:117)
>  [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:828) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at io.netty.buffer.WrappedByteBuf.readBytes(WrappedByteBuf.java:616) 
> [netty-all-4.1.9.Final.jar:4.1.9.Final]
>   at 
> org.apache.activemq.artemis.core.buffers.impl.ChannelBufferWrapper.readBytes(ChannelBufferWrapper.java:315)
>  [artemis-commons-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendServerMessage(MQTTPublishManager.java:277)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>   at 
> org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.sendMessage(MQTTPublishManager.java:121)
>  [artemis-mqtt-protocol-2.2.0-SNAPSHOT.jar:]
>

52 matches

Mail list logo