Re: ActiveMQ 5.11.0: Out Of Memory, caused by PageFile object tree filling up heap

[Tom Hall]

Hi Art,
What per destination limit are you looking for? The per destination limits will 
deal with memory used per queue.
http://activemq.apache.org/per-destination-policies.html 

 
   

-Tom

> On Jul 20, 2018, at 3:50 AM, art.licis  wrote:
> 
> 300mb memory limit is on a VM having -XmX 4Gb for heap; on 8Gb (most likely)
> it was more. While I totally agree 300 Mb memory limit is far from the
> optimal setting, and even better per-destination limits are needed.
> 
> However, I have stress tested ActiveMQ with much less memory, and I've seen
> the following in logs:
> 
> topic://topic-3, Usage Manager memory
> limit reached 5242880. Producers will be throttled to the rate at which
> messages are removed from this destination to prevent flooding it.
> 
> << just 5Mb of memory limit on this test!
> 
> .. and eventually this:
> 
> TopicSubscription: consumer=ID:a-thinkpad-44715-1532031318503-1:1:1:1,
> destinations=1, dispatched=32766, delivered=15963293, matched=1460,
> discarded=0, prefetchExtension=0: Pending message cursor
> [org.apache.activemq.broker.region.cursors.FilePendingMessageCursor@6bba423c]
> is full, temp usag (100%) or memory usage (0%) limit reached, blocking
> message add() pending the release of resources.
> 
> << all 10Gb of temp store filled!
> 
> 
> Which means filling up temp store and blocking the producer, but NOT
> crashing with out of memory, having all heap filled with PageFile-related
> data.
> 
> - Art
> 
> 
> 
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: ActiveMQ 5.11.0: Out Of Memory, caused by PageFile object tree filling up heap

[art.licis]

300mb memory limit is on a VM having -XmX 4Gb for heap; on 8Gb (most likely)
it was more. While I totally agree 300 Mb memory limit is far from the
optimal setting, and even better per-destination limits are needed.

However, I have stress tested ActiveMQ with much less memory, and I've seen
the following in logs:

topic://topic-3, Usage Manager memory
 limit reached 5242880. Producers will be throttled to the rate at which
messages are removed from this destination to prevent flooding it.

<< just 5Mb of memory limit on this test!

.. and eventually this:

TopicSubscription: consumer=ID:a-thinkpad-44715-1532031318503-1:1:1:1,
destinations=1, dispatched=32766, delivered=15963293, matched=1460,
discarded=0, prefetchExtension=0: Pending message cursor
[org.apache.activemq.broker.region.cursors.FilePendingMessageCursor@6bba423c]
is full, temp usag (100%) or memory usage (0%) limit reached, blocking
message add() pending the release of resources.

<< all 10Gb of temp store filled!


Which means filling up temp store and blocking the producer, but NOT
crashing with out of memory, having all heap filled with PageFile-related
data.

- Art



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Re: ActiveMQ 5.11.0: Out Of Memory, caused by PageFile object tree filling up heap

[Tom Hall]

Are you really limiting ram to 300 mb? Why so low if you are on a 8G box?
Thanks,
Tom

> On Jul 19, 2018, at 7:13 AM, art.licis  wrote:
> 
> Dear community,
> 
> We've been evaluation AMQ which is based on version 5.11.0. Several tests in
> staging environments which would have a big number of producers and
> consumers ended up with OutOfMemory: GC Overhead Limit exceeded exception.
> However, it looks like it's just one slow consumer that is causing this
> memory problem (details below).
> 
> Initially, we tried running AMQ with 4Gb of max heap, later with 8Gb but it
> had the same outcome.
> 
> Configuration: we have intentionally producer flow control turned off, and
> expecting temp store to fill up before the producer is blocked
> (non-persistent topic messages). I can simulate this with a simple and
> controlled tests involving very fast producer, and several fast and some
> slow consumers. With manual testing we achieve the desired result: when temp
> store reaches the limit, the producer is blocked.
> 
> We analyzed GC outputs and heap dump which we got upon the crash. Time spent
> in GC keeps significantly increasing before the crash, and heap dump shows
> the majority of memory occupied by PageFile object tree (lots of
> PageFile$PageWrite objects, which indicate that too much data remains in
> memory uncommitted to temp store; also related DataFileAppender$WriteBatch).
> PageFile is the correct one, the one used in PListStoreImpl (i.e. temp store
> for non-persistent messages).
> 
> During my analysis I also discovered some bug that was fixed recently -
> *AMQ-6815*. It looks like it affects the exact class family that could have
> caused the memory leak. 
> 
> I'd be grateful for any help or suggestions. I'm currently preparing the
> bigger test environment with many producers/consumers (and I'm also gonna
> run Activemq in docker to limit I/O speed while I'm sure it wasn't the
> issue), but the fact is that out of potentially thousands of
> producers/consumers there was only one slow consumer. Eventually, we will
> try running whole system against latest ActiveMQ, however, it will take lots
> of arrangements and I'd like to see if I can make reproducable and
> controlled test environment, and if there're some pitfalls in current setup.
> 
> activemq.xml is below (the slow consumer was not the one listening to
> virtual topic). And I will post more info if I'm able to solve the issue or
> at least get more hints.
> 
> Thank you.
> -Art
> 
>   xmlns="http://www.springframework.org/schema/beans;
>  xmlns:amq="http://activemq.apache.org/schema/core;
>  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
>  xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd
>  http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd;>
> 
> 
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>
>
>
>
> 
>http://activemq.apache.org/schema/core;
> brokerName="${broker-name}" dataDirectory="${data}" start="false"
> restartAllowed="false">
> 
>
>
>  
>
>
> optimizedDispatch="true" >
>
>  
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
>
> 
>
>
> 
>
>
>
>
>
>
>
>
>
>
>
>
>
> 
>
> uri="tcp://0.0.0.0:0?maximumConnections=1000wireFormat.maxFrameSize=104857600"/>
>
> 
>
>
>
> forwardOnly="false">
>
> physicalName="rpc.request.virtual.notifications" />
>
>
> forwardOnly="false">
>
> physicalName="rpc.response.virtual.notifications" />
>
>
>
>
>
> 
>
> 
> 
> 
> 
> 
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html