Broker paging all available messages from store leading to resource exhaustion
------------------------------------------------------------------------------
Key: AMQ-2532
URL: https://issues.apache.org/activemq/browse/AMQ-2532
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.3.0
Environment: Java 1.6
Reporter: Fabrice Delaporte
There seems to be a regression (works fine in 5.2.0) related to the fix for the
following:
https://issues.apache.org/activemq/browse/AMQ-2413
I think the intent of the original code was to prevent uncontrolled paging of
pending messages. The doPageIn method is called in version 5.3.0 :
- each time a message is acked/committed
- each time the MemoryUsage is decreased
As a result, if you create a queue with, say, 1 000 000 messages in it, then
start a consumer, the broker will try to page in all the messages from the
store as consuming goes on (with default configuration values, 200 paged in
messages for one consumed message minimum - I say minimum because if the
MemoryUsage is decreased when you consume a message, then doPageIn is invoked
again). Guaranteed OutOfMemoryError as in general consuming messages is slower
than paging from store. I guess the same can occur when you have a fast
producer and a slow consumer, without producer flow control.
I would say that in any case, the broker should never page in more messages
than the maxPageSize attribute, plus the number of dispatched messages. More
formally:
{code:java}
int toPageIn = getMaxPageSize() - pagedInPendingDispatch.size();
{code}
And just to make sure we never get a negative value:
{code:java}
int toPageIn = Max(0, getMaxPageSize() -
pagedInPendingDispatch.size());
{code}
Note 1: this behavior is also exhibited by the background message expiration
logic, that forces message pagination each time it is run. With big queues
without consumers, by default you'll get 200 more messages paginated each time
the expiration task is run. Eventually leading to broker resource exhaustion if
no consumer comes into play in the near future with active producers queuing up
messages.
Note 2: this may also be related to
https://issues.apache.org/activemq/browse/AMQ-2468
Workarounds:
- setting lazyDispatch to true fixes the problem for message consuming only
- background expiration can be disabled by setting expireMessagesPeriod to -1
Cheers,
Fabrice
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
