[
https://issues.apache.org/jira/browse/AMQ-4495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Tully resolved AMQ-4495.
-----------------------------
Resolution: Fixed
On review - i have retained the status quo. The performance implications of
going to the store for a batch (page) of messages and dropping them due to no
space is too high.
Using a destination limit of X, and a cursorHighWaterMark of 70, only if usage
is < 0.7*X will we page in from the store. If we page In, it will be a pageSize
and we will accept the full page if a full page of messages exists. This may
cause usage to increase past X. However we won't go back to page in till usage
is again < 0.7*X.
The 0.7 is the cursorHighWaterMark and X is the memoryUsage (via policy
entries). PageSize is also configurable.
If per destinations limits are not set, then the global shared usage is
available to all destinations which can lead to starvation of consumers due to
the inability to page in because usage > 0.7*X due to messages in memory for
other destinations. Their consumers may have dropped off for example.
To divide a shared usage between N destinations:
0.7*X will be used for caching if the cache is enabled (useCache=true).
Then be prepared for potential increase of pageSize*messageSize usage when we
page in from the store for dispatch. This happens when the cache is exhausted
and usage drops below 0.7*X. Ideally, the 0.3*X that remains for pageIn is <
pageSize*messageSize.
For N destinations, the fraction available to each destinations needs to
account for the above.
This will ensure that the global shared usage is respected so the JVM heap
metrics can be sized sensibly.
Mitigation:
lazyDispatch - where we only page in what can be consumed can help limit the
pageSize dynamically if consumers are exact about their prefetch limit or use
pull consumers. With 3 pull consumers, we will only pageIn 3 messages if
lazyDispatch=true
> Improve cursor memory management
> --------------------------------
>
> Key: AMQ-4495
> URL: https://issues.apache.org/jira/browse/AMQ-4495
> Project: ActiveMQ
> Issue Type: Bug
> Reporter: Dejan Bosanac
> Assignee: Gary Tully
> Fix For: 5.9.0
>
> Attachments: FasterDispatchTest.java
>
>
> As currently stands, the store queue cursor will cache producer messages
> until it gets to the 70% (high watermark) of its usage. After that caching
> stops and messages goes only in store. When consumers comes, messages get
> dispatched to it, but memory isn't released until they are acked. The problem
> is with the use case where producer flow control is off and we have a
> prefetch large enough to get all our messages from the cache. Then, basically
> the cursor gets empty and as message acks release memory one by one, we go to
> the store and try to batch one message at the time. You can guess that things
> start to be really slow at that point.
> The solution for this scenario is to wait with batching until we have more
> space so that store access is optimized. We can do this by adding a new limit
> (smaller then the high watermark) which will be used as the limit after which
> we start filling cursor from the store again.
> All this led us to the following questions:
> 1. Why do we use 70% as the limit (instead of 100%) when we stop caching
> producer messages?
> 2. Would a solution that stop caching producer messages at 100% of usage and
> then start batching messages from the store when usage drops below high
> watermark value be enough. Of course, high watermark would be configurable,
> but 100% by default so we don't alter any behavior for regular use cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
