[
https://issues.apache.org/jira/browse/AMQ-2552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dejan Bosanac updated AMQ-2552:
-------------------------------
Fix Version/s: (was: 5.4.2)
5.5.0
> Automatic message expiration may not occur on a queue with no (or slow)
> consumers.
> ----------------------------------------------------------------------------------
>
> Key: AMQ-2552
> URL: https://issues.apache.org/jira/browse/AMQ-2552
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.3.1
> Environment: ActiveMQ 5.3.1 Snapshot r893661
> Reporter: Stirling Chow
> Fix For: 5.5.0
>
> Attachments: AMQTest.java
>
>
> Symptom
> ========
> If a queue is loaded with 400+ messages that never expire, and then a message
> that does expire is added to the end of the queue, the added message will not
> be processed by the Broker's expiration logic until some of the 400+
> unexpired messages are consumed. A unit test is provided to demonstrate this
> issue.
> These test cases worked for AMQ 5.3, but not for AMQ 5.3.1-SNAPSHOT r893661.
> Motivation
> ========
> Our application uses message expiration to trigger actions when messages are
> left in a queue for too long a period. In many cases, there are no consumers
> on the queue containing the messages that might expire. Prior versions of
> AMQ only expired messages when an attempt was made to dispatch them to a
> consumer, but AMQ-1112 introduced the notion of periodic, automatic
> expiration of pending messages (see the expireMessagesPeriod policy). With
> this periodic expiration, our expectation is that expired messages will be
> detected and processed in a timely fashion, regardless of the presence of
> consumers on the queue.
> Cause
> ======
> AMQ-2481 removed forced paging from the periodically-called expireMessages()
> method:
> >>
> doBrowse(true, browsedMessages, this.getMaxExpirePageSize());
> <<
> doBrowse(browsedMessages, this.getMaxExpirePageSize());
> The purpose of the change was to limit the number of messages that would be
> paged into memory from the message store. The side effect is that only the
> first 400 messages in the queue are ever inspected by expireMessages(). If
> these 400 messages do not contain the messages that are expiring and there
> are no active consumers to move the expiring messages forward, then
> expiration will never occur. With AMQ 5.3, successive calls to
> expireMessages() would eventually pull all messages into the page so they
> could be checked for expiration.
> It should also be noted that the test cases can be modified so there is a
> consumer (with a selector that doesn't choose any of the messages), and they
> still fail showing that the issue is not purely the result of having *no*
> consumers.
> Resolution
> =========
> I don't think the browse idiom works very well for the automated expiration
> task. It was my initial assumption (and expectation) that the
> expireMessagesPeriod task would process *all* expired messages in the queue
> *each time* the task fired. A related issue to limiting the number of
> messages that are examined during each expiration task is that expiring
> messages at the end of the queue will not be handled in a timely fashion
> since they must wait for multiple firings of the expiration task (this is on
> 5.3 code; 5.3.1 code will never get to those messages).
> If the desire/need is to scan all messages to determine which have expired,
> the existing paging strategy is limited because it would require all messages
> to be brought from store into memory, thus defeating the whole point of
> paging. An alternative might be to have a sliding page window that would
> allow batches of messages to be brought in while clearing out the previous
> batch.
> Another solution might be to have the expiration task page in directly from
> the store, but this is problematic since non-persistent messages would still
> have to come in from the pending message cursor. In addition, a complete
> scan of the store might be non-performant for large queues.
> I think the most satisfactory solution for minimizing unnecessary processing
> and providing timely expiration of messages is to implement a proper heap
> that indexes messages with expiration values and allows immediate access to
> *only* the messages that have expired. The references to these expired
> messages could then be used to remove the messages directly from the cursors
> and the store rather than having to iterate over these structures.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
