[
https://issues.apache.org/jira/browse/AMQ-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13809149#comment-13809149
]
Claus Ibsen commented on AMQ-4151:
----------------------------------
Did you ever manage to build an unit test?
> Duplicate non-persistent messages that are sent to a queue are either
> dispatched multiple times (i.e., not detected as duplicates) or cause the
> queue size to be miscalculated.
> -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: AMQ-4151
> URL: https://issues.apache.org/jira/browse/AMQ-4151
> Project: ActiveMQ
> Issue Type: Bug
> Reporter: Stirling Chow
>
> Symptom
> =======
> We have a virtual topic that is shared by a network of brokers. Each broker
> has consumers that process the corresponding Consumer.*.VirtualTopic.> queue.
> While testing the effects of memory limits on our system, we encountered
> AMQ-4148, which resulted in duplicate subscriptions being made to the virtual
> topic. These duplicate subcriptions were attempting to enqueue the same
> topic message multiple times to the corresponding Consumer.*.VirtualTopic.>.
> Although logic exists in queues to handle duplicate messages, we noticed lots
> of strange behaviour with duplicate handling. We decided to investigate
> further and found a race condition whereby duplicate non-persistent messages
> that were correctly ignored resulted in the queue size appearing to be
> non-empty when in fact there were no messages.
> While our original investigation was prompted by AMQ-4148, the test case we
> attached to this ticket does not rely on the misbehaviour caused by AMQ-4148,
> but can occur during normal operation of network bridges when there are
> multiple consumers for a virtual topic (as is the case, e.g., when conduit
> subscriptions are disabled).
> Cause
> =====
> If a virtual topic has multiple consumers, then sending a message to the
> topic results in multiple dispatches of the same message (one to each
> consumer). If the multiple consumers originate from the same remote broker
> (e.g., because conduit subscriptions are disabled), then a single message
> sent to the virtual topic on broker1 will result in multiple duplicate
> messages being sent to the virtual topic on broker2.
> If broker2 has corresponding Consumer.*.VirtualTopic.> queue, then the
> multiple duplicate messages sent to the virtual topic on broker2 will result
> in multiple duplicate messages being sent to the queue.
> If the messages are non-persistent, the only logic that prevents each
> duplicate message from being dispatched to a consumer is as follows:
> {code:title=Queue.java}
> private PendingList doPageInForDispatch(boolean force) throws Exception {
> ...
> // Only add new messages, not already pagedIn to avoid multiple
> // dispatch attempts
> pagedInMessagesLock.writeLock().lock();
> try {
> if(isPrioritizedMessages()) {
> resultList = new PrioritizedPendingList();
> } else {
> resultList = new OrderedPendingList();
> }
> for (QueueMessageReference ref : result) {
> if (!pagedInMessages.containsKey(ref.getMessageId())) {
> pagedInMessages.put(ref.getMessageId(), ref);
> resultList.addMessageLast(ref);
> } else {
> ref.decrementReferenceCount();
> }
> }
> } finally {
> pagedInMessagesLock.writeLock().unlock();
> }
> ...
> {code}
> If the consumers are fast and acknowledge the initial message dispatch before
> the next duplicate message is sent to the queue, then the
> {{pagedInMessages.constainsKey(...)}} check will *not* prevent the duplicate
> message from being dispatched since the message ID will have already been
> removed as part of the acknowledgement.
> This is problem 1: duplicate detection fails if acknowledgements are quick
> If the consumers are slow and don't acknowledge the initial message dispatch
> before the next duplicate message is sent to the queue, then duplicate
> detection will correctly ignore the message. However, by this time, the
> duplicate message has already incremented the queue size:
> {code:title=Queue.java}
> final void messageSent(final ConnectionContext context, final Message msg)
> throws Exception {
> destinationStatistics.getEnqueues().increment();
> destinationStatistics.getMessages().increment();
> messageDelivered(context, msg);
> {code}
> The call to {{messageSent}} is made by the thread sending the message to the
> queue. This thread is completely unaware that the message was ignored since
> the call to {{doPageInForDispatch}} is done by a separate thread (e.g., the
> taskrunner calling {{iterate()}}.
> Since the queue size is incremented, but the duplicate message is never
> dispatched, there is no subsequent acknowledgement ot reduce the queue size.
> As a result, each duplicate message that is ignored remains counted in the
> queue size even though it's no longer in the queue.
> This is problem 2: if duplication detection succeeds, the queue size
> incorrectly counts the duplicate/ignored message.
> Which problem gets exhibited depends on a race condition between the
> thread(s) enqueueing the duplicate virtual topic subscriptions and consumer
> threads acknowledging the dispatches.
> While the unit test's methodology of creating multiple virtual topic
> consumers is somewhat contrived, there must be other circumstances under
> which duplicate messages are sent to queues (otherwise, why would there be
> logic to handle this case?) In this context, the test case reveals a problem
> with the handling of duplicate messages.
> Solution
> ========
> None at this time. We worked around the problem by patching AMQ-4148 to
> prevent the duplicate subscriptions that cause this bug. However, our
> concern has been raised about AMQ's generally handling of duplicate messages
> sent to a queue.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
