[
https://issues.apache.org/jira/browse/ARTEMIS-3766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17519676#comment-17519676
]
Arthur Naseef commented on ARTEMIS-3766:
----------------------------------------
This one is a little complicated by the different scenarios that can occur,
especially with dual mirroring. If there were explicit broker "HOT" and "COLD"
states for the mirror source and target, that would make this simpler to
resolve, but of course adds the complexity of maintaining that HOT/COLD status.
Anytime we have "live/live" operations and attempt to keep both sides in-sync,
it gets to be complicated. Having one side designated as live (or HOT) and the
other side as idle/COLD can eliminate a lot of challenges there.
Here are some scenarios I can think of:
*Scenario 1*
* Message with TTL copied across mirror to target
* Message expiration processed on the source, causing the message to move into
the ExpiryQueue
* ExpiryQueue message copied across mirror to target
* ExpiryQueue message is consumed/removed on the mirror source
* Message expiration processed on the target
Expected outcome:
* Message is removed everywhere (i.e. removed from original queue and
ExpiryQueue on both source and target mirrors)
*Scenario 2*
* Message with TTL copied across mirror to target
* Source broker shuts down
* (A) Message expiration processed on the target, causing the message to move
into the ExpiryQueue
* (B) Source broker resumes operation
* Message expiration processed on the source, causing the message to move into
the ExpiryQueue
* ExpiryQueue message is consumed/removed on the mirror source
Expected outcome:
* After (A) completes, message exists in the target's ExpiryQueue
* After (B) completes, the source broker still has the message in the original
queue and does not see the ExpiryQueue message
* On completion, the message again is removed everywhere
*Scenario 3*
* Message with TTL copied across mirror to target
* Message is consumed from the mirror source
* Message expiration processed on the target before the mirrored ACK is
processed, causing the message to move into the ExpiryQueue
Expected outcome:
* On completion, the message again is removed everywhere
Is there a way to keep track of the ExpiryQueue message and tie its state to
the original message such that mirror operations can map incoming message ACKs
to both unexpired and expired messages?
Does DLQ handling create more of the same?
Hope this helps.
> Race condition with Dual Mirror and Expiry
> ------------------------------------------
>
> Key: ARTEMIS-3766
> URL: https://issues.apache.org/jira/browse/ARTEMIS-3766
> Project: ActiveMQ Artemis
> Issue Type: Bug
> Affects Versions: 2.21.0
> Reporter: Stephen Baker
> Priority: Major
>
> When dual mirroring is in play there is a race condition which can result in
> missing or duplicate messages on the brokers.
>
> Scenario 1:
> # Bring up two artemis instances with broker-connection mirroring between
> each other. (A and B)
> # Send a message to A with a short TTL (I used 30 seconds in my test)
> # After the message mirrors but before it expires, pause the Mirror queue on
> both sides
> # Wait for the message to expire, and the reaper thread to pick it up
> # Observe that on both sides the message has moved to the ExpiryQueue. In
> the Mirror queue there are 2 messages on A (a message destined to the
> ExpiryQueue and an ack on the original message). On B there is a single
> message in the mirror, which is the message to the expiry queue.
> # Resume the mirror, I'm not sure the order matters, but I did B and then A.
> # Observe that on A there is a single message in the ExpiryQueue, but on B
> there are now 2 messages in the expiry queue.
> Scenario 2:
> # Bring up two artemis instances with broker-connection mirroring between
> each other. (A and B). On A disable the reaper thread by setting
> `<message-expiry-scan-period>-1</message-expiry-scan-period>`
> # Send a message to A with a short TTL
> # Wait for the TTL to expire
> # On B the message moves to the ExpiryQueue, on A the message ends
> disappears (no longer in the ExpiryQueue nor the original queue)
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
