Hi all,
As discussed in team-wide review and mail threads [1] and [2], The
finalized queue purge flow is as follows.
Once a queue purge is requested from UI console (queue->browse page), the
AMQQueue.purge event is triggered in which the request is validated based
on the user's permissions (A new permission "queue/purgeQueue" was added
for this purpose).
Given that the user has permission;
1. The timestamp of the event is updated as the lastPurgedTimestamp for
the purged queue in QueueDeliveryWorker. This is used to compare with each
messages' arrival Time so that messages that came before the purge event
can be skipped at ultimate delivery stage. (This will handle the edge case
of old messages in delivery phase at the time of purge.)
2. All messages read into memory destined for the purged queue are
cleared.
3. Other MB nodes are notified of the purge event including the time it
happened. (This will clear in memory messages of other nodes.)
4. metadata and content of all messages of the queue are removed from
store. (message content is deleted in async mode to avoid blocking. There
are no node-specific entities in the store. So this is done through a
single node.)
5. Message counter for the purged queue is reset to 0.
6. Any messages in the DLC queue addressed to the purged queue are also
deleted.
*Current concerns : *
1. Given 10 concurrent publishers and subscribers on a single queue, if the
queue is purged in middle of pub/sub, delivery threads of Mb start
accumulating. Due to this, new messages sent to same queue will not be
received while at the same session. I have opened an L2 issue for this [3]
and checking on it.
*Relevant pull requests :*
https://github.com/wso2-dev/andes/pull/144
https://github.com/wso2-dev/carbon-business-messaging/pull/49
*Jira ticket : *
https://wso2.org/jira/browse/MB-143
[1] : [DEV] Way to optimize database calls during queue purge - MB 3.0.0
[2] : [DEV] Concern when purging during active subscriptions - MB 3.0.0
[3] : https://wso2.org/jira/browse/MB-853
On Thu, Oct 23, 2014 at 11:52 PM, Hasitha Amal De Silva <[email protected]>
wrote:
> *Rationale : *
>
> According to AMQP spec 0_9_1 : Once purged, We need to remove all messages
> which are not awaiting acknowledgement from the given queue. [1]
>
> *Requirements in terms of MB 3.0.0 architecture : *
>
> Once a queue purge is requested from UI console (queue->browse page), all
> messages addressed to that queue :
>
> 1. should be removed from the store (JDBC / Cassandra)
> 2. that are still queued for delivery in memory should be cleared
>
> Any message counters for the queue in store should be reset to 0.
>
> *Messages published to that queue immediately after a purge should still
> be received by the subscriber later.
>
> *Development Approach :*
>
> Refer sequence diagram [2] for implementation / flow. This is subject to
> change in line with [3] and [4] concerns.
>
> *Test Scenarios : (Not yet automated)*
>
> 1. Publish N messages -> purge queue -> subscribe and verify that no
> messages are received.
> 2. Publish N messages -> subscribe -> purge queue before receiving all
> messages -> verify that no messages are delivered / verify that only the
> immediately enqueued messages are delivered. (This test will need to access
> in-memory message collection sizes)
> 3. Publish N messages -> purge queue -> subscribe -> publish N
> messages -> verify that the second message batch is received.
> 4. Perform above tests with parallel publishers/subscribers
>
> *Current Concerns : *
>
> 1. Message content can be lost for some messages when purging in the
> middle of a pub/sub situation [3].
> 2. There is a performance hit in reading all message IDs from the
> metadata table prior to clearing. But given our current model, this is
> necessary to delete the message content relevant to a specific queue [4].
>
>
> Appreciate your feedback/suggestions on anything I have missed out or any
> improvements.
>
> [1] : http://www.rabbitmq.com/amqp-0-9-1-quickref.html
> [2] :
> https://drive.google.com/file/d/0B1soNraLsHdmdUdWSUVzdml5Mzg/view?usp=sharing
> <https://drive.google.com/a/wso2.com/file/d/0B1XfuoFd-MhENk02ZDNvcXpPUHM/view?usp=sharing>
> [3] : [DEV] Concern when purging during active subscriptions - MB 3.0.0
> [4] : [DEV] Way to optimize database calls during queue purge - MB 3.0.0
>
> --
> Cheers,
> Hasitha Amal De Silva
> Software Engineer
> Mobile : 0772037426
> Blog : http://devnutshell.tumblr.com/
> WSO2 Inc.: http://wso2.com ( lean.enterprise.middleware. )
>
--
Cheers,
Hasitha Amal De Silva
Software Engineer
Mobile : 0772037426
Blog : http://devnutshell.tumblr.com/
WSO2 Inc.: http://wso2.com ( lean.enterprise.middleware. )
_______________________________________________
Architecture mailing list
[email protected]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
