On 06/24/2011 11:07 AM, Ken Giusti wrote:
Message Groups
Status
Draft
Summary
This document describes a new feature that would allow a message producer to
enforce the order in which the data from a set of related messages are
processed by consumers.
Problem
While the broker currently employs an strict FIFO queuing model, it does not
guarantee that messages will be processed in that order when there are multiple
consumers subscribed to a queue. Therefore, it is impossible for a producer to
enforce a strict ordering to the processing of messages, even though it may be
required by the application.
For example, assume we have a shopping application that manages items in a virtual shopping cart. A user may add an
item to their shopping cart, then change their mind and remove it. If the application sends an "add" message
to the broker, immediately followed by a "remove" message, they will be queued in the proper order -
"add", then "remove".
However, if there are multiple consumers, it is possible that once a consumer acquires the "add" message, a
different consumer may acquire the "remove" message. This allows both messages to be processed in parallel,
which could result in the "remove" operation being performed before the "add" operation.
Solution
This problem can be solved by allowing a producer to mark a group of messages
as being related, and having the broker enforce strict ordering of consumption
of messages belonging to that group.
Specifically, for any given group of queued messages, the broker allows only
the first message in the group be available for consumption by a subscriber.
The broker blocks the remaining messages belonging to that group from
consumption by any subscriber. Once the broker has completed the transfered of
responsibility for that first message to a consumer, it then allows the next
message in the group to be available for consumption while all other messages
in that group remain blocked.
In order to guarantee messages are not processed in parallel, the application
has to ensure that it has completely processed the data in a received message
before accepting that message, as described in Section 2.6.2. Transfer of
Responsibility, of the AMQP-0.10 specification.
Note well that distinct message groups would not block each other. For example, assume a queue contains messages from two different message groups - say group
"A" and group "B" - and they are enqueued such that "A"'s messages are in front of "B". If the first message of group
"A" is in the process of being consumed, then the remaining "A" messages are blocked, but the first message of the "B" group is available
for consumption - even though it is "behind" group "A" in the queue.
This feature could be implemented by having the message producer set a group
identifier in the application headers of a message. Messages belonging to the
same group would have the same identifier value. The destination queue would be
configured with the key name of the application header used for group
identification. When a message is delivered to the queue, the broker would
check the application headers for the group identification key. It would then
classify which group the message belonged to based on the value of the key. The
queue subscriber logic on the broker would need to be modified to enforce the
rules for accessing grouped messages (as described above).
Rationale
The solution described above allows the message producer to enforce a strict
order to the processing of its messages by the consumers. The broker can
support such a feature with a minimum of configuration effort (merely
configuring the header key used per queue).
* Goal: allow dynamic values for the group identifiers (no
preconfiguration of identifiers necessary)
* Goal: the number of message groups "in flight" should only be limited by
the available resources (no need to configure a hard limit).
* Goal: the individual messages from a single message group may be
processed by different consumers, as long as the strict order is observed.
* Goal: manageability: visibility into the message groups currently on a
given queue
* Goal: manageability: purge, move, reroute messages at the group level.
* Goal: "sticky" subscribers - (optional) ensure that all messages from a
given group are consumed by the same client.
Implementation Notes
* Queues: support configuration of the group identifier header key.
* Messages: provide access to group identifier.
* Queues: identify head message of next available message group.
* Queues: block the trailing messages in a given message group from being
consumed.
* Consumers: track the message group of the currently acquired message(s).
Consequences
* Development: No changes to the development process.
* Release: No changes to the release process.
* Documentation: User documentation will be needed to explain the feature,
and the steps to configure and manage the new feature.
* Configuration: Yes: per-queue group identifier header key is
configurable. Queue state with regard to in-flight message groups needs to be
visible. Additional methods to purge/move/reroute a message group.
* Compatibility: Unlikely - new feature that would need to be enabled
manually. Applications wishing to use the feature would need to implement a
message grouping policy, and ensure the processing of received message data is
completed before signaling acceptance of the message.
I think we need to look at refactoring the queue so that functionality like this
can be added as a set of plug-in modules rather than the existing
swiss-army-knife approach. I don't think the existing approach will scale from a
maintenance point of view as we add more optional features to the queue.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
