Sorry, didn't read entire mail last time...
On 07/20/2011 01:39 PM, Ken Giusti wrote:
[snip]
In this case, it probably would be sufficient to maintain a counter
in the group's state. I'm assuming that we'll need a group lookup on
message arrival to find that state (or create it on the first
received message). Increment the counter when a message in that
group is enqueued, decrement when the message is dequeued. If
decremented to zero, delete the group. Performance impact of such an
approach is TBD, of course - probably need to pool the state objects
to limit memory trashing, etc....
That doesn't sound right. The first N messages of a group could be
dequeued
before the N+1 message is enqueued, which would put your counter to 0
before the
entire group is processed. I think we need an explicit end of group
marker on
the last message in the group to allow the user to close a group in a
sceanrio
with lots of dynamic group ids. You could ignore this in cases where
there is a
small fixed set of groups that are used over a long time.
Ah, yes - sorry. The lifetime of the group's state depends on the type of "Policy" that
is being used (see below). For the "Sequenced Consumers" policy - which is what I was
thinking of in my last reply - the broker doesn't need to maintain state across all messages of the
group. With that policy, the state can be dropped once there are no more messages for that group
present in the broker.
Again I don't get that. You're putting group boundaries at unpredictable
arbitrary points depending on relative speed of producer/consumer. The producer
may send the first N messages of a group containing N+M messages, then the
consumer consumes the first N messages. Now the queue is empty so the next M
messages are considered a new group, where N is impossibly to predict in
advance. So you randomly cut up the groups.
For the other proposed policy - Exclusive Consumer - then the group state would
be associated with a particular Consumer instance, and remain present as long
as the Consumer exists. So in that case, yes - there is the potential that an
unbounded number of group states could exist without some kind of reclaim
strategy. And end-of-group marker could be used, but that wouldn't prevent a
DOS by a misbehaving app that creates endless groups w/o setting the end
marker. Perhaps a hard max-groups-per-Consumer limit, or a TTL for idle groups?
I'm not so worried about that, it's no worse than the fact that a client can
create a huge number of of sessions or consumers etc. Solving that family of DOS
attacks is a separate issue I think.
[snip]
This QIP defines two message group policies. Additional policies
may
be
defined in the future.
Policy 1: Exclusive Consumer
With this policy, the broker would guarantee that all messages in
a
group
would be delivered to the same client.
This policy would be configured on a per-queue basis.
We could configure policy on a per-group basis with the value of the
"qpid.group" property. E.g. qpid.group=exclusive.
That's more flexible and puts control of the policy in the hands of
the
producer, rather than requiring an admin convention that associates a
queue name
with a policy.
I'm not sure I completely understand: are you proposing the producer sets both
the group identifier and a policy type identifier on a per-message basis?
As above I think we need an explicit "end group" message, so it would be
symmetric to have an explicit "start group" message also that specifies the
policy. This could be done with multiple properties (but that makes configuring
the property name more complicated) or with a configuration string e.g.
group-id=mygroup{policy=xxx} or somesuch. I'm not sure which approach I prefer
both are a little icky.
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
