James Strachan wrote:
On 2 August 2010 18:49, Don Santillan <donzym...@gmail.com> wrote:
Hello,
I am trying to create an application that would use a message queue that is
consumed by multiple consumers running concurrently but would still maintain
the order of message processing by a certain ID.
For example, I have a set of messages added to a queue:
ID1-M1
ID1-M2
ID2-M1
ID3-M1
ID2-M2
ID3-M2
I need the Messages coming from ID1 get consumed by Consumer1, ID2 to
Consumer2, and ID3 to Consumer3, that maintains order such as Consumer1 will
process ID1-M1 and then ID1-M2, Consumer 2 process ID2-M1 and then ID2-M2,
and Consumer3 process ID3-M1 and then ID3-M2.
I also like to have the Consumers come from a ConsumerPool so that if a
consumer is done processing with the messages, it will go back to the pool
and become available for other Messages owned by another ID. I also like to
have the pool a behavior that would create a new consumer in the pool if a
consumer dies so that a fix number of consumer is available from the pool.
I saw in the documentation that the Message Group, I think, fits into the
requirement perfectly. Process messages asynchronous, maintain message
order, and failover feature.
Unfortunately, I was not able to understand completely how to do it. I
understand that the producer needs to set a property "JMSXGroupID" to
identify which group the message will get associated and consumed by a
single consumer. What I don't understand is how the broker will select which
consumer to pass the message to? Even more, where will the consumers come
from? Do I need to create, for example, 20 consumers and subscribe them all
to the broker? What if the consumer one by one dies, can the broker
automatically recreate all 20 consumers?
It would be greatly appreciated if anybody can write a simple code that
would show how consumers get subscribed to the broker and have an
explanation that my requirement would really fit into Message Group. If this
is not possible through Message Group, any suggestions on how to implement
this?
The broker chooses a consumer on demand to fill a hash bucket. As
consumers die new consumers get chosen. So all you need to do is add
message group IDs to your message (as specific as possible to use the
biggest possible number of hash buckets) and create a number of
consumers. There's no need to manually assign consumers to individual
message group IDs or anything like that.
Sometimes prefetch can be set high which can reduce the amount of
concurrent consumers used; if you are only sending a small number of
messages you might find that only a small number of consumers are
actually used.
http://activemq.apache.org/i-do-not-receive-messages-in-my-second-consumer.html
http://activemq.apache.org/what-is-the-prefetch-limit-for.html
so you might want to set the prefetch to say 1 if you've only a
relatively small number of messages to ensure a good spread across
your consumers.
Thanks for replying James, now I got the point. I understand now that
all consumers that I created that is associated to the queue will get
passed depending on the first group id it has received.
More questions though. How does a consumer die? Can you site an example?
If I have 5 consumers created and associated to a queue, and if 5 of
them dies, does it mean the broker won't get any consumer to process the
pending messages anymore?
Another scenario, if the queue has 10 messages that belongs to different
groups, and there are only 5 consumers associated to the queue, what
will happen to the remaining 5 messages in the queue?
