On 06/29/2011 11:08 AM, Alan Conway wrote:
Gordon's mode 1 (stick to a consumer only as long as it has unacked messages) doesn't really do this: the only time it allows load to shift is when the queue is idle long enough for the consumer to ack all its pre-fetched messages. But if the queue is idle then we're not under heavy load which is where load balancing is most important.
That's not necessarily true. The queue could have plenty of messages on it and not be idle, but not have messages in a particular group.
Where I think this mode may make load balancing more adaptive is where there is variability in the groups and the processing times of messages.
For example, consumer 1 gets assigned group 1 and processes 1 available message and then accepts it, there are no more messages at present for that group. It then gets messages sent for groups 3, 5 and 6 say. If those groups now get a lot of messages, consumer 1 may become quite busy. If another set of messages for group 1 are published, why not allow them to be processed by a different consumer. Assuming of course that the requirement is simply in-order processing.
I'm certainly not claiming this is perfect. However it seems to me that it is not necessary in all cases to maintain the stickiness for the lifetime of the consumer.
This makes me think there is a case for a 3rd mode of stickiness: non-stick mode. In this mode we randomly chose the next consumer when the current consumer acks. Clearly this will lower the possible msg/sec throughput since you are effectively doing synchronous messaging. However it may still be relevant where the msg/sec throughput is low anyway but the need for fair load balancing is high - e.g. very large messages or messages that require a long time to process. In that case the extra time added by synchronous messages may be negligible, but the automatic distribution of load is very valuable.
Why not just set the prefetch on the consumers to 0 or 1, and have the consumer accept each message as it is processed. That would seem to achieve the same thing and is exactly what you would do to improve the fairness of distribution without message groups (again at the expense of throughput).
--------------------------------------------------------------------- Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:[email protected]
