Hi, I have read a lot of documentation over the past few days trying to answer what I thought was a very standard question: I have a farm of consumers (50-100) that needs to process messages from multiple sources (10-20). Messages from some of these sources will be coming in bursts, and I need to make sure that this won't affect messages from other sources. In other words I need to implement some sort of flow control logic.
I can imagine two solutions to these problem: Solution 1. Multiple input queues are all routed into one output queue. All consumers subscribe to this output queue. This output queue will use producer flow control to make sure that it remains small. There are several problems with this solution: * Flow control only kicks in when the output queue is full, so this queue would have to be really small so as not to introduce a lot of latency for messages from other sources. This would probably introduce a lot of contention. * This flow control is completely symmetric for all producers, so I can't treat messages from one producer as more important than messages from another producer. * Finally it looks like flow control in ActiveMQ is only used with non-durable queues. If that's indeed true then I can't use this approach. I do know about time-based throttling, but it won't work for me since it is completely agnostic of the actual message load across all queues. Solution 2. All consumers subscribe to all queues (using wildcard mechanism). Each queue in ActiveMQ has a "pending queue" (pagedInMessages). Controlling the maximum size of this pending queue I can assign relative priorities for different queues. This approach seems to have very nice properties, however I am concerned about performance implications of using small maxPageSize, as well as using having each consumer subscribe to so many queues. Has anyone done anything similar before? Any advice will be greatly appreciated. Thanks! -sergey -- View this message in context: http://www.nabble.com/Multiple-consumers-serving-multiple-queues-tp16765830s2354p16765830.html Sent from the ActiveMQ - User mailing list archive at Nabble.com.
