So if you want overlap - where 1 message may go to several groups, but each group processes it once - then multiple queues is the best solution I think. So either use a consumer to do the routing, or a broker interceptor.
BTW yeah am still based in Islington :) On 11/21/06, taxingmonk <[EMAIL PROTECTED]> wrote:
Hi James, To quote: "If you just want load balancing you could use 1 queue with a selector for each group." Yes, that'd be fine, the problem is that I want an overlap. Having done further testing I realise this is a flawed notion in and of itself, I'm inadvertently expecting a queue to duplicate messages for the overlapping selectors. Another way to describe the intending behaviour would be to have a set of consumers on a topic where groups of consumers could be made where one consumer in the group and only one receives a copy of the message for that group. Thus I can build a replicated distributed queue. The net result is actually identical to using multiple queues and composite destinations. I think your suggestion: "So send a message to an input queue then have a farm of routers consuming them and routing them to the correct queue for the right group to process it." is the closest, when I was first thinking up a solution for this using composite destinations I was hoping to hide the building of the composite destination from the sender. This seems like a nice way to do it. Thanks James, Max p.s. Are you in "Norf Lundun" ATM? James.Strachan wrote: > > On 11/21/06, taxingmonk <[EMAIL PROTECTED]> wrote: >> >> Hi Guys, >> >> I've got a use case where I want to load balance a given queue but >> paritioned accross selectors. I am convinced that somewhere between >> composite destinations and message groups I can achieve what I'm after. >> >> The use case is that I want to be able to send one message which is load >> balanced to a series of receievers which I group together somehow but at >> the >> same time I have multiple groups also receiving the same message. >> >> So In my example I have message M1 and I have Consumers (C1,C2,C3) Group >> 1 >> and (C4,C5,C6) Group 2 and (C7,C8,C9) Group 3. I want message M1 to be >> receieved by one member of Group 1, Group 2 and Group 3. > > You could just use 3 queues for each group - then you can browse what > messages are available for each group & monitor the queue depth etc. > Thats the main benefit over using selectors. > > Note that virtual destinations & composite destinations result in the > message going to all of the destinations - in this use case you want > it to only go to one of them. So selectors or a custom router is the > best solution. > > >> I thought that might be achieveable by using composite destinations but I >> didn't want to manage a seperate queue for each group so that I could >> drop a >> message on the bus and have to arrive at all of them without specifying >> send >> (a.group1, a.group2, a.group3), etc, etc. >> >> Furthermore, what would be even better is if this could derived from the >> message itself. If we consider a numberline where I can specify a range >> from say 1 to 10 and group 1 is interested in 1-3 and group 2 is >> interested >> 2-6 and group 3 is interested in 5-10. From there I would like to be able >> to >> say this message is for range 1 >> >> It's almost like I want layer selector behaviour accross the top of the >> message groups while load balancing :) > > Just to be clear - Message Groups are basically all about sticky load > balancing. All queues - whether using selectors or not - implement > load balancing. > > If you just want load balancing you could use 1 queue with a selector > for each group. Using message groups on top of this should give you > sticky load balancing if you need it. > > >> Ultimatly this allows me to spawn >> more consumers if I start to realise that the processes that are handling >> a >> particular range have a large data density, the key thing is that >> multiple >> groups can return responses to the one message, ranging specifying >> partitions of data, which may or may not overlap. This I fear may be a >> bridge too far. >> >> It should be noted that none of these messages is transactionally >> important, >> nor do I need the total guarentee that two or more consumers in the same >> group receieve the same message, although that shouldn't be the common >> case. > > You could write a custom Broker interceptor to implement whatever > routing policy you wish - using the message headers / body to route to > a different queue based on some rules... > http://incubator.apache.org/activemq/interceptors.html > > Though its a bit more MOM-like to do these kinds of routing policies > using a consumer which does the routing for you - which can then be as > simple or complex as you wish. So send a message to an input queue > then have a farm of routers consuming them and routing them to the > correct queue for the right group to process it. > > Am not sure if you need sticky load balancing from your mail - so > maybe just 1 queue with selectors would do the trick? > -- > > James > ------- > http://radio.weblogs.com/0112098/ > > -- View this message in context: http://www.nabble.com/Message-Group-Overlaps-and-Load-Balancing-tf2678713.html#a7474172 Sent from the ActiveMQ - User mailing list archive at Nabble.com.
-- James ------- http://radio.weblogs.com/0112098/
