this is a classic slow consumer problem. Have a peek at http://activemq.apache.org/slow-consumer-handling.html
You can use a single topic with non durable consumers and a low prefetch value. A slow consumer will then be one that has not consumed its prefetch before the next dispatch. You can determine what happens next, some or all messages are kept pending for that consumer in memory or paged to disk (if a memory limit is reached) or some are deleted using an eviction strategy. It is all configurable and plugable. As in, you can modify the existing strategies and implement your own if needs be. When the slow consumer accelerated again, it will get all the pending messages and resume normal operation. On 5 July 2011 14:15, detla <jonathan.schm...@detla.de> wrote: > Hi all, > > we currently consider the use of JMS implemented by ActiveMQ in the > following scenario: > > A producer will produce roughly 300 to 400 rather large (20-30 MB) data > objects per minute (in total ~17000 objects). Seven different > processor-objects process the data objects in parallel. The setting is > purely local, so the embedded message-broker is sufficient. Durable messages > are not required. Naively you would assume, that the best setup will be one > non-durable topic, where the seven processors register as subscribers and > the producer sending the data objects to that topic. Everything within this > setting is purely java. > > Our problem is the memory consumption of the setup. There is a limit of > roughly 100 of the data objects, which can be kept in memory at a given > time. Typically, the seven processors are roughly equally fast but there may > be situations, where one is lagging behind. In this case, the remaining fast > processors should proceed without decreasing speed, while for the lagging > processor, it is acceptable to complete its task with a delay. > > Until now we have identified several options how to deal with this problem > and neither makes us really happy: > 1) Intelligent data objects: The data objects get some intelligence and time > to live. Once the time to live is up, the objects starts persisting the > larger data chunks contained, thus freeing memory. Of course this solution > requires some rematerialization, once the object is needed again by the > lagging processor. This of course requires some tedious implementation (plus > the question as to how the data object gets notified, that it should persist > itself without interfering with gc). > > 2) (Ab)Use of JMS techniques: Try to achieve the following: Via priority > assure, that newest messages (and objects) are consumed first. If an old > object gets too old (or better: queue gets to long), the object is subject > to persisting via the JMS mechanisms. Question: is this a violation of the > design principles? Is it technically even possible? > > 3) Re-Aquiring data objects: Use the ActiveMQ mechanism to drop old > messages, via a second mechanism (simple java communication) re-request the > data-objects that were lost afterwards. This of course minimizes our chances > to ever make use of the distributed mechanisms of JMS and seems to involve a > lot of manual work. > > 4) Central observer: Use a central observer to notify about consumed > messages and filled queues. Controll persisting/dropping/redelivery of > messages via the central observer. Again: lots of manual work. > > > All in all our opinion is, that the requirements are somewhat standard and > we have the feeling, we might be missing some plain nice and easy way to > make use of standard-JMS-techniques (or ActiveMQ enhancements to the > standard). > > Has someone some idea where to start looking or some input to the > possibilities we already identified? > > Kind regards, > Jonathan > > > -- > View this message in context: > http://activemq.2283324.n4.nabble.com/Question-concerning-JMS-ActiveMQ-Design-tp3645904p3645904.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com. > -- http://fusesource.com http://blog.garytully.com
