On 5 January 2012 20:08, Vijay Devadhar <[email protected]> wrote:
> Great discussion folks; Original question Praveen posted was indeed > motivated by > A discussion we were having about "how to handle unexpected loads". > > In the past, I have come across producers who went awry and enqueued > millions > of messages. I guess we have to start throttling the producer that does > something > like this to protect the broker from running out of memory; > > Earlier in the week I updated the Qpid Broker to implement flow control (against the JMS client) for AMQP 0-10 in a similar manner to the existing flow control in the 0-9-1 codepath (essentially the broker revokes credit from the clients so that they may no longer send). At some point soon I shall also try to implement a mechanism identical to that in use on the C++ broker (where the broker delays acknowledging the completion of the transfer commands). Cheers,, Rob > I haven't read docs about flow control etc., so won't ask questions yet. > :-) > > -----Original Message----- > From: Rob Godfrey [mailto:[email protected]] > Sent: Thursday, January 05, 2012 10:58 AM > To: [email protected] > Subject: Re: Qpid Java Broker with persistance supports only finite number > of messages? > > On 5 January 2012 19:30, Fraser Adams <[email protected]> > wrote: > > > Just to jump in on this thread. > > > > Re " > > > > > > but my opinion > > is that if you have "millions of messages" then a Message Broker is the > > wrong solution to your problem - you want a Database. > > > > " > > I can't say I agree with Rob's assertion here!! > > > > Well maybe that's a reasonable comment if the *intention* is to have > > millions of messages hanging around, but what if it's due to an > unfortunate > > circumstance...... > > > > > Indeed - that is what I meant :-) Planning for failure scenarios should > always be carefully considered... As Robbie mentioned we have had people > build up millions of messages on persistent queues... and if you use the > 64-bit JVM and put large amounts of RAM in your box, then the physical > limitations of maximum queue size can be made acceptably large... However I > would always caution against designing a system where you expect to hold > this number of *persistent* messages as part of your regular process - > especially using the Java Broker as it is not designed for this sort of use > case. > > The case you describe is somewhat different in that you are looking at > transient messages (where one would expect the rate of > production/consumption to be higher), and you want the broker to be capable > of buffering for N minutes while temporary network outages occur. Flowing > to disk may make sense here as long as the rate at which messages can flow > through the disk is higher than the rate at which they are entering the > broker (and actually high enough so that the backlog can quickly be > reduced). This is where the flow-to-disk work that I hope to be starting > soon on the Java Broker would play. > > -- Rob > > --------------------------------------------------------------------- > Apache Qpid - AMQP Messaging Implementation > Project: http://qpid.apache.org > Use/Interact: mailto:[email protected] > >
