On Fri, Feb 26, 2010 at 3:29 PM, jerdavis <[email protected]> wrote: > > > In short I am using a persistent queue as a way to stretch scalability of a > database: > > My initial use case is this: > > I would like to queue 1million persistent&durable messages per minute. > This 1 million represents 1 million separate conversations(Queues?) at 1 msg > per minute (or 16.6KTransactions per second)
I million queues (around 200k per broker) is likely not going to be efficient no matter which vendor you use. All though each queue would occupy very little memory, the housekeeping and look up will be a bit expensive. Also since you want to use durable messaging, each queue will have it's own file. Since you are planning an exclusive consumer per queue, you are talking about a large number of connections. On linux the default file handle limit per process is 1024, all though this can be changed, I don't know what the maximum is. Either way your file handles will be shared by the socket and persistent storage. So I doubt that a particular instance could handle such a large no of queues. (Each socket, file handle structure will also take a little bit of memory and they all add up). I understand that the conversations are separate from each other. But finding a way to represent them with a lesser number of queues would be better. I would assume you plan to route messages based on some sort of a conversation ID? So how about you add a message header called conversation ID to each message? And then a single queue could take messages intended for more than one conversation (say X conversations). Then the consumer (Worker in your terminology) could take each message and distribute it among X workers who does the processing as you described. Also you could add a sequence number header for each message in a conversation. So in case redelivery happens and messages gets out of order you consumer has a reasonable way of handling out-of-order messages. You could argue that this solution is offloading part of the problem from the broker to the consumer, but it allows you to relieve the strain on the broker. Also more than one conversation on a queue allows you to balance the load more easily, when you have busy conversations and not so busy conversations. > ... My plan is to Federate 5 > Brokers to get ~3KTPS per broker which is reasonable for most disks. (I > assume I can also Federate 5 Clusters of 2 brokers for redundant copies of > data?) OK I didn't quite understand why you need to federate (or connect) the 5 broker entities? As I understood your conversations are distinct and I don't see what federation buys you. Why not use 5 distinct broker instances with the load equally spread across. You can certainly replicate each broker to have a standby. So you have 5 pairs of distinct clusters. > A Worker (that is an exclusive consumer of a given Queue) will take each > message in a conversation and roll it up in to an hourly product that will > be persisted to a DB. I will use 2PC to simultaneously commit the hourly > product to a DB and delete the hours worth of messages out of the Queue. > Concurrently with every new message arriving at the worker, I will be > updating the product (in memory only) and make it available via > caching(memcached/etc). > > In this way processing 1 Million Msgs per minute is only writing to a DB > 1M/3600~~280TPS (assuming perfectly staggered), also real time products are > available, and there is always a persistent copy in case of failure. > > So does this sort of thing seem reasonable? I think if you find a reasonable way of using a smaller number of queues to handle your 1 million conversations you could certainly do it. The rates you quote are certainly doable by the Qpid product. > > > -- > View this message in context: > http://n2.nabble.com/Total-Ordering-Guarantee-tp4636556p4641639.html > Sent from the Apache Qpid users mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > Apache Qpid - AMQP Messaging Implementation > Project: http://qpid.apache.org > Use/Interact: mailto:[email protected] > > -- Regards, Rajith Attapattu Red Hat http://rajith.2rlabs.com/ --------------------------------------------------------------------- Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:[email protected]
