On 07/28/2010 06:28 PM, Brian Crowell wrote:
On Wed, Jul 28, 2010 at 12:01 PM, Gordon Sim<[email protected]> wrote:
If you have four connections pumping in messages to a queue as fast as they
can and only one pulling them out, then the queue will indeed backup.
The best part is that only one publisher is really pushing as fast as
it can at any one time. One publisher is enough to overwhelm the
subscriber.
What are your settings for the subscriber? Try batching (or not
requiring) acks. I wouldn't expect a single producer to be
overwhelmingly faster than a subscriber.
Messages pile up in the queue
and I run out of memory fast. I suppose for that, I'll have to devote
one queue to each publisher.
Or perhaps build in some feedback/throttling into the application(?).
Funny thing: I'm actually using Qpid for sending around stock ticks,
which they say they designed AMQP to handle. So I can't slow down the
messages that come in. If I can't send them into Qpid fast enough,
they pile up at the publisher. If I can't read them out of Qpid fast
enough, they pile up in Qpid.
It sounds like I'm just going to have to create a more complicated
setup than I hoped for, trying to divide all the messages up into as
many queues as I can. I originally wanted all messages to go into a
few exchanges and let the subscriber bind to whichever stock it wanted
to know about.
So the subscriber isn't actually interested in all the messages coming
in? Again that should make a difference as the subscriber rate doesn't
need to match the total publisher rate.
So:
* More sessions at the publisher so it can feed more ticks.
Sessions won't really improve things; the unit of parallelism on the
broker is at the connection level. Each connection is processed by a
single thread/core at a time.
* As many queues as I can manage at the subscriber.
* Qpid on Linux with RT kernel.
--Brian
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
