Is this possibly related to the prefetch buffer? May not be the full
cause but could be playing a part. The 0.5 java client prefetches 5000
messages IIRC, with that reduced to 500 for 0.6.
You could try lowering the buffer size following the info on the lower
half of this page:
http://qpid.apache.org/use-priority-queues.html
Note that reducing the prefetch will lower peak performance, but in
this case it seems you have your own buffers in place which will
already be restricting individual client throughput to below peak
anyway.
Robbie
On 29 Jan 2010, at 15:54, mARK bLOORE <[email protected]> wrote:
Thanks for the reply, Rob.
That doesn't suggest a reason for the behaviour which I saw. I got a
similar thing today: one of the subscribers filled its buffer, and so
stopped taking messages as fast as possible. The message backlog
soared to tens of thousands, even though the other two subscribers'
buffers were not full, so that they could have taken the load. I
increased the first subscriber's buffer, and it it quickly absorbed
the backlog, while the others continued to take messages only as the
publisher added them.
This is a little different from what I saw yesterday, but again
suggests that message get dedicated to a single subscriber upon
publication, rather than being available to all. One thing the two
cases have in common is that the first subscriber is on the same box
as the broker, and the other two are on a different box. They are EC2
instances, so the pipe between them is very wide.
I really can't take the time to try to create a test case. This is a
large script running on realtime data.
On Wed, Jan 27, 2010 at 4:22 PM, Robert Godfrey <[email protected]
> wrote:
Hi Mark,
I can give an outline of how the Java Broker distributes messages
between subscriptions... I'm not familiar with anything the Python
client may do...
In general, when there is no backlog in the queue, the Java Broker
will round-robin between subscriptions which have available credit.
If a subscription runs out of credit then it is marked as suspended.
When such a subscription gets credit again, or when a new
subscription
is added to the queue then the queue will attempt to send as many
messages as it can to this new (or unsuspended) subscription. (To be
absolutely accurate it attempts to send up to 10 messages, then
yields
- but schedules another attempt to send 10 messages, and so on).
I'm not sure if the above totally explains what you are seeing -
certainly I don't see why the subscriptions should fail to keep up
with the publisher (until you kill the first subscriber) in the way
you describe.
If you could provide a simple example that shows the same behaviour,
that would be fantastic... In the meantime I will have a more
thorough dive into the broker code to see if I can spot anything
obvious.
Hope this helps,
Rob
2010/1/27 mARK bLOORE <[email protected]>:
I am using the java broker 0.5 and the python client. I have a
queue
with one publisher and (usually) three subscribers.
The publisher sends messages at a fairly constant rate.
The subscribers take messages in a single thread at an unlimited
rate,
put them in a buffer, and ack the messages immediately. If the
buffer
fills then they block before the ack.
Normally all the subscribers get messages at about the same rate,
and
the queue's message count is mostly zero. If a subscriber starts to
block then the message count may rise, and when it gets to the
tens of
thousands I reduce the publication rate. But an odd situation
appears: The blocked subscriber takes messages into its buffer as
fast as it takes them out, but the other two subscribers get
messages
at only one third of the rate that the publisher is adding them.
That
first subscriber is running many times faster.
If I add a fourth subscriber it gets messages at one quarter the
publication rate, and the other two start getting messages at that
rate too. If I kill the fourth subscriber rates return to what they
were.
If I kill the first subscriber then the other two start taking
messages very fast, and the backlog in the queue quickly disappears.
It seems as if the broker has earmarked the backlogged messages for
that one subscriber, and won't deliver them to any other unless that
one goes away. Could that be the case? Note that a subscriber may
have at most one unacked message.
I'm afraid I can't abstract any reasonable amount of code to
display.
I'm not sure that would help anyway.
--
mARK bLOORE <[email protected]>
---
------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
--
mARK bLOORE <[email protected]>
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
---------------------------------------------------------------------
Apache Qpid - AMQP Messaging Implementation
Project: http://qpid.apache.org
Use/Interact: mailto:[email protected]
