On 29 January 2010 16:54, mARK bLOORE <[email protected]> wrote: > Thanks for the reply, Rob. > > That doesn't suggest a reason for the behaviour which I saw. I got a > similar thing today: one of the subscribers filled its buffer, and so > stopped taking messages as fast as possible. The message backlog > soared to tens of thousands, even though the other two subscribers' > buffers were not full, so that they could have taken the load. I > increased the first subscriber's buffer, and it it quickly absorbed > the backlog, while the others continued to take messages only as the > publisher added them. > > This is a little different from what I saw yesterday, but again > suggests that message get dedicated to a single subscriber upon > publication, rather than being available to all. One thing the two > cases have in common is that the first subscriber is on the same box > as the broker, and the other two are on a different box. They are EC2 > instances, so the pipe between them is very wide. > > I really can't take the time to try to create a test case. This is a > large script running on realtime data. > >
OK, can you possibly raise a JIRA for this, and document which JVM / JVM version you are using... and some information on what sort of hardware and the approximate message rates. I'm also a bit snowed under at the moment, so I may not be able to look into this immediately... Thanks, Rob > > On Wed, Jan 27, 2010 at 4:22 PM, Robert Godfrey <[email protected]> > wrote: >> Hi Mark, >> >> I can give an outline of how the Java Broker distributes messages >> between subscriptions... I'm not familiar with anything the Python >> client may do... >> >> In general, when there is no backlog in the queue, the Java Broker >> will round-robin between subscriptions which have available credit. >> >> If a subscription runs out of credit then it is marked as suspended. >> When such a subscription gets credit again, or when a new subscription >> is added to the queue then the queue will attempt to send as many >> messages as it can to this new (or unsuspended) subscription. (To be >> absolutely accurate it attempts to send up to 10 messages, then yields >> - but schedules another attempt to send 10 messages, and so on). >> >> I'm not sure if the above totally explains what you are seeing - >> certainly I don't see why the subscriptions should fail to keep up >> with the publisher (until you kill the first subscriber) in the way >> you describe. >> >> If you could provide a simple example that shows the same behaviour, >> that would be fantastic... In the meantime I will have a more >> thorough dive into the broker code to see if I can spot anything >> obvious. >> >> Hope this helps, >> Rob >> >> 2010/1/27 mARK bLOORE <[email protected]>: >>> I am using the java broker 0.5 and the python client. I have a queue >>> with one publisher and (usually) three subscribers. >>> The publisher sends messages at a fairly constant rate. >>> The subscribers take messages in a single thread at an unlimited rate, >>> put them in a buffer, and ack the messages immediately. If the buffer >>> fills then they block before the ack. >>> >>> Normally all the subscribers get messages at about the same rate, and >>> the queue's message count is mostly zero. If a subscriber starts to >>> block then the message count may rise, and when it gets to the tens of >>> thousands I reduce the publication rate. But an odd situation >>> appears: The blocked subscriber takes messages into its buffer as >>> fast as it takes them out, but the other two subscribers get messages >>> at only one third of the rate that the publisher is adding them. That >>> first subscriber is running many times faster. >>> >>> If I add a fourth subscriber it gets messages at one quarter the >>> publication rate, and the other two start getting messages at that >>> rate too. If I kill the fourth subscriber rates return to what they >>> were. >>> >>> If I kill the first subscriber then the other two start taking >>> messages very fast, and the backlog in the queue quickly disappears. >>> >>> It seems as if the broker has earmarked the backlogged messages for >>> that one subscriber, and won't deliver them to any other unless that >>> one goes away. Could that be the case? Note that a subscriber may >>> have at most one unacked message. >>> >>> I'm afraid I can't abstract any reasonable amount of code to display. >>> I'm not sure that would help anyway. >>> >> >> >> >>> -- >>> mARK bLOORE <[email protected]> >>> >>> --------------------------------------------------------------------- >>> Apache Qpid - AMQP Messaging Implementation >>> Project: http://qpid.apache.org >>> Use/Interact: mailto:[email protected] >>> >>> >> >> --------------------------------------------------------------------- >> Apache Qpid - AMQP Messaging Implementation >> Project: http://qpid.apache.org >> Use/Interact: mailto:[email protected] >> >> > > > > -- > mARK bLOORE <[email protected]> > > --------------------------------------------------------------------- > Apache Qpid - AMQP Messaging Implementation > Project: http://qpid.apache.org > Use/Interact: mailto:[email protected] > > --------------------------------------------------------------------- Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:[email protected]
