Hi Jason, just to let you know I am investigating the issue where the broker suddenly stops receiving messages... I can see this too but I haven't yet got to the bottom of why. Apologies for any inconvenience.
Cheers, Rob On 27 March 2013 17:51, Jason Barto <[email protected]> wrote: > Rob, > first let me say thank you so much for your insight and quick responses. > I'm sorry to say that throttling the producers has not completely solved > the problem. I'm back in the office today and have been doing additional > testing and I think I may have detected a memory leak in the code that > handles AMQP 1.0 communications. > > My experience thus far: > I downloaded and installed the latest QPid revision using the git > directions on the website. With the Java QPid up and running I ran my > first test client (AMQP 0-9-1). I sent 60k messages/sec, each with a 1KB > payload, for 180 seconds. Using top I monitored the JVMs memory usage > which held steady at 233MB. The code for this client can be found on > pastebin at http://pastebin.com/fwDmhxG9. It uses the rabbitmq-client JAR > for AMQP 0-9-1. > > Again using 'top' and the latest Java QPid I ran my second test client at > 8k messages/sec, each with 1 KB payload, for 180 seconds. The memory > utilization of the JVM reported by 'top' rose steadily until the test > ended, at which time the JVM was using 1.1GB of memory. The code for this > second test client is at http://pastebin.com/DNTLCj0y and uses v0.23 of > the > qpid-amqp-1-0-client JAR. It may also be worth noting that at about second > 147 the consumer just stopped receiving messages, no exceptions were thrown > on either the client or broker, it simply stopped receiving messages. > > Both tests were throttled to ensure that the consumer would consume in a > timely fashion roughly all messages sent, to avoid any buildup of messages > in the broker's queue. > > I will admit that I have far more experience with the RabbitMQ client > library than the QPid library so my hope is that there is something someone > can spot in my test code that is not being done correctly that will explain > why the JVM is consuming so much memory. Does acknowledging a message with > Receiver.acknowledge not sufficiently inform the broker that the message > has been received and can be forgotten? > > On longer test runs, 500+ seconds, the broker eventually consumed all of > its allocated ram and as you mentioned the GC began taking over trying to > keep things running. > > Sincerely, > Jason > > > On Tue, Mar 26, 2013 at 12:43 PM, Rob Godfrey <[email protected] > >wrote: > > > Hi Jason, > > > > On 26 March 2013 13:26, Jason Barto <[email protected]> wrote: > > > > > Rob, > > > thanks for your quick response - I've consolidated the code into a > single > > > java file and would gladly publish it - frankly being new to AMQP 1.0 > and > > > its client libraries I'd value the feedback. I think I may have > > determined > > > the reason for the slow down - not sure why it didn't occur to me > earlier > > > to be honest. > > > > > > I have 2 threads, a Sender and Receiver. Both are trying to produce / > > > consume as rapidly as possible. Testing on my laptop the Sender is > > > publishing on average 20k messages / sec while the receiver seems to > top > > > out at 15K msg/sec, leading to an inevitable backlog of messages > building > > > up on the broker. When the queue on the broker is holding around 500k > > > messages (after about 90 seconds of testing) the performance of the > > broker > > > drops dramatically until both producer / consumer can only send / > receive > > > about 200 msg/s. > > > > > > > > > > OK - great (sort of) - we can understand the problem - basically the > broker > > is not sufficiently pushing back on the producing client when the queue > > becomes over full. > > > > The broker can be used to enforce flow control when using earlier > versions > > of the protocol (0-8, 0-9, 0-9-1, 0-10) but it's entirely possible that > > this is not yet implemented in the 1-0 codepath (I shall have to go > > check... if it is not I'll attempt to implement it shortly). > > > > In general the broker performance shouldn't degrade until it starts > running > > out of memory, whereupon the garbage collector will start dominating the > > performance. > > > > > > > If I put a limit on the number of messages / sec sent by the producer > the > > > problem goes away, ie the receiver can consume the messages at a rate > to > > > prevent a backlog of messages. I'd be happy to create a JIRA request > if > > > you feel one is necessary; is there perhaps a datastore I could > configure > > > QPid to use that would help the system to better deal with the growing > > > backlog mentioned? I ask as a mitigation technique. I'll be becoming > a > > > sysadmin for an enterprise-level message broker that will be expected > to > > > handle 1.75B messages per day (roughly 20k/s). If for some reason the > > > messages are not being consumed at the same rate they're being > produced - > > > what options do I have to ensure that QPid's performance is unaffected? > > > > > > > > I guess the question is how large do you want to let the queue grow > before > > you have to enforce some sort of restriction on the sender? Adding > > flow-to-disk style capabilities often just makes the problem worse > because > > the consumer starts to become limited by the speed at which messages can > be > > read from disk, which may be lower than it's optimal rate... so you start > > seeing even more queue growth. I've seen people who want to be able to > > handle spikes in queue growth of up to several million messages... and > the > > answer to their problem was just to stuff a lot more RAM into their > machine > > :) > > > > As an aside are you expecting these 20K/s messages to be AMQP 1.0? As I > > mentioned in my last mail, I've not yet done *any* perf tuning on the > > AMQP1.0 code (JMS client or broker)... Though your question yesterday has > > forced me to dig up my old copy of JProfiler and run it on the broker for > > the first time today :-) > > > > Cheers, > > Rob > > > > > > > Sincerely, > > > Jason > > > > > > > > > On Mon, Mar 25, 2013 at 7:08 PM, Rob Godfrey <[email protected] > > > >wrote: > > > > > > > Hi Jason, > > > > > > > > the AMQP 1-0 support in the java broker is somewhat experimental at > > > > this stage (and no perf tuning has been carried out), however it > > > > definitely seems like you've found a bug. Can you share some more > > > > details of your test (perhaps raise a JIRA and attach the test code?) > > > > and I'll try to look into it ASAP - hopefully that way we can get a > > > > fix in for 0.22, > > > > > > > > Thanks, > > > > Rob > > > > > > > > On 25 March 2013 19:20, Jason Barto <[email protected]> wrote: > > > > > Using amqp 0.9.1 I'm getting an average performance of 70k messages > > per > > > > > second throughput during an hour long test. > > > > > > > > > > When using amqp 1.0 I receive similar but smaller throughput > numbers > > > > > however after about 90 seconds of testing this quickly begins to > drop > > > > until > > > > > it gets to 10s or 100s of messages per second and a significant > > backlog > > > > of > > > > > messages begins building. > > > > > > > > > > Can anyone explain this drastic difference in performance? I am > using > > > > v0.20 > > > > > of qpid-j for the brokering. > > > > > > > > > > Sincerely, > > > > > Jason > > > > > > > > --------------------------------------------------------------------- > > > > To unsubscribe, e-mail: [email protected] > > > > For additional commands, e-mail: [email protected] > > > > > > > > > > > > > >
