Rob, will do. I'd also like to try taking a look at the source and see if I can identify the cause of the seeming memory leak. Although my java is so rusty I doubt I'll find anything.
Regarding your question about the protocol planned for the 20k messages, we have ever changing requirements so while I'm happy to build my solution on 0.9.1 I'd like to future proof by helping qpid develop to support the same incredible throughput in 1.0 that we see in 0.9.1. Sincerely, Jason On Wednesday, March 27, 2013, Rob Godfrey wrote: > Hi Jason, > > OK... so that's embarrassing :-) The reason that the broker stopped sending > out messages in my test case was due to a deadlock in the broker side AMQP > 1.0 codepath [1]. I've applied a fix to the trunk version of the broker, > which I'll presently be asking for inclusion into the 0.22 release. > > Can you see if this also fixes the issue you have been seeing? > > Cheers, > Rob > > [1] http://svn.apache.org/viewvc?view=revision&revision=r1461844 > > On 27 March 2013 20:41, Rob Godfrey <[email protected]> wrote: > > > Hi Jason, > > > > just to let you know I am investigating the issue where the broker > > suddenly stops receiving messages... I can see this too but I haven't yet > > got to the bottom of why. Apologies for any inconvenience. > > > > Cheers, > > Rob > > > > > > On 27 March 2013 17:51, Jason Barto <[email protected]> wrote: > > > >> Rob, > >> first let me say thank you so much for your insight and quick responses. > >> I'm sorry to say that throttling the producers has not completely solved > >> the problem. I'm back in the office today and have been doing > additional > >> testing and I think I may have detected a memory leak in the code that > >> handles AMQP 1.0 communications. > >> > >> My experience thus far: > >> I downloaded and installed the latest QPid revision using the git > >> directions on the website. With the Java QPid up and running I ran my > >> first test client (AMQP 0-9-1). I sent 60k messages/sec, each with a > 1KB > >> payload, for 180 seconds. Using top I monitored the JVMs memory usage > >> which held steady at 233MB. The code for this client can be found on > >> pastebin at http://pastebin.com/fwDmhxG9. It uses the rabbitmq-client > >> JAR > >> for AMQP 0-9-1. > >> > >> Again using 'top' and the latest Java QPid I ran my second test client > at > >> 8k messages/sec, each with 1 KB payload, for 180 seconds. The memory > >> utilization of the JVM reported by 'top' rose steadily until the test > >> ended, at which time the JVM was using 1.1GB of memory. The code for > this > >> second test client is at http://pastebin.com/DNTLCj0y and uses v0.23 of > >> the > >> qpid-amqp-1-0-client JAR. It may also be worth noting that at about > >> second > >> 147 the consumer just stopped receiving messages, no exceptions were > >> thrown > >> on either the client or broker, it simply stopped receiving messages. > >> > >> Both tests were throttled to ensure that the consumer would consume in a > >> timely fashion roughly all messages sent, to avoid any buildup of > messages > >> in the broker's queue. > >> > >> I will admit that I have far more experience with the RabbitMQ client > >> library than the QPid library so my hope is that there is something > >> someone > >> can spot in my test code that is not being done correctly that will > >> explain > >> why the JVM is consuming so much memory. Does acknowledging a message > >> with > >> Receiver.acknowledge not sufficiently inform the broker that the message > >> has been received and can be forgotten? > >> > >> On longer test runs, 500+ seconds, the broker eventually consumed all of > >> its allocated ram and as you mentioned the GC began taking over trying > to > >> keep things running. > >> > >> Sincerely, > >> Jason > >> > >> > >> On Tue, Mar 26, 2013 at 12:43 PM, Rob Godfrey <[email protected] > >> >wrote: > >> > >> > Hi Jason, > >> > > >> > On 26 March 2013 13:26, Jason Barto <[email protected]> wrote: > >> > > >> > > Rob, > >> > > thanks for your quick response - I've consolidated the code into a > >> single > >> > > java file and would gladly publish it - frankly being new to AMQP > 1.0 > >> and > >> > > its client libraries I'd value the feedback. I think I may have > >> > determined > >> > > the reason for the slow down - not sure why it didn't occur to me > >> earlier > >> > > to be honest. > >> > > > >> > > I have 2 threads, a Sender and Receiver. Both are trying to > produce / > >> > > consume as rapidly as possible. Testing on my laptop the Sender is > >> > > publishing on average 20k messages / sec while the receiver seems to > >> top > >>
