[
https://issues.apache.org/jira/browse/QPID-5786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14126792#comment-14126792
]
Rob Godfrey commented on QPID-5786:
-----------------------------------
And doing some more querying of the heap... we can find the oldest and newest
messages on the queue and their arrival times:
oldest message Mon, 08 Sep 2014 11:42:02 GMT
newest message Mon, 08 Sep 2014 11:54:42 GMT
We can also navigate down to the connection and look at when the last write on
the connection took place:
last write time Mon, 08 Sep 2014 11:54:53 GMT
and then looking at the statistics counters on the connection object you can see
Messages Delivered By the Broker:
last statistics Mon, 08 Sep 2014 11:54:52 GMT
Peak message throughput : 1529 msg/s
Current message throughput: 2 msg/s
So... All the evidence is pointing to the broker just getting to the point
where (because the consumer was slower than the producer) the queue grew too
large, causing the broker to run out of heap, and then enter a GC loop slowing
it down further (throughput of 2 msg/s) until eventually it died through heap
exhaustion.
As above, enabling flow control limits on queues would potentially be one
solution (presuming the client respects the flow controlling requests), other
than that, 0.30 will help a bit... Or you could introduce a TTL onto the
messages if it is not vital that all messages get through... but ultimately any
scenario where you have a producer that is outpacing the consumer will
eventually cause a broker to die as they cannot provide infinite storage.
> Memory leak in qpid-java-broker-0.26.tar.gz
> -------------------------------------------
>
> Key: QPID-5786
> URL: https://issues.apache.org/jira/browse/QPID-5786
> Project: Qpid
> Issue Type: Bug
> Components: Java Broker
> Affects Versions: 0.26
> Environment: Linux Mint 15 (64 Bit) - any Linux version should do
> Java // OpenJDK Runtime Environment (IcedTea 2.4.4)
> also tested with Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
> Ruby 1.8.7
> Ruby Bunny Gem 1.2.2
> Reporter: Bobby Flitter
> Attachments: amqp_bunny_qpid.pcap
>
>
> When using the hello bunny example program from Ruby together with Apache
> qpid server the memory consumption of qpid seems to grow continuously.
> Steps to reproduce the problem in Linux:
> wget
> http://ftp-stud.hs-esslingen.de/pub/Mirrors/ftp.apache.org/dist/qpid/0.26/qpid-java-broker-0.26.tar.gz
> tar xzvf qpid-java-broker-0.26.tar.gz
> cd qpid-broker-0.26/bin/
> ./qpid-server
> - in a new terminal
> - meanwhile bunny 1.2.2. is available
> - install ruby amqp bunny
> sudo gem install bunny
> - download hello bunny example
> wget http://pastebin.com/raw.php?i=f6MesMCZ -O hello_bunny.rb
> while true; do ruby hello_bunny.rb ; done
> - in another terminal:
> while true; do grep "^VmRSS" /proc/$(pgrep -f qpid)/status ; sleep 1;done
> - memory consumption of java qpid is growing continuously until
> OutOfMemoryError ends the server
> Unhandled Exception java.lang.OutOfMemoryError: Java heap space in Thread
> Thread-1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
