[
https://issues.apache.org/jira/browse/CAMEL-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Pilone updated CAMEL-5683:
----------------------------------
Attachment: MAT Snapshot.png
Attached screenshot from MAT analysis.
> JMS connection leak with request/reply producer on temporary queues
> -------------------------------------------------------------------
>
> Key: CAMEL-5683
> URL: https://issues.apache.org/jira/browse/CAMEL-5683
> Project: Camel
> Issue Type: Bug
> Components: camel-jms
> Affects Versions: 2.10.0
> Environment: Apache Camel 2.10.0
> ActiveMQ 5.6.0
> Spring 3.2.1.RELEASE
> Java 1.6.0_27
> SunOS HOST 5.10 Generic_144488-09 sun4v sparc SUNW,SPARC-Enterprise-T5220
> Reporter: Michael Pilone
> Attachments: MAT Snapshot.png
>
>
> Over time I see the number of temporary queues in ActiveMQ slowly climb.
> Using JMX information and memory dumps in MAT, I believe the cause is a
> connection leak in Apache Camel.
> My environment contains 2 ActiveMQ brokers in a network of brokers
> configuration. There are about 15 separate applications which use Apache
> Camel to connect to the broker using the ActiveMQ/JMS component. The various
> applications have different load profiles and route configurations.
> In the more active client applications, I found that ActiveMQ was listing
> 300+ consumers when, based on my configuration, I would expect no more than
> 75. The vast majority of the consumers are sitting on a temporary queue. Over
> time, the 300 number increments by one or two over about a 4 hour period.
> I did a memory dump on one of the more active client applications and found
> about 275 DefaultMessageListenerContainers. Using MAT, I can see that some of
> the containers are referenced by JmsProducers in the ProducerCache; however I
> can also see a large number of listener containers that are no longer being
> referenced at all. I was also able to match up a soft-references
> producer/listener endpoint with an unreferenced listener which means a second
> producer was created at some point.
> Looking through the ProducerCache code, it looks like the LRU cache uses
> soft-references to producers, in my case a JmsProducer. This seems
> problematic for two reasons:
> - If memory gets constrained and the GC cleans up a producer, it is never
> properly stopped.
> - If the cache gets full and the map removes the LRU producer, it is never
> properly stopped.
> What I believe is happening, is that my application is sending a few
> request/reply messages to a JmsProducer. The producer creates a
> TemporaryReplyManager which creates a DefaultMessageListenerContainer. At
> some point, the JmsProducer is claimed by the GC (either via the
> soft-reference or because the cache is full) and the reply manager is never
> stopped. This causes the listener container to continue to listen on the
> temporary queue, consuming local resources and more importantly, consuming
> resources on the JMS broker.
> I haven't had a chance to write an application to reproduce this behavior,
> but I will attach one of my route configurations and a screenshot of the MAT
> analysis looking at DefaultMessageListenerContainers. If needed, I could
> provide the entire memory dump for analysis (although I rather not post it
> publicly). The leak depends on memory usage or producer count in the client
> application because the ProducerCache must have some churn. Like I said, in
> our production system we see about 12 temporary queues abandoned per client
> per day.
> Unless I'm missing something, it looks like the producer cache would need to
> be much smarter to support stopping a producer when the soft-reference is
> reclaimed or a member of the cache is ejected from the LRU list.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
