[
https://issues.apache.org/jira/browse/CAMEL-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13470327#comment-13470327
]
Michael Pilone commented on CAMEL-5683:
---------------------------------------
I compiled the code from the 2.10.x branch and confirmed that your change does
appear to fix the issue when using the SendProducer. However I also confirmed
my previous comment that the problem still exists when using the
DefaultProducerTemplate (or any other code that uses the ProducerCache with the
LRU map implementation). I'll attach an update test case which uses the
ProducerTemplate to reproduce the problem. The current cache implementation is
going to be a problem with any producer that requires a stop call to properly
cleanup.
You might want to look at modifying the ProducerCache to support a
ReferenceQueue with the SoftReferences. Then the ProducerCache could drain the
queue and stop all the reclaimed producers before creating a new producer.
Even with that fix, it might be a good idea to have an easy way (e.g. via a
context property) to disable the soft-references in the cache and rely only on
max cache size. If I know I'm only going to have 3 or 4 producers but a lot of
memory churn, it would be nice to know that my producers would stay in the
cache until I completely fill it. This could be really valuable if producer
construction/teardown were expensive.
> JMS connection leak with request/reply producer on temporary queues
> -------------------------------------------------------------------
>
> Key: CAMEL-5683
> URL: https://issues.apache.org/jira/browse/CAMEL-5683
> Project: Camel
> Issue Type: Bug
> Components: camel-jms
> Affects Versions: 2.10.0
> Environment: Apache Camel 2.10.0
> ActiveMQ 5.6.0
> Spring 3.2.1.RELEASE
> Java 1.6.0_27
> SunOS HOST 5.10 Generic_144488-09 sun4v sparc SUNW,SPARC-Enterprise-T5220
> Reporter: Michael Pilone
> Assignee: Claus Ibsen
> Fix For: 2.9.4, 2.11.0, 2.10.2
>
> Attachments: CamelConnectionLeak.zip, Consumer List.txt, MAT
> Snapshot.png, Route Configuration.txt
>
>
> Over time I see the number of temporary queues in ActiveMQ slowly climb.
> Using JMX information and memory dumps in MAT, I believe the cause is a
> connection leak in Apache Camel.
> My environment contains 2 ActiveMQ brokers in a network of brokers
> configuration. There are about 15 separate applications which use Apache
> Camel to connect to the broker using the ActiveMQ/JMS component. The various
> applications have different load profiles and route configurations.
> In the more active client applications, I found that ActiveMQ was listing
> 300+ consumers when, based on my configuration, I would expect no more than
> 75. The vast majority of the consumers are sitting on a temporary queue. Over
> time, the 300 number increments by one or two over about a 4 hour period.
> I did a memory dump on one of the more active client applications and found
> about 275 DefaultMessageListenerContainers. Using MAT, I can see that some of
> the containers are referenced by JmsProducers in the ProducerCache; however I
> can also see a large number of listener containers that are no longer being
> referenced at all. I was also able to match up a soft-references
> producer/listener endpoint with an unreferenced listener which means a second
> producer was created at some point.
> Looking through the ProducerCache code, it looks like the LRU cache uses
> soft-references to producers, in my case a JmsProducer. This seems
> problematic for two reasons:
> - If memory gets constrained and the GC cleans up a producer, it is never
> properly stopped.
> - If the cache gets full and the map removes the LRU producer, it is never
> properly stopped.
> What I believe is happening, is that my application is sending a few
> request/reply messages to a JmsProducer. The producer creates a
> TemporaryReplyManager which creates a DefaultMessageListenerContainer. At
> some point, the JmsProducer is claimed by the GC (either via the
> soft-reference or because the cache is full) and the reply manager is never
> stopped. This causes the listener container to continue to listen on the
> temporary queue, consuming local resources and more importantly, consuming
> resources on the JMS broker.
> I haven't had a chance to write an application to reproduce this behavior,
> but I will attach one of my route configurations and a screenshot of the MAT
> analysis looking at DefaultMessageListenerContainers. If needed, I could
> provide the entire memory dump for analysis (although I rather not post it
> publicly). The leak depends on memory usage or producer count in the client
> application because the ProducerCache must have some churn. Like I said, in
> our production system we see about 12 temporary queues abandoned per client
> per day.
> Unless I'm missing something, it looks like the producer cache would need to
> be much smarter to support stopping a producer when the soft-reference is
> reclaimed or a member of the cache is ejected from the LRU list.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
