On Thu, Feb 18, 2010 at 10:02 AM, Johan Haleby <[email protected]> wrote: > > I've now created a small Maven example project that demonstrate the issue. > http://old.nabble.com/file/p27635967/camel-mina-outofmemory-example.zip > camel-mina-outofmemory-example.zip > > You should run the test in "MinaProducerRetainedInMemoryExample". To speed > up the process I use the following VM arguments: > -XX:+HeapDumpOnOutOfMemoryError -XX:MaxPermSize=25m -Xmx25m > > Here is a screen dump from VisualVM where you can see the aggregated memory > consumption and eventually the high GC activity when memory is running out: > http://old.nabble.com/file/p27635967/mina_camel_leak_visualvm.png > > Here's the screen dump from Eclipse MAT: > http://old.nabble.com/file/p27635967/mat_problem_suspects.png > > If you like I could also provide the heap dump. If there's anything else I > can do to assist just tell me. > > /Johan >
Thanks Johan Can you attach the sample to the JIRA ticket I just created https://issues.apache.org/activemq/browse/CAMEL-2484 If you grant the Apache license when attaching the project we can use that to create unit tests etc. in camel-mina to help solve the issue > > Johan Haleby wrote: >> >> I'll try to find time for this tomorrow. >> >> /Johan >> >> >> Claus Ibsen-2 wrote: >>> >>> On Wed, Feb 17, 2010 at 4:59 PM, Johan Haleby <[email protected]> >>> wrote: >>>> >>>> I don't suppose it matters much but there are actually 5 different Mina >>>> endpoints involved (not 4 as I mentioned earlier). These are: >>>> mina:tcp://localhost:6200?sync=false&textline=false >>>> mina:tcp://localhost:6201?sync=false&textline=false >>>> mina:tcp://localhost:6202?sync=false&textline=false >>>> mina:tcp://localhost:6203?sync=false&textline=false >>>> mina:tcp://localhost:6204?sync=false&textline=false >>>> >>>> It's always to one of these addresses to which a message is sent. >>>> >>>> I've also tried to do a producer.stop() after each producer.sendBody(..) >>>> but >>>> it doesn't help at all. I still get the OutOfMemoryError and the same >>>> number >>>> of MinaProducers are still retained in memory. >>>> >>> >>> I wonder what keeps it in the memory then. Could you create a small >>> project and attach it as a ZIP to a JIRA ticket? >>> >>> >>> >>> >>>> /Johan >>>> >>>> >>>> Claus Ibsen-2 wrote: >>>>> >>>>> On Wed, Feb 17, 2010 at 4:39 PM, Johan Haleby <[email protected]> >>>>> wrote: >>>>>> >>>>>> In the test I'm sending messages between 4 different endpoints but >>>>>> they're >>>>>> treated as dynamic I suppose. What I mean is that the message body of >>>>>> each >>>>>> incoming message contains the reply address(es) (which could be >>>>>> completely >>>>>> dynamic). Thus I use a recipientList to distribute a reply message to >>>>>> each >>>>>> reply address stated in the message body. >>>>>> >>>>> >>>>> I am sure if you close the producer template after use you should >>>>> probably not see this as it contains an internal cache. >>>>> But that one should only hit 1000, so I wonder why it grows to 20000+. >>>>> >>>>> Well could you post a few samples of the endpoints you send to the >>>>> recipient list. If they are all 100% different then I would assume >>>>> Camel creates a new producer per message, as it would be a cache miss >>>>> in the producer template. >>>>> >>>>> >>>>> >>>>>> /Johan >>>>>> >>>>>> >>>>>> Claus Ibsen-2 wrote: >>>>>>> >>>>>>> Hi >>>>>>> >>>>>>> On Wed, Feb 17, 2010 at 4:17 PM, Johan Haleby >>>>>>> <[email protected]> >>>>>>> wrote: >>>>>>>> >>>>>>>> The debug message ("Closing session when complete at address") is >>>>>>>> printed >>>>>>>> all >>>>>>>> the time so it should close the session. I've investigated a bit >>>>>>>> further >>>>>>>> and >>>>>>>> I see that the problem seem to be that DefaultProducerTemplate holds >>>>>>>> a >>>>>>>> SpringCamelContext which in turn holds an ArrayList which seem to >>>>>>>> contain >>>>>>>> all MinaProducers that has ever been created. >>>>>>>> >>>>>>>> I've uploaded a screenshot of the heap dump: >>>>>>>> http://old.nabble.com/file/p27625230/mina_camel_retaining_resources.png >>>>>>>> mina_camel_retaining_resources.png >>>>>>>> >>>>>>> >>>>>>> Are you using a lot of dynamic endpoints which are different? >>>>>>> >>>>>>> >>>>>>>> /Johan >>>>>>>> >>>>>>>> >>>>>>>> Claus Ibsen-2 wrote: >>>>>>>>> >>>>>>>>> Hi >>>>>>>>> >>>>>>>>> Could you google that exception and see what other Mina users have >>>>>>>>> encountered. >>>>>>>>> >>>>>>>>> And you see this DEBUG logging when you send to that remote >>>>>>>>> endpoint? >>>>>>>>> >>>>>>>>> if (LOG.isDebugEnabled()) { >>>>>>>>> LOG.debug("Closing session when complete at >>>>>>>>> address: " >>>>>>>>> + endpoint.getAddress()); >>>>>>>>> } >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Feb 17, 2010 at 3:27 PM, Johan Haleby >>>>>>>>> <[email protected]> >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I'm running into problems when using Camel (2.1 and 2.2) with Mina >>>>>>>>>> endpoints >>>>>>>>>> configured to disconnect the session after each received message. >>>>>>>>>> When >>>>>>>>>> lots >>>>>>>>>> of messages are sent over a longer period of time I eventually run >>>>>>>>>> out >>>>>>>>>> of >>>>>>>>>> memory and the system either throws OutOfMemoryError or spends >>>>>>>>>> long >>>>>>>>>> intervals in GC. When analyzing the heap dump in Eclipse MAT it >>>>>>>>>> finds >>>>>>>>>> this >>>>>>>>>> leak suspect: >>>>>>>>>> >>>>>>>>>> 27,840 instances of >>>>>>>>>> "org.apache.mina.transport.socket.nio.SocketSessionImpl", loaded >>>>>>>>>> by >>>>>>>>>> "sun.misc.Launcher$AppClassLoader @ 0xad65d850" occupy 20,662,464 >>>>>>>>>> (20.42%) >>>>>>>>>> bytes >>>>>>>>>> >>>>>>>>>> The endpoint is configured like: >>>>>>>>>> mina:tcp://localhost:6200?sync=false&textline=false and the >>>>>>>>>> message >>>>>>>>>> is >>>>>>>>>> sent >>>>>>>>>> to the endpoint using a >>>>>>>>>> http://camel.apache.org/recipient-list.html >>>>>>>>>> recipient list . >>>>>>>>>> >>>>>>>>>> What could be the cause of this? E.g. do I need to stop the >>>>>>>>>> producer >>>>>>>>>> template after each sent message? Right now the ProducerTemplate >>>>>>>>>> is a >>>>>>>>>> singleton used concurrently by multiple threads. >>>>>>>>>> >>>>>>>>>> /Johan >>>>>>>>>> -- >>>>>>>>>> View this message in context: >>>>>>>>>> http://old.nabble.com/SocketSessionImpl-in-Mina-component-retained-in-memory-indefinitely-tp27624487p27624487.html >>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Claus Ibsen >>>>>>>>> Apache Camel Committer >>>>>>>>> >>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/ >>>>>>>>> Open Source Integration: http://fusesource.com >>>>>>>>> Blog: http://davsclaus.blogspot.com/ >>>>>>>>> Twitter: http://twitter.com/davsclaus >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> View this message in context: >>>>>>>> http://old.nabble.com/SocketSessionImpl-in-Mina-component-retained-in-memory-indefinitely-tp27624487p27625230.html >>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Claus Ibsen >>>>>>> Apache Camel Committer >>>>>>> >>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/ >>>>>>> Open Source Integration: http://fusesource.com >>>>>>> Blog: http://davsclaus.blogspot.com/ >>>>>>> Twitter: http://twitter.com/davsclaus >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> View this message in context: >>>>>> http://old.nabble.com/SocketSessionImpl-in-Mina-component-retained-in-memory-indefinitely-tp27624487p27625648.html >>>>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Claus Ibsen >>>>> Apache Camel Committer >>>>> >>>>> Author of Camel in Action: http://www.manning.com/ibsen/ >>>>> Open Source Integration: http://fusesource.com >>>>> Blog: http://davsclaus.blogspot.com/ >>>>> Twitter: http://twitter.com/davsclaus >>>>> >>>>> >>>> >>>> -- >>>> View this message in context: >>>> http://old.nabble.com/SocketSessionImpl-in-Mina-component-retained-in-memory-indefinitely-tp27624487p27625979.html >>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>> >>>> >>> >>> >>> >>> -- >>> Claus Ibsen >>> Apache Camel Committer >>> >>> Author of Camel in Action: http://www.manning.com/ibsen/ >>> Open Source Integration: http://fusesource.com >>> Blog: http://davsclaus.blogspot.com/ >>> Twitter: http://twitter.com/davsclaus >>> >>> >> >> > > -- > View this message in context: > http://old.nabble.com/SocketSessionImpl-in-Mina-component-retained-in-memory-indefinitely-tp27624487p27635967.html > Sent from the Camel - Users mailing list archive at Nabble.com. > > -- Claus Ibsen Apache Camel Committer Author of Camel in Action: http://www.manning.com/ibsen/ Open Source Integration: http://fusesource.com Blog: http://davsclaus.blogspot.com/ Twitter: http://twitter.com/davsclaus
