Done! Thanks a lot for your help so far.
/Johan Claus Ibsen-2 wrote: > > On Thu, Feb 18, 2010 at 10:02 AM, Johan Haleby <[email protected]> > wrote: >> >> I've now created a small Maven example project that demonstrate the >> issue. >> http://old.nabble.com/file/p27635967/camel-mina-outofmemory-example.zip >> camel-mina-outofmemory-example.zip >> >> You should run the test in "MinaProducerRetainedInMemoryExample". To >> speed >> up the process I use the following VM arguments: >> -XX:+HeapDumpOnOutOfMemoryError -XX:MaxPermSize=25m -Xmx25m >> >> Here is a screen dump from VisualVM where you can see the aggregated >> memory >> consumption and eventually the high GC activity when memory is running >> out: >> http://old.nabble.com/file/p27635967/mina_camel_leak_visualvm.png >> >> Here's the screen dump from Eclipse MAT: >> http://old.nabble.com/file/p27635967/mat_problem_suspects.png >> >> If you like I could also provide the heap dump. If there's anything else >> I >> can do to assist just tell me. >> >> /Johan >> > > Thanks Johan > > Can you attach the sample to the JIRA ticket I just created > https://issues.apache.org/activemq/browse/CAMEL-2484 > > If you grant the Apache license when attaching the project we can use > that to create unit tests etc. in camel-mina to help solve the issue > >> >> Johan Haleby wrote: >>> >>> I'll try to find time for this tomorrow. >>> >>> /Johan >>> >>> >>> Claus Ibsen-2 wrote: >>>> >>>> On Wed, Feb 17, 2010 at 4:59 PM, Johan Haleby <[email protected]> >>>> wrote: >>>>> >>>>> I don't suppose it matters much but there are actually 5 different >>>>> Mina >>>>> endpoints involved (not 4 as I mentioned earlier). These are: >>>>> mina:tcp://localhost:6200?sync=false&textline=false >>>>> mina:tcp://localhost:6201?sync=false&textline=false >>>>> mina:tcp://localhost:6202?sync=false&textline=false >>>>> mina:tcp://localhost:6203?sync=false&textline=false >>>>> mina:tcp://localhost:6204?sync=false&textline=false >>>>> >>>>> It's always to one of these addresses to which a message is sent. >>>>> >>>>> I've also tried to do a producer.stop() after each >>>>> producer.sendBody(..) >>>>> but >>>>> it doesn't help at all. I still get the OutOfMemoryError and the same >>>>> number >>>>> of MinaProducers are still retained in memory. >>>>> >>>> >>>> I wonder what keeps it in the memory then. Could you create a small >>>> project and attach it as a ZIP to a JIRA ticket? >>>> >>>> >>>> >>>> >>>>> /Johan >>>>> >>>>> >>>>> Claus Ibsen-2 wrote: >>>>>> >>>>>> On Wed, Feb 17, 2010 at 4:39 PM, Johan Haleby >>>>>> <[email protected]> >>>>>> wrote: >>>>>>> >>>>>>> In the test I'm sending messages between 4 different endpoints but >>>>>>> they're >>>>>>> treated as dynamic I suppose. What I mean is that the message body >>>>>>> of >>>>>>> each >>>>>>> incoming message contains the reply address(es) (which could be >>>>>>> completely >>>>>>> dynamic). Thus I use a recipientList to distribute a reply message >>>>>>> to >>>>>>> each >>>>>>> reply address stated in the message body. >>>>>>> >>>>>> >>>>>> I am sure if you close the producer template after use you should >>>>>> probably not see this as it contains an internal cache. >>>>>> But that one should only hit 1000, so I wonder why it grows to >>>>>> 20000+. >>>>>> >>>>>> Well could you post a few samples of the endpoints you send to the >>>>>> recipient list. If they are all 100% different then I would assume >>>>>> Camel creates a new producer per message, as it would be a cache miss >>>>>> in the producer template. >>>>>> >>>>>> >>>>>> >>>>>>> /Johan >>>>>>> >>>>>>> >>>>>>> Claus Ibsen-2 wrote: >>>>>>>> >>>>>>>> Hi >>>>>>>> >>>>>>>> On Wed, Feb 17, 2010 at 4:17 PM, Johan Haleby >>>>>>>> <[email protected]> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> The debug message ("Closing session when complete at address") is >>>>>>>>> printed >>>>>>>>> all >>>>>>>>> the time so it should close the session. I've investigated a bit >>>>>>>>> further >>>>>>>>> and >>>>>>>>> I see that the problem seem to be that DefaultProducerTemplate >>>>>>>>> holds >>>>>>>>> a >>>>>>>>> SpringCamelContext which in turn holds an ArrayList which seem to >>>>>>>>> contain >>>>>>>>> all MinaProducers that has ever been created. >>>>>>>>> >>>>>>>>> I've uploaded a screenshot of the heap dump: >>>>>>>>> http://old.nabble.com/file/p27625230/mina_camel_retaining_resources.png >>>>>>>>> mina_camel_retaining_resources.png >>>>>>>>> >>>>>>>> >>>>>>>> Are you using a lot of dynamic endpoints which are different? >>>>>>>> >>>>>>>> >>>>>>>>> /Johan >>>>>>>>> >>>>>>>>> >>>>>>>>> Claus Ibsen-2 wrote: >>>>>>>>>> >>>>>>>>>> Hi >>>>>>>>>> >>>>>>>>>> Could you google that exception and see what other Mina users >>>>>>>>>> have >>>>>>>>>> encountered. >>>>>>>>>> >>>>>>>>>> And you see this DEBUG logging when you send to that remote >>>>>>>>>> endpoint? >>>>>>>>>> >>>>>>>>>> if (LOG.isDebugEnabled()) { >>>>>>>>>> LOG.debug("Closing session when complete at >>>>>>>>>> address: " >>>>>>>>>> + endpoint.getAddress()); >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Wed, Feb 17, 2010 at 3:27 PM, Johan Haleby >>>>>>>>>> <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I'm running into problems when using Camel (2.1 and 2.2) with >>>>>>>>>>> Mina >>>>>>>>>>> endpoints >>>>>>>>>>> configured to disconnect the session after each received >>>>>>>>>>> message. >>>>>>>>>>> When >>>>>>>>>>> lots >>>>>>>>>>> of messages are sent over a longer period of time I eventually >>>>>>>>>>> run >>>>>>>>>>> out >>>>>>>>>>> of >>>>>>>>>>> memory and the system either throws OutOfMemoryError or spends >>>>>>>>>>> long >>>>>>>>>>> intervals in GC. When analyzing the heap dump in Eclipse MAT it >>>>>>>>>>> finds >>>>>>>>>>> this >>>>>>>>>>> leak suspect: >>>>>>>>>>> >>>>>>>>>>> 27,840 instances of >>>>>>>>>>> "org.apache.mina.transport.socket.nio.SocketSessionImpl", loaded >>>>>>>>>>> by >>>>>>>>>>> "sun.misc.Launcher$AppClassLoader @ 0xad65d850" occupy >>>>>>>>>>> 20,662,464 >>>>>>>>>>> (20.42%) >>>>>>>>>>> bytes >>>>>>>>>>> >>>>>>>>>>> The endpoint is configured like: >>>>>>>>>>> mina:tcp://localhost:6200?sync=false&textline=false and the >>>>>>>>>>> message >>>>>>>>>>> is >>>>>>>>>>> sent >>>>>>>>>>> to the endpoint using a >>>>>>>>>>> http://camel.apache.org/recipient-list.html >>>>>>>>>>> recipient list . >>>>>>>>>>> >>>>>>>>>>> What could be the cause of this? E.g. do I need to stop the >>>>>>>>>>> producer >>>>>>>>>>> template after each sent message? Right now the ProducerTemplate >>>>>>>>>>> is a >>>>>>>>>>> singleton used concurrently by multiple threads. >>>>>>>>>>> >>>>>>>>>>> /Johan >>>>>>>>>>> -- >>>>>>>>>>> View this message in context: >>>>>>>>>>> http://old.nabble.com/SocketSessionImpl-in-Mina-component-retained-in-memory-indefinitely-tp27624487p27624487.html >>>>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Claus Ibsen >>>>>>>>>> Apache Camel Committer >>>>>>>>>> >>>>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/ >>>>>>>>>> Open Source Integration: http://fusesource.com >>>>>>>>>> Blog: http://davsclaus.blogspot.com/ >>>>>>>>>> Twitter: http://twitter.com/davsclaus >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> View this message in context: >>>>>>>>> http://old.nabble.com/SocketSessionImpl-in-Mina-component-retained-in-memory-indefinitely-tp27624487p27625230.html >>>>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Claus Ibsen >>>>>>>> Apache Camel Committer >>>>>>>> >>>>>>>> Author of Camel in Action: http://www.manning.com/ibsen/ >>>>>>>> Open Source Integration: http://fusesource.com >>>>>>>> Blog: http://davsclaus.blogspot.com/ >>>>>>>> Twitter: http://twitter.com/davsclaus >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> View this message in context: >>>>>>> http://old.nabble.com/SocketSessionImpl-in-Mina-component-retained-in-memory-indefinitely-tp27624487p27625648.html >>>>>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Claus Ibsen >>>>>> Apache Camel Committer >>>>>> >>>>>> Author of Camel in Action: http://www.manning.com/ibsen/ >>>>>> Open Source Integration: http://fusesource.com >>>>>> Blog: http://davsclaus.blogspot.com/ >>>>>> Twitter: http://twitter.com/davsclaus >>>>>> >>>>>> >>>>> >>>>> -- >>>>> View this message in context: >>>>> http://old.nabble.com/SocketSessionImpl-in-Mina-component-retained-in-memory-indefinitely-tp27624487p27625979.html >>>>> Sent from the Camel - Users mailing list archive at Nabble.com. >>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Claus Ibsen >>>> Apache Camel Committer >>>> >>>> Author of Camel in Action: http://www.manning.com/ibsen/ >>>> Open Source Integration: http://fusesource.com >>>> Blog: http://davsclaus.blogspot.com/ >>>> Twitter: http://twitter.com/davsclaus >>>> >>>> >>> >>> >> >> -- >> View this message in context: >> http://old.nabble.com/SocketSessionImpl-in-Mina-component-retained-in-memory-indefinitely-tp27624487p27635967.html >> Sent from the Camel - Users mailing list archive at Nabble.com. >> >> > > > > -- > Claus Ibsen > Apache Camel Committer > > Author of Camel in Action: http://www.manning.com/ibsen/ > Open Source Integration: http://fusesource.com > Blog: http://davsclaus.blogspot.com/ > Twitter: http://twitter.com/davsclaus > > -- View this message in context: http://old.nabble.com/SocketSessionImpl-in-Mina-component-retained-in-memory-indefinitely-tp27624487p27636292.html Sent from the Camel - Users mailing list archive at Nabble.com.
