Re: References to objects processed in routes held, eventually run out of memory
Hi Can you try using bean(Object, methodname) in your route and pass in the same object, just to be sure its not Camel loading your bean constantly? And you are sure your bean in Spring XML is defined as singleton scope? In 2.0 I have adding TRACE logging to ObjectHelper.loadClass that loads a class. Then we will be able to TRACE if Camel is loading the same class over and over again. Another try is to use .process() instead of bean and put the code in there so you are sure the class is not loaded. On Thu, Mar 19, 2009 at 2:24 PM, ee7arh andrew.hu...@2e-systems.com wrote: Hi, Thanks for response. I'm using Camel 1.6.0 and ActiveMQ 5.2.0. I also had the same results with camel 1.5 and activeMQ 5.1 I have attached a graph from Jconsole showing what happens as the application starts, runs for a about 1.5 hours then crashes with out of memory. http://www.nabble.com/file/p22600122/jconsole-graphs.jpg Notes: - I set the maximum memory to 64MB using -Xmx64M -server. normally I would use more but then the problem takes longer to reproduce - Notice the number of loaded classes never reduces. I also have another application which shows the same pattern. - As the application starts to run out of memory, the CPU usage goes extremely high Here is my route: from(jms:queue:mobileExtTriggerInvites) .errorHandler(deadLetterChannel(seda:errors).maximumRedeliveries(0)) // Exception Handling .onException(JAXBException.class) .to(jms:queue:unmarshallableRequest).end() .onException(UnexpectedEventException.class) .to(jms:queue:unexpectedEvent).end() // Log Incoming XML .to(log:incomingEventXmlLogger?level=INFO) // Attempt to Unmarshall object expected on this queue .to(bean:eventMarshaller?methodName=unmarshallTriggerInvites) // Generate Service Events from this External Event .to(bean:serviceEventGenerator?methodName=procecssTriggerInvites) // Split generated Events out of List into individual events .splitter(body()) // Destination queue for further processing .to(jms:queue:serviceEventQueue); You can see that I have 2 bean processors, I have also attached the code for those. http://www.nabble.com/file/p22600122/EventMarhaller.txt EventMarhaller.txt http://www.nabble.com/file/p22600122/ServiceEventGenerator.txt ServiceEventGenerator.txt Perhaps it's useful, perhaps not but I read a blog article which sounded very similar to the problem I am having, link below. Basically it looks like Camel or ActiveMQ still has a reference to every single object I create and send through the routing - this would surely explain why the number of loaded classes gets bigger and bigger the more the application runs. http://dertompson.com/2007/08/01/fighting-the-outofmemoryerror/ http://dertompson.com/2007/08/01/fighting-the-outofmemoryerror using JConsole, I also periodically purged the end queue and this made no difference. If it helps, I am able to provide a heapdump to anyone for analysis with jHat. As mentioned in my 1st post, all instances of classes created during the application lifecycle continue to live on in memory. It appears that my objects are referenced by ActiveMQMessage For example, I will trace 1 of those classes created in code attached: Class: class com.ee.berbe.mobile.servicelogic.flightops.events.service.TriggerMMSInvitations References to this object: org.apache.activemq.command.activemqobjectmess...@0xafab8fd8 (174 bytes) : field object -- -- Class: class org.apache.activemq.command.ActiveMQObjectMessage References to this object: java.util.linkedhashmap$en...@0xafab8970 (32 bytes) : field value java.util.linkedhashmap$en...@0xafab8810 (32 bytes) : field value So it seems that ActiveMQ is still storing all my objets on a LinkedHashMap but I've no idea why. Thanks again for any help Andrew willem.jiang wrote: Hi Which version of Camel are you using ? Can you show us you routing rules and your processor codes? They will help us to dig the issue. Willem -- View this message in context: http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22600122.html Sent from the Camel - Users mailing list archive at Nabble.com. -- Claus Ibsen Apache Camel Committer Open Source Integration: http://fusesource.com Blog: http://davsclaus.blogspot.com/
Re: References to objects processed in routes held, eventually run out of memory
Hi, I can confirm from JHat that only a single instance of the bean classes are loaded: 1 instance of class com.ee.berbe.mobile.external.EventMarshalling 1 instance of class com.ee.berbe.mobile.external.EventRouter 1 instance of class com.ee.berbe.mobile.servicelogic.ServiceEventGenerator 1 instance of class com.ee.berbe.mobile.servicelogic.ServiceDataRouter If I look in JHat at the total number of classes loaded and order it ascending, if I search down for anything camel related I see: 121 instances of class org.apache.camel.impl.converter.DefaultTypeConverter$TypeMapping If I do the same for activemq I see many more instances: 4466 instances of class org.apache.activemq.command.SessionId 4456 instances of class org.apache.activemq.command.ConnectionId 4382 instances of class org.apache.activemq.command.ProducerId 3750 instances of class org.apache.activemq.util.ByteSequence 3748 instances of class org.apache.activemq.command.MessageId 3565 instances of class org.apache.activemq.command.ActiveMQQueue If I do the same for my internal object type I see: 3358 instances of class com.ee.berbe.mobile.servicelogic..events.external.TriggerCkiInvites Does that answer your question satisfactorily or should I perform your tests to double check? Thanks Andrew Claus Ibsen-2 wrote: Hi Can you try using bean(Object, methodname) in your route and pass in the same object, just to be sure its not Camel loading your bean constantly? And you are sure your bean in Spring XML is defined as singleton scope? In 2.0 I have adding TRACE logging to ObjectHelper.loadClass that loads a class. Then we will be able to TRACE if Camel is loading the same class over and over again. Another try is to use .process() instead of bean and put the code in there so you are sure the class is not loaded. -- View this message in context: http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22617746.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: References to objects processed in routes held, eventually run out of memory
Hi again, I added this component into my camel xml config and added a couple of extra jar files from activemq to get it working: !-- configure the Camel JMS consumer to use the ActiveMQ broker declared above -- bean id=jms class=org.apache.activemq.camel.component.ActiveMQComponent property name=brokerURL value=vm://localhost/ /bean I continued to use the id=jms to avoid modifying all my routes. Unfortunately, it did not make any difference. I left the application running for about 1 hour now and the classes continue to increase steadily just as before :-( Thanks Andrew Claus Ibsen-2 wrote: Hi You should use the AMQ component instead of the generic JMS component http://camel.apache.org/activemq.html -- View this message in context: http://www.nabble.com/References-to-objects-processed-in-routes-held%2C-eventually-run-out-of-memory-tp22580791p22619315.html Sent from the Camel - Users mailing list archive at Nabble.com.
Re: References to objects processed in routes held, eventually run out of memory
Hi again, A bit more information which might help. I have been running the application for the last 4 hours with 256MB memory and so far it looks like the outOfMemory problem will probably not occur anytime soon because the heap usage drops dramatically after the initial high load processing. In addition, I have been running messages through further routes to perform more processing. In my last example, i ran the applciation only through a simple route but using less memory Here is the graph: http://www.nabble.com/file/p22601000/jconsole-graphs-2.jpg When the application was first started it was under high load so you can see it took almsot 200MB but after the first 30 mns, fewer messages were processed and so the application looks to gradually recover the memory. however, having said all that, notice once again the classes loaded into memory, it does not follow the same pattern. Number of loaded classes keeps on increasing so I guess although it may now take a long time, the application will eventually run out of memory. Perhaps then there are 2 reasons which can cause an out of memory: 1) When I simply do not have enough memory to facilitate high-load procesing 2) When the nunber of classes in memory continues to increase until no more memory is left... Andrew ee7arh wrote: Hi, Thanks but we are not using camel-jaxb, everything is being done manually using the JaxB classes directly (although now I see camel has this built in I will change well for R2.0 anyway!) As way of double checking, I found only 7 instances of my unmarshalled objects in the memory dump, not 1600 like I see for the aforementioned object in previous post. Andrew Claus Ibsen-2 wrote: Hi Just a very quick reply without digging into it yet. There was/is a leak in camel-jaxb. I cant remember if we got it fixed in 1.6.0 or its in 1.6.1. At least its fixed in 2.0m1. Could you either: - try without any JAXB - try with 1.6.1-SNAPSHOT or 2.0m1/2.0-SNAPSHOT On Thu, Mar 19, 2009 at 2:24 PM, ee7arh andrew.hu...@2e-systems.com wrote: Hi, Thanks for response. I'm using Camel 1.6.0 and ActiveMQ 5.2.0. I also had the same results with camel 1.5 and activeMQ 5.1 I have attached a graph from Jconsole showing what happens as the application starts, runs for a about 1.5 hours then crashes with out of memory. http://www.nabble.com/file/p22600122/jconsole-graphs.jpg Notes: - I set the maximum memory to 64MB using -Xmx64M -server. normally I would use more but then the problem takes longer to reproduce - Notice the number of loaded classes never reduces. I also have another application which shows the same pattern. - As the application starts to run out of memory, the CPU usage goes extremely high Here is my route: from(jms:queue:mobileExtTriggerInvites) .errorHandler(deadLetterChannel(seda:errors).maximumRedeliveries(0)) // Exception Handling .onException(JAXBException.class) .to(jms:queue:unmarshallableRequest).end() .onException(UnexpectedEventException.class) .to(jms:queue:unexpectedEvent).end() // Log Incoming XML .to(log:incomingEventXmlLogger?level=INFO) // Attempt to Unmarshall object expected on this queue .to(bean:eventMarshaller?methodName=unmarshallTriggerInvites) // Generate Service Events from this External Event .to(bean:serviceEventGenerator?methodName=procecssTriggerInvites) // Split generated Events out of List into individual events .splitter(body()) // Destination queue for further processing .to(jms:queue:serviceEventQueue); You can see that I have 2 bean processors, I have also attached the code for those. http://www.nabble.com/file/p22600122/EventMarhaller.txt EventMarhaller.txt http://www.nabble.com/file/p22600122/ServiceEventGenerator.txt ServiceEventGenerator.txt Perhaps it's useful, perhaps not but I read a blog article which sounded very similar to the problem I am having, link below. Basically it looks like Camel or ActiveMQ still has a reference to every single object I create and send through the routing - this would surely explain why the number of loaded classes gets bigger and bigger the more the application runs. http://dertompson.com/2007/08/01/fighting-the-outofmemoryerror/ http://dertompson.com/2007/08/01/fighting-the-outofmemoryerror using JConsole, I also periodically purged the end queue and this made no difference. If it helps, I am able to provide a heapdump to anyone for analysis with jHat. As mentioned in my 1st post, all instances of classes created during the application lifecycle continue to live on in memory. It appears that my objects are referenced by ActiveMQMessage For example, I will trace 1 of those classes created in code attached: Class: class
