Thanks Hiranya, I have done some more research, and found out the it leaks memory only on the failures. Not on successful deliver.
To give you the back ground, I had a typo in one of the endpoint URL. And I had only two endpoints, all the message that were attempted to be delivered to that "wrong" url; failed. That is what lead to this leaking issue. On the other hand the server I was testing had -Xms1024m, and -Xmx2048m. So its little better than what I was thinking. I am still digging into and trying to find where the leak is. I will update if I found anything but so far it's a leak on the failures only. If you can find out if this is correct that Synapse could leak those two object on failures. That will help. Thanks, Abid -----Original Message----- From: Hiranya Jayathilaka [mailto:[email protected]] Sent: Thursday, May 19, 2011 12:09 PM To: [email protected] Subject: Re: Please help: does Synapse leak memory? Synapse creates an Axis2MessageContext for each message received. Each Axis2MessageContext instance encapsulates a MessageContext object. But Synapse does not keep them in memory for too long. As soon as messages are mediated and sent to the corresponding endpoints/clients, they are discarded. These objects could be fairly big since they contain message payload, properties and all other context information of messages. So if Synapse is keeping them in memory forever, we won't be able to run any scenario with Synapse for more than several hours. But as Paul mentioned we have seen some pretty heavy duty deployments of Synapse dealing with hundreds of transactions every second. One of the key strengths of Synapse is that it can handle very high volumes of messages with a very small memory footprint. How much memory have you allocated for JBoss server? Also what is your Synapse configuration like? What's the memory usage pattern when you run Synapse standalone (ie without JBoss)? Thanks, Hiranya On Thu, May 19, 2011 at 9:17 PM, Abid Khan-EXT <[email protected]>wrote: > Hello Paul, > > What you are telling me is a good news. The facts I have here, are > different. I ran the server for a load-test and ran only 26000 message (of > about 1K each message size), that made the GC threads taking over every > thing, the java process was so stuck that the jboss-shutdown command could > not bring it down. > > And my trace log (memory profiling) showed that one instance of both > MessageContext and Axis2MessageContext were leaked for every message that is > delivered/processed. > > I am using 1.2 GA of synapse. And I am running it in JBoss. > > And will be more than happy to send the thread dump, but that is very big. > > Once again thanks for getting back to me and I am looking for good pointers > from you very soon. > > Regards, > Abid > > > > > -----Original Message----- > From: Paul Fremantle [mailto:[email protected]] > Sent: Wednesday, May 18, 2011 8:39 PM > To: [email protected] > Subject: Re: Please help: does Synapse leak memory? > > Abid > > Can you give us some idea of the flows? I know that Synapse is usually > memory safe - I have thread dumps from people who have pumped 134 > terabytes of data, 1.4 billion messages and all this with a max heap > size of <2Gb and no leaks. > > Paul > > On Thu, May 19, 2011 at 1:49 AM, Abid Khan-EXT <[email protected]> > wrote: > > Hello all, > > > > I am troubleshooting an instance where I had used Synapse, the issue is > that after delivering few thousand message server runs out of memory. > > > > I investigated and found out that for each message that is delivered > Synapse is leaking following two objects: > > > > 1) org.apache.axis2.context.MessageContext > > > > 2) org.apache.synapse.core.axis2.Axis2MessageContext > > > > I thought that it might be related to threads and ThreadLocal, but when I > scanned the thread local of all 160+ threads those objects were not there. > > > > It seems that synapse is keeping those two object in its some internal > object, that is difficult to find. > > > > Because of this problem those objects; server leaks memory at very high > rate, and freezes after few hours of running? > > > > Any immediate response will be appreciated, > > > > > > Thanks, > > Abid > > > > > > -- > Paul Fremantle > Co-Founder and CTO, WSO2 > Apache Synapse PMC Chair > OASIS WS-RX TC Co-chair > > blog: http://pzf.fremantle.org > [email protected] > > "Oxygenating the Web Service Platform", www.wso2.com > -- Hiranya Jayathilaka Senior Software Engineer; WSO2 Inc.; http://wso2.org E-mail: [email protected]; Mobile: +94 77 633 3491 Blog: http://techfeast-hiranya.blogspot.com
