Can you attach your Synapse configuration if possible? And tell which endpoint has a typo ?
Rajika On Thu, May 19, 2011 at 11:47 PM, Abid Khan-EXT <[email protected]>wrote: > Thanks Hiranya, > > I have done some more research, and found out the it leaks memory only on > the failures. Not on successful deliver. > > To give you the back ground, I had a typo in one of the endpoint URL. And I > had only two endpoints, all the message that were attempted to be delivered > to that "wrong" url; failed. That is what lead to this leaking issue. > > On the other hand the server I was testing had -Xms1024m, and -Xmx2048m. So > its little better than what I was thinking. > > I am still digging into and trying to find where the leak is. I will update > if I found anything but so far it's a leak on the failures only. > > If you can find out if this is correct that Synapse could leak those two > object on failures. That will help. > > Thanks, > Abid > > > > > -----Original Message----- > From: Hiranya Jayathilaka [mailto:[email protected]] > Sent: Thursday, May 19, 2011 12:09 PM > To: [email protected] > Subject: Re: Please help: does Synapse leak memory? > > Synapse creates an Axis2MessageContext for each message received. Each > Axis2MessageContext instance encapsulates a MessageContext object. But > Synapse does not keep them in memory for too long. As soon as messages are > mediated and sent to the corresponding endpoints/clients, they are > discarded. These objects could be fairly big since they contain message > payload, properties and all other context information of messages. So if > Synapse is keeping them in memory forever, we won't be able to run any > scenario with Synapse for more than several hours. But as Paul mentioned we > have seen some pretty heavy duty deployments of Synapse dealing with > hundreds of transactions every second. One of the key strengths of Synapse > is that it can handle very high volumes of messages with a very small > memory > footprint. > > How much memory have you allocated for JBoss server? Also what is your > Synapse configuration like? What's the memory usage pattern when you run > Synapse standalone (ie without JBoss)? > > Thanks, > Hiranya > > On Thu, May 19, 2011 at 9:17 PM, Abid Khan-EXT <[email protected] > >wrote: > > > Hello Paul, > > > > What you are telling me is a good news. The facts I have here, are > > different. I ran the server for a load-test and ran only 26000 message > (of > > about 1K each message size), that made the GC threads taking over every > > thing, the java process was so stuck that the jboss-shutdown command > could > > not bring it down. > > > > And my trace log (memory profiling) showed that one instance of both > > MessageContext and Axis2MessageContext were leaked for every message that > is > > delivered/processed. > > > > I am using 1.2 GA of synapse. And I am running it in JBoss. > > > > And will be more than happy to send the thread dump, but that is very > big. > > > > Once again thanks for getting back to me and I am looking for good > pointers > > from you very soon. > > > > Regards, > > Abid > > > > > > > > > > -----Original Message----- > > From: Paul Fremantle [mailto:[email protected]] > > Sent: Wednesday, May 18, 2011 8:39 PM > > To: [email protected] > > Subject: Re: Please help: does Synapse leak memory? > > > > Abid > > > > Can you give us some idea of the flows? I know that Synapse is usually > > memory safe - I have thread dumps from people who have pumped 134 > > terabytes of data, 1.4 billion messages and all this with a max heap > > size of <2Gb and no leaks. > > > > Paul > > > > On Thu, May 19, 2011 at 1:49 AM, Abid Khan-EXT <[email protected]> > > wrote: > > > Hello all, > > > > > > I am troubleshooting an instance where I had used Synapse, the issue is > > that after delivering few thousand message server runs out of memory. > > > > > > I investigated and found out that for each message that is delivered > > Synapse is leaking following two objects: > > > > > > 1) org.apache.axis2.context.MessageContext > > > > > > 2) org.apache.synapse.core.axis2.Axis2MessageContext > > > > > > I thought that it might be related to threads and ThreadLocal, but when > I > > scanned the thread local of all 160+ threads those objects were not > there. > > > > > > It seems that synapse is keeping those two object in its some internal > > object, that is difficult to find. > > > > > > Because of this problem those objects; server leaks memory at very high > > rate, and freezes after few hours of running? > > > > > > Any immediate response will be appreciated, > > > > > > > > > Thanks, > > > Abid > > > > > > > > > > > -- > > Paul Fremantle > > Co-Founder and CTO, WSO2 > > Apache Synapse PMC Chair > > OASIS WS-RX TC Co-chair > > > > blog: http://pzf.fremantle.org > > [email protected] > > > > "Oxygenating the Web Service Platform", www.wso2.com > > > > > > -- > Hiranya Jayathilaka > Senior Software Engineer; > WSO2 Inc.; http://wso2.org > E-mail: [email protected]; Mobile: +94 77 633 3491 > Blog: http://techfeast-hiranya.blogspot.com >
