Yes please, a sample configuration to re-produce the issue would be great. Rajika
On Fri, May 20, 2011 at 9:43 PM, Abid Khan-EXT <[email protected]>wrote: > The configuration that I have is huge, not only in number of mediators and > sequences, but I am using lot of class mediators. That makes the > configuration un useful for any who does not have the code. > > What I will do is that I will try to reproduce it with synapse server and > post the results here. > > Thanks, > Abid > > > > -----Original Message----- > From: Rajika Kumarasiri [mailto:[email protected]] > Sent: Thursday, May 19, 2011 11:22 PM > To: [email protected] > Subject: Re: Please help: does Synapse leak memory? > > Can you attach your Synapse configuration if possible? And tell which > endpoint has a typo ? > > Rajika > > On Thu, May 19, 2011 at 11:47 PM, Abid Khan-EXT <[email protected] > >wrote: > > > Thanks Hiranya, > > > > I have done some more research, and found out the it leaks memory only on > > the failures. Not on successful deliver. > > > > To give you the back ground, I had a typo in one of the endpoint URL. And > I > > had only two endpoints, all the message that were attempted to be > delivered > > to that "wrong" url; failed. That is what lead to this leaking issue. > > > > On the other hand the server I was testing had -Xms1024m, and -Xmx2048m. > So > > its little better than what I was thinking. > > > > I am still digging into and trying to find where the leak is. I will > update > > if I found anything but so far it's a leak on the failures only. > > > > If you can find out if this is correct that Synapse could leak those two > > object on failures. That will help. > > > > Thanks, > > Abid > > > > > > > > > > -----Original Message----- > > From: Hiranya Jayathilaka [mailto:[email protected]] > > Sent: Thursday, May 19, 2011 12:09 PM > > To: [email protected] > > Subject: Re: Please help: does Synapse leak memory? > > > > Synapse creates an Axis2MessageContext for each message received. Each > > Axis2MessageContext instance encapsulates a MessageContext object. But > > Synapse does not keep them in memory for too long. As soon as messages > are > > mediated and sent to the corresponding endpoints/clients, they are > > discarded. These objects could be fairly big since they contain message > > payload, properties and all other context information of messages. So if > > Synapse is keeping them in memory forever, we won't be able to run any > > scenario with Synapse for more than several hours. But as Paul mentioned > we > > have seen some pretty heavy duty deployments of Synapse dealing with > > hundreds of transactions every second. One of the key strengths of > Synapse > > is that it can handle very high volumes of messages with a very small > > memory > > footprint. > > > > How much memory have you allocated for JBoss server? Also what is your > > Synapse configuration like? What's the memory usage pattern when you run > > Synapse standalone (ie without JBoss)? > > > > Thanks, > > Hiranya > > > > On Thu, May 19, 2011 at 9:17 PM, Abid Khan-EXT <[email protected] > > >wrote: > > > > > Hello Paul, > > > > > > What you are telling me is a good news. The facts I have here, are > > > different. I ran the server for a load-test and ran only 26000 message > > (of > > > about 1K each message size), that made the GC threads taking over every > > > thing, the java process was so stuck that the jboss-shutdown command > > could > > > not bring it down. > > > > > > And my trace log (memory profiling) showed that one instance of both > > > MessageContext and Axis2MessageContext were leaked for every message > that > > is > > > delivered/processed. > > > > > > I am using 1.2 GA of synapse. And I am running it in JBoss. > > > > > > And will be more than happy to send the thread dump, but that is very > > big. > > > > > > Once again thanks for getting back to me and I am looking for good > > pointers > > > from you very soon. > > > > > > Regards, > > > Abid > > > > > > > > > > > > > > > -----Original Message----- > > > From: Paul Fremantle [mailto:[email protected]] > > > Sent: Wednesday, May 18, 2011 8:39 PM > > > To: [email protected] > > > Subject: Re: Please help: does Synapse leak memory? > > > > > > Abid > > > > > > Can you give us some idea of the flows? I know that Synapse is usually > > > memory safe - I have thread dumps from people who have pumped 134 > > > terabytes of data, 1.4 billion messages and all this with a max heap > > > size of <2Gb and no leaks. > > > > > > Paul > > > > > > On Thu, May 19, 2011 at 1:49 AM, Abid Khan-EXT <[email protected] > > > > > wrote: > > > > Hello all, > > > > > > > > I am troubleshooting an instance where I had used Synapse, the issue > is > > > that after delivering few thousand message server runs out of memory. > > > > > > > > I investigated and found out that for each message that is delivered > > > Synapse is leaking following two objects: > > > > > > > > 1) org.apache.axis2.context.MessageContext > > > > > > > > 2) org.apache.synapse.core.axis2.Axis2MessageContext > > > > > > > > I thought that it might be related to threads and ThreadLocal, but > when > > I > > > scanned the thread local of all 160+ threads those objects were not > > there. > > > > > > > > It seems that synapse is keeping those two object in its some > internal > > > object, that is difficult to find. > > > > > > > > Because of this problem those objects; server leaks memory at very > high > > > rate, and freezes after few hours of running? > > > > > > > > Any immediate response will be appreciated, > > > > > > > > > > > > Thanks, > > > > Abid > > > > > > > > > > > > > > > > -- > > > Paul Fremantle > > > Co-Founder and CTO, WSO2 > > > Apache Synapse PMC Chair > > > OASIS WS-RX TC Co-chair > > > > > > blog: http://pzf.fremantle.org > > > [email protected] > > > > > > "Oxygenating the Web Service Platform", www.wso2.com > > > > > > > > > > > -- > > Hiranya Jayathilaka > > Senior Software Engineer; > > WSO2 Inc.; http://wso2.org > > E-mail: [email protected]; Mobile: +94 77 633 3491 > > Blog: http://techfeast-hiranya.blogspot.com > > >
