I think there is a possibility of leakage of message contexts in *none* normal flows, *depending* on the configuration. It is subjective and most of the time a configuration tune up can be done to overcome this situation. So you better try to re-produce the problem with a fragment of the configuration and share it with us so that we can have a look into possible configuration tuneups.
Ruwan On Sat, May 21, 2011 at 11:01 PM, Hiranya Jayathilaka <[email protected]>wrote: > On Sat, May 21, 2011 at 1:32 AM, Andreas Veithen > <[email protected]>wrote: > > > On Thu, May 19, 2011 at 20:09, Hiranya Jayathilaka <[email protected] > > > > wrote: > > > Synapse creates an Axis2MessageContext for each message received. Each > > > Axis2MessageContext instance encapsulates a MessageContext object. But > > > Synapse does not keep them in memory for too long. As soon as messages > > are > > > mediated and sent to the corresponding endpoints/clients, they are > > > discarded. > > > > Not sure that this is entirely correct. In a request-response > > exchange, doesn't the callback keep a reference to the outgoing > > message context? > > > It does. I was speaking about the general case. When Synapse receives the > response from the backend server, the callback will get removed and the > message context instances will be cleaned up due course. So for normal HTTP > invocations things should get cleaned up within a matter of seconds at > most. > > > > What I've seen in the past is that some configuration > > problems may cause Synapse to believe that a message exchange is > > request-response while the outgoing transport is actually doing > > out-only. In that case, the callback handlers just continue to pile up > > in memory until they expire. > > > This could happen due to a configuration error. For instance if you haven't > set the OUT_ONLY property on a flow where you should be setting it. > > > > IIRC, at the time I observed this, the > > default timeout was 24h. > > > > Abid's observation that this only occurs for failures may point to the > > same root cause, i.e. callbacks not being removed. > > > One possible cause is that callbacks may have accumulated for all the > messages sent to the wrong endpoint. However if endpoint suspension is not > disabled that should guard against this. From Abid's description it looks > like suspension is disabled. Otherwise the very first error should have > suspended the endpoint and subsequent messages would not have been sent. > > Thanks, > Hiranya > > > > > > > These objects could be fairly big since they contain message > > > payload, properties and all other context information of messages. So > if > > > Synapse is keeping them in memory forever, we won't be able to run any > > > scenario with Synapse for more than several hours. But as Paul > mentioned > > we > > > have seen some pretty heavy duty deployments of Synapse dealing with > > > hundreds of transactions every second. One of the key strengths of > > Synapse > > > is that it can handle very high volumes of messages with a very small > > memory > > > footprint. > > > > > > How much memory have you allocated for JBoss server? Also what is your > > > Synapse configuration like? What's the memory usage pattern when you > run > > > Synapse standalone (ie without JBoss)? > > > > > > Thanks, > > > Hiranya > > > > > > On Thu, May 19, 2011 at 9:17 PM, Abid Khan-EXT <[email protected] > > >wrote: > > > > > >> Hello Paul, > > >> > > >> What you are telling me is a good news. The facts I have here, are > > >> different. I ran the server for a load-test and ran only 26000 message > > (of > > >> about 1K each message size), that made the GC threads taking over > every > > >> thing, the java process was so stuck that the jboss-shutdown command > > could > > >> not bring it down. > > >> > > >> And my trace log (memory profiling) showed that one instance of both > > >> MessageContext and Axis2MessageContext were leaked for every message > > that is > > >> delivered/processed. > > >> > > >> I am using 1.2 GA of synapse. And I am running it in JBoss. > > >> > > >> And will be more than happy to send the thread dump, but that is very > > big. > > >> > > >> Once again thanks for getting back to me and I am looking for good > > pointers > > >> from you very soon. > > >> > > >> Regards, > > >> Abid > > >> > > >> > > >> > > >> > > >> -----Original Message----- > > >> From: Paul Fremantle [mailto:[email protected]] > > >> Sent: Wednesday, May 18, 2011 8:39 PM > > >> To: [email protected] > > >> Subject: Re: Please help: does Synapse leak memory? > > >> > > >> Abid > > >> > > >> Can you give us some idea of the flows? I know that Synapse is usually > > >> memory safe - I have thread dumps from people who have pumped 134 > > >> terabytes of data, 1.4 billion messages and all this with a max heap > > >> size of <2Gb and no leaks. > > >> > > >> Paul > > >> > > >> On Thu, May 19, 2011 at 1:49 AM, Abid Khan-EXT < > [email protected]> > > >> wrote: > > >> > Hello all, > > >> > > > >> > I am troubleshooting an instance where I had used Synapse, the issue > > is > > >> that after delivering few thousand message server runs out of memory. > > >> > > > >> > I investigated and found out that for each message that is delivered > > >> Synapse is leaking following two objects: > > >> > > > >> > 1) org.apache.axis2.context.MessageContext > > >> > > > >> > 2) org.apache.synapse.core.axis2.Axis2MessageContext > > >> > > > >> > I thought that it might be related to threads and ThreadLocal, but > > when I > > >> scanned the thread local of all 160+ threads those objects were not > > there. > > >> > > > >> > It seems that synapse is keeping those two object in its some > internal > > >> object, that is difficult to find. > > >> > > > >> > Because of this problem those objects; server leaks memory at very > > high > > >> rate, and freezes after few hours of running? > > >> > > > >> > Any immediate response will be appreciated, > > >> > > > >> > > > >> > Thanks, > > >> > Abid > > >> > > > >> > > >> > > >> > > >> -- > > >> Paul Fremantle > > >> Co-Founder and CTO, WSO2 > > >> Apache Synapse PMC Chair > > >> OASIS WS-RX TC Co-chair > > >> > > >> blog: http://pzf.fremantle.org > > >> [email protected] > > >> > > >> "Oxygenating the Web Service Platform", www.wso2.com > > >> > > > > > > > > > > > > -- > > > Hiranya Jayathilaka > > > Senior Software Engineer; > > > WSO2 Inc.; http://wso2.org > > > E-mail: [email protected]; Mobile: +94 77 633 3491 > > > Blog: http://techfeast-hiranya.blogspot.com > > > > > > > > > -- > Hiranya Jayathilaka > Senior Software Engineer; > WSO2 Inc.; http://wso2.org > E-mail: [email protected]; Mobile: +94 77 633 3491 > Blog: http://techfeast-hiranya.blogspot.com > -- Ruwan Linton Member, Apache Software Foundation; http://www.apache.org <http://wso2.org> phone: +94 11 282 7532 email: [email protected]; cell: +94 77 341 3097 blog: http://blog.ruwan.org linkedin: http://www.linkedin.com/in/ruwanlinton google: http://www.google.com/profiles/ruwan.linton tweet: http://twitter.com/ruwanlinton
