On Sat, May 21, 2011 at 1:32 AM, Andreas Veithen <[email protected]>wrote:
> On Thu, May 19, 2011 at 20:09, Hiranya Jayathilaka <[email protected]> > wrote: > > Synapse creates an Axis2MessageContext for each message received. Each > > Axis2MessageContext instance encapsulates a MessageContext object. But > > Synapse does not keep them in memory for too long. As soon as messages > are > > mediated and sent to the corresponding endpoints/clients, they are > > discarded. > > Not sure that this is entirely correct. In a request-response > exchange, doesn't the callback keep a reference to the outgoing > message context? It does. I was speaking about the general case. When Synapse receives the response from the backend server, the callback will get removed and the message context instances will be cleaned up due course. So for normal HTTP invocations things should get cleaned up within a matter of seconds at most. > What I've seen in the past is that some configuration > problems may cause Synapse to believe that a message exchange is > request-response while the outgoing transport is actually doing > out-only. In that case, the callback handlers just continue to pile up > in memory until they expire. This could happen due to a configuration error. For instance if you haven't set the OUT_ONLY property on a flow where you should be setting it. > IIRC, at the time I observed this, the > default timeout was 24h. > > Abid's observation that this only occurs for failures may point to the > same root cause, i.e. callbacks not being removed. One possible cause is that callbacks may have accumulated for all the messages sent to the wrong endpoint. However if endpoint suspension is not disabled that should guard against this. From Abid's description it looks like suspension is disabled. Otherwise the very first error should have suspended the endpoint and subsequent messages would not have been sent. Thanks, Hiranya > > > These objects could be fairly big since they contain message > > payload, properties and all other context information of messages. So if > > Synapse is keeping them in memory forever, we won't be able to run any > > scenario with Synapse for more than several hours. But as Paul mentioned > we > > have seen some pretty heavy duty deployments of Synapse dealing with > > hundreds of transactions every second. One of the key strengths of > Synapse > > is that it can handle very high volumes of messages with a very small > memory > > footprint. > > > > How much memory have you allocated for JBoss server? Also what is your > > Synapse configuration like? What's the memory usage pattern when you run > > Synapse standalone (ie without JBoss)? > > > > Thanks, > > Hiranya > > > > On Thu, May 19, 2011 at 9:17 PM, Abid Khan-EXT <[email protected] > >wrote: > > > >> Hello Paul, > >> > >> What you are telling me is a good news. The facts I have here, are > >> different. I ran the server for a load-test and ran only 26000 message > (of > >> about 1K each message size), that made the GC threads taking over every > >> thing, the java process was so stuck that the jboss-shutdown command > could > >> not bring it down. > >> > >> And my trace log (memory profiling) showed that one instance of both > >> MessageContext and Axis2MessageContext were leaked for every message > that is > >> delivered/processed. > >> > >> I am using 1.2 GA of synapse. And I am running it in JBoss. > >> > >> And will be more than happy to send the thread dump, but that is very > big. > >> > >> Once again thanks for getting back to me and I am looking for good > pointers > >> from you very soon. > >> > >> Regards, > >> Abid > >> > >> > >> > >> > >> -----Original Message----- > >> From: Paul Fremantle [mailto:[email protected]] > >> Sent: Wednesday, May 18, 2011 8:39 PM > >> To: [email protected] > >> Subject: Re: Please help: does Synapse leak memory? > >> > >> Abid > >> > >> Can you give us some idea of the flows? I know that Synapse is usually > >> memory safe - I have thread dumps from people who have pumped 134 > >> terabytes of data, 1.4 billion messages and all this with a max heap > >> size of <2Gb and no leaks. > >> > >> Paul > >> > >> On Thu, May 19, 2011 at 1:49 AM, Abid Khan-EXT <[email protected]> > >> wrote: > >> > Hello all, > >> > > >> > I am troubleshooting an instance where I had used Synapse, the issue > is > >> that after delivering few thousand message server runs out of memory. > >> > > >> > I investigated and found out that for each message that is delivered > >> Synapse is leaking following two objects: > >> > > >> > 1) org.apache.axis2.context.MessageContext > >> > > >> > 2) org.apache.synapse.core.axis2.Axis2MessageContext > >> > > >> > I thought that it might be related to threads and ThreadLocal, but > when I > >> scanned the thread local of all 160+ threads those objects were not > there. > >> > > >> > It seems that synapse is keeping those two object in its some internal > >> object, that is difficult to find. > >> > > >> > Because of this problem those objects; server leaks memory at very > high > >> rate, and freezes after few hours of running? > >> > > >> > Any immediate response will be appreciated, > >> > > >> > > >> > Thanks, > >> > Abid > >> > > >> > >> > >> > >> -- > >> Paul Fremantle > >> Co-Founder and CTO, WSO2 > >> Apache Synapse PMC Chair > >> OASIS WS-RX TC Co-chair > >> > >> blog: http://pzf.fremantle.org > >> [email protected] > >> > >> "Oxygenating the Web Service Platform", www.wso2.com > >> > > > > > > > > -- > > Hiranya Jayathilaka > > Senior Software Engineer; > > WSO2 Inc.; http://wso2.org > > E-mail: [email protected]; Mobile: +94 77 633 3491 > > Blog: http://techfeast-hiranya.blogspot.com > > > -- Hiranya Jayathilaka Senior Software Engineer; WSO2 Inc.; http://wso2.org E-mail: [email protected]; Mobile: +94 77 633 3491 Blog: http://techfeast-hiranya.blogspot.com
