Hi Stan, Would like your help in replicating this issue.
I have devised a process which calls an external service. I have a continuous looping call to the process from the soapUI. I have run about 4000 instances a couple of times, and I don't see AxisService Instances going beyond 110 as my threads per host is equal to 100. regards, sathwik On Tue, Jul 15, 2014 at 12:59 PM, Sathwik B P <sathwik...@gmail.com> wrote: > Hi Stan, > > Thanks for reporting and your patch. > > Will look into it when I get some freetime. > > For your question about building ODE with axis2 1.6, it's not possible at > the moment as it needs some changes in the Axiom usage within ODE. > > Will see if we can bring it on 1.3.7 release, though the timelines are not > fixed yet. > > regards, > sathwik > > > On Mon, Jul 14, 2014 at 10:15 PM, Stan Angeloff <stani...@angeloff.name> > wrote: > >> For those interested, I ended up patching Axis2 1.5.6 from source with >> >> https://gist.github.com/StanAngeloff/e245a1a349d04efcfad6#file-axis2-1-5-6-cleanup-patch >> The patch has been adapted from two commits on the 1_6 branch. I >> re-compiled the kernel module and swapped the one that comes with the >> binary distribution of ODE. This seems to have resolved the issue for me. >> >> I also found https://issues.apache.org/jira/browse/ODE-585, however it's >> tagged for 2.0. IMHO memory leaks are a pretty major issue and should get >> addressed in whatever is the current stable release? >> >> >> On 14 July 2014 14:31, Stan Angeloff <stani...@angeloff.name> wrote: >> >> > *tl;dr;* Is it possible to use Axis 1.6 in ODE? How does one go about >> > building ODE and bundling it with Axis 1.6? >> > >> > Digging a bit deeper in the log files with >> > log4j.category.org.apache.axis2=DEBUG and >> > log4j.category.org.apache.ode=DEBUG configured yields interesting >> results. >> > When a ServiceClient instance is created, the second parameter is NULL. >> > When this is the case, Axis2 will create a new anonymous service ( >> > >> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L137 >> ). >> > The anonymous service is then added to the AxisConfiguration instance ( >> > >> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L176 >> > ). >> > >> > Internally, this creates a new AxisServiceGroup ( >> > >> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L315 >> ). >> > This then adds the new anonymous service to an allEndpoints property ( >> > >> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L373 >> > ). >> > >> > When ODE calls setAxisService, Axis2 clean up the service ( >> > >> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L612 >> ). >> > It does not, however, clean up the reference from allEndpoints. >> > >> > The filtered catalina.out >> > < >> https://gist.githubusercontent.com/StanAngeloff/9f1798e34963012bc190/raw/catalina.out.filtered >> > >> > log is attached for reference. Analysing the heap dumps indicates that >> > allEndpoints is indeed the culprit. >> > >> > Axis 1.6 seems to have a fix for this issue: >> > >> > >> https://github.com/apache/axis2-java/blob/1_6/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L679 >> > >> > Is it possible to use Axis 1.6 in ODE? How does one go about building >> ODE >> > and bundling it with Axis 1.6? >> > Should the ODE 1.3.x branch be updated to use the latest stable Axis2? >> > >> > Regards, >> > Stan >> > >> > >> > >> > >> > On 14 July 2014 13:45, Stan Angeloff <stani...@angeloff.name> wrote: >> > >> >> Hello, >> >> >> >> I've been playing with ODE for quite a while now and one of my most >> >> recent attempts tried to run 5, 000 simultaneous instances of one >> process. >> >> Unfortunately after a while ODE uses up all of the allowed memory and >> barfs >> >> 'Java heap space' continuously in the log. >> >> >> >> My set up is very basic, starting from >> >> https://github.com/vanto/apache-ode-tomcat-bundle (which has been an >> >> excellent learning tool). The main .bpel file consists of a continuous >> >> Repeat Until inside of which there are two sequential invoke activities >> >> calling external SOAP services. >> >> >> >> When the 5, 000 instances are started, the memory footprint is fairly >> >> stable. After a while, though, >> org.apache.axis2.engine.AxisConfiguration >> >> accumulates a lot of org.apache.axis2.description.AxisService instances >> >> (all of which have their name property set to 'anonService' + hash). >> >> Those keep growing until eventually the server stops responding due to >> >> exhausted memory. I have taken heap dumps at various stages and >> examined >> >> those. Eclipse was even smart enough to detect the potential memory >> leak on >> >> the latest dump just as the server crashed. >> >> >> >> One instance of "org.apache.axis2.engine.AxisConfiguration" loaded by >> >>> "org.apache.catalina.loader.WebappClassLoader @ 0xe07e53e0" occupies >> >>> 390,800,064 (75.91%) bytes. The memory is accumulated in one instance >> of >> >>> "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "<system >> class >> >>> loader>". >> >> >> >> >> >> I'll be happy to export the complete project, note however it doesn't >> >> matter what services are invoked or what the .bpel does so long it's a) >> >> continuous, e.g., does not stop b) calls an external service. >> >> >> >> I tried looking into ODE/Axis2 to try and figure out what might be >> going >> >> wrong. My understanding is that service clients are cached and recycled >> >> in SoapExternalService.java ( >> >> >> https://github.com/apache/ode/blob/master/axis2/src/main/java/org/apache/ode/axis2/SoapExternalService.java#L265 >> ) >> >> When a new ServiceClient instance is created, it calls the >> >> configureServiceClient method within Axis2 with axisService = null ( >> >> >> https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L143 >> ). >> >> That in turn calls createAnonymousService. Later on, ODE calls >> >> setAxisService which forces Axis2 to dispose of the anonymous service >> it >> >> created earlier. >> >> >> >> If the above is indeed what is happening, then there should be no >> memory >> >> leaks. However, in the memory dumps the number of instances of >> >> org.apache.axis2.description.AxisService keeps growing. I've tried >> >> increasing the maximum heap size up to 8GB and it will fail again, it >> is >> >> just a matter of time. >> >> >> >> Has anyone else experienced this? Is anyone interested in trying to >> help >> >> me narrow down the cause of this? Is it possible it's not even a memory >> >> issue? >> >> >> >> Regards, >> >> Stan >> >> >> > >> > >> > >
