Hi Stan, Thanks for reporting and your patch.
Will look into it when I get some freetime. For your question about building ODE with axis2 1.6, it's not possible at the moment as it needs some changes in the Axiom usage within ODE. Will see if we can bring it on 1.3.7 release, though the timelines are not fixed yet. regards, sathwik On Mon, Jul 14, 2014 at 10:15 PM, Stan Angeloff <stani...@angeloff.name> wrote: > For those interested, I ended up patching Axis2 1.5.6 from source with > > https://gist.github.com/StanAngeloff/e245a1a349d04efcfad6#file-axis2-1-5-6-cleanup-patch > The patch has been adapted from two commits on the 1_6 branch. I > re-compiled the kernel module and swapped the one that comes with the > binary distribution of ODE. This seems to have resolved the issue for me. > > I also found https://issues.apache.org/jira/browse/ODE-585, however it's > tagged for 2.0. IMHO memory leaks are a pretty major issue and should get > addressed in whatever is the current stable release? > > > On 14 July 2014 14:31, Stan Angeloff <stani...@angeloff.name> wrote: > > > *tl;dr;* Is it possible to use Axis 1.6 in ODE? How does one go about > > building ODE and bundling it with Axis 1.6? > > > > Digging a bit deeper in the log files with > > log4j.category.org.apache.axis2=DEBUG and > > log4j.category.org.apache.ode=DEBUG configured yields interesting > results. > > When a ServiceClient instance is created, the second parameter is NULL. > > When this is the case, Axis2 will create a new anonymous service ( > > > https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L137 > ). > > The anonymous service is then added to the AxisConfiguration instance ( > > > https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L176 > > ). > > > > Internally, this creates a new AxisServiceGroup ( > > > https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L315 > ). > > This then adds the new anonymous service to an allEndpoints property ( > > > https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L373 > > ). > > > > When ODE calls setAxisService, Axis2 clean up the service ( > > > https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L612 > ). > > It does not, however, clean up the reference from allEndpoints. > > > > The filtered catalina.out > > < > https://gist.githubusercontent.com/StanAngeloff/9f1798e34963012bc190/raw/catalina.out.filtered > > > > log is attached for reference. Analysing the heap dumps indicates that > > allEndpoints is indeed the culprit. > > > > Axis 1.6 seems to have a fix for this issue: > > > > > https://github.com/apache/axis2-java/blob/1_6/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L679 > > > > Is it possible to use Axis 1.6 in ODE? How does one go about building ODE > > and bundling it with Axis 1.6? > > Should the ODE 1.3.x branch be updated to use the latest stable Axis2? > > > > Regards, > > Stan > > > > > > > > > > On 14 July 2014 13:45, Stan Angeloff <stani...@angeloff.name> wrote: > > > >> Hello, > >> > >> I've been playing with ODE for quite a while now and one of my most > >> recent attempts tried to run 5, 000 simultaneous instances of one > process. > >> Unfortunately after a while ODE uses up all of the allowed memory and > barfs > >> 'Java heap space' continuously in the log. > >> > >> My set up is very basic, starting from > >> https://github.com/vanto/apache-ode-tomcat-bundle (which has been an > >> excellent learning tool). The main .bpel file consists of a continuous > >> Repeat Until inside of which there are two sequential invoke activities > >> calling external SOAP services. > >> > >> When the 5, 000 instances are started, the memory footprint is fairly > >> stable. After a while, though, org.apache.axis2.engine.AxisConfiguration > >> accumulates a lot of org.apache.axis2.description.AxisService instances > >> (all of which have their name property set to 'anonService' + hash). > >> Those keep growing until eventually the server stops responding due to > >> exhausted memory. I have taken heap dumps at various stages and examined > >> those. Eclipse was even smart enough to detect the potential memory > leak on > >> the latest dump just as the server crashed. > >> > >> One instance of "org.apache.axis2.engine.AxisConfiguration" loaded by > >>> "org.apache.catalina.loader.WebappClassLoader @ 0xe07e53e0" occupies > >>> 390,800,064 (75.91%) bytes. The memory is accumulated in one instance > of > >>> "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "<system > class > >>> loader>". > >> > >> > >> I'll be happy to export the complete project, note however it doesn't > >> matter what services are invoked or what the .bpel does so long it's a) > >> continuous, e.g., does not stop b) calls an external service. > >> > >> I tried looking into ODE/Axis2 to try and figure out what might be going > >> wrong. My understanding is that service clients are cached and recycled > >> in SoapExternalService.java ( > >> > https://github.com/apache/ode/blob/master/axis2/src/main/java/org/apache/ode/axis2/SoapExternalService.java#L265 > ) > >> When a new ServiceClient instance is created, it calls the > >> configureServiceClient method within Axis2 with axisService = null ( > >> > https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L143 > ). > >> That in turn calls createAnonymousService. Later on, ODE calls > >> setAxisService which forces Axis2 to dispose of the anonymous service it > >> created earlier. > >> > >> If the above is indeed what is happening, then there should be no memory > >> leaks. However, in the memory dumps the number of instances of > >> org.apache.axis2.description.AxisService keeps growing. I've tried > >> increasing the maximum heap size up to 8GB and it will fail again, it is > >> just a matter of time. > >> > >> Has anyone else experienced this? Is anyone interested in trying to help > >> me narrow down the cause of this? Is it possible it's not even a memory > >> issue? > >> > >> Regards, > >> Stan > >> > > > > >
