*tl;dr;* Is it possible to use Axis 1.6 in ODE? How does one go about building ODE and bundling it with Axis 1.6?
Digging a bit deeper in the log files with log4j.category.org.apache.axis2=DEBUG and log4j.category.org.apache.ode=DEBUG configured yields interesting results. When a ServiceClient instance is created, the second parameter is NULL. When this is the case, Axis2 will create a new anonymous service ( https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L137). The anonymous service is then added to the AxisConfiguration instance ( https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L176 ). Internally, this creates a new AxisServiceGroup ( https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L315). This then adds the new anonymous service to an allEndpoints property ( https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L373 ). When ODE calls setAxisService, Axis2 clean up the service ( https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L612). It does not, however, clean up the reference from allEndpoints. The filtered catalina.out <https://gist.githubusercontent.com/StanAngeloff/9f1798e34963012bc190/raw/catalina.out.filtered> log is attached for reference. Analysing the heap dumps indicates that allEndpoints is indeed the culprit. Axis 1.6 seems to have a fix for this issue: https://github.com/apache/axis2-java/blob/1_6/modules/kernel/src/org/apache/axis2/engine/AxisConfiguration.java#L679 Is it possible to use Axis 1.6 in ODE? How does one go about building ODE and bundling it with Axis 1.6? Should the ODE 1.3.x branch be updated to use the latest stable Axis2? Regards, Stan On 14 July 2014 13:45, Stan Angeloff <stani...@angeloff.name> wrote: > Hello, > > I've been playing with ODE for quite a while now and one of my most recent > attempts tried to run 5, 000 simultaneous instances of one process. > Unfortunately after a while ODE uses up all of the allowed memory and barfs > 'Java heap space' continuously in the log. > > My set up is very basic, starting from > https://github.com/vanto/apache-ode-tomcat-bundle (which has been an > excellent learning tool). The main .bpel file consists of a continuous > Repeat Until inside of which there are two sequential invoke activities > calling external SOAP services. > > When the 5, 000 instances are started, the memory footprint is fairly > stable. After a while, though, org.apache.axis2.engine.AxisConfiguration > accumulates a lot of org.apache.axis2.description.AxisService instances > (all of which have their name property set to 'anonService' + hash). > Those keep growing until eventually the server stops responding due to > exhausted memory. I have taken heap dumps at various stages and examined > those. Eclipse was even smart enough to detect the potential memory leak on > the latest dump just as the server crashed. > > One instance of "org.apache.axis2.engine.AxisConfiguration" loaded by >> "org.apache.catalina.loader.WebappClassLoader @ 0xe07e53e0" occupies >> 390,800,064 (75.91%) bytes. The memory is accumulated in one instance of >> "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "<system class >> loader>". > > > I'll be happy to export the complete project, note however it doesn't > matter what services are invoked or what the .bpel does so long it's a) > continuous, e.g., does not stop b) calls an external service. > > I tried looking into ODE/Axis2 to try and figure out what might be going > wrong. My understanding is that service clients are cached and recycled > in SoapExternalService.java ( > https://github.com/apache/ode/blob/master/axis2/src/main/java/org/apache/ode/axis2/SoapExternalService.java#L265) > When a new ServiceClient instance is created, it calls the > configureServiceClient method within Axis2 with axisService = null ( > https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L143). > That in turn calls createAnonymousService. Later on, ODE calls > setAxisService which forces Axis2 to dispose of the anonymous service it > created earlier. > > If the above is indeed what is happening, then there should be no memory > leaks. However, in the memory dumps the number of instances of > org.apache.axis2.description.AxisService keeps growing. I've tried > increasing the maximum heap size up to 8GB and it will fail again, it is > just a matter of time. > > Has anyone else experienced this? Is anyone interested in trying to help > me narrow down the cause of this? Is it possible it's not even a memory > issue? > > Regards, > Stan >
