Hello, I've been playing with ODE for quite a while now and one of my most recent attempts tried to run 5, 000 simultaneous instances of one process. Unfortunately after a while ODE uses up all of the allowed memory and barfs 'Java heap space' continuously in the log.
My set up is very basic, starting from https://github.com/vanto/apache-ode-tomcat-bundle (which has been an excellent learning tool). The main .bpel file consists of a continuous Repeat Until inside of which there are two sequential invoke activities calling external SOAP services. When the 5, 000 instances are started, the memory footprint is fairly stable. After a while, though, org.apache.axis2.engine.AxisConfiguration accumulates a lot of org.apache.axis2.description.AxisService instances (all of which have their name property set to 'anonService' + hash). Those keep growing until eventually the server stops responding due to exhausted memory. I have taken heap dumps at various stages and examined those. Eclipse was even smart enough to detect the potential memory leak on the latest dump just as the server crashed. One instance of "org.apache.axis2.engine.AxisConfiguration" loaded by > "org.apache.catalina.loader.WebappClassLoader @ 0xe07e53e0" occupies > 390,800,064 (75.91%) bytes. The memory is accumulated in one instance of > "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "<system class > loader>". I'll be happy to export the complete project, note however it doesn't matter what services are invoked or what the .bpel does so long it's a) continuous, e.g., does not stop b) calls an external service. I tried looking into ODE/Axis2 to try and figure out what might be going wrong. My understanding is that service clients are cached and recycled in SoapExternalService.java ( https://github.com/apache/ode/blob/master/axis2/src/main/java/org/apache/ode/axis2/SoapExternalService.java#L265) When a new ServiceClient instance is created, it calls the configureServiceClient method within Axis2 with axisService = null ( https://github.com/apache/axis2-java/blob/1_5/modules/kernel/src/org/apache/axis2/client/ServiceClient.java#L143). That in turn calls createAnonymousService. Later on, ODE calls setAxisService which forces Axis2 to dispose of the anonymous service it created earlier. If the above is indeed what is happening, then there should be no memory leaks. However, in the memory dumps the number of instances of org.apache.axis2.description.AxisService keeps growing. I've tried increasing the maximum heap size up to 8GB and it will fail again, it is just a matter of time. Has anyone else experienced this? Is anyone interested in trying to help me narrow down the cause of this? Is it possible it's not even a memory issue? Regards, Stan
