I gathered some data on the otest PermGen problems, last night. Would
take a fair amount of digging around for me to make any progress. So,
thought I'd drop a note to the list to see what ideas others have...
I used the java arg '-XX:+HeapDumpOnOutOfMemoryError' to generate a
heapdump, when the PermGen failure occurs. With the heap dump, you can
use a variety of post-mortem analysis tools to investigate the
problem. In this case, I used YourKit, which I really like. I recently
used the Eclipse Memory Analyzer (MAT) plugin and found it worked
pretty well.
For PermGen failures, I always start looking for ClassLoader memory
leaks. PermGen failures with a relatively finite number of
ClassLoaders typically means you need to bump your max permgen setting.
In my heapdump, there are 1760
o.a.t.sca.interfacedef.java.jaxws.GeneratedClassLoader instances. I
don't see any ClassLoaders holding onto many/any Classes, but just the
number of ClassLoader instances seems pretty excessive to me. I doubt
that would be expected... Most/all of them are kept alive by a chain
of references similar to the following (formatting might not survive
email processing, very well):
Paths from "GC Roots" to "Object GeneratedClassLoader #00000001057f4be0"
+---
org.apache.tuscany.sca.interfacedef.java.jaxws.GeneratedClassLoader
|
+---classLoader of
org.apache.tuscany.sca.interfacedef.java.jaxws.GeneratedDataTypeImpl
|
+---inputWrapperType of
org.apache.tuscany.sca.interfacedef.util.WrapperInfo
|
+---wrapper of
org.apache.tuscany.sca.interfacedef.java.impl.JavaOperationImpl
|
+---[0] of java.lang.Object[10]
|
+---elementData of
org.apache.tuscany.sca.interfacedef.impl.InterfaceImpl$OperationList
|
+---operations of
org.apache.tuscany.sca.interfacedef.java.impl.JavaInterfaceImpl
|
+---value of java.util.HashMap$Entry
|
+---[227] of java.util.HashMap$Entry[256]
|
+---table of java.util.HashMap
|
+---map of
org.apache.tuscany.sca.contribution.resolver.ExtensibleModelResolver
|
+---modelResolver of
org.apache.tuscany.sca.core.assembly.impl.RuntimeComponentImpl
|
+---component of
org.apache.tuscany.sca.core.assembly.impl.RuntimeComponentReferenceImpl
|
+---reference of
org.apache.tuscany.sca.core.assembly.impl.RuntimeEndpointReferenceImpl
|
+---[60] of java.lang.Object[88]
|
+---elementData of java.util.ArrayList
|
+---endpointreferences of
org.apache.tuscany.sca.endpoint.impl.EndpointRegistryImpl
|
+---[855] of java.lang.Object[5120]
|
+---elementData of
java.util.Vector
|
+---classes of
sun.misc.Launcher$AppClassLoader [Stack Local]
My guess would be that EndpointRegistryImpl.endpointreferences would
be the place to break this chain...
I also noted that o.a.t.sca.core.assembly.impl.RuntimeComponentImpl
instances are holding onto a lot of memory. There are over 14000
objects retaining over 115 meg of heap. They have a pretty similar
chain of references. They, again, seem to be rooted from
EndpointRegistryImpl:
Paths from "GC Roots" to "Object RuntimeComponentImpl #000000010f27adc8"
+---org.apache.tuscany.sca.core.assembly.impl.RuntimeComponentImpl
|
+---[0] of java.lang.Object[10]
|
+---elementData of java.util.ArrayList
|
+---components of
org.apache.tuscany.sca.assembly.impl.CompositeImpl
|
+---model of
org.apache.tuscany.sca.contribution.impl.ArtifactImpl
|
+---[108] of java.lang.Object[133]
|
+---elementData of java.util.ArrayList
|
+---artifacts of
org.apache.tuscany.sca.contribution.impl.ContributionImpl
|
+---[1] of java.lang.Object[10]
|
+---elementData of java.util.ArrayList
|
+---dependencies of
org.apache.tuscany.sca.contribution.impl.ContributionImpl
|
+---contribution of
org.apache.tuscany.sca.contribution.resolver.ExtensibleModelResolver
|
+---modelResolver of
org.apache.tuscany.sca.core.assembly.impl.RuntimeComponentImpl
|
+---component of
org.apache.tuscany.sca.core.assembly.impl.RuntimeEndpointReferenceImpl
|
+---[34] of java.lang.Object[88]
|
+---elementData of java.util.ArrayList
|
+---endpointreferences of
org.apache.tuscany.sca.endpoint.impl.EndpointRegistryImpl
|
+---[855] of java.lang.Object[5120]
|
+---elementData of
java.util.Vector
|
+---classes of
sun.misc.Launcher$AppClassLoader [Stack Local]
Let me know if you have any thoughts on the above. Afraid I'm not
going to have anytime to look at this until later today/this evening.
--kevan
