I'm still battling this PermGen leak and frankly I'm really starting to
doubt that I know what I'm doing anymore. I'd be very happy if anyone
would care to explain that to me...
Since my last post Scott and I have discussed potential class loader
leaks and some of them have been fixed in the 3.1.8 release. It seems
there is (at least) one leak that didn't get fixed in 3.1.8. I have made
a quick and dirty patch to avoid that leak. If anyone would care to try,
the patch (which includes a few other things probably fixed in 3.1.8
already) can be found here:
However, even with that patch, it seems there is still some kind of
PermGen leak that eventually leads to OutOfMemoryError. I have created a
small application with the sole purpose of detecting these leaks. If
anyone would care to try, it can be found here (sources included):
You will need to add some JARs to the WEB-INF/lib directory; preferrably
a couple of large ones like spring.jar and hibernate.jar (don't use
Resin JARs though).
Then just drop the WAR in a clean installation of Resin 3.1.8
(preferrably patched with the patch above).
Hit http://...:nn/leak (once is enough)
Force a redeploy by either deleting the webapps/leak dir or touch:ing
Hit http://...:nn/leak again
Repeat the last two steps for as long as you'd like
What you should see - or at least what I see on one Linux machine and
one Windows machine - is the (ClassLoadingMXBean) loadedClassCount and
the (MemoryPoolMXBean) Used Perm Gen steadily increasing (while the
unloadedClassCount remains pretty stable) for every redeploy, which
indicates a classloader leak. But I just can't find that leak.
Now, here are the things really bugging me:
1. If I keep redeploying over and over, I will eventually get closer and
closer to the Perm Gen Max (in some instances, I have seen the following
behaivour instead turn up when Used reaches Init if Init is large
enough). Then suddenly the unloadedClassCount is increased, but not with
all the unused classes - only about the amount of one redeployment.
Redeploy again, and it will increase another step. Meanwhile, the
loadedClassCount remains pretty stable, since we are loading as many new
classes as are unloaded. It's as if there was a FIFO queue/LRU cache of
classloaders, so that the oldest one is garbage collected once there is
not enough space for a new one. However, after a while there is
(assumably) not enough space to create the new classloader before the
old one is garbage collected, and I get OutOfMemoryError somewhere in
the middle. Sometimes I am actually able to recover from this error by
waiting for the GC to do it's job and then just try again.
2. Now I attached YourKit, looking for dangling classloaders as of the
inital post. I found none. In fact, the Classes without Instances
inspection only shows the classes in the added JARs from the last
redeployment, so when tracing back to GC root, it goes via the current
EnvironmentClassLoader which is correct. There are also no excessive
instances of EnvironmentClassLoader. Hmm... Now wait a minute. Look at
the total number of java.lang.Class objects. It does not match with the
totalLoadedClassCount. In fact, the total number of classes found by
YourKit is about the same as the totalLoadedClassCount on the very first
hit of the application, before any redeployments. So from YourKits point
of view, there is no classloader leak! But why then isn't the PermGen
space reclaimed. This led me to wonder if there was some kind of JVM
bug. (As a side note, I have yet to try with some other profiler)
3. So, I modified the test application (see commented out code in
MyServlet.java) to load the classes of the JARs in a regular
java.net.URLClassLoader which is then immediately thrown away. No leak.
loadedClassCount is immediately decreased (and unloadedClassCount
increased), as is the Used Perm Gen. That is, it behave the way we want
the redeployment to behave.
Ok, lets take one step down in the classloader hierachy and load all the
classes via a disposable com.caucho.loader.DynamicClassLoader. No leak.
So, what if I load them with a com.caucho.loader.EnvironmentClassLoader
which is then destroy()ed and left for garbage collection. The leak is
back! EnvironmentClassLoader has now applied to become prime suspect. In
order to track things down I subclassed EnvironmentClassLoader inside my
application and planned to make changes there. Well ... before any
changes, the leak is nowhere to be found.
What is going on here?????
Anything that might help my sanity would be appreciated.
Mattias Jiderhamn wrote (2008-11-05 06:43):
> In support of this latest theory is the fact that YourKit shows two GC
> roots for the HttpRequest. Apart from the
> com.caucho.server.port.TcpConnection._request reference, the request
> is a root itself by being on the stack of a thread (http--8080-...).
> This could indicate a thread currently waiting in the
> com.caucho.server.port.TcpConnection.run() method, where the
> ServerRequest of the parent is also a local variable.
> Even if the request is reused, I believe the Invocation or the
> ClassLoader of the invocation needs to be reset somehow, to release
> the webapp classloader.
> I can help try out a proposed fix to see if it solves the problem (=
> feel free to mail me off list). I might even give it a shot myself.
resin-interest mailing list