I'm still battling this PermGen leak and frankly I'm really starting to doubt that I know what I'm doing anymore. I'd be very happy if anyone would care to explain that to me...
Since my last post Scott and I have discussed potential class loader leaks and some of them have been fixed in the 3.1.8 release. It seems there is (at least) one leak that didn't get fixed in 3.1.8. I have made a quick and dirty patch to avoid that leak. If anyone would care to try, the patch (which includes a few other things probably fixed in 3.1.8 already) can be found here: http://jiderhamn.se/resin-leak.patch However, even with that patch, it seems there is still some kind of PermGen leak that eventually leads to OutOfMemoryError. I have created a small application with the sole purpose of detecting these leaks. If anyone would care to try, it can be found here (sources included): http://jiderhamn.se/leak.war You will need to add some JARs to the WEB-INF/lib directory; preferrably a couple of large ones like spring.jar and hibernate.jar (don't use Resin JARs though). Then just drop the WAR in a clean installation of Resin 3.1.8 (preferrably patched with the patch above). Hit http://...:nn/leak (once is enough) Force a redeploy by either deleting the webapps/leak dir or touch:ing leak.war Hit http://...:nn/leak again Repeat the last two steps for as long as you'd like What you should see - or at least what I see on one Linux machine and one Windows machine - is the (ClassLoadingMXBean) loadedClassCount and the (MemoryPoolMXBean) Used Perm Gen steadily increasing (while the unloadedClassCount remains pretty stable) for every redeploy, which indicates a classloader leak. But I just can't find that leak. Now, here are the things really bugging me: 1. If I keep redeploying over and over, I will eventually get closer and closer to the Perm Gen Max (in some instances, I have seen the following behaivour instead turn up when Used reaches Init if Init is large enough). Then suddenly the unloadedClassCount is increased, but not with all the unused classes - only about the amount of one redeployment. Redeploy again, and it will increase another step. Meanwhile, the loadedClassCount remains pretty stable, since we are loading as many new classes as are unloaded. It's as if there was a FIFO queue/LRU cache of classloaders, so that the oldest one is garbage collected once there is not enough space for a new one. However, after a while there is (assumably) not enough space to create the new classloader before the old one is garbage collected, and I get OutOfMemoryError somewhere in the middle. Sometimes I am actually able to recover from this error by waiting for the GC to do it's job and then just try again. 2. Now I attached YourKit, looking for dangling classloaders as of the inital post. I found none. In fact, the Classes without Instances inspection only shows the classes in the added JARs from the last redeployment, so when tracing back to GC root, it goes via the current EnvironmentClassLoader which is correct. There are also no excessive instances of EnvironmentClassLoader. Hmm... Now wait a minute. Look at the total number of java.lang.Class objects. It does not match with the totalLoadedClassCount. In fact, the total number of classes found by YourKit is about the same as the totalLoadedClassCount on the very first hit of the application, before any redeployments. So from YourKits point of view, there is no classloader leak! But why then isn't the PermGen space reclaimed. This led me to wonder if there was some kind of JVM bug. (As a side note, I have yet to try with some other profiler) 3. So, I modified the test application (see commented out code in MyServlet.java) to load the classes of the JARs in a regular java.net.URLClassLoader which is then immediately thrown away. No leak. loadedClassCount is immediately decreased (and unloadedClassCount increased), as is the Used Perm Gen. That is, it behave the way we want the redeployment to behave. Ok, lets take one step down in the classloader hierachy and load all the classes via a disposable com.caucho.loader.DynamicClassLoader. No leak. So, what if I load them with a com.caucho.loader.EnvironmentClassLoader which is then destroy()ed and left for garbage collection. The leak is back! EnvironmentClassLoader has now applied to become prime suspect. In order to track things down I subclassed EnvironmentClassLoader inside my application and planned to make changes there. Well ... before any changes, the leak is nowhere to be found. What is going on here????? Anything that might help my sanity would be appreciated. /Mattias Jiderhamn Mattias Jiderhamn wrote (2008-11-05 06:43): > In support of this latest theory is the fact that YourKit shows two GC > roots for the HttpRequest. Apart from the > com.caucho.server.port.TcpConnection._request reference, the request > is a root itself by being on the stack of a thread (http--8080-...). > This could indicate a thread currently waiting in the > com.caucho.server.port.TcpConnection.run() method, where the > ServerRequest of the parent is also a local variable. > > Even if the request is reused, I believe the Invocation or the > ClassLoader of the invocation needs to be reset somehow, to release > the webapp classloader. > I can help try out a proposed fix to see if it solves the problem (= > feel free to mail me off list). I might even give it a shot myself. > > /Mattias
_______________________________________________ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
