Scott Ferguson wrote (2008-11-26 16:53):
> On Nov 24, 2008, at 11:08 PM, Mattias Jiderhamn wrote:
>> I'm still battling this PermGen leak and frankly I'm really starting
>> to doubt that I know what I'm doing anymore. I'd be very happy if
>> anyone would care to explain that to me...
>> Since my last post Scott and I have discussed potential class loader
>> leaks and some of them have been fixed in the 3.1.8 release. It seems
>> there is (at least) one leak that didn't get fixed in 3.1.8. I have
>> made a quick and dirty patch to avoid that leak. If anyone would care
>> to try, the patch (which includes a few other things probably fixed
>> in 3.1.8 already) can be found here:
>> However, even with that patch, it seems there is still some kind of
>> PermGen leak that eventually leads to OutOfMemoryError. I have
>> created a small application with the sole purpose of detecting these
>> leaks. If anyone would care to try, it can be found here (sources
>> included):
>> You will need to add some JARs to the WEB-INF/lib directory;
>> preferrably a couple of large ones like spring.jar and hibernate.jar
>> (don't use Resin JARs though).
>> Then just drop the WAR in a clean installation of Resin 3.1.8
>> (preferrably patched with the patch above).
>> Hit http://...:nn/leak (once is enough)
>> Force a redeploy by either deleting the webapps/leak dir or touch:ing
>> leak.war
>> Hit http://...:nn/leak again
>> Repeat the last two steps for as long as you'd like
>> What you should see - or at least what I see on one Linux machine and
>> one Windows machine - is the (ClassLoadingMXBean) loadedClassCount
>> and the (MemoryPoolMXBean) Used Perm Gen steadily increasing (while
>> the unloadedClassCount remains pretty stable) for every redeploy,
>> which indicates a classloader leak. But I just can't find that leak.
> Thanks.   
> Right now, our code base is a bit stuck due to the WebBeans/OSGi
> upgrade (for Resin 4.0.0, was 3.2.2).  Once that's cleaned up and I
> can put up a snapshot I can take a look.

Scott Ferguson wrote (2008-12-29 20:41):
> I've just made an early Resin 4.0 snapshot available. 

Does this mean we can assume you will be able to look more closely at
the memory leaks in 3.1 sometime soon...?


> -- Scott
>> Now, here are the things really bugging me:
>> 1. If I keep redeploying over and over, I will eventually get closer
>> and closer to the Perm Gen Max (in some instances, I have seen the
>> following behaivour instead turn up when Used reaches Init if Init is
>> large enough). Then suddenly the unloadedClassCount is increased, but
>> not with all the unused classes - only about the amount of one
>> redeployment. Redeploy again, and it will increase another step.
>> Meanwhile, the loadedClassCount remains pretty stable, since we are
>> loading as many new classes as are unloaded. It's as if there was a
>> FIFO queue/LRU cache of classloaders, so that the oldest one is
>> garbage collected once there is not enough space for a new one.
>> However, after a while there is (assumably) not enough space to
>> create the new classloader before the old one is garbage collected,
>> and I get OutOfMemoryError somewhere in the middle. Sometimes I am
>> actually able to recover from this error by waiting for the GC to do
>> it's job and then just try again.
>> 2. Now I attached YourKit, looking for dangling classloaders as of
>> the inital post. I found none. In fact, the Classes without Instances
>> inspection only shows the classes in the added JARs from the last
>> redeployment, so when tracing back to GC root, it goes via the
>> current EnvironmentClassLoader which is correct. There are also no
>> excessive instances of EnvironmentClassLoader. Hmm... Now wait a
>> minute. Look at the total number of java.lang.Class objects. It does
>> not match with the totalLoadedClassCount. In fact, the total number
>> of classes found by YourKit is about the same as the
>> totalLoadedClassCount on the very first hit of the application,
>> before any redeployments. So from YourKits point of view, there is no
>> classloader leak! But why then isn't the PermGen space reclaimed.
>> This led me to wonder if there was some kind of JVM bug. (As a side
>> note, I have yet to try with some other profiler)
>> 3. So, I modified the test application (see commented out code in
>> to load the classes of the JARs in a regular
>> which is then immediately thrown away. No
>> leak. loadedClassCount is immediately decreased (and
>> unloadedClassCount increased), as is the Used Perm Gen. That is, it
>> behave the way we want the redeployment to behave.
>> Ok, lets take one step down in the classloader hierachy and load all
>> the classes via a disposable com.caucho.loader.DynamicClassLoader. No
>> leak.
>> So, what if I load them with a
>> com.caucho.loader.EnvironmentClassLoader which is then destroy()ed
>> and left for garbage collection. The leak is back!
>> EnvironmentClassLoader has now applied to become prime suspect. In
>> order to track things down I subclassed EnvironmentClassLoader inside
>> my application and planned to make changes there. Well ... before any
>> changes, the leak is nowhere to be found.
>> What is going on here?????
>> Anything that might help my sanity would be appreciated.
>>  /Mattias Jiderhamn
>> Mattias Jiderhamn wrote (2008-11-05 06:43):
>>> In support of this latest theory is the fact that YourKit shows two
>>> GC roots for the HttpRequest. Apart from the
>>> com.caucho.server.port.TcpConnection._request reference, the request
>>> is a root itself by being on the stack of a thread (http--8080-...).
>>> This could indicate a thread currently waiting in the
>>> method, where the
>>> ServerRequest of the parent is also a local variable.
>>> Even if the request is reused, I believe the Invocation or the
>>> ClassLoader of the invocation needs to be reset somehow, to release
>>> the webapp classloader.
>>> I can help try out a proposed fix to see if it solves the problem (=
>>> feel free to mail me off list). I might even give it a shot myself.
>>>  /Mattias

resin-interest mailing list

Reply via email to