On Jan 9, 2009, at 12:22 AM, Mattias Jiderhamn wrote:
> Scott Ferguson wrote (2008-11-26 16:53):
>> Right now, our code base is a bit stuck due to the WebBeans/OSGi
>> upgrade (for Resin 4.0.0, was 3.2.2). Once that's cleaned up and I
>> can put up a snapshot I can take a look.
> Scott Ferguson wrote (2008-12-29 20:41):
>> I've just made an early Resin 4.0 snapshot available.
> Does this mean we can assume you will be able to look more closely at
> the memory leaks in 3.1 sometime soon...?
I was hoping you could verify the 4.0 memory fixes before looking at
3.1 Because several of those changes were more involved than I'm
comfortable with for 3.1, I'd prefer to make sure they work in 4.0
>> -- Scott
>>> Now, here are the things really bugging me:
>>> 1. If I keep redeploying over and over, I will eventually get closer
>>> and closer to the Perm Gen Max (in some instances, I have seen the
>>> following behaivour instead turn up when Used reaches Init if Init
>>> large enough). Then suddenly the unloadedClassCount is increased,
>>> not with all the unused classes - only about the amount of one
>>> redeployment. Redeploy again, and it will increase another step.
>>> Meanwhile, the loadedClassCount remains pretty stable, since we are
>>> loading as many new classes as are unloaded. It's as if there was a
>>> FIFO queue/LRU cache of classloaders, so that the oldest one is
>>> garbage collected once there is not enough space for a new one.
>>> However, after a while there is (assumably) not enough space to
>>> create the new classloader before the old one is garbage collected,
>>> and I get OutOfMemoryError somewhere in the middle. Sometimes I am
>>> actually able to recover from this error by waiting for the GC to do
>>> it's job and then just try again.
>>> 2. Now I attached YourKit, looking for dangling classloaders as of
>>> the inital post. I found none. In fact, the Classes without
>>> inspection only shows the classes in the added JARs from the last
>>> redeployment, so when tracing back to GC root, it goes via the
>>> current EnvironmentClassLoader which is correct. There are also no
>>> excessive instances of EnvironmentClassLoader. Hmm... Now wait a
>>> minute. Look at the total number of java.lang.Class objects. It does
>>> not match with the totalLoadedClassCount. In fact, the total number
>>> of classes found by YourKit is about the same as the
>>> totalLoadedClassCount on the very first hit of the application,
>>> before any redeployments. So from YourKits point of view, there is
>>> classloader leak! But why then isn't the PermGen space reclaimed.
>>> This led me to wonder if there was some kind of JVM bug. (As a side
>>> note, I have yet to try with some other profiler)
>>> 3. So, I modified the test application (see commented out code in
>>> MyServlet.java) to load the classes of the JARs in a regular
>>> java.net.URLClassLoader which is then immediately thrown away. No
>>> leak. loadedClassCount is immediately decreased (and
>>> unloadedClassCount increased), as is the Used Perm Gen. That is, it
>>> behave the way we want the redeployment to behave.
>>> Ok, lets take one step down in the classloader hierachy and load all
>>> the classes via a disposable com.caucho.loader.DynamicClassLoader.
>>> So, what if I load them with a
>>> com.caucho.loader.EnvironmentClassLoader which is then destroy()ed
>>> and left for garbage collection. The leak is back!
>>> EnvironmentClassLoader has now applied to become prime suspect. In
>>> order to track things down I subclassed EnvironmentClassLoader
>>> my application and planned to make changes there. Well ... before
>>> changes, the leak is nowhere to be found.
>>> What is going on here?????
>>> Anything that might help my sanity would be appreciated.
>>> /Mattias Jiderhamn
>>> Mattias Jiderhamn wrote (2008-11-05 06:43):
>>>> In support of this latest theory is the fact that YourKit shows two
>>>> GC roots for the HttpRequest. Apart from the
>>>> com.caucho.server.port.TcpConnection._request reference, the
>>>> is a root itself by being on the stack of a thread
>>>> This could indicate a thread currently waiting in the
>>>> com.caucho.server.port.TcpConnection.run() method, where the
>>>> ServerRequest of the parent is also a local variable.
>>>> Even if the request is reused, I believe the Invocation or the
>>>> ClassLoader of the invocation needs to be reset somehow, to release
>>>> the webapp classloader.
>>>> I can help try out a proposed fix to see if it solves the problem
>>>> feel free to mail me off list). I might even give it a shot myself.
> resin-interest mailing list
resin-interest mailing list