Re: [Resin-interest] Potential class loader leaks
Scott, can you tell me whether you've been able to reproduce this scenario?
Mattias Jiderhamn wrote (2009-12-11 08:34):
Scott Ferguson wrote (2009-12-11 00:23):
Mattias Jiderhamn wrote:
So I've spent another day hunting that lo(ooo)ng standing PermGen memory
leak in our application and/or Resin.
I made a new discovery which "shouldn't" be an issue, but could
potentially fix problems.
From my investigation it seems that whenever the application is
reloaded, a reference to the old
com.caucho.loader.EnvironmentClassLoader is kept in
com.caucho.hessian.io.SerializerFactory._defaultFactory. I agree that in
theory this shouldn't happen, since the ClassLoader is a weak reference
key, /however/ if I add some code to the application shutdown,
explicitly removing the EnvironmentClassLoader from _defaultFactory
using reflection, the garbage collector is able to unload these classes.
I changed this in 4.0.2 (with some more changes in 4.0.3). Even though
the key is a weak reference, the value is a strong reference.
Indeed, this has been fixed in the latest snapshot! Good job!
Sorry for not testing against that...
I should mention though, that there is still a minimum of two
EnvironmentClassLoaders for the given application after reloading at
least once. The former one seem to stick around somehow. We have
discussed this before, Scott; how references are kept inside
com.caucho.server.dispatch.Invocation, at least in a low traffic (dev /
debugging) environment.
On a web-app change the Invocation cache is cleared, so there shouldn't
be any old references there.
Could you please double check that? For example, what happens to the
503:ed requests received during redeployment?
I have an examplifying YourKit (freely available EAP) snapshot I could
send you if it helps.
Here is what I'm getting, which seems repeatable over and over in my
environment:
Startup: 9k loaded classes, no unloaded classes, 49 MB PermGen
First redeployment: 15k loaded classes, no unloaded classes, 77 MB
PermGen = one extra instance of the app with 6k classes loaded.
Assumed a total of 2 classloaders
Second redeployment: 15k loaded classes, 6k unloaded classes, 77 MB
PermGen = initial/first(?) instance GC:ed and a new one loaded. Still
2 classloaders.
Third redeployment: 21k loaded classes, 6k unloaded classes, 105 MB
PermGen = one extra instance of the app with 6k classes loaded,
nothing GC:ed. Assumed a total of 3 classloaders
Fourth redeployment: 21k loaded classes, 12k unloaded classes, 105 MB
PermGen = second(?) instance GC:ed and a new one loaded. Still 3
classloaders.
(Fifth redeployment: 15k loaded classes, 24k unloaded classes, 77 MB
PermGen = third and fourth(?) instance GC:ed and a new one loaded. 2
classloaders. Haven't repeated this test enough to build a thesis)
Not sure if this a real problem, but ideally the
class loader of the previous version should be available for garbage
collection before the classes of the new version are loaded ('cause
there is no turning back anyway, is there...?).
In the past this could be a problem with JNI because a JNI library can
only be loaded in one classloader (that may have been changed but it
always was a JDK restriction.)
Are you saying this shouldn't be a issue in 4.0.2/snapshot running on
JDK 1.6?
--
___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Potential class loader leaks
Mattias Jiderhamn wrote (2009-12-11 08:34): I should mention though, that there is still a minimum of two EnvironmentClassLoaders for the given application after reloading at least once. The former one seem to stick around somehow. We have discussed this before, Scott; how references are kept inside com.caucho.server.dispatch.Invocation, at least in a low traffic (dev / debugging) environment. On a web-app change the Invocation cache is cleared, so there shouldn't be any old references there. Could you please double check that? For example, what happens to the 503:ed requests received during redeployment? I have an examplifying YourKit (freely available EAP) snapshot I could send you if it helps. Here is what I'm getting, which seems repeatable over and over in my environment: Startup: 9k loaded classes, no unloaded classes, 49 MB PermGen First redeployment: 15k loaded classes, no unloaded classes, 77 MB PermGen = one extra instance of the app with 6k classes loaded. Assumed a total of 2 classloaders Second redeployment: 15k loaded classes, 6k unloaded classes, 77 MB PermGen = initial/first(?) instance GC:ed and a new one loaded. Still 2 classloaders. Third redeployment: 21k loaded classes, 6k unloaded classes, 105 MB PermGen = one extra instance of the app with 6k classes loaded, nothing GC:ed. Assumed a total of 3 classloaders Fourth redeployment: 21k loaded classes, 12k unloaded classes, 105 MB PermGen = second(?) instance GC:ed and a new one loaded. Still 3 classloaders. (Fifth redeployment: 15k loaded classes, 24k unloaded classes, 77 MB PermGen = third and fourth(?) instance GC:ed and a new one loaded. 2 classloaders. Haven't repeated this test enough to build a thesis) I should probably point out that garbage collection was forced a couple of times before noting these numbers, so it shouldn't be a cased of delayed GC. -- ___ resin-interest mailing list resin-interest@caucho.com http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Potential class loader leaks
Scott Ferguson wrote (2009-12-11 00:23):
Mattias Jiderhamn wrote:
So I've spent another day hunting that lo(ooo)ng standing PermGen memory
leak in our application and/or Resin.
I made a new discovery which "shouldn't" be an issue, but could
potentially fix problems.
From my investigation it seems that whenever the application is
reloaded, a reference to the old
com.caucho.loader.EnvironmentClassLoader is kept in
com.caucho.hessian.io.SerializerFactory._defaultFactory. I agree that in
theory this shouldn't happen, since the ClassLoader is a weak reference
key, /however/ if I add some code to the application shutdown,
explicitly removing the EnvironmentClassLoader from _defaultFactory
using reflection, the garbage collector is able to unload these classes.
I changed this in 4.0.2 (with some more changes in 4.0.3). Even though
the key is a weak reference, the value is a strong reference.
Indeed, this has been fixed in the latest snapshot! Good job!
Sorry for not testing against that...
I should mention though, that there is still a minimum of two
EnvironmentClassLoaders for the given application after reloading at
least once. The former one seem to stick around somehow. We have
discussed this before, Scott; how references are kept inside
com.caucho.server.dispatch.Invocation, at least in a low traffic (dev /
debugging) environment.
On a web-app change the Invocation cache is cleared, so there shouldn't
be any old references there.
Could you please double check that? For example, what happens to the
503:ed requests received during redeployment?
I have an examplifying YourKit (freely available EAP) snapshot I could
send you if it helps.
Here is what I'm getting, which seems repeatable over and over in my
environment:
Startup: 9k loaded classes, no unloaded classes, 49 MB PermGen
First redeployment: 15k loaded classes, no unloaded classes, 77 MB
PermGen = one extra instance of the app with 6k classes loaded. Assumed
a total of 2 classloaders
Second redeployment: 15k loaded classes, 6k unloaded classes, 77 MB
PermGen = initial/first(?) instance GC:ed and a new one loaded. Still 2
classloaders.
Third redeployment: 21k loaded classes, 6k unloaded classes, 105 MB
PermGen = one extra instance of the app with 6k classes loaded, nothing
GC:ed. Assumed a total of 3 classloaders
Fourth redeployment: 21k loaded classes, 12k unloaded classes, 105 MB
PermGen = second(?) instance GC:ed and a new one loaded. Still 3
classloaders.
(Fifth redeployment: 15k loaded classes, 24k unloaded classes, 77 MB
PermGen = third and fourth(?) instance GC:ed and a new one loaded. 2
classloaders. Haven't repeated this test enough to build a thesis)
Not sure if this a real problem, but ideally the
class loader of the previous version should be available for garbage
collection before the classes of the new version are loaded ('cause
there is no turning back anyway, is there...?).
In the past this could be a problem with JNI because a JNI library can
only be loaded in one classloader (that may have been changed but it
always was a JDK restriction.)
Are you saying this shouldn't be a issue in 4.0.2/snapshot running on
JDK 1.6?
--
___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest
Re: [Resin-interest] Potential class loader leaks
Mattias Jiderhamn wrote:
> So I've spent another day hunting that lo(ooo)ng standing PermGen memory
> leak in our application and/or Resin.
>
> I made a new discovery which "shouldn't" be an issue, but could
> potentially fix problems.
> From my investigation it seems that whenever the application is
> reloaded, a reference to the old
> com.caucho.loader.EnvironmentClassLoader is kept in
> com.caucho.hessian.io.SerializerFactory._defaultFactory. I agree that in
> theory this shouldn't happen, since the ClassLoader is a weak reference
> key, /however/ if I add some code to the application shutdown,
> explicitly removing the EnvironmentClassLoader from _defaultFactory
> using reflection, the garbage collector is able to unload these classes.
>
I changed this in 4.0.2 (with some more changes in 4.0.3). Even though
the key is a weak reference, the value is a strong reference.
> I should mention though, that there is still a minimum of two
> EnvironmentClassLoaders for the given application after reloading at
> least once. The former one seem to stick around somehow. We have
> discussed this before, Scott; how references are kept inside
> com.caucho.server.dispatch.Invocation, at least in a low traffic (dev /
> debugging) environment.
On a web-app change the Invocation cache is cleared, so there shouldn't
be any old references there.
> Not sure if this a real problem, but ideally the
> class loader of the previous version should be available for garbage
> collection before the classes of the new version are loaded ('cause
> there is no turning back anyway, is there...?).
In the past this could be a problem with JNI because a JNI library can
only be loaded in one classloader (that may have been changed but it
always was a JDK restriction.)
> Otherwise you always
> have to set MaxPermSize to at least twice as much as the application
> actually needs.
>
> In some profiling session I've also seen references from logging and
> "index-cache-writer" threads, which might be worth looking into.
>
The index writer shouldn't have any references to any non-system
classloaders.
-- Scott
> Scott, could you tell me if anything can be done to the issues mentioned
> above?
>
>
___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest
[Resin-interest] Potential class loader leaks
So I've spent another day hunting that lo(ooo)ng standing PermGen memory
leak in our application and/or Resin.
I made a new discovery which "shouldn't" be an issue, but could
potentially fix problems.
From my investigation it seems that whenever the application is
reloaded, a reference to the old
com.caucho.loader.EnvironmentClassLoader is kept in
com.caucho.hessian.io.SerializerFactory._defaultFactory. I agree that in
theory this shouldn't happen, since the ClassLoader is a weak reference
key, /however/ if I add some code to the application shutdown,
explicitly removing the EnvironmentClassLoader from _defaultFactory
using reflection, the garbage collector is able to unload these classes.
I should mention though, that there is still a minimum of two
EnvironmentClassLoaders for the given application after reloading at
least once. The former one seem to stick around somehow. We have
discussed this before, Scott; how references are kept inside
com.caucho.server.dispatch.Invocation, at least in a low traffic (dev /
debugging) environment. Not sure if this a real problem, but ideally the
class loader of the previous version should be available for garbage
collection before the classes of the new version are loaded ('cause
there is no turning back anyway, is there...?). Otherwise you always
have to set MaxPermSize to at least twice as much as the application
actually needs.
In some profiling session I've also seen references from logging and
"index-cache-writer" threads, which might be worth looking into.
Scott, could you tell me if anything can be done to the issues mentioned
above?
--
___
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest
