On Mar 30, 2009, at 3:54 AM, Mattias Jiderhamn wrote:

After drawing the conclusion below, it isn't very far away to realize it probably has to do with some static (i.e. class loader specific) member of EnvironmentClassLoader.
And just as I thought, the heart of the problem is
private static EnvironmentLocal<ArrayList<AddLoaderListener>> _addLoaderListeners

More specifically, if line 234 of com.caucho.server.resin.Resin is commented out, the leak is removed (assuming my HttpRequest._invocation patch is also applied) // Environment.addChildLoaderListener(new WebBeansAddLoaderListener())

Scott, you said the WebBeans caches were not to blame because they are classloader local. Well, it seems something about this classloader locality isn't working the way it should, doesn't it...? Do you have enough info to try to fix this in a future (hopefully soon to be released) 3.1 release?

In the early work with 4.0, I'd cleaned a number of dead links (primarily WebBeans, but also some JMX). I believe the current 4.0 still has a few left, so I'll need to run a final pass at the end of the release cycle.

-- Scott



Maybe we are now seeing the end to all these memory leaks!!!

 /Mattias


Mattias Jiderhamn wrote (2009-03-30 11:40):

Just tried a new idea: I subclassed EnvironmentClassLoader within the Resin sources and compiled into resin.jar. There is still a leak (in contrast to when I subclassed EnvironmentClassLoader within my own application).

Conclusion: The classloader that loads the (Environment)ClassLoader class is somehow of importance.

 /Mattias

Mattias Jiderhamn wrote (2009-03-30 08:40):

Daniel, your findings confirms EnvironmentClassLoader as the prime suspect (my own tests are a bit more detailed in an earlier post, http://maillist.caucho.com/pipermail/resin-interest/2008-November/003158.html)

Regarding your GC root paths for this class:
1. I haven't noticed any uses of java.util.prefs - could this be part of any of your added JARs (like groovy)? 2. This is the error I have found within Resin that is still to be fixed. As mentioned earlier, there is a quick and dirty patch for Resin 3.1 available at http://jiderhamn.se/resin-leak.patch to avoid this leak. However even with the patch applied I'm still having leaks. 3. This obviously seems like a groovy problem, probably a static initialization triggered at class load.

So would you mind giving this another try with groovy removed and with a patched version of Resin? (I can provide a patched resin.jar if that helps)

Cheers,
  Mattias Jiderhamn

Daniel Lopez wrote (2009-03-29 14:52):

Hi,

Using that .war as a sample application, I did some more tests and
found some things that might be interesting:

I added groovy-1.5.7.jar, hibernate-3.2.6.ga.jar and
scala-compiler-2.7.2.jar to WEB-INF/lib. Nothing is done with them,
they simply live there. Important data: total size for WEB-INF/lib:
10.3 MB.

I then deployed the application, start it and here's some data I
gathered after accessing the application a couple of times and hitting
"Force Garbage Collector" until the memory is stable:

Heap Memory used: 31MB
Instances of com.caucho.loader.EnvironmentClassLoader: 1

I then re-save web.xml so the context is reloaded, access and force GC
until memory is stable again:

Heap Memory used: 41MB
Instances of com.caucho.loader.EnvironmentClassLoader: 2

Once again:

Heap Memory used: 51MB
Instances of com.caucho.loader.EnvironmentClassLoader: 3

Focusing on the last memory snapshot the 3 EnvironmentClassLoader come
from (shortest GC roots):
1.- contextClassLoader of
java.util.prefs.AbstractPreferences$EventDispatchThread [Stack Local,
Thread]
2.- _classLoader of com.caucho.server.dispatch.Invocation ->
_invocation of com.caucho.server.http.HttpRequest [Stack Local] -> 3.- <loader> of org.codehaus.groovy.tools.shell.util.Preferences $1 -> <class> of org.codehaus.groovy.tools.shell.util.Preferences $1 ->
     [1] of java.util.prefs.PreferenceChangeListener[3] ->
     prefListeners of java.util.prefs.WindowsPreferences ->
     STORE of org.codehaus.groovy.tools.shell.util.Preferences ->
     [771] of java.lang.Object[2560] ->
     elementData of java.util.Vector ->
     classes of com.caucho.loader.EnvironmentClassLoader ->
     contextClassLoader of
java.util.prefs.AbstractPreferences$EventDispatchThread [Stack Local,
Thread]

They are ordered using order of appearance (in the first memory SS,
one like 1 is present, after the first restart one like the 2 is
present).

And looking at one of the suspected classes to be replicated,
org.codehaus.groovy.tools.shell.util.Preferences, you can see that
indeed there are 3 instances of that class, each one loaded by a
different EnvironmentClassLoader.

So, I would say there is indeed a leak and the jackpot would be to
find out why the EnvironmentClassLoader instances are not being
cleaned. Without knowing how Resin is internally supposed to work and which loaders are supposed not to be there, it's difficult to know. As
the classes inside the lib directory are not even used at all, one
would suspect is a Resin internal issue, but seeing Preferences
classes and ThreadLocal in the mix makes the whole thing quite
suspicious :).

Any idea or test to find out more?
D.


Seems to me nothing has changed in this regard in the 4.0 snapshot. I did my leak test as previously described, see below, and no classes were
unloaded.

- Dowload http://jiderhamn.se/leak.war
- Add some JARs to the WEB-INF/lib directory; preferrably a couple of large ones like spring.jar and hibernate.jar (don't use Resin JARs though).
- Drop the WAR in a clean installation of Resin
- Hit http://...:nn/leak (once is enough)
- Force a redeploy by either deleting the webapps/leak dir or touch:ing
leak.war
- Hit http://...:nn/leak again
- Repeat the last two steps for as long as you'd like

/Mattias





_______________________________________________
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest

Reply via email to