I did some more tests today, as, like you, I suspected that the  
listener pointing to the Groovy classes where quite suspicious but ran  
out of time. I repeated the tests substituting the groovy jar file  
with Xalan and the leak is gone.
I performed the same tests and re-initialised the application several  
times. After each initialisation, the non-heap memory would grow, then  
go back to a stable number, and the number of instances of  
EnvironmentClassLoader would also grow and then schrink.
To confirm, I re-added the Groovy.jar file and there it was: after  
each context restart, one extra EnvironmentClassLoader could not be  
GC-ed and the non-Heap memory would keep growing with the aprox.  
amount of WEB-INF/lib size.

That's the problem with these kind of leaks, the container  
classloaders always look suspicious because they are always involved  
in the process and they are the ones that hold the bigger space. But  
while they can be the problem, just a single problem with one of the  
libraries and the EnvironmentClassLoader cannot be GC-ed effectively.

Well, that means that Groovy is also on the list of buggy libraries  
for such kind of environment, along with some JDBC drivers, like  
HSQLDB that starts a thread that is never stopped unless you add some  
custom-driver-specific code, and with some jakarta commons that do not  
clean ThreadLocal references appropriately.

Not sure what it can be done, apart of notifying the library  
developers so they can fix those bugs, but it really is a global  
problem as there are way too many libraries not ready for a throw-away  
classloader environment :(.


S'està citant Mattias Jiderhamn <mj-li...@expertsystems.se>:

> After drawing the conclusion below, it isn't very far away to realize it
> probably has to do with some static (i.e. class loader specific) member
> of EnvironmentClassLoader.
> And just as I thought, the heart of the problem is
>   private static EnvironmentLocal<ArrayList<AddLoaderListener>>
> _addLoaderListeners
> More specifically, if line 234 of com.caucho.server.resin.Resin is
> commented out, the leak is removed (assuming my HttpRequest._invocation
> patch is also applied)
>    // Environment.addChildLoaderListener(new WebBeansAddLoaderListener())
> Scott, you said the WebBeans caches were not to blame because they are
> classloader local. Well, it seems something about this classloader
> locality isn't working the way it should, doesn't it...?
> Do you have enough info to try to fix this in a future (hopefully soon
> to be released) 3.1 release?
> Maybe we are now seeing the end to all these memory leaks!!!
>  /Mattias
> Mattias Jiderhamn wrote (2009-03-30 11:40):
>> Just tried a new idea: I subclassed EnvironmentClassLoader within the
>> Resin sources and compiled into resin.jar. There is still a leak (in
>> contrast to when I subclassed EnvironmentClassLoader within my own
>> application).
>> Conclusion: The classloader that loads the (Environment)ClassLoader
>> class is somehow of importance.
>>  /Mattias
>> Mattias Jiderhamn wrote (2009-03-30 08:40):
>>> Daniel, your findings confirms EnvironmentClassLoader as the prime
>>> suspect (my own tests are a bit more detailed in an earlier post,
>>> http://maillist.caucho.com/pipermail/resin-interest/2008-November/003158.html)
>>> Regarding your GC root paths for this class:
>>> 1. I haven't noticed any uses of java.util.prefs - could this be part
>>> of any of your added JARs (like groovy)?
>>> 2. This is the error I have found within Resin that is still to be
>>> fixed. As mentioned earlier, there is a quick and dirty patch for
>>> Resin 3.1 available at http://jiderhamn.se/resin-leak.patch to avoid
>>> this leak. However even with the patch applied I'm still having leaks.
>>> 3. This obviously seems like a groovy problem, probably a static
>>> initialization triggered at class load.
>>> So would you mind giving this another try with groovy removed and
>>> with a patched version of Resin? (I can provide a patched resin.jar
>>> if that helps)
>>> Cheers,
>>>   Mattias Jiderhamn
>>> Daniel Lopez wrote (2009-03-29 14:52):
>>>> Hi,
>>>> Using that .war as a sample application, I did some more tests and
>>>> found some things that might be interesting:
>>>> I added groovy-1.5.7.jar, hibernate-3.2.6.ga.jar and
>>>> scala-compiler-2.7.2.jar to WEB-INF/lib. Nothing is done with them,
>>>> they simply live there. Important data: total size for WEB-INF/lib:
>>>> 10.3 MB.
>>>> I then deployed the application, start it and here's some data I
>>>> gathered after accessing the application a couple of times and hitting
>>>> "Force Garbage Collector" until the memory is stable:
>>>> Heap Memory used: 31MB
>>>> Instances of com.caucho.loader.EnvironmentClassLoader: 1
>>>> I then re-save web.xml so the context is reloaded, access and force GC
>>>> until memory is stable again:
>>>> Heap Memory used: 41MB
>>>> Instances of com.caucho.loader.EnvironmentClassLoader: 2
>>>> Once again:
>>>> Heap Memory used: 51MB
>>>> Instances of com.caucho.loader.EnvironmentClassLoader: 3
>>>> Focusing on the last memory snapshot the 3 EnvironmentClassLoader come
>>>> from (shortest GC roots):
>>>> 1.- contextClassLoader of
>>>> java.util.prefs.AbstractPreferences$EventDispatchThread [Stack Local,
>>>> Thread]
>>>> 2.- _classLoader of com.caucho.server.dispatch.Invocation ->
>>>>      _invocation of com.caucho.server.http.HttpRequest [Stack Local] ->
>>>> 3.- <loader> of org.codehaus.groovy.tools.shell.util.Preferences$1 ->
>>>>      <class> of org.codehaus.groovy.tools.shell.util.Preferences$1 ->
>>>>      [1] of java.util.prefs.PreferenceChangeListener[3] ->
>>>>      prefListeners of java.util.prefs.WindowsPreferences ->
>>>>      STORE of org.codehaus.groovy.tools.shell.util.Preferences ->
>>>>      [771] of java.lang.Object[2560] ->
>>>>      elementData of java.util.Vector ->
>>>>      classes of com.caucho.loader.EnvironmentClassLoader ->
>>>>      contextClassLoader of
>>>> java.util.prefs.AbstractPreferences$EventDispatchThread [Stack Local,
>>>> Thread]
>>>> They are ordered using order of appearance (in the first memory SS,
>>>> one like 1 is present, after the first restart one like the 2 is
>>>> present).
>>>> And looking at one of the suspected classes to be replicated,
>>>> org.codehaus.groovy.tools.shell.util.Preferences, you can see that
>>>> indeed there are 3 instances of that class, each one loaded by a
>>>> different EnvironmentClassLoader.
>>>> So, I would say there is indeed a leak and the jackpot would be to
>>>> find out why the EnvironmentClassLoader instances are not being
>>>> cleaned. Without knowing how Resin is internally supposed to work and
>>>> which loaders are supposed not to be there, it's difficult to know. As
>>>> the classes inside the lib directory are not even used at all, one
>>>> would suspect is a Resin internal issue, but seeing Preferences
>>>> classes and ThreadLocal in the mix makes the whole thing quite
>>>> suspicious :).
>>>> Any idea or test to find out more?
>>>> D.
>>>>> Seems to me nothing has changed in this regard in the 4.0 snapshot. I
>>>>> did my leak test as previously described, see below, and no classes were
>>>>> unloaded.
>>>>> - Dowload http://jiderhamn.se/leak.war
>>>>> - Add some JARs to the WEB-INF/lib directory; preferrably a couple of
>>>>> large ones like spring.jar and hibernate.jar (don't use Resin   
>>>>> JARs though).
>>>>> - Drop the WAR in a clean installation of Resin
>>>>> - Hit http://...:nn/leak (once is enough)
>>>>> - Force a redeploy by either deleting the webapps/leak dir or touch:ing
>>>>> leak.war
>>>>> - Hit http://...:nn/leak again
>>>>> - Repeat the last two steps for as long as you'd like
>>>>> /Mattias


resin-interest mailing list

Reply via email to