Hi again,
Thanks for the info. I have some experience with the GC so I know more
or less how it works ;). I was more interested to see if other had found
some similar issues with Resin or some specific libraries.
As Scott mentions, the problem is usually caused by having a class
loaded with a child classloader referenced from a class loaded from a
parent classloader, as the "throwing away" the child classloader won't
allow the reference to be GC-ed.
If I'm interpreting correctly the data, it seems to me that some
libraries, loaded using a "webapp classloader", are storing some
references using ThreadLocal, which is referenced from the root
classloader. That keeps the webapp classloader "alive" and that causes
many other classes that should be GC-ed to remain alive.
I know it's not my classes as I don't use ThreadLocal :).
Now I "just" have to find out the library that is using ThreadLocal to
store references and see if they are cleaning it up correctly.
I haven't used ThreadLocal much, so that would be the occasion to learn
more about it. If I understand correctly, the problem seems to be caused
by some threads are still being kept alive after a context restart, as
the thread dying would imply all its thread-local instances to be ready
for GC.
The libraries that look more suspicious are the HSQLSDB driver, the
Quartz scheduler and a JMS client, as they seem to spawn some threads
for house-keeping and other tasks.
I'll try to find more time to investigate further and look at it from a
Thread analysis point of view. I'll get back to you if I find something
else that looks Resin related.
Thanks for the feedback,
D.
PD: Scott, I'm trying to understand your example og properly using
ThreadLocal and I don't see why oldValue would never be GC-ed unless it
is set again. Does ThreadLocal keep a reference to old values even when
set is used with a new object?
PPD: Investigating lightly over the web, I already came up with some
people complaining about the usage of ThreadLocal in some of the
libraries I use (Xalan, for example). Ummm, that does not look good :).
Not that I want to restart my contexts too often in a production
environment, but...
Scott Ferguson escribió:
> On Jun 11, 2007, at 10:11 AM, Gary Zhu wrote:
>
>> It could be as simple as your ThreadLocal, in addition to that,
>> this could also be caused by others.
>
> True, but in this case, Daniel has noticed the extra ThreadLocal as
> the object holding the link in the profile, so it's a good place to
> start looking.
>
>> private static final Level CUSTOMLEVEL = new Level("test", 555) {};
>> ....
>> Logger.getLogger("test").log("CUSTOMLEVEL, "something");
>> ....
>> };
>>
>> Since CUSTOMLEVEL contains java.util.logging.Level.INFO (SEVERE,
>> etc), and they are 'reachable' from other part of JVM, so
>> your_class is not garbage collected.
>
> By the way, this is a bug in java.util.logging.Level. The
> application code would be fine if it wasn't for the JDK bug.
>
> The general rule is: never store an object from a child classloader
> context in a parent classloader context unless you use WeakReferences.
>
> It's the main reason why Singletons are dangerous.
>
> ThreadLocal is an instance of this problem, because it's essentially
> storing the object in the root classloader context.
>
> -- Scott
>
>> On that FOB session, Edward used JHat to trace PermGen leaks, and
>> found 5-6 violations in that application server and a couple of
>> places in the user application. A note to Caucho engineers: that
>> case study was not based on Resin.
>>
>> You can find these discussions from the blogs of two Sun engineers
>> who presented this FOB:
>>
>> http://blogs.sun.com/fkieviet
>> http://blogs.sun.com/edwardchou
>>
>> one of the good articles is on Frank's blog:
>> http://blogs.sun.com/fkieviet/entry/classloader_leaks_the_dreaded_java
>>
>>
>>
>>> -----Original Message-----
>>> From: [EMAIL PROTECTED]
>>> [mailto:[EMAIL PROTECTED] Behalf Of Scott Ferguson
>>> Sent: Monday, June 11, 2007 9:12 AM
>>> To: General Discussion for the Resin application server
>>> Subject: Re: [Resin-interest] Application Memory Profiling
>>>
>>>
>>>
>>> On Jun 11, 2007, at 3:00 AM, Daniel López wrote:
>>>
>>>> Something similar happens with com.caucho.xml.Xml: I start with 2
>>>> instances, one referenced from a class of mine and another from
>>>> java.lang.ThreadLocal. After a restart I get 2 instances referenced
>>>> from
>>>> java.lang.ThreadLocal. Another restart and I have 3 referenced from
>>>> java.lang.ThreadLocal... In this case, the problem seems to be a
>>>> growing
>>>> number of instances from
>>> org.apache.xml.utils.XMLReaderManager... that
>>>> are themselves related to the aforementioned
>>>> com.caucho.loader.EnvironmentClassLoader.
>>> Do you know what is holding the ThreadLocal references?
>>>
>>> This definitely looks like a threading/ThreadLocal issue. There have
>>> been libraries which have neglected to clear the ThreadLocals
>>> properly, which will cause major problems with garbage collection and
>>> classloading. That would be the first thing to look at.
>>>
>>> ThreadLocal needs to be used with the following pattern:
>>>
>>> ThreadLocal _local = new ThreadLocal();
>>>
>>> ...
>>>
>>> Object oldValue = _local.get();
>>> try {
>>> _local.set(newValue);
>>>
>>> ...
>>> } finally {
>>> _local.set(oldValue);
>>> }
>>>
>>> If the application forgets the second set, then the oldValue will not
>>> be garbage collected.
>>>
>>>> So my question would be if anybody has run into similar
>>> issues or has
>>>> any idea what could be causing the proliferation of
>>>> com.caucho.loader.EnvironmentClassLoader instances. I
>>> suspect it might
>>>> be that some library is keeping a reference to the
>>> classloader through
>>>> some daemon thread that is not being properly initialised when the
>>>> context is restarted.
>>>>
>>>> Any hints on how to further debug this issue? Any similar
>>> experience?
>>>
>>> Look for the ThreadLocal reference. Also, double check that there
>>> isn't a daemon thread that isn't properly closed on restart.
>>>
>>> -- Scott
>>>
>>>> Thanks,
>>>> D.
_______________________________________________
resin-interest mailing list
resin-interest@caucho.com
http://maillist.caucho.com/mailman/listinfo/resin-interest
