Hi Mark.
We made a very simple change to
org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler in the
line 85: the code "this.componentHolder.set(null);" was replaced by
"this.componentHolder.remove();".
We are using a normal maven webapp project in which we add all of our
custom classes. This webapp porject has the xmlui module
(org.dspace.modules.xmlui) as a war dependency. Then we lean on the maven's
overlay process to ensure those custom classes are included in the final
war.
This is how we redefined this cocoon's class (and many other dspace's
classes). Using this approach we don't need to download and recompile the
entire set of classes for dspace or cocoon. We just compile our classes
whilst maven gets the jars and wars and make the overlays.
I'm looking forward for your comments
Regards
Nestor
Servicio de Difusión de la Creación Intelectual - http://sedici.unlp.edu.ar
Universidad Nacional de La Plata
Buenos Aires, Argentina
On Fri, Jun 8, 2012 at 6:06 PM, Mark Diggory <mdigg...@atmire.com> wrote:
> Nestor, this is very promising.
>
> Can you forward any code changes you made? Are these directly
> in org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler or
> in code calling it?
>
> Mark
>
>
> On Fri, Jun 8, 2012 at 10:11 AM, Nestor Oviedo <oviedones...@gmail.com>wrote:
>
>> Hi all.
>> Thanks Mark a Tim for your answers.
>> We took a few weeks to review this issue and these are the news.
>>
>> We didn't think our extensions could be causing this because they do not
>> open any strange connection o resource. Nevertheless, we disabled them all
>> and the problem was still there.
>> Then we started to search into the DSpace and Cocoon code, looking for
>> ThreadLocal variables and we found three suspect classes:
>>
>> 1)
>> org.apache.cocoon.components.treeprocessor.variables.VariableResolverFactory
>> Here there is a private static ThreadLocal called disposableCollector
>> that is set but never read or cleaned. In our tests we saw this variable
>> was set around 10 different times with different values (allways instances
>> of ArrayList) only with the first few requests.
>> We commented out this variable and tested the app. Unfortunately without
>> success. The out of memory was still there.
>>
>> 2) org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler
>> Here there is a private final ThreadLocal variable called
>> componentHolder. This variable is cleaned in the run() method, using a
>> set(null) on the TreadLocal. We've found lots of forums saying that
>> remove() method should be used instead of the set(null) to clean a
>> ThreadLocal, and that the set(null) appoach could lead to memory leaks
>> problems. Also, we found the remove() method was added in the Java 1.5 to
>> replace the dangerous set(null) approach.
>> So, we replaced the set(null) for a remove(). After a some tests, we saw
>> this cleaning line was executed a lot of times (almos 100 times) per
>> request.
>> Fortunately, this patch did work!!! or at least in part. The OutOfMemory
>> errors dissapeared, and now the memory consumption is much more normal.
>>
>> Although we have a more stable service (before, the webapp died in 2-3
>> hs, and now after several days working, we haven't seen any OutOfMemory
>> error), the memory consumption shows a very little slight upward trend.
>> Also, the clearThreadLocalMaps errors still came up when tomcat stops.
>>
>> Looking forward for your comments.
>> Regards
>>
>> Nestor Oviedo
>>
>> Servicio de Difusión de la Creación Intelectual -
>> http://sedici.unlp.edu.ar
>> Universidad Nacional de La Plata
>> Buenos Aires, Argentina
>>
>> On Thu, May 17, 2012 at 7:33 PM, Mark Diggory <mdigg...@atmire.com>wrote:
>>
>>> I theory, even 1GB of memory dedicated to DSpace would be sufficient if
>>> even necessary at all.
>>>
>>> __MultiThreadedHttpConnectionMan__ager cleanup]
>>> org.apache.solr.common.util.__DateUtil.__ThreadLocalDateFormat
>>>
>>> This is suspicious and seems like a bug. Do you have any new
>>> customizations opening http connections on the backend with solr?
>>>
>>> Mark
>>>
>>>
>>> On Thu, May 17, 2012 at 8:20 AM, Tim Donohue <tdono...@duraspace.org>wrote:
>>>
>>>> Hi Nestor,
>>>>
>>>> I would actually suspect that 2GB of heap memory should be plenty, in a
>>>> normal DSpace install.
>>>>
>>>> Have you been able to track any more information about when these
>>>> "OutOfMemoryError: Java heap space" errors are occurring?
>>>>
>>>> You had mentioned you are running "1.8.2 (with our extensions)". What
>>>> extensions are you using? Are you sure that each of those extensions is
>>>> properly closing any DSpace Context objects it is using? In DSpace,
>>>> unclosed Context objects can cause DB connection issues or even memory
>>>> usage issues. (We are very careful to ensure that these Context objects
>>>> are freed up / closed in the main DSpace code. But, obviously we don't
>>>> have the same control over third-party or custom extensions.)
>>>>
>>>> As for the "ThreadLocal" errors, here's a reference online which seems
>>>> to detail the same issue:
>>>>
>>>> http://stackoverflow.com/questions/5292349/this-is-very-likely-to-create-a-memory-leak-tomcat
>>>>
>>>> It links off to the Tomcat Memory Leak Protection page:
>>>> http://wiki.apache.org/tomcat/MemoryLeakProtection
>>>>
>>>> From the messages you sent, it looks like these may be coming from
>>>> Cocoon (which is the underlying technology that XMLUI is based on). As
>>>> mentioned in the Tomcat instructions above, it looks like as of Tomcat
>>>> 7.0.6 and above, Tomcat will now forceably stop any "ThreadLocal" leaks
>>>> it notices on shutdown.
>>>>
>>>> That's all just based on quick searches around. It's possible there's
>>>> something we can do in the DSpace XMLUI to kill these "ThreadLocals" on
>>>> shutdown (I'd need to dig around more to be sure). But it's also very
>>>> possible that Cocoon is the one that is failing to shutdown all its
>>>> thread (and unfortunately we don't have control over Cocoon).
>>>>
>>>> Has anyone else on this list run across these "ThreadLocal" issues
>>>> before?
>>>>
>>>> At the very least, it looks like an upgrade to Tomcat 7.0.6 or above
>>>> would mean that the ThreadLocals will be cleaned up by Tomcat
>>>> automatically.
>>>>
>>>> - Tim
>>>>
>>>>
>>>>
>>>> On 5/17/2012 9:54 AM, Nestor Oviedo wrote:
>>>> > Hi Tim, thanks for your answer.
>>>> > I'd like your opinion about the amount of memory needed for a
>>>> repository
>>>> > with about 20000 items. Maybe 2GB for the heap isn't enough (the
>>>> PermGem
>>>> > has a limit of 256MB).
>>>> >
>>>> > Now, I wonder why tomcat log those errors when I stop it ( The web
>>>> > application [] created a ThreadLocal with key of type
>>>> > [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@__4b70ca96])
>>>> and a
>>>> > value of type [org.apache.cocoon.__serialization.XMLSerializer] (value
>>>> > [org.apache.cocoon.__serialization.XMLSerializer@__5a04648b]) but
>>>> failed
>>>> > to remove it when the web application was stopped. This is very likely
>>>> > to create a memory leak.).
>>>> > Also, assuming 2GB is enough heap memory, it's at least suspect that
>>>> the
>>>> > Old Gen memory keeps growing up to the memory limit and never gets
>>>> > Garbage collected ...
>>>> >
>>>> > Regards.
>>>> > Nestor
>>>> >
>>>> >
>>>> > On Thu, May 17, 2012 at 11:11 AM, Tim Donohue <tdono...@duraspace.org
>>>> > <mailto:tdono...@duraspace.org>> wrote:
>>>> >
>>>> > Hi Nestor,
>>>> >
>>>> > Which type of "OutOfMemory" error do you sometimes see in your
>>>> > Tomcat logs? There are two possible "OutOfMemoryErrors" you may
>>>> > encounter:
>>>> >
>>>> > (1) java.lang.OutOfMemoryError: Java heap space
>>>> >
>>>> > This one means you may need to tweak the -Xmx and/or -Xms settings
>>>> > for Tomcat (though it looks like you may already be doing that).
>>>> >
>>>> >
>>>> > (2) java.lang.OutOfMemoryError: PermGen space
>>>> >
>>>> > This one means that you need to increase the "-XX:MaxPermSize"
>>>> > settings for Tomcat.
>>>> >
>>>> > For more information on both OutOfMemory errors, and how you can
>>>> > tweak your Tomcat memory settings see:
>>>> >
>>>> https://wiki.duraspace.org/__display/DSDOC18/Performance+__Tuning+DSpace
>>>> > <
>>>> https://wiki.duraspace.org/display/DSDOC18/Performance+Tuning+DSpace>
>>>> >
>>>> > - Tim
>>>> >
>>>> >
>>>> > On 5/17/2012 8:03 AM, Nestor Oviedo wrote:
>>>> >
>>>> > Hi all.
>>>> > We have a Dspace 1.8.2 (with our extensions), and almost 20000
>>>> > items and
>>>> > 1000 collections/communities. Our Tomcat 6 get started with
>>>> -Xmx=2GB
>>>> > (Java 1.6.0_26 64bit).
>>>> > When tomcat starts everything work fine, but after a few
>>>> hours the
>>>> > memory is full and the application gets unresponsive. Some
>>>> times
>>>> > we see
>>>> > an OutOfMemory error in the log.
>>>> > When we stop the tomcat, the log shows thousands of lines
>>>> like the
>>>> > following (with many different types and values):
>>>> >
>>>> > org.apache.catalina.loader.__WebappClassLoader
>>>> > clearReferencesThreads
>>>> > GRAVE: The web application [] appears to have started a thread
>>>> > named [__MultiThreadedHttpConnectionMan__ager cleanup] but has
>>>> > failed to stop it. This is very likely to create a memory
>>>> leak.
>>>> >
>>>> > org.apache.catalina.loader.__WebappClassLoader
>>>> > clearReferencesThreads
>>>> > GRAVE: The web application [] appears to have started a thread
>>>> > named [Timer-0] but has failed to stop it. This is very likely
>>>> > to create a memory leak.
>>>> >
>>>> > org.apache.catalina.loader.__WebappClassLoader
>>>> clearThreadLocalMap
>>>> > GRAVE: The web application [] created a ThreadLocal with key
>>>> of
>>>> > type [java.lang.ThreadLocal] (value
>>>> > [java.lang.ThreadLocal@__29a9b1ec]) and a value of type
>>>> > [org.apache.xerces.parsers.__SAXParser] (value
>>>> > [org.apache.xerces.parsers.__SAXParser@8a61d64]) but failed
>>>> to
>>>> > remove it when the web application was stopped. This is very
>>>> > likely to create a memory leak.
>>>> >
>>>> > org.apache.catalina.loader.__WebappClassLoader
>>>> clearThreadLocalMap
>>>> > GRAVE: The web application [] created a ThreadLocal with key
>>>> of
>>>> > type [java.lang.ThreadLocal] (value
>>>> > [java.lang.ThreadLocal@__43e01252]) and a value of type
>>>> > [org.dspace.app.xmlui.wing.__IncludePageMeta] (value
>>>> > [org.dspace.app.xmlui.wing.__IncludePageMeta@2bd2e84e]) but
>>>> > failed to remove it when the web application was stopped. This
>>>> > is very likely to create a memory leak.
>>>> >
>>>> > org.apache.catalina.loader.__WebappClassLoader
>>>> clearThreadLocalMap
>>>> > GRAVE: The web application [] created a ThreadLocal with key
>>>> of
>>>> > type [null] (value [null]) and a value of type
>>>> > [org.apache.cocoon.__transformation.__TraxTransformer] (value
>>>> > [org.apache.cocoon.__transformation.__TraxTransformer@23916f5c
>>>> ])
>>>> > but failed to remove it when the web application was stopped.
>>>> > This is very likely to create a memory leak.
>>>> >
>>>> > org.apache.catalina.loader.__WebappClassLoader
>>>> clearThreadLocalMap
>>>> > GRAVE: The web application [] created a ThreadLocal with key
>>>> of
>>>> > type [java.lang.ThreadLocal] (value
>>>> > [java.lang.ThreadLocal@__21261342]) and a value of type
>>>> > [org.apache.cocoon.components.__xslt.TraxProcessor] (value
>>>> > [org.apache.cocoon.components.__xslt.TraxProcessor@4aba630e])
>>>> > but failed to remove it when the web application was stopped.
>>>> > This is very likely to create a memory leak.
>>>> >
>>>> > org.apache.catalina.loader.__WebappClassLoader
>>>> clearThreadLocalMap
>>>> > GRAVE: The web application [] created a ThreadLocal with key
>>>> of
>>>> > type
>>>> >
>>>> [org.apache.solr.common.util.__DateUtil.__ThreadLocalDateFormat]
>>>> > (value
>>>> >
>>>> [org.apache.solr.common.util.__DateUtil$__ThreadLocalDateFormat@12bd5276__
>>>> ])
>>>> > and a value of type [java.text.SimpleDateFormat] (value
>>>> > [java.text.SimpleDateFormat@__5af7aed5]) but failed to
>>>> remove it
>>>> > when the web application was stopped. This is very likely to
>>>> > create a memory leak.
>>>> >
>>>> > org.apache.catalina.loader.__WebappClassLoader
>>>> clearThreadLocalMap
>>>> > GRAVE: The web application [] created a ThreadLocal with key
>>>> of
>>>> > type [java.lang.ThreadLocal] (value
>>>> > [java.lang.ThreadLocal@__4b70ca96]) and a value of type
>>>> > [org.apache.cocoon.__serialization.XMLSerializer] (value
>>>> > [org.apache.cocoon.__serialization.XMLSerializer@__5a04648b])
>>>> > but failed to remove it when the web application was stopped.
>>>> > This is very likely to create a memory leak.
>>>> >
>>>> >
>>>> > I've found some old info about this issue, but it seems there
>>>> is
>>>> > no fix:
>>>> >
>>>> http://www.mail-archive.com/__dspace-tech@lists.sourceforge.__net/msg11969.html
>>>> > <
>>>> http://www.mail-archive.com/dspace-tech@lists.sourceforge.net/msg11969.html
>>>> >
>>>> > http://dspace.2283337.n4.__
>>>> nabble.com/Tomcat-memory-leak-__warning-prevents-restarting-__td3544827.html
>>>> > <
>>>> http://dspace.2283337.n4.nabble.com/Tomcat-memory-leak-warning-prevents-restarting-td3544827.html
>>>> >
>>>> >
>>>> http://stackoverflow.com/__questions/5292349/this-is-__very-likely-to-create-a-__memory-leak-tomcat
>>>> > <
>>>> http://stackoverflow.com/questions/5292349/this-is-very-likely-to-create-a-memory-leak-tomcat
>>>> >
>>>> >
>>>> > Is there somenting new on this issue?
>>>> >
>>>> > Thanks in advance
>>>> > Nestor
>>>> >
>>>> >
>>>> >
>>>> ------------------------------__------------------------------__------------------
>>>> > Live Security Virtual Conference
>>>> > Exclusive live event will cover all the ways today's security
>>>> and
>>>> > threat landscape has changed and how IT managers can respond.
>>>> > Discussions
>>>> > will include endpoint security, mobile security and the latest
>>>> > in malware
>>>> > threats.
>>>> > http://www.accelacomm.com/jaw/__sfrnl04242012/114/50122263/
>>>> > <http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/>
>>>> >
>>>> >
>>>> >
>>>> > _________________________________________________
>>>> > Dspace-devel mailing list
>>>> > Dspace-devel@lists.__sourceforge.net
>>>> > <mailto:Dspace-devel@lists.sourceforge.net>
>>>> > https://lists.sourceforge.net/__lists/listinfo/dspace-devel
>>>> > <https://lists.sourceforge.net/lists/listinfo/dspace-devel>
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> ------------------------------------------------------------------------------
>>>> > Live Security Virtual Conference
>>>> > Exclusive live event will cover all the ways today's security and
>>>> > threat landscape has changed and how IT managers can respond.
>>>> Discussions
>>>> > will include endpoint security, mobile security and the latest in
>>>> malware
>>>> > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>> >
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > Dspace-devel mailing list
>>>> > Dspace-devel@lists.sourceforge.net
>>>> > https://lists.sourceforge.net/lists/listinfo/dspace-devel
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Live Security Virtual Conference
>>>> Exclusive live event will cover all the ways today's security and
>>>> threat landscape has changed and how IT managers can respond.
>>>> Discussions
>>>> will include endpoint security, mobile security and the latest in
>>>> malware
>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>> _______________________________________________
>>>> Dspace-devel mailing list
>>>> Dspace-devel@lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/dspace-devel
>>>>
>>>
>>>
>>>
>>> --
>>> [image: @mire Inc.]
>>> *Mark Diggory *(Schedule a Meeting <https://tungle.me/markdiggory>)
>>> *2888 Loker Avenue East, Suite 305, Carlsbad, CA. 92010*
>>> *Esperantolaan 4, Heverlee 3001, Belgium*
>>> http://www.atmire.com
>>>
>>>
>>>
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Dspace-devel mailing list
>> Dspace-devel@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/dspace-devel
>>
>>
>
>
> --
> [image: @mire Inc.]
> *Mark Diggory *(Schedule a Meeting <https://tungle.me/markdiggory>)
> *2888 Loker Avenue East, Suite 305, Carlsbad, CA. 92010*
> *Esperantolaan 4, Heverlee 3001, Belgium*
> http://www.atmire.com
>
>
>
------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
_______________________________________________
Dspace-devel mailing list
Dspace-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/dspace-devel
