Hi Nestor, I would actually suspect that 2GB of heap memory should be plenty, in a normal DSpace install.
Have you been able to track any more information about when these "OutOfMemoryError: Java heap space" errors are occurring? You had mentioned you are running "1.8.2 (with our extensions)". What extensions are you using? Are you sure that each of those extensions is properly closing any DSpace Context objects it is using? In DSpace, unclosed Context objects can cause DB connection issues or even memory usage issues. (We are very careful to ensure that these Context objects are freed up / closed in the main DSpace code. But, obviously we don't have the same control over third-party or custom extensions.) As for the "ThreadLocal" errors, here's a reference online which seems to detail the same issue: http://stackoverflow.com/questions/5292349/this-is-very-likely-to-create-a-memory-leak-tomcat It links off to the Tomcat Memory Leak Protection page: http://wiki.apache.org/tomcat/MemoryLeakProtection From the messages you sent, it looks like these may be coming from Cocoon (which is the underlying technology that XMLUI is based on). As mentioned in the Tomcat instructions above, it looks like as of Tomcat 7.0.6 and above, Tomcat will now forceably stop any "ThreadLocal" leaks it notices on shutdown. That's all just based on quick searches around. It's possible there's something we can do in the DSpace XMLUI to kill these "ThreadLocals" on shutdown (I'd need to dig around more to be sure). But it's also very possible that Cocoon is the one that is failing to shutdown all its thread (and unfortunately we don't have control over Cocoon). Has anyone else on this list run across these "ThreadLocal" issues before? At the very least, it looks like an upgrade to Tomcat 7.0.6 or above would mean that the ThreadLocals will be cleaned up by Tomcat automatically. - Tim On 5/17/2012 9:54 AM, Nestor Oviedo wrote: > Hi Tim, thanks for your answer. > I'd like your opinion about the amount of memory needed for a repository > with about 20000 items. Maybe 2GB for the heap isn't enough (the PermGem > has a limit of 256MB). > > Now, I wonder why tomcat log those errors when I stop it ( The web > application [] created a ThreadLocal with key of type > [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@__4b70ca96]) and a > value of type [org.apache.cocoon.__serialization.XMLSerializer] (value > [org.apache.cocoon.__serialization.XMLSerializer@__5a04648b]) but failed > to remove it when the web application was stopped. This is very likely > to create a memory leak.). > Also, assuming 2GB is enough heap memory, it's at least suspect that the > Old Gen memory keeps growing up to the memory limit and never gets > Garbage collected ... > > Regards. > Nestor > > > On Thu, May 17, 2012 at 11:11 AM, Tim Donohue <tdono...@duraspace.org > <mailto:tdono...@duraspace.org>> wrote: > > Hi Nestor, > > Which type of "OutOfMemory" error do you sometimes see in your > Tomcat logs? There are two possible "OutOfMemoryErrors" you may > encounter: > > (1) java.lang.OutOfMemoryError: Java heap space > > This one means you may need to tweak the -Xmx and/or -Xms settings > for Tomcat (though it looks like you may already be doing that). > > > (2) java.lang.OutOfMemoryError: PermGen space > > This one means that you need to increase the "-XX:MaxPermSize" > settings for Tomcat. > > For more information on both OutOfMemory errors, and how you can > tweak your Tomcat memory settings see: > https://wiki.duraspace.org/__display/DSDOC18/Performance+__Tuning+DSpace > <https://wiki.duraspace.org/display/DSDOC18/Performance+Tuning+DSpace> > > - Tim > > > On 5/17/2012 8:03 AM, Nestor Oviedo wrote: > > Hi all. > We have a Dspace 1.8.2 (with our extensions), and almost 20000 > items and > 1000 collections/communities. Our Tomcat 6 get started with -Xmx=2GB > (Java 1.6.0_26 64bit). > When tomcat starts everything work fine, but after a few hours the > memory is full and the application gets unresponsive. Some times > we see > an OutOfMemory error in the log. > When we stop the tomcat, the log shows thousands of lines like the > following (with many different types and values): > > org.apache.catalina.loader.__WebappClassLoader > clearReferencesThreads > GRAVE: The web application [] appears to have started a thread > named [__MultiThreadedHttpConnectionMan__ager cleanup] but has > failed to stop it. This is very likely to create a memory leak. > > org.apache.catalina.loader.__WebappClassLoader > clearReferencesThreads > GRAVE: The web application [] appears to have started a thread > named [Timer-0] but has failed to stop it. This is very likely > to create a memory leak. > > org.apache.catalina.loader.__WebappClassLoader clearThreadLocalMap > GRAVE: The web application [] created a ThreadLocal with key of > type [java.lang.ThreadLocal] (value > [java.lang.ThreadLocal@__29a9b1ec]) and a value of type > [org.apache.xerces.parsers.__SAXParser] (value > [org.apache.xerces.parsers.__SAXParser@8a61d64]) but failed to > remove it when the web application was stopped. This is very > likely to create a memory leak. > > org.apache.catalina.loader.__WebappClassLoader clearThreadLocalMap > GRAVE: The web application [] created a ThreadLocal with key of > type [java.lang.ThreadLocal] (value > [java.lang.ThreadLocal@__43e01252]) and a value of type > [org.dspace.app.xmlui.wing.__IncludePageMeta] (value > [org.dspace.app.xmlui.wing.__IncludePageMeta@2bd2e84e]) but > failed to remove it when the web application was stopped. This > is very likely to create a memory leak. > > org.apache.catalina.loader.__WebappClassLoader clearThreadLocalMap > GRAVE: The web application [] created a ThreadLocal with key of > type [null] (value [null]) and a value of type > [org.apache.cocoon.__transformation.__TraxTransformer] (value > [org.apache.cocoon.__transformation.__TraxTransformer@23916f5c]) > but failed to remove it when the web application was stopped. > This is very likely to create a memory leak. > > org.apache.catalina.loader.__WebappClassLoader clearThreadLocalMap > GRAVE: The web application [] created a ThreadLocal with key of > type [java.lang.ThreadLocal] (value > [java.lang.ThreadLocal@__21261342]) and a value of type > [org.apache.cocoon.components.__xslt.TraxProcessor] (value > [org.apache.cocoon.components.__xslt.TraxProcessor@4aba630e]) > but failed to remove it when the web application was stopped. > This is very likely to create a memory leak. > > org.apache.catalina.loader.__WebappClassLoader clearThreadLocalMap > GRAVE: The web application [] created a ThreadLocal with key of > type > [org.apache.solr.common.util.__DateUtil.__ThreadLocalDateFormat] > (value > > [org.apache.solr.common.util.__DateUtil$__ThreadLocalDateFormat@12bd5276__]) > and a value of type [java.text.SimpleDateFormat] (value > [java.text.SimpleDateFormat@__5af7aed5]) but failed to remove it > when the web application was stopped. This is very likely to > create a memory leak. > > org.apache.catalina.loader.__WebappClassLoader clearThreadLocalMap > GRAVE: The web application [] created a ThreadLocal with key of > type [java.lang.ThreadLocal] (value > [java.lang.ThreadLocal@__4b70ca96]) and a value of type > [org.apache.cocoon.__serialization.XMLSerializer] (value > [org.apache.cocoon.__serialization.XMLSerializer@__5a04648b]) > but failed to remove it when the web application was stopped. > This is very likely to create a memory leak. > > > I've found some old info about this issue, but it seems there is > no fix: > > http://www.mail-archive.com/__dspace-tech@lists.sourceforge.__net/msg11969.html > > <http://www.mail-archive.com/dspace-tech@lists.sourceforge.net/msg11969.html> > > http://dspace.2283337.n4.__nabble.com/Tomcat-memory-leak-__warning-prevents-restarting-__td3544827.html > > <http://dspace.2283337.n4.nabble.com/Tomcat-memory-leak-warning-prevents-restarting-td3544827.html> > > http://stackoverflow.com/__questions/5292349/this-is-__very-likely-to-create-a-__memory-leak-tomcat > > <http://stackoverflow.com/questions/5292349/this-is-very-likely-to-create-a-memory-leak-tomcat> > > Is there somenting new on this issue? > > Thanks in advance > Nestor > > > > ------------------------------__------------------------------__------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. > Discussions > will include endpoint security, mobile security and the latest > in malware > threats. > http://www.accelacomm.com/jaw/__sfrnl04242012/114/50122263/ > <http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/> > > > > _________________________________________________ > Dspace-devel mailing list > Dspace-devel@lists.__sourceforge.net > <mailto:Dspace-devel@lists.sourceforge.net> > https://lists.sourceforge.net/__lists/listinfo/dspace-devel > <https://lists.sourceforge.net/lists/listinfo/dspace-devel> > > > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > > > > _______________________________________________ > Dspace-devel mailing list > Dspace-devel@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/dspace-devel ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Dspace-devel mailing list Dspace-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-devel
