Re: [Dspace-tech] Tomcat memory leak warning prevents restarting
Hi, I'm sure that this issue has already been discussed here more than once and as I can remember without any final and useful universal solution (maybe there's not one). I have been fighting with it for years (and still don't know where lies the bug). My solution is to restart Tomcat (via cron) once a day and use a special piece of code to do this - I test if Tomcat is really stopped correctly and if not I kill the Tomcat process (bash script): /opt/tomcat/bin/shutdown.sh echo Waiting 45 second for Tomcat shutdown... # This is necessary, because shutting down process can take some time, # 45 second is enough according to my experience sleep 45 RUNNING=`ps ax | grep opt\/tomcat | grep 2048` if [ -z $RUNNING ] then echo Tomcat has been stopped...; else echo Shutdown failure... Trying to do hard kill. PID=`echo $RUNNING | cut -d ' ' -f 1` kill -9 $PID # give system some time to kill it sleep 3 # check it again RUNNING=`ps ax | grep opt\/tomcat | grep 2048` if [ -z $RUNNING ] then echo Tomcat has been killed with -9. else echo Cannot kill Tomcat. Administrator needed! exit 1 fi fi This shutdown Tomcat for sure. Vlastik Vlastimil Krejčíř Library and Information Centre, Institute of Computer Science Masaryk University, Brno, Czech Republic Email: krejcir (at) ics (dot) muni (dot) cz Phone: +420 549 49 3872 ICQ: 163963217 Jabber: kre...@jabber.org On Tue, 24 May 2011, Jason Stirnaman wrote: I found a long, interesting thread from September 2010 at http://www.mail-archive.com/dspace-tech@lists.sourceforge.net/msg11969.html around this topic and an excellent description of the Tomcat MemoryLeakProtection at http://wiki.apache.org/tomcat/MemoryLeakProtection. I obviously didn't do my homework before posting. It looks like Tomcat 7.0.6 *might* solve this problem. However, I'm still unclear whether there is anything else I can do now to work around it - to at least stop the leak detection from thwarting my restarts? As Tim suggested in that earlier thread I'm including more details from my production server while Tomcat is up and DSpace is running OK. Hardware: Solaris 10 Zone virtual machine with 1167MHz processor Output from top (I'm open to better methods, but I'm limited to the single Zone.) 40 processes: 39 sleeping, 1 on cpu CPU states: 93.5% idle, 3.7% user, 2.8% kernel, 0.0% iowait, 0.0% swap Kernel: 18117 ctxsw, 6164 trap, 8377 intr, 45826 syscall, 4 fork, 543 flt, 16 pgin, 40 pgout Memory: 64G phys mem, 40G free mem, 4104M total swap, 4104M free swap RAM: CATALINA_OPTS value=-Xms256m -Xmx1028m -XX:MaxPermSize=256m I recently increased RAM to 1028 from (I think) 256M and I doubled PermGen. Since then the Java process has consistently been using around 490M and usually less than 1% CPU. Postgres uses around 520M. DSpace: version 1.7.1 with XMLUI and Discovery Issues: Prior to my increasing the allotted Catalina RAM and Permgen, Tomcat would stop responding around every 3 days (? number of requests) due to the database pool filling up (cannot get a connection - pool exhausted). I decided to also restart Tomcat every night to prevent this from happening. That's when I encountered the memory leak warnings preventing my restarts. Jason Jason Stirnaman Biomedical Librarian, Digital Projects A.R. Dykes Library, University of Kansas Medical Center jstirna...@kumc.edu 913-588-7319 On 5/23/2011 at 01:02 PM, in message 4DDAA130.D22 : 5 : 23711, Jason Stirnaman wrote: I get the error below when stopping Tomcat. In this particular case, I was trying to use Solaris' SMF to restart the Tomcat service. Instead, it never completed shutting down and it stopped answering requests. I got this error stanza repeated over and over in the log. Any ideas what's going on? Thanks, Jason Details: DSpace 1.7.1 w/ Discovery Tomcat 6.0.32 Solaris 10 May 21, 2011 12:00:07 AM org.apache.coyote.http11.Http11Protocol pause INFO: Pausing Coyote HTTP/1.1 on http-80 May 21, 2011 12:00:08 AM org.apache.catalina.core.StandardService stop May 21, 2011 12:00:07 AM org.apache.coyote.http11.Http11Protocol pause May 21, 2011 12:00:07 AM org.apache.coyote.http11.Http11Protocol pause INFO: Pausing Coyote HTTP/1.1 on http-80 May 21, 2011 12:00:08 AM org.apache.catalina.core.StandardService stop INFO: Stopping service Catalina May 21, 2011 12:00:49 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [] appears to have started a thread named [MultiThreadedHttpConnectionMa nager cleanup] but has failed to stop it. This is very likely to create a memory leak. May 21,
Re: [Dspace-tech] Tomcat memory leak warning prevents restarting
I found a long, interesting thread from September 2010 at http://www.mail-archive.com/dspace-tech@lists.sourceforge.net/msg11969.html around this topic and an excellent description of the Tomcat MemoryLeakProtection at http://wiki.apache.org/tomcat/MemoryLeakProtection. I obviously didn't do my homework before posting. It looks like Tomcat 7.0.6 *might* solve this problem. However, I'm still unclear whether there is anything else I can do now to work around it - to at least stop the leak detection from thwarting my restarts? As Tim suggested in that earlier thread I'm including more details from my production server while Tomcat is up and DSpace is running OK. Hardware: Solaris 10 Zone virtual machine with 1167MHz processor Output from top (I'm open to better methods, but I'm limited to the single Zone.) 40 processes: 39 sleeping, 1 on cpu CPU states: 93.5% idle, 3.7% user, 2.8% kernel, 0.0% iowait, 0.0% swap Kernel: 18117 ctxsw, 6164 trap, 8377 intr, 45826 syscall, 4 fork, 543 flt, 16 pgin, 40 pgout Memory: 64G phys mem, 40G free mem, 4104M total swap, 4104M free swap RAM: CATALINA_OPTS value=-Xms256m -Xmx1028m -XX:MaxPermSize=256m I recently increased RAM to 1028 from (I think) 256M and I doubled PermGen. Since then the Java process has consistently been using around 490M and usually less than 1% CPU. Postgres uses around 520M. DSpace: version 1.7.1 with XMLUI and Discovery Issues: Prior to my increasing the allotted Catalina RAM and Permgen, Tomcat would stop responding around every 3 days (? number of requests) due to the database pool filling up (cannot get a connection - pool exhausted). I decided to also restart Tomcat every night to prevent this from happening. That's when I encountered the memory leak warnings preventing my restarts. Jason Jason Stirnaman Biomedical Librarian, Digital Projects A.R. Dykes Library, University of Kansas Medical Center jstirna...@kumc.edu 913-588-7319 On 5/23/2011 at 01:02 PM, in message 4DDAA130.D22 : 5 : 23711, Jason Stirnaman wrote: I get the error below when stopping Tomcat. In this particular case, I was trying to use Solaris' SMF to restart the Tomcat service. Instead, it never completed shutting down and it stopped answering requests. I got this error stanza repeated over and over in the log. Any ideas what's going on? Thanks, Jason Details: DSpace 1.7.1 w/ Discovery Tomcat 6.0.32 Solaris 10 May 21, 2011 12:00:07 AM org.apache.coyote.http11.Http11Protocol pause INFO: Pausing Coyote HTTP/1.1 on http-80 May 21, 2011 12:00:08 AM org.apache.catalina.core.StandardService stop May 21, 2011 12:00:07 AM org.apache.coyote.http11.Http11Protocol pause May 21, 2011 12:00:07 AM org.apache.coyote.http11.Http11Protocol pause INFO: Pausing Coyote HTTP/1.1 on http-80 May 21, 2011 12:00:08 AM org.apache.catalina.core.StandardService stop INFO: Stopping service Catalina May 21, 2011 12:00:49 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [] appears to have started a thread named [MultiThreadedHttpConnectionMa nager cleanup] but has failed to stop it. This is very likely to create a memory leak. May 21, 2011 12:00:49 AM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap SEVERE: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1de92e]) and a value of type [org.apache.xerces.parsers.SAXParser] (value [o rg.apache.xerces.parsers.SAXParser@17533df]) but failed to remove it when the web application was st opped. This is very likely to create a memory leak. May 21, 2011 12:00:49 AM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap SEVERE: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1b7fc4e]) and a value of type [org.apache.cocoon.environment.internal.Enviro nmentStack] (value [[]]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak. Jason Stirnaman Biomedical Librarian, Digital Projects A.R. Dykes Library, University of Kansas Medical Center jstirna...@kumc.edu 913-588-7319 -- vRanger cuts backup time in half-while increasing security. With the market-leading solution for virtual backup and recovery, you get blazing-fast, flexible, and affordable data protection. Download your free trial now. http://p.sf.net/sfu/quest-d2dcopy1___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
[Dspace-tech] Tomcat memory leak warning prevents restarting
I get the error below when stopping Tomcat. In this particular case, I was trying to use Solaris' SMF to restart the Tomcat service. Instead, it never completed shutting down and it stopped answering requests. I got this error stanza repeated over and over in the log. Any ideas what's going on? Thanks, Jason Details: DSpace 1.7.1 w/ Discovery Tomcat 6.0.32 Solaris 10 May 21, 2011 12:00:07 AM org.apache.coyote.http11.Http11Protocol pause INFO: Pausing Coyote HTTP/1.1 on http-80 May 21, 2011 12:00:08 AM org.apache.catalina.core.StandardService stop May 21, 2011 12:00:07 AM org.apache.coyote.http11.Http11Protocol pause May 21, 2011 12:00:07 AM org.apache.coyote.http11.Http11Protocol pause INFO: Pausing Coyote HTTP/1.1 on http-80 May 21, 2011 12:00:08 AM org.apache.catalina.core.StandardService stop INFO: Stopping service Catalina May 21, 2011 12:00:49 AM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads SEVERE: The web application [] appears to have started a thread named [MultiThreadedHttpConnectionMa nager cleanup] but has failed to stop it. This is very likely to create a memory leak. May 21, 2011 12:00:49 AM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap SEVERE: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1de92e]) and a value of type [org.apache.xerces.parsers.SAXParser] (value [o rg.apache.xerces.parsers.SAXParser@17533df]) but failed to remove it when the web application was st opped. This is very likely to create a memory leak. May 21, 2011 12:00:49 AM org.apache.catalina.loader.WebappClassLoader clearThreadLocalMap SEVERE: The web application [] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1b7fc4e]) and a value of type [org.apache.cocoon.environment.internal.Enviro nmentStack] (value [[]]) but failed to remove it when the web application was stopped. This is very likely to create a memory leak. Jason Stirnaman Biomedical Librarian, Digital Projects A.R. Dykes Library, University of Kansas Medical Center jstirna...@kumc.edu 913-588-7319 -- What Every C/C++ and Fortran developer Should Know! Read this article and learn how Intel has extended the reach of its next-generation tools to help Windows* and Linux* C/C++ and Fortran developers boost performance applications - including clusters. http://p.sf.net/sfu/intel-dev2devmay___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
