Thanks so much Chris this has been very very helpful ... appreciate you taking
time out. Would stopping Tomcat6 service in Windows 2008 R2 take care of this
"orphan threads" if you will ? Also, does force kill taskkill tomcat6 help
either ?
I understand its not a premanent solution but as a stopgap for now ? If so , we
can put it as part of daily cycle to bounce tomcat6.
Many thanks.
> Date: Tue, 2 Apr 2013 15:58:53 -0400
> From: ch...@christopherschultz.net
> To: users@tomcat.apache.org
> Subject: Re: Re : Memory leak in Tomcat 6.0.35 ( 64 bit)
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Saumil,
>
> On 4/1/13 11:01 PM, saumil shah wrote:
> > I recently deployed one of the COTS products SAP Business Objects.
> > When the product was deployed , everything seemed to run fine but
> > yesterday we started experiencing "Service is unavailable" error ,
> > upon enabling DEBUG logs in Tomcat , we saw the error below with
> > memory leak message. We were wondering what could have caused it
> > ....
> >
> > The Tomcat is 64 bit , JVM is 64 bit but the applications deployed
> > are 32 bit webapps .... we say Tomcat6 process becoming
> > unresponsive around 2GB mark.
> >
> >
> > Apr 1, 2013 8:53:18 PM org.apache.coyote.http11.Http11Protocol
> > pauseINFO: Pausing Coyote HTTP/1.1 on http-8080Apr 1, 2013 8:53:19
> > PM org.apache.catalina.core.StandardService stopINFO: Stopping
> > service CatalinaApr 1, 2013 8:53:19 PM
> > org.apache.catalina.loader.WebappClassLoader
> > clearReferencesThreadsSEVERE: The web application
> > [/AnalyticalReporting] appears to have started a thread named
> > [Business Objects - Sessions Clean up] but has failed to stop it.
> > This is very likely to create a memory leak.
>
> This means what it says it means: your webapp (not Tomcat) started a
> thread that wasn't stopped by the time the webapp was stopped. This
> will pin the webapp's ClassLoader in memory and you'll get a whole
> bunch of stuff that can't be GC'd. The simple explanation is that you
> need to be sure that your threads are stopped. The solution may be
> complicated.
>
> Presumably any component that starts a thread also knows how to stop
> that same thread. Check the documentation for those components to make
> sure that you are shutting them down properly (usually in a
> ServletContextListener, etc.).
>
> Sometimes, it takes a few seconds for a thread to complete its work
> and actually terminate, and the messages you get from Tomcat are
> inaccurate. That's for you to determine.
>
> > Apr 1, 2013 8:53:19 PM
> > org.apache.catalina.loader.WebappClassLoader
> > clearReferencesThreadsSEVERE: The web application
> > [/AnalyticalReporting] appears to have started a thread named
> > [AWT-Windows] but has failed to stop it. This is very likely to
> > create a memory leak.
>
> Read the documentation for the JreMemoryLeakLeakPreventionListener.
> This is a standard Tomcat component that can be used to work-around
> well-known memory leaks, etc. that are triggered by certain JVM calls.
> There are specific configurations for preventing AWT threads from
> being problematic, but they are not enabled by default. You probably
> want to enable them if you are using AWT for anything.
>
> > Apr 1, 2013 8:53:19 PM
> > org.apache.catalina.loader.WebappClassLoader
> > clearReferencesThreadsSEVERE: The web application
> > [/AnalyticalReporting] appears to have started a thread named
> > [ORBacus:Client:SenderThread] but has failed to stop it. This is
> > very likely to create a memory leak.
>
> Anything with "ORB" in the name usually means a CORBA-related
> component: check to see that anything that start threads is also
> stopping them.
>
> > Apr 1, 2013 8:53:19 PM
> > org.apache.catalina.loader.WebappClassLoader
> > clearThreadLocalMapSEVERE: The web application
> > [/AnalyticalReporting] created a ThreadLocal with key of type
> > [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@63e3d2dd])
> > and a value of type [com.businessobjects.wp.xml.jaxp.XMLJaxpParser]
> > (value [com.businessobjects.wp.xml.jaxp.XMLJaxpParser@228de67]) but
> > failed to remove it when the web application was stopped. This is
> > very likely to create a memory leak.
>
> Looks like your Business Objects version has a bug in it that does not
> clear-out ThreadLocal values when requests are completed. Talk to them
> about this problem.
>
> > Apr 1, 2013 8:53:20 PM
> > org.apache.catalina.loader.WebappClassLoader
> > clearThreadLocalMapSEVERE: The web application [/VoyagerClient]
> > created a ThreadLocal with key of type
> > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl.ThreadCounterImpl]
> >
> >
> (value
> > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl$ThreadCounterImpl@13123b1c])
> >
> >
> and a value of type
> > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl.ThreadCounterImpl.Counter]
> >
> >
> (value
> > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl$ThreadCounterImpl$Counter@11e6b50c])
> >
> >
> but failed to remove it when the web application was stopped. This is
> > very likely to create a memory leak.Apr 1, 2013 8:53:20 PM
> > org.apache.catalina.loader.WebappClassLoader
> > clearThreadLocalMapSEVERE: The web application [/VoyagerClient]
> > created a ThreadLocal with key of type
> > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl.ThreadCounterImpl]
> >
> >
> (value
> > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl$ThreadCounterImpl@41783002])
> >
> >
> and a value of type
> > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl.ThreadCounterImpl.Counter]
> >
> >
> (value
> > [org.aspectj.runtime.internal.cflowstack.ThreadStackFactoryImpl$ThreadCounterImpl$Counter@3997f4e2])
> >
> >
> but failed to remove it when the web application was stopped. This is
> > very likely to create a memory leak.
>
> Looks like AspectJ has a similar problem: ThreadLocals need to be
> removed from threads after every request.
>
> You are probably seeing a pattern, here, so I won't bother looking at
> every other error message you have posted.
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG/MacGPG2 v2.0.17 (Darwin)
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJRWzh6AAoJEBzwKT+lPKRYUoQQALSx4IVs9Pd/QfAQKeB6klIP
> 9bfoG8bRk/fcqdQiJfaNJGw8xaKm0WVMalYB/U1xH7dZSdFDlz4LoQuitRLEMnsX
> WyRspvI3N3KUIhGUuU9GiVz0nn1vwaQF87K81g/7IpCOKcFsWfruHnict2UMjrg1
> pEI7xiNoC3oPsbd6+APhAy9JUip8gt4j7I0VUVAvCOg4L7PLvKGY3y4TXy594uFO
> D/QZ6cN/9SdImOO9wunwHk1+Ot3B3ZKuurdbJk6djXyt6K2Et4lHGJ8KldhZfgfj
> QdQjRg/ybTune+gWJ9QTPo5p7FOgTJxS2x94a+pXx9aTkCBqbBiKCWWU47O+wtf4
> P5Nc71m0N1zeLsSjThjDoaH8DA77AbwtNWHQEwemvqp6y8CQy8ypp/f58lrzdOza
> lFhER7mMtzlNNCHdxZKUEY9yBaEIzWAV9vPP51yWEUBCJaGRZ9icTZNvIQPUAcLB
> JzoIfgEHFjrVlLXCKb0phIq6He0B/yDZ4xwA2rn9bXKhRJEuZbBoYXTM6jsRJDNk
> ab/wMZKXfK5On5N9U9TVj3kzGni35GI2wApJ9OGvYWdKv+9oX4vKQNOl8xAdm8IM
> wAE3ooZk0XvsotLphMP+V9PHXNCaeZeWORsNKKUlkxZDAPU5vdmaVfcdU0um2lm2
> yqAz4cJjThxGwbh62xe3
> =UtMt
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
