RE: Tomcat 8 Hung State

[Caldarale, Charles R]

> From: Theo Sweeny [mailto:theo.swe...@avios.com] 
> Subject: Tomcat 8 Hung State

> yesterday Tomcat 8 went into a hung state, because an external API went 
> offline.

> Eventually the external service was restored but Tomcat was unable to 
> recovery from it.

> When a new connection is sent to Tomcat the connection hangs as no additional 
> thread 
> can be allocated to the new request.

> Here is an example of the log -

> com.spring.application.repository.product.price.IlogPriceRepository.getPrices_aroundBody0
> (IlogPriceRepository.java:104)

Note that this is application, not Tomcat, code.

> Are there any ideas as to why Tomcat is unable to free up its threads?

Because your webapp code never returns to Tomcat.  Whatever requests your 
webapp made to the external API were likely lost, but your webapp was still 
expecting response(s).  The webapp needs to include some kind of timeout and 
recovery logic when accessing an external API to make sure it doesn't get stuck 
there forever.  Nothing Tomcat can do about anti-social behavior inside your 
webapp, other than report it.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat 8 Hung State

[Mark Thomas]

On 17 March 2016 16:13:28 GMT+00:00, Theo Sweeny  wrote:
>Hello - yesterday Tomcat 8 went into a hung state, because an external
>API went offline.
>
>
>Eventually the external service was restored but Tomcat was unable to
>recovery from it.
>
>
>When a new connection is sent to Tomcat the connection hangs as no
>additional thread can be allocated to the new request.
>
>
>Here is an example of the log -
>
>
>17-Mar-2016 15:53:36.671 WARNING
>[ContainerBackgroundProcessor[StandardEngine[Catalina].StandardHost[localhost]]]
>org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected
>Thread "http-nio-10180-exec-40" (id=422) has been active for 33,717
>milliseconds (since 3/17/16 3:53 PM) to serve the same request for
>http://localhost:10180/retrieve-1.2/retrieve-prices and may be stuck
>(configured threshold for this StuckThreadDetectionValve is 30
>seconds). There is/are 1 thread(s) in total that are monitored by this
>Valve and may be stuck.
> java.lang.Throwable
>at sun.misc.Unsafe.park(Native Method)
>   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:425)
>at java.util.concurrent.FutureTask.get(FutureTask.java:187)
>at
>com.spring.application.repository.product.price.IlogPriceRepository.getPrices_aroundBody0(IlogPriceRepository.java:104)
>at
>com.spring.application.repository.product.price.IlogPriceRepository$AjcClosure1.run(IlogPriceRepository.java:1)
>at
>org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
>at
>com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61)
>at
>com.spring.application.repository.product.price.IlogPriceRepository.getPrices(IlogPriceRepository.java:86)
>at
>com.spring.application.repository.product.price.IlogPriceRepository$$FastClassBySpringCGLIB$$142c62f1.invoke()
>at
>org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
>at
>org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717)
>at
>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
>at
>org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
>at
>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>at
>org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653)
>at
>com.spring.application.repository.product.price.IlogPriceRepository$$EnhancerBySpringCGLIB$$4fdb7af0.getPrices()
>at
>com.spring.application.service.product.price.RealTimePriceApplicationService.getPrices_aroundBody0(RealTimePriceApplicationService.java:48)
>at
>com.spring.application.service.product.price.RealTimePriceApplicationService$AjcClosure1.run(RealTimePriceApplicationService.java:1)
>at
>org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
>at
>com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61)
>at
>com.spring.application.service.product.price.RealTimePriceApplicationService.getPrices(RealTimePriceApplicationService.java:47)
>at
>com.spring.application.service.product.price.GetPriceApplicationController.getPrices_aroundBody0(GetPriceApplicationController.java:74)
>at
>com.spring.application.service.product.price.GetPriceApplicationController$AjcClosure1.run(GetPriceApplicationController.java:1)
>at
>org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
>at
>com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61)
>at
>com.spring.application.service.product.price.GetPriceApplicationController.getPrices(GetPriceApplicationController.java:70)
>at sun.reflect.GeneratedMethodAccessor1306.invoke(Unknown Source)
>at
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>at java.lang.reflect.Method.invoke(Method.java:606)
>at
>org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
>at
>org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)
>at
>org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
>at
>org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:777)
>at
>org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:706)
>at
>org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
>at
>org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:943)
>at

Re: Tomcat 8 Hung State

[Theo Sweeny]

Hi Mark,


From: Mark Thomas <ma...@apache.org>
Sent: 17 March 2016 17:36
To: Tomcat Users List
Subject: Re: Tomcat 8 Hung State

On 17 March 2016 16:13:28 GMT+00:00, Theo Sweeny <theo.swe...@avios.com> wrote:
>Hello - yesterday Tomcat 8 went into a hung state, because an external
>API went offline.
>
>
>Eventually the external service was restored but Tomcat was unable to
>recovery from it.
>
>
>When a new connection is sent to Tomcat the connection hangs as no
>additional thread can be allocated to the new request.
>
>
>Here is an example of the log -
>
>
>17-Mar-2016 15:53:36.671 WARNING
>[ContainerBackgroundProcessor[StandardEngine[Catalina].StandardHost[localhost]]]
>org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected
>Thread "http-nio-10180-exec-40" (id=422) has been active for 33,717
>milliseconds (since 3/17/16 3:53 PM) to serve the same request for
>http://localhost:10180/retrieve-1.2/retrieve-prices and may be stuck
>(configured threshold for this StuckThreadDetectionValve is 30
>seconds). There is/are 1 thread(s) in total that are monitored by this
>Valve and may be stuck.
> java.lang.Throwable
>at sun.misc.Unsafe.park(Native Method)
>   at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
>at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:425)
>at java.util.concurrent.FutureTask.get(FutureTask.java:187)
>at
>com.spring.application.repository.product.price.IlogPriceRepository.getPrices_aroundBody0(IlogPriceRepository.java:104)
>at
>com.spring.application.repository.product.price.IlogPriceRepository$AjcClosure1.run(IlogPriceRepository.java:1)
>at
>org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
>at
>com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61)
>at
>com.spring.application.repository.product.price.IlogPriceRepository.getPrices(IlogPriceRepository.java:86)
>at
>com.spring.application.repository.product.price.IlogPriceRepository$$FastClassBySpringCGLIB$$142c62f1.invoke()
>at
>org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
>at
>org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717)
>at
>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
>at
>org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
>at
>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>at
>org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653)
>at
>com.spring.application.repository.product.price.IlogPriceRepository$$EnhancerBySpringCGLIB$$4fdb7af0.getPrices()
>at
>com.spring.application.service.product.price.RealTimePriceApplicationService.getPrices_aroundBody0(RealTimePriceApplicationService.java:48)
>at
>com.spring.application.service.product.price.RealTimePriceApplicationService$AjcClosure1.run(RealTimePriceApplicationService.java:1)
>at
>org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
>at
>com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61)
>at
>com.spring.application.service.product.price.RealTimePriceApplicationService.getPrices(RealTimePriceApplicationService.java:47)
>at
>com.spring.application.service.product.price.GetPriceApplicationController.getPrices_aroundBody0(GetPriceApplicationController.java:74)
>at
>com.spring.application.service.product.price.GetPriceApplicationController$AjcClosure1.run(GetPriceApplicationController.java:1)
>at
>org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
>at
>com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61)
>at
>com.spring.application.service.product.price.GetPriceApplicationController.getPrices(GetPriceApplicationController.java:70)
>at sun.reflect.GeneratedMethodAccessor1306.invoke(Unknown Source)
>at
>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>at java.lang.reflect.Method.invoke(Method.java:606)
>at
>org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
>at
>org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137)
>at
>org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)
>at
>org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.

Re: Tomcat 8 Hung State

[Mark Thomas]

On 18/03/2016 15:06, Theo Sweeny wrote:
> Hi Mark,



> hanks for the reply - the developers are pushing back as they believe Tomcat 
> should kill off any rogue webapps after a certain timeframe as this is how 
> they have it done in Weblogic.

I believe WebLogic runs each web application in a separate JVM (or at
least provides an option to do so). That allows developers to get away
with all sorts of poor practice because the container can simply kill
the JVM to clean things up.

Tomcat doesn't work like that. In Tomcat all web applications are hosted
in a single JVM. Java provides no safe mechanism for stopping a
currently processing thread. Therefore there is very little / nothing
Tomcat can do.

The developers need to fix their broken code.

> When I do an Strace on a broken webapp, the following can be seen -
> 
> 11297 11:39:23.023838 write(105, "2016-03-18 11:39:23,021 138066033 
> [http-nio-10080-exec-27] ERROR 
> com.spring.business.service.security.ldap.filter.CustomAuthFilter  - 
> Exception thrown in doFilterInternal: 
> {}\norg.apache.catalina.connector.ClientAbortException: java.io.IOException: 
> Broken pipe\n\tat 
> org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:393)
>  ~[catalina.jar:8.0.21]\n\tat 
> org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:426) 
> ~[tomcat-util.jar:8.0.21]\n\tat org.apache.catalina.connector."..., 17236 
> 
> 
> Is this due to Tomcat's inability to recover from a lost datasource?

That looks like a client dropping the connection. Tomcat will handle
that happily. Whether the app does is a different question.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org