RE: Tomcat 8 Hung State
> From: Theo Sweeny [mailto:theo.swe...@avios.com] > Subject: Tomcat 8 Hung State > yesterday Tomcat 8 went into a hung state, because an external API went > offline. > Eventually the external service was restored but Tomcat was unable to > recovery from it. > When a new connection is sent to Tomcat the connection hangs as no additional > thread > can be allocated to the new request. > Here is an example of the log - > com.spring.application.repository.product.price.IlogPriceRepository.getPrices_aroundBody0 > (IlogPriceRepository.java:104) Note that this is application, not Tomcat, code. > Are there any ideas as to why Tomcat is unable to free up its threads? Because your webapp code never returns to Tomcat. Whatever requests your webapp made to the external API were likely lost, but your webapp was still expecting response(s). The webapp needs to include some kind of timeout and recovery logic when accessing an external API to make sure it doesn't get stuck there forever. Nothing Tomcat can do about anti-social behavior inside your webapp, other than report it. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 8 Hung State
On 17 March 2016 16:13:28 GMT+00:00, Theo Sweeny wrote: >Hello - yesterday Tomcat 8 went into a hung state, because an external >API went offline. > > >Eventually the external service was restored but Tomcat was unable to >recovery from it. > > >When a new connection is sent to Tomcat the connection hangs as no >additional thread can be allocated to the new request. > > >Here is an example of the log - > > >17-Mar-2016 15:53:36.671 WARNING >[ContainerBackgroundProcessor[StandardEngine[Catalina].StandardHost[localhost]]] >org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected >Thread "http-nio-10180-exec-40" (id=422) has been active for 33,717 >milliseconds (since 3/17/16 3:53 PM) to serve the same request for >http://localhost:10180/retrieve-1.2/retrieve-prices and may be stuck >(configured threshold for this StuckThreadDetectionValve is 30 >seconds). There is/are 1 thread(s) in total that are monitored by this >Valve and may be stuck. > java.lang.Throwable >at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) >at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:425) >at java.util.concurrent.FutureTask.get(FutureTask.java:187) >at >com.spring.application.repository.product.price.IlogPriceRepository.getPrices_aroundBody0(IlogPriceRepository.java:104) >at >com.spring.application.repository.product.price.IlogPriceRepository$AjcClosure1.run(IlogPriceRepository.java:1) >at >org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) >at >com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61) >at >com.spring.application.repository.product.price.IlogPriceRepository.getPrices(IlogPriceRepository.java:86) >at >com.spring.application.repository.product.price.IlogPriceRepository$$FastClassBySpringCGLIB$$142c62f1.invoke() >at >org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) >at >org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) >at >org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) >at >org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136) >at >org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) >at >org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) >at >com.spring.application.repository.product.price.IlogPriceRepository$$EnhancerBySpringCGLIB$$4fdb7af0.getPrices() >at >com.spring.application.service.product.price.RealTimePriceApplicationService.getPrices_aroundBody0(RealTimePriceApplicationService.java:48) >at >com.spring.application.service.product.price.RealTimePriceApplicationService$AjcClosure1.run(RealTimePriceApplicationService.java:1) >at >org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) >at >com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61) >at >com.spring.application.service.product.price.RealTimePriceApplicationService.getPrices(RealTimePriceApplicationService.java:47) >at >com.spring.application.service.product.price.GetPriceApplicationController.getPrices_aroundBody0(GetPriceApplicationController.java:74) >at >com.spring.application.service.product.price.GetPriceApplicationController$AjcClosure1.run(GetPriceApplicationController.java:1) >at >org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) >at >com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61) >at >com.spring.application.service.product.price.GetPriceApplicationController.getPrices(GetPriceApplicationController.java:70) >at sun.reflect.GeneratedMethodAccessor1306.invoke(Unknown Source) >at >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >at java.lang.reflect.Method.invoke(Method.java:606) >at >org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221) >at >org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137) >at >org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110) >at >org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:777) >at >org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:706) >at >org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) >at >org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:943) >at >org.springframework.web.servlet.DispatcherServlet.doServic
Re: Tomcat 8 Hung State
Hi Mark, From: Mark Thomas Sent: 17 March 2016 17:36 To: Tomcat Users List Subject: Re: Tomcat 8 Hung State On 17 March 2016 16:13:28 GMT+00:00, Theo Sweeny wrote: >Hello - yesterday Tomcat 8 went into a hung state, because an external >API went offline. > > >Eventually the external service was restored but Tomcat was unable to >recovery from it. > > >When a new connection is sent to Tomcat the connection hangs as no >additional thread can be allocated to the new request. > > >Here is an example of the log - > > >17-Mar-2016 15:53:36.671 WARNING >[ContainerBackgroundProcessor[StandardEngine[Catalina].StandardHost[localhost]]] >org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected >Thread "http-nio-10180-exec-40" (id=422) has been active for 33,717 >milliseconds (since 3/17/16 3:53 PM) to serve the same request for >http://localhost:10180/retrieve-1.2/retrieve-prices and may be stuck >(configured threshold for this StuckThreadDetectionValve is 30 >seconds). There is/are 1 thread(s) in total that are monitored by this >Valve and may be stuck. > java.lang.Throwable >at sun.misc.Unsafe.park(Native Method) > at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) >at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:425) >at java.util.concurrent.FutureTask.get(FutureTask.java:187) >at >com.spring.application.repository.product.price.IlogPriceRepository.getPrices_aroundBody0(IlogPriceRepository.java:104) >at >com.spring.application.repository.product.price.IlogPriceRepository$AjcClosure1.run(IlogPriceRepository.java:1) >at >org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) >at >com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61) >at >com.spring.application.repository.product.price.IlogPriceRepository.getPrices(IlogPriceRepository.java:86) >at >com.spring.application.repository.product.price.IlogPriceRepository$$FastClassBySpringCGLIB$$142c62f1.invoke() >at >org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) >at >org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) >at >org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) >at >org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136) >at >org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) >at >org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) >at >com.spring.application.repository.product.price.IlogPriceRepository$$EnhancerBySpringCGLIB$$4fdb7af0.getPrices() >at >com.spring.application.service.product.price.RealTimePriceApplicationService.getPrices_aroundBody0(RealTimePriceApplicationService.java:48) >at >com.spring.application.service.product.price.RealTimePriceApplicationService$AjcClosure1.run(RealTimePriceApplicationService.java:1) >at >org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) >at >com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61) >at >com.spring.application.service.product.price.RealTimePriceApplicationService.getPrices(RealTimePriceApplicationService.java:47) >at >com.spring.application.service.product.price.GetPriceApplicationController.getPrices_aroundBody0(GetPriceApplicationController.java:74) >at >com.spring.application.service.product.price.GetPriceApplicationController$AjcClosure1.run(GetPriceApplicationController.java:1) >at >org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) >at >com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61) >at >com.spring.application.service.product.price.GetPriceApplicationController.getPrices(GetPriceApplicationController.java:70) >at sun.reflect.GeneratedMethodAccessor1306.invoke(Unknown Source) >at >sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >at java.lang.reflect.Method.invoke(Method.java:606) >at >org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221) >at >org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137) >at >org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110) >at >org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:7
Tomcat 8 Hung State
Hello - yesterday Tomcat 8 went into a hung state, because an external API went offline. Eventually the external service was restored but Tomcat was unable to recovery from it. When a new connection is sent to Tomcat the connection hangs as no additional thread can be allocated to the new request. Here is an example of the log - 17-Mar-2016 15:53:36.671 WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina].StandardHost[localhost]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread "http-nio-10180-exec-40" (id=422) has been active for 33,717 milliseconds (since 3/17/16 3:53 PM) to serve the same request for http://localhost:10180/retrieve-1.2/retrieve-prices and may be stuck (configured threshold for this StuckThreadDetectionValve is 30 seconds). There is/are 1 thread(s) in total that are monitored by this Valve and may be stuck. java.lang.Throwable at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:425) at java.util.concurrent.FutureTask.get(FutureTask.java:187) at com.spring.application.repository.product.price.IlogPriceRepository.getPrices_aroundBody0(IlogPriceRepository.java:104) at com.spring.application.repository.product.price.IlogPriceRepository$AjcClosure1.run(IlogPriceRepository.java:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61) at com.spring.application.repository.product.price.IlogPriceRepository.getPrices(IlogPriceRepository.java:86) at com.spring.application.repository.product.price.IlogPriceRepository$$FastClassBySpringCGLIB$$142c62f1.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653) at com.spring.application.repository.product.price.IlogPriceRepository$$EnhancerBySpringCGLIB$$4fdb7af0.getPrices() at com.spring.application.service.product.price.RealTimePriceApplicationService.getPrices_aroundBody0(RealTimePriceApplicationService.java:48) at com.spring.application.service.product.price.RealTimePriceApplicationService$AjcClosure1.run(RealTimePriceApplicationService.java:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61) at com.spring.application.service.product.price.RealTimePriceApplicationService.getPrices(RealTimePriceApplicationService.java:47) at com.spring.application.service.product.price.GetPriceApplicationController.getPrices_aroundBody0(GetPriceApplicationController.java:74) at com.spring.application.service.product.price.GetPriceApplicationController$AjcClosure1.run(GetPriceApplicationController.java:1) at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) at com.spring.framework.aspects.LoggingAspect.logMethodEntryAndExit(LoggingAspect.java:61) at com.spring.application.service.product.price.GetPriceApplicationController.getPrices(GetPriceApplicationController.java:70) at sun.reflect.GeneratedMethodAccessor1306.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:137) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:777) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:706) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:943) at org.springframework.web.servlet.DispatcherS
Re: Tomcat 8 Hung State
On 18/03/2016 15:06, Theo Sweeny wrote: > Hi Mark, > hanks for the reply - the developers are pushing back as they believe Tomcat > should kill off any rogue webapps after a certain timeframe as this is how > they have it done in Weblogic. I believe WebLogic runs each web application in a separate JVM (or at least provides an option to do so). That allows developers to get away with all sorts of poor practice because the container can simply kill the JVM to clean things up. Tomcat doesn't work like that. In Tomcat all web applications are hosted in a single JVM. Java provides no safe mechanism for stopping a currently processing thread. Therefore there is very little / nothing Tomcat can do. The developers need to fix their broken code. > When I do an Strace on a broken webapp, the following can be seen - > > 11297 11:39:23.023838 write(105, "2016-03-18 11:39:23,021 138066033 > [http-nio-10080-exec-27] ERROR > com.spring.business.service.security.ldap.filter.CustomAuthFilter - > Exception thrown in doFilterInternal: > {}\norg.apache.catalina.connector.ClientAbortException: java.io.IOException: > Broken pipe\n\tat > org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:393) > ~[catalina.jar:8.0.21]\n\tat > org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:426) > ~[tomcat-util.jar:8.0.21]\n\tat org.apache.catalina.connector."..., 17236 > > > Is this due to Tomcat's inability to recover from a lost datasource? That looks like a client dropping the connection. Tomcat will handle that happily. Whether the app does is a different question. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org