Hello Christopher,
Let me post my thread dump here.
All HTTP threads are in WAITING state.
"http-12453-346" daemon prio=10 tid=0x00007fbd3061f800 nid=0x59d0 in
Object.wait() [0x00007fbbe124f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1104)
- locked <0x000000072b18f320> (a
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool$Latch)
at
org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
at
org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
at
org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:82)
at
org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
at
org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
at
org.hibernate.jdbc.AbstractBatcher.prepareQueryStatement(AbstractBatcher.java:161)
at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1577)
at org.hibernate.loader.Loader.doQuery(Loader.java:696)
at
org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:259)
at org.hibernate.loader.Loader.doList(Loader.java:2232)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2129)
at org.hibernate.loader.Loader.list(Loader.java:2124)
at org.hibernate.loader.custom.CustomLoader.list(CustomLoader.java:312)
at org.hibernate.impl.SessionImpl.listCustomQuery(SessionImpl.java:1723)
at
org.hibernate.impl.AbstractSessionImpl.list(AbstractSessionImpl.java:165)
at org.hibernate.impl.SQLQueryImpl.list(SQLQueryImpl.java:175)
at
com.adobe_services.orderservice.persistence.dao.impl.StoreEmailDAOImpl.findPendingEmailOidsForOrderNumber(StoreEmailDAOImpl.java:237)
at
com.adobe_services.orderservice.service.OrderServiceImpl.findPendingEmailOidsForOrderNumber_aroundBody4(OrderServiceImpl.java:670)
at
com.adobe_services.orderservice.service.OrderServiceImpl.findPendingEmailOidsForOrderNumber_aroundBody5$advice(OrderServiceImpl.java:37)
at
com.adobe_services.orderservice.service.OrderServiceImpl.findPendingEmailOidsForOrderNumber(OrderServiceImpl.java:1)
at sun.reflect.GeneratedMethodAccessor948.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy112.findPendingEmailOidsForOrderNumber(Unknown Source)
at sun.reflect.GeneratedMethodAccessor948.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at
org.springframework.orm.hibernate3.HibernateInterceptor.invoke(HibernateInterceptor.java:111)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy113.findPendingEmailOidsForOrderNumber(Unknown Source)
at sun.reflect.GeneratedMethodAccessor948.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
at
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy114.findPendingEmailOidsForOrderNumber(Unknown Source)
at
com.adobe_services.orderservice.facade.impl.EmailProcessorFacadeImpl.executeTask(EmailProcessorFacadeImpl.java:76)
at
com.adobe_services.orderservice.service.OrderProcessorService.postTask_aroundBody2(OrderProcessorService.java:174)
at
com.adobe_services.orderservice.service.OrderProcessorService.postTask_aroundBody3$advice(OrderProcessorService.java:37)
at
com.adobe_services.orderservice.service.OrderProcessorService.postTask(OrderProcessorService.java:1)
at sun.reflect.GeneratedMethodAccessor1539.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:188)
at
org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:104)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:205)
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:102)
at
org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58)
at
org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:94)
at
org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:272)
- locked <0x000000072b1b4128> (a
org.apache.cxf.phase.PhaseInterceptorChain)
at
org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121)
at
org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:239)
at
org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:248)
at
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:222)
at
org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:153)
at
org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:167)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:286)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:206)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
at
org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:262)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
com.adobe_services.monitor.MonitorContextFilter.doFilter(MonitorContextFilter.java:97)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:554)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)
oracle.jdbc.driver.BlockSource.ThreadedCachingBlockSource.BlockReleaser"
daemon prio=10 tid=0x00007fbd3cb13000 nid=0x762b in Object.wait()
[0x00007fbcdf5fd000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser.run(BlockSource.java:327)
- locked <0x00000007000535d0> (a
oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource$BlockReleaser)
On Mon, May 16, 2016 at 2:42 PM, Christopher Schultz <
ch...@christopherschultz.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Pradeep,
>
> On 5/16/16 12:34 PM, pradeep reddy wrote:
> > Hello Community, I have a weird issue, which I am not able to
> > figure it out and needed some help. I am a Web Administrator and in
> > my organization we use Tomcat + CXF to run our commerce services.
> >
> > We are running on Tomcat-6.0.32 and jdk1.6.0_24, anyways we have
> > recently started to see issues with our connection pool and from
> > our monitoring tools its evident that we are running out of
> > connections. Now I have read some manuals on Apache Tomcat Page to
> > use below three settings to fail gracefully and to Log those
> > sessions which are either causing connection leak or abandoned
> >
> > removeAbandoned - set to true if we want to detect leaked
> > connections removeAbandonedTimeout logAbandoned="true"
> >
> >
> > But unfortunately even after having configured those above three
> > settings, I still see some sessions just hanging in and not
> > releasing the connection and I need help in understanding what I am
> > doing wrong.
> >
> > My DS cnfiguration <Resource name="jdbc/XXXXXDataSource"
> > auth="Container" type="javax.sql.DataSource"
> > driverClassName="oracle.jdbc.OracleDriver" url="XXXXXXXXXXX"
> > username="XXXXXX" password="XXXXXX" initialSize="1" maxActive="10"
> > minIdle="1" maxIdle="1" connectionCachingEnabled="true"
> > fastConnectionFailoverEnabled="true"
> > connectionCacheProperties="{MinLimit=1, MaxLimit=20,
> > InitialLimit=1, oracle.net.ns.SQLnetDef.TCP_CONNTIMEOUT_STR=3000}"
> > ONSConfiguration="nodes=XXXXX" validationQuery="SELECT 1 from
> > DUAL" removeAbandoned="true" removeAbandonedTimeout="30"
> > logAbandoned="true" maxWait="-1"/>
>
> What do the stack traces look like or the hung threads?
>
> - -chris
> -----BEGIN PGP SIGNATURE-----
> Comment: GPGTools - http://gpgtools.org
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQIcBAEBCAAGBQJXOjC7AAoJEBzwKT+lPKRYl5oQAL+tSB6ZkBpMOqihbN79UHdG
> LAKsRdenKPhphA2yjahoXIx9nWvSd03jsFy8uzJYe3ztyRkJGW4h7VAm37gAHkKl
> sg/jtQzvtl4HYO6YI1v5OgljxR5GnK8CvGad5PhOg/s6/mLRZ9kfUdGgv6tCYHXe
> lpoEOA8Em6tMaTwg1u07gXAqMEJyqg5QA3tYR3MU+0RALhNeVWgXNoEULol0FEil
> PJVcAVN1WC6AX0qVa3JlXIK+PHHLDDZWLH7qrpo2nGVvcf/5oFf27N5AnnnarH87
> RtOWHjWgK4APRVkEEOSX8Pef7YAb2dj6E+6fgf+3v117eEwYJlkLWU5pS/6Q3zWm
> 23obZllDks1clyYzs7C96zJukFb60wSRRGPdbHoSiKspH1J3/GavebaG7Sbg2ADB
> DNLcWG0fd4FUmy+kbG+pPi3hdEdwOQ0c6kgaqH7ARDjdYtfVKV8kgCe6wpJlKx+k
> WXfDK3cE2jWhUT/zx9VMutM+38gYk9G6ArUIZvOg5qdhcv6LvX299so3x8dEaL7w
> D/NNz4+yTJGVfQMnJ4LTnZR+R6wCUdhpS0quSYo9ZJNpXGERo3NY0aV80g4S8MQJ
> HObHoGDSUHDw4lScKCpIJYT7ZZ+Ok25fqyrlx+JN1fhsq7PcPtDQz/7FrRuhKEGa
> z8kfsKuyduF8cW1gDj5h
> =s/bM
> -----END PGP SIGNATURE-----
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
