Many Threads in TIMED_WAITING (parking) - [org.apache.tomcat.util.threads.TaskQueue]
Hi, Sometimes, in moments of high traffic for our patterns (170 req/sec), we have a lot of threads like that: ajp-apr-8009-exec-115 ^ 16/07/2015 - 11:13:37 - End of Execute daemon prio=10 tid=0x2aaab5c36800 nid=0x12f9 waiting on condition [0x4ddcb000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x000609a53198 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Thus, Tomcat hangs (all available threads stay in that state) and we have to restart the service. Server version: Apache Tomcat/7.0.63 Server built: Jun 30 2015 08:08:33 UTC Server number: 7.0.63.0 OS Name:Linux OS Version: 2.6.18-194.17.1.el5 Architecture: amd64 JVM Version:1.7.0_80-b15 JVM Vendor: Oracle Corporation We are using tomcat-native (APR connector) and Apache (2.2.3) as load balancer with mod_jk. server.xml ?xml version='1.0' encoding='utf-8'? Server port=8005 shutdown=SHUTDOWN Listener className=org.apache.catalina.core.AprLifecycleListener SSLEngine=off / Listener className=org.apache.catalina.core.JasperListener / Listener className=org.apache.catalina.core.JreMemoryLeakPreventionListener / Listener className=org.apache.catalina.mbeans.GlobalResourcesLifecycleListener / Listener className=org.apache.catalina.core.ThreadLocalLeakPreventionListener / GlobalNamingResources Resource name=UserDatabase auth=Container type=org.apache.catalina.UserDatabase description=User database that can be updated and saved factory=org.apache.catalina.users.MemoryUserDatabaseFactory pathname=conf/tomcat-users.xml / /GlobalNamingResources Service name=Catalina Connector port=8080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / Connector port=8009 enableLookups=false connectionTimeout=6 protocol=AJP/1.3 redirectPort=8443 / Engine name=Catalina defaultHost=localhost jvmRoute=tomsrv04 Realm className=org.apache.catalina.realm.LockOutRealm Realm className=org.apache.catalina.realm.UserDatabaseRealm resourceName=UserDatabase digest=SHA/ /Realm Host name=localhost appBase=webapps unpackWARs=true autoDeploy=true Valve className=org.apache.catalina.valves.StuckThreadDetectionValve threshold=60 interruptThreadThreshold=120 / /Host /Engine /Service /Server Any suggestions? Thanks in advance. Robert
Re: Many Threads in TIMED_WAITING (parking) - [org.apache.tomcat.util.threads.TaskQueue]
On Thu, Jul 16, 2015 at 10:37 AM, Robert Anderson ranom...@gmail.com wrote: Hi, Sometimes, in moments of high traffic for our patterns (170 req/sec), we have a lot of threads like that: ajp-apr-8009-exec-115 ^ 16/07/2015 - 11:13:37 - End of Execute daemon prio=10 tid=0x2aaab5c36800 nid=0x12f9 waiting on condition [0x4ddcb000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x000609a53198 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) This is not a problem. The thread is in the pool waiting for work. It's what you'd normally see on a server that's not handling any requests. Thus, Tomcat hangs (all available threads stay in that state) and we have to restart the service. Server version: Apache Tomcat/7.0.63 Server built: Jun 30 2015 08:08:33 UTC Server number: 7.0.63.0 OS Name:Linux OS Version: 2.6.18-194.17.1.el5 Architecture: amd64 JVM Version:1.7.0_80-b15 JVM Vendor: Oracle Corporation We are using tomcat-native (APR connector) and Apache (2.2.3) as load balancer with mod_jk. server.xml ?xml version='1.0' encoding='utf-8'? Server port=8005 shutdown=SHUTDOWN Listener className=org.apache.catalina.core.AprLifecycleListener SSLEngine=off / Listener className=org.apache.catalina.core.JasperListener / Listener className=org.apache.catalina.core.JreMemoryLeakPreventionListener / Listener className=org.apache.catalina.mbeans.GlobalResourcesLifecycleListener / Listener className=org.apache.catalina.core.ThreadLocalLeakPreventionListener / GlobalNamingResources Resource name=UserDatabase auth=Container type=org.apache.catalina.UserDatabase description=User database that can be updated and saved factory=org.apache.catalina.users.MemoryUserDatabaseFactory pathname=conf/tomcat-users.xml / /GlobalNamingResources Service name=Catalina Connector port=8080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / Connector port=8009 enableLookups=false connectionTimeout=6 protocol=AJP/1.3 redirectPort=8443 / Engine name=Catalina defaultHost=localhost jvmRoute=tomsrv04 Realm className=org.apache.catalina.realm.LockOutRealm Realm className=org.apache.catalina.realm.UserDatabaseRealm resourceName=UserDatabase digest=SHA/ /Realm Host name=localhost appBase=webapps unpackWARs=true autoDeploy=true Valve className=org.apache.catalina.valves.StuckThreadDetectionValve threshold=60 interruptThreadThreshold=120 / /Host /Engine /Service /Server Any suggestions? This could be an issue with HTTPD talking to Tomcat. - When the problem occurs, check the access logs for both and see if the request is making it from HTTPD to Tomcat. - Also when the problem occurs, try hitting Tomcat directly while the problem is occurring. See if you can access your app when you bypass HTTPD. - Post your HTTPD config - Make sure timeouts are configured appropriately - http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html Do you have a firewall in between HTTPD Tomcat? Is there anything that could be breaking the connection? Dan
Re: Many Threads in TIMED_WAITING (parking) - [org.apache.tomcat.util.threads.TaskQueue]
On 16 July 2015 16:37:53 CEST, Robert Anderson ranom...@gmail.com wrote: Hi, Sometimes, in moments of high traffic for our patterns (170 req/sec), we have a lot of threads like that: ajp-apr-8009-exec-115 ^ 16/07/2015 - 11:13:37 - End of Execute daemon prio=10 tid=0x2aaab5c36800 nid=0x12f9 waiting on condition [0x4ddcb000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x000609a53198 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Thus, Tomcat hangs (all available threads stay in that state) and we have to restart the service. Server version: Apache Tomcat/7.0.63 Server built: Jun 30 2015 08:08:33 UTC Server number: 7.0.63.0 OS Name:Linux OS Version: 2.6.18-194.17.1.el5 Architecture: amd64 JVM Version:1.7.0_80-b15 JVM Vendor: Oracle Corporation We are using tomcat-native (APR connector) and Apache (2.2.3) as load balancer with mod_jk. server.xml ?xml version='1.0' encoding='utf-8'? Server port=8005 shutdown=SHUTDOWN Listener className=org.apache.catalina.core.AprLifecycleListener SSLEngine=off / Listener className=org.apache.catalina.core.JasperListener / Listener className=org.apache.catalina.core.JreMemoryLeakPreventionListener / Listener className=org.apache.catalina.mbeans.GlobalResourcesLifecycleListener / Listener className=org.apache.catalina.core.ThreadLocalLeakPreventionListener / GlobalNamingResources Resource name=UserDatabase auth=Container type=org.apache.catalina.UserDatabase description=User database that can be updated and saved factory=org.apache.catalina.users.MemoryUserDatabaseFactory pathname=conf/tomcat-users.xml / /GlobalNamingResources Service name=Catalina Connector port=8080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / Connector port=8009 enableLookups=false connectionTimeout=6 protocol=AJP/1.3 redirectPort=8443 / Engine name=Catalina defaultHost=localhost jvmRoute=tomsrv04 Realm className=org.apache.catalina.realm.LockOutRealm Realm className=org.apache.catalina.realm.UserDatabaseRealm resourceName=UserDatabase digest=SHA/ /Realm Host name=localhost appBase=webapps unpackWARs=true autoDeploy=true Valve className=org.apache.catalina.valves.StuckThreadDetectionValve threshold=60 interruptThreadThreshold=120 / /Host /Engine /Service /Server Any suggestions? Thanks in advance. Robert Short since I'm on my phone. Connection exhaustion? Disable connection reuse or use NIO. Mark
Re: Many Threads in TIMED_WAITING (parking) - [org.apache.tomcat.util.threads.TaskQueue]
Interesting. What's the CPU usage like at the time? If it's high, you might want to look at gc logging to see if you've got a lot of GC happening. CPU is low. You'll probably also want to configure the access logs to log the amount of time the request took to process. I'll analyze this option. Thanks. 2015-07-16 13:44 GMT-03:00 Daniel Mikusa dmik...@pivotal.io: On Thu, Jul 16, 2015 at 12:04 PM, Robert Anderson ranom...@gmail.com wrote: Thanks, Dan. This is not a problem. The thread is in the pool waiting for work. It's what you'd normally see on a server that's not handling any requests. It's awkward. Everything is going normal and suddenly, many threads start waiting on the same monitor. After restart Tomcat, everything works without even make any changes in Apache (service or config files). The scenario you're describing can happen and it doesn't necessarily mean there's a problem with Tomcat. If you're seeing lots of threads waiting for work, it just means that you've got lots of free workers. The server could still be really busy if you got a couple threads running and working like mad. A full thread dump would be helpful, to see what else is happening at the time. You can also look at multiple thread dumps 10 - 15 seconds apart to see what the threads are doing over time (i.e. is thread #X at the same place across all the threads?) - When the problem occurs, check the access logs for both and see if the request is making it from HTTPD to Tomcat. OK. I'll check You'll probably also want to configure the access logs to log the amount of time the request took to process. That'll give you some understanding of how much time was spent at each hop. In Tomcat that's done by adding %D to the access log pattern. https://tomcat.apache.org/tomcat-7.0-doc/config/valve.html#Access_Log_Valve/Attributes HTTPD is the same, but it logs in micro seconds instead of milliseconds. http://httpd.apache.org/docs/2.2/mod/mod_log_config.html#formats - Also when the problem occurs, try hitting Tomcat directly while the problem is occurring. See if you can access your app when you bypass HTTPD. I've already tried this. Tomcat becomes very very very slow when it happens. Interesting. What's the CPU usage like at the time? If it's high, you might want to look at gc logging to see if you've got a lot of GC happening. Dan - Post your HTTPD config worker.properties: worker.list=balancer worker.tomsrv01.type=ajp13 worker.tomsrv01.host=x.y.z.1 worker.tomsrv01.port=8009 worker.tomsrv01.lbfactor=1 worker.tomsrv01.connect_timeout=1 worker.tomsrv01.prepost_timeout=1 worker.tomsrv01.socket_timeout=1800 worker.tomsrv01.connection_pool_timeout=60 worker.tomsrv02.type=ajp13 worker.tomsrv02.host=x.y.z.2 worker.tomsrv02.port=8009 worker.tomsrv02.lbfactor=1 worker.tomsrv02.connect_timeout=1 worker.tomsrv02.prepost_timeout=1 worker.tomsrv02.socket_timeout=1800 worker.tomsrv02.connection_pool_timeout=60 worker.tomsrv04.type=ajp13 worker.tomsrv04.host=x.y.z.3 worker.tomsrv04.port=8009 worker.tomsrv04.lbfactor=1 worker.tomsrv04.connect_timeout=1 worker.tomsrv04.prepost_timeout=1 worker.tomsrv04.socket_timeout=1800 worker.tomsrv04.connection_pool_timeout=60 worker.balancer.type=lb worker.balancer.balance_workers=tomsrv01,tomsrv02,tomsrv04 worker.balancer.sticky_session=1 - Make sure timeouts are configured appropriately - Ok Do you have a firewall in between HTTPD Tomcat? Is there anything that could be breaking the connection? No. Best regards. 2015-07-16 12:44 GMT-03:00 Daniel Mikusa dmik...@pivotal.io: On Thu, Jul 16, 2015 at 10:37 AM, Robert Anderson ranom...@gmail.com wrote: Hi, Sometimes, in moments of high traffic for our patterns (170 req/sec), we have a lot of threads like that: ajp-apr-8009-exec-115 ^ 16/07/2015 - 11:13:37 - End of Execute daemon prio=10 tid=0x2aaab5c36800 nid=0x12f9 waiting on condition [0x4ddcb000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x000609a53198 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at
Re: Many Threads in TIMED_WAITING (parking) - [org.apache.tomcat.util.threads.TaskQueue]
Hi Mark, How can I disable it? 2015-07-16 13:56 GMT-03:00 Mark Thomas ma...@apache.org: On 16 July 2015 16:37:53 CEST, Robert Anderson ranom...@gmail.com wrote: Hi, Sometimes, in moments of high traffic for our patterns (170 req/sec), we have a lot of threads like that: ajp-apr-8009-exec-115 ^ 16/07/2015 - 11:13:37 - End of Execute daemon prio=10 tid=0x2aaab5c36800 nid=0x12f9 waiting on condition [0x4ddcb000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x000609a53198 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) Thus, Tomcat hangs (all available threads stay in that state) and we have to restart the service. Server version: Apache Tomcat/7.0.63 Server built: Jun 30 2015 08:08:33 UTC Server number: 7.0.63.0 OS Name:Linux OS Version: 2.6.18-194.17.1.el5 Architecture: amd64 JVM Version:1.7.0_80-b15 JVM Vendor: Oracle Corporation We are using tomcat-native (APR connector) and Apache (2.2.3) as load balancer with mod_jk. server.xml ?xml version='1.0' encoding='utf-8'? Server port=8005 shutdown=SHUTDOWN Listener className=org.apache.catalina.core.AprLifecycleListener SSLEngine=off / Listener className=org.apache.catalina.core.JasperListener / Listener className=org.apache.catalina.core.JreMemoryLeakPreventionListener / Listener className=org.apache.catalina.mbeans.GlobalResourcesLifecycleListener / Listener className=org.apache.catalina.core.ThreadLocalLeakPreventionListener / GlobalNamingResources Resource name=UserDatabase auth=Container type=org.apache.catalina.UserDatabase description=User database that can be updated and saved factory=org.apache.catalina.users.MemoryUserDatabaseFactory pathname=conf/tomcat-users.xml / /GlobalNamingResources Service name=Catalina Connector port=8080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / Connector port=8009 enableLookups=false connectionTimeout=6 protocol=AJP/1.3 redirectPort=8443 / Engine name=Catalina defaultHost=localhost jvmRoute=tomsrv04 Realm className=org.apache.catalina.realm.LockOutRealm Realm className=org.apache.catalina.realm.UserDatabaseRealm resourceName=UserDatabase digest=SHA/ /Realm Host name=localhost appBase=webapps unpackWARs=true autoDeploy=true Valve className=org.apache.catalina.valves.StuckThreadDetectionValve threshold=60 interruptThreadThreshold=120 / /Host /Engine /Service /Server Any suggestions? Thanks in advance. Robert Short since I'm on my phone. Connection exhaustion? Disable connection reuse or use NIO. Mark
Re: Many Threads in TIMED_WAITING (parking) - [org.apache.tomcat.util.threads.TaskQueue]
On Thu, Jul 16, 2015 at 12:04 PM, Robert Anderson ranom...@gmail.com wrote: Thanks, Dan. This is not a problem. The thread is in the pool waiting for work. It's what you'd normally see on a server that's not handling any requests. It's awkward. Everything is going normal and suddenly, many threads start waiting on the same monitor. After restart Tomcat, everything works without even make any changes in Apache (service or config files). The scenario you're describing can happen and it doesn't necessarily mean there's a problem with Tomcat. If you're seeing lots of threads waiting for work, it just means that you've got lots of free workers. The server could still be really busy if you got a couple threads running and working like mad. A full thread dump would be helpful, to see what else is happening at the time. You can also look at multiple thread dumps 10 - 15 seconds apart to see what the threads are doing over time (i.e. is thread #X at the same place across all the threads?) - When the problem occurs, check the access logs for both and see if the request is making it from HTTPD to Tomcat. OK. I'll check You'll probably also want to configure the access logs to log the amount of time the request took to process. That'll give you some understanding of how much time was spent at each hop. In Tomcat that's done by adding %D to the access log pattern. https://tomcat.apache.org/tomcat-7.0-doc/config/valve.html#Access_Log_Valve/Attributes HTTPD is the same, but it logs in micro seconds instead of milliseconds. http://httpd.apache.org/docs/2.2/mod/mod_log_config.html#formats - Also when the problem occurs, try hitting Tomcat directly while the problem is occurring. See if you can access your app when you bypass HTTPD. I've already tried this. Tomcat becomes very very very slow when it happens. Interesting. What's the CPU usage like at the time? If it's high, you might want to look at gc logging to see if you've got a lot of GC happening. Dan - Post your HTTPD config worker.properties: worker.list=balancer worker.tomsrv01.type=ajp13 worker.tomsrv01.host=x.y.z.1 worker.tomsrv01.port=8009 worker.tomsrv01.lbfactor=1 worker.tomsrv01.connect_timeout=1 worker.tomsrv01.prepost_timeout=1 worker.tomsrv01.socket_timeout=1800 worker.tomsrv01.connection_pool_timeout=60 worker.tomsrv02.type=ajp13 worker.tomsrv02.host=x.y.z.2 worker.tomsrv02.port=8009 worker.tomsrv02.lbfactor=1 worker.tomsrv02.connect_timeout=1 worker.tomsrv02.prepost_timeout=1 worker.tomsrv02.socket_timeout=1800 worker.tomsrv02.connection_pool_timeout=60 worker.tomsrv04.type=ajp13 worker.tomsrv04.host=x.y.z.3 worker.tomsrv04.port=8009 worker.tomsrv04.lbfactor=1 worker.tomsrv04.connect_timeout=1 worker.tomsrv04.prepost_timeout=1 worker.tomsrv04.socket_timeout=1800 worker.tomsrv04.connection_pool_timeout=60 worker.balancer.type=lb worker.balancer.balance_workers=tomsrv01,tomsrv02,tomsrv04 worker.balancer.sticky_session=1 - Make sure timeouts are configured appropriately - Ok Do you have a firewall in between HTTPD Tomcat? Is there anything that could be breaking the connection? No. Best regards. 2015-07-16 12:44 GMT-03:00 Daniel Mikusa dmik...@pivotal.io: On Thu, Jul 16, 2015 at 10:37 AM, Robert Anderson ranom...@gmail.com wrote: Hi, Sometimes, in moments of high traffic for our patterns (170 req/sec), we have a lot of threads like that: ajp-apr-8009-exec-115 ^ 16/07/2015 - 11:13:37 - End of Execute daemon prio=10 tid=0x2aaab5c36800 nid=0x12f9 waiting on condition [0x4ddcb000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x000609a53198 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) This is not a problem. The thread is in the pool waiting for work. It's what you'd normally see on a server that's not handling any requests. Thus, Tomcat hangs (all available threads stay in that state) and we have
Re: Many Threads in TIMED_WAITING (parking) - [org.apache.tomcat.util.threads.TaskQueue]
Thanks, Dan. This is not a problem. The thread is in the pool waiting for work. It's what you'd normally see on a server that's not handling any requests. It's awkward. Everything is going normal and suddenly, many threads start waiting on the same monitor. After restart Tomcat, everything works without even make any changes in Apache (service or config files). - When the problem occurs, check the access logs for both and see if the request is making it from HTTPD to Tomcat. OK. I'll check - Also when the problem occurs, try hitting Tomcat directly while the problem is occurring. See if you can access your app when you bypass HTTPD. I've already tried this. Tomcat becomes very very very slow when it happens. - Post your HTTPD config worker.properties: worker.list=balancer worker.tomsrv01.type=ajp13 worker.tomsrv01.host=x.y.z.1 worker.tomsrv01.port=8009 worker.tomsrv01.lbfactor=1 worker.tomsrv01.connect_timeout=1 worker.tomsrv01.prepost_timeout=1 worker.tomsrv01.socket_timeout=1800 worker.tomsrv01.connection_pool_timeout=60 worker.tomsrv02.type=ajp13 worker.tomsrv02.host=x.y.z.2 worker.tomsrv02.port=8009 worker.tomsrv02.lbfactor=1 worker.tomsrv02.connect_timeout=1 worker.tomsrv02.prepost_timeout=1 worker.tomsrv02.socket_timeout=1800 worker.tomsrv02.connection_pool_timeout=60 worker.tomsrv04.type=ajp13 worker.tomsrv04.host=x.y.z.3 worker.tomsrv04.port=8009 worker.tomsrv04.lbfactor=1 worker.tomsrv04.connect_timeout=1 worker.tomsrv04.prepost_timeout=1 worker.tomsrv04.socket_timeout=1800 worker.tomsrv04.connection_pool_timeout=60 worker.balancer.type=lb worker.balancer.balance_workers=tomsrv01,tomsrv02,tomsrv04 worker.balancer.sticky_session=1 - Make sure timeouts are configured appropriately - Ok Do you have a firewall in between HTTPD Tomcat? Is there anything that could be breaking the connection? No. Best regards. 2015-07-16 12:44 GMT-03:00 Daniel Mikusa dmik...@pivotal.io: On Thu, Jul 16, 2015 at 10:37 AM, Robert Anderson ranom...@gmail.com wrote: Hi, Sometimes, in moments of high traffic for our patterns (170 req/sec), we have a lot of threads like that: ajp-apr-8009-exec-115 ^ 16/07/2015 - 11:13:37 - End of Execute daemon prio=10 tid=0x2aaab5c36800 nid=0x12f9 waiting on condition [0x4ddcb000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x000609a53198 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:86) at org.apache.tomcat.util.threads.TaskQueue.poll(TaskQueue.java:32) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745) This is not a problem. The thread is in the pool waiting for work. It's what you'd normally see on a server that's not handling any requests. Thus, Tomcat hangs (all available threads stay in that state) and we have to restart the service. Server version: Apache Tomcat/7.0.63 Server built: Jun 30 2015 08:08:33 UTC Server number: 7.0.63.0 OS Name:Linux OS Version: 2.6.18-194.17.1.el5 Architecture: amd64 JVM Version:1.7.0_80-b15 JVM Vendor: Oracle Corporation We are using tomcat-native (APR connector) and Apache (2.2.3) as load balancer with mod_jk. server.xml ?xml version='1.0' encoding='utf-8'? Server port=8005 shutdown=SHUTDOWN Listener className=org.apache.catalina.core.AprLifecycleListener SSLEngine=off / Listener className=org.apache.catalina.core.JasperListener / Listener className=org.apache.catalina.core.JreMemoryLeakPreventionListener / Listener className=org.apache.catalina.mbeans.GlobalResourcesLifecycleListener / Listener className=org.apache.catalina.core.ThreadLocalLeakPreventionListener / GlobalNamingResources Resource name=UserDatabase auth=Container type=org.apache.catalina.UserDatabase description=User database that can be updated and saved factory=org.apache.catalina.users.MemoryUserDatabaseFactory pathname=conf/tomcat-users.xml / /GlobalNamingResources Service name=Catalina Connector port=8080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 /
