> -----Original Message----- > From: Rainer Jung [mailto:rainer.j...@kippdata.de] > Sent: Friday, May 22, 2009 2:53 AM > To: Tomcat Users List > Subject: Re: Running out of tomcat threads - why many threads in > RUNNABLEstage even with no activity > > My point is: persistent connections are good, but connections which are > idle for a long time are not as good, so close them after some idle > time, like e.g. 10 minutes. Of course this means you need to create new > ones once your load goes up again, but that's not a big problem.
[Pantvaidya, Vishwajit] Why are connections idle for a long time not good? I thought threads when idle take only a little memory and cpu. Are there any other reasons? Thanks a lot Rainer, Chuck, Chris, Andre, Pid, Martin and everyone else I missed. I spent quite some time yesterday chewing on everything I gathered in the last few days' interactions and the conflicting behavior we are seeing in our systems - that led to following conclusions and action plan: Behavior observed in diff production systems: a. medium-to-large thread count whether firewall exists or not b. % of runnable threads is much higher where firewall between httpd/tomcat c. atleast 1 server where firewall exists has run out of threads d. atleast 1 server where no firewall exists has run out of threads Conclusions: 1. In general, runnable threads should not be a prob, unless they correspond to dropped connections. Since on our servers that have firewall between httpd and tomcat, runnable connections are not being used for new requests and tomcat keeps on creating new threads (leading to #b/c above), those threads could correspond to: i. connections dropped by firewall or ii. hanging tomcat threads as httpd recycle timeout disconnected the connection from that side (and there was no connectiontimeout in server.xml so that tomcat could do the same) or iii. combination of these "i" and "ii" 2. Runnable threads on servers where no firewall exist (and we do not see server running out of threads) should not be a point of concerns as they do not correspond to dropped connections, as seen from netstat o/p at the end of this email. So #a above could be ignored. 3. Observation #d above is puzzling and currently I have no answers for that Action: - check both sides by using "netstat -anop" (Apache side and the Tomcat side without connectionTimeout, so you can see the problem in the original form). See whether the number of AJP connections in the various TCP states differs much between the netstat output on the Apache and on the Tomcat system. - Bring workers.properties settings in line with Apache recommendations: - Worker...cachesize=10 - set to 1 - Worker...cache_timeout=600 - remove - Worker...recycle_timeout=300 - remove Netstat o/p's: connector running on 21005, no firewall between httpd/tomcat Httpd Side: Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name Timer tcp 0 0 129.41.29.241:53777 129.41.29.48:21005 ESTABLISHED - keepalive (2869.65/0/0) tcp 0 0 129.41.29.241:53943 129.41.29.48:21005 ESTABLISHED - keepalive (3341.39/0/0) tcp 0 0 129.41.29.241:49950 129.41.29.48:21005 ESTABLISHED - keepalive (6701.51/0/0) tcp 0 0 129.41.29.241:49927 129.41.29.48:21005 ESTABLISHED - keepalive (6240.25/0/0) tcp 0 0 129.41.29.241:49926 129.41.29.48:21005 ESTABLISHED - keepalive (6239.47/0/0) tcp 0 0 129.41.29.241:49971 129.41.29.48:21005 ESTABLISHED - keepalive (6931.40/0/0) tcp 0 0 129.41.29.241:49868 129.41.29.48:21005 ESTABLISHED - keepalive (5743.83/0/0) tcp 0 0 129.41.29.241:49865 129.41.29.48:21005 ESTABLISHED - keepalive (5741.65/0/0) tcp 0 0 129.41.29.241:49867 129.41.29.48:21005 ESTABLISHED - keepalive (5743.16/0/0) tcp 0 0 129.41.29.241:49901 129.41.29.48:21005 ESTABLISHED - keepalive (5906.92/0/0) tcp 0 0 129.41.29.241:49795 129.41.29.48:21005 ESTABLISHED - keepalive (4659.11/0/0) tcp 0 0 129.41.29.241:49558 129.41.29.48:21005 ESTABLISHED - keepalive (1705.06/0/0) tcp 0 0 129.41.29.241:50796 129.41.29.48:21005 ESTABLISHED - keepalive (4551.79/0/0) tcp 0 0 129.41.29.241:50784 129.41.29.48:21005 ESTABLISHED - keepalive (4539.53/0/0) tcp 0 0 129.41.29.241:50711 129.41.29.48:21005 ESTABLISHED - keepalive (4237.02/0/0) tcp 0 0 129.41.29.241:50841 129.41.29.48:21005 ESTABLISHED - keepalive (4544.81/0/0) tcp 0 0 129.41.29.241:50526 129.41.29.48:21005 ESTABLISHED - keepalive (3799.53/0/0) tcp 0 0 129.41.29.241:50558 129.41.29.48:21005 ESTABLISHED - keepalive (3766.94/0/0) tcp 0 0 129.41.29.241:50642 129.41.29.48:21005 ESTABLISHED - keepalive (4022.29/0/0) tcp 0 0 129.41.29.241:50387 129.41.29.48:21005 ESTABLISHED - keepalive (3723.14/0/0) tcp 0 0 129.41.29.241:50404 129.41.29.48:21005 ESTABLISHED - keepalive (3807.13/0/0) tcp 0 0 129.41.29.241:52042 129.41.29.48:21005 ESTABLISHED - keepalive (6770.05/0/0) tcp 0 0 129.41.29.241:52092 129.41.29.48:21005 ESTABLISHED - keepalive (542.41/0/0) tcp 0 0 129.41.29.241:52222 129.41.29.48:21005 ESTABLISHED - keepalive (555.67/0/0) tcp 0 0 129.41.29.241:52109 129.41.29.48:21005 ESTABLISHED - keepalive (580.33/0/0) tcp 0 0 129.41.29.241:51907 129.41.29.48:21005 ESTABLISHED - keepalive (6761.11/0/0) tcp 0 0 129.41.29.241:52379 129.41.29.48:21005 ESTABLISHED - keepalive (864.69/0/0) tcp 0 0 ::ffff:129.41.29.112:443 ::ffff:198.70.193.2:21005 TIME_WAIT - timewait (32.66/0/0) $ date Thu May 21 17:07:17 PDT 2009 Tomcat Side: Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name Timer tcp 0 0 :::21005 :::* LISTEN 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50796 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50784 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:52092 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50558 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:52042 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50526 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49971 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 865 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49926 ESTABLISHED 18131/java on (0.20/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49927 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49950 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:53777 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49901 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50404 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:52222 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49868 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49865 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49867 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:51907 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50642 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50387 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:53943 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:53943 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:52109 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49795 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50841 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:52379 ESTABLISHED 18131/java off (0.00/0/0) tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49558 ESTABLISHED 18131/java off (0.00/0/0) $ date Thu May 21 17:07:05 PDT 2009 --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org