On 22.05.2009 21:09, Pantvaidya, Vishwajit wrote: >> -----Original Message----- >> From: Rainer Jung [mailto:rainer.j...@kippdata.de] >> Sent: Friday, May 22, 2009 2:53 AM >> To: Tomcat Users List >> Subject: Re: Running out of tomcat threads - why many threads in >> RUNNABLEstage even with no activity >> >> My point is: persistent connections are good, but connections which are >> idle for a long time are not as good, so close them after some idle >> time, like e.g. 10 minutes. Of course this means you need to create new >> ones once your load goes up again, but that's not a big problem. > > [Pantvaidya, Vishwajit] Why are connections idle for a long time not good? I > thought threads when idle take only a little memory and cpu. Are there any > other reasons?
Because you might want to monitor connections in order to learn how many threads you need for your load and how things grow or shrink over time. If you keep connections open an infinite number of time, you'll only monitor the biggest need since restart, which is often not very interesting, because it often is artificial (triggered by some performance slowness you might have a very big connection number created during a short time). Second: because they are making trouble so often in combination with firewalls. So in general I like persistent connections as long as they are closed when idle for a longer time. So usually I set the pool min size to 0 and the idle connection timeout to 10 minutes. > Thanks a lot Rainer, Chuck, Chris, Andre, Pid, Martin and everyone else I > missed. I spent quite some time yesterday chewing on everything I gathered in > the last few days' interactions and the conflicting behavior we are seeing in > our systems - that led to following conclusions and action plan: > > Behavior observed in diff production systems: > a. medium-to-large thread count whether firewall exists or not > b. % of runnable threads is much higher where firewall between httpd/tomcat > c. atleast 1 server where firewall exists has run out of threads > d. atleast 1 server where no firewall exists has run out of threads Concurrency = Load * ResponseTime Concurrency: number of requests being processed in parallel Load: Number of Requests per Second being handled ResponseTime: Average Response time in seconds. So in case you have a performance problem and for a given load your response time goes up by a factor of ten, the number of connections will also go up by a factpr of 10. That's most often the reason for d) and was the reason, why we asked for thread dumps. > Conclusions: > 1. In general, runnable threads should not be a prob, unless they correspond > to dropped connections. Since on our servers that have firewall between httpd > and tomcat, runnable connections are not being used for new requests and > tomcat keeps on creating new threads (leading to #b/c above), those threads > could correspond to: > i. connections dropped by firewall or > ii. hanging tomcat threads as httpd recycle timeout disconnected the > connection from that side (and there was no connectiontimeout in server.xml > so that tomcat could do the same) or > iii. combination of these "i" and "ii" > 2. Runnable threads on servers where no firewall exist (and we do not see > server running out of threads) should not be a point of concerns as they do > not correspond to dropped connections, as seen from netstat o/p at the end of > this email. So #a above could be ignored. > 3. Observation #d above is puzzling and currently I have no answers for that If d) happens again, do some thread dumps. > Action: > - check both sides by using "netstat -anop" (Apache side and the Tomcat side > without connectionTimeout, so you can see the problem in the original form). > See whether the number of AJP connections in the various TCP states differs > much between the netstat output on the Apache and on the Tomcat system. > - Bring workers.properties settings in line with Apache recommendations: > - Worker...cachesize=10 - set to 1 respectively when using Apache, remove this. Rely on the defaults for that one. > - Worker...cache_timeout=600 - remove > - Worker...recycle_timeout=300 - remove Hmmm. > Netstat o/p's: connector running on 21005, no firewall between httpd/tomcat > > Httpd Side: > > Proto Recv-Q Send-Q Local Address Foreign Address > State PID/Program name Timer > tcp 0 0 129.41.29.241:53777 129.41.29.48:21005 > ESTABLISHED - keepalive (2869.65/0/0) > tcp 0 0 129.41.29.241:53943 129.41.29.48:21005 > ESTABLISHED - keepalive (3341.39/0/0) > tcp 0 0 129.41.29.241:49950 129.41.29.48:21005 > ESTABLISHED - keepalive (6701.51/0/0) > tcp 0 0 129.41.29.241:49927 129.41.29.48:21005 > ESTABLISHED - keepalive (6240.25/0/0) > tcp 0 0 129.41.29.241:49926 129.41.29.48:21005 > ESTABLISHED - keepalive (6239.47/0/0) > tcp 0 0 129.41.29.241:49971 129.41.29.48:21005 > ESTABLISHED - keepalive (6931.40/0/0) > tcp 0 0 129.41.29.241:49868 129.41.29.48:21005 > ESTABLISHED - keepalive (5743.83/0/0) > tcp 0 0 129.41.29.241:49865 129.41.29.48:21005 > ESTABLISHED - keepalive (5741.65/0/0) > tcp 0 0 129.41.29.241:49867 129.41.29.48:21005 > ESTABLISHED - keepalive (5743.16/0/0) > tcp 0 0 129.41.29.241:49901 129.41.29.48:21005 > ESTABLISHED - keepalive (5906.92/0/0) > tcp 0 0 129.41.29.241:49795 129.41.29.48:21005 > ESTABLISHED - keepalive (4659.11/0/0) > tcp 0 0 129.41.29.241:49558 129.41.29.48:21005 > ESTABLISHED - keepalive (1705.06/0/0) > tcp 0 0 129.41.29.241:50796 129.41.29.48:21005 > ESTABLISHED - keepalive (4551.79/0/0) > tcp 0 0 129.41.29.241:50784 129.41.29.48:21005 > ESTABLISHED - keepalive (4539.53/0/0) > tcp 0 0 129.41.29.241:50711 129.41.29.48:21005 > ESTABLISHED - keepalive (4237.02/0/0) > tcp 0 0 129.41.29.241:50841 129.41.29.48:21005 > ESTABLISHED - keepalive (4544.81/0/0) > tcp 0 0 129.41.29.241:50526 129.41.29.48:21005 > ESTABLISHED - keepalive (3799.53/0/0) > tcp 0 0 129.41.29.241:50558 129.41.29.48:21005 > ESTABLISHED - keepalive (3766.94/0/0) > tcp 0 0 129.41.29.241:50642 129.41.29.48:21005 > ESTABLISHED - keepalive (4022.29/0/0) > tcp 0 0 129.41.29.241:50387 129.41.29.48:21005 > ESTABLISHED - keepalive (3723.14/0/0) > tcp 0 0 129.41.29.241:50404 129.41.29.48:21005 > ESTABLISHED - keepalive (3807.13/0/0) > tcp 0 0 129.41.29.241:52042 129.41.29.48:21005 > ESTABLISHED - keepalive (6770.05/0/0) > tcp 0 0 129.41.29.241:52092 129.41.29.48:21005 > ESTABLISHED - keepalive (542.41/0/0) > tcp 0 0 129.41.29.241:52222 129.41.29.48:21005 > ESTABLISHED - keepalive (555.67/0/0) > tcp 0 0 129.41.29.241:52109 129.41.29.48:21005 > ESTABLISHED - keepalive (580.33/0/0) > tcp 0 0 129.41.29.241:51907 129.41.29.48:21005 > ESTABLISHED - keepalive (6761.11/0/0) > tcp 0 0 129.41.29.241:52379 129.41.29.48:21005 > ESTABLISHED - keepalive (864.69/0/0) > tcp 0 0 ::ffff:129.41.29.112:443 ::ffff:198.70.193.2:21005 > TIME_WAIT - timewait (32.66/0/0) > $ date > Thu May 21 17:07:17 PDT 2009 > > Tomcat Side: > Proto Recv-Q Send-Q Local Address Foreign Address > State PID/Program name Timer > tcp 0 0 :::21005 :::* > LISTEN 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50796 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50784 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:52092 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50558 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:52042 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50526 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49971 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 865 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49926 > ESTABLISHED 18131/java on (0.20/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49927 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49950 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:53777 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49901 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50404 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:52222 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49868 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49865 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49867 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:51907 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50642 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50387 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:53943 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:53943 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:52109 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49795 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:50841 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:52379 > ESTABLISHED 18131/java off (0.00/0/0) > tcp 0 0 ::ffff:129.41.29.48:21005 ::ffff:129.41.29.241:49558 > ESTABLISHED 18131/java off (0.00/0/0) > $ date > Thu May 21 17:07:05 PDT 2009 No problem here. Regards, Rainer --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
