Re: Tomcat 5.5.23 keeps starting threads until OS runs out of memory
I have some more info about the issue that I think is related. I found our Tomcat executable is also establishing a lot of TCP connections. just doing a simple netstat just after I start tomcat and comparing it to when thread count for tomcat reaches 2000, I found a lot more lines like TCP 127.0.0.1:4912 127.0.0.1:2170 ESTABLISHED 4064 [tomcat5.exe] TCP 127.0.0.1:4912 127.0.0.1:4913 ESTABLISHED 4064 [tomcat5.exe] TCP 127.0.0.1:4913 127.0.0.1:4912 ESTABLISHED 4064 [tomcat5.exe] TCP 127.0.0.1:4914 127.0.0.1:4917 ESTABLISHED 4064 [tomcat5.exe] Is this some sort of tomcat ping? Our tomcat connectors are set up to limit the max http threads as below Connector port=8080 maxHttpHeaderSize=8192 maxThreads=250 minSpareThreads=25 maxSpareThreads=75 enableLookups=false redirectPort=8443 acceptCount=100 connectionTimeout=2 disableUploadTimeout=true / Connector port=8443 maxHttpHeaderSize=8192 maxThreads=100 minSpareThreads=25 maxSpareThreads=50 enableLookups=false disableUploadTimeout=true connectionTimeout=2 acceptCount=100 scheme=https secure=false proxyPort=443/ The only timeouts are for the http requests, are there any other timeouts I can configure that could potentially stop all those selector threads from persisting? - Original Message From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Tuesday, 14 July, 2009 2:59:36 Subject: RE: Tomcat 5.5.23 keeps starting threads until OS runs out of memory From: Christopher Schultz [mailto:ch...@christopherschultz.net] Subject: Re: Tomcat 5.5.23 keeps starting threads until OS runs out of memory I'm surprised you're not hitting a thread maximum in the OS and halting the JVM. I'm not aware of any hard limit in Windows; regardless, hitting such a limit just returns an error status on the system call, not a process abort. Showing more of the stack trace will certainly help reveal the problem. Not likely; that will only show the stack of the started thread, not where it was started from. A heap profiler should show who created the Thread objects, if the profiler captures enough of the call stack at object creation time. One would think a grep of the webapp source for calls to start() would be a rather quick first cut if a profiler can't be used. If the source isn't available, then one possible way to trap the origin of the Thread.start() call is to enable a security manager and only allow start() calls from Tomcat, not webapp, code. This would probably require several iterations and would be somewhat tedious. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 5.5.23 keeps starting threads until OS runs out of memory
I found the cause of the issue. A call was being made to a jsp that opened a connection to another application sitting on the same server (via orbd.exe). This jsp is being called approx 10 times a second (as it is a monitor jsp) and the majority of requests are being served but on occasion it would seem that the request is not being served and a connection is left dangling. So I guess it was not Tomcats fault in the end! I found some error messages in the catalina log alluding to a connection problem related to a jsp that is not actually part of the webapp - hence why I didn't think of it until I saw the stack trace. To prove it was this jsp, I took out the code that creates a connection and found the windows threads just stood at the same thread count. Gonna have to have a word with the TA about this... - Original Message From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Tuesday, 14 July, 2009 13:00:53 Subject: RE: Tomcat 5.5.23 keeps starting threads until OS runs out of memory From: Rahman Akhlaqur [mailto:aki...@yahoo.co.uk] Subject: Re: Tomcat 5.5.23 keeps starting threads until OS runs out of memory I have some more info about the issue that I think is related. I found our Tomcat executable is also establishing a lot of TCP connections. Tomcat isn't, your webapp is. Don't blame Tomcat for your misbehaving code. Is this some sort of tomcat ping? No, it's your webapp opening up some sort of connection - likely RMI, judging from your stack trace. Our tomcat connectors are set up to limit the max http threads as below That's irrelevant, since it's your webapp starting the extra threads. The only timeouts are for the http requests, are there any other timeouts I can configure that could potentially stop all those selector threads from persisting? Whatever you can configure in your webapp - this is a problem with your code, not with Tomcat. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat 5.5.23 keeps starting threads until OS runs out of memory
Hi I am having an issue with Tomcat starting too many selector threads. I have got some stack trace info about these threads as below: SelectorThread sun.nio.ch.PollArrayWrapper.poll0 ( native code ) sun.nio.ch.PollArrayWrapper.poll ( PollArrayWrapper.java:74 ) sun.nio.ch.WindowsSelectorImpl.doSelect ( WindowsSelectorImpl.java:61 ) sun.nio.ch.SelectorImpl.lockAndDoSelect ( SelectorImpl.java:69 ) sun.nio.ch.SelectorImpl.select ( SelectorImpl.java:80 ) com.sun.corba.se.impl.transport.SelectorImpl.run ( SelectorImpl.java:249 ) When Tomcat starts approx 1800-1900 hundred of these selector threads, Windows runs out of memory and we have to restart tomcat to restore the website. I think these are started by our webapp, but not sure how to trace this back to the bit of code that is doing this. Any suggestions would be appreciated. Best Regards, Akik Rahman - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 5.5.23 stops listening to requests on SSL port
Thanks for the suggestion of taking a thread dump... I think I have some instructions on how to do this on a windows server. The apache httpd server is on a different machine to Tomcat, thats how they can both listen on port 8443. - Original Message From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Cc: p...@pidster.com Sent: Friday, 1 May, 2009 18:07:42 Subject: Re: Tomcat 5.5.23 stops listening to requests on SSL port -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rahman, On 5/1/2009 9:24 AM, Rahman Akhlaqur wrote: The SSL is terminated at the load balancers, so the request is then a http request on port 8443 to our apache webserver (we use this to resolve multiple hostnames to just a few virtual hosts) which then just proxys the http request to Tomcat. At which point we want the request to be passed on to the content server as a secure request on port 443 - to match content hosts set up on port 443. It seems to work okay as the reverse path is fine, from the content server back to the end user. This sounds fishy. You terminate SSL at your lb, so there's no SSL anymore, right? Tomcat is listening on 8443 (as per your Connector configuration) so how can your lb send the request to Apache httpd on port 8443? Maybe you just reversed the ports in your post. It sounds like you are doing this: client [HTTPS] - lb:443 [HTTP] - httpd (port?) - Tomcat:8443 (If you are expecting a previously-encrypted connection coming to Tomcat, why not set secure=true in your Connector?) There is nothing in the Tomcat logs, the request is not even logged in the localhost access log - this points to Tomcat not even listening properly on port 8443. The other port (8080) is working okay though. Have you taken a thread dump to see what is happening? Since you are running 3 connectors, you might want to use a shared executor to manage threads, though the result with no further changes will be that /all/ threads will likely be tied-up, rather than only those servicing port 8443. I highly recommend a thread dump to see what all your threads are doing. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkn7LF4ACgkQ9CaO5/Lv0PCG5ACfcuG/xvslTxhXzvfp25inr/at InMAoKOvUg5QSxowVKUhwBxk8kSft96z =VMmX -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 5.5.23 stops listening to requests on SSL port
Hi P The SSL is terminated at the load balancers, so the request is then a http request on port 8443 to our apache webserver (we use this to resolve multiple hostnames to just a few virtual hosts) which then just proxys the http request to Tomcat. At which point we want the request to be passed on to the content server as a secure request on port 443 - to match content hosts set up on port 443. It seems to work okay as the reverse path is fine, from the content server back to the end user. There is nothing in the Tomcat logs, the request is not even logged in the localhost access log - this points to Tomcat not even listening properly on port 8443. The other port (8080) is working okay though. - Original Message From: Pid p...@pidster.com To: Tomcat Users List users@tomcat.apache.org Sent: Thursday, 30 April, 2009 16:59:25 Subject: Re: Tomcat 5.5.23 stops listening to requests on SSL port Rahman Akhlaqur wrote: Hi I am having an issue with my Tomcat server not responding on the SSL port. I reported this as a possible bug but this was rejected as a config issue. More info as below: The Tomcat connector is configured as below !-- Define a non-SSL HTTP/1.1 Connector on port 8080 -- Connector port=8080 maxHttpHeaderSize=8192 maxThreads=350 minSpareThreads=25 maxSpareThreads=75 enableLookups=false redirectPort=8443 acceptCount=100 connectionTimeout=2 disableUploadTimeout=true / !-- Define a SSL HTTP/1.1 Connector on port 8443 -- Connector port=8443 maxHttpHeaderSize=8192 maxThreads=150 minSpareThreads=25 maxSpareThreads=75 enableLookups=false disableUploadTimeout=true connectionTimeout=2 acceptCount=100 scheme=https secure=false proxyPort=443/ !-- Define an AJP 1.3 Connector on port 8009 -- Connector port=8009 enableLookups=false redirectPort=8443 protocol=AJP/1.3 / The issue is that Tomcat stops listening on port 8443 after a while of running. This means no HTML is returned to end users browsers. I know this as I am getting a HTTP error code 502 from the apache 2.2.4 webserver that is proxying the request to Tomcat. The setup I have is that SSL is terminated at a load balancer and forwards the request to port 8443 to apache, which then proxy passes the request to Tomcat to port 8443 and then Tomcat passes this request to the content server on port 443 (as this is a secure request). So you're sending SSL at each stage via 2 proxies? ssl- LB:8443 ssl- HTTPD:8443 ssl- Tomcat:8443 ssl- ContentSrv One theory is that end users are abruptly terminating their session with Tomcat (by cancelling their secure requests or closing their browsers) and Tomcat is not closing the thread when this happens. Therefore running out of threads. I am not sure about this and need some way to prove it. What do the Tomcat logs say when this occurs? p Akik Rahman - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat 5.5.23 stops listening to requests on SSL port
Hi I am having an issue with my Tomcat server not responding on the SSL port. I reported this as a possible bug but this was rejected as a config issue. More info as below: The Tomcat connector is configured as below !-- Define a non-SSL HTTP/1.1 Connector on port 8080 -- Connector port=8080 maxHttpHeaderSize=8192 maxThreads=350 minSpareThreads=25 maxSpareThreads=75 enableLookups=false redirectPort=8443 acceptCount=100 connectionTimeout=2 disableUploadTimeout=true / !-- Define a SSL HTTP/1.1 Connector on port 8443 -- Connector port=8443 maxHttpHeaderSize=8192 maxThreads=150 minSpareThreads=25 maxSpareThreads=75 enableLookups=false disableUploadTimeout=true connectionTimeout=2 acceptCount=100 scheme=https secure=false proxyPort=443/ !-- Define an AJP 1.3 Connector on port 8009 -- Connector port=8009 enableLookups=false redirectPort=8443 protocol=AJP/1.3 / The issue is that Tomcat stops listening on port 8443 after a while of running. This means no HTML is returned to end users browsers. I know this as I am getting a HTTP error code 502 from the apache 2.2.4 webserver that is proxying the request to Tomcat. The setup I have is that SSL is terminated at a load balancer and forwards the request to port 8443 to apache, which then proxy passes the request to Tomcat to port 8443 and then Tomcat passes this request to the content server on port 443 (as this is a secure request). One theory is that end users are abruptly terminating their session with Tomcat (by cancelling their secure requests or closing their browsers) and Tomcat is not closing the thread when this happens. Therefore running out of threads. I am not sure about this and need some way to prove it. Akik Rahman - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
