Re: Tomcat HTTPS Connector Stops working after an hour
*Point 1 and 2. * 1. When I remove acceptCount=1 My tomcat 7 works fine. 2. But I had the same property in tomcat 6 and it worked fine on that. A. I tried remove and altering some variables to check this problem. Mistakenly I thought acceptCount was the culprit, because after removing acceptCount, tomcat was running for 2 hours then 443 connector stopped. Tomcat 6 with the same configuration (server.xml) works fine. *Point 4* 4. Also found out that when it reaches 9000 connections with my existing connector [See below] the tomcat https [Http works fine] stops responding and the browser says 'Connection reset by peer'. A. When the request count reaches 9000, the tomcat fails within minutes. This depends , the last two times I tested, I was able to reproduce this. Log from Tomcat Server status. (after 1/half hour) Max threads: 5000 Current thread count: 209 Current thread busy: 2 Keeped alive sockets count: 56 Max processing time: 300477 ms Processing time: 8788.765 s Request count: 9926 Error count: 783 Bytes received: 0.22 MB Bytes sent: 5.43 MB On 12/28/2011 2:37 PM, ma...@apache.org wrote: Saravanan Lsaravan...@te-soft.com wrote: Tomcat does not pass through any proxy. My firefox browser has a proxy plugin configured which RELAYED this message. I enabled this plugin because, I wanted to be sure of whats happening. (I guess I should made it clear) So the end line is tomcat does not responds to the request at all. Firefox says- Connection reset by peer. Chrome says - Same as above but in a different manner Firefox with Proxy - 504 Q. how your system is configured A. Just tomcat running in linux box (Fedora) with APR and Native compiled. 3. A 504 response code is a gateway timeout. Tomcat does not issue this on its own. Either you have a proxy in front of Tomcat or your application is returning this value. A. Yes the proxy relayed this message, it seems the tomcat did not respond with any message at all. - Original Message - From: Mark Thomasma...@apache.org To: Tomcat Users Listusers@tomcat.apache.org Sent: Tuesday, December 27, 2011 10:58:29 PM Subject: Re: Tomcat HTTPS Connector Stops working after an hour On 27/12/2011 10:47, Saravanan L wrote: 3. A 504 response code is a gateway timeout. Tomcat does not issue this on its own. Either you have a proxy in front of Tomcat or your application is returning this value. A. Yes the proxy relayed this message, it seems the tomcat did not respond with any message at all. Which is the first mention in this thread that you are accessing Tomcat via a proxy which makes most of the previous contents of this thread null and void. I think you need to start again with a more complete description of how your system is configured. This is look less and less like a Tomcat issue and more and more like a poorly configured proxy. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Ok. So go back a few messages and address points 1, 2 and 4. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Regards *Saravanan.L* This message and any attachment(s) contained here are information that is confidential, proprietary to TE Software Services and its customers. Contents may be privileged or otherwise protected by law. The information is solely intended for the individual or the entity it is addressed to. If you are not the intended recipient of this message, you are not authorized to read, forward, print, retain, copy or disseminate this message or any part of it. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete it from your computer.
Re: Tomcat HTTPS Connector Stops working after an hour
On 29/12/2011 08:51, Saravanan L wrote: *Point 1 and 2. * 1. When I remove acceptCount=1 My tomcat 7 works fine. So remove it and don't put it back. 2. But I had the same property in tomcat 6 and it worked fine on that. Clearly something is broken with your app / Tomcat 7 configuration. Using maxThreads=5000 when you are only servicing 9000 requests in 1800 seconds with an average processing time of less than 1s per request is not only excessive it will mask any connection related problems you do have. The default of 200 should be fine. Since you are using APR, you don't need a thread to maintain a keep-alive connection so I would reduce maxThreads to no more than 20 for now to aid debugging. Once you have fixed the problem, then you can restore the default. A. I tried remove and altering some variables to check this problem. Mistakenly I thought acceptCount was the culprit, because after removing acceptCount, tomcat was running for 2 hours then 443 connector stopped. Tomcat 6 with the same configuration (server.xml) works fine. That is pretty much completely unhelpful. a) you need to tell us what you are changing and what the results were and b) you should only change one thing at a time. *Point 4* 4. Also found out that when it reaches 9000 connections with my existing connector [See below] the tomcat https [Http works fine] stops responding and the browser says 'Connection reset by peer'. Connections != requests. Please make sure you use the correct terminology because if you don't it will cause confusion. Point was was related to old, invalid attributes you had in server.xml that should be removed. A. When the request count reaches 9000, the tomcat fails within minutes. This depends , the last two times I tested, I was able to reproduce this. Log from Tomcat Server status. (after 1/half hour) Max threads: 5000 Current thread count: 209 Current thread busy: 2 Keeped alive sockets count: 56 Max processing time: 300477 ms Processing time: 8788.765 s Request count: 9926 Error count: 783 Bytes received: 0.22 MB Bytes sent: 5.43 MB So, actions for you are: - remove accepCount from the connector configuration - reduce maxThreads to 20 - remove use of attributes that are not supported in Tomcat 7 (check that each attribute is in the docs) - explain exactly how you are testing these SSL connections (JMeter, browser, something else?) - repeat your test with the above settings and then report the results. Include relevant extracts from: - logs - netstat output - thread dump Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat HTTPS Connector Stops working after an hour
Saravanan L saravan...@te-soft.com wrote: Tomcat does not pass through any proxy. My firefox browser has a proxy plugin configured which RELAYED this message. I enabled this plugin because, I wanted to be sure of whats happening. (I guess I should made it clear) So the end line is tomcat does not responds to the request at all. Firefox says- Connection reset by peer. Chrome says - Same as above but in a different manner Firefox with Proxy - 504 Q. how your system is configured A. Just tomcat running in linux box (Fedora) with APR and Native compiled. 3. A 504 response code is a gateway timeout. Tomcat does not issue this on its own. Either you have a proxy in front of Tomcat or your application is returning this value. A. Yes the proxy relayed this message, it seems the tomcat did not respond with any message at all. - Original Message - From: Mark Thomas ma...@apache.org To: Tomcat Users List users@tomcat.apache.org Sent: Tuesday, December 27, 2011 10:58:29 PM Subject: Re: Tomcat HTTPS Connector Stops working after an hour On 27/12/2011 10:47, Saravanan L wrote: 3. A 504 response code is a gateway timeout. Tomcat does not issue this on its own. Either you have a proxy in front of Tomcat or your application is returning this value. A. Yes the proxy relayed this message, it seems the tomcat did not respond with any message at all. Which is the first mention in this thread that you are accessing Tomcat via a proxy which makes most of the previous contents of this thread null and void. I think you need to start again with a more complete description of how your system is configured. This is look less and less like a Tomcat issue and more and more like a poorly configured proxy. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Ok. So go back a few messages and address points 1, 2 and 4. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat HTTPS Connector Stops working after an hour
On 27 dec. 2011, at 06:25, Saravanan L saravan...@te-soft.com wrote: Please find the server.xml attached. The real problem is I dont know where to look at. -There are no error in logs or the linux sys logs. - I cannot diagnose as the connector(443) does not even connect. I saw the same with the HTTP BIO connector on Tomcat 7.0.22. It would start responding intermittently and then stop responding completely. This particular Tomcat has a second HTTP connector pool for an Apache httpd using mod_proxy_http. That pool kept working beautifully. I went back to 6.0.x and now it all works beautifully, as I am used from Tomcat. I tried reproducing it on a load test machine, but it never showed there. I am at a loss where to start looking for this. Kees Jan On 12/24/2011 3:31 PM, Pid wrote: On 23/12/2011 14:47, Christopher Schultz wrote: Saravanan, On 12/23/11 7:03 AM, Saravanan L wrote: Please find the latest thread dump attached with mail. Tomcat 7 still crashes without acceptCount. I am wrong on this assumption. The response code is 504. It takes about 2 hours for this occur. The last status in http-apr-443 (In Server status ) is : Max threads: 5000 Seems high, as Chris says. Current thread count: 249 Can you post your full server.xml please, inline and with all XML comments/usernames/passwords removed. Current thread busy: 2 Low. Keeped alive sockets count: 40 Hmm? Where does that come from? Max processing time: 300477 ms Processing time: 8788.765 s Seems low for 2 hours of operation. Where does that number come from? Request count: 9223 Error count: 783 That's about 10%. What are those errors, exactly? Bytes received: 0.22 MB Bytes sent: 5.43 MB Both of those seem low for 2 hours of operation. What is your application/server doing? p What happens if you configure your HTTPS connector with fewer max threads, say, 50? Does the connector lock-up more quickly? What does your load profile look like? Is this observable in a testing environment or only in production? If you could set up a simple jmeter test against a test webapp, that would be ideal for trying to debug this issue. If you switch to NIO or BIO connector, does Tomcat become more stable, or do you experience the same phenomenon? -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Regards Saravanan.L This message and any attachment(s) contained here are information that is confidential, proprietary to TE Software and Services Pvt Ltd. and its customers. Contents may be privileged or otherwise protected by law. The information is solely intended for the individual or the entity it is addressed to. If you are not the intended recipient of this message, you are not authorized to read, forward, print, retain, copy or disseminate this message or any part of it. server.xml - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat HTTPS Connector Stops working after an hour
3. A 504 response code is a gateway timeout. Tomcat does not issue this on its own. Either you have a proxy in front of Tomcat or your application is returning this value. A. Yes the proxy relayed this message, it seems the tomcat did not respond with any message at all. looks like 200 idle threads to me. In comet idle threads are fine rights. Thats how comet/nio works right. Please correct me on this. On 12/27/2011 4:03 PM, Mark Thomas wrote: On 27/12/2011 08:55, Saravanan L wrote: Tried netstat. 443 is listening, but I cannot connect to it. I assume it shows no open connections either. Secondly Thread dump - I could not find any signs of problem or anything relative to the current issue in the threaddump [Please find the threaddump in my previous post]. Yep, that looks like 200 idle threads to me. Looking back through this thread I see a number of most likely unrelated issues being treated as the same problem. Reviewing the thread: 1. acceptCount=1 is a crazily high number. That you see odd effects with this does not surprise me. acceptCount is only for handling infrequent, short, very high spikes in load. 2. maxThreads=5000 is very high. Given that you handled ~10k requests in 2 hours it appears that maxThreads should be reduced by several orders of magnitude. 3. A 504 response code is a gateway timeout. Tomcat does not issue this on its own. Either you have a proxy in front of Tomcat or your application is returning this value. 4. There are settings in your server.xml that were dropped a couple of versions ago. I suggest the following: - restore the threadpool and acceptcount defaults - remove the old settings Run your tests and then diagnose the fault that appears. If you need pointers, ask here. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Regards *Saravanan.L* This message and any attachment(s) contained here are information that is confidential, proprietary to TE Software Services and its customers. Contents may be privileged or otherwise protected by law. The information is solely intended for the individual or the entity it is addressed to. If you are not the intended recipient of this message, you are not authorized to read, forward, print, retain, copy or disseminate this message or any part of it. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete it from your computer.
Re: Tomcat HTTPS Connector Stops working after an hour
Saravanan L saravan...@te-soft.com wrote: Please find the server.xml attached. The real problem is I dont know where to look at. -There are no error in logs or the linux sys logs. - I cannot diagnose as the connector(443) does not even connect. Of course you can diagnose this further. netstat and a thread dump would be a good place to start. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat HTTPS Connector Stops working after an hour
On 27/12/2011 08:55, Saravanan L wrote: Tried netstat. 443 is listening, but I cannot connect to it. I assume it shows no open connections either. Secondly Thread dump - I could not find any signs of problem or anything relative to the current issue in the threaddump [Please find the threaddump in my previous post]. Yep, that looks like 200 idle threads to me. Looking back through this thread I see a number of most likely unrelated issues being treated as the same problem. Reviewing the thread: 1. acceptCount=1 is a crazily high number. That you see odd effects with this does not surprise me. acceptCount is only for handling infrequent, short, very high spikes in load. 2. maxThreads=5000 is very high. Given that you handled ~10k requests in 2 hours it appears that maxThreads should be reduced by several orders of magnitude. 3. A 504 response code is a gateway timeout. Tomcat does not issue this on its own. Either you have a proxy in front of Tomcat or your application is returning this value. 4. There are settings in your server.xml that were dropped a couple of versions ago. I suggest the following: - restore the threadpool and acceptcount defaults - remove the old settings Run your tests and then diagnose the fault that appears. If you need pointers, ask here. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat HTTPS Connector Stops working after an hour
On 27 Dec 2011, at 10:48, Saravanan L saravan...@te-soft.com wrote: 3. A 504 response code is a gateway timeout. Tomcat does not issue this on its own. Either you have a proxy in front of Tomcat or your application is returning this value. A. Yes the proxy relayed this message, it seems the tomcat did not respond with any message at all. looks like 200 idle threads to me. In comet idle threads are fine rights. Idle threads do not inherently represent a problem with any of the Connectors. It just means they have no work to do. Thats how comet/nio works right. Please correct me on this. I don't know what you mean by this. When you stop getting responses are you still connecting via a proxy or directly to Tomcat? Is the previous thread dump taken shortly after Tomcat stopped responding, or before? p On 12/27/2011 4:03 PM, Mark Thomas wrote: On 27/12/2011 08:55, Saravanan L wrote: Tried netstat. 443 is listening, but I cannot connect to it. I assume it shows no open connections either. Secondly Thread dump - I could not find any signs of problem or anything relative to the current issue in the threaddump [Please find the threaddump in my previous post]. Yep, that looks like 200 idle threads to me. Looking back through this thread I see a number of most likely unrelated issues being treated as the same problem. Reviewing the thread: 1. acceptCount=1 is a crazily high number. That you see odd effects with this does not surprise me. acceptCount is only for handling infrequent, short, very high spikes in load. 2. maxThreads=5000 is very high. Given that you handled ~10k requests in 2 hours it appears that maxThreads should be reduced by several orders of magnitude. 3. A 504 response code is a gateway timeout. Tomcat does not issue this on its own. Either you have a proxy in front of Tomcat or your application is returning this value. 4. There are settings in your server.xml that were dropped a couple of versions ago. I suggest the following: - restore the threadpool and acceptcount defaults - remove the old settings Run your tests and then diagnose the fault that appears. If you need pointers, ask here. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Regards *Saravanan.L* This message and any attachment(s) contained here are information that is confidential, proprietary to TE Software Services and its customers. Contents may be privileged or otherwise protected by law. The information is solely intended for the individual or the entity it is addressed to. If you are not the intended recipient of this message, you are not authorized to read, forward, print, retain, copy or disseminate this message or any part of it. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete it from your computer. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat HTTPS Connector Stops working after an hour
Tried netstat. 443 is listening, but I cannot connect to it. Secondly Thread dump - I could not find any signs of problem or anything relative to the current issue in the threaddump [Please find the threaddump in my previous post]. On 12/27/2011 2:11 PM, ma...@apache.org wrote: Saravanan Lsaravan...@te-soft.com wrote: Please find the server.xml attached. The real problem is I dont know where to look at. -There are no error in logs or the linux sys logs. - I cannot diagnose as the connector(443) does not even connect. Of course you can diagnose this further. netstat and a thread dump would be a good place to start. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Regards *Saravanan.L* This message and any attachment(s) contained here are information that is confidential, proprietary to TE Software Services and its customers. Contents may be privileged or otherwise protected by law. The information is solely intended for the individual or the entity it is addressed to. If you are not the intended recipient of this message, you are not authorized to read, forward, print, retain, copy or disseminate this message or any part of it. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete it from your computer.
Re: Tomcat HTTPS Connector Stops working after an hour
On 27/12/2011 10:47, Saravanan L wrote: 3. A 504 response code is a gateway timeout. Tomcat does not issue this on its own. Either you have a proxy in front of Tomcat or your application is returning this value. A. Yes the proxy relayed this message, it seems the tomcat did not respond with any message at all. Which is the first mention in this thread that you are accessing Tomcat via a proxy which makes most of the previous contents of this thread null and void. I think you need to start again with a more complete description of how your system is configured. This is look less and less like a Tomcat issue and more and more like a poorly configured proxy. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat HTTPS Connector Stops working after an hour
Tomcat does not pass through any proxy. My firefox browser has a proxy plugin configured which RELAYED this message. I enabled this plugin because, I wanted to be sure of whats happening. (I guess I should made it clear) So the end line is tomcat does not responds to the request at all. Firefox says- Connection reset by peer. Chrome says - Same as above but in a different manner Firefox with Proxy - 504 Q. how your system is configured A. Just tomcat running in linux box (Fedora) with APR and Native compiled. 3. A 504 response code is a gateway timeout. Tomcat does not issue this on its own. Either you have a proxy in front of Tomcat or your application is returning this value. A. Yes the proxy relayed this message, it seems the tomcat did not respond with any message at all. - Original Message - From: Mark Thomas ma...@apache.org To: Tomcat Users List users@tomcat.apache.org Sent: Tuesday, December 27, 2011 10:58:29 PM Subject: Re: Tomcat HTTPS Connector Stops working after an hour On 27/12/2011 10:47, Saravanan L wrote: 3. A 504 response code is a gateway timeout. Tomcat does not issue this on its own. Either you have a proxy in front of Tomcat or your application is returning this value. A. Yes the proxy relayed this message, it seems the tomcat did not respond with any message at all. Which is the first mention in this thread that you are accessing Tomcat via a proxy which makes most of the previous contents of this thread null and void. I think you need to start again with a more complete description of how your system is configured. This is look less and less like a Tomcat issue and more and more like a poorly configured proxy. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat HTTPS Connector Stops working after an hour
Please find the server.xml attached. The real problem is I dont know where to look at. -There are no error in logs or the linux sys logs. - I cannot diagnose as the connector(443) does not even connect. On 12/24/2011 3:31 PM, Pid wrote: On 23/12/2011 14:47, Christopher Schultz wrote: Saravanan, On 12/23/11 7:03 AM, Saravanan L wrote: Please find the latest thread dump attached with mail. Tomcat 7 still crashes without acceptCount. I am wrong on this assumption. The response code is 504. It takes about 2 hours for this occur. The last status in http-apr-443 (In Server status ) is : Max threads: 5000 Seems high, as Chris says. Current thread count: 249 Can you post your full server.xml please, inline and with all XML comments/usernames/passwords removed. Current thread busy: 2 Low. Keeped alive sockets count: 40 Hmm? Where does that come from? Max processing time: 300477 ms Processing time: 8788.765 s Seems low for 2 hours of operation. Where does that number come from? Request count: 9223 Error count: 783 That's about 10%. What are those errors, exactly? Bytes received: 0.22 MB Bytes sent: 5.43 MB Both of those seem low for 2 hours of operation. What is your application/server doing? p What happens if you configure your HTTPS connector with fewer max threads, say, 50? Does the connector lock-up more quickly? What does your load profile look like? Is this observable in a testing environment or only in production? If you could set up a simple jmeter test against a test webapp, that would be ideal for trying to debug this issue. If you switch to NIO or BIO connector, does Tomcat become more stable, or do you experience the same phenomenon? -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Regards *Saravanan.L* This message and any attachment(s) contained here are information that is confidential, proprietary to TE Software Services and its customers. Contents may be privileged or otherwise protected by law. The information is solely intended for the individual or the entity it is addressed to. If you are not the intended recipient of this message, you are not authorized to read, forward, print, retain, copy or disseminate this message or any part of it. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete it from your computer.?xml version='1.0' encoding='utf-8'? Server port=8005 shutdown=SHUTDOWN Listener className=org.apache.catalina.core.AprLifecycleListener SSLEngine=on / !--Initialize Jasper prior to webapps are loaded. Documentation at /docs/jasper-howto.html -- Listener className=org.apache.catalina.core.JasperListener / !-- Prevent memory leaks due to use of particular java/javax APIs-- Listener className=org.apache.catalina.core.JreMemoryLeakPreventionListener / Listener className=org.apache.catalina.mbeans.GlobalResourcesLifecycleListener / Listener className=org.apache.catalina.core.ThreadLocalLeakPreventionListener / GlobalNamingResources Resource name=UserDatabase auth=Container type=org.apache.catalina.UserDatabase description=User database that can be updated and saved factory=org.apache.catalina.users.MemoryUserDatabaseFactory pathname=conf/tomcat-users.xml / /GlobalNamingResources Service name=Catalina Connector port=80 protocol=org.apache.coyote.http11.Http11AprProtocol connectionTimeout=30 tomcatAuthentication=false keepaliveTimeout=5000 backlog=50 maxThreads=5000 / Connector port=443 protocol=org.apache.coyote.http11.Http11AprProtocol maxThreads=5000 maxSpareThreads=150 enableLookups=true disableUploadTimeout=true SSLEnabled=true connectionTimeout=30 scheme=https secure=true SSLCertificateFile=x.crt SSLCertificateKeyFile=x.private clientAuth=false sslProtocol=All / Engine name=Catalina defaultHost=localhost Realm className=org.apache.catalina.realm.LockOutRealm Realm className=org.apache.catalina.realm.UserDatabaseRealm resourceName=UserDatabase/ /Realm Host name=localhost appBase=webapps unpackWARs=true autoDeploy=true Valve className=org.apache.catalina.valves.AccessLogValve directory=logs prefix=localhost_access_log. suffix=.txt pattern=%h %l %u %t quot;%rquot; %s %b / /Host /Engine /Service /Server - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat HTTPS Connector Stops working after an hour
0x7fc5bc84e560 (a sun.misc.GC$LatencyLock) Low Memory Detector daemon prio=10 tid=0x7fc5b0078800 nid=0x6792 runnable [0x] java.lang.Thread.State: RUNNABLE CompilerThread1 daemon prio=10 tid=0x7fc5b0075000 nid=0x6791 waiting on condition [0x] java.lang.Thread.State: RUNNABLE CompilerThread0 daemon prio=10 tid=0x7fc5b0073000 nid=0x6790 waiting on condition [0x] java.lang.Thread.State: RUNNABLE Signal Dispatcher daemon prio=10 tid=0x7fc5b0071000 nid=0x678f waiting on condition [0x] java.lang.Thread.State: RUNNABLE Finalizer daemon prio=10 tid=0x7fc5b0053800 nid=0x678e in Object.wait() [0x4112d000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x7fc5bc7eb950 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked 0x7fc5bc7eb950 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) Reference Handler daemon prio=10 tid=0x7fc5b0051800 nid=0x678d in Object.wait() [0x4102c000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x7fc5bc81cbd8 (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked 0x7fc5bc81cbd8 (a java.lang.ref.Reference$Lock) main prio=10 tid=0x40112000 nid=0x6789 runnable [0x41b16000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390) - locked 0x7fc5bda66c88 (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:453) at java.net.ServerSocket.accept(ServerSocket.java:421) at org.apache.catalina.core.StandardServer.await(StandardServer.java:447) at org.apache.catalina.startup.Catalina.await(Catalina.java:706) at org.apache.catalina.startup.Catalina.start(Catalina.java:652) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:322) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:450) VM Thread prio=10 tid=0x7fc5b004b000 nid=0x678c runnable GC task thread#0 (ParallelGC) prio=10 tid=0x4011c000 nid=0x678a runnable GC task thread#1 (ParallelGC) prio=10 tid=0x4011e000 nid=0x678b runnable VM Periodic Task Thread prio=10 tid=0x7fc5b007b000 nid=0x6793 waiting on condition JNI global references: 646 Heap PSYoungGen total 136448K, used 68467K [0x7fc5d145, 0x7fc5dbaf, 0x7fc5dbaf) eden space 102272K, 50% used [0x7fc5d145,0x7fc5d4722640,0x7fc5d783) from space 34176K, 48% used [0x7fc5d783,0x7fc5d883a8d8,0x7fc5d999) to space 33600K, 0% used [0x7fc5d9a2,0x7fc5d9a2,0x7fc5dbaf) PSOldGentotal 121536K, used 78653K [0x7fc5bc6f, 0x7fc5c3da, 0x7fc5d145) object space 121536K, 64% used [0x7fc5bc6f,0x7fc5c13bf708,0x7fc5c3da) PSPermGen total 64000K, used 47797K [0x7fc5b72f, 0x7fc5bb17, 0x7fc5bc6f) object space 64000K, 74% used [0x7fc5b72f,0x7fc5ba19d448,0x7fc5bb17) ---DUMP END- On 12/22/2011 6:45 PM, Caldarale, Charles R wrote: From: Saravanan L [mailto:saravan...@te-soft.com] Subject: Fwd: Tomcat HTTPS Connector Stops working after an hour We have setup Tomcat 7 with APR in linux box. The issue is that after an hour or two, the https stops responding, but the HTTP connector is still working. What does a thread dump show? http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org * * Original Message Subject: Fwd: Tomcat HTTPS Connector Stops working after an hour Date: Thu, 22
Re: Tomcat HTTPS Connector Stops working after an hour
On 23/12/2011 14:47, Christopher Schultz wrote: Saravanan, On 12/23/11 7:03 AM, Saravanan L wrote: Please find the latest thread dump attached with mail. Tomcat 7 still crashes without acceptCount. I am wrong on this assumption. The response code is 504. It takes about 2 hours for this occur. The last status in http-apr-443 (In Server status ) is : Max threads: 5000 Seems high, as Chris says. Current thread count: 249 Can you post your full server.xml please, inline and with all XML comments/usernames/passwords removed. Current thread busy: 2 Low. Keeped alive sockets count: 40 Hmm? Where does that come from? Max processing time: 300477 ms Processing time: 8788.765 s Seems low for 2 hours of operation. Where does that number come from? Request count: 9223 Error count: 783 That's about 10%. What are those errors, exactly? Bytes received: 0.22 MB Bytes sent: 5.43 MB Both of those seem low for 2 hours of operation. What is your application/server doing? p What happens if you configure your HTTPS connector with fewer max threads, say, 50? Does the connector lock-up more quickly? What does your load profile look like? Is this observable in a testing environment or only in production? If you could set up a simple jmeter test against a test webapp, that would be ideal for trying to debug this issue. If you switch to NIO or BIO connector, does Tomcat become more stable, or do you experience the same phenomenon? -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- [key:62590808] signature.asc Description: OpenPGP digital signature
Re: Tomcat HTTPS Connector Stops working after an hour
() [0x4112d000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x7fc5bc7eb950 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked 0x7fc5bc7eb950 (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) Reference Handler daemon prio=10 tid=0x7fc5b0051800 nid=0x678d in Object.wait() [0x4102c000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x7fc5bc81cbd8 (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked 0x7fc5bc81cbd8 (a java.lang.ref.Reference$Lock) main prio=10 tid=0x40112000 nid=0x6789 runnable [0x41b16000] java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:390) - locked 0x7fc5bda66c88 (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:453) at java.net.ServerSocket.accept(ServerSocket.java:421) at org.apache.catalina.core.StandardServer.await(StandardServer.java:447) at org.apache.catalina.startup.Catalina.await(Catalina.java:706) at org.apache.catalina.startup.Catalina.start(Catalina.java:652) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:322) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:450) VM Thread prio=10 tid=0x7fc5b004b000 nid=0x678c runnable GC task thread#0 (ParallelGC) prio=10 tid=0x4011c000 nid=0x678a runnable GC task thread#1 (ParallelGC) prio=10 tid=0x4011e000 nid=0x678b runnable VM Periodic Task Thread prio=10 tid=0x7fc5b007b000 nid=0x6793 waiting on condition JNI global references: 646 Heap PSYoungGen total 136448K, used 68467K [0x7fc5d145, 0x7fc5dbaf, 0x7fc5dbaf) eden space 102272K, 50% used [0x7fc5d145,0x7fc5d4722640,0x7fc5d783) from space 34176K, 48% used [0x7fc5d783,0x7fc5d883a8d8,0x7fc5d999) to space 33600K, 0% used [0x7fc5d9a2,0x7fc5d9a2,0x7fc5dbaf) PSOldGentotal 121536K, used 78653K [0x7fc5bc6f, 0x7fc5c3da, 0x7fc5d145) object space 121536K, 64% used [0x7fc5bc6f,0x7fc5c13bf708,0x7fc5c3da) PSPermGen total 64000K, used 47797K [0x7fc5b72f, 0x7fc5bb17, 0x7fc5bc6f) object space 64000K, 74% used [0x7fc5b72f,0x7fc5ba19d448,0x7fc5bb17) ---DUMP END- On 12/22/2011 6:45 PM, Caldarale, Charles R wrote: From: Saravanan L [mailto:saravan...@te-soft.com] Subject: Fwd: Tomcat HTTPS Connector Stops working after an hour We have setup Tomcat 7 with APR in linux box. The issue is that after an hour or two, the https stops responding, but the HTTP connector is still working. What does a thread dump show? http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org * * Original Message Subject:Fwd: Tomcat HTTPS Connector Stops working after an hour Date: Thu, 22 Dec 2011 16:13:45 +0530 From: Saravanan L saravan...@te-soft.com To: users@tomcat.apache.org CC: 'Binu R' bi...@te-soft.com Hi, We have setup Tomcat 7 with APR in linux box. The issue is that after an hour or two, the https stops responding, but the HTTP connector is still working. When we try to open https://192.168.1.32/, it does not respond.[ it does not timeout but the browser says server could not reached] . We tried connecting using telnet and nc, but tomcat refuses to connect. There is no log of is problem in system network logs and tomcat logs aswell. One more thing, when tomcat is restarted https works for an hour and again the same problem. Please help us out on this. Note: We also run a Tomcat 6 server with the same configuration. But we
Re: Tomcat HTTPS Connector Stops working after an hour
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Saravanan, On 12/23/11 7:03 AM, Saravanan L wrote: Please find the latest thread dump attached with mail. Tomcat 7 still crashes without acceptCount. I am wrong on this assumption. The response code is 504. It takes about 2 hours for this occur. The last status in http-apr-443 (In Server status ) is : Max threads: 5000 Current thread count: 249 Current thread busy: 2 Keeped alive sockets count: 40 Max processing time: 300477 ms Processing time: 8788.765 s Request count: 9223 Error count: 783 Bytes received: 0.22 MB Bytes sent: 5.43 MB What happens if you configure your HTTPS connector with fewer max threads, say, 50? Does the connector lock-up more quickly? What does your load profile look like? Is this observable in a testing environment or only in production? If you could set up a simple jmeter test against a test webapp, that would be ideal for trying to debug this issue. If you switch to NIO or BIO connector, does Tomcat become more stable, or do you experience the same phenomenon? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk70lJQACgkQ9CaO5/Lv0PBIWwCfcH1xXusj3LQw/YVEozd3tHGh N5oAn3nNrDwYvDdZ2SwHiospx4pvNRH6 =D9tY -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Fwd: Tomcat HTTPS Connector Stops working after an hour
Hi, We have setup Tomcat 7 with APR in linux box. The issue is that after an hour or two, the https stops responding, but the HTTP connector is still working. When we try to open https://192.168.1.32/, it does not respond.[ it does not timeout but the browser says server could not reached] . We tried connecting using telnet and nc, but tomcat refuses to connect. There is no log of is problem in system network logs and tomcat logs aswell. One more thing, when tomcat is restarted https works for an hour and again the same problem. Please help us out on this. Note: We also run a Tomcat 6 server with the same configuration. But we dont get this problem in 6. Tomcat version : 7.0.23 Tomcat Native : 1.1.22 Apr: 1.4.5 Tomcat connector configuration, Connector protocol=org.apache.coyote.http11.Http11AprProtocol port=443 minSpareThreads=5 enableLookups=true disableUploadTimeout=true acceptCount=100 maxThreads=200 connectionTimeout=300 scheme=https secure=true SSLEnabled=true SSLCertificateFile=developer.tess.com.cert SSLCertificateKeyFile=developer.tess.com.key clientAuth=false sslProtocol=TLS / Connector port=80 protocol=org.apache.coyote.http11.Http11AprProtocol connectionTimeout=2 tomcatAuthentication=false backlog=50 maxThreads=300 / -- Regards *Saravanan.L* This message and any attachment(s) contained here are information that is confidential, proprietary to TE Software Services and its customers. Contents may be privileged or otherwise protected by law. The information is solely intended for the individual or the entity it is addressed to. If you are not the intended recipient of this message, you are not authorized to read, forward, print, retain, copy or disseminate this message or any part of it. If you have received this e-mail in error, please notify the sender immediately by return e-mail and delete it from your computer.
RE: Tomcat HTTPS Connector Stops working after an hour
From: Saravanan L [mailto:saravan...@te-soft.com] Subject: Fwd: Tomcat HTTPS Connector Stops working after an hour We have setup Tomcat 7 with APR in linux box. The issue is that after an hour or two, the https stops responding, but the HTTP connector is still working. What does a thread dump show? http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
