Re: Error 503 ocurring when server under load
On 12 October 2010 19:47, Christopher Schultz ch...@christopherschultz.net wrote: I would. I believe that is true. On the other hand, there is another case where you might have problems. If you have, say, 512 worker threads in Apache httpd but you only have, say, 200 request processor threads configured in Tomcat, then you will get mod_jk connection failures on the httpd side. I would recommend that you have enough request processors configured in Tomcat to handle the expected load. - -chris Thanks Chris (and everyone else for their comments). As an update: I updated Tomcat from 6.0.24 to 6.0.29 and commented out the cachesize directive from workers.properties. The server ran for a full day yesterday and not a single 503 error. :) So with a bit of luck it will continue to work as expected. Of course I'll monitor for any other issues for the next few days. Rob - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Error 503 ocurring when server under load
I just realized that I somehow replied to the wrong thread on this one. It was meant for Jane Muse's thread. I'll repost there in case someone missed it. Sorry to interrupt your thread with irrelevant information/spleculation. Jeff -Original Message- From: Jeffrey Janner [mailto:jeffrey.jan...@polydyne.com] Sent: Tuesday, October 12, 2010 2:23 PM To: Tomcat Users List Subject: RE: Error 503 ocurring when server under load I just occurred to me that I don't think anyone's asked if these are net-mounted file systems. I've seen this timestamp-shifting before, but only on net-mounted filesystems. Usually the source and local systems are set to different timezones (or DST settings). -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Tuesday, October 12, 2010 1:47 PM To: Tomcat Users List Subject: Re: Error 503 ocurring when server under load -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rob, On 10/11/2010 4:40 PM, Rob G wrote: So if I'm reading your email and the docs correctly. I should just comment out the cachesize=10 from the workers.properties. I would. And since for connection_pool_size (that replaced it) JK will discover this number for the Apache web server automatically and set the pool size to this value, I don't need to add anything to the workers.properties file? I believe that is true. On the other hand, there is another case where you might have problems. If you have, say, 512 worker threads in Apache httpd but you only have, say, 200 request processor threads configured in Tomcat, then you will get mod_jk connection failures on the httpd side. I would recommend that you have enough request processors configured in Tomcat to handle the expected load. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAky0rToACgkQ9CaO5/Lv0PCvjACgwttZ9YfINxpWP+DI1+VlKfvI OTAAoK+2RQRibL56GdYWlaWxx6obZVln =hY3s -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org ___ ___ Confidentiality Notice: This Transmission (including any attachments) may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this transmission in error, please immediately reply to the sender or telephone (512) 343-9100 and delete this transmission from your system. __ Confidentiality Notice: This Transmission (including any attachments) may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this transmission in error, please immediately reply to the sender or telephone (512) 343-9100 and delete this transmission from your system.
Re: Error 503 ocurring when server under load
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rob, On 10/11/2010 4:40 PM, Rob G wrote: So if I'm reading your email and the docs correctly. I should just comment out the cachesize=10 from the workers.properties. I would. And since for connection_pool_size (that replaced it) JK will discover this number for the Apache web server automatically and set the pool size to this value, I don't need to add anything to the workers.properties file? I believe that is true. On the other hand, there is another case where you might have problems. If you have, say, 512 worker threads in Apache httpd but you only have, say, 200 request processor threads configured in Tomcat, then you will get mod_jk connection failures on the httpd side. I would recommend that you have enough request processors configured in Tomcat to handle the expected load. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAky0rToACgkQ9CaO5/Lv0PCvjACgwttZ9YfINxpWP+DI1+VlKfvI OTAAoK+2RQRibL56GdYWlaWxx6obZVln =hY3s -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Error 503 ocurring when server under load
I just occurred to me that I don't think anyone's asked if these are net-mounted file systems. I've seen this timestamp-shifting before, but only on net-mounted filesystems. Usually the source and local systems are set to different timezones (or DST settings). -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Tuesday, October 12, 2010 1:47 PM To: Tomcat Users List Subject: Re: Error 503 ocurring when server under load -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rob, On 10/11/2010 4:40 PM, Rob G wrote: So if I'm reading your email and the docs correctly. I should just comment out the cachesize=10 from the workers.properties. I would. And since for connection_pool_size (that replaced it) JK will discover this number for the Apache web server automatically and set the pool size to this value, I don't need to add anything to the workers.properties file? I believe that is true. On the other hand, there is another case where you might have problems. If you have, say, 512 worker threads in Apache httpd but you only have, say, 200 request processor threads configured in Tomcat, then you will get mod_jk connection failures on the httpd side. I would recommend that you have enough request processors configured in Tomcat to handle the expected load. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAky0rToACgkQ9CaO5/Lv0PCvjACgwttZ9YfINxpWP+DI1+VlKfvI OTAAoK+2RQRibL56GdYWlaWxx6obZVln =hY3s -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org __ Confidentiality Notice: This Transmission (including any attachments) may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this transmission in error, please immediately reply to the sender or telephone (512) 343-9100 and delete this transmission from your system.
Re: Error 503 ocurring when server under load
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jeffrey, On 10/12/2010 3:22 PM, Jeffrey Janner wrote: I just occurred to me that I don't think anyone's asked if these are net-mounted file systems. I've seen this timestamp-shifting before, but only on net-mounted filesystems. Usually the source and local systems are set to different timezones (or DST settings). That's a very good thought. I've never seen good things happen when using a network filesystem for webapp deployments. Feel free to store your data on NFS, just don't use NFS to store your WAR files ;) - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAky0xJcACgkQ9CaO5/Lv0PCEQwCfdh1CXe7Z0MYqZPpgpsDdsoGA 838An1uHbLlpeQxNTqnF44zl04rqnE2d =PgBy -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Error 503 ocurring when server under load
Christopher Schultz wrote This directive has been deprecated since 1.2.16. Cachesize defines the number of connections made to the AJP backend that are maintained as a connection pool. It will limit the number of those connection that each web server child process can make. Cachesize property is used only for multi threaded web servers such as Apache 2.0 (worker), IIS and Netscape. The cachesize property should reflect the number of threads per child process. JK will discover the number of threads per child process on Apache 2 web server with worker-mpm and set its default value to match the ThreadsPerChild Apache directive. [...] Since mpm_winnt (like mpm_worker) uses ThreadsPerChild to set the number of worker threads (and therefore simultaneous requests that can be accepted), you'll have to know what ThreadsPerChild is set to. That's easy: just search for that directive in httpd.conf. If it's not there, or commented-out, the default is 64 on mpm_winnt (http://httpd.apache.org/docs/2.0/mod/mpm_common.html#threadsperchild). If you've left it as the default (64) and you've set worker.cachesize (really worker.connection_pool_size) to 10, then you can accept 64 connections from clients but only call-out to Tomcat over 10 of those connections: the others will have to wait. I'm not sure what the scheduling algorithm is for choosing which thread gets the next available connection from the pool (FIFO?), but it's very possible to starve your clients even after 10 simultaneous connections. - -chris So if I'm reading your email and the docs correctly. I should just comment out the cachesize=10 from the workers.properties. And since for connection_pool_size (that replaced it) JK will discover this number for the Apache web server automatically and set the pool size to this value, I don't need to add anything to the workers.properties file? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Error 503 ocurring when server under load
On 08/10/2010 10:08, André Warnier wrote: Pid wrote: ... Is this info in the original email? Yes. (If so, I think my mailer isn't behaving well). It seems also somewhat spastic. We've been receiving multiple copies of what you send. Or did you just have too much coffee lately ? Hmm. My apologies. I was on a tiny unreliable network connection; I think client server dropped restarted transactions. p 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: Error 503 ocurring when server under load
On 08/10/2010 00:07, Rob G wrote: Thanks to all for their replies to date. Much appreciated, I'll respond to some of the points raised. Apologies if I'm asking basic stuff but still getting my ahead around Tomcat and Apache integration. On 7 October 2010 18:31, André Warnier wrote: Your configuration looks very clean to me (no unnecessary settings etc.), which in this case is a plus (a good base to start tuning). Thank you You may want to upgrade Tomcat to the latest version (6.0.29). Noted For example, at the moment these errors happen, what are these Tomcats really doing ? Are they really busy each processing 200 requests, with 200 threads running and actually doing something ? (200 is the default for the maxThreads attribute of the AJP Connector). Unfortunately I'm not in a position to conduct realtime monitoring of the server and Tomcat. Is there a way to log this thread data? How do you get log data? If yes, then you may just need a leaner application, or a bigger system (more RAM, faster CPU), or more systems. What does the Task Manager tell you about the total system load ? Again, unfortunately not in a position to monitor in real time. You can enable JMX and connect remotely on a specified port to get a look inside the JVM If not, and many of these threads are waiting, then you may have an issue with a keepAlive that is too long. See http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html, for the connectionTimeout and keepAliveTimeout attributes. Will do Whatever you do, first get an idea of the starting situation. Then modify one setting at a time, and observe (and note) the effects. Sensible advice Pid wrote: OK. Is there anything else different apart from the Servlet container? I'm not sure what you mean? Did you change DB too, or is was it a simple migration? Why do you have two Tomcat instances? (It's not a trick question, I'm interested in your reasoning.) Load balancing. I thought/guessed that two instances would be sufficent. Perhaps this was incorrect and I need more? Maybe, maybe not. I was trying to understand why you had two instances, rather than one with, say, more threads configured. Load balancing two Tomcat instances which are both using the same underlying computing resource will not offer a tangible benefit in terms of throughput / response time compared to a suitably configured single instance. If you're on a 32bit system and are hitting process memory limits, you might need two instances to cope with the load. How many threads server instances did you have before? What were your Java -Xmx etc settings before, and now? Are you using an Executor? I'll have to check, but I think it's safe to say that they are set to the default values as I don't recall changing anything relating to them. Thread dumps from a maxed out Tomcat will tell you what each Thread is waiting for. Collect a series of these during high load periods to find out what's happening. I'll have to figure out how to do this, but sounds a good idea. Do you have remote access to the machine? There are command line tools with JDK 6 (6.0.21 is the latest) which are useful. E.g. jstack, jmap. If you have remote access to the GUI, JDK 6 also has JConsole and VisualVM which provide metric tonnes of useful information. Otherwise you can connect those tools, for slightly less output, remotely via JMX. Be sure to use : org.apache.catalina.mbeans.JmxRemoteLifecycleListener http://tomcat.apache.org/tomcat-6.0-doc/config/listeners.html Please also check the port numbers you have in use for each Tomcat instance. Are the Tomcats installed and run as a service? p Christopher Schultz wrote I think you might want to compare the request processor counts you have for Tomcat versus mod_jk: the default maxThreads for a Tomcat Connector is 200, while your MaxClients settings might be different on your Apache httpd configuration. Excuse my ignorance, but can you clarify where I check these settings? I believe that only the Worker MPM is available on Microsoft Windows. Have you explicitly configured an MPM? Since I don't know what an MPM is I'm going to say no. Since you're using Apache 2.x, mod_jk should be able to auto-configure this value for you. Do I need to set anything to allow this auto config to happen? From my reading, you may be limiting yourself to 10 simultaneous connections from Apache httpd to Tomcat, and others will get a 503. What lead you to that conclusion? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: Error 503 ocurring when server under load
Pid wrote: ... Is this info in the original email? Yes. (If so, I think my mailer isn't behaving well). It seems also somewhat spastic. We've been receiving multiple copies of what you send. Or did you just have too much coffee lately ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Error 503 ocurring when server under load
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rob, On 10/7/2010 7:07 PM, Rob G wrote: Christopher Schultz wrote I think you might want to compare the request processor counts you have for Tomcat versus mod_jk: the default maxThreads for a Tomcat Connector is 200, while your MaxClients settings might be different on your Apache httpd configuration. Excuse my ignorance, but can you clarify where I check these settings? Obviously, the worker settings are in your Workers.properties file that you've already posted. The settings for Apache httpd are usually in a file called conf/httpd.conf in the Apache httpd installation directory. On win32, that might be something like C:\Program Files\Apache Foundation\Apache Web Server\conf\httpd.conf Tomcat's server settings are in conf/server.xml in the Apache Tomcat installation directory. On win32, that might be something like C:\Program Files\Apache Foundation\Apache Tomcat\conf\server.xml Since you have two copies of Tomcat working, you should check both of them. In fact, if you could post the server.xml files from both instances and tell us how you have them configured to run side-by-side, that would be helpful. Pid pointed out that you might not have both instances running due to a shutdownPort conflict. That might also be the case if you have other ports that conflict. I believe that only the Worker MPM is available on Microsoft Windows. Have you explicitly configured an MPM? Since I don't know what an MPM is I'm going to say no. :) That stands for Multi-Processing Module and it's how the Apache Web Server serves more than one request at a time. On *NIX, you can choose the prefork module which gives you one child process per simultaneous request, or the worker module which gets you a single (child?) process with many threads running within it. I just checked the documentation and it looks like on windows, you get mpm_winnt, which is different from even those, though it quite resembles the mpm_worker. Since you're using Apache 2.x, mod_jk should be able to auto-configure this value for you. Do I need to set anything to allow this auto config to happen? No. If you don't set any value, mod_jk will compute a reasonable (and usually correct) default. For a prefork MPM, having more than a single connection in the connection pool is a huge waste, since a single httpd request processor can't make more than one connection to Tomcat on the backend. With the worker MPM, the connection pool should probably be the same size as the number of worker threads you have configured. Otherwise, you'll have a deficit that may cause the errors you are encountering. From my reading, you may be limiting yourself to 10 simultaneous connections from Apache httpd to Tomcat, and others will get a 503. What lead you to that conclusion? Here's the text from the docs: This directive has been deprecated since 1.2.16. Cachesize defines the number of connections made to the AJP backend that are maintained as a connection pool. It will limit the number of those connection that each web server child process can make. Cachesize property is used only for multi threaded web servers such as Apache 2.0 (worker), IIS and Netscape. The cachesize property should reflect the number of threads per child process. JK will discover the number of threads per child process on Apache 2 web server with worker-mpm and set its default value to match the ThreadsPerChild Apache directive. [...] Since mpm_winnt (like mpm_worker) uses ThreadsPerChild to set the number of worker threads (and therefore simultaneous requests that can be accepted), you'll have to know what ThreadsPerChild is set to. That's easy: just search for that directive in httpd.conf. If it's not there, or commented-out, the default is 64 on mpm_winnt (http://httpd.apache.org/docs/2.0/mod/mpm_common.html#threadsperchild). If you've left it as the default (64) and you've set worker.cachesize (really worker.connection_pool_size) to 10, then you can accept 64 connections from clients but only call-out to Tomcat over 10 of those connections: the others will have to wait. I'm not sure what the scheduling algorithm is for choosing which thread gets the next available connection from the pool (FIFO?), but it's very possible to starve your clients even after 10 simultaneous connections. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkyvKLwACgkQ9CaO5/Lv0PBAqgCgh3/TNdasW5b6cnHXOP99BxrL jCIAmgMt/kFdRWIut1m9QF8TmzbZ7zW3 =LXSf -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Error 503 ocurring when server under load
Rob G wrote: Hey all, Recently migrated a production site (mixture of Servlets and JSPs) from Oracle Application Server to Apache/Tomcat. Since then we have seen numerous HTTP Error 503 - Service unavailable errors at peak times when site is under load. mod_jk.log has the following error message(s): [2184:1952] [error] jk_lb_worker.c (1473): All tomcat instances failed, no more workers left I'm looking for help in trying to tweak settings to prevent this, or confirmation that I've configured the setup correctly. Many thanks Rob Platform: Windows Server 2003 SP2 Setup: Two tomcat instances with a single Apache front end, all on the same server Versions Tomcat: 6.0.24 Apache: 2.2.16 mod SSL: 2.2.16 Open SSL: 0.9.8 mod_JK:1.2.30 Your configuration looks very clean to me (no unnecessary settings etc.), which in this case is a plus (a good base to start tuning). You may want to upgrade Tomcat to the latest version (6.0.29). But before you start tuning, you should get some idea of what is actually going on. For example, at the moment these errors happen, what are these Tomcats really doing ? Are they really busy each processing 200 requests, with 200 threads running and actually doing something ? (200 is the default for the maxThreads attribute of the AJP Connector). If yes, then you may just need a leaner application, or a bigger system (more RAM, faster CPU), or more systems. What does the Task Manager tell you about the total system load ? If not, and many of these threads are waiting, then you may have an issue with a keepAlive that is too long. See http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html, for the connectionTimeout and keepAliveTimeout attributes. Whatever you do, first get an idea of the starting situation. Then modify one setting at a time, and observe (and note) the effects. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Error 503 ocurring when server under load
On 07/10/2010 18:31, André Warnier wrote: Rob G wrote: Hey all, Recently migrated a production site (mixture of Servlets and JSPs) from Oracle Application Server to Apache/Tomcat. Since then we have seen numerous HTTP Error 503 - Service unavailable errors at peak times when site is under load. mod_jk.log has the following error message(s): OK. Is there anything else different apart from the Servlet container? [2184:1952] [error] jk_lb_worker.c (1473): All tomcat instances failed, no more workers left Seems like your Tomcats are maxed out. I'm looking for help in trying to tweak settings to prevent this, or confirmation that I've configured the setup correctly. Platform: Windows Server 2003 SP2 Setup: Two tomcat instances with a single Apache front end, all on the same server Why do you have two Tomcat instances? (It's not a trick question, I'm interested in your reasoning.) Versions Tomcat: 6.0.24 Apache: 2.2.16 mod SSL: 2.2.16 Open SSL: 0.9.8 There's newer OpenSSL available, with important security fixes, if I'm not mistaken. mod_JK:1.2.30 Your configuration looks very clean to me (no unnecessary settings etc.), which in this case is a plus (a good base to start tuning). Was there an attachment I didn't see? You may want to upgrade Tomcat to the latest version (6.0.29). +1 But before you start tuning, you should get some idea of what is actually going on. +1 For example, at the moment these errors happen, what are these Tomcats really doing ? Are they really busy each processing 200 requests, with 200 threads running and actually doing something ? (200 is the default for the maxThreads attribute of the AJP Connector). Q: How many threads server instances did you have before? If yes, then you may just need a leaner application, or a bigger system (more RAM, faster CPU), or more systems. What does the Task Manager tell you about the total system load ? What were your Java -Xmx etc settings before, and now? If not, and many of these threads are waiting, then you may have an issue with a keepAlive that is too long. Or several other things. What is your Connector config? Are you using an Executor? Is there a database behind this, if so, what are the DataSource pool size settings? See http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html, for the connectionTimeout and keepAliveTimeout attributes. Whatever you do, first get an idea of the starting situation. Then modify one setting at a time, and observe (and note) the effects. Thread dumps from a maxed out Tomcat will tell you what each Thread is waiting for. Collect a series of these during high load periods to find out what's happening. Enable JMX and check the Connector, (Executor if enabled), and DataSources; I'd be looking at backlog of requests, active + idle pool members, total pool size. p - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: Error 503 ocurring when server under load
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rob, On 10/7/2010 12:59 PM, Rob G wrote: Recently migrated a production site (mixture of Servlets and JSPs) from Oracle Application Server to Apache/Tomcat. Since then we have seen numerous HTTP Error 503 - Service unavailable errors at peak times when site is under load. mod_jk.log has the following error message(s): [2184:1952] [error] jk_lb_worker.c (1473): All tomcat instances failed, no more workers left Just to reinforce a comment by Andre, I think you might want to compare the request processor counts you have for Tomcat versus mod_jk: the default maxThreads for a Tomcat Connector is 200, while your MaxClients settings might be different on your Apache httpd configuration. If you have httpd.MaxClients=250 and connector.maxThreads=200 then you have a deficit of 50 who will get 503 errors if they try to access your webapp during peak load. Apache: 2.2.16 I believe that only the Worker MPM is available on Microsoft Windows. Have you explicitly configured an MPM? worker.worker1.cachesize=10 You might want to read the documentation for worker.cachesize in http://tomcat.apache.org/connectors-doc/reference/workers.html Note that it's deprecated and you should use connection_pool_size instead. Since you're using Apache 2.x, mod_jk should be able to auto-configure this value for you. - From my reading, you may be limiting yourself to 10 simultaneous connections from Apache httpd to Tomcat, and others will get a 503. Rainer or Mladen will have to comment to clarify: I am certainly no expert in mod_jk configurations. Hope that helps, - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkyuRwUACgkQ9CaO5/Lv0PC3nACePwYgc2w4td8jSdRCM5rp9IB9 PB4AoKj4cR9F+FhsR20W4vIvcWkRevyL =nuW/ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Error 503 ocurring when server under load
On 07/10/2010 23:17, Christopher Schultz wrote: Rob, On 10/7/2010 12:59 PM, Rob G wrote: Recently migrated a production site (mixture of Servlets and JSPs) from Oracle Application Server to Apache/Tomcat. Since then we have seen numerous HTTP Error 503 - Service unavailable errors at peak times when site is under load. mod_jk.log has the following error message(s): [2184:1952] [error] jk_lb_worker.c (1473): All tomcat instances failed, no more workers left Just to reinforce a comment by Andre, I think you might want to compare the request processor counts you have for Tomcat versus mod_jk: the default maxThreads for a Tomcat Connector is 200, while your MaxClients settings might be different on your Apache httpd configuration. If you have httpd.MaxClients=250 and connector.maxThreads=200 then you have a deficit of 50 who will get 503 errors if they try to access your webapp during peak load. Apache: 2.2.16 I believe that only the Worker MPM is available on Microsoft Windows. Have you explicitly configured an MPM? worker.worker1.cachesize=10 Is this info in the original email? (If so, I think my mailer isn't behaving well). p - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: Error 503 ocurring when server under load
On 07/10/2010 17:59, Rob G wrote: Hey all, Recently migrated a production site (mixture of Servlets and JSPs) from Oracle Application Server to Apache/Tomcat. Since then we have seen numerous HTTP Error 503 - Service unavailable errors at peak times when site is under load. mod_jk.log has the following error message(s): [2184:1952] [error] jk_lb_worker.c (1473): All tomcat instances failed, no more workers left I'm looking for help in trying to tweak settings to prevent this, or confirmation that I've configured the setup correctly. Many thanks Rob Platform: Windows Server 2003 SP2 Setup: Two tomcat instances with a single Apache front end, all on the same server Versions Tomcat: 6.0.24 Apache: 2.2.16 mod SSL: 2.2.16 Open SSL: 0.9.8 mod_JK:1.2.30 Conf: httpd.conf: IfModule mod_jk.c JkWorkersFile conf/workers.properties JkLogFile logs/mod_jk.log JkLogLevel error JkLogStampFormat [%a %b %d %H:%M:%S %Y] JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories JkRequestLogFormat %w %V %T Location /*/WEB-INF/* AllowOverride None deny from all /Location JkMount /examples balancer JkMount /examples/* balancer JkMount /App1/* balancer JkMount /App2/* balancer /IfModule Workers.properties: workers.tomcat_home=C:\tomcat1 workers.java_home=C:\Program Files\Java\jdk1.6.0_18 ps=\ # Define workers #worker.list=worker1,worker2 worker.list=balancer # Set properties for worker worker1 (ajp13) worker.worker1.type=ajp13 worker.worker1.host=localhost worker.worker1.port=8009 worker.worker1.lbfactor=1 worker.worker1.cachesize=10 worker.worker1.connection_pool_timeout=300 worker.worker1.socket_keepalive=FALSE #worker.worker1.recycle_timeout=300 # Set properties for worker worker2 (ajp13) worker.worker2.type=ajp13 worker.worker2.host=localhost worker.worker2.port=8019 worker.worker2.lbfactor=1 worker.worker2.cachesize=10 worker.worker2.connection_pool_timeout=300 worker.worker2.socket_keepalive=FALSE #worker.worker2.recycle_timeout=300 worker.balancer.type=lb worker.balancer.balance_workers=worker1,worker2 worker.balancer.method=R worker.balancer.sticky_session=True Server.xml (same for both tomcat instances apart from jvmRoute): ?xml version='1.0' encoding='utf-8'? Server port=8005 shutdown=SHUTDOWN Listener className=org.apache.catalina.core.AprLifecycleListener SSLEngine=on / Listener className=org.apache.catalina.core.JasperListener / Listener className=org.apache.catalina.core.JreMemoryLeakPreventionListener / Listener className=org.apache.catalina.mbeans.ServerLifecycleListener / Listener className=org.apache.catalina.mbeans.GlobalResourcesLifecycleListener / GlobalNamingResources Resource name=UserDatabase auth=Container type=org.apache.catalina.UserDatabase description=User database that can be updated and saved factory=org.apache.catalina.users.MemoryUserDatabaseFactory pathname=conf/tomcat-users.xml / /GlobalNamingResources Service name=Catalina Connector port=8080 protocol=HTTP/1.1 connectionTimeout=4 redirectPort=8443 / Connector port=8009 protocol=AJP/1.3 redirectPort=8443 / Engine name=Catalina defaultHost=localhost jvmRoute=worker1 Realm className=org.apache.catalina.realm.UserDatabaseRealm resourceName=UserDatabase/ Host name=localhost appBase=webapps unpackWARs=true autoDeploy=true xmlValidation=false xmlNamespaceAware=false /Host /Engine /Service /Server [Answers own previous question] Weirdness. In addition to the previous relevant questions: Are both of the two Tomcat instances actually working? If the shutdown port is identical on both (8005), then I think the second one won't start up. The connectors certainly won't. Also, if you're using mod_jk 1.2.30 there's an example config in the install dir which uses the template config method and has good default settings. p 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: Error 503 ocurring when server under load
Thanks to all for their replies to date. Much appreciated, I'll respond to some of the points raised. Apologies if I'm asking basic stuff but still getting my ahead around Tomcat and Apache integration. On 7 October 2010 18:31, André Warnier wrote: Your configuration looks very clean to me (no unnecessary settings etc.), which in this case is a plus (a good base to start tuning). Thank you You may want to upgrade Tomcat to the latest version (6.0.29). Noted For example, at the moment these errors happen, what are these Tomcats really doing ? Are they really busy each processing 200 requests, with 200 threads running and actually doing something ? (200 is the default for the maxThreads attribute of the AJP Connector). Unfortunately I'm not in a position to conduct realtime monitoring of the server and Tomcat. Is there a way to log this thread data? If yes, then you may just need a leaner application, or a bigger system (more RAM, faster CPU), or more systems. What does the Task Manager tell you about the total system load ? Again, unfortunately not in a position to monitor in real time. If not, and many of these threads are waiting, then you may have an issue with a keepAlive that is too long. See http://tomcat.apache.org/tomcat-6.0-doc/config/ajp.html, for the connectionTimeout and keepAliveTimeout attributes. Will do Whatever you do, first get an idea of the starting situation. Then modify one setting at a time, and observe (and note) the effects. Sensible advice Pid wrote: OK. Is there anything else different apart from the Servlet container? I'm not sure what you mean? Why do you have two Tomcat instances? (It's not a trick question, I'm interested in your reasoning.) Load balancing. I thought/guessed that two instances would be sufficent. Perhaps this was incorrect and I need more? Was there an attachment I didn't see? No, all details for .conf files and server.xml was posted inline. How many threads server instances did you have before? What were your Java -Xmx etc settings before, and now? Are you using an Executor? I'll have to check, but I think it's safe to say that they are set to the default values as I don't recall changing anything relating to them. Thread dumps from a maxed out Tomcat will tell you what each Thread is waiting for. Collect a series of these during high load periods to find out what's happening. I'll have to figure out how to do this, but sounds a good idea. Christopher Schultz wrote I think you might want to compare the request processor counts you have for Tomcat versus mod_jk: the default maxThreads for a Tomcat Connector is 200, while your MaxClients settings might be different on your Apache httpd configuration. Excuse my ignorance, but can you clarify where I check these settings? I believe that only the Worker MPM is available on Microsoft Windows. Have you explicitly configured an MPM? Since I don't know what an MPM is I'm going to say no. Since you're using Apache 2.x, mod_jk should be able to auto-configure this value for you. Do I need to set anything to allow this auto config to happen? From my reading, you may be limiting yourself to 10 simultaneous connections from Apache httpd to Tomcat, and others will get a 503. What lead you to that conclusion? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
