Re: mod_jk fails to forward request on high traffic
Thanks guys, that seems to solve the problem. I also tried Mark's DisableReuse and it works fine, but I reverted since the timeouts solution works well and looks better. Another resource I used to get a better understanding of the issue was http://kbase.redhat.com/faq/docs/DOC-15866 Thanks again Ioannis André Warnier wrote: John Cherouvim wrote: ... Is there a possibility that there is a problem with the fact that I use: IfModule mpm_prefork_module ... ServerLimit1024 MaxClients 1024 and default Connector settings? Connector port=8010 protocol=AJP/1.3 redirectPort=8443 address=127.0.0.1 / Do those 2 need to have the same thread count (MaxClients=maxThreads)? I think you're on the right track there. See the previous response from Mark : ... My guess is that all of your Tomcat AJP threads are tied up with idle httpd threads. .. doing nothing. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- Ioannis Cherouvim Software Engineer mail: j...@eworx.gr web: www.eworx.gr - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
mod_jk fails to forward request on high traffic
Hello I have a website which during peak time (peak lasts around ~4 hours with 14 pageviews/sec, 140 http requests/sec) starts to drop pageview requests. My setup is: - SUSE Linux Enterprise Server 10 (x86_64) with 8 cpus, 16GB ram - 1 instance of apache 2.2.10 with mpm prefork - mod_jk/1.2.28 - 1 instance of tomcat 6.0.20 the O/S, tomcat and mysql seem to be completely underutilized: - cpu is at 2% to 4% - tomcat page render time is very low, around 50-200ms - iostat shows minimal disk utilization - lots of free disk space even when apache is not saturated, 1 out of ~6 requests (to be routed to tomcat) will never return (or will return after 5-10 seconds). mod_status at that point shows me the following which means that it can handle more requests: KK.K_CKK___KC_KK.__C__K___K__._CCKKKC_K__KK_KW_KKKW..KC_ KKK_KCK_KCC_.___W_CC__CC__KC.CK_K.CKK___KK_C_CKKK_CCKK._ K___KW_CK_K.CK_.CC.C..KCRW.._KC.._.KK..K...K_.._..K..K.K .KC._W.C_.K_CC._.K__K..KC_._K...K.K._..._..._._.._CKK_.. _.CKK_K_.KK...R. All requests to static content (not served via tomcat) have no problem. my apache config is: KeepAlive On MaxKeepAliveRequests 1024 KeepAliveTimeout 5 JkLogFile logs/mod_jk.log JkShmFile logs/jk-runtime-status JkLogLevelinfo JkWorkerProperty worker.list=tc1 JkWorkerProperty worker.tc1.type=ajp13 JkWorkerProperty worker.tc1.host=localhost JkWorkerProperty worker.tc1.port=8010 VirtualHost *:80 ServerName www.foobar.com DocumentRoot ... JkMount /* tc1 JkUnMount /*.css tc1 JkUnMount /*.js tc1 ... JkUnMount /*.jpg tc1 /VirtualHost IfModule mpm_prefork_module StartServers 5 MinSpareServers 5 MaxSpareServers 10 ServerLimit1024 MaxClients 1024 MaxRequestsPerChild 10 /IfModule my tomcat config is very plain: Service name=Catalina Connector port=8010 protocol=AJP/1.3 redirectPort=8443 address=127.0.0.1 / Engine name=Catalina defaultHost=www.foobar.com Host name=www.foobar.com appBase=webapps/foobar / /Engine /Service After I stop/start apache I start receiving these logs many times on mod_jk.log: [5449:902382544] [info] ajp_process_callback::jk_ajp_common.c (1748): Writing to client aborted or client network problems [5449:902382544] [info] ajp_service::jk_ajp_common.c (2407): (tc1) sending request to tomcat failed (unrecoverable), because of client write error (attempt=1) [5449:902382544] [info] jk_handler::mod_jk.c (2469): Aborting connection for worker=tc1 Exactly 200 seconds after the restart (confirmed it 10 times) these start appearing as well: [5761:902382544] [info] jk_open_socket::jk_connect.c (593): connect to 127.0.0.1:8010 failed (errno=110) [5761:902382544] [info] ajp_connect_to_endpoint::jk_ajp_common.c (922): Failed opening socket to (127.0.0.1:8010) (errno=110) [5761:902382544] [error] ajp_send_request::jk_ajp_common.c (1467): (tc1) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=110) These logs may be unrelated to the slow or dropped responses from mod_jk. On tomcat catalina.out I'm getting lots of: org.apache.jk.common.ChannelSocket processConnection WARNING: processCallbacks status 2 Any ideas on where should I look at? Will the mod_jk status worker give me any insights on this? Is there any possibility that mod_jk is getting blocked at the O/S level (maybe from iptables)? Also, I'm not sure if I can successfully reproduce the whole situation locally with JMeter since I don't have these many hosts to hit a staging server with. All I'm left with is wait each day for the peak time and fiddle around with the settings of the production server. thanks a lot Ioannis - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk fails to forward request on high traffic
John Cherouvim wrote: Hello I have a website which during peak time (peak lasts around ~4 hours with 14 pageviews/sec, 140 http requests/sec) starts to drop pageview requests. My guess is that all of your Tomcat AJP threads are tied up with idle httpd threads. Try the following in your httpd.conf: JkOptions +DisableReuse Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk fails to forward request on high traffic
On 08.09.2009 17:49, Mark Thomas wrote: John Cherouvim wrote: Hello I have a website which during peak time (peak lasts around ~4 hours with 14 pageviews/sec, 140 http requests/sec) starts to drop pageview requests. My guess is that all of your Tomcat AJP threads are tied up with idle httpd threads. Try the following in your httpd.conf: JkOptions +DisableReuse ... or read about the necessary timeouts on http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk fails to forward request on high traffic
Thanks for your reply. I'll try DisableReuse on the next peak, although I read in http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html that: For this you set JkOptions +DisableReuse in your Apache httpd configuration. This will have a huge negative performance impact! Is there a possibility that there is a problem with the fact that I use: IfModule mpm_prefork_module ... ServerLimit1024 MaxClients 1024 and default Connector settings? Connector port=8010 protocol=AJP/1.3 redirectPort=8443 address=127.0.0.1 / Do those 2 need to have the same thread count (MaxClients=maxThreads)? thanks Ioannis Mark Thomas wrote: John Cherouvim wrote: Hello I have a website which during peak time (peak lasts around ~4 hours with 14 pageviews/sec, 140 http requests/sec) starts to drop pageview requests. My guess is that all of your Tomcat AJP threads are tied up with idle httpd threads. Try the following in your httpd.conf: JkOptions +DisableReuse Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk fails to forward request on high traffic
John Cherouvim wrote: Thanks for your reply. I'll try DisableReuse on the next peak, although I read in http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html that: For this you set JkOptions +DisableReuse in your Apache httpd configuration. This will have a huge negative performance impact! Is there a possibility that there is a problem with the fact that I use: IfModule mpm_prefork_module ... ServerLimit1024 MaxClients 1024 and default Connector settings? Connector port=8010 protocol=AJP/1.3 redirectPort=8443 address=127.0.0.1 / Do those 2 need to have the same thread count (MaxClients=maxThreads)? There is no one right answer. The best way to do this will vary from case to case. DisableReuse is usually easier to set up than getting the timeouts and thread pool sizes correct. As long as httpd and Tomcat are on a fast local network, performance is very rarely an issue. Mark thanks Ioannis Mark Thomas wrote: John Cherouvim wrote: Hello I have a website which during peak time (peak lasts around ~4 hours with 14 pageviews/sec, 140 http requests/sec) starts to drop pageview requests. My guess is that all of your Tomcat AJP threads are tied up with idle httpd threads. Try the following in your httpd.conf: JkOptions +DisableReuse Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk fails to forward request on high traffic
Rainer Jung wrote: On 08.09.2009 17:49, Mark Thomas wrote: John Cherouvim wrote: Hello I have a website which during peak time (peak lasts around ~4 hours with 14 pageviews/sec, 140 http requests/sec) starts to drop pageview requests. My guess is that all of your Tomcat AJP threads are tied up with idle httpd threads. Try the following in your httpd.conf: JkOptions +DisableReuse ... or read about the necessary timeouts on http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html Or start a lot more threads on that AJP Connector, since you have a lot of spare everything on that machine. And if these are a lot of small requests by many separate clients, you may want to reduce your Apache KeepAliveTimeout to 4, then 3,.. The idea is : - a browser connects to Apache and issues a request - Apache is passing this request to one free Apache child - this child sees that the request is for mod_jk (and, in the background, Tomcat) - so one instance of an Apache child and mod_jk is now busy - mod_jk connects to Tomcat - to handle the request, the Tomcat Connector allocates one Tomcat thread (possibly starting a new one) - the thread works and returns the response to mod_jk - mod_jk returns the response to the Apache child - the Apache child returns the response to the browser but... since the connection is keep-alive, if the same browser does not, on the same connection, issue any more requests, then the Apache child anyway waits on that connection for the duration of the KeepAliveTimeout. Only when the timeout expires without new request, does the child close the connection and return itself to the available child pool. So if the request takes 100 milliseconds to fulfill, and your KeepAliveTimeout is 5000 milliseconds, your Apache children sit there doing nothing (and being unavailable for other browsers requests) for : (5000 ms + 100 ms) = 5100 ms = total time of which 100 ms = time to actually fulfill request thus 100 ms / 5100 ms = 0.0196.. = 1.9 % productive and 100 - 1.96 = 98.04 % unproductive during which time they probably hold on to their mod_jk connection to Tomcat also, and thus there is also a Tomcat thread waiting, and doing nothing for the same unproductive time, and not being available for other clients and requests. This is a horror scenario and, probably, in reality things are not so bad, depending on how your pages are and how smart all these little Apache/mod_jk/Tomcat pieces are. But maybe worth thinking about anyway ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: mod_jk fails to forward request on high traffic
John Cherouvim wrote: ... Is there a possibility that there is a problem with the fact that I use: IfModule mpm_prefork_module ... ServerLimit1024 MaxClients 1024 and default Connector settings? Connector port=8010 protocol=AJP/1.3 redirectPort=8443 address=127.0.0.1 / Do those 2 need to have the same thread count (MaxClients=maxThreads)? I think you're on the right track there. See the previous response from Mark : ... My guess is that all of your Tomcat AJP threads are tied up with idle httpd threads. .. doing nothing. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
