Dear All, We were runnig our site in PHP based CMS tool earlier, and normally 20-30K users will access our sites daily. But in new system with Tomcat, we are facing performance and availability issue frequently, when i access the tomcat url directly the page is loading within 3seconds, but if we access webServer URL then its taking more than 9seconds.
Also, Each day I am seeing more and more of these in my error_logs, and when the Total Children value is reached 999 the Apache is not responding and Server reboot only help to bring the site back. Every day atleast 4-5 times we are facing this issue (we are using mod_jk to connect with tomcat). Kindly please help on this. Usually I am seeing this on my error_log: [Sat Apr 15 20:49:33 2017] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 4 idle, and 31 total children [Sat Apr 15 20:51:14 2017] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 0 idle, and 20 total children [Sat Apr 15 20:51:15 2017] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 16 children, there are 0 idle, and 28 total children [Sat Apr 15 20:51:16 2017] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 32 children, there are 0 idle, and 44 total children We are using two Apache Nodes and Connected with Two Tomcat (at Application Level Clustering). Apache Servers: 4 Core 64-bit, Rhel System running on 16GB RAM (Both Servers) Server version: Apache/2.2.21 (Unix) *httpd.conf* KeepAlive On Timeout 300 MaxKeepAliveRequests 100 KeepAliveTimeout 15 <IfModule prefork.c> StartServers 80 ServerLimit 3500 MaxClients 3500 MaxRequestsPerChild 0 </IfModule> *workers.properties* worker.list=loadbalancer,status worker.qu_prod_live_svr.type=ajp13 worker.qu_prod_live_svr.host=cmsp1 worker.qu_prod_live_svr.port=8009 worker.qu_prod_live_svr.socket_keepalive=1 worker.qu_prod_live_svr.socket_timeout=300 worker.qu_prod_live_svr1.type=ajp13 worker.qu_prod_live_svr1.host=cmsp2 worker.qu_prod_live_svr1.port=8009 worker.qu_prod_live_svr1.socket_keepalive=1 worker.qu_prod_live_svr1.socket_timeout=300 worker.qu_prod_live_svr.lbfactor=1 worker.qu_prod_live_svr1.lbfactor=1 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=qu_prod_live_svr,qu_prod_live_svr1 worker.status.type=status *Tomcat Servers:* 4 Core 64-bit, Rhel System running on 16GB RAM (Both Servers) Server version: Apache Tomcat/7.0.42 <Connector port="9090" protocol="HTTP/1.1" redirectPort="8443" URIEncoding="UTF-8" emptySessionPath="true" maxThreads="500" minSpareThreads="10" connectionTimeout="-1" /> <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" URIEncoding="UTF-8" /> *error_log:* [Sat Apr 15 21:52:36 2017] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 32 children, there are 0 idle, and 839 total children [Sat Apr 15 21:52:37 2017] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 32 children, there are 0 idle, and 871 total children [Sat Apr 15 21:52:38 2017] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 32 children, there are 0 idle, and 903 total children [Sat Apr 15 21:52:39 2017] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 32 children, there are 0 idle, and 935 total children [Sat Apr 15 21:52:40 2017] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 32 children, there are 0 idle, and 967 total children [Sat Apr 15 21:52:41 2017] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 32 children, there are 0 idle, and 999 total children [Sat Apr 15 21:52:41 2017] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2 [Sat Apr 15 21:52:41 2017] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2 [Sat Apr 15 21:52:41 2017] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2 [Sat Apr 15 21:52:41 2017] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2 [Sat Apr 15 21:52:41 2017] [alert] Child 9351 returned a Fatal error... Apache is exiting! [Sat Apr 15 21:52:41 2017] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2 [Sat Apr 15 21:52:41 2017] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2 [Sat Apr 15 21:52:41 2017] [alert] (11)Resource temporarily unavailable: setuid: unable to change to uid: 2 [Sat Apr 15 21:53:06 2017] [error] (22)Invalid argument: apr_global_mutex_lock(jk_log_lock) failed [Sat Apr 15 21:53:06 2017] [error] mod_jk: jk_log_to_file [Sat Apr 15 21:53:06 2017][8752:4177577728] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1150): (qu_prod_live_svr1) can't receive the response header message from tomcat, network problems or tomcat (10.11.11.32:8009) is down (errno=104)\n failed: Broken pipe [Sat Apr 15 21:53:06 2017] [error] (22)Invalid argument: apr_global_mutex_unlock(jk_log_lock) failed [Sat Apr 15 21:53:06 2017] [error] (22)Invalid argument: apr_global_mutex_lock(jk_log_lock) failed [Sat Apr 15 21:53:06 2017] [error] mod_jk: jk_log_to_file [Sat Apr 15 21:53:06 2017][8752:4177577728] [error] ajp_get_reply::jk_ajp_common.c (1962): (qu_prod_live_svr1) Tomcat is down or refused connection. No response has been sent to the client (yet)\n failed: Broken pipe [Sat Apr 15 21:53:06 2017] [error] (22)Invalid argument: apr_global_mutex_unlock(jk_log_lock) failed *Thanks & Regards,* *Jay*
