https://bz.apache.org/bugzilla/show_bug.cgi?id=52865
janani <[email protected]> changed: What |Removed |Added ---------------------------------------------------------------------------- Version|2.4.1 |2.4.43 --- Comment #8 from janani <[email protected]> --- Apache crash was first noticed in 2.4.41 .Error stack looked similar to the bug which was fixed in 2.4.43 but still the issue was seen in 2.4.43.We made system level memory setting changes even after that we could see that the issue was happening . # vi rc.local # (Add the lines below to the end. Replace eth0, eth1 with the actual names) # sysctl -w net.core.rmem_max=16777216 # sysctl -w net.core.wmem_max=16777216 # sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216" # sysctl -w net.ipv4.tcp_wmem="4096 65536 16777216" # sysctl -w net.ipv4.tcp_fin_timeout=10 # ifconfig eth0 txqueuelen 10000 # ifconfig eth2 txqueuelen 10000 # reboot 14) Configure new ulimit settings # pbrun su-root # cd /etc/security # vi limits.conf # (Add these lines to the end) # root hard nofile 16384 # root soft nofile 16384 # apache hard nofile 16384 # apache soft nofile 16384 # :wq # reboot We suspected that the load on the server is more which may be due increase in the number of application users . We increased the apache mpm values and since we are using worker settings which deals with increase load on the server with respect to the increased connection . /apps/hrs/tt/bin/ ./apachectl -V |grep MPM Server MPM: worker Server limit set was 300 .We are increasing the value to 600. Start servers from 200 to 500 and Max client to 15000 Minspare and Maxspare thread values are changed as per the recommendation(75 and 250) from the apache documentation. ServerLimit 600 StartServers 500 MaxClients 15000 MinSpareThreads 75 (recommended value) MaxSpareThreads 250 (recommended value) ThreadsPerChild 25 MaxRequestsPerChild 0 But the issue was still seen with below exception post implementing the change I was researching on this issue again and fortunately got the source code for apache . Please find the source code link for apache. people.apache.org/~igalic/checks/httpd/2012-09-14-1/report-69Y1He.html After the below exception Server gets hung and cannot serve any request. [Fri Aug 07 16:12:12.684173 2020] [mpm_event:debug] [pid 19580:tid 139740527363840] event.c(1810): Too many open connections (25), not accepting new conns in this process This exception is seen from the time we enabled debugs . [Fri Aug 07 16:12:12.889710 2020] [mpm_event:debug] [pid 19609:tid 139740820498176] event.c(2314): AH02471: start_threads: Using epoll (wakeable) [Fri Aug 07 16:12:12.889934 2020] [mpm_event:debug] [pid 19610:tid 139740820498176] event.c(2314): AH02471: start_threads: Using epoll (wakeable) Lines in the source code referring to the exception 1805 else if (connections_above_limit()) { 1806 disable_listensocks(); 1807 ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, ap_server_conf, 1808 "Too many open connections (%u), " 1809 "not accepting new conns in this process", 1810 apr_atomic_read32(&connection_count)); 1811 ap_log_error(APLOG_MARK, APLOG_TRACE1, 0, ap_server_conf, 1812 "Idle workers: %u", This %u refers to the threads per child value in mpm which we is the default recommended value but looks like that request on server is high where there is lack of this parameter . Increased the thread per child to 50 and made other parameter changes to match the value .but still the issue was seen . From ServerLimit 600 StartServers 500 MaxClients 15000 MinSpareThreads 75 MaxSpareThreads 250 ThreadsPerChild 25 New value : ServerLimit 300 StartServers 200 MaxClients 15000 MinSpareThreads 75 MaxSpareThreads 250 ThreadsPerChild 50 we were getting other exception post the changes as below AH00486: server seems busy, (you may need to increase StartServers, ThreadsPerChild or Min/MaxSpareThreads), spawning 8 children, there are around 54 idle threads, 6 active children, and 6 children that are shutting down. Made the threads per child to 40 and min and max spare s 25 and 75 respectively with the semaphore changes at the system level but still the issue is seen Changes made on the server level # 17) Increase semaphore limits # pbrun su-root # vi /etc/sysctl.conf # Add this line: # # Add additional semaphores for Channel Secure and mod_rewrite # kernel.sem = 4096 512000 1600 9000 # :wq # sysctl -p /etc/sysctl.conf What I notice is webagent fails to initialise all the time and the server loses the connection ( no connections hit from F5 or the server does not take u the connection which like very low count of user when netstat performed say 25 and then the apacheURL goes down for the individual apache server though apache process is up . -- You are receiving this mail because: You are the assignee for the bug. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
