https://bz.apache.org/bugzilla/show_bug.cgi?id=53555
--- Comment #57 from Thomas Jarosch <[email protected]> --- Hi Stefan, thanks for trying to solve the "scoreboard full" issue :) I've been hit by it badly today, the affected machine is a forward proxy and stalls the traffic almost completely. Some background info: - event mpm on httpd 2.4.23 - forward proxy setup via mod_proxy - 280 real users + other machines. ~370 clients - server load is around 0.2, plenty of free RAM - file descriptor limit is 1024 - logrotate sends a graceful restart every hour If the problem occurs, httpd doesn't even respond to the /server-status page reliably. A small script logs the /server-status page every 30s to disk. Specific case: logrotate sends a "graceful restart" at 13h. /server-status output at 13:04:24h: ------------------- Total accesses: 8801 - Total Traffic: 74.6 MB 75 requests currently being processed, 125 idle workers +---------------------------------------------------------------------------+ | | Connections | Threads | Async connections | | PID |-------------------+-------------+---------------------------------| | | total | accepting | busy | idle | writing | keep-alive | closing || |-------+-------+-----------+------+------+---------+------------+---------|| | 14906 | 7 | yes | 6 | 44 | 0 | 1 | 0 || |-------+-------+-----------+------+------+---------+------------+---------|| | 14959 | 9 | yes | 9 | 41 | 0 | 0 | 0 || |-------+-------+-----------+------+------+---------+------------+---------|| | 15014 | 3 | no | 0 | 0 | 0 | 0 | 0 || |-------+-------+-----------+------+------+---------+------------+---------|| | 15015 | 49 | yes | 50 | 0 | 0 | 0 | 0 || |-------+-------+-----------+------+------+---------+------------+---------|| | 15329 | 3 | no | 0 | 0 | 0 | 0 | 0 || |-------+-------+-----------+------+------+---------+------------+---------|| | 15893 | 15 | no | 0 | 0 | 0 | 0 | 0 || |-------+-------+-----------+------+------+---------+------------+---------|| | 17762 | 11 | yes | 10 | 40 | 0 | 1 | 0 || |-------+-------+-----------+------+------+---------+------------+---------|| | Sum | 97 | | 75 | 125 | 0 | 2 | 0 || +---------------------------------------------------------------------------+ _________R_____R__________________R___R___R__R________R______R_R R_____R__R_________________R__R____RGGGGGGGGGGGGGGGGGGGGGGGGGGGG GGGGGGGGGGGGGGGGGGGGGGRRRRRRRRRRRRRRRRRRRRRRRRRRRWRRRRRRRRRRRRRR RRRRRRRRGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGR__________R__R_____ _______R_RR_________R_RR_R____ ------------------- /server-status output at 13:15:25h: ------------------- Total accesses: 12929 - Total Traffic: 90.9 MB 87 requests currently being processed, 63 idle workers +---------------------------------------------------------------------------+ | | Connections | Threads | Async connections | | PID |-------------------+-------------+---------------------------------| | | total | accepting | busy | idle | writing | keep-alive | closing || |-------+-------+-----------+------+------+---------+------------+---------|| | 14906 | 18 | yes | 16 | 34 | 0 | 2 | 0 || |-------+-------+-----------+------+------+---------+------------+---------|| | 14959 | 27 | yes | 26 | 24 | 0 | 2 | 0 || |-------+-------+-----------+------+------+---------+------------+---------|| | 15014 | 2 | no | 0 | 0 | 0 | 0 | 0 || |-------+-------+-----------+------+------+---------+------------+---------|| | 15015 | 2 | no | 0 | 0 | 0 | 0 | 0 || |-------+-------+-----------+------+------+---------+------------+---------|| | 15329 | 2 | no | 0 | 0 | 0 | 0 | 0 || |-------+-------+-----------+------+------+---------+------------+---------|| | 18564 | 45 | yes | 45 | 5 | 0 | 0 | 0 || |-------+-------+-----------+------+------+---------+------------+---------|| | 17762 | 39 | no | 0 | 0 | 0 | 0 | 0 || |-------+-------+-----------+------+------+---------+------------+---------|| | 18078 | 44 | no | 0 | 0 | 0 | 0 | 0 || |-------+-------+-----------+------+------+---------+------------+---------|| | Sum | 179 | | 87 | 63 | 0 | 4 | 0 || +---------------------------------------------------------------------------+ _____R__R___R_RR_RR_R_RR__R_____R_R___R_R_____R___W_RR__RR_RR__R RR__R_RR____RRRRR_R_RR___R_RR_RR____GGGGGGGGGGGGGGGGGGGGGGGGGGGG GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGRRRRRR RRRRRRRRR_RRRRRRRRR_RRRR_RRRRRRRRRRR_R_RRRRRGGGGGGGGGGGGGGGGGGGG GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG GGGGGGGGGGGGGGGG ------------------- /server-status at 13:25:20h: (httpd hardly responding anymore): ------------------- Total accesses: 14630 - Total Traffic: 97.4 MB 50 requests currently being processed, 0 idle workers +---------------------------------------------------------------------------+ | | Connections | Threads | Async connections | | PID |-------------------+-------------+---------------------------------| | | total | accepting | busy | idle | writing | keep-alive | closing || |-------+-------+-----------+------+------+---------+------------+---------|| | 14906 | 36 | no | 0 | 0 | 0 | 0 | 0 || |-------+-------+-----------+------+------+---------+------------+---------|| | 14959 | 2 | yes | 0 | 0 | 0 | 0 | 0 || |-------+-------+-----------+------+------+---------+------------+---------|| | 15014 | 2 | no | 0 | 0 | 0 | 0 | 0 || |-------+-------+-----------+------+------+---------+------------+---------|| | 15015 | 2 | no | 0 | 0 | 0 | 0 | 0 || |-------+-------+-----------+------+------+---------+------------+---------|| | 15329 | 2 | no | 0 | 0 | 0 | 0 | 0 || |-------+-------+-----------+------+------+---------+------------+---------|| | 18564 | 50 | yes | 50 | 0 | 0 | 1 | 0 || |-------+-------+-----------+------+------+---------+------------+---------|| | 17762 | 3 | no | 0 | 0 | 0 | 0 | 0 || |-------+-------+-----------+------+------+---------+------------+---------|| | 18078 | 1 | no | 0 | 0 | 0 | 0 | 0 || |-------+-------+-----------+------+------+---------+------------+---------|| | Sum | 98 | | 50 | 0 | 0 | 1 | 0 || +---------------------------------------------------------------------------+ GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGWRRRRR RRRRWRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRRGGGGGGGGGGGGGGGGGGGG GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG GGGGGGGGGGGGGGGG ------------------- I can provide more /server-status output if needed. After around 30 mins, the external "mon" watchdog kills httpd and restarts it. Traffic continues to flow. httpd config: ------------------- Timeout 300 KeepAliveTimeout 300 <IfModule mpm_event_module> # Number of concurrent connections is: ServerLimit * ThreadsPerChild # Result: 16 * 50 -> 800 # StartServers 1 ServerLimit 16 ThreadLimit 50 ThreadsPerChild 50 MaxConnectionsPerChild 1000 </IfModule> No other performance related settings. ------------------- I've now increased ServerLimit to 32 and disabled logrotate as a quick fix. It holds so far. Occasionally I still see the "scoreboard full" message, even though there are just ~160 active connections and some processes are (still?) in the graceful shutdown state. I'll put the patch from #55 on the productive machine tomorrow :o) It already runs on my own proxy and the one from my department. Anything else to watch out for? I can provide gdb backtraces if you tell me to look for something specific, too. Triggering a graceful restart during peak traffic might be a good test... Cheers, Thomas -- You are receiving this mail because: You are the assignee for the bug. --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
