On Mon, Oct 11, 2021 at 2:32 PM <bugzi...@apache.org> wrote: > > https://bz.apache.org/bugzilla/show_bug.cgi?id=65626 > > Bug ID: 65626 > Summary: MPM Event doesn't shutdown idle children after working > under high load > Product: Apache httpd-2 > Version: 2.4.37 > Hardware: PC > Status: NEW > Severity: normal > Priority: P2 > Component: mpm_event > Assignee: b...@httpd.apache.org > Reporter: gregvoro...@gmail.com > Target Milestone: --- > > After running ab with concurrency equal or more than MaxRequestWorkers for > some > time, apache doesn't kill idle processes any longer. Following is logged every > second: > > [Mon Oct 11 15:17:29.416964 2021] [mpm_event:trace5] [pid 71:tid > 140381265582400] event.c(2834): Not shutting down child: total daemons 16 / > active limit 16 / ServerLimit 16
At https://github.com/apache/httpd/blob/trunk/server/mpm/event/event.c#L3181 we handle MaxSpareThreads like this: if (idle_thread_count > max_spare_threads / num_buckets) { /* * Child processes that we ask to shut down won't die immediately * but may stay around for a long time when they finish their * requests. If the server load changes many times, many such * gracefully finishing processes may accumulate, filling up the * scoreboard. To avoid running out of scoreboard entries, we * don't shut down more processes when the total number of processes * is high. * [...] */ if (retained->total_daemons <= active_daemons_limit && retained->total_daemons < server_limit) { /* Kill off one child */ ap_mpm_podx_signal(retained->buckets[child_bucket].pod, AP_MPM_PODX_GRACEFUL); retained->idle_spawn_rate[child_bucket] = 1; } else { ap_log_error(APLOG_MARK, APLOG_TRACE5, 0, ap_server_conf, "Not shutting down child: total daemons %d / " [...]); } } with active_daemons_limit = MaxRequestWorkers / ThreadsPerChild (thus a constant), and retained->total_daemons which is incremented/decremented when a child forks/exits. So it seems that besides MaxRequestsPerChild, there is nothing to recover after active_daemons_limit was reached at one point in time. Shouldn't we also compute a busy_thread_count and kill children if/when later things settle down and thus busy_thread_count < max_spare_threads (like in the attached patch)? Regards; Yann.
Index: server/mpm/event/event.c =================================================================== --- server/mpm/event/event.c (revision 1894074) +++ server/mpm/event/event.c (working copy) @@ -3094,6 +3094,7 @@ static void perform_idle_server_maintenance(int ch { int i, j; int idle_thread_count = 0; + int busy_thread_count = 0; worker_score *ws; process_score *ps; int free_length = 0; @@ -3139,6 +3140,9 @@ static void perform_idle_server_maintenance(int ch ++idle_thread_count; } if (status >= SERVER_READY && status < SERVER_GRACEFUL) { + if (status != SERVER_READY) { + ++busy_thread_count; + } ++child_threads_active; } } @@ -3187,7 +3191,7 @@ static void perform_idle_server_maintenance(int ch * gracefully finishing processes may accumulate, filling up the * scoreboard. To avoid running out of scoreboard entries, we * don't shut down more processes when the total number of processes - * is high. + * is high, until there are less than MaxSpareThreads busy threads. * * XXX It would be nice if we could * XXX - kill processes without keepalive connections first @@ -3195,8 +3199,9 @@ static void perform_idle_server_maintenance(int ch * XXX depending on server load, later be able to resurrect them * or kill them */ - if (retained->total_daemons <= active_daemons_limit && - retained->total_daemons < server_limit) { + if (busy_thread_count < max_spare_threads / num_buckets + || (retained->total_daemons <= active_daemons_limit + && retained->total_daemons < server_limit)) { /* Kill off one child */ ap_mpm_podx_signal(retained->buckets[child_bucket].pod, AP_MPM_PODX_GRACEFUL); @@ -3204,9 +3209,9 @@ static void perform_idle_server_maintenance(int ch } else { ap_log_error(APLOG_MARK, APLOG_TRACE5, 0, ap_server_conf, "Not shutting down child: total daemons %d / " - "active limit %d / ServerLimit %d", + "active limit %d / ServerLimit %d / busy threads %d", retained->total_daemons, active_daemons_limit, - server_limit); + server_limit, busy_thread_count); } } else if (idle_thread_count < min_spare_threads / num_buckets) {