Still Failing: apache/httpd#2047 (trunk - bf7b392)
Build Update for apache/httpd - Build: #2047 Status: Still Failing Duration: 12 mins and 23 secs Commit: bf7b392 (trunk) Author: Stefan Eissing Message: * mod_http2: fixing some compiler warnings. length of output written now correctly calculated after buckets have been read. test cases updated. git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1894172 13f79535-47bb-0310-9956-ffa450edef68 View the changeset: https://github.com/apache/httpd/compare/59b7c104ce06...bf7b392b259f View the full build log and details: https://app.travis-ci.com/github/apache/httpd/builds/239678574?utm_medium=notification_source=email -- You can unsubscribe from build emails from the apache/httpd repository going to https://app.travis-ci.com/account/preferences/unsubscribe?repository=16806660_medium=notification_source=email. Or unsubscribe from *all* email updating your settings at https://app.travis-ci.com/account/preferences/unsubscribe?utm_medium=notification_source=email. Or configure specific recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications.
Failed: apache/httpd#2046 (trunk - 59b7c10)
Build Update for apache/httpd - Build: #2046 Status: Failed Duration: 11 mins and 46 secs Commit: 59b7c10 (trunk) Author: Yann Ylavic Message: *) core: Be safe with ap_lingering_close() called with a socket NULL-ed. PR 65627. mod_itk seems to: ap_set_core_module_config(c->conn_config, NULL) before calling ap_lingering_close(), causing a crash after r1891721. Until we have an API to no-op ap_lingering_close(), let's be safe. * server/connection.c(ap_start_lingering_close): The socket should not be NULL here, add an assertion. * server/connection.c(ap_lingering_close): Set c->aborted if the socket is NULL, and give up. Submitted by: acmondor , ylavic git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1894171 13f79535-47bb-0310-9956-ffa450edef68 View the changeset: https://github.com/apache/httpd/compare/17471dfb938c...59b7c104ce06 View the full build log and details: https://app.travis-ci.com/github/apache/httpd/builds/239664908?utm_medium=notification_source=email -- You can unsubscribe from build emails from the apache/httpd repository going to https://app.travis-ci.com/account/preferences/unsubscribe?repository=16806660_medium=notification_source=email. Or unsubscribe from *all* email updating your settings at https://app.travis-ci.com/account/preferences/unsubscribe?utm_medium=notification_source=email. Or configure specific recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications.
Still Failing: apache/httpd#2045 (trunk - 17471df)
Build Update for apache/httpd - Build: #2045 Status: Still Failing Duration: 17 mins and 22 secs Commit: 17471df (trunk) Author: Stefan Eissing Message: updated log tag to resolve duplicate. git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1894169 13f79535-47bb-0310-9956-ffa450edef68 View the changeset: https://github.com/apache/httpd/compare/01288f399d68...17471dfb938c View the full build log and details: https://app.travis-ci.com/github/apache/httpd/builds/239663408?utm_medium=notification_source=email -- You can unsubscribe from build emails from the apache/httpd repository going to https://app.travis-ci.com/account/preferences/unsubscribe?repository=16806660_medium=notification_source=email. Or unsubscribe from *all* email updating your settings at https://app.travis-ci.com/account/preferences/unsubscribe?utm_medium=notification_source=email. Or configure specific recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications.
Errored: apache/httpd#2044 (trunk - 01288f3)
Build Update for apache/httpd - Build: #2044 Status: Errored Duration: 19 mins and 34 secs Commit: 01288f3 (trunk) Author: Stefan Eissing Message: more numbers git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1894168 13f79535-47bb-0310-9956-ffa450edef68 View the changeset: https://github.com/apache/httpd/compare/6a355db082d0...01288f399d68 View the full build log and details: https://app.travis-ci.com/github/apache/httpd/builds/239663376?utm_medium=notification_source=email -- You can unsubscribe from build emails from the apache/httpd repository going to https://app.travis-ci.com/account/preferences/unsubscribe?repository=16806660_medium=notification_source=email. Or unsubscribe from *all* email updating your settings at https://app.travis-ci.com/account/preferences/unsubscribe?utm_medium=notification_source=email. Or configure specific recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications.
Re: [Bug 65626] New: MPM Event doesn't shutdown idle children after working under high load
On 10/12/21 4:58 PM, Yann Ylavic wrote:
> On Mon, Oct 11, 2021 at 2:32 PM wrote:
>>
>> https://bz.apache.org/bugzilla/show_bug.cgi?id=65626
>>
>> Bug ID: 65626
>>Summary: MPM Event doesn't shutdown idle children after working
>> under high load
>>Product: Apache httpd-2
>>Version: 2.4.37
>> Hardware: PC
>> Status: NEW
>> Severity: normal
>> Priority: P2
>> Component: mpm_event
>> Assignee: b...@httpd.apache.org
>> Reporter: gregvoro...@gmail.com
>> Target Milestone: ---
>>
>> After running ab with concurrency equal or more than MaxRequestWorkers for
>> some
>> time, apache doesn't kill idle processes any longer. Following is logged
>> every
>> second:
>>
>> [Mon Oct 11 15:17:29.416964 2021] [mpm_event:trace5] [pid 71:tid
>> 140381265582400] event.c(2834): Not shutting down child: total daemons 16 /
>> active limit 16 / ServerLimit 16
>
> At https://github.com/apache/httpd/blob/trunk/server/mpm/event/event.c#L3181
> we handle MaxSpareThreads like this:
>
> if (idle_thread_count > max_spare_threads / num_buckets)
> {
> /*
> * Child processes that we ask to shut down won't die immediately
> * but may stay around for a long time when they finish their
> * requests. If the server load changes many times, many such
> * gracefully finishing processes may accumulate, filling up the
> * scoreboard. To avoid running out of scoreboard entries, we
> * don't shut down more processes when the total number of processes
> * is high.
> * [...]
> */
> if (retained->total_daemons <= active_daemons_limit &&
> retained->total_daemons < server_limit) {
> /* Kill off one child */
> ap_mpm_podx_signal(retained->buckets[child_bucket].pod,
>AP_MPM_PODX_GRACEFUL);
> retained->idle_spawn_rate[child_bucket] = 1;
> } else {
> ap_log_error(APLOG_MARK, APLOG_TRACE5, 0, ap_server_conf,
> "Not shutting down child: total daemons %d / "
> [...]);
> }
> }
>
> with active_daemons_limit = MaxRequestWorkers / ThreadsPerChild (thus
> a constant), and retained->total_daemons which is
> incremented/decremented when a child forks/exits.
> So it seems that besides MaxRequestsPerChild, there is nothing to
> recover after active_daemons_limit was reached at one point in time.
Good catch, I think we need to have a conditions where we start shuting down
again.
>
> Shouldn't we also compute a busy_thread_count and kill children
> if/when later things settle down and thus busy_thread_count <
> max_spare_threads (like in the attached patch)?
Hm. Wouldn't it be better instead to ensure that idle_thread_count is "far"
larger than the min_spare_threads something like.
idle_thread_count > active_daemons_limit/2/num_buckets*ThreadsPerChild +
min_spare_threads
(2 is arbitrary choice could be also larger to start earlier reducing daemons)
This would prevent the situation that we have a dying daemon and a rise of
requests would cause the start of a new daemon "soon".
?
Regards
Rüdiger
Re: Ineract with travis? (was: Errored: apache/httpd#1888 (trunk - 47e6ece))
On Tue, Oct 12, 2021 at 11:31:51AM +0200, Yann Ylavic wrote: > Gavin (infra) gave all the httpd-committers team the "triage" access > right on github, which follows on travis. > This also includes managing PRs on github, great! Ah, brilliant. Thank you Yann and Gavin. Regards, Joe
Re: [Bug 65626] New: MPM Event doesn't shutdown idle children after working under high load
On Mon, Oct 11, 2021 at 2:32 PM wrote:
>
> https://bz.apache.org/bugzilla/show_bug.cgi?id=65626
>
> Bug ID: 65626
>Summary: MPM Event doesn't shutdown idle children after working
> under high load
>Product: Apache httpd-2
>Version: 2.4.37
> Hardware: PC
> Status: NEW
> Severity: normal
> Priority: P2
> Component: mpm_event
> Assignee: b...@httpd.apache.org
> Reporter: gregvoro...@gmail.com
> Target Milestone: ---
>
> After running ab with concurrency equal or more than MaxRequestWorkers for
> some
> time, apache doesn't kill idle processes any longer. Following is logged every
> second:
>
> [Mon Oct 11 15:17:29.416964 2021] [mpm_event:trace5] [pid 71:tid
> 140381265582400] event.c(2834): Not shutting down child: total daemons 16 /
> active limit 16 / ServerLimit 16
At https://github.com/apache/httpd/blob/trunk/server/mpm/event/event.c#L3181
we handle MaxSpareThreads like this:
if (idle_thread_count > max_spare_threads / num_buckets)
{
/*
* Child processes that we ask to shut down won't die immediately
* but may stay around for a long time when they finish their
* requests. If the server load changes many times, many such
* gracefully finishing processes may accumulate, filling up the
* scoreboard. To avoid running out of scoreboard entries, we
* don't shut down more processes when the total number of processes
* is high.
* [...]
*/
if (retained->total_daemons <= active_daemons_limit &&
retained->total_daemons < server_limit) {
/* Kill off one child */
ap_mpm_podx_signal(retained->buckets[child_bucket].pod,
AP_MPM_PODX_GRACEFUL);
retained->idle_spawn_rate[child_bucket] = 1;
} else {
ap_log_error(APLOG_MARK, APLOG_TRACE5, 0, ap_server_conf,
"Not shutting down child: total daemons %d / "
[...]);
}
}
with active_daemons_limit = MaxRequestWorkers / ThreadsPerChild (thus
a constant), and retained->total_daemons which is
incremented/decremented when a child forks/exits.
So it seems that besides MaxRequestsPerChild, there is nothing to
recover after active_daemons_limit was reached at one point in time.
Shouldn't we also compute a busy_thread_count and kill children
if/when later things settle down and thus busy_thread_count <
max_spare_threads (like in the attached patch)?
Regards;
Yann.
Index: server/mpm/event/event.c
===
--- server/mpm/event/event.c (revision 1894074)
+++ server/mpm/event/event.c (working copy)
@@ -3094,6 +3094,7 @@ static void perform_idle_server_maintenance(int ch
{
int i, j;
int idle_thread_count = 0;
+int busy_thread_count = 0;
worker_score *ws;
process_score *ps;
int free_length = 0;
@@ -3139,6 +3140,9 @@ static void perform_idle_server_maintenance(int ch
++idle_thread_count;
}
if (status >= SERVER_READY && status < SERVER_GRACEFUL) {
+if (status != SERVER_READY) {
+++busy_thread_count;
+}
++child_threads_active;
}
}
@@ -3187,7 +3191,7 @@ static void perform_idle_server_maintenance(int ch
* gracefully finishing processes may accumulate, filling up the
* scoreboard. To avoid running out of scoreboard entries, we
* don't shut down more processes when the total number of processes
- * is high.
+ * is high, until there are less than MaxSpareThreads busy threads.
*
* XXX It would be nice if we could
* XXX - kill processes without keepalive connections first
@@ -3195,8 +3199,9 @@ static void perform_idle_server_maintenance(int ch
* XXX depending on server load, later be able to resurrect them
* or kill them
*/
-if (retained->total_daemons <= active_daemons_limit &&
-retained->total_daemons < server_limit) {
+if (busy_thread_count < max_spare_threads / num_buckets
+|| (retained->total_daemons <= active_daemons_limit
+&& retained->total_daemons < server_limit)) {
/* Kill off one child */
ap_mpm_podx_signal(retained->buckets[child_bucket].pod,
AP_MPM_PODX_GRACEFUL);
@@ -3204,9 +3209,9 @@ static void perform_idle_server_maintenance(int ch
} else {
ap_log_error(APLOG_MARK, APLOG_TRACE5, 0, ap_server_conf,
"Not shutting down child: total daemons %d / "
- "active limit %d / ServerLimit %d",
+ "active limit %d /
Broken: apache/httpd#2043 (trunk - 6a355db)
Build Update for apache/httpd - Build: #2043 Status: Broken Duration: 19 mins and 33 secs Commit: 6a355db (trunk) Author: Stefan Eissing Message: *) mod_http2: - Fixed an issue since 1.15.24 that "Server" headers in proxied requests were overwritten instead of preserved. [PR by @daum3ns] - Added directove 'H2StreamTimeout' to configure a separate value for HTTP/2 streams, overriding server's 'Timeout' configuration. [rpluem] - HTTP/2 connections now use pollsets to monitor the status of the ongoing streams and their main connection when host OS allows this. - Removed work-arounds for older versions of libnghttp2 and checking during configure that at least version 1.15.0 is present. - The HTTP/2 connection state handler, based on an experiment and draft at the IETF http working group (abandoned for some time), has been removed. - H2SerializeHeaders no longer has an effect. A warning is logged when it is set to "on". The switch enabled the internal writing of requests to be parsed by the internal HTTP/1.1 protocol handler and was introduced to avoid potential incompatibilities during the introduction of HTTP/2. - Removed the abort/redo of tasks when mood swings lower the active limit. git-svn-id: https://svn.apache.org/repos/asf/httpd/httpd/trunk@1894163 13f79535-47bb-0310-9956-ffa450edef68 View the changeset: https://github.com/apache/httpd/compare/b1a8055deea4...6a355db082d0 View the full build log and details: https://app.travis-ci.com/github/apache/httpd/builds/239647985?utm_medium=notification_source=email -- You can unsubscribe from build emails from the apache/httpd repository going to https://app.travis-ci.com/account/preferences/unsubscribe?repository=16806660_medium=notification_source=email. Or unsubscribe from *all* email updating your settings at https://app.travis-ci.com/account/preferences/unsubscribe?utm_medium=notification_source=email. Or configure specific recipients for build notifications in your .travis.yml file. See https://docs.travis-ci.com/user/notifications.
Re: Ineract with travis? (was: Errored: apache/httpd#1888 (trunk - 47e6ece))
On Sat, Oct 9, 2021 at 6:12 PM Yann Ylavic wrote: > > On Fri, Oct 8, 2021 at 10:30 AM Joe Orton wrote: > > > > Did you work out how to do this? I never asked for special rights here > > AFAIK, I assumed everybody in the "httpd-committers" group could do it, > > but your github user is listed there too: > > > > https://github.com/orgs/apache/teams/httpd-committers/members > > Yeah but no :/ > I'll create an infra ticket to ask.. Gavin (infra) gave all the httpd-committers team the "triage" access right on github, which follows on travis. This also includes managing PRs on github, great! Cheers; Yann.
