https://bz.apache.org/bugzilla/show_bug.cgi?id=70308

            Bug ID: 70308
           Summary: mod_http2: configured receive window can collapse
                    during uploads, and adaptive growth can remain stuck
           Product: Apache httpd-2
           Version: 2.4.69
          Hardware: All
                OS: All
            Status: NEW
          Severity: normal
          Priority: P2
         Component: mod_http2
          Assignee: [email protected]
          Reporter: [email protected]
  Target Milestone: ---

I am seeing intermittent HTTP/2 upload throttling on Amazon Linux 2023 with
httpd-2.4.69-1.amzn2023.0.2.x86_64 and mod_http2-2.4.69-1.amzn2023.0.2.x86_64.
The running module identifies itself as mod_http2 2.0.42, with nghttp2 1.59.0.
The client is Chrome 156 on Windows, connecting directly to EC2 over HTTPS.
H2WindowSize is 4194304.

A 100 MiB upload takes about 3 seconds over HTTP/1.1, while ordinary HTTP/2
form submissions have taken 34–66 seconds. Increasing H2WindowSize from 1 MiB
to 4 or 8 MiB has not reliably prevented the slowdown. Apache mod_http2 trace
logs and targeted follow-ups were analyzed.

Basically, the Apache window size does not increase appropriately (even with
high values set in the Apache config) so the client sits waiting for the next
send window instead of continuously sending data.

An example excerpt from a session:
13:13:35.966938  INITIAL_WINDOW_SIZE=4194304
13:13:35.967107  POST /raw-upload-test.php starts
13:13:36.014412  consumed 8183 bytes, window now 49160/65535
13:13:36.158457  SETTINGS ACK received
13:13:36.158510  consumed 8227 bytes, window now 4169702/65535
13:13:50.238566  consumed 65500 bytes, window now 3252737/131071
13:14:17.612391  final consumption report, HALF_CLOSED_REMOTE

Chrome reported HTTP 200 in 42.03 seconds.

The internal adaptive target is stream->in_window_size. In the log above, when
mentioning "window now X/Y", X is available credit sampled before adjustment,
and Y is that cached target. The target remains 65,535 after SETTINGS
acknowledgement enlarges the effective window. A subsequent "growth" operation
produces 131,071 and supplies it to an absolute window setter. That is despite
having H2WindowSize set to 4194304 in the config.

For comparison, connection reuse via some Javascript executed in the Chrome
console seems to show what happens when the H2WindowSize is properly observed,
but is not a substitute for reliably observing the setting. Uploads following a
completed GET on the same HTTP/2 connection retained the 4 MiB target and
completed in 2.49–3.18 seconds. Connection reuse was verified in Chrome and
Apache logs. 

PR #710 (https://github.com/apache/httpd/pull/710) directly addresses the
collapse by enforcing H2WindowSize as a floor. With my existing 4 MiB setting,
it should prevent the failure I experienced. I have not tested the patch, but
suggest reviewing and merging that correctness fix first rather than delaying
it for the broader autotuning changes suggested below.

-------------------------------

Separately, the proposed 2:1 range in pull request #710 appear too restrictive
for adapting across different user connections. A 4:1 range such as 512 KiB–2
MiB, or preferably an 8:1 range of 256 KiB–2 MiB, seems worth evaluating. 

Calculating the BDP of connections ranging in RTT and bandwidth, a 2 MiB max
window covers the calculated buffer requirements for 30 Mbps connections with
RTT under 120ms, 100Mbps connections under 80ms RTT, and 300 Mbps RTTs under
20ms. This seems reasonable since the higher bandwidth connections generally
have lower RTT. Likewise, setting the value at 2 MiB seems unnecessarily high
for a basic/smaller upload. 

An independently configurable ceiling via another config variable may be
preferable to setting the 2:1 ratio in pull request #710 or the ratio suggested
here of 4:1 or 8:1.

-------------------------------

Finally, the single-report growth condition in the existing
h2_stream_in_consumed() rarely triggers. As mentioned in pull request #710, in
practice, the growth chunks are often much smaller than thigh. I too found the
growth condition rarely met as the data consumption rate is not necessarily
related to the data send rate.

https://blog.cloudflare.com/delivering-http-2-upload-speed-improvements/
CloudFlare discusses their autotuning work in nginx at that URL. Their work
provides a useful model: double the request-body buffer when estimated BDP
exceeds one quarter of the receive window. 

Their implementation measures actual TCP RTT and throughput. The diff below
uses a simpler heuristic instead that involves less code modification:
accumulate consumption across callbacks and evaluate data size over time. It
does not requiring platform-specific RTT handling.

The longer explanation: accumulate consumption across callbacks, evaluate after
40 ms, discard samples older than 200 ms, and double when accumulated
consumption exceeds one quarter of the target. It preserves the existing shrink
condition and uses an 8× ceiling.

The diff is against unmodified 2.4.69
(https://github.com/apache/httpd/blob/2.4.69/modules/http2/h2_stream.c) and
includes the floor logic from #710; it should not be applied on top of that PR
unchanged. 

It has not been compiled or run. For that reason, the timing constants could
use some real world evaluation.

diff --git a/modules/http2/h2_stream.h b/modules/http2/h2_stream.h
--- a/modules/http2/h2_stream.h
+++ b/modules/http2/h2_stream.h
@@ -94,2 +94,4 @@
     int in_window_size;
     apr_time_t in_last_write;
+    apr_off_t in_window_consumed;
+    apr_time_t in_window_check_at;
diff --git a/modules/http2/h2_stream.c b/modules/http2/h2_stream.c
--- a/modules/http2/h2_stream.c
+++ b/modules/http2/h2_stream.c
@@ -559,2 +559,4 @@
     stream->created      = apr_time_now();
+    stream->in_window_consumed = 0;
+    stream->in_window_check_at = stream->created;
     stream->state        = H2_SS_IDLE;
@@ -1264,30 +1266,40 @@
             int cur_size = nghttp2_session_get_stream_local_window_size(
                 session->ngh2, stream->id);
             int win = stream->in_window_size;
-            int thigh = win * 8/10;
-            int tlow = win * 2/10;
-            const int win_max = 2*1024*1024;
-            const int win_min = 32*1024;
-
-            /* Work in progress, probably should add directives for these
-             * values once this stabilizes somewhat. The general idea is
-             * to adapt stream window sizes if the input window changes
-             * a) very quickly (< good RTT) from full to empty
-             * b) only a little bit (> bad RTT)
-             * where in a) it grows and in b) it shrinks again.
-             */
-            if (cur_size > thigh && amount > thigh && win < win_max) {
-                /* almost empty again with one reported consumption, how
-                 * long did this take? */
-                long ms = apr_time_msec(apr_time_now() -
stream->in_last_write);
-                if (ms < 40) {
-                    win = H2MIN(win_max, win + (64*1024));
-                }
-            }
-            else if (cur_size < tlow && amount < tlow && win > win_min) {
-                /* staying full, for how long already? */
-                long ms = apr_time_msec(apr_time_now() -
stream->in_last_write);
+            const int win_min = h2_config_sgeti(session->s, H2_CONF_WIN_SIZE);
+            const int win_max = (win_min <= NGHTTP2_MAX_WINDOW_SIZE / 8)?
+                                win_min * 8 : NGHTTP2_MAX_WINDOW_SIZE;
+            apr_time_t now = apr_time_now();
+            apr_interval_time_t elapsed = now - stream->in_window_check_at;
+            int tlow;
+
+            /* Retain PR #710's floor before evaluating adaptation. */
+            if (win < win_min) {
+                win = win_min;
+            }
+            tlow = (int)(((apr_int64_t)win * 2) / 10);
+
+            /* Portable activity heuristic, not a measured BDP estimate.
+             * Combine callbacks over 40-200 ms; discard stale samples.
+             * Saturate accounting to keep this bounded on long uploads. */
+            stream->in_window_consumed += H2MIN(amount,
+                (apr_off_t)win_max - stream->in_window_consumed);
+            if (elapsed < 0 || elapsed >= apr_time_from_msec(40)) {
+                if (elapsed >= apr_time_from_msec(40)
+                    && elapsed <= apr_time_from_msec(200)
+                    && stream->in_window_consumed > win / 4
+                    && win < win_max) {
+                    win = (win > win_max / 2)? win_max : win * 2;
+                }
+                stream->in_window_consumed = 0;
+                stream->in_window_check_at = now;
+            }
+
+            /* Preserve shrinking, without undoing growth or the floor. */
+            if (win == stream->in_window_size
+                && cur_size < tlow && amount < tlow && win > win_min) {
+                long ms = apr_time_msec(now - stream->in_last_write);
                 if (ms > 700) {
                     win = H2MAX(win_min, win - (32*1024));
                 }
             }

-- 
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to