https://bz.apache.org/bugzilla/show_bug.cgi?id=70257
Bug ID: 70257
Summary: mod_http2: GOAWAY INTERNAL_ERROR on whole connection
when client RST_STREAM races HEADERS send
(on_frame_not_send_cb)
Product: Apache httpd-2
Version: 2.4.58
Hardware: PC
OS: Linux
Status: NEW
Severity: normal
Priority: P2
Component: mod_http2
Assignee: [email protected]
Reporter: [email protected]
Target Milestone: ---
mod_http2: GOAWAY INTERNAL_ERROR on whole connection when client RST_STREAM
races HEADERS send (on_frame_not_send_cb)
Description (grote tekstbox):
mod_http2 2.0.42 (Apache 2.4.58); the same code is in current trunk.
SYMPTOM
When a browser navigates away while fetches are still in flight, it sends
RST_STREAM (CANCEL) for those streams. Rarely (about 3 in 1000 resets in our
tests) the server then sends GOAWAY INTERNAL_ERROR ("The user callback function
failed") and ALL other streams on that connection die. Chromium reports
net::ERR_HTTP2_PROTOCOL_ERROR for unrelated subresources (scripts, CSS,
beacons) of the page the user is moving to.
CAUSE
There is a window of roughly 20 microseconds between "submit response" and the
HEADERS frame actually being sent. If the client's RST_STREAM arrives in that
window, the stream is already cleaned up and nghttp2 reports
NGHTTP2_ERR_STREAM_CLOSED (-510) to on_frame_not_send_cb (h2_session.c). That
callback falls through to "return NGHTTP2_ERR_CALLBACK_FAILURE", which nghttp2
turns into a connection-level GOAWAY.
For a stream the peer has just reset, not sending the frame is the correct
outcome, so this should not be fatal for the session.
PROPOSED PATCH
--- modules/http2/h2_session.c.orig
+++ modules/http2/h2_session.c
@@ -652,6 +652,13 @@
h2_stream_rst(stream, NGHTTP2_PROTOCOL_ERROR);
return 0;
}
+ if (stream_id && ngh2_err == NGHTTP2_ERR_STREAM_CLOSED) {
+ /* the client reset the stream (RST_STREAM) between submitting the
+ * response and sending its HEADERS; the stream is already cleaned
+ * up. Failing the callback here would GOAWAY the whole connection
+ * (INTERNAL_ERROR) and kill all other streams. */
+ return 0;
+ }
return NGHTTP2_ERR_CALLBACK_FAILURE;
}
REPRODUCTION
No special server config. Drive Chromium (Playwright) to navigate rapidly
between pages that load several subresources over one HTTP/2 connection, so
navigations cancel in-flight requests. About 400 navigations produced about 4
GOAWAYs; with LogLevel http2:debug the on_frame_not_send_cb path with -510 is
visible.
EFFECT OF THE PATCH
3 x 400 navigations on a test instance: 0 failures, 0 GOAWAY (the race was
caught 13 times). In production after deploying the patch, a nightly browser
test fleet of 200 runs showed 0 ERR_HTTP2_PROTOCOL_ERROR, against 371
occurrences in 59 of 200 runs the day before.
Ruled out: CDN/proxy, ModSecurity, graceful restarts, MaxConnectionsPerChild,
KeepAlive races, and a rapid-reset flood (800 resets on one connection:
nothing).
Happy to test an alternative fix if you prefer different handling.
--
You are receiving this mail because:
You are the assignee for the bug.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
