On Thu, Sep 14, 2023 at 12:35 PM Christopher Faulet <cfau...@haproxy.com> wrote:
> After a discussion with Willy, we've hopefully found a way to fix the issue by
> delaying detection of the server abort on the request processing side when
> there
> is a response to forward to the client. It should do the trick in your case
> and
> it should be safe.
>
> However, the fix remains sensitive. It is really hard to be sure it will not
> introduce a regression. The worst could be to block a session or to get a loop
> because of an unhandled event.
>
> Thus it could be go to test it on your side if it is possible. The patch is in
> attachment. It can be applied on top of the 2.9 or 2.8. Is this possible for
> you ?
I applied the patch on top of v2.9-dev5. For what it's worth, I build
on Ubuntu 22.04 with defaults from INSTALL: TARGET=linux-glibc
USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 USE_SYSTEMD=1.
The delayed processing patch does not appear to change anything for
me. I am still seeing HTTP 502 (Bad Gateway) and HAProxy logs report
session state as SH--. According to documentation, that is "the TCP
session was unexpectedly aborted by the server, or the server
explicitly refused it" together with "the proxy was waiting for
complete, valid response HEADERS from the server (HTTP only)".
Somehow, if the H2 connection to backend is half-open(local), HAProxy
is expecting more HEADERS from the backend server.
I am providing a short diff below, which force-closes the H2
connection to backend upon remote ES (which would break long-running
upload from frontend to client). The change is interesting, because it
highlights something weird with processing. Without this change, even
with your provided patch, HAProxy never finishes processing HEADERS.
However, with this change, response is delivered to the frontend
client when the stream closes.
The session state is probably not relevant, but for what it's worth,
it becomes CL--. C for "the TCP session was unexpectedly aborted by
the client" -- not sure why unexpectedly, because from the client side
everything received successfully -- and L for "the proxy was still
transmitting LAST data to the client while the server had already
finished. This one is very rare as it can only happen when the client
dies while receiving the last packets". Maybe this gives some other
ideas.
diff --git a/src/mux_h2.c b/src/mux_h2.c
index cc698b66b..c1ff014dd 100644
--- a/src/mux_h2.c
+++ b/src/mux_h2.c
@@ -2934,7 +2934,7 @@ static struct h2s *h2c_bck_handle_headers(struct
h2c *h2c, struct h2s *h2s)
else if (h2s->flags & H2_SF_ES_RCVD) {
if (h2s->st == H2_SS_OPEN)
h2s->st = H2_SS_HREM;
- else if (h2s->st == H2_SS_HLOC)
+ if (h2s->st == H2_SS_HLOC || h2s->st == H2_SS_HREM)
h2s_close(h2s);
}
The clients should send proper and timely END_STREAMs, so this should
not be a major issue, but something still feels strange here with
connection state processing.
Let me know if I can help brainstorm this further.
Best regards,
Valters Jansons
