Hello,
Trying to wrap my head around strange gRPC behavior I am seeing. I
took a look at the trace output, but I am not familiar with the H2
processing logic so I'll try to describe my observations the best I
can.
Four scenarios of gRPC requests that I have looked at. These are all
looking at how HAProxy communicates with the backend:
First one to look at is the happy path of a single message. If H2 mux
sends HEADER and DATA, where DATA includes End Stream flag, then the
response from backend server is HEADER (initial 200 OK), DATA (gRPC
protobuf), and HEADER (grpc-status header, includes End Stream flag
too).
Second scenario is that the backend service returns a failure right
away, for a missing service. The response skips the additional DATA
and HEADER. Response from backend is simply a HEADER (200 OK, with
grpc-status header and End Stream flag).
If the write stream is not closed by the frontend client, mux does not
set the End Stream flag, obviously. Backend responds with HEADER and
DATA, the backend is sent additional DATA, then the backend responds
with HEADER (End Stream here).
Where everything breaks apart is if the request fails. The gRPC
backend server seems to return HEADER frame of 200 OK, which right
away includes `grpc-status: 12` and `grpc-message: unknown service
grpc.reflection.v1.ServerReflection` headers. There is no follow-up
DATA or a later HEADER frame. Instead, the following frame from the
backend server is RST_STREAM as the server is done with the request
and wants the next request.
HAProxy sees this situation as SH-- termination state, as it is still
in mux_h2 FRAME_P state. It seems, it really wants the DATA.. but the
server doesn't have any to offer. This results in HTTP/1.1 502 Bad
Gateway which is not helpful for H2 gRPC clients. Instead, when the
stream is reset, the HTTP/2 200 OK with gRPC error headers should be
sent to the client at frontend.
Test requests are being made via github.com/jhump/protoreflect as it
keeps the stream open on client side for gRPC server reflection.
Unredacted TCP dumps can be provided in private, if preferred.
---
HAProxy version 2.8.2-1ppa1~jammy (from Ubuntu PPA)
global
log /dev/log local0
log /dev/log local1 notice
defaults
log global
mode http
option httplog
timeout connect 5000
timeout client 50000
timeout server 50000
frontend fe
bind 127.0.0.1:80 proto h2
default_backend be
backend be
server srv 65.109.88.38:12790 proto h2
---
Best regards,
Valters Jansons
