Hi,
Since 2aed0c9be ("BUG/MINOR: h3: fix missing EOM after trailers without
FIN"), an HTTP/3 client can crash haproxy on master by sending a
request with a body and trailers and never closing the stream. It isn't
in any stable branch yet, but it is tagged for backport up to 2.8, so
it would be good to have this fixed before it goes further. The patch
is attached, and its commit message has the full analysis.
On a stock build of master (68a8abcbe):
FATAL: bug condition "blk == ((void *)0)" matched at src/htx.c:937
call trace(15):
| 0x706884 <42 90 00 e8 ac 76 f4 ff]: htx_xfer+0x634/0x668 >
complain_with_dbg
| 0x52dd31 <4c 89 ef e8 1f 85 1d 00]: qcs_http_rcv_buf+0x221/0x484 >
htx_xfer
| 0x4c6d41 <48 89 df e8 cf 6d 06 00]: qcm_strm_rcv_buf+0x51/0x6b3 >
qcs_http_rcv_buf
| 0x63ca3a <d9 0f 29 45 a0 ff 50 10]: sc_conn_recv+0x47a/0xae1
| 0x640503 <89 75 e8 e8 bd c0 ff ff]: sc_conn_io_cb+0x93/0xf2 >
sc_conn_recv
The patch keeps the unterminated trailers in the QCS Rx buffer until the
FIN adds the EOT. I picked that over the flag approach mentioned in the
commit message because the HTX tail already holds that state, but I'm
fine either way if you'd rather have it explicit.
Bisect and versions, stock builds (make TARGET=linux-glibc
USE_OPENSSL=1 USE_QUIC=1), with the attached reproducer:
68a8abcbe (master) died 10 of 10
2aed0c9be died 3 of 3
14fa727d4 (its parent) died 0 of 5
3.4.7 died 0 of 5
68a8abcbe, no buffer-request died 2 of 10
68a8abcbe + patch died 0 of 10, also 0 of 10 without
buffer-request
With the patch, trailers without FIN followed by a late FIN, a
RESET_STREAM or a CONNECTION_CLOSE, with and without "option
http-buffer-request", gave no crash in 36 runs (18 stock, 18 ASAN with
no ASAN report). With the late FIN, the backend got the body first and
the trailers once the FIN arrived. reg-tests on the ASAN build give the
same result with and without the patch (357 passed, 4 failed in
ssl_client_auth / ssl_client_samples on this host's crypto policy).
Reproducer, attached as plain text (repro.cfg, repro.py, run.sh). It
needs openssl and python3 with aioquic. run.sh makes a throwaway cert
and starts haproxy, then repro.py does a GET (baseline), the POST
without FIN, and another GET:
HAPROXY=/path/to/haproxy ./run.sh
baseline: GET answered
sent: POST with HEADERS + DATA + trailers HEADERS, no FIN
after: GET not answered, haproxy is gone
haproxy: died, wait status 132, FATAL: bug condition "blk == ((void *)0)"
matched at src/htx.c:937
I found this with AI-assisted tooling. The reproducer and the patch
were tested on real builds as described.
Daniel
repro.cfg
Description: Binary data
repro.py
Description: Binary data
#!/bin/sh
# usage: HAPROXY=/path/to/haproxy PYTHON=/path/to/python3 ./run.sh
# Makes a throwaway self-signed cert, starts haproxy with repro.cfg on
# 127.0.0.1:8443 (QUIC) and 127.0.0.1:8081, runs repro.py, then reports how
# haproxy ended. Exit status is repro.py's: 0 reproduced, 1 not, 2 setup problem.
cd "$(dirname "$0")" || exit 2
HAPROXY=${HAPROXY:-haproxy}
PYTHON=${PYTHON:-python3}
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -days 1 \
-subj /CN=localhost -keyout cert.pem -out cert.pem 2>/dev/null || exit 2
"$HAPROXY" -db -f repro.cfg > haproxy.out 2>&1 &
pid=$!
sleep 1
"$PYTHON" repro.py 127.0.0.1:8443
rc=$?
if kill -0 $pid 2>/dev/null; then
kill $pid; wait $pid 2>/dev/null
echo "haproxy: still running at the end"
else
wait $pid; st=$?
echo "haproxy: died, wait status $st$(grep -m1 -o 'FATAL: bug condition.*' haproxy.out | sed 's/^/, /')"
fi
rm -f cert.pem
exit $rc
0001-BUG-MEDIUM-mux_quic-do-not-transfer-trailers-before-.patch
Description: Binary data

