Hi,

Since 2aed0c9be ("BUG/MINOR: h3: fix missing EOM after trailers without
FIN"), an HTTP/3 client can crash haproxy on master by sending a
request with a body and trailers and never closing the stream. It isn't
in any stable branch yet, but it is tagged for backport up to 2.8, so
it would be good to have this fixed before it goes further. The patch
is attached, and its commit message has the full analysis.

On a stock build of master (68a8abcbe):

  FATAL: bug condition "blk == ((void *)0)" matched at src/htx.c:937
    call trace(15):
    |       0x706884 <42 90 00 e8 ac 76 f4 ff]: htx_xfer+0x634/0x668 > 
complain_with_dbg
    |       0x52dd31 <4c 89 ef e8 1f 85 1d 00]: qcs_http_rcv_buf+0x221/0x484 > 
htx_xfer
    |       0x4c6d41 <48 89 df e8 cf 6d 06 00]: qcm_strm_rcv_buf+0x51/0x6b3 > 
qcs_http_rcv_buf
    |       0x63ca3a <d9 0f 29 45 a0 ff 50 10]: sc_conn_recv+0x47a/0xae1
    |       0x640503 <89 75 e8 e8 bd c0 ff ff]: sc_conn_io_cb+0x93/0xf2 > 
sc_conn_recv

The patch keeps the unterminated trailers in the QCS Rx buffer until the
FIN adds the EOT. I picked that over the flag approach mentioned in the
commit message because the HTX tail already holds that state, but I'm
fine either way if you'd rather have it explicit.

Bisect and versions, stock builds (make TARGET=linux-glibc
USE_OPENSSL=1 USE_QUIC=1), with the attached reproducer:

  68a8abcbe (master)            died 10 of 10
  2aed0c9be                     died 3 of 3
  14fa727d4 (its parent)        died 0 of 5
  3.4.7                         died 0 of 5
  68a8abcbe, no buffer-request  died 2 of 10
  68a8abcbe + patch             died 0 of 10, also 0 of 10 without
                                buffer-request

With the patch, trailers without FIN followed by a late FIN, a
RESET_STREAM or a CONNECTION_CLOSE, with and without "option
http-buffer-request", gave no crash in 36 runs (18 stock, 18 ASAN with
no ASAN report). With the late FIN, the backend got the body first and
the trailers once the FIN arrived. reg-tests on the ASAN build give the
same result with and without the patch (357 passed, 4 failed in
ssl_client_auth / ssl_client_samples on this host's crypto policy).

Reproducer, attached as plain text (repro.cfg, repro.py, run.sh). It
needs openssl and python3 with aioquic. run.sh makes a throwaway cert
and starts haproxy, then repro.py does a GET (baseline), the POST
without FIN, and another GET:

  HAPROXY=/path/to/haproxy ./run.sh

  baseline: GET answered
  sent: POST with HEADERS + DATA + trailers HEADERS, no FIN
  after: GET not answered, haproxy is gone
  haproxy: died, wait status 132, FATAL: bug condition "blk == ((void *)0)" 
matched at src/htx.c:937

I found this with AI-assisted tooling. The reproducer and the patch
were tested on real builds as described.

Daniel

Attachment: repro.cfg
Description: Binary data

Attachment: repro.py
Description: Binary data

#!/bin/sh
# usage: HAPROXY=/path/to/haproxy PYTHON=/path/to/python3 ./run.sh
# Makes a throwaway self-signed cert, starts haproxy with repro.cfg on
# 127.0.0.1:8443 (QUIC) and 127.0.0.1:8081, runs repro.py, then reports how
# haproxy ended. Exit status is repro.py's: 0 reproduced, 1 not, 2 setup problem.
cd "$(dirname "$0")" || exit 2
HAPROXY=${HAPROXY:-haproxy}
PYTHON=${PYTHON:-python3}
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -days 1 \
    -subj /CN=localhost -keyout cert.pem -out cert.pem 2>/dev/null || exit 2
"$HAPROXY" -db -f repro.cfg > haproxy.out 2>&1 &
pid=$!
sleep 1
"$PYTHON" repro.py 127.0.0.1:8443
rc=$?
if kill -0 $pid 2>/dev/null; then
    kill $pid; wait $pid 2>/dev/null
    echo "haproxy: still running at the end"
else
    wait $pid; st=$?
    echo "haproxy: died, wait status $st$(grep -m1 -o 'FATAL: bug condition.*' haproxy.out | sed 's/^/, /')"
fi
rm -f cert.pem
exit $rc

Attachment: 0001-BUG-MEDIUM-mux_quic-do-not-transfer-trailers-before-.patch
Description: Binary data

Reply via email to