Hi,
HAProxy 2.8.6 was released on 2024/02/15. It added 88 new commits after
version 2.8.5. It essentially groups the relevant fixes present in 2.9.2
to 2.9.5, most importantly:
- an API issue with OpenSSL. The SSL_do_handshake() function returns
SSL_ERROR_WANT_READ when it needs more data, but in certain obscure
circumstances related to internal error handling, it was found that
it may stop trying to read available data and continue to return that
status! This results in wakeup loops that prevent the process from
sleeping, hence it consumes 100% of the CPU (but it's still working
fine). The code does what the doc suggests (but the doc is basically
a one-liner), and neither aws-lc nor wolfSSL exhibit this problem.
- the QUIC compatibility layer didn't support setting the client
hello callback that's needed to extract some information from the
incoming connection.
- another QUIC issue whereby the some streams reset with pending outgoing
data may clog the output buffer until the connection closes, possibly
causing the connection to slow down or even stall.
- a possible crash in the QPACK encoder when encoding HTTP/3 responses
carrying status codes above 599.
- the HTTP/1 chunk and header parsers were strengthened a bit. Indeed,
Ben Kallus kindly reminded us that we would still accept the NUL byte
in header values and plain LF in chunks, while we were (wrongly) quite
certain that these had long been rejected. Ben is currently not aware
of situations where this could help convey an attack to any existing
component, but given the surprises he certainly faces in his reviews,
it's probably only a matter of time before one implementation shows to
be too weak and we fail to properly protect it. So it was better to
address both at once. In the extremely unlikely case that anyone would
discover such an invalid byte on their network with an application that
heavily relies on it, option accept-invalid-http* will work as usual to
bypass the check. We'll backport that to older versions as well, and I
think it would be prudent for distros to take that as well.
- A new global setting was added to the H2 multiplexer to limit the
total number of H2 streams per client connection. To do so,
'tune.h2.fe.max-total-streams' can now be used. It sets the HTTP/2
maximum number of total streams processed per incoming
connection. Once this limit is reached, HAProxy will send a graceful
GOAWAY frame informing the client that it will close the connection
after all pending streams have been closed. In practice, clients tend
to close as fast as possible when receiving this, and to establish a
new connection for next requests. Doing this is sometimes useful and
desired in situations where clients stay connected for a very long
time and cause some imbalance inside a farm. For example, in some
highly dynamic environments, it is possible that new load balancers
are instantiated on the fly to adapt to a load increase, and that
once the load goes down they should be stopped without breaking
established connections. By setting a limit here, the connections
will have a limited lifetime and will be frequently renewed, with
some possibly being established to other nodes, so that existing
resources are quickly released. Of course nothing changes by default
when not touching this setting.
- an interesting arch-specific bug in the JWT parser: by initializing
a 64-bit variable a bit too early, everything was fine on 64-bit
platforms, but on 32-bit ones, a pointer located closer to the
beginning of the structure got reset by this initialization before it
was used, causing a crash! The fact this was only noticed now by running
VTest on a 32-bit platform just shows that 32-bit users are less common
these days and that their configs are probably simple enough not to use
JWT ;-)
- a rare deadlock was found on the pools code, it can be triggered at
stopping time and crash the old process. It's been there since 2.5,
and is difficult to trigger, but a user faced it and that's how we
learned about it (GH issue #2427, thanks to user @JB0925).
- we finished our round of analysis, documentation and fixes for the
QUIC congestion control code and figured a few causes of corner cases
that can occasionally cause more performance degradation than expected,
especially on the loss detection and reordering part. For this, a new
global setting "tune.quic.reorder-ratio" allows to indicate what portion
of the in flight window can appear to be reordered before declaring a
loss. There's now a separate counter per connection for the reordered
packets that will help us figure what's happening. Tests on degraded
networks showed a x10 performance increase with the new default setting.
- some more CLI commands were found to occasionally miss the trailing
line feed, possibly confusing scripts and APIs. These were corrected,
and a more general fix was applied to the command line processor to
make sure each command always finishes with exactly one LF.
- the status of agent checks is returned as-is in the stats CSV output,
resulting in mangling the CLI's output if it contains line feeds. It
has been there since 2.0.
- an OCSP update reference counting issue was fixed, which was apparently
causing some certificates to reference a just freed OCSP response. Also
on errors, the reported message from the OCSP updater was confusing
(this one is in issue #2432, thanks to Frank Wall for the report).
- when deleting a crt-list line from the CLI, a dangling pointer reference
could be left, with the possible effect of causing a crash. Apparently
it has been the case since 2.4 so it seems that not that many people
use "del ssl crt-list" or that the occurrence is quite rare.
- the diag warnings (enabled using -dD) were not all run when checking
a config with "-c"! One would have to explicitly try to start the config
to run the last ones. The only missing ones were the duplicate server
cookie check.
- and a few other low-importance stuff and doc updates.
- Abhijeet Rastogi found that we still didn't recommend to the PCRE2 over
PCRE that's no longer maintained. It was just an overlook and the doc
was updated.
- and the usual CI updates (support for cache API v4, thanks to Tim), doc
cleanups and updates.
And I think that's all. There is no emergency in deploying this version,
but it contains a significant number of quite annoying bugs that are hard
to troubleshoot, so I would really appreciate it if we soon see this one
replace all other existing 2.8.
Please find the usual URLs below :
Site index : https://www.haproxy.org/
Documentation : https://docs.haproxy.org/
Wiki : https://github.com/haproxy/wiki/wiki
Discourse : https://discourse.haproxy.org/
Slack channel : https://slack.haproxy.org/
Issue tracker : https://github.com/haproxy/haproxy/issues
Sources : https://www.haproxy.org/download/2.8/src/
Git repository : https://git.haproxy.org/git/haproxy-2.8.git/
Git Web browsing : https://git.haproxy.org/?p=haproxy-2.8.git
Changelog : https://www.haproxy.org/download/2.8/src/CHANGELOG
Dataplane API :
https://github.com/haproxytech/dataplaneapi/releases/latest
Pending bugs : https://www.haproxy.org/l/pending-bugs
Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs
Code reports : https://www.haproxy.org/l/code-reports
Latest builds : https://www.haproxy.org/l/dev-packages
Willy
---
Complete changelog :
Abhijeet Rastogi (1):
DOC: install: recommend pcre2
Amaury Denoyelle (15):
BUG/MINOR: mux-quic: always report error to SC on RESET_STREAM emission
MINOR: h3: check connection error during sending
BUG/MINOR: h3: close connection on header list too big
BUG/MINOR: h3: properly handle alloc failure on finalize
BUG/MINOR: h3: close connection on sending alloc errors
BUG/MEDIUM: h3: fix incorrect snd_buf return value
BUG/MINOR: mux-quic: do not prevent non-STREAM sending on flow control
BUG/MINOR: h3: fix checking on NULL Tx buffer
BUG/MEDIUM: mux-quic: report early error on stream
MINOR: quic: extract qc_stream_buf free in a dedicated function
BUG/MEDIUM: quic: remove unsent data from qc_stream_desc buf
MINOR: h3: add traces for stream sending function
BUG/MEDIUM: h3: do not crash on invalid response status code
BUG/MEDIUM: qpack: allow 6xx..9xx status codes
BUG/MEDIUM: quic: fix crash on invalid qc_stream_buf_free() BUG_ON
Aurelien DARRAGON (5):
MINOR: stats: store the parent proxy in stats ctx (http)
BUG/MEDIUM: stats: unhandled switching rules with TCP frontend
REGTESTS: add a test to ensure map-ordering is preserved
BUG/MINOR: ext-check: cannot use without preserve-env
DEV: makefile: fix POSIX compatibility for "range" target
Christopher Faulet (8):
BUG/MEDIUM: mux-h2: Report too large HEADERS frame only when rxbuf is
empty
DOC: config: Update documentation about local haproxy response
BUG/MEDIUM: stconn: Forward shutdown on write timeout only if it is
forwardable
BUG/MEDIUM: spoe: Never create new spoe applet if there is no server up
BUG/MEDIUM: stconn: Allow expiration update when READ/WRITE event is
pending
BUG/MEDIUM: stconn: Don't check pending shutdown to wake an applet up
BUG/MINOR: h1: Don't support LF only at the end of chunks
BUG/MEDIUM: h1: Don't support LF only to mark the end of a chunk size
Emeric Brun (1):
BUG/MEDIUM: cli: some err/warn msg dumps add LR into CSV output on stat's
CLI
Frederic Lecaille (11):
BUG/MEDIUM: quic: keylog callback not called (USE_OPENSSL_COMPAT)
CLEANUP: quic: Remove unused CUBIC_BETA_SCALE_FACTOR_SHIFT macro.
MINOR: quic: Stop hardcoding a scale shifting value
(CUBIC_BETA_SCALE_FACTOR_SHIFT)
BUG/MINOR: quic: Wrong ack ranges handling when reaching the limit.
CLEANUP: quic: Code clarifications for QUIC CUBIC (RFC 9438)
BUG/MINOR: quic: fix possible integer wrap around in cubic window
calculation
MINOR: quic: Stop using 1024th of a second.
BUG/MEDIUM: quic: Wrong K CUBIC calculation.
MINOR: quic: Update K CUBIC calculation (RFC 9438)
MINOR: quic: Dynamic packet reordering threshold
MINOR: quic: Add a counter for reordered packets
Frédéric Lécaille (5):
BUG/MEDIUM: quic: Possible buffer overflow when building TLS records
BUG/MEDIUM: quic: QUIC CID removed from tree without locking
BUG/MINOR: quic: Wrong keylog callback setting.
BUG/MINOR: quic: Missing call to TLS message callbacks
CLEANUP: quic: Remaining useless code into server part
Lukas Tribus (1):
DOC: httpclient: add dedicated httpclient section
Miroslav Zagorac (1):
DOC: configuration: corrected description of keyword
tune.ssl.ocsp-update.mindelay
Olivier Houchard (1):
BUG/MAJOR: ssl_sock: Always clear retry flags in read/write functions
Remi Tricot-Le Breton (10):
BUG/MINOR: ssl: Fix error message after ssl_sock_load_ocsp call
BUG/MINOR: ssl: Duplicate ocsp update mode when dup'ing ckch
BUG/MINOR: ssl: Clear the ckch instance when deleting a crt-list line
MINOR: ssl: Use OCSP_CERTID instead of ckch_store in
ckch_store_build_certid
BUG/MEDIUM: ocsp: Separate refcount per instance and per store
BUG/MINOR: ssl: Destroy ckch instances before the store during deinit
BUG/MINOR: ssl: Reenable ocsp auto-update after an "add ssl crt-list"
REGTESTS: ssl: Fix empty line in cli command input
REGTESTS: ssl: Add OCSP related tests
BUG/MEDIUM: ssl: Fix crash when calling "update ssl ocsp-response" when
an update is ongoing
Thayne McCombs (1):
DOC: configuration: clarify http-request wait-for-body
Tim Duesterhus (1):
CI: Update to actions/cache@v4
William Lallemand (4):
DOC: configuration: typo req.ssl_hello_type
BUG/MINOR: mworker/cli: fix set severity-output support
BUG/MINOR: resolvers: default resolvers fails when network not configured
MINOR: errors: ha_alert() and ha_warning() uses warn_exec_path()
Willy Tarreau (23):
MINOR: mux-h2: support limiting the total number of H2 streams per
connection
BUG/MINOR: mux-h2: also count streams for refused ones
MINOR: compiler: add a new DO_NOT_FOLD() macro to prevent code folding
MINOR: debug: make sure calls to ha_crash_now() are never merged
MINOR: debug: make ABORT_NOW() store the caller's line number when using
abort
MINOR: debug: make BUG_ON() catch build errors even without DEBUG_STRICT
MINOR: mux-h2/traces: also suggest invalid header upon parsing error
MINOR: mux-h2/traces: explicitly show the error/refused stream states
MINOR: mux-h2/traces: clarify the "rejected H2 request" event
BUG/MEDIUM: mux-h2: refine connection vs stream error on headers
MINOR: mux-h2/traces: add a missing trace on connection WU with negative
inc
BUG/MINOR: vars/cli: fix missing LF after "get var" output
BUG/MEDIUM: cli: fix once for all the problem of missing trailing LFs
BUG/MINOR: jwt: fix jwt_verify crash on 32-bit archs
BUG/MEDIUM: pool: fix rare risk of deadlock in pool_flush()
BUG/MINOR: h1-htx: properly initialize the err_pos field
BUG/MEDIUM: h1: always reject the NUL character in header values
BUG/MINOR: diag: always show the version before dumping a diag warning
BUG/MINOR: diag: run the final diags before quitting when using -c
MINOR: ext-check: add an option to preserve environment variables
BUILD: address a few remaining calloc(size, n) cases
DOC: internal: update missing data types in peers-v2.0.txt
DEV: makefile: add a new "range" target to iteratively build all commits
---
