Hi, HAProxy 2.9.4 was released on 2024/01/31. It added 24 new commits after version 2.9.3.
This version addresses various long-term stability issues that popped all at once, so I preferred to issue it shortly after they were all addressed rather than risking to let more accumulate and see users forced to roll back later due to any possible regression that may later happen. We'll also issue new versions of older branches progressively as time permits. The issues fixes in this version are: - an API issue with OpenSSL. The SSL_do_handshake() function returns SSL_ERROR_WANT_READ when it needs more data, but in certain obscure circumstances related to internal error handling, it was found that it may stop trying to read available data and continue to return that status! This results in wakeup loops that prevent the process from sleeping, hence it consumes 100% of the CPU (but it's still working fine). The code does what the doc suggests (but the doc is basically a one-liner), and neither aws-lc nor wolfSSL exhibit this problem. Regardless, we decided to do like openssl does in their socket BIO which doesn't show this problem and which always clears both direction flags before any attempt in any direction, and this addressed the issue without degrading anything even for the other libs. This problem has been there since 2.0 and is very hard to reproduce without a prior trace (it's the first time it's reported). I'd like to thank Valentin Gutierrez for his invaluable report with a capture and a working reproducer, and Olivier Houchard for the quick fix that clearly looks more robust than my early workaround. That's issue #2403. - a regression in the cache's handling of secondary keys in 2.9 that may sometimes cause a crash (issue #2417). - a possible crash in the QPACK encoder when encoding HTTP/3 responses carrying status codes above 599. - another QUIC issue whereby the some streams reset with pending outgoing data may clog the output buffer until the connection closes, possibly causing the connection to slow down or even stall. - in H2, certain errors would only trigger a stream error (i.e. RESET) instead of a connection error consecutive to an insufficient fix that was merged in 2.9.3. - the status of agent checks is returned as-is in the stats CSV output, resulting in mangling the CLI's output if it contains line feeds. It has been there since 2.0. - the HTTP/1 chunk and header parsers were strengthened a bit. Indeed, Ben Kallus kindly reminded us that we would still accept the NUL byte in header values and plain LF in chunks, while we were (wrongly) quite certain that these had long been rejected. Ben is currently not aware of situations where this could help convey an attack to any existing component, but given the surprises he certainly faces in his reviews, it's probably only a matter of time before one implementation shows to be too weak and we fail to properly protect it. So it was better to address both at once. In the extremely unlikely case that anyone would discover such an invalid byte on their network with an application that heavily relies on it, option accept-invalid-http* will work as usual to bypass the check. We'll backport that to older versions as well, and I think it would be prudent for distros to take that as well. - an interesting arch-specific bug in the JWT parser: by initializing a 64-bit variable a bit too early, everything was fine on 64-bit platforms, but on 32-bit ones, a pointer located closer to the beginning of the structure got reset by this initialization before it was used, causing a crash! The fact this was only noticed now by running VTest on a 32-bit platform just shows that 32-bit users are less common these days and that their configs are probably simple enough not to use JWT ;-) - the "newreno" congestion control algorithm for QUIC was misspelled "newrno" in the code, making the config parser not recognize it. - and a few other low-importance stuff and doc updates. I'd suggest all users of 2.9 to adopt this one now so that we can later switch to less important fixes and backports if needed. There's currently nothing else in the pipe concerning bugs, but we're still investigating a case we triggered in the lab were the QUIC congestion window sometimes doesn't open enough, and which could be responsible for lower than expected performance on large objects when using the default Cubic algorithm (as Tristan observes). But that's quite difficult because the original RFC was barely exploitable due to numerous ambiguities, and fortunately there's a new very recent one that allows to recheck the code against it (and we'll take this opportunity to rename some parts according to the updated spec). We're hopeful that we'll get some good news from this front soon! Please find the usual URLs below : Site index : https://www.haproxy.org/ Documentation : https://docs.haproxy.org/ Wiki : https://github.com/haproxy/wiki/wiki Discourse : https://discourse.haproxy.org/ Slack channel : https://slack.haproxy.org/ Issue tracker : https://github.com/haproxy/haproxy/issues Sources : https://www.haproxy.org/download/2.9/src/ Git repository : https://git.haproxy.org/git/haproxy-2.9.git/ Git Web browsing : https://git.haproxy.org/?p=haproxy-2.9.git Changelog : https://www.haproxy.org/download/2.9/src/CHANGELOG Dataplane API : https://github.com/haproxytech/dataplaneapi/releases/latest Pending bugs : https://www.haproxy.org/l/pending-bugs Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs Code reports : https://www.haproxy.org/l/code-reports Latest builds : https://www.haproxy.org/l/dev-packages And thanks again to all issue reporters, it was dense but very efficient this time! Willy --- Complete changelog : Amaury Denoyelle (7): BUG/MINOR: h3: fix checking on NULL Tx buffer MINOR: quic: extract qc_stream_buf free in a dedicated function BUG/MEDIUM: quic: remove unsent data from qc_stream_desc buf MINOR: h3: add traces for stream sending function BUG/MEDIUM: h3: do not crash on invalid response status code BUG/MEDIUM: qpack: allow 6xx..9xx status codes BUG/MEDIUM: quic: fix crash on invalid qc_stream_buf_free() BUG_ON Aurelien DARRAGON (2): DOC: configuration: fix set-dst in actions keywords matrix BUG/MINOR: hlua: fix uninitialized var in hlua_core_get_var() Christopher Faulet (2): BUG/MINOR: h1: Don't support LF only at the end of chunks BUG/MEDIUM: h1: Don't support LF only to mark the end of a chunk size Emeric Brun (1): BUG/MEDIUM: cli: some err/warn msg dumps add LR into CSV output on stat's CLI Frederic Lecaille (3): BUG/MINOR: quic: newreno QUIC congestion control algorithm no more available CLEANUP: quic: Remove unused CUBIC_BETA_SCALE_FACTOR_SHIFT macro. MINOR: quic: Stop hardcoding a scale shifting value (CUBIC_BETA_SCALE_FACTOR_SHIFT) Lukas Tribus (1): DOC: httpclient: add dedicated httpclient section Olivier Houchard (1): BUG/MAJOR: ssl_sock: Always clear retry flags in read/write functions Remi Tricot-Le Breton (1): BUG/MEDIUM: cache: Fix crash when deleting secondary entry Thayne McCombs (1): DOC: configuration: clarify http-request wait-for-body Willy Tarreau (5): BUG/MEDIUM: mux-h2: refine connection vs stream error on headers MINOR: mux-h2/traces: add a missing trace on connection WU with negative inc BUG/MINOR: jwt: fix jwt_verify crash on 32-bit archs BUG/MINOR: h1-htx: properly initialize the err_pos field BUG/MEDIUM: h1: always reject the NUL character in header values ---