Hi, HAProxy 2.9.12 was released on 2024/11/08. It added 81 new commits after version 2.9.11.
Changes brought in this release are very similar to the 3.0.6. All commits regarding the debug tools backported to 3.0 were also backported to the 2.9: * The watchdog now emits warnings when it detects apparently locked up threads. By default, a warning is emitted if a thread is blocked for more than one second. But this may be configured thanks to the global parameter "warn-blocked-traffic-after". The "debug dev loop" command was also improved to be able to emit such warning when "warn" argument is set. * The dump of threads info on panic was improved. During a panic, each thread now uses its own buffer instead of a global one to dump its info. This way, all these buffers remain available in the core dump and can be retrieved from gdb. This should help bug analysis. * Memory profiling was also improved. Some entries were displayed with a NULL return address, causing confusion. Now, undecodable stacks causing an apparent NULL return address all lead to the "other" bin. In addition, per-DSO stats are displayed before showing the total. It is more convenient on systems where many libraries are loaded. * A magic pattern was placed at the beginning of the post_mortem structure, in order to ease finding it in core dumps. It now starts with the 32-chars pattern "POST-MORTEM STARTS HERE+7654321\0". The post_mortem structure is now also placed in its own section, still to ease its finding. Finally, several important pointers were added in it, such as pointers on the pools list or on the proxies list. * Non-printable characters are now removed from the "debug dev fd" cli command output. * Some GDB hints are added when crashing, for instance on a BUG_ON(). * The backtraces of all threads are now dumped, instead of only for the stuck ones. * The version and the command line are now added in the "show dev" cli command output. * Two new sample fetch functions were added to retrieve the internal error name of the frontend (fc_err_name) or the backend (bc_err_name) connections. In addition, connection error codes corresponding to common errno were added, and they are now set when such errors are encountered during recv/send/splice() calls. * The current number of alive streams and the total number of streams ever created are now tracked and reported in stats. This may be useful to diagnose some bugs, like sessions leaks. On bugs fixed in this release, we are also more or less aligned with the 3.0.6: * It was possible to truncate data with the HTTP compression filter because of a bug in the filter API. When a filter may alter the message payload, it is important to properly update the HTX message metadata to not emit the wrong payload length. But this was not systematically performed. * In 2.4, it was decided to reject HTTP/1.1 protocol upgrade requests with a payload because it is incompatible with the H2 on server side. Indeed, such upgrade requests must be converted to CONNECT requests in H2. So no payload are supported. However, it remains valid in HTTP/1.1. So instead of rejecting it on client side, these requests are now accepted and properly handled when sent to a H1 server. They are only rejected when they are sent to a H2 server. * No special care about H2C protocol upgrade were took. But this could be a security issue if accepted by a server because it could be possible for a client to bypass all filtering rules. To fix the issue, the Upgrade header is removed from the requests if "h2c" or "h2" token are found. * The H1 multiplexer was only able to handle timeouts if the client or server timeouts were defined, depending on the side. So, it was possible to ignore client-fin/server-fin and http-keep-alive/http-request timeouts. * It was able to have some blocked transferts in H2 because of an issue with the zero-copy data forwarding. It was possible to never remove an H2 stream from the send list. * An issue with the zero-copy data forwarding of H1 requests waiting for a TUNNEL established was fixed. SE_FL_EOI flag was erroneously set on the client sedesc. * On QUIC side, it was possible to experience some freezes with 0-RTT connections; A leak was possible on post handshake frames on the error path ; A stream could be erroneously closed with an empty frame with FIN bit set instead of a RESET_STREAM frame when not data was sent at all; The server timeout was never armed for small requests, fully received when the stream is created; A possible connection freeze because of 0-RTT undeciphered content; 1nother freeze because of early QUIC stream closure, before transmitting any data; Retransmit for empty STREAM frames with FIN flag set was not properly handled and could cause the transfer to freeze with the client waiting indefinitely for the FIN notification. All these bugs were fixed. * A server abort was reported on an invalid HTTP response payload instead of an internal error. And it was also possible to report a client abort instead of a server abort during the HTTP response forwarding. The right termination states are now reported in both cases. * It was possible to experienced a deadlock by setting the maxconn of a frontend on the CLI, because of a double lock on the proxy lock. * "set ssl cert" CLI command was not properly checking the transaction name. That could lead to commit accidentally a transaction on the wrong certificate. * There were some issues with early connection shutdowns that could lead to truncated messages because some tests on blocked data were missing. In addition, blocked data by an error on the sending path were not always properly detected, leaving streams blocked without any timeout armed. * Dequeuing process was refined to fix some bugs revealed by recent fixes in this area. * Inter-thread stream shutdown, used by "shutdown sessions server XXX" CLI command or "on-error shutdown-sessions" server option, was not thread safe. * It was possible to reuse HTTP connections for requests to different endpoints because some address families where not properly handled. The issue was encountered with the HTTP client and UNIX socket combination. * A memory leak was possible if a failure is encountered when a dynamic server is added with a check or agent-check options. In that case, the server cannot be released because its refcount was incremented too early. In addition access to the global server list during a dynamic server deletion was not protected against concurrent accesses. In the longterm, this could cause list corruption and crashes. Thanks everyone for your help ! Please find the usual URLs below : Site index : https://www.haproxy.org/ Documentation : https://docs.haproxy.org/ Wiki : https://github.com/haproxy/wiki/wiki Discourse : https://discourse.haproxy.org/ Slack channel : https://slack.haproxy.org/ Issue tracker : https://github.com/haproxy/haproxy/issues Sources : https://www.haproxy.org/download/2.9/src/ Git repository : https://git.haproxy.org/git/haproxy-2.9.git/ Git Web browsing : https://git.haproxy.org/?p=haproxy-2.9.git Changelog : https://www.haproxy.org/download/2.9/src/CHANGELOG Dataplane API : https://github.com/haproxytech/dataplaneapi/releases/latest Pending bugs : https://www.haproxy.org/l/pending-bugs Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs Code reports : https://www.haproxy.org/l/code-reports Latest builds : https://www.haproxy.org/l/dev-packages --- Complete changelog : Amaury Denoyelle (10): BUG/MINOR: h1: do not forward h2c upgrade header token BUG/MINOR: h2: reject extended connect for h2c protocol MINOR: quic: convert qc_stream_desc release field to flags MINOR: quic: implement function to check if STREAM is fully acked BUG/MEDIUM: quic: handle retransmit for standalone FIN STREAM BUG/MINOR: quic: prevent freeze after early QCS closure BUG/MEDIUM: mux-quic: ensure timeout server is active for short requests BUG/MINOR: mux-quic: do not close STREAM with empty FIN if no data sent BUG/MINOR: server: fix dynamic server leak with check on failed init BUG/MEDIUM: server: fix race on servers_list during server deletion Aurelien DARRAGON (5): BUG/MEDIUM: server: server stuck in maintenance after FQDN change BUG/MEDIUM: hlua: make hlua_ctx_renew() safe BUG/MEDIUM: hlua: properly handle sample func errors in hlua_run_sample_{fetch,conv}() DOC: config: fix rfc7239 forwarded typo in desc BUG/MEDIUM: connection/http-reuse: fix address collision on unhandled address families Christopher Faulet (17): MINOR: mux-h1: Set EOI on SE during demux when both side are in DONE state BUG/MEDIUM: mux-h1/mux-h2: Reject upgrades with payload on H2 side only REGTESTS: h1/h2: Update script testing H1/H2 protocol upgrades BUG/MINOR: mux-h1: Fix condition to set EOI on SE during zero-copy forwarding BUG/MINOR: http-ana: Disable fast-fwd for unfinished req waiting for upgrade BUG/MEDIUM: stconn: Wait iobuf is empty to shut SE down during a check send BUG/MINOR: http-ana: Don't report a server abort if response payload is invalid BUG/MEDIUM: stconn: Check FF data of SC to perform a shutdown in sc_notify() BUG/MAJOR: filters/htx: Add a flag to state the payload is altered by a filter REGTESTS: Never reuse server connection in http-messaging/truncated.vtc BUG/MEDIUM: mux-h2: Remove H2S from send list if data are sent via 0-copy FF BUG/MEDIUM: stconn: Report blocked send if sends are blocked by an error BUG/MINOR: http-ana: Fix wrong client abort reports during responses forwarding BUG/MINOR: stconn: Don't disable 0-copy FF if EOS was reported on consumer side BUG/MEDIUM: mux-h1: Fix how timeouts are applied on H1 connections BUG/MINOR: http-ana: Report internal error if an action yields on a final eval MINOR: stream: Save last evaluated rule on invalid yield Frederic Lecaille (2): BUG/MINOR: quic: avoid leaking post handshake frames BUG/MEDIUM: quic: avoid freezing 0RTT connections Oliver Dala (1): BUG/MEDIUM: cli: Deadlock when setting frontend maxconn Valentine Krasnobaeva (3): BUG/MINOR: cfgparse-global: fix allowed args number for setenv BUG/MINOR: mworker: fix mworker-max-reloads parser MINOR: cli/debug: show dev: add cmdline and version William Lallemand (4): BUG/MINOR: httpclient: return NULL when no proxy available during httpclient_new() MINOR: cli: remove non-printable characters from 'debug dev fd' BUG/MINOR: trace: stop rewriting argv with -dt BUG/MINOR: ssl/cli: 'set ssl cert' does not check the transaction name correctly Willy Tarreau (39): REGTESTS: shorten a bit the delay for the h1/h2 upgrade test BUG/MINOR: server: make sure the HMAINT state is part of MAINT MINOR: task: define two new one-shot events for use with WOKEN_OTHER or MSG BUG/MEDIUM: stream: make stream_shutdown() async-safe BUG/MINOR: queue: make sure that maintenance redispatches server queue MINOR: server: make srv_shutdown_sessions() call pendconn_redistribute() BUG/MEDIUM: queue: always dequeue the backend when redistributing the last server MINOR: debug: make mark_tainted() return the previous value MINOR: chunk: drop the global thread_dump_buffer MINOR: debug: split ha_thread_dump() in two parts MINOR: debug: slightly change the thread_dump_pointer signification MINOR: debug: make ha_thread_dump_done() take the pointer to be used MINOR: debug: replace ha_thread_dump() with its two components MEDIUM: debug: on panic, make the target thread automatically allocate its buf BUG/MEDIUM: queue: make sure never to queue when there's no more served conns MINOR: activity/memprofile: always return "other" bin on NULL return address MINOR: activity/memprofile: show per-DSO stats BUILD: debug: silence a build warning with threads disabled MINOR: pools: export the pools variable MINOR: debug: place a magic pattern at the beginning of post_mortem MINOR: debug: place the post_mortem struct in its own section. MINOR: debug: store important pointers in post_mortem DOC: config: document connection error 44 (reverse connect failure) CLEANUP: connection: properly name the CO_ER_SSL_FATAL enum entry MINOR: connection: add more connection error codes to cover common errno MINOR: rawsock: set connection error codes when returning from recv/send/splice MINOR: connection: add new sample fetch functions fc_err_name and bc_err_name MINOR: stream: maintain a counter of the number of active streams. MINOR: debug: print gdb hints when crashing MINOR: debug: do not limit backtraces to stuck threads MINOR: debug: also add a pointer to struct global to post_mortem MINOR: debug: also add fdtab and acitvity to struct post_mortem MINOR: debug: remove the redundant process.thread_info array from post_mortem MINOR: wdt: move the local timers to a struct MINOR: debug: add a function to dump a stuck thread DEBUG: wdt: better detect apparently locked up threads and warn about them DEBUG: cli: make it possible for "debug dev loop" to trigger warnings DEBUG: wdt: make the blocked traffic warning delay configurable MINOR: debug: move the "recover now" warn message after the optional notes -- Christopher Faulet