Hi, HAProxy 3.4-dev5 was released on 2026/02/19. It added 134 new commits after version 3.4-dev4.
43 bugs were fixed since last version two weeks ago, in various areas, most of which also concern stable branches and were already backported. Despite this, I'm pleased to see that work has been advancing well in plenty of areas, which is great if we want a feature freeze at the end of March: - dynamic settings: the "set-timeout" action now supports setting connect, tarpit and queue timeouts. This may allow to simplify some configurations by generic values and rely on maps for application-specific ones. - task profiling: we got a report of intense CPU usage since 3.3 when task profiling was on due to lock profiling being enabled with it. In the end it wasn't a wise idea to combine them all together so the lock and malloc task profiling settings were taken apart and are now optional when task profiling is enabled (they're mostly for developers, but it turns out that power users love to have a quick glance at task profiling to spot anomalies). This was backported into 3.3. - a very old dream is coming true: for a very long time (since 1.3 or 1.4) I've been hoping to be able to support multiple buffer sizes. We already made a step in that direction with the QUIC mux supporting small buffers for small requests/responses, but that didn't leave the mux. Now the stream layer (the topmost, where all analyzers are operating and executing rules and load balancing) supports working with large buffers. This means that if you have to look up a user ID in a request body, or even scan for some bad patterns, it's no longer necessary to increase tune.bufsize, it's possible to set tune.bufsize.large to the given value, and by adjusting the "wait-for-body" rule, you can allow it to allocate a large buffer if the default one fills up. This is great because we've seen many configs with large buffers that occupy a huge amount of memory in muxes, and this will no longer be necessary. A next step will be to try to make it possible to automatically fall back to the small ones for queued requests. But given that the work needed to support multiple buffer sizes was now done, it should be less difficult. If you are using a non-standard bufsize, please give this a try and report any possible issue! - the "balance random" algorithm wasn't fair when using hundreds to thousands of servers with less concurrent connections than servers, because it would solely rely on the concurrent connection count to decide between servers. This could result in frequency ratios of 2-3:1 between least and most used ones. Now when compared servers have the same number of concurrent connections, it will also compare their recent traffic history (req/s over the last second) to pick the (weighted) least loaded one. This has changed the imbalance to only a few percent. This was backported into 3.3 already since the default algo changed there. - the storage of objects into the cache now avoids one useless memory copy, thus saving CPU cycles and memory bandwidth on misses. Similarly, when waiting on body, the pending payload that might be fragmented due to headers manipulation is now first defragmented before being appended to. This will avoid a later copy when performing content analysis. - the internal filters architecture is being reworked so that the filters, which were previously attached to the stream and would decide on which channel (direction) to work based on their state, can now be specific to one direction. This means that the compression filter which can act both on the request and response can now have two cleanly distinct contexts. This is some preliminary work to implement decompression. - another feature that has been requested for a long time is the dynamic backend creation from the CLI. The principle at the moment is to create a backend from an existing named "defaults" section, then to add servers to it, then to enable it. It is not yet possible to delete the backend, this is still a work in progress (and that we really hope to have by 3.4). The feature is still experimental but given that it now relies on solid foundations, we think that it would be reasonable to imagine dropping the experimental status for 3.4. But for this we'll definitely need testers and feedback (ranging from "it works, cool" to "I have a reproducible crash", passing by "cannot work for me for reason X"), so please have a look at "add backend", "add server" and "publish backend" in the management doc. I suspect that backend removal might remain experimental longer though it would be a shame, so we're still very active on this topic in order to avoid lots of reloads in orchestrated environments. - the good old httpterm that I have been maintaining for 20 years as a fork of haproxy-1.2 and attempted to port to 1.6 was finally rewritten on top of current master to benefit from the flexible and scalable architecture, which will significantly ease testing and benchmarks in the future. We now have an "haterm" mode for frontends, but in order to preserve the spirit of extreme simplicity of httpterm, there's also an "haterm" program that's basically haproxy with a different command line parser that produces an in-memory config based on the arguments, and just like httpterm, it only needs one or two listening ports to create everything. The great thing is that it now supports SSL, H1/H2/H3, automatic thread binding (it's just haproxy in fact). Initial tests show that each second it can deliver 4.3M H1 requests, 7.5M H2 requests or 11M h3 requests on a 80-core ARM, and that locally with only 40 threads it can push half a terabit/s of clear HTTP/1 or 250 Gbps of TLS traffic, and there's still room for improvement. This indicates that it should require little resource for easily reproducible benchmarks. Some httpterm features are still missing (e.g. splicing and funky responses), some of haproxy such as KTLS are not yet easily exploitable but it's in good shape. Oh and a first nice benefit is that it puts even more stress on the lower layers and allows us to zoom on some contention points with some ways to measure new gains. - JWT: a new jwt_decrypt_jwk() converter that takes an RSA key in JWK format. - some optimizations (locking reduction in the queue code and in the mux selection code), - a number of fixes, minor updates and cleanups to the DeviceAtlas extension. - the supported features list in "haproxy -vv" now shows a separate line for runtime-detected features list (e.g. HAVE_WORKING_TCP_MD5SIG indicates whether the OS supports MD5 sigs or not), and features are now sorted. A few more minor info were updated in -vv. - some CI fixes and updates -windows, OSX, vtest2 new URL, QUIC interop), and various cleanups. That's a massive update in terms of features and improvements, so please beat it hard, really! On a side note, after last week's release of the QUIC bugfixes, I was curious whether targetted fuzzing was frequent in the wild or not, and for this I purposely left the vunlerable version running on haproxy.org, (the issue is not dramatic, in the worst case it crashes and restarts on the stable version). And nothing happened in one week, which seems to suggest that there are much lower hanging fruits to play with than QUIC stacks at the moment. That's not a reason for skipping fixes though :-) Please find the usual URLs below : Site index : https://www.haproxy.org/ Documentation : https://docs.haproxy.org/ Wiki : https://github.com/haproxy/wiki/wiki Discourse : https://discourse.haproxy.org/ Slack channel : https://slack.haproxy.org/ Issue tracker : https://github.com/haproxy/haproxy/issues Q&A from devs : https://github.com/orgs/haproxy/discussions Sources : https://www.haproxy.org/download/3.4/src/ Git repository : https://git.haproxy.org/git/haproxy.git/ Git Web browsing : https://git.haproxy.org/?p=haproxy.git Changelog : https://www.haproxy.org/download/3.4/src/CHANGELOG Dataplane API : https://github.com/haproxytech/dataplaneapi/releases/latest Pending bugs : https://www.haproxy.org/l/pending-bugs Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs Code reports : https://www.haproxy.org/l/code-reports Latest builds : https://www.haproxy.org/l/dev-packages Willy --- Complete changelog : Amaury Denoyelle (24): MINOR: cfgparse: validate defaults proxies separately MINOR: cfgparse: move proxy post-init in a dedicated function MINOR: proxy: refactor proxy inheritance of a defaults section MINOR: proxy: refactor mode parsing MINOR: backend: add function to check support for dynamic servers MINOR: proxy: define "add backend" handler MINOR: proxy: parse mode on dynamic backend creation MINOR: proxy: parse guid on dynamic backend creation MINOR: proxy: check default proxy compatibility on "add backend" MEDIUM: proxy: implement dynamic backend creation MINOR: proxy: assign dynamic proxy ID REGTESTS: add dynamic backend creation test BUG/MINOR: proxy: fix clang build error on "add backend" handler BUG/MINOR: proxy: fix null dereference in "add backend" handler BUG/MINOR: proxy: fix default ALPN bind settings BUG/MINOR: quic: ensure handshake speed up is only run once per conn BUG/MAJOR: quic: reject invalid token BUG/MAJOR: quic: fix parsing frame type BUG/MAJOR: Revert "MEDIUM: mux-quic: add BUG_ON if sending on locally closed QCS" BUG/MEDIUM: h3: reject frontend CONNECT as currently not implemented MINOR: mux-quic: add BUG_ON_STRESS() when draining data on closed stream REGTESTS: fix quoting in feature cmd which prevents test execution BUG/MINOR: backend: check delay MUX before conn_prepare() OPTIM: backend: reduce contention when checking MUX init with ALPN Aurelien DARRAGON (7): MINOR: filters: rework RESUME_FILTER_* macros as inline functions MINOR: filters: rework filter iteration for channel related callback functions MEDIUM: filters: use per-channel filter list when relevant MEDIUM: backend: make "balance random" consider tg local req rate when loads are equal BUG/MEDIUM: stats-file: fix shm-stats-file recover when all process slots are full BUG/MINOR: stats-file: manipulate shm-stats-file heartbeat using unsigned int BUG/MEDIUM: stats-file: detect and fix inconsistent shared clock when resuming from shm-stats-file Christopher Faulet (36): BUG/MEDIUM: mux-h2/quic: Stop sending via fast-forward if stream is closed BUG/MEDIUM: mux-h1: Stop sending vi fast-forward for unexpected states BUG/MEDIUM: applet: Fix test on shut flags for legacy applets (v2) DEV: term-events: Fix hanshake events decoding BUG/MINOR: flt-trace: Properly compute length of the first DATA block MINOR: flt-trace: Add an option to limit the amount of data forwarded CLEANUP: compression: Remove unused static buffers BUG/MEDIUM: shctx: Use the next block when data exactly filled a block BUG/MINOR: http-ana: Stop to wait for body on client error/abort MINOR: stconn: Add missing SC_FL_NO_FASTFWD flag in sc_show_flags REORG: stconn: Move functions related to channel buffers to sc_strm.h MINOR: tree-wide: Use the buffer size instead of global setting when possible MINOR: buffers: Swap buffers of same size only BUG/MINOR: config: Check buffer pool creation for failures MEDIUM: cache: Don't rely on a chunk to store messages payload MEDIUM: stream: Limit number of synchronous send per stream wakeup MEDIUM: compression: Be sure to never compress more than a chunk at once MEDIUM: mux-h1/mux-h2/mux-fcgi/h3: Disable 0-copy for buffers of different size MEDIUM: applet: Disable 0-copy for buffers of different size MINOR: h1-htx: Disable 0-copy for buffers of different size MEDIUM: stream: Offer buffers of default size only BUG/MEDIUM: htx: Fix function used to change part of a block value when defrag MEDIUM: htx: Refactor transfer of htx blocks to merge DATA blocks if possible MEDIUM: htx: Refactor htx defragmentation to merge data blocks MEDIUM: htx: Improve detection of fragmented/unordered HTX messages MINOR: http-ana: Do a defrag on unaligned HTX message when waiting for payload MINOR: http-fetch: Use pointer to HTX DATA block when retrieving HTX body MEDIUM: dynbuf: Add a pool for large buffers with a configurable size MEDIUM: chunk: Add support for large chunks MEDIUM: stconn: Properly handle large buffers during a receive MEDIUM: sample: Get chunks with a size dependent on input data when necessary MEDIUM: http-fetch: Be able to use large chunks when necessary MINPR: htx: Get large chunk if necessary to perform a defrag MEDIUM: http-ana: Use a large buffer if necessary when waiting for body MINOR: dynbuf: Add helpers to know if a buffer is a default or a large buffer MINOR: config: reject configs using HTTP with large bufsize >= 256 MB David Carlier (16): BUG/MINOR: deviceatlas: add missing return on error in config parsers BUG/MINOR: deviceatlas: add NULL checks on strdup() results in config parsers BUG/MEDIUM: deviceatlas: fix resource leaks on init error paths BUG/MINOR: deviceatlas: fix off-by-one in da_haproxy_conv() BUG/MINOR: deviceatlas: fix cookie vlen using wrong length after extraction BUG/MINOR: deviceatlas: fix double-checked locking race in checkinst BUG/MINOR: deviceatlas: fix resource leak on hot-reload compile failure BUG/MINOR: deviceatlas: fix deinit to only finalize when initialized BUG/MINOR: deviceatlas: set cache_size on hot-reloaded atlas instance MINOR: deviceatlas: check getproptype return and remove pprop indirection MINOR: deviceatlas: increase DA_MAX_HEADERS and header buffer sizes MINOR: deviceatlas: define header_evidence_entry in dummy library header MINOR: deviceatlas: precompute maxhdrlen to skip oversized headers early CLEANUP: deviceatlas: add unlikely hints and minor code tidying BUG/MEDIUM: jwe: fix timing side-channel and dead code in JWE decryption BUG/MINOR: acme: fix X509_NAME leak when X509_set_issuer_name() fails Egor Shestakov (4): BUG/MINOR: startup: fix allocation error message of progname string BUG/MINOR: startup: handle a possible strdup() failure CLEANUP: initcall: adjust comments to INITCALL{0,1} macros CLEANUP: mux-h1: Remove unneeded null check Frederic Lecaille (9): BUG/MEDIUM: ssl: SSL backend sessions used after free MINOR: ssl/ckch: Move EVP_PKEY and cert code generation from acme MINOR: ssl/ckch: certificates generation from "load" "crt-store" directive MINOR: trace: add definitions for haterm streams MINOR: init: allow a fileless init mode MEDIUM: init: allow the redefinition of argv[] parsing function MINOR: stconn: stream instantiation from proxy callback MINOR: haterm: add haterm HTTP server MINOR: haterm: new "haterm" utility Ilia Shipitsin (1): CI: do not use ghcr.io for Quic Interop workflows Nenad Merdanovic (1): MEDIUM: Add connect/queue/tarpit timeouts to set-timeout Olivier Houchard (4): BUG/MINOR: threads: Initialize maxthrpertgroup earlier. BUG/MEDIUM: threads: Differ checking the max threads per group number MINOR: queues: Check minconn first in srv_dynamic_maxconn() MINOR: servers: Call process_srv_queue() without lock when possible Remi Tricot-Le Breton (4): MINOR: ssl: Missing '\n' in error message MINOR: jwt: Convert an RSA JWK into an EVP_PKEY MINOR: jwt: Add new jwt_decrypt_jwk converter REGTESTS: jwt: Add new "jwt_decrypt_jwk" tests William Lallemand (17): DOC: internals: addd mworker V3 internals MINOR: startup: Add the SSL lib verify directory in haproxy -vv BUG/MINOR: ssl: SSL_CERT_DIR environment variable doesn't affect haproxy MINOR: startup: Add HAVE_WORKING_TCP_MD5SIG in haproxy -vv MINOR: startup: sort the feature list in haproxy -vv MINOR: startup: show the list of detected features at runtime with haproxy -vv SCRIPTS: build-vtest: allow to set a TMPDIR and a DESTDIR BUG/MINOR: ssl: lack crtlist_dup_ssl_conf() declaration BUG/MINOR: ssl: double-free on error path w/ ssl-f-use parser BUG/MINOR: ssl: fix leak in ssl-f-use parser upon error BUG/MINOR: ssl: clarify ssl-f-use errors in post-section parsing BUG/MINOR: ssl: error with ssl-f-use when no "crt" CI: vtest: move the vtest2 URL to vinyl-cache.org CI: github: disable windows.yml by default on unofficials repo DOC: remove openssl no-deprecated CI image DOC: configuration: add the ACME wiki page link CI: github: only enable OS X on development branches Willy Tarreau (11): MINOR: net_helper: extend the ip.fp output with an option presence mask CLEANUP: lb-chash: free lb_nodes from chash's deinit(), not global BUG/MEDIUM: lb-chash: always properly initialize lb_nodes with dynamic servers CLEANUP: haproxy: fix bad line wrapping in run_poll_loop() MINOR: activity: support setting/clearing lock/memory watching for task profiling MEDIUM: activity: apply and use new finegrained task profiling settings MINOR: activity: allow to switch per-task lock/memory profiling at runtime DOC: proxy-proto: underline the packed attribute for struct pp2_tlv_ssl DEV: gdb: add a utility to find the post-mortem address from a core DEV: gdb: use unsigned longs to display pools memory usage MINOR: haterm: increase thread-local pool size ---
