Hi,
HAProxy 3.4-dev7 was released on 2026/03/20. It added 130 new commits
after version 3.4-dev6.
I know that we've been slow to respond to certain messages on the list
lately and some reviews are still pending. Some of us have been very busy
these last few weeks, first exchanging around all the topics that need
to be completed soon (feature freeze coming at the end of this month),
second, chasing some difficult bugs and dealing with some reports, and
third, helping those dealing with large scale attacks (you know who you
are, thanks for your patience), so this unfortunately leaves less time
for the rest.
Regardless, some progress was made in various areas, mainly focusing on
stability:
- 38 bugs were fixed in various areas, in part due to some reports from
people running code analysis, and some found while chasing others.
Given that the vast majority of them are minor, it seems to indicate
we're observing a stabilization here.
- JWT now supports ECDH-ES algorithms and supports EC certificates in
the jwt_decrypt_cert() converter.
- stream connectors: the internal stconn mechanism used to rely on
operation descriptors called "app_ops" which we've been carrying under
different names since 1.5 or so, and that has always been annoying to
handle since they maintained code for certain impossible cases just for
API purposes. Each attempt to get rid of that failed due to some
dependenies. Recently after some dedicated code had to be added just for
haterm, we had the intuition that we had probably reached the point
where we could get rid of this old thing. And we were right because it
was done. Good riddance! The stconn code is now way cleaner and does
not have to carry a combination of special cases anymore (shutdown
propagation used to be a real mess for example).
- memory profiling: it's still difficult to spot memory leaks when some
parts such as the SSL stack use a single function for allocation and
spread the free() calls as macros because call places are not the same
in both cases so there's no way to easily match a free to an alloc.
For this reason we've now introduced the notion of execution context,
which can be a sample fetch function, a converter, a filter, an applet,
a group of CLI keywords etc, and the memory profiling is now refined by
context. It's still not perfect of course, but it now becomes easier to
distinguish certain allocations because a same allocation function will
now have distinct lines by execution context. Initial tests are promising,
we'll see how it goes in the real world.
- debugging: it's becoming increasingly common to face unexploitable core
dumps. Sometimes the binaries do not match because packages were deployed
over a whole fleet but restarts are done by steps. Or it happens that in
containers it's difficult to extract some libs, etc. A good solution was
found to this: the global "set-dumpable" directive now has a new option
"libs" which asks haproxy to load all of its dependencies in memory and
to keep them there preciously. In case of a crash they will be in the
core, an a tool (libs-from-core) permits to extract them. This way, only
a core is needed, the binary, libs etc are no longer needed. Of course
it uses a bit more memory (a few MB to a few tens of MB depending on the
setups) but for the majority of deployments this will not even be noticed
(and this remains an opt-in anyway).
- mworker: a few refinements and updates were brought to the master-worker
code to make it more robust against certain startup races and corner
cases.
- filters: the compression filter was now split into two distinct filters,
one per direction. This will help implement decompression soon.
- H2: the maximum number of concurrent streams, that is configured via the
global tune.h2.fe.max-concurrent-streams directive, can now be made
dynamic and depend on the executing thread's load (still via the same
directive). The goal here is to smoothly reduce the number of connections
to avoid adding too much load instead of rejecting extraneous streams.
In local tests, the following setting allowed to significantly reduce
the run queue size, hence the request latency during attacks:
tune.h2.fe.max-total-streams 10000
tune.h2.fe.max-concurrent-streams 100 rq-load auto min 12
This says that until the run queue reaches the maximum number of entries
processed in one loop, we advertise 100 concurrent streams, and then we
progressively decrease over that level, whil never going below 12. As
usual there's no universally good value and you need to experiment to
find optimal settings for your infrastructure and sites.
- scripts: the haproxy-reload script was converted to POSIX shell for
better portability.
- promex: stick-table update counts are now reported as new metric
"haproxy_sticktable_local_updates". It will allow to monitor the update
rate.
- stats-file: the clock in the shared stats is now always monotonic, and
the startup offset has been removed from it to move to the processes,
so that restarting with a slightly off clock will no longer produce
confusion during the resyncronization.
- startup with -dt (traces) is processed slightly later. Normally it
should not have any visible effect. The goal is to later permit to
register traces at runtime or from Lua.
- the build with OpenSSL 4.0 is fixed now.
- the rest are a few doc updates (Egor, Tom), optimization for haterm
(Aleks).
There are still pending issues and reviews (and even patches to probably
be taken as-is). Given the low available time for all developers at the
moment, I'd kindly ask that new features are kept away from now till the
release so that we can focus on stabilizing what's already merged or
about to be merged. Fixes and tests are more important for now. Thanks!
Please find the usual URLs below :
Site index : https://www.haproxy.org/
Documentation : https://docs.haproxy.org/
Wiki : https://github.com/haproxy/wiki/wiki
Discourse : https://discourse.haproxy.org/
Slack channel : https://slack.haproxy.org/
Issue tracker : https://github.com/haproxy/haproxy/issues
Q&A from devs : https://github.com/orgs/haproxy/discussions
Sources : https://www.haproxy.org/download/3.4/src/
Git repository : https://git.haproxy.org/git/haproxy.git/
Git Web browsing : https://git.haproxy.org/?p=haproxy.git
Changelog : https://www.haproxy.org/download/3.4/src/CHANGELOG
Dataplane API :
https://github.com/haproxytech/dataplaneapi/releases/latest
Pending bugs : https://www.haproxy.org/l/pending-bugs
Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs
Code reports : https://www.haproxy.org/l/code-reports
Latest builds : https://www.haproxy.org/l/dev-packages
Willy
---
Complete changelog :
Aleksandar Lazic (1):
OPTIM: haterm: use chunk builders for generated response headers
Alexander Stephan (1):
BUG/MINOR: mworker/cli: fix show proc pagination losing entries on resume
Amaury Denoyelle (2):
BUG/MAJOR: h3: check body size with content-length on empty FIN
BUG/MEDIUM: h3: reject unaligned frames except DATA
Aurelien DARRAGON (5):
MINOR: flt_http_comp: define and use proxy_get_comp() helper function
MEDIUM: flt_http_comp: split "compression" filter in 2 distinct filters
CLEANUP: flt_http_comp: comp_state doesn't bother about the direction
anymore
MEDIUM: stats-file/clock: automatically update now_offset based on shared
clock
MINOR: promex: export "haproxy_sticktable_local_updates" metric
Christopher Faulet (37):
BUG/MINOR: stconn: Increase SC bytes_out value in se_done_ff()
BUG/MINOR: ssl-sample: Fix sample_conv_sha2() by checking EVP_Digest*
failures
BUG/MINOR: backend: Don't get proto to use for webscoket if there is no
server
MINOR: stconn: Call sc_conn_process from the I/O callback if
TASK_WOKEN_MSG state was set
MINOR: mux-h2: Rely on h2s_notify_send() when resuming h2s for sending
MINOR: mux-spop: Rely on spop_strm_notify_send() when resuming streams
for sending
MINOR: muxes: Wakup the data layer from a mux stream with TASK_WOKEN_IO
state
MAJOR: muxes: No longer use app_ops .wake() callback function from muxes
MINOR: applet: Call sc_applet_process() instead of .wake() callback
function
MINOR: connection: Call sc_conn_process() instead of .wake() callback
function
MEDIUM: stconn: Remove .wake() callback function from app_ops
MINOR: check: Remove wake_srv_chk() function
MINOR: haterm: Remove hstream_wake() function
MINOR: stconn: Wakup the SC with TASK_WOKEN_IO state from opposite side
MEDIUM: stconn: Merge all .chk_rcv() callback functions in sc_chk_rcv()
MINOR: stconn: Remove .chk_rcv() callback functions
MEDIUM: stconn: Merge all .chk_snd() callback functions in sc_chk_snd()
MINOR: stconn: Remove .chk_snd() callback functions
MEDIUM: stconn: Merge all .abort() callback functions in sc_abort()
MINOR: stconn: Remove .abort() callback functions
MEDIUM: stconn: Merge all .shutdown() callback functions in sc_shutdown()
MINOR: stconn: Remove .shutdown() callback functions
MINOR: stconn: Totally app_ops from the stconns
MINOR: stconn: Simplify sc_abort/sc_shutdown by merging calls to
se_shutdown
DEBUG: stconn: Add a CHECK_IF() when I/O are performed on a orphan SC
BUG/MEDIUM: stconn: Don't forget to wakeup applets on shutdown
BUG/MINOR: spoe: Properly switch SPOE filter to WAITING_ACK state
BUG/MEDIUM: spoe: Properly abort processing on client abort
BUG/MEDIUM: stconn: Fix abort on close when a large buffer is used
BUG/MEDIUM: stconn: Don't perform L7 retries with large buffer
BUG/MINOR: h2/h3: Only test number of trailers inserted in HTX message
MINOR: htx: Add function to truncate all blocks after a specific block
BUG/MINOR: h2/h3: Never insert partial headers/trailers in an HTX message
BUG/MINOR: http-ana: Swap L7 buffer with request buffer by hand
BUG/MINOR: stream: Fix crash in stream dump if the current rule has no
keyword
BUG/MINOR: spoe: Fix condition to abort processing on client abort
BUILD: spoe: Remove unsused variable
Egor Shestakov (1):
DOC/CLEANUP: config: update mentions of the old "Global parameters"
section
Maxime Henrion (1):
MINOR: traces: defer processing of "-dt" options
Mia Kanashi (1):
BUG/MINOR: jws: fix memory leak in jws_b64_signature
Olivier Houchard (2):
BUG/MEDIUM: ssl: Handle receiving early data with BoringSSL/AWS-LC
BUG/MEDIUM: ssl: Don't report read data as early data with AWS-LC
Remi Tricot-Le Breton (8):
BUG/MINOR: jwt: Missing 'jwt_tokenize' return value check
MINOR: jwt: Improve 'jwt_tokenize' function
MINOR: jwt: Convert EC JWK to EVP_PKEY
MINOR: jwt: Parse ec-specific fields in jose header
MINOR: jwt: Manage ECDH-ES algorithm in jwt_decrypt_jwk function
MINOR: jwt: Add ecdh-es+axxxkw support in jwt_decrypt_jwk converter
MINOR: jwt: Manage ec certificates in jwt_decrypt_cert
DOC: jwt: Add ECDH support in jwt_decrypt converters
Tim Duesterhus (1):
BUG/MINOR: tcpcheck: Fix typo in error error message for `http-check
expect`
Tom Braarup (1):
DOC: configuration: http-check expect example typo
William Lallemand (20):
BUG/MINOR: admin: haproxy-reload use explicit socat address type
MEDIUM: admin: haproxy-reload conversion to POSIX sh
BUG/MINOR: admin: haproxy-reload rename -vv long option
MINOR: mworker: add a BUG_ON() on mproxy_li in _send_status
BUG/MINOR: mworker: don't set the PROC_O_LEAVING flag on master process
Revert "BUG/MINOR: jwt: Missing 'jwt_tokenize' return value check"
MEDIUM: mworker: exiting when couldn't find the master mworker_proc
element
BUILD: ssl: use ASN1_STRING accessors for OpenSSL 4.0 compatibility
BUILD: ssl: make X509_NAME usage OpenSSL 4.0 ready
BUG/MINOR: mworker: always stop the receiving listener
BUG/MINOR: mworker: only match worker processes when looking for
unspawned proc
BUG/MINOR: mworker: fix typo &= instead of & in proc list serialization
BUG/MINOR: mworker: set a timeout on the worker socketpair read at startup
BUG/MINOR: mworker: avoid passing NULL version in proc list serialization
BUG/MINOR: sockpair: set FD_CLOEXEC on fd received via SCM_RIGHTS
BUG/MINOR: mjson: make mystrtod() length-aware to prevent out-of-bounds
reads
BUG/MINOR: mworker: don't try to access an initializing process
CI: github: treat vX.Y.Z release tags as stable like haproxy-* branches
BUG/MINOR: mworker: fix sort order of mworker_proc in 'show proc'
CLEANUP: mworker: fix tab/space mess in mworker_env_to_proc_list()
Willy Tarreau (49):
SCRIPTS: git-show-backports: hide the common ancestor warning in quiet
mode
SCRIPTS: git-show-backports: add a restart-from-last option
BUILD: makefile: fix range build without test command
BUG/MINOR: memprof: avoid a small memory leak in "show profiling"
BUG/MINOR: proxy: do not forget to validate quic-initial rules
MINOR: activity: use dynamic allocation for "show profiling" entries
MINOR: tools: extend the pointer hashing code to ease manipulations
MINOR: tools: add a new pointer hash function that also takes an argument
MINOR: memprof: attempt different retry slots for different hashes on
collision
MINOR: tinfo: start to add basic thread_exec_ctx
MINOR: memprof: prepare to consider exec_ctx in reporting
MINOR: memprof: also permit to sort output by calling context
MINOR: tools: add a function to write a thread execution context.
MINOR: debug: report the execution context on thread dumps
MINOR: memprof: report the execution context on profiling output
MINOR: initcall: record the file and line declaration of an INITCALL
MINOR: tools: decode execution context TH_EX_CTX_INITCALL
MINOR: tools: support decoding ha_caller type exec context
MINOR: sample: store location for fetch/conv via initcalls
MINOR: sample: also report contexts registered directly
MINOR: tools: support an execution context that is just a function
MINOR: actions: store the location of keywords registered via initcalls
MINOR: actions: also report execution contexts registered directly
MINOR: filters: set the exec context to the current filter config
MINOR: ssl: set the thread execution context during message callbacks
MINOR: connection: track mux calls to report their allocation context
MINOR: task: set execution context on task/tasklet calls
MINOR: applet: set execution context on applet calls
MINOR: cli: keep the info of the current keyword being processed in the
appctx
MINOR: cli: keep track of the initcall context since kw registration
MINOR: cli: implement execution context for manually registered keywords
MINOR: activity: support aggregating by caller also for memprofile
MINOR: activity: raise the default number of memprofile buckets to 4k
DOC: internals: short explanation on how thread_exec_ctx works
MINOR: tools: add a function to create a tar file header
MINOR: tools: add a function to load a file into a tar archive
MINOR: config: support explicit "on" and "off" for "set-dumpable"
MINOR: debug: read all libs in memory when set-dumpable=libs
DEV: gdb: add a new utility to extract libs from a core dump:
libs-from-core
MINOR: debug: copy debug symbols from /usr/lib/debug when present
MINOR: debug: opportunistically load libthread_db.so.1 with
set-dumpable=libs
BUG/MEDIUM: peers: enforce check on incoming table key type
BUG/MINOR: mux-h2: properly ignore R bit in GOAWAY stream ID
BUG/MINOR: mux-h2: properly ignore R bit in WINDOW_UPDATE increments
MINOR: freq_ctr: add a function to add values with a peak
MINOR: task: maintain a per-thread indicator of the peak run-queue size
MINOR: mux-h2: store the concurrent streams hard limit in the h2c
MINOR: mux-h2: permit to moderate the advertised streams limit depending
on load
MINOR: mux-h2: permit to fix a minimum value for the advertised streams
limit
---
