Hi,
HAProxy 3.5-dev11 was released on 2026/11/04. It added 96 new commits
after version 3.5-dev10.
We're starting to receive bug reports, indicating that some users are
testing, though some are also for 3.4, which is less fun as it forces to
do a lot of context-switching between finalizing -dev and fixing bugs. I
know some issues can lag a bit, I'm sorry and we're trying to deal with
the incoming stream as well as we can. The good thing is that the feature
freeze now relieves a bit of pressure.
In this version, aside bug fixes, it's reasonably calm, as expected:
- slz update: it fixes the crashing bugs of dev10, and incidentally
slightly improves the compression ratio and speed. The new text-aware
compressor was finally merged, gaining 15-20% on "text/" content
types.
- HTTP hardening: as mentioned last week, this is the complement to
the violation exceptions. There are a number of gray areas in HTTP
which nowadays are suggested to be rejected when received, but that
haproxy cannot systematically do by default, at the risk of breaking
the many legacy applications running in internal networks (tried many
times). On the other hand, it's also annoying when certain features
are used from hostile clients to harm servers. So here what is done
is that for each incoming and outgoing connection, we assign a
hardening profile which defines how to react on certain border-line
features. The choice is made either via the "http-hardening" keyword
on the "bind" and "server" lines, or by default: servers default to
profile "trusted", while incoming connections will be assigned to
profile "trusted" when coming from a private network, or "untrusted"
when coming from a public network. "trusted" for now doesn't enable
any hardening rule, while "untrusted" enables a few. Profiles can be
modified, added, an even changed at runtime (i.e. instantly block or
unblock something). We continue to remain extremely conservative, and
the only things that are rejected by default from untrusted networks
are things that regular clients never do and that are systematically
emitted by attackers when found on the net:
- mixed content-length + transfer-encoding (already normalized when
it reaches us but could confuse downstream proxies/caches) ;
- content-length with leading zeroes, sometimes parsed as octal
(already normalized when it reaches us but can confuse downstream
proxies/caches)
- presence of control characters in headers or trailers (used to
attack backend servers or connections), till now we'd only block
the strictly required ones by default (CR, LF, NUL) ;
- folding of long header/trailer lines: has been deprecated for 12
years and not used by any client for even more. Was already
normalized by us but is used to confuse downstream proxies and
caches.
- lower case characters in method: no official method uses them,
and some servers mistakenly take methods for case insensitive,
causing semantic confusion and desynchronization (think "Head").
The other ones are a bit more involved (reject/delete headers "X-",
underscores in header names, removal of trailers etc) are suggested
but shouldn't be done by default yet. I think we could enable the
rejection of duplicate Host header as well by default, as many public
servers already reject it without issues. It is possible to see what
is enabled via "show http-hardening" and change it via "set
http-hardening" on the CLI.
These have been running on haproxy.org with everything turned on over
the past week without issues. I expect that such a mechanism can help
all of us buy a bit more time when facing the next waves of AI-driven
reports against HTTP implementations. More checks are clearly needed
and will be added, and now they're relatively easy to add (and I have
a check list of missing ones that I'm routinely completing). What's
still missing is the reporting: what was seen and what was fixed is
reported in a bit field but not yet exposed in logs nor as sample
fetches. That's the easy part but everything takes a bit of time. The
doc is still fairly light IMHO, I need to improve it. One thing I'm
happy about is that it's the first feature which uses the exact same
keyword definition for config, CLI updates, reporting and even fuzzy
matching; the list provided by -dKcfg has the same source as the one
provided on the CLI.
- for the rest, we have a bunch of QUIC code cleanups, symbol resolver
improvements (hopefully I won't see WDT warnings anymore at -O0 on
haproxy.org when dumping memory profiling), some cleanups and
improvement in the build system that should help us better unify
makefiles between addons.
- worth noting, there's a workaround for a bug affecting kTLS in some
versions of OpenSSL. Apparently it was fixed in the most recent
branches. HAProxy will detect the bug (by version) and automatically
disable kTLS on such versions, with a message at boot explaining it.
There's still a keyword to forcefully enable it. Anyway usually if
you're using kTLS you're focused on performance and closely following
specific versions that work best for you, so you shouldn't be
impacted.
- On the CI front, AWS-LC FIPS is now enabled, and VTest was fixed in
the FreeBSD image.
- and of course reg-test and doc updates.
Regarding the new cache, we ran some extra benchmarks yesterday evening
and found some rare corner cases where we'll still have to tune between
CPU savings and hit ratio. That's a bit frustrating because it's almost
there, but at the same time I don't want to rush something at the last
minute to replace something which is not perfect but whose weaknesses are
well known by now. So we'll postpone and that will buy us more time to
make it even more robust and 100% safe to replace the existing
implementation.
What's not mentioned is likely still pending. There are still issues to
be dealt with here on the ML and on GitHub and some patches pending for
review. We still have approx 360 fixes for small issues with reproducers
pending in queue to be qualified, reviewed, merged, addressed differently
or postponed. I know that there are a few for the resolvers (not really
surprising) and the cache (not surprising either) there to be handled
before the small ones. The nice lesson from this is that beyond that
pain, we're preparing one of the best releases ever ;-)
This dev11 is safe again for deployment, so please go test it!
Please find the usual URLs below :
Site index : https://www.haproxy.org/
Documentation : https://docs.haproxy.org/
Wiki : https://github.com/haproxy/wiki/wiki
Discourse : https://discourse.haproxy.org/
Slack channel : https://slack.haproxy.org/
Issue tracker : https://github.com/haproxy/haproxy/issues
Q&A from devs : https://github.com/orgs/haproxy/discussions
Sources : https://www.haproxy.org/download/3.5/src/
Git repository : https://git.haproxy.org/git/haproxy.git/
Git Web browsing : https://git.haproxy.org/?p=haproxy.git
Changelog : https://www.haproxy.org/download/3.5/src/CHANGELOG
Dataplane API :
https://github.com/haproxytech/dataplaneapi/releases/latest
OpenTelemetry : https://github.com/haproxytech/haproxy-opentelemetry
Pending bugs : https://www.haproxy.org/l/pending-bugs
Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs
Code reports : https://www.haproxy.org/l/code-reports
Latest builds : https://www.haproxy.org/l/dev-packages
Willy
---
Complete changelog :
Aleksandar Lazic (1):
BUG/MEDIUM: leastconn: reset the smallest key when backups take over
Amaury Denoyelle (2):
BUG/MINOR: debug: fix "debug dev fd" crash on unknown address families
BUG/MEDIUM: proxy: fix stick table expired entries cleanup on stopping
Aurelien DARRAGON (1):
BUG/MINOR: checks/log: force implicit UDP checks as TCP for log backend
Chris Staite (3):
BUG/MEDIUM: ssl: skip TLS 1.3 suites in the FIPS TLS 1.2 cipher check
BUG/MINOR: ssl/ckch: free the new instance when ckch_inst_rebuild() fails
BUG/MEDIUM: ssl/cli: reset the ongoing transaction when "commit ssl cert"
is interrupted
Christopher Faulet (3):
DOC: configuration: Clarify wait-for-body doc for the response side
MINOR: http-act: Add 'no-fail' option on 'wait-for-body' action
BUG/MINOR: filters: Reject id-less implicit instances of multi-instance
classes
Frederic Lecaille (13):
CLEANUP: quic: remove duplicated key phase pointer assignments
MINOR: quic: use the right trace event in quic_tls_key_update()
CLEANUP: quic: remove duplicated rx offset initialization in
quic_cstream_new()
CLEANUP: quic: remove the unused tx part of struct quic_cstream
CLEANUP: quic: remove unused tx crypto size and offset counters
BUG/MINOR: quic: fix the header protection key length dump with AWS-LC
CLEANUP: quic: remove the unused key parameter from quic_tls_decrypt()
CLEANUP: quic: remove the unused count and flags from struct quic_tls_kp
CLEANUP: quic: remove the stale initial_salt declaration
CLEANUP: quic: remove duplicated quic_cstream_new() declaration
CLEANUP: quic: remove the unneeded quic_rx.h include from quic_tls.c
CLEANUP: quic: update the outdated comment of quic_tls_derive_keys()
BUG/MINOR: sample: realign the ungrpc converter on message boundaries
Martin Mirchev (3):
BUG/MINOR: sample: fix NULL dereference in the ltime/utime converters
BUG/MINOR: httpclient/lua: fix stack overread when copying the headers
BUG/MINOR: httpclient/lua: raise an error on too many request headers
Olivier Houchard (7):
BUILD: sym: Fix build on recent glibc
CLEANUP: sym: factor out the short object name extraction
MINOR: sym: use the symbol table to resolve a DSO name
BUG/MEDIUM: mux-h1: don't drop pending output on a short content-length
BUG/MINOR: haterm: fix the response body phase on short writes
BUG/MEDIUM: haload: restore request accounting after user timeouts
BUG/MEDIUM: ssl: disable kTLS on OpenSSL versions affected by a known bug
William Lallemand (10):
BUG/MINOR: errors: reset diagnostic buffers after thread cleanup
BUILD: Makefile: extend the capabilities of EXTRA_MAKE
MINOR: thread: add EXTRAS_LOCK so addons can name their locks
BUILD: Makefile: allow external makefiles to provide build targets
REGTESTS: ssl: skip AWS-LC FIPS rejected crypto in ciphersuites/curves
tests
CI: github: add AWS-LC FIPS testing workflow
BUILD: github: run FreeBSD CI on every branch
CI: github: cache VTest binary in FreeBSD workflow
BUG/MEDIUM: ssl/cli: don't unlock the certificate lock after a failed
trylock
MINOR: stream: add per-stream memory slots
Willy Tarreau (52):
BUG/MINOR: http-ana: fix response compression for applets
BUG/MEDIUM: compression: fix incompatibility between compression and
large-buffers
CLEANUP: tools: make inetaddr_host_lim*() usable on untrusted inputs
BUG/MINOR: flt-http-comp: check the block returned by
htx_replace_blk_value()
BUG/MINOR: compression: pass the configured level to slz
OPTIM: flt-http-comp: only flush the compressor after the last DATA block
IMPORT: slz: update slz to version 1.4-dev4
MINOR: compression: pass the content type to the algorithms' init function
MINOR: flt-http-comp: size the compression reserve using
SLZ_MAX_OVERHEAD()
REGTESTS: compression: check hard to compress contents and nearly full
buffers
DOC: config: add missing empty line before tune.ssl.ktls-unsafe
BUG/MINOR: ist: make ist_find_range() accept non-zero min boundaries
MINOR: ist: add ist_find_http_ctl() to match http CTL chars
MINOR: ist: add ist_find_outside() to find offending characters
MINOR: http: use ist_find_http_ctl() instead of ist_find_ctl()
MINOR: h1: indicate the trailers state to h1_parse_msg_tlrs()
MINOR: http: introduce the notion of http-hardening flags and sections
MINOR: cfgparse/http: add a simple parser for new section
"http-fingerprint"
MINOR: http: support default hardening names "untrusted" and "trusted"
MINOR: cfgparse: dump the "http-hardening" keywords with -dKcfg
MINOR: cli/http: add the "show http-hardening" CLI command
MINOR: listener: add the "http-hardening" bind keyword
MINOR: server: add the "http-hardening" server keyword
MINOR: tools: add is_ip_untrusted() to check for non-local addresses
DOC: config: document the "http-hardening" section and server/bind options
MINOR: http: define http_hdng_chk for ambient hardening checks to be run
MINOR: mux-h1: pass the profile's HTTP hardening checks into the h1c and
h1s
MINOR: mux-h2: pass the profile's HTTP hardening checks into the h2c and
h2s
MINOR: h3: pass the HTTP hardening checks to the h3s and h3c
MINOR: http: add the "set http-hardening" CLI command
MINOR: http/hardening: implement hardening "reject-duplicate-host-header"
MINOR: http/hardening: implement hardening "reject-lowercase-method"
MINOR: http/hardening: implement hardening
"reject-underscore-in-header-name"
MINOR: http/hardening: implement hardening
"reject-underscore-in-trailer-name"
MINOR: http/harderning: implement "delete-underscore-in-header-name"
MINOR: http/harderning: implement "delete-underscore-in-trailer-name"
MINOR: http/hardening: add "delete-unregistered-header-fields"
MINOR: http/hardening: add "delete-unregistered-trailer-fields"
MINOR: http/hardening: add "delete-all-trailer-fields"
MINOR: http/hardening: implement hardening "reject-header-trailer-folding"
MINOR: h1: rearrange the header value parser to better deal with CTL chars
MINOR: http: speed up http_header_has_forbidden_char() to make it work
alone
MINOR: http: don't call ist_find_http_ctl() before up
http_header_has_forbidden_char()
MEDIUM: http/hardening: implement hardening "reject-ctl-in-header-trailer"
MINOR: http/hardening: implement hardening "reject-content-length-with-te"
MINOR: http/hardening: implement hardening
"reject-content-length-leading-zero"
MINOR: http/hardening: add aliases "all-reject" and "all-delete"
MEDIUM: http: enable a few important hardening rules on untrusted
connections
REGTESTS: http-messaging: test the http-hardening checks on HTTP/1 headers
REGTESTS: http-messaging: test the http-hardening checks on HTTP/2 headers
REGTESTS: http-messaging: test the http-hardening checks on trailers
REGTESTS: http-messaging: test http-hardening profiles and their CLI
maxml (1):
BUG/MINOR: resolvers: validate TXT RDATA before reading
---
