Hi,
HAProxy 3.2-dev8 was released on 2025/03/21. It added 119 new commits
after version 3.2-dev7.
As mentioned in the 3.1.6 announcement, a few bugs were addressed, but
nothing critical.
For the new stuff:
- automatic CPU binding (formerly known as "NUMA patches"): this work that
started almost two years ago and which I hoped to see merged into each
version since 2.9 was finally completed! This extends the current CPU
topology detection to better bind threads and thread groups. First, by
default, nothing will change in 3.2 compared to previous versions. The
new features will consist in detecting the detailed CPU topology, hence
nodes, packages, CCX, L3 caches, cores, clusters, threads, etc and do
the best to optimally bind to them and arrange the groups to limit the
costly inter-CCX communications. It comes with a "cpu-set" directive
that allows to only bind to, or exclude, certain CPUs based on their
node/core/thread/cluster number. For example if one wants to only bind
to odd or even threads to leave the other ones for the NIC drivers,
it is trivial to do with a single directive. Second, another directive,
"cpu-policy", describes how to use the selected CPUs. The default one,
"first-usable-node", does exactly like today, i.e. it will only bind
to the first node with available CPUs and limit itself to a single
group and 64 threads max. Another policy is "group-by-cluster", it
will create one thread group per CCX/L3 cache and configure as many
threads as there are enabled CPUs on them. It can also create multiple
groups if there are more than 64 CPUs in one of them. It's possible
that it will be come the default policy starting with 3.3, as it can
use the full machine in an efficient way. Just using this one was
sufficient to multiply the performance by 3 on a 64-core EPYC, i.e.
it was the same as what can be achieved using precise "cpu-map"
directives which become quite difficult to use with many-core systems.
A few other policies are available for CPUs with P+E cores to prefer
"Performance" cores or "Efficiency" cores.
We're interested in feedback from those dealing with large systems,
particularly multi-socket ones, as well as VMs and containers, to
make sure we haven't missed anything. Many tests were run on about
20-25 different systems, as well as emulations of about 10 other
ones based on /sys captures. For those who prefer, I have created
a discussion here on GitHub, feel free to participate and share
feedback (successes, failures and suggestions):
https://github.com/orgs/haproxy/discussions/2901
- Prometheus and stats convergence: those using Prometheus probably
noticed it from time to time, it's difficult to keep the two
synchronized, so sometimes we add some new stats and forget to
do the same to Prometheus. Some changes were made to extend the
stats internal representation so that Prometheus can rely on this.
This way there is now a single place to declare new metrics that
should be exposed at the two places. If well done, it should not
change anything (actually the only thing is that the warnings
counter will finally be exported by Prometheus). Please give it
a try to confirm that everything runs as smoothly as expected.
- the log-forward sections now support an "option host" to decide
how to fill the host part of outgoing log messages (leave it as-is,
replace it, append), since different users expect different behaviors.
- some new converters are provided to support JWS signing and verify
JSON Web Token (JWT). Please just bear with me, I have zero idea
about what JWS means nor what it's used for, but there are info in
the doc about it :-) Apparently it's related to authentication.
- some changes were made to the internal representation of certificates
that are not expected to have any visible effect. If you're using
complex setups, please give it a quick try to verify that you don't
face any error at load time.
- the "wait ... srv-removable" CLI command was optimised so that it
consumes much less CPU while waiting for a server to be removable.
It used to force thread isolation during the check but thanks to
some recent changes this is no longer necessary, so those with
many servers being constantly added and removed at run time and
who used to notice CPU spikes when a whole farm went down will see
a significant improvement.
- a small "show pools detailed" CLI command will now show all pools
registered behind a single entry. That's useless for normal users
but developers might ask about this in the future when chasing a
memory error.
- we found a case on a 128-thread EPYC where some watchdog warnings
could be emitted from time to time under extreme contention on the
mt_lists, indicating that some CPUs were blocked for
