Hi,
HAProxy 2.7.0 was released on 2022/12/01. It added 41 new commits
after version 2.7-dev10.
There was really nothing fancy over this last week. A handful of minor bug
fixes (possibly the last encountered cause for QUIC crashes was addressed),
minor code cleanups, various doc updates, and a few harmless improvements
in debug-oriented features (e.g. debug dev anon and memstats no longer
require expert mode). Those who were rebuilding their own SSL lib might
notice that they will no longer see the warning about not being able to
load the system CA files (those used by the httpclient). And a pair of
ifdefs were added to allow to build with QUIC and WolfSSL together.
Now let's see what we've done since 2.6.0 without entering too much into
details.
The new bandwidth limitation filters were merged. The very first day
Christopher showed me his filter design 7 years or so ago, I remember
saying "oh cool, with this I'll finally be able to implement traffic
shaping". But there has always been something more important to work
on and I never did it. Christopher finally did it, in combination with
stick-tables to handle traffic classes, so that it is possible to limit
both a stream or a class of streams (e.g. bandwidth per URL or per /24
source network etc). The bandwidth limitation can be applied from
constants or even using sample expressions so that it can be the result
of a complex calculation or just be extracted from a map or a header.
The thread groups that I couldn't finish in time for 2.6 are now ready.
With this, it's possible not only to break the 64-thread barrier, but
also to be much more efficient on NUMA systems or processors by splitting
the threads into multiple independent groups. Very little is shared
between these groups so the impact of such machines is reduced a lot.
In addition, we've finally enabled exponential back-off in our locks
so as to increase the fairness between multiple groups of cores. This
means that users who were seeing the watchdog trigger from time to time
on EPYC processors when using more cores than fit in a single core
complex will see a significant performance and stability improvement.
Speaking of threads, stick-tables locking was relaxed a lot, with
measured performance gains up to 11x on some heavy workloads involving
track-sc rules on large systems. Before 2.7 they were still relying on
the simple locking that was brought in 1.8, and that sensitive code had
never been revisited since. Testing on large systems showed that it was
possible to spend 95% of the time waiting for a lock, that's not
acceptable anymore! This rework significantly improved the situation.
I'm convinced we can still do better, but it will require some important
and delicate changes that do not exactly make me feel excited. I'd rather
work first on improving the queues to benefit from thread-groups.
QUIC received something like 252 patches total since 2.6.0, many of
which were fixes, and others being infrastructure updates to better
address certain limitations. At the time of writing this, the last
reported bug was addressed, though there are still a number of important
improvements to come. All these patches were already backported to the
latest 2.6 versions once sufficiently tested because we said that we'd
keep 2.6 on par with 2.7 given the youth of the implementation. Now
that 2.7 is released and that QUIC starts to work pretty well, we'll
backport much less QUIC stuff to 2.6 (essentially fixes), though we
don't exclude doing some batches from time to time, especially if it
becomes necessary to backport some fixes. 2.7 will continue to receive
more frequent updates though. So if you're just curious about QUIC, 2.6
will be calm and fine for you. If you're really interested in QUIC, 2.7
is a better choice. And if you want to be the leader, running on the
development branch will always bring the latest improvements.
There were quite some updates on the SSL front, in relation with QUIC.
Those interested in QUIC remember that the OpenSSL project rejected [1]
after 2.5 years the de-facto standard API that *all* other stacks have
adopted (BoringSSL, QuicTLS, LibreSSL, GNUTLS and WolfSSL at least) and
that is necessary for any agent to implement QUIC. This forces our users
to rely on the QuicTLS fork that they have to build themselves since it's
not packaged in distros, and this is not fun at all. Some progress seems
to be slowly happening with some distros having at least discussed the
opportunity of integrating this patch set [2]. But for now the outcome
is uncertain, and this combined with the outstanding performance
regressions [3] between OpenSSL 1.1.1 and OpenSSL 3.0 made us think that
it has become really urgent to seriously consider alternatives given our
sensitive dependency on this component that is isolating itself from the
rest of the world. Fortunately, the work done over the years to support
some forks of OpenSSL (BoringSSL and LibreSSL) has already taught us to
have a minimal compatibility layer that makes it easier to selectively
enable or disable certain features. During the 2.7 development period,
LibreSSL integrated the QUIC patchset, and we could adapt, test and
exchange with them on the subject. This patchset is now released in
experimental status in LibreSSL 3.6, and usable in HAProxy 2.7. This
means that operating systems which provide LibreSSL 3.6 with QUIC
support enabled should already be able to support QUIC with HAProxy
without effort. In addition, we've recently had very fruitful exchanges
with the WolfSSL project who was very responsive in implementing the
few things that were missing to support HAProxy, and we obviously did
the same on our side. As a result, HAProxy 2.7 can also be built with
the very latest development branch of WolfSSL. The support is still
very early and a number of features still don't work, but we could
at least verify that QUIC does work, so there is great hope on this
front as well in the near term. Honestly at the moment it is too early
to be used yet but we wanted to integrate this early so as to reduce
the burden for the WolfSSL team. Given the activity I'm observing right
now as I'm typing, I wouldn't be surprised if in a few weeks from now we
already see something very satisfying. We'll see. But seeing the light
at the end of the tunnel is encouraging. Whether it will be with or
without OpenSSL starts to be less of a concern now. Finally, another
update on the SSL part is that the ca-ignore-err and crt-ignore-err
now also support symbolic names instead of numerical values found in
the man page, and that x509_v_err_str() returns these names from an
error code. This allows to provide stable names in configs and logs
across SSL library variants and versions.
Small updates happened to Lua. The most visible ones concern the ability
to pass arguments to a script at the moment it's loaded in the config.
We've seen many cases where users either were presetting environment
variables to use there, or directly modifying the scripts. Neither is
convenient, and passing arguments at load time is the most natural
approach since it makes the script work like any regular program or shell
script.
We've improved the debugging stuff, as always, with the aim of reducing
the number of round trips between bug reporters and developers, and
exposing the least possible users to bugs in the process of being
diagnosed. One thing we noticed that was taking some users a lot of
time was redacting configs and dumps to remove section names that may
disclose hosted customers or internal IP addresses. In order to help
here, we've added native support for anonymizing configurations and
dumps. For configurations, running passing "-dC" on the command line
will dump the current configuration without comments. Adding a key
like "-dC1234" will hash sensitive stuff (IPs, names etc) and trim
extra arguments that may contain sensitive info. This is often sufficient
for developers to get a detailed idea of the enabled options without the
user having to trim too much of it. And on the CLI, entering "set anon on"
will also do the same for the dumps, using the configured key, producing
the same hashes that developers can match against the config. The hash
is very small (24 bits) so that it collides sufficiently not to be
reversible. If we were to find that it's still too much, it could be
further reduced. Two other features were made to help with debugging.
Now it's possible to enable traces directly in the config section so
that it's no longer necessary to script some commands on the CLI upon
restart, and it's possible to map a ring to a file so that traces can
be sent there and retrieved post-mortem without having to connect a
socat, tail or whatever to the current process. These ideas came from
discussions with bug reporters, do not hesitate to continue to share
concerns, difficulties and ideas to improve your experience.
In master-worker mode, the CLI will now wait for a "reload" action to
complete, and produce the startup logs on output. Previously the reload
was a bit blind, it was not possible to know what happened without
reconnecting, and since 2.5 those startup logs were not even readable
anymore. This should significantly simplify config deployment tools.
Furthermore, my colleage Marko Juraga who's in charge of the Data Plane
API project presented at HAProxyConf last month a much welcome change
of release cycle of his project: it turns out that the Data Plane API
was following a different version numbering scheme and was reaching 2.7
just now while they were finishing the support for HAProxy 2.7's new
stuff, so he decided that from now on the Data Plane API would follow
the same version numbering scheme as HAProxy so that it's way more
convenient for users (i.e. you'll need DPAPI 2.8 for HAProxy 2.8 and
so on). I wanted to mention it here so that those who might have tested
previous versions can give it a new try. The download link was added to
the long list of links at the end of the message.
One less visible change that could possibly open new possibilities is
the introduction of shards in the peers sections. The idea is that for
users with extreme peers traffic, it's possible to send only portions
of the traffic to different peers, based on the table name and the key.
This is particularly useful when all the updates are sent to central
components that are expected to take actions or aggregate data at levels
that do not fit into a single machine anymore. I'm still suspecting that
it might have other benefits in certain large-cluster deployments where
it could save a lot of RAM by making each peer store less info, and
reduce the total network traffic from N*(N-1) to N, but for now I'm not
yet clearly seeing the exact use case. However I wouldn't be surprised
if we'd find new use cases for this in the near future.
Finally a few deprecated keywords were now dropped as with each version
("process" and "bind-process" on "bind" lines and frontends).
For more details and usage examples, my colleagues Nick and Senad are
in the process of finishing a more in-depth article on the HAProxyTech
blog about this release with examples that will be posted here:
https://www.haproxy.com/blog/
That's about all from me now. We already have some pending stuff in the
-next branch to open 2.8-dev, that's great! As a reminder, HAProxy 1.8
is reaching its end of life. We'll probably emit a last one and close
it, marking the end of 1.x versions. I'd like to sincerely thank all
participants (code, discussions, bug reporting and triaging, tests, CI,
package maintenance, doc, user help, etc). Continue to congratulate the
people you meet who wear a HAProxy T-shirt, they really deserve it! I
hope I didn't mess up with the release, if you get some 404 or 403, as
usual, do not hesitate to report them so that I can fix them ASAP.
Please find the usual URLs below :
Site index : https://www.haproxy.org/
Documentation : https://docs.haproxy.org/
Wiki : https://github.com/haproxy/wiki/wiki
Discourse : https://discourse.haproxy.org/
Slack channel : https://slack.haproxy.org/
Issue tracker : https://github.com/haproxy/haproxy/issues
Sources : https://www.haproxy.org/download/2.7/src/
Git repository : https://git.haproxy.org/git/haproxy-2.7.git/
Git Web browsing : https://git.haproxy.org/?p=haproxy-2.7.git
Changelog : https://www.haproxy.org/download/2.7/src/CHANGELOG
Dataplane API :
https://github.com/haproxytech/dataplaneapi/releases/latest
Pending bugs : https://www.haproxy.org/l/pending-bugs
Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs
Code reports : https://www.haproxy.org/l/code-reports
Latest builds : https://www.haproxy.org/l/dev-packages
Have fun!
Willy
[1] https://github.com/openssl/openssl/pull/8797
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1011391
[3] https://github.com/openssl/openssl/issues/17627#issuecomment-1060123659
---
Complete changelog from 2.7-dev10:
Amaury Denoyelle (3):
CLEANUP: ncbuf: remove ncb_blk args by value
CLEANUP: ncbuf: inline small functions
CLEANUP: ncbuf: use standard BUG_ON with DEBUG_STRICT
Christopher Faulet (2):
BUG/MEDIUM: mux-h1: Close client H1C on EOS when there is no output data
MINOR: mux-h1: add the expire task and its expiration date in "show fd"
Frédéric Lécaille (1):
BUG/MINOR: quic: Endless loop during retransmissions
Ilya Shipitsin (1):
CLEANUP: assorted typo fixes in the code and comments
Stefan Eissing (1):
BUILD: quic: allow build with USE_QUIC and USE_OPENSSL_WOLFSSL
William Lallemand (2):
MINOR: ssl: forgotten newline in error messages on ca-file
BUG/MINOR: ssl: shut the ca-file errors emitted during httpclient init
Willy Tarreau (31):
DOC: config: provide some configuration hints for "http-reuse"
DOC: config: refer to section about quoting in the "add_item" converter
DOC: halog: explain how to use -ac and -ad in the help message
DOC: config: clarify the fact that SNI should not be used in HTTP
scenarios
DOC: config: mention that a single monitor-uri rule is supported
DOC: config: explain how default matching method for ACL works
DOC: config: clarify the fact that "retries" is not just for connections
BUILD: halog: fix missing double-quote at end of help line
DOC: config: clarify the -m dir and -m dom pattern matching methods
MINOR: activity: report uptime in "show activity"
REORG: activity/cli: move the "show activity" handler to activity.c
DEV: poll: add support for epoll
DEV: tcploop: centralize the polling code into wait_for_fd()
DEV: tcploop: add support for POLLRDHUP when supported
DEV: tcploop: do not report an error on POLLERR
DEV: tcploop: add optional support for epoll
SCRIPTS: announce-release: add a link to the data plane API
CLEANUP: stick-table: fill alignment holes in the stktable struct
MINOR: stick-table: store a per-table hash seed and use it
MINOR: stick-table: show the shard number in each entry's "show table"
output
MINOR: mux-h2: add the expire task and its expiration date in "show fd"
BUG/MINOR: peers: always initialize the stksess shard value
REGTESTS: fix peers-related regtests regarding "show table"
MINOR: stick-table: change the API of the function used to calculate the
shard
CLEANUP: peers: factor out the key len calculation in received updates
BUG/MINOR: peers: always update the stksess shard number on incoming
updates
MINOR: debug: improve error handling on the memstats command parser
CLEANUP: anon: clarify the help message on "debug dev hash"
MINOR: debug: relax access restrictions on "debug dev hash" and "memstats"
SCRIPTS: run-regtests: add a version check
MINOR: version: mention that it's stable now
---
