HAProxy 3.0.0 was released on 2024/05/29. It added 21 new commits
after version 3.0-dev13. I do appreciate that everything was only

We're having a total of 1108 patches in this release among which 850 ones
not concerning a bug, which makes it the smallest LTS release of all times
(2.6 and 2.4 still remain the largest ones, respectively 65% and 58%
larger). This is a good news in terms of expected stability, which might
possibly break the old myth of "better avoid dot zero".

Let's try to summarize what's new in this release. It has been one of the
most difficult for me to summarize because I'm not seeing one big killer
feature, instead it's an LTS as we like them: mostly a nice polishing of
existing stuff and small improvements all over the place as permitted by
the previous version's architectural changes. I tried to classify this
into a few categories, depending on the intended benefits.

First, let's enumerate the new features, and improvements of existing ones:

  - stats can finally be preserved across reloads for frontends,
    listeners, backends and servers. When using this, the config objects
    of the new process are preloaded with the relevant values from a dump
    of the previous process. This essentially concerns counters, ages and
    rates. Please have a look at "stats-file" and "dump stats-file" for
    more information.
  - the log outgoing load-balancing now relies on a regular backend,
    meaning that the load balancing algorithms could finally be unified
    with the ones used by other protocols, and servers now support
  - log-format now supports JSON and CBOR output encoding. In such a case,
    the field name is taken from a new naming scheme that is placed within
    the log-format itself, allowing to assign a name to each field.
  - the load balancing algorithm "sticky" that was initially reserved for
    logs was generalized to other protocols.
  - the HTTP/2 RST_STREAM reason code can finally be forwarded to the
    server for client aborts. This addresses the problem a few users were
    facing with gRPC where request cancellation appeared as communication
    errors the server side. For now this is purposely limited to only a
    few reason codes that are relevant to gRPC so that we don't ruin the
    possibility to later extend that to H3 and maybe H1.
  - QUIC now supports the HyStart++ (RFC9406) alternative to slowstart
    with the Cubic algorithm. It's supposed to show better recovery
    patterns. It's not yet enabled by default.
  - a new set of converters, map_*_key, will report the matching part of
    the key itself instead of the associated pattern. The main target use
    cases for this is to know what address mask an address did match, or
    what regex a pattern did match.
  - the "uuid()" sample fetch function, which takes an optional version in
    argument now also supports "7" for UUIDv7. These UUIDs regroup many
    properties found in ULID and other mechanisms, one of the most
    interesting one being time-based locality that, for example, eases the
    archiving of old data, or the grouping of events on systems where
    they'll be processed together.
  - the name associated with servers in connection pools can now be
    overridden by the expression in "pool-conn-name" when SNI is not
    desired (useful with rhttp without SSL for example, but may also make
    sense when reaching remote servers over SSL tunnels). It also allows
    to entirely drop SSL from the server.
  - the "namespace" argument now works for "bind" and "server" lines using
    UNIX sockets.
  - Linux capabilities: the use of namespaces on the server side used to
    require capability "cap_sys_admin" but it was neither checked nor
    reported on startup to it would silently fail. The capability is now
    supported and is being checked for. Similarly, the need for
    capabilities for transparent proxying or QUIC are checked and reported
    on startup. Finally, file-system capabilities set on the executable are
    also supported now.
  - the set-mark/set-tos actions were extended to support an expression in
    addition of the constant, and were extended to also support the backend
    side. This can for example be used to select an outgoing link from a
    single IP address. The new backend actions are called "set-bc-mark" and
    "set-bc-tos", and by analogy new frontend actions called "set-fc-mark"
    and "set-fc-tos" were created, and the old actions are aliases of these
    last ones.
  - QUIC built with latest AWS-LC TLS library now correctly supports 0-RTT.
  - a new global setting "ssl-security-level" allows to adjust OpenSSL's
    internal security level beween 0 and 5. Previously it could only be
    done in openssl.cnf.
  - the key used by consistent hash to map to a server used to always be
    the server's id (either explicit or implicit, position-based), but
    that was not always convenient when dealing with fast added-removed
    server within a large fleet of LBs. Now the "hash-key" directive will
    also allow to use the server's address or address+port for this so
    that the same key ends up on the same server for all LBs.
  - The HTTP client now has an option to use either origin or absolute
    URIs. This should make it easier to configure it to talk to old
    servers which are not spec-compliant and do not support absolute
    URIs. The ocsp_update agent already exploits this ability via a new
    setting "ocsp-update.httpproxy".
  - it is now possible to suppress Content-Length and Transfer-Encoding
    headers from HTTP/1 requests and responses. It must never be done of
    course but there are rare situations where users dealing with bogus
    clients or server need to perform such cleanups. Most of the time
    when done, this will mark a connection non-reusable and it will be
    closed at the end of the transfer.
  - the proxy protocol now also parses TLV for LOCAL mode and supports
    sending them without a stream so that elements can be passed during
    the preconnect phase of a reverse-HTTP instance to a next stage that
    will no longer ignore them.
  - the new sched_setaffinity() of FreeBSD 14 and newer is now supported.
  - the new certificate selection callback for WolfSSL was now enabled
    since it's finally available in the upstream project.

Second, there were a reasonable set of usability improvements, all the
small features that make config management and day-to-day operations

  - maps are often used to operate at run time on some parts of the
    configuration. When no initial value is desired, it was still needed
    to have an empty file (/dev/null is not usable since a map is indexed
    by its name). As such, some users have expressed their desire to have
    virtual and/or optional maps. Both are brought by this version. When
    a map is loaded from a file whose name begins with "opt@", the file
    will only be loaded if it exists otherwise an empty map will be created
    with this name. And maps whose name begins with "virt@" are exclusively
    virtual and never backed by a file. They're always created empty at
    boot, for use at run time.
  - the default certificate selection method was improved: till now, the
    default certificate was the first one mentioned on the bind line. This
    causes issues with sites that want to support both RSA and ECDSA. A
    new approach was brought, with an optional "default-crt" keyword that
    designates the default certs on the bind line, and its equivalent in
    the crt-list files designated by "*" in the name. This allows the right
    cert to be picked based on the desired algorithm. Of course the default
    behavior doesn't change.
  - the list of status codes that increment the http_err_cnt and
    http_fail_cnt counters can now be changed with the global directives
    "http-err-codes" and "http-fail-codes". This has long been requested,
    both by those whose applications randomly return 500 that are not
    server failures, and those where 404 happen a lot and does not
    necessarily indicate a URL scanner. All of the 1xx-5xx range is
    permitted for both classes.
  - cookies, both static and dynamic, are now permitted for dynamically
    added servers.
  - API clients will find the CLI more friendly when it comes to removing
    a server. First, idle connections are now automatically closed when
    trying to delete a server, so that it's no longer needed to wait for
    them to vanish. Second, a new "wait" command pauses operations for at
    most as long as specified, optionally waiting for a condition. A new
    such condition is "srv-removable", which checks when a server may
    safely be removed. This means that issuing this "wait" command before
    a "del server" command will save the client from having to
    periodically retry the operation.
  - a new "crt-store" configuration section is supported. It allows to
    declare certificates by specifying the path for each element. The aim
    is essentially to decorellate the storage from the instantiation, both
    of which are currently correlated in crt-lists, and to allow easier
    specification of individual components. This section supports
    "crt-base" and "key-base" to ease the splitting of certificates and
    keys into distinct directories, as well as "ocsp-update" to indicate
    which certificates need to have their OCSP partperiodically updated.
    The certificates also support aliases so that they can be referenced
    from a bind line with a more convenient names than a file name.
    crt-lists may now make use of these certificates to only decide which
    ones to instantiate for a given listener, without having to deal with
    deployment concerns such as paths and file names.
  - the "thread-hard-limit" global parameter was added. It allows to only
    set a hard limit on the number of threads without enforcing that value
    as the thread count (like nbthread does). This is convenient to
    prepare portable configs with no more than X threads when one knows
    it's only a waste of resources to use more.
  - certain warnings about the presence of HTTP rules in TCP frontends
    that are going to be upgraded to HTTP when switching to a backend will
    now no longer be reported when it is certain that they will work as
  - a new "guid" keyword was added for servers, listeners and proxies.
    The purpose will be to make it possible for external APIs to assign a
    globally unique object identifier to each of them in stats dumps or
    CLI accesses, and to later reliably recognize a server upon reloads.
    One usage example right now is stats preservation across reloads where
    this GUID uniquely identifies a server between two configs.
  - it has become easier to pass extra CFLAGS / LDFLAGS to the Makefile,
    just pass them into these variables (and a few other ones). Many were
    removed as the result of the simplification. The removed ones will
    trigger a build warning indicating what to use instead. A warning will
    also be emitted when passing an unknown USE_* setting, and such
    settings now support to be set to zero to disable them.

In addition to this, some changes aim at improving the reliability:

  - the draining of HTTP/1 request body was finally implemented. It is
    needed when an early response is sent before the end of a POST
    request, typically due to a redirect or authentication issue. It used
    to cause difficulties due to the TCP stack emitting an RST that would
    sometimes destroy the response before it had a chance to be sent, but
    this is now something of the past.
  - the buffer allocator's behavior on out-of-memory condition was finally
    fixed. It had been flaky since version 1.7, with possibilities for all
    requesters to deadlock if none had enough room to complete their work.
    A new, more robust algorithm was finally implemented, making sure that
    at least one requester has enough resources to make forward progress
    and let the system recover by itself.

Other ones put a particular focus on robustness against various threats in

  - H2, H3 and QUIC now maintain a counter of per-connection glitches,
    which are characterized by not strictly illegal but suspicious or
    bogus protocol handling and behavior from a peer. Such counters are
    reported at upper layers, are trackable in stick-tables, and can be
    used to kill a misbehaving connection past a threshold. The goal here
    is to significantly reduce the CPU impact and log pollution caused by
    bots that blindly try to exploit various well-known vulnerabilities or
    limitations of some implementations. Since this works on both sides it
    can also be used to detect faulty applications that would need to be
  - H2 now supports to forcefully close connections after a configurable
    number of streams. This can be used to accelerate the switchover during
    reloads, as well as maintain an optimal balance between multiple front
    nodes, and force the re-evaluation of sanity checks at the connection
    level regarding tracked metrics to more easily get rid of abusers.
  - two new global settings now make it possible to simply prevent HAProxy
    from accepting traffic from privileged ports; one setting is for TCP
    and the other one for QUIC. QUIC was configured by default to refuse
    such traffic, because by relying on UDP it's particularly exposed to
    DNS and NTP amplification attacks, and while it's more efficient to
    filter such ports upstream, it's still very simple and cheap to just
    drop such undesirable packets before processing them.
  - the code no longer depends on libsystemd, so that we will not pull in
    a myriad of questionable dependencies anymore. This also allows to
    enable USE_SYSTEMD by default (it's only done on linux-glibc though),
    thus reducing configuration combinations.

As with every version comes a comprehensive collection of performance

  - quic: the fast-forwarding mechanism now considers the flow control
    state, resulting in a reduction of the number of wakeups and better
    filling of packets. The internal send API was reworked and simplified
    and one buffer copy could be removed. Some minor fixes and cleanups
    were done in the cubic congestion controller.
  - a new QUIC setting, "tune.quic.reorder-ratio" was added to let the
    user adjust the size of holes over the in-flight window before we
    declare a loss. Normally QUIC users should observe much better
    performance now, even with the default setting (50%), which was
    sufficient for us to observe x10-20 at 3% losses. The send path was
    improved and cleaned up, by using exclusively sendmsg() and avoiding
    some copies where possible. Some CPU savings are expected on intense
  - the H1 mux now also supports zero-copy forwarding for chunks of unknown
    size (i.e. those larger than a buffer).
  - the fast forward zero-copy mechanism is now supported by applets. This
    will ultimately result in lower memory usage and higher performance
    for some applets such as the cache by carefully avoiding to queue more
    data than the mux can take without buffering. This can still be
    disabled by unsetting tune.cache.zero-copy-forwarding.
  - a few ebtree backports improved the performance on non-x86 machines
    (typically ~2% faster string lookups were measured on ARM and ~3%
    task switching rate was measured).
  - some of the remaining server name lookups that were still linear moved
    to use the tree instead, speeding up certain operations or config
  - ring: the ring internal API used to represent a bottleneck for traces
    at TCP logs, especially on multi-threaded systems due to the initially
    unplanned locking that resulted from the underlying buffer API. All of
    this was entirely rewritten so that the code is almost lockfree and
    waiting threads can prepare their work as groups in parallel. The
    performance increased by a factor of 2.5 on NUMA systems and even by
    20 on uniform systems, reaching up to around 7 million messages per
    second. This is sufficient to enable traces at the "developer" level
    even on moderately loaded systems. The "haring" utility was updated to
    automatically detect the new, slightly different format and support
    both the old and the new ones (the old haring tool will still read the
    new format in repair mode).
  - stick-tables are now sharded over multiple tree heads each with their
    own locks. This significantly reduces locking contention on systems
    with many threads (gains of ~6x measured on a 80-thread systems). In
    addition, the locking could be reduced even with low thread counts,
    particulary when using peers, where the performance could be doubled.
    This is particularly noticeable when using the bandwidth limiting
    filter "bwlim".
  - The Lua latency with single-threaded scripts (loaded by "lua-load")
    running on multi-thread instances was improved a lot by reducing the
    amount of consecutive instructions a thread may run when there are
    many threads.

A few changes that improve observability:

  - a few more sample fetches corresponding to certain log-format aliases
    were added (txn.redispatched, bc_be_queue, bc_srv_queue, etc).
  - new sample fetch functions retrieve the number of concurrent streams
    over the same connection for a frontend or a backend, as well as the
    maximum number negotiated. This can be useful to sort out connection
    performance from stream performance when looking at timings in logs.
  - the Prometheus exporter now exposes a bunch of new metrics (resolvers,
    more server stuff) and supports applying filters to limit the metrics
    that have to be returned.

Some debugging aid to save experts time in field, speed up recovery and
reduce the number of round trips in issues:

  - stick-table operations over the CLI using commands like "show table",
    "set table" and "clear table" now supports a "ptr" argument to directly
    use the pointer retrieved from a previous "show" command. This is
    convenient to remove bogus entries manually for example.
  - haproxy -dD will now report suspicious ACL pattern values which look
    like known ACL/sample fetch keywords.
  - the "insecure-fork-wanted" option now has an equivalent on the command
    line, "-dI". It's convenient to obtain decoded ASAN outputs for
    example, without having to edit a config
  - QUIC and HTTP/3 added some traces, refined some error reporting, and
    improved the accuracy of the "show quic" output.
  - the backend equivalent of the frontend keylog mechanism was
    implemented, so that it is now possible to decipher TLS captures on
    the backend side.  The log-format to be used becomes a bit large,
    please refer to the example in the doc.
  - some internal large memory areas (file descriptor tables, HTTP and SSL
    session caches, ring buffers etc) now have a name that is visible on
    Linux >= 5.17 in /proc/$pid/maps or using pmap. This will help figure
    out where the memory is being used and why.
  - traces are way faster on multi-threaded systems thanks to the ring
    locking changes, making them usable without risks on moderately loaded
Some possibly (but unlikely) breaking changes:

  - an update of the DeviceAtlas addon was made to support the new version
    of the library. It slightly changes the build system but so far no issue
    was reported.
  - a mistake I accidentally introduced two years ago with a bug fix had
    the undesired side effect of randomly accepting chained commands on
    the CLI in non-interactive mode, when delimited by line feeds. The
    likelihood that it would work is essentially time-based, so a short
    string of multiple commands had great chances of working while a large
    one almost none. This started to cause side effects to other issues and
    had to be fixed, so that we no longer accept multiple commands delimited
    by '\n' in non-interactive mode, as documented. If you happen to have
    such scripts sending multiple commands this way, you may have to fix
    them (either use the semi-colon ';' to delimit the commands, or switch
    to interactive mode via the "prompt" command). A warning is emitted when
    this unreliable behavior is detected, to ease detection of faulty
  - the "enabled" server keyword used to be silently ignored when adding a
    dynamic server. Now it's properly rejected to avoid confusing scripts.
  - the way the memory limitation specified by "-m" on the command line
    was handled on Linux using RLIMIT_AS got completely useless over time
    due to much more fragmented memory spaces on 64-bit platforms, ASLR,
    and the fact that it had been chosen exclusively to avoid
    underestimating the allocated buffers' cost, which originally were
    allocated all the time even when empty. Nowadays this is no longer
    relevant since buffers are only allocated when used, and the current
    state had the nasty effect of causing OOMs way below the configured
    limit, rendering it pretty useless. The use of RLIMIT_AS was now
    dropped in favor of the more reliable RLIMIT_DATA like on other
    operating systems.
  - the "namespace" keyword used to be silently ignored on "bind" and
    "server" lines using UNIX sockets. Now it is properly used and
    checked, thus it may fail if it references an invalid value. If the
    previous configuration used to work, it probably means the keyword was
    not needed. In addition, the presence of the keyword on a "server"
    line may also cause a boot failure that was previously only detected
    at run time, if permissions are insufficient. There's no loss of
    functionality here, only a check performed earlier to ensure the
    process boots in a properly working state.
  - the HTTP/1 URI parser no longer accepts invalid origin-form URIs that
    start neither with a '/' nor a '*' (e.g. "index.html" without leading
    slash). Even if some servers would still accept that, clients that
    would be compatible with this have disappeared way more than a decade
    ago, and continuing to support this for such broken applications would
    probably lead to an abuse sooner or later, so better put an end to
    this now.
  - a workaround for an issue affecting QUIC on LibreSSL when running on
    non-x86 machines was developed jointly with the LibreSSL team. There's
    an issue with the CHACHA20_POLY1305 cipher when used in-place (for
    QUIC) that has been well identified and will be fixed in version 4.0
    of LibreSSL. The workaround consists in making the QUIC connection
    fail fast so that the client can quickly retry using TCP. We'll
    disable it once a stable LibreSSL version is out with the fix. A
    config-based workaround consists in forcing the ciphers, and exclude
    this one.

And we even found some room to improve the code's maintainability and
clarity, which will hopefully further lower the barrier to contribution:

  - applet: most of the internal API rework was done, which simpifies the
    upper layers and the applet code as well (for those that were
    converted). New applet code will have its own buffers and even less
    stuff to care about. This is also true for the CLI keyword handlers
    which can now be written in a more natural way and may now yield even
    when not blocked.
  - a significant part of the internal "shutdown" API was cleaned up so
    that there is now only one function at each layer instead of one per
    direction. Not only this did eliminate very old legacy code ported
    over the years, it also made it possible to forward gRPC
  - prometheus: a new registration mechanism was added to permit to
    register metrics per module (e.g. stick-tables, resolvers etc). The
    extra counters are also dumped if requested now (frontend, backend,
    listener, server).

I'm fairly certain that I forgot a few things. As usual, I'm told that my
coworkers at HAProxyTech also went through this tedious task of enumerating
the changes, and it will be posted soon here:


My understanding is that there will be some followups with a focus on
selected points. I'm not surprised by the difficulty of the exercise
this time ;-)

For this version, we've got an increased help from various testers who
accepted to run one (or a few) servers with the development version, and
who were able to report a few problems with accurate version ranges, as
well as traces and info that permitted to fix the issues quickly. It
worked amazingly well and allowed us to address some nasty bugs that are
fairly hard to reproduce and that were present for several versions
already. At the risk of repeating myself, thanks for that! I know that
operating a -dev version requires a bit more involvement than a stable one
but it's also a win-win: when something doesn't please you, it's not too
late to suggest a change, and you can benefit from the latest debugging
features and performance improvements. I sincerely hope that this success
will encourage other users into that direction. The nice benefit for the
user of facing a bug in -dev vs -stable is that we have no problem
developing new debugging extensions just for that issue, so a git pull is
enough to suddenly make the problem much more observable and require less
amount of work to filter data than with a stable version. And something
that's human is that developers tend to be much more attracted by issues
affecting areas that are still fresh in their heads and will tend to treat
them with higher priority.

I also noticed more exchanges from various participants on the issues
and here on the list, so big thanks as well to those who take time to
review other users' problem reports and requests for help. Especially
for first-time reporters, it gives them a great experience of the
project and its community.

As usual with a new major release comes the death of an old one. This time
it's 2.0 that passed away after 5 years serving as a transition between
the old legacy versions and the newer HTX-enabled ones. I'm fairly sure
there are still some here and there, so please consider this as a reminder
that it's about time to upgrade. And 2.4 turned to critical fixes only

On a side note (not very funny but surprising), apparently there was a big
GitHub outage last night, and this morning we're getting a "Ooops 500" page
on the haproxy repository there: https://github.com/haproxy/haproxy
The issues seem to be working, the wiki and docs projects as well. So I
suspect that an error page got cached during the outage and continues to
be delivered for whetever reason. I opened a ticket to their support and
we'll see when we get a response. Fortunately we're not completely blocked,
but it feels strange to release on a day of outage. After all, that's a
form of resilience that also makes one use a load balancer, so there's
some logic there.

Speaking of resilience, I'm going to take a bit of vacation next week and
the week after (maybe I should have postponed given the heavy rain here),
but you're in good hands with the rest of the team, and Christopher is
back on Monday, fresh an in full force. Maybe you'll even manage to
convince him to emit -dev1 himself, who knows :-)

Please find the usual URLs below :
   Site index       : https://www.haproxy.org/
   Documentation    : https://docs.haproxy.org/
   Wiki             : https://github.com/haproxy/wiki/wiki
   Discourse        : https://discourse.haproxy.org/
   Slack channel    : https://slack.haproxy.org/
   Issue tracker    : https://github.com/haproxy/haproxy/issues
   Sources          : https://www.haproxy.org/download/3.0/src/
   Git repository   : https://git.haproxy.org/git/haproxy-3.0.git/
   Git Web browsing : https://git.haproxy.org/?p=haproxy-3.0.git
   Changelog        : https://www.haproxy.org/download/3.0/src/CHANGELOG
   Dataplane API    : 
   Pending bugs     : https://www.haproxy.org/l/pending-bugs
   Reviewed bugs    : https://www.haproxy.org/l/reviewed-bugs
   Code reports     : https://www.haproxy.org/l/code-reports
   Latest builds    : https://www.haproxy.org/l/dev-packages

I verified what I had in mind for 3.0 and 3.1-dev0 (that just opened),
and I think all is good (Tim already fixed an incorrect color on the
docs index). As usual, if (should I say when?) you detect a broken link,
just let me know so I can fix it.

Have fun!
Complete changelog from 3.0-dev13:
Amaury Denoyelle (2):
      DOC: streamline http-reuse and connection naming definition
      REGTESTS: complete http-reuse test with pool-conn-name

Aurelien DARRAGON (3):
      MINOR: log: rename 'log-format tag' to 'log-format alias'
      DOC: config: document logformat item naming and typecasting features
      DOC: config: add %ID logformat alias alternative

Valentine Krasnobaeva (3):
      CLEANUP: ssl/ocsp: readable ifdef in ssl_sock_load_ocsp
      BUG/MINOR: ssl/ocsp: init callback func ptr as NULL
      BUG/MINOR: activity: fix Delta_calls and Delta_bytes count

William Lallemand (2):
      MINOR: sample: implement the uptime sample fetch
      CI: github: upgrade the WolfSSL job to 5.7.0

Willy Tarreau (11):
      CI: scripts: fix build of vtest regarding option -C
      CI: scripts: build vtest using multiple CPUs
      BUILD: makefile: yearly reordering of objects by build time
      BUILD: fd: errno is also needed without poll()
      DOC: config: fix two typos "RST_STEAM" vs "RST_STREAM"
      DOC: config: refer to the non-deprecated keywords in ocsp-update on/off
      CLEANUP: ssl_sock: move dirty openssl-1.0.2 wrapper to openssl-compat
      DOC: install: update quick build reminders with some missing options
      DOC: install: update the range of tested openssl version to cover 3.3
      DEV: patchbot: prepare for new version 3.1-dev
      MINOR: version: mention that it's 3.0 LTS now.


Reply via email to