HAProxy 2.8.4 was released on 2023/11/17. It added 126 new commits
after version 2.8.3.

This release is pretty huge. We were busy in chasing annoying bugs on the
2.9, especially because the release date is getting closer. Unfortunately
and paradoxically, a stable release with so many fixes is susceptible to
breakage. So monitor it carefully after update and report bugs. That's said,
it remains important to upgrade. Now, let's describe changes brought by this

For QUIC, a lot of changes were introduced with this release. As usual, a
bunch of minor bugfixes, some of them related to QUIC Retry
mechanism. Crashes occurrences which could happened under memory exhaustion
were also prevented.  Along this, there is also some significant
changes. One of the most important is the introduction of a wrapper for QUIC
over OpenSSL. This allows to support QUIC with the standard OpenSSL library
without using QuicTLS patched version. First, haproxy must be recompiled
with USE_QUIC_OPENSSL_COMPAT defined. This wrapper is however considered as
a degraded alternative over a SSL library with true QUIC support. One of the
wrapper downside is that 0-RTT is not supported so haproxy will force
disable its advertising. This is the reason why it is necessary to use the
additional "limited-quic" in haproxy global configuration to run with the

Another important change is that now QUIC connections are accounted against
maxconn since their allocation. Previously, this was only done once the
handshake succeeded and the MUX layer was allocated. This is important as it
should ensure that haproxy ressources remains under control even regarding
QUIC handshakes. Along this change, QUIC connections are also accounted in
SSL connections which was never the case before.

The last noticeable change concerns connections on CLOSING or DRAINING
state.  These states are used when a CONNECTION_CLOSE was sent or
received. When entering it, a connection must cease all transmission, except
resending of a CONNECTION_CLOSE frame. This could be compared in some way to
TCP FIN_WAIT.  Before this release, idle timeout was used to kept the
connection longer than necessary. Now, recommandations from RFC 9000 are
followed more strictly and the RTT estimation is used to release the
connection earlier. This should improve haproxy ressources consumption if
CONNECTION_CLOSE frames are exchanged frequently.

On H2 side, a possible crash was fixed when processing a response containing
a DATA frame after an 1xx response (or more generally before final
headers). It is of course forbidden by the HTTP/2 RFC and a protocol error
is now reported.  When a congested H2 connection is shut done, we now take
care to wait to send the final empty DATA frame with the ES flag, if
necessary, instead of sending a RST_STREAM.

When working on the mux-to-mux forwarding for the 2.9, we've found an
interesting issue. On some rare occasions, it was possible to not wake up H2
streams waiting in the send_list or fctl_list. These lists are used to
preserve fairness between streams. The issue is very limited because
blocked streams are quickly woken up, after some client WINDOW_UPDATE frames
or streams creation for instance, reason why it was left that long. But, in
a constraint environment a test involving 32 conns of 20 streams each, all
downloading a 10 MB object previously showed a limitation of 17 Gbps with
lots of idle CPU time, and now filled the cable at 25 Gbps, thanks to the

Still on H2, handling of http-request and http-keep-alive timeouts was
fixed. It was broken on 2.8.0. Since then, they don't expire anymore as long
as there's some activity. Finally, The stream ID is now committed even if the
stream is rejected. The H2 spec says that a HEADERS frame turns an idle
stream to the open state, and it may then turn to half-closed(remote) on ES,
then to close, all at once, if we respond with RST (e.g. on error). Due to
the fact that we process a complete frame at once since h2_dec_hdrs() may
reassemble CONTINUATION frames until everything is complete, the state was
only committed after the frame was completely valid (otherwise multiple
passes could result in subsequent frames being rejected as the stream ID
would be equal to the highest one). However this is not correct because it
means that a client may retry on the same ID as a previously failed one,
which technically is forbidden (for example the client couldn't know which
of them a WINDOW_UPDATE or RST_STREAM frame is for). However, in practive,
the issue is very limited.

In the H1 multiplexer, http-request and http-keep-alive timeouts were also
broken since a while (since the 2.2). For idle connections on client side,
the smallest value between the client timeout and the
http-request/http-keep-alive timeout was used while the client timeout had
to be used only if other ones were not defined. So, if the client timeout
was the smallest value, the keep-alive timeout was not respected. It was
only an issue for idle client connections. The http-request timeout was
respected from the moment part of the next request was received. It is now
fixed. A bug prevented sending of 400-bad-request response on shutdown
before the first request was fixed. Except if we must silently ignore empty
connections by enabling http-ignore-probes or dontlognull options, when a
client connection is closed before the first request, a 400-bad-request
response must be sent with the corresponding log message. However, that was
broken since the 2.7-dev9. Finally, some sanitizing was performed on
headers during parsing when Content-Length and Transfer-Encoding headers
were both present. The Content-Length header is ignored, of course, but its
value must also be ignored. Otherwise, the wrong chunk size was able to be
used and led to miss the last chunk.

Related to H1 but at the applicative level, the abortonclose option was not
properly handled when set on the backend only. It was only usable on
defaults section. This was due to the fact the backend was not known yet
when the option was evaluated. To fix the bug, a specific control option was
added on the multiplexer, to notify that it should wait for reads again to
properly report aborts. By default, a frontend H1 multiplexer stops to wait
for read events at the end of the request to save some syscalls, except if
abortonclose option is enabled.

A race in the Lua co-socket connect code was addressed, by which if it's
interrupted by the Lua scheduler in the middle of the operation, it would
fail and not be able to recover. Now it will be able to reconnect. Thanks to
Tristan, lua's logging can now be selectively enabled for stderr and for
loggers. However default behavior remains unchanged, both are enabled.

There was an issue when mixing lua actions loaded from 'lua-load' and
'lua-load-per-thread' directives within a single http/tcp session. When
executing action defined in another running context from the one of the
previously executed action (from lua-load, then from lua-load-per-thread or
the opposite, order doesn't matter), an lua error was triggered. Thus, in
order to make streams capable of chaining lua actions, fetches and
converters loaded in different lua stacks, a new detection logic was added
in hlua_stream_ctx_prepare() to be able to recreate the lua context in the
proper stack space when the existing one conflicts with the expected stack

Several issues was reported on 2.9 about the client and server timeouts
management at the stream-connector level, leading to an excess of stream's
wakeup. Most of issues should be fixed for the 2.8.4. However there is still
a strange behavior we don't understand for now that leads sc_notify() to set
a stream expiration date in the past. The issue should be quite limited. In
addition, we changed our mind about the reporting of send activities,
because it was not accurate. Instead of reporting a send activities when all
outgoing data were sent at once and a blocked send otherwise, we now report
full sends, partial sends or blocked sends. These events are used to update
internal timers to be able to trigger write timeouts when necessary. This
should prevent to report write timeouts on very congested streams.

An annoying issue was met when testing the reverse-http mechanism on the
2.9, by which failed connection attempts would apparently not be attempted
again when there was no connect timeout. It turned out to be more
generalized than the rhttp system, and actually affects all outgoing
connections relying on NPN or ALPN to choose the mux, on which no mux is
installed and for which the subscriber (ssl_sock) must be notified
instead. The issue appeared during 2.2-dev1 development.

To finish, here are the list of fixed bugs:

  - A possible case where deleting a server from the CLI was possible if the
    server didn't have any streams anymore but was being retried on by one
    stream. I.e. that stream still had a reference to it and could possibly
    end up on it again after the retry.

  - We now make sure not to interrupt HTTP responses that are delivered
    before requests when the server terminates with a reset. That's
    particularly visible in H2 with gRPC.

  - Master-worker mode failed to work with thread-groups > 1 due to the CLI
    not being pinned to group 1 only.

  - A possible crash in fcgi with stderr records due to a zero-copy
    operation that should not be allowed in this case.

  - Streamers detection, used to perform SSL sends bigger than
    une.ssl.maxrecord, was no longer working for HTX streams.

  - The "proto" keyword was not working for dynamic servers

  - Matching of action's arguments was not working as expected because the
    parser stopped on the first match instead of looking for the longest
    matching name.

  - Some huge pauses were erroneously imposed by the bandwidth limitation
    filter because of an overflow on the overshoot computation after a long
    inactivity period.

  - Crashes were possible if an applet was released while it was waiting for
    a buffer. It was not properly removed from the list of entities waiting
    for a buffer. It only happened if the memory was limited.

  - On reload, it was possible to exit the master on bind error because
    distinction between the master process and children ones was made
    too late. Identifying the master process earlier fixed the issue.

  - It was possible to enter into a deadlock when purging a pattern because
    pools were trimmed while the operation was under a lock. Thus during a
    clearing of a map, if another thread tried to access or update an entry
    in the same map, it had to wait for the pattern lock to be released,
    while the pools trimming function was waiting for all threads to be
    harmless, thus causing a deadlock. To fix the issue, the pools are now
    trimmed by the caller.

  - On peers, it was no longer possible to perform a full resync if the
    number of tables exceeded the number of updates allowed at once. The
    loop responsible to send updates to other peers was always interrupted
    after the end. To fix the issue, restart conditions for a teaching loop
    were changed.

  - The method used to decide how many pool entries could be released at
    once was buggy. Comparaison between the allocated count and the used
    count was inverted. In some cases, this led to very small batches to be
    released, increasing the memory consumption. Not really a memory leak

  - @system-ca was not properly loaded because the ca-base directory was
    still added.

  - With TLSv1.3, the certificate selection favored RSA certificated over
    ECDSA when both were available for a domain while it should be the

It was a long description for a huge release. I hope it will be stable
enough to wait a bit before releasing the 2.8.5. However your are encouraged
to update.

Thanks everyone for your help and your contributions !

Please find the usual URLs below :
   Site index       : https://www.haproxy.org/
   Documentation    : https://docs.haproxy.org/
   Wiki             : https://github.com/haproxy/wiki/wiki
   Discourse        : https://discourse.haproxy.org/
   Slack channel    : https://slack.haproxy.org/
   Issue tracker    : https://github.com/haproxy/haproxy/issues
   Sources          : https://www.haproxy.org/download/2.8/src/
   Git repository   : https://git.haproxy.org/git/haproxy-2.8.git/
   Git Web browsing : https://git.haproxy.org/?p=haproxy-2.8.git
   Changelog        : https://www.haproxy.org/download/2.8/src/CHANGELOG
   Dataplane API    : 
   Pending bugs     : https://www.haproxy.org/l/pending-bugs
   Reviewed bugs    : https://www.haproxy.org/l/reviewed-bugs
   Code reports     : https://www.haproxy.org/l/code-reports
   Latest builds    : https://www.haproxy.org/l/dev-packages

Complete changelog :
Aleksandar Lazic (1):
      DOC: internal: filters: fix reference to entities.pdf

Amaury Denoyelle (17):
      BUG/MINOR: mux-quic: remove full demux flag on ncbuf release
      BUG/MINOR: hq-interop: simplify parser requirement
      BUG/MINOR: quic: reject packet with no frame
      BUG/MEDIUM: mux-quic: fix RESET_STREAM on send-only stream
      BUG/MINOR: mux-quic: support initial 0 max-stream-data
      BUG/MINOR: h3: strengthen host/authority header parsing
      BUG/MINOR: mux-quic: fix free on qcs-new fail alloc
      BUG/MEDIUM: quic-conn: free unsent frames on retransmit to prevent crash
      BUG/MINOR: quic: do not consider idle timeout on CLOSING state
      BUG/MINOR: ssl: use a thread-safe sslconns increment
      MINOR: frontend: implement a dedicated actconn increment function
      MEDIUM: quic: count quic_conn instance for maxconn
      MEDIUM: quic: count quic_conn for global sslconns
      BUG/MINOR: mux-quic: fix early close if unset client timeout
      BUG/MEDIUM: quic: fix actconn on quic_conn alloc failure
      BUG/MEDIUM: quic: fix sslconns on quic_conn alloc failure
      BUG/MINOR: quic: fix retry token check inconsistency

Aurelien DARRAGON (13):
      MINOR: hlua: add hlua_stream_ctx_prepare helper function
      BUG/MEDIUM: hlua: streams don't support mixing lua-load with 
      BUG/MEDIUM: hlua: don't pass stale nargs argument to lua_resume()
      BUG/MINOR: hlua/init: coroutine may not resume itself
      BUG/MINOR: server: add missing free for server->rdr_pfx
      MINOR: pattern: fix pat_{parse,match}_ip() function comments
      BUG/MEDIUM: server/cli: don't delete a dynamic server that has streams
      MINOR: connection: add conn_pr_mode_to_proto_mode() helper func
      BUG/MEDIUM: server: "proto" not working for dynamic servers
      BUG/MINOR: stktable: missing free in parse_stick_table()
      BUG/MINOR: cfgparse/stktable: fix error message on stktable_init() failure
      BUG/MINOR: stick-table/cli: Check for invalid ipv4 key
      BUG/MINOR: sink: don't learn srv port from srv addr

Cedric Paillet (1):
      BUG/MINOR: promex: fix backend_agg_check_status

Christopher Faulet (42):
      BUG/MEDIUM: mux-fcgi: Don't swap trash and dbuf when handling STDERR 
      BUG/MEDIUM: master/cli: Pin the master CLI on the first thread of the 
group 1
      BUG/MAJOR: mux-h2: Report a protocol error for any DATA frame before 
      BUG/MEDIUM: http-ana: Try to handle response before handling server abort
      MINOR: hlua: Set context's appctx when the lua socket is created
      MINOR: hlua: Don't preform operations on a not connected socket
      MINOR: hlua: Save the lua socket's timeout in its context
      MINOR: hlua: Save the lua socket's server in its context
      MINOR: hlua: Test the hlua struct first when the lua socket is connecting
      BUG/MEDIUM: hlua: Initialize appctx used by a lua socket on connect only
      BUG/MINOR: mux-h1: Handle read0 in rcv_pipe() only when data receipt was 
      BUG/MINOR: mux-h1: Ignore C-L when sending H1 messages if T-E is also set
      BUG/MEDIUM: h1: Ignore C-L value in the H1 parser if T-E is also set
      BUG/MEDIUM: stconn: Fix comparison sign in sc_need_room()
      BUG/MINOR: mux-h1: Send a 400-bad-request on shutdown before the first 
      BUG/MEDIUM: mux-h2: Don't report an error on shutr if a shutw is pending
      BUG/MEDIUM: peers: Be sure to always refresh recconnect timer in sync task
      BUG/MEDIUM: peers: Fix synchro for huge number of tables
      BUG/MINOR: tcpcheck: Report hexstring instead of binary one on check 
      BUG/MEDIUM: stconn: Don't report rcv/snd expiration date if SC cannot 
      BUG/MEDIUM: Don't apply a max value on room_needed in sc_need_room()
      BUG/MINOR: stconn: Sanitize report for read activity
      CLEANUP: htx: Properly indent htx_reserve_max_data() function
      BUG/MINOR: mux-h1: Properly handle http-request and http-keep-alive 
      BUG/MEDIUM: freq-ctr: Don't report overshoot for long inactivity period
      BUG/MEDIUM: stconn: Don't update stream expiration date if already expired
      BUG/MEDIUM: applet: Remove appctx from buffer wait list on release
      BUG/MINOR: stconn: Handle abortonclose if backend connection was already 
set up
      MINOR: connection: Add a CTL flag to notify mux it should wait for reads 
      MEDIUM: mux-h1: Handle MUX_SUBS_RECV flag in h1_ctl() and susbscribe for 
      BUG/MEDIUM: stream: Properly handle abortonclose when set on backend only
      REGTESTS: http: Improve script testing abortonclose option
      BUG/MEDIUM: stconn: Report a send activity everytime data were sent
      BUG/MEDIUM: applet: Report a send activity everytime data were sent
      BUG/MEDIUM: stream: Don't call mux .ctl() callback if not implemented
      BUG/MEDIUM: stconn: Update fsb date on partial sends
      MINOR: htx: Use a macro for overhead induced by HTX
      MINOR: channel: Add functions to get info on buffers and deal with HTX 
      BUG/MINOR: stconn: Fix streamer detection for HTX streams
      BUG/MINOR: stconn: Use HTX-aware channel's functions to get info on buffer
      BUG/MINOR: stconn/applet: Report send activity only if there was output 
      BUG/MINOR: stconn: Report read activity on non-indep streams for partial 

Emeric Brun (1):
      Revert "BUG/MEDIUM: quic: missing check of dcid for init pkt including a 

Frédéric Lécaille (25):
      BUG/MINOR: quic: Leak of frames to send.
      BUG/MINOR: quic: Wrong cluster secret initialization
      MINOR: quic: QUIC openssl wrapper implementation
      MINOR: quic: Include QUIC opensssl wrapper header from TLS stacks 
compatibility header
      MINOR: quic: Do not enable O-RTT with USE_QUIC_OPENSSL_COMPAT
      MINOR: quic: Set the QUIC connection as extra data before calling 
      MINOR: quic: Do not enable 0RTT with SSL_set_quic_early_data_enabled()
      MINOR: quic: Add a compilation option for the QUIC OpenSSL wrapper
      MINOR: quic: Export some KDF functions (QUIC-TLS)
      MINOR: quic: Initialize TLS contexts for QUIC openssl wrapper
      MINOR: quic: Call the keylog callback for QUIC openssl wrapper from 
      MINOR: quic: Add a quic_openssl_compat struct to quic_conn struct
      MINOR: quic: SSL context initialization with QUIC OpenSSL wrapper.
      MINOR: quic: Add "limited-quic" new tuning setting
      DOC: quic: Add "limited-quic" new tuning setting
      BUG/MINOR: quic+openssl_compat: Non initialized TLS encryption levels
      MINOR: quic: Warning for OpenSSL wrapper QUIC bindings without 
      MINOR: quic+openssl_compat: Do not start without "limited-quic"
      MINOR: quic+openssl_compat: Emit an alert for "allow-0rtt" option
      BUG/MINOR: quic: Avoid crashing with unsupported cryptographic algos
      BUG/MINOR: quic: idle timer task requeued in the past
      BUG/MEDIUM: quic: Avoid trying to send ACK frames from an empty ack 
ranges tree
      BUG/MEDIUM: quic: Possible crashes when sending too short Initial packets
      BUG/MEDIUM: quic: Avoid some crashes upon TX packet allocation failures
      DOC: quic: Wrong syntax for "quic-cc-algo" keyword.

Ilya Shipitsin (2):
      CI: musl: highlight section if there are coredumps
      CI: musl: drop shopt in workflow invocation

Tristan (1):
      MINOR: lua: Add flags to configure logging behaviour

William Lallemand (9):
      BUILD: Makefile: add USE_QUIC_OPENSSL_COMPAT to make help
      BUG/MINOR: quic: allow-0rtt warning must only be emitted with quic bind
      BUG/MINOR: quic: ssl_quic_initial_ctx() uses error count not error code
      BUILD: quic: fix build on centos 8 and USE_QUIC_OPENSSL_COMPAT
      BUG/MINOR: ssl: load correctly @system-ca when ca-base is define
      BUG/MINOR: ssl: suboptimal certificate selection with TLSv1.3 and dual 
      BUG/MEDIUM: ssl: segfault when cipher is NULL
      DOC: management: -q is quiet all the time
      BUG/MEDIUM: mworker: set the master variable earlier

Willy Tarreau (14):
      BUILD: bug: make BUG_ON() void to avoid a rare warning
      BUG/MINOR: freq_ctr: fix possible negative rate with the scaled API
      BUG/MEDIUM: actions: always apply a longest match on prefix lookup
      BUG/MEDIUM: quic_conn: let the scheduler kill the task when needed
      BUG/MINOR: mux-h2: make up other blocked streams upon removal from list
      BUG/MINOR: mux-h2: fix http-request and http-keep-alive timeouts again
      BUG/MINOR: trace: fix trace parser error reporting
      BUG/MINOR: mux-h2: commit the current stream ID even on reject
      BUG/MINOR: mux-h2: update tracked counters with req cnt/req err
      DEBUG: mux-h2/flags: fix list of h2c flags used by the flags decoder
      BUG/MEDIUM: pattern: don't trim pools under lock in pat_ref_purge_range()
      BUG/MEDIUM: pool: fix releasable pool calculation when overloaded
      DOC: config: use the word 'backend' instead of 'proxy' in 'track' 
      BUG/MEDIUM: connection: report connection errors even when no mux is 

Christopher Faulet

Reply via email to