Hi,
HAProxy 2.8.4 was released on 2023/11/17. It added 126 new commits
after version 2.8.3.
This release is pretty huge. We were busy in chasing annoying bugs on the
2.9, especially because the release date is getting closer. Unfortunately
and paradoxically, a stable release with so many fixes is susceptible to
breakage. So monitor it carefully after update and report bugs. That's said,
it remains important to upgrade. Now, let's describe changes brought by this
release.
For QUIC, a lot of changes were introduced with this release. As usual, a
bunch of minor bugfixes, some of them related to QUIC Retry
mechanism. Crashes occurrences which could happened under memory exhaustion
were also prevented. Along this, there is also some significant
changes. One of the most important is the introduction of a wrapper for QUIC
over OpenSSL. This allows to support QUIC with the standard OpenSSL library
without using QuicTLS patched version. First, haproxy must be recompiled
with USE_QUIC_OPENSSL_COMPAT defined. This wrapper is however considered as
a degraded alternative over a SSL library with true QUIC support. One of the
wrapper downside is that 0-RTT is not supported so haproxy will force
disable its advertising. This is the reason why it is necessary to use the
additional "limited-quic" in haproxy global configuration to run with the
wrapper.
Another important change is that now QUIC connections are accounted against
maxconn since their allocation. Previously, this was only done once the
handshake succeeded and the MUX layer was allocated. This is important as it
should ensure that haproxy ressources remains under control even regarding
QUIC handshakes. Along this change, QUIC connections are also accounted in
SSL connections which was never the case before.
The last noticeable change concerns connections on CLOSING or DRAINING
state. These states are used when a CONNECTION_CLOSE was sent or
received. When entering it, a connection must cease all transmission, except
resending of a CONNECTION_CLOSE frame. This could be compared in some way to
TCP FIN_WAIT. Before this release, idle timeout was used to kept the
connection longer than necessary. Now, recommandations from RFC 9000 are
followed more strictly and the RTT estimation is used to release the
connection earlier. This should improve haproxy ressources consumption if
CONNECTION_CLOSE frames are exchanged frequently.
On H2 side, a possible crash was fixed when processing a response containing
a DATA frame after an 1xx response (or more generally before final
headers). It is of course forbidden by the HTTP/2 RFC and a protocol error
is now reported. When a congested H2 connection is shut done, we now take
care to wait to send the final empty DATA frame with the ES flag, if
necessary, instead of sending a RST_STREAM.
When working on the mux-to-mux forwarding for the 2.9, we've found an
interesting issue. On some rare occasions, it was possible to not wake up H2
streams waiting in the send_list or fctl_list. These lists are used to
preserve fairness between streams. The issue is very limited because
blocked streams are quickly woken up, after some client WINDOW_UPDATE frames
or streams creation for instance, reason why it was left that long. But, in
a constraint environment a test involving 32 conns of 20 streams each, all
downloading a 10 MB object previously showed a limitation of 17 Gbps with
lots of idle CPU time, and now filled the cable at 25 Gbps, thanks to the
fix.
Still on H2, handling of http-request and http-keep-alive timeouts was
fixed. It was broken on 2.8.0. Since then, they don't expire anymore as long
as there's some activity. Finally, The stream ID is now committed even if the
stream is rejected. The H2 spec says that a HEADERS frame turns an idle
stream to the open state, and it may then turn to half-closed(remote) on ES,
then to close, all at once, if we respond with RST (e.g. on error). Due to
the fact that we process a complete frame at once since h2_dec_hdrs() may
reassemble CONTINUATION frames until everything is complete, the state was
only committed after the frame was completely valid (otherwise multiple
passes could result in subsequent frames being rejected as the stream ID
would be equal to the highest one). However this is not correct because it
means that a client may retry on the same ID as a previously failed one,
which technically is forbidden (for example the client couldn't know which
of them a WINDOW_UPDATE or RST_STREAM frame is for). However, in practive,
the issue is very limited.
In the H1 multiplexer, http-request and http-keep-alive timeouts were also
broken since a while (since the 2.2). For idle connections on client side,
the smallest value between the client timeout and the
http-request/http-keep-alive timeout was used while the client timeout had
to be used only if other ones were not defined. So, if the client timeout
was the smallest value, the keep-alive timeout was not respected. It was
only an issue for idle client connections. The http-request timeout was
respected from the moment part of the next request was received. It is now
fixed. A bug prevented sending of 400-bad-request response on shutdown
before the first request was fixed. Except if we must silently ignore empty
connections by enabling http-ignore-probes or dontlognull options, when a
client connection is closed before the first request, a 400-bad-request
response must be sent with the corresponding log message. However, that was
broken since the 2.7-dev9. Finally, some sanitizing was performed on
headers during parsing when Content-Length and Transfer-Encoding headers
were both present. The Content-Length header is ignored, of course, but its
value must also be ignored. Otherwise, the wrong chunk size was able to be
used and led to miss the last chunk.
Related to H1 but at the applicative level, the abortonclose option was not
properly handled when set on the backend only. It was only usable on
defaults section. This was due to the fact the backend was not known yet
when the option was evaluated. To fix the bug, a specific control option was
added on the multiplexer, to notify that it should wait for reads again to
properly report aborts. By default, a frontend H1 multiplexer stops to wait
for read events at the end of the request to save some syscalls, except if
abortonclose option is enabled.
A race in the Lua co-socket connect code was addressed, by which if it's
interrupted by the Lua scheduler in the middle of the operation, it would
fail and not be able to recover. Now it will be able to reconnect. Thanks to
Tristan, lua's logging can now be selectively enabled for stderr and for
loggers. However default behavior remains unchanged, both are enabled.
There was an issue when mixing lua actions loaded from 'lua-load' and
'lua-load-per-thread' directives within a single http/tcp session. When
executing action defined in another running context from the one of the
previously executed action (from lua-load, then from lua-load-per-thread or
the opposite, order doesn't matter), an lua error was triggered. Thus, in
order to make streams capable of chaining lua actions, fetches and
converters loaded in different lua stacks, a new detection logic was added
in hlua_stream_ctx_prepare() to be able to recreate the lua context in the
proper stack space when the existing one conflicts with the expected stack
id.
Several issues was reported on 2.9 about the client and server timeouts
management at the stream-connector level, leading to an excess of stream's
wakeup. Most of issues should be fixed for the 2.8.4. However there is still
a strange behavior we don't understand for now that leads sc_notify() to set
a stream expiration date in the past. The issue should be quite limited. In
addition, we changed our mind about the reporting of send activities,
because it was not accurate. Instead of reporting a send activities when all
outgoing data were sent at once and a blocked send otherwise, we now report
full sends, partial sends or blocked sends. These events are used to update
internal timers to be able to trigger write timeouts when necessary. This
should prevent to report write timeouts on very congested streams.
An annoying issue was met when testing the reverse-http mechanism on the
2.9, by which failed connection attempts would apparently not be attempted
again when there was no connect timeout. It turned out to be more
generalized than the rhttp system, and actually affects all outgoing
connections relying on NPN or ALPN to choose the mux, on which no mux is
installed and for which the subscriber (ssl_sock) must be notified
instead. The issue appeared during 2.2-dev1 development.
To finish, here are the list of fixed bugs:
- A possible case where deleting a server from the CLI was possible if the
server didn't have any streams anymore but was being retried on by one
stream. I.e. that stream still had a reference to it and could possibly
end up on it again after the retry.
- We now make sure not to interrupt HTTP responses that are delivered
before requests when the server terminates with a reset. That's
particularly visible in H2 with gRPC.
- Master-worker mode failed to work with thread-groups > 1 due to the CLI
not being pinned to group 1 only.
- A possible crash in fcgi with stderr records due to a zero-copy
operation that should not be allowed in this case.
- Streamers detection, used to perform SSL sends bigger than
une.ssl.maxrecord, was no longer working for HTX streams.
- The "proto" keyword was not working for dynamic servers
- Matching of action's arguments was not working as expected because the
parser stopped on the first match instead of looking for the longest
matching name.
- Some huge pauses were erroneously imposed by the bandwidth limitation
filter because of an overflow on the overshoot computation after a long
inactivity period.
- Crashes were possible if an applet was released while it was waiting for
a buffer. It was not properly removed from the list of entities waiting
for a buffer. It only happened if the memory was limited.
- On reload, it was possible to exit the master on bind error because
distinction between the master process and children ones was made
too late. Identifying the master process earlier fixed the issue.
- It was possible to enter into a deadlock when purging a pattern because
pools were trimmed while the operation was under a lock. Thus during a
clearing of a map, if another thread tried to access or update an entry
in the same map, it had to wait for the pattern lock to be released,
while the pools trimming function was waiting for all threads to be
harmless, thus causing a deadlock. To fix the issue, the pools are now
trimmed by the caller.
- On peers, it was no longer possible to perform a full resync if the
number of tables exceeded the number of updates allowed at once. The
loop responsible to send updates to other peers was always interrupted
after the end. To fix the issue, restart conditions for a teaching loop
were changed.
- The method used to decide how many pool entries could be released at
once was buggy. Comparaison between the allocated count and the used
count was inverted. In some cases, this led to very small batches to be
released, increasing the memory consumption. Not really a memory leak
however.
- @system-ca was not properly loaded because the ca-base directory was
still added.
- With TLSv1.3, the certificate selection favored RSA certificated over
ECDSA when both were available for a domain while it should be the
opposite.
It was a long description for a huge release. I hope it will be stable
enough to wait a bit before releasing the 2.8.5. However your are encouraged
to update.
Thanks everyone for your help and your contributions !
Please find the usual URLs below :
Site index : https://www.haproxy.org/
Documentation : https://docs.haproxy.org/
Wiki : https://github.com/haproxy/wiki/wiki
Discourse : https://discourse.haproxy.org/
Slack channel : https://slack.haproxy.org/
Issue tracker : https://github.com/haproxy/haproxy/issues
Sources : https://www.haproxy.org/download/2.8/src/
Git repository : https://git.haproxy.org/git/haproxy-2.8.git/
Git Web browsing : https://git.haproxy.org/?p=haproxy-2.8.git
Changelog : https://www.haproxy.org/download/2.8/src/CHANGELOG
Dataplane API :
https://github.com/haproxytech/dataplaneapi/releases/latest
Pending bugs : https://www.haproxy.org/l/pending-bugs
Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs
Code reports : https://www.haproxy.org/l/code-reports
Latest builds : https://www.haproxy.org/l/dev-packages
---
Complete changelog :
Aleksandar Lazic (1):
DOC: internal: filters: fix reference to entities.pdf
Amaury Denoyelle (17):
BUG/MINOR: mux-quic: remove full demux flag on ncbuf release
BUG/MINOR: hq-interop: simplify parser requirement
BUG/MINOR: quic: reject packet with no frame
BUG/MEDIUM: mux-quic: fix RESET_STREAM on send-only stream
BUG/MINOR: mux-quic: support initial 0 max-stream-data
BUG/MINOR: h3: strengthen host/authority header parsing
BUG/MINOR: mux-quic: fix free on qcs-new fail alloc
BUG/MEDIUM: quic-conn: free unsent frames on retransmit to prevent crash
BUG/MINOR: quic: do not consider idle timeout on CLOSING state
BUG/MINOR: ssl: use a thread-safe sslconns increment
MINOR: frontend: implement a dedicated actconn increment function
MEDIUM: quic: count quic_conn instance for maxconn
MEDIUM: quic: count quic_conn for global sslconns
BUG/MINOR: mux-quic: fix early close if unset client timeout
BUG/MEDIUM: quic: fix actconn on quic_conn alloc failure
BUG/MEDIUM: quic: fix sslconns on quic_conn alloc failure
BUG/MINOR: quic: fix retry token check inconsistency
Aurelien DARRAGON (13):
MINOR: hlua: add hlua_stream_ctx_prepare helper function
BUG/MEDIUM: hlua: streams don't support mixing lua-load with
lua-load-per-thread
BUG/MEDIUM: hlua: don't pass stale nargs argument to lua_resume()
BUG/MINOR: hlua/init: coroutine may not resume itself
BUG/MINOR: server: add missing free for server->rdr_pfx
MINOR: pattern: fix pat_{parse,match}_ip() function comments
BUG/MEDIUM: server/cli: don't delete a dynamic server that has streams
MINOR: connection: add conn_pr_mode_to_proto_mode() helper func
BUG/MEDIUM: server: "proto" not working for dynamic servers
BUG/MINOR: stktable: missing free in parse_stick_table()
BUG/MINOR: cfgparse/stktable: fix error message on stktable_init() failure
BUG/MINOR: stick-table/cli: Check for invalid ipv4 key
BUG/MINOR: sink: don't learn srv port from srv addr
Cedric Paillet (1):
BUG/MINOR: promex: fix backend_agg_check_status
Christopher Faulet (42):
BUG/MEDIUM: mux-fcgi: Don't swap trash and dbuf when handling STDERR
records
BUG/MEDIUM: master/cli: Pin the master CLI on the first thread of the
group 1
BUG/MAJOR: mux-h2: Report a protocol error for any DATA frame before
headers
BUG/MEDIUM: http-ana: Try to handle response before handling server abort
MINOR: hlua: Set context's appctx when the lua socket is created
MINOR: hlua: Don't preform operations on a not connected socket
MINOR: hlua: Save the lua socket's timeout in its context
MINOR: hlua: Save the lua socket's server in its context
MINOR: hlua: Test the hlua struct first when the lua socket is connecting
BUG/MEDIUM: hlua: Initialize appctx used by a lua socket on connect only
BUG/MINOR: mux-h1: Handle read0 in rcv_pipe() only when data receipt was
tried
BUG/MINOR: mux-h1: Ignore C-L when sending H1 messages if T-E is also set
BUG/MEDIUM: h1: Ignore C-L value in the H1 parser if T-E is also set
BUG/MEDIUM: stconn: Fix comparison sign in sc_need_room()
BUG/MINOR: mux-h1: Send a 400-bad-request on shutdown before the first
request
BUG/MEDIUM: mux-h2: Don't report an error on shutr if a shutw is pending
BUG/MEDIUM: peers: Be sure to always refresh recconnect timer in sync task
BUG/MEDIUM: peers: Fix synchro for huge number of tables
BUG/MINOR: tcpcheck: Report hexstring instead of binary one on check
failure
BUG/MEDIUM: stconn: Don't report rcv/snd expiration date if SC cannot
epxire
BUG/MEDIUM: Don't apply a max value on room_needed in sc_need_room()
BUG/MINOR: stconn: Sanitize report for read activity
CLEANUP: htx: Properly indent htx_reserve_max_data() function
BUG/MINOR: mux-h1: Properly handle http-request and http-keep-alive
timeouts
BUG/MEDIUM: freq-ctr: Don't report overshoot for long inactivity period
BUG/MEDIUM: stconn: Don't update stream expiration date if already expired
BUG/MEDIUM: applet: Remove appctx from buffer wait list on release
BUG/MINOR: stconn: Handle abortonclose if backend connection was already
set up
MINOR: connection: Add a CTL flag to notify mux it should wait for reads
again
MEDIUM: mux-h1: Handle MUX_SUBS_RECV flag in h1_ctl() and susbscribe for
reads
BUG/MEDIUM: stream: Properly handle abortonclose when set on backend only
REGTESTS: http: Improve script testing abortonclose option
BUG/MEDIUM: stconn: Report a send activity everytime data were sent
BUG/MEDIUM: applet: Report a send activity everytime data were sent
BUG/MEDIUM: stream: Don't call mux .ctl() callback if not implemented
BUG/MEDIUM: stconn: Update fsb date on partial sends
MINOR: htx: Use a macro for overhead induced by HTX
MINOR: channel: Add functions to get info on buffers and deal with HTX
streams
BUG/MINOR: stconn: Fix streamer detection for HTX streams
BUG/MINOR: stconn: Use HTX-aware channel's functions to get info on buffer
BUG/MINOR: stconn/applet: Report send activity only if there was output
data
BUG/MINOR: stconn: Report read activity on non-indep streams for partial
sends
Emeric Brun (1):
Revert "BUG/MEDIUM: quic: missing check of dcid for init pkt including a
token"
Frédéric Lécaille (25):
BUG/MINOR: quic: Leak of frames to send.
BUG/MINOR: quic: Wrong cluster secret initialization
MINOR: quic: QUIC openssl wrapper implementation
MINOR: quic: Include QUIC opensssl wrapper header from TLS stacks
compatibility header
MINOR: quic: Do not enable O-RTT with USE_QUIC_OPENSSL_COMPAT
MINOR: quic: Set the QUIC connection as extra data before calling
SSL_set_quic_method()
MINOR: quic: Do not enable 0RTT with SSL_set_quic_early_data_enabled()
MINOR: quic: Add a compilation option for the QUIC OpenSSL wrapper
MINOR: quic: Export some KDF functions (QUIC-TLS)
MINOR: quic: Initialize TLS contexts for QUIC openssl wrapper
MINOR: quic: Call the keylog callback for QUIC openssl wrapper from
SSL_CTX_keylog()
MINOR: quic: Add a quic_openssl_compat struct to quic_conn struct
MINOR: quic: SSL context initialization with QUIC OpenSSL wrapper.
MINOR: quic: Add "limited-quic" new tuning setting
DOC: quic: Add "limited-quic" new tuning setting
BUG/MINOR: quic+openssl_compat: Non initialized TLS encryption levels
MINOR: quic: Warning for OpenSSL wrapper QUIC bindings without
"limited-quic"
MINOR: quic+openssl_compat: Do not start without "limited-quic"
MINOR: quic+openssl_compat: Emit an alert for "allow-0rtt" option
BUG/MINOR: quic: Avoid crashing with unsupported cryptographic algos
BUG/MINOR: quic: idle timer task requeued in the past
BUG/MEDIUM: quic: Avoid trying to send ACK frames from an empty ack
ranges tree
BUG/MEDIUM: quic: Possible crashes when sending too short Initial packets
BUG/MEDIUM: quic: Avoid some crashes upon TX packet allocation failures
DOC: quic: Wrong syntax for "quic-cc-algo" keyword.
Ilya Shipitsin (2):
CI: musl: highlight section if there are coredumps
CI: musl: drop shopt in workflow invocation
Tristan (1):
MINOR: lua: Add flags to configure logging behaviour
William Lallemand (9):
BUILD: Makefile: add USE_QUIC_OPENSSL_COMPAT to make help
BUG/MINOR: quic: allow-0rtt warning must only be emitted with quic bind
BUG/MINOR: quic: ssl_quic_initial_ctx() uses error count not error code
BUILD: quic: fix build on centos 8 and USE_QUIC_OPENSSL_COMPAT
BUG/MINOR: ssl: load correctly @system-ca when ca-base is define
BUG/MINOR: ssl: suboptimal certificate selection with TLSv1.3 and dual
ECDSA/RSA
BUG/MEDIUM: ssl: segfault when cipher is NULL
DOC: management: -q is quiet all the time
BUG/MEDIUM: mworker: set the master variable earlier
Willy Tarreau (14):
BUILD: bug: make BUG_ON() void to avoid a rare warning
BUG/MINOR: freq_ctr: fix possible negative rate with the scaled API
BUG/MEDIUM: actions: always apply a longest match on prefix lookup
BUG/MEDIUM: quic_conn: let the scheduler kill the task when needed
BUG/MINOR: mux-h2: make up other blocked streams upon removal from list
BUG/MINOR: mux-h2: fix http-request and http-keep-alive timeouts again
BUG/MINOR: trace: fix trace parser error reporting
BUG/MINOR: mux-h2: commit the current stream ID even on reject
BUG/MINOR: mux-h2: update tracked counters with req cnt/req err
DEBUG: mux-h2/flags: fix list of h2c flags used by the flags decoder
BUG/MEDIUM: pattern: don't trim pools under lock in pat_ref_purge_range()
BUG/MEDIUM: pool: fix releasable pool calculation when overloaded
DOC: config: use the word 'backend' instead of 'proxy' in 'track'
description
BUG/MEDIUM: connection: report connection errors even when no mux is
installed
--
Christopher Faulet
