Hi,
HAProxy 2.6.21 was released on 2025/01/29. It added 47 new commits
after version 2.6.20.
Following bugs were fixed in this release:
* A bug was found in the way QUIC CRYPTO frames were stored internally to
handle unordered frames reception. There was no control on the relative
frame position that could lead to a crash when this position was too
high to be stored. Now, when this happens, the connection is closed with
CRYPTO_BUFFER_EXCEEDED error, as required by the QUIC specification.
* It was possible to provoke a crash if a QUIC ACK was received just
before the frames retransmit. In that case, the packet build was not
interrupted, leading to build an empty packet which should be
ack-elicting. The crash itself was the result of a BUG_ON() which
detected the issue.
* Most of remaining issues with the queues management were fixed. The
dequeuing process is now called when a stream is closed. This should
unsure no stream remains infinitely blocked in the queue and prevent any
infinite loop in some extreme cases. It was also possible to exceed the
configured maxconn when a server was brought back up. It appears that
only the proxy queue was evaluated at this stage while the server queue
must also be processed. Note that the issue it not totally fixed in
3.0. We can occasionally see a few more connections than maxconn, but
the max that have been observed is 4 more connections, we no longer get
multiple times maxconn. This was improved in the 3.2 to strictly respect
the maxconn value.
* Some locks were missing on some stick-table converters. No issue was
reported about this bug but it could lead to inconsistent values being
reported by "table_*" converters.
* Crashes could be encountered because of a use-after-free bug after two
QUIC packets were coalesced.
* QUIC NEW_TOKEN frames emitted by clients were not reject as expected.
Now when this happens, the connection is closed with PROTOCOL_VIOLATION
error code.
* The QUIC congestion window was increased each time a new acknowledge was
received without considering the window filling level. On a network
condition with negligible loss, this would cause the window to be
incremented until the maximum value, even though the application does
not have enough data to fill it. In most case, this issue was not
noticeable. However, it could lead to an excessive memory consumption
when a QUIC connection was suddenly interrupted, as in this case HAProxy
would fill the window with retransmission. It even caused OOM crash when
thousands of clients were interrupted at once on a local network
benchmark.
* An issue could be experienced on big-endian architecture in stick-table
sample fetch functions and converters because of conversions from 64-bit
integers and 32-bit ones.
* The HAPROXY_STARTUP_VERSION environment variable, defined during the
init stage, was set from a macro instead of being set from a
variable. Depending on how HAProxy is rebuilt, this could make debugging
sessions confusing because the version exposed in the environment was
not necessarily the same as the one presented in "haproxy -vv".
* The H1 multiplexer was only able to handle timeouts if the client or
server timeouts were defined, depending on the side. So, it was possible
to ignore client-fin/server-fin and http-keep-alive/http-request
timeouts.
* The way to deal with too many headers in received H2/H3 messages was
fixed. In H2, the maximum number of headers allowed in HEADERS frames on
sending path was lower than on receiving path. This could lead to report
sending errors while the message was accepted. It could be confusing. In
H3, the number of headers was tested before the decoding. However,
pseudo headers and cookie headers consumed extra slots. So in practice,
this lowered the maximum number of headers that could be received. To
workaround these issues, The number of headers in received messages is
now always tested after the decoding stage. In addition, unlike H1, the
number of headers must be limited when H2/H3 messages are sent to comply
to limitation imposed by the protocols. This limit was increased to
support headers rewriting without issue.
* On the H2 multiplexer, on server side, it was possible to send
RST_STREAM frame for streams with unassigned ID, so before the
formatting of the HEADERS frame, because the session was aborted during
the connection stage. It was an issue if this happened before the H2
PREFACE was sent because this prevent the servers to recognize it as a
H2 connection, leading to an early connection closure. We now take care
to not emit RST_STREAM frame in that case.
* Four issues with the L7 retries were fixed. First, the server status was
not adjusted at each retry, while it should be. Only the last connection
attempt was considered. Then, the buffer used to save the request to be
able to perform a L7 retry was released to early in some rare cases and
the request could be lost. It is of course unexpected and this could
lead to crash. The request state was not properly reset on L7 retry. The
request channel flag stating some data were sent was not reset on
retry. This could lead to consider a subsequent connection error as a L7
error while the request was never sent. In that case too, the request
could be lost, leading to crash. Finally, the L7 retries could be
ignored if a server abort was detected during the request forwarding
when the request was already in DONE state. In that case, the server
abort must be handled on the response analysis side to be able to
properly handle the L7 retries.
* In logs, the server response time (%Tr) was erroneously reported as -1
when it was intercepted by HAProxy. -1 is reserved to the case where
response headers were not fully received.
* The reason was missing in H2 responses forwarded to H1 clients while it
was stated in the configuration manual that HAProxy should add one that
matched the status code. It is now fixed.
* Inter-thread stream shutdown, used by "shutdown sessions server XXX" CLI
command or "on-error shutdown-sessions" server option, was not thread
safe.
* HAPROXY_CLI and HAPROXY_MASTER_CLI could exposed the internal sockpairs
which should be only used for the master CLI. These internal sockpairs
are now always hidden.
* The SIGINT signal could be missed by HAProxy when it was started in
background in a subshell. It is the root cause of some unexpected
timeouts with Vtest scripts. To fix the issue, the default signal
handler is registered for the SIGINT signal during init.
* A weird issue was fixed about the epoll poller. Over the last two years,
there were few reports about immediate closes spuriously happening on
connections where network captures proved that the server had not closed
at all (and sometimes even received the request and responded to it
after HAProxy had closed). The logs shown that a successful connection
was immediately reported on error after the request was sent. After
investigations, it appeared that a EPOLLUP, or eventually a EPOLLRDHUP,
can be reported by epool_wait() during the connect() but in
sock_conn_check(), the connect() reports a success. So the connection
was validated but the HUP was handled on the first receive and an error
was reported. So, to workaround the issue, we have decided to remove
FD_POLL_HUP flag on the FD during the connection establishment if
FD_POLL_ERR is not reported too in sock_conn_check(). This way, the call
to connect() is able to validate or reject the connection. At the end,
if the HUP or RDHUP flags were valid, either connect() would report the
error itself, or the next recv() would return 0 confirming the closure
that the poller tried to report. EPOLL_RDHUP is only an optimization to
save a syscall anyway, and this pattern is so rare that nobody will ever
notice the extra call to recv(). Please note that at least one reporter
confirmed that using poll() instead of epoll() also addressed the
problem, so that can also be a temporary workaround for those
discovering the problem without the ability to immediately upgrade.
* The expiration date for the task responsible to clean up the server
resolution status when outdated info were inherited from the state file
was fixed. "hold.timeout" was initially used. But is was not accurate,
especially because it could be set to a high value or 0. Now the
expiration date is based on the resolver "resolve" and "retry"
timeouts. In addition, when a resolver was woken up to process DNS
resolutions, it was possible to trigger an infinite loop on the
resolver's wait list because delayed resolutions were always reinserted
at the end of this list. This led the watchdog to kill the process. By
re-inserting them in front of the list fixed the issue.
* A crash could happen in mux-pt if an error happened on the connection
just before an abort that is going to emit a shutdown, and with a
pending wakeup that completes some work on a connection having no
transport layer anymore. This only affects TCP (e.g. peers and master
CLI; GH #2656).
In addition to these bug fixes, an improvement was backported. Some invalid
Transfer-Encoding values are now accepted during the H1 response parsing
when accept-invalid-http-response option is enabled, even if it is forbidden
by the RFC-9112. So, now, with this option, multiple "chunked" values are
accepted, as well as empty values. When several "chunked" values are found,
the payload will still be considered as encoded once and the header will be
sanitized when sent to the client. The request parsing was not changed. This
remains forbidden because it is highly suspicious if a client is sending an
invalid T-E header. On server side, we can consider the server as
trusted. But you must still remain careful with such behavior. And, of
course, the best is to fix the application.
To finish, some issues in the configuration manual were fixed, the
documentation of "%Tr" was improved and an explanation about quotes and
spaces in conditional blocks was added.
Thanks everyone for your help !
Please find the usual URLs below :
Site index : https://www.haproxy.org/
Documentation : https://docs.haproxy.org/
Wiki : https://github.com/haproxy/wiki/wiki
Discourse : https://discourse.haproxy.org/
Slack channel : https://slack.haproxy.org/
Issue tracker : https://github.com/haproxy/haproxy/issues
Sources : https://www.haproxy.org/download/2.6/src/
Git repository : https://git.haproxy.org/git/haproxy-2.6.git/
Git Web browsing : https://git.haproxy.org/?p=haproxy-2.6.git
Changelog : https://www.haproxy.org/download/2.6/src/CHANGELOG
Dataplane API :
https://github.com/haproxytech/dataplaneapi/releases/latest
Pending bugs : https://www.haproxy.org/l/pending-bugs
Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs
Code reports : https://www.haproxy.org/l/code-reports
Latest builds : https://www.haproxy.org/l/dev-packages
---
Complete changelog :
Amaury Denoyelle (5):
MINOR: quic: notify connection layer on handshake completion
BUG/MINOR: stream: unblock stream on wait-for-handshake completion
BUG/MINOR: quic: reject NEW_TOKEN frames from clients
BUG/MAJOR: quic: reject too large CRYPTO frames
BUG/MINOR: quic: do not increase congestion window if app limited
Aurelien DARRAGON (4):
DOC: lua: fix yield-dependent methods expected contexts
BUG/MEDIUM: pattern: prevent uninitialized reads in pat_match_{str,beg}
BUG/MINOR: stktable: fix big-endian compatiblity in smp_to_stkey()
BUG/MEDIUM: stktable: fix missing lock on some table converters
Christopher Faulet (20):
MEDIUM: h1: Accept invalid T-E values with accept-invalid-http-response
option
BUG/MEDIUM: mux-pt: Never fully close the connection on shutdown
BUG/MEDIUM: resolvers: Insert a non-executed resulution in front of the
wait list
BUG/MEDIUM: mux-h2: Don't send RST_STREAM frame for streams with no ID
BUG/MINOR: http_ana: Report -1 for %Tr for invalid response only
DOC: config: Slightly improve the %Tr documentation
BUG/MINOR: http-ana: Adjust the server status before the L7 retries
BUG/MEDIUM: mux-h2: Increase max number of headers when encoding HEADERS
frames
BUG/MEDIUM: mux-h2: Check the number of headers in HEADERS frame after
decoding
BUG/MEDIUM: h3: Properly limit the number of headers received
BUG/MEDIUM: h3: Increase max number of headers when sending headers
BUG/MEDIUM: http-ana: Don't release too early the L7 buffer
BUG/MEDIUM: sock: Remove FD_POLL_HUP during connect() if FD_POLL_ERR is
not set
BUG/MEDIUM: http-ana: Reset request flag about data sent to perform a L7
retry
BUG/MINOR: h1-htx: Use default reason if not set when formatting the
response
BUG/MINOR: server-state: Fix expiration date of srvrq_check tasks
BUG/MEDIUM: mux-h1: Fix how timeouts are applied on H1 connections
BUG/MEDIUM: stconn: Don't forward shut for SC in connecting state
MINOR: config: Alert about extra arguments for errorfile and errorloc
BUG/MINOR: stream: Properly handle "on-marked-up shutdown-backup-sessions"
Frederic Lecaille (4):
BUG/MAJOR: quic: fix wrong packet building due to already acked frames
BUG/MINOR: quic: ensure a detached coalesced packet can't access its
neighbours
MINOR: quic: Add a BUG_ON() on quic_tx_packet refcount
BUILD: quic: Move an ASSUME_NONNULL() for variable which is not null
Olivier Houchard (3):
BUG/MEDIUM: queues: Make sure we call process_srv_queue() when leaving
BUG/MEDIUM: queues: Do not use pendconn_grab_from_px().
BUG/MEDIUM: queue: Make process_srv_queue return the number of streams
Valentine Krasnobaeva (3):
BUG/MINOR: cli: don't show sockpairs in HAPROXY_CLI and HAPROXY_MASTER_CLI
BUG/MINOR: signal: register default handler for SIGINT in signal_init()
BUG/MINOR: ssl: put ssl_sock_load_ca under SSL_NO_GENERATE_CERTIFICATES
Willy Tarreau (8):
MINOR: task: define two new one-shot events for use with WOKEN_OTHER or
MSG
BUG/MEDIUM: stream: make stream_shutdown() async-safe
BUG/MEDIUM: queue: always dequeue the backend when redistributing the
last server
BUG/MEDIUM: queue: make sure never to queue when there's no more served
conns
BUG/MEDIUM: checks: make sure to always apply offsets to now_ms in
expiration
BUG/MEDIUM: mailers: make sure to always apply offsets to now_ms in
expiration
BUG/MINOR: peers: make sure to always apply offsets to now_ms in
expiration
DOC: configuration: explain quotes and spaces in conditional blocks
--
Christopher Faulet
