Hi,
HAProxy 3.0.8 was released on 2025/01/29. It added 55 new commits
after version 3.0.7.
As for the 3.1.3, this release fixes some annoying bugs. First one is about
the way QUIC CRYPTO frames were stored internally to handle unordered frames
reception. There was no control on the relative frame position that could
lead to a crash when this position was too high to be stored. Now, when this
happens, the connection is closed with CRYPTO_BUFFER_EXCEEDED error, as
required by the QUIC specification. The second one is about the logs and
rings. Log messages were sent to all servers sharing the same name but
defined in distinct log backends, because all these servers shared the same
sink. To fix the issue, the sink name used for a log server is now prefixed
with the backend name, making it truly unique. Finally, the last major bug
is about the Prometheus exporter. When backends extra counters were dumped,
the wrong pointer was used in the promex context to retrieve the
corresponding stats module, leading to memory corruptions and crashes. This
issue is mitigated by the fact the Prometheus exporter should never be
exposed. Other Crashes (or undefined behaviors) with the Prometheus exporter
could be experienced when resolvers metrics were dumped. The issue was
observed when a "resolvers" section was defined without any nameserver. To
fix the issue, we now take care to skip these sections when resolvers
metrics are dumped.
Some other bugs were fixed by this released:
* In the H2 multiplexer, when the reverse HTTP was used, invalid value
could be reported for the counters tracking the current and the total
numbers of HTTP/2 connections. First the increment was not performed on
the expected side. Then integer overflows could be experienced because
the decrement was also performed on the wrong side. To fix the issue,
the increment operations are now delayed on reverse HTTP. Both counters
are updated only after the reverse has completed.
* Crashes were also reported when "show schema json" command was issued.
The issue was introduced in 3.0-dev8 when the stats were refactored to
be preserved between reloads. The command context was not properly
initialised. It is now fixed.
* Most of remaining issues with the queues management were fixed. The
dequeuing process is now called when a stream is closed. This should
unsure no stream remains infinitely blocked in the queue and prevent any
infinite loop in some extreme cases. It was also possible to exceed the
configured maxconn when a server was brought back up. It appears that
only the proxy queue was evaluated at this stage while the server queue
must also be processed. Note that the issue it not totally fixed in
3.0. We can occasionally see a few more connections than maxconn, but
the max that have been observed is 4 more connections, we no longer get
multiple times maxconn. This was improved in the 3.2 to strictly respect
the maxconn value.
* A crash because of a BUG_ON() could be experienced because a stream task
could be queued in the past. Conditions to trigger the issue are quite
hard to meet. A buffer allocation must have failed while an analysis
timer is set. In that case, if an I/O event is received, the stream task
could be queued in past because an expired analysis timer is used. It
appeared that only I/O timers (read and write timeout) must be
considered when an I/O event is handled. Only the stream is responsible
to consider analysis timers.
* The CLI was skipping the '\r' at the end of all lines, even in command
payloads. It was not expected. It should only be performed on the
command line. The issue was observed when setting certificates
containing CRLFs via the CLI.
* Some locks were missing on some stick-table converters. No issue was
reported about this bug but it could lead to inconsistent values being
reported by "table_*" converters.
* In H1, it was possible to have unusable client connections waiting for
the client timeout while they should be closed. This happened when a
connection error was immediately encountered after the connection
establishment, in same time of the connection closure. It was not a leak
because connections were finally closed but it was a waste of
ressources, especially with a high client timeout.
* Crashes could be encountered because of a use-after-free bug after two
QUIC packets were coalesced.
* QUIC NEW_TOKEN frames emitted by clients were not reject as expected. Now
when this happens, the connection is closed with PROTOCOL_VIOLATION error
code.
* The QUIC congestion window was increased each time a new acknowledge was
received without considering the window filling level. On a network
condition with negligible loss, this would cause the window to be
incremented until the maximum value, even though the application does not
have enough data to fill it. In most case, this issue was not
noticeable. However, it could lead to an excessive memory consumption when a
QUIC connection was suddenly interrupted, as in this case HAProxy would
fill the window with retransmission. It even caused OOM crash when thousands
of clients were interrupted at once on a local network benchmark.
* "on-marked-up shutdown-backup-sessions" action was no longer working. It was
broken when the shutdown of a stream was modified to be async-safe. When
this part was fixed, the active stream shutdowns, via the CLI for instance,
and "on-marked-down shutdown-sessions" action were properly handled. But not
the shutdown of backup sessions.
* An issue could be experienced on big-endian architecture in stick-table
sample fetch functions and converters because of conversions from 64-bit
integers and 32-bit ones.
* The HAPROXY_STARTUP_VERSION environment variable, defined during the init
stage, was set from a macro instead of being set from a variable. Depending
on how HAProxy is rebuilt, this could make debugging sessions confusing
because the version exposed in the environment was not necessarily the same
as the one presented in "haproxy -vv".
* The parsing of errorfile and errorloc was improved to properly alert
about unexpected extra arguments.
* H1 responses to HEAD requests with a "transfer-encoding" header are now
properly parsed. A regression was introduced in 2.9. the H1 parser was
waiting for a message body for such responses, because of the
"transfer-encoding" header, while it should instead consider the
response as fully received.
* With dynamic servers, deleting a server that's currently about to be
displayed in stats but interrupted leaves a dangling pointer in the
stats dumper thread that most often will crash when it resumes
dumping. The mechanism was changed for a more reliable (albeit a bit
more complex) one.
In addition to all these bug fixes, some debugging tools were backported.
* fc.debug_str / bc.debug_str: reports lots of internal information from
the front and back connections (stream ID, connection flags, etc). These
are mostly used by developers and can take a lot of room on the log
line. See converter "when" below.
* Converter "when": allows to pass the contents of its input to the output
only when a defined condition is met. The condition might be "error",
"stopping", "acl:<acl_name>" and a few others, in order to define what's
of interest to extend the log output. Combined with "fc/bc.debug_str",
it allows to only emit detailed info when something abnormal happens
(e.g. unusually long response).
* A new COUNT_IF() macro allows to place event counters anywhere in the
code. Those are also automatically added to track glitches. The "debug
counters" command then allows to list all known counters with their
exact location in the code, and a likely cause when a meaning is
known. This allows to detect and count some unexpected conditions, as
well as figure what is causing trouble (typically by looking at glitch
counters even when no request was instantiated and no log produced).
Finally the documentation was improved: An exemple for server "track"
keyword was added, the "trask-sc0" action was missing from the action
keywords matrix, "tune.lua.*" keywords were not correctly ordered.
Thanks everyone for your help !
Please find the usual URLs below :
Site index : https://www.haproxy.org/
Documentation : https://docs.haproxy.org/
Wiki : https://github.com/haproxy/wiki/wiki
Discourse : https://discourse.haproxy.org/
Slack channel : https://slack.haproxy.org/
Issue tracker : https://github.com/haproxy/haproxy/issues
Sources : https://www.haproxy.org/download/3.0/src/
Git repository : https://git.haproxy.org/git/haproxy-3.0.git/
Git Web browsing : https://git.haproxy.org/?p=haproxy-3.0.git
Changelog : https://www.haproxy.org/download/3.0/src/CHANGELOG
Dataplane API :
https://github.com/haproxytech/dataplaneapi/releases/latest
Pending bugs : https://www.haproxy.org/l/pending-bugs
Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs
Code reports : https://www.haproxy.org/l/code-reports
Latest builds : https://www.haproxy.org/l/dev-packages
---
Complete changelog :
Amaury Denoyelle (12):
BUG/MINOR: stats: decrement srv refcount on stats-file release
MINOR: list: define a watcher type
BUG/MEDIUM: stats/server: use watcher to track server during stats dump
BUG/MINOR: h2/rhttp: fix HTTP2 conn counters on reverse
MINOR: mux-quic: define dump functions for QCC and QCS
MINOR: mux-quic: implement debug string for logs
MINOR: quic: dump quic_conn debug string for logs
MINOR: time: define tot_time structure
MINOR: mux-quic: measure QCS lifetime and its blocking state
BUG/MINOR: quic: reject NEW_TOKEN frames from clients
BUG/MAJOR: quic: reject too large CRYPTO frames
BUG/MINOR: quic: do not increase congestion window if app limited
Aurelien DARRAGON (8):
DOC: config: add example for server "track" keyword
DOC: config: reorder "tune.lua.*" keywords by alphabetical order
DOC: config: add "tune.lua.burst-timeout" to the list of global parameters
BUG/MINOR: stats: fix segfault caused by uninitialized value in "show schema
json"
DOC: config: add missing "track-sc0" in action keywords matrix
BUG/MINOR: stktable: fix big-endian compatiblity in smp_to_stkey()
BUG/MEDIUM: stktable: fix missing lock on some table converters
BUG/MAJOR: log/sink: possible sink collision in sink_new_from_srv()
Christopher Faulet (10):
BUG/MEDIUM: stconn: Don't forward shut for SC in connecting state
BUG/MEDIUM: stconn: Only consider I/O timers to update stream's
expiration date
MINOR: config: Alert about extra arguments for errorfile and errorloc
BUG/MEDIUM: promex/resolvers: Don't dump metrics if no nameserver is
defined
BUG/MEDIUM: h1-htx: Properly handle bodyless messages
MINOR: mux-h1: Add support of the debug string for logs
DEV: sock: Add a debug counter to track strange flag on fd during
connect()
BUG/MEDIUM: promex: Use right context pointers to dump backends
extra-counters
BUG/MEDIUM: mux-h1: Properly close H1C if an error is reported before
sending data
BUG/MINOR: stream: Properly handle "on-marked-up shutdown-backup-sessions"
Dragan Dosen (1):
MINOR: arg: add an argument type for identifier
Frederic Lecaille (3):
BUG/MINOR: quic: ensure a detached coalesced packet can't access its
neighbours
MINOR: quic: Add a BUG_ON() on quic_tx_packet refcount
BUILD: quic: Move an ASSUME_NONNULL() for variable which is not null
Olivier Houchard (3):
BUG/MEDIUM: queues: Make sure we call process_srv_queue() when leaving
BUG/MEDIUM: queues: Do not use pendconn_grab_from_px().
BUG/MEDIUM: queue: Make process_srv_queue return the number of streams
Valentine Krasnobaeva (3):
BUG/MINOR: cli: cli_snd_buf: preserve \r\n for payload lines
REGTESTS: ssl: add a PEM with mix of LF and CRLF line endings
BUG/MINOR: ssl: put ssl_sock_load_ca under SSL_NO_GENERATE_CERTIFICATES
Willy Tarreau (15):
BUG/MEDIUM: clock: make sure now_ms cannot be TICK_ETERNITY
MINOR: stconn: add a new pair of sf functions {bs,fs}.debug_str
MINOR: mux-h2: implement the debug string for logs
MINOR: sample: add the "when" converter to condition some expressions
MINOR: acl: export find_acl_default()
MINOR: sample: extend the "when" converter to support an ACL
CLEANUP: debug: make the BUG_ON() macros check the condition in the outer
one
MEDIUM: debug: add match counters for BUG_ON/WARN_ON/CHECK_IF
MINOR: debug: add a new debug macro COUNT_IF()
MINOR: debug: add "debug dev counters" to list code counters
MINOR: debug/cli: replace "debug dev counters" with "debug counters"
MINOR: activity/memprofile: also monitor strdup() activity
MINOR: activity/memprofile: monitor non-portable calls as well
BUILD: activity/memprofile: fix a build warning in the posix_memalign
handler
BUG/MINOR: init: set HAPROXY_STARTUP_VERSION from the variable, not the
macro
--
Christopher Faulet
