Hi,
HAProxy 2.9-dev8 was released on 2023/10/20. It added 99 new commits
after version 2.9-dev7.
Some crash issues were fixed, and the long-sought task leak affecting
QUIC was fixed as well. Some of the last important parts were finally
merged. These include:
- mux-to-mux zero-copy forwarding (work started around 2.4). In short,
the principle is to avoid buffering whenever we can avoid it, so as
to save memory and preserve data in CPU caches as much as possible.
The receiving side (typically the backend mux) will request the
sending side (frontend mux) to borrow its buffer and receive the
data directly into it, respecting flow control and available space.
This means that if the output is clogged, we don't add to the buffer
bloat. On a test involving h2 clients and h1 servers, we saw 18% more
bandwidth and 8% CPU saving at the same time (the 100G link reached
saturation) while saving ~10% of the RAM thanks to filling less
buffers. For now only tcp, H1 and H2 are supported, QUIC will follow
and I doubt fcgi would benefit from this at all.
- log backends: this starts to address the 4yr old issue #401 about
log server management. The first step here consists in supporting a
new mode "log" for backends and to permit to log to such a backend
from a "log" statement. In the backend, there are servers, in dgram
or connected mode (udp/unix/tcp etc), and various LB algorithms
(round-robin, hash, sticky, random). Since it's a backend, regular
health check methods are supported. There's still a lot of work to
be done there (we still do not identify named fields for hashing or
masking, nor permit them to be represented differently, nor can we
produce multiple logs at various steps). But at least it opens the
way to this long rework. I know that some server keywords are still
a bit too permissive and that there's ongoing work to better control
them, but that looks functional enough. Please note that the logging
performance remains basically unaffected, and that TCP continues to
rely on a ring with its infamous lock that doesn't scale well with
threads, but for moderate uses this will be fine anyway.
And the rest is less sensitive but nonetheless interesting:
- some cleanups were done for wolfssl and awslc, enabling some of the
features that were previously ifdefed out, and refining some regtests.
- some config updates to the reverse-http mechanism were brought:
the protocol name to use is now consistently "rhttp@" instead of
"rev@" or "@reverse". The number of connections a listener must set up
is now defined by a new keyword "nbconn" on the bind line instead of
abusing the closely related but confusing "maxconn". Most "bind"
keywords that do not apply to reverse-http are now rejected there,
just like the mix of both address types (rhttp and regular ones). As
such if you already tried it, your config will fail but will be easy
to adjust.
- the h2 mux didn't update the counter of request errors when facing
unparsable requests, in part because that started as an H2->H1
converter and also because for some time, any error would result in
an instant breakage of the connection. But over time we've refined
the lower-layer controls, offering more reasons to reject a request,
so it was logical to update the tracked counters in this case, like
h1 does. This means that if you've fine-tuned some http_req_rate
thresholds, they may become slightly more sensitive, but also more
accurate.
- the json_query() sample fetch function can now also extract arrays
- the "set-timeout" rule is now supported in "http-response" rulesets
(very convenient to adapt to an expected response time advertised by
the server via a header, e.g. for slow requests).
- some traces and debugging improvements, and regtest cleanups
Tristan reported a recent regression in quic-to-h1 transfers that
brought back a transfer slowdown, and could bisect it so we'll have
to have a look at this soon. We've also got a non-reproducible but
quite analyzable report of one of the rare "spinning stream" bugs,
so we might finally be able to understand what happens in this rare
case. I could instrument the ssl_capture leak I'm observing on
haproxy.org and figure that it's the SSL stack that doesn't release
it! I have no idea why for now but placing counters there definitely
shows that there are less objects released than calls to SSL_free()
and friends, we'll need to compare to older versions. I also noticed
a huge number of buffer pools allocated in the shared cache that do
not seem to be release:
> show pools
(...)
- Pool buffer (32768 bytes) : 28259 allocated (925990912 bytes), 27 used (~18
by thread caches), needed_avg 21, 0 failures, 1 users, @0x51182c0 [SHARED]
- Pool ssl-capture (296 bytes) : 225525 allocated (66755400 bytes), 221862
used (~3 by thread caches), needed_avg 221736, 0 failures, 1 users, @0x1d2c200
[SHARED]
It looks like at one point we needed to allocate a lot (since the
needed_avg count went down very low) but I don't know why they're not
flushed, I'll have to look at this. I already faced once a strange
behaviour where launching violent injections and interrupting them
would increase the memory usage, and could bisect it to my changes
to shard the pools in 2.9-dev4, but and couldn't reproduce it since.
It might be in this situation where the global cache is enabled, so
now I have an idea what to look for.
Anyway, we're approximately 1 month away from the release so there should
not be important changes anymore, mostly cleanups, doc updates and bug
fixes. Given the nature of the issues we're facing, I think we'll be on
schedule. I would really like it if someone had a look at a better way to
present the actions in the doc to put an end to the horrible copy-paste
job that's done for each of them in tcp-request, http-request, response...
Ideally we'd fix that before the release, but first we need to know how
to present them. I doubt a matrix would work, but maybe starting from
the actions and indicating what rule sets they are compatible with would
be an improvement.
Please find the usual URLs below :
Site index : https://www.haproxy.org/
Documentation : https://docs.haproxy.org/
Wiki : https://github.com/haproxy/wiki/wiki
Discourse : https://discourse.haproxy.org/
Slack channel : https://slack.haproxy.org/
Issue tracker : https://github.com/haproxy/haproxy/issues
Sources : https://www.haproxy.org/download/2.9/src/
Git repository : https://git.haproxy.org/git/haproxy.git/
Git Web browsing : https://git.haproxy.org/?p=haproxy.git
Changelog : https://www.haproxy.org/download/2.9/src/CHANGELOG
Dataplane API :
https://github.com/haproxytech/dataplaneapi/releases/latest
Pending bugs : https://www.haproxy.org/l/pending-bugs
Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs
Code reports : https://www.haproxy.org/l/code-reports
Latest builds : https://www.haproxy.org/l/dev-packages
Willy
---
Complete changelog :
Amaury Denoyelle (15):
REORG: quic: cleanup traces definition
BUG/MINOR: quic: reject packet with no frame
BUG/MEDIUM: mux-quic: fix RESET_STREAM on send-only stream
BUG/MINOR: mux-quic: support initial 0 max-stream-data
BUG/MINOR: h3: strengthen host/authority header parsing
BUG/MINOR: quic: fix qc.cids access on quic-conn fail alloc
BUG/MINOR: quic: fix free on quic-conn fail alloc
BUG/MINOR: mux-quic: fix free on qcs-new fail alloc
BUG/MEDIUM: quic-conn: free unsent frames on retransmit to prevent crash
MINOR: cfgparse: forbid mixing reverse and standard listeners
MINOR: listener: add nbconn kw for reverse connect
MINOR: server: convert @reverse to rev@ standard format
MINOR: cfgparse: rename "rev@" prefix to "rhttp@"
REGTESTS: remove maxconn from rhttp bind line
MINOR: listener: forbid most keywords for reverse HTTP bind
Aurelien DARRAGON (18):
MEDIUM: tree-wide: logsrv struct becomes logger
MEDIUM: log: introduce log target
DOC: config: log <address> becomes log <target> in "log" related doc
MEDIUM: sink/log: stop relying on AF_UNSPEC for rings
MINOR: log: support explicit log target as argument in __do_send_log()
MINOR: log: remove the logger dependency in do_send_log()
MEDIUM: log/sink: simplify log header handling
MEDIUM: sink: inherit from caller fmt in ring_write() when rings didn't
set one
MINOR: sink: add sink_new_from_srv() function
MAJOR: log: introduce log backends
MINOR: log/balance: support for the "sticky" lb algorithm
MINOR: log/balance: support for the "random" lb algorithm
MINOR: lbprm: support for the "none" hash-type function
MINOR: lbprm: compute the hash avalanche in gen_hash()
MINOR: sample: add sample_process_cnv() function
MEDIUM: log/balance: support for the "hash" lb algorithm
REGTEST: add a test for log-backend used as a log target
MINOR: server: introduce "log-bufsize" kw
Christopher Faulet (35):
BUG/MEDIUM: stconn: Report a send activity everytime data were sent
BUG/MEDIUM: applet: Report a send activity everytime data were sent
BUG/MINOR: mux-h1: Send a 400-bad-request on shutdown before the first
request
CLEANUP: hlua: Remove dead-code on error path in hlua_socket_new()
BUG/MEDIUM: mux-h1: do not forget TLR/EOT even when no data is sent
BUG/MINOR: htpp-ana/stats: Specify that HTX redirect messages have a C-L
header
BUG/MEDIUM: mux-h2: Don't report an error on shutr if a shutw is pending
MEDIUM: stconn/channel: Move pipes used for the splicing in the SE
descriptors
MINOR: stconn: Start to introduce mux-to-mux fast-forwarding notion
MINOR: stconn: Extend iobuf to handle a buffer in addition to a pipe
MINOR: connection: Add new mux callbacks to perform data fast-forwarding
MINOR: stconn: Temporarily remove kernel splicing support
MINOR: mux-pt: Temporarily remove splicing support
MINOR: mux-h1: Temporarily remove splicing support
MINOR: connection: Remove mux callbacks about splicing
MEDIUM: stconn: Add mux-to-mux fast-forward support
MINOR: mux-h1: Use HTX extra field only for responses with known length
MEDIUM: mux-h1: Properly handle state transitions of chunked outgoing
messages
MEDIUM: raw-sock: Specifiy amount of data to send via snd_pipe callback
MINOR: mux-h1: Add function to add size of a chunk to an outgoind message
MEDIUM: mux-h1: Simplify zero-copy on sending path
MEDIUM: mux-h1: Simplify payload formatting based on HTX blocks on
sending path
MEDIUM: mux-h1: Add fast-forwarding support
MINOR: h2: Set the BODYLESS_RESP flag on the HTX start-line if necessary
MEDIUM: mux-h2: Add consumer-side fast-forwarding support
MEDIUM: channel: don't look at iobuf to report an empty channel
MINOR: tree-wide: Only rely on co_data() to check channel emptyness
REGTESTS: Reenable HTTP tests about splicing
CLEAN: mux-h1: Remove useless __maybe_unused attribute on h1_make_chunk()
MEDIUM: mux-pt: Add fast-forwarding support
MINOR: global: Add an option to disable the zero-copy forwarding
BUILD: mux-h1: Fix build without kernel splicing support
REORG: stconn/muxes: Rename init step in fast-forwarding
BUG/MEDIUM: peers: Be sure to always refresh recconnect timer in sync task
BUG/MEDIUM: peers: Fix synchro for huge number of tables
Frédéric Lécaille (1):
BUG/MINOR: quic: Avoid crashing with unsupported cryptographic algos
Jens Popp (1):
MINOR: sample: Added support for Arrays in sample_conv_json_query in
sample.c
Vladimir Vdovin (1):
MINOR: support for http-response set-timeout
William Lallemand (16):
MINOR: ssl: add an explicit error when 'ciphersuites' are not supported
BUILD: ssl: enable 'ciphersuites' for WolfSSL
BUILD: ssl: add 'ssl_c_r_dn' fetch for WolfSSL
BUILD: ssl: add 'secure_memcmp' converter for WolfSSL and awslc
BUILD: ssl: enable keylog for awslc
CLEANUP: ssl: remove compat functions for openssl < 1.0.0
BUILD: ssl: enable keylog for WolfSSL
REGTESTS: pki: add a pki for SSL tests
REGTESTS: ssl: update common.pem with the new pki
REGTESTS: ssl: disable ssl_dh.vtc for WolfSSL
REGTESTS: wolfssl: temporarly disable some failing reg-tests
CI: ssl: add wolfssl to build-ssl.sh
CI: ssl: add git id support for wolfssl download
CI: github: add a wolfssl entry to the CI
CI: github: update wolfssl to git revision d83f2fa
CI: github: add awslc 1.16.0 to the push CI
Willy Tarreau (12):
CLEANUP: connection: drop an uneeded leftover cast
BUG/MAJOR: connection: make sure to always remove a connection from the
tree
BUG/MINOR: mux-h2: make up other blocked streams upon removal from list
DEBUG: pool: store the memprof bin on alloc() and update it on free()
BUG/MEDIUM: quic_conn: let the scheduler kill the task when needed
MINOR: dgram: allow to set rcv/sndbuf for dgram sockets as well
BUG/MINOR: mux-h2: fix http-request and http-keep-alive timeouts again
BUG/MINOR: trace: fix trace parser error reporting
MINOR: mux-h2/traces: explicitly show the error/refused stream states
MINOR: mux-h2/traces: clarify the "rejected H2 request" event
BUG/MINOR: mux-h2: commit the current stream ID even on reject
BUG/MINOR: mux-h2: update tracked counters with req cnt/req err
---
