HAProxy 2.9-dev10 was released on 2023/11/18. It added 154 new commits
after version 2.9-dev9.

It's not yet as calm as I would like it to be but that's essentially due
to a number of long-lasting bugs being worked on at the same time, and
many of those fixed here were already in 2.8, so instead it's a sign of

Rémi's updates on the cache were finally merged. The patches were clean
and the review easy enough. Given that this series has been under stress
test for a while now and that its main goal is to significantly reduce
locking contention, there was no reason to further delay it. During the
review we even noticed new opportunities for future improvements, such
as changing the default block size etc. But this can come later and even
be backported if needed.

The automatic reconnection issue that I mentioned was affecting reverse-http
was finally not caused by reverse-http but by an old bug which prevents
connection errors from being reported immediately when ALPN is configured
on a server. This was fixed, but it gave the opportunity to reverse-http
to also honor the connect timeout. Speaking about reverse-http, there were
issues in which some incoming connections would be considered in excess and
rejected too early, sometimes causing 503 to appear. Now it seems to be
working quite well.

On the QUIC front, we now have a "timeout client-hs" that permits to set
a timeout for the client handshake phase without being bound by the client
timeout. We indeed anticipate that in the future some users will need a
large client timeout for certain applications and still want a small
handshake timeout to avoid easy attacks on the early stages of the
connection. The way the accept queue and handshakes was counted has
been refined to better match reality: retry tokens will now only be
sent when too many unconfirmed connections were seen, and the listener
will stop responding once the accept queue is full, just like for TCP
finally. This allows to reuse existing tunables without creating new
ones. Just like for TCP, the QUIC TLS handshakes are now performed in
a task tagged "heavy" which allows to maintain low processing latency
even when under attack. Finally, all congestion control algorithms now
support an optional argument that sets their maximum window size. Just
like for TCP with sysctls, this will be useful to balance memory usage
versus performance.

Some changes were brought to the log backend. Initially a "log-balance"
keyword had been added to select the LB algorithms without having to
rework the "balance" parser to adapt to the mode, but we finally thought
it was a mistake that we would risk to have to maintain for a long time
and justify by "this is historic". So now in a backend in mode "log",
the "balance" keyword sets the balance algorithm. Those that are not
supported in "mode log" will trigger a warning, and those specific to
this mode will trigger a warning when used in tcp/http. There are
differences between modes for the same algorithm, which are related to
how they're used, but these are documented. Right now we have "log-hash"
for the hash, which differs from "hash" in that it takes a list of
converters to apply to the log line. Possibly in the future we'll implement
a dummy sample-fetch function returning the log line so that "hash" can
be used naturally on it, I don't know. Similarly, there's a "log-sticky"
which sticks to the same server as long as it's up. I'm just realizing
while writing this that some users already do that for data base servers
and might like this with regular servers, so maybe this will later turn
into "sticky" and be usable everywhere, again I don't really know. Let's
not try to anticipate too much, at least what I want is to make sure the
config remains understandable, manageable and non-confusing. In previous
dev version there were a number of commits that were added to exclude
some backend keywords depending on the mode, but these were not reliable
enough (depends if "mode" was seen before or after). These commits were
reverted and the check moved to a more reliable place.

After careful inspection of the behavior under zero-copy fast-forward
transfers, we noticed the memory usage hadn't dropped as we'd anticipate,
and finally found a few places with missing synchronous flushes. On large
h2 transfers we've observed an increase of 60% of the forwarding badnwidth
together with a reduction of 30% of the memory usage! That was worth it!
The bandwidth increase is in fact caused by the lower memory usage: data
don't have the time to rot in the L3 cache anymore, and that was the
original goal. The gains will definitely vary depending on the CPUs and
traffic patterns though. For now QUIC doesn't benefit from it. We also
found that the H2 mux was wasting a lot of memory on single-stream
transfers (i.e. someone watching a video), because all of the 32 buffers
a connection can have could be allocated for this single stream. This
was changed so that on average there are no longer more buffers than
streams, and again this significantly reduced memory usage on such a
traffic pattern. I don't exclude that a few more low-hanging fruits
like this are found in the next few weeks since a lot of testing is
still ongoing for the release.

The config check with "haproxy -c" is now quiet by default in that it
won't pollute logs with "config is valid" every time it's called. However
it can still be enabled using -V. Apparently some users were running
"haproxy -c -q" to remove it but in this case they didn't see warnings
nor errors so that was definitely not a great idea. And the warning
about -S missing -W was finally upgraded to an error (everyone has been
warned since 1.8, it should be sufficient by now).

Finally the previously mentioned issue concerning pool allocation that
seemed to be leaking was found and fixed, it was a stupidly inverted
min/max calculation in the pool usage estimation which would go wild
after the first inconsistent result. A few tiny improvements were
brought in the debugging tools, such as the duration of measuring period
for "show profiling", history of the last caller of "task_queue" that
helps figure when a task goes wild, ability to filter on suspicious or
too old streams in "show sess" and stuff like this. That's mostly for
advanced users who want to quickly spot what's going on.

That's about all for this one. If things continue to calm down, I think
it's reasonable to expect a release by the end of the month, maybe the
29th or the 30th. If new problems surface we'll postpone. I'll probably
emit yet another dev release next week to help with testing fixes.

Till now there's still some doc to update (I still want to see a section
dedicated to actions), some tidying of some files to ease backports later,
and some remaining low-risk stuff depending on how it goes. For example
the reverse http client creates all the connections from a same thread
and it seems that spreading them over multiple threads is no big deal,
so that may appear soon. I also reopened my branch on the CPU selection
that wasn't touched since July and found that the initial patches are
mostly a cleanup of certain hacks we currently have. I may have a look
there to estimate if it's worth taking some of them.

In the mean time, continue to test and report issues. It's always easier
to fix them before than after a release.

Please find the usual URLs below :
   Site index       : https://www.haproxy.org/
   Documentation    : https://docs.haproxy.org/
   Wiki             : https://github.com/haproxy/wiki/wiki
   Discourse        : https://discourse.haproxy.org/
   Slack channel    : https://slack.haproxy.org/
   Issue tracker    : https://github.com/haproxy/haproxy/issues
   Sources          : https://www.haproxy.org/download/2.9/src/
   Git repository   : https://git.haproxy.org/git/haproxy.git/
   Git Web browsing : https://git.haproxy.org/?p=haproxy.git
   Changelog        : https://www.haproxy.org/download/2.9/src/CHANGELOG
   Dataplane API    : 
   Pending bugs     : https://www.haproxy.org/l/pending-bugs
   Reviewed bugs    : https://www.haproxy.org/l/reviewed-bugs
   Code reports     : https://www.haproxy.org/l/code-reports
   Latest builds    : https://www.haproxy.org/l/dev-packages

Complete changelog :
Amaury Denoyelle (17):
      BUG/MEDIUM: quic: fix actconn on quic_conn alloc failure
      BUG/MEDIUM: quic: fix sslconns on quic_conn alloc failure
      MINOR: proto_reverse_connect: use connect timeout
      MEDIUM: quic: adjust address validation
      MINOR: quic: reduce half open counters scope
      MEDIUM: quic: limit handshake per listener
      MEDIUM: quic: define an accept queue limit
      BUG/MINOR: quic: fix retry token check inconsistency
      BUG/MINOR: quic: remove dead code in error path
      MEDIUM: quic: respect closing state even on soft-stop
      MEDIUM: quic: release conn socket before using quic_cc_conn
      BUG/MINOR: quic: fix crash on qc_new_conn alloc failure
      BUG/MINOR: quic: fix decrement of half_open counter on qc alloc failure
      BUG/MEDIUM: quic: fix FD for quic_cc_conn
      MINOR: connection: update rhttp flags usage
      BUG/MINOR: mux_h2: reject passive reverse conn if error on add to idle
      MINOR: server: force add to idle on reverse

Aurelien DARRAGON (29):
      BUG/MINOR: stick-table/cli: Check for invalid ipv4 key
      MEDIUM: stktable/cli: simplify entry key handling
      MINOR: stktable/cli: support v6tov4 and v4tov6 conversions
      MINOR: tools: make str2sa_range() directly return type hints
      BUG/MEDIUM: server: invalid address (post)parsing checks
      BUG/MINOR: sink: don't learn srv port from srv addr
      CLEANUP: sink: bad indent in sink_new_from_logger()
      CLEANUP: sink: useless leftover in sink_add_srv()
      BUG/MINOR: log: keep the ref in dup_logger()
      MINOR: stktable: add stktable_deinit function
      BUG/MINOR: proxy/stktable: missing frees on proxy cleanup
      CLEANUP: backend: removing unused LB param
      MEDIUM: lbprm: store algo params on 32bits
      MEDIUM: log/balance: merge tcp/http algo with log ones
      Revert "MINOR: proxy: report a warning for max_ka_queue in 
      Revert "MINOR: tcp_rules: tcp-{request,response} requires TCP or HTTP 
      Revert "MINOR: stktable: "stick" requires TCP or HTTP mode"
      Revert "MINOR: cfgparse-listen: "http-send-name-header" requires TCP or 
HTTP mode"
      Revert "MINOR: cfgparse-listen: "dynamic-cookie-key" requires TCP or HTTP 
      Revert "MINOR: cfgparse-listen: "http-reuse" requires TCP or HTTP mode"
      Revert "MINOR: fcgi-app: "use-fcgi-app" requires TCP or HTTP mode"
      Revert "MINOR: http_htx/errors: prevent the use of some keywords when not 
in tcp/http mode"
      Revert "MINOR: flt_http_comp: "compression" requires TCP or HTTP mode"
      Revert "MINOR: filter: "filter" requires TCP or HTTP mode"
      MINOR: log/backend: ensure log exclusive params are not used in other 
      MINOR: log/backend: prevent tcp-{request,response} use with LOG mode
      MINOR: log/backend: prevent stick table and stick rules with LOG mode
      MINOR: log/backend: prevent "http-send-name-header" use with LOG mode
      MINOR: log/backend: prevent "dynamic-cookie-key" use with LOG mode

Christopher Faulet (32):
      BUG/MEDIUM: stconn: Report send activity during mux-to-mux fast-forward
      BUG/MEDIUM: stconn: Don't report rcv/snd expiration date if SC cannot 
      MINOR: stconn: Don't queue stream task in past in sc_notify()
      BUG/MEDIUM: Don't apply a max value on room_needed in sc_need_room()
      BUG/MINOR: stconn: Sanitize report for read activity
      CLEANUP: htx: Properly indent htx_reserve_max_data() function
      DOC: stconn: Improve comments about lra and fsb usage
      BUG/MEDIUM: mux-h1: Be sure xprt support splicing to use it during 
      BUG/MINOR: mux-h1: Release empty ibuf during data fast-forwarding
      BUG/MINOR: mux-h1: Properly handle http-request and http-keep-alive 
      BUG/MEDIUM: freq-ctr: Don't report overshoot for long inactivity period
      MEDIUM: stconn/muxes: Loop on data fast-forwarding to forward at least a 
      MINOR: stconn/mux-h2: Use a iobuf flag to report EOI to consumer side 
during FF
      BUG/MEDIUM: stconn: Don't update stream expiration date if already expired
      BUG/MEDIUM: mux-h1: Exit early if fast-forward is not supported by 
opposite SC
      BUG/MEDIUM: applet: Remove appctx from buffer wait list on release
      BUG/MINOR: sample: Fix bytes converter if offset is bigger than sample 
      DOC: config: Fix name for tune.disable-zero-copy-forwarding global param
      BUG/MINOR: stconn: Handle abortonclose if backend connection was already 
set up
      MINOR: connection: Add a CTL flag to notify mux it should wait for reads 
      MEDIUM: mux-h1: Handle MUX_SUBS_RECV flag in h1_ctl() and susbscribe for 
      BUG/MEDIUM: stream: Properly handle abortonclose when set on backend only
      MINOR: stconn: Use SC to detect frontend connections in sc_conn_recv()
      REGTESTS: http: Improve script testing abortonclose option
      BUG/MEDIUM: stream: Don't call mux .ctl() callback if not implemented
      BUG/MEDIUM: stconn: Update fsb date on partial sends
      MINOR: htx: Use a macro for overhead induced by HTX
      MINOR: channel: Add functions to get info on buffers and deal with HTX 
      BUG/MINOR: stconn: Fix streamer detection for HTX streams
      BUG/MINOR: stconn: Use HTX-aware channel's functions to get info on buffer
      BUG/MINOR: stconn/applet: Report send activity only if there was output 
      BUG/MINOR: stconn: Report read activity on non-indep streams for partial 

Frédéric Lécaille (26):
      MEDIUM: quic: Heavy task mode during handshake
      MEDIUM: quic: Heavy task mode with non contiguously bufferized CRYPTO data
      MINOR: quic: release the TLS context asap from quic_conn_release()
      MINOR: quic: Add idle timer task pointer to traces
      BUG/MINOR: quic: idle timer task requeued in the past
      CLEANUP: quic: Indentation fix in qc_do_build_pkt()
      MINOR: quic: Avoid zeroing frame structures
      BUG/MEDIUM: quic: Too short Initial packet sent (enc. level allocation 
      BUG/MEDIUM: quic: Avoid trying to send ACK frames from an empty ack 
ranges tree
      BUG/MEDIUM: quic: Possible crashes when sending too short Initial packets
      BUG/MEDIUM: quic: Avoid some crashes upon TX packet allocation failures
      BUG/MEDIUM: quic: Possible crashes during secrets allocations (heavy load)
      BUG/MINOR: quic: Useless use of non-contiguous buffer for in order CRYPTO 
      BUG/MEDIUM: quic: Non initialized CRYPTO data stream deferencing
      MINOR: quic: Add a max window parameter to congestion control algorithms
      MINOR: quic: Maximum congestion control window for each algo
      DOC: quic: Wrong syntax for "quic-cc-algo" keyword.
      DOC: quic: Maximum congestion control window configuration
      BUG/MINOR: quic: maximum window limits do not match the doc
      MINOR: proxy: Add "handshake" new timeout (frontend side)
      MEDIUM: quic: Add support for "handshake" timeout setting.
      MINOR: quic: Dump the expiration date of the idle timer task
      BUG/MINOR: quic: Malformed CONNECTION_CLOSE frame
      MEDIUM: session: handshake timeout (TCP)
      DOC: proxy: Add "handshake" timeout documentation.
      MINOR: quic: Rename "handshake" timeout to "client-hs"

Remi Tricot-Le Breton (19):
      MINOR: shctx: Set last_append to NULL when reserving block in hot list
      MEDIUM: shctx: Move list between hot and avail list in O(1)
      MEDIUM: shctx: Simplify shctx_row_reserve_hot loop
      MINOR: shctx: Remove explicit 'from' param from shctx_row_data_append
      MEDIUM: cache: Use dedicated cache tree lock alongside shctx lock
      MINOR: cache: Remove expired entry delete in "show cache" command
      MINOR: cache: Add option to avoid removing expired entries in lookup 
      MEDIUM: cache: Use rdlock on cache in cache_use
      MEDIUM: shctx: Remove 'hot' list from shared_context
      MINOR: cache: Use dedicated trash for "show cache" cli command
      MEDIUM: cache: Switch shctx spinlock to rwlock and restrict its scope
      MEDIUM: cache: Add refcount on cache_entry
      MEDIUM: shctx: Descend shctx_lock calls into the shctx_row_reserve_hot
      MINOR: shctx: Add new reserve_finish callback call to 
      MAJOR: cache: Delay cache entry delete in reserve_hot function
      MINOR: shctx: Remove redundant arg from free_block callback
      MINOR: shctx: Remove 'use_shared_mem' variable
      DOC: cache: Specify when function expects a cache lock
      BUG/MINOR: shctx: Remove old HA_SPIN_INIT

Tim Duesterhus (1):
      CLEANUP: Re-apply xalloc_size.cocci (3)

William Lallemand (9):
      MINOR: errors: ha_alert() and ha_warning() uses warn_exec_path()
      MINOR: errors: does not check MODE_STARTING for log emission
      MEDIUM: errors: move the MODE_QUIET test in print_message()
      DOC: management: -q is quiet all the time
      MEDIUM: mworker: -W is mandatory when using -S
      MEDIUM: startup: 'haproxy -c' is quiet when valid
      REGTESTS: startup: -conf-OK requires -V with current VTest
      BUG/MEDIUM: mworker: set the master variable earlier
      REGTESTS: try to activate again the seamless reload test with the master 

Willy Tarreau (21):
      BUG/MEDIUM: pool: fix releasable pool calculation when overloaded
      BUG/MINOR: pool: check one other random bucket on alloc conflict
      BUG/MEDIUM: pool: try once to allocate from another bucket if empty
      MINOR: task/debug: explicitly support passing a null caller to wakeup 
      MINOR: task/debug: make task_queue() and task_schedule() possible callers
      OPTIM: mux-h2: don't allocate more buffers per connections than streams
      DOC: config: use the word 'backend' instead of 'proxy' in 'track' 
      MINOR: server: always initialize pp_tlvs for default servers
      BUG/MEDIUM: proxy: always initialize the default settings after init
      BUG/MEDIUM: connection: report connection errors even when no mux is 
      MINOR: activity: report profiling duration and age in "show profiling"
      CLEANUP: haproxy: remove old comment from 1.1 from the file header
      BUG/MEDIUM: mux-h2: fail earlier on malloc in takeover()
      BUG/MEDIUM: mux-h1: fail earlier on malloc in takeover()
      BUG/MEDIUM: mux-fcgi: fail earlier on malloc in takeover()
      MINOR: rhttp: remove the unused outgoing connect() function
      MINOR: backend: without ->connect(), allow to pick another thread's 
      BUG/MINOR: stream/cli: report correct stream age in "show sess"
      MINOR: stream/cli: add an optional "older" filter for "show sess"
      MINOR: stream/cli: add another filter "susp" to "show sess"
      REGTESTS: http: add a test to validate chunked responses delivery


Reply via email to