Hi,
HAProxy 2.7-dev8 was released on 2022/10/14. It added 81 new commits
after version 2.7-dev7.
Not much changed since dev7, we're seeing a few fixes and twice as many
cleanup and doc patches. That's nice. Other improvements are roughly
spread like this:
- master-cli: the "reload" command now reports the operation status and
dumps the startup logs. This is particularly useful in environments
like systemd where there's normally no feedback reload on actions.
The feature requires shm_open() and is currently only enabled on the
linux-glibc target, though I could verify that it does also work on
freebsd. Testers are welcome, just pass USE_SHM_OPEN=1 to the make
command for this.
- health checks: the checks were being scheduled on any thread, which
caused all threads to periodically wake up and compete on the shared
wait queue lock. I noticed that on a machine with 48 threads, only 40
servers were enough to cause fast wakeups and 2% CPU! And as a side
effect, threads were picking many checks at once, causing a significant
imbalance that was directly responsible for some of the higher latency
tails in tests. This was modified so that a check is now rescheduled
on a random thread if that other thread is les loaded. This resulted
in significantly smoothing the load over just a few seconds, and
eliminating the latency spikes and the CPU load since there's no more
contention on the shared lock. However, checks are now run by more
threads, which may cause more CPU usage to be reported in a very
specific case (many servers with very short check interval, ~10ms),
because threads do not have the time to sleep a full time slice
anymore. Such setups are clearly unlikely (and even less likely to
run idle with no load at all) so that's normally not a problem,
considering how it improves a number of shortcomings. But I'm
interested in any reports of increased idle CPU usage if that would
happen.
- quic: the "thread" keyword on bind lines was not respected, all
threads were always used. This was now changed so that it becomes
possible to dedicate only certain threads to QUIC, or possibly to
start several listeners on different thread groups.
- quic: the frontend's destination address was not retrieved previously
due to limitations of the BSD socket API. One of the limitations is
that having a listener on 0.0.0.0 doesn't work if several addresses
are present on the machine, as the response will take the source
address that can reach the target from the routing table. Switching
to recvmsg() allows on certain systems to receive packet info with
datagrams, and among these info is the packet's destination address.
This is automatically detected at build time, and we could verify
that Linux and FreeBSD both work (IPv4 + IPv6). Maybe we should add
an entry for this in "haproxy -vv" to help diagnose incompatible
setups. There were also various other low-level improvements.
- stick-tables: since 1.8, all the locking needed to protect against
concurrent accesses from different threads was being performed using
spinlocks. This doesn't scale with many threads, as can be seen when
using track-sc on a large-scale machine. The code was adapted to
support R/W locks and that completely unleashed the unused performance.
A 48-thread machine jumped from 180k to 2.1M requests per second with
3 track-sc, that's a 11x improvement!
- init: we've seen a few reports of strange errors in certain docker
setups roughly saying "impossible to set the FD limit to 8k, leaving
it to 1 billion". This sounds absurd, but might be caused by certain
syscall restrictions in such environments. There's no point reducing
the number of FDs, it's just a limit. So the code was adjusted so
that we only try to raise the limit but do nothing in case where we
need to lower it.
- lua: some fixes to better match what the doc says, and to avoid
a risk of crash when passing wrong arguments to certain functions
related to stick-tables.
I'm also aware of an upcoming improvement in the Lua area to speed up
startup of large configs (Lua even if unused builds a table of all
backends and servers, and that takes time). And some updates to QUIC
should happen soon to better deal with packet drops, polling and reloads.
With the haproxyconf in less than one month and the amazing amount of time
it takes to the speakers to prepare their talks, I'm anticipating a bit of
slowdown on new patches and bug fixes for the forthcoming weeks (it has
already started for me), so we need to keep reasonable expectations and
goals for the upcoming release to keep a target on end of November, and
at the same time test reports are more than ever well appreciated.
Please find the usual URLs below :
Site index : https://www.haproxy.org/
Documentation : https://docs.haproxy.org/
Wiki : https://github.com/haproxy/wiki/wiki
Discourse : https://discourse.haproxy.org/
Slack channel : https://slack.haproxy.org/
Issue tracker : https://github.com/haproxy/haproxy/issues
Sources : https://www.haproxy.org/download/2.7/src/
Git repository : https://git.haproxy.org/git/haproxy.git/
Git Web browsing : https://git.haproxy.org/?p=haproxy.git
Changelog : https://www.haproxy.org/download/2.7/src/CHANGELOG
Pending bugs : https://www.haproxy.org/l/pending-bugs
Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs
Code reports : https://www.haproxy.org/l/code-reports
Latest builds : https://www.haproxy.org/l/dev-packages
Willy
---
Complete changelog :
Amaury Denoyelle (17):
CLEANUP: mux-quic: remove usage of non-standard ull type
CLEANUP: quic: remove global var definition in quic_tls header
BUG/MINOR: quic: adjust quic_tls prototypes
CLEANUP: quic: fix headers
CLEANUP: quic: remove unused function prototype
CLEANUP: quic: remove duplicated varint code from xprt_quic.h
CLEANUP: quic: create a dedicated quic_conn module
BUG/MINOR: mux-quic: ignore STOP_SENDING for locally closed stream
MINOR: quic: limit usage of ssl_sock_ctx in favor of quic_conn
MINOR: mux-quic: check quic-conn return code on Tx
CLEANUP: quic: fix indentation
MEDIUM: quic: retrieve frontend destination address
BUG/MINOR: quic: set IP_PKTINFO socket option for QUIC receivers only
BUG/MINOR: stick-table: fix build with DEBUG_THREAD
CLEANUP: quic: remove unused rxbufs member in receiver
CLEANUP: quic: improve naming for rxbuf/datagrams handling
MINOR: quic: implement datagram cleanup for quic_receiver_buf
Aurelien DARRAGON (7):
BUG/MINOR: hlua: hlua_channel_insert_data() behavior conflicts with
documentation
MINOR: hlua: some luaL_checktype() calls were not guarded with MAY_LJMP
DOC: configuration: missing 'if' in tcp-request content example
MINOR: hlua: removing ambiguous lua_pushvalue with 0 index
DOC/CLEANUP: lua-api: some minor corrections
DOC: lua-api: updating toolbox link
DOC/CLEANUP: lua-api: removing duplicate core.proxies attribute
Christopher Faulet (8):
DOC: config: Fix pgsql-check documentation to make user param mandatory
BUG/MINOR: http-fetch: Update method after a prefetch in smp_fetch_meth()
REORG: mux-fcgi: Extract flags and enums into mux_fcgi-t.h
MINOR: flags/mux-fcgi: Decode FCGI connection and stream flags
BUG/MEDIUM: mux-h1: Add connection error handling when reading/sending on
a pipe
BUG/MEDIUM: mux-h1: Handle abort with an incomplete message during parsing
BUG/MEDIUM: httpclient: Don't set EOM flag on an empty HTX message
MINOR: httpclient/lua: Don't set req_payload callback if body is empty
Fatih Acar (1):
BUG/MINOR: checks: update pgsql regex on auth packet
Frédéric Lécaille (6):
MINOR: quic: New quic_cstream object implementation
MINOR: quic: Extract CRYPTO frame parsing from qc_parse_pkt_frms()
MINOR: quic: Use a non-contiguous buffer for RX CRYPTO data
BUG/MINOR: quic: Stalled 0RTT connections with big ClientHello TLS message
MINOR: quic: Split the secrets key allocation in two parts
BUILD: ssl_sock: bind_conf uninitialized in ssl_sock_bind_verifycbk()
Ilya Shipitsin (2):
CI: SSL: use proper version generating when "latest" semantic is used
CI: SSL: temporarily stick to LibreSSL=3.5.3
Olivier Houchard (2):
BUG/MEDIUM: lua: Don't crash in hlua_lua2arg_check on failure
BUG/MEDIUM: lua: handle stick table implicit arguments right.
Tim Duesterhus (4):
CLEANUP: Reapply ist.cocci (2)
CLEANUP: Reapply strcmp.cocci
CI: Replace the deprecated `::set-output` command by writing to
$GITHUB_OUTPUT in matrix.py
CI: Replace the deprecated `::set-output` command by writing to
$GITHUB_OUTPUT in workflow definition
William Lallemand (7):
MINOR: ring: ring_cast_from_area() cast from an allocated area
MINOR: buffers: split b_force_xfer() into b_cpy() and b_force_xfer()
MINOR: logs: startup-logs can use a shm for logging the reload
MINOR: mworker/cli: reload command displays the startup-logs
DOC: management: update the "reload" command of the master CLI
DOC: management: add forgotten "show startup-logs"
DOC: management: "show startup-logs" for master CLI
Willy Tarreau (27):
BUILD: h1: silence an initiialized warning with gcc-4.7 and -Os
MINOR: fd: add a new function to only raise RLIMIT_NOFILE
MINOR: init: do not try to shrink existing RLIMIT_NOFIlE
BUILD: http_fetch: silence an uninitiialized warning with gcc-4/5/6 at -Os
CLEANUP: quic/receiver: remove the now unused tx_qring list
BUG/MAJOR: stick-tables: do not try to index a server name for applets
MINOR: plock: support disabling exponential back-off
MINOR: freq_ctr: use the thread's local time whenever possible
MEDIUM: stick-table: switch the table lock to rwlock
MINOR: stick-table: do not take an exclusive lock when downing ref_cnt
MINOR: stick-table: move the write lock inside stktable_touch_with_exp()
MEDIUM: stick-table: only take the lock when needed in
stktable_touch_with_exp()
MEDIUM: stick-table: make stksess_kill_if_expired() avoid the exclusive
lock
MEDIUM: stick-table: return inserted entry in __stktable_store()
MEDIUM: stick-table: free newly allocated stkess if it couldn't be
inserted
MEDIUM: stick-table: switch to rdlock in stktable_lookup() and
lookup_key()
MEDIUM: stick-table: make stktable_get_entry() look up under a read lock
MEDIUM: stick-table: do not take a lock to update t->current anymore.
MEDIUM: stick-table: make stktable_set_entry() look up under a read lock
MEDIUM: stick-table: requeue the expiration task out of the exclusive lock
MINOR: stick-table: split stktable_store() between key and requeue
MEDIUM: stick-table: always use atomic ops to requeue the table's task
MEDIUM: stick-table: requeue the wakeup task out of the write lock
BUG/MINOR: server: make sure "show servers state" hides private bits
MINOR: checks: use the lighter PRNG for spread checks
MEDIUM: checks: spread the checks load over random threads
MEDIUM: quic: respect the threads assigned to a bind line
---
