Hi,
HAProxy 3.0-dev6 was released on 2024/03/26. It added 114 new commits
after version 3.0-dev5.
This release is slightly late for having been busier than expected last
week and accumulating last-minute regressions (now fixed). Here's what's
new this time:
- a bit less bug fixes than usual (~20), some of which were long-standing
ones, such as the issues around "del server" handling regarding idle
connections, those affecting H2's handling of NTLM on the backend, and
the final fix for the OCSP refcounting that should now be OK.
- some recent build regressions affecting old compilers were fixed. It
was verified that gcc-4.2.1 now works fine (albeit with tons of
warnings, but those dealing with old OSes might appreciate). gcc-4.4
is mostly clean and 4.8.5 produces zero warning for me.
- ring: after having been complaining a lot about the ring locking that
used to seriously limit our ability to produce debugging traces, I
finally managed to attack the problem and make them almost lockless
with parallel writes in groups of threads. The performance went from
x2.5 on NUMA machines to x20 on more uniform machines, caping to around
7 million messages/s. I would have expected even more but it appears
that we're really reaching the limits of the CPU's L3 cache latency
when touching the same offsets (to keep messages ordered). After all
it's not that bad already and can definitely help enable traces in
production again. I couldn't trigger a single watchdog anymore after
that change. The "haring" utility was updated to automatically detect
the new, slightly different format and support both the old and the
new ones (the old haring tool will read the new one in repair mode).
- the H2 mux now supports a setting to fix the maximum number of glitches
that are acceptable over a connection before it's forcefully closed.
I've already got some reports confirming that it's already quite
effective against certain classes of annoying attacks that consist in
eating your CPU with useless traffic. I started to backport this to
2.9 and 2.8 as we've got some reports from users facing a growing
trend of such attacks, as we could anticipate a few months ago.
- the idle connection takeover code was revisited a little bit in order
to permit to forcefully close some idle conns. This allows the "del
server" command to automatically close idle connections instead of
having to wait for them to die by themselves. This will significantly
speed up hot server removal.
- there is now a global setting "ssl-security-level" that allows to
adjust OpenSSL's internal security level, beween 0 and 5. Like probably
most of those reading this, I had never heard about this before :-)
- the "insecure-fork-wanted" option now has an equivalent on the command
line, "-dI". It's convenient to obtain decoded ASAN outputs for
example, without having to edit a config
- spoe: the SPOE engine for now was marked as deprecated by lack of time
assigned to it and the inability to maintain it in its current state as
its existence is making lots of low-level development more complicated.
However, since then, some concerns were raised about this (so apparently
there are more users than initially imagined), and several of us have
had some private discussions on this topic to see how to address the
problem without making it a pain for existing users. It seems that
we're aiming at finally rewriting it according to our modern standards
that will make it faster, more efficient, more reliable and more
importantly: maintainable. There are still ongoing discussions about
this, more on that later. For now we've left the deprecation warning
in the code, but it may possibly change to just something that will
warn about possible future incompatibilities in order to ease the
transition to 3.1, and maybe we'll anticipate the support for some
future keywords or options to make it easier to switch back-and-forth
between 3.0 and 3.1. To be honest, thinking that we'll have to keep it
in the current state for 5 years in 3.0 depresses me, so I think that
as soon as we have a good alternative, I'll aggressively press users
to upgrade so we don't have to deal with it anymore.
- support for FreeBSD 14+ was added.
- some CI updates to increase coverage (debug lists, enable forks for
ASAN traces)
- various doc updates and cleanups
At this point we're getting close to roughly 2 months before the release
and I urge anyone with sensitive stuff to finish it. There are still a huge
number of pending issues in the issue tracker, it seems they arrive at a rate
of roughly one per day and each of them takes a week to address, so at this
pace we'll never finish everything that's started if we don't try to narrow
our focus down a little bit.
I'm still having some reviews pending (log updates, capabilities fixes and
updates). There are also a number of long-pending things that I'd like to
see addressed before the release:
- mt_lists: try again to merge the updated ones. A few functions need to
be rewritten for the new attempt, no time for now to work on this, we'll
see close to the end of the cycle I guess. Importance: +++ for large
machines.
- stick tables sharding: very encouraging tests by Felipe Damasio and
Ricardo Sanchez from Taghos, showing that 15 minutes of code could
dramatically change the locking overhead. It just indicates that we
must absolutely finish this before 3.0 (not much either, iterate over
the peers and the CLI dump; the peers part might be trickier since I
don't know it).
- http-request yielding: we've got a report of a case where a super
expensive converter (a json decoder possibly under attack) was used
in 50 successive http-request rules. The sad thing is that during all
this time the process doesn't make progress anywhere else, and when
it lasts more than 2 seconds (an eternity in network sime scale),
the watchdog is irritated enough to bite. This could be improved by
making it possible to yield after every N rules. It's just a little
bit tricky because right now we yield only inside a rule and not
between, so an extra state needs to be added. Annoying for little
value except for users victim of expensive rules.
- reduce the max latency on lua-load: those running some single-threaded
Lua scripts using "lua-load" could benefit from a significant latency
improvement if the max number of instructions were divided by the number
of threads so as to share the time slot more evenly. Seems easy to do
and could also help those suffering from such issues.
- the "wait" command on the CLI that can wait for a server to be
removable should probably be merged back into the "del server" command,
or renamed.
- "balance pack": was planned for 2.5 or so, always forgotten at the end.
That's basically an "inverted leastconn": send traffic to the most
loaded server so as to focus on the least possible number of servers,
allowing to recycle other ones (requires maxconn of course).
- "add server" with cookie: there was a report about "add server" not
supporting cookies. We think we did that on purpose back then as a
first step to avoid having to deal with all implications, but after
looking at that there doesn't seem to be a compelling technical reason
anymore so that should definitely be done.
- "abns2" aka "supporting abstract namespace sockets with other tools
that use a zero-terminated path". We'll need to continue the discussion
and see how to reach something fine and sustainable. Based on the
current analysis made by Tristan, it seems that we should already
have most of it, but that the devil is in the details that are not
covered by "most of it".
- sample fetch to report number of streams per connection, that could
help understand why some objects take more time than others to download
over certain multiplexed protocols.
- for the NUMA-aware automatic binding, I'm giving up again for this one,
it's become hopeless, it's the 3rd year I'm trying to finish it :-(
- for the compact ebtrees, we'll see, we can experiment close to the end
if we estimate that the memory savings are worth a try for stick tables,
config elements names, LRU cache and possibly HTTP cache.
I'm certain that there was a lot of other stuff but I don't have it in
mind at the moment, too many places to check in parallel. But this easily
illustrates that even if everyone stops right now what they're doing and
spend half time on the bugs, the time is scarce already, so let's focus
on what can be finished.
Also we worked on continuing the backports to 2.7 and 2.6 but for now these
were not released. We'll try an see how to improve the maintenance process
so that it's not always the oldest releases that suffer from the lack of
time. In an ideal world we should be able to produce more versions at once
with certain patches kept on hold for the next one. We just never managed
to do it since it takes more than a day to produce 2 versions...
With that said, 3.0-dev6 up and running on haproxy.org now, let us know
how it goes for you :-)
Please find the usual URLs below :
Site index : https://www.haproxy.org/
Documentation : https://docs.haproxy.org/
Wiki : https://github.com/haproxy/wiki/wiki
Discourse : https://discourse.haproxy.org/
Slack channel : https://slack.haproxy.org/
Issue tracker : https://github.com/haproxy/haproxy/issues
Sources : https://www.haproxy.org/download/3.0/src/
Git repository : https://git.haproxy.org/git/haproxy.git/
Git Web browsing : https://git.haproxy.org/?p=haproxy.git
Changelog : https://www.haproxy.org/download/3.0/src/CHANGELOG
Dataplane API :
https://github.com/haproxytech/dataplaneapi/releases/latest
Pending bugs : https://www.haproxy.org/l/pending-bugs
Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs
Code reports : https://www.haproxy.org/l/code-reports
Latest builds : https://www.haproxy.org/l/dev-packages
Willy
---
Complete changelog :
Amaury Denoyelle (13):
MINOR: quic: simplify rescheduling for handshake
MINOR: quic: remove qc_treat_rx_crypto_frms()
MINOR: session: rename private conns elements
BUG/MAJOR: server: do not delete srv referenced by session
BUG/MINOR: session: ensure conn owner is set after insert into session
BUG/MEDIUM: http_ana: ignore NTLM for reuse aggressive/always and no H1
BUG/MAJOR: connection: fix server used_conns with H2 + reuse safe
MINOR: connection: implement conn_release()
MINOR: connection: extend takeover with release option
MEDIUM: server: close idle conn on server deletion
MEDIUM: mux: prepare for takeover on private connections
MEDIUM: server: close private idle connection before server deletion
BUG/MINOR: mux-quic: close all QCS before freeing QCC tasklet
Aurelien DARRAGON (9):
BUG/MINOR: hlua: segfault when loading the same filter from different
contexts
BUG/MINOR: hlua: missing lock in hlua_filter_new()
BUG/MINOR: hlua: fix missing lock in hlua_filter_delete()
DEBUG: lua: precisely identify if stream is stuck inside lua or not
MINOR: hlua: use accessors for stream hlua ctx
BUG/MEDIUM: hlua: streams don't support mixing lua-load with
lua-load-per-thread (2nd try)
BUILD: server: fix build regression on old compilers (<= gcc-4.4)
OPTIM: http_ext: avoid useless copy in http_7239_extract_{ipv4,ipv6}
BUG/MINOR: server: 'source' interface ignored from 'default-server'
directive
Brooks Davis (1):
MINOR: tools: use public interface for FreeBSD get_exec_path()
Christopher Faulet (9):
BUG/MINOR: listener: Wake proxy's mngmt task up if necessary on session
release
BUG/MINOR: listener: Don't schedule frontend without task in
listener_release()
BUG/MEDIUM: spoe: Don't rely on stream's expiration to detect processing
timeout
BUG/MINOR: spoe: Be sure to be able to quickly close IDLE applets on
soft-stop
MAJOR: spoe: Deprecate the SPOE filter
MINOR: cfgparse: Add a global option to expose deprecated directives
MINOR: spoe: Add SPOE filters in the exposed deprecated directives
BUG/MEDIUM: spoe: Return an invalid frame on recv if size is too small
BUG/MEDIUM: mux-fcgi: Properly handle EOM flag on end-of-trailers HTX
block
Dragan Dosen (2):
BUG/MINOR: ssl: fix possible ctx memory leak in sample_conv_aes_gcm()
BUG/MINOR: ssl: do not set the aead_tag flags in sample_conv_aes_gcm()
Ilia Shipitsin (2):
CLEANUP: assorted typo fixes in the code and comments
CI: temporarily adjust kernel entropy to work with ASAN/clang
Remi Tricot-Le Breton (8):
BUG/MAJOR: ocsp: Separate refcount per instance and per store
REGTESTS: ssl: Add OCSP related tests
BUG/MEDIUM: ssl: Fix crash when calling "update ssl ocsp-response" when
an update is ongoing
BUG/MEDIUM: ssl: Fix crash in ocsp-update log function
MEDIUM: ssl: Change output of ocsp-update log
MINOR: ssl: Change level of ocsp-update logs
CLEANUP: ssl: Remove undocumented ocsp fetches
REGTESTS: ssl: Add checks on ocsp-update log format
William Lallemand (7):
DOC: configuration: clarify ciphersuites usage (V2)
MEDIUM: ssl: initialize the SSL stack explicitely
MEDIUM: ssl: allow to change the OpenSSL security level from global
section
CLEANUP: ssl: remove useless #ifdef in openssl-compat.h
CI: github: add -DDEBUG_LIST to the default builds
MINOR: debug: enable insecure fork on the command line
CI: github: add -dI to haproxy arguments
Willy Tarreau (63):
MINOR: mux-h2: always use h2c_report_glitch()
MEDIUM: mux-h2: allow to set the glitches threshold to kill a connection
BUG/MINOR: server: fix first server template not being indexed
MINOR: debug: add "debug dev trace" to flood with traces
MINOR: atomic: add a read-specific variant of __ha_cpu_relax()
MINOR: applet: add new function applet_append_line()
MINOR: log/applet: add new function syslog_applet_append_event()
MEDIUM: ring/sink: use applet_append_line()/syslog_applet_append_event()
for readers
REORG: dns/ring: split the ring between the generic one and the DNS one
MEDIUM: ring: move the ring reader code to ring_dispatch_messages()
MEDIUM: sink: move the generic ring forwarder code use
ring_dispatch_messages()
MEDIUM: log/sink: make the log forwarder code use ring_dispatch_messages()
MINOR: buf: add b_add_ofs() to add a count to an absolute position
MINOR: buf: add b_rel_ofs() to turn an absolute offset into a relative one
MINOR: buf: add b_putblk_ofs() to copy a block at a specific position
MINOR: buf: add b_getblk_ofs() that works relative to area and not head
MINOR: ring: make the ring reader use only absolute offsets
MINOR: ring: reserve one special value for the readers count
MINOR: vecpair: add new vector pair based data manipulation mechanisms
MINOR: vecpair: add necessary functions to use vecpairss from/to ring APIs
MINOR: ring: rename totlen vs msglen in ring_write()
MINOR: ring: add ring_data() to report the amount of data in a ring
MINOR: ring: add ring_size() to return the ring's size
MINOR: ring: add ring_dup() to copy a ring into another one
MINOR: ring: also add ring_area(), ring_head(), ring_tail()
MINOR: ring: make callers use ring_data() and ring_size(), not ring->buf
MINOR: errors: use ring_dup() to duplicate the startup_logs
MINOR: ring: use ring_size(), ring_area(), ring_head() and ring_tail()
MINOR: ring: add a flag to indicate a mapped file
MAJOR: ring: insert an intermediary ring_storage level
MINOR: ring: resize only under thread isolation
MINOR: ring: allow to reduce a ring size
MEDIUM: ring: replace the buffer API in ring_write() with the vec<->ring
API
MEDIUM: ring: change the ring reader to use the new vector-based API now
MEDIUM: ring: remove the struct buffer from the ring
MEDIUM: ring: align the head and tail fields in the ring_storage structure
MINOR: ring: make the reader check the readers count before inc/dec
MEDIUM: ring: lock the tail's readers counters before proceeding with the
changes
MEDIUM: ring: protect the reader's positions against writers
MEDIUM: ring: use the topmost bit of the tail as a lock
MEDIUM: move the ring's lock to only protect the readers list
MEDIUM: ring: unlock the ring's tail earlier
MINOR: ring: don't take the readers lock if there are no readers
MEDIUM: ring/applet: turn the wait_entry list to an mt_list instead
MEDIUM: ring: protect the initialization of the initial reader offset
MINOR: ring: make sure ring_dispatch waits when facing a changing message
MAJOR: ring: drop the now unneeded lock
OPTIM: ring: don't even try to update offset when failed to read
OPTIM: ring: have only one thread at a time wake up all readers
MINOR: ring: keep a few frequently used pointers in the local stack
MINOR: ring: add the definition of a ring waiting cell
MINOR: ring: make the number of queues configurable
MAJOR: ring: implement a waiting queue in front of the ring
MEDIUM: ring: significant boost in the loop by checking the ring queue
ptr first
MEDIUM: ring: improve speed in the queue waiting loop on x86_64
MINOR: ring: simplify the write loop a little bit
CLEANUP: ring: further simplify the write loop
MINOR: ring: it's not x86 but all non-ARMv8.1 which needs the read before
OR
MINOR: ring: avoid writes to cells during copy
OPTIM: ring: use relaxed stores to release the threads
CLEANUP: ring: use only curr_cell and not next_cell in the main write loop
BUILD: ssl: fix build error on older compilers with openssl-3.2
BUG/MAJOR: ring: free the ring storage not the ring itself when using maps
---
