Hi, HAProxy 3.0-dev6 was released on 2024/03/26. It added 114 new commits after version 3.0-dev5.
This release is slightly late for having been busier than expected last week and accumulating last-minute regressions (now fixed). Here's what's new this time: - a bit less bug fixes than usual (~20), some of which were long-standing ones, such as the issues around "del server" handling regarding idle connections, those affecting H2's handling of NTLM on the backend, and the final fix for the OCSP refcounting that should now be OK. - some recent build regressions affecting old compilers were fixed. It was verified that gcc-4.2.1 now works fine (albeit with tons of warnings, but those dealing with old OSes might appreciate). gcc-4.4 is mostly clean and 4.8.5 produces zero warning for me. - ring: after having been complaining a lot about the ring locking that used to seriously limit our ability to produce debugging traces, I finally managed to attack the problem and make them almost lockless with parallel writes in groups of threads. The performance went from x2.5 on NUMA machines to x20 on more uniform machines, caping to around 7 million messages/s. I would have expected even more but it appears that we're really reaching the limits of the CPU's L3 cache latency when touching the same offsets (to keep messages ordered). After all it's not that bad already and can definitely help enable traces in production again. I couldn't trigger a single watchdog anymore after that change. The "haring" utility was updated to automatically detect the new, slightly different format and support both the old and the new ones (the old haring tool will read the new one in repair mode). - the H2 mux now supports a setting to fix the maximum number of glitches that are acceptable over a connection before it's forcefully closed. I've already got some reports confirming that it's already quite effective against certain classes of annoying attacks that consist in eating your CPU with useless traffic. I started to backport this to 2.9 and 2.8 as we've got some reports from users facing a growing trend of such attacks, as we could anticipate a few months ago. - the idle connection takeover code was revisited a little bit in order to permit to forcefully close some idle conns. This allows the "del server" command to automatically close idle connections instead of having to wait for them to die by themselves. This will significantly speed up hot server removal. - there is now a global setting "ssl-security-level" that allows to adjust OpenSSL's internal security level, beween 0 and 5. Like probably most of those reading this, I had never heard about this before :-) - the "insecure-fork-wanted" option now has an equivalent on the command line, "-dI". It's convenient to obtain decoded ASAN outputs for example, without having to edit a config - spoe: the SPOE engine for now was marked as deprecated by lack of time assigned to it and the inability to maintain it in its current state as its existence is making lots of low-level development more complicated. However, since then, some concerns were raised about this (so apparently there are more users than initially imagined), and several of us have had some private discussions on this topic to see how to address the problem without making it a pain for existing users. It seems that we're aiming at finally rewriting it according to our modern standards that will make it faster, more efficient, more reliable and more importantly: maintainable. There are still ongoing discussions about this, more on that later. For now we've left the deprecation warning in the code, but it may possibly change to just something that will warn about possible future incompatibilities in order to ease the transition to 3.1, and maybe we'll anticipate the support for some future keywords or options to make it easier to switch back-and-forth between 3.0 and 3.1. To be honest, thinking that we'll have to keep it in the current state for 5 years in 3.0 depresses me, so I think that as soon as we have a good alternative, I'll aggressively press users to upgrade so we don't have to deal with it anymore. - support for FreeBSD 14+ was added. - some CI updates to increase coverage (debug lists, enable forks for ASAN traces) - various doc updates and cleanups At this point we're getting close to roughly 2 months before the release and I urge anyone with sensitive stuff to finish it. There are still a huge number of pending issues in the issue tracker, it seems they arrive at a rate of roughly one per day and each of them takes a week to address, so at this pace we'll never finish everything that's started if we don't try to narrow our focus down a little bit. I'm still having some reviews pending (log updates, capabilities fixes and updates). There are also a number of long-pending things that I'd like to see addressed before the release: - mt_lists: try again to merge the updated ones. A few functions need to be rewritten for the new attempt, no time for now to work on this, we'll see close to the end of the cycle I guess. Importance: +++ for large machines. - stick tables sharding: very encouraging tests by Felipe Damasio and Ricardo Sanchez from Taghos, showing that 15 minutes of code could dramatically change the locking overhead. It just indicates that we must absolutely finish this before 3.0 (not much either, iterate over the peers and the CLI dump; the peers part might be trickier since I don't know it). - http-request yielding: we've got a report of a case where a super expensive converter (a json decoder possibly under attack) was used in 50 successive http-request rules. The sad thing is that during all this time the process doesn't make progress anywhere else, and when it lasts more than 2 seconds (an eternity in network sime scale), the watchdog is irritated enough to bite. This could be improved by making it possible to yield after every N rules. It's just a little bit tricky because right now we yield only inside a rule and not between, so an extra state needs to be added. Annoying for little value except for users victim of expensive rules. - reduce the max latency on lua-load: those running some single-threaded Lua scripts using "lua-load" could benefit from a significant latency improvement if the max number of instructions were divided by the number of threads so as to share the time slot more evenly. Seems easy to do and could also help those suffering from such issues. - the "wait" command on the CLI that can wait for a server to be removable should probably be merged back into the "del server" command, or renamed. - "balance pack": was planned for 2.5 or so, always forgotten at the end. That's basically an "inverted leastconn": send traffic to the most loaded server so as to focus on the least possible number of servers, allowing to recycle other ones (requires maxconn of course). - "add server" with cookie: there was a report about "add server" not supporting cookies. We think we did that on purpose back then as a first step to avoid having to deal with all implications, but after looking at that there doesn't seem to be a compelling technical reason anymore so that should definitely be done. - "abns2" aka "supporting abstract namespace sockets with other tools that use a zero-terminated path". We'll need to continue the discussion and see how to reach something fine and sustainable. Based on the current analysis made by Tristan, it seems that we should already have most of it, but that the devil is in the details that are not covered by "most of it". - sample fetch to report number of streams per connection, that could help understand why some objects take more time than others to download over certain multiplexed protocols. - for the NUMA-aware automatic binding, I'm giving up again for this one, it's become hopeless, it's the 3rd year I'm trying to finish it :-( - for the compact ebtrees, we'll see, we can experiment close to the end if we estimate that the memory savings are worth a try for stick tables, config elements names, LRU cache and possibly HTTP cache. I'm certain that there was a lot of other stuff but I don't have it in mind at the moment, too many places to check in parallel. But this easily illustrates that even if everyone stops right now what they're doing and spend half time on the bugs, the time is scarce already, so let's focus on what can be finished. Also we worked on continuing the backports to 2.7 and 2.6 but for now these were not released. We'll try an see how to improve the maintenance process so that it's not always the oldest releases that suffer from the lack of time. In an ideal world we should be able to produce more versions at once with certain patches kept on hold for the next one. We just never managed to do it since it takes more than a day to produce 2 versions... With that said, 3.0-dev6 up and running on haproxy.org now, let us know how it goes for you :-) Please find the usual URLs below : Site index : https://www.haproxy.org/ Documentation : https://docs.haproxy.org/ Wiki : https://github.com/haproxy/wiki/wiki Discourse : https://discourse.haproxy.org/ Slack channel : https://slack.haproxy.org/ Issue tracker : https://github.com/haproxy/haproxy/issues Sources : https://www.haproxy.org/download/3.0/src/ Git repository : https://git.haproxy.org/git/haproxy.git/ Git Web browsing : https://git.haproxy.org/?p=haproxy.git Changelog : https://www.haproxy.org/download/3.0/src/CHANGELOG Dataplane API : https://github.com/haproxytech/dataplaneapi/releases/latest Pending bugs : https://www.haproxy.org/l/pending-bugs Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs Code reports : https://www.haproxy.org/l/code-reports Latest builds : https://www.haproxy.org/l/dev-packages Willy --- Complete changelog : Amaury Denoyelle (13): MINOR: quic: simplify rescheduling for handshake MINOR: quic: remove qc_treat_rx_crypto_frms() MINOR: session: rename private conns elements BUG/MAJOR: server: do not delete srv referenced by session BUG/MINOR: session: ensure conn owner is set after insert into session BUG/MEDIUM: http_ana: ignore NTLM for reuse aggressive/always and no H1 BUG/MAJOR: connection: fix server used_conns with H2 + reuse safe MINOR: connection: implement conn_release() MINOR: connection: extend takeover with release option MEDIUM: server: close idle conn on server deletion MEDIUM: mux: prepare for takeover on private connections MEDIUM: server: close private idle connection before server deletion BUG/MINOR: mux-quic: close all QCS before freeing QCC tasklet Aurelien DARRAGON (9): BUG/MINOR: hlua: segfault when loading the same filter from different contexts BUG/MINOR: hlua: missing lock in hlua_filter_new() BUG/MINOR: hlua: fix missing lock in hlua_filter_delete() DEBUG: lua: precisely identify if stream is stuck inside lua or not MINOR: hlua: use accessors for stream hlua ctx BUG/MEDIUM: hlua: streams don't support mixing lua-load with lua-load-per-thread (2nd try) BUILD: server: fix build regression on old compilers (<= gcc-4.4) OPTIM: http_ext: avoid useless copy in http_7239_extract_{ipv4,ipv6} BUG/MINOR: server: 'source' interface ignored from 'default-server' directive Brooks Davis (1): MINOR: tools: use public interface for FreeBSD get_exec_path() Christopher Faulet (9): BUG/MINOR: listener: Wake proxy's mngmt task up if necessary on session release BUG/MINOR: listener: Don't schedule frontend without task in listener_release() BUG/MEDIUM: spoe: Don't rely on stream's expiration to detect processing timeout BUG/MINOR: spoe: Be sure to be able to quickly close IDLE applets on soft-stop MAJOR: spoe: Deprecate the SPOE filter MINOR: cfgparse: Add a global option to expose deprecated directives MINOR: spoe: Add SPOE filters in the exposed deprecated directives BUG/MEDIUM: spoe: Return an invalid frame on recv if size is too small BUG/MEDIUM: mux-fcgi: Properly handle EOM flag on end-of-trailers HTX block Dragan Dosen (2): BUG/MINOR: ssl: fix possible ctx memory leak in sample_conv_aes_gcm() BUG/MINOR: ssl: do not set the aead_tag flags in sample_conv_aes_gcm() Ilia Shipitsin (2): CLEANUP: assorted typo fixes in the code and comments CI: temporarily adjust kernel entropy to work with ASAN/clang Remi Tricot-Le Breton (8): BUG/MAJOR: ocsp: Separate refcount per instance and per store REGTESTS: ssl: Add OCSP related tests BUG/MEDIUM: ssl: Fix crash when calling "update ssl ocsp-response" when an update is ongoing BUG/MEDIUM: ssl: Fix crash in ocsp-update log function MEDIUM: ssl: Change output of ocsp-update log MINOR: ssl: Change level of ocsp-update logs CLEANUP: ssl: Remove undocumented ocsp fetches REGTESTS: ssl: Add checks on ocsp-update log format William Lallemand (7): DOC: configuration: clarify ciphersuites usage (V2) MEDIUM: ssl: initialize the SSL stack explicitely MEDIUM: ssl: allow to change the OpenSSL security level from global section CLEANUP: ssl: remove useless #ifdef in openssl-compat.h CI: github: add -DDEBUG_LIST to the default builds MINOR: debug: enable insecure fork on the command line CI: github: add -dI to haproxy arguments Willy Tarreau (63): MINOR: mux-h2: always use h2c_report_glitch() MEDIUM: mux-h2: allow to set the glitches threshold to kill a connection BUG/MINOR: server: fix first server template not being indexed MINOR: debug: add "debug dev trace" to flood with traces MINOR: atomic: add a read-specific variant of __ha_cpu_relax() MINOR: applet: add new function applet_append_line() MINOR: log/applet: add new function syslog_applet_append_event() MEDIUM: ring/sink: use applet_append_line()/syslog_applet_append_event() for readers REORG: dns/ring: split the ring between the generic one and the DNS one MEDIUM: ring: move the ring reader code to ring_dispatch_messages() MEDIUM: sink: move the generic ring forwarder code use ring_dispatch_messages() MEDIUM: log/sink: make the log forwarder code use ring_dispatch_messages() MINOR: buf: add b_add_ofs() to add a count to an absolute position MINOR: buf: add b_rel_ofs() to turn an absolute offset into a relative one MINOR: buf: add b_putblk_ofs() to copy a block at a specific position MINOR: buf: add b_getblk_ofs() that works relative to area and not head MINOR: ring: make the ring reader use only absolute offsets MINOR: ring: reserve one special value for the readers count MINOR: vecpair: add new vector pair based data manipulation mechanisms MINOR: vecpair: add necessary functions to use vecpairss from/to ring APIs MINOR: ring: rename totlen vs msglen in ring_write() MINOR: ring: add ring_data() to report the amount of data in a ring MINOR: ring: add ring_size() to return the ring's size MINOR: ring: add ring_dup() to copy a ring into another one MINOR: ring: also add ring_area(), ring_head(), ring_tail() MINOR: ring: make callers use ring_data() and ring_size(), not ring->buf MINOR: errors: use ring_dup() to duplicate the startup_logs MINOR: ring: use ring_size(), ring_area(), ring_head() and ring_tail() MINOR: ring: add a flag to indicate a mapped file MAJOR: ring: insert an intermediary ring_storage level MINOR: ring: resize only under thread isolation MINOR: ring: allow to reduce a ring size MEDIUM: ring: replace the buffer API in ring_write() with the vec<->ring API MEDIUM: ring: change the ring reader to use the new vector-based API now MEDIUM: ring: remove the struct buffer from the ring MEDIUM: ring: align the head and tail fields in the ring_storage structure MINOR: ring: make the reader check the readers count before inc/dec MEDIUM: ring: lock the tail's readers counters before proceeding with the changes MEDIUM: ring: protect the reader's positions against writers MEDIUM: ring: use the topmost bit of the tail as a lock MEDIUM: move the ring's lock to only protect the readers list MEDIUM: ring: unlock the ring's tail earlier MINOR: ring: don't take the readers lock if there are no readers MEDIUM: ring/applet: turn the wait_entry list to an mt_list instead MEDIUM: ring: protect the initialization of the initial reader offset MINOR: ring: make sure ring_dispatch waits when facing a changing message MAJOR: ring: drop the now unneeded lock OPTIM: ring: don't even try to update offset when failed to read OPTIM: ring: have only one thread at a time wake up all readers MINOR: ring: keep a few frequently used pointers in the local stack MINOR: ring: add the definition of a ring waiting cell MINOR: ring: make the number of queues configurable MAJOR: ring: implement a waiting queue in front of the ring MEDIUM: ring: significant boost in the loop by checking the ring queue ptr first MEDIUM: ring: improve speed in the queue waiting loop on x86_64 MINOR: ring: simplify the write loop a little bit CLEANUP: ring: further simplify the write loop MINOR: ring: it's not x86 but all non-ARMv8.1 which needs the read before OR MINOR: ring: avoid writes to cells during copy OPTIM: ring: use relaxed stores to release the threads CLEANUP: ring: use only curr_cell and not next_cell in the main write loop BUILD: ssl: fix build error on older compilers with openssl-3.2 BUG/MAJOR: ring: free the ring storage not the ring itself when using maps ---