Hi, HAProxy 2.7.7 was released on 2023/04/27. It added 163 new commits after version 2.7.6.
This release is pretty huge. In one month, the QUIC team achieved an amazing work to improve the stack and make it more stable. A big thanks to Tristan for his priceless help. More than half of commits concern the QUIC stack. It is hard to sum up all changes. Many bugs were fixed, the most visible are: * The Congestion algorithms state was shared between connections instead of being private. * HTTP/1.0 responses with an unknown content length and finished on close were not properly handled. It was considered as an early close by the QUIC multiplexer, leading to a RESET_STREAM emission. * The streams fairness was improved to prevent timeouts. A stream sending a large object could block other streams. With a small client timeout, blocked streams could be aborted via a RESET_STREAM. * Some contradictions in code could lead to very long loops sending empty packets (PADDING only packets). One visible effect was a very low throughput performance when the client serialized its requests. * The control window in congestion algorithms could be zero because of a wrong calculation and could lead to a SIGFPE crash. * Padding was missing in very short probe packets * Possible memory leaks were fixed And of course, some improvements were brought. The main one is the support of the thread loadbalancing on accept. A series of changes allowed to use the default mechanism to accept and handle connections. The less loaded thread is now selected, improving the global performance of the QUIC stack. In addition, a timer was added to delay the acknowledgments. Recent refactoring about the stream-connector layer introduced a regression since the 2.7.4. The read timer was no longer rearmed if the end of the message was reached. This change was introduced to avoid server timeouts when the server replies before the end of the request. But it revealed several bugs, some was fixed, but some others pretty are hard to fix without changing some internals. It is too sensitive for a stable version. Thus for now we decided to revert this change, waiting for a better solution. Note that it is not a big deal because we only restore a behavior that has been there for ages. On soft-stop or reload, idle DNS session are now killed. Since the 2.7.5, these sessions were no longer killed, preventing the process to finish. In addition, we now force the connect timeout for the DNS resolution. The "resolve" timeout is used to set its value. Have no connect timeout was an issue for resolution over TCP. Connection failures might take quite long to report, leading to an excess of unusable DNS sessions in connecting state. It was especially visible on soft-stop because this prevented the process to quickly exit. Still on the DNS, errors are now properly handled when a response is consumed. This was an issue for truncated responses followed by an abort. The applet could ignore the abort and loop waiting for more data until a timeout is triggered. A similar issue was fixed in the syslog applet. Several bugs in lua part were fixed. First, except for lua tasks, it is no longer possible to register functions at runtime. It was clearly stated in the documentation, but nothing forbidden it in the code. An error is now triggered if this happens, preventing potential segfaults. Memory leaks on references were fixed and the lua locking was simplified to be re-entrant to prevent deadlocks. Aurélien fixed several issues on the servers management. The "visible" server list consistency was fixed. It was possible, at least in theory, to access an invalid server if several dynamic server deletions were performed while the list was accessed. For instance it might happen when the server list was dumped in the stats. He also fixed wrong report for tracking servers leaving drain state. Finally, he centralized proxy and server stats updates on server state transition to be sure to not miss an update on some transitions. The issues that were occasionally met around the use of malloc_trim() that had been addressed in 2.8 were finally backported after one month of exposure in 2.8. The issue was that not only malloc_trim() could still sometimes be used when jemalloc (or any other allocator) was used, but our attempts at plugging these special cases didn't work when linking with external libs that also explicitly call it. In the end, the opposite was done: we redefine our own version of malloc_trim(), which contains the tests for the presence of an alternate lib, and call the original if the allocator comes from libc, or call the equivalent function from other allocators. This way external libs that would use it are safe as well. The mixed library version detection in dlopen() was still a bit sensitive, and could sometimes detect anomalies related to an external lib depending on libcrypto but not libssl for example, as well as libs that were linked with an ABI-compatible version of the lib, but not exactly the same one. The tests were improved to only validate the grouped presence of a combination of relevant symbols that allow to distinguish between different ABI versions. It manages to catch libs loaded from Lua that were compiled against a totally different libcrypto version without being triggered when the ABI is compatible, which was the initial purpose of the test. Typically, loading luaossl built against openssl with haproxy build against quictls is properly detected as an error, but loading luaossl built against a slightly older but compatible libssl version than haproxy's (or conversely) is OK. Similarly this has been in 2.8 for one month and a few -dev versions, and some users continue to experience problems in 2.7 so it was about time to backport this. The remaining are the usual bunch of bug fixes: * In the H2 multiplexer, connection errors are now properly detected during handshake. This avoids to insert invalid connection into an idle list. It can be an issue if such connection is the only idle connection. If the traffic is too low to create new connections but sufficient to always reuse it before purging it, no connection to the server is possible. * It was possible to trigger the watchdog purging stick-tables on soft-strop. To not spend too much time purging expired entries, we now enforce a budget limitation and the purge is performed in several steps. In addition, memory is reclaimed only when entries are released. Indeed, this operation involves a call to malloc_trim() on glibc, which is rather expensive. * NUMA topology detected on FreeBSD was fixed. * It was not possible to use the lua filter API if used in conjunction with a "wait-for-body" action. Switching the HTTP message in DATA state preventing the call to most of lua filter functions. It was fixed by keep the HTTP message in BODY state at this stage. * The read expiration date is now updated on synchronous sends for all streams except independent ones. This fixed an old bug when a filter is configured. Write activities on synchronous sends were lost. With slow clients uploading large object, it was possible to reach the server timeout. * More internal variables are unset from program section. More specifically, HAPROXY_STARTUPLOGS_FD, HAPROXY_MWORKER_WAIT_ONLY and HAPROXY_PROCESSES variables are now unset. * ssl-min-ver and ss-max-ver parameters are now duplicated for bundles in crt-list. * A warning is now emitted during configuration parsing if a header field name contains forbidden chars in an HTTP action. A special case is made for the colon at the beginning so that it remains possible to place any future pseudo-headers that may appear. * An error is reported during configuration parsing if when the "len" argument of a stick table type contains incorrect characters. * DeviceAtlas compile options were updated to support the API v3 from 3.1.7 and onwards. * The strict-sni documentation was updated to state it is possible to tart without certificate on a bind line. Thanks everyone for you help and your contributions ! Please find the usual URLs below : Site index : https://www.haproxy.org/ Documentation : https://docs.haproxy.org/ Wiki : https://github.com/haproxy/wiki/wiki Discourse : https://discourse.haproxy.org/ Slack channel : https://slack.haproxy.org/ Issue tracker : https://github.com/haproxy/haproxy/issues Sources : https://www.haproxy.org/download/2.7/src/ Git repository : https://git.haproxy.org/git/haproxy-2.7.git/ Git Web browsing : https://git.haproxy.org/?p=haproxy-2.7.git Changelog : https://www.haproxy.org/download/2.7/src/CHANGELOG Dataplane API : https://github.com/haproxytech/dataplaneapi/releases/latest Pending bugs : https://www.haproxy.org/l/pending-bugs Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs Code reports : https://www.haproxy.org/l/code-reports Latest builds : https://www.haproxy.org/l/dev-packages --- Complete changelog : Amaury Denoyelle (39): MINOR: quic: derive first DCID from client ODCID MINOR: quic: remove ODCID dedicated tree MINOR: quic: remove address concatenation to ODCID BUG/MINOR: task: allow to use tasklet_wakeup_after with tid -1 CLEANUP: quic: remove unused QUIC_LOCK label CLEANUP: quic: remove unused scid_node CLEANUP: quic: remove unused qc param on stateless reset token CLEANUP: quic: rename quic_connection_id vars MINOR: quic: remove uneeded tasklet_wakeup after accept MINOR: quic: adjust Rx packet type parsing MINOR: quic: adjust quic CID derive API MINOR: quic: remove TID ref from quic_conn BUG/MINOR: quic: transform qc_set_timer() as a reentrant function BUG/MEDIUM: quic: prevent crash on Retry sending BUG/MINOR: mux-quic: fix crash with app ops install failure BUG/MINOR: mux-quic: properly handle STREAM frame alloc failure BUG/MINOR: h3: fix crash on h3s alloc failure BUG/MINOR: quic: prevent crash on qc_new_conn() failure BUG/MINOR: quic: consume Rx datagram even on error MEDIUM: quic: use a global CID trees list MINOR: quic: remove TID encoding in CID MEDIUM: quic: handle conn bootstrap/handshake on a random thread MINOR: quic: do not proceed to accept for closing conn MINOR: protocol: define new callback set_affinity MINOR: quic: delay post handshake frames after accept MINOR: fd: implement fd_migrate_on() to migrate on a non-local thread MEDIUM: quic: implement thread affinity rebinding MINOR: quic: properly finalize thread rebinding MAJOR: quic: support thread balancing on accept MINOR: listener: remove unneeded local accept flag CLEANUP: quic: rename frame types with an explicit prefix CLEANUP: quic: rename frame variables BUG/MEDIUM: mux-quic: do not emit RESET_STREAM for unknown length BUG/MEDIUM: mux-quic: improve streams fairness to prevent early timeout BUG/MINOR: quic: prevent buggy memcpy for empty STREAM MINOR: mux-quic: do not set buffer for empty STREAM frame MINOR: mux-quic: do not allocate Tx buf for empty STREAM frame MINOR: quic: finalize affinity change as soon as possible BUG/MINOR: quic: fix race on quic_conns list during affinity rebind Aurelien DARRAGON (23): MINOR: proxy/pool: prevent unnecessary calls to pool_gc() BUG/MEDIUM: proxy/sktable: prevent watchdog trigger on soft-stop BUG/MINOR: backend: make be_usable_srv() consistent when stopping MINOR: server: add SRV_F_DELETED flag BUG/MINOR: server/del: fix srv->next pointer consistency BUG/MINOR: stats: properly handle server stats dumping resumption BUG/MINOR: sink: free forward_px on deinit() BUG/MINOR: log: free log forward proxies on deinit() BUG/MINOR: hlua: hook yield does not behave as expected BUG/MINOR: hlua: enforce proper running context for register_x functions CLEANUP: hlua: fix conflicting comment in hlua_ctx_destroy() MINOR: hlua: add simple hlua reference handling API BUG/MINOR: hlua: fix reference leak in core.register_task() BUG/MINOR: hlua: fix reference leak in hlua_post_init_state() BUG/MINOR: hlua: prevent function and table reference leaks on errors MINOR: hlua: simplify lua locking BUG/MEDIUM: hlua: prevent deadlocks with main lua lock BUG/MINOR: errors: invalid use of memprintf in startup_logs_init() BUG/MINOR: server: incorrect report for tracking servers leaving drain MINOR: server: explicitly commit state change in srv_update_status() BUG/MINOR: server: don't miss proxy stats update on server state transitions BUG/MINOR: server: don't miss server stats update on server state transitions BUG/MINOR: server: don't use date when restoring last_change from state file Christopher Faulet (14): Revert "BUG/MEDIUM: stconn: Don't rearm the read expiration date if EOI was reached" BUG/MEDIUM: mux-h2: Be able to detect connection error during handshake BUG/MEDIUM: channel: Improve reports for shut in co_getblk() BUG/MEDIUM: dns: Properly handle error when a response consumed MINOR: http-ana: Add a HTTP_MSGF flag to state the Expect header was checked BUG/MINOR: http-ana: Don't switch message to DATA when waiting for payload BUG/MEDIUM: dns: Kill idle DNS sessions during stopping stage BUG/MINOR: resolvers: Wakeup DNS idle task on stopping BUG/MEDIUM: resolvers: Force the connect timeout for DNS resolutions BUG/MINOR: stream: Fix test on SE_FL_ERROR on the wrong entity REGTESTS: fix the race conditions in log_uri.vtc BUG/MEDIUM: log: Properly handle client aborts in syslog applet CLEANUP: backend: Remove useless debug message in assign_server() BUG/MEDIUM: Update read expiration date on synchronous send David Carlier (1): BUILD: da: extends CFLAGS to support API v3 from 3.1.7 and onwards. Frédéric Lécaille (56): BUG/MINOR: quic: Missing padding in very short probe packets BUG/MINOR: quic: Wrong use of now_ms timestamps (cubic algo) MINOR: quic: Add recovery related information to "show quic" BUG/MINOR: quic: Wrong use of now_ms timestamps (newreno algo) BUG/MINOR: quic: Missing max_idle_timeout initialization for the connection MINOR: quic: Implement cubic state trace callback MINOR: quic: Adjustments for generic control congestion traces MINOR: quic: Traces adjustments at proto level. MEDIUM: quic: Ack delay implementation BUG/MINOR: quic: Wrong rtt variance computing BUG/MINOR: quic: Remaining useless statements in cubic slow start callback BUG/MINOR: quic: Cubic congestion control window may wrap MINOR: quic: Add missing traces in cubic algorithm implementation BUG/MAJOR: quic: Congestion algorithms states shared between the connection BUG/MINOR: quic: Remove useless BUG_ON() in newreno and cubic algo implementation MINOR: quic: Add trace to debug idle timer task issues BUG/MINOR: quic: Unexpected connection closures upon idle timer task execution BUG/MINOR: quic: Wrong idle timer expiration (during 20s) BUILD: quic: 32bits compilation issue in cli_io_handler_dump_quic() BUG/MINOR: quic: Possible wrong PTO computing BUG/MINOR: quic: Possible crashes in qc_idle_timer_task() MINOR: quic: Trace fix in quic_pto_pktns() (handshaske status) BUG/MINOR: quic: Wrong packet number space probing before confirmed handshake MINOR: quic: Modify qc_try_rm_hp() traces MINOR: quic: Dump more information at proto level when building packets MINOR: quic: Add a trace for packet with an ACK frame MINOR: quic: Add packet loss and maximum cc window to "show quic" BUG/MINOR: quic: Ignored less than 1ms RTTs MINOR: quic: Add connection flags to traces BUG/MEDIUM: quic: Code sanitization about acknowledgements requirements BUG/MINOR: quic: Possible wrapped values used as ACK tree purging limit. BUG/MINOR: quic: SIGFPE in quic_cubic_update() MINOR: quic: Display the packet number space flags in traces MINOR: quic: Remove a useless test about probing in qc_prep_pkts() BUG/MINOR: quic: Wrong Application encryption level selection when probing BUG/MINOR: quic: Do not use ack delay during the handshakes BUG/MINOR: quic: Stop removing ACK ranges when building packets MINOR: quic: Do not allocate too much ack ranges BUG/MINOR: quic: Unchecked buffer length when building the token BUG/MINOR: quic: Wrong Retry token generation timestamp computing MINOR: quic: Move traces at proto level BUG/MINOR: quic: Possible memory leak from TX packets BUG/MINOR: quic: Possible leak during probing retransmissions BUG/MINOR: quic: Useless probing retransmission in draining or killing state BUG/MINOR: quic: Useless I/O handler task wakeups (draining, killing state) CLEANUP: quic: Remove useless parameters passes to qc_purge_tx_buf() CLEANUP: quic: Rename <buf> variable to <token> in quic_generate_retry_token() CLEANUP: quic: Rename <buf> variable into quic_padding_check() CLEANUP: quic: Rename <buf> variable into quic_rx_pkt_parse() CLEANUP: quic: Rename <buf> variable for several low level functions CLEANUP: quic: Make qc_build_pkt() be more readable CLEANUP: quic: Rename quic_get_dgram_dcid() <buf> variable CLEANUP: quic: Rename several <buf> variables at low level CLEANUP: quic: Rename <buf> variable into quic_packet_read_long_header() CLEANUP: quic: Rename <buf> variable into qc_parse_hd_form() CLEANUP: quic: Rename several <buf> variables into quic_sock.c Ilya Shipitsin (2): CI: bump "actions/checkout" to v3 for cross zoo matrix CI: cirrus-ci: bump FreeBSD image to 13-1 Marcos de Oliveira (1): DOC/MINOR: reformat configuration.txt's "quoting and escaping" table Miroslav Zagorac (1): BUG/MINOR: illegal use of the malloc_trim() function if jemalloc is used Olivier Houchard (1): BUG/MEDIUM: fd: don't wait for tmask to stabilize if we're not in it. Remi Tricot-Le Breton (1): BUG/MINOR: ssl: ssl-(min|max)-ver parameter not duplicated for bundles in crt-list William Lallemand (4): DOC: config: strict-sni allows to start without certificate BUG/MINOR: mworker: unset more internal variables from program section BUG/MINOR: stick_table: alert when type len has incorrect characters MINOR: ssl: remove OpenSSL 1.0.2 mention into certificate loading error Willy Tarreau (20): MINOR: http-act: emit a warning when a header field name contains forbidden chars BUILD: compiler: fix __equals_1() on older compilers MINOR: activity: add a line reporting the average CPU usage to "show activity" BUG/MINOR: cfgparse: make sure to include openssl-compat MINOR: fd: optimize fd_claim_tgid() for use in fd_insert() MINOR: fd: add a lock bit with the tgid BUG/MINOR: cli: clarify error message about stats bind-process BUG/MINOR: config: fix NUMA topology detection on FreeBSD BUILD: sock_inet: forward-declare struct receiver BUILD: proto_tcp: export the correct names for proto_tcpv[46] MINOR: pools: make sure 'no-memory-trimming' is always used MINOR: pools: intercept malloc_trim() instead of trying to plug holes MEDIUM: pools: move the compat code from trim_all_pools() to malloc_trim() MINOR: pools: export trim_all_pools() MINOR: pattern: use trim_all_pools() instead of a conditional malloc_trim() MINOR: tools: relax dlopen() on malloc/free checks MEDIUM: tools: further relax dlopen() checks too consider grouped symbols BUG/MINOR: pools: restore detection of built-in allocator MINOR: pools: report a replaced memory allocator instead of just malloc_trim() BUG/MINOR: tools: check libssl and libcrypto separately -- Christopher Faulet