Hi, HAProxy 2.3.0 was released on 2020/11/05. It added 33 new commits after version 2.3-dev9. I was right to wait a few more days before releasing, we could spot two late regressions and fix them in time!
This time we're on schedule and that's great. It's also one of the benefits of working more on features than for a target version. We could start with some accumulated stuff pending in -next (which we'll do again by the way) and focus more on features at the beginning of the cycle and more on fixes and cleanups at the end. Most of the changes in this version are not user-visible, as frustrating as this can be, as they're more low-level stuff to prepare the ground to accept new features for 2.4. With this said, I can still cite a few nice new features which were already mentioned as they were merged but I know not everyone reads development announces: - syslog forwarding / load balancing: we can now receive UDP/TCP syslog and forward the messages to any set of servers over UDP/TCP/unix etc, and even transcode the RFC3164/5424 formats as needed. A typical use case is to receive log from your infrastructure and load-balance them to a two-servers farm while keeping a local copy to a 3rd server. It makes use of the existing logging infrastructure and as such, benefits from its facilities. Check "log-forward" in the doc for more info. - the stats engine was improved to allow "modules" to register and provide their own counters. Actually these modules are all of the non-core stuff that was hard to integrate with the stats. Muxes are such an example, SSL is another one, more will come soon (actually I'm seeing that the SSL part was not merged in time for 2.3 but we may backport it later if needed). The problem was to add optional counters without breaking existing tools, and for this the stats output is cut in two parts, the fixed metrics first, a column named "-" and dynamic metrics which may change with your version, build options, etc. On the HTML stats page, these are optionally reported when "stats show-modules" is specified, in which case a new column is added with links to the relevant modules and their respective stats. The output isn't as pretty there since these stats by definition are of any type. But they're properly typed and available to all outputs, including Prometheus. - the second improvement in the stats is that they now support "domains", which allow modules to register stats for everything not a proxy or a server. The DNS stats that were available in "show resolvers" are now available in a generic way under "show stat domain dns". We can expect that peers, SPOE etc which do not directly interact with proxies will appear there in the near future. - another improvement to the stats is that it's now possible not to list servers that are in maintenance, typically because they were reserved using a server-template. Users with huge configs had to consume gigabytes of data because of this. This change was really trivial, and if this is something you're suffering from in an LTS version, let me know, maybe I'll accept to backport it to 2.2. Maybe. - the cache is now able to respond "304 not modified" to conditional requests instead of returning the full object. This will not change the cache hit ratio but could slightly lower the amount of data sent over the wire when the client already has an up-to-date content. - in "http-reuse safe" mode (the default one), we don't merge multiple clients' requests anymore. The reason behind this is to avoid the head-of-line blocking that results from merging multiple client connections into a same server connection when one of these clients is slow. This may result in slightly higher H2 connection counts on your servers if using H2 on the backend at high request rates, but in lower and stable response times for your users. - backend connections using a constant "sni()" expression will now support being reused. In the past they were closed, but some users have fixed strings there and there was no reason for not supporting them in an optimal way. - an option was added to decide how to match SSL file name extensions (either appending ".key" or replacing ".crt" with ".key"), because it used to be unconventional for some users to have "foo.crt.key" next to "foo.crt". - some minor adjustments were made to the non-deterministic LB algos to improve the resistance and ability to gracefully recover from bad situations (e.g. huge queues after a temporary network outage). They will avoid searching for a server if it's known that all are full. In addition, leastconn will now consider the server's queue length in addition to the connection count and will accept to append directly into the server's queue if that's considered better than any other server. This helps flush spikes better. - speaking of LB algorithms, "balance URI" got a new "path-only" option to only use the path and not the full URI so that origin/absolute URIs as found in HTTP/1 and HTTP/2 requests respectively hash similarly. This will improve the cache hit ratio for those using it to load balance cache farms. - there is a new "iif" converter that I think will help us simplify our configurations. It's a ternary operator, it returns arg1 or arg2 depending on the input. This is convenient when deciding to report a protocol name versus another one, or "miss" versus "hit", or be combined after an strcmp() converter, etc. - we're progressively becoming less sensitive to version abuses by OpenSSL derivatives that claim to be similar to OpenSSL version X but do not fully implement its API. The work has begun to rely on more reliable patterns whenever possible to detect support of various features. - some long-announced option removals were finally done, particularly "option http-tunnel", "monitor-net" and "mode health", which were only working in some rare (and irrelevant) situations nowadays and forcing some architectural issues that prevented the code from evoling. - the "nbproc" directive was marked as deprecated and will be killed in 2.5. We've long past the point of its unsuitability to plenty of use cases, and now we're at a point where just keeping its support is regularly source of vicious bugs (like listeners not always being in the expected state), and will not be usable at all with any UDP-based protocol such QUIC. It will emit a warning inviting you to try without it or to use "nbthread 1" to shut the warning. If you're having trouble getting rid of it, I'm interested in knowing why. - the strict-limits are now on by default, so that if you start with a bogus configuration that it known to break under load (due to missing FDs to satisfy your maxconn for example), haproxy will now refuse to start so that you don't discover the hard way after the incident that it already warned you. Of course this can be disabled and developers or support teams will certainly continue to do so :-) - those running with large numbers of threads should observe a slightly lower CPU usage, as we managed to further reduce contention and locking cost in several hot code paths. Some important changes were brought to the listeners code in preparation of the extension to modern protocols such as QUIC, and resulted in a few inevitable user-visible changes. One of them (which could be considered as an improvement) is that you won't see "proxy blah started" anymore in your boot logs, because it's not the proxies that are started but the listeners. Another one, less visible, is that if you completely mess up with a failed reload while having a conflicting port still listening in another daemon, sometimes you could end up with a listener that would stay in pause, with the proxy in error state and the impossibility to try to rebind that listener without restarting the process. Now since the listeners are totally autonomous, a reload cycle again (or just a SIGTTIN) are enough to retry the binding and recover the listening port. I don't know why I'm explaining this, I'm pretty sure nobody does that. Or at least I hope... :-) Now for the next version, there's already quite a bunch of stuff queued up into -next, and other stuff that I refused a few days ago that is going to arrive soon. By the way on this last point, when I announce end of merging of features, it's not to annoy people but because we need to stabilize everything to test and fix issues, and we owe the testers some guarantees that what they're testing doesn't change each time they update. Plus when developers are busy fixing bugs they're not available for reviewing. I've seen that a few trees continue to fill up with patches so I can expect some features to land into 2.4 early in the cycle. For this version I'd like to further shorten the merge window. I won't anticipate a strict end date too early, but I would like that we don't perform any sensitive change past February, and that we don't merge any feature at all past end of March. That will leave us roughly two months to debug and document before a release end of May, which is not too much considering that it will be an LTS one. Depending how things go, we may even advance these dates and close the doors earlier if we consider there is enough stuff to keep everyone busy. So if you expect to get your changes in, don't wait to raise your hand and show them! I've been notified that Daniel has just published a detailed changelog of 2.3 with config examples below, it contains more info and details than what I wrote above so it's worth having a look at: https://www.haproxy.com/blog/announcing-haproxy-2-3/ As a reminder, please be kind with your favorite OS package maintainers, it always takes time to prepare a new release, so there's no need to repeatedly ask them when their packages will be available, they will be available when they're ready, as usual. Let's just give them a few days to catch up. Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse : http://discourse.haproxy.org/ Slack channel : https://slack.haproxy.org/ Issue tracker : https://github.com/haproxy/haproxy/issues Wiki : https://github.com/haproxy/wiki/wiki Sources : http://www.haproxy.org/download/2.3/src/ Git repository : http://git.haproxy.org/git/haproxy-2.3.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-2.3.git Changelog : http://www.haproxy.org/download/2.3/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ This time I think I got all the upload right, otherwise you know whom to blame :-) Have fun, Willy --- Complete changelog since 2.3-dev9: Amaury Denoyelle (3): MINOR: mux_h2: capitalize frame type in stats MINOR: mux_h2: add stat for total count of connections/streams MINOR: stats: do not display empty stat module title on html Christopher Faulet (8): BUG/MEDIUM: filters: Don't try to init filters for disabled proxies BUG/MINOR: proxy/server: Skip per-proxy/server post-check for disabled proxies BUG/MINOR: checks: Report a socket error before any connection attempt BUG/MINOR: server: Set server without addr but with dns in RMAINT on startup MINOR: server: Copy configuration file and line for server templates BUG/MEDIUM: mux-pt: Release the tasklet during an HTTP upgrade BUG/MINOR: filters: Skip disabled proxies during startup only CLEANUP: mux-h2: Remove the h1 parser state from the h2 stream Daniel Corbett (1): DOC: Add dns as an available domain to show stat Ilya Shipitsin (4): BUILD: ssl: use SSL_CTRL_GET_RAW_CIPHERLIST instead of OpenSSL versions BUILD: ssl: use HAVE_OPENSSL_KEYLOG instead of OpenSSL versions CI: github actions: limit OpenSSL no-deprecated builds to "default,bug,devel" reg-tests BUILD: ssl: use feature macros for detecting ec curves manipulation support William Lallemand (1): MINOR: mworker/cli: the master CLI use its own applet Willy Tarreau (16): CLEANUP: pattern: remove unused entry "tree" in pattern.val MINOR: debug: don't count free(NULL) in memstats BUG/MEDIUM: stick-table: limit the time spent purging old entries BUG/MEDIUM: listener: only enable a listening listener if needed BUG/MEDIUM: listener: never suspend inherited sockets BUG/MEDIUM: listener: make the master also keep workers' inherited FDs MINOR: fd: add fd_want_recv_safe() MEDIUM: listeners: make use of fd_want_recv_safe() to enable early receivers REGTESTS: mark abns_socket as working now MINOR: sock: add a check against cross worker<->master socket activities BUG/MEDIUM: server: make it possible to kill last idle connections MINOR: ssl: define SSL_CTX_set1_curves_list to itself on BoringSSL BUILD: makefile: usual reorder of objects for faster builds DOC: update INSTALL to mention that TCC is supported DOC: mention in INSTALL that haproxy 2.3 is a stable version MINOR: version: mention that it's stable now ---