Hi, Thanks for the release! :) I'm raising my hand for the backport of "del-header -m" if possible. It is definitely not critical but would certainly make my life easier. :)
Kind regards, sob., 21 lis 2020 o 16:56 Willy Tarreau <[email protected]> napisał(a): > Hi, > > HAProxy 2.4-dev1 was released on 2020/11/21. It added 107 new commits > after version 2.4-dev0. > > Let me say that despite having spent quite some time recently on a few > really irritating bugs, I'm quite happy to see that the new development > cycle starts to pay off. It's probably the first ever new branch that > gets 107 patches 2 weeks after having been opened, and the majority are > updates and not bug fixes. > > First, let's have a quick round on the bugs fixed since last release > (2.3.2 should be issued shortly). The latest SSL changes in 2.3 had a > little bit of crt-list breakage that was quickly addressed. A bug in > the http-after-response rules could possibly cause random crashes. An > old bug in the SPOE with a dangling pointer could cause random crashes > (Many thanks to Maciej Zdeb for working hard for two months to isolate > this one). Checks could crash if a "proto" directive was set on the > server lines. And finally the last one, a dangling session pointer in > the idle connections was sometimes used after the session had been > detached, causing random crashes. These were detected on 2.3 which > amplifies the issue. While it's possible to crash 2.2 on the same issue > by applying a minor patch, it's impossible to say if a similar code path > is used without that patch. And a few crash reports there look suspiciously > related, so given that backporting this one requires extreme care, I'd > appreciate it if the rare ones who occasionally experience a crash in 2.2 > could run an instance on 2.4-dev1 and report if they think their issue is > gone. > > Now the new stuff. First there was a nice liftup of the CI to migrate to > GitHub actions (thanks Tim and Ilya for the work). We now have a larger > test matrix which seems more reliable and is more controllable than the > one we previously used on Travis. It was also an opportunity to start to > make the SSL build process more resistant to the non-linear evolutions of > the various OpenSSL forks. > > Some changes were made to the pattern code to stop freezing the whole > process each time a del-acl or del-map action is performed. While it used > to be OK when dealing with only a few hundreds of thousands of entries, > it's not fun anymore with maps containing 20 million IP addresses where it > used to cause long pauses that sometimes managed to trigger the watchdog! > With this change, the ACL/map entries are now versionned and atomically > updated, so that it is possible to perform a delete in the background in > small batches, and even if it requires a full scan for certain types, it > will be done in small batches. > > The cache used not to fully comply with the standards, as it would cache > an object that didn't have an explicit expiration time nor validator. Rémi > fixed this. Normally nobody should notice anything because such objects are > almost non-existent nowadays. However maybe some broken applications will > not be cached anymore, but for good, in that it was not really possible to > check for validity there. > > It used to be possible to change a server's IP address at run time form the > CLI but it was not possible to enable SSL at run time because it required > to allocate an SSL context. William Dauchy worked on this so that this > limitation doesn't exist anymore. > > The memory of the old process usage during reloads should significantly > on systems supporting malloc_trim() (i.e. glibc for now). While working > on optimizing the patterns I was annoyed by the huge memory usage after > replacing a whole map and have been looking for a way to compact unused > memory. I discovered this malloc_trim() that does exactly what one would > expect, i.e. unmap all unused pages from the allocator's caches. My old > process went down form 1.7 GB to 260 MB! Those doing frequent reloads > might be interested in giving it a try. > > Amaury added some SSL stats so that it will now be possible to count > handshakes an errors on the two sides. More detailed info will likely > come over time but for me this will be related to the ability to report > better SSL logs as well. > > Christopher and Baptiste finally finished their work on the MQTT and FIX > parsers. These can be used to extract information from initial messages > and steer the traffic to one server or another (or to drop it). > > Fred added some traces to the peers so that exchanges can now be observed. > This is essentially useful for debugging, but always interesting to see > what flows between nodes during tables synchronization. This should be > improved over time. > > Maciej implemented the "-m" argument to the "del-header" action, that was > initially planned for 2.2 and that everyone forgot about. This one allows > to specify if the argument to del-header designates a full header name, > a substring, a prefix, a suffix, or even a regex. This is something a lot > of users have been missing after "rspdel" was removed. I suspect that his > work is safe for backporting, so if anyone currently uses 2.2 or 2.3 and > is using ugly tricks (or Lua) just to remove header names by prefix for > example, just raise your hand to ask for a backport, we'll see what can > be done. > > I also want to thank those who contributed new regression tests, we've got > 12 new ones after 2.3, this is very useful and significantly contributes > to the code's quality and reliability. > > Last point, for those who always want to live on the bleeding edge, the > changes in this version are still pretty minor and relatively safe, I'm > going to put it on haproxy.org, feel free to do so as well (but not on > all servers, as usual). > > Please find the usual URLs below : > Site index : http://www.haproxy.org/ > Discourse : http://discourse.haproxy.org/ > Slack channel : https://slack.haproxy.org/ > Issue tracker : https://github.com/haproxy/haproxy/issues > Wiki : https://github.com/haproxy/wiki/wiki > Sources : http://www.haproxy.org/download/2.4/src/ > Git repository : http://git.haproxy.org/git/haproxy.git/ > Git Web browsing : http://git.haproxy.org/?p=haproxy.git > Changelog : http://www.haproxy.org/download/2.4/src/CHANGELOG > Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ > > Willy > --- > Complete changelog : > Amaury Denoyelle (10): > MINOR: ssl: instantiate stats module > MINOR: ssl: count client hello for stats > MINOR: ssl: add counters for ssl sessions > BUG/MINOR: stats: free dynamically stats fields/lines on shutdown > BUG/MEDIUM: stats: prevent crash if counters not alloc with dummy one > BUG/MEDIUM: check: reuse srv proto only if using same mode > MINOR: check: report error on incompatible proto > MINOR: check: report error on incompatible connect proto > MINOR: ssl: remove client hello counters > MEDIUM: stats: add counters for failed handshake > > Baptiste Assmann (2): > MINOR: sample: Add converters to parse FIX messages > MINOR: sample: Add converts to parses MQTT messages > > Christopher Faulet (21): > MINOR: ist: Add istend() function to return a pointer to the end of > the string > REGTEST: converter: Add a regtest for fix converters > REGTEST: converter: Add a regtest for MQTT converters > MINOR: http-htx: Add understandable errors for the errorfiles parsing > DOC: config: Fix a typo on ssl_c_chain_der > BUG/MINOR: http-fetch: Fix calls w/o parentheses of the cookie > sample fetches > BUG/MINOR: http-htx: Handle warnings when parsing http-error and > http-errors > BUG/MAJOR: spoe: Be sure to remove all references on a released spoe > applet > MINOR: spoe: Don't close connection in sync mode on processing > timeout > BUG/MINOR: tcpcheck: Don't warn on unused rules if check option is > after > MINOR: init: Fix the prototype for per-thread free callbacks > MINOR: config/mux-h2: Return ERR_ flags from init_h2() instead of a > status > CLEANUP: config: Return ERR_NONE from config callbacks instead of 0 > REGTEST: make ssl_client_samples and ssl_server_samples require to > 2.2 > BUG/MEDIUM: filters: Forward all filtered data at the end of http > filtering > BUG/MINOR: http-ana: Don't wait for the body of CONNECT requests > CLEANUP: flt-trace: Remove unused random-parsing option > MINOR: flt-trace: Add an option to inhibits trace messages > MINOR: flt-trace: Use a bitfield for the trace options > REGTESTS: Add a script to test the random forwarding with several > filters > BUG/MEDIUM: http-ana: Don't eval http-after-response ruleset on > empty messages > > Eric Salama (1): > MINOR: cfgparse: tighten the scope of newnameserver variable, free > it on error. > > Frédéric Lécaille (3): > MINOR: peers: Add traces to peer_treat_updatemsg(). > BUG/MINOR: peers: Do not ignore a protocol error for dictionary > entries. > BUG/MINOR: peers: Missing TX cache entries reset. > > Ilya Shipitsin (9): > CI: travis-ci: remove amd64, osx builds > CI: travis-ci: arm64 are not allowed to fail anymore > BUILD: ssl: use SSL_MODE_ASYNC macro instead of OPENSSL_VERSION > CI: Github Actions: enable prometheus exporter > CI: Github Actions: remove LibreSSL-3.0.2 builds > CI: Github Actions: enable BoringSSL builds > CI: travis-ci: remove builds migrated to GH actions > CI: Github Action: run "apt-get update" before packages restore > BUILD: SSL: guard TLS13 ciphersuites with > HAVE_SSL_CTX_SET_CIPHERSUITES > > Jerome Magnin (1): > CLEANUP: cfgparse: remove duplicate registration for transparent > build options > > Joao Morais (1): > DOC: clarify how to create a fallback crt > > Maciej Zdeb (3): > BUG/MINOR: http-fetch: Extract cookie value even when no cookie name > BUG/MINOR: http_htx: Fix searching headers by substring > MINOR: http_act: Add -m flag for del-header name matching method > > Matthieu Guegan (1): > BUILD: makefile: enable crypt(3) for OpenBSD > > Remi Tricot-Le Breton (1): > MEDIUM: cache: Change caching conditions > > Thierry Fournier (2): > BUG/MINOR: pattern: a sample marked as const could be written > BUG/MINOR: lua: set buffer size during map lookups > > Tim Duesterhus (6): > CI: Expand use of GitHub Actions for CI > REGTESTS: Add sample_fetches/cook.vtc > CI: Stop hijacking the hosts file > CI: Make the h2spec workflow more consistent with the VTest workflow > CI: Pass the github.event_name to matrix.py > CI: Clean up Windows CI > > William Dauchy (3): > REGTESTS: converter: add url_dec test > MINOR: ssl: create common ssl_ctx init > MEDIUM: cli/ssl: configure ssl on server at runtime > > William Lallemand (10): > REGTEST: ssl: test wildcard and multi-type + exclusions > BUG/MEDIUM: ssl/crt-list: correctly insert crt-list line if crt > already loaded > REGTEST: ssl: mark reg-tests/ssl/ssl_crt-list_filters.vtc as broken > DOC: add missing 3.10 in the summary > REGTEST: server/cli_set_ssl.vtc requires OpenSSL > BUG/MINOR: ssl: segv on startup when AKID but no keyid > BUG/MEDIUM: ssl/crt-list: bundle support broken in crt-list > BUG/MEDIUM: ssl: error when no certificate are found > BUG/MINOR: ssl/crt-list: load bundle in crt-list only if activated > BUG/MEDIUM: ssl/crt-list: fix error when no file found > > Willy Tarreau (33): > MINOR: compat: automatically include malloc.h on glibc > MEDIUM: pools: call malloc_trim() from pool_gc() > MEDIUM: pattern: call malloc_trim() on pat_ref_reload() > MINOR: pattern: move the update revision to the pat_ref, not the > expression > CLEANUP: pattern: delete the back refs at once during > pat_ref_reload() > MINOR: pattern: new sflag PAT_SF_REGFREE indicates regex_free() is > needed > MINOR: pattern: make the delete and prune functions more generic > MEDIUM: pattern: link all final elements from the reference > MEDIUM: pattern: change the pat_del_* functions to delete from the > references > MINOR: pattern: remerge the list and tree deletion functions > MINOR: pattern: perform a single call to pat_delete_gen() under the > expression > CLEANUP: acl: don't reference the generic pattern deletion function > anymore > CLEANUP: pattern: remove pat_delete_fcts[] and pattern_head->delete() > MINOR: pattern: introduce pat_ref_delete_by_ptr() to delete a valid > reference > MINOR: pattern: store a generation number in the reference patterns > MEDIUM: pattern: only match patterns that match the current > generation > MINOR: pattern: add pat_ref_commit() to commit a previously inserted > element > MINOR: pattern: implement pat_ref_load() to load a pattern at a > given generation > MINOR: pattern: add pat_ref_purge_older() to purge old entries > MEDIUM: pattern: make pat_ref_prune() rely on pat_ref_purge_older() > MINOR: pattern: during reload, delete elements frem the ref, not the > expression > MINOR: pattern: prepare removal of a pattern from the list head > MEDIUM: pattern: turn the pattern chaining to single-linked list > BUG/MINOR: ssl: don't report 1024 bits DH param load error when it's > higher > MINOR: server: remove idle lock in srv_cleanup_connections > BUILD: ssl: silence build warning on uninitialised counters > BUILD: http-htx: fix build warning regarding long type in printf > BUG/MEDIUM: peers: fix decoding of multi-byte length in stick-table > messages > REGTESTS: mark the abns test as broken again > DOC: coding-style: update a few rules about pointers > CLEANUP: connection: do not use conn->owner when the session is known > BUG/MAJOR: connection: reset conn->owner when detaching from session > list > REGTESTS: mark proxy_protocol_random_fail as broken > > --- > >
