Hi, HAProxy 2.2-dev2 was released on 2020/02/07. It added 115 new commits after version 2.2-dev1.
Over the last two weeks, 36 bugs and a few build warnings were addressed. On the features front, I mainly noted these : - CLI: Adis added support for multiple filters on the "show table" output. I haven't tried it yet but it looks promising as I guess I'll be able to simplify a few anti-abuse scripts here and there that heavily rely on grep. - connection: the ongoing code cleanup work was continued. It should normally be harmless (famous last words). Once this is complete, I intend to write some doc about it so that we know this tricky area better in the future. - HTTP: Christopher implemented about all the things we've been debating over the last 5 years around the "return" directive. Now we can forge a response at any instant based on a status code, an error file, a body from an argument string, a body from a log-format argument, a body from a file, a body processed from a log-format stored in a file (effectively making it a response template), a set of headers, and I think that's about all. We may have to remind people more often that haproxy is not a file server, but at least it will be extremely convenient for some to be able to return rich reject pages or tailored sets of headers for rate shaping based on 503+retry for example. This should allow some of us to finally remove the dirty hacks consisting in using an backend with a specific 503 error file, or something similar based on a deny rule, just for the sake of delivering a robots, favicon, or any ".well-known" content. Responses remain limited to the size of a buffer, and I'm not willing to see this change for now. - HTTP: Christopher also added a new "http-after-response" ruleset to manipulate headers after the final response is sent. This is mainly used to append headers after redirects or haproxy's error responses. The main use case definitely concerns HSTS, but given that all regular actions were implemented, one could also think about using this to delete some Server headers for example. - Lua: it is now possible to directly build a response to be injected from an HTTP action by passing a reply object to txn:done(). In the past it used to be only possible from services. This means that some new HTTP actions could first be implemented in Lua for the time it takes to get a broad consensus on them, before doing them natively. - SSL: William brought significant startup time savings when using large amounts of certificates thanks to a new option indicating what extensions to look for. By default, for backwards compatibility we look for ".ocsp", ".sctl", ".issuer" and all cert types extensions. But when you know exactly what you're using and know it's pointless to check for the ones above, you can now explicitly tell haproxy not to look them up, and all these extra syscalls start to account for real when you have 100k certs. - Lua: Tim added options to prepend the lookup path for Lua modules. - a bunch of dead code cleanups and/or minor fixes by Ilya and William Dauchy (I noticed a few other ones arrived since the release). - splicing: a thread-local pool of recently used pipes was added to improve cache locality and eliminate locking on allocation, resulting in ~5-6% performance increase on spliced traffic. - scheduler: the scheduler now becomes latency aware. I was particularly irritated by seeing some pathological cases in which a "show info" on the CLI could take tens of seconds on a machine saturated under high traffic rates just because I/O tasks requeue themselves and find some new data available. Now we have 3 latency classes and tasks are placed there based on their behavior. The result is that now even on a machine saturating 16 threads at 100% forwarding 90 Gbps of traffic, the CLI responds in 70ms and not one minute anymore. And the small objects now experience a much lower latency on mixed traffic. Last, a few potentially user-visible changes: - you'll now get an error if an ACL is called "or" since you'll never be able to match it, as the "or" word is taken by the expression parser. It was backported but will only warn in stable versions. - too large error files that fill the rewrite reserved area will warn you at load time, as they are potentially incompatible with http-after-reponse rules. - the number of connections reported in the logs output of a quitting proxy now clearly indicates that the value is the cumulated conns and not the active conns. I wouldn't be surprised to learn that some people parse this output and got it wrong as I did a few times :-) This branch is starting to be interesting, I'll deploy it on haproxy.org and see if it allows me to simplify some of the configuration. Let's continue to get good stuff like this merged till end of March, and let's try to get dev3 in two weeks. Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse : http://discourse.haproxy.org/ Slack channel : https://slack.haproxy.org/ Issue tracker : https://github.com/haproxy/haproxy/issues Sources : http://www.haproxy.org/download/2.2/src/ Git repository : http://git.haproxy.org/git/haproxy.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy.git Changelog : http://www.haproxy.org/download/2.2/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ Willy --- Complete changelog : Adis Nezirovic (3): MEDIUM: cli: Allow multiple filter entries for "show table" BUG/MINOR: cli: Missing arg offset for filter data values. MINOR: cli: Report location of errors or any extra data for "show table" Christopher Faulet (30): BUG/MINOR: http-ana: Increment the backend counters on the backend BUG/MINOR: stream: Be sure to have a listener to increment its counters BUG/MINOR: http-rules: Always init log-format expr for common HTTP actions BUG/MINOR: http-act: Use the good message to test strict rewritting mode MINOR: global: Set default tune.maxrewrite value during global structure init MINOR: http-rules: Set SF_ERR_PRXCOND termination flag when a header rewrite fails MINOR: http-htx: Emit a warning if an error file runs over the buffer's reserve MINOR: htx: Add a function to append an HTX message to another one MINOR: htx/channel: Add a function to copy an HTX message in a channel's buffer BUG/MINOR: http-ana: Don't overwrite outgoing data when an error is reported MINOR: dns: Dynamically allocate dns options to reduce the act_rule size MINOR: dns: Add function to release memory allocated for a do-resolve rule BUG/MINOR: http-ana: Reset HTX first index when HAPRoxy sends a response BUG/MINOR: http-ana: Set HTX_FL_PROXY_RESP flag if a server perform a redirect MINOR: http-rules: Add a flag on redirect rules to know the rule direction MINOR: http-rules: Handle the rule direction when a redirect is evaluated MINOR: http-ana: Rely on http_reply_and_close() to handle server error MINOR: http-ana: Add a function for forward internal responses MINOR: http-ana/http-rules: Use dedicated function to forward internal responses MEDIUM: http: Add a ruleset evaluated on all responses just before forwarding MEDIUM: http-rules: Add the return action to HTTP rules MEDIUM: http-rules: Support extra headers for HTTP return actions CLEANUP: lua: Remove consistency check for sample fetches and actions BUG/MINOR: http-ana: Increment failed_resp counters on invalid response MINOR: lua: Get the action return code on the stack when an action finishes MINOR: lua: Create the global 'act' object to register all action return codes MINOR: lua: Add act:wake_time() function to set a timeout when an action yields MEDIUM: lua: Add ability for actions to intercept HTTP messages REGTESTS: Add reg tests for the HTTP return action REGTESTS: Add a reg test for http-after-response rulesets Emmanuel Hocdet (2): BUG/MINOR: ssl: ssl_sock_load_pem_into_ckch is not consistent BUG/MINOR: ssl/cli: ocsp_issuer must be set w/ "set ssl cert" Frédéric Lécaille (1): BUG/MINOR: ssl: Possible memleak when allowing the 0RTT data buffer. Ilya Shipitsin (2): BUILD: CI: temporarily mark openssl-1.0.2 as allowed failure BUILD: CI: move cygwin builds to Github Actions Jerome Magnin (1): DOC: word converter ignores delimiters at the start or end of input string Olivier Houchard (16): BUG/MEDIUM: netscaler: Don't forget to allocate storage for conn->src/dst. MEDIUM: streams: Always create a conn_stream in connect_server(). MEDIUM: connections: Get ride of the xprt_done callback. BUG/MEDIUM: connections: Set CO_FL_CONNECTED in conn_complete_session(). BUG/MEDIUM: 0rtt: Only consider the SSL handshake. BUG/MEDIUM: streams: Move the conn_stream allocation outside #IF USE_OPENSSL. MINOR: ssl: Remove dead code. BUG/MEDIUM: ssl: Don't forget to free ctx->ssl on failure. BUG/MEDIUM: stream: Don't install the mux in back_handle_st_con(). MEDIUM: streams: Don't close the connection in back_handle_st_con(). MEDIUM: streams: Don't close the connection in back_handle_st_rdy(). BUG/MEDIUM: connections: Don't forget to unlock when killing a connection. BUG/MEDIUM: memory_pool: Update the seq number in pool_flush(). MINOR: memory: Only init the pool spinlock once. BUG/MEDIUM: memory: Add a rwlock before freeing memory. BUG/MAJOR: memory: Don't forget to unlock the rwlock if the pool is empty. Tim Duesterhus (7): MINOR: lua: Add hlua_prepend_path function MINOR: lua: Add lua-prepend-path configuration option MINOR: lua: Add HLUA_PREPEND_C?PATH build option CLEANUP: peers: Remove unused static function `free_dcache` CLEANUP: peers: Remove unused static function `free_dcache_tx` MINOR: acl: Warn when an ACL is named 'or' BUG/MINOR: acl: Fix type of log message when an acl is named 'or' William Dauchy (3): BUG/MINOR: connection: fix ip6 dst_port copy in make_proxy_line_v2 BUG/MINOR: dns: allow 63 char in hostname MINOR: proxy: clarify number of connections log when stopping William Lallemand (7): BUG/MINOR: ssl/cli: free the previous ckch content once a PEM is loaded BUG/MINOR: ssl: increment issuer refcount if in chain BUG/MINOR: ssl: memory leak w/ the ocsp_issuer BUG/MINOR: ssl: typo in previous patch BUG/MINOR: ssl/cli: fix unused variable with openssl < 1.0.2 MINOR: ssl: ssl-load-extra-files configure loading of files BUG/MINOR: ssl: clear the SSL errors on DH loading failure Willy Tarreau (43): BUILD: stick-table: fix build errors introduced by last stick-table change CLEANUP: changelog: remove the duplicate entry for 2.2-dev1 CLEANUP: backend: remove useless test for inexistent connection CLEANUP: backend: shut another false null-deref in back_handle_st_con() CLEANUP: stats: shut up a wrong null-deref warning from gcc 9.2 MEDIUM: connection: remove CO_FL_CONNECTED and only rely on CO_FL_WAIT_* MINOR: stream-int: always report received shutdowns MINOR: connection: remove CO_FL_SSL_WAIT_HS from CO_FL_HANDSHAKE MEDIUM: connection: use CO_FL_WAIT_XPRT more consistently than L4/L6/HANDSHAKE MINOR: connection: remove checks for CO_FL_HANDSHAKE before I/O MINOR: connection: do not check for CO_FL_SOCK_RD_SH too early MINOR: connection: don't check for CO_FL_SOCK_WR_SH too early in handshakes MINOR: raw-sock: always check for CO_FL_SOCK_WR_SH before sending MINOR: connection: remove some unneeded checks for CO_FL_SOCK_WR_SH BUG/MINOR: stktable: report the current proxy name in error messages BUG/MEDIUM: mux-h2: make sure we don't emit TE headers with anything but "trailers" BUILD: cfgparse: silence a bogus gcc warning on 32-bit machines REGTESTS: make the set_ssl_cert test require version 2.2 BUILD: CI: disable slow regtests on Travis BUG/MINOR: tcpchecks: fix the connect() flags regarding delayed ack MEDIUM: raw-sock: remove obsolete calls to fd_{cant,cond,done}_{send,recv} MEDIUM: pipe/thread: reduce the locking overhead MEDIUM: pipe/thread: maintain a per-thread local cache of recently used pipes BUG/MEDIUM: pipe/thread: fix atomicity of pipe counters MINOR: tasks: move the list walking code to its own function MEDIUM: tasks: implement 3 different tasklet classes with their own queues MEDIUM: tasks: automatically requeue into the bulk queue an already running tasklet OPTIM: task: refine task classes default CPU bandwidth ratios MINOR: task: permanently flag tasklets waking themselves up MINOR: task: make sched->current also reflect tasklets MINOR: task: detect self-wakeups on tl==sched->current instead of TASK_RUNNING OPTIM: task: readjust CPU bandwidth distribution since last update MINOR: task: don't set TASK_RUNNING on tasklets SCRIPTS: add a new "backport" script to simplify long series of backports BUG/MINOR: ssl: we may only ignore the first 64 errors SCRIPTS: use /usr/bin/env bash instead of /bin/bash for scripts CLEANUP: hpack: remove a redundant test in the decoder CONTRIB: debug: add missing flags SF_HTX and SF_MUX CONTRIB: debug: add the possibility to decode the value as certain types only CONTRIB: debug: support reporting multiple values at once BUILD: lua: silence a warning on systems where longjmp is not marked as noreturn CONTRIB: debug: also support reading values from stdin SCRIPTS: backport: use short revs and resolve the initial commit ---