Hi, HAProxy 2.1-dev3 was released on 2019/10/25. It added 155 new commits after version 2.1-dev2.
It's two weeks later than initially expected due to being diverted by bugs but the main point is that we're converging towards something better :-) So now we've finally merged the tail of pending features. There are still some rough edges but these ones will be progressively addressed in the upcoming weeks. The last user-visible changes since 2.1-dev2 include : - SSL: refactoring of how certificates are loaded and indexed in memory so that they're loaded only once each even if referenced on multiple bind lines (CPU and memory savings), and ability to update them from the CLI ("set ssl cert"), as well as OCSP/issuer/SCTL etc. There are still a few limitations, I think certain corner cases are not supported (yet) but I can't tell what so I'll rather shut up. At least it's a great improvement because certs updates were one reason for some users to reload often, and these ones were experiencing long reload operations due to a massive amount of certs. - H1/H2: properly handle authority and scheme. When H2 was implemented on top of H1, H2 requests were turned to H1 requests in "origin form" (i.e. GET /path/to/file + Host header). But H2 agents are encouraged to use absolute form (GET https://authority/path/to/file) which they do. Our conversion always used the origin form, which resulted in the loss of the scheme on end-to-end transfers, and a loss of representation if using H2 to convery H1 requests. Now that HTX is the only internal representation, it was possible to maintain the request in its original form (typically absolute for H2 and origin for H1) and preserve all elements end-to-end. One visible effect though is that logs will now show "GET https://authority/path" instead of "GET /path" since the URI really is this. Some will find this better, others may be annoyed but it's still possible to change the format if desired. What matters is that we do not denaturate requests anymore. - the cache can now cache requests for absolute URIs as well, as a byproduct of having to support these for H2. - HTX: we now maintain the authority and the host synchronized when using set-uri or when touching the Host header. In addition, requests with conflicting Host/authority are now rejected as required by the standards. - H1/FCGI: implement traces just like in H2, this can be used to provide detailed captures of issues to developers, or just for you to observe the traffic. - H2: add the ability to emit CONTINUATION frames for too large headers or trailers to fit into a single frame. This was needed in environments where more than 16kB of headers need to be sent to a client. So now our support for CONTINUATION is complete, we can both receive and send large header blocks. Note that this part is easy to backport and might at some point be backported into 2.0 if there is demand for it. - HTTP: http-send-name-header would previously not remove any existing occurrence of the header in HTX mode, this is now done so that it behaves exactly like in old legacy mode. - H1: smarter handler of internally generated responses (mostly errorfiles) which now support keep-alive when the messages are properly formatted. - stats: the new output modifier "desc" to "show info" and "show stat" will provide a short description of the meaning of each metric. This is an attempt at saving a few monthly hours of sleep to a number of admins :-) - build: threads and CPU affinity are now enabled on OSX. Performance improvements: - the scheduler now uses a combination of a locked and a lockfree list to regain 5-10% performance on workloads involving high connection rates. Debugging: - the "debug dev" commands that were only available when building with -DDEBUG_DEV are now always built-in, but only shown and available when the CLI is in "expert-mode". These are sometimes needed by developers to extract some extra information about a sick session, or to perform fault injection. Do not try to use them in production without being invited to do so, you'll very likely crash your process before you understand what you did. - more prominent version strings: among the difficulties faced when analyzing a core for a very strange issue, there is the permanent doubt about whether or not the core file was really issued from the reported version. The version string used to be built as a constant and as such did not appear in core files. Now it's copied into a variable so that it is as simple as running "strings core | fgrep -A2 'HAProxy version'" to see the exact version string. And roughly 50 bugs were addressed since -dev2, many of which were already backported into 2.0.8. We've noticed that a few issues that are still being worked on : - problems with how connection errors are reported on the backend side when several streams are multiplexed: only one of them can be retried at the moment and some issues look a bit dirty. Some of these will also affect 2.0 and 1.9 to some extents. - there's still a known minor issue by which if you trigger an error on the CLI with the new "set ssl cert" command, the lock remains held and you won't be able to update again. >From now on it's important to stick to fixes only if we want to have a chance to release something in good shape before the end of next month. Trivially valid improvements can be merged into the -next branch but submissions which require review take time and distract bug fixing, so please all be nice with developers and focus on current code's correctness only. I'd also like to kindly remind all subsystem maintainers (i.e. all those listed in the MAINTAINERS file) to devote some time to quickly verify that nothing broke in their areas before the release. Raising an issue early so that everyone is aware is often more desirable than a late report with a fix :-) I'd like to emit one version per week now till the release, though I'm not sure I'll manage to sustain the rhythm with the HAProxyConf arriving quickly (November 12th and 13th). We'll see :-) Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse : http://discourse.haproxy.org/ Slack channel : https://slack.haproxy.org/ Issue tracker : https://github.com/haproxy/haproxy/issues Sources : http://www.haproxy.org/download/2.1/src/ Git repository : http://git.haproxy.org/git/haproxy.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy.git Changelog : http://www.haproxy.org/download/2.1/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ Willy --- Complete changelog : Baptiste Assmann (1): BUG/MINOR: dns: allow srv record weight set to 0 Christopher Faulet (33): BUG/MINOR: mux-h2/trace: Fix traces on h2c initialization MINOR: h1-htx: Update h1_copy_msg_data() to ease the traces in the mux-h1 MINOR: htx: Adapt htx_dump() to be used from traces MINOR: mux-h1/trace: register a new trace source with its events MINOR: proxy: Store http-send-name-header in lower case MINOR: http: Remove headers matching the name of http-send-name-header option BUG/MINOR: mux-h1: Adjust header case when the server name is add to a request BUG/MINOR: mux-h1: Adjust header case when chunked encoding is add to a message MINOR: mux-h1: Try to wakeup the stream on output buffer allocation MINOR: fcgi: Add function to get the string representation of a record type MINOR: mux-fcgi/trace: Register a new trace source with its events BUG/MINOR: mux-h1/mux-fcgi/trace: Fix position of the 4th arg in some traces MINOR: htx: Add 2 flags on the start-line to have more info about the uri MINOR: http: Add a function to get the authority into a URI MINOR: h1-htx: Set the flag HTX_SL_F_HAS_AUTHORITY during the request parsing MEDIUM: http-htx: Keep the Host header and the request start-line synchronized MINOR: h1-htx: Only use the path of a normalized URI to format a request line BUG/MEDIUM: htx: Catch chunk_memcat() failures when HTX data are formatted to h1 BUG/MINOR: chunk: Fix tests on the chunk size in functions copying data BUG/MINOR: mux-h1: Mark the output buffer as full when the xfer is interrupted MINOR: mux-h1: Xfer as much payload data as possible during output processing CLEANUP: h1-htx: Move htx-to-h1 formatting functions from htx.c to h1_htx.c BUG/MINOR: mux-h1: Capture ignored parsing errors MINOR: h1: Reject requests with different occurrences of the header host MINOR: h1: Reject requests if the authority does not match the header host REGTESTS: Send valid URIs in peers reg-tests and fix HA config to avoid warnings REGTESTS: Adapt proxy_protocol_random_fail.vtc to match normalized URI too BUG/MINOR: http-htx: Properly set htx flags on error files to support keep-alive MINOR: htx: Add a flag on HTX to known when a response was generated by HAProxy MINOR: mux-h1: Force close mode for proxy responses with an unfinished request BUG/MINOR: tcp: Don't alter counters returned by tcp info fetchers BUG/MINOR: ssl: Fix fd leak on error path when a TLS ticket keys file is parsed BUG/MINOR: stick-table: Never exceed (MAX_SESS_STKCTR-1) when fetching a stkctr David Carlier (3): BUILD/MEDIUM: threads: rename thread_info struct to ha_thread_info BUILD/SMALL: threads: enable threads on osx BUILD/MEDIUM: threads: enable cpu_affinity on osx Emeric Brun (7): CLEANUP: ssl: make cli_parse_set_cert handle errcode and warnings. CLEANUP: ssl: make ckch_inst_new_load_(multi_)store handle errcode/warn CLEANUP: ssl: make ssl_sock_put_ckch_into_ctx handle errcode/warn CLEANUP: ssl: make ssl_sock_load_dh_params handle errcode/warn CLEANUP: bind: handle warning label on bind keywords parsing. BUG/MEDIUM: ssl: 'tune.ssl.default-dh-param' value ignored with openssl > 1.1.1 BUG/MINOR: ssl: fix memcpy overlap without consequences. Frédéric Lécaille (1): BUG/MINOR: peers: crash on reload without local peer. Miroslav Zagorac (1): BUG/MINOR: WURFL: fix send_log() function arguments Olivier Houchard (10): BUG/MEDIUM: tasks: Don't forget to decrement tasks_run_queue. MEDIUM: task: Split the tasklet list into two lists. MINOR: h2: Document traps to be avoided on multithread. MINOR: lists: Try to use local variables instead of macro arguments. MINOR: lists: Fix alignement of \ when relevant. BUG/MEDIUM: lists: Handle 1-element-lists in MT_LIST_BEHEAD(). BUG/MEDIUM: mux_pt: Make sure we don't have a conn_stream before freeing. Revert e8826ded5fea3593d89da2be5c2d81c522070995. BUG/MEDIUM: mux_pt: Don't destroy the connection if we have a stream attached. BUG/MEDIUM: mux_pt: Only call the wake emthod if nobody subscribed to receive. Rick Rackow (1): DOC: fix typo in Prometheus exporter doc Tim Duesterhus (1): BUG/MINOR: sample: Make the `field` converter compatible with `-m found` Vedran Furac (1): BUG/MINOR: server: check return value of fopen() in apply_server_state() William Dauchy (1): MINOR: tcp: avoid confusion in time parsing init William Lallemand (45): MINOR: ssl: crt-list do ckchn_lookup REORG: ssl: rename ckch_node to ckch_store REORG: ssl: move structures to ssl_sock.h MINOR: ssl: initialize the sni_keytypes_map as EB_ROOT MINOR: ssl: initialize explicitly the sni_ctx trees BUG/MINOR: ssl: abort on sni allocation failure BUG/MINOR: ssl: free the sni_keytype nodes BUG/MINOR: ssl: abort on sni_keytypes allocation failure MEDIUM: ssl: introduce the ckch instance structure MEDIUM: ssl: split ssl_sock_add_cert_sni() MINOR: ssl: ssl_sock_load_ckchn() can properly fail MINOR: ssl: ssl_sock_load_multi_ckchs() can properly fail MEDIUM: ssl: ssl_sock_load_ckchs() alloc a ckch_inst MINOR: ssl: ssl_sock_load_crt_file_into_ckch() is filling from a BIO MEDIUM: ssl/cli: 'set ssl cert' updates a certificate from the CLI MINOR: ssl: load the sctl in/from the ckch MINOR: ssl: load the ocsp in/from the ckch BUG/MEDIUM: ssl: NULL dereference in ssl_sock_load_cert_sni() BUG/MINOR: ssl: fix build without SSL BUG/MINOR: ssl: fix build without multi-cert bundles BUILD: ssl: wrong #ifdef for SSL engines code BUG/MINOR: ssl: fix OCSP build with BoringSSL BUG/MINOR: ssl: fix error messages for OCSP loading BUG/MINOR: ssl: can't load ocsp files BUG/MINOR: mworker/ssl: close openssl FDs unconditionally REGTEST: mcli/mcli_show_info: launch a 'show info' on the master CLI BUG/MINOR: mworker/cli: reload fail with inherited FD BUG/MINOR: cache: alloc shctx after check config CLEANUP: ssl: remove old TODO commentary CLEANUP: ssl: fix SNI/CKCH lock labels MINOR: ssl: OCSP functions can load from file or buffer MINOR: ssl: load sctl from buf OR from a file MINOR: ssl: load issuer from file or from buffer MINOR: ssl: split ssl_sock_load_crt_file_into_ckch() BUG/MINOR: ssl/cli: fix looking up for a bundle MINOR: ssl/cli: update ocsp/issuer/sctl file from the CLI MINOR: ssl: update ssl_sock_free_cert_key_and_chain_contents MINOR: ssl: copy a ckch from src to dst MINOR: ssl: new functions duplicate and free a ckch_store MINOR: ssl/cli: assignate a new ckch_store MEDIUM: cli/ssl: handle the creation of SSL_CTX in an IO handler BUG/MINOR: ssl/cli: fix build of SCTL and OCSP BUG/MINOR: ssl/cli: out of bounds when built without ocsp/sctl BUG/MINOR: ssl: fix build with openssl < 1.1.0 BUG/MINOR: ssl: fix build of X509_chain_up_ref() w/ libreSSL Willy Tarreau (50): MINOR: mux-h2/trace: missing conn pointer in demux full message MINOR: mux-h2: add a per-connection list of blocked streams BUILD: ebtree: make eb_is_empty() and eb_is_dup() take a const BUG/MEDIUM: mux-h2: do not enforce timeout on long connections BUG/MEDIUM: cache: make sure not to cache requests with absolute-uri DOC: clarify some points around http-send-name-header's behavior MEDIUM: mux-h2: support emitting CONTINUATION frames after HEADERS MINOR: h2: clarify the rules for how to convert an H2 request to HTX MEDIUM: h2: make the request parser rebuild a complete URI MINOR: h2: report in the HTX flags when the request has an authority MEDIUM: mux-h2: do not map Host to :authority on output MEDIUM: h2: use the normalized URI encoding for absolute form requests MINOR: stats: mention in the help message support for "json" and "typed" MINOR: stats: get rid of the ST_CONVDONE flag MINOR: stats: replace the ST_* uri_auth flags with STAT_* MINOR: stats: always merge the uri_auth flags into the appctx flags MINOR: stats: set the appctx flags when initializing the applet only MINOR: stats: get rid of the STAT_SHOWADMIN flag MINOR: stats: make stats_dump_fields_json() directly take flags MINOR: stats: uniformize the calling convention of the dump functions MINOR: stats: support the "desc" output format modifier for info and stat MINOR: stats: prepare to add a description with each stat/info field MINOR: stats: make "show stat" and "show info" MINOR: stats: fill all the descriptions for "show info" and "show stat" BUG/MEDIUM: applet: always check a fast running applet's activity before killing BUILD: stats: fix missing '=' sign in array declaration MINOR: lists: add new macro LIST_SPLICE_END_DETACHED MINOR: list: add new macro MT_LIST_BEHEAD MINOR: mux-h2: also support emitting CONTINUATION on trailers MINOR: version: make the version strings variables, not constants BUILD: travis-ci: limit build to branches "master" and "next" MINOR: istbuf: add b_fromist() to make a buffer from an ist BUG/MINOR: cache: also cache absolute URIs BUG/MEDIUM: tasklet: properly compute the sleeping threads mask in tasklet_wakeup() BUG/MAJOR: idle conns: schedule the cleanup task on the correct threads BUG/MEDIUM: task: make tasklets either local or shared but not both at once CLEANUP: ssl: make ssl_sock_load_cert*() return real error codes CLEANUP: ssl: make ssl_sock_load_ckchs() return a set of ERR_* REGTESTS: make seamless-reload depend on 1.9 and above REGTESTS: server/cli_set_fqdn requires version 1.8 minimum BUG/MINOR: stick-table: fix an incorrect 32 to 64 bit key conversion BUG/MEDIUM: pattern: make the pattern LRU cache thread-local and lockless BUG/MINOR: mux-h2: do not emit logs on backend connections MINOR: debug: add a new "debug dev stream" command MINOR: cli/debug: validate addresses using may_access() in "debug dev stream" REORG: move CLI access level definitions to cli.h MINOR: cli: add an expert mode to hide dangerous commands MINOR: debug: make most debug CLI commands accessible in expert mode MINOR: stats/debug: maintain a counter of debug commands issued BUG/MEDIUM: debug: address a possible null pointer dereference in "debug dev stream" ---