Hi, HAProxy 2.0-dev6 was released on 2019/06/07. It added 73 new commits after version 2.0-dev5.
The good thing is that all the pending stuff I was aware of was finally merged in time. The bad thing is that we had to address yet another problem with connection setup expiration causing loops in process_stream() and that the fix required deep changes that I didn't feel comfortable with at this stage of the cycle. But better fix them before the release than papering over the issue afterwards. I'm now certain that 1.9 is affected as well, and just thinking that at some point I'll have to backport this changes to fix 1.9 isn't the first thing that makes me want to wake up every morning. But while I'm still not certain that I perfectly addressed 100% of the remaining issues in this category, I'm at least certain that it's way better now than it was and it will be easier in the future to address any possible remaining issue. So we'll observe on 2.0 and decide to backport on 1.9 once we're more confident. I'm quite happy with the latest merges because they will be useful over the long term for this release, present very low risks and will really help with the maintenance, so special thanks to those who made efforts to send their code in time! Among the latest changes I can enumerate in no particular order : - Fix of the HTX encoding of HTTP trailers : it used to be a plain copy of the HTTP/1 representation which is good neither for H2 nor for captures or later processing. Now they are encoded just like headers, which means that it'll be easier to perform minor updates on them later if needed (e.g. Patrick asked for a way to capture them, which I still don't know how to do but will surely be easier now). - The new stick-table data type "server_name" was added. It works just like server_id except that what is exchanged over the wire is the server's name in addition of its ID (and MagicFred managed to keep it backwards compatible with older versions, so no protocol upgrade is needed). While IDs are useful for globally maintained configurations, they don't work well in highly dynamic environments where servers are inserted and removed in any order. Now nothing really requires to use a specific server ID anymore, which confirms our move towards making the config easier to integrate with modern tools. By the way we've seen how we could further extend the current mechanism to stick on a server's address later if needed (which could be nice with DNS). - The server-side SOCKS4 layer was merged. The code was clean and represents very little risk. Theorically we ought to have postponed this to 2.1 since it was not initially planned but I'm not in favor of bureaucratic rules so I prefered to take it anyway :-) - Completion of the stacking of the connection layers : for a long time (since 1.5) we used to have two polling modes for connections, depending on handshakes or data. This was extremely tricky and regularly caused some issues (such as the recent SOCKS4 issues). The split of these in layers was started just after 1.9 and the remaining parts (replacement of flags with subscriptions) was just completed, making the whole stack work the same from bottom to top, and removing lots of exceptions everywhere in the code. This will help us get rid of the old model to improve long term maintenance and reduce the number of bugs. - A few sample fetch functions for SSL were added. They're supposed to help debugging from traffic captures. I don't know if they're needed in older versions (nor if they are compatible), but I'm not against backporting low-risk stuff that helps with debugging like this. - The server connection pool default purge delay was extended from 1 to 5 seconds : I noticed during some testing that when you have many servers (hence few connections per server), starting to kill idle connections after just one second provided a very low reuse rate, so let's raise this one a little bit. This is just default tuning anyway. The rest is just cleanups, build fixes and bug fixes. Overall the bug fixing rate is going down significantly and their severity as well, so I'm reasonably confident in what we have. I recently said that I expect to release next week, but we've slipped a little bit due to the painful issue described above. Well, we'll see how things go. I'm aware of a few minor things that need to be addressed but that are not even show-stoppers for the release : - tcp-request inspect delay is currently not respected in HTX mode, but Christopher knows why, this is just a silly thing that needs to be addressed, hopefully before the release. - the state-file still compares server IDs and names and tries to do its best. It would be nice if it now favored the name over the ID (or even ignored the ID). Baptiste told me he'll have a look at it. - DNS SRV records : if I understood well, we currently perform lookups for all records in the response instead of using the address provided with them. Baptiste has some fix for this. - I'm seeing an irregular H2 performance with low numbers of connections and H1 + reuse + pools on the server side which I'd like to figure. It may just be that we can't always benefit from certain optimizations, which I'd be fine with, but I'd at least like to be certain it doesn't hide another bug. - I'd like to add more counters on show stat, show info etc, especially for H2. It's not trivial since the H2 code is totally independent on the rest, but I'll have a look. - I wanted to support a very basic "ifdef" to isolate some config parts, which could for example never match at all in 2.0 so that we can start to add macros in 2.1. This would allow some configs to start to become more portable across versions (especially for testing). Still not sure if this will be done as I don't want to rush a stupid design causing more trouble than it solves. - some github issues might still be relevant, need to recheck - and finally some code cleanups, as usual (nothing functional, mostly variable/function renaming for easier long-term maintenance). So from this point the only things I'm willing to merge are related to the points above, to bug fixes of course, and in general anything that improves debugging or correct usage (including warnings and doc updates). Anything else will be for 2.1 and could go into -next (I already spotted quite a number of interesting things to work on for 2.1 to further improve scalability). We'll see next week if we need another -dev version or if it is reasonable to bring a few bottles of Champagne at the office and declare 2.0 ready. On another point, I was told that very few proposals for talks at the HAProxyConf were sent to date. That doesn't surprize me and I haven't even sent mine yet either :-) Please keep in mind that the deadline for proposals submissions is in two weeks and that time flies. So instead of wasting your time reading my lengthy boring e-mails, please take a few minutes to scratch an idea of something that could be of interest to others, such as "how we deal with logs at XXX", "how we keep our configs in sync", "how we built our own CDN", "12 things to take care of for scalability", "how we perform A/B testing", etc. Also if you contribute to other OSS projects and see certain things done well there that you think haproxy could benefit from, that could bring interesting discussions as well! Anyway I'll ping again and more aggressively next week! Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse : http://discourse.haproxy.org/ Slack channel : https://slack.haproxy.org/ Issue tracker : https://github.com/haproxy/haproxy/issues Sources : http://www.haproxy.org/download/2.0/src/ Git repository : http://git.haproxy.org/git/haproxy.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy.git Changelog : http://www.haproxy.org/download/2.0/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ Willy --- Complete changelog : Christopher Faulet (12): BUG/MINOR: channel/htx: Don't alter channel during forward for empty HTX message BUG/MINOR: contrib/prometheus-exporter: Add HTX data block in one time BUG/MINOR: mux-h1: errflag must be set on H1S and not H1M during output processing MEDIUM: mux-h1: refactor output processing MINOR: mux-h1: Add the flag HAVE_O_CONN on h1s MINOR: mux-h1: Add h1_eval_htx_hdrs_size() to estimate size of the HTX headers MINOR: mux-h1: Don't count the EOM in the estimated size of headers MEDIUM: cache/htx: Always store info about HTX blocks in the cache MEDIUM: htx: Add the parsing of trailers of chunked messages MINOR: htx: Don't use end-of-data blocks anymore BUG/MINOR: mux-h1: Don't send more data than expected BUG/MINOR: flt_trace/htx: Only apply the random forwarding on the message body. Emmanuel Hocdet (1): CLEANUP: ssl: remove unneeded defined(OPENSSL_IS_BORINGSSL) Frédéric Lécaille (18): CLEANUP: peers: Remove tabs characters. CLEANUP: peers: Replace hard-coded values by macros. BUG/MINOR: peers: Wrong stick-table update message building. MINOR: dict: Add dictionary new data structure. MINOR: peers: Add a LRU cache implementation for dictionaries. MINOR: stick-table: Add "server_name" new data type. MINOR: cfgparse: Space allocation for "server_name" stick-table data type. MINOR: proxy: Add a "server by name" tree to proxy. MINOR: server: Add a dictionary for server names. MINOR: stream: Stickiness server lookup by name. MINOR: peers: Make peers protocol support new "server_name" data type. MINOR: stick-table: Make the CLI stick-table handler support dictionary entry data type. REGTEST: Add a basic server by name stickiness reg test. MINOR: peers: Add dictionary cache information to "show peers" CLI command. MINOR: peers: Replace hard-coded for peer protocol 64-bits value encoding by macros. MINOR: peers: Replace hard-coded values for peer protocol messaging by macros. BUG/MINOR: peers: Wrong "server_name" decoding. BUG/MINOR: peers: Wrong server name parsing. Ilya Shipitsin (1): BUILD: travis-ci improvements Olivier Houchard (11): BUG/MEDIUM: servers: Don't attempt to destroy idle connections if disabled. MEDIUM: checks: Make sure we unsubscribe before calling cs_destroy(). MEDIUM: connections: Wake the upper layer even if sending/receiving is disabled. MEDIUM: ssl: Handle subscribe by itself. MINOR: ssl: Make ssl_sock_handshake() static. MINOR: connections: Add a new xprt method, remove_xprt. MINOR: connections: Add a new xprt method, add_xprt(). MEDIUM: connections: Introduce a handshake pseudo-XPRT. MEDIUM: connections: Remove CONN_FL_SOCK* BUG/MEDIUM: ssl: Don't forget to initialize ctx->send_recv and ctx->recv_wait. BUG/MEDIUM: tcp: Make sure we keep the polling consistent in tcp_probe_connect. Patrick Hemmer (1): MINOR: SSL: add client/server random sample fetches Willy Tarreau (29): BUG/MEDIUM: connection: fix multiple handshake polling issues MINOR: connection: also stop receiving after a SOCKS4 response MINOR: mux-h1: don't try to recv() before the connection is ready BUG/MEDIUM: mux-h1: only check input data for the current stream, not next one MEDIUM: mux-h1: don't use CS_FL_REOS anymore CLEANUP: connection: remove the now unused CS_FL_REOS flag CONTRIB: debug: add 4 missing connection/conn_stream flags MEDIUM: stream: make a full process_stream() loop when completing I/O on exit MINOR: server: increase the default pool-purge-delay to 5 seconds BUILD: tools: do not use the weak attribute for trace() on obsolete linkers BUG/MEDIUM: vars: make sure the scope is always valid when accessing vars BUG/MEDIUM: vars: make the tcp/http unset-var() action support conditions BUILD: task: fix a build warning when threads are disabled MINOR: server: really increase the pool-purge-delay default to 5 seconds BUG/MINOR: stream: don't emit a send-name-header in conn error or disconnect states MINOR: stream-int: use bit fields to match multiple stream-int states at once MEDIUM: stream-int: remove dangerous interval checks for stream-int states MEDIUM: stream-int: introduce a new state SI_ST_RDY MAJOR: stream-int: switch from SI_ST_CON to SI_ST_RDY on I/O MEDIUM: stream-int: make idle-conns switch to ST_RDY MEDIUM: stream: re-arrange the connection setup status reporting MINOR: stream-int: split si_update() into si_update_rx() and si_update_tx() MINOR: stream-int: make si_sync_send() from the send code of si_update_both() MEDIUM: stream: rearrange the events to remove the loop MEDIUM: stream: only loop on flags relevant to the analysers MEDIUM: stream: don't abusively loop back on changes on CF_SHUT*_NOW BUILD: stream-int: avoid a build warning in dev mode in si_state_bit() BUILD: peers: fix a build warning about an incorrect intiialization BUG/MINOR: time: make sure only one thread sets global_now at boot ---