Hi,
HAProxy 2.0-dev6 was released on 2019/06/07. It added 73 new commits
after version 2.0-dev5.
The good thing is that all the pending stuff I was aware of was finally
merged in time. The bad thing is that we had to address yet another
problem with connection setup expiration causing loops in process_stream()
and that the fix required deep changes that I didn't feel comfortable with
at this stage of the cycle. But better fix them before the release than
papering over the issue afterwards. I'm now certain that 1.9 is affected
as well, and just thinking that at some point I'll have to backport this
changes to fix 1.9 isn't the first thing that makes me want to wake up
every morning. But while I'm still not certain that I perfectly addressed
100% of the remaining issues in this category, I'm at least certain that
it's way better now than it was and it will be easier in the future to
address any possible remaining issue. So we'll observe on 2.0 and decide
to backport on 1.9 once we're more confident.
I'm quite happy with the latest merges because they will be useful over
the long term for this release, present very low risks and will really
help with the maintenance, so special thanks to those who made efforts
to send their code in time! Among the latest changes I can enumerate in
no particular order :
- Fix of the HTX encoding of HTTP trailers : it used to be a plain
copy of the HTTP/1 representation which is good neither for H2 nor
for captures or later processing. Now they are encoded just like
headers, which means that it'll be easier to perform minor updates
on them later if needed (e.g. Patrick asked for a way to capture
them, which I still don't know how to do but will surely be easier
now).
- The new stick-table data type "server_name" was added. It works just
like server_id except that what is exchanged over the wire is the
server's name in addition of its ID (and MagicFred managed to keep it
backwards compatible with older versions, so no protocol upgrade is
needed). While IDs are useful for globally maintained configurations,
they don't work well in highly dynamic environments where servers are
inserted and removed in any order. Now nothing really requires to use
a specific server ID anymore, which confirms our move towards making
the config easier to integrate with modern tools. By the way we've
seen how we could further extend the current mechanism to stick on a
server's address later if needed (which could be nice with DNS).
- The server-side SOCKS4 layer was merged. The code was clean and
represents very little risk. Theorically we ought to have postponed
this to 2.1 since it was not initially planned but I'm not in favor
of bureaucratic rules so I prefered to take it anyway :-)
- Completion of the stacking of the connection layers : for a long
time (since 1.5) we used to have two polling modes for connections,
depending on handshakes or data. This was extremely tricky and
regularly caused some issues (such as the recent SOCKS4 issues).
The split of these in layers was started just after 1.9 and the
remaining parts (replacement of flags with subscriptions) was just
completed, making the whole stack work the same from bottom to top,
and removing lots of exceptions everywhere in the code. This will
help us get rid of the old model to improve long term maintenance
and reduce the number of bugs.
- A few sample fetch functions for SSL were added. They're supposed
to help debugging from traffic captures. I don't know if they're
needed in older versions (nor if they are compatible), but I'm not
against backporting low-risk stuff that helps with debugging like
this.
- The server connection pool default purge delay was extended from 1
to 5 seconds : I noticed during some testing that when you have many
servers (hence few connections per server), starting to kill idle
connections after just one second provided a very low reuse rate, so
let's raise this one a little bit. This is just default tuning anyway.
The rest is just cleanups, build fixes and bug fixes.
Overall the bug fixing rate is going down significantly and their severity
as well, so I'm reasonably confident in what we have. I recently said that
I expect to release next week, but we've slipped a little bit due to the
painful issue described above. Well, we'll see how things go.
I'm aware of a few minor things that need to be addressed but that are not
even show-stoppers for the release :
- tcp-request inspect delay is currently not respected in HTX mode,
but Christopher knows why, this is just a silly thing that needs to
be addressed, hopefully before the release.
- the state-file still compares server IDs and names and tries to do its
best. It would be nice if it now favored the name over the ID (or even
ignored the ID). Baptiste told me he'll have a look at it.
- DNS SRV records : if I understood well, we currently perform lookups
for all records in the response instead of using the address provided
with them. Baptiste has some fix for this.
- I'm seeing an irregular H2 performance with low numbers of connections
and H1 + reuse + pools on the server side which I'd like to figure. It
may just be that we can't always benefit from certain optimizations,
which I'd be fine with, but I'd at least like to be certain it doesn't
hide another bug.
- I'd like to add more counters on show stat, show info etc, especially
for H2. It's not trivial since the H2 code is totally independent on
the rest, but I'll have a look.
- I wanted to support a very basic "ifdef" to isolate some config parts,
which could for example never match at all in 2.0 so that we can start
to add macros in 2.1. This would allow some configs to start to become
more portable across versions (especially for testing). Still not sure
if this will be done as I don't want to rush a stupid design causing
more trouble than it solves.
- some github issues might still be relevant, need to recheck
- and finally some code cleanups, as usual (nothing functional, mostly
variable/function renaming for easier long-term maintenance).
So from this point the only things I'm willing to merge are related to
the points above, to bug fixes of course, and in general anything that
improves debugging or correct usage (including warnings and doc updates).
Anything else will be for 2.1 and could go into -next (I already spotted
quite a number of interesting things to work on for 2.1 to further improve
scalability). We'll see next week if we need another -dev version or if
it is reasonable to bring a few bottles of Champagne at the office and
declare 2.0 ready.
On another point, I was told that very few proposals for talks at the
HAProxyConf were sent to date. That doesn't surprize me and I haven't
even sent mine yet either :-) Please keep in mind that the deadline for
proposals submissions is in two weeks and that time flies. So instead of
wasting your time reading my lengthy boring e-mails, please take a few
minutes to scratch an idea of something that could be of interest to
others, such as "how we deal with logs at XXX", "how we keep our configs
in sync", "how we built our own CDN", "12 things to take care of for
scalability", "how we perform A/B testing", etc. Also if you contribute
to other OSS projects and see certain things done well there that you
think haproxy could benefit from, that could bring interesting discussions
as well! Anyway I'll ping again and more aggressively next week!
Please find the usual URLs below :
Site index : http://www.haproxy.org/
Discourse : http://discourse.haproxy.org/
Slack channel : https://slack.haproxy.org/
Issue tracker : https://github.com/haproxy/haproxy/issues
Sources : http://www.haproxy.org/download/2.0/src/
Git repository : http://git.haproxy.org/git/haproxy.git/
Git Web browsing : http://git.haproxy.org/?p=haproxy.git
Changelog : http://www.haproxy.org/download/2.0/src/CHANGELOG
Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/
Willy
---
Complete changelog :
Christopher Faulet (12):
BUG/MINOR: channel/htx: Don't alter channel during forward for empty HTX
message
BUG/MINOR: contrib/prometheus-exporter: Add HTX data block in one time
BUG/MINOR: mux-h1: errflag must be set on H1S and not H1M during output
processing
MEDIUM: mux-h1: refactor output processing
MINOR: mux-h1: Add the flag HAVE_O_CONN on h1s
MINOR: mux-h1: Add h1_eval_htx_hdrs_size() to estimate size of the HTX
headers
MINOR: mux-h1: Don't count the EOM in the estimated size of headers
MEDIUM: cache/htx: Always store info about HTX blocks in the cache
MEDIUM: htx: Add the parsing of trailers of chunked messages
MINOR: htx: Don't use end-of-data blocks anymore
BUG/MINOR: mux-h1: Don't send more data than expected
BUG/MINOR: flt_trace/htx: Only apply the random forwarding on the message
body.
Emmanuel Hocdet (1):
CLEANUP: ssl: remove unneeded defined(OPENSSL_IS_BORINGSSL)
Frédéric Lécaille (18):
CLEANUP: peers: Remove tabs characters.
CLEANUP: peers: Replace hard-coded values by macros.
BUG/MINOR: peers: Wrong stick-table update message building.
MINOR: dict: Add dictionary new data structure.
MINOR: peers: Add a LRU cache implementation for dictionaries.
MINOR: stick-table: Add "server_name" new data type.
MINOR: cfgparse: Space allocation for "server_name" stick-table data type.
MINOR: proxy: Add a "server by name" tree to proxy.
MINOR: server: Add a dictionary for server names.
MINOR: stream: Stickiness server lookup by name.
MINOR: peers: Make peers protocol support new "server_name" data type.
MINOR: stick-table: Make the CLI stick-table handler support dictionary
entry data type.
REGTEST: Add a basic server by name stickiness reg test.
MINOR: peers: Add dictionary cache information to "show peers" CLI
command.
MINOR: peers: Replace hard-coded for peer protocol 64-bits value encoding
by macros.
MINOR: peers: Replace hard-coded values for peer protocol messaging by
macros.
BUG/MINOR: peers: Wrong "server_name" decoding.
BUG/MINOR: peers: Wrong server name parsing.
Ilya Shipitsin (1):
BUILD: travis-ci improvements
Olivier Houchard (11):
BUG/MEDIUM: servers: Don't attempt to destroy idle connections if
disabled.
MEDIUM: checks: Make sure we unsubscribe before calling cs_destroy().
MEDIUM: connections: Wake the upper layer even if sending/receiving is
disabled.
MEDIUM: ssl: Handle subscribe by itself.
MINOR: ssl: Make ssl_sock_handshake() static.
MINOR: connections: Add a new xprt method, remove_xprt.
MINOR: connections: Add a new xprt method, add_xprt().
MEDIUM: connections: Introduce a handshake pseudo-XPRT.
MEDIUM: connections: Remove CONN_FL_SOCK*
BUG/MEDIUM: ssl: Don't forget to initialize ctx->send_recv and
ctx->recv_wait.
BUG/MEDIUM: tcp: Make sure we keep the polling consistent in
tcp_probe_connect.
Patrick Hemmer (1):
MINOR: SSL: add client/server random sample fetches
Willy Tarreau (29):
BUG/MEDIUM: connection: fix multiple handshake polling issues
MINOR: connection: also stop receiving after a SOCKS4 response
MINOR: mux-h1: don't try to recv() before the connection is ready
BUG/MEDIUM: mux-h1: only check input data for the current stream, not
next one
MEDIUM: mux-h1: don't use CS_FL_REOS anymore
CLEANUP: connection: remove the now unused CS_FL_REOS flag
CONTRIB: debug: add 4 missing connection/conn_stream flags
MEDIUM: stream: make a full process_stream() loop when completing I/O on
exit
MINOR: server: increase the default pool-purge-delay to 5 seconds
BUILD: tools: do not use the weak attribute for trace() on obsolete
linkers
BUG/MEDIUM: vars: make sure the scope is always valid when accessing vars
BUG/MEDIUM: vars: make the tcp/http unset-var() action support conditions
BUILD: task: fix a build warning when threads are disabled
MINOR: server: really increase the pool-purge-delay default to 5 seconds
BUG/MINOR: stream: don't emit a send-name-header in conn error or
disconnect states
MINOR: stream-int: use bit fields to match multiple stream-int states at
once
MEDIUM: stream-int: remove dangerous interval checks for stream-int states
MEDIUM: stream-int: introduce a new state SI_ST_RDY
MAJOR: stream-int: switch from SI_ST_CON to SI_ST_RDY on I/O
MEDIUM: stream-int: make idle-conns switch to ST_RDY
MEDIUM: stream: re-arrange the connection setup status reporting
MINOR: stream-int: split si_update() into si_update_rx() and
si_update_tx()
MINOR: stream-int: make si_sync_send() from the send code of
si_update_both()
MEDIUM: stream: rearrange the events to remove the loop
MEDIUM: stream: only loop on flags relevant to the analysers
MEDIUM: stream: don't abusively loop back on changes on CF_SHUT*_NOW
BUILD: stream-int: avoid a build warning in dev mode in si_state_bit()
BUILD: peers: fix a build warning about an incorrect intiialization
BUG/MINOR: time: make sure only one thread sets global_now at boot
---
