Hi,

HAProxy 2.0-dev6 was released on 2019/06/07. It added 73 new commits
after version 2.0-dev5.

The good thing is that all the pending stuff I was aware of was finally
merged in time. The bad thing is that we had to address yet another
problem with connection setup expiration causing loops in process_stream()
and that the fix required deep changes that I didn't feel comfortable with
at this stage of the cycle. But better fix them before the release than
papering over the issue afterwards. I'm now certain that 1.9 is affected
as well, and just thinking that at some point I'll have to backport this
changes to fix 1.9 isn't the first thing that makes me want to wake up
every morning. But while I'm still not certain that I perfectly addressed
100% of the remaining issues in this category, I'm at least certain that
it's way better now than it was and it will be easier in the future to
address any possible remaining issue. So we'll observe on 2.0 and decide
to backport on 1.9 once we're more confident.

I'm quite happy with the latest merges because they will be useful over
the long term for this release, present very low risks and will really
help with the maintenance, so special thanks to those who made efforts
to send their code in time! Among the latest changes I can enumerate in
no particular order :

  - Fix of the HTX encoding of HTTP trailers : it used to be a plain
    copy of the HTTP/1 representation which is good neither for H2 nor
    for captures or later processing. Now they are encoded just like
    headers, which means that it'll be easier to perform minor updates
    on them later if needed (e.g. Patrick asked for a way to capture
    them, which I still don't know how to do but will surely be easier
    now).

  - The new stick-table data type "server_name" was added. It works just
    like server_id except that what is exchanged over the wire is the
    server's name in addition of its ID (and MagicFred managed to keep it
    backwards compatible with older versions, so no protocol upgrade is
    needed). While IDs are useful for globally maintained configurations,
    they don't work well in highly dynamic environments where servers are
    inserted and removed in any order. Now nothing really requires to use
    a specific server ID anymore, which confirms our move towards making
    the config easier to integrate with modern tools. By the way we've
    seen how we could further extend the current mechanism to stick on a
    server's address later if needed (which could be nice with DNS).

  - The server-side SOCKS4 layer was merged. The code was clean and
    represents very little risk. Theorically we ought to have postponed
    this to 2.1 since it was not initially planned but I'm not in favor
    of bureaucratic rules so I prefered to take it anyway :-)

  - Completion of the stacking of the connection layers : for a long
    time (since 1.5) we used to have two polling modes for connections,
    depending on handshakes or data. This was extremely tricky and
    regularly caused some issues (such as the recent SOCKS4 issues).
    The split of these in layers was started just after 1.9 and the
    remaining parts (replacement of flags with subscriptions) was just
    completed, making the whole stack work the same from bottom to top,
    and removing lots of exceptions everywhere in the code. This will
    help us get rid of the old model to improve long term maintenance
    and reduce the number of bugs.

  - A few sample fetch functions for SSL were added. They're supposed
    to help debugging from traffic captures. I don't know if they're
    needed in older versions (nor if they are compatible), but I'm not
    against backporting low-risk stuff that helps with debugging like
    this.

  - The server connection pool default purge delay was extended from 1
    to 5 seconds : I noticed during some testing that when you have many
    servers (hence few connections per server), starting to kill idle
    connections after just one second provided a very low reuse rate, so
    let's raise this one a little bit. This is just default tuning anyway.

The rest is just cleanups, build fixes and bug fixes.

Overall the bug fixing rate is going down significantly and their severity
as well, so I'm reasonably confident in what we have. I recently said that
I expect to release next week, but we've slipped a little bit due to the
painful issue described above. Well, we'll see how things go.

I'm aware of a few minor things that need to be addressed but that are not
even show-stoppers for the release :
  - tcp-request inspect delay is currently not respected in HTX mode,
    but Christopher knows why, this is just a silly thing that needs to
    be addressed, hopefully before the release.

  - the state-file still compares server IDs and names and tries to do its
    best. It would be nice if it now favored the name over the ID (or even
    ignored the ID). Baptiste told me he'll have a look at it.

  - DNS SRV records : if I understood well, we currently perform lookups
    for all records in the response instead of using the address provided
    with them. Baptiste has some fix for this.

  - I'm seeing an irregular H2 performance with low numbers of connections
    and H1 + reuse + pools on the server side which I'd like to figure. It
    may just be that we can't always benefit from certain optimizations,
    which I'd be fine with, but I'd at least like to be certain it doesn't
    hide another bug.

  - I'd like to add more counters on show stat, show info etc, especially
    for H2. It's not trivial since the H2 code is totally independent on
    the rest, but I'll have a look.

  - I wanted to support a very basic "ifdef" to isolate some config parts,
    which could for example never match at all in 2.0 so that we can start
    to add macros in 2.1. This would allow some configs to start to become
    more portable across versions (especially for testing). Still not sure
    if this will be done as I don't want to rush a stupid design causing
    more trouble than it solves.

  - some github issues might still be relevant, need to recheck

  - and finally some code cleanups, as usual (nothing functional, mostly
    variable/function renaming for easier long-term maintenance).

So from this point the only things I'm willing to merge are related to
the points above, to bug fixes of course, and in general anything that
improves debugging or correct usage (including warnings and doc updates).
Anything else will be for 2.1 and could go into -next (I already spotted
quite a number of interesting things to work on for 2.1 to further improve
scalability). We'll see next week if we need another -dev version or if
it is reasonable to bring a few bottles of Champagne at the office and
declare 2.0 ready.

On another point, I was told that very few proposals for talks at the
HAProxyConf were sent to date. That doesn't surprize me and I haven't
even sent mine yet either :-)  Please keep in mind that the deadline for
proposals submissions is in two weeks and that time flies. So instead of
wasting your time reading my lengthy boring e-mails, please take a few
minutes to scratch an idea of something that could be of interest to
others, such as "how we deal with logs at XXX", "how we keep our configs
in sync", "how we built our own CDN", "12 things to take care of for
scalability", "how we perform A/B testing", etc. Also if you contribute
to other OSS projects and see certain things done well there that you
think haproxy could benefit from, that could bring interesting discussions
as well! Anyway I'll ping again and more aggressively next week!

Please find the usual URLs below :
   Site index       : http://www.haproxy.org/
   Discourse        : http://discourse.haproxy.org/
   Slack channel    : https://slack.haproxy.org/
   Issue tracker    : https://github.com/haproxy/haproxy/issues
   Sources          : http://www.haproxy.org/download/2.0/src/
   Git repository   : http://git.haproxy.org/git/haproxy.git/
   Git Web browsing : http://git.haproxy.org/?p=haproxy.git
   Changelog        : http://www.haproxy.org/download/2.0/src/CHANGELOG
   Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/

Willy
---
Complete changelog :
Christopher Faulet (12):
      BUG/MINOR: channel/htx: Don't alter channel during forward for empty HTX 
message
      BUG/MINOR: contrib/prometheus-exporter: Add HTX data block in one time
      BUG/MINOR: mux-h1: errflag must be set on H1S and not H1M during output 
processing
      MEDIUM: mux-h1: refactor output processing
      MINOR: mux-h1: Add the flag HAVE_O_CONN on h1s
      MINOR: mux-h1: Add h1_eval_htx_hdrs_size() to estimate size of the HTX 
headers
      MINOR: mux-h1: Don't count the EOM in the estimated size of headers
      MEDIUM: cache/htx: Always store info about HTX blocks in the cache
      MEDIUM: htx: Add the parsing of trailers of chunked messages
      MINOR: htx: Don't use end-of-data blocks anymore
      BUG/MINOR: mux-h1: Don't send more data than expected
      BUG/MINOR: flt_trace/htx: Only apply the random forwarding on the message 
body.

Emmanuel Hocdet (1):
      CLEANUP: ssl: remove unneeded defined(OPENSSL_IS_BORINGSSL)

Frédéric Lécaille (18):
      CLEANUP: peers: Remove tabs characters.
      CLEANUP: peers: Replace hard-coded values by macros.
      BUG/MINOR: peers: Wrong stick-table update message building.
      MINOR: dict: Add dictionary new data structure.
      MINOR: peers: Add a LRU cache implementation for dictionaries.
      MINOR: stick-table: Add "server_name" new data type.
      MINOR: cfgparse: Space allocation for "server_name" stick-table data type.
      MINOR: proxy: Add a "server by name" tree to proxy.
      MINOR: server: Add a dictionary for server names.
      MINOR: stream: Stickiness server lookup by name.
      MINOR: peers: Make peers protocol support new "server_name" data type.
      MINOR: stick-table: Make the CLI stick-table handler support dictionary 
entry data type.
      REGTEST: Add a basic server by name stickiness reg test.
      MINOR: peers: Add dictionary cache information to "show peers" CLI 
command.
      MINOR: peers: Replace hard-coded for peer protocol 64-bits value encoding 
by macros.
      MINOR: peers: Replace hard-coded values for peer protocol messaging by 
macros.
      BUG/MINOR: peers: Wrong "server_name" decoding.
      BUG/MINOR: peers: Wrong server name parsing.

Ilya Shipitsin (1):
      BUILD: travis-ci improvements

Olivier Houchard (11):
      BUG/MEDIUM: servers: Don't attempt to destroy idle connections if 
disabled.
      MEDIUM: checks: Make sure we unsubscribe before calling cs_destroy().
      MEDIUM: connections: Wake the upper layer even if sending/receiving is 
disabled.
      MEDIUM: ssl: Handle subscribe by itself.
      MINOR: ssl: Make ssl_sock_handshake() static.
      MINOR: connections: Add a new xprt method, remove_xprt.
      MINOR: connections: Add a new xprt method, add_xprt().
      MEDIUM: connections: Introduce a handshake pseudo-XPRT.
      MEDIUM: connections: Remove CONN_FL_SOCK*
      BUG/MEDIUM: ssl: Don't forget to initialize ctx->send_recv and 
ctx->recv_wait.
      BUG/MEDIUM: tcp: Make sure we keep the polling consistent in 
tcp_probe_connect.

Patrick Hemmer (1):
      MINOR: SSL: add client/server random sample fetches

Willy Tarreau (29):
      BUG/MEDIUM: connection: fix multiple handshake polling issues
      MINOR: connection: also stop receiving after a SOCKS4 response
      MINOR: mux-h1: don't try to recv() before the connection is ready
      BUG/MEDIUM: mux-h1: only check input data for the current stream, not 
next one
      MEDIUM: mux-h1: don't use CS_FL_REOS anymore
      CLEANUP: connection: remove the now unused CS_FL_REOS flag
      CONTRIB: debug: add 4 missing connection/conn_stream flags
      MEDIUM: stream: make a full process_stream() loop when completing I/O on 
exit
      MINOR: server: increase the default pool-purge-delay to 5 seconds
      BUILD: tools: do not use the weak attribute for trace() on obsolete 
linkers
      BUG/MEDIUM: vars: make sure the scope is always valid when accessing vars
      BUG/MEDIUM: vars: make the tcp/http unset-var() action support conditions
      BUILD: task: fix a build warning when threads are disabled
      MINOR: server: really increase the pool-purge-delay default to 5 seconds
      BUG/MINOR: stream: don't emit a send-name-header in conn error or 
disconnect states
      MINOR: stream-int: use bit fields to match multiple stream-int states at 
once
      MEDIUM: stream-int: remove dangerous interval checks for stream-int states
      MEDIUM: stream-int: introduce a new state SI_ST_RDY
      MAJOR: stream-int: switch from SI_ST_CON to SI_ST_RDY on I/O
      MEDIUM: stream-int: make idle-conns switch to ST_RDY
      MEDIUM: stream: re-arrange the connection setup status reporting
      MINOR: stream-int: split si_update() into si_update_rx() and 
si_update_tx()
      MINOR: stream-int: make si_sync_send() from the send code of 
si_update_both()
      MEDIUM: stream: rearrange the events to remove the loop
      MEDIUM: stream: only loop on flags relevant to the analysers
      MEDIUM: stream: don't abusively loop back on changes on CF_SHUT*_NOW
      BUILD: stream-int: avoid a build warning in dev mode in si_state_bit()
      BUILD: peers: fix a build warning about an incorrect intiialization
      BUG/MINOR: time: make sure only one thread sets global_now at boot

---

Reply via email to