Hi,
HAProxy 2.3-dev7 was released on 2020/10/17. It added 56 new commits
after version 2.3-dev6.
I'm pleased to see that development has significantly stabilized and
that the code looks in good shape, with the issue list not dramatically
increasing (and not for 2.3-only stuff, which is great).
A few long-standing bugs have been addressed. One of them, found by
Christopher, is a missing lock when using the "leastconn" or "first" load
balancing algorithms, which at first glance could only trigger a divide by
zero, but it can do much more and was proven to be responsible for causing
spinning loops. And while reviewing the fix, I noticed that the main reason
for slow server lookups when using many servers at their maxconn setting is
that we're still using a spinlock instead of an rwlock to lock the LB
algorithms. The fix is simple, it just requires to carefully pass over all
places, so I want to address it before 2.3-final.
I also detected a snowball effect in the scheduler. Any thread may wake up
expired global tasks. The problem was that this was done by taking a write
lock for each of them until the task list is empty. In this situation, in
a highly threaded environment, nothing prevents one thread from removing
expired entries while a few other threads are having fun of it by feeding
the queue with many more tasks (e.g. checks), preventing it from ever
finishing. I'm pretty sure it was the cause of at least one crash report
in the github issues. I've changed this by improving the locking (which
BTW reduced from 2 to 1% the cost of the scheduler on 8 threads), and by
enforcing a bound on the number of tasks woken up in one round (which is
the backportable part).
There was a crash in "haproxy -c" introduced by my recent protocol layer
changes in -dev5 (fdtab was not allocated yet), that was fixed by William
Dauchy (and that I noticed after fixing it myself in parallel, sorry for
this William).
As recently discussed, some obsolete keywords were removed ("mode health",
"monitor-net"). The proposed warning on "nbproc" was not added yet.
Also there were some long-standing limitations that were addressed by
Amaury. One of them was some head-of-line blocking when using H2 to
connect to backends. This is a limitation of doing multiplexing over an
in-order protocol like TCP. The problem is when requests from multiple
frontend connections are merged into the same backend connection. If one
client reads large objects slower than others, it will sometimes block
the demultiplexer with some unparsed frames. The solution against this
is to reduce the per-stream window, but then each stream may face a very
low bandwidth if servers are far away. We opted for a different solution
here, consisting in merging only requests from the same connection in an
H2 connection by default. When I'm saying "by default", I mean that it's
what is done in "http-reuse safe", because by "safe" we mean "no bad
surprises". Doing this will slightly lower the reuse rate on H2 backends
by default but significantly reduce the jitter in response time experienced
by the users. Other levels (http-reuse aggressive & always) continue to
merge the requests from any connection.
Another limitation is that a connection made to a server over TLS with an
SNI was marked as private because we didn't want to share it with another
request if there was a risk that the expected SNI wouldn't match. However
some users rightfully pointed in the past that very often they just hard
code the SNI in the configuration (e.g. "sni str(example.com)") and in
this case there is no such risk, and it's too bad not to reuse such
connections. Amaury refined the test so that only connections using a
variable expression for the SNI will be marked private. Now it will be
possible to reuse them for configuration-based constants such as above.
This is definitely something we'll have to further improve to support
a small number of valid SNIs for a given server.
UDP receivers now support being paused during reloads, and the TCP ones
have been cleaned up to use connect(AF_UNSPEC) instead of the shutdown+
listen+shutdown dance that was used to try to detect incompatible OSes
that would silently succeed but not work as expected. An issue with the
recent changes to the listener causing the backends never to stop was
addressed, and listeners are now in a relatively clean state so that we
should not significantly break them in the near future.
Fred found a bug causing peers sessions to be reset sometimes. He thinks
there's another bug (that predates 2.3) but it was not identified yet.
Among the remaining ideas floating around for the release, we've had a
discussion yesterday on various points regarding dynamic server management
and one point concerned the verbosity of the "show server-state" and
"show stats" outputs when many server-templates are used. We thought
about adding a possibility not to dump disabled servers, and I progressively
started to see several common points with previous demands such as
hiding some information or not displaying the prompt. That made me
think that we could have a new CLI command "display-options" which
would allow us to enable/disable a number of options for the session,
like the prompt, the interactive mode, anonymous mode, display disabled
servers, and/or display elements counts (e.g if you dump a backend of
100 servers of which only 4 are active and reported, you may want to know
that there are 96 left). Given that it always takes time for tools to
adapt to new outputs, I was thinking that this is a small harmless change
that would be nice to have before 2.3 so that tools are already updated
when 2.4 LTS is out (by then we'll then extend it). If things go well
and we're not flooded by too many bugs, I'll possibly work on this to
have something minimalist at first.
I'll try to stick to one version per week till the release in order to
ease testing. And if you don't want to put it on test on your production
(which I perfectly understand despite doing it myself), you can at least
use it with "-c" on your existing configs to check that everything still
looks good with no warning!
Please find the usual URLs below :
Site index : http://www.haproxy.org/
Discourse : http://discourse.haproxy.org/
Slack channel : https://slack.haproxy.org/
Issue tracker : https://github.com/haproxy/haproxy/issues
Wiki : https://github.com/haproxy/wiki/wiki
Sources : http://www.haproxy.org/download/2.3/src/
Git repository : http://git.haproxy.org/git/haproxy.git/
Git Web browsing : http://git.haproxy.org/?p=haproxy.git
Changelog : http://www.haproxy.org/download/2.3/src/CHANGELOG
Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/
Willy
---
Complete changelog :
Amaury Denoyelle (12):
BUG/MINOR: connection: fix loop iter on connection takeover
BUG/MEDIUM: connection: fix srv idle count on conn takeover
MINOR: connection: improve list api usage
MINOR: mux/connection: add a new mux flag for HOL risk
MINOR: connection: don't check priv flag on free
MEDIUM: backend: add new conn to session if mux marked as HOL blocking
MEDIUM: backend: add reused conn to sess if mux marked as HOL blocking
MEDIUM: h2: remove conn from session on detach
MEDIUM: fcgi: remove conn from session on detach
DOC: Describe reuse safe for HOL handling
MINOR: ssl: add volatile flags to ssl samples
MEDIUM: backend: reuse connection if using a static sni
Christopher Faulet (3):
BUG/MEDIUM: spoe: Unset variable instead of set it if no data provided
BUG/MEDIUM: mux-h1: Get the session from the H1S when capturing bad
messages
BUG/MEDIUM: lb: Always lock the server when calling
server_{take,drop}_conn
Frédéric Lécaille (1):
BUG/MINOR: peers: Possible unexpected peer seesion reset after collisions.
Ilya Shipitsin (1):
CI: travis-ci: replace not defined SSL_LIB, SSL_INC for BotringSSL builds
Matteo Contrini (1):
DOC: fix typo in MAX_SESS_STKCTR
Willy Tarreau (38):
BUG/MINOR: init: only keep rlim_fd_cur if max is unlimited
BUG/MINOR: mux-h2: do not stop outgoing connections on stopping
MINOR: fd: report an error message when failing initial allocations
MINOR: proto-tcp: make use of connect(AF_UNSPEC) for the pause
MINOR: sock: add sock_accept_conn() to test a listening socket
MINOR: protocol: make proto_tcp & proto_uxst report listening sockets
MINOR: sockpair: implement the .rx_listening function
CLEANUP: tcp: make use of sock_accept_conn() where relevant
CLEANUP: unix: make use of sock_accept_conn() where relevant
BUG/MINOR: listener: detect and handle shared sockets stopped in other
processes
CONTRIB: tcploop: implement a disconnect operation 'D'
CLEANUP: protocol: intitialize all of the sockaddr when disconnecting
BUG/MEDIUM: deinit: check fdtab before fdtab[fd].owner
MEDIUM: proxy: remove obsolete "mode health"
MEDIUM: proxy: remove obsolete "monitor-net"
CLEANUP: protocol: remove the ->drain() function
CLEANUP: fd: finally get rid of fd_done_recv()
MINOR: connection: make sockaddr_alloc() take the address to be copied
MEDIUM: listener: allocate the connection before queuing a new connection
MINOR: session: simplify error path in session_accept_fd()
MINOR: connection: add new error codes for accept_conn()
MINOR: sock: rename sock_accept_conn() to sock_accepting_conn()
MINOR: protocol: add a new function accept_conn()
MINOR: sock: implement sock_accept_conn() to accept a connection
MINOR: sockpair: implement sockpair_accept_conn() to accept a connection
MEDIUM: listener: use protocol->accept_conn() to accept a connection
MEDIUM: listener: remove the second pass of fd manipulation at the end
MINOR: protocol: add a default I/O callback and put it into the receiver
MINOR: log: set the UDP receiver's I/O handler in the receiver
MINOR: protocol: register the receiver's I/O handler and not the
protocol's
CLEANUP: protocol: remove the now unused <handler> field of
proto_fam->bind()
DOC: improve the documentation for "option nolinger"
BUG/MEDIUM: proxy: properly stop backends
BUG/MEDIUM: task: bound the number of tasks picked from the wait queue at
once
MINOR: threads: augment rwlock debugging stats to report seek lock stats
MINOR: threads: add the transitions to/from the seek state
MEDIUM: task: use an upgradable seek lock when scanning the wait queue
BUILD: listener: avoir a build warning when threads are disabled
---
