Hi,
HAProxy 2.3-dev7 was released on 2020/10/17. It added 56 new commits
after version 2.3-dev6.
I'm pleased to see that development has significantly stabilized and
that the code looks in good shape, with the issue list not dramatically
increasing (and not for 2.3-only stuff, which is great).
A few long-standing bugs have been addressed. One of them, found by
Christopher, is a missing lock when using the "leastconn" or "first" load
balancing algorithms, which at first glance could only trigger a divide by
zero, but it can do much more and was proven to be responsible for causing
spinning loops. And while reviewing the fix, I noticed that the main reason
for slow server lookups when using many servers at their maxconn setting is
that we're still using a spinlock instead of an rwlock to lock the LB
algorithms. The fix is simple, it just requires to carefully pass over all
places, so I want to address it before 2.3-final.
I also detected a snowball effect in the scheduler. Any thread may wake up
expired global tasks. The problem was that this was done by taking a write
lock for each of them until the task list is empty. In this situation, in
a highly threaded environment, nothing prevents one thread from removing
expired entries while a few other threads are having fun of it by feeding
the queue with many more tasks (e.g. checks), preventing it from ever
finishing. I'm pretty sure it was the cause of at least one crash report
in the github issues. I've changed this by improving the locking (which
BTW reduced from 2 to 1% the cost of the scheduler on 8 threads), and by
enforcing a bound on the number of tasks woken up in one round (which is
the backportable part).
There was a crash in "haproxy -c" introduced by my recent protocol layer
changes in -dev5 (fdtab was not allocated yet), that was fixed by William
Dauchy (and that I noticed after fixing it myself in parallel, sorry for
this William).
As recently discussed, some obsolete keywords were removed ("mode health",
"monitor-net"). The proposed warning on "nbproc" was not added yet.
Also there were some long-standing limitations that were addressed by
Amaury. One of them was some head-of-line blocking when using H2 to
connect to backends. This is a limitation of doing multiplexing over an
in-order protocol like TCP. The problem is when requests from multiple
frontend connections are merged into the same backend connection. If one
client reads large objects slower than others, it will sometimes block
the demultiplexer with some unparsed frames. The solution against this
is to reduce the per-stream window, but then each stream may face a very
low bandwidth if servers are far away. We opted for a different solution
here, consisting in merging only requests from the same connection in an
H2 connection by default. When I'm saying "by default", I mean that it's
what is done in "http-reuse safe", because by "safe" we mean "no bad
surprises". Doing this will slightly lower the reuse rate on H2 backends
by default but significantly reduce the jitter in response time experienced
by the users. Other levels (http-reuse aggressive & always) continue to
merge the requests from any connection.
Another limitation is that a connection made to a server over TLS with an
SNI was marked as private because we didn't want to share it with another
request if there was a risk that the expected SNI wouldn't match. However
some users rightfully pointed in the past that very often they just hard
code the SNI in the configuration (e.g. "sni str(example.com)") and in
this case there is no such risk, and it's too bad not to reuse such
connections. Amaury refined the test so that only connections using a
variable expression for the SNI will be marked private. Now it will be
possible to reuse them for configuration-based constants such as above.
This is definitely something we'll have to further improve to support
a small number of valid SNIs for a given server.
UDP receivers now support being paused during reloads, and the TCP ones
have been cleaned up to use connect(AF_UNSPEC) instead of the shutdown+
listen+shutdown dance that was used to try to detect incompatible OSes
that would silently succeed but not work as expected. An issue with the
recent changes to the listener causing the backends never to stop was
addressed, and listeners are now in a relatively clean state so that we
should not significantly break them in the near future.
Fred found a bug causing peers sessions to be reset sometimes. He thinks
there's another bug (that predates 2.3) but it was not identified yet.
Among the remaining ideas floating around for the release, we've had a
discussion yesterday on various points regarding dynamic server management
and one point concerned the verbosity of the "show server-state" and
"show stats" outputs when many server-templates are used. We thought
about adding a possibility not to dump disabled servers, and I progressively
started to see several common points with
