Hi,
HAProxy 2.6.10 was released on 2023/03/10. It added 78 new commits
after version 2.6.9.
A bit more than half of the commits are HTTP3/QUIC fixes. However, as
indicated in the 2.8-dev5 announce, a concurrency bug introduced in 2.5
was fixed in this version, that may cause freezes and crashes when some
HTTP/1 backend connections are closed by the server exactly at the same
time they're going to be reused by another thread. Another different bug
also affecting idle connections since 2.2 was fixed, possibly causing an
occasional crash. One possible work-around if you've faced such issues
recently is to disable inter-thread connection reuse with this directive
in the global section:
tune.idle-pool.shared off
But beware that this may increase the total number of connections kept
established with your backend servers depending the reuse frequency and
the number of threads.
I want to be clear on one point: the issue is structural, and trying to
port these fixes to 2.6 just made the situation much worse! The only
solution we found to address it relies on some facilities that were
integrated in 2.7 and that offer the guarantees we need during certain
critical transitions of a file descriptor state (refcount and owning
thread group). I have not found any workable solution to the problem
without these facilities, so this required that I backported the strict
minimum amount of patches (18) to bring these facilities there. I hate
having to do that but this time there was no other option. And that made
me realize that instead of keeping 2.6 on its own half-way architecture
like it was, it's not a bad thing that it ressembles next versions to
make backports of fixes more reliable in the future. Despite the large
amount of tests in legacy and master-worker modes, with/without threads,
with reloads, FD passing, saturated listeners etc, it remains possible
that I failed on a corner case. So please watch a little bit more than
usual after you update, and do not hesitate to report any issue you
think you might face consecutive to this.
Other, less critical, issues are described below.
In master-worker mode, when performing an upgrade from an old version
(before 1.9) to a newer version (>=2.5) the HAPROXY_PROCESSES environment
variable was missing, and this combined with a missing element in an
internal structure representing old processes will result in a null-deref
which will crash the master process after the reload. It's very unlikely
to hit this one, except during migration attempts where it can make one
think the new version doesn't work, and encourage to roll back to the
older one. The reported uptime for processes was also fixed so that wall
clock time is used instead of the internal timer.
A few issues affecting the Lua mapping of the HTTP client were addressed;
one of them is a small memory leak by which a few bytes could leak per
request, which could become problematic if used heavily. Another one is
a concurrency issue with Lua's garbage collector that didn't sufficiently
lock other threads' items while trying to free them.
It was found that the low-latency scheduling of TLS handshakes can
degenerate during extreme loads, and take a long time to recover. The
problem is that in order to prevent TLS handshakes from causing high
latency spikes to the rest of the traffic, they're placed in a dedicated
scheduling class that executes one of them per polling loop. But if there
are too many pending due to a big burst, the extra latency caused to the
pending ones can make clients give up and try again, reaching the point
where none of the processed tasks yields anything useful since they were
already abandonned. Now the number of handshakes per loop will grow as
the number of pending ones grows, and this addresses the problem without
adding extra latency even under extreme loads.
There were various QUIC fixes aiming at addressing some issues reported
by users and tests.
The cache failed to cache a response for a request that had the "no-cache"
directive (typically a forced reload). This prevented from refreshing the
cache this way, this is now fixed.
In some rare cases it was possible to freeze a compressing stream if there
was exactly one byte left at the end of the buffer, which was insufficient
to place a new HTX block and prevented any progress from being made. This
has been the case since 2.5 so it doesn't seem easy to trigger!
Layer7 retries did not work anymore on the "empty-response" condition due
to a change that was made in 2.4.
The dump of the supported config language keywords with -dK incorrectly
attributed some of the crt-list specific keywords to "bind ... ssl", which
could cause confusion for those designing config parsers or generators by
regularly checking for new stuff there. Now an explicit "crt-list" sub-
section is dumped and "bind ssl" only dumps keywords really supported on
"bind" lines.
The global directive "no numa-cpu-mapping" that forces haproxy to bind to
multiple CPU
