Hi,
HAProxy 2.2-dev12 was released on 2020/07/04. It added 72 new commits
after version 2.2-dev11.
Yes, a 12th development release. But the good news is that it's just
here to help with testing because we've finally managed to address the
performance regression issue spotted by William Dauchy! And it was quite
a tough one, so it was a good decision to invested so many efforts on
this one before the release.
To make a long story short, when you have many very fast servers, almost
all of a server's idle connections could be used then released by a thread
at once, and taken over by the next thread and so on, never leaving spare
ones for other threads. And this takeover would go through the global run
queue and cause contention there when using a moderate number of threads.
So that wouldn't affect low-performance users but definitely was a
performance killer for high performance ones dealing with tens to hundreds
of thousands of requests per second.
The great thing is that by fixing all these issues we've had to implement
a few improvements that were anticipated for later and that this made the
internal infrastructure a bit better and further improved the overall
performance gap from 2.1.
Since 2.2-dev11, the most user-visible changes are:
- fixed the performance regression above
- addition of the new "pool-low-conn" server setting to improve distribution
of idle connections on very fast servers (sub-millisecond response time).
We've found that using twice the number of threads seems to provide very
good performance.
- added a few new fields in the stats page to report the number of idle
and used connections per server
- new "show servers conn" command on the CLI to visualize the state of used
and idle connections of a server, including per-thread
- small change on the log-format processing: historically, multiple spaces
were merged together as a single separator. This was OK for real logs,
but is a bit annoying when building headers, and very annoying for error
pages. So this was changed so that only logs merge spaces. This should
probably be addressed in a more generic way later, but this was the most
reasonble approach for this release.
- the RFC5424 log format was missing the sub-second and timezone fields,
the former being highly recommended and the latter being mandatory. So
this was addressed right before having a new LTS version. I'm not much
tempted by backporting this to stable releases because that could result
in visible changes that are not welcome in the middle of a stable version,
that's why I asked to have it right now.
- a few sample fetches and patterns were missing the trailing NUL character
and wouldn't always match (I don't remember which ones, sorry). This will
likely be backported as it was a bug.
- threads are now disabled by default on OpenBSD which lacks thread-local
storage and fails to build. Clang seems to emulate it so users of clang
can enable USE_THREAD=1 if they want.
- "show sess" would endlessly dump new streams when they arrive too fast.
It was a real pain so now it will only dump past the last stream known
at the moment the command is typed. This means that it may show less
streams than the total, but will not result in multi-gigabyte dumps
anymore.
- for developers, building with DEBUG_MEM_STATS provides a new expert
command "debug dev memstats" which shows the total counts (calls and
sizes) of memory allocations per line of code. This is very cheap and
can be enabled on production servers if suspecting a memory leak
somewhere (and it served to spot a regression in a recent fix).
In addition to this, William is finishing the addition of a sample fetch
to extract the equivalent of the TLS pre-master key for TLS 1.3, which
is needed in order to decrypt TLS traffic with Wireshark. It would be
useful to have this early so that those who upgrade can place that in
their logs if that can help them.
Christopher addressed a few other low-importance bugs in the private
connections management. Since they were made available just before this
release and we've all been very tired by looking at these bugs over the
last weeks, I preferred that we let these cool down and we look at them
after a small rest; we've accidently broken enough stuff while working
on the fixes above, I didn't want to take the risk of creating new
breakage. It looks like 3 of these patches could be merged before the
release (they already affect previous versions) and the other ones could
be merged post-2.2 then backported once considered safe enough.
Tim also had some post-2.2 fixes pending to improve free() calls and
remove some valgrind complaints on exit.
With all the energy spent on the bugs above I couldn't work at all on
the doc review I wanted to do. I'll try to do this shortly but it will
not be as refined as I'd had expected.
Anyway, I now consider 2.2 ready. I'm particularly interested in build
issues (please give it a try especially on less common platforms), and
regressions from 2.1. I can retest on an AIX server. My Sparc is still
dead so if anyone could retry on Solaris, that would be nice, especially
since it's been a while since we've last built evports; Manu maybe ?
I've just addressed a last minute build issue affecting DEBUG_LOCK, so
if you build with it and get a failure, just pick the last commit as well.
If all goes well, I'd like to release 2.2-final on Tuesday, with or without
the minor updates mentioned above. I expect this one to be a really good
release, seeing that the usual month of post-release fixes happened before
the release this time!
Please find the usual URLs below :
Site index : http://www.haproxy.org/
Discourse : http://discourse.haproxy.org/
Slack channel : https://slack.haproxy.org/
Issue tracker : https://github.com/haproxy/haproxy/issues
Sources : http://www.haproxy.org/download/2.2/src/
Git repository : http://git.haproxy.org/git/haproxy.git/
Git Web browsing : http://git.haproxy.org/?p=haproxy.git
Changelog : http://www.haproxy.org/download/2.2/src/CHANGELOG
Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/
Willy
---
Complete changelog :
Christopher Faulet (2):
BUG/MINOR: http-rules: Fix ACLs parsing for http deny rules
BUG/MEDIUM: pattern: Add a trailing \0 to match strings only if possible
Dragan Dosen (3):
BUG/MEDIUM: log-format: fix possible endless loop in
parse_logformat_string()
MINOR: 51d: silence a warning about null pointer dereference
MINOR: log-format: allow to preserve spacing in log format strings
Emeric Brun (2):
MINOR: log: add time second fraction field to rfc5424 log timestamp.
BUG/MINOR: log: missing timezone on iso dates.
Ilya Shipitsin (1):
CI: cirrus-ci: exclude slow reg-tests
Olivier Houchard (10):
BUG/MEDIUM: connections: Don't increase curr_used_conns for shared
connections.
BUG/MEDIUM: checks: Increment the server's curr_used_conns
BUG/MINOR: threads: Don't forget to init each thread toremove_lock.
BUG/MEDIUM: lists: Lock the element while we check if it is in a list.
MINOR: list: Add MT_LIST_DEL_SAFE_NOINIT() and MT_LIST_ADDQ_NOCHECK()
CLEANUP: connections: rename the toremove_lock to takeover_lock
MEDIUM: connections: Don't use a lock when moving connections to remove.
BUG/MEDIUM: muxes: Make sure nobody stole the connection before using it.
BUG/MEDIUM: connections: Set the tid for the old tasklet on takeover.
BUG/MEDIUM: connections: Let the xprt layer know a takeover happened.
Tim Duesterhus (1):
BUG/MINOR: http_act: don't check capture id in backend (2)
William Lallemand (1):
DOC: ssl: add "allow-0rtt" and "ciphersuites" in crt-list
Willy Tarreau (52):
BUG/MINOR: mux_h2: don't lose the leaving trace in h2_io_cb()
MINOR: cli: make "show sess" stop at the last known session
CLEANUP: buffers: remove unused buffer_wq_lock lock
BUG/MEDIUM: buffers: always allocate from the local cache first
MINOR: connection: align toremove_{lock,connections} and cleanup into
idle_conns
CONTRIB: debug: add missing flags SI_FL_L7_RETRY & SI_FL_D_L7_RETRY
REORG: buffer: rename buffer.c to dynbuf.c
REORG: includes: create tinfo.h for the thread_info struct
CLEANUP: pool: only include the type files from types
MINOR: pools: move the LRU cache heads to thread_info
BUG/MINOR: debug: fix "show fd" null-deref when built with DEBUG_FD
MINOR: stats: add 3 new output values for the per-server idle conn state
MINOR: activity: add per-thread statistics on FD takeover
BUG/MINOR: server: start cleaning idle connections from various points
MEDIUM: server: improve estimate of the need for idle connections
MINOR: stats: add the estimated need of concurrent connections per server
Revert "BUG/MEDIUM: lists: Lock the element while we check if it is in a
list."
BUG/MINOR: haproxy: don't wake already stopping threads on exit
BUG/MINOR: server: always count one idle slot for current thread
MEDIUM: server: use the two thresholds for the connection release
algorithm
BUG/MINOR: sched: properly cover for a rare MT_LIST_ADDQ() race
MINOR: mux-h1: avoid taking the toremove_lock in on dying tasks
MINOR: mux-h2: avoid taking the toremove_lock in on dying tasks
MINOR: mux-fcgi: avoid taking the toremove_lock in on dying tasks
MINOR: pools: increase MAX_BASE_POOLS to 64
BUG/MINOR: proxy: fix dump_server_state()'s misuse of the trash
BUG/MINOR: proxy: always initialize the trash in show servers state
MINOR: cli/proxy: add a new "show servers conn" command
MINOR: server: skip servers with no idle conns earlier
BUG/MINOR: server: fix the connection release logic regarding nearly full
conditions
MEDIUM: server: add a new pool-low-conn server setting
BUG/MEDIUM: backend: always search in the safe list after failing on the
idle one
MINOR: backend: don't always takeover from the same threads
MINOR: sched: make sched->task_list_size atomic
MEDIUM: sched: create a new TASK_KILLED task flag
MEDIUM: sched: implement task_kill() to kill a task
MEDIUM: mux-h1: use task_kill() during h1_takeover() instead of
task_wakeup()
MEDIUM: mux-h2: use task_kill() during h2_takeover() instead of
task_wakeup()
MEDIUM: mux-fcgi: use task_kill() during fcgi_takeover() instead of
task_wakeup()
DOC: configuration: add missing index entries for
tune.pool-{low,high}-fd-ratio
DOC: configuration: fix alphabetical ordering for
tune.pool-{high,low}-fd-ratio
MINOR: config: add a new tune.idle-pool.shared global setting.
MINOR: debug: add a new "debug dev memstats" command
BUILD: debug: avoid build warnings with DEBUG_MEM_STATS
BUG/MAJOR: sched: make sure task_kill() always queues the task
BUG/MEDIUM: cli/proxy: don't try to dump idle connection state if there's
none
BUILD: haproxy: fix build error when RLIMIT_AS is not set
BUG/MAJOR: sched: make it work also when not building with DEBUG_STRICT
BUG/MEDIUM: server: don't kill all idle conns when there are not enough
MINOR: sched: split tasklet_wakeup() into tasklet_wakeup_on()
BUILD: makefile: disable threads by default on OpenBSD
BUILD: peers: fix build warning with gcc 4.2.1
---
