Hi,
HAProxy 2.4-dev19 was released on 2021/05/10. It added 155 new commits
after version 2.4-dev18.
It took longer than I expected because of late discoveries of some
monstrosities with the includes which are responsible for the long
build times. When you see that the task scheduler or arguments
parser implicitly include openssl and resulted in 41k lines having
to be compiled, that's quite annoying. At least now I figured the
root cause. It's because openssl uses typedefs to define types, and
doing so prevents one from using forward declaration of incomplete
types, forcing all struct members that require a pointer to include
the file (these are in struct server and struct listener, so virtually
everywhere). On the forthcoming week I'll still look if a cleaner cut
is possible between the include files, othrewise we'll defer this to
2.5.
With that aside, I think we're done, with everything planed having been
completed by the end of last week.
Among the user-visible changes, here's what I'm having in mind in random
order:
- CLI: fixed a memory leak on write errors. Also the "help" output was
re-aligned and sorted for better legibility. Larger lines allowed to
report the usage syntax that most commands were missing.
- experimental commands/directives: using experimental directives in
the configuration will require an opt-in by placing the directive
"expose-experimental-directives" in the global section. The goal is
to make sure that random copy-paste from blogs doesn't result in users
being exposed without being aware of it. For now this affects the
"tune.fd.edge-triggered" global option and "http-request normalize-uri"
action.
- use of declaration and execution of experimental or expert features
is reported with a new "tainted" flag reported in "show info", that
should primarily help developers know if anything dangerous or less
known was done on the process, but could also possibly help an admin
figure that someone used some tricky commands on the CLI.
- the spelling inconsistencies between "haproxy/Haproxy/HAproxy/HA-Proxy"
etc were (hopefully) all addressed, including the one that appears in
"haproxy -v" that I already forgot to fix before previous release.
- the "abortonclose" option was fixed for HTTP/1, as in some cases the
mux wouldn't try to read anymore thus wouldn't detect a client close.
- an optional memory profiler was added to detect where memory is used.
It's enabled by "set profiling memory on" on the CLI and consulted
using "show profiling memory". It shows the number of allocations and
releases per caller as well as their size. It's interesting to watch
when using external libs like openssl or Lua. It works well on Linux
and glibc, I managed to get it to build and work on FreeBSD as well,
though with a bit less visibility for now. It uses very little
resources so I wanted to have it merged so that in the future we
could ask some issue reporters to enable it.
- idle frontend connections will now be closed on reload. Till now they
could only be closed when after receiving and processing a new request,
forcing many users to add a hard-stop-after directive. Now for the vast
majority of users this should be cleaner and the old process should
disappear much faster.
- on reload, idle backend connections were not explicitly closed,
resulting in a "normal" death when the process exited. This used to
cause some TIME_WAIT sockets to be left, consuming source ports for
a long time. In some environments it's a problem. Now there's an
explicit kill of these connections when quitting so as not to leave
such entries lying around.
- the "ALERT/WARNING/NOTICE/DIAG" output messages in foreground were
cleaned up to drop the unused time code and to align them so that
they read better.
- the .if/.elif config directives now support a few predicates to isolate
a config block. The strict minimum was done to help with migrations,
test configs for developers, and more portable regtests in the future.
We currently have "version_atleast()" and "version_before()" which test
for the current version being at least or before the mentioned one,
"feature()" to test for a feature to appear in the "Feature list" of
"haproxy -vv", "streq()" and "strneq()" to test for string equality or
difference, and "defined()" to test for the existence of an environment
variable. I think that some of us will stop modifying their test configs
10 times a day to enable/disable SSL or PROMEX for example, and that
we won't replace as many VTCs between 2.4 and 2.5 as we replaced between
2.3 and 2.4.
- the diagnostic mode (-dD) will not complain anymore about peers having
a zero weight, and will also report annotations left by the admin using
the new ".diag" config directive.
- "show info" now supports an optional "float" option that will make
some fields being emitted as floats instead of integers. This is true
for the start time, uptime, and some rates for now. Some graphs could
look smoother, especially at very low rates where discrete values were
a problem.
- the new pseudo-environment variables ".FILE", ".LINE" and ".SECTION"
will reflect respectively the config file name, the line and the section
being parsed. These can be useful in health checks, headers, or error
log messages to help the admin locate the offending rule.
- the DEBUG_DONT_SHARE_POOLS build option that some use in production
for improved debugging was suffering from a misdesign: while its goal
has always been to avoid sharing of pools of identical size, some
pools are multiply defined (like stick-tables or captures) and those
were never merged, resulting in a high memory usage for those doing
it with many stick-tables or captures. Now those of same name will
still be merged if they're of identical size.
- the regtests that were recently failing about 30-50% of the time in
the CI have now been stabilized. The cause turned out to be that idle
connection sharing works so well that some dummy server connections
were abandoned by the server at the moment we were about to reuse
them, occasionally causing an "incomplete HTTP header" report in the
VTest output. The affected tests were adjusted to limit the connection
sharing and now the tests appear stable again.
- those building with USE_QUIC=1 should have less surprises now that the
CI builds it
- the keyword argument parser was improved to better report missing
arguments and missing parenthesis instead of saying things like
"error before ''".
I really don't expect any more significant changes being applied before
the release now, so we'll focus on cosmetic updates, doc and build tests
on various platforms. I'll add a few CPU entries in the Makefile to ease
building on modern ARM platforms and recheck the docs aimed at newcomers
(contributing, reporting issues etc).
Unless a huge bug falls in front of us blocking any progress, I think
we'll release this week, ideally on Friday morning so that those in search
of a distraction for their boring Friday afternoon have something to play
with :-)
So if you still have a comment to formulate, hurry up!
Please find the usual URLs below :
Site index : http://www.haproxy.org/
Discourse : http://discourse.haproxy.org/
Slack channel : https://slack.haproxy.org/
Issue tracker : https://github.com/haproxy/haproxy/issues
Wiki : https://github.com/haproxy/wiki/wiki
Sources : http://www.haproxy.org/download/2.4/src/
Git repository : http://git.haproxy.org/git/haproxy.git/
Git Web browsing : http://git.haproxy.org/?p=haproxy.git
Changelog : http://www.haproxy.org/download/2.4/src/CHANGELOG
Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/
Willy
---
Complete changelog :
Amaury Denoyelle (15):
MINOR: srv: close all idle connections on shutdown
MINOR: connection: move session_list member in a union
MEDIUM: mux_h1: release idling frontend conns on soft-stop
MEDIUM: connection: close front idling connection on soft-stop
MINOR: global: define tainted flag
MINOR: cfgparse: add a new field flags in cfg_keyword
MINOR: cfgparse: implement experimental config keywords
MINOR: action: replace match_pfx by a keyword flags field
MINOR: action: implement experimental actions
MINOR: cli: set tainted when using CLI expert/experimental mode
MINOR: stats: report tainted on show info
MINOR: http_act: mark normalize-uri as experimental
BUILD: fix usage of ha_alert without format string
MINOR: proxy: define PR_CAP_LB
BUG/MINOR: server: do not report diag for peer servers with null weight
Christopher Faulet (17):
BUG/MINOR: hlua: Don't rely on top of the stack when using Lua buffers
MINOR: channel: Rely on HTX version if appropriate in channel_may_recv()
BUG/MINOR: stream-int: Don't block reads in si_update_rx() if chn may
receive
MINOR: conn-stream: Force mux to wait for read events if abortonclose is
set
MEDIUM: mux-h1: Don't block reads when waiting for the other side
BUG/MEDIUM: mux-h1: Properly report client close if abortonclose option
is set
REGTESTS: Add script to test abortonclose option
MINOR: mux-h1: clean up conditions to enabled and disabled splicing
MINOR: mux-h1: Subscribe for sends if output buffer is not empty in
h1_snd_pipe
MINOR: mux-h1: Always subscribe for reads when splicing is disabled
MEDIUM: mux-h1: Wake H1 stream when both sides a synchronized
CLEANUP: mux-h1: rename WAIT_INPUT/WAIT_OUTPUT flags
MINOR: mux-h1: Manage processing blocking flags on the H1 stream
BUG/MINOR: stream: Decrement server current session counter on L7 retry
BUG/MINOR: stream: Reset stream final state and si error type on L7 retry
BUG/MINOR: checks: Handle synchronous connect when a tcpcheck is started
BUG/MINOR: checks: Reschedule check on observe mode only if fastinter is
set
Daniel Corbett (1):
DOC: Fix a few grammar/spelling issues and casing of HAProxy
Ilya Shipitsin (2):
CI: Github Actions: switch to LibreSSL-3.3.3
CI: Github Actions: enable USE_QUIC=1 for BoringSSL builds
Remi Tricot-Le Breton (2):
DOC: ssl: Extra files loading now works for backends too
DOC: ssl: Add information about crl-file option
William Lallemand (1):
BUG/MINOR: ssl/cli: fix a lock leak when no memory available
Willy Tarreau (117):
BUG/MEDIUM: cli: prevent memory leak on write errors
MINOR: debug: add a new "debug dev sym" command in expert mode
MINOR: pools/debug: slightly relax DEBUG_DONT_SHARE_POOLS
MINOR: tools: add functions to retrieve the address of a symbol
CLEANUP: activity: mark the profiling and task_profiling_mask
__read_mostly
MINOR: activity: add a "memory" entry to "profiling"
MINOR: activity: declare the storage for memory usage statistics
MEDIUM: activity: collect memory allocator statistics with
USE_MEMORY_PROFILING
MINOR: activity: clean up the show profiling io_handler a little bit
MINOR: activity: make "show profiling" support a few arguments
MINOR: activity: make "show profiling" also dump the memoery usage
MINOR: activity: add the profiling.memory global setting
BUILD: makefile: add new option USE_MEMORY_PROFILING
BUG/MINOR: config: fix uninitialized initial state in ".if" block
evaluator
BUG/MINOR: config: add a missing "ELIF_TAKE" test for ".elif" condition
evaluator
BUG/MINOR: config: .if/.elif should also accept negative integers
MINOR: config: centralize the ".if"/".elif" condition parser and evaluator
MINOR: config: keep up-to-date current file/line/section in the global
struct
MINOR: config: support some pseudo-variables for file/line/section
BUILD: activity: do not include malloc.h
MINOR: arg: improve the error message on missing closing parenthesis
MINOR: global: export the build features string list
MINOR: global: add version comparison functions
MINOR: config: improve .if condition error reporting
MINOR: config: make cfg_eval_condition() support predicates with arguments
MINOR: config: add predicate "defined()" to conditional expression blocks
MINOR: config: add predicates "streq()" and "strneq()" to conditional
expressions
MINOR: config: add predicate "feature" to detect certain built-in features
MINOR: config: add predicates "version_atleast" and "version_before" to
cond blocks
BUG/MINOR: activity: use the new pointer to calculate the new size in
realloc()
BUG/MINOR: stream: properly clear the previous error mask on L7 retries
MEDIUM: log: slightly refine the output format of alerts/warnings/etc
MINOR: config: add a new message directive: .diag
CLEANUP: cli/tree-wide: properly re-align the CLI commands' help messages
ADDONS: make addons/ discoverable by git via .gitignore
MINOR: sample: improve error reporting on missing arg to strcmp()
converter
DOC: management: mention that some fields may be emitted as floats
MINOR: tools: implement trimming of floating point numbers
MINOR: tools: add a float-to-ascii conversion function
MINOR: freq_ctr: add new functions to report float measurements
MINOR: stats: avoid excessive padding of float values with trailing zeroes
MINOR: stats: add the HTML conversion for float types
MINOR: stats: pass the appctx flags to stats_fill_info()
MINOR: stats: support an optional "float" option to "show info"
MINOR: stats: use tv_remain() to precisely compute the uptime
MINOR: stats: report uptime and start time as floats with subsecond
resolution
MINOR: stats: make "show info" able to report rates as floats when asked
MINOR: config: mark tune.fd.edge-triggered as experimental
REORG: vars: move the "proc" scope variables out of the global struct
REORG: threads: move all_thread_mask() to thread.h
BUILD: wdt: include signal-t.h
BUILD: auth: include missing list.h
REORG: mworker: move proc_self from global to mworker
BUILD: ssl: ssl_utils requires chunk.h
BUILD: config: cfgparse-ssl.c needs tools.h
BUILD: wurfl: wurfl.c needs tools.h
BUILD: spoe: flt_spoe.c needs tools.h
BUILD: promex: service-prometheus.c needs tools.h
BUILD: resolvers: include tools.h
BUILD: config: include tools.h in cfgparse-listen.c
BUILD: htx: include tools.h in http_htx.c
BUILD: proxy: include tools.h in proxy.c
BUILD: session: include tools.h in session.c
BUILD: cache: include tools.h in cache.c
BUILD: sink: include tools.h in sink.c
BUILD: connection: include tools.h in connection.c
BUILD: server-state: include tools.h from server_state.c
BUILD: dns: include tools.h in dns.c
BUILD: payload: include tools.h in payload.c
BUILD: vars: include tools.h in vars.c
BUILD: compression: include tools.h in compression.c
BUILD: mworker: include tools.h from mworker.c
BUILD: queue: include tools.h from queue.c
BUILD: udp: include tools.h from proto_udp.c
BUILD: stick-table: include freq_ctr.h from stick_table.h
BUILD: server: include tools.h from server.c
BUILD: server: include missing proxy.h in server.c
BUILD: sink: include proxy.h in sink.c
BUILD: mworker: include proxy.h in mworker.c
BUILD: filters: include proxy.h in filters.c
BUILD: fcgi-app: include proxy.h in fcgi-app.c
BUILD: connection: move list_mux_proto() to connection.c
REORG: stick-table: uninline stktable_alloc_data_type()
REORG: stick-table: move composite address functions to stick_table.h
REORG: config: uninline warnifnotcap() and failifnotcap()
BUILD: task: remove unused includes from task.c
MINOR: task: stop including stream.h from task.c
BUILD: connection: stop including listener-t.h
BUILD: hlua: include proxy.h from hlua.c
BUILD: mux-h1: include proxy.h from mux-h1.c
BUILD: mux-fcgi: include proxy.h from mux-fcgi.c
BUILD: listener: include proxy.h from listener.c
BUILD: http-rules: include proxy.h from http_rules.c
BUILD: thread: include log.h from thread.c
BUILD: comp: include proxy.h from flt_http_comp.c
BUILD: fd: include log.h from fd.c
BUILD: config: do not include proxy.h nor errors.h anymore in cfgparse.h
BUILD: makefile: reorder object files by build time
REGTESTS: run-regtests: match both "HAProxy" and "HA-Proxy" in the version
MINOR: version: report "HAProxy" not "HA-Proxy" in the version output
DOC: remove last occurrences of "HA-Proxy" syntax
DOC: peers: fix the protocol tag name in the doc
ADMIN: netsnmp: report "HAProxy" and not "Haproxy" in output descriptions
MEDIUM: mailers: use "HAProxy" nor "HAproxy" in the subject of messages
DOC: fix a few remainig cases of "Haproxy" and "HAproxy" in doc and
comments
MINOR: tools/rnd: compute the result outside of the CAS loop
BUILD: http_fetch: address a few aliasing warnings with older compilers
BUILD: ssl: define HAVE_CRYPTO_memcmp() based on the library version
BUILD: errors: include stdarg in errors.h
REGTESTS: disable inter-thread idle connection sharing on sensitive tests
MINOR: cli: make "help" support a command in argument
MINOR: cli: sort the output of the "help" keywords
CLEANUP: cli/mworker: properly align the help messages
BUILD: memprof: make the old caller pointer a const in get_prof_bin()
BUILD: compat: include malloc_np.h for USE_MEMORY_PROFILING on FreeBSD
BUG/MEDIUM: quic: fix null deref on error path in qc_conn_init()
BUILD: cli: appease a null-deref warning in cli_gen_usage_msg()
---
