Hi,

HAProxy 2.4-dev17 was released on 2021/04/23. It added 168 new commits
after version 2.4-dev16.

Yes there are more commits than usual because I simply forgot to emit a
release last week, being busy in the code, sorry about that.

There was a non-negligible number of small bug fixes in various locations,
most of which were in fact detected on older versions and applied there.
A few more important ones that Christopher already detailed in the 2.3.10
announce are there as well, most notably the problem with too short
CONTINUATION frames in the H2 parser that can cause spurious wakeups,
and the recent regression on failure to detect some H2 shutdowns that's
still in the comet tail of the initial truncated frames issues. But as
usual I'm not going to enumerate the bugs fixed in development since it's
mostly uninteresting.

Among the most visible changes, I can quickly enumerate:

  - Amaury implemented experimental removal of added servers. It's still
    under the "experimental-mode on" on the CLI. It's only for servers
    that were added that way. We're *really* interested in feedback on
    add+del server. We were extremely careful to try to spot all conditions
    that can result in a safe removal (no queue, no connections, no check,
    etc) but I could easily admit that we might have forgotten a special
    case that we couldn't imagine. Similarly if you see that they seem to
    work for you but are unusable because of some missing options, please
    report it.

  - Aleks added the new json_query() converter to extract any element
    from a JSON payload.

  - Moemen added the ub64dec/ub64enc converters to encode/decode messages
    using the URL-compatible variant of base64. This one is used in JWT.

  - Tim added the normalize-uri list of actions, which are usable to clean
    up paths or query strings. It would be nice to have some feedback on
    these ones because as I've already expressed a year ago and recently,
    I'm still not fond of how the configuration looks like and I would
    prefer if we could have something durable for merging in 2.4, because
    even though they're marked experimental, users will copy-paste config
    snippets and we might have to maintain this for a very long time.
    Regardless of the configuration aspect, the feature is really great
    to have because it opens new possibilities to perform stricter checks
    on requests. So please test that!

  - based on the previous discussions on the subject, I finally imported
    libslz and we now have memory-less compression enabled by default. It
    is not possible to build with an external copy of the lib any more
    (given that aside Fedora no other distro packages it, it will not be
    missed). It's still possible to disable it or to enable zlib instead.

  - Amaury implemented NUMA topology detection. The idea is that we all
    know how the performance is horrible when starting unbounded threads
    on multi-socket machines, and since v2.0 threads are automatically
    enabled, so some unsuspecting users could get worse performance than
    before when not setting anything. So here what is done is that if
    haproxy detects that the user didn't configure the thread count,
    instead of using all available CPUs, it will use all available CPUs
    of a single physical socket. This results in much better performance
    by default while saving resources at the same time. The detection
    only works on Linux for now.

  - As part of the NUMA work above, the maximum CPU number was raised to
    what the OS supports, so that it's now possible to bind 18 threads to
    CPUs 0..17 and 18 other threads to CPUs 60..77 for example. The total
    number of threads still remains limited to 64 however (but ideas are
    flotting on this subject for 2.5, still too early to discuss for now).

  - the contrib/ directory is finally gone! Yeah, at last! All that
    remained there was version-agnostic (only SPOA agents) so they were
    moved where their authors/maintainers had easiest access to maintain
    them and the links were updated in the Wiki. The example ones were
    added on github/haproxy for convenience. This may even make them
    easier to find, especially the examples.

  - a few improvements were made to some sample fetch functions to be
    usable in tcp-checks. My understanding is that this may improve
    the logs verbosity and may also allow to pass some info to servers
    while building the check request.

More for developers:

  - some list macros were renamed to remove the ADD/ADDQ confusion and
    special care was taken to avoid conflicts with old names so that any
    out-of-tree code can still have its own defines if needed (thinking
    about Nuster and haproxytech's modules among others)

  - the most significant part of the pools cleanup is now done. The
    cache is now globally enabled or disabled, with an option for the
    shared one. Options (such as fault injections) now even apply to
    cache requests. I still didn't manage to sort out the 3 versions of
    the shared cache that slows down the rest of the cleanups. Maybe
    that will be for 2.5 if that takes too much time.

  - the freq-counters were finally merged so that the ones working on
    a per-second period are just the same as the arbitrary period ones
    with a period of 1 second. There was no more benefit in keeping
    two implementations and it resulted in bugs already.

  - the time keeping code was simplified and cleaned up since the last
    user of the current millisecond was the per-second freq counter.
    Interestingly we've already noticed some CPU spent on time keeping
    in the past (when threads would disagree on the current time), and
    while the previous method would make it hard for them to converge,
    the new one makes this way simpler and avoids the vast majority of
    write conflicts.

  - introduction of a __read_mostly keyword to group variables that are
    rarely written to into their own section for lower false-sharing of
    cache lines between CPUs (e.g. global array pointers). It's a good
    habit to take when you're thinking "too bad it cannot be made const".

As usual there are more details in the short log below.

I'm thinking about implementing the relative-path next week and a few
".if" macros as well. During a discussion with Moemen last week we figured
that we were still missing an CLI command to atomically replace a map/acl
while the code already permits it (we just need to call pat_ref_load(next_gen)
instead of pat_ref_add() in map_add_key_value(), but of course the devil
hides in the details). I'll see if that ends up being easy and might
possibly do it. If done I think it might even be backportable to 2.3. We'll
see.

Anyway, do not forget to test and report feedback, especially those who've
met issues recently (including with stable versions, because the 2.4-dev
compatibility is excellent and can even help spot config bugs when run with
-dD). And if you're interested in a feature, test it now, after the release
it will be too late to change it!

Please find the usual URLs below :
   Site index       : http://www.haproxy.org/
   Discourse        : http://discourse.haproxy.org/
   Slack channel    : https://slack.haproxy.org/
   Issue tracker    : https://github.com/haproxy/haproxy/issues
   Wiki             : https://github.com/haproxy/wiki/wiki
   Sources          : http://www.haproxy.org/download/2.4/src/
   Git repository   : http://git.haproxy.org/git/haproxy.git/
   Git Web browsing : http://git.haproxy.org/?p=haproxy.git
   Changelog        : http://www.haproxy.org/download/2.4/src/CHANGELOG
   Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/

Willy
---
Complete changelog :
Alex (2):
      MINOR: sample: converter: Add mjson library.
      MINOR: sample: converter: Add json_query converter

Amaury Denoyelle (17):
      BUG/MINOR: server: free srv.lb_nodes in free_server
      BUG/MINOR: logs: free logsrv.conf.file on exit
      BUG/MEDIUM: server: ensure thread-safety of server runtime creation
      MINOR: server: add log on dynamic server creation
      MINOR: server: implement delete server cli command
      BUG/MINOR: server: fix potential null gcc error in delete server
      BUG/MEDIUM: config: fix cpu-map notation with both process and threads
      MINOR: config: add a diag for invalid cpu-map statement
      MINOR: cpuset: define a platform-independent cpuset type
      MINOR: cfgparse: use hap_cpuset for parse_cpu_set
      MEDIUM: config: use platform independent type hap_cpuset for cpu-map
      MINOR: thread: implement the detection of forced cpu affinity
      MINOR: cfgparse: support the comma separator on parse_cpu_set
      MEDIUM: cfgparse: detect numa and set affinity if needed
      MINOR: global: add option to disable numa detection
      BUG/MINOR: haproxy: fix compilation on macOS
      BUG/MINOR: cpuset: fix compilation on platform without cpu affinity

Christopher Faulet (32):
      MINOIR: mux-pt/trace: Register a new trace source with its events
      BUG/MINOR: mux-pt: Fix a possible UAF because of traces in mux_pt_io_cb
      MINOR: trace: Add the checks as a possible trace source
      MINOIR: checks/trace: Register a new trace source with its events
      MINOR: hlua: Add function to release a lua function
      BUG/MINOR: hlua: Fix memory leaks on error path when registering a task
      BUG/MINOR: hlua: Fix memory leaks on error path when registering a 
converter
      BUG/MINOR: hlua: Fix memory leaks on error path when registering a fetch
      BUG/MINOR: hlua: Fix memory leaks on error path when parsing a lua action
      BUG/MINOR: hlua: Fix memory leaks on error path when registering an action
      BUG/MINOR: hlua: Fix memory leaks on error path when registering a service
      BUG/MINOR: hlua: Fix memory leaks on error path when registering a cli 
keyword
      BUG/MINOR: cfgparse/proxy: Fix some leaks during proxy section parsing
      BUG/MINOR: listener: Handle allocation error when allocating a new 
bind_conf
      BUG/MINOR: cfgparse/proxy: Hande allocation errors during proxy section 
parsing
      MINOR: cfgparse/proxy: Group alloc error handling during proxy section 
parsing
      BUG/MEDIUM: threads: Ignore current thread to end its harmless period
      MINOR: threads: Only consider running threads to end a thread harmeless 
period
      BUG/MINOR: checks: Set missing id to the dummy checks frontend
      MINOR: logs: Add support of checks as session origin to format lf strings
      BUG/MINOR: connection: Fix fc_http_major and bc_http_major for TCP 
connections
      MINOR: connection: Make bc_http_major compatible with tcp-checks
      BUG/MINOR: ssl-samples: Fix ssl_bc_* samples when called from a 
health-check
      BUG/MINOR: http-fetch: Make method smp safe if headers were already 
forwarded
      MINOR: tcp_samples: Add samples to get src/dst info of the backend 
connection
      MINOR: tcp_samples: Be able to call bc_src/bc_dst from the health-checks
      BUG/MINOR: http_htx: Remove BUG_ON() from http_get_stline() function
      BUG/MINOR: logs: Report the true number of retries if there was no 
connection
      BUILD: makefile: Redirect stderr to /dev/null when probing options
      BUG/MAJOR: mux-h2: Properly detect too large frames when decoding headers
      BUG/MEDIUM: mux-h2: Fix dfl calculation when merging CONTINUATION frames
      BUG/MEDIUM: mux-h2: Properly handle shutdowns when received with data

Emeric Brun (3):
      BUG/MINOR: peers: remove useless table check if initial resync is finished
      BUG/MEDIUM: peers: re-work connection to new process during reload.
      BUG/MEDIUM: peers: re-work refcnt on table to protect against flush

Ilya Shipitsin (2):
      CI: cirrus: install "pcre" package
      CI: travis-ci: enable weekly graviton2 builds

Maximilian Mader (3):
      BUG/MINOR: uri_normalizer: Use delim parameter when building the sorted 
query in uri_normalizer_query_sort
      CLEANUP: uri_normalizer: Remove trailing whitespace
      MINOR: uri_normalizer: Add a `strip-dot` normalizer

Miroslav Zagorac (2):
      MINOR: opentracing: correct calculation of the number of arguments in the 
args[]
      MINOR: opentracing: transfer of context names without prefix

Moemen MHEDHBI (2):
      MINOR: sample: add ub64dec and ub64enc converters
      CLEANUP: sample: align samples list in sample.c

Remi Tricot-Le Breton (3):
      DOC: internals: update the SSL architecture schema
      DOC: ssl: Certificate hot update only works on fronted certificates
      DOC: ssl: Certificate hot update works on server certificates

Thayne McCombs (1):
      BUG/MEDIUM: sample: Fix adjusting size in field converter

Tim Duesterhus (16):
      CI: travis: Drastically clean up .travis.yml
      MINOR: ist: Add `istclear(struct ist*)`
      MINOR: uri_normalizer: Add uri_normalizer module
      MINOR: uri_normalizer: Add `enum uri_normalizer_err`
      MINOR: uri_normalizer: Add `http-request normalize-uri`
      MINOR: uri_normalizer: Add a `merge-slashes` normalizer to http-request 
normalize-uri
      MINOR: uri_normalizer: Add a `dotdot` normalizer to http-request 
normalize-uri
      MINOR: uri_normalizer: Add support for supressing leading `../` for 
dotdot normalizer
      MINOR: uri_normalizer: Add a `sort-query` normalizer
      MINOR: uri_normalizer: Add a `percent-upper` normalizer
      MEDIUM: http_act: Rename uri-normalizers
      DOC: Add introduction to http-request normalize-uri
      DOC: Note that URI normalization is experimental
      CLEANUP: sample: Improve local variables in sample_conv_json_query
      CLEANUP: sample: Explicitly handle all possible enum values from mjson
      CLEANUP: sample: Use explicit return for successful `json_query`s

William Lallemand (2):
      BUG/MINOR: mworker/init: don't reset nb_oldpids in non-mworker cases
      BUG/MINOR: mworker: don't use oldpids[] anymore for reload

Willy Tarreau (83):
      CLEANUP: pattern: make all pattern tables read-only
      MINOR: trace: replace the trace() inline function with an equivalent macro
      MINOR: initcall: uniformize the section names between MacOS and other 
unixes
      CLEANUP: initcall: rename HA_SECTION to HA_INIT_SECTION
      MINOR: compiler: add macros to declare section names
      CLEANUP: initcall: rely on HA_SECTION_* instead of defining its own
      MINOR: global: declare a read_mostly section
      MINOR: fd: move a few read-mostly variables to their own section
      MINOR: epoll: move epoll_fd to read_mostly
      MINOR: kqueue: move kqueue_fd to read_mostly
      MINOR: pool: move pool declarations to read_mostly
      MINOR: threads: mark all_threads_mask as read_mostly
      MINOR: server: move idle_conn_task to read_mostly
      MINOR: protocol: move __protocol_by_family to read_mostly
      MINOR: pattern: make the pat_lru_seed read_mostly
      MINOR: trace: make trace sources read_mostly
      MINOR: freq_ctr: add a generic function to report the total value
      MEDIUM: freq_ctr: make read_freq_ctr_period() use freq_ctr_total()
      MEDIUM: freq_ctr: reimplement freq_ctr_remain_period() from 
freq_ctr_total()
      MINOR: freq_ctr: add the missing next_event_delay_period()
      MINOR: freq_ctr: unify freq_ctr and freq_ctr_period into freq_ctr
      MEDIUM: freq_ctr: replace the per-second counters with the generic ones
      MINOR: freq_ctr: add cpu_relax in the rotation loop of 
update_freq_ctr_period()
      MINOR: freq_ctr: simplify and improve the update function
      CLEANUP: time: remove the now unused ms_left_scaled
      MINOR: time: move the time initialization out of tv_update_date()
      MINOR: time: remove useless variable copies in tv_update_date()
      MINOR: time: change the global timeval and the the global tick at once
      MEDIUM: time: make the clock offset global and no per-thread
      MINOR: atomic: reimplement the relaxed version of x86 BTS/BTR
      BUG/MINOR: pools: maintain consistent ->allocated count on alloc failures
      BUG/MINOR: pools/buffers: make sure to always reserve the required buffers
      MINOR: pools: drop the unused static history of artificially failed allocs
      CLEANUP: pools: remove unused arguments to pool_evict_from_cache()
      MEDIUM: pools: move the cache into the pool header
      MINOR: pool: remove the size field from pool_cache_head
      MINOR: pools: rename CONFIG_HAP_LOCAL_POOLS to CONFIG_HAP_POOLS
      MINOR: pools: enable the fault injector in all allocation modes
      MINOR: pools: make the basic pool_refill_alloc()/pool_free() update 
needed_avg
      MEDIUM: pools: unify pool_refill_alloc() across all models
      CLEANUP: pools: re-merge pool_refill_alloc() and __pool_refill_alloc()
      MINOR: pools: call pool_alloc_nocache() out of the pool's lock
      CLEANUP: pools: move the lock to the only __pool_get_first() that needs it
      CLEANUP: pools: rename __pool_get_first() to pool_get_from_shared_cache()
      CLEANUP: pools: rename pool_*_{from,to}_cache() to *_local_cache()
      CLEANUP: pools: rename __pool_free() to pool_put_to_shared_cache()
      MINOR: tools: add statistical_prng_range() to get a random number over a 
range
      MINOR: pools: use cheaper randoms for fault injections
      MINOR: pools: move the fault injector to __pool_alloc()
      MINOR: pools: split the OS-based allocator in two
      MINOR: pools: always use atomic ops to maintain counters
      MINOR: pools: move pool_free_area() out of the lock in the locked version
      MINOR: pools: factor the release code into pool_put_to_os()
      MEDIUM: pools: make CONFIG_HAP_POOLS control both local and shared pools
      MINOR: pools: create unified pool_{get_from,put_to}_cache()
      MINOR: pools: evict excess objects using pool_evict_from_local_cache()
      MEDIUM: pools: make pool_put_to_cache() always call 
pool_put_to_local_cache()
      CLEANUP: pools: make the local cache allocator fall back to the shared 
cache
      CLEANUP: pools: merge pool_{get_from,put_to}_local_caches with generic 
ones
      CLEANUP: pools: uninline pool_put_to_cache()
      CLEANUP: pools: declare dummy pool functions to remove some ifdefs
      BUILD: pools: fix build with DEBUG_FAIL_ALLOC
      BUG/MINOR: server: make srv_alloc_lb() allocate lb_nodes for consistent 
hash
      CONTRIB: mod_defender: import the minimal number of includes
      CONTRIB: mod_defender: make the code build with the embedded includes
      CONTRIB: modsecurity: import the minimal number of includes
      CONTRIB: modsecurity: make the code build with the embedded includes
      CLEANUP: lists/tree-wide: rename some list operations to avoid some 
confusion
      CONTRIB: move spoa_example out of the tree
      CONTRIB: move spoa_server out of the tree
      CONTRIB: move modsecurity out of the tree
      CONTRIB: move mod_defender out of the tree
      CLEANUP: contrib: remove the last references to the now dead contrib/ 
directory
      BUILD: makefile: fix the "make clean" target on strict bourne shells
      IMPORT: slz: import slz into the tree
      BUILD: compression: switch SLZ from out-of-tree to in-tree
      CI: github: do not build libslz any more
      CLEANUP: compression: remove calls to SLZ init functions
      MINOR: time: avoid unneeded updates to now_offset
      MINOR: time: avoid overwriting the same values of global_now
      CLEANUP: time: use __tv_to_ms() in tv_update_date() instead of open-coding
      MINOR: time: avoid u64 needlessly expensive computations for the 32-bit 
now_ms
      BUG/MEDIUM: config: fix missing initialization in numa_detect_topology()

---

Reply via email to