Hi,

HAProxy 2.2-dev8 was released on 2020/05/22. It added 124 new commits
after version 2.2-dev7.

The codebase for the forthcoming release is not stabilizing as fast as I
would like to, but this is mainly due to quite some time being spent on
version-specific bugs affecting only older versions, way too much time
being spent in code reviews, and context-switching 5 times more than usual.
That's not dramatic though, there's no emergency to release 2.2 and as
long as every day invested before the release saves more than this spent
troubleshooting after, the balance is positive.

On the good side, the code is cleaning up, and some low hanging fruits
were addressed for good by assembling many of the pieces that were ready.
Among the noticeable changes since dev7, I can see:

  - SSL code cleanup: the ssl_sock.c file really got huge over time, now it
    was split over 5 different files. Time to train our fingers again :-)
    The code was also adjusted so that we don't need to register outdated
    callbacks like the one dealing with the old heartbleed attack on modern
    TLS libs. Finally we switch default-dh-param to 2048 to get rid of the
    annoying warning which was suggesting to set it above 1024.

  - checks: support of log-format expressions on the send/expect rules,
    which allows to build way more complex and dynamic check rules

  - the MySQL health check now defaults to post-4.1 format

  - the reported c/q/r time averages in the stats are now more accurate
    when the process starts because they are averaged over the number of
    recorded samples. This will essentially affect those who reload often
    and who monitor these values for anomalies (Marcin).

  - new "http-error" directive and unification with the "return", "deny"
    and "tarpit" rules. This means that it's now possible to handle a
    deny or a return exactly the same way by specifiying headers and body
    independently using raw text or log-format, and that all processing
    errors can now be dynamic. I know this used to be a very long awaited
    feature which will for example allow to define errorfile templates
    which embed a unique ID or at least be a bit more user-friendly.

  - config: invalid hex sequences are now fatal in the config file. In
    the past they used to all be reported, but if you write 100k of
    "\xZZ" you essentially get 100k errors and it takes a huge time
    (Tim).

  - contrib: the spoa_server example code was ported to support python 3
    without losing compatibility with python 2 (Gilchrist Dadaglo). This
    was essential as python 2 is not supported anymore and haproxy 2.2 is
    going to be an LTS version. If you use this server for your own
    processing, please give it a try to make sure it still works for you,
    or please report problems.

  - an option was added to revert to the old (bogus) behavior on the
    server-side proxy-protocol-v2 headers in health checks to help
    detect and work around non-compliant server implementations which
    fail on the LOCAL command.

  - soft-stop: there used to be a pause of 1-to-2 seconds when stopping
    in multi-thread because we used to rely on other threads' poll timeout
    to synchronize all of them. Now the signal is immediately broadcasted
    and all threads are instantly informed. This can significantly help
    those who reload often.

  - roughly two tens of bugs were fixed, which is not much and is a good
    indication that things are overall stabilizing.

In addition we merged than reverted a piece of code to create new ring
buffers for logs and traces in the global section. Having them as a
single directive was too limiting and problematic to deal with, instead
there will be a section, which should allow to even forward them to a
remote server. Emeric is currently trying to get this done before the
release, which will allow to support syslog over TCP and even possibly
other logging protocols later if needed. His initial work consisting in
splitting the log generation and transport was already done, so this can
be merged late without any incidence on existing features, and I'd really
like to get this done for 2.2 because likewise, it's been requested for a
very long time and we already have all the technical pieces, they "just"
need to be assembled.

William proposed to have a look at issue #534 to see if we can reasonably
support escapes on the CLI. The problem is that people who use maps/ACLs
indexed on a user-agent cannot update them because it's not possible to
enter a space. Since it can impact the compatibility with homegrown scripts
I'd rather have this done before the release.

I had some patches that I forgot to merge that allow to release memory
pools which contain too many objects after a traffic surge, and which
significantly reduce the memory usage in tests. The typical use case is
that when a server suddenly slows down (I/O or a VM), haproxy will see
a sudden increase in concurrent connections, and all these allocations
currently remain in the pools for later use. That was fine when the
pools used to only contain sessions or buffers but now we have a lot of
different objects and it's important not to keep too many unused ones.
During a test I saw my memory usage fall down from 938 to 69 MB, which
is quite appreciable. This is also particularly interesting for users who
reload often (more memory for the new process) or those who need a lot of
memory for dynamic stuff like stick-tables. So I'll merge them soon.

I wanted to reorganize the include files, as some will have noticed, I
could never start. At least now I know what I want to do and nobody's
supposed to significantly interfer with these parts anymore so I'll attack
this next week if everything goes well, so that backports of fixes from
2.3 to 2.2 can go smoothly.

I'm aware of a bug that William Dauchy reported at the end of issue #552
that really looks like a regression between 2.1 and 2.2 but I'm confident
we'll eventually nail it down since William's traces are generally quite
detailed, so that does not worry me at all.

I also noted that we still have to recheck all the docs. Last time I put
my noze in intro.txt I learned that haproxy didn't cache, so I fixed it,
but it's quite likely that a number of the other ones need to be updated
as well.

That's about all that's left on my notes here. Enough talking, let's test
it.

Please find the usual URLs below :
   Site index       : http://www.haproxy.org/
   Discourse        : http://discourse.haproxy.org/
   Slack channel    : https://slack.haproxy.org/
   Issue tracker    : https://github.com/haproxy/haproxy/issues
   Sources          : http://www.haproxy.org/download/2.2/src/
   Git repository   : http://git.haproxy.org/git/haproxy.git/
   Git Web browsing : http://git.haproxy.org/?p=haproxy.git
   Changelog        : http://www.haproxy.org/download/2.2/src/CHANGELOG
   Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/

Willy

PS: we should probably issue a 2.1 and a 2.0 next week since a number of
    annoying bugs were addressed there.

---
Complete changelog :
Adis Nezirovic (1):
      BUG/MEDIUM: lua: Fix dumping of stick table entries for STD_T_DICT

Aleksandar Lazi (1):
      DOC/MINOR: halog: Add long help info for ic flag

Christopher Faulet (49):
      MINOR: checks: Improve report of unexpected errors for expect rules
      MEDIUM: checks: Add matching on log-format string for expect rules
      DOC: Fix req.body and co documentation to be accurate
      MEDIUM: checks: Remove dedicated sample fetches and use response ones 
instead
      CLEANUP: checks: sort and rename tcpcheck_expect_type types
      MINOR: checks: Use dedicated actions to send log-format strings in send 
rules
      MINOR: checks: Simplify matching on HTTP headers in HTTP expect rules
      MINOR: checks/sample: Remove unnecessary tests on the sample session
      REGTEST: checks: Adapt SSL error message reported when connection is 
rejected
      MINOR: checks: Support log-format string to set the URI for HTTP send 
rules
      MINOR: checks: Support log-format string to set the body for HTTP send 
rules
      DOC: Be more explicit about configurable check ok/error/timeout status
      MINOR: checks: Make matching on HTTP headers for expect rules less obscure
      BUG/MINOR: config: Make use_backend and use-server post-parsing less 
obscur
      BUG/MEDIUM: checks: Subscribe to I/O events on an unfinished connect
      BUG/MINOR: checks: Don't subscribe to I/O events if it is already done
      BUG/MINOR: checks: Rely on next I/O oriented rule when waiting for a 
connection
      MINOR: checks: Don't try to send outgoing data if waiting to be able to 
send
      BUG/MEDIUM: checks: Subscribe to I/O events only if a mux was installed
      BUG/MINOR: sample/ssl: Fix digest converter for openssl < 1.1.0
      DOC: SPOE is no longer experimental
      BUG/MINOR: cache: Don't needlessly test "cache" keyword in 
parse_cache_flt()
      MINOR: config: Don't dump keywords if argument is NULL
      MEDIUM: checks: Make post-41 the default mode for mysql checks
      BUG/MAJOR: mux-fcgi: Stop sending loop if FCGI stream is blocked for any 
reason
      CLEANUP: http_ana: Remove unused TXN flags
      BUG/MINOR: http-rules: Mark http return rules as final
      MINOR: http-htx: Add http_reply type based on what is used for http 
return rules
      CLEANUP: http-htx: Rename http_error structure into http_error_msg
      MINOR: http-rules: Use http_reply structure for http return rules
      MINOR: http-htx: Use a dedicated function to release http_reply objects
      MINOR: http-htx: Use a dedicated function to parse http reply arguments
      MINOR: http-htx: Use a dedicated function to check http reply validity
      MINOR: http-ana: Use a dedicated function to send a response from an http 
reply
      MEDIUM: http-rules: Rely on http reply for http deny/tarpit rules
      MINOR: http-htx: Store default error messages in a global http reply array
      MINOR: http-htx: Store messages of an http-errors section in a http reply 
array
      MINOR: http-htx: Store errorloc/errorfile messages in http replies
      MINOR: proxy: Add references on http replies for proxy error messages
      MINOR: http-htx: Use http reply from the http-errors section
      MINOR: http-ana: Use a TXN flag to prevent after-response ruleset 
evaluation
      MEDIUM: http-ana: Use http replies for HTTP error messages
      CLEANUP: http-htx: Remove unused storage of error messages in buffers
      MINOR: htx: Add a function to copy a buffer in an HTX message
      CLEANUP: channel: Remove channel_htx_copy_msg() function
      MINOR: http-ana: Add a function to write an http reply in an HTX message
      MINOR: http-htx/proxy: Add http-error directive using http return syntax
      DOC: Fix "errorfile" description in the configuration manual
      BUG/MINOR: checks: Respect check-ssl param when a port or an addr is 
specified

Dragan Dosen (4):
      MEDIUM: ssl: allow to register callbacks for SSL/TLS protocol messages
      MEDIUM: ssl: split ssl_sock_msgcbk() and use a new callback mechanism
      MINOR: ssl: add a new function ssl_sock_get_ssl_object()
      MEDIUM: ssl: use ssl_sock_get_ssl_object() in fetchers where appropriate

Emeric Brun (4):
      BUG/MINOR: logs: prevent double line returns in some events.
      MEDIUM: sink: build header in sink_write for log formats
      MEDIUM: logs: buffer targets now rely on new sink_write
      MEDIUM: sink: add global statement to create a new ring (sink buffer)

Gilchrist Dadaglo (1):
      MAJOR: contrib: porting spoa_server to support python3

Ilya Shipitsin (8):
      CI: travis-ci: enable arm64 builds again
      CI: travis-ci: skip pcre2 on arm64 build
      CI: travis-ci: upgrade LibreSSL versions
      DOC: assorted typo fixes in the documentation
      CI: extend spellchecker whitelist
      CLEANUP: assorted typo fixes in the code and comments
      CLEANUP: acl: remove unused assignment
      CI: travis-ci: fix libslz download URL

Jerome Magnin (2):
      BUILD: select: only declare existing local labels to appease clang
      DOC: retry-on can only be used with mode http

Marcin Deranek (3):
      MINOR: stats: Prepare for more accurate moving averages
      MINOR: stats: Expose native cum_req metric for a server
      MEDIUM: stats: Enable more accurate moving average calculation for stats

Martin Tzvetanov Grigorov (1):
      CI: travis-ci: switch arm64 builds to use openssl from distro

Olivier Houchard (2):
      BUG/MEDIUM: streams: Remove SF_ADDR_SET if we're retrying due to L7 retry.
      BUG/MEDIUM: stream: Only allow L7 retries when using HTTP.

Patrick Gansterer (2):
      MINOR: sample: Move aes_gcm_dec implementation into sample.c
      MINOR: sample: Add digest and hmac converters

Tim Duesterhus (2):
      BUG/MINOR: cfgparse: Abort parsing the current line if an invalid \x 
sequence is encountered
      MINOR: cfgparse: Improve error message for invalid \x sequences

William Dauchy (3):
      BUG/MINOR: pollers: remove uneeded free in global init
      CLEANUP: select: enhance readability in init
      BUILD: ssl: include buffer common headers for ssl_sock_ctx

William Lallemand (21):
      MINOR: mworker: replace ha_alert by ha_warning when exiting successfuly
      REORG: ssl: move macros and structure definitions to ssl_sock.h
      CLEANUP: ssl: remove the shsess_* macros
      REORG: move the crt-list structures in their own .h
      REORG: ssl: move the ckch structures to types/ssl_ckch.h
      CLEANUP: ssl: add ckch prototypes in proto/ssl_ckch.h
      REORG: ssl: move crtlist functions to src/ssl_crtlist.c
      CLEANUP: ssl: avoid circular dependencies in ssl_crtlist.h
      REORG: ssl: move the ckch_store related functions to src/ssl_ckch.c
      REORG: ssl: move ckch_inst functions to src/ssl_ckch.c
      REORG: ssl: move the crt-list CLI functions in src/ssl_crtlist.c
      REORG: ssl: move the CLI 'cert' functions to src/ssl_ckch.c
      REORG: ssl: move ssl configuration to cfgparse-ssl.c
      MINOR: ssl: remove static keyword in some SSL utility functions
      REORG: ssl: move ssl_sock_ctx and fix cross-dependencies issues
      REORG: ssl: move sample fetches to src/ssl_sample.c
      REORG: ssl: move utility functions to src/ssl_utils.c
      DOC: ssl: update MAINTAINERS file
      BUILD: ssl: include errno.h in ssl_crtlist.c
      BUILD: ssl: fix build without OPENSSL_NO_ENGINE
      MINOR: ssl: split config and runtime variable for ssl-{min,max}-ver

Willy Tarreau (20):
      REGTESTS: make the http-check-send test require version 2.2
      BUG/MINOR: http-ana: fix NTLM response parsing again
      BUG/MEDIUM: http_ana: make the detection of NTLM variants safer
      MEDIUM: ssl: increase default-dh-param to 2048
      CI: travis-ci: extend the build time for SSL to 60 minutes
      CLEANUP: config: drop unused setting CONFIG_HAP_MEM_OPTIM
      CLEANUP: config: drop unused setting CONFIG_HAP_INLINE_FD_SET
      CLENAUP: config: move CONFIG_HAP_LOCKLESS_POOLS out of config.h
      CLEANUP: remove THREAD_LOCAL from config.h
      BUG/MINOR: pools: use %u not %d to report pool stats in "show pools"
      BUG/MINOR: soft-stop: always wake up waiting threads on stopping
      MINOR: soft-stop: let the first stopper only signal other threads
      MEDIUM: hpack: use a pool for the hpack table
      BUG/MEDIUM: ring: write-lock the ring while attaching/detaching
      MINOR: applet: adopt the wait list entry from the CLI
      MINOR: ring: make the applet code not depend on the CLI
      Revert "MEDIUM: sink: add global statement to create a new ring (sink 
buffer)"
      BUILD: hpack: make sure the hpack table can still be built standalone
      CONTRIB: hpack: make use of the simplified standalone HPACK API
      MINOR: connection: add pp2-never-send-local to support old PP2 behavior

---

Reply via email to