Hi Willy, > - the H2 mux had an incorrect buffer full detection causing the send > phase to stop on a fragment boundary then to immediately wake up all > waiting threads to go on, resulting in an excessive CPU usage in some > tricky situations. It is possible that those using H2 with many streams > per connection and moderately large objects, like Luke's maps servers, > could observe a CPU usage drop (maybe Luke on his map servers).
We just deployed 2.1.3 across our PoP network last night, and I can indeed verify we're seeing better CPU usage—anywhere from 40-50% aggregate reduction! Once we have a few more days of data, I'll send a pretty chart so you can enjoy the fruits of your hard work. Best, Luke — Luke Seelenbinder Stadia Maps | Founder stadiamaps.com > On 12 Feb 2020, at 17:44, Willy Tarreau <w...@1wt.eu> wrote: > > Hi, > > HAProxy 2.1.3 was released on 2020/02/12. It added 86 new commits > after version 2.1.2. > > It's clear that 2.1 has been one of the calmest releases in a while, to > the point of making us forget that it still had a few fixes pending that > would be pleasant to have in a released version! So after accumulating > fixes for 7 weeks, it's about time to have another one! > > Here are the most relevant fixes: > > - pools: there is an ABA race condition in pool_flush() (which is called > when stopping as well as under memory pressure) which can lead to a > crash. It's been there since 1.9 and is very hard to trigger, but if > you run with many threads and reload very often you may occasionally > hit it, seeing a trace of the old process crashing in your system > logs. > > - there was a bug in the way our various hashes were calculated, some > of them were considering the inputs as signed chars instead of > unsigned ones, so some non-ASCII characters would hash differently > across different architectures and wouldn't match another component's > calculation (e.g. a CRC32 inserted in a header would differ when given > values with the 8th bit set, or applied to the PROXY protocol header). > The bug has been there since 1.5-dev20 but became visible since it > affected Postfix's validation of the PROXY protocol's CRC32. It's > unlikely that anyone will ever witness it if it didn't happen already, > but I tagged it "major" to make sure it is properly backported to > distro packages, since not having it on certain nodes may sometimes > result in hash inconsistencies which can be very hard to diagnose. > > - the addition of the Early-Data header when using 0rtt could wrongly > be emitted during SSL handshake as well. > > - health checks could crash if using handshakes (e.g. SSL) mixed with > DNS that takes time to retrieve an address, causing an attempt to > use an incompletely initialized connection. > > - the peers listening socket was missing from the seamless reload, > possibly causing some failed bindings when not using reuseport, > resulting in the new process giving up. > > - splicing could often end up on a timeout because after the last block > we did not switch back to HTX to complete the message. > > - fixed a small race affecting idle connections, allowing one thread to > pick a connection at the same moment another one would decide to free > it because there are too many idle. > > - response redirects were appended to the actual response instead of > replacing it. This could cause various errors, including data > corruption on the client if the entire response didn't fit into the > buffer at once. > > - when stopping or when releasing a few connections after a listener's > maxconn was reached, we could distribute some work to inexistent > threads if the listener had "1/odd" or "1/even" while the process > had less than 64 threads. An easy workaround for this is to explicitly > reference the thread numbers instead. > > - when proxying an HTTP/1 client to an HTTP/2 server, make sure to clean > up the "TE" header from anything but "trailers", otherwise the server > may reject a request if it came from a browser placing "gzip" there. > > - the H2 mux had an incorrect buffer full detection causing the send > phase to stop on a fragment boundary then to immediately wake up all > waiting threads to go on, resulting in an excessive CPU usage in some > tricky situations. It is possible that those using H2 with many streams > per connection and moderately large objects, like Luke's maps servers, > could observe a CPU usage drop (maybe Luke on his map servers). > > - it was possible to lose the master-worker status after a failed reload > when it was only mentioned in the config and not on the command line. > > - when decoding the Netscaler's CIP protocol we forgot to allocate the > storage for the src/dst addresses, crashing the process. > > - upon pipe creation failure due to shortage of file descriptors, the > struct pipe was still returned after having been released, quickly > crashing the process. Fortunately the automatic maxconn/maxpipe > settings do not allow this situation to happen but very old configs > still having "ulimit-n" could have been affected. > > - the "tcp-request session" rules would report an error upon a "reject" > action, making the listener throttle itself to protect resources, > which could actually amplify the problem. > > - the "commit ssl cert" command on the CLI used the old SSL_CTX instead > of the new one, which caused some certs not to work anymore (found on > openssl-1.0.2 with ECDSA+ECDHE). There is quite a number of other SSL > SSL fixes for small bugs that were found while troubleshooting this > issue, mainly in relation with dynamic cert updates. > > - the H1 mux could attempt to perform a sendto() when facing new data > after having already failed, resulting in excess calls to sendto(). > > The rest has less impact or is less likely to be noticed, but feel free > to dig into the appended changelog. > > I'm hearing 2.0 jealously complain that it's always his youngest brother > that's served first and that it would like to get a release as well, so > maybe I'll please the rest of the family this week, as the other stable > co-maintainers look busy right now (or they play it well :-)). > > Speaking of this, for those currently testing 2.2-dev2, please make sure > to be up to date, as I was hit by two dirty crashes after deploying it on > haproxy.org this week-end, that were both addressed in the master branch. > > Please find the usual URLs below : > Site index : http://www.haproxy.org/ > Discourse : http://discourse.haproxy.org/ > Slack channel : https://slack.haproxy.org/ > Issue tracker : https://github.com/haproxy/haproxy/issues > Sources : http://www.haproxy.org/download/2.1/src/ > Git repository : http://git.haproxy.org/git/haproxy-2.1.git/ > Git Web browsing : http://git.haproxy.org/?p=haproxy-2.1.git > Changelog : http://www.haproxy.org/download/2.1/src/CHANGELOG > Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ > > Willy > --- > Complete changelog : > Baptiste Assmann (1): > BUG/MINOR: http_act: don't check capture id in backend > > Ben51Degrees (1): > BUG/MINOR: 51d: Fix bug when HTX is enabled > > Christian Lachner (1): > MINOR: build: add aix72-gcc build TARGET and power{8,9} CPUs > > Christopher Faulet (13): > BUG/MINOR: h1: Report the right error position when a header value is > invalid > BUG/MINOR: proxy: Fix input data copy when an error is captured > BUG/MINOR: channel: inject output data at the end of output > BUG/MINOR: stream-int: Don't trigger L7 retry if max retries is already > reached > BUG/MINOR: http-ana/filters: Wait end of the http_end callback for all > filters > BUG/MINOR: http-rules: Remove buggy deinit functions for HTTP rules > BUG/MINOR: stick-table: Use MAX_SESS_STKCTR as the max track ID during > parsing > BUG/MINOR: tcp-rules: Fix memory releases on error path during action > parsing > MINOR: proxy/http-ana: Add support of extra attributes for the cookie > directive > MINOR: htx: Add a function to append an HTX message to another one > MINOR: htx/channel: Add a function to copy an HTX message in a channel's > buffer > BUG/MINOR: http-ana: Don't overwrite outgoing data when an error is > reported > BUG/MINOR: http-ana: Set HTX_FL_PROXY_RESP flag if a server perform a > redirect > > Emmanuel Hocdet (5): > BUG/MINOR: ssl: ssl_sock_load_ocsp_response_from_file memory leak > BUG/MINOR: ssl: ssl_sock_load_issuer_file_into_ckch memory leak > BUG/MINOR: ssl: ssl_sock_load_sctl_from_file memory leak > BUG/MINOR: ssl: ssl_sock_load_pem_into_ckch is not consistent > BUG/MINOR: ssl/cli: ocsp_issuer must be set w/ "set ssl cert" > > Frédéric Lécaille (1): > BUG/MINOR: ssl: Possible memleak when allowing the 0RTT data buffer. > > Jerome Magnin (4): > BUG/MINOR: stream: don't mistake match rules for store-request rules > BUG/MINOR: pattern: handle errors from fgets when trying to load patterns > BUILD: pattern: include errno.h > DOC: word converter ignores delimiters at the start or end of input > string > > Kevin Zhu (1): > BUG/MEDIUM: http-ana: Truncate the response when a redirect rule is > applied > > Olivier Houchard (13): > BUG/MEDIUM: checks: Only attempt to do handshakes if the connection is > ready. > BUG/MEDIUM: connections: Hold the lock when wanting to kill a connection. > MINOR: ssl: Remove unused variable "need_out". > BUG/MEDIUM: tasks: Use the MT macros in tasklet_free(). > BUG/MEDIUM: mux_h1: Don't call h1_send if we subscribed(). > BUG/MEDIUM: netscaler: Don't forget to allocate storage for > conn->src/dst. > BUG/MEDIUM: 0rtt: Only consider the SSL handshake. > BUG/MEDIUM: ssl: Don't forget to free ctx->ssl on failure. > BUG/MEDIUM: connections: Don't forget to unlock when killing a > connection. > BUG/MEDIUM: memory_pool: Update the seq number in pool_flush(). > MINOR: memory: Only init the pool spinlock once. > BUG/MEDIUM: memory: Add a rwlock before freeing memory. > BUG/MAJOR: memory: Don't forget to unlock the rwlock if the pool is > empty. > > Tim Duesterhus (6): > BUG/MINOR: cache: Fix leak of cache name in error path > BUG/MINOR: dns: Make dns_query_id_seed unsigned > MINOR: lua: Add hlua_prepend_path function > MINOR: lua: Add lua-prepend-path configuration option > MINOR: lua: Add HLUA_PREPEND_C?PATH build option > MINOR: acl: Warn when an ACL is named 'or' > > William Dauchy (5): > MINOR: config: disable busy polling on old processes > BUG/MINOR: connection: fix ip6 dst_port copy in make_proxy_line_v2 > BUG/MINOR: dns: allow 63 char in hostname > BUG/MINOR: tcp: avoid closing fd when socket failed in tcp_bind_listener > BUG/MINOR: tcp: don't try to set defaultmss when value is negative > > William Lallemand (11): > BUG/MINOR: cli/mworker: can't start haproxy with 2 programs > REGTEST: mcli/mcli_start_progs: start 2 programs > BUG/MEDIUM: mworker: remain in mworker mode during reload > BUG/MEDIUM: cli: _getsocks must send the peers sockets > BUG/MINOR: ssl/cli: free the previous ckch content once a PEM is loaded > BUG/MINOR: ssl: increment issuer refcount if in chain > BUG/MINOR: ssl: memory leak w/ the ocsp_issuer > BUG/MINOR: ssl: typo in previous patch > BUG/MINOR: ssl: clear the SSL errors on DH loading failure > BUG/MEDIUM: ssl/cli: 'commit ssl cert' wrong SSL_CTX init > DOC: schematic of the SSL certificates architecture > > Willy Tarreau (24): > BUG/MINOR: checks: refine which errno values are really errors. > BUG/MEDIUM: session: do not report a failure when rejecting a session > BUG/MINOR: mux-h2: use a safe list_for_each_entry in h2_send() > BUG/MEDIUM: mux-h2: fix missing test on sending_list in previous patch > BUG/MEDIUM: mux-h2: don't stop sending when crossing a buffer boundary > BUG/MAJOR: hashes: fix the signedness of the hash inputs > REGTEST: add sample_fetches/hashes.vtc to validate hashes > BUG/MEDIUM: connection: add a mux flag to indicate splice usability > CLEANUP: stats: shut up a wrong null-deref warning from gcc 9.2 > BUG/MINOR: stktable: report the current proxy name in error messages > BUG/MEDIUM: mux-h2: make sure we don't emit TE headers with anything but > "trailers" > BUILD: cfgparse: silence a bogus gcc warning on 32-bit machines > BUG/MINOR: tcpchecks: fix the connect() flags regarding delayed ack > BUG/MEDIUM: pipe: fix a use-after-free in case of pipe creation error > BUG/MINOR: ssl: we may only ignore the first 64 errors > CONTRIB: debug: add missing flags SF_HTX and SF_MUX > CONTRIB: debug: add the possibility to decode the value as certain types > only > CONTRIB: debug: support reporting multiple values at once > CONTRIB: debug: also support reading values from stdin > SCRIPTS: announce-release: place the send command in the mail's header > SCRIPTS: announce-release: allow the user to force to overwrite old files > BUG/MINOR: unix: better catch situations where the unix socket path > length is close to the limit > BUG/MEDIUM: listener: only consider running threads when resuming > listeners > BUG/MINOR: listener: enforce all_threads_mask on bind_thread on init > > --- >
