Hi, HAProxy 1.9.7 was released on 2019/04/25. It added 100 new commits after version 1.9.6.
After a month since the last release, a huge number of bugs were addressed into this release. The most significant are fixes of 100% CPU usage issues. Olivier and Willy worked hard on it. They fixed several bugs in the task scheduler and improved it. Another issue concerned the streams and the way aborts was handled. The flag reporting the abort on the stream-interface was cleared too early. So the stream was not properly released, looping infinitely on itself. Thanks to Maciej for helping Olivier investigating on this issue. Finally, two bugs leading to a loop were fixed in the H2 multiplexer. The first one occurred when an incomplete frame was received and no more data was available. The second one when the subscription to the send events was made twice, which should never happen. This last issue was also able to lead to random crashes. Willy, with the help of Maksim, fixed a locking problem in the round-robin LB causing occasional crashes of HAProxy because of a divide by zero when the next server's position was computed. Willy also fixed an AB/BA locking issue in the listener's accept code thanks to Olivier and his self-locked list. At the end, all this part was reworked. The lock on the listener queue disappeared and the accept code was significantly improved. As mentioned, Olivier implemented self-locked lists. It is now safe to manipulate such lists with multiple threads without having to worry about concurrency issues. A very old bug on how HTTP sample fetches work was fixed. All HTTP sample fetches were buggy because the channel used was chosen depending on the sample direction and not on the keyword really used. The request channel was used when called during the request analysis and the response one was used when called during the response analysis, regardless the sample really called. It could cause a whole bunch of bugs, from undefined behavior because the data were extracted from the wrong buffer to crash of HAProxy. Many bugs in the HTX part were fixed. The most significant was a problem of data corruption when an header was added while some data were already received. It was addressed by defragmenting the HTX message. It is a quick and dirty fix, waiting for a better solution. The TCP splicing was fixed in the H1 multiplexer. The handling of the CONNECT method was refactored to work when h1 and h2 are mixed (an h1 client sending requests to an h2 server or this opposite). And finally, the option "http-tunnel" was deprecated in HTX. It is still supported in legacy HTTP, but in HTX, this options is now ignored. Otherwise, as always, there was a bunch of bugfixes and changes here and there. The most noticeable are: - A memory leak was fixed in the map converter. And now when a map file is loaded, the default value is parsed only when it is present. This fixes segfaults at parsing time when no default value is provided. - Pattern IDs are now assigned after checking the config validity. It fixes a bug where some map identifiers were not assigned (appearing as -1 in show map). Thanks to Pavlos to report this bug. - Several bugs was fixed on the SPOE, mainly on the way fragmented frames was handled internally. - A bug was fixed in the peers. Peer sessions were not always cleanly reset on release, resulting in a bad state for new sessions. In addition, the CLI command "show peers" was added. It dumps info about the peers configured in "peers" sections. - Nenad fixed a security issue in the TLS ticket key rotation. Whenever HAProxy was reloaded with rotated keys, the resumption would be broken for previous encryption key. CVE-2019-11323 was assigned to this bug. - Build of HAProxy on AIX 5.1 was fixed. - Some small improvements were brought to the master-worker mode. The busy polling was disabled on the master process to not waste a dedicated processor on it. And an unambiguous status is now set on exit. - The global directive "set-dumpable" was added to enable core dumps. It's always a pain to get a core dump when enabling user/group setting (which disables the dumpable flag on Linux), when using a chroot and/or when HAProxy is started by a service management tool which requires complex operations to just raise the core dump limit. It becomes much easier to retrieve a core file by using this new directive. Temporarily moving the chroot to a user-writable place generally enough. - Missing locks was added in set-map and add-acl HTTP rules. - The multithreading mode is now enabled on wurlf. And it is now compatible with the new buffer API, so it builds again on 1.9. WARNING: Device detection modules (51Degrees, DeviceAtlas and Wurfl) are not compatible with the HTX mode. Respective maintainers works on it. But for this release, don't enable the HTX if you need to use one of them. There is still an unresolved issue when the SSL is heavily used with multiple threads. It is unclear for now what the problem is, but an extra latency has been observed by a user. After digging, we suspect a problem with OpenSSL. You can have more details on this issue on Github (issue #67). Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse : http://discourse.haproxy.org/ Slack channel : https://slack.haproxy.org/ Issue tracker : https://github.com/haproxy/haproxy/issues Sources : http://www.haproxy.org/download/1.9/src/ Git repository : http://git.haproxy.org/git/haproxy-1.9.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-1.9.git Changelog : http://www.haproxy.org/download/1.9/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ --- Complete changelog : Christopher Faulet (28): BUG/MINOR: htx: Preserve empty HTX messages with an unprocessed parsing error BUG/MEDIUM: spoe: Queue message only if no SPOE applet is attached to the stream BUG/MEDIUM: spoe: Return an error if nothing is encoded for fragmented messages BUG/MINOR: spoe: Be sure to set tv_request when each message fragment is encoded BUG/MEDIUM: htx: Defrag if blocks position is changed and the payloads wrap BUG/MEDIUM: htx: Don't crush blocks payload when append is done on a data block MEDIUM: htx: Deprecate the option 'http-tunnel' and ignore it in HTX MINOR: proto_htx: Don't adjust transaction mode anymore in HTX analyzers BUG/MEDIUM: htx: Fix the process of HTTP CONNECT with h2 connections MINOR: mux-h1: Simplify handling of 1xx responses BUG/MINOR: mux-h1: Handle the flag CS_FL_KILL_CONN during a shutdown read/write BUG/MINOR: mux-h1: Process input even if the input buffer is empty BUG/MINOR: mux-h1: Don't switch the parser in busy mode if other side has done BUG/MEDIUM: mux-h1: Notify the stream waiting for TCP splicing if ibuf is empty BUG/MEDIUM: mux-h1: Enable TCP splicing to exchange data only MINOR: mux-h1: Handle read0 during TCP splicing BUG/MEDIUM: htx: Don't return the start-line if the HTX message is empty BUG/MAJOR: http_fetch: Get the channel depending on the keyword used BUG/MINOR: http_fetch/htx: Allow permissive sample prefetch for the HTX BUG/MEDIUM: h1: Don't parse chunks CRLF if not enough data are available BUG/MEDIUM: thread/http: Add missing locks in set-map and add-acl HTTP rules BUG/MEDIUM: stream: Don't request a server connection if a shutw was scheduled BUG/MINOR: 51d: Get the request channel to call CHECK_HTTP_MESSAGE_FIRST() BUG/MINOR: da: Get the request channel to call CHECK_HTTP_MESSAGE_FIRST() BUG/MEDIUM: stream: Fix the way early aborts on the client side are handled BUG/MINOR: spoe: Don't systematically wakeup SPOE stream in the applet handler BUG/MAJOR: muxes: Use the HTX mode to find the best mux for HTTP proxies only BUG/MINOR: htx: Exclude TCP proxies when the HTX mode is handled during startup David Carlier (1): BUILD/MINOR: listener: Silent a few signedness warnings. Emeric Brun (3): BUG/MEDIUM: peers: fix a case where peer session is not cleanly reset on release. MINOR: peers: adds counters on show peers about tasks calls. MINOR: ssl/cli: async fd io-handlers printable on show fd Frédéric Lécaille (2): MINOR: peers: Add a new command to the CLI for peers. DOC: update for "show peers" CLI command. Ilya Shipitsin (1): REGTESTS: exclude tests that require ssl, pcre if no such feature is enabled Nenad Merdanovic (2): BUG/MEDIUM: map: Fix memory leak in the map converter BUG/MINOR: ssl: Fix 48 byte TLS ticket key rotation Olivier Houchard (15): BUG/MEDIUM: h2: Don't attempt to recv from h2_process_demux if we subscribed. BUG/MEDIUM: streams: Don't remove the SI_FL_ERR flag in si_update_both(). BUG/MEDIUM: streams: Store prev_state before calling si_update_both(). BUG/MEDIUM: stream: Don't clear the stream_interface flags in si_update_both. BUG/MEDIUM: streams: Only re-run process_stream if we're in a connected state. BUG/MEDIUM: stream_interface: Don't bother doing chk_rcv/snd if not connected. MINOR: lists: Implement locked variations. BUG/MEDIUM: lists: Properly handle the case we're removing the first elt. MINOR: initcall: Don't forget to define the __start/stop_init_##stg symbols. BUG/MEDIUM: h2: Make sure we're not already in the send_list in h2_subscribe(). BUG/MEDIUM: tasks: Make sure we modify global_tasks_mask with the rq_lock. MINOR: tasks: Don't consider we can wake task with tasklet_wakeup(). MEDIUM: tasks: No longer use rq.node.leaf_p as a lock. MINOR: tasks: Don't set the TASK_RUNNING flag when adding in the tasklet list. BUG/MEDIUM: applets: Don't use task_in_rq(). Robin H. Johnson (1): MINOR: skip get_gmtime where tm is unused William Lallemand (4): BUG/MINOR: cli: correctly handle abns in 'show cli sockets' MINOR: cli: start addresses by a prefix in 'show cli sockets' BUG/MINOR: mworker: don't exit with an ambiguous value BUG/MINOR: mworker: ensure that we still quits with SIGINT Willy Tarreau (40): BUILD: makefile: work around an old bug in GNU make-3.80 BUILD: http: properly mark some struct as extern BUILD: chunk: properly declare pool_head_trash as extern BUILD: cache: avoid a build warning with some compilers/linkers MINOR: tools: make memvprintf() never pass a NULL target to vsnprintf() BUILD: re-implement an initcall variant without using executable sections BUILD: makefile: fix build of IPv6 header on aix51 BUILD: makefile: add _LINUX_SOURCE_COMPAT to build on AIX-51 BUILD: Makefile: disable shared cache on AIX 5.1 BUILD: fix backport of initcall variant BUILD: use inttypes.h instead of stdint.h BUILD: connection: fix naming of ip_v field BUG/MEDIUM: htx: fix random premature abort of data transfers BUG/MEDIUM: pattern: assign pattern IDs after checking the config validity BUG/MEDIUM: task/threads: address a fairness issue between local and global tasks BUG/MINOR: tasks: make sure the first task to be queued keeps its nice value BUG/MINOR: threads: fix the process range of thread masks BUG/MEDIUM: list: fix the rollback on addq in the locked liss BUG/MEDIUM: list: fix LIST_POP_LOCKED's removal of the last pointer BUG/MEDIUM: list: add missing store barriers when updating elements and head MINOR: list: make the delete and pop operations idempotent BUG/MEDIUM: list: correct fix for LIST_POP_LOCKED's removal of last element BUG/MEDIUM: list: fix again LIST_ADDQ_LOCKED BUG/MEDIUM: list: fix incorrect pointer unlocking in LIST_DEL_LOCKED() MAJOR: listener: do not hold the listener lock in listener_accept() BUG/MEDIUM: listener: use a self-locked list for the dequeue lists BUG/MEDIUM: listener: make sure the listener never accepts too many conns BUILD: htx: fix a used uninitialized warning on is_cookie2 BUG/MAJOR: lb/threads: fix insufficient locking on round-robin LB MEDIUM: tasks: improve fairness between the local and global queues BUILD: task/thread: fix single-threaded build of task.c MEDIUM: tasks: only base the nice offset on the run queue depth MINOR: tasks: restore the lower latency scheduling when niced tasks are present BUG/MEDIUM: tasks: Make sure we set TASK_QUEUED before adding a task to the rq. BUG/MAJOR: task: make sure never to delete a queued task BUG/MINOR: mworker: disable busy polling in the master process BUG/MEDIUM: maps: only try to parse the default value when it's present BUG/MINOR: acl: properly detect pattern type SMP_T_ADDR MINOR: init: add a "set-dumpable" global directive to enable core dumps BUG/MAJOR: lb/threads: fix AB/BA locking issue in round-robin LB paulborile (3): BUILD: wurfl: build fix for 1.9/2.0 code base MINOR: wurfl: enabled multithreading mode DOC: wurfl: added point of contact in MAINTAINERS file -- Christopher Faulet