Hi, HAProxy 2.8.13 was released on 2024/12/12. It added 48 new commits after version 2.8.12.
A major bug were fixed on QUIC in this released. It was possible to provoke a crash if a ACK was received just before the frames retransmit. In that case, the packet build was not interrupted, leading to build an empty packet which should be ack-elicting. The crash itself was the result of a BUG_ON() which detected the issue. The way to deal with too many headers in received H2/H3 messages was fixed. In H2, the maximum number of headers allowed in HEADERS frames on sending path was lower than on receiving path. This could lead to report sending errors while the message was accepted. It could be confusing. In H3, the number of headers was tested before the decoding. However, pseudo headers and cookie headers consumed extra slots. So in practice, this lowered the maximum number of headers that could be received. To workaround these issues, The number of headers in received messages is now always tested after the decoding stage. In addition, unlike H1, the number of headers must be limited when H2/H3 messages are sent to comply to limitation imposed by the protocols. This limit was increased to support headers rewriting without issue. On QUIC, the alert message about the 'socket-owner connection' support was replaced by a diagnostic warning because there is an automatic fallback mechanism. Retransmit for empty STREAM frames with FIN flag set was not properly handled and could cause the transfer to freeze with the client waiting indefinitely for the FIN notification. And a freeze was possible because of early QUIC stream closure, before transmitting any data. These bugs were fixed. On the H2 multiplexer, on server side, it was possible to send RST_STREAM frame for streams with unassigned ID, so before the formatting of the HEADERS frame, because the session was aborted during the connection stage. It was an issue if this happened before the H2 PREFACE was sent because this prevent the servers to recognize it as a H2 connection, leading to an early connection closure. We now take care to not emit RST_STREAM frame in that case. In 2.4, it was decided to reject HTTP/1.1 protocol upgrade requests with a payload because it is incompatible with the H2 on server side. Indeed, such upgrade requests must be converted to CONNECT requests in H2. So no payload are supported. However, it remains valid in HTTP/1.1. So instead of rejecting it on client side, these requests are now accepted and properly handled when sent to a H1 server. They are only rejected when they are sent to a H2 server. No special care about H2C protocol upgrade were took. But this could be a security issue if accepted by a server because it could be possible for a client to bypass all filtering rules. To fix the issue, the Upgrade header is removed from the requests if "h2c" or "h2" token are found. The code was reviewed to never use now_ms as an expiration date for a task. It was an internal issue that could lead to unexpected behavior when now_ms variable wrapped and was exactly equal to 0. We now always make sure to apply an offsets to now_ms when an expiration data must be set. Four issues with the L7 retries were fixed. First, the server status was not adjusted at each retry, while it should be. Only the last connection attempt was considered. Then, the buffer used to save the request to be able to perform a L7 retry was released to early in some rare cases and the request could be lost. It is of course unexpected and this could lead to crash. The request state was not properly reset on L7 retry. The request channel flag stating some data were sent was not reset on retry. This could lead to consider a subsequent connection error as a L7 error while the request was never sent. In that case too, the request could be lost, leading to crash. Finally, the L7 retries could be ignored if a server abort was detected during the request forwarding when the request was already in DONE state. In that case, the server abort must be handled on the response analysis side to be able to properly handle the L7 retries. In logs, the server response time (%Tr) was erroneously reported as -1 when it was intercepted by HAProxy. -1 is reserved to the case where response headers were not fully received. An old issue with the watchdog was fixed. It was possible to consider a thread as stuck by error because it was flagged in the debug handler. The issue was really visible on the 3.1, but on older versions, it was possible to encounter it if a "show threads" command executed while the watchdog timer was about to trigger before going back to the scheduler. Now, only the watchdog is responsible to detect stuck threads. The reason was missing in H2 responses forwarded to H1 clients while it was stated in the configuration manual that HAProxy should add one that matched the status code. It is now fixed. Random crap reports or even crashes could be experienced during a "show profiling memory" because some pools are replaced during startup and pool pointers were not properly unreferenced. It is now fixed. The expiration date for the task responsible to clean up the server resolution status when outdated info were inherited from the state file was fixed. "hold.timeout" was initially used. But is was not accurate, especially because it could be set to a high value or 0. Now the expiration date is based on the resolver "resolve" and "retry" timeouts. In addition, when a resolver was woken up to process DNS resolutions, it was possible to trigger an infinite loop on the resolver's wait list because delayed resolutions were always reinserted at the end of this list. This led the watchdog to kill the process. By re-inserting them in front of the list fixed the issue. A weird issue was fixed about the epoll poller. Over the last two years, there were few reports about immediate closes spuriously happening on connections where network captures proved that the server had not closed at all (and sometimes even received the request and responded to it after HAProxy had closed). The logs shown that a successful connection was immediately reported on error after the request was sent. After investigations, it appeared that a EPOLLUP, or eventually a EPOLLRDHUP, can be reported by epool_wait() during the connect() but in sock_conn_check(), the connect() reports a success. So the connection was validated but the HUP was handled on the first receive and an error was reported. So, to workaround the issue, we have decided to remove FD_POLL_HUP flag on the FD during the connection establishment if FD_POLL_ERR is not reported too in sock_conn_check(). This way, the call to connect() is able to validate or reject the connection. At the end, if the HUP or RDHUP flags were valid, either connect() would report the error itself, or the next recv() would return 0 confirming the closure that the poller tried to report. EPOLL_RDHUP is only an optimization to save a syscall anyway, and this pattern is so rare that nobody will ever notice the extra call to recv(). Please note that at least one reporter confirmed that using poll() instead of epoll() also addressed the problem, so that can also be a temporary workaround for those discovering the problem without the ability to immediately upgrade. The SIGINT signal could be missed by HAProxy when it was started in background in a subshell. It is the root cause of some unexpected timeouts with Vtest scripts. To fix the issue, the default signal handler is registered for the SIGINT signal during init. HAPROXY_CLI and HAPROXY_MASTER_CLI could exposed the internal sockpairs which should be only used for the master CLI. These internal sockpairs are now always hidden. An issue with the zero-copy data forwarding of H1 requests waiting for a TUNNEL established was fixed. SE_FL_EOI flag was erroneously set on the client sedesc. Dequeuing process was refined to fix some bugs revealed by recent fixes in this area. Inter-thread stream shutdown, used by "shutdown sessions server XXX" CLI command or "on-error shutdown-sessions" server option, was not thread safe. To finish, some issues in the configuration manual were fixed, the documention of "%Tr" was improved and an explanation about quotes and spaces in conditional blocks was added. Thanks everyone for your help ! Please find the usual URLs below : Site index : https://www.haproxy.org/ Documentation : https://docs.haproxy.org/ Wiki : https://github.com/haproxy/wiki/wiki Discourse : https://discourse.haproxy.org/ Slack channel : https://slack.haproxy.org/ Issue tracker : https://github.com/haproxy/haproxy/issues Sources : https://www.haproxy.org/download/2.8/src/ Git repository : https://git.haproxy.org/git/haproxy-2.8.git/ Git Web browsing : https://git.haproxy.org/?p=haproxy-2.8.git Changelog : https://www.haproxy.org/download/2.8/src/CHANGELOG Dataplane API : https://github.com/haproxytech/dataplaneapi/releases/latest Pending bugs : https://www.haproxy.org/l/pending-bugs Reviewed bugs : https://www.haproxy.org/l/reviewed-bugs Code reports : https://www.haproxy.org/l/code-reports Latest builds : https://www.haproxy.org/l/dev-packages --- Complete changelog : Amaury Denoyelle (7): BUG/MINOR: h1: do not forward h2c upgrade header token BUG/MINOR: h2: reject extended connect for h2c protocol MINOR: quic: convert qc_stream_desc release field to flags MINOR: quic: implement function to check if STREAM is fully acked BUG/MEDIUM: quic: handle retransmit for standalone FIN STREAM BUG/MINOR: quic: prevent freeze after early QCS closure BUG/MINOR: quic: remove startup alert if conn socket-owner unsupported Aurelien DARRAGON (2): DOC: lua: fix yield-dependent methods expected contexts BUG/MEDIUM: event_hdl: fix uninitialized value in async mode when no data is provided Christopher Faulet (21): MINOR: mux-h1: Set EOI on SE during demux when both side are in DONE state BUG/MEDIUM: mux-h1/mux-h2: Reject upgrades with payload on H2 side only REGTESTS: h1/h2: Update script testing H1/H2 protocol upgrades BUG/MINOR: http-ana: Disable fast-fwd for unfinished req waiting for upgrade BUG/MEDIUM: resolvers: Insert a non-executed resulution in front of the wait list BUG/MEDIUM: mux-h2: Don't send RST_STREAM frame for streams with no ID BUG/MINOR: Don't report early srv aborts on request forwarding in DONE state BUG/MINOR: http_ana: Report -1 for %Tr for invalid response only DOC: config: Slightly improve the %Tr documentation DOC: config: Move wait_end in section about internal samples BUG/MINOR: http-ana: Adjust the server status before the L7 retries BUG/MEDIUM: mux-h2: Increase max number of headers when encoding HEADERS frames BUG/MEDIUM: mux-h2: Check the number of headers in HEADERS frame after decoding BUG/MEDIUM: h3: Properly limit the number of headers received BUG/MEDIUM: h3: Increase max number of headers when sending headers DEV: lags/show-sess-to-flags: Properly handle fd state on server side BUG/MEDIUM: http-ana: Don't release too early the L7 buffer BUG/MEDIUM: sock: Remove FD_POLL_HUP during connect() if FD_POLL_ERR is not set BUG/MEDIUM: http-ana: Reset request flag about data sent to perform a L7 retry BUG/MINOR: h1-htx: Use default reason if not set when formatting the response BUG/MINOR: server-state: Fix expiration date of srvrq_check tasks Frederic Lecaille (1): BUG/MAJOR: quic: fix wrong packet building due to already acked frames Valentine Krasnobaeva (2): BUG/MINOR: cli: don't show sockpairs in HAPROXY_CLI and HAPROXY_MASTER_CLI BUG/MINOR: signal: register default handler for SIGINT in signal_init() Willy Tarreau (15): BUG/MINOR: ssl_sock: fix xprt_set_used() to properly clear the TASK_F_USR1 bit REGTESTS: shorten a bit the delay for the h1/h2 upgrade test MINOR: task: define two new one-shot events for use with WOKEN_OTHER or MSG BUG/MEDIUM: stream: make stream_shutdown() async-safe BUG/MEDIUM: queue: always dequeue the backend when redistributing the last server BUG/MEDIUM: queue: make sure never to queue when there's no more served conns BUG/MEDIUM: checks: make sure to always apply offsets to now_ms in expiration BUG/MEDIUM: mailers: make sure to always apply offsets to now_ms in expiration BUG/MINOR: mux_quic: make sure to always apply offsets to now_ms in expiration BUG/MINOR: peers: make sure to always apply offsets to now_ms in expiration DOC: configuration: explain quotes and spaces in conditional blocks DOC: configuration: wrap long line for "strstr()" conditional expression BUG/MEDIUM: debug: don't set the STUCK flag from debug_handler() MINOR: activity/memprofile: offer a function to unregister stale info BUG/MEDIUM: pools/memprofile: always clean stale pool info on pool_destroy() -- Christopher Faulet
