Re: Maybe stupid question but, I don't see a fetch method for %rt => StreamID
Le 6/1/21 à 8:26 PM, Aleksandar Lazic a écrit : On 01.06.21 14:23, Tim Düsterhus wrote: Aleks, On 6/1/21 10:30 AM, Aleksandar Lazic wrote: This phrasing is understandable to me, but now I'm wondering if this is the best solution. Maybe the already existing user-configurable unique request ID should instead be sent to the SPOE and then logged? https://cbonte.github.io/haproxy-dconv/2.2/configuration.html#7.3.6-unique-id The request_counter (%rt) you mentioned could be embedded into this unique-id. Well this uniqe-id is not send as Stream ID to SPOA receiver, due to this fact can't you debug which stream is the troubled one. Yes, that's why I suggested that the SPOE is extended to also include this specific ID somewhere (just) for logging purposes. Yep. Any opinion from the other community Members? The SID provided in the SPOE log message is the one used in the SPOP frame header. This way it is possible to match a corresponding log message emitted by the agent. Regarding the format for this log message, its original purpose was to diagnose problems. Instead of adding custom information, I guess the best would be to have a "log-format" directive. At least to not break existing tools parsing those log messages. But to do so, all part of the current message must be available via log variables and/or sample fetches. And, at first glance, it will be hard to achieve (sample fetches are probably easier though). Regarding the stream_uniq_id sample fetch, it is a good idea to add it. In fact, when it makes sense, a log variable must also be accessible via a sample fetch. Tim's remarks about the patch are valid. For the scope, INTRN or L4CLI, I don't know. I'm inclined to choose INTRN. -- Christopher Faulet
Re: [PATCH] DOC/MINOR: move uuid in the configuration to the right, alphabetical order
Le 6/1/21 à 12:35 AM, Aleksandar Lazic a écrit : Fix alphabetical order of uuid merged now, thanks ! -- Christopher Faulet
Re: Weird behavior of spoe between http and https requests
16:01:01 reggata-001 spoe-url[112969]: 2021/06/11 16:01:01 Msg Count :5: Jun 11 16:01:01 reggata-001 spoe-url[112969]: 2021/06/11 16:01:01 Arg Name :my_path: Jun 11 16:01:01 reggata-001 spoe-url[112969]: 2021/06/11 16:01:01 Arg Value :/test: Jun 11 16:01:01 reggata-001 spoe-url[112969]: 2021/06/11 16:01:01 Arg Name :my_src: Jun 11 16:01:01 reggata-001 spoe-url[112969]: 2021/06/11 16:01:01 Arg Value :: Jun 11 16:01:01 reggata-001 spoe-url[112969]: 2021/06/11 16:01:01 Arg Name :my_referer: Jun 11 16:01:01 reggata-001 spoe-url[112969]: 2021/06/11 16:01:01 Arg Value :%!s(): Jun 11 16:01:01 reggata-001 spoe-url[112969]: 2021/06/11 16:01:01 Arg Name :my_sid: Jun 11 16:01:01 reggata-001 spoe-url[112969]: 2021/06/11 16:01:01 Arg Value :11: Jun 11 16:01:01 reggata-001 spoe-url[112969]: 2021/06/11 16:01:01 Arg Name :my_req_host: Jun 11 16:01:01 reggata-001 spoe-url[112969]: 2021/06/11 16:01:01 Arg Value :: ``` But when I make a https request I get only the path. ``` Jun 11 15:55:32 reggata-001 spoe-url[112869]: 2021/06/11 15:55:32 Msg Name :agent-on-http-req: Jun 11 15:55:32 reggata-001 spoe-url[112869]: 2021/06/11 15:55:32 Msg Count :5: Jun 11 15:55:32 reggata-001 spoe-url[112869]: 2021/06/11 15:55:32 Arg Name :my_path: Jun 11 15:55:32 reggata-001 spoe-url[112869]: 2021/06/11 15:55:32 Arg Value :/test: Jun 11 15:55:32 reggata-001 spoe-url[112869]: 2021/06/11 15:55:32 Arg Name :my_src: Jun 11 15:55:32 reggata-001 spoe-url[112869]: 2021/06/11 15:55:32 Arg Value :0.0.0.0: Jun 11 15:55:32 reggata-001 spoe-url[112869]: 2021/06/11 15:55:32 Arg Name :: Jun 11 15:55:32 reggata-001 spoe-url[112869]: 2021/06/11 15:55:32 Arg Value :%!s(): Jun 11 15:55:32 reggata-001 spoe-url[112869]: 2021/06/11 15:55:32 Arg Name :: Jun 11 15:55:32 reggata-001 spoe-url[112869]: 2021/06/11 15:55:32 Arg Value :%!s(): Jun 11 15:55:32 reggata-001 spoe-url[112869]: 2021/06/11 15:55:32 Arg Name :: Jun 11 15:55:32 reggata-001 spoe-url[112869]: 2021/06/11 15:55:32 Arg Value :%!s(): ``` Please can somebody tell me what's my mistake, thank you? The problem can be easily reproduces when the bind lines is replaces with '::' Working: *:80 Not Working: :::80 Then works also the HTTPS part. It looks like that '*:80' goes different Way then ':::80' Hi Alex, I'm unable to reproduce the issue. Everything works as expected, with all combinations of HTTP/HTTPS and IPv4/IPv6. It may be an issue with your agent. "my_src" value is displayed as an IPv4 while it should be an IPv6. Could you check your agent is properly decoding IPv6 values ? You may also try to do a network capture between HAProxy and your agent. -- Christopher Faulet
Re: [PATCH]: BUILD/MEDIUM: set-mark openbsd support
Le 7/3/21 à 10:22 AM, David CARLIER a écrit : Hi here a follow-up of the previous patch but this time for OpenBSD. Thanks, applied now ! -- Christopher Faulet
Re: [PATCH] DOC: use CREATE USER for mysql-check
Le 7/1/21 à 4:09 AM, Daniel Black a écrit :
CREATE USER has been the standard way of creating users since
MySQL-5.0 (2005).
The current syntax of INSERT INTO mysql.user won't actually work
on MariaDB-10.4+.
Because haproxy doesn't use any resources the MySQL executable comment
syntax provides resource contraints to make it more palatable
to risk adverse users.
/*!50701 is a syntax recognised by MySQL and MariaDB 5.7.1+ when
resource contraints where added.
/*M!100201 is a MariaDB executable comment syntax recognised for MariaDB
for the 10.2.1 where the MAX_STATEMENT_TIME was added.
This patch may be backported as far as 2.0.
---
doc/configuration.txt | 11 ++-
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/doc/configuration.txt b/doc/configuration.txt
index 5d1c97bd3..a9bbf6fa4 100644
--- a/doc/configuration.txt
+++ b/doc/configuration.txt
@@ -8982,12 +8982,13 @@ option mysql-check [ user [ { post-41 |
pre-41 } ] ]
one Client Authentication packet, and one QUIT packet, to correctly close
MySQL session. We then parse the MySQL Handshake Initialization packet
and/or
Error packet. It is a basic but useful test which does not produce error nor
- aborted connect on the server. However, it requires adding an authorization
- in the MySQL table, like this :
+ aborted connect on the server. However, it requires an unlocked authorised
+ user without a password. To create a basic limited user in MySQL with
+ optional resource limits:
- USE mysql;
- INSERT INTO user (Host,User) values ('','');
- FLUSH PRIVILEGES;
+ CREATE USER ''@''
+ /*!50701 WITH MAX_QUERIES_PER_HOUR 1 MAX_UPDATES_PER_HOUR 0 */
+ /*M!100201 MAX_STATEMENT_TIME 0.0001 */;
If you don't specify a username (it is deprecated and not recommended), the
check only consists in parsing the Mysql Handshake Initialization packet or
Merged now, thanks !
--
Christopher Faulet
Re: proposed enhancement to mysql-check - accept account locked/password expired errors
Le 7/1/21 à 7:14 AM, Daniel Black a écrit : It seems users are still disturbed at creating passwordless users in mysql for mysql-check. https://discourse.haproxy.org/t/haproxy-mysql-check-user-removal/6685 I certainly understand not wanting to implement the truly ugly implementation that is the protocol to support multiple password mechanisms ( mysql_native_password, mysql_caching_sha256_password etc). As a middle ground, would you be willing to take a haproxy patch that: * accepts an account locked error on the client as a valid state?; and/or * offer the CLIENT_CAN_HANDLE_EXPIRED_PASSWORDS client capability and accept the password expired error ( 1820) as a valid state? This would enable users to deploy a user account for MySQL and MariaDB that is truly unusable and make the assessment to use this functionality easier from a risk management point of view. On the mysql/mariadb server side both of these are Notes like below: 2021-07-01T04:41:03.144498Z 4554 [Note] Your password has expired. To log in you must change it using a client that supports expired passwords. 2021-07-01T04:57:30.706593Z 5056 [Note] Access denied for user 'haproxy'@'10.0.2.100'. Account is locked. This has the consequences that: * these aren't counted as connection errors therefore reaching the default 100 max_connect_errors isn't hit because of this. * dropping MySQL's log_error_verbosity=2 from 3(default) will hide these informational messages from the logs that would otherwise be quite noisy while not hiding anything else too significant. In the same area of code, dropping the MySQL-4.0 protocol support is probably called for given the MySQL-5.0 was end of support in January 9, 2012 https://endoflife.software/applications/databases/mysql . 4.1 protocol is still in use. Hi, First of all, it is a good idea to remove support of "pre-41" option. It is not the default option anymore and I guess it is old enough to be removed. It is now on my todo list. About the MySQL check itself and the way it works, I'm open to any suggestion. From the check point of view, your proposal seems reasonable. Of course, the actual behavior must still be supported (passwordless account). However, from the MySQL administration point of view, I don't know if it is acceptable or not. I'm not a MySQL admin and I don't know if it is acceptable or not to change the log level. I guess that if no one has a better idea or any objections about this change, you can provide a patch. -- Christopher Faulet
[ANNOUNCE] haproxy-2.4.2
_get_scheme MEDIUM: http: implement scheme-based normalization MEDIUM: h1-htx: apply scheme-based normalization on h1 requests MEDIUM: h2: apply scheme-based normalization on h2 requests REGTESTS: add http scheme-based normalization test Christopher Faulet (19): BUG/MINOR: server-state: load SRV resolution only if params match the config BUG/MINOR: server: Forbid to set fqdn on the CLI if SRV resolution is enabled BUG/MEDIUM: server/cli: Fix ABBA deadlock when fqdn is set from the CLI MINOR: resolvers: Clean server in a dedicated function when removing a SRV item MINOR: resolvers: Remove server from named_servers tree when removing a SRV item BUG/MEDIUM: resolvers: Add a task on servers to check SRV resolution status BUG/MINOR: resolvers: Use resolver's lock in resolv_srvrq_expire_task() BUG/MINOR: server/cli: Fix locking in function processing "set server" command MINOR: tcp-act: Add set-src/set-src-port for "tcp-request content" rules DOC: config: Add missing actions in "tcp-request session" documentation CLEANUP: dns: Remove a forgotten debug message BUG/MINOR: resolvers: Always attach server on matching record on resolution BUG/MINOR: resolvers: Reset server IP when no ip is found in the response MINOR: resolvers: Reset server IP on error in resolv_get_ip_from_response() BUG/MINOR: tcpcheck: Fix numbering of implicit HTTP send/expect rules BUG/MINOR: mqtt: Fix parser for string with more than 127 characters BUG/MINOR: mqtt: Support empty client ID in CONNECT message BUG/MEDIUM: resolvers: Make 1st server of a template take part to SRV resolution Revert "MINOR: tcp-act: Add set-src/set-src-port for "tcp-request content" rules" Daniel Black (1): DOC: config: use CREATE USER for mysql-check David Carlier (1): BUILD: Makefile: fix linkage for Haiku. Dirkjan Bussink (1): BUG/MINOR: checks: return correct error code for srv_parse_agent_check Emeric Brun (3): BUG/MINOR: stick-table: fix several printf sign errors dumping tables BUG/MINOR: peers: fix data_type bit computation more than 32 data_types DOC: stick-table: add missing documentation about gpt0 stored type Tim Duesterhus (1): BUG/MINOR: cache: Correctly handle existing-but-empty 'accept-encoding' header Willy Tarreau (2): BUG/MEDIUM: sock: make sure to never miss early connection failures BUG/MINOR: cli: fix server name output in "show fd" -- Christopher Faulet
[ANNOUNCE] haproxy-2.3.11
even with fixed id BUG/MAJOR: server: fix deadlock when changing maxconn via agent-check REGTESTS: fix maxconn update with agent-check Christopher Faulet (39): BUG/MINOR: mux-fcgi: Don't send normalized uri to FCGI application BUG/MINOR: htx: Preserve HTX flags when draining data from an HTX message BUG/MINOR: applet: Notify the other side if data were consumed by an applet BUG/MINOR: hlua: Don't rely on top of the stack when using Lua buffers BUG/MINOR: stream: Decrement server current session counter on L7 retry BUG/MINOR: stream: Reset stream final state and si error type on L7 retry BUG/MINOR: checks: Handle synchronous connect when a tcpcheck is started BUG/MINOR: checks: Reschedule check on observe mode only if fastinter is set MINOR: channel: Rely on HTX version if appropriate in channel_may_recv() BUG/MINOR: stream-int: Don't block reads in si_update_rx() if chn may receive MINOR: conn-stream: Force mux to wait for read events if abortonclose is set MEDIUM: mux-h1: Don't block reads when waiting for the other side BUG/MEDIUM: mux-h1: Properly report client close if abortonclose option is set REGTESTS: Add script to test abortonclose option BUG/MEDIUM: filters: Exec pre/post analysers only one time per filter BUG/MINOR: http-comp: Preserve HTTP_MSGF_COMPRESSIONG flag on the response BUG/MINOR: http-ana: Handle L7 retries on refused early data before K/A aborts BUG/MAJOR: stream-int: Release SI endpoint on server side ASAP on retry BUG/MEDIUM: compression: Add a flag to know the filter is still processing data BUG/MAJOR: htx: Fix htx_defrag() when an HTX block is expanded BUG/MINOR: mux-fcgi: Expose SERVER_SOFTWARE parameter by default DOC: lua: Add a warning about buffers modification in HTTP BUILD: cfgparse-ssl: Remove const from defpx param in keylog parsing function BUG/MEDIUM: server/cli: Fix ABBA deadlock when fqdn is set from the CLI BUG/MINOR: server/cli: Fix locking in function processing "set server" command MINOR: tcp-act: Add set-src/set-src-port for "tcp-request content" rules DOC: config: Add missing actions in "tcp-request session" documentation BUG/MINOR: tcpcheck: Fix numbering of implicit HTTP send/expect rules BUG/MINOR: server-state: load SRV resolution only if params match the config BUG/MINOR: server: Forbid to set fqdn on the CLI if SRV resolution is enabled MINOR: resolvers: Clean server in a dedicated function when removing a SRV item MINOR: resolvers: Remove server from named_servers tree when removing a SRV item BUG/MEDIUM: resolvers: Add a task on servers to check SRV resolution status BUG/MINOR: resolvers: Use resolver's lock in resolv_srvrq_expire_task() BUG/MINOR: resolvers: Always attach server on matching record on resolution BUG/MINOR: resolvers: Reset server IP when no ip is found in the response MINOR: resolvers: Reset server IP on error in resolv_get_ip_from_response() BUG/MEDIUM: resolvers: Make 1st server of a template take part to SRV resolution Revert "MINOR: tcp-act: Add set-src/set-src-port for "tcp-request content" rules" Daniel Black (1): DOC: config: use CREATE USER for mysql-check Dirkjan Bussink (1): BUG/MINOR: checks: return correct error code for srv_parse_agent_check Emeric Brun (17): BUG/MEDIUM: peers: initialize resync timer to get an initial full resync BUG/MEDIUM: peers: register last acked value as origin receiving a resync req BUG/MEDIUM: peers: stop considering ack messages teaching a full resync BUG/MEDIUM: peers: reset starting point if peers appears longly disconnected BUG/MEDIUM: peers: reset commitupdate value in new conns BUG/MEDIUM: peers: re-work updates lookup during the sync on the fly BUG/MEDIUM: peers: reset tables stage flags stages on new conns MINOR: peers: add informative flags about resync process for debugging BUG/MEDIUM: dns: reset file descriptor if send returns an error BUG/MEDIUM: dns: send messages on closed/reused fd if fd was detected broken BUG/MINOR: resolvers: answser item list was randomly purged or errors MEDIUM: resolvers: add a ref on server to the used A/ answer item MEDIUM: resolvers: add a ref between servers and srv request or used SRV record BUG/MAJOR: resolvers: segfault using server template without SRV RECORDs BUG/MINOR: stick-table: fix several printf sign errors dumping tables DOC: stick-table: add missing documentation about gpt0 stored type BUG/MINOR: peers: fix data_type bit computation more than 32 data_types Remi Tricot-Le Breton (15): BUG/MEDIUM: ebtree: Invalid read when looking for dup entry BUG/MINOR: server: Missing call
[ANNOUNCE] haproxy-2.2.15
Hi, HAProxy 2.2.15 was released on 2021/07/16. It added 90 new commits after version 2.2.14. This release is very similar to the 2.3.11/2.3.12. To sum up, most noticeable bugs fixed in this release are: * A possible deadlock if "set maxconn server" command was used when there was a pending connection ready to be dequeued. * A possible infinite loop in process_stream() when a connection error was reported while the stream was waiting for a retry. * A possible race between free() and pool_alloc() in the pools lockless variant. * A bug in the HTX defragmentation leading to crash. The bug might be encountered in the HTTP compression filter or in HTTP header replacement. * An old bug preventing the dequeuing for servers with a very low maxconn because the load balancing was not skipped when a new connection was picked from the proxy's or server's queue. * A bug in the sock part leading to high CPU usage because some early connection failures might be missed. * A thread-safety issue with the SHCTX code when compiled with USE_PRIVATE_CACHE mode. It was not using any locks. * Most of resolvers performance issues and several other bugs in this area. * An issue with the abortonclose option. It was not working since a while. * A bug in the HTTP compression leading to truncated or corrupted responses. * A bug with synchronous connect in tcpcheck when several connections come one after the other. * "url_ip"/"url_port" sample fetches not properly handling url parsing errors. In addition, the http-ignore-probes is now respected for H2 connections. When this option is set, no errors are reported anymore when connections are aborted during preface. And the FCGI multiplexer was slightly improved to send a relative path instead of a normalized URI to an application and to expose SERVER_SOFTWARE parameter by default. Finally, as a consequence of the bug fixed in the pools, the code was simplified. The lockless implementation is used everywhere, resulting in the removal of the very old locked implementation that was kept for non-capable architectures. As a result, threads will now be faster on less common architectures (e.g. i686, MIPS, PPC64, ...). The rest is less visible but contains, as usual, cleanups, small fixes here and there, improvements... It is strongly advised to update to this version. Thanks everyone for your help and your contributions! Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse: http://discourse.haproxy.org/ Slack channel: https://slack.haproxy.org/ Issue tracker: https://github.com/haproxy/haproxy/issues Wiki : https://github.com/haproxy/wiki/wiki Sources : http://www.haproxy.org/download/2.2/src/ Git repository : http://git.haproxy.org/git/haproxy-2.2.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-2.2.git Changelog: http://www.haproxy.org/download/2.2/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ --- Complete changelog : Alex (1): DOC: use the req.ssl_sni in examples Alexandar Lazic (1): DOC/MINOR: move uuid in the configuration to the right alphabetical order Amaury Denoyelle (5): BUG/MINOR: http_fetch: fix possible uninit sockaddr in fetch_url_ip/port BUG/MAJOR: server: prevent deadlock when using 'set maxconn server' BUG/MINOR: stick-table: insert srv in used_name tree even with fixed id BUG/MAJOR: server: fix deadlock when changing maxconn via agent-check REGTESTS: fix maxconn update with agent-check Christopher Faulet (35): BUG/MINOR: hlua: Don't rely on top of the stack when using Lua buffers BUG/MINOR: stream: Decrement server current session counter on L7 retry BUG/MINOR: stream: Reset stream final state and si error type on L7 retry BUG/MINOR: checks: Handle synchronous connect when a tcpcheck is started BUG/MINOR: checks: Reschedule check on observe mode only if fastinter is set MINOR: channel: Rely on HTX version if appropriate in channel_may_recv() BUG/MINOR: stream-int: Don't block reads in si_update_rx() if chn may receive MINOR: conn-stream: Force mux to wait for read events if abortonclose is set MEDIUM: mux-h1: Don't block reads when waiting for the other side BUG/MEDIUM: mux-h1: Properly report client close if abortonclose option is set REGTESTS: Add script to test abortonclose option BUG/MEDIUM: filters: Exec pre/post analysers only one time per filter BUG/MINOR: http-comp: Preserve HTTP_MSGF_COMPRESSIONG flag on the response BUG/MINOR: http-ana: Handle L7 retries on refused early data before K/A aborts BUG/MAJOR: stream-int: Release SI endpoint on server side ASAP on retry BUG/MEDIUM: compression: Add a flag to know the filter is
[ANNOUNCE] haproxy-2.0.23
Hi, HAProxy 2.0.23 was released on 2021/07/16. It added 97 new commits after version 2.0.22. All commits were already mentioned in announcements of the last 2.3 and 2.2 releases. So I will be brief. Here is the list of main bugs fixed by this release: * An issue in the CONTINUATION frames parsing in h2 leading to spurious wakeups. * A bug in the shutdowns detection for h2 connections. * A possible deadlock if "set maxconn server" command was used when there was a pending connection ready to be dequeued. * A bug in the HTX defragmentation leading to crash. The bug might be encountered in the HTTP compression filter or in HTTP header replacement. * An old bug preventing the dequeuing for servers with a very low maxconn because the load balancing was not skipped when a new connection was picked from the proxy's or server's queue. * A bug in the sock part leading to high CPU usage because some early connection failures might be missed. * A thread-safety issue with the SHCTX code when compiled with USE_PRIVATE_CACHE mode. It was not using any locks. * An issue with the abortonclose option. It was not working since a while. * A bug in the HTTP compression leading to truncated or corrupted responses. * "url_ip"/"url_port" sample fetches not properly handling url parsing errors. * A bug in the cpu-map notation when both processes and threads were specified, most specifically P-Q/1 or 1/P-Q notation. * Some issues affecting the peers synchronization. * H1 idle connections on server side receiving data not closed. Because of this bug, it was possible to send pending 408-Request-time-out responses to clients. * Wrong number of retries reported in logs if no connection was attempted. Since the beginning, when the session was aborted before any connection attempt to any server, the backend retries value was reported, instead of 0. * A bug with the method sample fetch when an exotic method is found. Note that resolvers performance issues fixed in upper versions were not fixed in this one because the code is too different. This part is a perpetual source of bugs. It was too risky to backport the fixes. If you experience any issue with the resolvers, please consider to upgrade to a newer version. Thanks everyone for your help and your contributions! Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse: http://discourse.haproxy.org/ Slack channel: https://slack.haproxy.org/ Issue tracker: https://github.com/haproxy/haproxy/issues Wiki : https://github.com/haproxy/wiki/wiki Sources : http://www.haproxy.org/download/2.0/src/ Git repository : http://git.haproxy.org/git/haproxy-2.0.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-2.0.git Changelog: http://www.haproxy.org/download/2.0/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ --- Complete changelog : Amaury Denoyelle (6): BUG/MINOR: server: free srv.lb_nodes in free_server BUG/MEDIUM: config: fix cpu-map notation with both process and threads BUG/MINOR: http_fetch: fix possible uninit sockaddr in fetch_url_ip/port BUG/MAJOR: server: prevent deadlock when using 'set maxconn server' BUG/MINOR: stick-table: insert srv in used_name tree even with fixed id BUG/MAJOR: server: fix deadlock when changing maxconn via agent-check Christopher Faulet (34): DOC: Explicitly state only IPv4 are supported by forwardfor/originalto options BUG/MEDIUM: threads: Ignore current thread to end its harmless period BUG/MINOR: http-fetch: Make method smp safe if headers were already forwarded BUG/MINOR: http_htx: Remove BUG_ON() from http_get_stline() function BUG/MINOR: logs: Report the true number of retries if there was no connection BUG/MINOR: mux-h1: Release idle server H1 connection if data are received BUG/MAJOR: mux-h2: Properly detect too large frames when decoding headers BUG/MEDIUM: mux-h2: Fix dfl calculation when merging CONTINUATION frames BUG/MEDIUM: mux-h2: Properly handle shutdowns when received with data BUG/MINOR: htx: Preserve HTX flags when draining data from an HTX message BUG/MINOR: applet: Notify the other side if data were consumed by an applet BUG/MINOR: hlua: Don't rely on top of the stack when using Lua buffers BUG/MINOR: stream: Decrement server current session counter on L7 retry BUG/MINOR: stream: Reset stream final state and si error type on L7 retry MINOR: channel: Rely on HTX version if appropriate in channel_may_recv() BUG/MINOR: stream-int: Don't block reads in si_update_rx() if chn may receive MEDIUM: mux-h1: Don't block reads when waiting for the other side REGTESTS: Add script to test
Re: http-request return bytes_read from v2.3 to v2.4
Le 7/25/21 à 1:26 PM, William Dauchy a écrit : Hello, While upgrading from v2.3.x to v2.4.x I noticed a difference of bytes read in requests http logs with this simple frontend config: frontend foo bind 127.0.0.1:8000 http-request return in v2.3, the logs (%B) tells me 54 bytes, in v2.4, I am getting 49. So a difference of 5 bytes per request. I was simply curious to understand that behavior change. Is it a change in the way we count bytes in stats, or is there really a difference in the payload we write? When I look at a dump, the TCP payloads are identical with 57 bytes in both versions. And the overall length of the packet is 123 bytes. Maybe that's expected, but I wanted some clarification. Hi William, The response size reported in the log messages is related to the internal representation of the HTTP message. It is a limitation of the current design. It means this size is not equal to the raw size of the message and may change when the HTX representation changes. In this case, in 2.4, the start-line representation was changed, a 32-bits integer was removed from hx_sl structure. In addition, the end-of-message block (EOM) was removed from the HTX representation. It counted for 1 byte. This explains the difference of 5 bytes between 2.3 and 2.4. -- Christopher Faulet
Re: Is it possible to capture the body of http responses?
Le 8/11/21 à 2:53 AM, Ryan Burn a écrit : I'm working on integrating HAProxy with traceable.ai <http://traceable.ai>'s security product. As part of the integration, we'd like to capture the contents of any http responses processed by HAProxy and send them to a service either via SPOA or an RPC call from Lua. The response contents are used by the product to help identify possible security threats. I've tried a few things, but haven't found a reliable way to capture the contents of response bodies. Is this possible with HAProxy? Here are the approaches I've explored so far: 1. I used the "res.body" fetch but that only provides the contents sometimes (I presume if it's available in a buffer): https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/extcap.conf#L19 <https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/extcap.conf#L19> 2. I also tried accessing the contents of the response channel from a Lua action, but that fails with "Cannot manipulate HAProxy channels in HTTP mode" https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/response.lua#L5 <https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/response.lua#L5> Hi Ryan, About the SPOE, it is on the todo-list to add streaming feature to be able to send request/response payload to an agent and be able to rewrite it. Unfortunately, to make it clean and usable, it requires a total refactoring and for now, I haven't found the time to work on it. About the sample fetches, on HAProxy 2.3 and lower, there is no way to get the response payload because it is not possible to wait for it. There is no equivalent to the "http-buffer-request" option on the response side. On HAProxy-2.4, it is possible by using "wait-for-body" HTTP rule, available on the request and the response side. However, it is still limited by the buffer size. With the Lua, it is only possible by writing a filter using the experimental Lua filter API, available in HAProxy 2.5. This API is really experimental for now, but it may be a solution to analyze the whole message payload, regardless its size. However, It may be painful because the API may be incomplete and because dealing with multiple buffers is not simple, especially if you don't want to forward the payload before the end of analysis. -- Christopher Faulet
Re: Is it possible to capture the body of http responses?
Le 9/14/21 à 3:14 AM, Ryan Burn a écrit : On Thu, Sep 9, 2021 at 12:22 AM Christopher Faulet <mailto:cfau...@haproxy.com>> wrote: Le 8/11/21 à 2:53 AM, Ryan Burn a écrit : > I'm working on integrating HAProxy with traceable.ai <http://traceable.ai> <http://traceable.ai <http://traceable.ai>>'s > security product. > > As part of the integration, we'd like to capture the contents of any http > responses processed by HAProxy and send them to a service either via SPOA or an > RPC call from Lua. The response contents are used by the product to help > identify possible security threats. > > I've tried a few things, but haven't found a reliable way to capture the > contents of response bodies. Is this possible with HAProxy? > > Here are the approaches I've explored so far: > > 1. I used the "res.body" fetch but that only provides the contents sometimes (I > presume if it's available in a buffer): > https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/extcap.conf#L19 <https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/extcap.conf#L19> > <https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/extcap.conf#L19 <https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/extcap.conf#L19>> > > 2. I also tried accessing the contents of the response channel from a Lua > action, but that fails with "Cannot manipulate HAProxy channels in HTTP mode" > https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/response.lua#L5 <https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/response.lua#L5> > <https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/response.lua#L5 <https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/response.lua#L5>> About the sample fetches, on HAProxy 2.3 and lower, there is no way to get the response payload because it is not possible to wait for it. There is no equivalent to the "http-buffer-request" option on the response side. On HAProxy-2.4, it is possible by using "wait-for-body" HTTP rule, available on the request and the response side. However, it is still limited by the buffer size. Thanks Christopher! Do you know how to access the response body from a SPOA if you add the "wait-for-body"? I added the wait-for-proxy rules to my example project, but the "res.body" argument still doesn't consistently provide the full body. https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/haproxy.cfg#L15-L16 <https://github.com/rnburn/haproxy-extcap/blob/master/test/docker/haproxy.cfg#L15-L16> I've checked your configuration and your SPOE message is sent on the 'on-http-response' event. This event is triggered before 'http-response' ruleset evaluation. Thus the 'wait-for-body' action is not performed yet at this stage. Here, you should use a SPOE group and send it using 'send-spoe-group' action. The same should be done on the request side. -- Christopher Faulet
Re: [PATCH 1/2] CLEANUP: Include check.h in flt_spoe.c
Le 9/16/21 à 5:38 PM, Tim Duesterhus a écrit : This is required for the prototype of spoe_prepare_healthcheck_request(). --- src/flt_spoe.c | 1 + 1 file changed, 1 insertion(+) diff --git a/src/flt_spoe.c b/src/flt_spoe.c index 28a5a24f8..aeb68c889 100644 --- a/src/flt_spoe.c +++ b/src/flt_spoe.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include #include Thanks Tim! Both patches merged now. -- Christopher Faulet
Re: Disabling HTTP/1.1 pipelining
Le 9/17/21 à 1:20 PM, Stefan Behte a écrit : Hi everyone, surely many on this list have heard about the meris botnet (https://krebsonsecurity.com/2021/09/krebsonsecurity-hit-by-huge-new-iot-botnet-meris/) which uses HTTP/1.1 pipelining for layer 7 attacks. As far as I can see, it's not possible to disallow HTTP pipelining in haproxy, so the best possibility could be "option httpclose"? Of course, this does not solve everything when a ~100k botnet is attacking, but it could ease the initial load / mitigate the pipelining vector a bit, as the attack clients have longer RTT. Or maybe I am missing something? Hi, HAproxy does not support HTTP pipelining. But it may be configured to mitigate ddos attack. There are several mechanisms that you can use, depending on your applications. A quick search on the net about "haproxy ddos prevention" will give you several hints. Regards, -- Christopher Faulet
Re: AW: Disabling HTTP/1.1 pipelining
Le 9/21/21 à 6:00 PM, Stefan Behte a écrit : Hi Christopher, thank you for the hint, I'm aware of the different ways to mitigate DDoS with rate limits etc., I was just curious about the pipelining vector. :) http://www.haproxy.org/download/2.4/doc/configuration.txt says: " By default HAProxy operates in keep-alive mode with regards to persistent connections: for each connection it processes each request and response, and leaves the connection idle on both sides between the end of a response and the start of a new request. This mode may be changed by several options such as "option http-server-close" or "option httpclose". Setting "option http-server-close" enables HTTP connection-close mode on the server side while keeping the ability to support HTTP keep-alive and pipelining on the client side." "1.1. The HTTP transaction model" and " timeout http-keep-alive" also mention pipelining. Section 1.1 mainly describes generalities about the HTTP protocol. Only the end of the section is focused on HAProxy and it is specified it only supports keep-alive mode, not the pipelining. However, I agree it is pretty confusing because pipelining is mentioned in "option http-server-close" and "timeout http-keep-alive" descriptions. In fact, the ambiguities comes from the fact that HAProxy does not performed any HTTP pipelining. But the client is free to send several requests in same time. No error will be triggered. However, the requests will be processed the one after the other. Thus, HAProxy does not perform any HTTP pipelining but it does not forbid it. So I guess I did just misunderstand the documentation and it would be nice to just clarify it in the docs that haproxy does not support HTTP/1.1 pipelining. I agree. Pipelining should at least be removed from "option http-server-close" description. And section 1.1 should be reword to be clear on this point. -- Christopher Faulet
[ANNOUNCE] haproxy-2.4.5
http://www.haproxy.org/download/2.4/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ --- Complete changelog : Amaury Denoyelle (8): BUG/MINOR: connection: prevent null deref on mux cleanup task allocation BUILD: ist: prevent gcc11 maybe-uninitialized warning on istalloc BUG/MINOR: server: allow 'enable health' only if check configured MINOR: server: implement a refcount for dynamic servers MINOR: global: define MODE_STOPPING BUG/MINOR: server: do not use refcount in free_server in stopping mode MINOR: server: return the next srv instance on free_server BUG/MINOR: stats: use refcount to protect dynamic server on dump Christopher Faulet (26): MINOR: lua: Add a flag on lua context to know the yield capability at run time BUG/MINOR: lua: Yield in channel functions only if lua context can yield BUG/MINOR: lua: Don't yield in channel.append() and channel.set() BUG/MINOR: stream: Don't release a stream if FLT_END is still registered BUG/MEDIUM: http-ana: Reset channels analysers when returning an error BUG/MINOR: filters: Always set FLT_END analyser when CF_FLT_ANALYZE flag is set BUG/MINOR: filters: Set right FLT_END analyser depending on channel BUG/MEDIUM: mux-h1: Remove "Upgrade:" header for requests with payload MINOR: htx: Skip headers with no value when adding a header list to a message CLEANUP: mux-h1: Remove condition rejecting upgrade requests with payload BUG/MEDIUM: stream-int: Don't block SI on a channel policy if EOI is reached BUG/MAJOR: mux-h1: Don't eval input data if an error was reported BUG/MINOR: tcpcheck: Improve LDAP response parsing to fix LDAP check BUG/MINOR: h1-htx: Fix a typo when request parser is reset BUG/MEDIUM: mux-h1: Adjust conditions to ask more space in the channel buffer BUG/MEDIUM: stream-int: Notify stream that the mux wants more room to xfer data BUG/MEDIUM: stream: Stop waiting for more data if SI is blocked on RXBLK_ROOM MINOR: stream-int: Set CO_RFL transient/persistent flags apart in si_cs_rcv() MINOR: htx: Add an HTX flag to know when a message is fragmented MINOR: htx: Add a function to know if the free space wraps BUG/MEDIUM: stream-int: Defrag HTX message in si_cs_recv() if necessary MINOR: stream-int: Notify mux when the buffer is not stuck when calling rcv_buf BUG/MINOR: mux-h1/mux-fcgi: Sanitize TE header to only send "trailers" MINOR: arg: Be able to forbid unresolved args when building an argument list BUG/MINOR: tcpcheck: Don't use arg list for default proxies during parsing BUG/MINOR: tcp-rules: Stop content rules eval on read error and end-of-input David Carlier (1): BUILD: tools: get the absolute path of the current binary on NetBSD. Dragan Dosen (2): BUG/MINOR: flt-trace: fix an infinite loop when random-parsing is set BUG/MINOR: http-ana: increment internal_errors counter on response error Emeric Brun (1): DOC: peers: fix doc "enable" statement on "peers" sections William Lallemand (3): BUG/MINOR: systemd: ExecStartPre must use -Ws DOC: management: certificate files must be sanitized before injection MINOR: Makefile: add MEMORY_POOLS to the list of DEBUG_xxx options Willy Tarreau (25): BUG/MINOR: compat: make sure __WORDSIZE is always defined CLEANUP: pools: factor all malloc_trim() calls into trim_all_pools() MINOR: pools: automatically disable malloc_trim() with external allocators MINOR: pools: use mallinfo2() when available instead of mallinfo() BUG/MINOR: cli/payload: do not search for args inside payload BUILD: activity: use #ifdef not #if on USE_MEMORY_PROFILING BUILD/MINOR: defaults: eliminate warning on MAXHOSTNAMELEN with -Wundef BUILD/MINOR: ssl: avoid a build warning on LIBRESSL_VERSION with -Wundef IMPORT: slz: silence a build warning with -Wundef BUILD/MINOR: regex: avoid a build warning on USE_PCRE2 with -Wundef BUILD: ssl: next round of build warnings on LIBRESSL_VERSION_NUMBER BUILD: ssl: fix two remaining occurrences of #if USE_OPENSSL BUILD: tools: properly guard __GLIBC__ with defined() BUG/MINOR: vars: improve accuracy of the rules used to check expression validity MINOR: sample: add missing ARGC_ entries BUG/MINOR: vars: properly set the argument parsing context in the expression BUG/MINOR: vars: truncate the variable name in error reports about scope. BUG/MINOR: vars: do not talk about global section in CLI errors for set-var BUILD: compiler: fixed a missing test on defined(__GNUC__) BUILD: halog: fix a -Wundef warning on non-glibc systems BUILD: threads: fix -Wundef for _POSIX_PRIORITY_SCHEDULING on libmusl BUG/MEDIUM: leastconn: f
Re: [ANNOUNCE] haproxy-2.4.5
Le 10/2/21 à 10:54 AM, Matthias Fechner a écrit :
Am 01.10.2021 um 18:09 schrieb Christopher Faulet:
HAProxy 2.4.5 was released on 2021/10/01. It added 69 new commits
after version 2.4.4.
Damned ! You're right...
It is a typo in the commit feca2a453 ("BUG/MINOR: filters: Always set FLT_END
analyzer when CF_FLT_ANALYZE flag is set"). It also affects the 2.5-DEV.
The patch is pretty simple:
diff --git a/src/filters.c b/src/filters.c
index 136a3e80b..f64c192bd 100644
--- a/src/filters.c
+++ b/src/filters.c
@@ -475,7 +475,7 @@ flt_stream_start(struct stream *s)
}
if (strm_li(s) && (strm_li(s)->analysers & AN_REQ_FLT_START_FE)) {
s->req.flags |= CF_FLT_ANALYZE;
- s->req.analysers |= AN_RES_FLT_END;
+ s->req.analysers |= AN_REQ_FLT_END;
}
return 0;
}
I will push a fix. As a workaround, you can temporarily disable the HTTP
compression filter.
Thanks for the report !
--
Christopher Faulet
Re: [ANNOUNCE] haproxy-2.4.5
Le 10/3/21 à 9:06 AM, Vincent Bernat a écrit : ❦ 3 October 2021 08:53 +02, Christopher Faulet: I will push a fix. As a workaround, you can temporarily disable the HTTP compression filter. Will you release 2.4.6 or should we push packages for 2.4.5 with the patch? For Debian/Ubuntu, I didn't push packages for 2.4.5 yet. It is probably better to make a new release. At least for everyone relying on the sources instead of a distribution package. -- Christopher Faulet
Re: [ANNOUNCE] haproxy-2.4.5
Le 10/3/21 à 7:46 PM, Matthias Fechner a écrit :
Am 03.10.2021 um 08:53 schrieb Christopher Faulet:
Damned ! You're right...
It is a typo in the commit feca2a453 ("BUG/MINOR: filters: Always set FLT_END
analyzer when CF_FLT_ANALYZE flag is set"). It also affects the 2.5-DEV.
thanks a lot Christopher for the quick fix.
Just prepared a new package for FreeBSD and it is working fine.
I will ask the maintainer to include this patch till a new release is out.
Thanks Matthias. FYI, I will emit a new release this morning.
--
Christopher Faulet
Re: [ANNOUNCE] haproxy-2.4.5
Le 10/4/21 à 11:05 AM, Herzfeld, Patrick a écrit : Ciao Christopher I see the new release is already on the haproxy.org website, will there be an official release mail to the mailing list or no? I'm adding the mailing list to notify everyone. We are hunting another bug. Unfortunately, it was reported after the 2.4.6 was released. We try to figure out if it is a major problem or not for now. But it crashes HAProxy. So, a 2.4.7 will probably be released very soon. -- Christopher Faulet
[ANNOUNCE] haproxy-2.4.7
Hi,
HAProxy 2.4.7 was released on 2021/10/04. It added 2 new commits
after version 2.4.5.
No, it is no an error. The 2.4.6 was released few hours ago to fix a
regression of the 2.4.5. But I was a little too enthusiastic because
another bug was reported by Yves Lafon. Thus we decided to not announce the
2.4.6 to immediately release the 2.4.7.
The first regression was introduced by the commit feca2a453 ("BUG/MINOR:
filters: Always set FLT_END analyser when CF_FLT_ANALYZE flag is
set"). Because of a typo in the request analyzers when a filter was attached
to a stream, it was possible to hang on the filter release stage. HTTP
filters (HTTP compression and cache) didn't seem to be affected by the bug
in HTTP mode. However, in TCP mode, data forwarding was blocked until the
client or the server timeout expiration.
The second regression was introduced by the commit db8ba1069 ("BUG/MEDIUM:
http-ana: Reset channels analysers when returning an error"). The request
analyzers were not properly cleared when a redirect rule from a redirect
ruleset (via the "redirect ..." directive) was applied. This means that some
HTTP request processing were possible on an already flushed request channel,
leading to crash.
If you planned to upgrade to the 2.4.5 or the 2.4.6, please use the 2.4.7
instead.
Please find the usual URLs below :
Site index : http://www.haproxy.org/
Discourse: http://discourse.haproxy.org/
Slack channel: https://slack.haproxy.org/
Issue tracker: https://github.com/haproxy/haproxy/issues
Wiki : https://github.com/haproxy/wiki/wiki
Sources : http://www.haproxy.org/download/2.4/src/
Git repository : http://git.haproxy.org/git/haproxy-2.4.git/
Git Web browsing : http://git.haproxy.org/?p=haproxy-2.4.git
Changelog: http://www.haproxy.org/download/2.4/src/CHANGELOG
Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/
---
Complete changelog :
Christopher Faulet (2):
BUG/MEDIUM: http-ana: Clear request analyzers when applying redirect rule
BUG/MEDIUM: filters: Fix a typo when a filter is attached blocking the
release
--
Christopher Faulet
Re: Use variables in healthchecks
Le 10/7/21 à 11:59 AM, Basti Schubert a écrit : Hi there, i'd like to use variables in healtchecks to set the host header to the ip of the server that is beeing checked, something like this: http-check connect port 443 ssl http-check send meth GET uri /ecp/healthcheck.htm ver HTTP/1.1 hdr host $SERVERIP http-check expect status 200 Problem is: i don't know if this is possible and if there is already a variable for such configs (can %si from the log format section be used for this?) I'm on haproxy 2.2.x Hi, It is possible to use sample fetches and log-format variables. Thus, you can indeed rely on "%si". But you must use 2.2.14 or newer. -- Christopher Faulet
Re: [PATCH] BUG/MINOR: lua: Fix lua error handling in `hlua_config_prepend_path()`
Le 10/11/21 à 6:51 PM, Tim Duesterhus a écrit :
Set an `lua_atpanic()` handler before calling `hlua_prepend_path()` in
`hlua_config_prepend_path()`.
This prevents the process from abort()ing when `hlua_prepend_path()` fails
for some reason.
see GitHub Issue #1409
This is a very minor issue that can't happen in practice. No backport needed.
---
src/hlua.c | 27 +--
1 file changed, 25 insertions(+), 2 deletions(-)
diff --git a/src/hlua.c b/src/hlua.c
index b03d8bb46..1dbaaa3e0 100644
--- a/src/hlua.c
+++ b/src/hlua.c
@@ -11167,8 +11167,31 @@ static int hlua_config_prepend_path(char **args, int
section_type, struct proxy
}
LIST_APPEND(&prepend_path_list, &p->l);
- hlua_prepend_path(hlua_states[0], type, path);
- hlua_prepend_path(hlua_states[1], type, path);
+ /* Handle the global state and the per-thread state for the first
+* thread. The remaining threads will be initialized based on
+* prepend_path_list.
+*/
+ for (size_t i = 0; i < 2; i++) {
+ lua_State *L = hlua_states[i];
+ const char *error;
+
+ if (setjmp(safe_ljmp_env) != 0) {
+ lua_atpanic(L, hlua_panic_safe);
+ if (lua_type(L, -1) == LUA_TSTRING)
+ error = lua_tostring(L, -1);
+ else
+ error = "critical error";
+ fprintf(stderr, "lua-prepend-path: %s.\n", error);
+ exit(1);
+ } else {
+ lua_atpanic(L, hlua_panic_ljmp);
+ }
+
+ hlua_prepend_path(L, type, path);
+
+ lua_atpanic(L, hlua_panic_safe);
+ }
+
return 0;
err2:
Thanks Tim, now merged !
--
Christopher Faulet
Re: compression offload and http2
Le 10/15/21 à 12:47 AM, Björn Jacke a écrit : Hi, I noticed that the compression offload feature is not working with backends using h2. I couldn't find any note in the documentation that the compression offload feature is limited to http 1 only. Is it a bug that it doesn't work with http2 or is it by design and just the documentation might need some clarification here. Hi, It should work. What is your HAProxy version ? -- Christopher Faulet
Re: compression offload and http2
Le 10/15/21 à 12:04 PM, Björn Jacke a écrit : On 15.10.21 10:10, Christopher Faulet wrote: It should work. What is your HAProxy version ? 2.4.7 Björn Sorry Björn, I missed your reply. It is strange, there is no known bug in this area for now. There is probably something in the request or response headers preventing the compression to be enabled. If it is easy to reproduce on a test platform, you may try to start HAproxy in foreground in debug mode (-db -d on the command line). This will print the request and the response on stderr. -- Christopher Faulet
Re: PH disconnects, but "show errors" has 0 entries ?
Le 10/13/21 à 8:30 PM, Jim Freeman a écrit : In adding a couple of new security response headers via haproxy.cfg (one is 112 bytes, the other 32), a few requests are now getting 500 status (PH session state) responses, but "show errors" has 0 entries? Most responses succeed (all have the additional headers), so it's not a problem with the new headers themselves. If haproxy generates a PH/500, shouldn't "show errors" show details of the offending response ? Thanks, ...jfree == # echo "show info" | socat stdio /run/haproxy/stats.sock | grep ^Version: Version: 2.2.8-1~bpo10+1 # echo "show errors -1" | socat - /run/haproxy/stats.sock Total events captured on [13/Oct/2021:18:24:15.819] : 0 # cat /etc/debian_version 10.11 Hi, Only parsing errors are reported by "show errors" command. Here PH/500 error is most probably due to a header rewrite error. I have not deeply checked however. You can verify my assumption by checking the "wrew" counter in "show stats" command output on the stats socket. Header rewrite errors are triggered when there is not enough space in the buffer to perform the rewrites. By default, 1024 Bytes are reserved in the buffer, to be sure to have enough space to perform some rewrites. If you add many headers in the response, it may be the problem. You can increase the reserve by setting "tune.maxrewrite" global parameter. When such error is encountered, HAProxy returns a 500-Internal-Error response. You can change that to make it fails silently. To do so, take a look at the "strict-mode" http-response action. -- Christopher Faulet
Re: PH disconnects, but "show errors" has 0 entries ?
Le 10/19/21 à 16:49, Jim Freeman a écrit : OK - this is weird (so don't shoot the messenger?). With more tcpdump-ing and examination, the back-end service logs that it sent a response, but 1) tcpdump running on the haproxy instance never sees the response ! a) 2 proxies - an AWS ELB and on-instance nginx - lie between HAProxy instance and the service 2) sans any response (and within 0.2 to 13 seconds of the request send), HAProxy initiates the PH/500 to the client! It would make sense to me if any timeouts or disconnects were involved - HAProxy would report an [sS][DH] or somesuch. And reverting the sending of the "content-security-policy: frame-ancestors ..." and "x-frame-options: ..." response(!) headers makes the problem disappear again. You'll rightly point out that HTTP/1.1 is stateless, and that the prior history of the request/response stream (and response headers sent to the client) shouldn't affect the (non-)response to a given request. Any clues as to how/why the PH/500 might be generated without a response to trigger it would be most eagerly received. While it is entirely likely this will wind up being a "nut loose on the keyboard" issue, I just thought I'd share my observations and befuddlement ... First of all, I missed a point. The 2.2.8 is quite old. You must upgrade first. Then, have you check the rewrite error counters on your backend ? Because, AFAIK, it is the only place where a 500 is possible with the PH termination state. If you are using http-after-response rules, it may explain this error. However, share your redacted configuration too. It can help me to explain what you observe. -- Christopher Faulet
Re: compression offload ... in default section
Le 10/20/21 à 01:24, Björn Jacke a écrit : Hi, On 19.10.21 11:06, Christopher Faulet wrote: Sorry Björn, I missed your reply. It is strange, there is no known bug in this area for now. There is probably something in the request or response headers preventing the compression to be enabled. I found the error: the "compression offload" is not honored in the default section. This behavior is documented but it slipped from my memory. What is actually the reason why this is silently ignored when it is set in the the default section? If someone has a reason to set that in the default section, why does haproxy ignore it intentionally? If the support of compression offload shall stay intentionally unsupported in the default section, then it would be good if this would trigger a configure check warning, wouldn't it? Thanks Björn Good catch. Indeed, this parameter is ignored in defaults sections. However there is no warning and the documentation is not really helpful on this point because it is only mentioned as a side note. I guess it may be more visible. This parameter is not supported in defaults sections because there is no way to cancel it in backend sections. I guess we can add a directive to disable the compression offloading. This way, we will be able to support "compression offload" in defaults section. -- Christopher Faulet
Re: Problem with the var() sample fetch function
Le 10/27/21 à 18:32, Willy Tarreau a écrit :
Christopher also found that the set-var() converter already mandates a
matching method, as the following will be rejected:
... if { int(12),set-var(txn.truc) 12 }
while this one will work:
... if { int(12),set-var(txn.truc) eq 12 }
Just a little clarification. Both are rejected with an error message. "-m int"
is required:
... if { int(12),set-var(txn.truc) -m int eq 12 }
Emitting an error seems to be an obvious choice if var() sample is used in the
same situations. However, this also means following ACLs will be rejected while
it works today:
http-request set-var(txn.foo) str("forbidden")
http-request deny if { var(txn.foo) "forbidden" }
With the proposed change, "-m str" will be required.
--
Christopher Faulet
Re: [PATCH] REGTESTS: Use `feature cmd` for 2.5+ tests (2)
Le 11/4/21 à 21:12, Tim Duesterhus a écrit : This patch effectively is identical to 7ba98480cc5b2ede0fd4cca162959f66beb82c82. Merged, thanks Tim! -- Christopher Faulet
[ANNOUNCE] haproxy-2.2.18
close ones).
- the validity checks for sample fetch functions were only applied to
the frontend capability of a proxy. This means that using a small
set of sample fetch functions (like "be_name()") in proxies that are
both a frontend and a backend ("listen" or "defaults") would lead to
a config error while it is technically valid. This problem has always
been there and never reported.
- automatic cast of variables to other types would fail to first verify
if a cast method was known, possibly causing a crash at runtime when
calling them for the first time (e.g. using a variable of type address
as an argument to strcmp() or a boolean with secure_memcmp()).
- some streams could sometimes be frozen when filters were enabled (such
as compression) and an error was raised with data still left to be
processed.
- HTTP health check could report L7 timeout when facing a parse error,
because the response is dropped before being translated to HTX, while
the check waiting for a response didn't explicitly check for a possible
end-of-input.
- http-after-response rules must stop after an "allow" action, to match
their http-response counter-part.
- the parsing of the "Authorization" header field would fail if more
than one space was present between the scheme and the value.
We planned to emit a new 2.0 release the next week if no major bug is
reported on this one affecting the 2.0 too. Thanks everyone for your help
and your contributions!
Please find the usual URLs below :
Site index : http://www.haproxy.org/
Discourse: http://discourse.haproxy.org/
Slack channel: https://slack.haproxy.org/
Issue tracker: https://github.com/haproxy/haproxy/issues
Wiki : https://github.com/haproxy/wiki/wiki
Sources : http://www.haproxy.org/download/2.2/src/
Git repository : http://git.haproxy.org/git/haproxy-2.2.git/
Git Web browsing : http://git.haproxy.org/?p=haproxy-2.2.git
Changelog: http://www.haproxy.org/download/2.2/src/CHANGELOG
Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/
---
Complete changelog :
Amaury Denoyelle (3):
BUILD: ist: prevent gcc11 maybe-uninitialized warning on istalloc
BUG/MINOR: server: allow 'enable health' only if check configured
BUILD: fix compilation on NetBSD
Christopher Faulet (31):
BUG/MEDIUM: stream-int: Don't block SI on a channel policy if EOI is
reached
Revert "REGTESTS: mark http_abortonclose as broken"
BUG/MINOR: tcpcheck: Improve LDAP response parsing to fix LDAP check
BUG/MINOR: h1-htx: Fix a typo when request parser is reset
BUG/MEDIUM: mux-h1: Adjust conditions to ask more space in the channel
buffer
BUG/MEDIUM: stream-int: Notify stream that the mux wants more room to
xfer data
BUG/MEDIUM: stream: Stop waiting for more data if SI is blocked on
RXBLK_ROOM
BUG/MINOR: mux-h1/mux-fcgi: Sanitize TE header to only send "trailers"
MINOR: arg: Be able to forbid unresolved args when building an argument
list
BUG/MINOR: tcpcheck: Don't use arg list for default proxies during parsing
BUG/MINOR: tcp-rules: Stop content rules eval on read error and
end-of-input
BUG/MINOR: stream: Don't release a stream if FLT_END is still registered
BUG/MEDIUM: http-ana: Reset channels analysers when returning an error
BUG/MINOR: filters: Always set FLT_END analyser when CF_FLT_ANALYZE flag
is set
BUG/MINOR: filters: Set right FLT_END analyser depending on channel
BUG/MEDIUM: filters: Fix a typo when a filter is attached blocking the
release
BUG/MEDIUM: http-ana: Clear request analyzers when applying redirect rule
MINOR: htx: Add an HTX flag to know when a message is fragmented
MINOR: htx: Add a function to know if the free space wraps
BUG/MEDIUM: stream-int: Defrag HTX message in si_cs_recv() if necessary
BUG/MEDIUM: mux_h2: Handle others remaining read0 cases on partial frames
BUG/MINOR: http-ana: Don't eval front after-response rules if stopped on
back
BUG/MEDIUM: stream: Keep FLT_END analyzers if a stream detects a channel
error
BUG/MEDIUM: tcpcheck: Properly catch early HTTP parsing errors
BUG/MINOR: mux-h1: Save shutdown mode if the shutdown is delayed
BUG/MEDIUM: mux-h1: Perform a connection shutdown when the h1c is released
BUG/MEDIUM: resolvers: Don't recursively perform requester unlink
BUG/MEDIUM: resolvers: Track api calls with a counter to free resolutions
BUG/MEDIUM: http-ana: Drain request data waiting the tarpit timeout
expiration
DOC: config: Fix alphabetical order of fc_* samples
MINOR: stream: Improve dump of bogus streams
Dragan Dosen (1):
BUG/MINOR: http-ana: increment internal_errors counter on res
Re: [PATCH] MINOR: promex: backend aggregated server check status
Le 11/8/21 à 14:31, William Dauchy a écrit : On Mon, Nov 8, 2021 at 1:52 PM Willy Tarreau wrote: Just to be sure, is this something you want to merge into 2.5 or is it to be queued next ? I'm fine with both, but I prefer to ask as it's not just a one-liner and I don't know the impact on promex. I know Christopher was possibly thinking about a more advanced implementation but I thought it could be ok for a first version. Probably a good idea to wait for Christopher feedbacks anyway. I was indeed targeting a late v2.5, despite being a new thing. Sorry for that and I forgot to add a message about it. If it works well enough, I will also probably ask for a backport in 2.4 before the end of the year as I know large clusters are waiting for this patch to lower their memory consumption in prometheus. Thanks, Hi William, The change seems reasonable for a first implementation. I will add backports info in the commit message and merge it. Thanks ! -- Christopher Faulet
[ANNOUNCE] haproxy-2.4.9
Hi,
HAProxy 2.4.9 was released on 2021/11/23. It added 36 new commits
after version 2.4.8.
In the previous release, fixes about shutdowns management in the muxes have
exposed some hidden bugs. Since the muxes were introduced, in the 1.8,
shutdowns at the conn-stream level were not fully idempotent. Until
recently, it was not an issue. But in the 2.4.8, some users observed delays
to close client connections on the HAProxy side corresponding to the client
timeout because the silent mode was used instead of the clean one to
shutdown the connection. In addition, true silent shutdowns were not
properly handled in the H1 multiplexer when outgoing data were blocked,
leading too to delay to close connections.
A H2 multiplexer fix to drain data and be sure to send GOAWAY frame was
announced in the 2.4.8. However a patch was missing. Another side effect of
this missing patch was the TLS sessions were not cached as expected. It is
now fixed. Still on the H2 multiplexer, an old fix for H2 partial frames was
incomplete and caused some high CPU usages in h2_io_cb() on some rare
occasions.
Some users reported occasional crashes in the cache (#1284 and #1451). We
finally had an explanation (a missing break). This was fixed. "show cache"
cli command was also fixed to be thread-safe. Under high load, it was
possible to dereference a node already reassigned, leading to
crash. Finally, parsing of "max-age" or "s-maxage" was improved to properly
ignore unparsable value in quotes.
A bug with the "program" post-parser was fixed. It could be called with an
empty programs list in case of a config parsing error on reload after
another error, and could crash.
Recent adjustments about the backend support for WebSocket over HTTP/2 were
backported. They allow to fallback on a HTTP/1 connection if the WebSockets
are not support in HTTP/2. In addition the server keyword "ws" can be used
to tune this.
http-response rulesets evaluation was not aligned with what is said in the
documentation. It was possible to inhibit the frontend rules evaluation with
an "allow" rule in the backend section while it should instead only stop
backend rules evaluation. This bug exists since the beginning and only
concerns the "allow" rule. It was fixed and http-after-response rulesets
evaluation was also fixed in the same way.
The support for backend aggregated server check status in the Prometheus
exporter was backported. Thanks to this feature, the number of server per
health-check status are now reported at the backend level.
William fixed some bugs in the SSL part. First, outgoing TLS connections
involving SNI couldn't be resumed in TLS 1.3 because the call to
SSL_get_servername() on a resumed connection doesn't return the previous SNI
with TLS 1.3. Then, the wrong error was reported during SSL handshake when a
non-matching SNI was found with the strict-sni option enabled because the
clientHello callback was returning with a success code. An "handshake
failure" was reported instead of "unrecognized name". As a side effect of
this bug, the connections was accepted in case of TLS resume. Finally,
thanks to Willy, the SSL counter are now atomically updated.
The detection of the need for libatomic in the makefile was modified so that
it's not hard-coded on the architecture but instead detects what the
compiler says it needs. This allowed to remove the arm/aarch64 hacks on
linux and also allows MIPS and RISCV to work as expected. In addition it's
now trivial to force it if desired.
In addition, the usual bunch of some of small fixes and cleanups.
The 2.3.16 will be emitted quite soon. The next 2.2 and 2.0 releases are
planned for the next week.
Thanks everyone for your help and your contributions!
Please find the usual URLs below :
Site index : http://www.haproxy.org/
Discourse: http://discourse.haproxy.org/
Slack channel: https://slack.haproxy.org/
Issue tracker: https://github.com/haproxy/haproxy/issues
Wiki : https://github.com/haproxy/wiki/wiki
Sources : http://www.haproxy.org/download/2.4/src/
Git repository : http://git.haproxy.org/git/haproxy-2.4.git/
Git Web browsing : http://git.haproxy.org/?p=haproxy-2.4.git
Changelog: http://www.haproxy.org/download/2.4/src/CHANGELOG
Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/
---
Complete changelog :
Amaury Denoyelle (7):
MINOR: mux-h2: add trace on extended connect usage
BUG/MEDIUM: mux-h2: reject upgrade if no RFC8441 support
MINOR: stream/mux: implement websocket stream flag
MINOR: connection: implement function to update ALPN
MINOR: connection: add alternative mux_ops param for conn_install_mux_be
MEDIUM: server/backend: implement websocket protocol selection
MINOR: server: add ws keyword
Christopher Faulet (10):
DOC: config: Fix
[ANNOUNCE] haproxy-2.3.16
Hi, HAProxy 2.3.16 was released on 2021/11/24. It added 18 new commits after version 2.3.15. As announced for the 2.4.9, this release contains fixes about hidden bugs recently exposed about the shutdowns management at the conn-stream level. The client connections close could be delayed by the client timeout. In addition, because of a failed backport, affecting the 2.2 too, H1 responses could be truncated. All these bugs was fixed. The H2 multiplexer fix to drains data and be sure to send GOAWAY frame was finally backported. It was erroneously announced for the 2.3.15. As side effect, the caching of TLS sessions is now fixed for H2 connections. Still on the H2 multiplexer, an incomplete old fix for H2 partial frames was fixed. It caused some high CPU usages in h2_io_cb() on some rare occasions. Issues reported about occasional crashed in the cache (#1284 and #1451) was fixed. A missing break statement was the explanation. A bug with the "program" post-parser was fixed. It could be called with an empty programs list in case of a config parsing error on reload after another error, and could crash. http-response rulesets evaluation was not aligned with what is said in the documentation. It was possible to inhibit the frontend rules evaluation with an "allow" rule in the backend section while it should instead only stop backend rules evaluation. This bug exists since the beginning and only concerns the "allow" rule. It was fixed and http-after-response rulesets evaluation was also fixed in the same way. William's fixes about the SSL was backported. First, outgoing TLS connections involving SNI can now be resumed in TLS 1.3. Then, the right error is not reported during SSL handshake when a non-matching SNI is found with the strict-sni option enabled. A "unrecognized name" error is returned instead of "handshake failure". As a side effect, this fixes the TLS resume for non-matching SNI, rejecting the connections. Thanks everyone for your help and your contributions! Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse: http://discourse.haproxy.org/ Slack channel: https://slack.haproxy.org/ Issue tracker: https://github.com/haproxy/haproxy/issues Wiki : https://github.com/haproxy/wiki/wiki Sources : http://www.haproxy.org/download/2.3/src/ Git repository : http://git.haproxy.org/git/haproxy-2.3.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-2.3.git Changelog: http://www.haproxy.org/download/2.3/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ --- Complete changelog : Christopher Faulet (8): BUG/MEDIUM: mux-h1: Fix H1C_F_ST_SILENT_SHUT value DOC: config: Fix typo in ssl_fc_unique_id description BUG/MINOR: http-ana: Apply stop to the current section for http-response rules Revert "BUG/MINOR: http-ana: Don't eval front after-response rules if stopped on back" DOC: lua: Be explicit with the Reply object limits BUG/MEDIUM: conn-stream: Don't reset CS flags on close BUG/MINOR: mux-h2: Fix H2_CF_DEM_SHORT_READ value BUG/MINOR: stick-table/cli: Check for invalid ipv6 key William Lallemand (3): BUG/MEDIUM: ssl: backend TLS resumption with sni and TLSv1.3 BUG/MINOR: mworker: doesn't launch the program postparser BUG/MEDIUM: ssl: abort with the correct SSL error when SNI not found Willy Tarreau (7): BUG/MEDIUM: connection: make cs_shutr/cs_shutw//cs_close() idempotent MINOR: connection: add a new CO_FL_WANT_DRAIN flag to force drain on close MINOR: mux-h2: perform a full cycle shutdown+drain on close BUG/MEDIUM: mux-h2: always process a pending shut read BUG/MEDIUM: shctx: leave the block allocator when enough blocks are found BUG/MINOR: shctx: do not look for available blocks when the first one is enough MINOR: shctx: add a few BUG_ON() for consistency checks -- Christopher Faulet
Re: [PATCH 1/1] BUG/MINOR: lua: remove loop initial declarations
Le 11/24/21 à 22:16, Bertrand Jacquin a écrit :
HAProxy is documented to support gcc >= 3.4 as per INSTALL file, however
hlua.c makes use of c11 only loop initial declarations leading to build
failure when using gcc-4.9.4:
x86_64-unknown-linux-gnu-gcc -Iinclude -Wchar-subscripts -Wcomment -Wformat -Winit-self -Wmain
-Wmissing-braces -Wno-pragmas -Wparentheses -Wreturn-type -Wsequence-point -Wstrict-aliasing
-Wswitch -Wtrigraphs -Wuninitialized -Wunknown-pragmas -Wunused-label -Wunused-variable
-Wunused-value -Wpointer-sign -Wimplicit -pthread -fdiagnostics-color=auto -D_LARGEFILE_SOURCE
-D_FILE_OFFSET_BITS=64 -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -O3 -msse -mfpmath=sse
-march=core2 -g -fPIC -g -Wall -Wextra -Wundef -Wdeclaration-after-statement -fwrapv
-Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered
-Wno-missing-field-initializers -Wtype-limits -DUSE_EPOLL -DUSE_NETFILTER -DUSE_PCRE2
-DUSE_PCRE2_JIT -DUSE_POLL -DUSE_THREAD -DUSE_BACKTRACE -DUSE_TPROXY -DUSE_LINUX_TPROXY
-DUSE_LINUX_SPLICE -DUSE_LIBCRYPT -DUSE_CRYPT_H -DUSE_GETADDRINFO -DUSE_OPENSSL -DUSE_LUA
-DUSE_ACCEPT4 -DUSE_SLZ -DUSE_CPU_AFFINITY -DUSE_TFO -DUSE_NS -DUSE_DL -DUSE_RT -DUSE_PRCTL
-DUSE_THREAD_DUMP-DUSE_PCRE2 -DPCRE2_CODE_UNIT_WIDTH=8 -I/usr/local/include
-DCONFIG_HAPROXY_VERSION=\"2.5.0\" -DCONFIG_HAPROXY_DATE=\"2021/11/23\" -c -o
src/connection.o src/connection.c
src/hlua.c: In function 'hlua_config_prepend_path':
src/hlua.c:11292:2: error: 'for' loop initial declarations are only allowed
in C99 or C11 mode
for (size_t i = 0; i < 2; i++) {
^
src/hlua.c:11292:2: note: use option -std=c99, -std=gnu99, -std=c11 or
-std=gnu11 to compile your code
This commit moves loop iterator to an explicit declaration.
No backport needed as this issue was introduced in v2.5-dev10~69 with
commit 9e5e586e35c5 ("BUG/MINOR: lua: Fix lua error handling in
`hlua_config_prepend_path()`")
---
src/hlua.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/hlua.c b/src/hlua.c
index 08735374af77..8dea91e75832 100644
--- a/src/hlua.c
+++ b/src/hlua.c
@@ -11249,6 +11249,7 @@ static int hlua_config_prepend_path(char **args, int
section_type, struct proxy
char *path;
char *type = "path";
struct prepend_path *p = NULL;
+ size_t i;
if (too_many_args(2, args, err, NULL)) {
goto err;
@@ -11289,7 +11290,7 @@ static int hlua_config_prepend_path(char **args, int
section_type, struct proxy
* thread. The remaining threads will be initialized based on
* prepend_path_list.
*/
- for (size_t i = 0; i < 2; i++) {
+ for (i = 0; i < 2; i++) {
lua_State *L = hlua_states[i];
const char *error;
Thanks, merged now with backports info updated.
--
Christopher Faulet
[ANNOUNCE] haproxy-2.2.19
Hi,
HAProxy 2.2.19 was released on 2021/11/29. It added 21 new commits
after version 2.2.18.
Main changes in this release are about the shutdowns management in the H1
and H2 multiplexers to be able to perform a clean shutdown when delayed.
This fixed an issue with caching of TLS sessions. Indeed, with the migration
to the muxes in 1.9-2.0, we've lost the clean shutdown at the end of
connection that's also used to commit the TLS session cache entry. That
allowed too to fix hidden bugs. The shutdowns at the conn-stream level are
now idempotent.
In addition to the above changes, the H2 multiplexer now drains data on
shutdown to be sure to send GOAWAY frame. Among other things, this fixed
issues with h2spec tests in the CI. Finally, an old and incomplete fix for
H2 partial frames causing some high CPU usages in h2_io_cb() on some rare
occasions was fixed.
Some users reported occasional crashes in the cache (#1284 and #1451). We
finally had an explanation (a missing break). This was fixed.
http-response rulesets evaluation was not aligned with what is said in the
documentation. It was possible to inhibit the frontend rules evaluation with
an "allow" rule in the backend section while it should instead only stop
backend rules evaluation. This bug exists since the beginning and only
concerns the "allow" rule. It was fixed and http-after-response rulesets
evaluation was also fixed in the same way.
William fixed some bugs in the SSL part. First, outgoing TLS connections
involving SNI couldn't be resumed in TLS 1.3 because the call to
SSL_get_servername() on a resumed connection doesn't return the previous SNI
with TLS 1.3. Then, the wrong error was reported during SSL handshake when a
non-matching SNI was found with the strict-sni option enabled because the
clientHello callback was returning with a success code. An "handshake
failure" was reported instead of "unrecognized name". As a side effect of
this bug, the connections was accepted in case of TLS resume.
A bug in the validity checks for sample fetch functions was fixed. The
checks were only applied to the frontend capability of a proxy. This means
that using a small set of sample fetch functions (like "be_name()") in
proxies that are both a frontend and a backend ("listen" or "defaults")
would lead to a config error while it is technically valid. This problem has
always been there and never reported.
A bug with the "program" post-parser was fixed. It could be called with an
empty programs list in case of a config parsing error on reload after
another error, and could crash.
In addition, the usual bunch of some of small fixes and cleanups.
The next 2.0 release will be emitted at the end of the week.
Thanks everyone for your help and your contributions!
Please find the usual URLs below :
Site index : http://www.haproxy.org/
Discourse: http://discourse.haproxy.org/
Slack channel: https://slack.haproxy.org/
Issue tracker: https://github.com/haproxy/haproxy/issues
Wiki : https://github.com/haproxy/wiki/wiki
Sources : http://www.haproxy.org/download/2.2/src/
Git repository : http://git.haproxy.org/git/haproxy-2.2.git/
Git Web browsing : http://git.haproxy.org/?p=haproxy-2.2.git
Changelog: http://www.haproxy.org/download/2.2/src/CHANGELOG
Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/
---
Complete changelog :
Christopher Faulet (11):
BUG/MEDIUM: stream-int: Block reads if channel cannot receive more data
BUG/MEDIUM: sample: Cumulate frontend and backend sample validity flags
BUG/MEDIUM: mux-h1: Fix H1C_F_ST_SILENT_SHUT value
DOC: config: Fix typo in ssl_fc_unique_id description
BUG/MINOR: http-ana: Apply stop to the current section for http-response
rules
Revert "BUG/MINOR: http-ana: Don't eval front after-response rules if stopped
on back"
DOC: lua: Be explicit with the Reply object limits
BUG/MEDIUM: conn-stream: Don't reset CS flags on close
BUG/MINOR: mux-h2: Fix H2_CF_DEM_SHORT_READ value
BUG/MINOR: stick-table/cli: Check for invalid ipv6 key
CLEANUP: ssl: Release cached SSL sessions on deinit
William Lallemand (3):
BUG/MINOR: mworker: doesn't launch the program postparser
BUG/MEDIUM: ssl: backend TLS resumption with sni and TLSv1.3
BUG/MEDIUM: ssl: abort with the correct SSL error when SNI not found
Willy Tarreau (7):
BUG/MINOR: sample: fix backend direction flags consecutive to last fix
BUG/MEDIUM: connection: make cs_shutr/cs_shutw//cs_close() idempotent
MINOR: connection: add a new CO_FL_WANT_DRAIN flag to force drain on close
MINOR: mux-h2: perform a full cycle shutdown+drain on close
BUG/MEDIUM: mux-h2: always process a pending shut read
BUG/MEDIUM: shctx: leave the block allocator when enough blocks are found
BUG/MINOR: shctx: do not look for available blocks when the first one is
enough
--
Christopher Faulet
Re: default-server behavior different in 2.2 vs. 2.4?
Le 11/29/21 à 09:56, Christian Ruppert a écrit :
Hey,
we have something like:
server maint 192.168.70.98:80 weight 1 backup non-stick
default-server check maxconn 100 ssl verify required sni
str(somestr) ca-file /etc/ssl/certs/ca-certificates.crt observe layer7
server s200010 192.168.200.10:8443 cookie somecookie weight 100
check addr 127.0.0.1 port 62041 check inter 1 fall 2 rise 2
...
In 2.2 (2.2.17) it is totally valid and the stats say the "maint" is
still "no check". In 2.4.9 the config verify fails. In 2.2 it seems to
only affects server's below default-server.
[ALERT](23456) : Proxy '...', server 'maint'
[/etc/haproxy/haproxy.cfg:104096] verify is enabled by default but no CA
file specified. If you're running on a LAN where you're certain to trust
the server's certificate, please set an explicit 'verify none' statement
on the 'server' line, or use 'ssl-server-verify none' in the global
section to disable server-side verifications by default.
We're using templates and share the maint server with several hundred
other listeners/backends. Only 5 are using a config like this here, with
SSL and verify.
Another problem here seems: "verify is enabled by default but no CA file
specified" while in fact it is?
Is this intended?
Indeed. The change was introduced by the commit f63704488e ("MEDIUM: cli/ssl:
configure ssl on server at runtime"). During the post-parsing stage, when the
configuration is validated, we rely on the last "default-server" line parsed in
the backend section to finalize the SSL configuration of servers in this
backend. It is of course a bug. We may probably rely on a flag instead. I'll
investigate.
Thanks !
--
Christopher Faulet
Re: default-server behavior different in 2.2 vs. 2.4?
Le 11/29/21 à 09:56, Christian Ruppert a écrit :
Hey,
we have something like:
server maint 192.168.70.98:80 weight 1 backup non-stick
default-server check maxconn 100 ssl verify required sni
str(somestr) ca-file /etc/ssl/certs/ca-certificates.crt observe layer7
server s200010 192.168.200.10:8443 cookie somecookie weight 100
check addr 127.0.0.1 port 62041 check inter 1 fall 2 rise 2
...
In 2.2 (2.2.17) it is totally valid and the stats say the "maint" is
still "no check". In 2.4.9 the config verify fails. In 2.2 it seems to
only affects server's below default-server.
[ALERT](23456) : Proxy '...', server 'maint'
[/etc/haproxy/haproxy.cfg:104096] verify is enabled by default but no CA
file specified. If you're running on a LAN where you're certain to trust
the server's certificate, please set an explicit 'verify none' statement
on the 'server' line, or use 'ssl-server-verify none' in the global
section to disable server-side verifications by default.
We're using templates and share the maint server with several hundred
other listeners/backends. Only 5 are using a config like this here, with
SSL and verify.
Another problem here seems: "verify is enabled by default but no CA file
specified" while in fact it is?
Is this intended?
I pushed a patch (https://github.com/haproxy/haproxy/commit/4ab26796) that
should fix this issue. It will be backported ASAP. In the mean time, the patch
for the 2.4 is attached to this mail.
--
Christopher FauletFrom 7f689d5265a973d66b702e57ba02e866c1db400c Mon Sep 17 00:00:00 2001
From: Christopher Faulet
Date: Wed, 1 Dec 2021 09:50:41 +0100
Subject: [PATCH] BUG/MINOR: server: Don't rely on last default-server to init
server SSL context
During post-parsing stage, the SSL context of a server is initialized if SSL
is configured on the server or its default-server. It is required to be able
to enable SSL at runtime. However a regression was introduced, because the
last parsed default-server is used. But it is not necessarily the
default-server line used to configure the server. This may lead to
erroneously initialize the SSL context for a server without SSL parameter or
the skip it while it should be done.
The problem is the default-server used to configure a server is not saved
during configuration parsing. So, the information is lost during the
post-parsing. To fix the bug, the SRV_F_DEFSRV_USE_SSL flag is
introduced. It is used to know when a server was initialized with a
default-server using SSL.
For the record, the commit f63704488e ("MEDIUM: cli/ssl: configure ssl on
server at runtime") has introduced the bug.
This patch must be backported as far as 2.4.
---
include/haproxy/server-t.h | 3 +++
reg-tests/server/cli_set_ssl.vtc | 20 +---
src/cfgparse.c | 4 ++--
src/server.c | 4
4 files changed, 22 insertions(+), 9 deletions(-)
diff --git a/include/haproxy/server-t.h b/include/haproxy/server-t.h
index 4291953888..402ea79f0a 100644
--- a/include/haproxy/server-t.h
+++ b/include/haproxy/server-t.h
@@ -149,6 +149,9 @@ enum srv_initaddr {
#define SRV_F_SOCKS4_PROXY 0x0400/* this server uses SOCKS4 proxy */
#define SRV_F_NO_RESOLUTION 0x0800 /* disable runtime DNS resolution on this server */
#define SRV_F_DYNAMIC 0x1000/* dynamic server instantiated at runtime */
+/* 0x2000 unused */
+#define SRV_F_DEFSRV_USE_SSL 0x4000 /* default-server uses SSL */
+
/* configured server options for send-proxy (server->pp_opts) */
#define SRV_PP_V1 0x0001 /* proxy protocol version 1 */
diff --git a/reg-tests/server/cli_set_ssl.vtc b/reg-tests/server/cli_set_ssl.vtc
index 638debea0d..f97bf4de3a 100644
--- a/reg-tests/server/cli_set_ssl.vtc
+++ b/reg-tests/server/cli_set_ssl.vtc
@@ -27,8 +27,9 @@ haproxy h1 -conf {
default_backend test0
backend test0
-default-server ssl
server www0 ${s1_addr}:${s1_port} no-ssl
+default-server ssl
+server www1 ${s1_addr}:${s1_port} no-ssl
backend test1
server www0 ${s1_addr}:${s1_port} no-ssl
@@ -37,17 +38,22 @@ haproxy h1 -conf {
haproxy h1 -cli {
# supported case
send "show servers state test0"
-expect ~ "test0 1 www0 ${s1_addr} .* - ${s1_port} - -1"
-send "set server test0/www0 ssl on"
+expect ~ "test0 2 www1 ${s1_addr} .* - ${s1_port} - -1"
+send "set server test0/www1 ssl on"
expect ~ "server ssl setting updated"
send "show servers state test0"
-expect ~ "test0 1 www0 ${s1_addr} .* - ${s1_port} - 1"
-send "set server test0/www0 ssl off"
+expect ~ "test0 2 www1 ${s1_addr} .* - ${s1_port} - 1"
+send "set server test0/www1 ssl off
Re: [PATCH] BUG/MEDIUM: sample: Fix memory leak in sample_conv_jwt_member_query
Le 12/1/21 à 23:04, Tim Duesterhus a écrit : The function leaked one full buffer per invocation. Fix this by simply removing the call to alloc_trash_chunk(), the static chunk from get_trash_chunk() is sufficient. This bug was introduced in 0a72f5ee7c2a61bdb379436461269315c776b50a, which is 2.5-dev10. This fix needs to be backported to 2.5+. --- src/sample.c | 4 1 file changed, 4 deletions(-) diff --git a/src/sample.c b/src/sample.c index 5abf4712a..63816be5d 100644 --- a/src/sample.c +++ b/src/sample.c @@ -3584,10 +3584,6 @@ static int sample_conv_jwt_member_query(const struct arg *args, struct sample *s if (item_num < member + 1) goto end; - decoded_header = alloc_trash_chunk(); - if (!decoded_header) - goto end; - ret = base64urldec(items[member].start, items[member].length, decoded_header->area, decoded_header->size); if (ret == -1) Thanks, merged now ! -- Christopher Faulet
Re: Is it expected that "capture response" does not get headers when "http-request return" is used
Le 12/4/21 à 13:25, Aleksandar Lazic a écrit :
Hi.
I try to capture the response header "dst_conn" from "http-request return" but
in %hs isn't the value.
```
podman logs -f haproxy-dest
[NOTICE] (1) : New worker #1 (3) forked
<6>[04/Dec/2021:12:14:34.437] 200 58 - - LR-- {} "GET / HTTP/1.1"
<6>[04/Dec/2021:12:14:34.437] 200 58 - - LR-- {} "GET / HTTP/1.1"
<6>[04/Dec/2021:12:14:34.438] 200 58 - - LR-- {} "GET / HTTP/1.1"
```
I haven't seen any "capture" in "http-after-response".
The question is also makes sense to have a capture after "http-request return"
as in the documenation is
written that return stops the evaluation of any other rules also from capture?
http://cbonte.github.io/haproxy-dconv/2.4/configuration.html#http-response%20return
"This stops the evaluation of the rules and immediately returns a response."
Hi Alex,
Unfortunately, it is indeed not possible for now. First, the captures via
"capture request" and "capture response" directives are performed very early,
and on received messages only. Thus it is not possible to capture info from
generated responses at this stage. However, it is probably possible to add a
"capture" action to the "http-afer-response" ruleset. This would able to you to
capture your header with the following config:
declare capture response len 4
http-after-response capture hdr(dst_conn) id 0
At first glance it seems trivial. I will check that.
--
Christopher Faulet
Re: Is it expected that "capture response" does not get headers when "http-request return" is used
Le 12/6/21 à 08:25, Christopher Faulet a écrit :
Le 12/4/21 à 13:25, Aleksandar Lazic a écrit :
Hi.
I try to capture the response header "dst_conn" from "http-request return" but
in %hs isn't the value.
```
podman logs -f haproxy-dest
[NOTICE] (1) : New worker #1 (3) forked
<6>[04/Dec/2021:12:14:34.437] 200 58 - - LR-- {} "GET / HTTP/1.1"
<6>[04/Dec/2021:12:14:34.437] 200 58 - - LR-- {} "GET / HTTP/1.1"
<6>[04/Dec/2021:12:14:34.438] 200 58 - - LR-- {} "GET / HTTP/1.1"
```
I haven't seen any "capture" in "http-after-response".
The question is also makes sense to have a capture after "http-request return"
as in the documenation is
written that return stops the evaluation of any other rules also from capture?
http://cbonte.github.io/haproxy-dconv/2.4/configuration.html#http-response%20return
"This stops the evaluation of the rules and immediately returns a response."
Hi Alex,
Unfortunately, it is indeed not possible for now. First, the captures via
"capture request" and "capture response" directives are performed very early,
and on received messages only. Thus it is not possible to capture info from
generated responses at this stage. However, it is probably possible to add a
"capture" action to the "http-afer-response" ruleset. This would able to you to
capture your header with the following config:
declare capture response len 4
http-after-response capture hdr(dst_conn) id 0
At first glance it seems trivial. I will check that.
Hi,
I added it to 2.6-DEV. The patch is small enough to be backported to 2.5.
--
Christopher Faulet
Re: HAP 2.3.16 A bogus STREAM [0x559faa07b4f0] at "cache store filter"
Le 12/25/21 à 23:59, Aleksandar Lazic a écrit :
Hi.
as the message tell us that we should report this to the developers I do so :-)
```
Dec 24 01:10:31 lb1 haproxy[20008]: A bogus STREAM [0x559faa07b4f0] is spinning
at 204371 calls per second
and refuses to die, aborting now!
Please report this error to developers
[strm=0x559faa07b4f0,12390e src=:::79.183.184.235 fe=https-in be=be_api
dst=api_main2
txn=0x559faab233e0,44000 txn.req=MSG_DONE,d txn.rsp=MSG_RPBEFORE,0
rqf=48c4e068 rqa=4
rpf=a000a860 rpa=0 sif=CLO,2c8002 sib=CLO,1280112 af=(nil),0
csf=0x559faa07ba10,1059a0
ab=(nil),0 csb=0x559faad7dcf0,1a0
cof=0x7f224212e5d0,80003300:H2(0x559faa7d7b00)/SSL(0x7f22424fc7a0)/tcpv6(2162)
cob=0x7f2240f79fe0,8982300:H1(0x559faa0ab840)/SSL(0x7f2263517770)/tcpv4(1490)
filters={0x559faa29c520="cache store filter"}]
Hi Alex,
I think I found the issue. I'm unable to reproduce the spinning loop but I can
freeze infinitely a stream. It is probably just a matter of timing. On my side,
it is related to L7 retries. Could you confirm you have a "retry-on" parameter
in your configuration ?
Thanks !
--
Christopher Faulet
Re: HAP 2.3.16 A bogus STREAM [0x559faa07b4f0] at "cache store filter"
Le 1/4/22 à 10:26, Aleksandar Lazic a écrit :
On 04.01.22 10:16, Christopher Faulet wrote:
Le 12/25/21 à 23:59, Aleksandar Lazic a écrit :
Hi.
as the message tell us that we should report this to the developers I do so :-)
```
Dec 24 01:10:31 lb1 haproxy[20008]: A bogus STREAM [0x559faa07b4f0] is spinning
at 204371 calls per second
and refuses to die, aborting now!
Please report this error to developers
[strm=0x559faa07b4f0,12390e src=:::79.183.184.235 fe=https-in be=be_api
dst=api_main2
txn=0x559faab233e0,44000 txn.req=MSG_DONE,d txn.rsp=MSG_RPBEFORE,0
rqf=48c4e068 rqa=4
rpf=a000a860 rpa=0 sif=CLO,2c8002 sib=CLO,1280112 af=(nil),0
csf=0x559faa07ba10,1059a0
ab=(nil),0 csb=0x559faad7dcf0,1a0
cof=0x7f224212e5d0,80003300:H2(0x559faa7d7b00)/SSL(0x7f22424fc7a0)/tcpv6(2162)
cob=0x7f2240f79fe0,8982300:H1(0x559faa0ab840)/SSL(0x7f2263517770)/tcpv4(1490)
filters={0x559faa29c520="cache store filter"}]
Hi Alex,
I think I found the issue. I'm unable to reproduce the spinning loop but I can
freeze infinitely a stream.
It is probably just a matter of timing. On my side, it is related to L7
retries. Could you confirm you have
a "retry-on" parameter in your configuration ?
Yes I can confirm.
```
defaults http
log global
mode http
retry-on all-retryable-errors
option forwardfor
option redispatch
option http-ignore-probes
option httplog
option dontlognull
option ssl-hello-chk
option log-health-checks
option socket-stats
timeout connect 5s
timeout client 50s
timeout server 50s
http-reuse safe
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
...
```
Thanks Alex, I pushed a fix. It will be backported as far as the 2.0 ASAP.
--
Christopher Faulet
Re: Issue with uploads and HAProxy 2.4.11
Le 1/10/22 à 14:46, Christian Ruppert a écrit :
On 2022-01-10 13:33, Sander Klein wrote:
Hi,
I've upgraded to HAProxy 2.4.11 and now I seem to have a problem with
bigger file uploads (>70MB).
When uploading a file I get a 500 back from HAProxy, and if I retry it
immediately it most of the time succeeds. Downgrading to 2.4.10 fixes
the issue. The log I get is:
Jan 10 12:09:45 [redacted] haproxy[21823]: 2001:67c:[redacted]
[10/Jan/2022:12:09:20.543] [redacted]~ [redacted]/[redacted]
11198/0/0/-1/25137 500 1991 - - IH-- 957/282/0/0/0 0/0
{[redacted].[redacted].com|Mozilla/5.0
(Mac|80349066|https://[redacted].[redacted].com/upload} {} “POST
https://[redacted].[redacted].com/upload/process?projectId=3431&setId=149
HTTP/2.0”
The frontend is HTTP/2.0 and the backend is NGINX talking HTTP/1.1
(non-TLS).
The config is quite large, but I think it boils down to:
---
frontend [redacted]
bind [redactes]]:80 transparent
bind 2001:67c:[redacted]:80 transparent
bind [redacted]:443 transparent ssl crt /etc/haproxy/ssl/[redacted]
strict-sni alpn h2,http/1.1 npn h2,http/1.1
bind 2001:67c:[redacted]:443 transparent ssl crt
/etc/haproxy/ssl/[redacted] strict-sni alpn h2,http/1.1 npn
h2,http/1.1
mode http
maxconn 16384
option httplog
option dontlog-normal
option http-ignore-probes
option forwardfor
option http-buffer-request
capture request header Host len 64
capture request header User-Agent len 16
capture request header Content-Length len 10
capture request header Referer len 256
capture response header Content-Length len 10
acl [some ACLs here]
acl [some ACLs here]
http-request deny if [an ACL]
http-request deny if [another ACL]
use_backend [failing-backend] if [ACL]
use_backend
%[req.hdr(host),lower,regsub(^www\.,,i),map(/etc/haproxy/map.d/file.map,yes-backend)]
default_backend another-backend
backend failing-backend
fullconn256
modehttp
balance roundrobin
option abortonclose
option prefer-last-server
option redispatch
option httpchk GET /check-thingy HTTP/1.0
http-check expect status 200
default-server weight 100 maxconn 20 check inter 2s rise 3 fall 3
slowstart 5m agent-check agent-port 8081 agent-inter 20s
server server1 [redacted]:80 cookie cookie1
server server2 [redacted]:80 cookie cookie2
# Sorry Server
server outage 127.0.0.1:80 backup
retries 1
---
If any more info is needed, please let me know.
Regards,
Sander Klein
Hi Sander,
this might be also related to
https://github.com/haproxy/haproxy/issues/1510
Thanks Christian. Indeed, it is most probably the same issue. The proposed patch
is in attachment. The 2.6 and 2.5 are also affected.
--
Christopher FauletFrom 2984ccfc1f5c4ca1157f7a498dcdc7de2f7ba934 Mon Sep 17 00:00:00 2001
From: Christopher Faulet
Date: Mon, 10 Jan 2022 17:27:51 +0100
Subject: [PATCH] BUG/MAJOR: mux-h1: Don't decrement .curr_len for unsent data
A regression was introduced by commit 140f1a585 ("BUG/MEDIUM: mux-h1: Fix
splicing by properly detecting end of message"). To detect end of the
outgoing message, when the content-length is announced, we count amount of
data already sent. But only data really sent must be counted.
If the output buffer is full, we can fail to send data (fully or
partially). In this case, we must take care to only count sent
data. Otherwise we may think too much data were sent and an internal error
may be erroneously reported.
This patch should solve issue #1511. It must be backported as far as 2.4.
---
src/mux_h1.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/src/mux_h1.c b/src/mux_h1.c
index f9a6120fe4..c2dc80834d 100644
--- a/src/mux_h1.c
+++ b/src/mux_h1.c
@@ -2180,7 +2180,6 @@ static size_t h1_process_output(struct h1c *h1c, struct buffer *buf, size_t coun
H1_EV_TX_DATA|H1_EV_STRM_ERR|H1_EV_H1C_ERR|H1_EV_H1S_ERR, h1c->conn, h1s);
goto error;
}
- h1m->curr_len -= vlen;
}
if ((h1m->flags & H1_MF_RESP) && (h1s->flags & H1S_F_BODYLESS_RESP)) {
TRACE_PROTO("Skip data for bodyless response", H1_EV_TX_DATA|H1_EV_TX_BODY, h1c->conn, h1s, chn_htx);
@@ -2226,6 +2225,8 @@ static size_t h1_process_output(struct h1c *h1c, struct buffer *buf, size_t coun
H1_EV_TX_DATA|H1_EV_TX_BODY, h1c->conn, h1s, 0, (size_t[]){v.len});
skip_data:
+if (h1m->state == H1_MSG_DATA && (h1m->flags & H1_MF_CLEN))
+ h1m->curr_len -= vlen;
if (last_data)
goto done;
break;
--
2.33.1
[ANNOUNCE] haproxy-2.4.12
Hi, HAProxy 2.4.12 was released on 2022/01/11. It added 2 new commits after version 2.4.11. The 2.4.11 introduced a major regression into the H1 multiplexer. The bug affects all HTTP messages announcing a content length from the time there are some contentions on the output buffer. The result is that an internal error is erroneously reported when HAproxy tries to emit the last block of payload and the message is unexpectedly truncated. In addition, William fixed a bug in the master-worker when the master is executed in wait mode. In this case, the master must never try to to get the listeners FD from the previous process using _getsocks on the stat socket. Otherwise, if a reload fails, the master exists with a EXIT_FAILURE status, killing all the workers. A 2.5.1 will be emitted very soon with these fixes (and many others). However the 2.5.0 is not affected by the first issue. The 2.4.12 was emitted first to avoid any trouble for anyone who would like to update or who have already upgraded. Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse: http://discourse.haproxy.org/ Slack channel: https://slack.haproxy.org/ Issue tracker: https://github.com/haproxy/haproxy/issues Wiki : https://github.com/haproxy/wiki/wiki Sources : http://www.haproxy.org/download/2.4/src/ Git repository : http://git.haproxy.org/git/haproxy-2.4.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-2.4.git Changelog: http://www.haproxy.org/download/2.4/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ --- Complete changelog : Christopher Faulet (1): BUG/MAJOR: mux-h1: Don't decrement .curr_len for unsent data William Lallemand (1): BUG/MEDIUM: mworker: don't use _getsocks in wait mode -- Christopher Faulet
Re: [PATCH] BUG/MEDIUM: server: avoid changing healthcheck ctx with set server ssl
Le 1/10/22 à 23:19, Willy Tarreau a écrit : w options were still configurable on the CLI by then. "check-ssl" has been available for a long time, so that's not the reason behind it, but I guess you were referring to something else. I suspect I did a dumb copy/paste from the new_server function and probably never thought was possibly wrong as my previous production never had any check using tls. Maybe but then I don't remember about all the detailed rules in place that indicate when check-ssl *has* to be used. I'll have to read the doc. For a health-check, if no port or address is specified and no transport layer is forced, then the transport layer used by the check is the same than for the production traffic. So, the same must be done for dynamic changes. But it is not so simple because, when the check inherits from the server settings, "srv->check.use_ssl" is also changed. I don't remember why this field is updated. But this may prevent any dynamic change on healtcheck. I must read the code to be sure. -- Christopher Faulet
[ANNOUNCE] haproxy-2.5.1
uild with libreSSL 3.5 and newer (some macros didn't work anymore). - David Carlier fixed the build with FreeBSD 14, which changes the cpuset API to better match Linux's. - a small improvement, in order to help users provide exploitable cores, there's now a new command-line option "-dL" which dumps the dynamic libraries that were detected at run time just before forking. This possibly includes dependencies from Lua or various other libs that do not always appear in "ldd". Typically libgcc_s is listed. The output format allows to pipe that to tar to produce an archive of all executable code that apparently tends to open well with a core, irrelevant to the distros in use. Since it eases bug reports, we've decided to backport it. - another build issue, this time with clang on i386. It tries to make use of the CMPXCHG8B instruction to perform 64-bit atomics but incorrectly expects the operands to be 64-bit aligned while neither the ABI nor the instruction have this requirement. So basically it complains about the code it produces itself. The analysis showed that working around this would require tens to hundreds of isolated hacks and that the least dirty solution is to disable the warning. Firefox faced the same issue 3 years ago and adopted the same work around. I guess nobody's interested anymore in i386 for anyone to expect a fix there anyway. - a workaround for a possibly slow malloc_trim() in modern libcs upon reload when using many threads, that could be slow enough to panic the old process. - a few other build fixes and doc fixes. - "show version" command was added to display the current process version. - "capture" action is now supported in http-after-response rulesets. - Empty lines were removed from "show ssl ocsp-response" command output. Thanks everyone for your help and your contributions! Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse: http://discourse.haproxy.org/ Slack channel: https://slack.haproxy.org/ Issue tracker: https://github.com/haproxy/haproxy/issues Wiki : https://github.com/haproxy/wiki/wiki Sources : http://www.haproxy.org/download/2.5/src/ Git repository : http://git.haproxy.org/git/haproxy-2.5.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-2.5.git Changelog: http://www.haproxy.org/download/2.5/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ --- Complete changelog : Bertrand Jacquin (1): BUG/MINOR: lua: remove loop initial declarations Christopher Faulet (15): BUG/MINOR: cache: Fix loop on cache entries in "show cache" BUG/MEDIUM: cli: Properly set stream analyzers to process one command at a time BUG/MINOR: server: Don't rely on last default-server to init server SSL context BUG/MEDIUM: resolvers: Detach query item on response error BUG/MEDIUM: h1: Properly reset h1m flags when headers parsing is restarted MINOR: mux-h1: Improve H1 traces by adding info about http parsers BUILD: bug: Fix error when compiling with -DDEBUG_STRICT_NOCRASH MINOR: http-rules: Add capture action to http-after-response ruleset BUG/MINOR: cli/server: Don't crash when a server is added with a custom id DOC: spoe: Clarify use of the event directive in spoe-message section DOC: config: Specify %Ta is only available in HTTP mode BUG/MEDIUM: mux-h1: Fix splicing by properly detecting end of message BUG/MINOR: mux-h1: Fix splicing for messages with unknown length BUG/MEDIUM: http-ana: Preserve response's FLT_END analyser on L7 retry BUG/MAJOR: mux-h1: Don't decrement .curr_len for unsent data Daniel Jakots (1): BUILD: ssl: unbreak the build with newer libressl David CARLIER (3): MINOR: cpuset: switch to sched_setaffinity for FreeBSD 14 and above. BUILD/MINOR: cpuset FreeBSD 14 build fix. BUILD: cpuset: fix build issue on macos introduced by previous change David Carlier (1): BUILD/MINOR: tools: solaris build fix on dladdr. Emeric Brun (1): BUG/MAJOR: segfault using multiple log forward sections. Ilya Shipitsin (3): CI: Github Actions: do not show VTest failures if build failed CI: github actions: update OpenSSL to 3.0.1 CI: github actions: clean default step conditions Lukas Tribus (2): DOC: config: retry-on list is space-delimited DOC: config: fix error-log-format example Miroslav Zagorac (1): BUILD: opentracing: display warning in case of using OT_USE_VARS at compile time Remi Tricot-Le Breton (3): BUG/MINOR: vars: Fix the set-var and unset-var converters MINOR: ssl: Remove empty lines from "show ssl ocsp-response" output BUG/MINOR:
Re: [PATCH] BUG/MEDIUM: server: avoid changing healthcheck ctx with set server ssl
Le 1/10/22 à 23:19, Willy Tarreau a écrit :
At this point I'm starting to think that we should probably avoid as
much as possible to use implicit settings for whatever is dynamic.
Originally a lot of settings were implicit because we don't want to
have huge config lines to enforce lots of obvious settings. On the CLI
it's less of a problem and for example if "ssl" only deals with the
traffic without manipulating the checks, I'm personally not shocked,
because these are normally sent by bots and we don't care if they
have to set a few more settings to avoid multiple implicit changes
that may not always be desired. This is where the CLI (or any future
API) might differ a bit from the config, and an agent writing some
config might have to explicitly state certain things like "no-check-ssl"
for example to stay safe and avoid such implicit dependencies.
I agree with Willy on this point. Especially because, depending the order of
commands, the result can be different because of implicit changes and it is
pretty hard to predict how it will behave in all cases.
For instance, to fix William's bug about "set server ssl" command, in a way or
another, we must stop to dynamically update the health-check if its port or its
address is explicitly specified. With this change, the result of following set
of commands will be different:
$ set server BK/SRV ssl on
$ set server BK/SRV check-port XXX
==> SSL is enabled for the server and the health-check
$ set server BK/SRV check-port XXX
$ set server BK/SRV ssl on
==> because the check's port was updated first, the SSL is only enabled for the
server.
Note that such a brainstorming doesn't apply to your fix and should
not hold it from being merged in any way, I'm just speaking in the
general sense, as I don't feel comfortable with keep all these special
cases in a newly introduced API, that are only there for historical
reasons.
Agree. But, if possible, a warning may be added in the documentation to warn
about implicit changes.
About the fix, I checked the code, and, at first glance, there is no reason to
change "srv->check.use_ssl" value when the health-check uses the server
settings. Thus, we may fix "set server ssl" command this way:
diff --git a/src/check.c b/src/check.c
index f0ae81504..8cf8a1c5b 100644
--- a/src/check.c
+++ b/src/check.c
@@ -1565,10 +1565,8 @@ int init_srv_check(struct server *srv)
* default, unless one is specified.
*/
if (!srv->check.port && !is_addr(&srv->check.addr)) {
- if (!srv->check.use_ssl && srv->use_ssl != -1) {
- srv->check.use_ssl = srv->use_ssl;
- srv->check.xprt= srv->xprt;
- }
+ if (!srv->check.use_ssl && srv->use_ssl != -1)
+ srv->check.xprt = srv->xprt;
else if (srv->check.use_ssl == 1)
srv->check.xprt = xprt_get(XPRT_SSL);
srv->check.send_proxy |= (srv->pp_opts);
diff --git a/src/server.c b/src/server.c
index 2061153bc..a18836a71 100644
--- a/src/server.c
+++ b/src/server.c
@@ -2113,10 +2113,11 @@ void srv_set_ssl(struct server *s, int use_ssl)
return;
s->use_ssl = use_ssl;
- if (s->use_ssl)
- s->xprt = xprt_get(XPRT_SSL);
- else
- s->xprt = s->check.xprt = s->agent.xprt = xprt_get(XPRT_RAW);
+ s->xprt = (s->use_ssl == 1) ? xprt_get(XPRT_SSL) : xprt_get(XPRT_RAW);
+
+ if ((s->check.state & CHK_ST_CONFIGURED) && !s->check.use_ssl &&
+ !s->check.port && !is_addr(&s->check.addr))
+ s->check.xprt = s->xprt;
}
#endif /* USE_OPENSSL */
This may be done with the following 3 steps:
* First, stop to change "srv->check.use_ssl" value
* Then, stop to change the agent settings in srv_set_ssl() because there is
no ssl support for agent-check.
* Finally, fix the bug identified by William, adding the according
documentation.
Note I don't know if all this stuff works properly with the server-state file...
--
Christopher Faulet
Re: [PATCH] BUG/MEDIUM: server: avoid changing healthcheck ctx with set server ssl
Le 1/11/22 à 22:47, William Dauchy a écrit : Agree. But, if possible, a warning may be added in the documentation to warn about implicit changes. From the discussion, I would be tempted to say the opposite, as I feel like keeping implicit things for this command is worse. I don't know what is the expected behavior on the stable releases for users. The actual state is buggy because health-check are only updated when ssl is disabled. When SSL is enabled on a server, there is no implicit change on the check. when SSL is disabled, there is an implicit change. So we must first state for the expected behavior on stable releases. But, keep in mind there is no way to dynamically enable/disable SSL on health-checks. So if a user configures the SSL on its server but disables it on startup, when he enables ssl, he probably want to do so on the health-check, explicitly or not. as mentioned above I am not sure I am aligned here; I would rather remove the side effect of changing s->check. Honestly, I've misread you patch and kept in mind you alternative solution... But as said, if there is no implicit change (and I'm fine with this solution), a new command or an option to current ones must be introduced to be able to change SSL setting for health-check too. Otherwise, it will be unusable. Note I don't know if all this stuff works properly with the server-state file... I am always scared to look at the server state functionality... I truly understand :( ... -- Christopher Faulet
Re: invalid request
Le 1/13/22 à 02:57, Aleksandar Lazic a écrit : On 12.01.22 21:52, Andrew Anderson wrote: On Wed, Jan 12, 2022 at 11:58 AM Aleksandar Lazic mailto:al-hapr...@none.at>> wrote: Well, looks like you want a forward proxy like squid not a reverse proxy like haproxy. The application being load balanced is a proxy, so http_proxy is not a good fit (and as you mention on the deprecation list), but haproxy as a load balancer is a much better at front-ending this environment than any other solution available. We upgraded to 2.4 recently, and a Java application that uses these proxy servers is what exposed this issue for us. Even if we were to use squid, we would still run into this, as I would want to ensure that squid was highly available for the environment, and we would hit the same code path when going through haproxy to connect to squid. The only option currently available in 2.4 that I am aware of is to setup internal-only frontend/backend paths with accept-invalid-http-request configured on those paths exclusively for Java clients to use. This is effectively how we have worked around this for now: listen proxy bind :8080 mode http option httplog server proxy1 192.0.2.1:8080 server proxy2 192.0.2.2:8080 listen proxy-internal bind :8081 mode http option httplog option accept-invalid-http-request server proxy1 192.0.2.1:8080 track proxy/proxy1 server proxy2 192.0.2.2:8080 track proxy/proxy2 This is a viable workaround for us in the short term, but this would not be a solution that would work for everyone. If the uri parser patches I found in the 2.5/2.6 branches are the right ones to make haproxy more permissive on matching the authority with the host in CONNECT requests, that will remove the need for the parallel frontend/backends without validation enabled. I hope to be able to have time to test a 2.4 build with those patches included over the next few days. By design is HAProxy a reverse proxy to a origin server not to a forwarding proxy which is the reason why the CONNECT method is a invalid method. Because of that fact I would not use "mode http" for the squid backend/servers because of the issues you described. Why not "mode tcp" with proxy protocol http://www.squid-cache.org/Doc/config/proxy_protocol_access/ if you need the client ip. Hi, CONNECT method is valid and HAproxy supports it. It doesn't handle it by itself however. It forwards it to the server. Since the 2.4, a scheme-based normalization is performed on the requests after the parsing but with some limitation. The target-uri must be in absolute-form. Thus CONNECT requests are not normalized. But as said, the normalization is performed after the H1 parsing. Idea is to sanitize the request before sending it to the server and to simplify host-based routing configurations. However, during H1 parsing, the authority found in the URI is validated against the Host header. At this stage, both must be identical. Otherwise an error is reported. "accept-invalid-http-request" option is a valid workaround in this case. So now the question is to know if a pre-normalization must be performed during H1 parsing or not (in addition to the one performed during the request analysis). And if it should be extended to CONNECT requests. In practice, it is only an issue for CONNECT requests because the absolue-form is not the common form for URIs in H1. -- Christopher Faulet
Re: DOC issue with http-request return
Le 2/1/22 à 18:31, Jay Wren (jawren) a écrit :
I think I've found an error in the docs. The entry for `http-request return` shows
the last component as [ {if | unless} ] but if I leave that off in
a backend I get [ALERT] parsing... 'http-request return' expects either 'if' or
'unless' followed by a condition but found ‘…'.
Hi,
The ACL part in the "http-request return" rule is optional. Otherwise it means
there is a bug in the parser. But, first of all, you should share the failing
rule and the HAProxy version you use.
Thanks,
--
Christopher Faulet
[ANNOUNCE] haproxy-2.5.4
Hi, HAProxy 2.5.4 was released on 2022/02/25. It added 8 new commits after version 2.5.3. A major issue in the H2 multiplexer was fixed in this release. An error during the response processing, after the HEADERS frame parsing, led to a wakeup loop consuming all the CPU because the error was not properly reported to the upper layer. For instance, this happened if an invalid header value, an invalid status code or a forbidden header was found in the response. Note that only HAProxy >= 2.4 are affected by this issue. In addition, some issues about errors on buffers allocation were fixed. First, in the H1 multiplexer. If we failed to send data because we failed to allocate the H1 output buffer, the H1 stream was erroneously woken up. This led to a wakeup loop to send more data while it is not possible because there is no output buffer. Then, in process_stream(), if we failed to allocate the channel response buffer while a connect or an analysis timeout occurred, the stream was woken up in loop because its task was requeued with an expired date. Now an error is reported when this happens and the stream processing is interrupted. Note there is a mechanism to deal with errors on buffers allocation. Unfortunately, since the 1.7, this mechanism is broken. And it is even worse now with the multiplexers. All this part must be refactored. But for now, HAProxy may be partially Frozen if too many entities are waiting for a buffer. Other commits are doc improvements and small fixes here and there. As usual, people using the 2.5 branch are encouraged to migrate to this version. Thanks everyone for your help and your contributions! Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse: http://discourse.haproxy.org/ Slack channel: https://slack.haproxy.org/ Issue tracker: https://github.com/haproxy/haproxy/issues Wiki : https://github.com/haproxy/wiki/wiki Sources : http://www.haproxy.org/download/2.5/src/ Git repository : http://git.haproxy.org/git/haproxy-2.5.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-2.5.git Changelog: http://www.haproxy.org/download/2.5/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ --- Complete changelog : Christian Ruppert (1): DOC: Fix usage/examples of deprecated ACLs Christopher Faulet (4): BUG/MEDIUM: htx: Be sure to have a buffer to perform a raw copy of a message BUG/MEDIUM: mux-h1: Don't wake h1s if mux is blocked on lack of output buffer BUG/MAJOR: mux-h2: Be sure to always report HTX parsing error to the app layer BUG/MEDIUM: stream: Abort processing if response buffer allocation fails Willy Tarreau (3): BUG/MINOR: proxy: preset the error message pointer to NULL in parse_new_proxy() REGTESTS: fix the race conditions in 40be_2srv_odd_health_checks CI: github: enable pool debugging by default -- Christopher Faulet
[ANNOUNCE] haproxy-2.4.14
Hi, HAProxy 2.4.14 was released on 2022/02/25. It added 26 new commits after version 2.4.13. The main issues fixed in this version are: - A major issue in the H2 multiplexer. An error during the response processing, after the HEADERS frame parsing, led to a wakeup loop consuming all the CPU because the error was not properly reported to the upper layer. For instance, this happened if an invalid header value, an invalid status code or a forbidden header was found in the response. Note that only HAProxy >= 2.4 are affected by this issue. - A FD leak on reload failures. When the master process is reloaded on a new config, it will try to connect to the previous process' socket to retrieve all known listening FDs to be reused by the new listeners. If listeners were removed, their unused FDs are simply closed. However there's a catch. In case a socket fails to bind, the master will cancel its startup and switch to wait mode for a new operation to happen. In this case it didn't close the possibly remaining FDs that were left unused. - A FD leak of a sockpair upon a failed reload. When starting HAProxy in master-worker, the master pre-allocate a struct mworker_proc and do a socketpair() before the configuration parsing. If the configuration loading failed, the FD was never closed because they aren't part of listener, they are not even in the fdtab. - Some issues about errors on buffers allocation. First, in the H1 multiplexer. If we failed to send data because we failed to allocate the H1 output buffer, the H1 stream was erroneously woken up. This led to a wakeup loop to send more data while it is not possible because there is no output buffer. Then, in process_stream(), if we failed to allocate the channel response buffer while a connect or an analysis timeout occurred, the stream was woken up in loop because its task was requeued with an expired date. Now an error is reported when this happens and the stream processing is interrupted. Note there is a mechanism to deal with errors on buffers allocation. Unfortunately, since the 1.7, this mechanism is broken. And it is even worse now with the multiplexers. All this part must be refactored. But for now, HAProxy may be partially frozen if too many entities are waiting for a buffer. - Some alignment problems that were found when using gcc-11 + RHEL8, resulting in instant crashes on startup. - An issue with multi-line ESMTP response in the mailer code. - An issue in the resolvers code with domain names with a trailing dot. The trailing dot was not ignored as expected and a junk character was added at the end of the encoded part of the domain name. The remaining is the usual bunch of fixes and improvements. As usual, people using the 2.4 branch are encouraged to migrate to this version. Thanks everyone for your help and your contributions! Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse: http://discourse.haproxy.org/ Slack channel: https://slack.haproxy.org/ Issue tracker: https://github.com/haproxy/haproxy/issues Wiki : https://github.com/haproxy/wiki/wiki Sources : http://www.haproxy.org/download/2.4/src/ Git repository : http://git.haproxy.org/git/haproxy-2.4.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-2.4.git Changelog: http://www.haproxy.org/download/2.4/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ --- Complete changelog : Christopher Faulet (6): BUG/MINOR: sink: Use the right field in appctx context in release callback BUG/MEDIUM: resolvers: Really ignore trailing dot in domain names BUG/MEDIUM: htx: Be sure to have a buffer to perform a raw copy of a message BUG/MEDIUM: mux-h1: Don't wake h1s if mux is blocked on lack of output buffer BUG/MAJOR: mux-h2: Be sure to always report HTX parsing error to the app layer BUG/MEDIUM: stream: Abort processing if response buffer allocation fails Ilya Shipitsin (4): BUILD: adopt script/build-ssl.sh for OpenSSL-3.0.0beta2 CI: github actions: add OpenSSL-3.0.0 builds CI: github actions: relax OpenSSL-3.0.0 version comparision CI: github actions: update OpenSSL to 3.0.1 Lukas Tribus (1): BUG/MINOR: mailers: negotiate SMTP, not ESMTP William Lallemand (5): BUG/MINOR: mworker: fix a FD leak of a sockpair upon a failed reload BUILD: fix compilation for OpenSSL-3.0.0-alpha17 CI: github actions: -Wno-deprecated-declarations with OpenSSL 3.0.0 CI: github: switch to OpenSSL 3.0.0 BUG/MINOR: tools: url2sa reads ipv4 too far Willy Tarreau (10): MINOR: sock: move the unused socket cleaning code into its own function BUG/MEDIUM: mworker: close unused transferred FDs on load fai
Re: Incompatible with 'frontend http-request header rule'
Le 3/1/22 à 22:00, Henning Svane a écrit :
http-request track-sc0 src table table_login_limiter if { url_beg /login } {
status 401 }
http-request tarpit deny_status 429 if { sc_http_req_rate(0) gt 10 } { url_beg
/login }
Hi,
You cannot match on the response status in a request rule. At this stage, the
response is not received yet. So, you should rely on an http-response rule
instead. But, at this stage, url_beg is no longer available because the request
was already sent. You must use capture.req.uri instead.
In addition, because the tracking will be performed during the response
evaluation, you must use table_http_req_rate() converter to look up in your
stick-table. (Note that in your tarpit rule, you must explicitly specify the
table name)
You can try the following rules :
http-request tarpit deny_status 429 if {
src,table_http_req_rate(table_login_limiter) gt 10 } { url_beg /login }
http-response track-sc0 src table table_login_limiter if { capture.req.uri -m
beg /login } { status 401 }
You can also match on the url in an http-request rule to set a variable and use
it in the http-response rule.
Regards,
--
Christopher Faulet
[ANNOUNCE] haproxy-2.3.18
which case the socket transfer from the older process couldn't happen. - some issues about errors on buffers allocation. First, in the H1 multiplexer. If we failed to send data because we failed to allocate the H1 output buffer, the H1 stream was erroneously woken up. This led to a wakeup loop to send more data while it is not possible because there is no output buffer. Then, in process_stream(), if we failed to allocate the channel response buffer while a connect or an analysis timeout occurred, the stream was woken up in loop because its task was requeued with an expired date. Now an error is reported when this happens and the stream processing is interrupted. Note there is a mechanism to deal with errors on buffers allocation. Unfortunately, since the 1.7, this mechanism is broken. And it is even worse now with the multiplexers. All this part must be refactored. But for now, HAProxy may be partially frozen if too many entities are waiting for a buffer. - some alignment problems that were found when using gcc-11 + RHEL8, resulting in instant crashes on startup. - an issue with multi-line ESMTP response in the mailer code. - an issue in the resolvers code with domain names with a trailing dot. The trailing dot was not ignored as expected and a junk character was added at the end of the encoded part of the domain name. - there were still a number of other issues of lower level of importance, such as the CLI being extremely slow to parse pipelined requests because it was looking for the line feed first, hence the larger the buffer, the slower it was with batch updates like ACL/map updates; a possibly truncated pidfile in master mode; a bug with the data transfer in the HTX layer for large data block; an inconsistency with the parsing of IPv4 addresses. Note that the EOL of the 2.3 is planned for the end of this quarter. So this release is probably one of the last 2.3 releases. For everyone running a 2.3, it could be a good idea to migrate to the 2.4. Thanks everyone for your help and your contributions! Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse: http://discourse.haproxy.org/ Slack channel: https://slack.haproxy.org/ Issue tracker: https://github.com/haproxy/haproxy/issues Wiki : https://github.com/haproxy/wiki/wiki Sources : http://www.haproxy.org/download/2.3/src/ Git repository : http://git.haproxy.org/git/haproxy-2.3.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-2.3.git Changelog: http://www.haproxy.org/download/2.3/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ --- Complete changelog : Andrew McDermott (1): BUG/MAJOR: http/htx: prevent unbounded loop in http_manage_server_side_cookies Christopher Faulet (9): BUG/MEDIUM: htx: Adjust length to add DATA block in an empty HTX buffer BUG/MEDIUM: cli: Never wait for more data on client shutdown BUG/MINOR: sink: Use the right field in appctx context in release callback BUG/MEDIUM: resolvers: Really ignore trailing dot in domain names BUG/MEDIUM: htx: Be sure to have a buffer to perform a raw copy of a message BUG/MEDIUM: mux-h1: Don't wake h1s if mux is blocked on lack of output buffer BUG/MAJOR: mux-h2: Be sure to always report HTX parsing error to the app layer BUG/MEDIUM: stream: Abort processing if response buffer allocation fails REGTESTS: fix the race conditions in secure_memcmp.vtc David Carlier (1): BUILD/MINOR: fix solaris build with clang. Ilya Shipitsin (5): BUILD: adopt script/build-ssl.sh for OpenSSL-3.0.0beta2 CI: github actions: add OpenSSL-3.0.0 builds CI: github actions: relax OpenSSL-3.0.0 version comparision CI: github actions: update OpenSSL to 3.0.1 CI: github actions: use cache for SSL libs Lukas Tribus (1): BUG/MINOR: mailers: negotiate SMTP, not ESMTP William Lallemand (6): BUG/MINOR: mworker: does not erase the pidfile upon reload BUG/MINOR: mworker: fix a FD leak of a sockpair upon a failed reload BUILD: fix compilation for OpenSSL-3.0.0-alpha17 CI: github actions: -Wno-deprecated-declarations with OpenSSL 3.0.0 CI: github: switch to OpenSSL 3.0.0 BUG/MINOR: tools: url2sa reads ipv4 too far Willy Tarreau (20): MEDIUM: cli: yield between each pipelined command MINOR: channel: add new function co_getdelim() to support multiple delimiters BUG/MINOR: cli: avoid O(bufsize) parsing cost on pipelined commands BUG/MEDIUM: mcli: do not try to parse empty buffers BUG/MEDIUM: mcli: always realign wrapping buffers before parsing them BUG/MEDIUM: mworker: don't lose the stats socket on failed reload MINOR: listener: replace the listener's spinlock with an rwlock
Re: SV: Incompatible with 'frontend http-request header rule'
Le 3/4/22 à 00:42, Henning Svane a écrit :
Hi Christopher
I tried your rule and it did not compile, but I am trying to understand it.
/haproxy02.cfg:20] : error detected while parsing an 'http-request tarpit'
condition : no such ACL : 'http-response'
I placed the rule in the frontend, but was thinking if it should be in the
backend, as it is here server is called and hereby produce the return code.
I understand the idea in your rule, but at the same time, I do not understand
the order of execution.
It looks like it has to be executed from the right with the " if { capture.req.uri
-m beg /login } { status 401 }" first.
But then what?
If I understand correctly
1) You save the request url in a table with capture.req.uri.
2) Then server try to execute the url
3) Based on the server return the http-response (this part I have not fully
understand yet)
4) If the response is 401 then " http-request tarpit deny_status 429"
I will try to work a little more with you suggestion and see if I can get to
work.
Regards
Henning
haproxy02.cfg:20] : error detected while parsing an 'http-request tarpit'
condition : no such ACL : 'http-response'.
Your email client seems to have mangled my reply. Or there is a formatting issue
on my side. Anyway, it is not one rule with everything on one line, but 2 rules.
An http-request one to deny the request on its own line and an http-response one
to track login failures, on another line.
Both can be specified in the frontend, the backend or split. It depends on your
other rules, if any. With your config snippet, it doesn't matter.
--
Christopher Faulet
Re: server check inter and timeout check relation
Le 3/14/22 à 10:53, Artur a écrit : Hello, I'd like to know how checks behaves depending on the "inter" and "timeout check" settings. Let's try this simplified setup : backend back mode tcp timeout check 5s server s1 1.2.3.4:80 check inter 2s server s2 1.2.3.5:80 check inter 2s "inter 2s" is the default setup. We should have there one check every 2s if everything is optimal. "timeout check 5s" specify that the server check can take up to 5s (once the connection established). In this configuration, what happens if the check takes more than 2 seconds ? Does haproxy wait (up to 5 seconds) for this check to finish before launching another check or it's still launching checks every 2s anyway ? Hi, For a given server, inter/fastinter/downinter timeouts are used to define the delay between the end of a health-check and the beginning of the following one. This is independent on the evaluation time. Thus in your example, a health-check will still run 2s after the end of the previous one, independently on its duration. -- Christopher Faulet
[ANNOUNCE] haproxy-2.5.5
Hi, HAProxy 2.5.5 was released on 2022/03/14. It added 39 new commits after version 2.5.4. The main issues fixed in this version are: * An issue in the pass-through multiplexer leading to a connection leak on the server side when timeout occurred during the connection establishment. In this case, the server connection was detached from the application stream but not closed. At this stage the connection could only be closed by the server, if it was finally accepted, or by the kernel, after all SYN retries. All versions as far as 2.3 are affected by this bug. * Two issues in the HTTP client applet. First it was possible to trigger an infinite loop when the same HTTP client lua instance was used to send several POST requests. A counter was not reset between the requests. Then, the applet was unexpectedly able to consume the response before its analysis by the application stream. To hit the bug, the applet's I/O handler had to be scheduled before the stream one. The result was a crash because of a NULL dereferenced pointer. * An issue in the master CLI. When a command was sent to a worker, the errors, especially write errors, during the response processing were not properly handled. The session could remain stuck if a client quickly closed the connection before the response was fully sent. The maxconn value of the master CLI is set 10. Thus, it could quickly be unresponsive if this happened several times. * A possible null deref in the htx_xfer_blks() function, when headers or trailers were partially transferred. Concretely, it was only possible when H2 trailers were copied from the mux to the channel buffer. * A crash with the FCGI health-checks. When the multi-level source and destination addresses were introduced, a bug was also introduced. The FCGI multiplexer was relying on the server stream-interface to set some parameters (REMOTE_ADDR/REMOTE_PORT and SERVER_NAME/SERVER_PORT). But there is no stream-interface with the health-check because there is no stream. Now, the server connection is used instead of the stream-interface when the origin is a health-check. * A design issue for listener-less streams. When a stream was created from a session without listener, the request analyzers were not properly set. Concretely, it is only an issue for client applets, more specifically the HTTP ones. Thus only the HTTP client was affected by this bug. However, there was no visible effect. * An issue with all HTX applets. The end of a message was only reported at the HTX level. The channel's flags were not updated accordingly. The only known visible effect of this bug was some server aborts erroneously reported in the stats counters. * A theoretical risk of memleak in session_accept_fd() because of a wrong goto label on the error path. * An alignment issue with pool_head structure. * Some build issues were fixed. kFreeBSD is now a distinct target, the old HA_ATOMIC_LOAD() macro now supports const pointers, few numeric constants are explicitly marked as long long, In addition, it adds some improvements: * Proxy mode (tcp, http, cli...) is not properly reported when displayed. Missing "syslog" and "peers" mode can now be reported. * "no-memory-trimming" global option was added to disable call to malloc_trim(). Some users with very large numbers of connections have been facing extremely long malloc_trim() calls on reload that managed to trigger the watchdog! That's a bit counter-productive. It's even possible that some implementations are not perfectly reliable or that their trimming time grows quadratically with the memory used. With this option, it is possible to disable this mechanism. * The dark mode support of the stat page was updated to be applied on socket rows. As usual, people using the 2.5 branch are encouraged to migrate to this version. Thanks everyone for your help and your contributions! Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse: http://discourse.haproxy.org/ Slack channel: https://slack.haproxy.org/ Issue tracker: https://github.com/haproxy/haproxy/issues Wiki : https://github.com/haproxy/wiki/wiki Sources : http://www.haproxy.org/download/2.5/src/ Git repository : http://git.haproxy.org/git/haproxy-2.5.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-2.5.git Changelog: http://www.haproxy.org/download/2.5/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ --- Complete changelog : Christopher Faulet (16): BUG/MEDIUM: mux-fcgi: Don't rely on SI src/dst addresses for FCGI health-checks BUG/MEDIUM: htx: Fix a possible null derefs in htx_xfer_blks() REGTESTS: fix the
[ANNOUNCE] haproxy-2.4.15
Hi, HAProxy 2.4.15 was released on 2022/03/14. It added 26 new commits after version 2.4.14. This one contains more or less the same fixes than the 2.5.5, except 2.5-specific ones : * An issue in the pass-through multiplexer leading to a connection leak on the server side when timeout occurred during the connection establishment. In this case, the server connection was detached from the application stream but not closed. At this stage the connection could only be closed by the server, if it was finally accepted, or by the kernel, after all SYN retries. All versions as far as 2.3 are affected by this bug. * An issue in the master CLI. When a command was sent to a worker, the errors, especially write errors, during the response processing were not properly handled. The session could remain stuck if a client quickly closed the connection before the response was fully sent. The maxconn value of the master CLI is set 10. Thus, it could quickly be unresponsive if this happened several times. * A possible null deref in the htx_xfer_blks() function, when headers or trailers were partially transferred. Concretely, it was only possible when H2 trailers were copied from the mux to the channel buffer. * An issue with all HTX applets. The end of a message was only reported at the HTX level. The channel's flags were not updated accordingly. The only known visible effect of this bug was some server aborts erroneously reported in the stats counters. * A theoretical risk of memleak in session_accept_fd() because of a wrong goto label on the error path. * An alignment issue with pool_head structure. * Proxy mode (tcp, http, cli...) is not properly reported when displayed. Missing "syslog" and "peers" mode can now be reported. * "no-memory-trimming" global option was added to disable call to malloc_trim(). Some users with very large numbers of connections have been facing extremely long malloc_trim() calls on reload that managed to trigger the watchdog! That's a bit counter-productive. It's even possible that some implementations are not perfectly reliable or that their trimming time grows quadratically with the memory used. With this option, it is possible to disable this mechanism. * The anti-loop protection in process_stream() was improved to only count the no-progress calls. Thanks everyone for your help and your contributions! Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse: http://discourse.haproxy.org/ Slack channel: https://slack.haproxy.org/ Issue tracker: https://github.com/haproxy/haproxy/issues Wiki : https://github.com/haproxy/wiki/wiki Sources : http://www.haproxy.org/download/2.4/src/ Git repository : http://git.haproxy.org/git/haproxy-2.4.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-2.4.git Changelog: http://www.haproxy.org/download/2.4/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ --- Complete changelog : Christian Ruppert (1): DOC: Fix usage/examples of deprecated ACLs Christopher Faulet (12): BUG/MEDIUM: htx: Fix a possible null derefs in htx_xfer_blks() REGTESTS: fix the race conditions in normalize_uri.vtc REGTESTS: fix the race conditions in secure_memcmp.vtc BUG/MINOR: hlua: Set conn-stream/channel EOI flags at the end of request BUG/MINOR: stats: Set conn-stream/channel EOI flags at the end of request BUG/MINOR: cache: Set conn-stream/channel EOI flags at the end of request BUG/MINOR: promex: Set conn-stream/channel EOI flags at the end of request DEBUG: cache: Update underlying buffer when loading HTX message in cache applet BUG/MEDIUM: mcli: Properly handle errors and timeouts during reponse processing DEBUG: stream: Add the missing descriptions for stream trace events DEBUG: stream: Fix stream trace message to print response buffer state BUG/MAJOR: mux-pt: Always destroy the backend connection on detach Ilya Shipitsin (3): CI: github actions: add OpenTracing builds CI: github actions: use cache for OpenTracing CI: github actions: use cache for SSL libs William Lallemand (2): BUG/MINOR: add missing modes in proxy_mode_str() BUG/MINOR: cli: shows correct mode in "show sess" Willy Tarreau (8): CI: github actions: add the output of $CC -dM -E- BUG/MINOR: pool: always align pool_heads to 64 bytes BUG/MEDIUM: pools: fix ha_free() on area in the process of being freed MINOR: pools: add a new global option "no-memory-trimming" BUILD: pools: fix backport of no-memory-trimming on non-linux OS BUG/MINOR: session: fix theoretical risk of memleak in session_accept_fd() BUG/MINOR: stream: make the call_ra
[ANNOUNCE] haproxy-2.3.19
Hi, HAProxy 2.3.19 was released on 2022/03/14. It added 14 new commits after version 2.3.18. All fixes included in this release were already described in the 2.4.14 announcement. Here is a cut-paste of relevant parts: * An issue in the pass-through multiplexer leading to a connection leak on the server side when timeout occurred during the connection establishment. In this case, the server connection was detached from the application stream but not closed. At this stage the connection could only be closed by the server, if it was finally accepted, or by the kernel, after all SYN retries. All versions as far as 2.3 are affected by this bug. * An issue in the master CLI. When a command was sent to a worker, the errors, especially write errors, during the response processing were not properly handled. The session could remain stuck if a client quickly closed the connection before the response was fully sent. The maxconn value of the master CLI is set 10. Thus, it could quickly be unresponsive if this happened several times. * An issue with all HTX applets. The end of a message was only reported at the HTX level. The channel's flags were not updated accordingly. The only known visible effect of this bug was some server aborts erroneously reported in the stats counters. * Proxy mode (tcp, http, cli...) is not properly reported when displayed. Missing "syslog" and "peers" mode can now be reported. * The anti-loop protection in process_stream() was improved to only count the no-progress calls. Thanks everyone for your help and your contributions! Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse: http://discourse.haproxy.org/ Slack channel: https://slack.haproxy.org/ Issue tracker: https://github.com/haproxy/haproxy/issues Wiki : https://github.com/haproxy/wiki/wiki Sources : http://www.haproxy.org/download/2.3/src/ Git repository : http://git.haproxy.org/git/haproxy-2.3.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-2.3.git Changelog: http://www.haproxy.org/download/2.3/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ --- Complete changelog : Christian Ruppert (1): DOC: Fix usage/examples of deprecated ACLs Christopher Faulet (9): BUG/MINOR: hlua: Set conn-stream/channel EOI flags at the end of request BUG/MINOR: stats: Set conn-stream/channel EOI flags at the end of request BUG/MINOR: cache: Set conn-stream/channel EOI flags at the end of request BUG/MINOR: promex: Set conn-stream/channel EOI flags at the end of request DEBUG: cache: Update underlying buffer when loading HTX message in cache applet BUG/MEDIUM: mcli: Properly handle errors and timeouts during reponse processing DEBUG: stream: Add the missing descriptions for stream trace events DEBUG: stream: Fix stream trace message to print response buffer state BUG/MAJOR: mux-pt: Always destroy the backend connection on detach William Lallemand (2): BUG/MINOR: add missing modes in proxy_mode_str() BUG/MINOR: cli: shows correct mode in "show sess" Willy Tarreau (2): BUG/MINOR: stream: make the call_rate only count the no-progress calls BUILD: tree-wide: mark a few numeric constants as explicitly long long -- Christopher Faulet
[ANNOUNCE] haproxy-2.2.22
Hi, HAProxy 2.2.22 was released on 2022/03/14. It added 13 new commits after version 2.2.21. This one contains the same fixes than the 2.3.19. So, I'm not going to be really original: * An issue in the pass-through multiplexer leading to a connection leak on the server side when timeout occurred during the connection establishment. In this case, the server connection was detached from the application stream but not closed. At this stage the connection could only be closed by the server, if it was finally accepted, or by the kernel, after all SYN retries. All versions as far as 2.3 are affected by this bug. * An issue in the master CLI. When a command was sent to a worker, the errors, especially write errors, during the response processing were not properly handled. The session could remain stuck if a client quickly closed the connection before the response was fully sent. The maxconn value of the master CLI is set 10. Thus, it could quickly be unresponsive if this happened several times. * An issue with all HTX applets. The end of a message was only reported at the HTX level. The channel's flags were not updated accordingly. The only known visible effect of this bug was some server aborts erroneously reported in the stats counters. * Proxy mode (tcp, http, cli...) is not properly reported when displayed. Missing "syslog" and "peers" mode can now be reported. * The anti-loop protection in process_stream() was improved to only count the no-progress calls. This release cycle was performed to be able to finally release the 2.0.28. It was announced few weeks ago. Twice. But it was delayed because of lack of time. This time, it must be released tomorrow morning ! Thanks everyone for your help and your contributions! Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse: http://discourse.haproxy.org/ Slack channel: https://slack.haproxy.org/ Issue tracker: https://github.com/haproxy/haproxy/issues Wiki : https://github.com/haproxy/wiki/wiki Sources : http://www.haproxy.org/download/2.2/src/ Git repository : http://git.haproxy.org/git/haproxy-2.2.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-2.2.git Changelog: http://www.haproxy.org/download/2.2/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ --- Complete changelog : Christian Ruppert (1): DOC: Fix usage/examples of deprecated ACLs Christopher Faulet (9): BUG/MINOR: hlua: Set conn-stream/channel EOI flags at the end of request BUG/MINOR: stats: Set conn-stream/channel EOI flags at the end of request BUG/MINOR: cache: Set conn-stream/channel EOI flags at the end of request BUG/MINOR: promex: Set conn-stream/channel EOI flags at the end of request DEBUG: cache: Update underlying buffer when loading HTX message in cache applet BUG/MEDIUM: mcli: Properly handle errors and timeouts during reponse processing DEBUG: stream: Add the missing descriptions for stream trace events DEBUG: stream: Fix stream trace message to print response buffer state BUG/MAJOR: mux-pt: Always destroy the backend connection on detach William Lallemand (1): BUG/MINOR: cli: shows correct mode in "show sess" Willy Tarreau (2): BUG/MINOR: stream: make the call_rate only count the no-progress calls BUILD: tree-wide: mark a few numeric constants as explicitly long long -- Christopher Faulet
[ANNOUNCE] haproxy-2.0.28
_stream(), if we failed to allocate the channel response buffer while a connect or an analysis timeout occurred, the stream was woken up in loop because its task was requeued with an expired date. Now an error is reported when this happens and the stream processing is interrupted. Note there is a mechanism to deal with errors on buffers allocation. Unfortunately, since the 1.7, this mechanism is broken. And it is even worse now with the multiplexers. All this part must be refactored. But for now, HAProxy may be partially frozen if too many entities are waiting for a buffer. * An issue with multi-line ESMTP response in the mailer code. * An issue in the resolvers code with domain names with a trailing dot. The trailing dot was not ignored as expected and a junk character was added at the end of the encoded part of the domain name. * An issue in the master CLI. When a command was sent to a worker, the errors, especially write errors, during the response processing were not properly handled. The session could remain stuck if a client quickly closed the connection before the response was fully sent. The maxconn value of the master CLI is set 10. Thus, it could quickly be unresponsive if this happened several times. * An issue with all HTX applets. The end of a message was only reported at the HTX level. The channel's flags were not updated accordingly. The only known visible effect of this bug was some server aborts erroneously reported in the stats counters. * Proxy mode (tcp, http, cli...) is not properly reported when displayed. Missing "syslog" and "peers" mode can now be reported. * The anti-loop protection in process_stream() was improved to only count the no-progress calls. Some other issues of lower level of importance were also fixed, such as the CLI being extremely slow to parse pipelined requests because it was looking for the line feed first, hence the larger the buffer, the slower it was with batch updates like ACL/map updates; a possibly truncated pidfile in master mode; an inconsistency with the parsing of IPv4 addresses. Thanks everyone for your help and your contributions! Please find the usual URLs below : Site index : http://www.haproxy.org/ Discourse: http://discourse.haproxy.org/ Slack channel: https://slack.haproxy.org/ Issue tracker: https://github.com/haproxy/haproxy/issues Wiki : https://github.com/haproxy/wiki/wiki Sources : http://www.haproxy.org/download/2.0/src/ Git repository : http://git.haproxy.org/git/haproxy-2.0.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-2.0.git Changelog: http://www.haproxy.org/download/2.0/src/CHANGELOG Cyril's HTML doc : http://cbonte.github.io/haproxy-dconv/ --- Complete changelog : Alex (1): DOC: use the req.ssl_sni in examples Andrew McDermott (1): BUG/MAJOR: http/htx: prevent unbounded loop in http_manage_server_side_cookies Christian Ruppert (1): DOC: Fix usage/examples of deprecated ACLs Christopher Faulet (11): BUG/MEDIUM: resolvers: Really ignore trailing dot in domain names BUG/MEDIUM: mux-h1: Don't wake h1s if mux is blocked on lack of output buffer BUG/MAJOR: mux-h2: Be sure to always report HTX parsing error to the app layer BUG/MEDIUM: stream: Abort processing if response buffer allocation fails BUG/MINOR: hlua: Set conn-stream/channel EOI flags at the end of request BUG/MINOR: stats: Set conn-stream/channel EOI flags at the end of request BUG/MINOR: cache: Set conn-stream/channel EOI flags at the end of request BUG/MINOR: promex: Set conn-stream/channel EOI flags at the end of request DEBUG: cache: Update underlying buffer when loading HTX message in cache applet BUG/MEDIUM: mcli: Properly handle errors and timeouts during reponse processing BUG/MAJOR: mux-pt: Always destroy the backend connection on detach Ilya Shipitsin (1): CI: github actions: use cache for SSL libs Lukas Tribus (2): BUG/MINOR: mailers: negotiate SMTP, not ESMTP DOC: ssl: req_ssl_sni needs implicit TLS William Lallemand (4): BUG/MINOR: mworker: does not erase the pidfile upon reload BUG/MINOR: mworker: fix a FD leak of a sockpair upon a failed reload BUG/MINOR: tools: url2sa reads ipv4 too far BUG/MINOR: cli: shows correct mode in "show sess" Willy Tarreau (16): MEDIUM: cli: yield between each pipelined command MINOR: channel: add new function co_getdelim() to support multiple delimiters BUG/MINOR: cli: avoid O(bufsize) parsing cost on pipelined commands BUG/MEDIUM: mcli: do not try to parse empty buffers BUG/MEDIUM: mcli: always realign wrapping buffers before parsing them BUG/MEDIUM: mworker: don't lose the stats socket on failed reload BUG/
Re: Check interval rise and fall behaviour
Le 3/29/22 à 18:02, Lais, Alexander a écrit : Dear all, We are using the backend health checks to disable flapping backends. The default values for rise and fall are 2 subsequent succeeded and 3 subsequent failed checks. Our check interval is at 1000ms (a little frequent, potentially part of the problem). Here is what we observed, using HAProxy 2.4.4: 1. Falling It started with the backend being up and then going down (fall). 2022-03-23T21:31:54.942ZHealth check for server http-routers-http1/node4 failed, reason: Layer4 timeout, check duration: 1000ms, status: 2/3 UP. 2022-03-23T21:31:56.920ZHealth check for server http-routers-http1/node4 failed, reason: Layer4 timeout, check duration: 1001ms, status: 1/3 UP. 2022-03-23T21:31:57.931ZHealth check for server http-routers-http1/node4 succeeded, reason: Layer7 check passed, code: 200, check duration: 1ms, status: 3/3 UP. 2022-03-24T10:03:27.223ZHealth check for server http-routers-http1/node4 failed, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 1ms, status: 2/3 UP. 2022-03-24T10:03:28.234ZHealth check for server http-routers-http1/node4 failed, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 1ms, status: 1/3 UP. 2022-03-24T10:03:29.237ZHealth check for server http-routers-http1/node4 failed, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 1ms, status: 0/2 DOWN. We go down from 3/3 to 2/3, 1/3 and back up again to 3/3. My assumption is that it then measured 2/3, but only needs 2 for rising, i.e. 2/2, which is bumped to 3/3 as the backend is now considered up. The backend stays up for a while and then goes down with my expected health checks, i.e. 3/3, 2/3, 1/3, 0/3 -> 0/2 (as we need 2 for rise). 2. Rising 2022-03-24T10:12:26.846ZHealth check for server http-routers-http1/node4 failed, reason: Layer4 timeout, check duration: 1000ms, status: 0/2 DOWN. 2022-03-24T10:12:29.843ZHealth check for server http-routers-http1/node4 failed, reason: Layer4 connection problem, info: "Connection refused", check duration: 1ms, status: 0/2 DOWN. 2022-03-24T10:13:43.902ZHealth check for server http-routers-http1/node4 failed, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 2ms, status: 0/2 DOWN. 2022-03-24T10:14:03.039ZHealth check for server http-routers-http1/node4 succeeded, reason: Layer7 check passed, code: 200, check duration: 1ms, status: 1/2 DOWN. 2022-03-24T10:14:04.079ZHealth check for server http-routers-http1/node4 succeeded, reason: Layer7 check passed, code: 200, check duration: 1ms, status: 3/3 UP. So coming up (rise), it goes from 0/2 probes to 1/2 to 3/3. My assumption that it goes to 2/2, is considered up and is bumped to 3/3 because for fall we now need 3 failed probes. The documentation describes rise / fall as “number of subsequent probes that succeeded / failed. From my observations it looks like it is a sliding window of the last n being successful, i.e. when the number of fall is larger than rise, it is easier to rise back up with a single successful probe. Maybe I’m misreading the log outputs or drawing the wrong conclusions. If someone knows by heart how it’s supposed to work based on the code that would be great. Otherwise we can dig some more ourselves. Hi, Rise and fall values are the number of consecutive successful/unsuccessful health checks. When a server is DOWN, we count the number of consecutive successful health checks. If the counter reaches the rise value, the server is considered as UP. Otherwise, on each failure, the counter is reset. The same is done when the server is UP. we count the number of consecutive unsuccessful health checks. If the counter reaches the fall value, the server is considered as DOWN. Otherwise, on each success, the counter is reset. Internally it is a bit more complex but the idea is the same. In logs, the rise value is reported when the server is DOWN (X/rise) and the counter is incremented on each success (so from 0 to rise-1). And the fall value is reported when the server is UP (Y/fall) and the counter is decremented on each failure (from fall to 1). So when the server is set to DOWN state, you will never see "0/3 UP" in logs but "0/2 DOWN" instead. The same is true when the server is set to UP state, "2/2 UP" is never reported because "0/3 DOWN" is reported. And you're right, with a rise value lower than the fall value it is quicker to consider a DOWN server as UP than the opposite. But with a rise to 2, we need 2 successful health checks to set a server UP. -- Christopher Faulet
[ANNOUNCE] haproxy-2.5.6
* The automatic frontend connection closing mechanism on reload that was
brought into 2.5 caused some concerns to some users, leading to an option
to disable it. Now there's a new global setting, "close-spread-time",
which can be used to indicate that the closure of idle connections should
be randomly spread over that interval, so that reconnecting clients don't
all rush at the same time on the new process. This applies both to
passive close ("connection: close" on responses), and to active close of
idle connections. For best efficiency, the interval should obviously be
shorter than the one used in "hard-stop-after" if any. We'll also see how
to extend the mechanism to allow never to close at all as there's also
some demand for this.
* Opentracing was updated. In 2.5 we had to disable the use of variables
between the plugin and the haproxy core because the code was relying on
an original misfeature of the variables which was that they would never
disappear after being created, and this misfeature was fixed in 2.5,
breaking that part of Opentracing. Miroslav finally found the time to
address this and rework it in an elegant way so that the module is fully
functional again.
* Support for MQTT 3.1 was added.
* Another improvement which is not related to the code, with the precious
help of Tim and Cyril, we could finally set up an automatic generation of
the HTML documentation. It's performed daily and published on github
pages at http://docs.haproxy.org.
As usual, people using the 2.5 branch are encouraged to migrate to this
version. Thanks everyone for your help and your contributions!
Please find the usual URLs below :
Site index : http://www.haproxy.org/
Documentation: http://docs.haproxy.org/
Wiki : https://github.com/haproxy/wiki/wiki
Discourse: http://discourse.haproxy.org/
Slack channel: https://slack.haproxy.org/
Issue tracker: https://github.com/haproxy/haproxy/issues
Sources : http://www.haproxy.org/download/2.5/src/
Git repository : http://git.haproxy.org/git/haproxy-2.5.git/
Git Web browsing : http://git.haproxy.org/?p=haproxy-2.5.git
Changelog: http://www.haproxy.org/download/2.5/src/CHANGELOG
Pending bugs : http://www.haproxy.org/l/pending-bugs
Reviewed bugs: http://www.haproxy.org/l/reviewed-bugs
Code reports : http://www.haproxy.org/l/code-reports
---
Complete changelog :
Aleksandar Lazic (1):
DOC: remove double blanks in configuration.txt
Christopher Faulet (21):
BUG/MINOR: rules: Initialize the list element when allocating a new rule
DOC: config: Explictly add supported MQTT versions
BUG/MEDIUM: mux-fcgi: Properly handle return value of headers/trailers
parsing
BUG/MEDIUM: mux-h1: Properly detect full buffer cases during message
parsing
BUG/MINOR: fcgi-app: Don't add C-L header on response to HEAD requests
BUG/MEDIUM: stats: Be sure to never set EOM flag on an empty HTX message
BUG/MEDIUM: hlua: Don't set EOM flag on an empty HTX message in HTTP
applet
BUG/MEDIUM: promex: Be sure to never set EOM flag on an empty HTX message
BUG/MEDIUM: mux-h1: Set outgoing message to DONE when payload length is
reached
BUG/MEDIUM: http-conv: Fix url_enc() to not crush const samples
BUG/MEDIUM: http-act: Don't replace URI if path is not found or invalid
BUG/MEDIUM: mux-h1: Don't request more room on partial trailers
BUG/MEDIUM: fcgi-app: Use http_msg flags to know if C-L header can be
added
BUG/MEDIUM: compression: Don't forget to update htx_sl and http_msg flags
BUG/MINOR: cache: Disable cache if applet creation fails
BUG/MAJOR: connection: Never remove connection from idle lists outside
the lock
BUG/MINOR: rules: Forbid captures in defaults section if used by a backend
BUG/MEDIUM: rules: Be able to use captures defined in defaults section
BUG/MINOR: rules: Fix check_capture() function to use the right rule
arguments
Revert "CI: github actions: disable -Wno-deprecated"
REGTESTS: fix the race conditions in be2dec.vtc ad field.vtc
Dhruv Jain (1):
MEDIUM: mqtt: support mqtt_is_valid and mqtt_field_value converters for
MQTTv3.1
Ilya Shipitsin (4):
CI: github actions: switch to LibreSSL-3.5.1
REGTESTS: ssl: use X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY for cert
check
CI: github actions: update OpenSSL to 3.0.2
CI: cirrus: switch to FreeBSD-13.0
Lukas Tribus (1):
DOC: reflect H2 timeout changes
Miroslav Zagorac (16):
BUG/MINOR: opentracing: setting the return value in function
flt_ot_var_set()
BUG/BUILD: opentracing: fixed OT_DEFINE variable setting
EXAMPLES: opentracing: refined shell scripts for testing filter
performance
DOC: opentracing: co
[ANNOUNCE] haproxy-2.4.16
e usual URLs below :
Site index : http://www.haproxy.org/
Documentation: http://docs.haproxy.org/
Wiki : https://github.com/haproxy/wiki/wiki
Discourse: http://discourse.haproxy.org/
Slack channel: https://slack.haproxy.org/
Issue tracker: https://github.com/haproxy/haproxy/issues
Sources : http://www.haproxy.org/download/2.4/src/
Git repository : http://git.haproxy.org/git/haproxy-2.4.git/
Git Web browsing : http://git.haproxy.org/?p=haproxy-2.4.git
Changelog: http://www.haproxy.org/download/2.4/src/CHANGELOG
Pending bugs : http://www.haproxy.org/l/pending-bugs
Reviewed bugs: http://www.haproxy.org/l/reviewed-bugs
Code reports : http://www.haproxy.org/l/code-reports
---
Complete changelog :
Christopher Faulet (19):
DOC: config: Explictly add supported MQTT versions
BUG/MEDIUM: mux-fcgi: Properly handle return value of headers/trailers
parsing
BUG/MEDIUM: mux-h1: Properly detect full buffer cases during message
parsing
BUG/MINOR: fcgi-app: Don't add C-L header on response to HEAD requests
BUG/MEDIUM: stats: Be sure to never set EOM flag on an empty HTX message
BUG/MEDIUM: hlua: Don't set EOM flag on an empty HTX message in HTTP
applet
BUG/MEDIUM: promex: Be sure to never set EOM flag on an empty HTX message
BUG/MEDIUM: mux-h1: Set outgoing message to DONE when payload length is
reached
BUG/MEDIUM: http-conv: Fix url_enc() to not crush const samples
BUG/MEDIUM: http-act: Don't replace URI if path is not found or invalid
BUG/MEDIUM: mux-h1: Don't request more room on partial trailers
BUG/MEDIUM: fcgi-app: Use http_msg flags to know if C-L header can be
added
BUG/MEDIUM: compression: Don't forget to update htx_sl and http_msg flags
BUG/MINOR: cache: Disable cache if applet creation fails
BUG/MAJOR: connection: Never remove connection from idle lists outside
the lock
BUG/MINOR: rules: Fix check_capture() function to use the right rule
arguments
REGTESTS: fix the race conditions in be2dec.vtc ad field.vtc
CLEANUP: acl: Remove unused variable when releasing an acl expression
BUILD: opentracing: Fix OT build due to misuse of var_clear()
Dhruv Jain (1):
MEDIUM: mqtt: support mqtt_is_valid and mqtt_field_value converters for
MQTTv3.1
Ilya Shipitsin (3):
CI: github actions: switch to LibreSSL-3.5.1
CI: github actions: update OpenSSL to 3.0.2
CI: cirrus: switch to FreeBSD-13.0
Lukas Tribus (1):
DOC: reflect H2 timeout changes
Miroslav Zagorac (8):
BUG/MINOR: opentracing: setting the return value in function
flt_ot_var_set()
EXAMPLES: opentracing: refined shell scripts for testing filter
performance
DOC: opentracing: corrected comments in function descriptions
CLEANUP: opentracing: removed unused function flt_ot_var_unset()
CLEANUP: opentracing: removed unused function flt_ot_var_get()
CLEANUP: opentracing: added flt_ot_smp_init() function
CLEANUP: opentracing: added variable to store variable length
DEBUG: opentracing: show return values of all functions in the debug
output
Tim Duesterhus (3):
CI: Update to actions/checkout@v3
CI: Update to actions/cache@v3
BUG/MINOR: resolvers: Fix memory leak in resolvers_deinit()
William Lallemand (3):
BUG/MINOR: tools: fix url2sa return value with IPv4
BUG/MINOR: server/ssl: free the SNI sample expression
BUG/MINOR: tools: url2sa reads too far when no port nor path
Willy Tarreau (27):
BUG/MEDIUM: mux-h1: only turn CO_FL_ERROR to CS_FL_ERROR with empty ibuf
BUG/MEDIUM: trace: avoid race condition when retrieving session from
conn->owner
BUG/MEDIUM: stream-int: do not rely on the connection error once
established
MEDIUM: mux-h2: slightly relax timeout management rules
BUG/MEDIUM: mux-h2: make use of http-request and keep-alive timeouts
BUG/MINOR: samples: add missing context names for sample fetch functions
BUG/MINOR: cli/stream: fix "shutdown session" to iterate over all threads
BUG/MAJOR: mux_pt: always report the connection error to the conn_stream
BUG/MINOR: mux-h2: do not send GOAWAY if SETTINGS were not sent
BUG/MINOR: cache: do not display expired entries in "show cache"
BUILD: debug: mark the __start_mem_stats/__stop_mem_stats symbols as weak
BUG/MINOR: mux-h2: do not use timeout http-keep-alive on backend side
BUG/MINOR: mux-h2: use timeout http-request as a fallback for
http-keep-alive
BUILD: sched: workaround crazy and dangerous warning in Clang 14
BUILD: compiler: use a more portable set of asm(".weak") statements
BUG/MEDIUM: stream: do not abort connection setup too early
SCRIPTS: announce-release: update the doc's URL
DOC: lua: update a few doc URLs
[ANNOUNCE] haproxy-2.3.20
Hi, HAProxy 2.3.20 was released on 2022/04/29. It added 41 new commits after version 2.3.19. The 2.3 branch was planned to be EOL last quarter. There are no longer bug reports for this specific branch. Thus, it is probably the last 2.3 release. Except if there are critical bugs in next few weeks, no further release should be expected. You should have no reason to deploy it anymore in a production environment. Use the 2.4 instead. No specific support should no longer be expected on the 2.3. Here are main changes for this release, cut-pasted from 2.4.16 announce: * An internal issue leading to truncated messages was fixed. When data were mixed with an error report, connection errors could be handled too early by the stream-interface. Now connection errors are only considered by the stream-interface during the connection establishment. After that, it relies on the conn-stream to be notified of any error. * An issue in the pass-through multiplexer, exposed by the previous fix, and that may lead to a loop at 100% CPU was fixed. Connection error was not properly reported to the conn-stream on the sending path. * An issue with the FCGI multiplexer when the response is compressed was fixed. The FCGI application was rewriting the response headers modifying HTX flags while the compression filter was doing so by modifying the HTTP message flags. Thus some modification performed on a side were not detected by the other, leading to produce invalid responses. Now, the flags of both structures are systematically updated. * An issue with responses to HEAD requests sent to FCGI servers was fixed. A "Content-Length: 0" header was erroneously added on the bodyless responses while it should not. Indeed, if the expected payload size is not specified by the server, HAProxy must not add this header because it cannot know it. In addition, still in the FCGI multiplexer, the parsing of headers and trailers was fixed to properly handle parsing errors. * Two issues in the H1 multiplexer were fixed. First, Connection error was reported to early, when there were still pending data for the stream. Because of this bug, last pending data could be truncated. Now the connection error is reported only if there is no pending data. The second issue is a problem about full buffer detection during the trailers parsing. Because of this bug, it was possible to block the message parsing till the timeout expiration. The same bug was fixed about processing of EOM block. * Some issues in the H2 multiplexers were fixed. First the GOAWAY frame is no longer sent if SETTINGS were not sent. Then, as announced, the "timeout http-keep-alive" and "timeout http-request" are now respected and work as documented, so that it will finally be possible to force such connections to be closed when no request comes even if they're seeing control traffic such as PING frames. This can typically happen in some server-to-server communications whereby the client application makes use of PING frames to make sure the connection is still alive. * A crash of HAproxy was fixed. It happened when HAproxy was compiled without the PCRE/PCRE2 support if it tried to replace part of the uri while the path is invalid or not specified. * An issue with url_enc() converter was fixed. It was able to crush HTTP headers. It is now fixed. * Expired entries were displayed in "show cache" output. These entries are now evicted instead of being listed. Thanks everyone for your help and your contributions ! Please find the usual URLs below : Site index : http://www.haproxy.org/ Documentation: http://docs.haproxy.org/ Wiki : https://github.com/haproxy/wiki/wiki Discourse: http://discourse.haproxy.org/ Slack channel: https://slack.haproxy.org/ Issue tracker: https://github.com/haproxy/haproxy/issues Sources : http://www.haproxy.org/download/2.3/src/ Git repository : http://git.haproxy.org/git/haproxy-2.3.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-2.3.git Changelog: http://www.haproxy.org/download/2.3/src/CHANGELOG Pending bugs : http://www.haproxy.org/l/pending-bugs Reviewed bugs: http://www.haproxy.org/l/reviewed-bugs Code reports : http://www.haproxy.org/l/code-reports --- Complete changelog : Christopher Faulet (14): BUG/MEDIUM: mux-fcgi: Properly handle return value of headers/trailers parsing BUG/MEDIUM: mux-h1: Properly detect full buffer cases during message parsing BUG/MEDIUM: mux-h1: Properly detect full buffer cases when adding EOM block BUG/MINOR: fcgi-app: Don't add C-L header on response to HEAD requests BUG/MEDIUM: http-conv: Fix url_enc() to not crush const samples BUG/MEDIUM: http-act: Don't replace URI if path is not found or invalid
[ANNOUNCE] haproxy-2.2.23
Hi, HAProxy 2.2.23 was released on 2022/05/06. It added 40 new commits after version 2.2.22. This release contains more or less the same fixes than the 2.3.20: * An internal issue leading to truncated messages was fixed. When data were mixed with an error report, connection errors could be handled too early by the stream-interface. Now connection errors are only considered by the stream-interface during the connection establishment. After that, it relies on the conn-stream to be notified of any error. * An issue in the pass-through multiplexer, exposed by the previous fix, and that may lead to a loop at 100% CPU was fixed. Connection error was not properly reported to the conn-stream on the sending path. * An issue with the FCGI multiplexer when the response is compressed was fixed. The FCGI application was rewriting the response headers modifying HTX flags while the compression filter was doing so by modifying the HTTP message flags. Thus some modification performed on a side were not detected by the other, leading to produce invalid responses. Now, the flags of both structures are systematically updated. * An issue with responses to HEAD requests sent to FCGI servers was fixed. A "Content-Length: 0" header was erroneously added on the bodyless responses while it should not. Indeed, if the expected payload size is not specified by the server, HAProxy must not add this header because it cannot know it. In addition, still in the FCGI multiplexer, the parsing of headers and trailers was fixed to properly handle parsing errors. * Two issues in the H1 multiplexer were fixed. First, Connection error was reported to early, when there were still pending data for the stream. Because of this bug, last pending data could be truncated. Now the connection error is reported only if there is no pending data. The second issue is a problem about full buffer detection during the trailers parsing. Because of this bug, it was possible to block the message parsing till the timeout expiration. The same bug was fixed about processing of EOM block. * Some issues in the H2 multiplexers were fixed. First the GOAWAY frame is no longer sent if SETTINGS were not sent. Then, as announced, the "timeout http-keep-alive" and "timeout http-request" are now respected and work as documented, so that it will finally be possible to force such connections to be closed when no request comes even if they're seeing control traffic such as PING frames. This can typically happen in some server-to-server communications whereby the client application makes use of PING frames to make sure the connection is still alive. * A crash of HAproxy was fixed. It happened when HAproxy was compiled without the PCRE/PCRE2 support if it tried to replace part of the uri while the path is invalid or not specified. * An issue with url_enc() converter was fixed. It was able to crush HTTP headers. It is now fixed. * Expired entries were displayed in "show cache" output. These entries are now evicted instead of being listed. Thanks everyone for your help and your contributions ! Please find the usual URLs below : Site index : http://www.haproxy.org/ Documentation: http://docs.haproxy.org/ Wiki : https://github.com/haproxy/wiki/wiki Discourse: http://discourse.haproxy.org/ Slack channel: https://slack.haproxy.org/ Issue tracker: https://github.com/haproxy/haproxy/issues Sources : http://www.haproxy.org/download/2.2/src/ Git repository : http://git.haproxy.org/git/haproxy-2.2.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-2.2.git Changelog: http://www.haproxy.org/download/2.2/src/CHANGELOG Pending bugs : http://www.haproxy.org/l/pending-bugs Reviewed bugs: http://www.haproxy.org/l/reviewed-bugs Code reports : http://www.haproxy.org/l/code-reports Latest builds: http://www.haproxy.org/l/dev-packages --- Complete changelog : Christopher Faulet (12): BUG/MEDIUM: mux-fcgi: Properly handle return value of headers/trailers parsing BUG/MEDIUM: mux-h1: Properly detect full buffer cases during message parsing BUG/MEDIUM: mux-h1: Properly detect full buffer cases when adding EOM block BUG/MINOR: fcgi-app: Don't add C-L header on response to HEAD requests BUG/MEDIUM: http-conv: Fix url_enc() to not crush const samples BUG/MEDIUM: http-act: Don't replace URI if path is not found or invalid BUG/MEDIUM: mux-h1: Don't request more room on partial trailers BUG/MEDIUM: fcgi-app: Use http_msg flags to know if C-L header can be added BUG/MEDIUM: compression: Don't forget to update htx_sl and http_msg flags BUG/MINOR: cache: Disable cache if applet creation fails BUG
Re: [PATCH] BUG/MEDIUM: lua: fix argument handling in data removal functions
Le 5/10/22 à 19:47, Boyang Li a écrit :
Lua API Channel.remove() and HTTPMessage.remove() expects 1 to 3
arguments (counting the manipulated object), with offset and length
being the 2nd and 3rd argument, respectively.
hlua_{channel,http_msg}_del_data() incorrectly gets the 3rd argument as
offset, and 4th (nonexistent) as length. hlua_http_msg_del_data() also
improperly checks arguments. This patch fixes argument handling in both.
Must be backported to 2.5.
Thanks ! Both patches were merged.
--
Christopher Faulet
[ANNOUNCE] haproxy-2.5.7
HANGELOG
Pending bugs : http://www.haproxy.org/l/pending-bugs
Reviewed bugs: http://www.haproxy.org/l/reviewed-bugs
Code reports : http://www.haproxy.org/l/code-reports
Latest builds: http://www.haproxy.org/l/dev-packages
---
Complete changelog :
Boyang Li (2):
BUG/MEDIUM: lua: fix argument handling in data removal functions
DOC/MINOR: fix typos in the lua-api document
Christopher Faulet (7):
BUG/MEDIUM: http-ana: Fix memleak in redirect rules with ignore-empty
option
BUG/MEDIUM: httpclient: Fix loop consuming HTX blocks from the response
channel
BUG/MEDIUM: mux-fcgi: Be sure to never set EOM flag on an empty HTX
message
BUG/MEDIUM: mux-h1: Be able to handle trailers when C-L header was
specified
DOC: config: Update doc for PR/PH session states to warn about rewrite
failures
MINOR: mux-h1: Add global option accpet payload for any HTTP/1.0 requests
CLEANUP: mux-h1: Fix comments and error messages for global options
Emeric Brun (1):
BUG/MAJOR: dns: multi-thread concurrency issue on UDP socket
Ilya Shipitsin (2):
CI: github actions: update LibreSSL to 3.5.2
CI: dynamically determine actual version of h2spec
Remi Tricot-Le Breton (2):
MINOR: connection: Add way to disable active connection closing during
soft-stop
BUG/MINOR: ssl: Fix typos in crl-file related CLI commands
Thomas Prückl (1):
MINOR: ssl: add a new global option "tune.ssl.hard-maxrecord"
Tim Duesterhus (1):
BUG/MINOR: resolvers: Fix memory leak in resolvers_deinit()
William Lallemand (4):
BUG/MINOR: tcp/http: release the expr of set-{src,dst}[-port]
BUG/MINOR: startup: usage() when no -cc arguments
BUG/MEDIUM: ssl/cli: fix yielding in show_cafile_detail
BUG/MEDIUM: wdt: don't trigger the watchdog when p is unitialized
Willy Tarreau (22):
BUILD: compiler: properly distinguish weak and global symbols
BUG/MINOR: pools: make sure to also destroy shared pools in
pool_destroy_all()
SCRIPTS: announce-release: add URL of dev packages
BUG/MINOR: mux-h2: mark the stream as open before processing it not after
MINOR: mux-h2: report a trace event when failing to create a new stream
BUG/MEDIUM: resolvers: make "show resolvers" properly yield
BUG/MEDIUM: cli: make "show cli sockets" really yield
BUG/MINOR: proxy/cli: don't enumerate internal proxies on "show backend"
BUG/MINOR: map/cli: protect the backref list during "show map" errors
BUG/MINOR: map/cli: make sure patterns don't vanish under "show map"'s
init
BUG/MINOR: ssl/cli: fix "show ssl ca-file/crl-file" not to mix cli+ssl
contexts
BUG/MINOR: ssl/cli: fix "show ssl ca-file " not to mix cli+ssl
contexts
BUG/MINOR: ssl/cli: fix "show ssl crl-file" not to mix cli+ssl contexts
BUG/MINOR: ssl/cli: fix "show ssl cert" not to mix cli+ssl contexts
DOC: fix typo "ant" for "and" in INSTALL
BUILD: ssl: work around bogus warning in gcc 12's -Wformat-truncation
BUILD: debug: work around gcc-12 excessive -Warray-bounds warnings
BUILD: listener: shut report of possible null-deref in listener_accept()
BUG/MEDIUM: ssl: fix the gcc-12 broken fix :-(
DOC: install: update gcc version requirements
BUG/MINOR: conn_stream: do not confirm a connection from the frontend path
CLEANUP: applet: make appctx_new() initialize the whole appctx
vigneshsp (1):
BUG/MINOR: server: Make SRV_STATE_LINE_MAXLEN value from 512 to 2kB (2000
bytes).
--
Christopher Faulet
[ANNOUNCE] haproxy-2.4.17
Hi,
HAProxy 2.4.17 was released on 2022/05/13. It added 24 new commits
after version 2.4.16.
Here are the issues fixed by this release:
* A regression in the H1 multiplexer was fixed. If an H2 message announced
the payload size with a Content-Length header and contained trailers, an
internal error was triggered during forwarding on the other side, in the
H1 multiplexer.
* A major issue in the DNS part was fixed. A concurrency issue that could
lead to a crash when a DNS request was failing. Because of some missing
locks on dgram structure, it was possible to set the UDP socket FD to -1
on a thread while it as used to access to fdtab array on another thread.
* A server abort or a server timeout could be experienced with FCGI backend
connections when the END_REQUEST record was delayed for responses with no
content-length.
* A timing issue could lead to some delay in the server-side connection
establishment. It was a tricky issue, but sometimes the server-side
connection attempts were only validated after the "timeout connect"
value, and only with H2 clients.
* H2 streams were marked as open after processing it instead of before. It
could be an issue when a client didn't respect the H2
MAX_CONCURRENT_STREAMS setting because the max_id was only updated on the
success path. Thus, under some circumstances a connection error was
reported instead of a stream error.
* The watchdog could be erroneously triggered because an uninitialized value
was not tested. It was possible to encounter this issue in the master
just after loading the configuration.
* It was reported the maximum line length on the server-state file was too
small. It was increased to 2kB.
* Some bugs in CLI commands were fixed. "show resolvers" and "show cli
sockets" commands were not properly yielding and some locks were missing
in "show map" command. It is very unlikely to have ever hit one of these
bugs, but not impossible though.
Thanks everyone for your help and your contributions !
Please find the usual URLs below :
Site index : http://www.haproxy.org/
Documentation: http://docs.haproxy.org/
Wiki : https://github.com/haproxy/wiki/wiki
Discourse: http://discourse.haproxy.org/
Slack channel: https://slack.haproxy.org/
Issue tracker: https://github.com/haproxy/haproxy/issues
Sources : http://www.haproxy.org/download/2.4/src/
Git repository : http://git.haproxy.org/git/haproxy-2.4.git/
Git Web browsing : http://git.haproxy.org/?p=haproxy-2.4.git
Changelog: http://www.haproxy.org/download/2.4/src/CHANGELOG
Pending bugs : http://www.haproxy.org/l/pending-bugs
Reviewed bugs: http://www.haproxy.org/l/reviewed-bugs
Code reports : http://www.haproxy.org/l/code-reports
Latest builds: http://www.haproxy.org/l/dev-packages
---
Complete changelog :
Christopher Faulet (4):
BUG/MEDIUM: mux-fcgi: Be sure to never set EOM flag on an empty HTX
message
BUG/MEDIUM: mux-h1: Be able to handle trailers when C-L header was
specified
DOC: config: Update doc for PR/PH session states to warn about rewrite
failures
CLEANUP: mux-h1: Fix comments and error messages for global options
Emeric Brun (1):
BUG/MAJOR: dns: multi-thread concurrency issue on UDP socket
Ilya Shipitsin (2):
CI: github actions: update LibreSSL to 3.5.2
CI: dynamically determine actual version of h2spec
William Lallemand (2):
BUG/MINOR: tcp/http: release the expr of set-{src,dst}[-port]
BUG/MEDIUM: wdt: don't trigger the watchdog when p is unitialized
Willy Tarreau (14):
SCRIPTS: announce-release: add URL of dev packages
BUG/MINOR: mux-h2: mark the stream as open before processing it not after
MINOR: mux-h2: report a trace event when failing to create a new stream
BUG/MEDIUM: resolvers: make "show resolvers" properly yield
BUG/MEDIUM: cli: make "show cli sockets" really yield
BUG/MINOR: map/cli: protect the backref list during "show map" errors
BUG/MINOR: map/cli: make sure patterns don't vanish under "show map"'s
init
DOC: fix typo "ant" for "and" in INSTALL
BUILD: ssl: work around bogus warning in gcc 12's -Wformat-truncation
BUILD: debug: work around gcc-12 excessive -Warray-bounds warnings
BUILD: listener: shut report of possible null-deref in listener_accept()
BUG/MEDIUM: ssl: fix the gcc-12 broken fix :-(
DOC: install: update gcc version requirements
BUG/MINOR: conn_stream: do not confirm a connection from the frontend path
vigneshsp (1):
BUG/MINOR: server: Make SRV_STATE_LINE_MAXLEN value from 512 to 2kB (2000
bytes).
--
Christopher Faulet
[ANNOUNCE] haproxy-2.2.24
Hi,
HAProxy 2.2.24 was released on 2022/05/13. It added 11 new commits
after version 2.2.23.
Here are the issues fixed by this release:
* A major issue in the DNS part was fixed. A concurrency issue that could
lead to a crash when a DNS request was failing. Because of some missing
locks on dgram structure, it was possible to set the UDP socket FD to -1
on a thread while it as used to access to fdtab array on another thread.
* The watchdog could be erroneously triggered because an uninitialized value
was not tested. It was possible to encounter this issue in the master
just after loading the configuration.
* H2 streams were marked as open after processing it instead of before. It
could be an issue when a client didn't respect the H2
MAX_CONCURRENT_STREAMS setting because the max_id was only updated on the
success path. Thus, under some circumstances an connection error was
reported instead of a stream error.
* It was reported the maximum line length on the server-state file was too
small. It was increased to 2kB.
* Some bugs in CLI commands were fixed. "show resolvers" and "show cli
sockets" commands were not properly yielding and some locks were missing
in "show map" command. It is very unlikely to have ever hit one of these
bugs, but not impossible though.
Thanks everyone for your help and your contributions !
Please find the usual URLs below :
Site index : http://www.haproxy.org/
Documentation: http://docs.haproxy.org/
Wiki : https://github.com/haproxy/wiki/wiki
Discourse: http://discourse.haproxy.org/
Slack channel: https://slack.haproxy.org/
Issue tracker: https://github.com/haproxy/haproxy/issues
Sources : http://www.haproxy.org/download/2.2/src/
Git repository : http://git.haproxy.org/git/haproxy-2.2.git/
Git Web browsing : http://git.haproxy.org/?p=haproxy-2.2.git
Changelog: http://www.haproxy.org/download/2.2/src/CHANGELOG
Pending bugs : http://www.haproxy.org/l/pending-bugs
Reviewed bugs: http://www.haproxy.org/l/reviewed-bugs
Code reports : http://www.haproxy.org/l/code-reports
Latest builds: http://www.haproxy.org/l/dev-packages
---
Complete changelog :
Christopher Faulet (2):
DOC: config: Update doc for PR/PH session states to warn about rewrite
failures
CLEANUP: mux-h1: Fix comments and error messages for global options
Emeric Brun (1):
BUG/MAJOR: dns: multi-thread concurrency issue on UDP socket
William Lallemand (2):
BUG/MINOR: tcp/http: release the expr of set-{src,dst}[-port]
BUG/MEDIUM: wdt: don't trigger the watchdog when p is unitialized
Willy Tarreau (5):
BUG/MINOR: mux-h2: mark the stream as open before processing it not after
BUG/MEDIUM: cli: make "show cli sockets" really yield
BUG/MINOR: map/cli: protect the backref list during "show map" errors
BUG/MINOR: map/cli: make sure patterns don't vanish under "show map"'s
init
DOC: fix typo "ant" for "and" in INSTALL
vigneshsp (1):
BUG/MINOR: server: Make SRV_STATE_LINE_MAXLEN value from 512 to 2kB (2000
bytes).
--
Christopher Faulet
[ANNOUNCE] haproxy-2.0.29
Hi, HAProxy 2.0.29 was released on 2022/05/13. It added 41 new commits after version 2.0.28. Here are the issues fixed by this release: * An internal issue leading to truncated messages was fixed. When data were mixed with an error report, connection errors could be handled too early by the stream-interface. Now connection errors are only considered by the stream-interface during the connection establishment. After that, it relies on the conn-stream to be notified of any error. * An issue in the pass-through multiplexer, exposed by the previous fix, and that may lead to a loop at 100% CPU was fixed. Connection error was not properly reported to the conn-stream on the sending path. * Still on the pass-through multiplexer, a fix of the previous release was reverted because it introduced a regression in legacy HTTP mode. A crash could be experienced when a keep-alive backend connection was reused. While the fix is valid for higher versions, it is not applicable for this one. * A major issue in the DNS part was fixed. A concurrency issue that could lead to a crash when a DNS request was failing. Because of some missing locks on dgram structure, it was possible to set the UDP socket FD to -1 on a thread while it as used to access to fdtab array on another thread. * Two issues in the H1 multiplexer were fixed. First, Connection error was reported to early, when there were still pending data for the stream. Because of this bug, last pending data could be truncated. Now the connection error is reported only if there is no pending data. The second issue is a problem about full buffer detection during the trailers parsing. Because of this bug, it was possible to block the message parsing till the timeout expiration. The same bug was fixed about processing of EOM block. * Some issues in the H2 multiplexers were fixed. First the GOAWAY frame is no longer sent if SETTINGS were not sent. Then, as announced, the "timeout http-keep-alive" and "timeout http-request" are now respected and work as documented, so that it will finally be possible to force such connections to be closed when no request comes even if they're seeing control traffic such as PING frames. This can typically happen in some server-to-server communications whereby the client application makes use of PING frames to make sure the connection is still alive. * A crash of HAproxy was fixed. It happened when HAproxy was compiled without the PCRE/PCRE2 support if it tried to replace part of the uri while the path is invalid or not specified. * Some bugs in CLI commands were fixed. "show resolvers" and "show cli sockets" commands were not properly yielding and some locks were missing in "show map" command. It is very unlikely to have ever hit one of these bugs, but not impossible though. In addition, expired entries were displayed in "show cache" output. These entries are now evicted instead of being listed. * The watchdog could be erroneously triggered because an unitialized value was not tested. It was possible to encounter this issue in the master just after loading the configuration. * It was reported the maximum line length on the server-state file was too small. It was increased to 2kB. * Some shared pools were not properly released on exit. * An improvement which is not related to the code, with the precious help of Tim and Cyril, we could finally set up an automatic generation of the HTML documentation. It's performed daily and published on github pages at http://docs.haproxy.org. Thanks everyone for your help and your contributions! Please find the usual URLs below : Site index : http://www.haproxy.org/ Documentation: http://docs.haproxy.org/ Wiki : https://github.com/haproxy/wiki/wiki Discourse: http://discourse.haproxy.org/ Slack channel: https://slack.haproxy.org/ Issue tracker: https://github.com/haproxy/haproxy/issues Sources : http://www.haproxy.org/download/2.0/src/ Git repository : http://git.haproxy.org/git/haproxy-2.0.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-2.0.git Changelog: http://www.haproxy.org/download/2.0/src/CHANGELOG Pending bugs : http://www.haproxy.org/l/pending-bugs Reviewed bugs: http://www.haproxy.org/l/reviewed-bugs Code reports : http://www.haproxy.org/l/code-reports Latest builds : http://www.haproxy.org/l/dev-packages --- Complete changelog : Christopher Faulet (7): Revert "BUG/MAJOR: mux-pt: Always destroy the backend connection on detach" BUG/MEDIUM: http-act: Don't replace URI if path is not found or invalid BUG/MEDIUM: mux-h1: Don't request more room on partial trailers BUG/MEDIUM: compression: Don't forget to update htx_sl a
Re: Peers using heavily single cpu core
Le 4/20/22 à 14:51, Maciej Zdeb a écrit : Hi Willy, I saw Christopher changes are now merged. I was wondering how to proceed with my issue. Right now in stream_new() I'm able to get cs_endpoint and appctx (if endpoint is applet), so I can get thread_mask of appctx to create a stream task on the same thread. Is this approach correct? Hi Maciej, I've finally finish my applet refactoring. Now it is possible to choose where to start an applet. I've also merged your patches. Peer applets must now be balanced across threads. So, you may give it a try to be sure it solves your issue and also validate everything works fine. -- Christopher Faulet
Re: SV: Traffic from HAproxy get error 401 and 500
Le 6/1/22 à 23:39, Henning Svane a écrit : Hi I have tried haproxy -d and here I saw 401 and 500. But I have also seen this, but I have and Error I do not how to fix: odin@haproxy01:~$ sudo haproxy -d -f /home/odin/haproxy07e.cfg Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result FAILED Total: 3 (2 usable), will use epoll. Available filters : [SPOE] spoe [CACHE] cache [FCGI] fcgi-app [COMP] compression [TRACE] trace Using epoll() as the polling mechanism. But to your question I have attached the file debug.txt which are the output from haproxy -d whenI try to open Outlook. There are some errors but I do not what they mean. Hi, You should specify your HAProxy version and share your redacted configuration. Otherwise it will be hard to help you. Log messages may also help. Note that NTLM authenticates a connection not a request. Thus you must be sure to keep connections alive. This may explain your 401 responses. About the 500 errors, it may be because of header rewrite failures. You may check 'wrew' counter in "show stat" command output or in the csv export of the HTTP state page. Regards, -- Christopher Faulet
Re: [PATCH] CLEANUP: Re-apply xalloc_size.cocci (2)
Le 6/1/22 à 21:58, Tim Duesterhus a écrit : This reapplies the xalloc_size.cocci patch across the whole `src/` tree. Merged. Thanks ! -- Christopher Faulet
Re: Ability to set response in lua via HTTP class
Le 6/8/22 à 15:22, Yuriy Ivkin a écrit : Greetings! Is the TXN.done(txn[, reply]) function in haproxy 2.6 what I am looking for ? 08.06.2022 12:53, Yuriy Ivkin пишет: Greetings! In the lua HTTP class did not find a method allows to override response body. May be it is possible via some other methods ? If not, are there any reasons should prevent me to write it ? Hi, It depends on what you want to achieve. If you want to return a custom response, you can indeed register a lua action and call it in an http-response rule. You can create a Reply object with some original response headers, if necessary, and with a custom body. In this case, TXN.done() should be used. However, this solution is limited to one buffer. You cannot stream the response payload. In the same way, you must wait for the response body to rewrite it. But only if the whole response fits in a buffer. With this solution, it is not possible to rewrite a too big response body. If you want to rewrite a large response, you must write a lua filter. It is more difficult. The API is a bit rough. But it is more versatile. -- Christopher Faulet
Re: Segfault on 2.6.0 with TCP switching to HTTP/2
Le 6/16/22 à 05:12, David Leadbeater a écrit : I tried upgrading to 2.6.0 (from 2.5.6) and I'm seeing a segfault when making HTTP/2 requests. I'm using a frontend in TCP mode and then switching it to HTTP/2. I've made a minimal config that exhibits the segfault, below. Simply doing curl -vk https://ip is enough to trigger it for me. Thread 1 "haproxy" received signal SIGSEGV, Segmentation fault. 0x555d1d07 in h2s_close (h2s=0x55a60b70) at src/mux_h2.c:1497 1497 HA_ATOMIC_DEC(&h2s->h2c->px_counters->open_streams); (gdb) bt #0 0x555d1d07 in h2s_close (h2s=0x55a60b70) at src/mux_h2.c:1497 #1 h2s_destroy (h2s=0x55a60b70) at src/mux_h2.c:1515 #2 0x555d3463 in h2_detach (sd=) at src/mux_h2.c:4432 The exact backtrace varies but always in h2s_destroy. (In case you're wondering what on earth I'm doing, there's a write-up of it at https://dgl.cx/2022/04/showing-you-your-actual-http-request) David --- global ssl-default-bind-options no-sslv3 no-tlsv10 user nobody defaults timeout connect 10s timeout client 30s timeout server 2m frontend tcp-https mode tcp bind [::]:443 v4v6 ssl crt /etc/haproxy/ssl/bodge.cloud.pem alpn h2,http/1.1 acl ipwtf hdr(Host),lower,field(1,:),word(-1,.,2) ip.wtf default_backend ipwtf tcp-request inspect-delay 10s tcp-request content switch-mode http if !ipwtf use_backend cloud-regions.bodge.cloud if !ipwtf backend ipwtf mode tcp server ipwtf localhost:8080 backend cloud-regions.bodge.cloud mode http server cr localhost:8080 Hi, Thanks ! I'm able to reproduce the segfault. I'm on it. -- Christopher Faulet
Re: Segfault on 2.6.0 with TCP switching to HTTP/2
Le 6/16/22 à 05:12, David Leadbeater a écrit : I tried upgrading to 2.6.0 (from 2.5.6) and I'm seeing a segfault when making HTTP/2 requests. I'm using a frontend in TCP mode and then switching it to HTTP/2. I've made a minimal config that exhibits the segfault, below. Simply doing curl -vk https://ip is enough to trigger it for me. Thread 1 "haproxy" received signal SIGSEGV, Segmentation fault. 0x555d1d07 in h2s_close (h2s=0x55a60b70) at src/mux_h2.c:1497 1497 HA_ATOMIC_DEC(&h2s->h2c->px_counters->open_streams); (gdb) bt #0 0x555d1d07 in h2s_close (h2s=0x55a60b70) at src/mux_h2.c:1497 #1 h2s_destroy (h2s=0x55a60b70) at src/mux_h2.c:1515 #2 0x555d3463 in h2_detach (sd=) at src/mux_h2.c:4432 The exact backtrace varies but always in h2s_destroy. (In case you're wondering what on earth I'm doing, there's a write-up of it at https://dgl.cx/2022/04/showing-you-your-actual-http-request) I pushed a fix in 2.7-dev : https://github.com/haproxy/haproxy/commit/82af3c68 It will be backported to 2.6 soon. A 2.6.1 should be released the next week. Thanks ! -- Christopher Faulet
Re: Haproxy [2.4.0] cvs stats return " * L7OK,0 " in check_status and check_code fields
Le 6/23/22 à 22:51, Rashid B. a écrit : Hi all, Maybe someone have already encountered the same issue: Sometimes I get " * L7OK" and "0" in check_status and check_code fields respectively in stats in cvs format response. The full response looks as follows: 'write_read,hostname1,0,0,1,2,500,866,238067,642557,,0,,0,0,0,0,UP,1,1,0,0,0,97441,0,,1,2,2,,866,,2,0,,1,** L7OK,0*,3,,,0,0,49,,,0,0,0,112047Layer7 check passed,,2,3,4,,tcp0,866,0,,,0,,0,7,0,540337,0,0,0,0,1,1,-,0,0,0,,\n' My configuration is as follows: listen write_read bind *:5000 option httpchk http-check expect status 200 default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions server hostname1 hostname1:1234 maxconn 500 check port 0910 inter 1s HAProxy version 2.4.0-6cbbecf 2021/05/14 CentOS Linux release 7.9.2009 Any help is appreciated. Thanks in advance. Hi, When the health-check is running, the status reported is the last known status and it is prefixed with an asterisks. In this case the code is set to 0 in the CSV output. So it is expected. The same is done for the agent-check. -- Christopher Faulet
Re: Issue - consecutive early-hint defined with "if" condition
Le 6/30/22 à 11:47, Krzysztof Kozłowski a écrit :
Hi,
I’m struggling with consecutive early-hint statements defined with if condition.
Considering below configuration:
http-request early-hint Link "<https://example.com/style1.css
<https://example.com/style1.css>>; rel="preload"; as="style"" if { path -i -m
beg /test1 }
http-request early-hint Link "<https://example.com/style2.css
<https://example.com/style2.css>>; rel="preload"; as="style"" if { path -i -m
beg /test2 }
I’m not getting HTTP 103 response to 2nd statement.
Response I’m getting with above config:
example.com/test1 <http://example.com/test1> response: HTTP/2 103 + HTTP/2 200
example.com/test2 <http://example.com/test2> response: HTTP/2 200
However if I add some dummy statement in between those early-hints like this:
http-request early-hint Link "<https://example.com/style1.css
<https://example.com/style1.css>>; rel="preload"; as="style"" if { path -i -m
beg /test1 }
http-request set-header X-Test "test-request"
http-request early-hint Link "<https://example.com/style2.css
<https://example.com/style2.css>>; rel="preload"; as="style"" if { path -i -m
beg /test2 }
I’m receiving HTTP 103 response to both statements as expected.
Response I’m getting with above config:
example.com/test1 <http://example.com/test1> response: HTTP/2 103 + HTTP/2 200
example.com/test2 <http://example.com/test2> response: HTTP/2 103 + HTTP/2 200
It looks like an issue with parsing early-hint statements which are defined with
"if” statement.
It’s my first message sent to this mailing list - so please excuse me if I miss
any information required to properly identify the problem. If I’m missing
anything - please do let me know.
I would appreciate any help here.
I’m using HAProxy version 2.4.15-7782e23
Thanks ! Indeed, I can confirm the issue. I'll try to fix it soon.
--
Christopher Faulet
Re: Issue - consecutive early-hint defined with "if" condition
Le 7/4/22 à 11:19, Christopher Faulet a écrit :
Le 6/30/22 à 11:47, Krzysztof Kozłowski a écrit :
Hi,
I’m struggling with consecutive early-hint statements defined with if condition.
Considering below configuration:
http-request early-hint Link "<https://example.com/style1.css
<https://example.com/style1.css>>; rel="preload"; as="style"" if { path -i -m
beg /test1 }
http-request early-hint Link "<https://example.com/style2.css
<https://example.com/style2.css>>; rel="preload"; as="style"" if { path -i -m
beg /test2 }
I’m not getting HTTP 103 response to 2nd statement.
Response I’m getting with above config:
example.com/test1 <http://example.com/test1> response: HTTP/2 103 + HTTP/2 200
example.com/test2 <http://example.com/test2> response: HTTP/2 200
However if I add some dummy statement in between those early-hints like this:
http-request early-hint Link "<https://example.com/style1.css
<https://example.com/style1.css>>; rel="preload"; as="style"" if { path -i -m
beg /test1 }
http-request set-header X-Test "test-request"
http-request early-hint Link "<https://example.com/style2.css
<https://example.com/style2.css>>; rel="preload"; as="style"" if { path -i -m
beg /test2 }
I’m receiving HTTP 103 response to both statements as expected.
Response I’m getting with above config:
example.com/test1 <http://example.com/test1> response: HTTP/2 103 + HTTP/2 200
example.com/test2 <http://example.com/test2> response: HTTP/2 103 + HTTP/2 200
It looks like an issue with parsing early-hint statements which are defined with
"if” statement.
It’s my first message sent to this mailing list - so please excuse me if I miss
any information required to properly identify the problem. If I’m missing
anything - please do let me know.
I would appreciate any help here.
I’m using HAProxy version 2.4.15-7782e23
Thanks ! Indeed, I can confirm the issue. I'll try to fix it soon.
Hi,
It should be fixed now in 2.7-DEV but not backported yet. The commit:
http://git.haproxy.org/?p=haproxy.git;a=commit;h=4c3d3d2
Thanks !
--
Christopher Faulet
[ANNOUNCE] haproxy-2.6.2
MINOR: h3: handle errors on HEADERS parsing/QPACK decoding MINOR: qpack: properly handle invalid dynamic table references BUG/MEDIUM: mux-quic: fix server chunked encoding response BUG/MINOR: quic: fix closing state on NO_ERROR code sent BUG/MINOR: quic: do not send CONNECTION_CLOSE_APP in initial/handshake Benoit DOLEZ (1): BUILD: quic: fix anonymous union for gcc-4.4 Brad Smith (1): BUILD: makefile: Fix install(1) handling for OpenBSD/NetBSD/Solaris/AIX Christian Ruppert (1): BUILD: Makefile: Add Lua 5.4 autodetect Christopher Faulet (16): BUG/MINOR: http-ana: Set method to HTTP_METH_OTHER when an HTTP txn is created BUG/MINOR: http-fetch: Use integer value when possible in "method" sample fetch BUG/MINOR: http-check: Preserve headers if not redefined by an implicit rule BUG/MINOR: http-act: Properly generate 103 responses when several rules are used BUG/MINOR: http-htx: Fix scheme based normalization for URIs wih userinfo MINOR: http: Add function to get port part of a host MINOR: http: Add function to detect default port BUG/MEDIUM: h1: Improve authority validation for CONNCET request MINOR: http-htx: Use new HTTP functions for the scheme based normalization BUG/MEDIUM: http-fetch: Don't fetch the method if there is no stream REGTEESTS: filters: Fix CONNECT request in random-forwarding script BUG/MINOR: mux-h1: Be sure to commit htx changes in the demux buffer BUG/MEDIUM: http-ana: Don't wait to have an empty buf to switch in TUNNEL state BUG/MEDIUM: mux-h1: Handle connection error after a synchronous send BUG/MEDIUM: stconn: Only reset connect expiration when processing backend side BUG/MINOR: backend: Fallback on RR algo if balance on source is impossible Emeric Brun (3): MINOR: fd: add a new FD_DISOWN flag to prevent from closing a deleted FD BUG/MEDIUM: ssl/fd: unexpected fd close using async engine MINOR: fd: Add BUG_ON checks on fd_insert() Frédéric Lécaille (13): BUG/MINOR: quic: Missing acknowledgments for trailing packets BUG/MINOR: quic: Wrong reuse of fulfilled dgram RX buffer BUG/MAJOR: quic: Big RX dgrams leak when fulfilling a buffer BUG/MAJOR: quic: Big RX dgrams leak with POST requests BUILD: quic+h3: 32-bit compilation errors fixes BUG/MINOR: quic: Dropped packets not counted (with RX buffers full) MINOR: quic: Add new stats counter to diagnose RX buffer overrun MINOR: quic: Duplicated QUIC_RX_BUFSZ definition MINOR: task: Add tasklet_wakeup_after() MINOR: quic: Improvements for the datagrams receipt MINOR: quic: Increase the QUIC connections RX buffer size (upto 64Kb) CLEANUP: h2: Typo fix in h2_unsubcribe() traces BUG/MAJOR: mux_quic: fix invalid PROTOCOL_VIOLATION on POST data overlap Ilya Shipitsin (1): CI: re-enable gcc asan builds Remi Tricot-Le Breton (1): BUG/MINOR: ssl: Do not look for key in extra files if already in pem William Lallemand (7): BUG/MINOR: peers: fix possible NULL dereferences at config parsing MEDIUM: mworker: set the iocb of the socketpair without using fd_insert() MINOR: resolvers: resolvers_destroy() deinit and free a resolver BUG/MINOR: resolvers: shut off the warning for the default resolvers BUG/MINOR: ssl: allow duplicate certificates in ca-file directories BUG/MINOR: mworker/cli: relative pid prefix not validated anymore BUG/MEDIUM: mworker: proc_self incorrectly set crashes upon reload Willy Tarreau (8): MEDIUM: mux-h2: try to coalesce outgoing WINDOW_UPDATE frames BUG/MINOR: peers/config: always fill the bind_conf's argument BUG/MEDIUM: cli/threads: make "show threads" more robust on applets BUG/MINOR: debug: enter ha_panic() only once BUG/MEDIUM: tools: avoid calling dlsym() in static builds BUG/MEDIUM: tools: avoid calling dlsym() in static builds (try 2) BUG/MINOR: tools: fix statistical_prng_range()'s output range BUILD: add detection for unsupported compiler models -- Christopher Faulet
[ANNOUNCE] haproxy-2.5.8
ose the FD when the it is removed from the fdtab array. * Crashes could be experienced during hot-upgrade from 2.4 to 2.6 because old worker was still identified as a running worker. * An internal error was reported when loadbalancing on source IP address was impossible. It could happens with SPOE applets or with clients connected to HAPRoxy via a unix socket. Now, when this happens, a fallback to round-robin is performed. * The HTTP scheme based normalization did not properly handle the URIs with userinfo. They were not preserved after the normalization process. * Duplicate certificates in ca-file directories were not properly handled because of an OpenSSL error. The error is now ignored. * Lookup for a private key in extra files was not ignored when it was already found in the pem file, while it should. * HAProxy could crash on old Glibc on dlsym() function call if it is statically built. Now, we avoid to call it in static builds. * Depending on the declaration order of "http-check send" and "option httpchk" directives, the configured headers could be ignored. Now a previous list of headers is replaced by a new one only if it is not empty. * Mailers healthchecks were causing a crash since the refactoring of the internal HAProxy connection stack introduced in 2.6. This is now fixed. * It was possible to crash HAProxy by defining multiple bind lines in a peers section. An error is now reported during configuration parsing. * A warning is now reported when some unsupported keywords are used in peers section instead of silently ignoring them. init_addr, resolvers, check, agent-check are concerned. * The DNS resolution is now ignored for disabled proxies preventing some crashes. Thanks everyone for your help and your contributions! Please find the usual URLs below : Site index : http://www.haproxy.org/ Documentation: http://docs.haproxy.org/ Wiki : https://github.com/haproxy/wiki/wiki Discourse: http://discourse.haproxy.org/ Slack channel: https://slack.haproxy.org/ Issue tracker: https://github.com/haproxy/haproxy/issues Sources : http://www.haproxy.org/download/2.5/src/ Git repository : http://git.haproxy.org/git/haproxy-2.5.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-2.5.git Changelog: http://www.haproxy.org/download/2.5/src/CHANGELOG Pending bugs : http://www.haproxy.org/l/pending-bugs Reviewed bugs: http://www.haproxy.org/l/reviewed-bugs Code reports : http://www.haproxy.org/l/code-reports Latest builds: http://www.haproxy.org/l/dev-packages --- Complete changelog : Brad Smith (1): BUILD: makefile: Fix install(1) handling for OpenBSD/NetBSD/Solaris/AIX Christian Ruppert (1): BUILD: Makefile: Add Lua 5.4 autodetect Christopher Faulet (51): MEDIUM: http-ana: Add a proxy option to restrict chars in request header names REGTESTS: abortonclose: Fix some race conditions BUG/MEDIUM: config: Reset outline buffer size on realloc error in readcfgfile() BUG/MINOR: check: Reinit the buffer wait list at the end of a check BUG/MEDIUM: resolvers: Don't defer resolutions release in deinit function BUG/MINOR: ssl_ckch: Free error msg if commit changes on a cert entry fails BUG/MINOR: ssl_ckch: Free error msg if commit changes on a CA/CRL entry fails BUG/MEDIUM: ssl_ckch: Don't delete a cert entry if it is being modified BUG/MEDIUM: ssl_ckch: Don't delete CA/CRL entry if it is being modified BUG/MINOR: ssl_ckch: Don't duplicate path when replacing a cert entry BUG/MINOR: ssl_ckch: Don't duplicate path when replacing a CA/CRL entry BUG/MEDIUM: ssl_ckch: Rework 'commit ssl cert' to handle full buffer cases BUG/MEDIUM: ssl_ckch: Rework 'commit ssl ca-file' to handle full buffer cases BUG/MEDIUM: ssl/crt-list: Rework 'add ssl crt-list' to handle full buffer cases BUG/MEDIUM: httpclient: Don't remove HTX header blocks before duplicating them BUG/MEDIUM: httpclient: Rework CLI I/O handler to handle full buffer cases MEDIUM: http-ana: Always report rewrite failures as PRXCOND in logs MEDIUM: httpclient: Don't close CLI applet at the end of a response REGTESTS: abortonclose: Add a barrier to not mix up log messages REGTESTS: http_request_buffer: Increase client timeout to wait "slow" clients BUG/MINOR: ssl_ckch: Dump CRL transaction only once if show command yield BUG/MINOR: ssl_ckch: Dump CA transaction only once if show command yield BUG/MINOR: ssl_ckch: Dump cert transaction only once if show command yield BUG/MINOR: ssl_ckch: Init right field when parsing "commit ssl crl-file" cmd BUG/MINOR: ssl_ckch: Fix possible uninitialized value in show_cert I/O
[ANNOUNCE] haproxy-2.4.18
Now a previous list of headers is replaced by a new one only if it is not empty. * It was possible to crash HAProxy by defining multiple bind lines in a peers section. An error is now reported during configuration parsing. * A warning is now reported when some unsupported keywords are used in peers section instead of silently ignoring them. init_addr, resolvers, check, agent-check are concerned. * The DNS resolution is now ignored for disabled proxies preventing some crashes. Thanks everyone for your help and your contributions! Please find the usual URLs below : Site index : http://www.haproxy.org/ Documentation: http://docs.haproxy.org/ Wiki : https://github.com/haproxy/wiki/wiki Discourse: http://discourse.haproxy.org/ Slack channel: https://slack.haproxy.org/ Issue tracker: https://github.com/haproxy/haproxy/issues Sources : http://www.haproxy.org/download/2.4/src/ Git repository : http://git.haproxy.org/git/haproxy-2.4.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-2.4.git Changelog: http://www.haproxy.org/download/2.4/src/CHANGELOG Pending bugs : http://www.haproxy.org/l/pending-bugs Reviewed bugs: http://www.haproxy.org/l/reviewed-bugs Code reports : http://www.haproxy.org/l/code-reports Latest builds: http://www.haproxy.org/l/dev-packages --- Complete changelog : Brad Smith (1): BUILD: makefile: Fix install(1) handling for OpenBSD/NetBSD/Solaris/AIX Christian Ruppert (1): BUILD: Makefile: Add Lua 5.4 autodetect Christopher Faulet (35): MEDIUM: http-ana: Add a proxy option to restrict chars in request header names REGTESTS: abortonclose: Fix some race conditions BUG/MEDIUM: config: Reset outline buffer size on realloc error in readcfgfile() BUG/MINOR: check: Reinit the buffer wait list at the end of a check BUG/MEDIUM: resolvers: Don't defer resolutions release in deinit function BUG/MINOR: ssl_ckch: Free error msg if commit changes on a cert entry fails BUG/MEDIUM: ssl_ckch: Don't delete a cert entry if it is being modified BUG/MINOR: ssl_ckch: Don't duplicate path when replacing a cert entry BUG/MEDIUM: ssl_ckch: Rework 'commit ssl cert' to handle full buffer cases BUG/MEDIUM: ssl/crt-list: Rework 'add ssl crt-list' to handle full buffer cases MEDIUM: http-ana: Always report rewrite failures as PRXCOND in logs REGTESTS: abortonclose: Add a barrier to not mix up log messages REGTESTS: http_request_buffer: Increase client timeout to wait "slow" clients BUG/MINOR: ssl_ckch: Dump cert transaction only once if show command yield BUG/MINOR: ssl_ckch: Fix possible uninitialized value in show_cert I/O handler REGTESTS: http_abortonclose: Extend supported versions REGTESTS: restrict_req_hdr_names: Extend supported versions BUG/MEDIUM: mailers: Set the object type for check attached to an email alert BUG/MINOR: trace: Test server existence for health-checks to get proxy BUG/MINOR: checks: Properly handle email alerts in trace messages REGTESTS: healthcheckmail: Update the test to be functionnal again REGTESTS: healthcheckmail: Relax health-check failure condition BUG/MINOR: tcp-rules: Make action call final on read error and delay expiration BUG/MINOR: http-ana: Set method to HTTP_METH_OTHER when an HTTP txn is created BUG/MINOR: http-fetch: Use integer value when possible in "method" sample fetch BUG/MINOR: http-check: Preserve headers if not redefined by an implicit rule BUG/MINOR: http-act: Properly generate 103 responses when several rules are used BUG/MINOR: http-htx: Fix scheme based normalization for URIs wih userinfo BUG/MEDIUM: http-fetch: Don't fetch the method if there is no stream REGTEESTS: filters: Fix CONNECT request in random-forwarding script BUG/MINOR: mux-h1: Be sure to commit htx changes in the demux buffer BUG/MEDIUM: http-ana: Don't wait to have an empty buf to switch in TUNNEL state BUG/MEDIUM: mux-h1: Handle connection error after a synchronous send REGTESTS: Fix some scripts to be compatible with 2.4 and prior BUG/MINOR: backend: Fallback on RR algo if balance on source is impossible David CARLIER (1): BUILD/MINOR: cpuset fix build for FreeBSD 13.1 David Carlier (2): BUILD: fix build warning on solaris based systems with __maybe_unused. MINOR: tools: add get_exec_path implementation for solaris based systems. Emeric Brun (7): BUG/MEDIUM: peers: fix segfault using multiple bind on peers sections BUG/MEDIUM: peers: prevent unitialized multiple listeners on peers section DOC: peers: clarify when entry expiration date is renewed. DOC: peers: fix port number and addresses on new peers section format
[ANNOUNCE] haproxy-2.2.25
: http://www.haproxy.org/download/2.2/src/ Git repository : http://git.haproxy.org/git/haproxy-2.2.git/ Git Web browsing : http://git.haproxy.org/?p=haproxy-2.2.git Changelog: http://www.haproxy.org/download/2.2/src/CHANGELOG Pending bugs : http://www.haproxy.org/l/pending-bugs Reviewed bugs: http://www.haproxy.org/l/reviewed-bugs Code reports : http://www.haproxy.org/l/code-reports Latest builds: http://www.haproxy.org/l/dev-packages --- Complete changelog : Christopher Faulet (29): MEDIUM: http-ana: Add a proxy option to restrict chars in request header names REGTESTS: abortonclose: Fix some race conditions BUG/MEDIUM: config: Reset outline buffer size on realloc error in readcfgfile() BUG/MINOR: check: Reinit the buffer wait list at the end of a check BUG/MEDIUM: resolvers: Don't defer resolutions release in deinit function BUG/MEDIUM: dns: Keep the right count of active nameservers for a resolver BUG/MINOR: ssl_ckch: Free error msg if commit changes on a cert entry fails BUG/MEDIUM: ssl_ckch: Don't delete a cert entry if it is being modified BUG/MINOR: ssl_ckch: Don't duplicate path when replacing a cert entry BUG/MEDIUM: ssl_ckch: Rework 'commit ssl cert' to handle full buffer cases BUG/MEDIUM: ssl/crt-list: Rework 'add ssl crt-list' to handle full buffer cases MEDIUM: http-ana: Always report rewrite failures as PRXCOND in logs REGTESTS: abortonclose: Add a barrier to not mix up log messages REGTESTS: http_request_buffer: Increase client timeout to wait "slow" clients BUG/MINOR: ssl_ckch: Dump cert transaction only once if show command yield BUG/MINOR: ssl_ckch: Fix possible uninitialized value in show_cert I/O handler REGTESTS: restrict_req_hdr_names: Extend supported versions BUG/MEDIUM: mailers: Set the object type for check attached to an email alert REGTESTS: healthcheckmail: Update the test to be functionnal again REGTESTS: healthcheckmail: Relax health-check failure condition BUG/MINOR: tcp-rules: Make action call final on read error and delay expiration BUG/MINOR: http-ana: Set method to HTTP_METH_OTHER when an HTTP txn is created BUG/MINOR: http-fetch: Use integer value when possible in "method" sample fetch BUG/MINOR: http-check: Preserve headers if not redefined by an implicit rule BUG/MINOR: http-act: Properly generate 103 responses when several rules are used BUG/MEDIUM: http-ana: Don't wait to have an empty buf to switch in TUNNEL state BUG/MEDIUM: mux-h1: Handle connection error after a synchronous send REGTESTS: Fix some scripts to be compatible with 2.4 and prior BUG/MINOR: backend: Fallback on RR algo if balance on source is impossible David Carlier (1): BUILD: fix build warning on solaris based systems with __maybe_unused. Emeric Brun (4): BUG/MEDIUM: peers: fix segfault using multiple bind on peers sections BUG/MEDIUM: peers: prevent unitialized multiple listeners on peers section DOC: peers: clarify when entry expiration date is renewed. DOC: peers: fix port number and addresses on new peers section format Ilya Shipitsin (3): CI: determine actual LibreSSL version dynamically CI: determine actual OpenSSL version dynamically CI: re-enable gcc asan builds Remi Tricot-Le Breton (2): BUG/MINOR: ssl: Fix crash when no private key is found in pem BUG/MINOR: ssl: Do not look for key in extra files if already in pem Thayne McCombs (1): BUG/MEDIUM: sample: Fix adjusting size in word converter Tim Duesterhus (2): BUG/MEDIUM: http: Properly reject non-HTTP/1.x protocols REGTESTS: Do not use REQUIRE_VERSION for HAProxy 2.5+ (2) William Lallemand (3): BUG/MEDIUM: ssl/cli: crash when crt inserted into a crt-list BUG/MINOR: peers: fix possible NULL dereferences at config parsing BUG/MINOR: sockpair: wrong return value for fd_send_uxst() Willy Tarreau (12): BUG/MINOR: cfgparse: abort earlier in case of allocation error BUG/MINOR: peers: fix error reporting of "bind" lines SCRIPTS: add make-releases-json to recreate a releases.json file in download dirs SCRIPTS: make publish-release try to launch make-releases-json DOC: peers: indicate that some server settings are not usable BUG/MINOR: conn_stream: do not confirm a connection from the frontend path BUILD: compiler: implement unreachable for older compilers too BUG/MINOR: cli/stats: add missing trailing LF after JSON outputs BUG/MINOR: server: do not enable DNS resolution on disabled proxies BUG/MINOR: cli/stats: add missing trailing LF after "show info json" MEDIUM: mux-h2: try to coalesce outgoing WINDOW_UPDATE frames BUG/MINOR: peers/config: always fill the bind_conf's argument -- Christopher Faulet
Re: "Success" logs in HTTP frontends
Le 7/29/22 à 10:13, Christian Ruppert a écrit : Hi list, so I noticed on my private HAProxy I have 2 of those logs within the past ~1-2 months: haproxy[28669]: 1.2.3.4:48596 [17/Jun/2022:13:55:18.530] public/HTTPSv4: Success So that's nothing so far but still no idea what that means. At work, of 250 mio log entries per day, there are about 600k of those "Success" ones. haproxy[27892]: 192.168.70.102:7904 [29/May/2022:00:13:37.316] genfrontend_35310-foobar/3: Success I'm not sure what it means by "3". Is it the third bind? I couldn't trigger those "Success" logs by either restarting or reloading. What is it for / where does it come from? Hi Christian, What is your version ? At first glance, I can't find such log message in the code. It could come from a lua module. In fact, I found something. It is probably because an "embryonic" session is killed with no connection/ssl error. For instance, an SSL connection rejected because of a "tcp-request session" rule (so after the SSL handshake). The same may happen with a listener using the PROXY protocol. Regards, -- Christopher Faulet
Re: "Success" logs in HTTP frontends
Le 7/29/22 à 11:21, Tim Düsterhus a écrit : Hi On 7/29/22 11:10, Christopher Faulet wrote: What is your version ? At first glance, I can't find such log message in the code. It could come from a lua module. I'm seeing the same for both 2.4.x and 2.6.x. Christian and I had a short chat about this in IRC. In fact, I found something. It is probably because an "embryonic" session is killed with no connection/ssl error. For instance, an SSL connection rejected because of a "tcp-request session" rule (so after the SSL handshake). The same may happen with a listener using the PROXY protocol. On one of the machines I'm seeing it, we neither have 'tcp-' rules, nor do we use PROXY protocol: Well, it may have several reason to kill an embryonic session with no error. a reject at the session level is one of them. Probably the most common. It may also be an error when we try to install the client mux. The configuration may help in this case. I don't know if it is possible to have an handshake failure without setting any error. However, at first glance, the error is always set in this case. Of course, it may be a bug. If not, such messages can be removed by setting "dontlognull" option. -- Christopher Faulet
Re: Server timeouts since HAProxy 2.2
Le 8/3/22 à 16:23, William Edwards a écrit :
Hi,
Two days ago, I upgraded my first production system from HAProxy 1.8.19
to 2.2.9. Since then, many HTTP requests are hitting the server timeout.
Before upgrade:
root@lb0-0:~# zgrep 'sD--' /var/log/haproxy.log.5.gz | wc -l
0
root@lb0-0:~# zgrep 'sD--' /var/log/haproxy.log.4.gz | wc -l
0
root@lb0-0:~# zgrep 'sD--' /var/log/haproxy.log.3.gz | wc -l
0
After upgrade:
# Day of upgrade
root@lb0-0:~# zgrep 'sD--' /var/log/haproxy.log.2.gz | wc -l
3798
# Yesterday
root@lb0-0:~# grep 'sD--' /var/log/haproxy.log.1 | wc -l
127176
# Today, so far
root@lb0-0:~# grep 'sD--' /var/log/haproxy.log | wc -l
85063
For this specific request, Ta ("total active time for the HTTP request")
is 3, and Tt ("total TCP session duration time, between the moment the
proxy accepted it and the moment both ends were closed") is 34 (5
minutes, the server timeout):
Aug 3 00:31:05 lb0-0 haproxy[16884]: $ip:62223
[03/Aug/2022:00:26:05.337] fr_other~
bk_http.lyr_http-lyr02.cf.ha.cyberfusion.cloud/http-lyr02.cf.ha.cyberfusion.cloud
0/0/0/3/34 200 27992 - - sD-- 616/602/226/226/0 0/0 "GET
https://$domain/wp-content/uploads/2022/07/20220712_155022-300x300.jpg
HTTP/2.0"
The backend server indeed served the request within Ta:
$domain $ip - - [03/Aug/2022:00:26:05 +0200] "GET
/wp-content/uploads/2022/07/20220712_155022-300x300.jpg HTTP/1.1" 200
28008 "https://$domain/stoffen/"; "Mozilla/5.0 (Windows NT 10.0; Win64;
x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0
Safari/537.36"
The timeouts only occur with 5 out of 13 backends. There is no clear
pattern, i.e. the timeouts don't come in bursts, and they aren't caused
by fixed clients.
Does anyone know why the TCP session is kept open, and the HTTP request
is not responded to by HAProxy after the backend server responded to the
HTTP request, but only after the server timeout is reached?
Hi,
The 2.2.9 is pretty old. It is affected by 369 known bugs
(http://www.haproxy.org/bugs/bugs-2.2.9.html). You must update it to 2.2.25 first.
Regards,
--
Christopher Faulet
Re: Possible problem with custom error pages -- backend server returns 503, haproxy logs 503, but the browser gets 403
Le 8/22/22 à 16:37, Shawn Heisey a écrit :
The same problem also happens with 2.6.4, built with the same options as
the dev version.
HAProxy version 2.6.4 2022/08/22 - https://haproxy.org/
I have documentation for the problem details in another project's bug
tracker:
https://issues.apache.org/jira/browse/SOLR-16327?focusedCommentId=17582990&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17582990
It appears so far as if haproxy is getting a 503 from the backend,
logging a 503, but actually sending a 403. Here is the config snippet
when it works correctly:
A top-level config section:
http-errors myerrors
errorfile 404 /etc/haproxy/errors/404.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/50x.http
errorfile 503 /etc/haproxy/errors/50x.http
errorfile 504 /etc/haproxy/errors/50x.http
In the frontend:
errorfiles myerrors
http-response return status 404 default-errorfiles if
!real_errors { status 404 }
http-response return status 403 default-errorfiles if
!real_errors { status 403 }
http-response return status 500 default-errorfiles if
!real_errors { status 500 }
http-response return status 502 default-errorfiles if
!real_errors { status 502 }
http-response return status 503 default-errorfiles if
!real_errors { status 503 }
http-response return status 504 default-errorfiles if
!real_errors { status 504 }
Removing the "!real_errors" part and restarting haproxy is when the
problem occurs. I created and used the real_errors acl as a working
bandaid for the issue -- turn off the custom error pages for the solr
hostname.
Hi,
It could be good to share your configuration and not only a snippet. However I'm
puzzled because in your case, the status code must be the one returned by the
server if no return rule matches or the one specified by the applied return rule.
There is also something I don't understand. In your bug report on solr project,
HAProxy logs report HTTP/3.0 requests but the screenshots show HTTP/2.0
requests. And the payload for the 403 response is talking about 50x errors. What
is the 50x.http error file content ?
--
Christopher Faulet
Re: Lua: processing expired on timeout when using core.msleep
Le 8/22/22 à 18:53, Bren a écrit : Greetings, This is a minor issue I've been meaning to ask about. I have a fairly simple Lua script that simply runs core.msleep on certain requests for a random number of ms to slow them down. I've noticed this in the logs: [ALERT] (3650884) : Lua function 'delay_request': aborting Lua processing on expired timeout. I've always been under the impression that a sleep shouldn't cause any timeouts. Both tune.lua.session-timeout and tune.lua.service-timeout says: If the Lua does a sleep, the sleep is not taken in account. Am I missing something? Bren Hi, It could be good to share your config, at least the part calling your lua script. But this error can be triggered when the inspect-delay for tcp rules evaluation is expires. It can also happen when a read error is detected or an abort with abortonclose option. -- Christopher Faulet
Re: Lua: processing expired on timeout when using core.msleep
Le 8/23/22 à 20:08, Bren a écrit : --- Original Message --- On Tuesday, August 23rd, 2022 at 4:26 AM, Christopher Faulet wrote: It could be good to share your config, at least the part calling your lua script. I think these are the only relevant bits: tcp-request inspect-delay 10s http-request lua.delay_request 15000 3 I'm delaying requests a random number of ms between 15000 and 3. But this error can be triggered when the inspect-delay for tcp rules evaluation is expires. Perhaps this is what is happening? You have an inspect delay for tcp-request rules but your script is called during http-request rules eval. So it is not related. Here I guess you have enable abortonclose option and a client abort is caught. In this case, the script execution is interrupted. Or a read error occurred on the client side. However, I must admit the warning is not accurate in this case... -- Christopher Faulet
Re: Lua hangs with get line at end of request body
Le 8/26/22 à 18:57, Robert Newson a écrit :
Hi All,
I'm upgrading some HAproxy nodes here (from 2.0.29 to 2.6.4) and everything is
going well except some Lua code I have. I've distilled the code and config to
this;
haproxy.cfg;
global
lua-load httpbug.lua
defaults
mode http
timeout client 15
timeout connect 5000
timeout queue 5000
timeout server 36
listen proxy
bind 127.0.0.1:10001
http-request use-service lua.httpbug
httpbug.lua:
core.register_service("httpbug", "http", function(applet)
repeat
local line = applet:getline()
core.log(core.crit, line)
until line == ''
applet:set_status(200)
applet:start_response()
applet:send("success")
end)
--
With a request like 'curl localhost: 10001 -XPOST -T body' and file called
'body' as follows;
line1
line2
line3
(where each line is newline-terminated)
With 2.0.29 I get a response (of 200 with body "success"), and line1, line2,
line3 are logged to screen (haproxy -d -f haproxy.cfg)
With 2.6.4 I see the logs of line1, line2, line3, but no response.
It appears applet:getline() blocks forever at the end of the request body,
rather than returning an empty string to indicate this like it used to (and is
documented to do).
I searched the mailing list (and SO, etc) for other examples of reading the
http request body line by line or for a report of this bug from someone else
but found nothing.
What did I do wrong?
Hi,
There is a bug. I'm able to reproduce it with your lua script. I will fix it
soon.
Thanks !
--
Christopher Faulet
Re: Lua hangs with get line at end of request body
Le 8/29/22 à 12:30, Christopher Faulet a écrit : There is a bug. I'm able to reproduce it with your lua script. I will fix it soon. FYI, I pushed a fix[1] to 2.7-DEV. It will be backported soon. [1] https://github.com/haproxy/haproxy/commit/4a20972a -- Christopher Faulet
Re: Health Checks and DNS lookups in stopping processes
Le 9/19/22 à 17:44, Tim Düsterhus a écrit : Hi recently our HAProxy nodes started handling long-running HTTP connections (similar to WebSockets). This causes old workers to stay around for several days after a reload. This isn't too bad from a memory perspective, we have sufficient RAM to keep around the old processes until the connections die naturally. It's much worse from a CPU perspective: The old workers appear to still perform health checks and DNS lookups and this takes away precious resources from the active workers. My understanding is that the old workers will only handle existing connections and thus will never need to connect to a backend (server) again. Thus it should not be necessary to waste CPU on DNS lookups and health checks for a stopping worker. Am I missing something here? It is not exactly how it works. When HAProxy is reloaded, it stops to accept new connections and it closes idle HTTP connections on the server side and also on the client side. However, on client sides, a connection is idle if at least one request was processed. So totally inactive clients are not closed. These clients may still perform a request. Connection to servers are not blocked on old workers. It is important when "idle-close-on-response" option is enabled. In this case, idle client connections are not closed and may try to perform a last request. This means all backend mechanisms must still be running (load-balancing, redispatch, l7-retry, health-checks, dns resolution ...). -- Christopher Faulet
Re: Health Checks and DNS lookups in stopping processes
Le 9/19/22 à 21:04, Tim Düsterhus a écrit : Christopher, On 9/19/22 20:22, Christopher Faulet wrote: It is not exactly how it works. When HAProxy is reloaded, it stops to accept new connections and it closes idle HTTP connections on the server side and also on the client side. However, on client sides, a connection is idle if at least one request was processed. So totally inactive clients are not closed. These clients may still perform a request. Connection to servers are not blocked on old workers. It is important when "idle-close-on-response" option is enabled. In this case, idle client connections are not closed and may try to perform a last request. This means all backend mechanisms must still be running (load-balancing, redispatch, l7-retry, health-checks, dns resolution ...). In other words: If I do not use 'idle-close-on-response' or if all existing connections have been used at least once, the health/DNS logic could technically be disabled? In that case I would file a feature request for that, because 'hard-stop-after' and 'mworker-max-reloads' both are "not great" for our use case. I want to keep the connections alive for as long at possible, but without adding unnecessary load on our backends or resolvers. I would say yes, it is possible. But I don't know if it is easy to achieve. -- Christopher
Re: [UK OFFICIAL] [PATCH] Issue 1812 - smtpchk option does not send a QUIT
Le 9/21/22 à 11:42, Wright Loz a écrit : Classification: UK OFFICIAL Hi, Reference Github issue 1812 ( https://github.com/haproxy/haproxy/issues/1812 <https://github.com/haproxy/haproxy/issues/1812> ) relating to unclean shutdown of “smtpchk” health checking, I attach a patch which implements an SMTP QUIT (and then a wait for successful confirmation) prior to closing the TCP connection on each SMTP service check. Hopefully relatively straightforward and obviously happy to discuss! It seems to be ok at first glance. However, I must fix the regex to properly match the EHLO reply first. It is a multi-line reply and, for now, the regex is only matching on the first line. I will fix it before merging your patch. In addition and if it is ok for you, I will amend you patch to add information about backports and to update reg-tests accordingly (reg-tests/checks/4be_1srv_smtpchk_httpchk_layer47errors.vtc and reg-tests/checks/smtp-check.vtc) Thanks ! -- Christopher Faulet
[ANNOUNCE] haproxy-2.6.6
MINOR: mux-quic: refactor snd_buf
BUG/MEDIUM: mux-quic: properly trim HTX buffer on snd_buf reset
Aurelien DARRAGON (8):
BUG/MEDIUM: proxy: ensure pause_proxy() and resume_proxy() own PROXY_LOCK
MINOR: listener: small API change
MINOR: proxy/listener: support for additional PAUSED state
BUG/MINOR: stats: fixing stat shows disabled frontend status as 'OPEN'
CLEANUP: listener: function comment typo in stop_listener()
BUG/MINOR: listener: null pointer dereference suspected by coverity
BUG/MEDIUM: server: segv when adding server with hostname from CLI
BUG/MINOR: log: improper behavior when escaping log data
Brad Smith (2):
MINOR: Revert part of clarifying samples support per os commit
BUILD: makefile: enable crypt(3) for NetBSD
Christopher Faulet (4):
BUG/MINOR: h1: Support headers case adjustment for TCP proxies
BUG/MINOR: task: Fix detection of tasks profiling in
tasklet_wakeup_after()
BUG/MINOR: mux-h1: Increment open_streams counter when H1 stream is
created
REGTESTS: healthcheckmail: Relax matching on the healthcheck log message
Emeric Brun (1):
BUG/MEDIUM: sink: bad init sequence on tcp sink from a ring.
Frédéric Lécaille (12):
BUG/MINOR: quic: Retransmitted frames marked as acknowledged
BUG/MINOR: quic: Possible crash with "tls-ticket-keys" on QUIC bind lines
BUG/MINOR: quic: Possible crash when verifying certificates
MINOR: quic: Add traces about sent or resent TX frames
MINOR: quic: No TRACE_LEAVE() in retrieve_qc_conn_from_cid()
BUG/MINOR: quic: Wrong connection ID to thread ID association
BUG/MINOR: quic: Speed up the handshake completion only one time
BUG/MINOR: quic: Trace fix about packet number space information.
BUG/MINOR: h3: Crash when h3 trace verbosity is "minimal"
MINOR: h3: Add the quic_conn object to h3 traces
MINOR: h3: Missing connection argument for a TRACE_LEAVE() argument
MINOR: h3: Send the h3 settings with others streams (requests)
Ilya Shipitsin (4):
CI: cirrus-ci: bump FreeBSD image to 13-1
REGTESTS: ssl: adopt tests to OpenSSL-3.0.N
REGTESTS: ssl: adopt tests to OpenSSL-3.0.N
REGTESTS: ssl: fix grep invocation to use extended regex in
ssl_generate_certificate.vtc
Mathias Weiersmueller (1):
DOC: fix TOC in starter guide for subsection 3.3.8. Statistics
Matthias Wirth (1):
BUG/MINOR: signals/poller: ensure wakeup from signals
William Lallemand (11):
BUILD: quic: add some ifdef around the SSL_ERROR_* for libressl
BUILD: ssl: fix ssl_sock_switchtx_cbk when no client_hello_cb
BUILD: quic: temporarly ignore chacha20_poly1305 for libressl
BUILD: quic: enable early data only with >= openssl 1.1.1
BUILD: ssl: fix the ifdef mess in ssl_sock_initial_ctx
BUILD: quic: fix the #ifdef in ssl_quic_initial_ctx()
MINOR: quic: add QUIC support when no client_hello_cb
BUG/MINOR: signals/poller: set the poller timeout to 0 when there are
signals
REGTESTS: log: test the log-forward feature
REGTESTS: ssl/log: test the log-forward with SSL
MEDIUM: quic: separate path for rx and tx with set_encryption_secrets
Willy Tarreau (14):
MEDIUM: peers: limit the number of updates sent at once
BUG/MINOR: task: always reset a new tasklet's call date
BUG/MINOR: task: make task_instant_wakeup() work on a task not a tasklet
MINOR: task: permanently enable latency measurement on tasklets
CLEANUP: task: rename ->call_date to ->wake_date
BUG/MINOR: sched: properly account for the CPU time of dying tasks
MINOR: sched: store the current profile entry in the thread context
BUG/MINOR: stream/sched: take into account CPU profiling for the last call
DEV: flags: fix usage message to reflect available options
DEV: flags: add missing CO_FL_FDLESS connection flag
CLEANUP: pollers: remove dead code in the polling loop
BUG/MEDIUM: captures: free() an error capture out of the proxy lock
BUILD: fd: fix a build warning on the DWCAS
SCRIPTS: announce-release: update some URLs to https
cui fliter (1):
CLEANUP: quic,ssl: fix tiny typos in C comments
--
Christopher Faulet
Re: [UK OFFICIAL] [PATCH] Issue 1812 - smtpchk option does not send a QUIT
Le 9/21/22 à 16:13, Wright Loz a écrit : Classification: UK OFFICIAL Hi Christopher, Many thanks - good point regarding EHLO reply and yes, if you could add the backports / reg-tests parts as well that would be brilliant. This is my first ever patch submission so am keen to learn all the necessary parts for next time! Hi, I merged your patch. Thanks ! -- Christopher Faulet
[ANNOUNCE] haproxy-2.5.9
HAProxy when adding a server with hostname from the CLI. In itself, it is not an issue but the server is created with no address and an operation was not guarded against NULL addresses. * There was a bug in the SPOE. In sync or pipelining modes, an unhealthy SPOA could led HAProxy to create a huge number of applets to process queued messages, slowing down all processing. * Willy managed to trigger an error on reload where the old process died saying "t->tid >= 0 && t->tid != tid". This is caused by the deinit code that needs to stop stuff initialized on other threads, and as such it violates some consistency checks. The check was relaxed to ignore the stopping condition. * Characters escaping process in log messages was not correctly processing strings coming from sample fetches truncating the output string. * Using HAProxy built with PCRE2_JIT with a lib built without would fail to match. Now it will fall back to the regular match. * Agent-check could be delayed by ~200ms due to TCP QUICKACK being disabled by default. * Reading from the rings could also occasionally freeze at high rate if the reader had to stop due to a buffer full while the writer had already stopped due to a ring full. * In Lua, it was possible to hand reading HTTP payload (by line or not) from an HTTP applet because we relied on a transiant HTX flags to detect the end of the message instead of relying on the channel flag. * Some ca-file elements could leak during "commit ssl ca-file". * On the CLI, no error was returned when an empty ca-file was added. This could be a problem is the file was malformated and did not contain any PEM header. * A 60s delay could be experienced after stopping HAProxy. This was happening when a signal was received before entering the poller and without any activity on the process. In mworker mode, if a worker exited and the SIGCHLD signal was delivered at the right time to the master, this one could be stuck for 60s. The timeout is now set to 0 in this specific case. The following improvements were also backported: * Headers case adjustment in H1 is now available for TCP proxies. It was an issue for HTTP health-checks on backend side or for TCP connections upgraded to HTTP on frontend side. * The stats applet was reported paused frontends as OPEN. Now, these frontends are reported as PAUSED. * Encrypted password in Userlists are now supported on NetBSD Thanks everyone for your help and your contributions! The 2.4.19 will be released the next week. Please find the usual URLs below : Site index : https://www.haproxy.org/ Documentation: https://docs.haproxy.org/ Wiki : https://github.com/haproxy/wiki/wiki Discourse: https://discourse.haproxy.org/ Slack channel: https://slack.haproxy.org/ Issue tracker: https://github.com/haproxy/haproxy/issues Sources : https://www.haproxy.org/download/2.5/src/ Git repository : https://git.haproxy.org/git/haproxy-2.5.git/ Git Web browsing : https://git.haproxy.org/?p=haproxy-2.5.git Changelog: https://www.haproxy.org/download/2.5/src/CHANGELOG Pending bugs : https://www.haproxy.org/l/pending-bugs Reviewed bugs: https://www.haproxy.org/l/reviewed-bugs Code reports : https://www.haproxy.org/l/code-reports Latest builds: https://www.haproxy.org/l/dev-packages --- Complete changelog : Aurelien DARRAGON (6): BUG/MEDIUM: proxy: ensure pause_proxy() and resume_proxy() own PROXY_LOCK MINOR: listener: small API change MINOR: proxy/listener: support for additional PAUSED state BUG/MINOR: stats: fixing stat shows disabled frontend status as 'OPEN' BUG/MEDIUM: server: segv when adding server with hostname from CLI BUG/MINOR: log: improper behavior when escaping log data Brad Smith (1): BUILD: makefile: enable crypt(3) for NetBSD Christopher Faulet (20): MINOR: peers: Use a dedicated reconnect timeout when stopping the local peer BUG/MEDIUM: peers: limit reconnect attempts of the old process on reload BUG/MINOR: peers: Use right channel flag to consider the peer as connected BUG/MEDIUM: dns: Properly initialize new DNS session MINOR: server: Constify source server to copy its settings REORG: server: Export srv_settings_cpy() function BUG/MEDIUM: proxy: Perform a custom copy for default server settings BUG/MINOR: tcpcheck: Disable QUICKACK only if data should be sent after connect REGTESTS: Fix prometheus script to perform HTTP health-checks BUG/MEDIUM: spoe: Properly update streams waiting for a ACK in async mode BUG/MEDIUM: peers: Add connect and server timeut to peers proxy BUG/MEDIUM: peers: Don't use resync timer when local resync is in progress BUG/MEDIUM: peers
