[ANNOUNCE] haproxy-2.6.19

[Christopher Faulet]

Hi,

HAProxy 2.6.19 was released on 2024/09/19. It added 64 new commits
after version 2.6.18.

Following the 2.8.11 release, it is now the turn to the 2.6. This release covers
the same period. So, fixed bugs are more or less the same, excluding those not
concerning the 2.6:

  * The SSL stack is now always completely initialized. Due to a change of
API in 3.x the old call was deprecated and used to emit a warning, but
it was later found to still be required in some cases.  This has been
cooking in 2.9 for 6 months now and is considered OK.

  * HTTP applets (stats, cache and promex) were starting to process the
request and reply without worrying about whether the request analysis
was finished or not. In the vast majority of cases, it is not an issue
because the request analysis is indeed finished in the same time the
applet on server side is created. But if a filter delayed the request
analysis, it might happens. In that case, some undefined and hardly
predictible behaviors were able to be experienced, like responses sent
too early or even crashes. Among others, the compression filter was
pretty sensitive in this case because it is mandatory to filter the
request before the response. To fix the issue, there is now a check in
backend HTTP applets to wait for the end of the request analysis.

  * The hard limit on the number of file descriptors now defaults to about 1
million, in order to match what has been done for a very long time on
many distros, and that recently changed to 1 billion on some of them,
causing a huge startup time (or even a watchdog at boot) and a massive
memory usage.

  * The SSL library functions used to validate a JWT token would leave an
error in the SSL stack, that will later be mistaken for an error on
another connection and cause it to be closed. "jwt_verify" converter was
fixed when called with a HMAC algorithm (HS256, HS384, HS512). In that
case, the converter must not try to load a file because the key should
hold a secret. It is especially important for runtime uses, like lua for
instance. But note, for dynamic calls, this remains an issue for other
algorithms because there is nothing to prevent I/O disk.

  * A time-of-check/time-of-use (TOCTOU) issue in the queue processing makes
it rare but possible to leave a server with no connection yet not take
any traffic. It's more likely to happen with maxconn 1, very hard at 2
and almost impossible at 3 or above. In addition, A flag is now used to
be sure only one thread is dequeuing session at once instead of relying
on a trylock to do so. No 2.6 release was affected, but 2.9/3.0 versions
were buggy because of TOCTOU fix.

  * Empty transfer-encoding headers in H1 are now properly blocked since
they may be used to try to build an attack against vulnerable
implementations.

  * An issue in SPOE that can cause a thread to refrain from creating an
applet to connect outside, causing failures on requests processed on
this thread.

  * It was possible to crash the process when performing an implicit
protocol upgrade (TCP to HTTP due to a transition from a TCP front to an
HTTP back) if an error happened on the connection just before the
transition.

  * Unhandled aborts were fixed in the H2 multiplexer. The end of
message could be reported twice for tunneled streams, leaving the second
one blocked at the channel level because of the first one.

  * Several bugs were fixed on QUIC:

- An incorrect computation was performed when encoding a STREAM frame in
  a single packet leading to datagrams smaller than expected, resulting
  in suboptimal for bandwidth.

- A few assorted minor fixes (possible crash on resource allocation
  error, slight loss of precision in Cubic parameters calculations etc).

- It was possible to freeze a connection because of 0-RTT undeciphered
  content.

- The MAX_STREAM ID value was not properly checked and it was possible
  to send too big value. It is now fixed. Thanks to this patch, this
  also ensure that the peer cannot open a stream with an invalid ID as
  this would cause a flow-control violation instead.

- Some issues with the QUIC traces were fixed.

  * On H3, when a response is formatted to be sent to the client, the
handling of responses with a too long header list was fixed to no longer
abort the process but to return proper error. In addition, the syntax
checks on :method and :scheme H3 headers were insufficient. This was
fixed too.

  * Some bugs related to pattern expressions handling loaded from file were
fixed.

  * When a listen() failed for TCP and Unix sockets, the file descriptor was
not removed from the fdtab[] array, leading to a possible crash because
of a BUG_ON() when this FD was reused. The FD is now properly removed
from fdtab

[ANNOUNCE] haproxy-2.8.11

[Christopher Faulet]

Hi,

HAProxy 2.8.11 was released on 2024/09/19. It added 90 new commits
after version 2.8.10.

This release covers fixes shipped into the last four 2.9 releases (from
2.9.8 to 2.9.11). Here is the unordered list of issues fixes by the 2.8.11:

  * The SSL stack is now always completely initialized. Due to a change of
API in 3.x the old call was deprecated and used to emit a warning, but
it was later found to still be required in some cases.  This has been
cooking in 2.9 for 6 months now and is considered OK.

  * A memory allocation error while loading OCSP parameters could leave the
tree locked and freeze subsequent operations.

  * The hard limit on the number of file descriptors now defaults to about 1
million, in order to match what has been done for a very long time on
many distros, and that recently changed to 1 billion on some of them,
causing a huge startup time (or even a watchdog at boot) and a massive
memory usage.

  * The memory cleanup on the proxy fields on deinit was improved (some
leaks were present till now as well as in 2.9 till 2.9.8).

  * A bug in the bwlim filter that can make it forget to update its timeout
and loop fast when woken up with nothing to forward.

  * A crash could happen in mux-pt if an error happened on the connection
just before an abort that is going to emit a shutdown, and with a
pending wakeup that completes some work on a connection having no
transport layer anymore. This only affects TCP (e.g. peers and master
CLI; GH #2656).

  * The "show threads" command would crash if issued with less than 16
threads (due to an area shared for two different things it would start
to dump threads from the 17th).

  * The SSL library functions used to validate a JWT token would leave an
error in the SSL stack, that will later be mistaken for an error on
another connection and cause it to be closed. "jwt_verify" converter was
fixed when called with a HMAC algorithm (HS256, HS384, HS512). In that
case, the converter must not try to load a file because the key should
hold a secret. It is especially important for runtime uses, like lua for
instance. But note, for dynamic calls, this remains an issue for other
algorithms because there is nothing to prevent I/O disk.

  * A time-of-check/time-of-use (TOCTOU) issue in the queue processing makes
it rare but possible to leave a server with no connection yet not take
any traffic. It's more likely to happen with maxconn 1, very hard at 2
and almost impossible at 3 or above. In addition, A flag is now used to
be sure only one thread is dequeuing session at once instead of relying
on a trylock to do so. No 2.8 release was affected, but upper versions
were buggy because of TOCTOU fix.

  * Empty transfer-encoding headers in H1 are now properly blocked since
they may be used to try to build an attack against vulnerable
implementations.  However, this will be slightly relaxed on the response
side in the next release. Parsing of multiple "chunked" values will also
be relaxed.

  * H1 multiplexer could repeat a 408 error multiple times in logs when
failing to send an empty message on a full output buffer. In this case,
it would attempt to close again every client timeout and produce a log
each time despite no data leaving.

  * An issue in SPOE that can cause a thread to refrain from creating an
applet to connect outside, causing failures on requests processed on
this thread.

  * A hang on lua CLI applets when trying to deal with their own buffers.

  * It was possible to crash the process when performing an implicit
protocol upgrade (TCP to HTTP due to a transition from a TCP front to an
HTTP back) if an error happened on the connection just before the
transition.

  * A temporary leak of sessions was fixed in the H1 multiplexer when the
zero-copy data forwarding was inuse. When the H1 connection was about to
be closed, the event was not properly handled in case of zero-copy data
forwarding, leaving the connection in CLOSING state till the timeout was
reached. This could be detected by an excess of connections in
CLOSE_WAIT state.

  * HTTP applets (stats, cache and promex) were starting to process the
request and reply without worrying about whether the request analysis
was finished or not. In the vast majority of cases, it is not an issue
because the request analysis is indeed finished in the same time the
applet on server side is created. But if a filter delayed the request
analysis, it might happens. In that case, some undefined and hardly
predictable behaviors were able to be experienced, like responses sent
too early or even crashes. Among others, the compression filter was
pretty sensitive in this case because it is mandatory to filter the
request before the response. To fix

[ANNOUNCE] haproxy-2.9.11

[Christopher Faulet]

Hi,

HAProxy 2.9.11 was released on 2024/09/19. It added 58 new commits
after version 2.9.10.

This release is pretty similar to the 3.0.5. Following bugs were fixed:

  * A temporary leak of sessions was fixed in the H1 multiplexer when the
zero-copy data forwarding was inuse. When the H1 connection was about to
be closed, the event was not properly handled in case of zero-copy data
forwarding, leaving the connection in CLOSING state till the timeout was
reached. This could be detected by an excess of connections in
CLOSE_WAIT state.

  * HTTP applets (stats, cache and promex) were starting to process the
request and reply without worrying about whether the request analysis
was finished or not. In the vast majority of cases, it is not an issue
because the request analysis is indeed finished in the same time the
applet on server side is created. But if a filter delayed the request
analysis, it might happens. In that case, some undefined and hardly
predictable behaviors were able to be experienced, like responses sent
too early or even crashes. Among others, the compression filter was
pretty sensitive in this case because it is mandatory to filter the
request before the response. To fix the issue, there is now a check in
backend HTTP applets to wait for the end of the request analysis.

  * Several commits concerned the clock part to fix handling of time
jumps. In case of large time jump, it was possible to no longer update
the global time offset, leading to a wrong "date" value. Among other
things, this could lead to wrong internal rates computation. By fixing
the clock issues, a bug in the busy polling was revealed. The time and
status passed to clock_update_local_date() was incorrect.

  * Some unhandled aborts were fixed in the H2 multiplexer. The end of
message could be reported twice for tunneled streams, leaving the second
one blocked at the channel level because of the first one. In additions,
termination flags were not always properly propagated from the H2 stream
to the stream-endpoint descriptor. Because of these both bugs, it was
possible from time to time to block streams infinitely.

  * The zero-copy data forwarding in H1 was erroneously disabled when a
tunnel was established because the response message was flagged to have
no data. While such response has no HTTP payload, tunneled data are
expected.

  * Write error on client side when HAProxy was waiting for the server
response was not properly handled. The stream was not properly aborted
as usual. It was not an issue if no filter was used. But with a filter,
it was possible to infinitely block the stream because data could
remain blocked in the response channel buffer.

  * Same kind of issue was fixed but at the H1/pass-though multiplexer
level. The pipe used for the kernel splicing was not properly released
on write error, preventing the stream to be released when a filter was
used because the corresponding channel always appeared as non-empty. On
write error, the pipe can be safely released because no more data can be
sent.

  * The pipeline modes on the master CLI was broken since the 3.0-dev4. On
older versions, this still works but a warning is emitted. When the
pipeline modes was fixed to match the documentation (having a semi-colon
between commands and a new-line at the end) for the worker CLI, we
forgot to reflect the change to the master CLI. It is now fixed.

  * The fix concerning the session queuing in the 2.9.10 introduced a
regression, leading to a infinite loop in assign_server_and_queue()
because of a race condition between dequeuing and queuing mechanisms.
The bug was mainly due to the fact that a trylock was used to dequeue a
session when a server slot was released. A trylock was used to be sure
no thread was waiting to dequeue sessions if another one was still doing
it, because it is an expensive task. However, the trylock could also
fail when a session is queued. So, to fix the bug, a flag is now used
instead of a trylock.

  * Several bugs were fixed on QUIC:

- A 0-RTT session may be opened with a spoofed IP address, trusted by
      HAProxy because of a previously established connection, by-passing
  this way IP allow/block list. The problem was reported by Michael
  Wedl. To mitigate this vulnerability, the NEW_TOKEN frame support was
  added so as to always be able to validate reachability of the client
  during 0-RTT. It allows to deliver an IP-based token to the client for
  use later, and if the address changes, then we can use a regular RETRY
  token.

- It was possible to freeze a connection because of 0-RTT undeciphered
  content.

- The MAX_STREAM ID value was not properly checked and it was possible
  to send too big value. It is now fixed. Thanks to this p

[ANNOUNCE] haproxy-3.0.5

[Christopher Faulet]

Hi,

HAProxy 3.0.5 was released on 2024/09/19. It added 68 new commits
after version 3.0.4.

As announced by Willy after the 3.0.4, a bunch of backports were missing. All
of those are included in this release with another set of fixes as usual. So
following bugs were fixed:

  * A temporary leak of sessions was fixed in the H1 multiplexer when the
zero-copy data forwarding was inuse. When the H1 connection was about to
be closed, the event was not properly handled in case of zero-copy data
forwarding, leaving the connection in CLOSING state till the timeout was
reached. This could be detected by an excess of connections in
CLOSE_WAIT state.

  * HTTP applets (stats, cache and promex) were starting to process the
request and reply without worrying about whether the request analysis
was finished or not. In the vast majority of cases, it is not an issue
because the request analysis is indeed finished in the same time the
applet on server side is created. But if a filter delayed the request
analysis, it might happens. In that case, some undefined and hardly
prédictible behaviors were able to be experienced, like responses sent
too early or even crashes. Among others, the compression filter was
pretty sensitive in this case because it is mandatory to filter the
request before the response. To fix the issue, there is now a check in
backend HTTP applets to wait for the end of the request analysis.

  * Several commits concerned the clock part to fix handling of time
jumps. In case of large time jump, it was possible to no longer update
the global time offset, leading to a wrong "date" value. Among other
things, this could lead to wrong internal rates computation. By fixing
the clock issues, a bug in the busy polling was revealed. The time and
status passed to clock_update_local_date() was incorrect.

  * Some unhandled aborts were fixed in the H2 multiplexer. The end of
message could be reported twice for tunneled streams, leaving the second
one blocked at the channel level because of the first one. In additions,
termination flags were not always properly propagated from the H2 stream
to the stream-endpoint descriptor. Because of these both bugs, it was
possible from time to time to block streams infinitely.

  * The zero-copy data forwarding in H1 was erroneously disabled when a
tunnel was established because the response message was flagged to have
no data. While such response has no HTTP payload, tunneled data are
expected.

  * Write error on client side when HAproxy was waiting for the server
response was not properly handled. The stream was not properly aborted
as usual. It was not an issue if no filter was used. But with a filter,
it was possible to infinitely block the stream because data could
remain blocked in the response channel buffer.

  * Same kind of issue was fixed but at the H1/pass-though multiplexer
level. The pipe used for the kernel splicing was not properly released
on write error, preventing the stream to be released when a filter was
used because the corresponding channel always appeared as non-empty. On
write error, the pipe can be safely released because no more data can be
sent.

  * The pipeline modes on the master CLI was broken since the 3.0-dev4. On
older versions, this still works but a warning is emitted. When the
pipeline modes was fixed to match the documentation (having a semi-colon
between commands and a new-line at the end) for the worker CLI, we
forgot to reflect the change to the master CLI. It is now fixed.

  * The fix concerning the session queuing in the 3.0.4 introduced a
regression, leading to a infinite loop in assign_server_and_queue()
because of a race condition between dequeuing and queuing mechanisms.
The bug was mainly due to the fact that a trylock was used to dequeue a
session when a server slot was released. A trylock was used to be sure
no thread was waiting to dequeue sessions if another one was still doing
it, because it is an expensive task. However, the trylock could also
fail when a session is queued. So, to fix the bug, a flag is now used
instead of a trylock.

  * Because of optimisations performed on the stick-tables during the 3.0
development cycle, when entries for a local table were learned from a
unique remote peer upon resynchro, local updates on such entries were
not advertised to the peer anymore, until they eventually expired and
were able to be recreated upon local updates. Still on peers, A wakeup
loop was fixed when a peer applet was waiting for a synchro with the
global synchronisation task. In that case, the applet must explicitly
state it won't consume data otherwise it is woken up again and again.

  * Several bugs were fixed on QUIC:

- A 0-RTT session may be opened with a spoofed IP addr

[ANNOUNCE] haproxy-3.1-dev8

[Willy Tarreau]

Hi,

HAProxy 3.1-dev8 was released on 2024/09/18. It added 50 new commits
after version 3.1-dev7.

The last two weeks have been mostly dedicated to fixing bugs in order to
update stable branches, so it will be no surprise that this version mostly
contains fixes as well. They are not even particularly interesting to
describe here (frozen H1 connections, crashes with cache+compression+extra
filters, second fix for the dequeue lockup).

A few points are still worth mentioning:
  - despite the code that tries to correct possible time jumps, there were
some uncovered cases (typically if the time changes outside the polling
loop instead of during poll). These were addressed as well, and since
we were already collecting the monotonic time on operating systems
supporting it, it was the right moment to opportunistically rely on it
when it ticks.

  - the previously discussed choice of sending warnings for all detected
cases of conflicts on proxy names has started. Now we detect frontend
vs backend, listen after backend, defaults vs other proxies. Some
cleanups are still deserved for defaults handling (right now they're
all preserved but the duplicate ones of same names should be dropped).
Detection of server name duplicates still needs to be done.

  - accept-invalid-http-response now also tolerates responses with two
"chunked" transfer-codings like before, because of a report of at
least one bogus application relying on that (GitHub issue #2677).
This is not accepted in requests though, as this remains particularly
dangerous.

  - the accept-invalid-http-request/response options were more and more
used to adapt bogus applications in ways that start to pose security
concerns. Lukas suggested that we'd rename them in a way that makes
this aspect more obvious, and it was the right moment! They're now
called: "accept-unsafe-violations-in-http-request/response". The old
ones are still supported but deprecated and will issue a warning
suggesting the new one. With a bit of luck some users seeing the
warning will reconsider whether or not they still need these.

  - the CLI now offers "dump ssl cert" to dump a certificate directly
in PEM format.

  - speaking of the CLI, the "init-state" server keyword wasn't permitted
on the "add server" directive, it is now.

  - configs with many variables should now run faster: they used to be
stored in a linked list, and lookups could take quite some time.
Now they're stored in a tree. This shows a performance gain of 67%
at max speed for 100 variables set and each read 10 times in a
section (OK that's a pretty empirical test).

We've created a new "Breaking Changes" page on the wiki. The official
purpose is to let everyone know what updates may require some efforts,
but we all know that the main goal is for me to stop forgetting about
deprecating stuff that was already planned, thus postponing that by a
year once noticed :-)  That can be useful over time to help users
making big jumps from older to newer versions spanning multiple cycles.

New stable versions are coming. We're done with the backports, and are
still orking on collecting the list of changes for the announce message,
and we'll hopefully issue them this Thursday.

Oh before I forget, some might not have noticed the news, but the date
of the HAProxyConf 2025 is now fixed to June 4-5 2025 in San Francisco,
and with some workshops starting on June 3. There's currently a Call For
Papers open. If you have any ideas of topics worth presenting, some cool
tricks you figured and would like to share, report a nice success story,
or enumerate all the points that annoy you in HAProxy as well, don't be
shy and please consider proposing a talk. The Call for Papers site is:
https://www.haproxyconf.com/call-for-papers/ . You can know more by
visiting the site, which is still: https://www.haproxyconf.com/ .

Please find the usual URLs below :
   Site index   : https://www.haproxy.org/
   Documentation: https://docs.haproxy.org/
   Wiki : https://github.com/haproxy/wiki/wiki
   Discourse: https://discourse.haproxy.org/
   Slack channel: https://slack.haproxy.org/
   Issue tracker: https://github.com/haproxy/haproxy/issues
   Sources  : https://www.haproxy.org/download/3.1/src/
   Git repository   : https://git.haproxy.org/git/haproxy.git/
   Git Web browsing : https://git.haproxy.org/?p=haproxy.git
   Changelog: https://www.haproxy.org/download/3.1/src/CHANGELOG
   Dataplane API: 
https://github.com/haproxytech/dataplaneapi/releases/latest
   Pending bugs : https://www.haproxy.org/l/pending-bugs
   Reviewed bugs: https://www.haproxy.org/l/reviewed-bugs
   Code reports : https://www.haproxy.org/l/code-reports
   Latest builds: https://www.haproxy.

Re: haproxy-3.x.x - Ubuntu Focal

[Vincent Bernat]

On 2024-09-11 08:40, Alexis Vachette wrote:

Just wanted to know if you had a plan to release package for Ubuntu 
20.04 Focal.


Mostly because of OpenSSL 3.0 regression performance.

The question is more for Vincent Bernat.


I've just pushed a build for Ubuntu Focal.

Re: haproxy-3.x.x - Ubuntu Focal

[Willy Tarreau]

On Wed, Sep 11, 2024 at 10:14:13AM +0200,  ??? wrote:
> ??, 11 . 2024 ?. ? 08:44, Alexis Vachette :
> 
> > Hi,
> >
> > Just wanted to know if you had a plan to release package for Ubuntu 20.04
> > Focal.
> >
> > Mostly because of OpenSSL 3.0 regression performance.
> >
> > The question is more for Vincent Bernat.
> >
> 
> I wonder what are your expectation of SSL lib for that package. I do not
> see good choice
> 
> 1) OpenSSL-1.1.1 (only limited QUIC unfortunately)
> 2) OpenSSL-3.X (bad perf)
> 3) QuicTLS (development frozen)
> 4) AWS-LC, WolfSSL, LibreSSL (requires efforts from packaging)
> 
> or, if you do not plan to use QUIC, OpenSSL-1.1.1 would b just nice

FWIW, at HaproxyTech, we decided to stick to OpenSSL-1.1.1 for now
and are providing packages built with it. We've already got several
reports of outages with 3.0 (not surprising) and are systematically
directing customers to the 1.1.1 packages to avoid any future problem.

I've read on the Ubuntu blog that they're going to maintain their 1.1.1
package up to 2030. I don't know if it's possible to install a package
of an older distro on a newer one, but that could be convenient.

Otherwise if you only want packaged stuff for Ubuntu 20, Vincent still
provides haproxy up to 2.9 (that includes 2.8 which is LTS). But at some
point you might have to build packages yourself if you need an extended
support on an older distro, or to update to a newer distro with a
different lib or version. It's sad but that the result of OpenSSL having
irresponsibly tagged 3.0 LTS before even testing it... It ended up in
distros while being totally unfit for a server, and distros can't easily
switch to an unsupported version, as security issues are even worse for
them than performance and stability issues.

Willy

Re: haproxy-3.x.x - Ubuntu Focal

[Илья Шипицин]

ср, 11 сент. 2024 г. в 08:44, Alexis Vachette :

> Hi,
>
> Just wanted to know if you had a plan to release package for Ubuntu 20.04
> Focal.
>
> Mostly because of OpenSSL 3.0 regression performance.
>
> The question is more for Vincent Bernat.
>

I wonder what are your expectation of SSL lib for that package. I do not
see good choice

1) OpenSSL-1.1.1 (only limited QUIC unfortunately)
2) OpenSSL-3.X (bad perf)
3) QuicTLS (development frozen)
4) AWS-LC, WolfSSL, LibreSSL (requires efforts from packaging)

or, if you do not plan to use QUIC, OpenSSL-1.1.1 would b just nice


>
> Regards,
> --
>
>
> *Alexis Vachette*Expert Network Engineer
>
> 
>

haproxy-3.x.x - Ubuntu Focal

[Alexis Vachette]

Hi,

Just wanted to know if you had a plan to release package for Ubuntu 20.04
Focal.

Mostly because of OpenSSL 3.0 regression performance.

The question is more for Vincent Bernat.

Regards,
-- 


*Alexis Vachette*Expert Network Engineer

[ANNOUNCE] haproxy-3.1-dev7

[Willy Tarreau]

Hi,

HAProxy 3.1-dev7 was released on 2024/09/05. It added 56 new commits
after version 3.1-dev6.

This version collects a non-negligible number of bug fixes (16) including
the important one affecting H2+zero-copy forwarding mentioned in the 3.0
and 2.9 announcements. The other bugs that were fixed concern frozen
connections, crashes, and an issue where rate counters could be bad and
remain bad for a while in case of time jumps.

Among the changes in this version, I'm seeing:
  - support for applying a factor to byte rates in stick-tables. The issue
is that right now the rate values are stored on 32 bits and that this
limits measurable rates to about 34 Gbps. When this was added in 2010,
we wouldn't imagine that users would require more than this *per key*.
But now we're in 2024, the bwlim filter has become popular even on
large systems where high limits need to be set. It was figured that
the best way to address this specific (and admittedly rare) need is to
just apply a factor to the rate stored in the stick-table. This allows
the bwlim filter to be configured with lower values and reported rates
to still make sense. The "brates-factor" configured on the stick-table
supports a range of 1 to 1024. 1024 could imply counting kbytes per
second instead of bytes per second. 125 could be nice to count kbit/s
instead for example.

  - the MPTCP support from Dorian, Anthony and Mathieu was finally merged.
MPTCP addresses can be explicitly specified using prefixes "mptcp@",
"mptcp4@", "mptcp6@". While writing this I'm realizing that I don't
know if we have an easy way to know if we're dealing with an MPTCP or a
regular TCP connection in order to feed stats. It could be interesting
to see in logs which connections were made using this and measure how
popular it is among clients. Hmmm maybe we can do that using one
listener each... More things to test :-)

  - the init-state patch from Aaron was also merged. It finally allows
a server *not* to instantly turn on upon startup or when leaving
maintenance, but only after it succeeds its checks. There have been
requests for this for a very long time (issue 51!), and now 4 possible
behaviors can be configured per server. One use case I'm aware of is
when you set a server in maintenance to upgrade it, you usually don't
want it to instantly be reintegrated when marking it ready, instead
you usually prefer that it performs its checks. Now it will be
possible via the "init-state" server parameter.

  - a few sample fetch functions (ssl_fc_supported_versions_bin and
ssl_fc_sigalgs_bin) were added to retrieve such information during
the SSL handshake that are otherwise not available later.

  - the environment variables for CLF log formats were added

  - the master process now restores its initial environment before
switching to wait mode. Indeed, previously it would re-exec itself
using the new variables parsed in the config. An example of problem
is if you do that in your config:

global
#setenv TRACE 1

.if defined(TRACE)
...
.endif

Then once you temporarily uncomment the "setenv", the TRACE variable
is set and passed to sub-processes, then if you comment the config
again it persists since it was inherited. This won't happen anymore
as we now keep a copy of the environment that is restored before the
execve(). Note that all such corner cases won't be an issue soon with
the ongoing change of the master/worker startup sequence anyway.

  - support for "retry-on 429" was added.

  - QUIC: the NEW_TOKEN frame support was added so as to always be able
to validate reachability of the client during 0-RTT. The problem was
reported by Michael Wedl, without this a client can change its address
between the connection establishment and a 0-RTT request, the last one
will be used and logged (as expected). The problem is that we don't
know if it's its real address and we need to validate it first. The
NEW_TOKEN allows to deliver an IP-based token to the client for use
later, and if the address changes, then we can use a regular RETRY
token. It was tested to be OK and now works fine even in the interop.
It was also backported to 3.0. Not sure how far it will go since it
relies on recent infrastructure updates.

The rest is extra debug info, cleanups and typo fixes, as well as CI
updates. I'm glad that we're finally merging long-standing changes that
have been discussed for several years.

If you're running any 3.1-dev, this one should be worth an update. It
contains essentially bug fixes (including the one against the risk of
crashing the H2 mux) and all other changes are really isolated, so except
by faili

Re: [ANNOUNCE] haproxy-3.0.4

[Willy Tarreau]

Hi,

On Tue, Sep 03, 2024 at 03:52:43PM +0200, Willy Tarreau wrote:
> HAProxy 3.0.4 was released on 2024/09/03. It added 42 new commits
> after version 3.0.3.
(...)
> Note that at this point this flushes the queue of pending bugs for 3.0,
> which is a good news. There remains one exception, a recently introduced
> QUIC patchset into 3.1 to implement NEW_TOKEN on 0-RTT that we'd like to
> backport since it addresses some bad corner cases. But the backport is
> non-trivial and the patches need to be exposed a bit longer in 3.1 first,
> this might come in 3.0.5.

I shouldn't have claimed that the queue was flushed, because in the
middle of the patches marked "backport after an observation period",
Christopher found a small cluster that could have been backported as
well with this release. Nothing critical, that doesn't deserve a new
release, but we'll probably make another one once the QUIC backports
above are done, in order to *really* flush the pipe this time. It will
likely be at the same time as the next 2.8 and lower then.

Sorry about that, it's always difficult to properly spot what's eligible
yet missing (we've improved the tooling again to help detect such misses
next time). All of these have been queued into the 3.0-maint and
2.9-maint branches.

In short, no worries, but don't be surprised if you were waiting for
3.0.4 regarding a specific fix and only find it queued for next release.

Willy

[ANNOUNCE] haproxy-2.9.10

[Christopher Faulet]

Hi,

HAProxy 2.9.10 was released on 2024/09/03. It added 52 new commits
after version 2.9.9.

This release groups all backportable fixes shipped in the 3.0.3 and
3.0.4. The main topic is about fixes of two issues affecting how the H2 mux
deals with incomplete frames:

  - in one case, certain errors happening while processing an incomplete
frame did not lead to the termination of the connection, and would cause
endless wakeups to try to handle the error, preventing the process from
sleeping, thus eating CPU.

  - another case, much harder to reproduce but also observed as actively
exploited in one case, can cause an endless loop in the h2_send()
function if a processing error requiring a GOAWAY is reported with an
almost full output buffer when no more progress can be made on the input
buffer due to an incomplete frame while many streams are transmitting
data in parallel in zero-copy mode. What happens in this case is that
the output buffer is cleared (due to the error) while still leaving the
full indication that prevents output data from being considered, and no
condition to exit the loop is met. In this case the loop will be
interrupted by the watchdog which will kill the process after two
seconds. A work-around consists in simply disabling zero-copy forwarding
for HTTP/2: "tune.h2.zero-copy-fwd-send off".  This issue was assigned
CVE-2024-45506.

Other than that, the following issues were fixed:

  - 16 threads (due to an area shared for two different things it would
start to dump threads from the 17th).

  - JWT: the SSL library functions used to validate a token would leave
an error in the SSL stack, that will later be mistaken for an error
on another connection and cause it to be closed.

  - a time-of-check/time-of-use (TOCTOU) issue in the queue processing
makes it rare but possible to leave a server with no connection yet
not take any traffic. It's more likely to happen with maxconn 1,
very hard at 2 and almost impossible at 3 or above.

  - QUIC: there was a case which could produce crashes when built with the
aws-lc TLS library.

  - OCSP: a memory allocation error while loading OCSP parameters could
leave the tree locked and freeze subsequent operations.

  - some uploads to H2 servers could freeze due to the zero-copy forwarding
not always setting the END_STREAM flag on the last DATA frame (GH
#2665).

  - it was possible to crash the process when performing an implicit
protocol upgrade (TCP to HTTP due to a transition from a TCP front to an
HTTP back) if an error happened on the connection just before the
transition.

  - a crash could happen in mux-pt if an error happened on the connection
just before an abort that is going to emit a shutdown, and with a
pending wakeup that completes some work on a connection having no
transport layer anymore. This only affects TCP (e.g. peers and master
CLI; GH #2656).

  - mux-h1 could repeat a 408 error multiple times in logs when failing to
send an empty message on a full output buffer. In this case, it would
attempt to close again every client timeout and produce a log each time
despite no data leaving.

  - a bug in the bwlim filter that can make it forget to update its timeout
and loop fast when woken up with nothing to forward.

  - the lack of filtering of empty transfer-encoding headers in the H1 mux
that was reported by Ben Kallus.

  - the insufficient syntax checks on :method and :scheme H3 headers that
was reported by Yuki Mogi.

  - a hang on lua CLI applets when trying to deal with their own buffers.

  - a possible crash in deinit() (when quitting/reloading) when using e-mail
alerts after a fix that went into 2.9.9.

  - a rare but possible deadlock (and crash) in QUIC after a recent fix for
a race condition in the CID tree, that also went into 2.9.9.

  - an unreproduced race condition affecting the QUIC CID tree. The impact
might be a possible crash but it was only found in code review and never
reported nor reproduced.

  - an issue in SPOE that can cause a thread to refrain from creating an
applet to connect outside, causing failures on requests processed on
this thread.

  - a race in stick-tables starting from 2.9 where an element may be
accessed immediately after decrementing the refcount, sometimes allowing
it to be purged in parallel and causing crashes (GH #2611).

Finally a change that is not exactly an issue but addressing a problem
encountered in some setups:

  - the hard limit on the number of file descriptors now defaults to about
1 million, in order to match what has been done for a very long time
on many distros, and that recently changed to 1 billion on some of
them, causing a huge startup time (or even a watchdog at boot) and a
massive memory usage.

And the rest is pretty minor. Everyone running on the 2.9.

[ANNOUNCE] haproxy-3.0.4

[Willy Tarreau]

Hi,

HAProxy 3.0.4 was released on 2024/09/03. It added 42 new commits
after version 3.0.3.

This version addresses two issues affecting how the H2 mux deals with
incomplete frames:
  - in one case, certain errors happening while processing an incomplete
frame did not lead to the termination of the connection, and would
cause endless wakeups to try to handle the error, preventing the
process from sleeping, thus eating CPU.

  - another case, much harder to reproduce but also observed as actively
exploited in one case, can cause an endless loop in the h2_send()
function if a processing error requiring a GOAWAY is reported with an
almost full output buffer when no more progress can be made on the
input buffer due to an incomplete frame while many streams are
transmitting data in parallel in zero-copy mode. What happens in this
case is that the output buffer is cleared (due to the error) while
still leaving the full indication that prevents output data from being
considered, and no condition to exit the loop is met. In this case the
loop will be interrupted by the watchdog which will kill the process
after two seconds. A work-around consists in simply disabling
zero-copy forwarding for HTTP/2:  "tune.h2.zero-copy-fwd-send off".
This issue was assigned CVE-2024-45506.

Other than that, the following issues were fixed:
  - CLI: the "show threads" command would crash if issued with less than
16 threads (due to an area shared for two different things it would
start to dump threads from the 17th).

  - JWT: the SSL library functions used to validate a token would leave
an error in the SSL stack, that will later be mistaken for an error
on another connection and cause it to be closed.

  - a time-of-check/time-of-use (TOCTOU) issue in the queue processing
makes it rare but possible to leave a server with no connection yet
not take any traffic. It's more likely to happen with maxconn 1,
very hard at 2 and almost impossible at 3 or above.

  - QUIC: when the accept queue is full and the incoming connection cannot
be migrated to another thread, it needed to be re-migrated to the local
thread, which was not supported because the migration had already
started, and would crash. Also there was a case which could produce
crashes when built with the aws-lc TLS library.

  - OCSP: a memory allocation error while loading OCSP parameters could
leave the tree locked and freeze subsequent operations.

  - some uploads to H2 servers could freeze due to the zero-copy
forwarding not always setting the END_STREAM flag on the last DATA
frame (GH #2665).

  - it was possible to crash the process when performing an implicit
protocol upgrade (TCP to HTTP due to a transition from a TCP front
to an HTTP back) if an error happened on the connection just before
the transition.

  - a crash could happen in mux-pt if an error happened on the connection
just before an abort that is going to emit a shutdown, and with a
pending wakeup that completes some work on a connection having no
transport layer anymore. This only affects TCP (e.g. peers and master
CLI; GH #2656).

  - mux-h1 could repeat a 408 error multiple times in logs when failing
to send an empty message on a full output buffer. In this case, it
would attempt to close again every client timeout and produce a log
each time despite no data leaving.

Finally these changes are not exactly issues but address problems
encountered in some setups:

  - the hard limit on the number of file descriptors now defaults to about
1 million, in order to match what has been done for a very long time
on many distros, and that recently changed to 1 billion on some of
them, causing a huge startup time (or even a watchdog at boot) and a
massive memory usage.

  - New global directive "h1-do-not-close-on-insecure-transfer-encoding"
was added to explicitly permit to maintain a connection alive when it
uses both Content-Length and Transfer-Encoding. This goes against the
latest version of the HTTP specification but may be needed with some
clients or servers which cause too many TLS reconnections because of
this despite being in well controlled environments.

  - The log-format parser became stricter in 3.0 as a side effect of some
of the log processing improvements. A fatal error was emitted when an
alias or expression was used in an incompatible context (e.g. HTTP info
in TCP logs, but there are more subtle ones), preventing some working
configs from starting. This has now been relaxed and is only produced
in diagnostic mode (-dD). This was Github issue #2642.

  - support for "retry-on 429" was added (GH #2687)

And the rest is pretty minor. For the more problematic issues above, a
2.9 release will follow shortly.

Thanks to all those who kindly shared 

Re: [PATCH] CLEANUP: haproxy: fix typos in code comment

[Willy Tarreau]

Hi Nicolas,

On Tue, Aug 27, 2024 at 10:18:51PM +0200, Nicolas CARPi wrote:
> Found these two typos while browsing the code :)
(...)
> Found this typo in macro name :)

Thank you, both patches applied.

> BTW, in mqtt.c mqtt_read_varint(), "off" is initialized to 0, but 
> initialized again in the for loop, which results in a dead assignment.
> 
> I believe this is done on purpose, as some sort of defensive
> programming, so I didn't touch it, but wanted to bring attention to it, 
> in case it is not something wanted.

Indeed it's not the prettiest but not a problem either. Sometimes
this can happen as the result of removing a check and an error path
for example. Let's leave it as is, it has no effect anyway :-)

Thank you!
Willy

[PATCH] CLEANUP: haproxy: fix typos in code comment

[Nicolas CARPi]

Hello,

Found these two typos while browsing the code :)

Wanted to see how haproxy gets random bytes. That seed initialization 
code really goes out of its way to grab all entropy it can find! ^^

Best,
~Nico
>From b71acb32c4122819d50dac11ae60bb5cac1faba8 Mon Sep 17 00:00:00 2001
From: Nicolas CARPi 
Date: Tue, 27 Aug 2024 22:02:17 +0200
Subject: [PATCH] CLEANUP: haproxy: fix typos in code comment

Use "from" instead of "form" in ha_random_boot function code comments.
---
 src/haproxy.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/src/haproxy.c b/src/haproxy.c
index f7bb5b5d8..21cdcb340 100644
--- a/src/haproxy.c
+++ b/src/haproxy.c
@@ -1417,11 +1417,11 @@ static void ha_random_boot(char *const *argv)
m += 4;
}
 
-   /* stack address (benefit form operating system's ASLR) */
+   /* stack address (benefit from operating system's ASLR) */
l = (unsigned long)&m;
memcpy(m, &l, sizeof(l)); m += sizeof(l);
 
-   /* argv address (benefit form operating system's ASLR) */
+   /* argv address (benefit from operating system's ASLR) */
l = (unsigned long)&argv;
memcpy(m, &l, sizeof(l)); m += sizeof(l);
 
-- 
2.46.0

Re: HAProxy returning 502 with SH--

[BJ Taylor]

Ok, will do. Thanks.

BJ


On Tue, Aug 27, 2024 at 10:42 AM Lukas Tribus  wrote:

> Also, before doing anything else, try using:
>
> tune.disable-zero-copy-forwarding or tune.h1.zero-copy-fwd-recv off
>
> as there is currently an open bug that doesn't fully match your case
> but is still close enough that it may be worth a try:
>
> https://github.com/haproxy/haproxy/issues/2665
>
>
>
> Lukas
>

Re: HAProxy returning 502 with SH--

[Lukas Tribus]

Also, before doing anything else, try using:

tune.disable-zero-copy-forwarding or tune.h1.zero-copy-fwd-recv off

as there is currently an open bug that doesn't fully match your case
but is still close enough that it may be worth a try:

https://github.com/haproxy/haproxy/issues/2665



Lukas

Re: HAProxy returning 502 with SH--

[Lukas Tribus]

Hello,

On Tue, 27 Aug 2024 at 18:09, BJ Taylor  wrote:
>
> Here are the 502 logs from the last run after the config changes.
>
> 2024-08-26T09:29:02.547581-06:00 testserver haproxy[284569]: <134>Aug 26 
> 09:29:02 haproxy[284569]: 192.168.69.101:45382 [26/Aug/2024:09:29:02.545] 
> www~ front3/pdafront32 0/0/0/-1/1 502 208 - - SH-- 5/5/3/3/0 0/0 
> {front3.domain.com|} "POST https://front3.domain.com/front1 HTTP/2.0"
> 2024-08-26T11:27:20.748921-06:00 testserver haproxy[284569]: <134>Aug 26 
> 11:27:20 haproxy[284569]: 192.168.69.101:50606 [26/Aug/2024:11:27:20.746] 
> www~ front3/pdafront32 0/0/0/-1/1 502 208 - - SH-- 5/5/3/3/0 0/0 
> {front3.domain.com|} "POST https://front3.domain.com/front1 HTTP/2.0"
> 2024-08-26T14:11:11.289987-06:00 testserver haproxy[284569]: <134>Aug 26 
> 14:11:11 haproxy[284569]: 192.168.69.101:40516 [26/Aug/2024:14:11:11.285] 
> www~ front3/pdafront32 0/0/0/-1/2 502 208 - - SH-- 15/15/7/7/0 0/0 
> {front3.domain.com|} "POST https://front3.domain.com/front1 HTTP/2.0"
> 2024-08-26T17:40:55.801154-06:00 testserver haproxy[284569]: <134>Aug 26 
> 17:40:55 haproxy[284569]: 192.168.69.101:53952 [26/Aug/2024:17:40:55.798] 
> www~ front3/pdafront32 0/0/0/-1/1 502 208 - - SH-- 10/10/1/1/0 0/0 
> {front3.domain.com|} "POST https://front3.domain.com/front1 HTTP/2.0"

This indicates that your backend applications crashes or at least does
not complete the HTTP response header after 208 bytes.

It possible that "show errors" on the haproxy admin socket gives your
more insight into what the HTTP response of your server looks like
(and where it suddenly aborts exactly after those 208 bytes).

Try switching off H2 in the backend, to see if this is H2 related.

If you can switch off SSL on the backend and you can still reproduce
the issue, you may have an easier time debugging this with network
traces.

Otherwise if you have no possibilities to troubleshoot at the backend
application, show errors is not useful and you cannot disable SSL on
the backend, you need to be able to decrypt the backend traffic from a
network trace. Reproducing with a non-FS cipher will allow you to
decrypt the SSL traffic with the certificates private key; otherwise
you have to use client random logging [1] and then decrypt the traffic
in wireshark before analyzing what happens in those last bytes of the
208 byte incomplete HTTP response.



Regards,
Lukas


[1] http://docs.haproxy.org/3.0/configuration.html#3.2-tune.ssl.keylog

Re: HAProxy returning 502 with SH--

[BJ Taylor]

Here are the 502 logs from the last run after the config changes.

2024-08-26T09:29:02.547581-06:00 testserver haproxy[284569]: <134>Aug 26
09:29:02 haproxy[284569]: 192.168.69.101:45382 [26/Aug/2024:09:29:02.545]
www~ front3/pdafront32 0/0/0/-1/1 502 208 - - SH-- 5/5/3/3/0 0/0 {
front3.domain.com|} "POST https://front3.domain.com/front1 HTTP/2.0"
2024-08-26T11:27:20.748921-06:00 testserver haproxy[284569]: <134>Aug 26
11:27:20 haproxy[284569]: 192.168.69.101:50606 [26/Aug/2024:11:27:20.746]
www~ front3/pdafront32 0/0/0/-1/1 502 208 - - SH-- 5/5/3/3/0 0/0 {
front3.domain.com|} "POST https://front3.domain.com/front1 HTTP/2.0"
2024-08-26T14:11:11.289987-06:00 testserver haproxy[284569]: <134>Aug 26
14:11:11 haproxy[284569]: 192.168.69.101:40516 [26/Aug/2024:14:11:11.285]
www~ front3/pdafront32 0/0/0/-1/2 502 208 - - SH-- 15/15/7/7/0 0/0 {
front3.domain.com|} "POST https://front3.domain.com/front1 HTTP/2.0"
2024-08-26T17:40:55.801154-06:00 testserver haproxy[284569]: <134>Aug 26
17:40:55 haproxy[284569]: 192.168.69.101:53952 [26/Aug/2024:17:40:55.798]
www~ front3/pdafront32 0/0/0/-1/1 502 208 - - SH-- 10/10/1/1/0 0/0 {
front3.domain.com|} "POST https://front3.domain.com/front1 HTTP/2.0"

I'm open to suggestions.
Thanks,
BJ

On Fri, Aug 23, 2024 at 2:20 PM BJ Taylor  wrote:

> I will make the suggested changes and try again. Here are the failure logs
> for the last run. Queues are 0, current connections are not all that high.
>
> Aug 22 01:12:12 haproxy[87118]:
>> {"host":"testserver","ident":"haproxy","pid":87118,"timestamp":"22/Aug/2024:01:12:12
>> -0600","haproxy":{"connections":{"active":27,"frontend":27,"backend":13,"server":13},"queue":{"backend":0,"server":0},"timing_ms":{"time_to_end_of_headers":0,"client_request_send_time":226,"queue_wait_time":0,"server_wait_time":0,"server_response_send_time":-1,"response_time":-1,"session_duration":226},"termination_state":"SH--","retries":0,"network":{"client_ip":"192.168.69.101","client_port":47794,"frontend_ip":"192.168.39.5","frontend_port":443},"ssl":{"version":"TLSv1.3","ciphers":"TLS_AES_128_GCM_SHA256"},"request":{"method":"POST","uri":"\/front1","protocol":"HTTP/2.0","header":{"host":"
>> front3.domain.com",
>>  
>> "xforwardfor":"-","referer":"-"}},"name":{"frontend":"www~","backend":"front3",
>>"server":"testbackend1"},
>>  
>> "response":{"status_code":502,"header":{"xrequestid":"-"}},"bytes":{"uploaded":65785,"read":208}}}
>> Aug 22 02:57:48 haproxy[87118]:
>> {"host":"testserver","ident":"haproxy","pid":87118,"timestamp":"22/Aug/2024:02:57:48
>> -0600","haproxy":{"connections":{"active":18,"frontend":18,"backend":3,
>> "server":3
>> },"queue":{"backend":0,"server":0},"timing_ms":{"time_to_end_of_headers":0,"client_request_send_time":226,"queue_wait_time":0,"server_wait_time":0,"server_response_send_time":-1,"response_time":-1,"session_duration":226},"termination_state":"SH--","retries":0,"network":{"client_ip":"192.168.69.101","client_port":51728,"frontend_ip":"192.168.39.5","frontend_port":443},"ssl":{"version":"TLSv1.3","ciphers":"TLS_AES_128_GCM_SHA256"},"request":{"method":"POST","uri":"\/front1","protocol":"HTTP/2.0","header":{"host":"
>> front3.domain.com",
>>  
>> "xforwardfor":"-","referer":"-"}},"name":{"frontend":"www~","backend":"front3",
>>"server":"testbackend1"},
>>  
>> "response":{"status_code":502,"header":{"xrequestid":"-"}},"bytes":{"uploaded":250,
>>  "read":208}}}
>> Aug 22 03:22:

[PATCH v2 0/4] Add MPTCP to haproxy

[Aperence]

Multipath TCP (MPTCP), standardized in RFC8684 [1], is a TCP extension
that enables a TCP connection to use different paths.

Multipath TCP has been used for several use cases. On smartphones, MPTCP
enables seamless handovers between cellular and Wi-Fi networks while
preserving established connections. This use-case is what pushed Apple
to use MPTCP since 2013 in multiple applications [2]. On dual-stack
hosts, Multipath TCP enables the TCP connection to automatically use the
best performing path, either IPv4 or IPv6. If one path fails, MPTCP
automatically uses the other path.

To benefit from MPTCP, both the client and the server have to support
it. Multipath TCP is a backward-compatible TCP extension that is enabled
by default on recent Linux distributions (Debian, Ubuntu, Redhat, ...).
Multipath TCP is included in the Linux kernel since version 5.6 [3]. To
use it on Linux, an application must explicitly enable it when creating
the socket. No need to change anything else in the application.


Link: https://www.rfc-editor.org/rfc/rfc8684.html [1]
Link: https://www.tessares.net/apples-mptcp-story-so-far/ [2]
Link: https://www.mptcp.dev [3]

---

Changelog:

v2:
  - Patch 1: (new)
- Separated the changes to str2sa_range into a different patch 
  (https://www.mail-archive.com/haproxy@formilux.org/msg45229.html).
  This function adds a new "alt" parameter that can be used
  to get an alternate protocol to use. (Willy)
  - Patch 2: (new)
- Add a protocol field to the server, and fill its value 
  with the modified version of str2sa_range that accepts an
  "alt" parameter. (Willy)
  - Patch 3: (new)
- Modify the call to create the socket, by passing the protocol 
  used. (Willy)
  - Patch 4:
- Regrouped the changes related to MPTCP and included the changes
  about TCP_MAXSEG being not supported (yet) with IPPROTO_MPTCP.
  (Willy)
v1:
  - Patch 1:
- Duplicate the mptcp(4|6) protocol structures from their tcp
  equivalent, instead of using a function that creates them by copying
  tcp structures
- Add support for MPTCP on the backend side (between the proxy and
  the servers)
  - Patch 2:
- Fix a warning about TCP_MAXSEG not being supported while MPTCP is
  used by declaring new constants sock_inet(6)_mptcp_maxseg_default.
  Those constants represent the MSS supported by MPTCP (== -1 for the
  moment, as long as TCP_MAXSEG is not supported). For the moment,
  MPTCP doesn't support TCP_MAXSEG socket option, mainly because it
  has apparently never been requested before, apparently. It should
  not be difficult to implement it, but is it an important option
  for HAProxy?
  - https://www.mail-archive.com/haproxy@formilux.org/msg45224.html
v0: (Dorian & Matthieu)
  - https://www.mail-archive.com/haproxy@formilux.org/msg44871.html

Aperence (4):
  FEATURE: extend str2sa_range to add an alt parameter
  FEATURE: add a alt_proto field for server
  REORG: use protocol when creating socket
  FEATURE: add MPTCP per address support

 doc/configuration.txt   |  21 +++
 examples/mptcp-backend.py   |  22 
 examples/mptcp.cfg  |  23 
 include/haproxy/compat.h    |  10 
 include/haproxy/protocol.h  |   4 +-
 include/haproxy/server-t.h  |   1 +
 include/haproxy/sock_inet.h |   5 ++
 include/haproxy/tools.h |   2 +-
 src/backend.c   |   3 +-
 src/cfgparse-listen.c   |   8 ++-
 src/cfgparse.c  |   6 +-
 src/check.c |   2 +-
 src/cli.c   |   2 +-
 src/hlua.c  |   3 +-
 src/http_client.c   |   2 +-
 src/log.c   |   2 +-
 src/proto_tcp.c | 109 +++-
 src/protocol.c  |   3 +-
 src/resolvers.c |   3 +-
 src/server.c|  10 ++--
 src/sock.c  |   7 ++-
 src/sock_inet.c |  30 ++
 src/ssl_ocsp.c  |   2 +-
 src/tcpcheck.c  |   3 +-
 src/tools.c |  36 +++-
 25 files changed, 289 insertions(+), 30 deletions(-)
 create mode 100644 examples/mptcp-backend.py
 create mode 100644 examples/mptcp.cfg

-- 
2.46.0

Re: HAProxy returning 502 with SH--

[BJ Taylor]

I will make the suggested changes and try again. Here are the failure logs
for the last run. Queues are 0, current connections are not all that high.

Aug 22 01:12:12 haproxy[87118]:
> {"host":"testserver","ident":"haproxy","pid":87118,"timestamp":"22/Aug/2024:01:12:12
> -0600","haproxy":{"connections":{"active":27,"frontend":27,"backend":13,"server":13},"queue":{"backend":0,"server":0},"timing_ms":{"time_to_end_of_headers":0,"client_request_send_time":226,"queue_wait_time":0,"server_wait_time":0,"server_response_send_time":-1,"response_time":-1,"session_duration":226},"termination_state":"SH--","retries":0,"network":{"client_ip":"192.168.69.101","client_port":47794,"frontend_ip":"192.168.39.5","frontend_port":443},"ssl":{"version":"TLSv1.3","ciphers":"TLS_AES_128_GCM_SHA256"},"request":{"method":"POST","uri":"\/front1","protocol":"HTTP/2.0","header":{"host":"
> front3.domain.com",
>  
> "xforwardfor":"-","referer":"-"}},"name":{"frontend":"www~","backend":"front3",
>"server":"testbackend1"},
>  
> "response":{"status_code":502,"header":{"xrequestid":"-"}},"bytes":{"uploaded":65785,"read":208}}}
> Aug 22 02:57:48 haproxy[87118]:
> {"host":"testserver","ident":"haproxy","pid":87118,"timestamp":"22/Aug/2024:02:57:48
> -0600","haproxy":{"connections":{"active":18,"frontend":18,"backend":3,
> "server":3
> },"queue":{"backend":0,"server":0},"timing_ms":{"time_to_end_of_headers":0,"client_request_send_time":226,"queue_wait_time":0,"server_wait_time":0,"server_response_send_time":-1,"response_time":-1,"session_duration":226},"termination_state":"SH--","retries":0,"network":{"client_ip":"192.168.69.101","client_port":51728,"frontend_ip":"192.168.39.5","frontend_port":443},"ssl":{"version":"TLSv1.3","ciphers":"TLS_AES_128_GCM_SHA256"},"request":{"method":"POST","uri":"\/front1","protocol":"HTTP/2.0","header":{"host":"
> front3.domain.com",
>  
> "xforwardfor":"-","referer":"-"}},"name":{"frontend":"www~","backend":"front3",
>"server":"testbackend1"},
>  
> "response":{"status_code":502,"header":{"xrequestid":"-"}},"bytes":{"uploaded":250,
>  "read":208}}}
> Aug 22 03:22:34 haproxy[87118]:
> {"host":"testserver","ident":"haproxy","pid":87118,"timestamp":"22/Aug/2024:03:22:34
> -0600","haproxy":{"connections":{"active":14,"frontend":14,"backend":4,
> "server":4
> },"queue":{"backend":0,"server":0},"timing_ms":{"time_to_end_of_headers":0,"client_request_send_time":222,"queue_wait_time":0,"server_wait_time":0,"server_response_send_time":-1,"response_time":-1,"session_duration":223},"termination_state":"SH--","retries":0,"network":{"client_ip":"192.168.69.101","client_port":36912,"frontend_ip":"192.168.39.5","frontend_port":443},"ssl":{"version":"TLSv1.3","ciphers":"TLS_AES_128_GCM_SHA256"},"request":{"method":"POST","uri":"\/front1","protocol":"HTTP/2.0","header":{"host":"
> front3.domain.com",
>  
> "xforwardfor":"-","referer":"-"}},"name":{"frontend":"www~","backend":"front3",
>"server":"testbacke

Re: HAProxy returning 502 with SH--

[Lukas Tribus]

On Fri, 23 Aug 2024 at 18:55, BJ Taylor  wrote:
>
> We are trying to deploy HAProxy into our environment. We have a script that
> does some 600k api calls during approximately 24 hours.

How many concurrent connections / transactions though?


>  During that time, when haproxy is in place, there are a handful (8-12) of
> responses that come back as 502 with SH--.

As per the documentation:

 SH   The server aborted before sending its full HTTP response headers, or
  it crashed while processing the request. Since a server aborting at
  this moment is very rare, it would be wise to inspect its logs to
  control whether it crashed and why. The logged request may indicate a
  small set of faulty requests, demonstrating bugs in the application.
  Sometimes this might also be caused by an IDS killing the connection
      between HAProxy and the server.


You should disable custom format logging, enable httplog format and
share the affected SH log line.



>tune.bufsize 8388608
>tune.maxrewrite 1024

bufsize should usually be 16K maybe 32K for some specific application
requiring huge headers, but not 8M. I think 8M is unreasonable and I
think it will lead to issues one way or another.


>timeout connect 86400s
>timeout client  86400s
>timeout server  86400s

1 day long timeouts will probably also lead to an issue at some point,
due to sessions not expiring.




Lukas

HAProxy returning 502 with SH--

[BJ Taylor]

We are trying to deploy HAProxy into our environment. We have a script that
does some 600k api calls during approximately 24 hours. During that time,
when haproxy is in place, there are a handful (8-12) of responses that come
back as 502 with SH--. We've tested this against Ubuntu's default 2.8
version of HAProxy, along with 3.0.3 and most recently 3.1-dev5-8427c5b. It
is the same in each version. The logs don't seem to tell us much. The
backend server doesn't show any issues, and indeed, when HAProxy is not in
place, there are no errors whatsoever.

I'm afraid that at the moment, I don't know how to reproduce this beyond
what we are doing in our environment, but that's not something we can share
with others. That said, if you have any guidance on how I can discover what
is going on, I'd be more than happy to dig in further. I turned the logging
to debug, but it doesn't seem to show me anything more than what info was
giving us.

Our config file looks like the following:

global
    log stdout len 65535 local0 debug
user haproxy
group haproxy
daemon
stats timeout 30

# Default SSL material locations
ca-base /etc/ssl/certs
crt-base /etc/ssl/private
ssl-default-bind-ciphers (list)
ssl-default-bind-ciphersuites (list)
ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets
tune.bufsize 8388608
tune.maxrewrite 1024

defaults
log global
mode http
option dontlognull
log-format
'{"host":"%H","ident":"haproxy","pid":%pid,"timestamp":"%Tl","haproxy":{"connections":{"active":%ac,"frontend":%fc,"backend":%bc,"server":%sc},"queue":{"backend":%bq,"server":%sq},"timing_ms":{"time_to_end_of_headers":%TR,"client_request_send_time":%Tq,"queue_wait_time":%Tw,"server_wait_time":%Tc,"server_response_send_time":%Tr,"response_time":%Td,"session_duration":%Tt},"termination_state":"%tsc","retries":%rc,"network":{"client_ip":"%ci","client_port":%cp,"frontend_ip":"%fi","frontend_port":%fp},"ssl":{"version":"%sslv","ciphers":"%sslc"},"request":{"method":"%HM","uri":"%[front4.req.uri,json(utf8s)]","protocol":"%HV","header":{"host":"%[front4.req.hdr(0),json(utf8s)]","xforwardfor":"%[front4.req.hdr(1),json(utf8s)]","referer":"%[front4.req.hdr(2),json(utf8s)]"}},"name":{"frontend":"%ft","backend":"%b","server":"%s"},"response":{"status_code":%ST,"header":{"xrequestid":"%[front4.res.hdr(0),json(utf8s)]"}},"bytes":{"uploaded":%U,"read":%B}}}'
timeout connect 86400s
timeout client  86400s
timeout server  86400s
errorfile 400 /etc/haproxy/errors/400.http
errorfile 403 /etc/haproxy/errors/403.http
errorfile 408 /etc/haproxy/errors/408.http
errorfile 500 /etc/haproxy/errors/500.http
errorfile 502 /etc/haproxy/errors/502.http
errorfile 503 /etc/haproxy/errors/503.http
errorfile 504 /etc/haproxy/errors/504.http

#listen prometheus #this doesn't work in 3.1
#bind *:9100
#mode http
#http-request use-service prometheus-exporter if { path /metrics }
#no log

frontend www
http-request front4 req.hdr(Host) len 1000
http-request front4 req.hdr(Referer) len 1000
bind *:80
bind *:443 ssl crt /etc/ssl/private/star.domain.com.pem crt
/etc/ssl/private/star.domain2.com.pem crt
/etc/ssl/private/star.domain3.com.pem alpn h2,http/1.1
option forwardfor
# stats stuff for admin path for web interface
stats enable
stats uri /admin
stats realm HAProxy\ Statistics
stats auth admin:"$AUTH"
stats admin if TRUE
stats show-node

 data frontends 
acl front1_acl hdr(host) -i front1.domain.com
use_backend front1 if front1_acl

acl front2_acl hdr(host) -i front2.domain.com
use_backend front2 if front2_acl

acl front3_pgst_acl hdr(host) -i front3.domain.com
use_backend front3 if front3_pgst_acl

acl front3_redx_acl hdr(host) -i front3.domain2.com
use_backend front3 if front3_redx_acl

acl front3_theredx_acl hdr(host) -i front3.domain3.com
use_backend front3 if front3_theredx_acl

acl front4_acl hdr(host) -i front4.domain.com
use_backend front4 if front4_acl

acl front5_acl hdr(host) -i front5.domain.com
use_backend front5 if front5_acl

acl front6_acl hdr(host) -i front6.domain.com
use_backend front6 if

Re: [PATCH 0/2] Add MPTCP to haproxy

[Matthieu Baerts]

On 23/08/2024 17:20, Willy Tarreau wrote:
> On Fri, Aug 23, 2024 at 05:11:11PM +0200, Matthieu Baerts wrote:

(...)

>> Maybe a new socket option would be better if the idea is only to
>> silently drop connections? :)
> 
> Yes, probably. Right now it's done directly in the action itself
> (tcp_exec_action_silent_drop()), and we already detect if the option
> is supported, so that would just be a matter of checking more
> explicitly for conn->ctrl->proto == mptcp and using the other option.

If such option is added, I think it is probably better to add it at TCP
level (or socket level), and then adds its support for MPTCP socket. But
yes, good to know it is easy to add alternatives there!

> In any case the code *tries* to make use of this, and will fall back
> to other methods. One of them is to try to change the TTL of the
> socket before closing (so that the RST doesn't reach the client).
> And when all of this fails, the connection is closed anyway, so it's
> just not silent. As such, there's no harm at all in not supporting this
> at all, it's just less optimal.

I sent patches more than one year ago to support these socket options to
modify the TTL in v4 and v6, as part of a refactoring, and they are
still not applied :(

(/me updates his TODO list)

>> (If these botnets are using "plain" TCP, the TCP_REPAIR socket option
>> will work!)
> 
> Ah, good to know! I guess we still have quite some time ahead before
> they start playing with mptcp by default, so for most use cases it will
> be fine!

I guess yes :)

(and currently MPTCP is not supported on Windows)

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: [PATCH 0/2] Add MPTCP to haproxy

[Willy Tarreau]

On Fri, Aug 23, 2024 at 05:11:11PM +0200, Matthieu Baerts wrote:
> >>> With that said, from an implementation perspective, it would seem right
> >>> to make sure that most TCP tunables also work with MPTCP.
> >>
> >> That's what we tried to do. All "common" ones are supported, but it is
> >> not always easy to define what is "common" and what is not! There are
> >> many obscure/specific socket options out there :)
> > 
> > I agree :-) For example we're using other horrors such as TCP_REPAIR,
> > which was initially defined for CRIU, and that we're using to silently
> > destroy a connection without responding. It's extremely effective against
> > HTTP botnets as they keep their connection pending and do not consume
> > anything locally. I don't know if you have it, but if you're interested
> > in giving it a try, I can help you set it up in haproxy for a test.
> 
> Wow, nice bazooka :-)

:-)

> Here is what we can currently find in the MPTCP code in the kernel:
> 
>   /* TCP_REPAIR, TCP_REPAIR_QUEUE, TCP_QUEUE_SEQ, TCP_REPAIR_OPTIONS,
>* TCP_REPAIR_WINDOW are not supported, better avoid this mess
>*/
> 
> Maybe a new socket option would be better if the idea is only to
> silently drop connections? :)

Yes, probably. Right now it's done directly in the action itself
(tcp_exec_action_silent_drop()), and we already detect if the option
is supported, so that would just be a matter of checking more
explicitly for conn->ctrl->proto == mptcp and using the other option.

In any case the code *tries* to make use of this, and will fall back
to other methods. One of them is to try to change the TTL of the
socket before closing (so that the RST doesn't reach the client).
And when all of this fails, the connection is closed anyway, so it's
just not silent. As such, there's no harm at all in not supporting this
at all, it's just less optimal.

> (If these botnets are using "plain" TCP, the TCP_REPAIR socket option
> will work!)

Ah, good to know! I guess we still have quite some time ahead before
they start playing with mptcp by default, so for most use cases it will
be fine!

Willy

Re: [PATCH 0/2] Add MPTCP to haproxy

[Matthieu Baerts]

On 23/08/2024 16:42, Willy Tarreau wrote:
> On Fri, Aug 23, 2024 at 04:13:16PM +0200, Matthieu Baerts wrote:

(...)

>>> I'll comment on each patch separately,
>>
>> Thank you, please take your time!
> 
> That's what I'm doing but I really want to make sure we won't discover
> last-minute show-stoppers that require important changes. It's still
> possible to do some important ones for a few weeks, so let's tackle all
> roadblocks while we can. I really feel like we're on a good track now!

It makes sense, thank you for having done the reviews so quickly!

I will check with Anthony if he needs some help there.

(...)

>>> With that said, from an implementation perspective, it would seem right
>>> to make sure that most TCP tunables also work with MPTCP.
>>
>> That's what we tried to do. All "common" ones are supported, but it is
>> not always easy to define what is "common" and what is not! There are
>> many obscure/specific socket options out there :)
> 
> I agree :-) For example we're using other horrors such as TCP_REPAIR,
> which was initially defined for CRIU, and that we're using to silently
> destroy a connection without responding. It's extremely effective against
> HTTP botnets as they keep their connection pending and do not consume
> anything locally. I don't know if you have it, but if you're interested
> in giving it a try, I can help you set it up in haproxy for a test.

Wow, nice bazooka :-)

Here is what we can currently find in the MPTCP code in the kernel:

  /* TCP_REPAIR, TCP_REPAIR_QUEUE, TCP_QUEUE_SEQ, TCP_REPAIR_OPTIONS,
   * TCP_REPAIR_WINDOW are not supported, better avoid this mess
   */

Maybe a new socket option would be better if the idea is only to
silently drop connections? :)

(If these botnets are using "plain" TCP, the TCP_REPAIR socket option
will work!)

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: [PATCH 0/2] Add MPTCP to haproxy

[Willy Tarreau]

Hi Matthieu,

On Fri, Aug 23, 2024 at 04:13:16PM +0200, Matthieu Baerts wrote:
> Hi Willy,
> 
> Thank you for your quick reply!

You're welcome!

> > I'll comment on each patch separately,
> 
> Thank you, please take your time!

That's what I'm doing but I really want to make sure we won't discover
last-minute show-stoppers that require important changes. It's still
possible to do some important ones for a few weeks, so let's tackle all
roadblocks while we can. I really feel like we're on a good track now!

(... mss ...)

> I just filled a new ticket on MPTCP side, not to forget about that:
> 
>   https://github.com/multipath-tcp/mptcp_net-next/issues/515

OK!

> > With that said, from an implementation perspective, it would seem right
> > to make sure that most TCP tunables also work with MPTCP.
> 
> That's what we tried to do. All "common" ones are supported, but it is
> not always easy to define what is "common" and what is not! There are
> many obscure/specific socket options out there :)

I agree :-) For example we're using other horrors such as TCP_REPAIR,
which was initially defined for CRIU, and that we're using to silently
destroy a connection without responding. It's extremely effective against
HTTP botnets as they keep their connection pending and do not consume
anything locally. I don't know if you have it, but if you're interested
in giving it a try, I can help you set it up in haproxy for a test.

Cheers,
Willy

Re: [PATCH 0/2] Add MPTCP to haproxy

[Matthieu Baerts]

Hi Willy,

Thank you for your quick reply!

On 23/08/2024 15:58, Willy Tarreau wrote:
> On Fri, Aug 23, 2024 at 03:34:08PM +0200, Aperence wrote:

(...)

> I'll comment on each patch separately,

Thank you, please take your time!

> though I'll respond to the
> question below:
> 
>>   - Patch 2:
>> - Fix a warning about TCP_MAXSEG not being supported while MPTCP is 
>> used by declaring new constants sock_inet(6)_mptcp_maxseg_default.
>> Those constants represent the MSS supported by MPTCP (== -1 for the
>> moment, as long as TCP_MAXSEG is not supported). For the moment, 
>> MPTCP doesn't support TCP_MAXSEG socket option, mainly because it 
>> has apparently never been requested before, apparently. It should 
>>     not be difficult to implement it, but is it an important option
>> for HAProxy?
> 
> It's not that common anymore but 15 years ago when ADSL over PPPoE was
> everywhere, it was super frequent to see users not able to upload large
> POST data over TCP because their PC connected with a 1500 MTU to their
> PPPoE modem/router was seeing its packets rejected due to the extra 8
> bytes of PPPoE encapsulation. Many people had by then implemented the
> set-mss rule in iptables on the server infrastructure to deal with this,
> but high-bandwidth hosting providers couldn't even just place a firewall
> in front of haproxy just to deal with that, so in this case it was very
> convenient to permit haproxy to simply configure the socket to negotiate
> a lower MSS compatible with everyone's setup.
> 
> Nowadays with everyone having a triple-play box provided by the ISP,
> all this stuff is transparent to the user (when needed at all). However
> I've been aware of users doing that as well to fixup trouble caused by
> inter-company VPN links. You know, the type of places where everyone
> says "it's right on my side, it must be the other side"...
> 
> So I guess that nowadays the mss setting mostly serves to fixup VPN
> environments, and it's not much likely that we'll see that with MPTCP
> soon I think (but I could of course be wrong).

Many thanks for this explanation!

I just filled a new ticket on MPTCP side, not to forget about that:

  https://github.com/multipath-tcp/mptcp_net-next/issues/515

> With that said, from an implementation perspective, it would seem right
> to make sure that most TCP tunables also work with MPTCP.

That's what we tried to do. All "common" ones are supported, but it is
not always easy to define what is "common" and what is not! There are
many obscure/specific socket options out there :)

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: [PATCH 0/2] Add MPTCP to haproxy

[Willy Tarreau]

Hello!

On Fri, Aug 23, 2024 at 03:34:08PM +0200, Aperence wrote:
> Multipath TCP (MPTCP), standardized in RFC8684 [1], is a TCP extension
> that enables a TCP connection to use different paths.
> 
> Multipath TCP has been used for several use cases. On smartphones, MPTCP
> enables seamless handovers between cellular and Wi-Fi networks while
> preserving established connections. This use-case is what pushed Apple
> to use MPTCP since 2013 in multiple applications [2]. On dual-stack
> hosts, Multipath TCP enables the TCP connection to automatically use the
> best performing path, either IPv4 or IPv6. If one path fails, MPTCP
> automatically uses the other path.
> 
> To benefit from MPTCP, both the client and the server have to support
> it. Multipath TCP is a backward-compatible TCP extension that is enabled
> by default on recent Linux distributions (Debian, Ubuntu, Redhat, ...).
> Multipath TCP is included in the Linux kernel since version 5.6 [3]. To
> use it on Linux, an application must explicitly enable it when creating
> the socket. No need to change anything else in the application.
> 
> These patches are a continuation of the work [4] realized by Dorian 
> Craps and Matthieu Baerts.

It's great to see Dorian's work pursued, thank you for this series!

I'll comment on each patch separately, though I'll respond to the
question below:

>   - Patch 2:
> - Fix a warning about TCP_MAXSEG not being supported while MPTCP is 
> used by declaring new constants sock_inet(6)_mptcp_maxseg_default.
> Those constants represent the MSS supported by MPTCP (== -1 for the
> moment, as long as TCP_MAXSEG is not supported). For the moment, 
> MPTCP doesn't support TCP_MAXSEG socket option, mainly because it 
> has apparently never been requested before, apparently. It should 
> not be difficult to implement it, but is it an important option
> for HAProxy?

It's not that common anymore but 15 years ago when ADSL over PPPoE was
everywhere, it was super frequent to see users not able to upload large
POST data over TCP because their PC connected with a 1500 MTU to their
PPPoE modem/router was seeing its packets rejected due to the extra 8
bytes of PPPoE encapsulation. Many people had by then implemented the
set-mss rule in iptables on the server infrastructure to deal with this,
but high-bandwidth hosting providers couldn't even just place a firewall
in front of haproxy just to deal with that, so in this case it was very
convenient to permit haproxy to simply configure the socket to negotiate
a lower MSS compatible with everyone's setup.

Nowadays with everyone having a triple-play box provided by the ISP,
all this stuff is transparent to the user (when needed at all). However
I've been aware of users doing that as well to fixup trouble caused by
inter-company VPN links. You know, the type of places where everyone
says "it's right on my side, it must be the other side"...

So I guess that nowadays the mss setting mostly serves to fixup VPN
environments, and it's not much likely that we'll see that with MPTCP
soon I think (but I could of course be wrong).

With that said, from an implementation perspective, it would seem right
to make sure that most TCP tunables also work with MPTCP.

Thanks!
Willy

[PATCH 0/2] Add MPTCP to haproxy

[Aperence]

Multipath TCP (MPTCP), standardized in RFC8684 [1], is a TCP extension
that enables a TCP connection to use different paths.

Multipath TCP has been used for several use cases. On smartphones, MPTCP
enables seamless handovers between cellular and Wi-Fi networks while
preserving established connections. This use-case is what pushed Apple
to use MPTCP since 2013 in multiple applications [2]. On dual-stack
hosts, Multipath TCP enables the TCP connection to automatically use the
best performing path, either IPv4 or IPv6. If one path fails, MPTCP
automatically uses the other path.

To benefit from MPTCP, both the client and the server have to support
it. Multipath TCP is a backward-compatible TCP extension that is enabled
by default on recent Linux distributions (Debian, Ubuntu, Redhat, ...).
Multipath TCP is included in the Linux kernel since version 5.6 [3]. To
use it on Linux, an application must explicitly enable it when creating
the socket. No need to change anything else in the application.

These patches are a continuation of the work [4] realized by Dorian 
Craps and Matthieu Baerts.

Changelog:
  - Patch 1:
- Duplicate the mptcp(4|6) protocol structures from their tcp 
equivalent, instead of using a function that creates them by copying
tcp structures
- Add support for MPTCP on the backend side (between the proxy and 
the servers)
  - Patch 2:
- Fix a warning about TCP_MAXSEG not being supported while MPTCP is 
used by declaring new constants sock_inet(6)_mptcp_maxseg_default.
Those constants represent the MSS supported by MPTCP (== -1 for the
moment, as long as TCP_MAXSEG is not supported). For the moment, 
MPTCP doesn't support TCP_MAXSEG socket option, mainly because it 
has apparently never been requested before, apparently. It should 
not be difficult to implement it, but is it an important option
    for HAProxy?

Link: https://www.rfc-editor.org/rfc/rfc8684.html [1]
Link: https://www.tessares.net/apples-mptcp-story-so-far/ [2]
Link: https://www.mptcp.dev [3]
Link: https://www.mail-archive.com/haproxy@formilux.org/msg44871.html [4]

Aperence (2):
  FEATURE: add MPTCP per address support
  BUG/MINOR: fix warning when setting MSS with MPTCP

 doc/configuration.txt   |  21 +++
 examples/mptcp-backend.py   |  22 +++
 examples/mptcp.cfg  |  23 
 include/haproxy/compat.h|   5 ++
 include/haproxy/protocol.h  |   4 +-
 include/haproxy/sock_inet.h |   3 +
 src/backend.c   |  11 +++-
 src/proto_tcp.c | 113 ++--
 src/protocol.c  |   4 +-
 src/sock.c  |   4 +-
 src/sock_inet.c |  28 +
 src/tools.c |  24 +++-
 12 files changed, 250 insertions(+), 12 deletions(-)
 create mode 100644 examples/mptcp-backend.py
 create mode 100644 examples/mptcp.cfg

-- 
2.46.0

[ANNOUNCE] haproxy-3.1-dev6

[Willy Tarreau]

Hi,

HAProxy 3.1-dev6 was released on 2024/08/21. It added 67 new commits
after version 3.1-dev5.

A few minor bugs were fixed, with the most important one probably be the
recent breakage of command pipeling on the master CLI (commands delimited
by semi-colons passed to a worker). Aside this, the focus was mostly set on
moderately sensitive changes this time:

  - the QUIC buffer allocator was replaced. Previously, up to 30 (or even
more if needed, as configured by tune.quic.frontend.conn-tx-buffers.limit)
buffers of 16kB were allocated to send data, regardless of the on-wire
window. This could often result in up to half a megabyte of data allocated
on a connection while distillating data to a slow client. It also wasn't
very efficient in terms of buffer recycling because a full 16kB buffer
could remain attached for just a few hundred bytes of headers. And it
wasn't very easy to experiment with the effects of new congestion control
algorithms. Now this is done diffrently. First, the QUIC mux will allocate
as many buffers as permitted by the congestion control algorithm's send
window, and a limit in bytes is set to limit how large we want a window
to grow. It now becomes easier to tune the maximum per-connection memory
size, knowing that this size will only be allocated if the link quality
to the peer is sufficient to keep these bytes in flight with no loss,
so most of these buffers are quite ephemeral. In addition, there are now
also small buffers (1kB) for use when small data are needed, typically
a short response, or just the headers. This should reduce the memory
usage of the QUIC stack. Since the change is not trivial, there may be
corner cases (we hope not). We're particularly interested in observations
such as increased CPU usage (e.g. maybe more frequent wakeups since working
with a lower latency), or timeouts (none were noticed in tests but we know
that tests != prod). The old tune.quic.frontend.conn-tx-buffers.limit
setting is now ignored and will emit a warning directing to the new one.

  - logs: after some discussion around GitHub issue 2642 regarding the recent
stricter checks for logformat expression, we realized that these checks
are in fact a thing of the past, an era where sample fetch functions
ought not be called from a wrong context at all. But this has been
addressed a bunch of versions ago (2.0 or so) and we don't need to be
strict anymore (and the proof is that previous versions wouldn't report
issues and were working, possibly silently reporting an empty field). So
we decided to relax that painful check and only emit it in diag mode (-dD)
as a hint that something might possibly not work as expected. Anyway, with
multiple log points coming, this situation was not sustainable anymore,
users would start to get crazy trying to create their log formats! This
will normally be backported to 3.0 as well to apease users facing this
problem.

  - log: the "option tcplog" now supports the optional "clf" argument, like
"httplog" and "httpslog", as apparently there are a few users who prefer
to rely on that format for everything. That definitely makes sense IMHO.
There's a pending patch to add the corresponding environment variables
defining the format but I forgot to review it before that release. Will
do when idle time increases again.

  - config: many of the hard-coded global keywords were moved to a
proper keyword list. That's the beginning of a serious long-term
cleanup that is needed anyway to simplify the master process startup
mechanism. In addition, files loaded from stdin are now limited to
about 10MB by default. Almost nobody uses stdin to feed files, and
10MB starts to be quite a serious size. That's an effective way to
avoid pre-loading a log file, a map file or such a huge thing by
accident. The limit could be changed if anyone had concerns about
this.

  - protocols: some super-rare binding failures could be incorrectly
reported with TCP and unix stream sockets. For TCP this would happen
on linux with more than 32k listeners on the same ip:port. The error
output would be freed before being displayed, leaving the user with
something as verbose as "failed to start listeners" or something in
that vein.

  - proto: the internal protocol registration mechanism was slightly updated
to permit to generalize the variants that are sometimes needed (abns,
mptcp etc) that sometimes need to resort to ugly hacks. We should try
again to integrate Tristan's past work on abns as, if my memory serves
me right, the code dealing with the parsing and setting of the addresses
was correct, and the trouble was on the registration part.

For the rest, it's usual CI updates, 

Re: [ANNOUNCE] haproxy-3.1-dev5

[Willy Tarreau]

Hi Ilya,

On Wed, Aug 07, 2024 at 08:30:46PM +0200,  ??? wrote:
> > HAProxy 3.1-dev5 was released on 2024/08/07. It added 88 new commits
> > after version 3.1-dev4.
> >
> > There were quite a bunch of fixes this time, spread over various areas
> > (h2, analysers, jwt, quic, 0-rtt, queues, traces), though nothing exciting
> > at this point.
> >
> 
> I'm not sure about roadmap of aws-lc support in HAProxy, but from my
> observation it reached parity with QuicTLS
> (*) performance of OpenSSL-1.1.1 (in some cases even 20% faster due to
> assembler implementation)
> (*) being actively developed by AWS (derived from BoringSSL)
> (*) all major QUIC features implemented
> (*) no LTS cycles ((

I totally agree. Its performance is even significantly higher than 1.1.1
(we've collected some measurements and are working on an article about
that). And indeed what's missing is LTS. I would really love it if the
QuicTLS and aws-lc teams could join efforts and try to maintain an LTS
version of this high-quality library! All the QUIC community would benefit
from this!

Willy

Re: [ANNOUNCE] haproxy-3.1-dev5

[Илья Шипицин]

ср, 7 авг. 2024 г. в 18:48, Willy Tarreau :

> Hi,
>
> HAProxy 3.1-dev5 was released on 2024/08/07. It added 88 new commits
> after version 3.1-dev4.
>
> There were quite a bunch of fixes this time, spread over various areas
> (h2, analysers, jwt, quic, 0-rtt, queues, traces), though nothing exciting
> at this point.
>

I'm not sure about roadmap of aws-lc support in HAProxy, but from my
observation it reached parity with QuicTLS
(*) performance of OpenSSL-1.1.1 (in some cases even 20% faster due to
assembler implementation)
(*) being actively developed by AWS (derived from BoringSSL)
(*) all major QUIC features implemented
(*) no LTS cycles ((


>
> We've got a report of a user facing higher loads due to one of the new
> safety rules enforcements in the HTTP spec mandating that requests that
> contain both Transfer-Encoding and Content-Length had to work in close
> mode (to avoid smuggling on possible incompatible intermediary HTTP/1.0
> proxies), so we've added an option to relax this rule when the chain is
> trusted.
>
> On the QUIC side, Chacha20 and 0-RTT were fixed when using the aws-lc
> crypto library. A new ruleset "quic-initial" allows to filter packets
> during the QUIC handshake. The currently supported actions are "reject",
> "accept", "dgram-drop" (for a silent drop), and "send-retry" (to force
> a retry when in 0-RTT for example). It can significantly help against
> abuses or simply to enforce source-based filtering so that the client
> cannot even engage in a handshake. The quic traces will now also indicate
> how long a stream spent waiting for flow control, buffers, etc, which
> should help us explain why certain requests appear to be slow. Some
> Cubic-specific info are now also dumped in "show quic".
>
> The traces now permit some sources to follow other ones, so that when
> a source automatically triggers, the followers will automatically be
> enabled as well. One use case is to track the session, allowing to
> watch a communication between the frontend and the backend without
> being disturbed by the rest of the traffic. We've verified that we
> can follow a series of requests from a front QUIC connection to an
> HTTP/2 backend. This will allow us to simplify some captures. Also
> a new meta-source "all" is supported for some "trace" commands, to
> set the sink, the level and the source to follow. This will save a
> lot of debugging commands.
>
> A new pair of sample fetch functions, fs.debug_str() and bs.debug_str()
> can be used to complete the logs with useful debugging info from the
> lower layers (stream ID, flow-control etc). It appears important to
> continue to provide detailed troubleshooting elements because it has
> happened quite a few times since we have muxes that some logs would
> report an error, a timeout or something unusual and that it was a bit
> hard to figure what happened at the lower layers. Obviously with
> protocols like H2 and QUIC we can't tell the whole history but it should
> help quite a bit. For example the stream's pause times mentioned above
> will be there so it will be possible to correlate the request timers
> with some such elements.
>
> A more significant change concerns the loading of configuration files.
> Previously they were opened and parsed on the fly just once. With the
> pending master startup changes, the starting process will need to check
> in the global section if it's supposed to be the master process and stop
> there, and the worker will parse its own config. Since /dev/stdin works
> and is supported, it's not possible to open it twice. Instead it was
> decided that the config is pre-loaded in memory and processed from there
> so that it stays buffered. The config size in memory is not much of a
> concern given the huge amplification factor (40 to 100x) of a config
> represented in memory, so the temporary extra copy of the text-based one
> is small. There could be pitfalls, though and it's always interesting to
> know if you find something that breaks it. We already know that loading
> /dev/zero will make it eat a lot of RAM for example, but we'll rather
> address all corner cases as a whole than each of them individually.
>
> And the rest is as usual, some build fixes, CI updates and doc updates.
>
> Quite honestly, if you're running large configs and/or are streaming your
> configs over SSH to a remote daemon for example, or doing anything fancy,
> your feedback on the config loading changes is really important to those
> working on this. Normally you should not notice a difference at this step.
>
> Please find the usual URLs below :
>Site index   : https://www.

[ANNOUNCE] haproxy-3.1-dev5

[Willy Tarreau]

Hi,

HAProxy 3.1-dev5 was released on 2024/08/07. It added 88 new commits
after version 3.1-dev4.

There were quite a bunch of fixes this time, spread over various areas
(h2, analysers, jwt, quic, 0-rtt, queues, traces), though nothing exciting
at this point.

We've got a report of a user facing higher loads due to one of the new
safety rules enforcements in the HTTP spec mandating that requests that
contain both Transfer-Encoding and Content-Length had to work in close
mode (to avoid smuggling on possible incompatible intermediary HTTP/1.0
proxies), so we've added an option to relax this rule when the chain is
trusted.

On the QUIC side, Chacha20 and 0-RTT were fixed when using the aws-lc
crypto library. A new ruleset "quic-initial" allows to filter packets
during the QUIC handshake. The currently supported actions are "reject",
"accept", "dgram-drop" (for a silent drop), and "send-retry" (to force
a retry when in 0-RTT for example). It can significantly help against
abuses or simply to enforce source-based filtering so that the client
cannot even engage in a handshake. The quic traces will now also indicate
how long a stream spent waiting for flow control, buffers, etc, which
should help us explain why certain requests appear to be slow. Some
Cubic-specific info are now also dumped in "show quic".

The traces now permit some sources to follow other ones, so that when
a source automatically triggers, the followers will automatically be
enabled as well. One use case is to track the session, allowing to
watch a communication between the frontend and the backend without
being disturbed by the rest of the traffic. We've verified that we
can follow a series of requests from a front QUIC connection to an
HTTP/2 backend. This will allow us to simplify some captures. Also
a new meta-source "all" is supported for some "trace" commands, to
set the sink, the level and the source to follow. This will save a
lot of debugging commands.

A new pair of sample fetch functions, fs.debug_str() and bs.debug_str()
can be used to complete the logs with useful debugging info from the
lower layers (stream ID, flow-control etc). It appears important to
continue to provide detailed troubleshooting elements because it has
happened quite a few times since we have muxes that some logs would
report an error, a timeout or something unusual and that it was a bit
hard to figure what happened at the lower layers. Obviously with
protocols like H2 and QUIC we can't tell the whole history but it should
help quite a bit. For example the stream's pause times mentioned above
will be there so it will be possible to correlate the request timers
with some such elements.

A more significant change concerns the loading of configuration files.
Previously they were opened and parsed on the fly just once. With the
pending master startup changes, the starting process will need to check
in the global section if it's supposed to be the master process and stop
there, and the worker will parse its own config. Since /dev/stdin works
and is supported, it's not possible to open it twice. Instead it was
decided that the config is pre-loaded in memory and processed from there
so that it stays buffered. The config size in memory is not much of a
concern given the huge amplification factor (40 to 100x) of a config
represented in memory, so the temporary extra copy of the text-based one
is small. There could be pitfalls, though and it's always interesting to
know if you find something that breaks it. We already know that loading
/dev/zero will make it eat a lot of RAM for example, but we'll rather
address all corner cases as a whole than each of them individually.

And the rest is as usual, some build fixes, CI updates and doc updates.

Quite honestly, if you're running large configs and/or are streaming your
configs over SSH to a remote daemon for example, or doing anything fancy,
your feedback on the config loading changes is really important to those
working on this. Normally you should not notice a difference at this step.

Please find the usual URLs below :
   Site index   : https://www.haproxy.org/
   Documentation: https://docs.haproxy.org/
   Wiki : https://github.com/haproxy/wiki/wiki
   Discourse    : https://discourse.haproxy.org/
   Slack channel: https://slack.haproxy.org/
   Issue tracker: https://github.com/haproxy/haproxy/issues
   Sources  : https://www.haproxy.org/download/3.1/src/
   Git repository   : https://git.haproxy.org/git/haproxy.git/
   Git Web browsing : https://git.haproxy.org/?p=haproxy.git
   Changelog: https://www.haproxy.org/download/3.1/src/CHANGELOG
   Dataplane API: 
https://github.com/haproxytech/dataplaneapi/releases/latest
   Pending bugs : https://www.haproxy.org/l/pending-bugs
   Reviewed bugs: https://www.haproxy.org/l/reviewed-bug

Re: [ANNOUNCE] haproxy-3.0.3

[Willy Tarreau]

On Sun, Aug 04, 2024 at 09:39:25PM +0200, Vincent Bernat wrote:
> 
> On 2024-07-22 21:59, Willy Tarreau wrote:
> 
> > > > HAProxy 3.0.3 was released on 2024/07/11. It added 42 new commits
> > > > after version 3.0.2.
> > > 
> > > I am late releasing this version on haproxy.debian.net. I have issues with
> > > compiling for ARM64. There is a known bug in recent version of QEMU
> > > segfaulting on ARM64 userland emulation. Previously, I was able to just
> > > downgrade to an older version of QEMU, but this does not work anymore. I
> > > suppose there was some changes in recent kernel versions. I'll need to try
> > > again in a VM.
> > 
> > OK, thanks for letting us know!
> > 
> > I could probably propose you an access to one physical machine, but I
> > guess this would not help much because it would likely completely
> > change your build process. Do not hesitate to let us know if there's
> > anything we can do to help you.
> 
> It seems the segfault is random while using user emulation. So, I have just
> to try a few times... Hoping it will be fixed at some point. Until then,
> 3.0.3 packages for Debian/Ubuntu are online.

OK, thank you Vincent!
Willy

Re: [ANNOUNCE] haproxy-3.0.3

[Vincent Bernat]

On 2024-07-22 21:59, Willy Tarreau wrote:


HAProxy 3.0.3 was released on 2024/07/11. It added 42 new commits
after version 3.0.2.


I am late releasing this version on haproxy.debian.net. I have issues with
compiling for ARM64. There is a known bug in recent version of QEMU
segfaulting on ARM64 userland emulation. Previously, I was able to just
downgrade to an older version of QEMU, but this does not work anymore. I
suppose there was some changes in recent kernel versions. I'll need to try
again in a VM.


OK, thanks for letting us know!

I could probably propose you an access to one physical machine, but I
guess this would not help much because it would likely completely
change your build process. Do not hesitate to let us know if there's
anything we can do to help you.


It seems the segfault is random while using user emulation. So, I have 
just to try a few times... Hoping it will be fixed at some point. Until 
then, 3.0.3 packages for Debian/Ubuntu are online.

Re: Some HTTP connections not closing properly on Haproxy 2.8.10

[Christopher Faulet]

Hi,

I finally found some issues. Not sure it is exactly yours. But it is very 
similar. So I'm confident. I pushed 3 new fixes in 3.1-DEV. It may be good to 
check on the 3.0 if this fully fixes all your issues. And eventually, if 
possible, on 2.8 too, with the splicing enabled.


In attachement, the whole series for 3.0 and 2.8: My first 3 patches and the 3 
new ones.


I'm going to be on vacation for 3 weeks. so there's no rush

Many thanks for your help.

--
Christopher Faulet
From db2e751bc8cc4bd8a96d2149c8a59022d15b9709 Mon Sep 17 00:00:00 2001
From: Christopher Faulet 
Date: Mon, 29 Jul 2024 17:48:16 +0200
Subject: [PATCH 1/6] BUG/MEDIUM: stconn: Report error on SC on send if a
 previous SE error was set

When a send on a connection is performed, if a SE error (or a pending error)
was already reported earlier, we leave immediately. No send is performed.
However, we must be sure to report the error at the SC level if necessary.
Indeed, the SE error may have been reported during the zero-copy data
forwarding. So during receive on the opposite side. In that case, we may
have missed the opportunity to report it at the SC level.

The patch must be backported as far as 2.8.

(cherry picked from commit 5dc45445ff18207dbacebf1f777e1f1abcd5065d)
Signed-off-by: Christopher Faulet 
---
 src/stconn.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/src/stconn.c b/src/stconn.c
index 607740305a..536b1a6bbc 100644
--- a/src/stconn.c
+++ b/src/stconn.c
@@ -1590,6 +1590,8 @@ int sc_conn_send(struct stconn *sc)
if (sc->state < SC_ST_CON)
return 0;
BUG_ON(sc_ep_test(sc, SE_FL_EOS|SE_FL_ERROR|SE_FL_ERR_PENDING) 
== (SE_FL_EOS|SE_FL_ERR_PENDING));
+   if (sc_ep_test(sc, SE_FL_ERROR))
+   sc->flags |= SC_FL_ERROR;
return 1;
}
 
-- 
2.45.2

From 0c98078e8a9dedad6e75f8b23c293ebdf85150a6 Mon Sep 17 00:00:00 2001
From: Christopher Faulet 
Date: Tue, 30 Jul 2024 08:41:03 +0200
Subject: [PATCH 2/6] BUG/MEDIUM: mux-pt/mux-h1: Release the pipe on connection
 error on sending path

When data are sent using the kernel splicing, if a connection error
occurred, the pipe must be released. Indeed, in that case, no more data can
be sent and there is no reason to not release the pipe. But it is in fact an
issue for the stream because the channel will appear are not empty. This may
prevent the stream to be released. This happens on 2.8 when a filter is also
attached on it. On 2.9 and upper, it seems there is not issue. But it is
hard to be sure and the current patch remains valid is all cases. On 2.6 and
lower, the code is not the same and, AFAIK, there is no issue.

This patch must be backported to 2.8. However, on 2.8, there is no zero-copy
data forwarding. The patch must be adapted. There is no done_ff/resume_ff
callback functions for muxes. The pipe must released in sc_conn_send() when
an error flag is set on the SE, after the call to snd_pipe callback
function.

(cherry picked from commit 760d26a8625f3af2b6939037a40f19b5f8063be1)
Signed-off-by: Christopher Faulet 
---
 src/mux_h1.c | 8 
 src/mux_pt.c | 8 
 2 files changed, 16 insertions(+)

diff --git a/src/mux_h1.c b/src/mux_h1.c
index a4118dc9b3..075acb62f4 100644
--- a/src/mux_h1.c
+++ b/src/mux_h1.c
@@ -4829,6 +4829,10 @@ static size_t h1_done_ff(struct stconn *sc)
h1c->conn->xprt->subscribe(h1c->conn, 
h1c->conn->xprt_ctx, SUB_RETRY_RECV, &h1c->wait_event);
}
se_fl_set_error(h1s->sd);
+   if (sd->iobuf.pipe) {
+   put_pipe(sd->iobuf.pipe);
+   sd->iobuf.pipe = NULL;
+   }
TRACE_DEVEL("connection error", 
H1_EV_STRM_ERR|H1_EV_H1C_ERR|H1_EV_H1S_ERR, h1c->conn, h1s);
}
 
@@ -5084,6 +5088,10 @@ static int h1_resume_fastfwd(struct stconn *sc, unsigned 
int flags)
h1c->conn->xprt->subscribe(h1c->conn, 
h1c->conn->xprt_ctx, SUB_RETRY_RECV, &h1c->wait_event);
}
se_fl_set_error(h1s->sd);
+   if (h1s->sd->iobuf.pipe) {
+   put_pipe(h1s->sd->iobuf.pipe);
+   h1s->sd->iobuf.pipe = NULL;
+   }
TRACE_DEVEL("connection error", 
H1_EV_STRM_ERR|H1_EV_H1C_ERR|H1_EV_H1S_ERR, h1c->conn, h1s);
}
 
diff --git a/src/mux_pt.c b/src/mux_pt.c
index 6dbbe04cd1..7af06e4962 100644
--- a/src/mux_pt.c
+++ b/src/mux_pt.c
@@ -631,6 +631,10 @@ static size_t mux_pt_done_ff(struct stconn *sc)
if (conn_xprt_read0_pending(conn))
se_fl_set(ctx->sd, SE_FL_EOS);
se_fl_set_error(ctx->sd);
+   if (se->iobuf.pipe) {
+   put_pipe(se->iobuf.pipe);
+   se->iobuf.pipe = NULL;
+   }
TRACE_DEVEL("error on connection", 
PT_EV_TX_DATA|PT_EV_CONN_ERR, conn, sc);
}
 
@@ -746

Re: Some HTTP connections not closing properly on Haproxy 2.8.10

[Christopher Faulet]

Le 30/07/2024 à 17:23, Jens Wahnes a écrit :

Thanks for the confirmation. On 3.0, I was unable to reproduce the
issue. So I'm not surprised.


On version 3.0.3 with splicing turned on, I actually did end up with a
backend connection in state CLOSE_WAIT that is still around after some
hours. But it is different from the other cases I saw (with version
2.8.10). This one on 3.0.3 is an HTTPS connection (likely using HTTP/2)
on the frontend side and the backend connection is HTTP/1. What is also
a bit special is that the HTTP response code is 204, so there is no
"real" data being transmitted.


That's interesting because here, there is no kernel splicing. So my fixes will 
not catch this case. But your are using the bandwidth limitation filter. It may 
be related. I must investigate a bit. It seems unrelated with the previous issue 
however. Because when the splicing is used, no filter can be active.


Could you share your bwlim configuration please ?



So I'm not sure if this one is really related to the other ones or if
it's just a coincidence that I'm seeing this happen as I'm looking for
other connections not being closed properly. :)

The associated `show sess` looks like this (IP addresses slightly altered):

```
0x7fc63caccc00: [30/Jul/2024:14:05:50.626096] id=123594 proto=tcpv4
source=10.80.119.118:53864
flags=0x3384a, conn_retries=0, conn_exp= conn_et=0x000
srv_conn=0x7fc63f0a6000, pend_pos=(nil) waiting=0 epoch=0
frontend=Loadbalancer (id=48 mode=http), listener=https (id=15)
addr=10.210.18.56:443
backend=projekt_piwik_2019 (id=94 mode=http) addr=172.16.240.53:50858
server=counterstrike (id=1) addr=172.16.240.99:2501
task=0x7fc63eb6f260 (state=0x00 nice=400 calls=6 rate=0 exp=
tid=1(1/1) age=2h20m)
txn=0x7fc63ed00320 flags=0x4 meth=3 status=-1 req.st=MSG_DONE
rsp.st=MSG_RPBEFORE req.f=0x4d rsp.f=0x00
scf=0x7fc642deb620 flags=0x00070006 ioto=1m state=CLO
endp=CONN,0x7fc63ebbdf00,0x5043d601 sub=0 rex= wex= rto=?
wto=
  iobuf.flags=0x .pipe=0 .buf=0@(nil)+0/0
h2s=0x7fc63ebbdf00 h2s.id=15 .st=CLO .flg=0x4109 .rxbuf=0@(nil)+0/0
 .sc=0x7fc642deb620(.flg=0x00070006 .app=0x7fc63caccc00)
.sd=0x7fc642d25770(.flg=0x5043d601)
 .subs=(nil)
 h2c=0x7fc642d8f200 h2c.st0=FRH .err=0 .maxid=15 .lastid=-1
.flg=0x1a60600 .nbst=0 .nbsc=1, .glitches=0
 .fctl_cnt=0 .send_cnt=0 .tree_cnt=1 .orph_cnt=0 .sub=0 .dsi=15
.dbuf=0@(nil)+0/0
 .mbuf=[1..1|32],h=[0@(nil)+0/0],t=[0@(nil)+0/0]
.task=0x7fc642c0af60 .exp=
co0=0x7fc63cb2d320 ctrl=tcpv4 xprt=SSL mux=H2 data=STRM
target=LISTENER:0x7fc641a5a400
flags=0x801c0300 fd=237 fd.state=1922 updt=0 fd.tmask=0x2
scb=0x7fc642da50e0 flags=0x1013 ioto=10m state=EST
endp=CONN,0x7fc63ebbb200,0x50404001 sub=1 rex= wex= rto=?
wto=
  iobuf.flags=0x .pipe=0 .buf=0@(nil)+0/0
h1s=0x7fc63ebbb200 h1s.flg=0x94010 .sd.flg=0x50404001
.req.state=MSG_DONE .res.state=MSG_DONE
 .meth=POST status=204 .sd.flg=0x50404001 .sc.flg=0x1013
.sc.app=0x7fc63caccc00
 .subs=0x7fc642da50f8(ev=1 tl=0x7fc642da9c40 tl.calls=4
tl.ctx=0x7fc642da50e0 tl.fct=sc_conn_io_cb)
 h1c=0x7fc63ca5d840 h1c.flg=0x8000 .sub=0 .ibuf=0@(nil)+0/0
.obuf=0@(nil)+0/0 .task=0x7fc63cad2940 .exp=
co1=0x7fc63ebb6820 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM
target=SERVER:0x7fc63f0a6000
flags=0x0300 fd=167 fd.state=11122 updt=0 fd.tmask=0x2
filters={0x7fc642d6fb30="bandwidth limitation filter"}
req=0x7fc63caccc28 (f=0x2084 an=0x48000 tofwd=0 total=993)
an_exp= buf=0x7fc63caccc30 data=0x7fc63cb52280 o=0 p=0
i=16384 size=16384
htx=0x7fc63cb52280 flags=0x10 size=16336 data=1 used=1 wrap=NO
extra=0
res=0x7fc63caccc70 (f=0x80008000 an=0x2000 tofwd=0 total=274)
an_exp= buf=0x7fc63caccc78 data=0x7fc63c6a7dc0 o=274 p=274
i=16110 size=16384
htx=0x7fc63c6a7dc0 flags=0x10 size=16336 data=274 used=11 wrap=NO
extra=0

```


I pushed some patches that should fix your issue. They cannot be applied
as-is on the 2.8. You can use attached patches for the 2.8 if you want
to try. It could help to be sure they properly fix your issue.


Thank you. I applied the patches and compiled a version of 2.8.10 based
on this. However, I'm hesitating to run it just yet, as I am uncertain
if the above example of a "stuck" session in version 3.0.3 is worth
another look. If you would like me to perform any action on that session
to diagnose further, like the "close FD" or anything else, please let me
know.

Otherwise, I'd try the version of 2.8.10 with your patched applied.


Well, I'm annoyed. At first glance both issues are unrelated but symptoms are 
too similar to be a coincidence. So I may have missed something the first time, 
distracted by the splicing. So, for now, it is better to wait a bit before 
testing my fixes. I hope to find the root cause of the issue quickly.


Thanks !

--
Christopher Faulet

Re: Some HTTP connections not closing properly on Haproxy 2.8.10

[Jens Wahnes]

Hi Christopher,

Christopher Faulet wrote:

Le 29/07/2024 à 16:30, Jens Wahnes a écrit :

Christopher Faulet wrote:

Le 29/07/2024 à 09:05, Christopher Faulet a écrit :


Thanks, I will investigate. It is indeed most probably an issue with 
the

splicing, as Willy said. I will try to find the bug on the 2.8 and
figure out if
upper versions are affected too.


I'm able to reproduce the issue by hacking the code, forcing a
connection error by hand. It occurs when an error is reported on the
connection when haproxy tries to send data using kernel splicing. But it
is only an issue when a filter is attached to the applicative stream. I
guess you have enabled the HTTP compression. The response is not
compressed of course, otherwise the kernel splicing would not be used.
But it is still attached to the stream and it has an effect in this 
case.


AFAIK, the older versions are not affected. On newer versions, I don't
really know. There is an issue with my hack, but timeouts are still
active and a true client abort is properly detected. So, I'm inclined to
think there is no issue on these versions. But my fix will probably be
applicable too.

I'm on the fix. I must test when this happens on server side, to be
sure. But it should be fixed soon.



Thank you for the update.

My results so far: Everything is fine on 2.8.10 without splicing.

On 3.0.3 with splicing turned on, I have also not seen any lingering
sessions, but I have only been running version 3.0.3 for some hours now,
so this could still happen. I'll rather let it run for some more time
before drawing conclusions.



Thanks for the confirmation. On 3.0, I was unable to reproduce the 
issue. So I'm not surprised.


On version 3.0.3 with splicing turned on, I actually did end up with a 
backend connection in state CLOSE_WAIT that is still around after some 
hours. But it is different from the other cases I saw (with version 
2.8.10). This one on 3.0.3 is an HTTPS connection (likely using HTTP/2) 
on the frontend side and the backend connection is HTTP/1. What is also 
a bit special is that the HTTP response code is 204, so there is no 
"real" data being transmitted.


So I'm not sure if this one is really related to the other ones or if 
it's just a coincidence that I'm seeing this happen as I'm looking for 
other connections not being closed properly. :)


The associated `show sess` looks like this (IP addresses slightly altered):

```
0x7fc63caccc00: [30/Jul/2024:14:05:50.626096] id=123594 proto=tcpv4 
source=10.80.119.118:53864
  flags=0x3384a, conn_retries=0, conn_exp= conn_et=0x000 
srv_conn=0x7fc63f0a6000, pend_pos=(nil) waiting=0 epoch=0
  frontend=Loadbalancer (id=48 mode=http), listener=https (id=15) 
addr=10.210.18.56:443

  backend=projekt_piwik_2019 (id=94 mode=http) addr=172.16.240.53:50858
  server=counterstrike (id=1) addr=172.16.240.99:2501
  task=0x7fc63eb6f260 (state=0x00 nice=400 calls=6 rate=0 exp= 
tid=1(1/1) age=2h20m)
  txn=0x7fc63ed00320 flags=0x4 meth=3 status=-1 req.st=MSG_DONE 
rsp.st=MSG_RPBEFORE req.f=0x4d rsp.f=0x00
  scf=0x7fc642deb620 flags=0x00070006 ioto=1m state=CLO 
endp=CONN,0x7fc63ebbdf00,0x5043d601 sub=0 rex= wex= rto=? 
wto=

iobuf.flags=0x .pipe=0 .buf=0@(nil)+0/0
  h2s=0x7fc63ebbdf00 h2s.id=15 .st=CLO .flg=0x4109 .rxbuf=0@(nil)+0/0
   .sc=0x7fc642deb620(.flg=0x00070006 .app=0x7fc63caccc00) 
.sd=0x7fc642d25770(.flg=0x5043d601)

   .subs=(nil)
   h2c=0x7fc642d8f200 h2c.st0=FRH .err=0 .maxid=15 .lastid=-1 
.flg=0x1a60600 .nbst=0 .nbsc=1, .glitches=0
   .fctl_cnt=0 .send_cnt=0 .tree_cnt=1 .orph_cnt=0 .sub=0 .dsi=15 
.dbuf=0@(nil)+0/0
   .mbuf=[1..1|32],h=[0@(nil)+0/0],t=[0@(nil)+0/0] 
.task=0x7fc642c0af60 .exp=
  co0=0x7fc63cb2d320 ctrl=tcpv4 xprt=SSL mux=H2 data=STRM 
target=LISTENER:0x7fc641a5a400

  flags=0x801c0300 fd=237 fd.state=1922 updt=0 fd.tmask=0x2
  scb=0x7fc642da50e0 flags=0x1013 ioto=10m state=EST 
endp=CONN,0x7fc63ebbb200,0x50404001 sub=1 rex= wex= rto=? 
wto=

iobuf.flags=0x .pipe=0 .buf=0@(nil)+0/0
  h1s=0x7fc63ebbb200 h1s.flg=0x94010 .sd.flg=0x50404001 
.req.state=MSG_DONE .res.state=MSG_DONE
   .meth=POST status=204 .sd.flg=0x50404001 .sc.flg=0x1013 
.sc.app=0x7fc63caccc00
   .subs=0x7fc642da50f8(ev=1 tl=0x7fc642da9c40 tl.calls=4 
tl.ctx=0x7fc642da50e0 tl.fct=sc_conn_io_cb)
   h1c=0x7fc63ca5d840 h1c.flg=0x8000 .sub=0 .ibuf=0@(nil)+0/0 
.obuf=0@(nil)+0/0 .task=0x7fc63cad2940 .exp=
  co1=0x7fc63ebb6820 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM 
target=SERVER:0x7fc63f0a6000

  flags=0x0300 fd=167 fd.state=11122 updt=0 fd.tmask=0x2
  filters={0x7fc642d6fb30="bandwidth limitation filter"}
  req=0x7fc63caccc28 (f=0x2084 an=0x48000 tofwd=0 total=993)
  an_exp= buf=0x7fc63caccc30 data=0x7fc63cb52280 o=0 p=0 
i=16384 size=16384
  htx=0x7fc63cb52280 flags=0x10 size=16336 data=1 used=1 wrap=NO 
extra=0

  res=0x7fc63caccc70 (f=0x8000

Re: Some HTTP connections not closing properly on Haproxy 2.8.10

[Christopher Faulet]

Le 29/07/2024 à 16:30, Jens Wahnes a écrit :

Christopher Faulet wrote:

Le 29/07/2024 à 09:05, Christopher Faulet a écrit :


Thanks, I will investigate. It is indeed most probably an issue with the
splicing, as Willy said. I will try to find the bug on the 2.8 and
figure out if
upper versions are affected too.


I'm able to reproduce the issue by hacking the code, forcing a
connection error by hand. It occurs when an error is reported on the
connection when haproxy tries to send data using kernel splicing. But it
is only an issue when a filter is attached to the applicative stream. I
guess you have enabled the HTTP compression. The response is not
compressed of course, otherwise the kernel splicing would not be used.
But it is still attached to the stream and it has an effect in this case.

AFAIK, the older versions are not affected. On newer versions, I don't
really know. There is an issue with my hack, but timeouts are still
active and a true client abort is properly detected. So, I'm inclined to
think there is no issue on these versions. But my fix will probably be
applicable too.

I'm on the fix. I must test when this happens on server side, to be
sure. But it should be fixed soon.



Thank you for the update.

My results so far: Everything is fine on 2.8.10 without splicing.

On 3.0.3 with splicing turned on, I have also not seen any lingering
sessions, but I have only been running version 3.0.3 for some hours now,
so this could still happen. I'll rather let it run for some more time
before drawing conclusions.



Thanks for the confirmation. On 3.0, I was unable to reproduce the issue. So I'm 
not surprised.


I pushed some patches that should fix your issue. They cannot be applied as-is 
on the 2.8. You can use attached patches for the 2.8 if you want to try. It 
could help to be sure they properly fix your issue.


--
Christopher Faulet
From 681f19beaaf95872f7e0a5e1442546571268472c Mon Sep 17 00:00:00 2001
From: Christopher Faulet 
Date: Mon, 29 Jul 2024 17:48:16 +0200
Subject: [PATCH 1/3] BUG/MEDIUM: stconn: Report error on SC on send if a
 previous SE error was set

When a send on a connection is performed, if a SE error (or a pending error)
was already reported earlier, we leave immediately. No send is performed.
However, we must be sure to report the error at the SC level if necessary.
Indeed, the SE error may have been reported during the zero-copy data
forwarding. So during receive on the opposite side. In that case, we may
have missed the opportunity to report it at the SC level.

The patch must be backported as far as 2.8.
---
 src/stconn.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/src/stconn.c b/src/stconn.c
index 20a1d9b0b2..2df175e7a4 100644
--- a/src/stconn.c
+++ b/src/stconn.c
@@ -1586,6 +1586,8 @@ static int sc_conn_send(struct stconn *sc)
 		if (sc->state < SC_ST_CON)
 			return 0;
 		BUG_ON(sc_ep_test(sc, SE_FL_EOS|SE_FL_ERROR|SE_FL_ERR_PENDING) == (SE_FL_EOS|SE_FL_ERR_PENDING));
+		if (sc_ep_test(sc, SE_FL_ERROR))
+			sc->flags |= SC_FL_ERROR;
 		return 1;
 	}
 
-- 
2.45.2

From 7e6bf5ea450b034bc090fd662fd5057f428c0f63 Mon Sep 17 00:00:00 2001
From: Christopher Faulet 
Date: Tue, 30 Jul 2024 08:41:03 +0200
Subject: [PATCH 2/3] BUG/MEDIUM: mux-pt/mux-h1: Release the pipe on connection
 error on sending path

When data are sent using the kernel splicing, if a connection error
occurred, the pipe must be released. Indeed, in that case, no more data can
be sent and there is no reason to not release the pipe. But it is in fact an
issue for the stream because the channel will appear are not empty. This may
prevent the stream to be released. This happens on 2.8 when a filter is also
attached on it. On 2.9 and upper, it seems there is not issue. But it is
hard to be sure and the current patch remains valid is all cases. On 2.6 and
lower, the code is not the same and, AFAIK, there is no issue.

This patch must be backported to 2.8. However, on 2.8, there is no zero-copy
data forwarding. The patch must be adapted. There is no done_ff/resume_ff
callback functions for muxes. The pipe must released in sc_conn_send() when
an error flag is set on the SE, after the call to snd_pipe callback
function.
---
 src/stconn.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/stconn.c b/src/stconn.c
index 2df175e7a4..ebe05bc585 100644
--- a/src/stconn.c
+++ b/src/stconn.c
@@ -1608,7 +1608,7 @@ static int sc_conn_send(struct stconn *sc)
 		if (ret > 0)
 			did_send = 1;
 
-		if (!oc->pipe->data) {
+		if (!oc->pipe->data || sc_ep_test(sc, SE_FL_ERROR | SE_FL_ERR_PENDING)) {
 			put_pipe(oc->pipe);
 			oc->pipe = NULL;
 		}
-- 
2.45.2

From 13e3f3b91a1f7f7da7d8eaad371d501b7c76530f Mon Sep 17 00:00:00 2001
From: Christopher Faulet 
Date: Tue, 30 Jul 2024 09:28:23 +0200
Subject: [PATCH 3/3] BUG/MINOR: stconn: Request to send something to be woken
 up when the pipe is full

After a receive via the kernel splicing,

Re: Some HTTP connections not closing properly on Haproxy 2.8.10

[Jens Wahnes]

Christopher Faulet wrote:

Le 29/07/2024 à 09:05, Christopher Faulet a écrit :


Thanks, I will investigate. It is indeed most probably an issue with the
splicing, as Willy said. I will try to find the bug on the 2.8 and 
figure out if

upper versions are affected too.


I'm able to reproduce the issue by hacking the code, forcing a 
connection error by hand. It occurs when an error is reported on the 
connection when haproxy tries to send data using kernel splicing. But it 
is only an issue when a filter is attached to the applicative stream. I 
guess you have enabled the HTTP compression. The response is not 
compressed of course, otherwise the kernel splicing would not be used. 
But it is still attached to the stream and it has an effect in this case.


AFAIK, the older versions are not affected. On newer versions, I don't 
really know. There is an issue with my hack, but timeouts are still 
active and a true client abort is properly detected. So, I'm inclined to 
think there is no issue on these versions. But my fix will probably be 
applicable too.


I'm on the fix. I must test when this happens on server side, to be 
sure. But it should be fixed soon.




Thank you for the update.

My results so far: Everything is fine on 2.8.10 without splicing.

On 3.0.3 with splicing turned on, I have also not seen any lingering 
sessions, but I have only been running version 3.0.3 for some hours now, 
so this could still happen. I'll rather let it run for some more time 
before drawing conclusions.


Thanks,
Jens

Re: Some HTTP connections not closing properly on Haproxy 2.8.10

[Christopher Faulet]

Le 29/07/2024 à 09:05, Christopher Faulet a écrit :


Thanks, I will investigate. It is indeed most probably an issue with the
splicing, as Willy said. I will try to find the bug on the 2.8 and figure out if
upper versions are affected too.


I'm able to reproduce the issue by hacking the code, forcing a connection error 
by hand. It occurs when an error is reported on the connection when haproxy 
tries to send data using kernel splicing. But it is only an issue when a filter 
is attached to the applicative stream. I guess you have enabled the HTTP 
compression. The response is not compressed of course, otherwise the kernel 
splicing would not be used. But it is still attached to the stream and it has an 
effect in this case.


AFAIK, the older versions are not affected. On newer versions, I don't really 
know. There is an issue with my hack, but timeouts are still active and a true 
client abort is properly detected. So, I'm inclined to think there is no issue 
on these versions. But my fix will probably be applicable too.


I'm on the fix. I must test when this happens on server side, to be sure. But it 
should be fixed soon.


--
Christopher Faulet

Re: Some HTTP connections not closing properly on Haproxy 2.8.10

[Christopher Faulet]

Le 26/07/2024 à 19:51, Jens Wahnes a écrit :

Hi everyone,

I'm trying to move from Haproxy 2.4 to 2.8 and encountered some trouble
with 2.8 that did not occur on 2.4.
Specifically, this seems to concern plain HTTP connections only, i.e.
non-HTTPS traffic. I have not seen an example of this happening with
HTTPS connections. However, it could also be that it is specific to
HTTP/1 as opposed to HTTP/2 (we do not have any unencrypted HTTP/2
connections).

Some of the backend connections are not properly closed with Haproxy 2.8
and keep lingering for hours or days. That is, the client connection is
gone, but the backend connection is still there in state CLOSE_WAIT.
This in turn causes trouble with seamless reloads: the old Haproxy
processes never terminate and as several newer ones pile up, they take
up memory.

I have not experienced this with Haproxy 2.4.27. On 2.4.27, these
backend TCP connections were properly closed. It only began to be a
problem with never versions like 2.8.10.

To give an example, a connection stuck in CLOSE_WAIT may look this in
`ss` output:

```
CLOSE-WAIT 0  0  172.16.240.53:48528
  172.16.240.86:2000
users:(("haproxy",pid=1099145,fd=239))
```

In Haproxy's own `show sess`, the above example would look like this:

```
0x7efc6b7b5400: [26/Jul/2024:18:24:50.308214] id=237988 proto=tcpv4
source=10.22.113.130:50205
flags=0x53c0a, conn_retries=0, conn_exp= conn_et=0x000
srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0
frontend=Loadbalancer (id=48 mode=http), listener=http (id=5)
addr=10.210.18.56:80
backend=_std_ (id=3 mode=http) addr=172.16.240.53:48528
server=hermes (id=1) addr=172.16.240.86:2000
task=0x7efc72205ec0 (state=0x00 nice=0 calls=6 rate=0 exp=
tid=1(1/1) age=17m41s)
txn=0x7efc6bf72aa0 flags=0x0 meth=1 status=403 req.st=MSG_CLOSED
rsp.st=MSG_CLOSING req.f=0x44 rsp.f=0x0d
scf=0x7efc7233e8a0 flags=0x00030406 state=CLO
endp=CONN,0x7efc6d080ec0,0x04014a01 sub=0 rex= wex=
h1s=0x7efc6d080ec0 h1s.flg=0x104010 .sd.flg=0x4014a01
.req.state=MSG_DONE .res.state=MSG_DATA
.meth=GET status=403 .sd.flg=0x04014a01 .sc.flg=0x00030406
.sc.app=0x7efc6b7b5400 .subs=(nil)
h1c=0x7efc6d5f5d40 h1c.flg=0x20d00 .sub=0 .ibuf=0@(nil)+0/0
.obuf=0@(nil)+0/0 .task=0x7efc6d004360 .exp=
co0=0x7efc7230f140 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM
target=LISTENER:0x7efc712d5e00
flags=0x801c0300 fd=194 fd.state=1c22 updt=0 fd.tmask=0x2
scb=0x7efc6b7c85e0 flags=0x0003501b state=CLO
endp=CONN,0x7efc6d095bc0,0x05004a01 sub=0 rex= wex=
h1s=0x7efc6d095bc0 h1s.flg=0x4010 .sd.flg=0x5004a01
.req.state=MSG_DONE .res.state=MSG_DONE
.meth=GET status=403 .sd.flg=0x05004a01 .sc.flg=0x0003501b
.sc.app=0x7efc6b7b5400 .subs=(nil)
h1c=0x7efc6d087280 h1c.flg=0x8100 .sub=0 .ibuf=0@(nil)+0/0
.obuf=0@(nil)+0/0 .task=0x7efc6d052500 .exp=
co1=0x7efc723ede00 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM
target=SERVER:0x7efc71e48000
flags=0x00040300 fd=239 fd.state=1122 updt=0 fd.tmask=0x2
req=0x7efc6b7b5420 (f=0x848000 an=0x4 pipe=0 tofwd=0 total=113)
an_exp= buf=0x7efc6b7b5428 data=(nil) o=0 p=0 i=0 size=0
htx=0x8c65c0 flags=0x0 size=0 data=0 used=0 wrap=NO extra=0
res=0x7efc6b7b5470 (f=0xa024 an=0x2400 pipe=2310 tofwd=0
total=25601)
an_exp= buf=0x7efc6b7b5478 data=(nil) o=0 p=0 i=0 size=0
htx=0x8c65c0 flags=0x0 size=0 data=0 used=0 wrap=NO extra=0
```

Issuing a `shutdown session` command does not resolve the issue (e.g.
`shutdown session 0x7efc6b7b5400` in this example). So far, killing the
whole process has been the only way for me to get rid of these leftover
connections.

So my question is: does this ring a bell for anyone? Maybe something
that has been fixed in newer versions (like 3.0)?



Thanks, I will investigate. It is indeed most probably an issue with the 
splicing, as Willy said. I will try to find the bug on the 2.8 and figure out if 
upper versions are affected too.


--
Christopher Faulet

Re: Some HTTP connections not closing properly on Haproxy 2.8.10

[Jens Wahnes]

Hi Willy,

thanks so much for looking into this. I certainly didn't expect an 
answer over the weekend and I really appreciate your taking time to look 
into this. Please find my comments below.


Willy Tarreau wrote:

On Fri, Jul 26, 2024 at 07:51:47PM +0200, Jens Wahnes wrote:

Some of the backend connections are not properly closed with Haproxy 2.8 and
keep lingering for hours or days. That is, the client connection is gone,
but the backend connection is still there in state CLOSE_WAIT. This in turn
causes trouble with seamless reloads: the old Haproxy processes never
terminate and as several newer ones pile up, they take up memory.

To give an example, a connection stuck in CLOSE_WAIT may look this in `ss`
output:

```
CLOSE-WAIT 0  0  172.16.240.53:48528
172.16.240.86:2000 users:(("haproxy",pid=1099145,fd=239))
```

In Haproxy's own `show sess`, the above example would look like this:

```
0x7efc6b7b5400: [26/Jul/2024:18:24:50.308214] id=237988 proto=tcpv4
source=10.22.113.130:50205
   flags=0x53c0a, conn_retries=0, conn_exp= conn_et=0x000
srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0
   frontend=Loadbalancer (id=48 mode=http), listener=http (id=5)
addr=10.210.18.56:80
   backend=_std_ (id=3 mode=http) addr=172.16.240.53:48528
   server=hermes (id=1) addr=172.16.240.86:2000
   task=0x7efc72205ec0 (state=0x00 nice=0 calls=6 rate=0 exp=
tid=1(1/1) age=17m41s)
   txn=0x7efc6bf72aa0 flags=0x0 meth=1 status=403 req.st=MSG_CLOSED
rsp.st=MSG_CLOSING req.f=0x44 rsp.f=0x0d
   scf=0x7efc7233e8a0 flags=0x00030406 state=CLO
endp=CONN,0x7efc6d080ec0,0x04014a01 sub=0 rex= wex=
   h1s=0x7efc6d080ec0 h1s.flg=0x104010 .sd.flg=0x4014a01
.req.state=MSG_DONE .res.state=MSG_DATA
   .meth=GET status=403 .sd.flg=0x04014a01 .sc.flg=0x00030406
.sc.app=0x7efc6b7b5400 .subs=(nil)
   h1c=0x7efc6d5f5d40 h1c.flg=0x20d00 .sub=0 .ibuf=0@(nil)+0/0
.obuf=0@(nil)+0/0 .task=0x7efc6d004360 .exp=
   co0=0x7efc7230f140 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM
target=LISTENER:0x7efc712d5e00
   flags=0x801c0300 fd=194 fd.state=1c22 updt=0 fd.tmask=0x2
   scb=0x7efc6b7c85e0 flags=0x0003501b state=CLO
endp=CONN,0x7efc6d095bc0,0x05004a01 sub=0 rex= wex=
   h1s=0x7efc6d095bc0 h1s.flg=0x4010 .sd.flg=0x5004a01
.req.state=MSG_DONE .res.state=MSG_DONE
   .meth=GET status=403 .sd.flg=0x05004a01 .sc.flg=0x0003501b
.sc.app=0x7efc6b7b5400 .subs=(nil)
   h1c=0x7efc6d087280 h1c.flg=0x8100 .sub=0 .ibuf=0@(nil)+0/0
.obuf=0@(nil)+0/0 .task=0x7efc6d052500 .exp=
   co1=0x7efc723ede00 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM
target=SERVER:0x7efc71e48000
   flags=0x00040300 fd=239 fd.state=1122 updt=0 fd.tmask=0x2
   req=0x7efc6b7b5420 (f=0x848000 an=0x4 pipe=0 tofwd=0 total=113)
   an_exp= buf=0x7efc6b7b5428 data=(nil) o=0 p=0 i=0 size=0
   htx=0x8c65c0 flags=0x0 size=0 data=0 used=0 wrap=NO extra=0
   res=0x7efc6b7b5470 (f=0xa024 an=0x2400 pipe=2310 tofwd=0
total=25601)
   an_exp= buf=0x7efc6b7b5478 data=(nil) o=0 p=0 i=0 size=0
   htx=0x8c65c0 flags=0x0 size=0 data=0 used=0 wrap=NO extra=0
```

Issuing a `shutdown session` command does not resolve the issue (e.g.
`shutdown session 0x7efc6b7b5400` in this example). So far, killing the
whole process has been the only way for me to get rid of these leftover
connections.


First, thanks a lot for all the details. I'm seeing a non-zero "pipe"
value above, indicating you're using splicing between the two sides.
Could you please comment whatever "option splice*" you have in your
config so that we can confirm whether it's related or not ?


Yes, there are splice options in the frontend configuration:

option splice-response
option splice-request


However I'm seeing a few post-2.9 changes related to splicing, so it may
be possible that it has accidently been fixed. As such it would be very
useful if you could test 2.9 or 3.0, as this could help us figure whether
it is only 2.8-specific or affects any version starting from 2.8.


Thanks for clarifying.

For now, I'm running the same config, just without "splice-response" and 
without "splice-request", with 2.8.10 to see if the problem still 
appears. If there are no new sockets in CLOSE_WAIT state coming up with 
this setup, it should narrow things down to being a issue that has to do 
with splicing.


So if that proves successful, I'll try version 3.0.3 with splicing 
enabled, probably next week.



Also, if you want to experiment something regarding the possibilities to
close such FDs, you can spot the FD number after the second "fd==" in
"show sess" (backend side), and forcefully close it by issuing:

   expert-mode on; debug dev clode XXX

where XXX is the FD number. Then issue the "show sess" again on the
faulty stream to see if it's gone or if it changed. If still there, its
new flags can also help figure the problem. Just be aware that forcef

Re: Some HTTP connections not closing properly on Haproxy 2.8.10

[Willy Tarreau]

Hi Jens,

On Fri, Jul 26, 2024 at 07:51:47PM +0200, Jens Wahnes wrote:
> Hi everyone,
> 
> I'm trying to move from Haproxy 2.4 to 2.8 and encountered some trouble with
> 2.8 that did not occur on 2.4.
> Specifically, this seems to concern plain HTTP connections only, i.e.
> non-HTTPS traffic. I have not seen an example of this happening with HTTPS
> connections. However, it could also be that it is specific to HTTP/1 as
> opposed to HTTP/2 (we do not have any unencrypted HTTP/2 connections).
> 
> Some of the backend connections are not properly closed with Haproxy 2.8 and
> keep lingering for hours or days. That is, the client connection is gone,
> but the backend connection is still there in state CLOSE_WAIT. This in turn
> causes trouble with seamless reloads: the old Haproxy processes never
> terminate and as several newer ones pile up, they take up memory.
> 
> I have not experienced this with Haproxy 2.4.27. On 2.4.27, these backend
> TCP connections were properly closed. It only began to be a problem with
> never versions like 2.8.10.
> 
> To give an example, a connection stuck in CLOSE_WAIT may look this in `ss`
> output:
> 
> ```
> CLOSE-WAIT 0  0  172.16.240.53:48528
> 172.16.240.86:2000 users:(("haproxy",pid=1099145,fd=239))
> ```
> 
> In Haproxy's own `show sess`, the above example would look like this:
> 
> ```
> 0x7efc6b7b5400: [26/Jul/2024:18:24:50.308214] id=237988 proto=tcpv4
> source=10.22.113.130:50205
>   flags=0x53c0a, conn_retries=0, conn_exp= conn_et=0x000
> srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0
>   frontend=Loadbalancer (id=48 mode=http), listener=http (id=5)
> addr=10.210.18.56:80
>   backend=_std_ (id=3 mode=http) addr=172.16.240.53:48528
>   server=hermes (id=1) addr=172.16.240.86:2000
>   task=0x7efc72205ec0 (state=0x00 nice=0 calls=6 rate=0 exp=
> tid=1(1/1) age=17m41s)
>   txn=0x7efc6bf72aa0 flags=0x0 meth=1 status=403 req.st=MSG_CLOSED
> rsp.st=MSG_CLOSING req.f=0x44 rsp.f=0x0d
>   scf=0x7efc7233e8a0 flags=0x00030406 state=CLO
> endp=CONN,0x7efc6d080ec0,0x04014a01 sub=0 rex= wex=
>   h1s=0x7efc6d080ec0 h1s.flg=0x104010 .sd.flg=0x4014a01
> .req.state=MSG_DONE .res.state=MSG_DATA
>   .meth=GET status=403 .sd.flg=0x04014a01 .sc.flg=0x00030406
> .sc.app=0x7efc6b7b5400 .subs=(nil)
>   h1c=0x7efc6d5f5d40 h1c.flg=0x20d00 .sub=0 .ibuf=0@(nil)+0/0
> .obuf=0@(nil)+0/0 .task=0x7efc6d004360 .exp=
>   co0=0x7efc7230f140 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM
> target=LISTENER:0x7efc712d5e00
>   flags=0x801c0300 fd=194 fd.state=1c22 updt=0 fd.tmask=0x2
>   scb=0x7efc6b7c85e0 flags=0x0003501b state=CLO
> endp=CONN,0x7efc6d095bc0,0x05004a01 sub=0 rex= wex=
>   h1s=0x7efc6d095bc0 h1s.flg=0x4010 .sd.flg=0x5004a01
> .req.state=MSG_DONE .res.state=MSG_DONE
>   .meth=GET status=403 .sd.flg=0x05004a01 .sc.flg=0x0003501b
> .sc.app=0x7efc6b7b5400 .subs=(nil)
>   h1c=0x7efc6d087280 h1c.flg=0x8100 .sub=0 .ibuf=0@(nil)+0/0
> .obuf=0@(nil)+0/0 .task=0x7efc6d052500 .exp=
>   co1=0x7efc723ede00 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM
> target=SERVER:0x7efc71e48000
>   flags=0x00040300 fd=239 fd.state=1122 updt=0 fd.tmask=0x2
>   req=0x7efc6b7b5420 (f=0x848000 an=0x4 pipe=0 tofwd=0 total=113)
>   an_exp= buf=0x7efc6b7b5428 data=(nil) o=0 p=0 i=0 size=0
>   htx=0x8c65c0 flags=0x0 size=0 data=0 used=0 wrap=NO extra=0
>   res=0x7efc6b7b5470 (f=0xa024 an=0x2400 pipe=2310 tofwd=0
> total=25601)
>   an_exp= buf=0x7efc6b7b5478 data=(nil) o=0 p=0 i=0 size=0
>   htx=0x8c65c0 flags=0x0 size=0 data=0 used=0 wrap=NO extra=0
> ```
> 
> Issuing a `shutdown session` command does not resolve the issue (e.g.
> `shutdown session 0x7efc6b7b5400` in this example). So far, killing the
> whole process has been the only way for me to get rid of these leftover
> connections.

First, thanks a lot for all the details. I'm seeing a non-zero "pipe"
value above, indicating you're using splicing between the two sides.
Could you please comment whatever "option splice*" you have in your
config so that we can confirm whether it's related or not ?

> So my question is: does this ring a bell for anyone? Maybe something that
> has been fixed in newer versions (like 3.0)?

No, it really doesn't ring any bell right now. In any case if such a bug
had been identified, the fix would have been backported to affected
versions. I'm not seeing anything related post-2.8 either.

However I'm seeing a few post-2.9 changes related to splicing, so it may
be possible that it has accidently been fixed. As such it would be very
useful if you could test 2.9 or 3.0, as this could help us figure whether
it is only 2.8-specific or affects any version st

Some HTTP connections not closing properly on Haproxy 2.8.10

[Jens Wahnes]

Hi everyone,

I'm trying to move from Haproxy 2.4 to 2.8 and encountered some trouble 
with 2.8 that did not occur on 2.4.
Specifically, this seems to concern plain HTTP connections only, i.e. 
non-HTTPS traffic. I have not seen an example of this happening with 
HTTPS connections. However, it could also be that it is specific to 
HTTP/1 as opposed to HTTP/2 (we do not have any unencrypted HTTP/2 
connections).


Some of the backend connections are not properly closed with Haproxy 2.8 
and keep lingering for hours or days. That is, the client connection is 
gone, but the backend connection is still there in state CLOSE_WAIT. 
This in turn causes trouble with seamless reloads: the old Haproxy 
processes never terminate and as several newer ones pile up, they take 
up memory.


I have not experienced this with Haproxy 2.4.27. On 2.4.27, these 
backend TCP connections were properly closed. It only began to be a 
problem with never versions like 2.8.10.


To give an example, a connection stuck in CLOSE_WAIT may look this in 
`ss` output:


```
CLOSE-WAIT 0  0  172.16.240.53:48528 
172.16.240.86:2000 
users:(("haproxy",pid=1099145,fd=239))

```

In Haproxy's own `show sess`, the above example would look like this:

```
0x7efc6b7b5400: [26/Jul/2024:18:24:50.308214] id=237988 proto=tcpv4 
source=10.22.113.130:50205
  flags=0x53c0a, conn_retries=0, conn_exp= conn_et=0x000 
srv_conn=(nil), pend_pos=(nil) waiting=0 epoch=0
  frontend=Loadbalancer (id=48 mode=http), listener=http (id=5) 
addr=10.210.18.56:80

  backend=_std_ (id=3 mode=http) addr=172.16.240.53:48528
  server=hermes (id=1) addr=172.16.240.86:2000
  task=0x7efc72205ec0 (state=0x00 nice=0 calls=6 rate=0 exp= 
tid=1(1/1) age=17m41s)
  txn=0x7efc6bf72aa0 flags=0x0 meth=1 status=403 req.st=MSG_CLOSED 
rsp.st=MSG_CLOSING req.f=0x44 rsp.f=0x0d
  scf=0x7efc7233e8a0 flags=0x00030406 state=CLO 
endp=CONN,0x7efc6d080ec0,0x04014a01 sub=0 rex= wex=
  h1s=0x7efc6d080ec0 h1s.flg=0x104010 .sd.flg=0x4014a01 
.req.state=MSG_DONE .res.state=MSG_DATA
  .meth=GET status=403 .sd.flg=0x04014a01 .sc.flg=0x00030406 
.sc.app=0x7efc6b7b5400 .subs=(nil)
  h1c=0x7efc6d5f5d40 h1c.flg=0x20d00 .sub=0 .ibuf=0@(nil)+0/0 
.obuf=0@(nil)+0/0 .task=0x7efc6d004360 .exp=
  co0=0x7efc7230f140 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM 
target=LISTENER:0x7efc712d5e00

  flags=0x801c0300 fd=194 fd.state=1c22 updt=0 fd.tmask=0x2
  scb=0x7efc6b7c85e0 flags=0x0003501b state=CLO 
endp=CONN,0x7efc6d095bc0,0x05004a01 sub=0 rex= wex=
  h1s=0x7efc6d095bc0 h1s.flg=0x4010 .sd.flg=0x5004a01 
.req.state=MSG_DONE .res.state=MSG_DONE
  .meth=GET status=403 .sd.flg=0x05004a01 .sc.flg=0x0003501b 
.sc.app=0x7efc6b7b5400 .subs=(nil)
  h1c=0x7efc6d087280 h1c.flg=0x8100 .sub=0 .ibuf=0@(nil)+0/0 
.obuf=0@(nil)+0/0 .task=0x7efc6d052500 .exp=
  co1=0x7efc723ede00 ctrl=tcpv4 xprt=RAW mux=H1 data=STRM 
target=SERVER:0x7efc71e48000

  flags=0x00040300 fd=239 fd.state=1122 updt=0 fd.tmask=0x2
  req=0x7efc6b7b5420 (f=0x848000 an=0x4 pipe=0 tofwd=0 total=113)
  an_exp= buf=0x7efc6b7b5428 data=(nil) o=0 p=0 i=0 size=0
  htx=0x8c65c0 flags=0x0 size=0 data=0 used=0 wrap=NO extra=0
  res=0x7efc6b7b5470 (f=0xa024 an=0x2400 pipe=2310 tofwd=0 
total=25601)

  an_exp= buf=0x7efc6b7b5478 data=(nil) o=0 p=0 i=0 size=0
  htx=0x8c65c0 flags=0x0 size=0 data=0 used=0 wrap=NO extra=0
```

Issuing a `shutdown session` command does not resolve the issue (e.g. 
`shutdown session 0x7efc6b7b5400` in this example). So far, killing the 
whole process has been the only way for me to get rid of these leftover 
connections.


So my question is: does this ring a bell for anyone? Maybe something 
that has been fixed in newer versions (like 3.0)?


Thanks,
Jens

Re: [ANNOUNCE] haproxy-3.1-dev2

[Aleksandar Lazic]

On 2024-07-26 (Fr.) 10:49, Willy Tarreau wrote:

On Fri, Jul 26, 2024 at 10:40:46AM +0200,  ??? wrote:

next point: 0-RTT


Sorry for the stupid questions but as I don't maintain any QUIC based apps
I'm
curious if this really makes any differences compared to "normal" requests.

How often is this really used in the real world setup?
Have anybody seen any real benefit for users out there with 0-RTT?



very beneficial
Introducing Zero Round Trip Time Resumption (0-RTT) (cloudflare.com)



When we were wondering whether or not to adopt the compat layer for
openssl that was suggested by NGINX and that does not support 0-RTT,
we asked the IETF QUIC WG what their opinion on the subject was, and
it was unanimous: "don't do that!". They considered that exposing QUIC
without 0-RTT would hurt the protocol's image because it's expected to
be used a lot. I think it can vary depending on whether users visit a
site often or not (e.g multiple times a day or once a day).

We know that on the OpenSSL front it will remain limited. But for libs
that don't need hacks, it's definitely important that we try to support
the protocol fully.

Willy


Thank you both for the answers.

Regards
Alex

Re: [ANNOUNCE] haproxy-3.1-dev2

[Willy Tarreau]

On Fri, Jul 26, 2024 at 10:40:46AM +0200,  ??? wrote:
> > > next point: 0-RTT
> >
> > Sorry for the stupid questions but as I don't maintain any QUIC based apps
> > I'm
> > curious if this really makes any differences compared to "normal" requests.
> >
> > How often is this really used in the real world setup?
> > Have anybody seen any real benefit for users out there with 0-RTT?
> >
> 
> very beneficial
> Introducing Zero Round Trip Time Resumption (0-RTT) (cloudflare.com)
> 

When we were wondering whether or not to adopt the compat layer for
openssl that was suggested by NGINX and that does not support 0-RTT,
we asked the IETF QUIC WG what their opinion on the subject was, and
it was unanimous: "don't do that!". They considered that exposing QUIC
without 0-RTT would hurt the protocol's image because it's expected to
be used a lot. I think it can vary depending on whether users visit a
site often or not (e.g multiple times a day or once a day).

We know that on the OpenSSL front it will remain limited. But for libs
that don't need hacks, it's definitely important that we try to support
the protocol fully.

Willy

Re: [ANNOUNCE] haproxy-3.1-dev2

[Илья Шипицин]

пт, 26 июл. 2024 г. в 09:55, Aleksandar Lazic :

> Hi.
>
> On 2024-07-25 (Do.) 23:22, Илья Шипицин wrote:
> >
> >
> > чт, 25 июл. 2024 г. в 22:00, Илья Шипицин  > <mailto:chipits...@gmail.com>>:
> >
> >
> >
> > чт, 25 июл. 2024 г. в 14:27, William Lallemand  > <mailto:wlallem...@irq6.net>>:
> >
> > On Wed, Jul 03, 2024 at 11:51:21PM +0200, William Lallemand
> wrote:
> >  > > aws-lc implements chacha20_poly1305 in a different way than
> QuicTLS.
> >  > > and if that gap is eliminated, it will be a good point to
> declare
> > aws-lc as
> >  > > a recommended QUIC lib.
> >  > >
> >  > > if we compare aws-lc against openssl-1.1 (not quictls), it
> is indeed
> >  > > matches (maybe except some niche features like async)
> >  > >
> >  >
> >  > BoringSSL, LibreSSL, and AWS-LC uses the EVP_AEAD API,
> basically we
> > would need
> >  > to use EVP_aead_chacha20_poly1305() instead of
> > EVP_chacha20_poly1305(), and
> >  > EVP_aead_chacha20_poly1305() instead of EVP_chacha20() and
> uses the
> > EVP_AEAD
> >  > functions. IMHO this is not a huge change and this is doable,
> we just
> > need to
> >  > invest some time on it.
> >  >
> >  > Regards,
> >
> > FYI I just push some patches that implements the
> CHACHA20_POLY1305 for
> > QUIC with AWS-LC.
> >
> >
> > nice, it passes chacha20 test!
> >
> > We now handle with AWS-LC for QUIC:
> > - TLS_AES_128_GCM_SHA256
> > - TLS_AES_256_GCM_SHA384
> > - TLS_CHACHA20_POLY1305_SHA256
> >
> > The TLS_AES_128_CCM_SHA256 cipher is not implemented by AWS-LC,
> and is
> > disabled by default in a lot of stack anyway.
> >
> >
> > next point: 0-RTT
>
> Sorry for the stupid questions but as I don't maintain any QUIC based apps
> I'm
> curious if this really makes any differences compared to "normal" requests.
>
> How often is this really used in the real world setup?
> Have anybody seen any real benefit for users out there with 0-RTT?
>

very beneficial
Introducing Zero Round Trip Time Resumption (0-RTT) (cloudflare.com)
<https://blog.cloudflare.com/introducing-0-rtt/>


>
> > according to haproxy/include/haproxy/openssl-compat.h at master ·
> > haproxy/haproxy (github.com)
> > <
> https://github.com/haproxy/haproxy/blob/master/include/haproxy/openssl-compat.h#L117-L119
> >
> > 0-RTT support depends on SSL_OP_NO_ANTI_REPLAY (which is not defined for
> AWS-LC)
> >
> > thus it is not possible to satisfy that condition
> >
> > haproxy/src/quic_ssl.c at master · haproxy/haproxy (github.com)
> > <https://github.com/haproxy/haproxy/blob/master/src/quic_ssl.c#L765-L772
> >
> >
> > AWS-LC (inherited from BoringSSL) supports 0-RTT in its own way not
> compatible
> > with QuicTLS ((
> >
> >
> > Regards,
> >
> > --
> > William Lallemand
>
> Regards
> Alex
>

Re: [ANNOUNCE] haproxy-3.1-dev2

[Aleksandar Lazic]

Hi.

On 2024-07-25 (Do.) 23:22, Илья Шипицин wrote:



чт, 25 июл. 2024 г. в 22:00, Илья Шипицин <mailto:chipits...@gmail.com>>:




чт, 25 июл. 2024 г. в 14:27, William Lallemand mailto:wlallem...@irq6.net>>:

On Wed, Jul 03, 2024 at 11:51:21PM +0200, William Lallemand wrote:
 > > aws-lc implements chacha20_poly1305 in a different way than 
QuicTLS.
 > > and if that gap is eliminated, it will be a good point to declare
aws-lc as
 > > a recommended QUIC lib.
 > >
 > > if we compare aws-lc against openssl-1.1 (not quictls), it is 
indeed
 > > matches (maybe except some niche features like async)
 > >
 >
 > BoringSSL, LibreSSL, and AWS-LC uses the EVP_AEAD API, basically we
would need
 > to use EVP_aead_chacha20_poly1305() instead of
EVP_chacha20_poly1305(), and
 > EVP_aead_chacha20_poly1305() instead of EVP_chacha20() and uses the
EVP_AEAD
 > functions. IMHO this is not a huge change and this is doable, we just
need to
 > invest some time on it.
 >
 > Regards,

FYI I just push some patches that implements the CHACHA20_POLY1305 for
QUIC with AWS-LC.


nice, it passes chacha20 test!

We now handle with AWS-LC for QUIC:
- TLS_AES_128_GCM_SHA256
- TLS_AES_256_GCM_SHA384
- TLS_CHACHA20_POLY1305_SHA256

The TLS_AES_128_CCM_SHA256 cipher is not implemented by AWS-LC, and is
disabled by default in a lot of stack anyway.


next point: 0-RTT


Sorry for the stupid questions but as I don't maintain any QUIC based apps I'm 
curious if this really makes any differences compared to "normal" requests.


How often is this really used in the real world setup?
Have anybody seen any real benefit for users out there with 0-RTT?

according to haproxy/include/haproxy/openssl-compat.h at master · 
haproxy/haproxy (github.com) 
<https://github.com/haproxy/haproxy/blob/master/include/haproxy/openssl-compat.h#L117-L119>

0-RTT support depends on SSL_OP_NO_ANTI_REPLAY (which is not defined for AWS-LC)

thus it is not possible to satisfy that condition

haproxy/src/quic_ssl.c at master · haproxy/haproxy (github.com) 
<https://github.com/haproxy/haproxy/blob/master/src/quic_ssl.c#L765-L772>


AWS-LC (inherited from BoringSSL) supports 0-RTT in its own way not compatible 
with QuicTLS ((



Regards,

-- 
William Lallemand


Regards
Alex

Re: [ANNOUNCE] haproxy-3.1-dev2

[Илья Шипицин]

пт, 26 июл. 2024 г. в 09:14, William Lallemand :

> On Thu, Jul 25, 2024 at 11:22:25PM +0200, Илья Шипицин wrote:
> > Subject: Re: [ANNOUNCE] haproxy-3.1-dev2
> > чт, 25 июл. 2024 г. в 22:00, Илья Шипицин :
> >
> > >
> > >
> > > чт, 25 июл. 2024 г. в 14:27, William Lallemand :
> > >
> > >> On Wed, Jul 03, 2024 at 11:51:21PM +0200, William Lallemand wrote:
> > >> > > aws-lc implements chacha20_poly1305 in a different way than
> QuicTLS.
> > >> > > and if that gap is eliminated, it will be a good point to declare
> > >> aws-lc as
> > >> > > a recommended QUIC lib.
> > >> > >
> > >> > > if we compare aws-lc against openssl-1.1 (not quictls), it is
> indeed
> > >> > > matches (maybe except some niche features like async)
> > >> > >
> > >> >
> > >> > BoringSSL, LibreSSL, and AWS-LC uses the EVP_AEAD API, basically we
> > >> would need
> > >> > to use EVP_aead_chacha20_poly1305() instead of
> EVP_chacha20_poly1305(),
> > >> and
> > >> > EVP_aead_chacha20_poly1305() instead of EVP_chacha20() and uses the
> > >> EVP_AEAD
> > >> > functions. IMHO this is not a huge change and this is doable, we
> just
> > >> need to
> > >> > invest some time on it.
> > >> >
> > >> > Regards,
> > >>
> > >> FYI I just push some patches that implements the CHACHA20_POLY1305 for
> > >> QUIC with AWS-LC.
> > >>
> > >
> > > nice, it passes chacha20 test!
> > >
> > >
> > >> We now handle with AWS-LC for QUIC:
> > >> - TLS_AES_128_GCM_SHA256
> > >> - TLS_AES_256_GCM_SHA384
> > >> - TLS_CHACHA20_POLY1305_SHA256
> > >>
> > >> The TLS_AES_128_CCM_SHA256 cipher is not implemented by AWS-LC, and is
> > >> disabled by default in a lot of stack anyway.
> > >>
> > >
> > next point: 0-RTT
> >
> > according to haproxy/include/haproxy/openssl-compat.h at master ·
> > haproxy/haproxy (github.com)
> > <
> https://github.com/haproxy/haproxy/blob/master/include/haproxy/openssl-compat.h#L117-L119
> >
> >
> > 0-RTT support depends on SSL_OP_NO_ANTI_REPLAY (which is not defined for
> > AWS-LC)
> >
> > thus it is not possible to satisfy that condition
> >
> > haproxy/src/quic_ssl.c at master · haproxy/haproxy (github.com)
> > <https://github.com/haproxy/haproxy/blob/master/src/quic_ssl.c#L765-L772
> >
> >
> > AWS-LC (inherited from BoringSSL) supports 0-RTT in its own way not
> > compatible with QuicTLS ((
>
> Yes, I already planned to do it.
>

great!

I ran QUIC Interop on quic-go client:

Run took 0:21:32.237143
+++
||haproxy |
+++
| go | HDCLRC20MSR3BAL1L2C1C2 |
||  EV2   |
||  ZU6   |
+++
++--+
||   haproxy|
++--+
| go | G: 7995 (± 98) kbps  |
|| C: 4721 (± 182) kbps |
++--+



>
> --
> William Lallemand
>

Re: [ANNOUNCE] haproxy-3.1-dev2

[William Lallemand]

On Thu, Jul 25, 2024 at 11:22:25PM +0200, Илья Шипицин wrote:
> Subject: Re: [ANNOUNCE] haproxy-3.1-dev2
> чт, 25 июл. 2024 г. в 22:00, Илья Шипицин :
> 
> >
> >
> > чт, 25 июл. 2024 г. в 14:27, William Lallemand :
> >
> >> On Wed, Jul 03, 2024 at 11:51:21PM +0200, William Lallemand wrote:
> >> > > aws-lc implements chacha20_poly1305 in a different way than QuicTLS.
> >> > > and if that gap is eliminated, it will be a good point to declare
> >> aws-lc as
> >> > > a recommended QUIC lib.
> >> > >
> >> > > if we compare aws-lc against openssl-1.1 (not quictls), it is indeed
> >> > > matches (maybe except some niche features like async)
> >> > >
> >> >
> >> > BoringSSL, LibreSSL, and AWS-LC uses the EVP_AEAD API, basically we
> >> would need
> >> > to use EVP_aead_chacha20_poly1305() instead of EVP_chacha20_poly1305(),
> >> and
> >> > EVP_aead_chacha20_poly1305() instead of EVP_chacha20() and uses the
> >> EVP_AEAD
> >> > functions. IMHO this is not a huge change and this is doable, we just
> >> need to
> >> > invest some time on it.
> >> >
> >> > Regards,
> >>
> >> FYI I just push some patches that implements the CHACHA20_POLY1305 for
> >> QUIC with AWS-LC.
> >>
> >
> > nice, it passes chacha20 test!
> >
> >
> >> We now handle with AWS-LC for QUIC:
> >> - TLS_AES_128_GCM_SHA256
> >> - TLS_AES_256_GCM_SHA384
> >> - TLS_CHACHA20_POLY1305_SHA256
> >>
> >> The TLS_AES_128_CCM_SHA256 cipher is not implemented by AWS-LC, and is
> >> disabled by default in a lot of stack anyway.
> >>
> >
> next point: 0-RTT
> 
> according to haproxy/include/haproxy/openssl-compat.h at master ·
> haproxy/haproxy (github.com)
> <https://github.com/haproxy/haproxy/blob/master/include/haproxy/openssl-compat.h#L117-L119>
> 
> 0-RTT support depends on SSL_OP_NO_ANTI_REPLAY (which is not defined for
> AWS-LC)
> 
> thus it is not possible to satisfy that condition
> 
> haproxy/src/quic_ssl.c at master · haproxy/haproxy (github.com)
> <https://github.com/haproxy/haproxy/blob/master/src/quic_ssl.c#L765-L772>
> 
> AWS-LC (inherited from BoringSSL) supports 0-RTT in its own way not
> compatible with QuicTLS ((

Yes, I already planned to do it.

-- 
William Lallemand

Re: [ANNOUNCE] haproxy-3.1-dev2

[Илья Шипицин]

чт, 25 июл. 2024 г. в 22:00, Илья Шипицин :

>
>
> чт, 25 июл. 2024 г. в 14:27, William Lallemand :
>
>> On Wed, Jul 03, 2024 at 11:51:21PM +0200, William Lallemand wrote:
>> > > aws-lc implements chacha20_poly1305 in a different way than QuicTLS.
>> > > and if that gap is eliminated, it will be a good point to declare
>> aws-lc as
>> > > a recommended QUIC lib.
>> > >
>> > > if we compare aws-lc against openssl-1.1 (not quictls), it is indeed
>> > > matches (maybe except some niche features like async)
>> > >
>> >
>> > BoringSSL, LibreSSL, and AWS-LC uses the EVP_AEAD API, basically we
>> would need
>> > to use EVP_aead_chacha20_poly1305() instead of EVP_chacha20_poly1305(),
>> and
>> > EVP_aead_chacha20_poly1305() instead of EVP_chacha20() and uses the
>> EVP_AEAD
>> > functions. IMHO this is not a huge change and this is doable, we just
>> need to
>> > invest some time on it.
>> >
>> > Regards,
>>
>> FYI I just push some patches that implements the CHACHA20_POLY1305 for
>> QUIC with AWS-LC.
>>
>
> nice, it passes chacha20 test!
>
>
>> We now handle with AWS-LC for QUIC:
>> - TLS_AES_128_GCM_SHA256
>> - TLS_AES_256_GCM_SHA384
>> - TLS_CHACHA20_POLY1305_SHA256
>>
>> The TLS_AES_128_CCM_SHA256 cipher is not implemented by AWS-LC, and is
>> disabled by default in a lot of stack anyway.
>>
>
next point: 0-RTT

according to haproxy/include/haproxy/openssl-compat.h at master ·
haproxy/haproxy (github.com)
<https://github.com/haproxy/haproxy/blob/master/include/haproxy/openssl-compat.h#L117-L119>

0-RTT support depends on SSL_OP_NO_ANTI_REPLAY (which is not defined for
AWS-LC)

thus it is not possible to satisfy that condition

haproxy/src/quic_ssl.c at master · haproxy/haproxy (github.com)
<https://github.com/haproxy/haproxy/blob/master/src/quic_ssl.c#L765-L772>

AWS-LC (inherited from BoringSSL) supports 0-RTT in its own way not
compatible with QuicTLS ((



>
>> Regards,
>>
>> --
>> William Lallemand
>>
>

Re: [ANNOUNCE] haproxy-3.1-dev2

[Илья Шипицин]

чт, 25 июл. 2024 г. в 14:27, William Lallemand :

> On Wed, Jul 03, 2024 at 11:51:21PM +0200, William Lallemand wrote:
> > > aws-lc implements chacha20_poly1305 in a different way than QuicTLS.
> > > and if that gap is eliminated, it will be a good point to declare
> aws-lc as
> > > a recommended QUIC lib.
> > >
> > > if we compare aws-lc against openssl-1.1 (not quictls), it is indeed
> > > matches (maybe except some niche features like async)
> > >
> >
> > BoringSSL, LibreSSL, and AWS-LC uses the EVP_AEAD API, basically we
> would need
> > to use EVP_aead_chacha20_poly1305() instead of EVP_chacha20_poly1305(),
> and
> > EVP_aead_chacha20_poly1305() instead of EVP_chacha20() and uses the
> EVP_AEAD
> > functions. IMHO this is not a huge change and this is doable, we just
> need to
> > invest some time on it.
> >
> > Regards,
>
> FYI I just push some patches that implements the CHACHA20_POLY1305 for
> QUIC with AWS-LC.
>

nice, it passes chacha20 test!


> We now handle with AWS-LC for QUIC:
> - TLS_AES_128_GCM_SHA256
> - TLS_AES_256_GCM_SHA384
> - TLS_CHACHA20_POLY1305_SHA256
>
> The TLS_AES_128_CCM_SHA256 cipher is not implemented by AWS-LC, and is
> disabled by default in a lot of stack anyway.
>
> Regards,
>
> --
> William Lallemand
>

Re: [ANNOUNCE] haproxy-3.1-dev4

[Lukas Tribus]

On Wed, 24 Jul 2024 at 23:19, William Lallemand  wrote:
>
> On Wed, Jul 24, 2024 at 10:32:16PM +0200, Aleksandar Lazic wrote:
> > Does this announcement have any impact to HAProxy?
> >
> > "Intent to End OCSP Service"
> > https://letsencrypt.org/2024/07/23/replacing-ocsp-with-crls.html
> > https://news.ycombinator.com/item?id=41046956
> >
>
> I read about this yesterday and my impression is that they are trying to use 
> the excuse of the privacy problems to end a
> service that they have difficulties to scale.

I agree.

Google disabled online/active OCSP requests a long time ago - more
than a decade.

Here's more argumentation:
https://docs.google.com/document/d/180T6cDSWPy54Rb5d6R4zN7MuLEMShaZ4IRLQgdPqE98/edit?pli=1

They claim OCSP must-staple is basically unused, OCSP stapling itself
is not much used (8%), it's complicated and they do not want to rely
on it.


I guess it's one less feature we have to care about, but I wish they
would have made up their mind 10 years ago and spared us all the pain.


Lukas

Re: [ANNOUNCE] haproxy-3.1-dev2

[William Lallemand]

On Wed, Jul 03, 2024 at 11:51:21PM +0200, William Lallemand wrote:
> > aws-lc implements chacha20_poly1305 in a different way than QuicTLS.
> > and if that gap is eliminated, it will be a good point to declare aws-lc as
> > a recommended QUIC lib.
> > 
> > if we compare aws-lc against openssl-1.1 (not quictls), it is indeed
> > matches (maybe except some niche features like async)
> > 
> 
> BoringSSL, LibreSSL, and AWS-LC uses the EVP_AEAD API, basically we would need
> to use EVP_aead_chacha20_poly1305() instead of EVP_chacha20_poly1305(), and
> EVP_aead_chacha20_poly1305() instead of EVP_chacha20() and uses the EVP_AEAD
> functions. IMHO this is not a huge change and this is doable, we just need to
> invest some time on it.
> 
> Regards,

FYI I just push some patches that implements the CHACHA20_POLY1305 for QUIC 
with AWS-LC.
We now handle with AWS-LC for QUIC:
- TLS_AES_128_GCM_SHA256
- TLS_AES_256_GCM_SHA384
- TLS_CHACHA20_POLY1305_SHA256

The TLS_AES_128_CCM_SHA256 cipher is not implemented by AWS-LC, and is disabled 
by default in a lot of stack anyway.

Regards,

-- 
William Lallemand

Re: [ANNOUNCE] haproxy-3.1-dev4

[Илья Шипицин]

ср, 24 июл. 2024 г. в 23:22, William Lallemand :
>
> On Wed, Jul 24, 2024 at 10:32:16PM +0200, Aleksandar Lazic wrote:
> > Does this announcement have any impact to HAProxy?
> >
> > "Intent to End OCSP Service"
> > https://letsencrypt.org/2024/07/23/replacing-ocsp-with-crls.html
> > https://news.ycombinator.com/item?id=41046956
> >
>

interesting part is that Chrome (which is a dominant browser) is using its
own CRLSets machinery  CRLSets (chromium.org)
<https://www.chromium.org/Home/chromium-security/crlsets/>

so, chrome will simply do not emit neither CRL, nor OCSP request (but it
honors stapling)


we've discovered that accidentally trying to "revoke" some of our certs and
wondering why it didn't work for chrome

> I read about this yesterday and my impression is that they are trying to
use the excuse of the privacy problems to end a
> service that they have difficulties to scale.
>
> However this does not make much sense to me because they don't talk about
OCSP stapling in this article which does not
> have these privacy problems since it is done by the web server, and it
honestly seems to be intentional...
>
> What it means for HAProxy deployments is that you just need to disable
OCSP stapling once it's not available anymore.
> When OCSP stapling is not enabled, the browser requests an OCSP response
on the OCSP responder. They are able to do the
> same with CRL.
>
> Honestly I don't get where this is a better approach with how CRL are
working in browsers currently, but things are
> probably going to evolve on the browsers side, and the HN discussion
seems to confirm that.
>
> > I know there is
https://docs.haproxy.org/3.0/configuration.html#5.1-crl-file
> > but maybe it's worth to add a blog post about that topic and what impact
> > this change have to HAProxy.
>
> The "crl-file" keyword on the bind line does not have anything to do with
this, it is only useful when you want to
> revoke client certificates when doing mTLS. The problem is only on the
browser side in fact.
>
> --
> William Lallemand
>
>

Re: [ANNOUNCE] haproxy-3.1-dev4

[William Lallemand]

On Wed, Jul 24, 2024 at 10:32:16PM +0200, Aleksandar Lazic wrote:
> Does this announcement have any impact to HAProxy?
> 
> "Intent to End OCSP Service"
> https://letsencrypt.org/2024/07/23/replacing-ocsp-with-crls.html
> https://news.ycombinator.com/item?id=41046956
> 

I read about this yesterday and my impression is that they are trying to use 
the excuse of the privacy problems to end a
service that they have difficulties to scale.

However this does not make much sense to me because they don't talk about OCSP 
stapling in this article which does not
have these privacy problems since it is done by the web server, and it honestly 
seems to be intentional...

What it means for HAProxy deployments is that you just need to disable OCSP 
stapling once it's not available anymore.
When OCSP stapling is not enabled, the browser requests an OCSP response on the 
OCSP responder. They are able to do the
same with CRL.

Honestly I don't get where this is a better approach with how CRL are working 
in browsers currently, but things are
probably going to evolve on the browsers side, and the HN discussion seems to 
confirm that.

> I know there is https://docs.haproxy.org/3.0/configuration.html#5.1-crl-file
> but maybe it's worth to add a blog post about that topic and what impact
> this change have to HAProxy.

The "crl-file" keyword on the bind line does not have anything to do with this, 
it is only useful when you want to
revoke client certificates when doing mTLS. The problem is only on the browser 
side in fact.

-- 
William Lallemand

Re: [ANNOUNCE] haproxy-3.1-dev4

[Илья Шипицин]

ср, 24 июл. 2024 г. в 22:39, Willy Tarreau :
>
> Hi Alex,
>
> On Wed, Jul 24, 2024 at 10:32:16PM +0200, Aleksandar Lazic wrote:
> > >- SPOE: the old applet-based architecture was replaced with the new
> > >  mux-based one which allows idle connections sharing between threads,
> > >  as well as queuing, load balancing, stickiness etc per request 
> > > instead
> > >  of per-connection and adds a lot of flexibility to the engine. We'd
> > >  appreciate it a lot if SPOE users would take some time to verify that
> > >  it works at least as well for them as before (and hopefully even
> > >  better). Some good ideas may spark. Please check Christopher's
> > >  response to the SPOE thread for more info.
> >
> > Cool. Thank you that you handle this topic, even I don't use it for now :-)
>
> Hehe, who knows, maybe one day you'll have a good use for it :-)
>
> > >- ocsp: some processing was refined to better handle a corner case 
> > > where
> > >  the issuer chain is not in the same PEM file, though it also slightly
> > >  changes how this is handled on the CLI.
> >
> > [snipp]
> >
> > Does this announcement have any impact to HAProxy?
> >
> > "Intent to End OCSP Service"
> > https://letsencrypt.org/2024/07/23/replacing-ocsp-with-crls.html
> > https://news.ycombinator.com/item?id=41046956
>
> I noticed it on LWN today but I really have no idea. I'll let the SSL
> experts chime in.

from user point of view disabling OCSP is a disadvantage, because web
browser will need extra CRL download (instead of OCSP staple)

for people setting up their servers to add OCSP stapling it would be
... a surprise :)

>
> Cheers,
> Willy
>
>

Re: [ANNOUNCE] haproxy-3.1-dev4

[Willy Tarreau]

Hi Alex,

On Wed, Jul 24, 2024 at 10:32:16PM +0200, Aleksandar Lazic wrote:
> >- SPOE: the old applet-based architecture was replaced with the new
> >  mux-based one which allows idle connections sharing between threads,
> >  as well as queuing, load balancing, stickiness etc per request instead
> >  of per-connection and adds a lot of flexibility to the engine. We'd
> >  appreciate it a lot if SPOE users would take some time to verify that
> >  it works at least as well for them as before (and hopefully even
> >  better). Some good ideas may spark. Please check Christopher's
> >  response to the SPOE thread for more info.
> 
> Cool. Thank you that you handle this topic, even I don't use it for now :-)

Hehe, who knows, maybe one day you'll have a good use for it :-)

> >- ocsp: some processing was refined to better handle a corner case where
> >  the issuer chain is not in the same PEM file, though it also slightly
> >  changes how this is handled on the CLI.
> 
> [snipp]
> 
> Does this announcement have any impact to HAProxy?
> 
> "Intent to End OCSP Service"
> https://letsencrypt.org/2024/07/23/replacing-ocsp-with-crls.html
> https://news.ycombinator.com/item?id=41046956

I noticed it on LWN today but I really have no idea. I'll let the SSL
experts chime in.

Cheers,
Willy

Re: [ANNOUNCE] haproxy-3.1-dev4

[Aleksandar Lazic]

On 2024-07-24 (Mi.) 18:50, Willy Tarreau wrote:

Hi,

HAProxy 3.1-dev4 was released on 2024/07/24. It added 113 new commits
after version 3.1-dev3.

Some nice goodies came in this version:


[snipp]


   - SPOE: the old applet-based architecture was replaced with the new
 mux-based one which allows idle connections sharing between threads,
 as well as queuing, load balancing, stickiness etc per request instead
 of per-connection and adds a lot of flexibility to the engine. We'd
 appreciate it a lot if SPOE users would take some time to verify that
 it works at least as well for them as before (and hopefully even
 better). Some good ideas may spark. Please check Christopher's
 response to the SPOE thread for more info.


Cool. Thank you that you handle this topic, even I don't use it for now :-)


   - ocsp: some processing was refined to better handle a corner case where
 the issuer chain is not in the same PEM file, though it also slightly
 changes how this is handled on the CLI.


[snipp]

Does this announcement have any impact to HAProxy?

"Intent to End OCSP Service"
https://letsencrypt.org/2024/07/23/replacing-ocsp-with-crls.html
https://news.ycombinator.com/item?id=41046956

I know there is https://docs.haproxy.org/3.0/configuration.html#5.1-crl-file but 
maybe it's worth to add a blog post about that topic and what impact this change 
have to HAProxy.


Regards
Alex

[ANNOUNCE] haproxy-3.1-dev4

[Willy Tarreau]

Hi,

HAProxy 3.1-dev4 was released on 2024/07/24. It added 113 new commits
after version 3.1-dev3.

Some nice goodies came in this version:
  - QUIC now has support for sending frames using GSO, which can save
quite some significant CPU cycles since a single syscall will be used
for multiple datagrams. We found that there are restrictions to this
such as certain network interfaces not supporting it (veth for example)
so the mechanism implements an automatic fallback when the failure is
detected.

  - SPOE: the old applet-based architecture was replaced with the new
mux-based one which allows idle connections sharing between threads,
as well as queuing, load balancing, stickiness etc per request instead
of per-connection and adds a lot of flexibility to the engine. We'd
appreciate it a lot if SPOE users would take some time to verify that
it works at least as well for them as before (and hopefully even
better). Some good ideas may spark. Please check Christopher's
response to the SPOE thread for more info.

  - rings: the outgoing connections to a log server are now balanced over
multiple threads, not just the first one, and the "max-reuse" server
parameter is honored to force connections to be broken and recreated
after a given number of messages in order to improve rebalancing as
well as allow to gracefully renew backend connections (e.g. in case
another layer of LB sits there).

  - ocsp: some processing was refined to better handle a corner case where
the issuer chain is not in the same PEM file, though it also slightly
changes how this is handled on the CLI.

  - variables: new "parent" scopes (psess, ptxn, preq, pres) to access the
parent stream's sess/txn/req/res variables. For now, only used with
SPOE to access the parent stream, but might be usable for other things
later.

  - more info collected in post_mortem for "show dev" (e.g. process'
boot and current limits etc).

  - speedup of startup with very high maxconn and thread counts by only
scanning the known used FD range.

We also fixed a few bugs, a very difficult to reproduce one which can make
the H2 mux loop on certain truncated frames, needlessly wasting CPU until
a restart, an issue with the bwlim filter which could cause wakeup loops
due to a timeout in the past, a bug where "show threads" can crash the
process on setups with less than 16 threads, a missing unlock in SSL OCSP
code's error path, some late boot-time warnings that were not properly
accounted regarding the zero-warning restriction, and a few other minor
issues.

And as usual there were some code cleanups (e.g. some "global" keywords
parsers started to move to their own functions), cleanup of FD limits
handling code, some doc updates (e.g. dedicated section about variables,
OCSP and http-keep-alive section), and regtests.

Even though it contains significant changes it's not expected to be
broken (and it's already running on haproxy.org). SPOE is a big change
that might have affect some of its users in case there would be an
uncaught issue there (which is one extra reason for testing now), so
please watch when testing it.

Please find the usual URLs below :
   Site index   : https://www.haproxy.org/
   Documentation: https://docs.haproxy.org/
   Wiki : https://github.com/haproxy/wiki/wiki
   Discourse: https://discourse.haproxy.org/
   Slack channel: https://slack.haproxy.org/
   Issue tracker: https://github.com/haproxy/haproxy/issues
   Sources  : https://www.haproxy.org/download/3.1/src/
   Git repository   : https://git.haproxy.org/git/haproxy.git/
   Git Web browsing : https://git.haproxy.org/?p=haproxy.git
   Changelog: https://www.haproxy.org/download/3.1/src/CHANGELOG
   Dataplane API: 
https://github.com/haproxytech/dataplaneapi/releases/latest
   Pending bugs : https://www.haproxy.org/l/pending-bugs
   Reviewed bugs: https://www.haproxy.org/l/reviewed-bugs
   Code reports : https://www.haproxy.org/l/code-reports
   Latest builds: https://www.haproxy.org/l/dev-packages

Willy
---
Complete changelog :
Amaury Denoyelle (12):
  CLEANUP: quic: remove obsolete comment on send
  MINOR: quic: extend detection of UDP API OS features
  MINOR: quic: activate UDP GSO for QUIC if supported
  MINOR: quic: define quic_cc_path MTU as constant
  MINOR: quic: add GSO parameter on quic_sock send API
  MAJOR: quic: support GSO when encoding datagrams
  MEDIUM: quic: implement GSO fallback mechanism
  MINOR: quic: add counters of sent bytes with and without GSO
  CLEANUP: proto: rename TID affinity callbacks
  CLEANUP: quic: rename TID affinity elements
  BUG/MINOR: do not close uninit FD in quic_test_socketops()
  BUG/MINOR: stick-table: fix crash for src_inc_gpc() without stkcounter

Re: [ANNOUNCE] haproxy-3.0.3

[Willy Tarreau]

Hi Vincent,

On Mon, Jul 22, 2024 at 09:25:45PM +0200, Vincent Bernat wrote:
> On 2024-07-11 16:51, Willy Tarreau wrote:
> > Hi,
> > 
> > HAProxy 3.0.3 was released on 2024/07/11. It added 42 new commits
> > after version 3.0.2.
> 
> I am late releasing this version on haproxy.debian.net. I have issues with
> compiling for ARM64. There is a known bug in recent version of QEMU
> segfaulting on ARM64 userland emulation. Previously, I was able to just
> downgrade to an older version of QEMU, but this does not work anymore. I
> suppose there was some changes in recent kernel versions. I'll need to try
> again in a VM.

OK, thanks for letting us know!

I could probably propose you an access to one physical machine, but I
guess this would not help much because it would likely completely
change your build process. Do not hesitate to let us know if there's
anything we can do to help you.

Cheers,
Willy

Re: [ANNOUNCE] haproxy-3.0.3

[Vincent Bernat]

On 2024-07-11 16:51, Willy Tarreau wrote:

Hi,

HAProxy 3.0.3 was released on 2024/07/11. It added 42 new commits
after version 3.0.2.


I am late releasing this version on haproxy.debian.net. I have issues 
with compiling for ARM64. There is a known bug in recent version of QEMU 
segfaulting on ARM64 userland emulation. Previously, I was able to just 
downgrade to an older version of QEMU, but this does not work anymore. I 
suppose there was some changes in recent kernel versions. I'll need to 
try again in a VM.

Re: [ANNOUNCE] haproxy-3.1-dev3 (more infos on the story with fd-hard-limit and systemd)

[Willy Tarreau]

On Wed, Jul 17, 2024 at 02:06:03PM +0200, Lukas Tribus wrote:
> On Wed, 17 Jul 2024 at 11:25, Willy Tarreau  wrote:
> >
> > At this point, do you (or anyone else) still have any objection against
> > backporting the DEFAULT_MAXFD patch so as to preserve the current
> > defaults for users, and/or do you have any alternate proposal, or just
> > want to discuss other possibilities ?
> 
> No, I don't have objections.

OK thank you!

Willy

Re: [ANNOUNCE] haproxy-3.1-dev3 (more infos on the story with fd-hard-limit and systemd)

[Lukas Tribus]

On Wed, 17 Jul 2024 at 11:25, Willy Tarreau  wrote:
>
> At this point, do you (or anyone else) still have any objection against
> backporting the DEFAULT_MAXFD patch so as to preserve the current
> defaults for users, and/or do you have any alternate proposal, or just
> want to discuss other possibilities ?

No, I don't have objections.

Regards,
Lukas


Lukas

Re: [ANNOUNCE] haproxy-3.1-dev3 (more infos on the story with fd-hard-limit and systemd)

[Willy Tarreau]

Hi Lukas,

On Tue, Jul 16, 2024 at 11:28:12PM +0200, Lukas Tribus wrote:
> Hi Valentine, hi Willy,
> 
> after spending some time testing I agree tuning maxconn/fd-limits is hard ...

In fact we know that it's hard for experts, and it's even harder for
new users.

> With 8GB RAM we can still OOM with 1M FDs / 500k maxconn (no TLS), but
> it appears to be around the sweetspot.

It happens to be the previous common limit, with plenty of distros
offering 1M FDs by default. For me it's already large, I would be
fine with a much lower value, but lowering a limit could possibly
break some use cases, and given that basically nobody complained
about OOM in regular usages recently, it was sort of a confirmation
that the default 1M that was applied for quite some time was OK for
most users.

> It thought it would require more memory considering that we suggest
> 1GB of memory for 20k non-TLS connections or 8k TLS connections, but
> my test was indeed synthetic with zero features used, and it's not
> only about haproxy userspace but the system as well.

Yes, it's really a whole. Its even more difficult nowadays because
it depends if data stay stuck in system buffers or not. Haproxy
aggressively recycles its buffers, and since 3.0 it even avoids
reading data if some are stuck downstream so it tries to be very
memory efficient, but we know that memory usage can increase a lot
with SSL and congested links.

> lukas@dev:~/haproxy$ git grep -B3 -A1 "GB of RAM"
(...)

I think we might be doing better than before on this but I don't
want to encourage users to put more connections on low memory. RAM
is so cheap that the smallest VM you can find has 2GB of RAM for a
single CPU core, so better encourage them to have enough RAM to
evacuate the data rather than risking OOM.

I would *love* to find a reasonable way to autosize all this based
on available RAM, but there's no portable solution to this, which
means that whatever hack we'll do would still require a fallback
anyway.

> > What I would really like is to no longer see any maxconn in a regular
> > configuration because there's no good value and we've seen them copied
> > over and over.
> 
> By setting a global maxconn you force yourself to at least think about it.

I totally agree with that. However our experience (particularly from
discussing appliance sizing with potential customers) is that nobody
has any idea of their maxconn, nor even within an order of magnitude.
Most users usually know more or less what their bandwidth is, sometimes
number of visitors per day, and that's about all. We really need to
consider that those who post here and on github are the exception and
not the norm. Another factor that comes into play is how connections
are accounted. Those coming from other LBs acting at a lower layer
(e.g. LVS) count the total number of connections tracked in the LB's
table, which includes TIME_WAIT, which can represent 90-99% depending
on the workloads! That's how you hear that port 80 in charge of the
redirect of a web site working exclusively in SSL, that gets 1000
conn/s requires 6 concurrent connections while the reality is
rather less than 100!

> I'm assuming by "regular configuration" you mean small scale/size? In
> this case I agree.

Yes that was it. The vast majority of users have way less than even
100 req/s on average and technical parameters like maxconn are the
least of their concerns.

> > I'm confused now, I don't see how, given that the change only *lowers*
> > an existing limit, it never raises it. It's precisely because of the
> > risk of OOM with OSes switching the default from one million FDs to one
> > billion that we're proposing to keep the previous limit of 1 million as
> > a sane upper bound. The only risk I'm seeing would be users discovering
> > that they cannot accept more than ~500k concurrent connections on a large
> > system. But I claim that those dealing with such loads *do* careful size
> > and configure their systems and services (RAM, fd, conntrack, monitoring
> > tools etc). Thus I'm not sure which scenario you have in mind that this
> > change could result in such a report as above.
> 
> True, I confused memory required for initialization with memory
> allocated when actually used.

OK no pb!

> On Thu, 11 Jul 2024 at 14:44, Willy Tarreau  wrote:
> >
> > My take on this limit is that most users should not care. Those dealing
> > with high loads have to do their homework and are used to doing this,
> > and those deploying in extremely small environments are also used to
> > adjusting limits (even sometimes rebuilding with specific options), and
> > I'm fine with leaving a bit of work for both extremities.
> 
> Considering ho

Re: [ANNOUNCE] haproxy-3.1-dev3 (more infos on the story with fd-hard-limit and systemd)

[Lukas Tribus]

Hi Valentine, hi Willy,


after spending some time testing I agree tuning maxconn/fd-limits is hard ...


With 8GB RAM we can still OOM with 1M FDs / 500k maxconn (no TLS), but
it appears to be around the sweetspot.

It thought it would require more memory considering that we suggest
1GB of memory for 20k non-TLS connections or 8k TLS connections, but
my test was indeed synthetic with zero features used, and it's not
only about haproxy userspace but the system as well.

lukas@dev:~/haproxy$ git grep -B3 -A1 "GB of RAM"
doc/configuration.txt-  global maxconn. Also, keep in mind that a
connection contains two buffers
doc/configuration.txt-  of tune.bufsize (16kB by default) each, as
well as some other data resulting
doc/configuration.txt-  in about 33 kB of RAM being consumed per
established connection. That means
doc/configuration.txt:  that a medium system equipped with 1GB of RAM
can withstand around
doc/configuration.txt-  2-25000 concurrent connections if properly tuned.
--
doc/intro.txt-  - 1300 HTTPS connections per second using TLS
connections renegotiated with
doc/intro.txt-RSA2048;
doc/intro.txt-
doc/intro.txt:  - 2 concurrent saturated connections per GB of
RAM, including the memory
doc/intro.txt-required for system buffers; it is possible to do
better with careful tuning
doc/intro.txt-but this result it easy to achieve.
doc/intro.txt-
doc/intro.txt:  - about 8000 concurrent TLS connections (client-side
only) per GB of RAM,
doc/intro.txt-including the memory required for system buffers;
lukas@dev:~/haproxy$



On Thu, 11 Jul 2024 at 08:05, Willy Tarreau  wrote:
>
> What I would really like is to no longer see any maxconn in a regular
> configuration because there's no good value and we've seen them copied
> over and over.

By setting a global maxconn you force yourself to at least think about it.

I'm assuming by "regular configuration" you mean small scale/size? In
this case I agree.



> I'm confused now, I don't see how, given that the change only *lowers*
> an existing limit, it never raises it. It's precisely because of the
> risk of OOM with OSes switching the default from one million FDs to one
> billion that we're proposing to keep the previous limit of 1 million as
> a sane upper bound. The only risk I'm seeing would be users discovering
> that they cannot accept more than ~500k concurrent connections on a large
> system. But I claim that those dealing with such loads *do* careful size
> and configure their systems and services (RAM, fd, conntrack, monitoring
> tools etc). Thus I'm not sure which scenario you have in mind that this
> change could result in such a report as above.

True, I confused memory required for initialization with memory
allocated when actually used.



On Thu, 11 Jul 2024 at 14:44, Willy Tarreau  wrote:
>
> My take on this limit is that most users should not care. Those dealing
> with high loads have to do their homework and are used to doing this,
> and those deploying in extremely small environments are also used to
> adjusting limits (even sometimes rebuilding with specific options), and
> I'm fine with leaving a bit of work for both extremities.

Considering how non-trivial tuning maxconn/fd-hard-limit/ulimit for a
specific memory size and configuration is, I (now) have to agree.



On Tue, 16 Jul 2024 at 16:22, Valentine Krasnobaeva
 wrote:
>
> Our issue in GITHUB: https://github.com/haproxy/haproxy/issues/2621
>
>>You agree that this is the environment systemd sets us up with, right?
>
> Yes, as it was investigated by Apollon systemd/256~rc3-3 now sets the
> file descriptor hard limit to kernel max on boot.

Yes, I will comment regarding some additional systemd context on the
github issue (but nothing changes for haproxy).



On Tue, 16 Jul 2024 at 16:22, Valentine Krasnobaeva
 wrote:
>
> It is obscure for some users 'fd-hard-limit'. And a lot of them
> may ask: "What is the best value, according to my environment,
> which I should put here ?", "What will be the impact?"

This is completely true and why I prefer users think about maxconn
instead, but like I said, even that is hard.



Regards,
Lukas

Re: [ANNOUNCE] haproxy-3.1-dev3 (more infos on the story with fd-hard-limit and systemd)

[Valentine Krasnobaeva]

Hi Lukas ! Hi All!

Below are some links and comments just to provide more information on 
"fd-hard-limit" default value subject :).


On 11/07/2024 12:17, Lukas Tribus wrote:

Hi,

I will get back to this for further research and discussion in about a week.

In the meantime, do we agree that the environment we are developing the fix
for is the following:

the hard limit is always set to the maximum available in the kernel which
on amd64 is one billion with a B, whether the systems has 128M or 2T of
memory is irrelevant.


Yes, we have a user's bug report, you could check all details here:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1074374

It's on DebianTrixie, amd64

Our issue in GITHUB: https://github.com/haproxy/haproxy/issues/2621



You agree that this is the environment systemd sets us up with, right?


Yes, as it was investigated by Apollon systemd/256~rc3-3 now sets the 
file descriptor hard limit to kernel max on boot.


https://lists.debian.org/debian-devel/2024/06/msg00041.html

And here is the link to this systemd patch:

https://github.com/systemd/systemd/pull/28968/commits/f470dafddcd688c3ea6031d4bbcbf934fd094711

https://github.com/systemd/systemd/pull/28968

1. We are trying to anticipate this change for haproxy packages 
mainteners and for the new users, who have big servers and just want to 
discover haproxy, compile and start it with a very simple config.


2. We try to avoid the situation, when haproxy installed from package in 
the new distro with systemd >=  256~rc3-3, refuses to start with such 
kind of errors at stderr:


[ALERT](968) : fd hard limit is very high ~1B, Please, set fd-hard-limit in 
your global section

Or:
[ALERT](30116) : Not enough memory to allocate 10 entries for fdtab!
[ALERT](30116) : No polling mechanism available. Please set fd-hard-limit 
in your global section...


It is obscure for some users 'fd-hard-limit'. And a lot of them may ask: "What is the best 
value, according to my environment, which I should put here ?", "What will be the impact 
?".


3. In case of small systems with less than 16 GB of RAM and fs.nr_open=32768, 
you could do the following test with the current master branch in order to 
confirm, that:

a) process can't not bump its runtime fd hard limit to 1048576, if the current 
fd hard limit is lower;
b) process does not exits at a startup with an error in this case.


root@corei7:# git diff
diff --git a/examples/quick-test.cfg b/examples/quick-test.cfg
index f27eeff43..abda4172d 100644
--- a/examples/quick-test.cfg
+++ b/examples/quick-test.cfg
@@ -2,6 +2,7 @@
 # support for HTTP/1 and 2.
 
 global

+   stats socket /tmp/haproxy.sock
strict-limits  # refuse to start if insufficient FDs/memory
# add some process-wide tuning here if required
 
diff --git a/src/limits.c b/src/limits.c

index bb7451060..70477d86b 100644
--- a/src/limits.c
+++ b/src/limits.c
@@ -148,6 +148,7 @@ int compute_ideal_maxconn()
 * if only one of these ha-specific limits is presented in config or in
 * the cmdline.
 */
+   ha_notice(">>> DEFAULT_MAXFD=%d\n", DEFAULT_MAXFD);
if (!is_any_limit_configured())
global.fd_hard_limit = DEFAULT_MAXFD;
 
root@corei7:#


root@corei7:# sysctl -w fs.nr_open=32768
fs.nr_open = 32768
root@corei7:# ulimit -n 32768
root@corei7:# ulimit -n
32768
root@corei7:# ulimit -n -H
32768

root@corei7:# cd haproxy && make -j 8 TARGET=linux-glibc USE_LUA=1 USE_PCRE2=1

root@corei7:/home/haproxy/haproxy# ./haproxy -D -f examples/quick-test.cfg
[NOTICE]   (30970) : config : >>> DEFAULT_MAXFD=1048576

root@corei7:# echo 'show dev' | socat /tmp/haproxy.sock stdio
HAProxy version 3.1-dev3-e8799d-67
Features
  -51DEGREES +ACCEPT4 +BACKTRACE -CLOSEFROM +CPU_AFFINITY +CRYPT_H -DEVICEATLAS 
+DL -ENGINE +EPOLL -EVPORTS +GETADDRINFO -KQUEUE -LIBATOMIC +LIBCRYPT 
+LINUX_CAP +LINUX_SPLICE +LINUX_TPROXY +LUA +MATH -MEMORY_PROFILING +NETFILTER 
+NS -OBSOLETE_LINKER -OPENSSL -OPENSSL_AWSLC -OPENSSL_WOLFSSL -OT -PCRE +PCRE2 
-PCRE2_JIT -PCRE_JIT +POLL +PRCTL -PROCCTL -PROMEX -PTHREAD_EMULATION -QUIC 
-QUIC_OPENSSL_COMPAT +RT +SHM_OPEN +SLZ -SSL -STATIC_PCRE -STATIC_PCRE2 
+SYSTEMD +TFO +THREAD +THREAD_DUMP +TPROXY -WURFL -ZLIB
Build options
  Built with multi-threading support (MAX_TGROUPS=16, MAX_THREADS=256, 
default=8).
  Built with Lua version : Lua 5.4.4
  Built with network namespace support.
  Built with libslz for stateless compression.
  Compression algorithms supported : identity("identity"), deflate("deflate"), 
raw-deflate("deflate"), gzip("gzip")
  Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT 
IP_FREEBIND
  Built with PCRE2 version : 10.39 2021-10-29
  PCRE2 library supports JIT : no (USE_PCRE2_JIT not set)
  Encrypted password support via crypt(3): yes
  Built with gcc compiler version

Re: [PATCH 1/1]: BUILD/MINOR: haproxy: fix SO_LINGER usage on macOs.

[Willy TARREAU]

On Tue, Jul 16, 2024 at 05:27:10AM +0100, David CARLIER wrote:
> Hi you are right I did not check properly, had a brain fog :) we can forget
> it. Thanks.

OK perfect, thanks :-)

Willy

Re: [PATCH 1/1]: BUILD/MINOR: haproxy: fix SO_LINGER usage on macOs.

[David CARLIER]

Hi you are right I did not check properly, had a brain fog :) we can forget
it. Thanks.

On Tue, 16 Jul 2024 at 03:59, Willy TARREAU  wrote:

> Hi David!
>
> On Mon, Jul 15, 2024 at 10:29:48PM +0100, David CARLIER wrote:
> > Hi here a little patch proposal targeted for macOs.
> >
> > Cheers.
>
> > From df5741a0d391a7107157d0051ba81ef48d87b8f5 Mon Sep 17 00:00:00 2001
> > From: David Carlier 
> > Date: Mon, 15 Jul 2024 22:20:33 +0100
> > Subject: [PATCH] BUILD/MEDIUM: haproxy : fix SO_LINGER usage on macOS.
> >
> > SO_LINGER on macOS works in term of clock ticks rather than seconds
> > leading to behavior differences. SO_LINGER_SEC is available to
> > be more in line so we create HA_SO_LINGER internal.
>
> How would that make any difference given that we're using it *only*
> with zero ? From what I'm seeing both use the same "linger" struct
> so 0 ticks or 0 seconds shouldn't make any difference. What different
> behavior did you observe there ?
>
> thanks!
> Willy
>

Re: [PATCH 1/1]: BUILD/MINOR: haproxy: fix SO_LINGER usage on macOs.

[Willy TARREAU]

Hi David!

On Mon, Jul 15, 2024 at 10:29:48PM +0100, David CARLIER wrote:
> Hi here a little patch proposal targeted for macOs.
> 
> Cheers.

> From df5741a0d391a7107157d0051ba81ef48d87b8f5 Mon Sep 17 00:00:00 2001
> From: David Carlier 
> Date: Mon, 15 Jul 2024 22:20:33 +0100
> Subject: [PATCH] BUILD/MEDIUM: haproxy : fix SO_LINGER usage on macOS.
> 
> SO_LINGER on macOS works in term of clock ticks rather than seconds
> leading to behavior differences. SO_LINGER_SEC is available to
> be more in line so we create HA_SO_LINGER internal.

How would that make any difference given that we're using it *only*
with zero ? From what I'm seeing both use the same "linger" struct
so 0 ticks or 0 seconds shouldn't make any difference. What different
behavior did you observe there ?

thanks!
Willy

[PATCH 1/1]: BUILD/MINOR: haproxy: fix SO_LINGER usage on macOs.

[David CARLIER]

Hi here a little patch proposal targeted for macOs.

Cheers.


0001-BUILD-MEDIUM-haproxy-fix-SO_LINGER-usage-on-macOS.patch
Description: Binary data

[ANNOUNCE] haproxy-3.0.3

[Willy Tarreau]

Hi,

HAProxy 3.0.3 was released on 2024/07/11. It added 42 new commits
after version 3.0.2.

This version closes the list of pending issues for 3.0. Fixed in this
release are:

  - a QUIC bug affecting 3.0 and above only, by which can trigger a BUG_ON
(hence a crash) sometimes when trying to mix a large initial packet
with a small 1-RTT short packet in the same datagram ;

  - a bug in the bwlim filter that can make it forget to update its
timeout and loop fast when woken up with nothing to forward ;

  - the lack of filtering of empty transfer-encoding headers in the H1 mux
that was reported by Ben Kallus ;

  - the insufficient syntax checks on :method and :scheme H3 headers that
was reported by Yuki Mogi ;

  - a hang on lua CLI applets when trying to deal with their own buffers ;

  - a possible crash in peers code due to a synchronization issue that is
occasionally triggered by one of the regtests (GH #2629) ;

  - a possible crash in deinit() (when quitting/reloading) when using e-mail
alerts after a fix that went into 3.0.2 ;

  - a rare but possible deadlock (and crash) in QUIC after a recent fix for
a race condition in the CID tree, that also went into 3.0.2 ;

  - an unreproduced race condition affecting the QUIC CID tree. The impact
might be a possible crash but it was only found in code review and never
reported nor reproduced ;

  - a remaining issue in DNS resolution timeout/error that may cause some
flapping due to the server's address not being fully wiped ;

  - a subtle race between server address change and server removal that
may sometimes try to update a just deleted server due to isolation
starting after the lookup and implicitly giving control back when
starting ;

  - an issue in SPOE that can cause a thread to refrain from creating an
applet to connect outside, causing failures on requests processed on
this thread ;

  - a race in stick-tables starting from 2.9 where an element may be
accessed immediately after decrementing the refcount, sometimes
allowing it to be purged in parallel and causing crashes (GH #2611).

  - and the rest is a bunch of small fixes for less impacting or likely
bugs.

Some might have been waiting on 3.0.1 after seeing that 3.0.2 got a few
incomplete or unstable fixes. If you're on 3.0, 3.0.3 definitely is the
one to start with. Of course, I'm always having a thought for those
reading that in the future when I'm writing things like this, but really
it's what 3.0.0 ought to have been, so please don't wait to upgrade,
particularly if you've experienced issues recently.

Please find the usual URLs below :
   Site index   : https://www.haproxy.org/
   Documentation: https://docs.haproxy.org/
   Wiki     : https://github.com/haproxy/wiki/wiki
   Discourse: https://discourse.haproxy.org/
   Slack channel: https://slack.haproxy.org/
   Issue tracker: https://github.com/haproxy/haproxy/issues
   Sources  : https://www.haproxy.org/download/3.0/src/
   Git repository   : https://git.haproxy.org/git/haproxy-3.0.git/
   Git Web browsing : https://git.haproxy.org/?p=haproxy-3.0.git
   Changelog: https://www.haproxy.org/download/3.0/src/CHANGELOG
   Dataplane API: 
https://github.com/haproxytech/dataplaneapi/releases/latest
   Pending bugs : https://www.haproxy.org/l/pending-bugs
   Reviewed bugs: https://www.haproxy.org/l/reviewed-bugs
   Code reports : https://www.haproxy.org/l/code-reports
   Latest builds: https://www.haproxy.org/l/dev-packages

Willy
---
Complete changelog :
Amaury Denoyelle (13):
  BUG/MAJOR: quic: fix padding with short packets
  SCRIPTS: git-show-backports: do not truncate git-show output
  BUG/MINOR: h3: fix crash on STOP_SENDING receive after GOAWAY emission
  BUG/MINOR: mux-quic: fix crash on qcs SD alloc failure
  BUG/MINOR: h3: fix BUG_ON() crash on control stream alloc failure
  BUG/MINOR: quic: fix BUG_ON() on Tx pkt alloc failure
  BUG/MEDIUM: h3: ensure the ":method" pseudo header is totally valid
  BUG/MEDIUM: h3: ensure the ":scheme" pseudo header is totally valid
  BUG/MEDIUM: quic: fix race-condition in quic_get_cid_tid()
  BUG/MINOR: quic: fix race condition in qc_check_dcid()
  BUG/MINOR: quic: fix race-condition on trace for CID retrieval
  BUG/MEDIUM: server: fix race on server_atomic_sync()
  DEV: flags/quic: decode quic_conn flags

Aurelien DARRAGON (8):
  BUG/MINOR: log: fix broken '+bin' logformat node option
  DEBUG: hlua: distinguish burst timeout errors from exec timeout errors
  BUG/MEDIUM: proxy: fix email-alert invalid free
  DOC: management: document ptr lookup for table commands
  DOC: api/event_hdl: small updates, fix an example and add some precisions
  BUG/MINOR: hlua: report proper context upon error in 
hlua_cli_io_handler_fct()
 

Re: [ANNOUNCE] haproxy-3.1-dev3

[Willy Tarreau]

Hi Lukas,

On Thu, Jul 11, 2024 at 12:17:53PM +0200, Lukas Tribus wrote:
> Hi,
> 
> I will get back to this for further research and discussion in about a week.

OK! In the mean time I'll revert the pending patches from 3.0 so that
we can issue 3.0.3 without them.

> In the meantime, do we agree that the environment we are developing the fix
> for is the following:
> 
> the hard limit is always set to the maximum available in the kernel which
> on amd64 is one billion with a B, whether the systems has 128M or 2T of
> memory is irrelevant.
> 
> You agree that this is the environment systemd sets us up with, right?

I'm having a doubt by not being certain I'm parsing the question correctly :-)

Just to rephrase the goal here, it's to make sure that when the service
is started with extreme (and absurd) limits, we don't use all of what is
offered but only a smaller subset that matches what was commonly encountered
till now.

So if we start with 1B fd regardless of the amount of RAM, we want to
limit to a lower value so as not to OOM for no reason. One FD takes
64 bytes. Starting haproxy with 1M FDs here takes ~80 MB, which I
consider not that extreme by todays sizes, and that in practice few
people have reported problems about. Of course establishing connections
on all of these will use way more memory but that already depends on
the traffic pattern and configuration (SSL front/back, etc).

My take on this limit is that most users should not care. Those dealing
with high loads have to do their homework and are used to doing this,
and those deploying in extremely small environments are also used to
adjusting limits (even sometimes rebuilding with specific options), and
I'm fine with leaving a bit of work for both extremities.

Willy

Re: [ANNOUNCE] haproxy-3.1-dev3

[Lukas Tribus]

Hi,

I will get back to this for further research and discussion in about a week.

In the meantime, do we agree that the environment we are developing the fix
for is the following:

the hard limit is always set to the maximum available in the kernel which
on amd64 is one billion with a B, whether the systems has 128M or 2T of
memory is irrelevant.

You agree that this is the environment systemd sets us up with, right?


Thanks,
Lukas

Re: [ANNOUNCE] haproxy-3.1-dev3

[Willy Tarreau]

Hi Lukas,

and first, many thanks for sharing your thoughts and opinions on this.

[ responding to both of your messages at once ]

On Wed, Jul 10, 2024 at 09:30:55PM +0200, Lukas Tribus wrote:
> On Wed, 10 Jul 2024 at 16:39, Willy Tarreau  wrote:
> >
> > Another change that will need to be backported after some time concerns
> > the handling of default FD limits.
(...)
> I wholeheartedly hate default implicit limits and I also pretty much
> disagree with fd-hard-limit in general, but allow me to quote your own
> post here from github issue #2043 comment
> https://github.com/haproxy/haproxy/issues/2043#issuecomment-1433593837

I don't like having had to deal with such limits for ~23 years now but
the facts is that it's one of the strict, non-bypassable, system-imposed
limits. The problem is that while the vast majority of users don't care
about the number of FDs, this value cannot be changed at runtime and does
have serious implications on RAM usage and even ou ability to accept
connections that we're engaging in processing cleanly. So in any case we
need to respect a limit, and for this we have to compose with what operating
systems are doing.

For decades they would present 1024 soft and more hard, but not that much
(e.g. 4k) and it was needed to start as root to go beyond. Then some OSes
started to expose much higher hard values by default (256k to 1M) so that
it was no longer requird to be root to start a service. During this time,
such limits were more or less tailored around RAM sizing. Now it seems
we're reaching a limit, with extreme values being advertised without any
relation with allocated RAM. I think that containers are part of the cause
of this.

> > we used to have a 2k maxconn limit for a very long time and it was causing
> > much more harm than such an error: the process used to start well and was
> > working perfectly fine until the day there was a big rush on the site and it
> > wouldn't accept more connections than the default limit. I'm not that much
> > tempted by setting new high default limits. We do have some users running
> > with 2+ million concurrent connections, or roughly 5M FDs. That's already 
> > way
> > above what most users would consider an acceptable default limit, and 
> > anything
> > below this could mean that such users wouldn't know about the setting and 
> > could
> > get trapped.
> 
> I disagree that we need to heuristically guess those values like I
> believe I said in the past.

My problem is that such limit *does* exist (and if you look at ulimit -a,
it's one of the rare ones that's never unlimited), we have to apply a
value, with too low one we reject traffic at the worst possible moments
(when there are the most possible witnesses of your site falling down)
and with too high one we cannot start anymore. Limits are imposed to the
process and it needs to work within.

> "But containers ..." should not be an argument to forgo the principle
> of least surprise.

I agree even though they're part of the problem (but no longer the only
one).

> There are ways to push defaults like this out if really needed: with
> default configuration files, like we have in examples/ and like
> distributions provide in their repositories. This default the users
> will then find in the configuration file and can look it up in the
> documentation if they want.

I'm not against encouraging users to find sane limits in the config
files they copy-paste all over the place. Just like I think that if
systemd starts to advertise very large values, we should probably
encourage to ship unit files setting the hard limit to 1M or so (i.e.
the previously implicit hard value presented to the daemon).

> At the very least we need a *stern* configuration warning that we now
> default to 1M fd, although I would personally consider this (lack of
> all fd-hard-limit, ulimit and global maxconn) leading to heuristic
> fd-hard-limit a critical error.

When Valentine, who worked on the patch, talked to me about the problem,
these were among the possibilities we thought about. I initially
disagreed with the error because I considered that having to set yet
another limit to keep your config running is quite a pain (a long time
ago users were really bothered by the relation between ulimit-n and
maxconn). But I was wrong on one point, I forgot that fd-hard-limit only
applies when maxconn/ulimit-n etc are not set, so that wouldn't affect
users who already set their values correctly.

> I also consider backporting this change - even with a configuration
> warning - dangerous.

I know, but we don't decide what distro users start their stable version
on :-/

> So here a few proposals:
> 
> Proposal 1:
> 
> - remove fd-hard-limit as it was a confusing mistake 

Re: [ANNOUNCE] haproxy-3.1-dev3

[Lukas Tribus]

On Wed, 10 Jul 2024 at 21:30, Lukas Tribus  wrote:
> There are ways to push defaults like this out if really needed: with
> default configuration files, like we have in examples/ and like
> distributions provide in their repositories. This default the users
> will then find in the configuration file and can look it up in the
> documentation if they want.


So here a few proposals:

Proposal 1:

- remove fd-hard-limit as it was a confusing mistake in the first place
- exit with a configuration error when global maxconn is not set
- put global maxconn in all example configurations, encourage
Debian/RH to do the same
- document accordingly


Proposal 2:

- keep fd-hard-limit
- exit with a configuration error when fd-hard-limit needs to guess 1M
- put fd-hard-limit in all example configurations, encourage Debian/RH
to do the same
- document accordingly



Otherwise the next bug report will be that haproxy OOM's (in
production and only when encountering load) by default with systems
with less than 16 GB of RAM. The same bug reporter just needs a VM
with 8 GB RAM or less.



Sometimes the hard choices need to be up to the user. I believe this
is some of those times.


cheers
lukas

Re: [ANNOUNCE] haproxy-3.1-dev3

[Lukas Tribus]

Hello,


On Wed, 10 Jul 2024 at 16:39, Willy Tarreau  wrote:
>
> Another change that will need to be backported after some time concerns
> the handling of default FD limits. For a few decades, operating systems
> would advertise safe limits (i.e. those they were able to deal with based
> on their amount of RAM). We've seen a few attempts at bumping the hard
> limit beyond 1 billion FDs many years ago that were reverted due to
> breaking many apps. Now it seems it's coming back, via systemd-256 setting
> the hard-limit from the kernel's nr_open variable (which in itself is not
> necessarily a bad thing -- proof that I'm not always bashing systemd, only
> when needed :-)). But with some machines showing extreme nr_open (I still
> don't know why) we're back to square one where it's possible for haproxy
> to try to start with a limit set to one billion FDs. Not only this would
> eat at least 64GB of RAM just for the fdtab itself, it also takes ages to
> start, and fortunately the watchdog quickly puts an end to this mess...
> We already have an fd-hard-limit global setting that allows to fix a hard
> limit to the number of FDs, but not everyone knows about it nor uses it.
> What we've found to be the most reasonable is to consider that
> fd-hard-limit now has a default value of 1048576, which matches what was
> almost always the default hard limit, so that when not set, it's like it
> used to be till now. That's sufficient for the vast majority of use cases,
> and trust me, the rare users who need to support more than 500k concurrent
> connections are pretty much aware of all related tunables and already make
> use of them, so it's expected that nobody should observe any change.

I wholeheartedly hate default implicit limits and I also pretty much
disagree with fd-hard-limit in general, but allow me to quote your own
post here from github issue #2043 comment
https://github.com/haproxy/haproxy/issues/2043#issuecomment-1433593837

> we used to have a 2k maxconn limit for a very long time and it was causing
> much more harm than such an error: the process used to start well and was
> working perfectly fine until the day there was a big rush on the site and it
> wouldn't accept more connections than the default limit. I'm not that much
> tempted by setting new high default limits. We do have some users running
> with 2+ million concurrent connections, or roughly 5M FDs. That's already way
> above what most users would consider an acceptable default limit, and anything
> below this could mean that such users wouldn't know about the setting and 
> could
> get trapped.

I disagree that we need to heuristically guess those values like I
believe I said in the past.

"But containers ..." should not be an argument to forgo the principle
of least surprise.

There are ways to push defaults like this out if really needed: with
default configuration files, like we have in examples/ and like
distributions provide in their repositories. This default the users
will then find in the configuration file and can look it up in the
documentation if they want.

At the very least we need a *stern* configuration warning that we now
default to 1M fd, although I would personally consider this (lack of
all fd-hard-limit, ulimit and global maxconn) leading to heuristic
fd-hard-limit a critical error.

I also consider backporting this change - even with a configuration
warning - dangerous.


cheers,
lukas

Re: [ANNOUNCE] haproxy-3.1-dev3

[Willy Tarreau]

On Wed, Jul 10, 2024 at 06:49:52PM +0200, Aleksandar Lazic wrote:
> 
> 
> On 2024-07-10 (Mi.) 16:39, Willy Tarreau wrote:
> > Hi,
> > 
> > HAProxy 3.1-dev3 was released on 2024/07/10. It added 35 new commits
> > after version 3.1-dev2.
> 
> [snipp]
> 
> > And I'm still trying to free some time for the pending reviews (I have not
> > forgotten you but stuff that depends on multiple persons cannot always
> > wait).
> 
> There is no hurry about the connect patch. I have created in the meantime
> another solution in rust. :-)
> 
> It's no the best code in the world but it solves my issue.
> https://github.com/git001/tls-proxy-tunnel/
> 
> So take your time for review.

Thanks for your kind understanding, Alex!
Willy

Re: [ANNOUNCE] haproxy-3.1-dev3

[Aleksandar Lazic]

On 2024-07-10 (Mi.) 16:39, Willy Tarreau wrote:

Hi,

HAProxy 3.1-dev3 was released on 2024/07/10. It added 35 new commits
after version 3.1-dev2.


[snipp]


And I'm still trying to free some time for the pending reviews (I have not
forgotten you but stuff that depends on multiple persons cannot always
wait).


There is no hurry about the connect patch. I have created in the meantime 
another solution in rust. :-)


It's no the best code in the world but it solves my issue.
https://github.com/git001/tls-proxy-tunnel/

So take your time for review.

Regards
Alex

[ANNOUNCE] haproxy-3.1-dev3

[Willy Tarreau]

Hi,

HAProxy 3.1-dev3 was released on 2024/07/10. It added 35 new commits
after version 3.1-dev2.

This version mostly addresses some issues affecting dev2, a few of which
could cause crashes. Some of them are marked for backporting because they
also affect older versions (races between server address updates and
deletion, and a few things like this).

There's a fix for the handling of the Transfer-encoding header in the H1
mux that was reported by Ben Kallus. It's believed to be harmless as the
only server that was found to be confused cannot parse a second request,
but we'll backport it soon.

Another change that will need to be backported after some time concerns
the handling of default FD limits. For a few decades, operating systems
would advertise safe limits (i.e. those they were able to deal with based
on their amount of RAM). We've seen a few attempts at bumping the hard
limit beyond 1 billion FDs many years ago that were reverted due to
breaking many apps. Now it seems it's coming back, via systemd-256 setting
the hard-limit from the kernel's nr_open variable (which in itself is not
necessarily a bad thing -- proof that I'm not always bashing systemd, only
when needed :-)). But with some machines showing extreme nr_open (I still
don't know why) we're back to square one where it's possible for haproxy
to try to start with a limit set to one billion FDs. Not only this would
eat at least 64GB of RAM just for the fdtab itself, it also takes ages to
start, and fortunately the watchdog quickly puts an end to this mess...
We already have an fd-hard-limit global setting that allows to fix a hard
limit to the number of FDs, but not everyone knows about it nor uses it.
What we've found to be the most reasonable is to consider that
fd-hard-limit now has a default value of 1048576, which matches what was
almost always the default hard limit, so that when not set, it's like it
used to be till now. That's sufficient for the vast majority of use cases,
and trust me, the rare users who need to support more than 500k concurrent
connections are pretty much aware of all related tunables and already make
use of them, so it's expected that nobody should observe any change.

I think that after some time we'll start to backport it, at least to the
latest LTS (3.0) so as to make distro packagers' lives easier. We may have
to backport it to older versions as well if they ask for it. As such,
feedback on this version, especially from those with large setups such as
Ricardo, Felipe and Christian, would be much appreciated so that we know
if any corner cases need to be adjusted for example.

Among other changes, the mt_lists that were added and reverted from 2.9
were finally fixed and added back. This time I'm no longer observing the
massive gain with QUIC because the stack has improved since then and
likely puts less pressure on memory writes; the perf is now well beyond
what it was with the fix a year ago. That's good news in any case.

Some doc updates (the obsolete architecture guide that I thought dropped
was still lying there after 18 years, now I got rid of it), cleanups, and
an attempt from Ilya at running the QUIC interop in the weekly CI, this
can definitely be cool, we'll see how it goes.

Some more important changes are coming soon, so I expect that next
versions will become a bit funkier. Thus if you want to give a test to
3.1-dev but don't feel like taking too many risks, this one should be the
best choice for a while.

And I'm still trying to free some time for the pending reviews (I have not
forgotten you but stuff that depends on multiple persons cannot always
wait).

Please find the usual URLs below :
   Site index   : https://www.haproxy.org/
   Documentation: https://docs.haproxy.org/
   Wiki : https://github.com/haproxy/wiki/wiki
   Discourse: https://discourse.haproxy.org/
   Slack channel: https://slack.haproxy.org/
   Issue tracker: https://github.com/haproxy/haproxy/issues
   Sources  : https://www.haproxy.org/download/3.1/src/
   Git repository   : https://git.haproxy.org/git/haproxy.git/
   Git Web browsing : https://git.haproxy.org/?p=haproxy.git
   Changelog: https://www.haproxy.org/download/3.1/src/CHANGELOG
   Dataplane API: 
https://github.com/haproxytech/dataplaneapi/releases/latest
   Pending bugs : https://www.haproxy.org/l/pending-bugs
   Reviewed bugs: https://www.haproxy.org/l/reviewed-bugs
   Code reports : https://www.haproxy.org/l/code-reports
   Latest builds: https://www.haproxy.org/l/dev-packages

Willy
---
Complete changelog :
Amaury Denoyelle (11):
  BUG/MEDIUM: server: fix race on server_atomic_sync()
  CLEANUP: quic: cleanup prototypes related to CIDs handling
  CLEANUP: quic: remove non-existing quic_cid_tree definition
  MINOR: quic: remove access to CID global tree outside of qu

【三井住友カード】不正利用からアカウントを保護するために追加情報が必要haproxy

[haproxy]

三井住友カード会員　様

いつも三井住友カードをご利用頂きありがとうございます。

カードがご利用いただけなかったお取引がございましたのでお知らせいたします。

◇利用カード：三井住友カード
◇利用日：2024/07/04 5:04:33
◇利用先：ＴＯＹＯＴＡＷａｌｌｅｔチャージ
◇利用金額：50,000円

不正利用検知システムで第三者による不正の可能性を検知したため、上記取引を制限させていただきました。
以下の回答ページよりご利用内容の確認にご協力ください。

▼回答ページはこちら（所要時間：30秒）

  3024.szsmxf.com/5205

URLをコピーしてWebブラウザーのアドレスバーに貼り付けると正しくWebサイトが開きますご


不便とご心配をおかけしまして誠に申し訳ございませんが、

何とぞご理解易りたくお願い申しあげます

【ご本人さまのご利用の場合】
ご利用制限を解除します。
＊上記取引は不成立のため、利用制限解除後に再決済が必要です。

【ご利用に覚えがない場合】 カードの停止（無効化）・再発行手続きへ進みます。

＊ご利用確認の回答時に個人情報を入力いただくことはございません
※一部、状況やカードの種類によって再発行手続きが完了しない場合がございます。
＊すでにSMS等でご回答いただいた場合は、本メールからのご回答は不要です。

※このメールアドレスは送信専用です。ご返信に回答できません。
━━━


■発行者
三井住友カード株式会社
〒100-6150 東京都千代田区永田町2丁目11番1号 山王パークタワー
お問い合わせは こちら
※送信元「ｍａｉｌ＠ｃｏｎｔａｃｔ．ｖｐａｓｓ．ｎｅ．ｊｐ」は送信専用です。
※本メールは重要なお知らせのため、メール配信を「否」にされている方にも送信しております。

━━━

Re: [ANNOUNCE] haproxy-3.1-dev2

[William Lallemand]

On Wed, Jul 03, 2024 at 11:18:12PM +0200, Илья Шипицин wrote:
> Subject: Re: [ANNOUNCE] haproxy-3.1-dev2
> сб, 29 июн. 2024 г. в 12:01, Willy Tarreau :
> 
> > Hi,
> >
> > HAProxy 3.1-dev2 was released on 2024/06/29. It added 45 new commits
> > after version 3.1-dev1.
> >
> > Several fixes related to the recent 3.0 release are present in this
> > version, as well as a few older ones. Most visible are three QUIC
> > crashes, a possible double free on stick-tables, issues affecting
> > aws-lc with ECDSA, a risk of server flapping when DNS resolution
> > times out, and the usual amount of small fixes all over the place.
> >
> > In addition we've been notified by Yuki Mogi of FFRI Security that
> > some of our pseudo-headers in H3 were not sanitized enough and that
> > these could theoretically be abused with some severely non-compliant
> > backend servers, so this was fixed as well. This will be backported
> > to the next stable versions as it's not dramatic enough to warrant a
> > release on its own.
> >
> > The "show dev" command was improved to report command-line args as
> > well as boot & current capabilities. Linux capabilities switched to
> > v3 instead of v1 in order to avoid a deprecation warning on recent
> > kernels. We've checked the impacts and apparently there are none,
> > beyond the lack of support of kernels prior to 2.6.26, so we'll soon
> > backport it to 3.0 so as to silence the annoying warning users are
> > facing.
> >
> > A date converter was added to take an HTTP date on input and produce
> > a UNIX timestamp on output. This will help calculate expiration delays
> > for example.
> >
> > The sigalg feature was added for aws-lc, which, I think, should now
> > match 1:1 the features level of openssl (unless I'm missing something).
> >
> 
> it is VERY close to openssl. however there are few gaps.
> the one I'm aware of is
> 
> haproxy/include/haproxy/quic_tls.h at master · haproxy/haproxy (github.com)
> <https://github.com/haproxy/haproxy/blob/master/include/haproxy/quic_tls.h#L143-L154>
> 
> aws-lc implements chacha20_poly1305 in a different way than QuicTLS.
> and if that gap is eliminated, it will be a good point to declare aws-lc as
> a recommended QUIC lib.
> 
> if we compare aws-lc against openssl-1.1 (not quictls), it is indeed
> matches (maybe except some niche features like async)
> 

BoringSSL, LibreSSL, and AWS-LC uses the EVP_AEAD API, basically we would need
to use EVP_aead_chacha20_poly1305() instead of EVP_chacha20_poly1305(), and
EVP_aead_chacha20_poly1305() instead of EVP_chacha20() and uses the EVP_AEAD
functions. IMHO this is not a huge change and this is doable, we just need to
invest some time on it.

Regards,

-- 
William Lallemand

Re: [ANNOUNCE] haproxy-3.1-dev2

[Илья Шипицин]

сб, 29 июн. 2024 г. в 12:01, Willy Tarreau :

> Hi,
>
> HAProxy 3.1-dev2 was released on 2024/06/29. It added 45 new commits
> after version 3.1-dev1.
>
> Several fixes related to the recent 3.0 release are present in this
> version, as well as a few older ones. Most visible are three QUIC
> crashes, a possible double free on stick-tables, issues affecting
> aws-lc with ECDSA, a risk of server flapping when DNS resolution
> times out, and the usual amount of small fixes all over the place.
>
> In addition we've been notified by Yuki Mogi of FFRI Security that
> some of our pseudo-headers in H3 were not sanitized enough and that
> these could theoretically be abused with some severely non-compliant
> backend servers, so this was fixed as well. This will be backported
> to the next stable versions as it's not dramatic enough to warrant a
> release on its own.
>
> The "show dev" command was improved to report command-line args as
> well as boot & current capabilities. Linux capabilities switched to
> v3 instead of v1 in order to avoid a deprecation warning on recent
> kernels. We've checked the impacts and apparently there are none,
> beyond the lack of support of kernels prior to 2.6.26, so we'll soon
> backport it to 3.0 so as to silence the annoying warning users are
> facing.
>
> A date converter was added to take an HTTP date on input and produce
> a UNIX timestamp on output. This will help calculate expiration delays
> for example.
>
> The sigalg feature was added for aws-lc, which, I think, should now
> match 1:1 the features level of openssl (unless I'm missing something).
>

it is VERY close to openssl. however there are few gaps.
the one I'm aware of is

haproxy/include/haproxy/quic_tls.h at master · haproxy/haproxy (github.com)
<https://github.com/haproxy/haproxy/blob/master/include/haproxy/quic_tls.h#L143-L154>

aws-lc implements chacha20_poly1305 in a different way than QuicTLS.
and if that gap is eliminated, it will be a good point to declare aws-lc as
a recommended QUIC lib.

if we compare aws-lc against openssl-1.1 (not quictls), it is indeed
matches (maybe except some niche features like async)


>
> Some hints were added to crash outputs to suggest how to decode the
> core file and what to report.
>
> And the rest is a start of files reorganization, cleanups and doc updates.
>
> Really nothing truly exciting but it's only dev2, be patient :-)
>
> Regarding the MPTCP and CONNECT patches that have been floating around,
> I just couldn't assign a single minute to them since my return from
> vacation. Maybe I'll have an eye on one of them next week, may be the
> week after. Don't worry, they're not forgotten. I'd also like that we
> revive Tristan's abns change proposal for 3.1, now that we're no longer
> at the end of a cycle. I'll also try better next time to advance the
> release to the middle of the week but this time it didn't work.
>
> Please find the usual URLs below :
>Site index   : https://www.haproxy.org/
>    Documentation    : https://docs.haproxy.org/
>Wiki : https://github.com/haproxy/wiki/wiki
>Discourse: https://discourse.haproxy.org/
>Slack channel: https://slack.haproxy.org/
>Issue tracker: https://github.com/haproxy/haproxy/issues
>Sources  : https://www.haproxy.org/download/3.1/src/
>Git repository   : https://git.haproxy.org/git/haproxy.git/
>Git Web browsing : https://git.haproxy.org/?p=haproxy.git
>Changelog: https://www.haproxy.org/download/3.1/src/CHANGELOG
>Dataplane API:
> https://github.com/haproxytech/dataplaneapi/releases/latest
>Pending bugs : https://www.haproxy.org/l/pending-bugs
>Reviewed bugs: https://www.haproxy.org/l/reviewed-bugs
>Code reports : https://www.haproxy.org/l/code-reports
>Latest builds: https://www.haproxy.org/l/dev-packages
>
> Willy
> ---
> Complete changelog :
> Amaury Denoyelle (12):
>   BUG/MAJOR: quic: fix padding with short packets
>   BUG/MAJOR: quic: do not loop on emission on closing/draining state
>   SCRIPTS: git-show-backports: do not truncate git-show output
>   BUG/MINOR: h3: fix crash on STOP_SENDING receive after GOAWAY
> emission
>   BUG/MINOR: mux-quic: fix crash on qcs SD alloc failure
>   BUG/MINOR: h3: fix BUG_ON() crash on control stream alloc failure
>   BUG/MINOR: quic: fix BUG_ON() on Tx pkt alloc failure
>   BUG/MEDIUM: h3: ensure the ":method" pseudo header is totally valid
>   BUG/MEDIUM: h3: ensure the ":scheme" pseudo header is totally valid
>   BUG/MEDIUM: quic: fix race-condition in qu

[ANNOUNCE] haproxy-3.1-dev2

[Willy Tarreau]

Hi,

HAProxy 3.1-dev2 was released on 2024/06/29. It added 45 new commits
after version 3.1-dev1.

Several fixes related to the recent 3.0 release are present in this
version, as well as a few older ones. Most visible are three QUIC
crashes, a possible double free on stick-tables, issues affecting
aws-lc with ECDSA, a risk of server flapping when DNS resolution
times out, and the usual amount of small fixes all over the place.

In addition we've been notified by Yuki Mogi of FFRI Security that
some of our pseudo-headers in H3 were not sanitized enough and that
these could theoretically be abused with some severely non-compliant
backend servers, so this was fixed as well. This will be backported
to the next stable versions as it's not dramatic enough to warrant a
release on its own.

The "show dev" command was improved to report command-line args as
well as boot & current capabilities. Linux capabilities switched to
v3 instead of v1 in order to avoid a deprecation warning on recent
kernels. We've checked the impacts and apparently there are none,
beyond the lack of support of kernels prior to 2.6.26, so we'll soon
backport it to 3.0 so as to silence the annoying warning users are
facing.

A date converter was added to take an HTTP date on input and produce
a UNIX timestamp on output. This will help calculate expiration delays
for example.

The sigalg feature was added for aws-lc, which, I think, should now
match 1:1 the features level of openssl (unless I'm missing something).

Some hints were added to crash outputs to suggest how to decode the
core file and what to report.

And the rest is a start of files reorganization, cleanups and doc updates.

Really nothing truly exciting but it's only dev2, be patient :-)

Regarding the MPTCP and CONNECT patches that have been floating around,
I just couldn't assign a single minute to them since my return from
vacation. Maybe I'll have an eye on one of them next week, may be the
week after. Don't worry, they're not forgotten. I'd also like that we
revive Tristan's abns change proposal for 3.1, now that we're no longer
at the end of a cycle. I'll also try better next time to advance the
release to the middle of the week but this time it didn't work.

Please find the usual URLs below :
   Site index   : https://www.haproxy.org/
   Documentation: https://docs.haproxy.org/
   Wiki : https://github.com/haproxy/wiki/wiki
   Discourse: https://discourse.haproxy.org/
   Slack channel: https://slack.haproxy.org/
   Issue tracker: https://github.com/haproxy/haproxy/issues
   Sources  : https://www.haproxy.org/download/3.1/src/
   Git repository   : https://git.haproxy.org/git/haproxy.git/
   Git Web browsing : https://git.haproxy.org/?p=haproxy.git
   Changelog: https://www.haproxy.org/download/3.1/src/CHANGELOG
   Dataplane API: 
https://github.com/haproxytech/dataplaneapi/releases/latest
   Pending bugs : https://www.haproxy.org/l/pending-bugs
   Reviewed bugs: https://www.haproxy.org/l/reviewed-bugs
   Code reports : https://www.haproxy.org/l/code-reports
   Latest builds: https://www.haproxy.org/l/dev-packages

Willy
---
Complete changelog :
Amaury Denoyelle (12):
  BUG/MAJOR: quic: fix padding with short packets
  BUG/MAJOR: quic: do not loop on emission on closing/draining state
  SCRIPTS: git-show-backports: do not truncate git-show output
  BUG/MINOR: h3: fix crash on STOP_SENDING receive after GOAWAY emission
  BUG/MINOR: mux-quic: fix crash on qcs SD alloc failure
  BUG/MINOR: h3: fix BUG_ON() crash on control stream alloc failure
  BUG/MINOR: quic: fix BUG_ON() on Tx pkt alloc failure
  BUG/MEDIUM: h3: ensure the ":method" pseudo header is totally valid
  BUG/MEDIUM: h3: ensure the ":scheme" pseudo header is totally valid
  BUG/MEDIUM: quic: fix race-condition in quic_get_cid_tid()
  BUG/MINOR: quic: fix race condition in qc_check_dcid()
  BUG/MINOR: quic: fix race-condition on trace for CID retrieval

Aurelien DARRAGON (11):
  BUG/MINOR: log: fix broken '+bin' logformat node option
  DEBUG: hlua: distinguish burst timeout errors from exec timeout errors
  BUG/MEDIUM: proxy: fix email-alert invalid free
  REORG: mailers: move free_email_alert() to mailers.c
  BUG/MINOR: proxy: fix email-alert leak on deinit() (2nd try)
  DOC: management: document ptr lookup for table commands
  DOC: api/event_hdl: small updates, fix an example and add some precisions
  BUG/MINOR: hlua: report proper context upon error in 
hlua_cli_io_handler_fct()
  MINOR: cfgparse/log: remove leftover dead code
  BUG/MINOR: server: fix first server template name lookup UAF
  BUG/MEDIUM: server/dns: prevent DOWN/UP flap upon resolution timeout or 
error

Christopher Faulet (2):
  BUG/MEDIUM: stick-table: Decre

[ANNOUNCE] haproxy-2.4.27

[Amaury Denoyelle]

Hi,

HAProxy 2.4.27 was released on 2024/06/18. It added 43 new commits after
version 2.4.26.

This version follows the first release of 3.0. Here is a summary of the
most notable changes.

One fix was applied for better HTTP error reporting. In some cases, 502
server initial errors were incorrectly hidden and are now properly
logged.

On the backend side, an issue was found when NTLM headers are used. This
caused the backend connection to be marked dynamically as private to
prevent HTTP reuse. However, this is conceptually wrong when using
HTTP/2 multiplexer on the backend side with http-reuse mode set to
aggressive or higher, as this connection can already be shared accross
several clients. Thus, NTLM headers are simply ignored in this case.

For the SSL stack, cipher algorithm negotiation was adjusted as haproxy
could have chosen an ECDSA certificate even if not compatible with
client algorithms instead of fallback to RSA.

Cache hits should be increased as previously cached HTTP responses which
used Vary header on anything other than Accept-encoding but with
Encoding header present were never returned from the cache.

On the LUA side, a serie of cleanups and minor bugfixes are merged. Most
of them are relevant to error handling which may improve script
debugging.

A Solaris user reported that external checks were causing an infinite
loop. In fact, this was due to a wrong signal handling in evports,
Solaris polling mechanism, present since its first introduction in
haproxy.

Thanks to everyone who contributed to this release.

Please find the usual URLs below :
   Site index   : https://www.haproxy.org/
   Documentation: https://docs.haproxy.org/
   Wiki : https://github.com/haproxy/wiki/wiki
   Discourse: https://discourse.haproxy.org/
   Slack channel: https://slack.haproxy.org/
   Issue tracker: https://github.com/haproxy/haproxy/issues
   Sources  : https://www.haproxy.org/download/2.4/src/
   Git repository   : https://git.haproxy.org/git/haproxy-2.4.git/
   Git Web browsing : https://git.haproxy.org/?p=haproxy-2.4.git
   Changelog: https://www.haproxy.org/download/2.4/src/CHANGELOG
   Dataplane API: 
https://github.com/haproxytech/dataplaneapi/releases/latest
   Pending bugs : https://www.haproxy.org/l/pending-bugs
   Reviewed bugs: https://www.haproxy.org/l/reviewed-bugs
   Code reports : https://www.haproxy.org/l/code-reports
   Latest builds: https://www.haproxy.org/l/dev-packages

---
Complete changelog :
Amaury Denoyelle (4):
  BUG/MINOR: backend: use cum_sess counters instead of cum_conn
  BUG/MINOR: connection: parse PROXY TLV for LOCAL mode
  BUG/MEDIUM: http_ana: ignore NTLM for reuse aggressive/always and no H1
  BUG/MAJOR: connection: fix server used_conns with H2 + reuse safe

Aurelien DARRAGON (10):
  BUG/MEDIUM: thread/sched: set proper scheduling context upon ha_set_tid()
  BUG/MINOR: log: fix lf_text_len() truncate inconsistency
  BUG/MINOR: tools/log: invalid encode_{chunk,string} usage
  BUG/MINOR: log: invalid snprintf() usage in sess_build_logline()
  BUG/MEDIUM: fd: prevent memory waste in fdtab array
  CLEANUP: hlua: use hlua_pusherror() where relevant
  BUG/MINOR: hlua: don't use lua_pushfstring() when we don't expect LJMP
  BUG/MINOR: hlua: fix unsafe hlua_pusherror() usage
  BUG/MINOR: hlua: prevent LJMP in hlua_traceback()
  CLEANUP: hlua: simplify ambiguous lua_insert() usage in hlua_ctx_resume()

Christopher Faulet (7):
  MINOR: cli: Remove useless loop on commands to find unescaped semi-colon
  BUG/MEDIUM: cli: Warn if pipelined commands are delimited by a \n
  BUG/MEDIUM: http-ana: Deliver 502 on keep-alive for fressh server 
connection
  BUG/MEDIUM: stconn: Don't forward channel data if input data must be 
filtered
  BUG/MINOR: htpp-ana/stats: Specify that HTX redirect messages have a C-L 
header
  BUG/MINOR: stats: Don't state the 303 redirect response is chunked
  BUG/MINOR: server: Don't reset resolver options on a new default-server 
line

Ilya Shipitsin (2):
  CI: revert kernel addr randomization introduced in 3a0fc864
  CI: introduce scripts/build-vtest.sh for installing VTest

Remi Tricot-Le Breton (1):
  BUG/MEDIUM: cache: Vary not working properly on anything other than 
accept-encoding

Valentine Krasnobaeva (3):
  BUG/MINOR: ssl/ocsp: init callback func ptr as NULL
  BUG/MINOR: activity: fix Delta_calls and Delta_bytes count
  BUG/MINOR: haproxy: only tid 0 must not sleep if got signal

William Lallemand (2):
  CLEANUP: ssl/cli: remove unused code in dump_crtlist_conf
  BUG/MEDIUM: ssl: wrong priority whem limiting ECDSA ciphers in ECDSA+RSA 
configuration

Willy Tarreau (14):
  BUG/MEDIUM: cli: fix once for all the problem of missing trailing LFs
  BUG/MEDIUM: peers/trace: fix crash when listing event types
  BUG/MEDIUM: evports: do 

[ANNOUNCE] haproxy-2.6.18

[Amaury Denoyelle]

Hi,

HAProxy 2.6.18 was released on 2024/06/18. It added 82 new commits after
version 2.6.17.

This version follows the first release of 3.0. Here is a summary of the
most notable changes.

Several fixes are applied for better HTTP conformance. In some cases,
502 server initial errors were incorrectly hidden and are now properly
logged.  CONNECT requests with a scheme are now rejected as they are
invalid according to RFC 7230. Empty paths are normalized to "/" for
aboslute-form URI.

Dynamic servers testing under heavy load have been performed during 3.0
development cycle. This revealed that crashes could occur due to the
removal of a server currently in used. Removal conditions were thus
adjusted to reject such operation. Also, some settings were not
completely initialized for dynamic servers which cause a difference of
behavior with static ones.

Still on the backend side, an issue was found when NTLM headers are
used. This caused the backend connection to be marked dynamically as
private to prevent HTTP reuse. However, this is conceptually wrong when
using HTTP/2 multiplexer on the backend side with http-reuse mode set to
aggressive or higher, as this connection can already be shared accross
several clients. Thus, NTLM headers are simply ignored in this case.

Minor fixes were merged for QUIC. Most of them are related to improve
the LibreSSL compatibility. Other than that, error handling was improved
to report more specific error codes from the different layers of QUIC
multiplexer, HTTP/3 or QPACK decoder.

For the SSL stack, cipher algorithm negotiation was adjusted as haproxy
could have chosen an ECDSA certificate even if not compatible with
client algorithms instead of fallback to RSA.

A bug was fixed for the peer applet where a blocking condition could
occured when reaching max-updates-at-once limit.

Cache hits should be increased as previously cached HTTP responses which
used Vary header on anything other than Accept-encoding but with
Encoding header present were never returned from the cache.

It is now possible to disable seamless reload on master-worker mode by
using the argument '-x /dev/null'. This may be necessary for some usage
since the introduction of automatic seamless reload for master-worker
mode.

An interesting security feature was backported to block traffic with
clients which use privileged port as their source port value. Such
behavior is highly suspect as it is often the sign of an amplification
attack. This can be activated using
harden.reject-privileged-ports.{tcp|quic} keywords. Note that on 3.0, we
chose to set it by default for QUIC. However, it remains disabled on 2.9
and lesser versions to keep the current behavior on stable haproxy
branches, but users are free to activate it if needed. It is
particularly useful when QUIC listeners are active to prevent DNS/NTP
amplification attack.  However, on TCP this protection may break some
protocols such as FTP.

On the LUA side, a serie of cleanups and minor bugfixes are merged. Most
of them are relevant to error handling which may improve script
debugging. Also a crash was fixed when using CacheCert module from init
context.

A Solaris user reported that external checks were causing an infinite
loop. In fact, this was due to a wrong signal handling in evports,
Solaris polling mechanism, present since its first introduction in
haproxy.

Thanks to everyone who contributed to this release.

Please find the usual URLs below :
   Site index   : https://www.haproxy.org/
   Documentation: https://docs.haproxy.org/
   Wiki : https://github.com/haproxy/wiki/wiki
   Discourse: https://discourse.haproxy.org/
   Slack channel: https://slack.haproxy.org/
   Issue tracker: https://github.com/haproxy/haproxy/issues
   Sources  : https://www.haproxy.org/download/2.6/src/
   Git repository   : https://git.haproxy.org/git/haproxy-2.6.git/
   Git Web browsing : https://git.haproxy.org/?p=haproxy-2.6.git
   Changelog: https://www.haproxy.org/download/2.6/src/CHANGELOG
   Dataplane API: 
https://github.com/haproxytech/dataplaneapi/releases/latest
   Pending bugs : https://www.haproxy.org/l/pending-bugs
   Reviewed bugs: https://www.haproxy.org/l/reviewed-bugs
   Code reports : https://www.haproxy.org/l/code-reports
   Latest builds: https://www.haproxy.org/l/dev-packages

---
Complete changelog :
Amaury Denoyelle (17):
  BUG/MEDIUM: mux-quic: report early error on stream
  BUG/MEDIUM: quic: remove unsent data from qc_stream_desc buf
  MINOR: server: allow cookie for dynamic servers
  BUG/MINOR: backend: use cum_sess counters instead of cum_conn
  BUG/MINOR: mux-quic: fix error code on shutdown for non HTTP/3
  BUG/MINOR: qpack: fix error code reported on QPACK decoding failure
  BUG/MINOR: connection: parse PROXY TLV for LOCAL mode
  MEDIUM: config: prevent communication with privileged ports
  BUG/MINOR: quic: adjust restr

Re: HAProxy 3.0 for Debian Bullseye?

[Tim Düsterhus]

Hi

On 6/16/24 14:39, Vincent Bernat wrote:

Yes, that's possible. I didn't want to do that unless there is at least
one user. I'll do it later today.



Thank you, greatly appreciated.

Best regards
Tim Düsterhus

Re: HAProxy 3.0 for Debian Bullseye?

[Vincent Bernat]

On 2024-06-16 14:37, Tim Düsterhus wrote:

Hi Vincent,

On 5/29/24 17:07, Willy Tarreau wrote:

HAProxy 3.0.0 was released on 2024/05/29. It added 21 new commits
after version 3.0-dev13. I do appreciate that everything was only
cosmetic.


I'm seeing that HAProxy 3.0 is already available on haproxy.debian.net 
since some time. Unfortunately there are only builds for Debian 
Bookworm, which comes with OpenSSL 3.0.11. I'm still holding out the 
upgrade to Bookworm due to the OpenSSL 3.x performance regressions.


Any chance to get a HAProxy 3.0 build for Debian Bullseye or is this not 
possible / too much effort on your end?


Yes, that's possible. I didn't want to do that unless there is at least 
one user. I'll do it later today.

HAProxy 3.0 for Debian Bullseye? (was: [ANNOUNCE] haproxy-3.0.0)

[Tim Düsterhus]

Hi Vincent,

On 5/29/24 17:07, Willy Tarreau wrote:

HAProxy 3.0.0 was released on 2024/05/29. It added 21 new commits
after version 3.0-dev13. I do appreciate that everything was only
cosmetic.


I'm seeing that HAProxy 3.0 is already available on haproxy.debian.net 
since some time. Unfortunately there are only builds for Debian 
Bookworm, which comes with OpenSSL 3.0.11. I'm still holding out the 
upgrade to Bookworm due to the OpenSSL 3.x performance regressions.


Any chance to get a HAProxy 3.0 build for Debian Bullseye or is this not 
possible / too much effort on your end?


Best regards
Tim Düsterhus

Re: [ANNOUNCE] haproxy-3.0.0

[Tim Düsterhus]

Hi

On 5/29/24 21:37, Willy Tarreau wrote:

- The version table on haproxy.org still has the EOL column for 2.0 in bold.
Other EOL versions are not bold, so that's inconsistent.


Ah, that makes sense, you're right. Now fixed!


I'm just noticing a few more things:

- For 3.0 the "Release date" has a tilde at the front. It should not be 
there.

- For 3.0 the link to "Bugs" is missing in the version table.

Best regards
Tim Düsterhus

Re: [ANNOUNCE] haproxy-3.1-dev1

[Christopher Faulet]

Le 14/06/2024 à 16:22, Christopher Faulet a écrit :

Le 14/06/2024 à 16:20, Willy Tarreau a écrit :

On Fri, Jun 14, 2024 at 04:12:03PM +0200, Christopher Faulet wrote:

Hi,

HAProxy 3.1-dev1 was released on 2024/06/14. It added 95 new commits
after version 3.1-dev0.

Because Willy announced publicly I should managed a -dev1 before his return
from vacations, I have no choice. So, faced with so much pressure, here is
the first 3.1 dev release.


Hehe, thank you!



J'ai buildé la dev1 sur flx04 mais je ne suis pas certain d'avoir les droits de
déployer et relancer haproxy.


Damned I replied to everyone !

--
Christopher Faulet

Re: [ANNOUNCE] haproxy-3.1-dev1

[Christopher Faulet]

Le 14/06/2024 à 16:20, Willy Tarreau a écrit :

On Fri, Jun 14, 2024 at 04:12:03PM +0200, Christopher Faulet wrote:

Hi,

HAProxy 3.1-dev1 was released on 2024/06/14. It added 95 new commits
after version 3.1-dev0.

Because Willy announced publicly I should managed a -dev1 before his return
from vacations, I have no choice. So, faced with so much pressure, here is
the first 3.1 dev release.


Hehe, thank you!



J'ai buildé la dev1 sur flx04 mais je ne suis pas certain d'avoir les droits de 
déployer et relancer haproxy.


--
Christopher Faulet

Re: [ANNOUNCE] haproxy-3.1-dev1

[Willy Tarreau]

On Fri, Jun 14, 2024 at 04:12:03PM +0200, Christopher Faulet wrote:
> Hi,
> 
> HAProxy 3.1-dev1 was released on 2024/06/14. It added 95 new commits
> after version 3.1-dev0.
> 
> Because Willy announced publicly I should managed a -dev1 before his return
> from vacations, I have no choice. So, faced with so much pressure, here is
> the first 3.1 dev release.

Hehe, thank you!

Willy

[ANNOUNCE] haproxy-3.1-dev1

[Christopher Faulet]

Hi,

HAProxy 3.1-dev1 was released on 2024/06/14. It added 95 new commits
after version 3.1-dev0.

Because Willy announced publicly I should managed a -dev1 before his return
from vacations, I have no choice. So, faced with so much pressure, here is
the first 3.1 dev release. After the feature freeze preceding the 3.0.0, the
developments now slowly restart.

Aurélien pushed several patchs related to the initial implementation of the
"log-profile" feature as described in Github issue #401. While some
directives such as "log-format", "log-format-sd", "error-log-format" or
"log-tag" make it possible to configure log formatting globally or at the
proxy level, it may be relevant to configure such settings as close as
possible to the log endpoints, that is, per "log" directive. This is what
"log-profile" section now offers when combined with a "log" directive. For
now, the use of log-profiles is somewhat limited because we lack the ability
to explicitly trigger the log building process at specific steps during the
stream handling, but it should gain more traction over the time as the
feature evolves and new mechanisms allowing the emission of logs at expected
processing steps will be added.

William, on his side, worked on improving the AWS-LC support, making closer
and closer to something isofunctional to OpenSSL-1.1.1 but with QUIC
support as a bonus. More we work on AWS-LC support, more it appears as one
of the best alternative to OpenSSL. On this release, William added the
support for ECDA+RSA certificate selection.

Finally, you may note that our acme.sh fork to be able to update dynamically
certificates without any reload was removed. The feature was merged in the
mainstream project.

The remaining is the bunch of usual bug fixes, documentation improvements,
CI upgrades and cleanups here and there. It may be good to mention that two
major bugs were fixed on the H1 multiplexer. A possible use-after-free on
the H1 connection when a request was drained and an issue with the zero-copy
data forwarding of the chunked messages. The ten first bytes of the first
chunk could be crushed, depending on the timing. So the very early adopters
and fans of the dev releases are highly encouraged to upgrade.

Thanks everyone for your help. Enjoy !

Please find the usual URLs below :
   Site index   : https://www.haproxy.org/
   Documentation: https://docs.haproxy.org/
   Wiki : https://github.com/haproxy/wiki/wiki
   Discourse: https://discourse.haproxy.org/
   Slack channel    : https://slack.haproxy.org/
   Issue tracker: https://github.com/haproxy/haproxy/issues
   Sources  : https://www.haproxy.org/download/3.1/src/
   Git repository   : https://git.haproxy.org/git/haproxy.git/
   Git Web browsing : https://git.haproxy.org/?p=haproxy.git
   Changelog: https://www.haproxy.org/download/3.1/src/CHANGELOG
   Dataplane API: 
https://github.com/haproxytech/dataplaneapi/releases/latest
   Pending bugs : https://www.haproxy.org/l/pending-bugs
   Reviewed bugs: https://www.haproxy.org/l/reviewed-bugs
   Code reports : https://www.haproxy.org/l/code-reports
   Latest builds: https://www.haproxy.org/l/dev-packages


---
Complete changelog :
Amaury Denoyelle (8):
  BUG/MINOR: quic: prevent crash on qc_kill_conn()
  BUG/MINOR: quic: fix computed length of emitted STREAM frames
  BUG/MINOR: quic: ensure Tx buf is always purged
  BUG/MINOR: quic: fix padding of INITIAL packets
  OPTIM: quic: fill whole Tx buffer if needed
  MINOR: quic: refactor qc_build_pkt() error handling
  MINOR: quic: use global datagram headlen definition
  MINOR: quic: refactor qc_prep_pkts() loop

Aurelien DARRAGON (44):
  MINOR: stktable: avoid ambiguous stktable_data_ptr() usage in 
cli_io_handler_table()
  BUG/MINOR: hlua: use CertCache.set() from various hlua contexts
  CLEANUP: hlua: fix CertCache class comment
  CLEANUP: hlua: use hlua_pusherror() where relevant
  BUG/MINOR: hlua: don't use lua_pushfstring() when we don't expect LJMP
  BUG/MINOR: hlua: fix unsafe hlua_pusherror() usage
  BUG/MINOR: hlua: prevent LJMP in hlua_traceback()
  CLEANUP: hlua: get rid of hlua_traceback() security checks
  BUG/MINOR: hlua: fix leak in hlua_ckch_set() error path
  CLEANUP: hlua: simplify ambiguous lua_insert() usage in hlua_ctx_resume()
  MINOR: log: fix "http-send-name-header" ignore warning message
  BUG/MINOR: proxy: fix server_id_hdr_name leak on deinit()
  BUG/MINOR: proxy: fix log_tag leak on deinit()
  BUG/MINOR: proxy: fix email-alert leak on deinit()
  BUG/MINOR: proxy: fix check_{command,path} leak on deinit()
  BUG/MINOR: proxy: fix dyncookie_key leak on deinit()
  BUG/MINOR: proxy: fix source interface and usesrc leaks on deinit()
  BUG/MINOR: proxy: fix header_unique_id leak on d

[ANNOUNCE] haproxy-2.8.10

[Amaury Denoyelle]

Hi,

HAProxy 2.8.10 was released on 2024/06/14. It added 80 new commits after
version 2.8.9.

This version is part of the serie of new stable releases which follows the new
3.0. Here is a summary of the most notable changes.

Several fixes are applied for better HTTP conformance. In some cases, 502
server initial errors were incorrectly hidden and are now properly logged.
CONNECT requests with a scheme are now rejected as they are invalid according
to RFC 7230. Empty paths are normalized to "/" for aboslute-form URI.

Dynamic servers testing under heavy load have been performed during 3.0
development cycle. This revealed that crashes could occur due to the removal of
a server currently in used. Removal conditions were thus adjusted to reject
such operation. Also, some settings were not completely initialized for dynamic
servers which cause a difference of behavior with static ones.

Still on the backend side, an issue was found when NTLM headers are used. This
caused the backend connection to be marked dynamically as private to prevent
HTTP reuse. However, this is conceptually wrong when using HTTP/2 multiplexer
on the backend side with http-reuse mode set to aggressive or higher, as this
connection can already be shared accross several clients. Thus, NTLM headers
are simply ignored in this case.

Minor fixes were merged for QUIC. Most of them are related to improve the
LibreSSL compatibility. Other than that, error handling was improved to report
more specific error codes from the different layers of QUIC multiplexer, HTTP/3
or QPACK decoder.

For the SSL stack, a few fixes were done, in particular for better OCSP
support. Also, cipher algorithm negotiation was adjusted as haproxy could have
chosen an ECDSA certificate even if not compatible with client algorithms
instead of fallback to RSA.

A bug was fixed for the peer applet where a blocking condition could occured
when reaching max-updates-at-once limit.

Cache hits should be increased as previously cached HTTP responses which used
Vary header on anything other than Accept-encoding but with Encoding header
present were never returned from the cache.

It is now possible to disable seamless reload on master-worker mode by using
the argument '-x /dev/null'. This may be necessary for some usage since the
introduction of automatic seamless reload for master-worker mode.

An interesting security feature was backported to block traffic with clients
which use privileged port as their source port value. Such behavior is highly
suspect as it is often the sign of an amplification attack. This can be
activated using harden.reject-privileged-ports.{tcp|quic} keywords. Note that
on 3.0, we chose to set it by default for QUIC. However, it remains disabled on
2.9 and lesser versions to keep the current behavior on stable haproxy
branches, but users are free to activate it if needed. It is particularly
useful when QUIC listeners are active to prevent DNS/NTP amplification attack.
However, on TCP this protection may break some protocols such as FTP.

Not so much on the SPOE front. A single bug-fix is merged which allowed applet
to retry later when reaching buffer exhaustion instead of returning a definite
I/O error.

On the LUA side, a serie of cleanups and minor bugfixes are merged. Most of
them are relevant to error handling which may improve script debugging. Also a
crash was fixed when using CacheCert module from init context.

The CLI applet now reports a proper error message when command larger than
buffer size is rejected. Previously, the CLI connection was silently closed
without further information.

A Solaris user reported that external checks were causing an infinite loop. In
fact, this was due to a wrong signal handling in evports, Solaris polling
mechanism, present since its first introduction in haproxy.

Thanks to everyone who contributed to this release.

Please find the usual URLs below :
   Site index   : https://www.haproxy.org/
   Documentation: https://docs.haproxy.org/
   Wiki : https://github.com/haproxy/wiki/wiki
   Discourse: https://discourse.haproxy.org/
   Slack channel: https://slack.haproxy.org/
   Issue tracker: https://github.com/haproxy/haproxy/issues
   Sources  : https://www.haproxy.org/download/2.8/src/
   Git repository   : https://git.haproxy.org/git/haproxy-2.8.git/
   Git Web browsing : https://git.haproxy.org/?p=haproxy-2.8.git
   Changelog: https://www.haproxy.org/download/2.8/src/CHANGELOG
   Dataplane API: 
https://github.com/haproxytech/dataplaneapi/releases/latest
   Pending bugs : https://www.haproxy.org/l/pending-bugs
   Reviewed bugs: https://www.haproxy.org/l/reviewed-bugs
   Code reports : https://www.haproxy.org/l/code-reports
   Latest builds: https://www.haproxy.org/l/dev-packages

---
Complete changelog :
Amaury Denoyelle (14):
  BUG/MINOR: backend: use cum_sess counters instead of cum_conn
  BUG/MINOR: mux-qui

[ANNOUNCE] haproxy-2.9.9

[Christopher Faulet]

Hi,

HAProxy 2.9.9 was released on 2024/06/14. It added 13 new commits
after version 2.9.8.

As for the 3.0, an major issue on the H1 multiplexer regarding the H1
requests draining for us to emit a new release just few days after the
previous one. This issue could lead to a crash becaues of a use-after-free
on the H1 connection.

In addition, memory cleanup on the proxy fields on deinit was improved, the
documentation for some missing command line options were added into the
management guide and the configuration manual was also slightly improved.

As said for the 3.0, if you are running on the 2.9, you are encouraged to
upgrade. Thanks everyone for your help.

#
Please find the usual URLs below :
   Site index   : https://www.haproxy.org/
   Documentation: https://docs.haproxy.org/
   Wiki : https://github.com/haproxy/wiki/wiki
   Discourse: https://discourse.haproxy.org/
   Slack channel: https://slack.haproxy.org/
   Issue tracker: https://github.com/haproxy/haproxy/issues
   Sources  : https://www.haproxy.org/download/2.9/src/
   Git repository   : https://git.haproxy.org/git/haproxy-2.9.git/
   Git Web browsing : https://git.haproxy.org/?p=haproxy-2.9.git
   Changelog: https://www.haproxy.org/download/2.9/src/CHANGELOG
   Dataplane API: 
https://github.com/haproxytech/dataplaneapi/releases/latest
   Pending bugs : https://www.haproxy.org/l/pending-bugs
   Reviewed bugs: https://www.haproxy.org/l/reviewed-bugs
   Code reports : https://www.haproxy.org/l/code-reports
   Latest builds: https://www.haproxy.org/l/dev-packages


---
Complete changelog :
Amaury Denoyelle (1):
  BUG/MINOR: quic: fix computed length of emitted STREAM frames

Aurelien DARRAGON (9):
  MINOR: log: fix "http-send-name-header" ignore warning message
  BUG/MINOR: proxy: fix server_id_hdr_name leak on deinit()
  BUG/MINOR: proxy: fix log_tag leak on deinit()
  BUG/MINOR: proxy: fix email-alert leak on deinit()
  BUG/MINOR: proxy: fix check_{command,path} leak on deinit()
  BUG/MINOR: proxy: fix dyncookie_key leak on deinit()
  BUG/MINOR: proxy: fix source interface and usesrc leaks on deinit()
  BUG/MINOR: proxy: fix header_unique_id leak on deinit()
  DOC: management: rename show stats domain cli "dns" to "resolvers"

Christopher Faulet (1):
  BUG/MAJOR: mux-h1: Prevent any UAF on H1 connection after draining a 
request

Valentine Krasnobaeva (2):
  DOC/MINOR: management: add missed -dR and -dv options
  DOC/MINOR: management: add -dZ option

--
Christopher Faulet

[ANNOUNCE] haproxy-3.0.2

[Christopher Faulet]

Hi,

HAProxy 3.0.2 was released on 2024/06/14. It added 21 new commits
after version 3.0.1.

Unfortunately, few days after the previous release, a bug on the H1
multiplexer was found, forcing us to emit a new release. The issue affected
the H1 requests draining that could lead to a crash because of a
use-after-free on the H1 connection.

In addition to this major issue, a possible crash was fixed in the
Prometheus exporter when resolvers metrics were dumped while there was no
resolvers section at all. This happened when the default resolver was not
loaded for any reason (because of an empty /etc/resolv.conf file for
instance).

On proxies, log-format strings on default section were no longer properly
initialized, preventing some options to be applied on regular proxies. As
consequences, some log-format expressions stored inside a default section
and executed by a regular proxy could not behavior properly and encoding
could be altered. A UAF on deinit was also fixed to prevent crashes when
log-format expressions were used on a disabled proxy.

Finally, memory cleanup on the proxy fields on deinit was improved, the
documentation for some missing command line options were added into the
management guide and the configuration manual was also slightly improved.

All 3.0 users are encouraged to upgrade. Thanks everyone for your
help. Especially Annika for her help on the H1 requests draining issue.

#
Please find the usual URLs below :
   Site index   : https://www.haproxy.org/
   Documentation: https://docs.haproxy.org/
   Wiki : https://github.com/haproxy/wiki/wiki
   Discourse: https://discourse.haproxy.org/
   Slack channel: https://slack.haproxy.org/
   Issue tracker: https://github.com/haproxy/haproxy/issues
   Sources  : https://www.haproxy.org/download/3.0/src/
   Git repository   : https://git.haproxy.org/git/haproxy-3.0.git/
   Git Web browsing : https://git.haproxy.org/?p=haproxy-3.0.git
   Changelog: https://www.haproxy.org/download/3.0/src/CHANGELOG
   Dataplane API: 
https://github.com/haproxytech/dataplaneapi/releases/latest
   Pending bugs : https://www.haproxy.org/l/pending-bugs
   Reviewed bugs: https://www.haproxy.org/l/reviewed-bugs
   Code reports : https://www.haproxy.org/l/code-reports
   Latest builds: https://www.haproxy.org/l/dev-packages


---
Complete changelog :
Amaury Denoyelle (1):
  BUG/MINOR: quic: fix padding of INITIAL packets

Aurelien DARRAGON (16):
  MINOR: log: fix "http-send-name-header" ignore warning message
  BUG/MINOR: proxy: fix server_id_hdr_name leak on deinit()
  BUG/MINOR: proxy: fix log_tag leak on deinit()
  BUG/MINOR: proxy: fix email-alert leak on deinit()
  BUG/MINOR: proxy: fix check_{command,path} leak on deinit()
  BUG/MINOR: proxy: fix dyncookie_key leak on deinit()
  BUG/MINOR: proxy: fix source interface and usesrc leaks on deinit()
  BUG/MINOR: proxy: fix header_unique_id leak on deinit()
  BUG/MEDIUM: log: fix lf_expr_postcheck() behavior with default section
  DOC: config: move "hash-key" from proxy to server options
  DOC: config: add missing section hint for "guid" proxy keyword
  DOC: config: add missing context hint for new server and proxy keywords
  MINOR: proxy: add proxy_free_common() helper function
  BUG/MEDIUM: proxy: fix UAF with {tcp,http}checks logformat expressions
  CLEANUP: log/proxy: fix comment in proxy_free_common()
  DOC: management: rename show stats domain cli "dns" to "resolvers"

Christopher Faulet (2):
  BUG/MINOR: promex: Skip resolvers metrics when there is no resolver 
section
  BUG/MAJOR: mux-h1: Prevent any UAF on H1 connection after draining a 
request

Valentine Krasnobaeva (2):
  DOC/MINOR: management: add missed -dR and -dv options
  DOC/MINOR: management: add -dZ option

--
Christopher Faulet

[ANNOUNCE] haproxy-2.9.8

[Amaury Denoyelle]

Hi,

HAProxy 2.9.8 was released on 2024/06/11. It added 106 new commits
after version 2.9.7.

This release is pretty huge as HAProxy development team was mainly
focused on finalizing the new 3.0 version. Now we are catching up the
delay and are preparing to emit a serie of new stable releases. I won't
enter into any details of this release but here is a summary of the most
notable changes.

Several fixes are applied for better HTTP conformance. Final responses
following an interim HTTP status code are now properly forwarded when
using HTTP/2 as frontend protocol. In some cases, 502 server initial
errors were incorrectly hidden and are now properly logged. CONNECT
requests with a scheme are now rejected as they are invalid according to
RFC 7230. Empty paths are normalized to "/" for aboslute-form URI.
Finally, manual removal or modification of Content-Length and
Transfer-Encoding headers via http-rules is now supported again, as it
was incompatible with the first zero-copy forwarding implementation.
However, this should never be used for production unless for testing
purpose.

Compression filter could have produce truncated responses. This was due
to an improper internal stream forwarding which is now correctly
handled.

Dynamic servers testing under heavy load have been performed during 3.0
development cycle. This revealed that crashes could occur due to the
removal of a server currently in used. Removal conditions were thus
adjusted to reject such operation. Also, some settings were not
completely initialized for dynamic servers which cause a difference of
behavior with static ones.

Still on the backend side, an issue was found when NTLM headers are
used. This caused the backend connection to be marked dynamically as
private to prevent HTTP reuse. However, this is conceptually wrong when
using HTTP/2 multiplexer on the backend side with http-reuse mode set to
aggressive or higher, as this connection can already be shared accross
several clients. Thus, NTLM headers are simply ignored in this case.

Minor fixes were merged for QUIC. Most of them are related to improve
the LibreSSL compatibility. Other than that, error handling was improved
to report more specific error codes from the different layers of QUIC
multiplexer, HTTP/3 or QPACK decoder.

For the SSL stack, a few fixes were done, in particular for better OCSP
support. Two patches also concern cipher algorithm negotiation. Firstly,
haproxy could have chosen an ECDSA certificate even if not compatible
with client algorithms instead of fallback to RSA. Secondly, ClientHello
handler for WolfSSL with TLSv1.2 was not behaving correctly.

A serie of patches were written on peers and stick-tables areas to
improve the multithreading contention. However, in one occurence a race
condition appeared due to a relaxed lock. This was corrected while still
preserving the performance improvment. Also a blocking condition could
have occured for the peer applet when reaching max-updates-at-once
limit.

Cache hits should be increased as previously cached HTTP responses which
used Vary header on anything other than Accept-encoding but with
Encoding header present were never returned from the cache.

It is now possible to disable seamless reload on master-worker mode by
using the argument '-x /dev/null'. This may be necessary for some usage
since the introduction of automatic seamless reload for master-worker
mode.

Reverse HTTP feature has been extended for the new 3.0 rendering it more
flexible. Along this, a fix was backported to the current 2.9 version to
properly support reverse HTTP on single thread mode. Previously, active
connections creation via rhttp@ bind lines would not be performed under
this particular condition.

An interesting security feature was backported to block traffic with
clients which use privileged port as their source port value. Such
behavior is highly suspect as it is often the sign of an amplification
attack. This can be activated using
harden.reject-privileged-ports.{tcp|quic} keywords. Note that on 3.0, we
chose to set it by default for QUIC. However, it remains disabled on 2.9
and lesser versions to keep the current behavior on stable haproxy
branches, but users are free to activate it if needed. It is
particularly useful when QUIC listeners are active to prevent DNS/NTP
amplification attack.  However, on TCP this protection may break some
protocols such as FTP.

Not so much on the SPOE front, with a proper rewrite fo the engine now
scheduled for upcoming releases. In 2.9, a single bug-fix is merged
which allowed applet to retry later when reaching buffer exhaustion
instead of returning a definite I/O error.

On the LUA side, a serie of cleanups and minor bugfixes are merged. Most
of them are relevant to error handling which may improve script
debugging. Also a crash was fixed when using CacheCert module from init
context.

The CLI applet now reports a proper error message when command larger
than buffer size is