Re: CPU Spikes
Hey Willy, On 2019-07-09 08:09, Willy Tarreau wrote: What's you CPU like between the peaks ? 1%, 10%, 50% ? Just to get a rough estimate of whether it's something reaching a critical point or if it's something doing its mess alone in its corner. In between the spikes it's about 7% System, 11% User, 6% Softirq, 76% Idle. Bandwidth is then about 500Mbit/s, mostly outbound. What I didn't notice before, but just saw while staring at my graphs, is I get more incoming traffic during the CPU spikes. So, I'm doing about 500Mbit/s, then the incoming traffic rises to about 100Mbit/s (probably a HTTP POST), CPU spikes, total traffic drops to about 200Mbit/s, everything starts getting slow. I had HAProxy running on physical hardware with an E5-2407 and 1Gbit NIC. Now it is running as a VM on an E5-2650 with 10Gbit NIC. With the same issues. Are you using threads ? I'm asking because I'm currently working on an issue which I found could cause exactly this behaviour. I'm fairly certain we've met it in the past without being able to attribute it to exactly this. Yes, I'm using threads. If you're using threads, attaching gdb to the process and issuing "info threads" will tell us where they are. If many of them are in fd_update_events() or fd_may_recv(), you're likely on the one I've been working on. Other possibilities (due to the regularity of your observation) are : - timeouts (check in your conf if a 10s timeout appears somewhere, maybe it triggers and is improperly caught) I have the following timeouts in defaults: timeout client 60s timeout connect 10s timeout http-keep-alive 4s timeout http-request15s timeout queue 30s timeout server 60s timeout tarpit 120s Looking at the spikes again it's more like a 20 second up, 20 second down. But that probably has more to do with the POST taking that long. - health checks (maybe you have 10s checks, or 2s checks with 4 retries or I don't know what, which causes a special event to occur after 10s) Check are every 2s with a rise of 3 and a fall of 3. In any case you're clearly facing a bug, but it's always difficult to tell. It could be useful to issue "show activity" twice 1 second apart when this happens, and maybe even "show fd" and "show sess all" if you don't have too many connections. Right, I will do the above steps. But, since this only happens on Mondays we have to wait a bit ;-) Regards, Sander 0x2E78FBE8.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: DOC: Suggest to replace the netstat commands
On Mon, Jul 08, 2019 at 04:51:24PM +0200, Alain Belkadi wrote: > > Hello, > > As the "netstat" command is deprecated since a long time (1), I suggest to > replace it with other commands like ss and ip. I disagree with this. netstat is not deprecated at all, it's superseded *on linux* because there we have netlink which provides a much faster and more complete interface than the one used by netstat. But netstat is the only command you'll find on about all systems and its output format is pretty consistent. However it might make sense to add a few lines close to the locations where netstat is mentioned to indicate that on Linux ss is preferred since it's much less resource intensive than netstat. Thanks, Willy
Re: DOC: Fix typo in management.txt
Hello Alain, On Mon, Jul 08, 2019 at 02:57:45PM +0200, Alain Belkadi wrote: > > Hello, > > An another patch for a typo in management.txt Please try to group your typo changes for a same file in a single patch as much as possible. A good hint to keep in mind is that we should try to avoid having two commits with the same subject so that it's possible to uniquely designate a commit by its subject. Proceeding like this helps figuring what is specific to one patch and not the others, and thin if they should be merged or not. Thanks, Willy
Re: Replace deprecated reqrep
On Mon, Jul 08, 2019 at 02:36:13PM +0200, Tim Düsterhus wrote: > Artur. > > Am 08.07.19 um 14:25 schrieb Artur: > > Hello, > > > > Could you please suggest how to rewrite following rules written with > > 'regrep' with 'http-request replace-uri' : > > > > frontend www > > reqrep ^([^\ ]*)\ /p3/js/(.*) \1\ /p3/js-min/\2 > > > > The idea is to rewrite something similar to "GET /p3/js/file.js > > HTTP/1.1" with "GET /p3/js-min/file.js HTTP/1.1". > > > > You basically can copy over the "second half" of your rule and adjust > the numbers referencing the capturing groups: > > http-request replace-uri /p3/js/(.*) /p3/js-min/\1 > > Consider either adding a `^` at the start to only match URLs starting > with /p3/ or add an `if` that ensures only URLs starting with /p3/ are > modified. Maybe we should add such examples in the reqrep doc ? Or maybe we should wait for 1 or 2 other ones to complement this one ? Willy
Re: CPU Spikes
Hi Sander, On Mon, Jul 08, 2019 at 02:44:44PM +0200, Sander Klein wrote: > Hi, > > I'm having an issue with HAProxy causing CPU spikes with certain traffic. We've actually fixed quite a number of issues causing this over the last few years, though most of them are already addressed by the versions you're running. > We have a client who is downloading lots of URL's during the night. When the > download starts there is not much other traffic going on and there doesn't > seem to be any problem. But, when the morning comes, 'normal' traffic starts > hitting HAProxy and every 10 seconds or so, HAProxy starts eating 100% of > CPU while network traffic drops. When HAProxy stops eating CPU after 10 > seconds, network traffic rises again. When the crawler is finished > everything returns to normal. So it looks like some kind of mix of traffic > which causes it. What's you CPU like between the peaks ? 1%, 10%, 50% ? Just to get a rough estimate of whether it's something reaching a critical point or if it's something doing its mess alone in its corner. > I've tested it with HAProxy 1.8.20, 1.9.8 (which I am running by default) > and 2.0.1. They all show the same behaviour. I also tried with 2 different > kernels to see if anything happens there. With kernel 4.9 top show HAProxy > using 100% CPU where 50% is user and 50% is system. With kernel 4.19 I see > 100% CPU usage with 70% user and 50% system. In fact once something stats to loop, all calls are so short that it's very difficult for the system to measure an accurate time spent in user/sys, so I am not surprised that it changes with the kernel. > I also tried with disabling H2, splicing, and some regexes I use. Even tried > new hardware, and moved it to a VM just to see if I could find any > difference, but none... Are you using threads ? I'm asking because I'm currently working on an issue which I found could cause exactly this behaviour. I'm fairly certain we've met it in the past without being able to attribute it to exactly this. > Does anyone have a good idea how to troubleshoot this any further? If you're using threads, attaching gdb to the process and issuing "info threads" will tell us where they are. If many of them are in fd_update_events() or fd_may_recv(), you're likely on the one I've been working on. Other possibilities (due to the regularity of your observation) are : - timeouts (check in your conf if a 10s timeout appears somewhere, maybe it triggers and is improperly caught) - health checks (maybe you have 10s checks, or 2s checks with 4 retries or I don't know what, which causes a special event to occur after 10s) In any case you're clearly facing a bug, but it's always difficult to tell. It could be useful to issue "show activity" twice 1 second apart when this happens, and maybe even "show fd" and "show sess all" if you don't have too many connections. Thanks, Willy
Re: global maxconn behaviour in haproxy2.0
On Wed, Jul 03, 2019 at 01:30:52PM +, William Dauchy wrote:
> On Wed, Jun 26, 2019 at 11:29:47AM +1000, Igor Cicimov wrote:
> > Those maxconn values are per frontend so if your backend is referenced by
> > two frontends you might end up with a limit of 2 x maxconn on the backend.
> > Hence it is recommended to set maxconn per server too to protect from
> > situation like this. So read about maxconn and even fullconn in the server
> > config and tuning guide for more details.
>
> thanks for the precision. I however later discovered in the code:
>
> ha_warning("Proxy %s stopped (FE: %lld conns, BE: %lld conns).\n",
> p->id, p->fe_counters.cum_conn, p->be_counters.cum_conn);
>
> which means this is simply a cumulative counter displayed in the log.
Indeed. It was added into 1.4 (and backported into 1.3) with this commit :
commit 1104614b57419dbe6f0407d8ce6bf6910d9aeba7
Author: Willy Tarreau
Date: Thu Mar 4 23:07:28 2010 +0100
[MINOR] report total number of processed connections when stopping a proxy
It's sometimes convenient to know if a proxy has processed any connection
at all when stopping it. Since a soft restart causes the "Proxy stopped"
message to be logged for each proxy, let's add the number of connections
so that it's possible afterwards to check whether a proxy had received
any connection.
I might have had a good reason for doing it like this back then because
9 years later I find it stupid! Or at least confusing. I too was pretty
sure that it was the number of active conns. After giving it some thoughts,
it wouldn't make sense that it reports the number of active conns since
the proxy only stops when there are no more. Still the message could be
improved, like "FE: %lld cumulated conns" or something like this.
Willy
Re: haproxy inappropriately sending rst_stream on http/2
Hi Patrick, On Mon, Jul 08, 2019 at 05:48:26PM -0400, Patrick Hemmer wrote: > So I sent the information to Willy, but I'm guessing he's occupied with > other things as I haven't heard back (which is perfectly fine, this is an > open source project with no support contract after all). Oh sorry, it was during the week I was in vacation, never too far away from the keyboard but definitely not as fast anymore to catch up with incoming emails, so yours might have been in the middle of a long series I received and that will be archived without ever being read. I can see it now. I'll go dig into this hopefully this week, once I'm done with some other problematic issues. > But if there is > someone else in the HAProxy org I can send the debug info to, I can do that. > I just don't want to post it publicly. I perfectly understand, no problem. Thanks for the ping, much appreciated! Willy
[PR] Corrected Typo
Dear list! Author: Rajesh Rajendran Number of patches: 1 This is an automated relay of the Github pull request: Corrected Typo Patch title(s): Corrected Typo Link: https://github.com/haproxy/haproxy/pull/160 Edit locally: wget https://github.com/haproxy/haproxy/pull/160.patch && vi 160.patch Apply locally: curl https://github.com/haproxy/haproxy/pull/160.patch | git am - Description: Instructions: This github pull request will be closed automatically; patch should be reviewed on the haproxy mailing list (haproxy@formilux.org). Everyone is invited to comment, even the patch's author. Please keep the author and list CCed in replies. Please note that in absence of any response this pull request will be lost.
Re: Proof of concept SPOE based SSO solution
On Fri, Jul 5, 2019 at 11:12 AM Andrew Heberle wrote: > Hi All, > > I have put together a Go based proof of concept SPOE agent that also > implements a SAML 2 Service Provider (SP) in order to do "SSO" in > HAProxy. > > The code is located here: > > https://gitlab.com/andrewheberle/go-http-auth-sso > > The basic process is that SPOA is used to check if the user is logged > in or not and then based on the set variables you can make decisions > via "http-request" rules. > > This originally started out without the SPOE part and was using the > Lua http-auth-request script > (https://github.com/TimWolla/haproxy-auth-request), however with the > release of the Go SPOE package > (https://github.com/Aestek/haproxy-connect/tree/master/spoe) I rewrote > it based on that. > > Our use case is to have the SP pointed to a IdP in Azure so we can do > single-sign-on to Office 365 and we have "http-request" rules in place > to set some custom headers that our application uses for > authentication/authorisation. These are set based on the variables > that come back from the SPOA, which come from the claims in the > authentication process. > > Hopefully this is of some use to people. > > Any feedback and constructive criticism is welcome. > > -- > Andrew Heberle > > Thanks for sharing Andrew! Cheers, Igor
Re: [PATCH v2] BUG/MINOR: ssl: revert empty handshake detection in OpenSSL <= 1.0.2
On Mon, Jul 08, 2019 at 02:29:15PM +0200, Lukas Tribus wrote:
> Commit 54832b97 ("BUILD: enable several LibreSSL hacks, including")
> changed empty handshake detection in OpenSSL <= 1.0.2 and LibreSSL,
> from accessing packet_length directly (not available in LibreSSL) to
> calling SSL_state() instead.
(...)
Now applied, thank you Lukas!
Willy
Re: haproxy inappropriately sending rst_stream on http/2
*From:* Patrick Hemmer [mailto:hapr...@stormcloud9.net] *Sent:* Wednesday, June 26, 2019, 08:06 EDT *To:* haproxy@formilux.org *Subject:* haproxy inappropriately sending rst_stream on http/2 I'm running haproxy 1.9.8 and am having an issue where haproxy is sending a http/2 rst_stream message to the client for some reason. When I look in the haproxy logs, the state termination flags are "CD--". However I know the client didn't abort the request as I have a packet capture showing the client did no such thing. Additionally there are other requests on the same connection, both before & after the one that gets reset, which go through fine. Willy, I'm going to send both the logs and packet capture off-list. -Patrick So I sent the information to Willy, but I'm guessing he's occupied with other things as I haven't heard back (which is perfectly fine, this is an open source project with no support contract after all). But if there is someone else in the HAProxy org I can send the debug info to, I can do that. I just don't want to post it publicly. Thanks -Patrick
Re: Get http connection client/server ip/port
Hello, as confirmed in the github issue and for the record here as well (so that we have the solution in the archives): This is not a haproxy bug, but turned out to be a kernel/conntrack thing. Disabling nf_conntrack_tcp_loose fixes this: https://www.spinics.net/lists/netdev/msg546371.html cheers, lukas
Re: DOC: Suggest to replace the netstat commands
*From:* Alain Belkadi [mailto:xigu...@linuxbeach.be]
*Sent:* Monday, July 8, 2019, 10:51 EDT
*To:* haproxy@formilux.org
*Subject:* DOC: Suggest to replace the netstat commands
Hello,
As the "netstat" command is deprecated since a long time (1), I
suggest to replace it with other commands like ss and ip.
I've made a first patch for this. As the number of columns is higher
than the 80 standard, I've made a second patch with less long lines
... but that don't fit inside 80 chars and if I remove more spaces
that don't looks good.
(1) https://en.wikipedia.org/wiki/Netstat
Regards,
This raises a potential issue. Netstat is deprecated on Linux yes, but
not other OSs (which HAProxy runs on). However the example usage that
was provided ("netstat -ltnp") is the Linux compatible flags, so the doc
was already somewhat Linux specific. However other references, such as
"netstat -i", are cross-platform.
So where do we draw the line on being Linux-specific in our
documentation, vs. OS agnostic?
-Patrick
Re: 1.9 external health checks fail suddenly
Hello, On Mon, 1 Jul 2019 at 12:27, Lukas Tribus wrote: > > > Sometimes (infrequently) all external checks hang and time out: > > > * Has happened with versions 1.9.4 and 1.9.8 on multiple servers with > > > nbproc 1 and nbthread set to (4-12) depending on server. > > > * Happens infrequently, last one happened after 10 days of uptime. > > > * External checks are written in python and write errors into their own > > > log file directly. When hanging happens, nothing is logged by external > > > check. > > > * Only external checks fail, common 'option httpcheck' does not fail at > > > the same time. > > > > External checks are not thread-safe, that's a bug. > > > > Could you try the suggest patch in: > > > > https://github.com/haproxy/haproxy/issues/140#issuecomment-507119534 > > Sorry, I think I got confused here with something else ... the bug is > about signals being blocked, which is not your problem. The bug you may be affected by is: https://github.com/haproxy/haproxy/issues/141 Can you check what happens with: nbthread 1 cheers, lukas
DOC: Suggest to replace the netstat commands
Hello,
As the "netstat" command is deprecated since a long time (1), I suggest
to replace it with other commands like ss and ip.
I've made a first patch for this. As the number of columns is higher
than the 80 standard, I've made a second patch with less long lines ...
but that don't fit inside 80 chars and if I remove more spaces that
don't looks good.
(1) https://en.wikipedia.org/wiki/Netstat
Regards,
--
[Alain Belkadi / LinuxBeach]From 0678b8610ee5931cd31edb66091a09faf21dc1ad Mon Sep 17 00:00:00 2001
From: Alain Belkadi
Date: Mon, 8 Jul 2019 16:17:17 +0200
Subject: [PATCH 1/2] DOC: Replace the deprecated "netstat" command with more
recent tools
---
doc/management.txt | 33 -
1 file changed, 16 insertions(+), 17 deletions(-)
diff --git a/doc/management.txt b/doc/management.txt
index 93bee671..59ffb1ed 100644
--- a/doc/management.txt
+++ b/doc/management.txt
@@ -103,16 +103,14 @@ connections and to initiate outgoing connections. An immediate effect of this is
that there is no relation between packets observed on the two sides of a
forwarded connection, which can be of different size, numbers and even family.
Since a connection may only be accepted from a socket in LISTEN state, all the
-sockets it is listening to are necessarily visible using the "netstat" utility
+sockets it is listening to are necessarily visible using the "ss" utility
to show listening sockets. Example :
- # netstat -ltnp
- Active Internet connections (only servers)
- Proto Recv-Q Send-Q Local Address Foreign Address StatePID/Program name
- tcp0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1629/sshd
- tcp0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2847/haproxy
- tcp0 0 0.0.0.0:443 0.0.0.0:* LISTEN 2847/haproxy
-
+ # ss -ltnp
+ StateRecv-Q Send-Q Local Address:Port Peer Address:Port
+ LISTEN 0128 0.0.0.0:80 0.0.0.0:* users:(("haproxy",pid=29687,fd=10))
+ LISTEN 0128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=18594,fd=3))
+ LISTEN 0128 [::]:22[::]:* users:(("sshd",pid=18594,fd=4))
3. Starting HAProxy
---
@@ -2779,7 +2777,7 @@ is ignored. The reason is that in fact even the new process is restarted with a
new configuration, the old one also gets some incoming connections and
processes them, returning unexpected results. When in doubt, just stop the new
process and try again. If it still works, it very likely means that an old
-process remains alive and has to be stopped. Linux's "netstat -lntp" is of good
+process remains alive and has to be stopped. Linux's "ss -lntp" is of good
help here.
When adding entries to an ACL from the command line (eg: when blacklisting a
@@ -2953,14 +2951,15 @@ not noticeable at the naked eye. If they appear a lot in the traces, it is
worth investigating exactly what happens and where the packets are lost. HTTP
doesn't cope well with TCP losses, which introduce huge latencies.
-The "netstat -i" command will report statistics per interface. An interface
-where the Rx-Ovr counter grows indicates that the system doesn't have enough
-resources to receive all incoming packets and that they're lost before being
-processed by the network driver. Rx-Drp indicates that some received packets
-were lost in the network stack because the application doesn't process them
-fast enough. This can happen during some attacks as well. Tx-Drp means that
-the output queues were full and packets had to be dropped. When using TCP it
-should be very rare, but will possibly indicate a saturated outgoing link.
+The "ip -s link" command will report statistics per interface. An interface
+where the 'RX overrun' counter grows indicates that the system doesn't have
+enough resources to receive all incoming packets and that they're lost before
+being processed by the network driver. 'RX dropped' indicates that some
+received packets were lost in the network stack because the application doesn't
+process them fast enough. This can happen during some attacks as well.
+TX dropped means that the output queues were full and packets had to be
+dropped. When using TCP it should be very rare, but will possibly indicate a
+saturated outgoing link.
13. Security considerations
--
2.11.0
From c94d7f2bd58947e220a560e0a7f42318051dc5db Mon Sep 17 00:00:00 2001
From: Alain Belkadi
Date: Mon, 8 Jul 2019 16:40:00 +0200
Subject: [PATCH 2/2] DOC: Replace the deprecated "netstat" command with more
recent tools
---
doc/management.txt | 8
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/doc/management.txt b/doc/management.t
DOC: Fix typo in management.txt
Hello, An another patch for a typo in management.txt Regards, -- [Alain Belkadi / LinuxBeach]From 492a84aa7f25444b89e5224a5cf015a8e12fb683 Mon Sep 17 00:00:00 2001 From: Alain Belkadi Date: Mon, 8 Jul 2019 14:27:10 +0200 Subject: [PATCH] DOC: Fix a typo in management.txt --- doc/management.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/management.txt b/doc/management.txt index 93bee671..d5f0 100644 --- a/doc/management.txt +++ b/doc/management.txt @@ -127,7 +127,7 @@ followed by one of more letters, and possibly followed by one or multiple extra arguments. Without any option, HAProxy displays the help page with a reminder about supported options. Available options may vary slightly based on the operating system. A fair number of these options overlap with an equivalent one -if the "global" section. In this case, the command line always has precedence +in the "global" section. In this case, the command line always has precedence over the configuration file, so that the command line can be used to quickly enforce some settings without touching the configuration files. The current list of options is : -- 2.11.0
CPU Spikes
Hi, I'm having an issue with HAProxy causing CPU spikes with certain traffic. We have a client who is downloading lots of URL's during the night. When the download starts there is not much other traffic going on and there doesn't seem to be any problem. But, when the morning comes, 'normal' traffic starts hitting HAProxy and every 10 seconds or so, HAProxy starts eating 100% of CPU while network traffic drops. When HAProxy stops eating CPU after 10 seconds, network traffic rises again. When the crawler is finished everything returns to normal. So it looks like some kind of mix of traffic which causes it. I've tested it with HAProxy 1.8.20, 1.9.8 (which I am running by default) and 2.0.1. They all show the same behaviour. I also tried with 2 different kernels to see if anything happens there. With kernel 4.9 top show HAProxy using 100% CPU where 50% is user and 50% is system. With kernel 4.19 I see 100% CPU usage with 70% user and 50% system. I also tried with disabling H2, splicing, and some regexes I use. Even tried new hardware, and moved it to a VM just to see if I could find any difference, but none... Does anyone have a good idea how to troubleshoot this any further? Regards, Sander 0x2E78FBE8.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: Replace deprecated reqrep
Artur. Am 08.07.19 um 14:25 schrieb Artur: > Hello, > > Could you please suggest how to rewrite following rules written with > 'regrep' with 'http-request replace-uri' : > > frontend www > reqrep ^([^\ ]*)\ /p3/js/(.*) \1\ /p3/js-min/\2 > > The idea is to rewrite something similar to "GET /p3/js/file.js > HTTP/1.1" with "GET /p3/js-min/file.js HTTP/1.1". > You basically can copy over the "second half" of your rule and adjust the numbers referencing the capturing groups: http-request replace-uri /p3/js/(.*) /p3/js-min/\1 Consider either adding a `^` at the start to only match URLs starting with /p3/ or add an `if` that ensures only URLs starting with /p3/ are modified. Best regards Tim Düsterhus
Re: [PATCH v2] BUG/MINOR: ssl: revert empty handshake detection in OpenSSL <= 1.0.2
Nice
On Mon, Jul 8, 2019, 5:30 PM Lukas Tribus wrote:
> Commit 54832b97 ("BUILD: enable several LibreSSL hacks, including")
> changed empty handshake detection in OpenSSL <= 1.0.2 and LibreSSL,
> from accessing packet_length directly (not available in LibreSSL) to
> calling SSL_state() instead.
>
> However, SSL_state() appears to be fully broken in both OpenSSL and
> LibreSSL.
>
> Since there is no possibility in LibreSSL to detect an empty handshake,
> let's not try (like BoringSSL) and restore this functionality for
> OpenSSL 1.0.2 and older, by reverting to the previous behavior.
>
> Should be backported to 2.0.
> ---
>
> changes in V2:
> add code comments
>
> ---
> src/ssl_sock.c | 18 --
> 1 file changed, 12 insertions(+), 6 deletions(-)
>
> diff --git a/src/ssl_sock.c b/src/ssl_sock.c
> index c9fffbe..3ddacb6 100644
> --- a/src/ssl_sock.c
> +++ b/src/ssl_sock.c
> @@ -5352,15 +5352,18 @@ static int ssl_sock_handshake(struct connection
> *conn, unsigned int flag)
> if (!errno && conn->flags &
> CO_FL_WAIT_L4_CONN)
> conn->flags &= ~CO_FL_WAIT_L4_CONN;
> if (!conn->err_code) {
> -#ifdef OPENSSL_IS_BORINGSSL /* BoringSSL */
> +#if defined(OPENSSL_IS_BORINGSSL) || defined(LIBRESSL_VERSION_NUMBER)
> + /* do not handle empty handshakes
> in BoringSSL or LibreSSL */
> conn->err_code =
> CO_ER_SSL_HANDSHAKE;
> #else
> int empty_handshake;
> #if (HA_OPENSSL_VERSION_NUMBER >= 0x101fL)
> + /* use SSL_get_state() in OpenSSL
> >= 1.1.0; SSL_state() is broken */
> OSSL_HANDSHAKE_STATE state =
> SSL_get_state((SSL *)ctx->ssl);
> empty_handshake = state ==
> TLS_ST_BEFORE;
> #else
> - empty_handshake = SSL_state((SSL
> *)ctx->ssl) == SSL_ST_BEFORE;
> + /* access packet_length directly
> in OpenSSL <= 1.0.2; SSL_state() is broken */
> + empty_handshake =
> !ctx->ssl->packet_length;
> #endif
> if (empty_handshake) {
> if (!errno) {
> @@ -5382,7 +5385,7 @@ static int ssl_sock_handshake(struct connection
> *conn, unsigned int flag)
> else
> conn->err_code =
> CO_ER_SSL_HANDSHAKE;
> }
> -#endif
> +#endif /* BoringSSL or LibreSSL */
> }
> goto out_error;
> }
> @@ -5433,15 +5436,18 @@ check_error:
> if (!errno && conn->flags & CO_FL_WAIT_L4_CONN)
> conn->flags &= ~CO_FL_WAIT_L4_CONN;
> if (!conn->err_code) {
> -#ifdef OPENSSL_IS_BORINGSSL /* BoringSSL */
> +#if defined(OPENSSL_IS_BORINGSSL) || defined(LIBRESSL_VERSION_NUMBER)
> + /* do not handle empty handshakes in
> BoringSSL or LibreSSL */
> conn->err_code = CO_ER_SSL_HANDSHAKE;
> #else
> int empty_handshake;
> #if (HA_OPENSSL_VERSION_NUMBER >= 0x101fL)
> + /* use SSL_get_state() in OpenSSL >=
> 1.1.0; SSL_state() is broken */
> OSSL_HANDSHAKE_STATE state =
> SSL_get_state(ctx->ssl);
> empty_handshake = state == TLS_ST_BEFORE;
> #else
> - empty_handshake = SSL_state((SSL
> *)ctx->ssl) == SSL_ST_BEFORE;
> + /* access packet_length directly in
> OpenSSL <= 1.0.2; SSL_state() is broken */
> + empty_handshake = !ctx->ssl->packet_length;
> #endif
> if (empty_handshake) {
> if (!errno) {
> @@ -5463,7 +5469,7 @@ check_error:
> else
> conn->err_code =
> CO_ER_SSL_HANDSHAKE;
> }
> -#endif
> +#endif /* BoringSSL or LibreSSL */
> }
> goto out_error;
> }
> --
> 2.7.4
>
[PATCH v2] BUG/MINOR: ssl: revert empty handshake detection in OpenSSL <= 1.0.2
Commit 54832b97 ("BUILD: enable several LibreSSL hacks, including")
changed empty handshake detection in OpenSSL <= 1.0.2 and LibreSSL,
from accessing packet_length directly (not available in LibreSSL) to
calling SSL_state() instead.
However, SSL_state() appears to be fully broken in both OpenSSL and
LibreSSL.
Since there is no possibility in LibreSSL to detect an empty handshake,
let's not try (like BoringSSL) and restore this functionality for
OpenSSL 1.0.2 and older, by reverting to the previous behavior.
Should be backported to 2.0.
---
changes in V2:
add code comments
---
src/ssl_sock.c | 18 --
1 file changed, 12 insertions(+), 6 deletions(-)
diff --git a/src/ssl_sock.c b/src/ssl_sock.c
index c9fffbe..3ddacb6 100644
--- a/src/ssl_sock.c
+++ b/src/ssl_sock.c
@@ -5352,15 +5352,18 @@ static int ssl_sock_handshake(struct connection *conn,
unsigned int flag)
if (!errno && conn->flags & CO_FL_WAIT_L4_CONN)
conn->flags &= ~CO_FL_WAIT_L4_CONN;
if (!conn->err_code) {
-#ifdef OPENSSL_IS_BORINGSSL /* BoringSSL */
+#if defined(OPENSSL_IS_BORINGSSL) || defined(LIBRESSL_VERSION_NUMBER)
+ /* do not handle empty handshakes in
BoringSSL or LibreSSL */
conn->err_code = CO_ER_SSL_HANDSHAKE;
#else
int empty_handshake;
#if (HA_OPENSSL_VERSION_NUMBER >= 0x101fL)
+ /* use SSL_get_state() in OpenSSL >=
1.1.0; SSL_state() is broken */
OSSL_HANDSHAKE_STATE state =
SSL_get_state((SSL *)ctx->ssl);
empty_handshake = state ==
TLS_ST_BEFORE;
#else
- empty_handshake = SSL_state((SSL
*)ctx->ssl) == SSL_ST_BEFORE;
+ /* access packet_length directly in
OpenSSL <= 1.0.2; SSL_state() is broken */
+ empty_handshake =
!ctx->ssl->packet_length;
#endif
if (empty_handshake) {
if (!errno) {
@@ -5382,7 +5385,7 @@ static int ssl_sock_handshake(struct connection *conn,
unsigned int flag)
else
conn->err_code =
CO_ER_SSL_HANDSHAKE;
}
-#endif
+#endif /* BoringSSL or LibreSSL */
}
goto out_error;
}
@@ -5433,15 +5436,18 @@ check_error:
if (!errno && conn->flags & CO_FL_WAIT_L4_CONN)
conn->flags &= ~CO_FL_WAIT_L4_CONN;
if (!conn->err_code) {
-#ifdef OPENSSL_IS_BORINGSSL /* BoringSSL */
+#if defined(OPENSSL_IS_BORINGSSL) || defined(LIBRESSL_VERSION_NUMBER)
+ /* do not handle empty handshakes in BoringSSL
or LibreSSL */
conn->err_code = CO_ER_SSL_HANDSHAKE;
#else
int empty_handshake;
#if (HA_OPENSSL_VERSION_NUMBER >= 0x101fL)
+ /* use SSL_get_state() in OpenSSL >= 1.1.0;
SSL_state() is broken */
OSSL_HANDSHAKE_STATE state =
SSL_get_state(ctx->ssl);
empty_handshake = state == TLS_ST_BEFORE;
#else
- empty_handshake = SSL_state((SSL *)ctx->ssl) ==
SSL_ST_BEFORE;
+ /* access packet_length directly in OpenSSL <=
1.0.2; SSL_state() is broken */
+ empty_handshake = !ctx->ssl->packet_length;
#endif
if (empty_handshake) {
if (!errno) {
@@ -5463,7 +5469,7 @@ check_error:
else
conn->err_code =
CO_ER_SSL_HANDSHAKE;
}
-#endif
+#endif /* BoringSSL or LibreSSL */
}
goto out_error;
}
--
2.7.4
Replace deprecated reqrep
Hello, Could you please suggest how to rewrite following rules written with 'regrep' with 'http-request replace-uri' : frontend www reqrep ^([^\ ]*)\ /p3/js/(.*) \1\ /p3/js-min/\2 The idea is to rewrite something similar to "GET /p3/js/file.js HTTP/1.1" with "GET /p3/js-min/file.js HTTP/1.1". -- Best regards, Artur
DOC: Fix typo in management.txt
Hello, A fix for a typo in §3. Regards, -- [Alain Belkadi / LinuxBeach]From 35c31e13d5b8c87f1781674c7a6434cda50716c1 Mon Sep 17 00:00:00 2001 From: Alain Belkadi Date: Mon, 8 Jul 2019 14:12:16 +0200 Subject: [PATCH] DOC: Fix a typo in management.txt --- doc/management.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/management.txt b/doc/management.txt index 93bee671..6a1dcf2a 100644 --- a/doc/management.txt +++ b/doc/management.txt @@ -123,7 +123,7 @@ passed on the command line. The actual syntax is : $ haproxy []* where []* is any number of options. An option always starts with '-' -followed by one of more letters, and possibly followed by one or multiple extra +followed by one or more letters, and possibly followed by one or multiple extra arguments. Without any option, HAProxy displays the help page with a reminder about supported options. Available options may vary slightly based on the operating system. A fair number of these options overlap with an equivalent one -- 2.11.0
Re: prometheus service kills ssl handshake
Hi Christopher. Am 08.07.2019 um 10:30 schrieb Christopher Faulet: > Le 06/07/2019 à 23:02, Aleksandar Lazic a écrit : >> Hi. >> >> I use HAP 2.0.1 with haproxy service with my image. >> After some times (~several hours) the ssl handshake stops working for the >> https >> frontend which offers the prom service. >> [snipp] > > Hi Aleks, > > Could you check with the latest 2.0 snapshot ? An issue about Prometheus was > fixed (#151 on GitHub). And some others about connections. > Okay I created the image with ss-20190706 . https://gitlab.com/aleks001/haproxy20-centos/commit/212ed6f4099dd92c72b426726afdf04022065798 Regards Aleks
Re: question about spoe doc
Hi.
Am 05.07.2019 um 15:28 schrieb Christopher Faulet:
> Le 03/07/2019 à 11:03, Aleksandar Lazic a écrit :
>> Hi.
>>
>> in the spoe doc in section **2.5. Example** are the following lines.
>>
>> ```
>> 620 spoe-message get-ip-reputation
>> 621 args ip=src
>> 622 event on-client-session if ! { src -f
>> /etc/haproxy/whitelist.lst }
>> ```
>>
>> As far as I understood the doc in line `496 args [name=] ...` right
>> could be the check also against `ip` not only `src`?
>>
>> git links:
>>
>> http://git.haproxy.org/?p=haproxy-1.8.git;a=blob;f=doc/SPOE.txt;h=9602df95dacc3aa5f6fa9154c0cf3f2a3f4ed092;hb=HEAD
>>
>>
>> http://git.haproxy.org/?p=haproxy.git;a=blob;f=doc/SPOE.txt;h=19f00ad98d51f155494b47e680f14d6973dd91bb;hb=HEAD
>>
>
> Sorry, I don't understand the question. "src" is a sample fetch. "ip" is just
> a
> label. It may be use by the agent to match arguments. It could be a way to
> pass
> arguments in any order and rely on their names instead of their positions.
>
> Hope it helps :)
Yes, thank you. I thought that "ip" can be used for testing, but as it is only a
label it's not possible to use it similar as "src".
Best regads
aleks
July 8, 2019 - GCPL Chairman Adi Godrej's FY19 remuneration 114-times higher than firm's median pay
Re: Send-proxy not modifying some traffic with proxy ip/port details instead retaining same client ip port
For the record: this was not a haproxy bug, but turned out to be a kernel/conntrack thing. Disabling nf_conntrack_tcp_loose fixes this: https://www.spinics.net/lists/netdev/msg546371.html cheers, lukas
Fix typo in management.txt
Hello, I've found a little typo in management.txt Regards, -- [Alain Belkadi / LinuxBeach]From b62365ce6c78157a1efd23ad4a150434fbe473d9 Mon Sep 17 00:00:00 2001 From: Alain Belkadi Date: Mon, 8 Jul 2019 11:09:54 +0200 Subject: [PATCH] [DOC] Fix typo in management.txt --- doc/management.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/management.txt b/doc/management.txt index 93bee671..e6b55d77 100644 --- a/doc/management.txt +++ b/doc/management.txt @@ -48,7 +48,7 @@ familiar with troubleshooting utilities such as strace and tcpdump. 2. Quick reminder about HAProxy's architecture -- -HAProxy is a multi-threaded, event-driven, non-blocking daemon. This means is +HAProxy is a multi-threaded, event-driven, non-blocking daemon. This means it uses event multiplexing to schedule all of its activities instead of relying on the system to schedule between multiple activities. Most of the time it runs as a single process, so the output of "ps aux" on a system will report only one -- 2.11.0
Re: prometheus service kills ssl handshake
Le 06/07/2019 à 23:02, Aleksandar Lazic a écrit :
Hi.
I use HAP 2.0.1 with haproxy service with my image.
After some times (~several hours) the ssl handshake stops working for the https
frontend which offers the prom service.
The config is mainly the same as showing at the blog post
https://www.me2digital.com/blog/2019/05/haproxy-sni-routing/
I have just added this line.
```
http-request use-service prometheus-exporter if { path /mymetrics }
```
This is the prometheus config snipplet.
```
- job_name: 'me2d-prom'
metrics_path: '/mymetrics'
scheme: 'https'
static_configs:
- targets: ['cloud.domain']
```
The lines in the logs does not show too much
```
:::85.10.XXX.XXX:40276 [05/Jul/2019:13:53:33.532] https-in/1: SSL handshake
failure
```
Have someone used this service in the main https config?
Which data (show ...) should I provide to debug this topic?
Hi Aleks,
Could you check with the latest 2.0 snapshot ? An issue about Prometheus was
fixed (#151 on GitHub). And some others about connections.
--
Christopher Faulet
