Re: Haproxy 1.8 with OpenSSL 1.1.1-pre4 stops working after 1 hour

[Lukas Tribus]

Hello,


On 23 May 2018 at 22:17, Jim Freeman  wrote:
> Or kludge around it with eg; http://www.issihosts.com/haveged/ ?

No, it's not about insufficient entropy in the kernel. It's about
interfacing with that entropy while in chroot.


Lukas

Re: Haproxy 1.8 with OpenSSL 1.1.1-pre4 stops working after 1 hour

[Jim Freeman]

Or kludge around it with eg; http://www.issihosts.com/haveged/ ?

On Wed, May 23, 2018 at 1:48 PM, Lukas Tribus  wrote:

> Hello,
>
>
> On 23 May 2018 at 18:29, Emeric Brun  wrote:
> > This issue was due to openssl-1.1.1 which re-seed after an elapsed time
> or number of request.
> >
> > If /dev/urandom is used as seeding source when haproxy is chrooted it
> fails to re-open /dev/urandom 
> >
> > By defaut the openssl-1.1.1 configure script uses the syscall getrandom
> as seeding source and fallback on /dev/urandom if not available.
> >
> > So you don't face the issue if your openssl-1.1.1 is compiled to use
> getrandom
> >
> > But getrandom syscall is available only since kernel > 3.17 and the main
> point: for glibc > 2.25.
> >
> > With openssl-1.1.1 you can check this this way:
> > # ./openssl-1.1.1/openssl version -r
> > Seeding source: getrandom-syscall
>
> I have glibc 2.23 (Ubuntu 16.04) and openssl shows "os-specific", even
> if kernel headers are installed while compiling, yet -pre6 does not
> hang for me in chroot (-pre4 did):
>
> lukas@dev:~/libsslbuild/bin$ uname -r
> 4.4.0-109-generic
> lukas@dev:~/libsslbuild/bin$ ./openssl version
> OpenSSL 1.1.1-pre6 (beta) 1 May 2018
> lukas@dev:~/libsslbuild/bin$ ./openssl version -r
> Seeding source: os-specific
> lukas@dev:~/libsslbuild/bin$
>
>
> But, stracing haproxy shows that the library IS ACTUALLY using
> getrandom(). So the "Seeding source" output of the executable is
> wrong. Gonna dig into this as well, but seeing how my haproxy
> executable uses getrandom() calls, this perfectly explains why I did
> not see this in -pre6 (which has the build-workaround for < libc 2.25,
> while pre4 did not, so it did not use the getrandom() call).
>
>
> @Sander it looks like openssl folks won't change their mind about
> this. You have to either upgrade to a kernel more recent than 3.17 so
> that getrandom() can be used, or make /dev/xrandom available within
> your chroot.
>
>
>
> Lukas
>
>

Re: Haproxy 1.8 with OpenSSL 1.1.1-pre4 stops working after 1 hour

[Lukas Tribus]

Hello,


On 23 May 2018 at 18:29, Emeric Brun  wrote:
> This issue was due to openssl-1.1.1 which re-seed after an elapsed time or 
> number of request.
>
> If /dev/urandom is used as seeding source when haproxy is chrooted it fails 
> to re-open /dev/urandom 
>
> By defaut the openssl-1.1.1 configure script uses the syscall getrandom as 
> seeding source and fallback on /dev/urandom if not available.
>
> So you don't face the issue if your openssl-1.1.1 is compiled to use getrandom
>
> But getrandom syscall is available only since kernel > 3.17 and the main 
> point: for glibc > 2.25.
>
> With openssl-1.1.1 you can check this this way:
> # ./openssl-1.1.1/openssl version -r
> Seeding source: getrandom-syscall

I have glibc 2.23 (Ubuntu 16.04) and openssl shows "os-specific", even
if kernel headers are installed while compiling, yet -pre6 does not
hang for me in chroot (-pre4 did):

lukas@dev:~/libsslbuild/bin$ uname -r
4.4.0-109-generic
lukas@dev:~/libsslbuild/bin$ ./openssl version
OpenSSL 1.1.1-pre6 (beta) 1 May 2018
lukas@dev:~/libsslbuild/bin$ ./openssl version -r
Seeding source: os-specific
lukas@dev:~/libsslbuild/bin$


But, stracing haproxy shows that the library IS ACTUALLY using
getrandom(). So the "Seeding source" output of the executable is
wrong. Gonna dig into this as well, but seeing how my haproxy
executable uses getrandom() calls, this perfectly explains why I did
not see this in -pre6 (which has the build-workaround for < libc 2.25,
while pre4 did not, so it did not use the getrandom() call).


@Sander it looks like openssl folks won't change their mind about
this. You have to either upgrade to a kernel more recent than 3.17 so
that getrandom() can be used, or make /dev/xrandom available within
your chroot.



Lukas

Re: Dynamically adding/deleting SSL certificates

[Aurélien Nephtali]

Hello Emeric,

On Tue, May 22, 2018 at 05:37:58PM +0200, Emeric Brun wrote:
> Hi Auréline
> 
> I see that you're using the domain to known the certificate to delete.
> 
> If you take a look to crt-list, you will see that the identifier of the 
> certificate
> is customizable and is not necessarily the domain.
> 
> I think to perform the adds/delete operation we should use the same 
> identifiers than the crt-list option
> 

Right, I forgot about these.
It's now possible to use one of the identifiers (if any) of a
certificate from a crt-list in the deletion.

I added a refcount in the struct sni_ctx, added wildcard support that I
forgot and corrected a typo in an error message.

It's still RFC patches.

-- 
Aurélien.
>From 9612ff9dcfda0611e5287ea0e9d9959d972e0061 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Aur=C3=A9lien=20Nephtali?= 
Date: Wed, 18 Apr 2018 15:34:33 +0200
Subject: [PATCH 1/2] MINOR/WIP: ssl: Refactor ssl_sock_load_cert_file() into
 ssl_sock_load_cert2()
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

- ssl_sock_get_dh_from_file() -> ssl_sock_get_dh()
- ssl_sock_load_dh_params() takes a BIO * instead of a char *
- ssl_sock_load_cert_chain_file() -> ssl_sock_load_cert_chain() + takes a
  BIO * instead of a char *

Signed-off-by: Aurélien Nephtali 
---
 src/ssl_sock.c | 210 +
 1 file changed, 136 insertions(+), 74 deletions(-)

diff --git a/src/ssl_sock.c b/src/ssl_sock.c
index 7a602ad57..eb0d43ded 100644
--- a/src/ssl_sock.c
+++ b/src/ssl_sock.c
@@ -2583,43 +2583,39 @@ static DH *ssl_get_tmp_dh(SSL *ssl, int export, int keylen)
 	return dh;
 }
 
-static DH * ssl_sock_get_dh_from_file(const char *filename)
+static DH * ssl_sock_get_dh(BIO *in)
 {
-	DH *dh = NULL;
-	BIO *in = BIO_new(BIO_s_file());
+	return PEM_read_bio_DHparams(in, NULL, NULL, NULL);
+}
 
-	if (in == NULL)
-		goto end;
+int ssl_sock_load_global_dh_param_from_file(const char *filename)
+{
+	BIO *in;
+	int ret = -1;
+
+	in = BIO_new(BIO_s_file());
+	if (!in)
+		return -1;
 
 	if (BIO_read_filename(in, filename) <= 0)
 		goto end;
 
-	dh = PEM_read_bio_DHparams(in, NULL, NULL, NULL);
+	global_dh = ssl_sock_get_dh(in);
+	if (global_dh)
+		ret = 0;
 
 end:
-if (in)
-BIO_free(in);
+	BIO_free(in);
 
-	return dh;
-}
-
-int ssl_sock_load_global_dh_param_from_file(const char *filename)
-{
-	global_dh = ssl_sock_get_dh_from_file(filename);
-
-	if (global_dh) {
-		return 0;
-	}
-
-	return -1;
+	return ret;
 }
 
 /* Loads Diffie-Hellman parameter from a file. Returns 1 if loaded, else -1
if an error occured, and 0 if parameter not found. */
-int ssl_sock_load_dh_params(SSL_CTX *ctx, const char *file)
+int ssl_sock_load_dh_params(SSL_CTX *ctx, BIO *in)
 {
 	int ret = -1;
-	DH *dh = ssl_sock_get_dh_from_file(file);
+	DH *dh = ssl_sock_get_dh(in);
 
 	if (dh) {
 		ret = 1;
@@ -3192,10 +3188,9 @@ static int ssl_sock_load_multi_cert(const char *path, struct bind_conf *bind_con
 /* Loads a certificate key and CA chain from a file. Returns 0 on error, -1 if
  * an early error happens and the caller must call SSL_CTX_free() by itelf.
  */
-static int ssl_sock_load_cert_chain_file(SSL_CTX *ctx, const char *file, struct bind_conf *s,
-	 struct ssl_bind_conf *ssl_conf, char **sni_filter, int fcount)
+static int ssl_sock_load_cert_chain(SSL_CTX *ctx, BIO *in, struct bind_conf *s,
+struct ssl_bind_conf *ssl_conf, char **sni_filter, int fcount)
 {
-	BIO *in;
 	X509 *x = NULL, *ca;
 	int i, err;
 	int ret = -1;
@@ -3211,14 +3206,6 @@ static int ssl_sock_load_cert_chain_file(SSL_CTX *ctx, const char *file, struct
 	STACK_OF(GENERAL_NAME) *names;
 #endif
 
-	in = BIO_new(BIO_s_file());
-	if (in == NULL)
-		goto end;
-
-	if (BIO_read_filename(in, file) <= 0)
-		goto end;
-
-
 	passwd_cb = SSL_CTX_get_default_passwd_cb(ctx);
 	passwd_cb_userdata = SSL_CTX_get_default_passwd_cb_userdata(ctx);
 
@@ -3308,44 +3295,110 @@ end:
 	if (x)
 		X509_free(x);
 
-	if (in)
-		BIO_free(in);
-
 	return ret;
 }
 
-static int ssl_sock_load_cert_file(const char *path, struct bind_conf *bind_conf, struct ssl_bind_conf *ssl_conf,
-   char **sni_filter, int fcount, char **err)
+struct ssl_input {
+const char *cert_path;
+struct chunk *cert_buf;
+};
+
+static int ssl_sock_load_cert2(struct ssl_input *si, struct bind_conf *bind_conf, struct ssl_bind_conf *ssl_conf,
+   char **sni_filter, int fcount, char **err)
 {
 	int ret;
 	SSL_CTX *ctx;
+	BIO *in;
+
+	if (!si->cert_path && (!si->cert_buf || !si->cert_buf->len))
+		return 1;
 
 	ctx = SSL_CTX_new(SSLv23_server_method());
 	if (!ctx) {
-		memprintf(err, "%sunable to allocate SSL context for cert '%s'.\n",
-		  err && *err ? *err : "", path);
+		if (si->cert_path)
+			memprintf(err, "%sunable to allocate SSL context for cert '%s'.\n",
+  err && *err ? *err : "", 

Re: remaining process after (seamless) reload

[William Dauchy]

On Wed, May 23, 2018 at 06:49:09PM +0200, William Dauchy wrote:
> We do frequent reloads (approximatively every 10s).
> After a while some processes remains alive and seem to never exit (waited >24
> hours). While stracing them, some of them are still handling traffic and
> doing healthchecks. Some of them are exiting normally after the reload.
> I was wondering how I can help to debug this issue. I assume I won't
> have any other info through the stats socket, since it concerns older
> processes but maybe I missed something.
> Do you have any hint to help me understand what is going on?

More details which could help understand what is going on:

ps output:

root 15928  0.3  0.0 255216 185268 ?   Ss   May21  10:11 
/usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 16988 
16912 6340 28271 30590 30334 -x /var/lib/haproxy/stats
haproxy   6340  2.0  0.0 526172 225476 ?   Ssl  May22  35:03  \_ 
/usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 6328 
6315 -x /var/lib/haproxy/stats
haproxy  28271  1.8  0.0 528720 229508 ?   Ssl  May22  27:13  \_ 
/usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 28258 
28207 28232 6340 -x /var/lib/haproxy/stats
haproxy  30590  265  0.0 527268 225032 ?   Rsl  04:35 2188:55  \_ 
/usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 30578 
28271 6340 -x /var/lib/haproxy/stats
haproxy  30334  197  0.0 526704 224544 ?   Rsl  09:17 1065:59  \_ 
/usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 30322 
30295 27095 6340 28271 30590 -x /var/lib/haproxy/stats
haproxy  16912  1.7  0.0 527544 216552 ?   Ssl  18:14   0:03  \_ 
/usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 16899 
28271 30590 30334 6340 -x /var/lib/haproxy/stats
haproxy  17001  2.2  0.0 528392 214656 ?   Ssl  18:17   0:00  \_ 
/usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf 16988 
16912 6340 28271 30590 30334 -x /var/lib/haproxy/stats


lsof output:

haproxy6340haproxy5u unix 0x883feec97000   0t0  
679289634 /var/lib/haproxy/stats.15928.tmp
haproxy6340  6341  haproxy5u unix 0x883feec97000   0t0  
679289634 /var/lib/haproxy/stats.15928.tmp
haproxy6340  6342  haproxy5u unix 0x883feec97000   0t0  
679289634 /var/lib/haproxy/stats.15928.tmp
haproxy6340  6343  haproxy5u unix 0x883feec97000   0t0  
679289634 /var/lib/haproxy/stats.15928.tmp
haproxy   17020haproxy5u unix 0x883feec97000   0t0  
679289634 /var/lib/haproxy/stats.15928.tmp
haproxy   17020 17021  haproxy5u unix 0x883feec97000   0t0  
679289634 /var/lib/haproxy/stats.15928.tmp
haproxy   17020 17022  haproxy5u unix 0x883feec97000   0t0  
679289634 /var/lib/haproxy/stats.15928.tmp
haproxy   17020 17023  haproxy5u unix 0x883feec97000   0t0  
679289634 /var/lib/haproxy/stats.15928.tmp
haproxy   28271haproxy5u unix 0x883feec97000   0t0  
679289634 /var/lib/haproxy/stats.15928.tmp
haproxy   28271 28272  haproxy5u unix 0x883feec97000   0t0  
679289634 /var/lib/haproxy/stats.15928.tmp
haproxy   28271 28273  haproxy5u unix 0x883feec97000   0t0  
679289634 /var/lib/haproxy/stats.15928.tmp
haproxy   28271 28274  haproxy5u unix 0x883feec97000   0t0  
679289634 /var/lib/haproxy/stats.15928.tmp

(So on unhealthy nodes, I find old processes which are still linked to
the socket.)

The provisioning part is also seeing data which are supposed to be
already updated through the runtime API. I suspect I am getting old
data when connecting to the unix socket. The later being still attached
to an old process?
Indeed, if I try
for i in {1..500}; do sudo echo "show info" | sudo socat stdio  
/var/lib/haproxy/stats  | grep Pid; done

I get "Pid: 17001" most of the time, which is the last process
but I sometimes get: "Pid: 28271"(!) which is a > 24 hours old
process.

Is there something we are doing wrongly?

Best,

-- 
William

Re: Rewrite image path based on HTTP_REQUEST

[Jarno Huuskonen]

Hi,

On Sat, May 19, Aleksandar Lazic wrote:
> On 17/05/2018, Lotic Lists wrote:
> > How can I rewrite a image path based on URL?
> > 
> > Example, users request the url www.example.com/images/logo.png, haproxy just
> > balance to backend servers normally.
> > 
> > Now users request www.newdomain.com, I need rewrite URI to
> > /images/new-logo.png
> 
> Well what have you already tried?
> 
> I would try, untested.
> 
> https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4.2-reqrep
> 
> acl old_dom hdr(host) -i www.example.com
> acl old_path path_beg -i /images/logo.png
> 
> reqrep "^([^ :]*) /images/logo.png /images/new-logo.png" if old_dom && 
> old_path
> reqirep "^Host: www.example.com   "Host: www.newdomain.com" if old_dom && 
> old_path

If you just need to change the path then http-request set-path should be
enough(https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#4-http-request).

Something like:
acl new_dom hdr_dom(host) -i www.newdomain.com
acl old_path path_beg -i /images/logo.png
http-request set-path "/images/new-logo.png" if new_dom old_path

-Jarno

-- 
Jarno Huuskonen - System Administrator |  jarno.huuskonen atsign uef.fi

remaining process after (seamless) reload

[William Dauchy]

Hello,

I am trying to understand a possible issue we have regarding haproxy (seamless)
reloads.
I am using haproxy v1.8.9 with the following config (using nbthread):

global
 log 127.0.0.1 local0 info
 maxconn 262144
 user haproxy
 group haproxy
 nbproc 1
 daemon
 stats socket /var/lib/haproxy/stats level admin mode 644 expose-fd 
listeners
 stats timeout 2m
 tune.bufsize 33792
 ssl-default-bind-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
 ssl-default-bind-ciphers 
ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
 ssl-default-server-options no-sslv3 no-tlsv10 no-tlsv11 no-tls-tickets
 ssl-default-server-ciphers 
ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
 ssl-server-verify none
 crt-base /etc/haproxy/tls/
 nbthread 4
 cpu-map auto:1/1-4 0-3

defaults
 log global
 mode http
 retries 3
 timeout connect 10s
 timeout client 180s
 timeout server 180s
 timeout http-keep-alive 10s
 timeout http-request 10s
 timeout queue 1s
 timeout check 5s
 option httplog
 option dontlognull
 option redispatch
 option prefer-last-server
 option dontlog-normal
 option http-keep-alive
 option forwardfor except 127.0.0.0/8
 balance roundrobin
 maxconn 262134
 http-reuse safe
 default-server inter 5s fastinter 1s fall 3 slowstart 20s observe layer7 
error-limit 1 on-error fail-check
 http-check send-state
 http-check expect status 200 string passing

listen stats
 bind *:8080
 stats enable
 stats uri /haproxy_stats


frontend fe_main
# arbitrary split in two for http/https traffic
bind *:80 name http_1 process 1/1
bind *:80 name http_2 process 1/2
bind *:443 name https_3 ssl crt /etc/haproxy/tls/fe_main process 1/3
bind *:443 name https_4 ssl crt /etc/haproxy/tls/fe_main process 1/4
[...]

The rest of the config contains lost of acl/backends (> 1000)

We do frequent reloads (approximatively every 10s).
After a while some processes remains alive and seem to never exit (waited >24
hours). While stracing them, some of them are still handling traffic and
doing healthchecks. Some of them are exiting normally after the reload.
I was wondering how I can help to debug this issue. I assume I won't
have any other info through the stats socket, since it concerns older
processes but maybe I missed something.
Do you have any hint to help me understand what is going on?

Best regards,

-- 
William

Re: Haproxy 1.8 with OpenSSL 1.1.1-pre4 stops working after 1 hour

[Emeric Brun]

Hi Sander, Lukas,

On 05/23/2018 02:32 PM, Lukas Tribus wrote:
> Hello,
> 
> On 23 May 2018 at 13:10, Sander Hoentjen  wrote:
>> I can confirm the issue is gone when I don't use chroot. I will try to
>> see if I can get more info like a strace soon. I won't be able to today
>> though. Thanks Lucas and Emeric!
> 
> 1.8.9 with 1.1.1-pre6 chrooted is now running for me for more than 22
> hours. I don't see the same issue.
> 
> So are probably we are facing a different issue.
> 
> 
> 
> Lukas
> 

This issue was due to openssl-1.1.1 which re-seed after an elapsed time or 
number of request.

If /dev/urandom is used as seeding source when haproxy is chrooted it fails to 
re-open /dev/urandom 

By defaut the openssl-1.1.1 configure script uses the syscall getrandom as 
seeding source and fallback on /dev/urandom if not available.

So you don't face the issue if your openssl-1.1.1 is compiled to use getrandom

But getrandom syscall is available only since kernel > 3.17 and the main point: 
for glibc > 2.25.

With openssl-1.1.1 you can check this this way:
# ./openssl-1.1.1/openssl version -r
Seeding source: getrandom-syscall


R,
Emeric

Re: DNS resolver + threads, 100% cpu usage / hang 1.9dev

[Willy Tarreau]

Hi Olivier,

On Wed, May 23, 2018 at 04:13:20PM +0200, Olivier Houchard wrote:
> Thanks a lot for testing, and your usual investigation work !
> 
> Willy, can you please apply this ?

Sure, now done, thanks!

Willy

Re: [RFC PATCH] MINOR: ssl: set SSL_OP_PRIORITIZE_CHACHA

[Willy Tarreau]

Hi Lukas,

On Wed, May 23, 2018 at 03:30:55PM +0200, Lukas Tribus wrote:
> You can merge as-is, it was just RFC to see if there are any
> objections to enable this flag unconditionally.

OK now merged, thank you!

Willy

Re: DNS resolver + threads, 100% cpu usage / hang 1.9dev

[Olivier Houchard]

Hi Pieter,

On Tue, May 22, 2018 at 09:00:24PM +0200, PiBa-NL wrote:
> Hi Olivier,
> 
> Op 22-5-2018 om 18:46 schreef Olivier Houchard:
> > Hi Pieter,
> > 
> > Does the attached patch fix it for you ? It's been generated from master,
> > but will probably apply against 1.8 as well.
> > 
> > Thanks !
> > 
> > Olivier
> 
> Patch works for me (on master, didn't try with 1.8). Or at least i'm running
> the same testbox for an hour now without issue.
> Thanks !
> 

Thanks a lot for testing, and your usual investigation work !

Willy, can you please apply this ?

Thanks !

Olivier
>From b938f86e1fe51e95adc73f9e583dd225f5ecf88d Mon Sep 17 00:00:00 2001
From: Olivier Houchard 
Date: Tue, 22 May 2018 18:40:07 +0200
Subject: [PATCH] BUG/MEDIUM: dns: Delay the attempt to run a DNS resolution on
 check failure.

When checks fail, the code tries to run a dns resolution, in case the IP
changed.
The old way of doing that was to check, in case the last dns resolution
hadn't expired yet, if there were an applicable IP, which should be useless,
because it has already be done when the resolution was first done, or to
run a new resolution.
Both are a locking nightmare, and lead to deadlocks, so instead, just wake the
resolvers task, that should do the trick.

This should be backported to 1.8.
---
 src/dns.c | 9 -
 1 file changed, 4 insertions(+), 5 deletions(-)

diff --git a/src/dns.c b/src/dns.c
index e42950212..e2d46a658 100644
--- a/src/dns.c
+++ b/src/dns.c
@@ -348,11 +348,10 @@ void dns_trigger_resolution(struct dns_requester *req)
/* The resolution must not be triggered yet. Use the cached response, if
 * valid */
exp = tick_add(res->last_resolution, resolvers->hold.valid);
-   if (res->status == RSLV_STATUS_VALID &&
-   tick_isset(res->last_resolution) && !tick_is_expired(exp, now_ms))
-   req->requester_cb(req, NULL);
-   else
-   dns_run_resolution(res);
+   if (resolvers->t && (res->status != RSLV_STATUS_VALID ||
+   !tick_isset(res->last_resolution) || tick_is_expired(exp, now_ms)))
+   task_wakeup(resolvers->t, TASK_WOKEN_OTHER);
+
 }
 
 
-- 
2.14.3

Re: [RFC PATCH] MINOR: ssl: set SSL_OP_PRIORITIZE_CHACHA

[Lukas Tribus]

Hi Willy,


On 22 May 2018 at 18:54, Willy Tarreau  wrote:
> On Tue, May 22, 2018 at 04:28:38PM +0200, Emeric Brun wrote:
>> I agree, we could merge it as it is.
>
> OK thanks Emeric.
>
> So Lukas, just let me know if you want me to merge it as-is or if you
> still have some polishing to do on it, since you marked it RFC.

You can merge as-is, it was just RFC to see if there are any
objections to enable this flag unconditionally.


Thanks,
Lukas

Re: SSL certs loading performance regression

[Emmanuel Hocdet]

Hi Hervé,

> Le 22 mai 2018 à 10:31, Hervé Commowick  a 
> écrit :
> 
> Hello HAProxy ML,
> 
> I tracked down a performance regression about loading bunch of
> certificates, at least 3x to 5x more time for loading 10 certs since
> this commit
> http://git.haproxy.org/?p=haproxy-1.8.git;a=commitdiff;h=f6b37c67be277b5f0ae60438d796ff29ef19be40
> 
> This regression is 1.8 specific, (no issue in 1.6 or 1.7 branch)
> 
> my bind line :
> bind 127.0.0.1:1443 ssl crt ssl10k ecdhe secp384r1
> 
> After some tests with William, it looks like it is also related to
> "ecdhe secp384r1" parameter, i don't really understand why, but without
> this i don't see any regression (and it looks like secp384r1 was
> effectively working in old version)
> 

can you try with « curves » parameter and not the old « ecdhe » ?

> Let me know if i can test something, from 1min30s to 5min has some
> impacts as you can understand :-)
> 

Manu.

Re: Haproxy 1.8 with OpenSSL 1.1.1-pre4 stops working after 1 hour

[Lukas Tribus]

Hello,

On 23 May 2018 at 13:10, Sander Hoentjen  wrote:
> I can confirm the issue is gone when I don't use chroot. I will try to
> see if I can get more info like a strace soon. I won't be able to today
> though. Thanks Lucas and Emeric!

1.8.9 with 1.1.1-pre6 chrooted is now running for me for more than 22
hours. I don't see the same issue.

So are probably we are facing a different issue.



Lukas

Process crash on reload with TLS tickets

[Janusz Dziemidowicz]

Hi,
this seems harmless, but haproxy processes crash on reload when using
TLS tickets with multiple sockets per port.

Following configuration reproduces the problem:
global
  nbproc 2
  user haproxy
  group haproxy
  daemon

defaults
  timeout connect 5000
  timeout client  5
  timeout server  5

frontend test
  bind 127.0.0.1:8443 process 1 ssl crt file tls-ticket-keys file
  bind 127.0.0.1:8443 process 2 ssl crt file tls-ticket-keys file

Every reload results with the following warning in logs:
[WARNING] 142/134019 (23389) : Former worker 23404 exited with code 134

gdb shows following:
Program received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x7fc87c8cc42a in __GI_abort () at abort.c:89
#2  0x7fc87c908c00 in __libc_message (do_abort=do_abort@entry=2,
fmt=fmt@entry=0x7fc87c9fdd98 "*** Error in `%s': %s: 0x%s ***\n") at
../sysdeps/posix/libc_fatal.c:175
#3  0x7fc87c90efc6 in malloc_printerr (action=3,
str=0x7fc87c9fde10 "double free or corruption (!prev)", ptr=, ar_ptr=) at malloc.c:5049
#4  0x7fc87c90f80e in _int_free (av=0x7fc87cc31b00 ,
p=0x55f389a1a4a0, have_lock=0) at malloc.c:3905
#5  0x55f387b9aa16 in ssl_sock_destroy_bind_conf
(bind_conf=0x55f389a1c400) at src/ssl_sock.c:4818
#6  0x55f387c25280 in deinit () at src/haproxy.c:2240
#7  0x55f387b8846e in main (argc=,
argv=0x7ffed2261cc8) at src/haproxy.c:3070

src/ssl_sock.c:4818 contains:
  free(bind_conf->keys_ref->tlskeys);

haproxy version (1.8.9 recompiled on Debian stretch with USE_TFO):
$ /usr/sbin/haproxy -vv
HA-Proxy version 1.8.9-1~tsg9+1 2018/05/21
Copyright 2000-2018 Willy Tarreau 

Build options :
  TARGET  = linux2628
  CPU = generic
  CC  = gcc
  CFLAGS  = -g -O2 -fdebug-prefix-map=/root/haproxy-1.8.9=.
-fstack-protector-strong -Wformat -Werror=format-security -Wdate-time
-D_FORTIFY_SOURCE=2
  OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1
USE_LUA=1 USE_SYSTEMD=1 USE_PCRE=1 USE_PCRE_JIT=1 USE_TFO=1 USE_NS=1

Default settings :
  maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with OpenSSL version : OpenSSL 1.1.0f  25 May 2017
Running on OpenSSL version : OpenSSL 1.1.0f  25 May 2017
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.3
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.39 2016-06-14
Running on PCRE version : 8.39 2016-06-14
PCRE library supports JIT : yes
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.

Available polling systems :
  epoll : pref=300,  test result OK
   poll : pref=200,  test result OK
 select : pref=150,  test result OK
Total: 3 (3 usable), will use epoll.

Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace


-- 
Janusz Dziemidowicz

Re: Fwd: [haproxy/haproxy] BUG/MAJOR: server: Segfault after parsing server state file. (0bedb8a)

[Tim Düsterhus]

Willy,(writing from my phone, blame it, if I mess up the quotes) Am 23.05.2018 11:20 vorm. schrieb Willy Tarreau :
Well, please post here instead, it's where people are present and

follow the activity. I'm adding Fred in CC since he's the one who fixed

the crash, and Baptiste as well since he's the state file designer.
Please note that the initial comment on GitHub is not written by me. I'm very well aware that the list is to be used.I'm just the messenger, I don't know the original author and I suspect that the original author will not read this mailing list thread, because GitHub does not give out their email address. Someone should probably comment on the GitHub commit, hinting to the mailing list. I can do so tomorrow when I'm back at my computer. Best regardsTim Düsterhus

gRPC protocol

[Aleksandar Lazic]

Hi.

Any plans to add the gRPC Protocol into haproxy?

Have anyone used https://github.com/jinq0123/grpc-lua or similar in
haproxy?

In some areas is the gRPC Protocol more and more famous due to this fact
I ask just ask what's the plan for this protocol is like mqtt?

Best regards
Aleks

Re: Haproxy 1.8 with OpenSSL 1.1.1-pre4 stops working after 1 hour

[Sander Hoentjen]

On 05/22/2018 04:31 PM, Sander Hoentjen wrote:
> On 05/22/2018 04:19 PM, Emeric Brun wrote:
>> Hi Sander,
>>
>> On 05/22/2018 02:04 PM, Sander Hoentjen wrote:
>>> On 05/22/2018 12:04 PM, Lukas Tribus wrote:
 Hello,

 On 22 May 2018 at 11:48, Sander Hoentjen  wrote:
> I did, but I still experience the same issues. What is your exact
> haproxy version you tested with? Mine is 1.8.8
> Built with OpenSSL version : OpenSSL 1.1.1-pre6 (beta) 1 May 2018
> Running on OpenSSL version : OpenSSL 1.1.1-pre6 (beta) 1 May 2018
 I'm using the development tree. So if it doesn't depend on openssl, it
 may be bug that has been fixed in haproxy itself.

 Can you try 1.8.9?
>>> I just tried HA-Proxy version 1.8.9-83616ec 2018/05/18, it has the same
>>> issue unfortunately.
>>>
>> Do you use chroot ?
>>
>> I remember a behavior change between 1.1.0 and 1.1.1. After an elapsed time 
>> (or a number of new sess) openssl
>> try to re-open some /dev/Xrandom sources. And it could fail in a chroot.
>>
>> https://github.com/openssl/openssl/issues/5330
>>
>>
> Thanks, I do use chroot. In that issue it is mentioned that this problem
> has been fixed, but I will test without chroot anyway. Will report
> tomorrow if this worked or not.
I can confirm the issue is gone when I don't use chroot. I will try to
see if I can get more info like a strace soon. I won't be able to today
though. Thanks Lucas and Emeric!

Re: SPOE and modsecurity contrib

[Willy Tarreau]

On Sun, May 20, 2018 at 10:59:02AM -0400, Daniel Corbett wrote:
> While I haven't been able to get 'tcp-request content reject' to work with
> this configuration -- I am able to get 'http-request deny' to work:
> 
> http-request deny if { var(txn.modsec.code) -m int gt 0 }

This is expected. The "tcp-request content" rules are evaluated before
HTTP rules, hence before the filters as well. Thus here the rule says
that it will reject the request based on a variable that holds its initial
value and never had a chance to be modified yet. Unfortunately there is
little we can do to detect this better, except improving the doc and
providing better config examples.

Willy

Re: Fwd: [haproxy/haproxy] BUG/MAJOR: server: Segfault after parsing server state file. (0bedb8a)

[Willy Tarreau]

Hi Tim,

On Tue, May 22, 2018 at 09:52:16PM +0200, Tim Düsterhus wrote:
> Hi list
> 
> the following comment has been posted to GitHub on commit
> 0bedb8ac90ffdf1498a999c44d1c91556fb726ee
> 
> https://github.com/haproxy/haproxy/commit/0bedb8ac90ffdf1498a999c44d1c91556fb726ee#commitcomment-29087381

Well, please post here instead, it's where people are present and
follow the activity. I'm adding Fred in CC since he's the one who fixed
the crash, and Baptiste as well since he's the state file designer.

So it's about commit 0bedb8a ("BUG/MAJOR: server: Segfault after parsing
server state file.").

> @wtarreau This patch causes the state of the haproxy to be discarded by
> the haproxy unless the server id of the backend server is not fixed.
> Normally they are auto-generated and removing e.g. 1st server on the
> list of backends results in renumerating all server ids :(

So if I understand right, the problem is in fact that servers from the
state file which don't exist in the config file are reported as mismatches,
thus the code for the ID/name mismatch was modified instead of modifying
the special case. I don't know what could be done to detect such cases.
At the very least, given that the previous code used to crash in this
case, it means this condition differs in a detectable way (possibly that
a server lookup returns null or something like this).

I'll let Fred and Baptiste take a look.

Thanks,
Willy

Re: [PATCH] MINOR: http: Log warning if (add|set)-header fails

[Willy Tarreau]

Hi Tim,

On Sun, May 20, 2018 at 05:55:06PM +0200, Tim Duesterhus wrote:
> Willy,
> 
> attached is a first attempt at a patch that adds logging (without any rate
> limiting). I have a few questions regarding the whole counters and logging
> infrastructure:
> 
> 1. I noticed that there is ha_warning(...) and send_log(p, LOG_WARNING, ...),
>usually both are used in the same place. What's the difference between
>them? Are the log lines in my patch going into the correct place?

ha_warning() is used to emit warnings on stderr during startup, mainly
when parsing the config. Example of such warnings are "missing timeouts
for proxy foobar". send_log() is used at run time to either report activity
(regular logs) or send notification about noticeable events like servers
going up/down, or risks for the reliability due to external events (no more
FDs) or issues like this one.

Historically we've accumulated a few ha_warning() in runtime code paths,
but each time you see this it's an error. Sometimes due to a thinko,
sometimes due to code being moved or reused.

> 2. How do you prefer the rate limiting of the log lines?
>- Is there an existing "flag" for rate limited logs?

Not that I'm aware of.

>- Where should the information about the rate limited logging be stored
>  (in a static variable?)?

At a few places (which ones, I don't remember) we have a local static
variable to report certain elements only once. It could be one option.
But the problem is that if the first event was missed, the new ones
will be missed as well.

>- Log only every Xth failed call to http_header_add_tail2?

It could be the easiest and most convenient solution for the mid term
I think. E.g. just log once every 256 using "if (!(counter++ & 255)) ..."

>- By time? This would require an additional call to gettimeofday, no?

It's not welcome there. We could rely on a freq_counter though, but I'm
not sure it's worth the extra pain (enthough it's likely that over the
long term that's what it will evolve to). If you prefer to do it anyway,
it will look roughly like this :

 #define LIMIT  10   // burst of 10 messages max
 #define PERIOD 1// over 10 seconds
 static struct freq_ctr_period warn_limiter;
 static unsigned int skipped;

 if (freq_ctr_remain_period(_limiter, PERIOD, LIMIT, 0)) {
  update_freq_ctr_period(_limiter, PERIOD, 1);
  send_log("log: blah (skipped %d messages)", skipped);
  skipped=0;
 } else {
  skipped++;
 }
  
>- Should the code have separate rate limiting per header (I guess no,
>  because the only meaningful thing a administrator can do is increase
>  tune.maxrewrite and that fixes all headers at once)?

It would really be overkill and would provide little to no value. As you
say, what matters is to alert against too low maxrewrite. However using a
dedicated log function to warn about such problems and maintain its own
counter definitely makes sense to me.

> 3. Regarding the counters I noticed frontends don't really seem to use
>error counters yet. Do I need to add a counter to both
>struct fe_counters as well as struct be_counters and increment depending
>on whether the (add|set)-header line is in a backend or frontend?

I'm not sure what you mean about frontends not using error counters,
because the request parsing issues are reported there, just like some of
the errors causing an error to be reported. The frontend currently tracks
failed_req, denied_{conn,sess,req,resp}, and {cli,srv}_aborts.

> diff --git a/src/proto_http.c b/src/proto_http.c
> index 3adb54f2..9739fc7a 100644
> --- a/src/proto_http.c
> +++ b/src/proto_http.c
> @@ -2631,7 +2631,11 @@ resume_execution:
>   }
>   }
>  
> - http_header_add_tail2(>req, >hdr_idx, 
> replace->str, replace->len);
> + if (http_header_add_tail2(>req, >hdr_idx, 
> replace->str, replace->len) < 0) {
> + replace->str[rule->arg.hdr_add.name_len] = 0;
> + ha_warning("Proxy %s failed to add or set the 
> request header '%s'. You might need to increase tune.maxrewrite.\n", px->id, 
> replace->str);
> + send_log(px, LOG_WARNING, "Proxy %s failed to 
> add or set the request header '%s'. You might need to increase 
> tune.maxrewrite.", px->id, replace->str);
> + }

In my opinion it could be useful to add "for request #%u",s->uniq_id.
This ID can be retrieved using "%rt" in the logs, so for those who need
to understand what request broke in order to know a bit more about it,
it will definitely help.

thanks,
Willy