1.8.0 stuck in write(threads_sync_pipe[1], "S", 1)
Hi!
Tonight all of mine haproxy 1.8.0 instances stopped answering. They didn't
forward traffic and even didn't answered over socket. They're compiled with
threads, but threads are not enabled in they configs (no nbthread option).
All of them stuck in same place:
# strace -f -p 831919
Process 831919 attached
write(2, "S", 1
Here's some debug stuff (from 1-threaded instance):
(gdb) bt
#0 0x7fef9bd2a330 in __write_nocancel () at
../sysdeps/unix/syscall-template.S:81
#1 0x558dea62275b in thread_want_sync () at src/hathreads.c:74
#2 0x558dea58f548 in srv_register_update (srv=srv@entry=0x558ded691e30)
at src/server.c:2596
#3 0x558dea5922f7 in server_recalc_eweight (sv=sv@entry=0x558ded691e30)
at src/server.c:1151
#4 0x558dea5c3028 in server_warmup (t=0x558def513120) at
src/checks.c:1448
#5 0x558dea619216 in process_runnable_tasks () at src/task.c:229
#6 0x558dea5cf237 in run_poll_loop () at src/haproxy.c:2326
#7 run_thread_poll_loop (data=) at src/haproxy.c:2375
#8 0x558dea53b6fe in main (argc=, argv=0x7ffe03a880d8)
at src/haproxy.c:2910
(gdb) bt full
#0 0x7fef9bd2a330 in __write_nocancel () at
../sysdeps/unix/syscall-template.S:81
No locals.
#1 0x558dea62275b in thread_want_sync () at src/hathreads.c:74
No locals.
#2 0x558dea58f548 in srv_register_update (srv=srv@entry=0x558ded691e30)
at src/server.c:2596
No locals.
#3 0x558dea5922f7 in server_recalc_eweight (sv=sv@entry=0x558ded691e30)
at src/server.c:1151
px =
w =
#4 0x558dea5c3028 in server_warmup (t=0x558def513120) at
src/checks.c:1448
s = 0x558ded691e30
#5 0x558dea619216 in process_runnable_tasks () at src/task.c:229
t = 0x558def513120
i =
max_processed = 173
rq_next =
local_tasks = {0xcbe, 0x558dea61d5d7 ,
0x558e1f68b5e0, 0x402770, 0x558dea894350 , 0x558e19e7f4e0, 0x0,
0x558dea539f47 , 0x202310, 0x558e19e7f4e0,
0x558dea60e979 , 0x7ffe03a87d80, 0x50004,
0x558dea61e6e2 , 0x7ffe03a87d80,
0x7ffe03a87d80}
local_tasks_count =
final_tasks_count =
#6 0x558dea5cf237 in run_poll_loop () at src/haproxy.c:2326
next =
#7 run_thread_poll_loop (data=) at src/haproxy.c:2375
ptif =
ptdf =
#8 0x558dea53b6fe in main (argc=, argv=0x7ffe03a880d8)
at src/haproxy.c:2910
tids = 0x558def33ad30
threads = 0x558def46d050
i =
err =
retry =
limit = {rlim_cur = 42419, rlim_max = 42419}
errmsg = "\000\000\000\000\000\000\000\000n\000\000\000w", '\000'
,
"\017\177\250\003\376\177\000\000\260\033e\354\215U\000\000`W&\233\357\177\000\000|\000\000\000\000\000\000\000
\000\000\000\000\000\000\000P
e\354\215U\000\000`W&\233\357\177\000\000\260o\001\000\000\000\000\000\200\000\000\000\000\000\000\000P
",
pidfd =
(gdb) thread
[Current thread is 1 (Thread 0x7fef9c59d980 (LWP 831919))]
root@mslb02e:/var/tmp# /usr/sbin/haproxy -vv
HA-Proxy version 1.8.0-4 2017/11/29
Copyright 2000-2017 Willy Tarreau
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS = -g -O2 -fPIE -fstack-protector --param=ssp-buffer-size=4
-Wformat -Werror=format-security -D_FORTIFY_SOURCE=2
OPTIONS = USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_THREAD=1
USE_OPENSSL=1 USE_LUA=1 USE_PCRE=1 USE_TFO=1 USE_NS=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 1024, maxpollevents = 200
Built with OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
Running on OpenSSL version : OpenSSL 1.0.1f 6 Jan 2014
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : SSLv3 TLSv1.0 TLSv1.1 TLSv1.2
Built with Lua version : Lua 5.3.1
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT
IP_FREEBIND
Encrypted password support via crypt(3): yes
Built with multi-threading support.
Built with PCRE version : 8.31 2012-07-06
Running on PCRE version : 8.31 2012-07-06
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with zlib version : 1.2.8
Running on zlib version : 1.2.8
Compression algorithms supported : identity("identity"),
deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with network namespace support.
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
Available filters :
[SPOE] spoe
[COMP] compression
[TRACE] trace
--
Best regards,
Maksim Kupriianov
Client cert verification on some paths
Hi, I have some apps that need to mimic an Apache httpd behavior on client certificate verification: require certificate only on some paths. Apache does this implementing SSL renegotiation as briefly explained here[1]. Of couse I can `mode tcp` proxy to an Apache instance to do that for me but my topology would be simplified if I could implement SSL renegotiation on HAProxy as soon as I can fetch the path sample. Is there a way to accomplish this without using Apache httpd? ~jm [1] http://httpd.apache.org/docs/2.4/mod/mod_ssl.html#sslverifyclient
What can cause RST from HAProxy 1.7.9
Hi, HAProxy 1.7.9 is being used to route traffic to a Kubernetes cluster on AWS. It was observed periodic spikes of RST from HAProxy on active connections. Full details in the following issue from GitHub: https://github.com/jcmoraisjr/haproxy-ingress/issues/77 . In which circumstances HAProxy send RST? Any help would be very much appreciated. ~jm
Re: Segfault with 1.8.0 build (RHEL5, old gcc).
Hi Christopher,
On Fri, Dec 01, 2017 at 11:59:14AM -0800, Christopher Lane wrote:
> gist with backtrace, -vv output, and config file. Also strace.
>
> https://gist.github.com/jayalane/c6dbe7918aa9635b62c874d20f57dfec
>
> It does all the listens and then right after the first epoll is done it has
> this segv. all the local variables are "optimized out"
>
> (We really want the hard-stop-after -- we get a leak of children with our
> super frequent soft-reloads).
>
> It's possible something is messed up in my compiling/linking environment,
> but I don't see anything, and I thought maybe a .0 release had a chance of
> issues on very old RHEL versions. We currently run 1.5 and have to
> manually clean up the left over children (but are hoping to scale up to
> nbproc 4 and use CPU pinning).
>
> I'm debugging also, but hoped someone would have something off the top of
> their head (apologies for not having read the list for a while before
> posting).
>
Thanks a lot for the very detailled report !
I think I know what's going on.
In your config file, you have :
server stage.qa.example.com:11302 stage.qa.example.com check inter 2m
fastinter 2s fall 2
It should probably be :
server stage.qa.example.com stage.qa.example.com:11302 check inter 2m
fastinter 2s fall 2
However this is of course no reason to segfault.
Can you please try the attached patch, and let me know if it fixes your issue ?
Thanks a lot !
Olivier
>From 5236a1a4ac19cc27c6f06d328b2df0c4cdfe220c Mon Sep 17 00:00:00 2001
From: Olivier Houchard
Date: Fri, 1 Dec 2017 22:04:05 +0100
Subject: [PATCH] MINOR: checks: Be sure we have a mux if we created a cs.
In connect_conn_chk(), there were one case we could return with a new
conn_stream created, but no mux attached. With no mux, cs_destroy() would
segfault. Fix that by setting the mux before we can fail.
This should be backported to 1.8.
---
src/checks.c | 18 --
1 file changed, 8 insertions(+), 10 deletions(-)
diff --git a/src/checks.c b/src/checks.c
index 63747201e..eaf84a225 100644
--- a/src/checks.c
+++ b/src/checks.c
@@ -1564,25 +1564,23 @@ static int connect_conn_chk(struct task *t)
conn->addr.to = s->addr;
}
+ proto = protocol_by_family(conn->addr.to.ss_family);
+
+ conn_prepare(conn, proto, check->xprt);
+ conn_install_mux(conn, &mux_pt_ops, cs);
+ cs_attach(cs, check, &check_conn_cb);
+ conn->target = &s->obj_type;
+
if ((conn->addr.to.ss_family == AF_INET) || (conn->addr.to.ss_family ==
AF_INET6)) {
int i = 0;
i = srv_check_healthcheck_port(check);
- if (i == 0) {
- cs->data = check;
+ if (i == 0)
return SF_ERR_CHK_PORT;
- }
set_host_port(&conn->addr.to, i);
}
- proto = protocol_by_family(conn->addr.to.ss_family);
-
- conn_prepare(conn, proto, check->xprt);
- conn_install_mux(conn, &mux_pt_ops, cs);
- cs_attach(cs, check, &check_conn_cb);
- conn->target = &s->obj_type;
-
/* no client address */
clear_addr(&conn->addr.from);
--
2.14.3
Segfault with 1.8.0 build (RHEL5, old gcc).
gist with backtrace, -vv output, and config file. Also strace. https://gist.github.com/jayalane/c6dbe7918aa9635b62c874d20f57dfec It does all the listens and then right after the first epoll is done it has this segv. all the local variables are "optimized out" (We really want the hard-stop-after -- we get a leak of children with our super frequent soft-reloads). It's possible something is messed up in my compiling/linking environment, but I don't see anything, and I thought maybe a .0 release had a chance of issues on very old RHEL versions. We currently run 1.5 and have to manually clean up the left over children (but are hoping to scale up to nbproc 4 and use CPU pinning). I'm debugging also, but hoped someone would have something off the top of their head (apologies for not having read the list for a while before posting). --Chris
[PATCH] Make thread affinity work on FreeBSD
Hi,
The attached patch makes the call to pthread_setaffinity_np() work on
FreeBSD.
Regards,
Olivier
>From fc204ac3d7f9323b6583465ff5b42a0cfa46b8b1 Mon Sep 17 00:00:00 2001
From: Olivier Houchard
Date: Fri, 1 Dec 2017 18:19:43 +0100
Subject: [PATCH] MINOR: threads: Fix pthread_setaffinity_np on FreeBSD.
As with the call to cpuset_setaffinity(), FreeBSD expects the argument to
pthread_setaffinity_np() to be a cpuset_t, not an unsigned long, so the call
was silently failing.
This should probably be backported to 1.8.
---
src/haproxy.c | 20 +---
1 file changed, 17 insertions(+), 3 deletions(-)
diff --git a/src/haproxy.c b/src/haproxy.c
index a1fe550e1..4e3c82b86 100644
--- a/src/haproxy.c
+++ b/src/haproxy.c
@@ -2900,10 +2900,24 @@ int main(int argc, char **argv)
global.cpu_map.thread[relative_pid-1][i] &=
global.cpu_map.proc[relative_pid-1];
if (i < LONGBITS && /* only the first 32/64
threads may be pinned */
- global.cpu_map.thread[relative_pid-1][i]) /* only
do this if the thread has a THREAD map */
+ global.cpu_map.thread[relative_pid-1][i]) {/* only
do this if the thread has a THREAD map */
+#if defined(__FreeBSD__) || defined(__NetBSD__)
+ cpuset_t cpuset;
+#else
+ cpu_set_t cpuset;
+#endif
+ int j;
+ unsigned long cpu_map =
global.cpu_map.thread[relative_pid-1][i];
+
+ CPU_ZERO(&cpuset);
+
+ while ((j = ffsl(cpu_map)) > 0) {
+ CPU_SET(j - 1, &cpuset);
+ cpu_map &= ~(1 << (j - 1));
+ }
pthread_setaffinity_np(threads[i],
- sizeof(unsigned long),
- (void
*)&global.cpu_map.thread[relative_pid-1][i]);
+ sizeof(cpuset), &cpuset);
+ }
}
#endif /* !USE_CPU_AFFINITY */
--
2.14.3
Re: [PATCH]: BUILD/MINOR : cfgparse: parse_cpu_set under condition
On Fri, Dec 01, 2017 at 03:00:13PM +, David CARLIER wrote: > Hi, > > Here a little patch proposal to compile this function only when needed. Good catch, thank you David, now merged and backported. Willy
[PATCH]: BUILD/MINOR : cfgparse: parse_cpu_set under condition
Hi,
Here a little patch proposal to compile this function only when needed.
Hope it is good.
Thanks.
From 7a1053f25331d21e164eae30dc1bad2400914f8d Mon Sep 17 00:00:00 2001
From: David Carlier
Date: Fri, 1 Dec 2017 09:14:02 +
Subject: [PATCH] BUILD/MINOR: haproxy: compiling config cpu parsing handling
when needed
parse_cpu_set is only relevant where there is cpu affinity,
avoiding in the process compilation warning as well.
---
src/cfgparse.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/src/cfgparse.c b/src/cfgparse.c
index e7e32cf9..b3202a68 100644
--- a/src/cfgparse.c
+++ b/src/cfgparse.c
@@ -646,6 +646,7 @@ int parse_process_number(const char *arg, unsigned long *proc, int *autoinc, cha
return 0;
}
+#ifdef USE_CPU_AFFINITY
/* Parse cpu sets. Each CPU set is either a unique number between 0 and
* or a range with two such numbers delimited by a dash
* ('-'). Multiple CPU numbers or ranges may be specified. On success, it
@@ -687,6 +688,8 @@ static unsigned long parse_cpu_set(const char **args, unsigned long *cpu_set, ch
}
return 0;
}
+#endif
+
/*
* parse a line in a section. Returns the error code, 0 if OK, or
* any combination of :
--
2.14.2
MQTT client id in Haproxy
Hi All, Is it possible to get client id in haproxy in tcp mode just like http mode. I'm trying to get mqtt client id in haproxy tcp mode Similar case in nginx: https://www.nginx.com/blog/nginx-plus-iot-load-balancing-mqtt/ Regards, CJ
Re: Question to halog
-- Originalnachricht -- Von: "Aleksandar Lazic" An: "haproxy" Gesendet: 01.12.2017 12:59:34 Betreff: Question to halog Hi all. I have the following log line and want to parse it with the halog tool. '2017-12-01T12:12:35+01:00 localhost haproxy[14]: 1.2.3.4:55866 [01/Dec/2017:12:12:02.628] public be_01/server_01 0/0/2/32379/32381 500 969 - - 9/9/2/3/0 0/0 "POST /PATH/Service HTTP/1.1"' |./halog -H I always get a parsing error. I tried also to skip some fields '-s 5' but still parsing errors. Answer to my self. ./halog - H -s -1 -e -u The default skipfield is 5, this line tells halog to start at field 4 Sorry for the rush It's halog tool from haproxy 1.5.19 This is a logline from a rsyslog with omstdout module. Thanks for any help. Regards Aleks
Question to halog
Hi all. I have the following log line and want to parse it with the halog tool. '2017-12-01T12:12:35+01:00 localhost haproxy[14]: 1.2.3.4:55866 [01/Dec/2017:12:12:02.628] public be_01/server_01 0/0/2/32379/32381 500 969 - - 9/9/2/3/0 0/0 "POST /PATH/Service HTTP/1.1"' |./halog -H I always get a parsing error. I tried also to skip some fields '-s 5' but still parsing errors. It's halog tool from haproxy 1.5.19 This is a logline from a rsyslog with omstdout module. Thanks for any help. Regards Aleks
