Got some results from strace. Running the reload with sudo takes about 3
minutes and shows that it spends most of the time on:
14:39:38.077925 poll([{fd=6, events=POLLIN}], 1, -1) = ?
ERESTART_RESTARTBLOCK (Interrupted by signal) <180.434742>
But running without sudo it's a bit faster (about 2 minutes) and it spends
a lot of time on multiple calls like this (note the 5 second timeout on the
last row):
14:47:35.040118 ioctl(4, FIONREAD, [149]) = 0 <0.000011>
14:47:35.040156 recvfrom(4,
"\201\t\201\200\0\1\0\1\0\5\0\0\10commodus\6loop54\2se\0"..., 1024, 0,
{sa_family=AF_INET, sin_port=htons(53), sin_addr=inet_addr("213.80.98.2")},
[16]) = 149 <0.000011>
14:47:35.040200 close(4) = 0 <0.000014>
14:47:35.040313 brk(0x3ba7000) = 0x3ba7000 <0.000013>
14:47:35.040440 brk(0x3bcb000) = 0x3bcb000 <0.000012>
14:47:35.040542 stat("/etc/resolv.conf", {st_mode=S_IFREG|0644,
st_size=198, ...}) = 0 <0.000014>
14:47:35.040591 stat("/etc/resolv.conf", {st_mode=S_IFREG|0644,
st_size=198, ...}) = 0 <0.000013>
14:47:35.040635 open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 4 <0.000015>
14:47:35.040678 fstat(4, {st_mode=S_IFREG|0644, st_size=183, ...}) = 0
<0.000010>
14:47:35.040718 read(4, "127.0.0.1\tlocalhost\n127.0.1.1\tha"..., 4096) =
183 <0.000011>
14:47:35.040760 read(4, "", 4096) = 0 <0.000010>
14:47:35.040795 close(4) = 0 <0.000011>
14:47:35.040831 stat("/etc/resolv.conf", {st_mode=S_IFREG|0644,
st_size=198, ...}) = 0 <0.000013>
14:47:35.040876 socket(PF_INET, SOCK_DGRAM|SOCK_NONBLOCK, IPPROTO_IP) = 4
<0.000014>
14:47:35.040914 connect(4, {sa_family=AF_INET, sin_port=htons(53),
sin_addr=inet_addr("213.80.98.2")}, 16) = 0 <0.000014>
14:47:35.040955 poll([{fd=4, events=POLLOUT}], 1, 0) = 1 ([{fd=4,
revents=POLLOUT}]) <0.000011>
14:47:35.041015 sendto(4,
"o\5\1\0\0\1\0\0\0\0\0\0\4jaws\6loop54\2se\0\0\1\0\1", 32, MSG_NOSIGNAL,
NULL, 0) = 32 <0.000023>
14:47:35.041067 poll([{fd=4, events=POLLIN}], 1, 5000) = 0 (Timeout)
<5.005048>
It seems to be doing this for each of our backends, but in some cases it
gets a timeout which stalls the process for 5 seconds. On our other servers
it seems to run the same calls but it doesn't timeout as often so the total
time is much faster.
So some follow up questions:
1. Does it try to connect to each backend when starting up and won't start
until it has tested all of them? If so, can we disable this?
2. What is the poll([{fd=6, events=POLLIN}], 1, -1) call that takes 3
minutes when running with sudo?
----------------------------
Joel W Kall
*Loop54*
Barnhusgatan 20
111 23 Stockholm
www.loop54.com
Mobil: 073-5933 255
[email protected]
2017-10-13 14:13 GMT+02:00 Joel W Kall <[email protected]>:
> Hi and thanks for your answer.
>
> We have 3 certificates. Since this only happens on one server and we have
> the exact same conf and certs on all servers I don't think that's the
> reason, though.
>
> This is our conf:
>
> ###
> global
> log 127.0.0.1 local0 notice
> maxconn 20000
> user haproxy
> group haproxy
> tune.ssl.default-dh-param 2048
>
> defaults
> log global
> mode http
> option httplog
> option dontlognull
> option http-keep-alive
> option forwardfor
> retries 3
> option redispatch
> timeout connect 5s
> timeout check 5s
> timeout client 60000
> timeout server 60000
>
> stats enable
> stats uri /haproxy?stats
> stats auth [REDACTED]
>
> option httpchk GET / HTTP/1.0
> balance roundrobin
> default-server inter 10s fall 2 rise 2
>
> frontend http-in
> bind *:80
>
> # Define hosts
> acl host_1 hdr(host) -i somehost.somedomain.com
> [more host header configurations]
>
> ## switches
> use_backend 1 if host_1
> [more if-clauses]
>
> frontend https-in
> bind *:443 ssl crt [REDACTED] crt [REDACTED] crt [REDACTED]
>
> # Define hosts
> acl host_1 hdr(host) -i somehost.somedomain.com
> [more host header configurations]
>
> ## switches
> use_backend 1 if host_1
> [more if-clauses]
>
> backend 1
> server node1 [some IP] check
> server node2 [some IP] check
>
> [more backends]
> ###
>
> Will try with strace and see what we see.
>
>
> ----------------------------
> Joel W Kall
>
>
