Hello list! I believe I've found a bug in haproxy related to multiproc and a set of DNS resolvers. What happens is, when combining these two features (multiproc and dynamic resolvers), I get the following problem: the DNS resolvers, one per process it seems, will fail intermittently and independently for no obvious reason, and this triggers a DOWN event in the backend; a short time later, the resolution succeeds and the backend goes back UP for a short time, before repeating indefinitely. This bug also seems to have a curious effect of causing the active record type to switch from A to AAAA and then back to A repeatedly in a dual-stack setup, though the test below shows that this bug occurs in an IPv4-only environment as well, and this failure is not documented in my tests.
First, some background. I'm attempting to set up an haproxy instance with multiple processes for SSL termination. At the same time, I'm also trying to use a IPv6 backend managed by DNS, so I set up a "resolvers" section so I could use resolved IPv6 addresses from AAAA records. As an aside, I've noticed that haproxy will not start up if the only record for a host is an AAAA record, reporting that the address can't be resolved. However since I run a dual-stack [A + AAAA] record setup normally, this is not a huge deal to me, though I think supporting an IPv6/AAAA-only backend should definitely be a future goal! First my config (figure 1). This is the most basic config I can construct that triggers the bug while keeping most of my important settings; note the host resolves to an A record only (figure 2); this record is provided by a dnsmasq process which has read it out of /etc/hosts, so the resolution here should be 100% stable. (figure 1) | global | log ::1:514 daemon debug | log-send-hostname | chroot /var/lib/haproxy | pidfile /run/haproxy/haproxy.pid | nbproc 2 | cpu-map 1 0 | cpu-map 2 1 | stats socket /var/lib/haproxy/admin-1.sock mode 660 level admin process 1 | stats socket /var/lib/haproxy/admin-2.sock mode 660 level admin process 2 | stats timeout 30s | user haproxy | group haproxy | daemon | maxconn 10000 | resolvers dns | nameserver dnsmasq 127.0.0.1:53 | resolve_retries 1 | hold valid 1s | hold timeout 1s | timeout retry 1s | defaults | log global | option http-keep-alive | option forwardfor except 127.0.0.0/8 | option redispatch | option dontlognull | option forwardfor | timeout connect 5s | timeout client 24h | timeout server 60m | listen back_deb-http | bind 0.0.0.0:80 | mode http | option httpchk GET /debian/haproxy | server deb deb.domain.net:80/debian check inter 1000 resolvers dns (figure 2) | [email protected] ~ $ dig @127.0.0.1 +short deb.domain.net | 10.9.0.13 When this config starts [using the command in (figure 3)], within seconds I receive errors that the backend host has gone down due to a DNS resolution failure. This continues indefinitely, however I stopped this test after one failure/return cycle for each process (figure 4). (figure 3) | [email protected] ~ $ sudo haproxy -f /etc/haproxy/haproxy.cfg; sudo strace -tp $(pgrep haproxy | head -1) &>~/haproxy-1.strace & sudo strace -tp $(pgrep haproxy | tail -1) &>~/haproxy-2.strace & (figure 4) | Nov 29 01:46:54 elb2.domain.net haproxy[29887]: Proxy back_deb-http started. | Nov 29 01:46:58 elb2.domain.net haproxy[29888]: Server back_deb-http/deb is going DOWN for maintenance (DNS timeout status). 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. | Nov 29 01:46:58 elb2.domain.net haproxy[29888]: proxy back_deb-http has no server available! | Nov 29 01:46:59 elb2.domain.net haproxy[29888]: Server back_deb-http/deb ('deb.domain.net') is UP/READY (resolves again). | Nov 29 01:46:59 elb2.domain.net haproxy[29888]: Server back_deb-http/deb administratively READY thanks to valid DNS answer. | Nov 29 01:47:00 elb2.domain.net haproxy[29889]: Server back_deb-http/deb is going DOWN for maintenance (DNS timeout status). 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. | Nov 29 01:47:00 elb2.domain.net haproxy[29889]: proxy back_deb-http has no server available! | Nov 29 01:47:02 elb2.domain.net haproxy[29889]: Server back_deb-http/deb ('deb.domain.net') is UP/READY (resolves again). | Nov 29 01:47:02 elb2.domain.net haproxy[29889]: Server back_deb-http/deb administratively READY thanks to valid DNS answer. I've attached straces of both processes as shown in the command in (figure 3) as well as a pcap file taken on my loopback interface of the DNS resolution requests to dnsmask. The most curious thing I see in the DNS pcap is requests for AAAA records that don't exist, but I don't know if that's the cause, though this would explain the swapping-A-and-AAAA-records I mentioned earlier. I've noticed this bug both on haproxy 1.6.9 (from the Debian jessie-backports repo) and also on my own self-built 1.7.0 package as well, and all the above testing was on 1.7.0. Please let me know if I can provide any further information! Thanks, Joshua M. Boniface Linux System Ærchitect - Boniface Labs Sigmentation fault: core dumped
haproxy.pcap
Description: application/vnd.tcpdump.pcap
Process 29888 attached
01:46:54 epoll_wait(0, {}, 200, 983) = 0
01:46:55 epoll_wait(0, {{EPOLLIN, {u32=4, u64=4}}}, 200, 0) = 1
01:46:55 recvfrom(4,
"\320)\201\200\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1", 512, 0, NULL,
NULL) = 32
01:46:55 recvfrom(4, 0x7ffe07e72510, 512, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:46:55 sendto(4, "\320)\1\0\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1",
32, 0, NULL, 0) = 32
01:46:55 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 1
01:46:55 fcntl(1, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
01:46:55 setsockopt(1, SOL_TCP, TCP_NODELAY, [1], 4) = 0
01:46:55 connect(1, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("10.9.0.13")}, 16) = -1 EINPROGRESS (Operation now in
progress)
01:46:55 epoll_wait(0, {{EPOLLIN, {u32=4, u64=4}}}, 200, 0) = 1
01:46:55 recvfrom(1, 0x563209ab8204, 16384, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:46:55 getsockopt(1, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
01:46:55 sendto(1, "GET /debian/haproxy HTTP/1.0\r\n\r\n", 32,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = -1 EAGAIN (Resource temporarily
unavailable)
01:46:55 recvfrom(4, 0x7ffe07e72510, 512, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:46:55 epoll_ctl(0, EPOLL_CTL_ADD, 1, {EPOLLOUT, {u32=1, u64=1}}) = 0
01:46:55 epoll_wait(0, {{EPOLLOUT, {u32=1, u64=1}}}, 200, 1000) = 1
01:46:55 getsockopt(1, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
01:46:55 sendto(1, "GET /debian/haproxy HTTP/1.0\r\n\r\n", 32,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 32
01:46:55 epoll_ctl(0, EPOLL_CTL_MOD, 1, {EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}) = 0
01:46:55 epoll_wait(0, {{EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}}, 200, 999) = 1
01:46:55 recvfrom(1, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 16384, 0, NULL,
NULL) = 243
01:46:55 close(1) = 0
01:46:55 epoll_wait(0, {}, 200, 998) = 0
01:46:56 epoll_wait(0, {}, 200, 0) = 0
01:46:56 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 1
01:46:56 fcntl(1, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
01:46:56 setsockopt(1, SOL_TCP, TCP_NODELAY, [1], 4) = 0
01:46:56 connect(1, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("10.9.0.13")}, 16) = -1 EINPROGRESS (Operation now in
progress)
01:46:56 epoll_wait(0, {{EPOLLIN, {u32=4, u64=4}}}, 200, 0) = 1
01:46:56 recvfrom(1, 0x563209ab8204, 16384, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:46:56 getsockopt(1, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
01:46:56 sendto(1, "GET /debian/haproxy HTTP/1.0\r\n\r\n", 32,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 32
01:46:56 recvfrom(4, 0x7ffe07e72510, 512, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:46:56 epoll_ctl(0, EPOLL_CTL_ADD, 1, {EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}) = 0
01:46:56 epoll_wait(0, {{EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}}, 200, 999) = 1
01:46:56 recvfrom(1, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 16384, 0, NULL,
NULL) = 243
01:46:56 close(1) = 0
01:46:56 epoll_wait(0, {}, 200, 998) = 0
01:46:57 epoll_wait(0, {}, 200, 0) = 0
01:46:57 sendto(4, "\223\311\1\0\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1",
32, 0, NULL, 0) = 32
01:46:57 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 1
01:46:57 fcntl(1, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
01:46:57 setsockopt(1, SOL_TCP, TCP_NODELAY, [1], 4) = 0
01:46:57 connect(1, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("10.9.0.13")}, 16) = -1 EINPROGRESS (Operation now in
progress)
01:46:57 epoll_wait(0, {{EPOLLIN, {u32=4, u64=4}}}, 200, 0) = 1
01:46:57 recvfrom(1, 0x563209ab8204, 16384, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:46:57 getsockopt(1, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
01:46:57 sendto(1, "GET /debian/haproxy HTTP/1.0\r\n\r\n", 32,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 32
01:46:57 recvfrom(4,
"n\256\201\200\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1", 512, 0, NULL,
NULL) = 32
01:46:57 recvfrom(4, 0x7ffe07e72510, 512, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:46:57 epoll_ctl(0, EPOLL_CTL_ADD, 1, {EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}) = 0
01:46:57 epoll_wait(0, {{EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}}, 200, 1000) = 1
01:46:57 recvfrom(1, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 16384, 0, NULL,
NULL) = 243
01:46:57 close(1) = 0
01:46:57 epoll_wait(0, {}, 200, 999) = 0
01:46:58 epoll_wait(0, {}, 200, 0) = 0
01:46:58 sendmsg(8, {msg_name(28)={sa_family=AF_INET6, sin6_port=htons(514),
inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0},
msg_iov(8)=[{"<25>Nov 29 01:46:58 elb2.i.bonil"..., 39}, {"haproxy", 7}, {"[",
1}, {"29888", 5}, {"]: ", 3}, {"", 0}, {"Server back_deb-http/deb is goin"...,
165}, {"\n", 1}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) =
221
01:46:58 sendmsg(8, {msg_name(28)={sa_family=AF_INET6, sin6_port=htons(514),
inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0},
msg_iov(8)=[{"<24>Nov 29 01:46:58 elb2.i.bonil"..., 39}, {"haproxy", 7}, {"[",
1}, {"29888", 5}, {"]: ", 3}, {"", 0}, {"proxy back_deb-http has no serve"...,
44}, {"\n", 1}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) =
100
01:46:58 epoll_wait(0, {}, 200, 999) = 0
01:46:59 epoll_wait(0, {}, 200, 0) = 0
01:46:59 sendto(4, "n\256\1\0\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1",
32, 0, NULL, 0) = 32
01:46:59 epoll_wait(0, {{EPOLLIN, {u32=4, u64=4}}}, 200, 994) = 1
01:46:59 recvfrom(4,
"n\256\201\200\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1", 512, 0, NULL,
NULL) = 32
01:46:59 sendto(4, "n\256\1\0\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\1\0\1", 32,
0, NULL, 0) = 32
01:46:59 recvfrom(4,
"\r\363\201\200\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1", 512, 0, NULL,
NULL) = 32
01:46:59 recvfrom(4,
"n\256\205\200\0\1\0\1\0\0\0\0\3deb\6domain\3net\0\0\1\0\1"..., 512, 0, NULL,
NULL) = 48
01:46:59 sendmsg(8, {msg_name(28)={sa_family=AF_INET6, sin6_port=htons(514),
inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0},
msg_iov(8)=[{"<29>Nov 29 01:46:59 elb2.i.bonil"..., 39}, {"haproxy", 7}, {"[",
1}, {"29888", 5}, {"]: ", 3}, {"", 0}, {"Server back_deb-http/deb ('deb.d"...,
73}, {"\n", 1}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) =
129
01:46:59 sendmsg(8, {msg_name(28)={sa_family=AF_INET6, sin6_port=htons(514),
inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0},
msg_iov(8)=[{"<29>Nov 29 01:46:59 elb2.i.bonil"..., 39}, {"haproxy", 7}, {"[",
1}, {"29888", 5}, {"]: ", 3}, {"", 0}, {"Server back_deb-http/deb adminis"...,
75}, {"\n", 1}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) =
131
01:46:59 recvfrom(4, 0x7ffe07e72510, 512, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:46:59 epoll_wait(0, {}, 200, 993) = 0
01:47:00 epoll_wait(0, {}, 200, 0) = 0
01:47:00 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 1
01:47:00 fcntl(1, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
01:47:00 setsockopt(1, SOL_TCP, TCP_NODELAY, [1], 4) = 0
01:47:00 connect(1, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("10.9.0.13")}, 16) = -1 EINPROGRESS (Operation now in
progress)
01:47:00 epoll_wait(0, {}, 200, 0) = 0
01:47:00 recvfrom(1, 0x563209ab8204, 16384, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:47:00 getsockopt(1, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
01:47:00 sendto(1, "GET /debian/haproxy HTTP/1.0\r\n\r\n", 32,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 32
01:47:00 epoll_ctl(0, EPOLL_CTL_ADD, 1, {EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}) = 0
01:47:00 epoll_wait(0, {{EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}}, 200, 3) = 1
01:47:00 recvfrom(1, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 16384, 0, NULL,
NULL) = 243
01:47:00 close(1) = 0
01:47:00 epoll_wait(0, {}, 200, 2) = 0
01:47:00 epoll_wait(0, {}, 200, 997) = 0
01:47:01 epoll_wait(0, {}, 200, 0) = 0
01:47:01 sendto(4, "\r\363\1\0\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1",
32, 0, NULL, 0) = 32
01:47:01 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 1
01:47:01 fcntl(1, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
01:47:01 setsockopt(1, SOL_TCP, TCP_NODELAY, [1], 4) = 0
01:47:01 connect(1, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("10.9.0.13")}, 16) = -1 EINPROGRESS (Operation now in
progress)
01:47:01 epoll_wait(0, {}, 200, 0) = 0
01:47:01 recvfrom(1, 0x563209ab8204, 16384, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:47:01 getsockopt(1, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
01:47:01 sendto(1, "GET /debian/haproxy HTTP/1.0\r\n\r\n", 32,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 32
01:47:01 epoll_ctl(0, EPOLL_CTL_ADD, 1, {EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}) = 0
01:47:01 epoll_wait(0, {{EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}}, 200, 1000) = 1
01:47:01 recvfrom(1, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 16384, 0, NULL,
NULL) = 243
01:47:01 close(1) = 0
01:47:01 epoll_wait(0, {}, 200, 999) = 0
01:47:02 epoll_wait(0, {}, 200, 0) = 0
01:47:02 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 1
01:47:02 fcntl(1, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
01:47:02 setsockopt(1, SOL_TCP, TCP_NODELAY, [1], 4) = 0
01:47:02 connect(1, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("10.9.0.13")}, 16) = -1 EINPROGRESS (Operation now in
progress)
01:47:02 epoll_wait(0, {}, 200, 0) = 0
01:47:02 recvfrom(1, 0x563209ab8204, 16384, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:47:02 getsockopt(1, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
01:47:02 sendto(1, "GET /debian/haproxy HTTP/1.0\r\n\r\n", 32,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 32
01:47:02 epoll_ctl(0, EPOLL_CTL_ADD, 1, {EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}) = 0
01:47:02 epoll_wait(0, {{EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}}, 200, 999) = 1
01:47:02 recvfrom(1, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 16384, 0, NULL,
NULL) = 243
01:47:02 close(1) = 0
01:47:02 epoll_wait(0, {{EPOLLIN, {u32=4, u64=4}}}, 200, 998) = 1
01:47:03 recvfrom(4,
">\"\201\200\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1", 512, 0, NULL, NULL)
= 32
01:47:03 recvfrom(4, 0x7ffe07e72510, 512, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:47:03 epoll_wait(0, {}, 200, 0) = 0
01:47:03 sendto(4, "\205(\1\0\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1",
32, 0, NULL, 0) = 32
01:47:03 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 1
01:47:03 fcntl(1, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
01:47:03 setsockopt(1, SOL_TCP, TCP_NODELAY, [1], 4) = 0
01:47:03 connect(1, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("10.9.0.13")}, 16) = -1 EINPROGRESS (Operation now in
progress)
01:47:03 epoll_wait(0, {}, 200, 0) = 0
01:47:03 recvfrom(1, 0x563209ab8204, 16384, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:47:03 getsockopt(1, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
01:47:03 sendto(1, "GET /debian/haproxy HTTP/1.0\r\n\r\n", 32,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 32
01:47:03 epoll_ctl(0, EPOLL_CTL_ADD, 1, {EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}) = 0
01:47:03 epoll_wait(0, {{EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}}, 200, 1000) = 1
01:47:03 recvfrom(1, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 16384, 0, NULL,
NULL) = 243
01:47:03 close(1) = 0
01:47:03 epoll_wait(0, 563209afb670, 200, 999) = -1 EINTR (Interrupted system
call)
01:47:04 --- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=29952,
si_uid=0} ---
01:47:04 +++ killed by SIGTERM +++
Process 29889 attached
01:46:54 epoll_wait(0, {}, 200, 982) = 0
01:46:55 epoll_wait(0, {}, 200, 0) = 0
01:46:55 sendto(4, "\320)\1\0\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1",
32, 0, NULL, 0) = 32
01:46:55 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 1
01:46:55 fcntl(1, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
01:46:55 setsockopt(1, SOL_TCP, TCP_NODELAY, [1], 4) = 0
01:46:55 connect(1, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("10.9.0.13")}, 16) = -1 EINPROGRESS (Operation now in
progress)
01:46:55 epoll_wait(0, {}, 200, 0) = 0
01:46:55 recvfrom(1, 0x563209ab8204, 16384, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:46:55 getsockopt(1, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
01:46:55 sendto(1, "GET /debian/haproxy HTTP/1.0\r\n\r\n", 32,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 32
01:46:55 epoll_ctl(0, EPOLL_CTL_ADD, 1, {EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}) = 0
01:46:55 epoll_wait(0, {{EPOLLIN, {u32=4, u64=4}}}, 200, 1000) = 1
01:46:55 recvfrom(4,
"\320)\201\200\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1", 512, 0, NULL,
NULL) = 32
01:46:55 sendto(4, "\320)\1\0\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\1\0\1", 32,
0, NULL, 0) = 32
01:46:55 recvfrom(4,
"\320)\205\200\0\1\0\1\0\0\0\0\3deb\6domain\3net\0\0\1\0\1"..., 512, 0, NULL,
NULL) = 48
01:46:55 recvfrom(4, 0x7ffe07e72510, 512, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:46:55 epoll_wait(0, {{EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}}, 200, 999) = 1
01:46:55 recvfrom(1, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 16384, 0, NULL,
NULL) = 243
01:46:55 close(1) = 0
01:46:55 epoll_wait(0, {}, 200, 998) = 0
01:46:56 epoll_wait(0, {}, 200, 0) = 0
01:46:56 sendto(4, "\223\311\1\0\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1",
32, 0, NULL, 0) = 32
01:46:56 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 1
01:46:56 fcntl(1, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
01:46:56 setsockopt(1, SOL_TCP, TCP_NODELAY, [1], 4) = 0
01:46:56 connect(1, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("10.9.0.13")}, 16) = -1 EINPROGRESS (Operation now in
progress)
01:46:56 epoll_wait(0, {{EPOLLIN, {u32=4, u64=4}}}, 200, 0) = 1
01:46:56 recvfrom(1, 0x563209ab8204, 16384, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:46:56 getsockopt(1, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
01:46:56 sendto(1, "GET /debian/haproxy HTTP/1.0\r\n\r\n", 32,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 32
01:46:56 recvfrom(4,
"\223\311\201\200\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1", 512, 0, NULL,
NULL) = 32
01:46:56 sendto(4, "\223\311\1\0\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\1\0\1",
32, 0, NULL, 0) = 32
01:46:56 recvfrom(4,
"\223\311\205\200\0\1\0\1\0\0\0\0\3deb\6domain\3net\0\0\1\0\1"..., 512, 0,
NULL, NULL) = 48
01:46:56 recvfrom(4, 0x7ffe07e72510, 512, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:46:56 epoll_ctl(0, EPOLL_CTL_ADD, 1, {EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}) = 0
01:46:56 epoll_wait(0, {{EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}}, 200, 1000) = 1
01:46:56 recvfrom(1, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 16384, 0, NULL,
NULL) = 243
01:46:56 close(1) = 0
01:46:56 epoll_wait(0, {}, 200, 998) = 0
01:46:57 epoll_wait(0, {{EPOLLIN, {u32=4, u64=4}}}, 200, 2) = 1
01:46:57 recvfrom(4,
"\223\311\201\200\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1", 512, 0, NULL,
NULL) = 32
01:46:57 recvfrom(4, 0x7ffe07e72510, 512, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:46:57 epoll_wait(0, {}, 200, 0) = 0
01:46:57 sendto(4, "n\256\1\0\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1",
32, 0, NULL, 0) = 32
01:46:57 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 1
01:46:57 fcntl(1, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
01:46:57 setsockopt(1, SOL_TCP, TCP_NODELAY, [1], 4) = 0
01:46:57 connect(1, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("10.9.0.13")}, 16) = -1 EINPROGRESS (Operation now in
progress)
01:46:57 epoll_wait(0, {}, 200, 0) = 0
01:46:57 recvfrom(1, 0x563209ab8204, 16384, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:46:57 getsockopt(1, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
01:46:57 sendto(1, "GET /debian/haproxy HTTP/1.0\r\n\r\n", 32,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 32
01:46:57 epoll_ctl(0, EPOLL_CTL_ADD, 1, {EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}) = 0
01:46:57 epoll_wait(0, {{EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}}, 200, 1000) = 1
01:46:57 recvfrom(1, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 16384, 0, NULL,
NULL) = 243
01:46:57 close(1) = 0
01:46:57 epoll_wait(0, {}, 200, 999) = 0
01:46:58 epoll_wait(0, {}, 200, 0) = 0
01:46:58 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 1
01:46:58 fcntl(1, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
01:46:58 setsockopt(1, SOL_TCP, TCP_NODELAY, [1], 4) = 0
01:46:58 connect(1, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("10.9.0.13")}, 16) = -1 EINPROGRESS (Operation now in
progress)
01:46:58 epoll_wait(0, {}, 200, 0) = 0
01:46:58 recvfrom(1, 0x563209ab8204, 16384, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:46:58 getsockopt(1, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
01:46:58 sendto(1, "GET /debian/haproxy HTTP/1.0\r\n\r\n", 32,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 32
01:46:58 epoll_ctl(0, EPOLL_CTL_ADD, 1, {EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}) = 0
01:46:58 epoll_wait(0, {{EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}}, 200, 1000) = 1
01:46:58 recvfrom(1, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 16384, 0, NULL,
NULL) = 243
01:46:58 close(1) = 0
01:46:58 epoll_wait(0, {}, 200, 999) = 0
01:46:59 epoll_wait(0, {}, 200, 0) = 0
01:46:59 sendto(4, "\r\363\1\0\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1",
32, 0, NULL, 0) = 32
01:46:59 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 1
01:46:59 fcntl(1, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
01:46:59 setsockopt(1, SOL_TCP, TCP_NODELAY, [1], 4) = 0
01:46:59 connect(1, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("10.9.0.13")}, 16) = -1 EINPROGRESS (Operation now in
progress)
01:46:59 epoll_wait(0, {}, 200, 0) = 0
01:46:59 recvfrom(1, 0x563209ab8204, 16384, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:46:59 getsockopt(1, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
01:46:59 sendto(1, "GET /debian/haproxy HTTP/1.0\r\n\r\n", 32,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 32
01:46:59 epoll_ctl(0, EPOLL_CTL_ADD, 1, {EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}) = 0
01:46:59 epoll_wait(0, {{EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}}, 200, 1000) = 1
01:46:59 recvfrom(1, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 16384, 0, NULL,
NULL) = 243
01:46:59 close(1) = 0
01:46:59 epoll_wait(0, {}, 200, 999) = 0
01:47:00 epoll_wait(0, {}, 200, 0) = 0
01:47:00 sendmsg(8, {msg_name(28)={sa_family=AF_INET6, sin6_port=htons(514),
inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0},
msg_iov(8)=[{"<25>Nov 29 01:47:00 elb2.i.bonil"..., 39}, {"haproxy", 7}, {"[",
1}, {"29889", 5}, {"]: ", 3}, {"", 0}, {"Server back_deb-http/deb is goin"...,
165}, {"\n", 1}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) =
221
01:47:00 sendmsg(8, {msg_name(28)={sa_family=AF_INET6, sin6_port=htons(514),
inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0},
msg_iov(8)=[{"<24>Nov 29 01:47:00 elb2.i.bonil"..., 39}, {"haproxy", 7}, {"[",
1}, {"29889", 5}, {"]: ", 3}, {"", 0}, {"proxy back_deb-http has no serve"...,
44}, {"\n", 1}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) =
100
01:47:00 epoll_wait(0, {{EPOLLIN, {u32=4, u64=4}}}, 200, 999) = 1
01:47:01 recvfrom(4,
"\r\363\201\200\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1", 512, 0, NULL,
NULL) = 32
01:47:01 recvfrom(4, 0x7ffe07e72510, 512, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:47:01 epoll_wait(0, {}, 200, 0) = 0
01:47:01 epoll_wait(0, {}, 200, 1000) = 0
01:47:02 epoll_wait(0, {}, 200, 0) = 0
01:47:02 sendto(4, "\205(\1\0\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1",
32, 0, NULL, 0) = 32
01:47:02 epoll_wait(0, {{EPOLLIN, {u32=4, u64=4}}}, 200, 999) = 1
01:47:02 recvfrom(4,
"\205(\201\200\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1", 512, 0, NULL,
NULL) = 32
01:47:02 sendto(4, "\205(\1\0\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\1\0\1", 32,
0, NULL, 0) = 32
01:47:02 recvfrom(4,
"\205(\205\200\0\1\0\1\0\0\0\0\3deb\6domain\3net\0\0\1\0\1"..., 512, 0, NULL,
NULL) = 48
01:47:02 sendmsg(8, {msg_name(28)={sa_family=AF_INET6, sin6_port=htons(514),
inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0},
msg_iov(8)=[{"<29>Nov 29 01:47:02 elb2.i.bonil"..., 39}, {"haproxy", 7}, {"[",
1}, {"29889", 5}, {"]: ", 3}, {"", 0}, {"Server back_deb-http/deb ('deb.d"...,
73}, {"\n", 1}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) =
129
01:47:02 sendmsg(8, {msg_name(28)={sa_family=AF_INET6, sin6_port=htons(514),
inet_pton(AF_INET6, "::1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0},
msg_iov(8)=[{"<29>Nov 29 01:47:02 elb2.i.bonil"..., 39}, {"haproxy", 7}, {"[",
1}, {"29889", 5}, {"]: ", 3}, {"", 0}, {"Server back_deb-http/deb adminis"...,
75}, {"\n", 1}], msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) =
131
01:47:02 recvfrom(4, 0x7ffe07e72510, 512, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:47:02 epoll_wait(0, {}, 200, 998) = 0
01:47:03 epoll_wait(0, {}, 200, 0) = 0
01:47:03 sendto(4, ">\"\1\0\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1", 32,
0, NULL, 0) = 32
01:47:03 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 1
01:47:03 fcntl(1, F_SETFL, O_RDONLY|O_NONBLOCK) = 0
01:47:03 setsockopt(1, SOL_TCP, TCP_NODELAY, [1], 4) = 0
01:47:03 connect(1, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr("10.9.0.13")}, 16) = -1 EINPROGRESS (Operation now in
progress)
01:47:03 epoll_wait(0, {}, 200, 0) = 0
01:47:03 recvfrom(1, 0x563209ab8204, 16384, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:47:03 getsockopt(1, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
01:47:03 sendto(1, "GET /debian/haproxy HTTP/1.0\r\n\r\n", 32,
MSG_DONTWAIT|MSG_NOSIGNAL, NULL, 0) = 32
01:47:03 epoll_ctl(0, EPOLL_CTL_ADD, 1, {EPOLLIN|EPOLLRDHUP, {u32=1, u64=1}}) = 0
01:47:03 epoll_wait(0, {{EPOLLIN, {u32=4, u64=4}}, {EPOLLIN|EPOLLRDHUP, {u32=1,
u64=1}}}, 200, 1000) = 2
01:47:03 recvfrom(4,
"\205(\201\200\0\1\0\0\0\0\0\0\3deb\6domain\3net\0\0\34\0\1", 512, 0, NULL,
NULL) = 32
01:47:03 recvfrom(4, 0x7ffe07e72510, 512, 0, 0, 0) = -1 EAGAIN (Resource
temporarily unavailable)
01:47:03 recvfrom(1, "HTTP/1.1 200 OK\r\nServer: nginx/1"..., 16384, 0, NULL,
NULL) = 243
01:47:03 close(1) = 0
01:47:03 epoll_wait(0, 563209afb670, 200, 999) = -1 EINTR (Interrupted system
call)
01:47:04 --- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=29952,
si_uid=0} ---
01:47:04 +++ killed by SIGTERM +++
