Re: haproxy stops responding to maskable signals

[Vincent Bernat]

 ❦  1 mai 2016 22:31 +0300, Alexander Piavlo  :

> Then trying to gracefully reload haproxy often it fails to reload
> saying that it cannot bind to sockets. From what i see the old process
> does not close the listening sockets, and thus replacement process
> fails. Trying to send old haproxy process SIGTTOU and actualy any
> other maskable signal does not have any affect. The old process keeps
> on serving data correctly the only thing that to reload configuration
> SIGKLILL is required which is a show stoppper for us.
>
> This happens both with haproxy 1.5 and 1.6 (so any of the new 1.6
> features we use seem unrelated to the cause). We tried all 1.6
> versions including latest 1.6.4
>
> During then we stripped the configuration to minimum (mainly removed
> all acls) it seemed stable (thought we did not test long enough to be
> 100% sure it helped)
>
> Any ideas what could be the reason for this?

If you happen to trigger reload of HAProxy processes from a Go program,
this may be a bug in Go:
 https://github.com/haproxy/haproxy/issues/48#issuecomment-211769303
-- 
Suspicion always haunts the guilty mind.
-- Wm. Shakespeare

haproxy stops responding to maskable signals

[Alexander Piavlo]

Then trying to gracefully reload haproxy often it fails to reload saying
that it cannot bind to sockets. From what i see the old process does not
close the listening sockets, and thus replacement process fails. Trying to
send old haproxy process SIGTTOU and actualy any other maskable signal does
not have any affect. The old process keeps on serving data correctly the
only thing that to reload configuration SIGKLILL is required which is a
show stoppper for us.

This happens both with haproxy 1.5  and 1.6 (so any of the new 1.6 features
we use seem unrelated to the cause). We tried all 1.6 versions including
latest 1.6.4

During then we stripped the configuration to minimum (mainly removed all
acls) it seemed stable (thought we did not test long enough to be 100% sure
it helped)

Any ideas what could be the reason for this?

tnx
Alex

Re: Erroneous error code on wrong configuration.

[Lukas Tribus]

Hi,


Am 01.05.2016 um 18:18 schrieb Jonathan Matthews:

> frontend my_frontend

> bind :80
> acl global hdr(host) -i blablabla
> use_backend my_backend if global
> backend my_backend
> server google www.google.com:80 

Given that you don't alter the Host header before submitting the 
request to Google, I'm not sure what you're expecting to happen.




This has been previously troubleshooted on discourse and I advised to 
post this to the mailing list.


This is not about when the Host header is actually blablabla. Its about 
when its not:


The ACL doesn't match, and no backend is selected, no server can be 
connected to. That's why we see NOSRV in the log. However, we also see 
the SC flag, which implies that there was a network problem while 
connecting to a backend server. Which is not true in this case, because 
we never even opened a socket to the backend.





Why do you think it's misleading?


We don't open a backend socket, so we shouldn't imply (by the SC flag) 
that there is a network problem on the backend.


Here's how this looks trough strace (you can see the only involved 
sockets are the one on the frontend and the syslog socket):


18:32:55.530183 epoll_wait(3, {{EPOLLIN, {u32=4, u64=4}}}, 200, 1000) = 1
18:32:55.874843 gettimeofday({1462120375, 874872}, NULL) = 0
18:32:55.874934 accept4(4, {sa_family=AF_INET, sin_port=htons(34934), 
sin_addr=inet_addr("10.0.0.55")}, [16], SOCK_NONBLOCK) = 6

18:32:55.875060 setsockopt(6, SOL_TCP, TCP_NODELAY, [1], 4) = 0
18:32:55.875222 accept4(4, 0xbf81501c, [128], SOCK_NONBLOCK) = -1 EAGAIN 
(Resource temporarily unavailable)
18:32:55.875356 recv(6, 0x926e318, 15360, 0) = -1 EAGAIN (Resource 
temporarily unavailable)
18:32:55.875392 epoll_ctl(3, EPOLL_CTL_ADD, 6, {EPOLLIN|EPOLLRDHUP, 
{u32=6, u64=6}}) = 0

18:32:55.875392 gettimeofday({1462120375, 875392}, NULL) = 0
18:32:55.875392 epoll_wait(3, {{EPOLLIN, {u32=6, u64=6}}}, 200, 1000) = 1
18:32:55.875898 gettimeofday({1462120375, 875920}, NULL) = 0
18:32:55.875972 recv(6, "GET / HTTP/1.1\r\nUser-Agent: curl"..., 15360, 
0) = 73

18:32:55.876758 epoll_ctl(3, EPOLL_CTL_DEL, 6, {0, {u32=6, u64=6}}) = 0
18:32:55.877220 gettimeofday({1462120375, 877361}, NULL) = 0
18:32:55.877643 epoll_wait(3, {}, 200, 0) = 0
18:32:55.878121 gettimeofday({1462120375, 878261}, NULL) = 0
18:32:55.878507 send(6, "HTTP/1.0 503 Service Unavailable"..., 212, 
MSG_DONTWAIT|MSG_NOSIGNAL|MSG_MORE) = 212

18:32:55.879024 shutdown(6, SHUT_WR)= 0
18:32:55.879460 close(6)= 0
18:32:55.879868 sendmsg(5, {msg_name(16)={sa_family=AF_INET, 
sin_port=htons(514), sin_addr=inet_addr("10.0.0.3")}, 
msg_iov(8)=[{"<46>May  1 18:32:55 ", 20}, {"haproxy", 7}, {"[", 1}, 
{"9363", 4}, {"]: ", 3}, {"", 0}, {"10.0.0.55:34934 
[01/May/2016:18:"..., 136}, {"\n", 1}], msg_controllen=0, msg_flags=0}, 
MSG_DONTWAIT|MSG_NOSIGNAL) = 172





Regards,

Lukas

Re: Erroneous error code on wrong configuration.

[Jonathan Matthews]

On 29 Apr 2016 11:29, "Mayank Jha"  wrote:
>
> I am facing the following in haproxy 1.5. I get the following error, with
error code "SC" which is very misleading, for the below mentioned config.

Why do you think it's misleading?

> haproxy[6379]: 127.0.0.1:53010 [29/Apr/2016:12:05:40.552] my_frontend
my_frontend/ -1/-1/-1/-1/1 503 212 - - SC-- 0/0/0/0/0 0/0 "GET /
HTTP/1.1"
>
> With the following config.
>
> frontend my_frontend
> bind :80
> acl global hdr(host) -i blablabla
> use_backend my_backend if global
> backend my_backend
> server google www.google.com:80

Given that you don't alter the Host header before submitting the request to
Google, I'm not sure what you're expecting to happen.

I think there's a fair bit of extra information you'll need to provide
before I (at least; not speaking for anyone else!) understand what your
problem actually *is*. You're assuming we know more than we do about your
setup, aims, and expected outcomes :-)

J