Re: healthchecks (to uwsgi) possible regression 1.9.8 -> 1.9.9
Hi Jarno, On Wed, Oct 02, 2019 at 01:08:14PM +0300, Jarno Huuskonen wrote: > Hello, > > I was testing haproxy -> uwsgi(alert.io) and noticed a possible regression > with healthchecks(httpchk). > With 1.9.9 uwsgi logs: > [uwsgi-http key: host.name.fi client_addr: 127.0.0.1 client_port: 45715] > hr_read(): Connection reset by peer [plugins/http/http.c line 917] > > health checks work > (option httpchk GET /_ HTTP/1.1\r\nHost:\ host.name.fi\r\nUser-Agent:\ > haproxy) > but uwsgi logs the hr_read() warning/error. > > I bisected 1.9.9 and this commit is probably the commit that changes > behaviour between 1.9.8 and 1.9.9: > 5d0cb90eb78f869e8801b34eddfdfd5dd8360e71 is the first bad commit > commit 5d0cb90eb78f869e8801b34eddfdfd5dd8360e71 > Author: Olivier Houchard > Date: Fri Jun 14 15:26:06 2019 +0200 > > BUG/MEDIUM: connections: Don't call shutdown() if we want to disable > linger. > > In conn_sock_shutw(), avoid calling shutdown() if linger_risk is set. Not > doing so will result in getting sockets in TIME_WAIT for some time. > This is particularly observable with health checks. > > This should be backported to 1.9. > > (cherry picked from commit fe4abe62c7c5206dff1802f42d17014e198b9141) > Signed-off-by: Christopher Faulet Hmmm that's annoying, really, because we've opened a huge can of worms when fixing the first failed check and we're constantly displacing the problem somewhere else :-/ Yes, please do provide an strace, and a tcpdump, that would be nice. I suspect that we'll possibly see a FIN from the server, without the equivalent recv()==0 in haproxy, and that the setsockopt() call resulting in the RST being sent doesn't ack the FIN. Normally we should perform a clean shutdown+close if the FIN was received with the response and detected in time. I'm seeing that your check request doesn't contain "connection: close", so actually it's possible that your server doesn't send the SYN, in which case we really need to close with RST. Could you please try to add "connection: close" to your httpchk line ? Regards, Willy
Re: Handling Idle clients without specifying timeouts
Hi, On Thu, Oct 03, 2019 at 06:40:08PM -0700, Shishir Kumar Yadav wrote: > Hi All, > > We are using Haproxy 1.8.3 and have a peculiar situation. We want to remove > idle client connections but without specifying HAProxy timeout, basically > we want to rely on backend to detect whether client is idle or not because > sometimes backend might not be able to send any data on a connection for > long time for some reasons and we do not want to close connection with > client because of this. What you're describing is exactly the default behaviour, so you have nothing to do to have this. > Currently backend is able to detect idle client and > kill the connection between HAproxy and backend, but looks like HAProxy is > not closing the connection with client. The connection b/w backend and > HAProxy goes in FIN_WAIT1 state on the backend side but it is still showing > ESTABLISHED on HAProxy side. If you have FIN_WAIT1 on one side, you must have CLOSE_WAIT on the other size, and the FIN_WAIT1 must immediately become FIN_WAIT2 once the other one acknowledges it. If it's not the case, it means that the FIN sent by the server never reached haproxy. This explains why this last one remained ESTABLISHED and why the other one doesn't switch to FIN_WAIT2. I suspect that you're having a firewall, NAT or L4 load balancer between haproxy and your server, with too short timeouts, and that the connection silently expired in the middle, which is why the FIN didn't pass and why this connection cannot be reused. Note that iptables+conntrack running on either of these are also firewalls, so if they are not properly configured (e.g. too short timeout for established connections), you will face such problems. Another possibility could be that you're in fact running in VMs and that it's the hypervisor which runs conntrack with very short timeouts. We've seen this a little bit over the last few years, report by users hosted by incompetent VM providers, who had to switch to a correct one. > Also the connection b/w client and HAProxy > remains in ESTABLISHED state. This situation does not change unless client > is killed. I have also tried 'option nolinger' and it did not help. > > tcp 5462444 0 127.0.0.1:40567 127.0.0.1:8081 > ESTABLISHED 6768/haproxy (Connection b/w Haproxy and backend - Haproxy > side) > tcp6 0 216432 10.60.106.229:8010.62.185.40:52924 > ESTABLISHED 6768/haproxy (Connection b/w client and HAProxy) > tcp6 0 4109430 127.0.0.1:8081 127.0.0.1:40567 > FIN_WAIT1 - (Connection b/w Haproxy and backend - backend side > ) Ah that's becoming amusing, it's on the loopback that you're losing packets! Thus I'm pretty sure you're having iptables and incorrectly configured timeouts. Have a look inside /proc/sys/net/netfilter/ and you'll probably find nf_conntrack_timeout_established with a low value in it. Willy
Handling Idle clients without specifying timeouts
Hi All, We are using Haproxy 1.8.3 and have a peculiar situation. We want to remove idle client connections but without specifying HAProxy timeout, basically we want to rely on backend to detect whether client is idle or not because sometimes backend might not be able to send any data on a connection for long time for some reasons and we do not want to close connection with client because of this. Currently backend is able to detect idle client and kill the connection between HAproxy and backend, but looks like HAProxy is not closing the connection with client. The connection b/w backend and HAProxy goes in FIN_WAIT1 state on the backend side but it is still showing ESTABLISHED on HAProxy side. Also the connection b/w client and HAProxy remains in ESTABLISHED state. This situation does not change unless client is killed. I have also tried 'option nolinger' and it did not help. tcp 5462444 0 127.0.0.1:40567 127.0.0.1:8081 ESTABLISHED 6768/haproxy (Connection b/w Haproxy and backend - Haproxy side) tcp6 0 216432 10.60.106.229:8010.62.185.40:52924 ESTABLISHED 6768/haproxy (Connection b/w client and HAProxy) tcp6 0 4109430 127.0.0.1:8081 127.0.0.1:40567 FIN_WAIT1 - (Connection b/w Haproxy and backend - backend side ) Below is haproxy config - = global # paths setup log 127.0.0.1 local0 log 127.0.0.1 local1 notice chroot /var/lib/haproxy # stats setup # Can use this socket to check the stats of the first process. stats socket /run/haproxy/admin.sock mode 660 level admin stats bind-process 1 stats timeout 30s ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS ssl-default-bind-options ssl-min-ver TLSv1.0 ssl-dh-param-file /etc/ssl/rfc3526_group14_dhparam.pem # mode setup # master-worker mode, introduced in 1.8, is the only mode suitable to our use case. # For more details, see https://www.haproxy.com/blog/whats-new-haproxy-1-8/. master-worker # muticore setup # This template value will be filled by haproxy.sh in run time on each blade. nbproc 4 # DH key setup # Sets the maximum size of the Diffie-Hellman parameters used for generating # the ephemeral/temporary Diffie-Hellman key in case of DHE key exchange. tune.ssl.default-dh-param 2048 defaults modehttp log global option httplog# Enable logging of HTTP request, session state and timers option dontlognull# Disable logging of null connections option dontlog-normal # only log HTTP requests that error'ed out frontend haproxy-frontend bind :::80 v4v6 bind :::443 v4v6 ssl crt /ssd/net/ca/gui.pem default_backend ir-http-server-backend backend ir-http-server-backend http-request add-header __pure-internal-http-tls-version "%[ssl_fc_protocol]" server server1 127.0.0.1:8081 option nolinger = Please let me know if you have any idea. Thanks, Shishir
Re: Use haproxy behind Squid
Hi Nikita. Am 03.10.19 um 12:02 schrieb Akhnin Nikita: > Hello, Aleksandar! > > Vice versa, actually: Client -> Haproxy -> Squid -> Internet > > Here's the situation. Haproxy instance stands in a private network and > interacts with the Internet through Firewall that performs NAT. Current > schema looks like this: > Client -> Haproxy -> FW (SNAT) -> Internet > > The firewall performs traffic filtering in addition to NAT (security > reasons), and in its policies it operates by destination hosts IP-addresses, > not domain names. And the problem comes when backend server hostname changes > its IP-addresses (e.g. CDN). We must update Firewall configuration with new > IP-addresses, and there is service downtime before firewall guys will do it. > And we cannot just open network access from Haproxy to any host in the > Internet. > > I'm looking for workaround for this. We have a Squid that can proxy HTTP > requests to the Internet bypassing the Firewall. Also it filters requests by > domain name. So I wonder if there is any way to proxy client requests to the > Internet through Squid transparently to client (no configuration on client > side). > Something like this, but with Haproxy instead of Httpd: > https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxyremote I don't see any reason to use haproxy in this setup. Of course you can make a listen like the snipplet below but why do you want to add haproxy into this setup? ``` global ... defaults mode tcp ... listen squid-gw bind ::3124 server squid squid.local:3124 check ``` Isn't this a much easier setup? Client -> Squid -> Internet For client configs can you take a look into this page, there are several possible solution described. https://wiki.squid-cache.org/SquidFaq/ConfiguringBrowsers For client's ip address can you setup PROXY Protocol in squid and haproxy http://www.squid-cache.org/Doc/config/proxy_protocol_access/ http://cbonte.github.io/haproxy-dconv/2.0/configuration.html#5.2-send-proxy Hth Aleks > -Original Message- > From: Aleksandar Lazic > Sent: Wednesday, October 2, 2019 6:24 PM > To: Ахнин Никита Андреевич ; haproxy@formilux.org > Subject: Re: Use haproxy behind Squid > > Am 02.10.19 um 13:10 schrieb Akhnin Nikita: >> Hey there! >> >> Is it possible to use Haproxy behind HTTP proxy like Squid to proxy >> incoming requests to the Internet through it? It will be awesome if >> someone will share the configuration example. > > Do you mean such a flow? > > Internet -> squid -> haproxy -> Client > > This statement confuses me a little bit. > >> to proxy incoming requests to the Internet > > From which point of view is incomming and outgoing? > > Regards > Aleks >
RE: Use haproxy behind Squid
Hello, Aleksandar! Vice versa, actually: Client -> Haproxy -> Squid -> Internet Here's the situation. Haproxy instance stands in a private network and interacts with the Internet through Firewall that performs NAT. Current schema looks like this: Client -> Haproxy -> FW (SNAT) -> Internet The firewall performs traffic filtering in addition to NAT (security reasons), and in its policies it operates by destination hosts IP-addresses, not domain names. And the problem comes when backend server hostname changes its IP-addresses (e.g. CDN). We must update Firewall configuration with new IP-addresses, and there is service downtime before firewall guys will do it. And we cannot just open network access from Haproxy to any host in the Internet. I'm looking for workaround for this. We have a Squid that can proxy HTTP requests to the Internet bypassing the Firewall. Also it filters requests by domain name. So I wonder if there is any way to proxy client requests to the Internet through Squid transparently to client (no configuration on client side). Something like this, but with Haproxy instead of Httpd: https://httpd.apache.org/docs/2.4/mod/mod_proxy.html#proxyremote -Original Message- From: Aleksandar Lazic Sent: Wednesday, October 2, 2019 6:24 PM To: Ахнин Никита Андреевич ; haproxy@formilux.org Subject: Re: Use haproxy behind Squid Am 02.10.19 um 13:10 schrieb Akhnin Nikita: > Hey there! > > Is it possible to use Haproxy behind HTTP proxy like Squid to proxy > incoming requests to the Internet through it? It will be awesome if > someone will share the configuration example. Do you mean such a flow? Internet -> squid -> haproxy -> Client This statement confuses me a little bit. > to proxy incoming requests to the Internet From which point of view is incomming and outgoing? Regards Aleks
