Hi, Today I noticed after a reload that previous process was alive for long time( >8hours). This is a HAProxy which runs in HTTP mode in front of few squid servers, conf is quite simple[1] and the version is 1.5.6[2]
I had a lsof watcher for the old pid and the number of connections were very slowly dropping from 2K to 200 right now. For few of the connections that were in established state( for the old process) I run tcpdump and saw no activity at all, I have attached a network trace from one those and you can see that client sends periodically every 10min 5bytes. The HAProxy is used by normal browsers but also from cronjobs with various languages(Perl,Python,C,Go etc) I was surprised about this very long inactivity period for TCP connection on a system which has reasonable settings for TCP keepalive[3]. But setting 'timeout tunnel' is not set, and since this HAProxy is serving proxy traffic to squid all client/server connections are treated as tunnels. am I right? My question is about TCP keepalive and tunnel. Are system keepalive settings ignored when HAProxy treads client/servers connections as tunnels? Cheers, Pavlos [1] global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 65536 tune.bufsize 65536 user haproxy group haproxy daemon stats socket /var/lib/haproxy/stats uid 0 gid 0 mode 0440 level admin defaults mode http log global option httplog clf option dontlognull option forwardfor except 10.0.0.0/8 option redispatch option http-server-close option http-use-proxy-header option tcp-smart-accept option tcp-smart-connect no option checkcache retries 3 maxconn 65536 timeout queue 1m timeout connect 4s timeout client 30m timeout server 30m timeout check 10s timeout http-request 10s timeout http-keep-alive 10s errorfile 408 /dev/null listen haproxy :8080 mode http stats enable stats uri / stats show-node stats refresh 10s frontend http_in *:3128 default_backend squid_http backend squid_http balance leastconn server squid-01 squid-01:3128 server squid-02 squid-02:3128 server squid-03 squid-03:3128 server squid-04 squid-04:3128 [2] haproxy -vv HA-Proxy version 1.5.6 2014/10/18 Copyright 2000-2014 Willy Tarreau <w...@1wt.eu> Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.3 Compression algorithms supported : identity, deflate, gzip Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 7.8 2008-09-05 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll. pparissis at corplbout-201 in ~ [3] sudo sysctl -a|grep keepalive net.ipv4.tcp_keepalive_time = 30 net.ipv4.tcp_keepalive_probes = 2 net.ipv4.tcp_keepalive_intvl = 1
tcpdump host 10.155.96.64 and port 64473 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes 18:56:52.528548 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [P.], seq 2900223928:2900223933, ack 3414209439, win 8210, options [nop,nop,TS val 1306751675 ecr 1014245525], length 5 18:56:52.548765 IP haproxyserver.squid > 10.155.96.64.64473: Flags [P.], seq 1:6, ack 5, win 31, options [nop,nop,TS val 1014785499 ecr 1306751675], length 5 18:56:52.575487 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [.], ack 6, win 8209, options [nop,nop,TS val 1306751720 ecr 1014785499], length 0 19:06:00.052136 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [P.], seq 5:10, ack 6, win 8209, options [nop,nop,TS val 1307290366 ecr 1014785499], length 5 19:06:00.069963 IP haproxyserver.squid > 10.155.96.64.64473: Flags [P.], seq 6:11, ack 10, win 31, options [nop,nop,TS val 1015333020 ecr 1307290366], length 5 19:06:00.094985 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [.], ack 11, win 8209, options [nop,nop,TS val 1307290408 ecr 1015333020], length 0 19:15:01.559202 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [P.], seq 10:15, ack 11, win 8209, options [nop,nop,TS val 1307822802 ecr 1015333020], length 5 19:15:01.577110 IP haproxyserver.squid > 10.155.96.64.64473: Flags [P.], seq 11:16, ack 15, win 31, options [nop,nop,TS val 1015874527 ecr 1307822802], length 5 19:15:01.602712 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [.], ack 16, win 8209, options [nop,nop,TS val 1307822844 ecr 1015874527], length 0 19:23:59.161132 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [P.], seq 15:20, ack 16, win 8209, options [nop,nop,TS val 1308351633 ecr 1015874527], length 5 19:23:59.179017 IP haproxyserver.squid > 10.155.96.64.64473: Flags [P.], seq 16:21, ack 20, win 31, options [nop,nop,TS val 1016412129 ecr 1308351633], length 5 19:23:59.204112 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [.], ack 21, win 8208, options [nop,nop,TS val 1308351673 ecr 1016412129], length 0 19:32:58.776691 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [P.], seq 20:25, ack 21, win 8208, options [nop,nop,TS val 1308882229 ecr 1016412129], length 5 19:32:58.794670 IP haproxyserver.squid > 10.155.96.64.64473: Flags [P.], seq 21:26, ack 25, win 31, options [nop,nop,TS val 1016951745 ecr 1308882229], length 5 19:32:58.819939 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [.], ack 26, win 8208, options [nop,nop,TS val 1308882272 ecr 1016951745], length 0 19:37:41.962975 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [F.], seq 25, ack 26, win 8208, options [nop,nop,TS val 1309161761 ecr 1016951745], length 0 19:37:41.963836 IP haproxyserver.squid > 10.155.96.64.64473: Flags [F.], seq 26, ack 26, win 31, options [nop,nop,TS val 1017234914 ecr 1309161761], length 0 19:37:41.988875 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [.], ack 27, win 8208, options [nop,nop,TS val 1309161785 ecr 1017234914], length 0
signature.asc
Description: OpenPGP digital signature