Hi, Today I noticed after a reload that previous process was alive for long time( >8hours). This is a HAProxy which runs in HTTP mode in front of few squid servers, conf is quite simple[1] and the version is 1.5.6[2]
I had a lsof watcher for the old pid and the number of connections were
very slowly dropping from 2K to 200 right now.
For few of the connections that were in established state( for the old
process) I run tcpdump and saw no activity at all, I have attached a
network trace from one those and you can see that client sends
periodically every 10min 5bytes. The HAProxy is used by normal browsers
but also from cronjobs with various languages(Perl,Python,C,Go etc)
I was surprised about this very long inactivity period for TCP
connection on a system which has reasonable settings for TCP keepalive[3].
But setting 'timeout tunnel' is not set, and since this HAProxy is
serving proxy traffic to squid all client/server connections are treated
as tunnels. am I right?
My question is about TCP keepalive and tunnel. Are system keepalive
settings ignored when HAProxy treads client/servers connections as tunnels?
Cheers,
Pavlos
[1]
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 65536
tune.bufsize 65536
user haproxy
group haproxy
daemon
stats socket /var/lib/haproxy/stats uid 0 gid 0 mode 0440 level admin
defaults
mode http
log global
option httplog clf
option dontlognull
option forwardfor except 10.0.0.0/8
option redispatch
option http-server-close
option http-use-proxy-header
option tcp-smart-accept
option tcp-smart-connect
no option checkcache
retries 3
maxconn 65536
timeout queue 1m
timeout connect 4s
timeout client 30m
timeout server 30m
timeout check 10s
timeout http-request 10s
timeout http-keep-alive 10s
errorfile 408 /dev/null
listen haproxy :8080
mode http
stats enable
stats uri /
stats show-node
stats refresh 10s
frontend http_in *:3128
default_backend squid_http
backend squid_http
balance leastconn
server squid-01 squid-01:3128
server squid-02 squid-02:3128
server squid-03 squid-03:3128
server squid-04 squid-04:3128
[2]
haproxy -vv
HA-Proxy version 1.5.6 2014/10/18
Copyright 2000-2014 Willy Tarreau <[email protected]>
Build options :
TARGET = linux2628
CPU = generic
CC = gcc
CFLAGS =
OPTIONS = USE_LINUX_TPROXY=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1
USE_PCRE=1
Default settings :
maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200
Encrypted password support via crypt(3): yes
Built with zlib version : 1.2.3
Compression algorithms supported : identity, deflate, gzip
Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 7.8 2008-09-05
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT
IPV6_TRANSPARENT IP_FREEBIND
Available polling systems :
epoll : pref=300, test result OK
poll : pref=200, test result OK
select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.
pparissis at corplbout-201 in ~
[3]
sudo sysctl -a|grep keepalive
net.ipv4.tcp_keepalive_time = 30
net.ipv4.tcp_keepalive_probes = 2
net.ipv4.tcp_keepalive_intvl = 1
tcpdump host 10.155.96.64 and port 64473 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes 18:56:52.528548 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [P.], seq 2900223928:2900223933, ack 3414209439, win 8210, options [nop,nop,TS val 1306751675 ecr 1014245525], length 5 18:56:52.548765 IP haproxyserver.squid > 10.155.96.64.64473: Flags [P.], seq 1:6, ack 5, win 31, options [nop,nop,TS val 1014785499 ecr 1306751675], length 5 18:56:52.575487 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [.], ack 6, win 8209, options [nop,nop,TS val 1306751720 ecr 1014785499], length 0 19:06:00.052136 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [P.], seq 5:10, ack 6, win 8209, options [nop,nop,TS val 1307290366 ecr 1014785499], length 5 19:06:00.069963 IP haproxyserver.squid > 10.155.96.64.64473: Flags [P.], seq 6:11, ack 10, win 31, options [nop,nop,TS val 1015333020 ecr 1307290366], length 5 19:06:00.094985 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [.], ack 11, win 8209, options [nop,nop,TS val 1307290408 ecr 1015333020], length 0 19:15:01.559202 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [P.], seq 10:15, ack 11, win 8209, options [nop,nop,TS val 1307822802 ecr 1015333020], length 5 19:15:01.577110 IP haproxyserver.squid > 10.155.96.64.64473: Flags [P.], seq 11:16, ack 15, win 31, options [nop,nop,TS val 1015874527 ecr 1307822802], length 5 19:15:01.602712 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [.], ack 16, win 8209, options [nop,nop,TS val 1307822844 ecr 1015874527], length 0 19:23:59.161132 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [P.], seq 15:20, ack 16, win 8209, options [nop,nop,TS val 1308351633 ecr 1015874527], length 5 19:23:59.179017 IP haproxyserver.squid > 10.155.96.64.64473: Flags [P.], seq 16:21, ack 20, win 31, options [nop,nop,TS val 1016412129 ecr 1308351633], length 5 19:23:59.204112 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [.], ack 21, win 8208, options [nop,nop,TS val 1308351673 ecr 1016412129], length 0 19:32:58.776691 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [P.], seq 20:25, ack 21, win 8208, options [nop,nop,TS val 1308882229 ecr 1016412129], length 5 19:32:58.794670 IP haproxyserver.squid > 10.155.96.64.64473: Flags [P.], seq 21:26, ack 25, win 31, options [nop,nop,TS val 1016951745 ecr 1308882229], length 5 19:32:58.819939 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [.], ack 26, win 8208, options [nop,nop,TS val 1308882272 ecr 1016951745], length 0 19:37:41.962975 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [F.], seq 25, ack 26, win 8208, options [nop,nop,TS val 1309161761 ecr 1016951745], length 0 19:37:41.963836 IP haproxyserver.squid > 10.155.96.64.64473: Flags [F.], seq 26, ack 26, win 31, options [nop,nop,TS val 1017234914 ecr 1309161761], length 0 19:37:41.988875 IP 10.155.96.64.64473 > haproxyserver.squid: Flags [.], ack 27, win 8208, options [nop,nop,TS val 1309161785 ecr 1017234914], length 0
signature.asc
Description: OpenPGP digital signature
