Hi, Lukas! Thanks for your reply!
And my OS is : FreeBSD Haproxy01 10.0-BETA2 FreeBSD 10.0-BETA2 #0 r257417: Thu Oct 31 13:02:48 CST 2013 haproxy version: root@Haproxy01:/usr/ports/net/haproxy-devel # haproxy -vv HA-Proxy version 1.5-dev21-6b07bf7 2013/12/17 Copyright 2000-2013 Willy Tarreau <[email protected]> Build options : TARGET = freebsd CPU = generic CC = gcc47 CFLAGS = -O2 -fno-strict-aliasing -pipe -msse3 -I/usr/local/include -L/usr/local/lib -fno-omit-frame-pointer -Wl,--eh-frame-hdr -DFREEBSD_PORTS OPTIONS = USE_TPROXY=1 USE_GETADDRINFO=1 USE_ZLIB=1 USE_REGPARM=1 USE_OPENSSL=1 USE_STATIC_PCRE=1 USE_PCRE_JIT=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built with zlib version : 1.2.8 Compression algorithms supported : identity, deflate, gzip Built with OpenSSL version : OpenSSL 1.0.1e-freebsd 11 Feb 2013 Running on OpenSSL version : OpenSSL 1.0.1e-freebsd 11 Feb 2013 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 8.33 2013-05-28 PCRE library supports JIT : yes Built with transparent proxy support using: IP_BINDANY IPV6_BINDANY Available polling systems : kqueue : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use kqueue. And my whole configuration below #2013############## global pidfile /var/run/haproxy.pid maxconn 100000 maxpipes 50000 daemon stats socket /tmp/haproxy.sock mode 755 level admin nbproc 1 spread-checks 5 tune.rcvbuf.client 16384 tune.rcvbuf.server 32768 tune.sndbuf.client 65536 tune.sndbuf.server 16384 node haproxy defaults #TCP SECTION maxconn 200000 backlog 32768 timeout connect 10s timeout client 60s timeout server 60s timeout queue 30s timeout check 5s timeout http-request 5s timeout http-keep-alive 10s timeout tunnel 3600s # option nolinger # option http-no-delay #HTTP SECTION option accept-invalid-http-request option accept-invalid-http-response option redispatch retries 2 option httplog no option checkcache option http-keep-alive #SYSTEM SECTION option dontlog-normal option dontlognull option log-separate-errors ########################## listen admin_stat bind :2101 mode http log global stats enable stats refresh 30s stats uri /admin?stats stats realm Haproxy\ Statistics stats auth admin:haproxy2012 stats hide-version ######### frontend ############## frontend tcp-in-tos02 bind :2001 mss 1360 transparent mode tcp log global option tcplog no option http-keep-alive no option accept-invalid-http-request #distingush HTTP and non-HTTP tcp-request inspect-delay 30s tcp-request content accept if HTTP #ACL DEFINE acl squid_incompatiable-Host hdr_reg(Host) -f /usr/local/etc/acl-define.d/squid_incompatiable-Host.txt #ACL DEFINE of websocket acl missing_host hdr_cnt(Host) eq 0 acl has_range hdr_cnt(Range) gt 0 acl check_SquidCluster-tos02 nbsrv(SquidCluster-tos02) 0 #ACL DEFINE of websocket acl is_websocket hdr(Upgrade) -i WebSocket acl is_websocket hdr_beg(Host) -i ws acl matches_media url_reg -i -f /usr/local/etc/acl-define.d/whitelist.txt acl check_bk_SquidMediaCluster-tos02 nbsrv(SquidMediaCluster-tos02) 0 #ACTION use_backend Direct if !HTTP use_backend Direct if HTTP_1.1 missing_host use_backend Direct if METH_CONNECT use_backend NginxClusterWebsockets if is_websocket use_backend NginxClusterNormal if HTTP squid_incompatiable-Host use_backend SquidMediaCluster-tos02 if HTTP matches_media !check_bk_SquidMediaCluster-tos02 use_backend SquidCluster-tos02 if !check_SquidCluster-tos02 default_backend Direct #default_backend SquidCluster-tos02 backend SquidCluster-tos02 mode http option forwardfor header X-Client balance hdr(Host) log global acl mgmt-src src -f /usr/local/etc/acl-define.d/mgmt-src.txt acl is_internal_error status ge 500 #reqadd Internal-Proto:\ 02 rspideny . if is_internal_error !mgmt-src rspidel ^via:.* unless mgmt-src rspidel ^x-cache:* unless mgmt-src rspidel ^x-cache-lookup:* unless mgmt-src rspidel ^X-Ecap:* unless mgmt-src source 0.0.0.0 option httpchk GET http://www.baidu.com server sq-L1-n1a 192.168.138.1:3001 weight 20 check inter 5s maxconn 10000 server sq-L1-n1b 192.168.138.1:3002 weight 20 check inter 5s maxconn 10000 server sq-L1-n1c 192.168.138.1:3003 weight 20 check inter 5s maxconn 10000 server sq-L1-n2a 192.168.138.2:3001 weight 20 check inter 5s maxconn 10000 server sq-L1-n2b 192.168.138.2:3002 weight 20 check inter 5s maxconn 10000 server sq-L1-n3a 192.168.138.3:3001 weight 20 check inter 5s maxconn 10000 server sq-L1-n3b 192.168.138.3:3002 weight 20 check inter 5s maxconn 10000 server sq-L1-n3c 192.168.138.3:3003 weight 20 check inter 5s maxconn 10000 server sq-L1-n3d 192.168.138.3:3004 weight 20 check inter 5s maxconn 10000 backend SquidMediaCluster-tos02 mode http option forwardfor header X-Client balance uri whole log global acl mgmt-src src -f /usr/local/etc/acl-define.d/mgmt-src.txt errorfile 502 /usr/local/etc/errorfiles/504.http acl is_internal_error status ge 500 rspideny . if is_internal_error !mgmt-src rspidel ^via:.* unless mgmt-src rspidel ^x-cache:* unless mgmt-src rspidel ^x-cache-lookup:* unless mgmt-src rspidel ^X-Ecap:* unless mgmt-src source 0.0.0.0 option httpchk GET http://www.baidu.com server Squid_layer1_backend1d 192.168.138.1:3004 weight 20 check inter 5s maxconn 10000 backend NginxClusterNormal mode http balance hdr(Host) log global acl mgmt-src src -f /usr/local/etc/acl-define.d/mgmt-src.txt rspidel ^via:.* unless mgmt-src rspidel ^x-cache:* unless mgmt-src rspidel ^x-cache-lookup:* unless mgmt-src source 0.0.0.0 option httpchk GET / HTTP/1.1\r\nHost:\ www.baidu.com server XEN-L2-xn1-n2 192.168.172.17:2222 weight 20 check inter 5s maxconn 10000 server XEN-L2-xn2-n2 192.168.172.16:2222 weight 20 check inter 5s maxconn 10000 backend NginxClusterWebsockets mode http balance hdr(Host) log global acl mgmt-src src -f /usr/local/etc/acl-define.d/mgmt-src.txt rspidel ^via:.* unless mgmt-src rspidel ^x-cache:* unless mgmt-src rspidel ^x-cache-lookup:* unless mgmt-src source 0.0.0.0 option httpchk GET / HTTP/1.1\r\nHost:\ www.baidu.com server XEN-L2-xn1 192.168.172.17:3333 weight 20 check inter 5s maxconn 10000 server XEN-L2-xn2 192.168.172.16:3333 weight 20 check inter 5s maxconn 10000 backend Direct mode tcp log global option tcplog no option http-keep-alive no option httpclose no option http-server-close no option accept-invalid-http-response no option http-pretend-keepalive source 0.0.0.0 usesrc clientip option transparent On Wed, Jan 8, 2014 at 3:12 AM, Lukas Tribus <[email protected]> wrote: > Hi, > >> Recently, we use haproxy1.5-dev21 in our product.And we want to get >> the benefit of http-keep-alive. But after we added the option >> http-keep-alive and deployed new version of haproxy. We found that the >> connection of FIN_WAIT_2 CLOSED ESTABLISHED increased quickly. when we >> change to the tunnel mode, it decreased. > > What release did you previously run? Please also specify your kernel release > and the output of "./haproxy -vv". > > > >> root@Haproxy01:~ # session-count.sh >> LISTEN 8 >> FIN_WAIT_1 245 >> FIN_WAIT_2 22836 >> SYN_SENT 46 >> LAST_ACK 943 >> CLOSING 4 >> CLOSE_WAIT 1151 >> CLOSED 21940 >> SYN_RCVD 11 >> TIME_WAIT 255 >> ESTABLISHED 13894 > > But we don't know where does high numbers are, backend or frontend (or both; > equally distributed). > > Can you try (by matching your frontend port): > netstat -nat | grep ":2001 " | wc -l > > > >> And some related configuration below. > > From your configuration its not clear to me what was really enabled > when you were having this problem, and what was later configured as a > workaround: > > - you globablly enable "option http-keep-alive" > - you disable "no option http-keep-alive" in your (only?) frontend tcp-in > - you disable "no option http-keep-alive" in your backend Direct > - your backend SquidCluster-tos02 doesn't mention http-keep-alive, but > it will not use it since its disabled on the frontend. > > > Please provide a consistent and complete configuration when your where > having this problem, otherwise its very difficult for us to understand the > situation. > > Also, if you disable keep-alive completely (which, it seems, you already > did), without downgrading the release to the old one, does this fix the > problem? > > I suspect there is some issue or limitation with keep-alive in combination > with TPROXY/CTTPROXY. > > > > Regards, > > Lukas
