Thanks Bryan/Cyril for trying to help me out....I am not super familiar with dealing with systems at that level so I may need a little hand holding...
Here is what the system currently tells me: [2.2.4-RELEASE][root@HTPL-PROXY-03]/root:* pfctl -si | grep current* current entries 6788 [2.2.4-RELEASE][root@HTPL-PROXY-03]/root: *pfctl -sm* states hard limit 6540000 src-nodes hard limit 6540000 frags hard limit 5000 table-entries hard limit 200000 and haproxy stats shows this: (i have offloaded my clients for now but no limits are reached when the slow down happens, not even close) *maxsock = *100043; *maxconn = *50000; *maxpipes = *0 current conns = 292; current pipes = 0/0; conn rate = 22/sec Running tasks: 1/311; idle = 99 % Based on the comments from Cyril I made the following changes (I did have the maxconn numbers set before for the frontend(s) when the slowness occurred but I took them out trying to solve the problem which probably made it worse) */*var/etc/haproxy.cfg file contents: global maxconn 50000 log /var/run/log local0 info stats socket /tmp/haproxy.socket level admin uid 80 gid 80 nbproc 1 chroot /tmp/haproxy_chroot daemon spread-checks 5 listen HAProxyLocalStats bind 127.0.0.1:2200 name localstats mode http stats enable stats admin if TRUE stats uri /haproxy_stats.php?haproxystats=1 timeout client 5000 timeout connect 5000 timeout server 5000 frontend HTPL_PROXY bind 10.1.4.105:8181 name 10.1.4.105:8181 mode http log global option http-server-close option forwardfor acl https ssl_fc reqadd X-Forwarded-Proto:\ http if !https reqadd X-Forwarded-Proto:\ https if https maxconn 40000 timeout client 5000 default_backend HTPL_WEB_PROXY_http_ipvANY frontend HTPL_CONTENT_FILTER bind 10.1.4.106:8182 name 10.1.4.106:8182 mode tcp log global maxconn 10000 timeout client 5000 default_backend HTPL_CONT_FILTER_tcp_ipvANY backend HTPL_WEB_PROXY_http_ipvANY mode http cookie SERVERID insert indirect balance roundrobin timeout connect 50000 timeout server 50000 retries 3 server HTPL-PROXY-01 10.1.4.103:3128 cookie HTPLPROXY01 check inter 60000 weight 150 fastinter 1000 fall 5 server HTPL-PROXY-02 10.1.4.104:3128 cookie HTPLPROXY02 check inter 60000 weight 100 fastinter 1000 fall 5 server HTPL-PROXY-03 10.1.4.107:3128 cookie HTPLPROXY03 check inter 60000 weight 50 fastinter 1000 fall 5 server HTPL-PROXY-04 10.1.4.108:3128 cookie HTPLPROXY04 check inter 60000 weight 200 fastinter 1000 fall 5 server HTHPL-PROXY-01 10.1.4.101:3128 cookie HTHPLPROXY01 check inter 60000 disabled weight 150 fastinter 1000 fall 5 server HTHPL-PROXY-02 10.1.4.102:3128 cookie HTHPLPROXY02 check inter 60000 disabled weight 100 fastinter 1000 fall 5 backend HTPL_CONT_FILTER_tcp_ipvANY mode tcp balance roundrobin timeout connect 50000 timeout server 50000 retries 3 server HTHPL-PROXY-01 10.1.4.101:1344 check inter 60000 disabled weight 100 fastinter 1000 fall 5 server HTHPL-PROXY-02 10.1.4.102:1344 check inter 60000 disabled weight 100 fastinter 1000 fall 5 server HTPL-WEB-01 10.1.4.153:1344 check inter 60000 weight 200 fastinter 1000 fall 5 server HTPL-WEB-02 10.1.4.154:1344 check inter 60000 weight 200 fastinter 1000 fall 5 On Fri, Oct 2, 2015 at 2:17 PM, Bryan Talbot <bryan.tal...@ijji.com> wrote: > On Fri, Oct 2, 2015 at 1:48 PM, Daren Sefcik <dsef...@hightechhigh.org> > wrote: > >> I Hope this is the right place to ask for help..if not please flame me >> and send me on my way.... >> >> So I had haproxy 1.5 installed (as a front end for a cluster of squid >> proxies) on a low end Dell server with pfsense(PFS) 2.1.5 and was >> experiencing slow down with 1500+ connections so I built up a new PFS >> 2.2.4 machine on a brand new Dell R630 with 64gb RAM, Dual CPU, bad ass >> raid disks etc....loaded and configured haproxy with several squid backends >> and some ICAP backends. Things work great until I hit about 1500 or more >> connections and then everything just slows to a crawl. Restarting haproxy >> helps momentarily but it will slow back down again very quickly. If I >> offload clients to the point of only 300-400 connections it will become >> responsive again. In the haproxy stats page it will show 97% idle or >> similar and the output from top will show maybe 5% cpu for haproxy. If I >> configure the browser client to use one of the squid backends directly it >> works fast but as soon as I put the broswer proxy config back to use the >> haproxy frontend IP it will slow down. >> > > > The problem seems consistent with your connection tracking tables filling > up. You don't say if the 1500 concurrent connections creates a lot of new > connections or if they are 1500 connections that last for a long time. If > your connection lifetime is short then the connection tracking tables > probably need to be tuned. > > I don't recall what the conntrack controls are for FreeBSD but it's > probably something in the pfctl utility, right? > > -Bryan > >
