Hi Baptiste, I'll try your suggestiion, but I'd like to understand why if I enable tcp_timestamp I have no problems and if I disable it, after few minutes on the live system I get the problem.
thanks, Luca 2015-10-22 14:08 GMT+02:00 Baptiste <[email protected]>: > Hi Luca, > > It seems your clients are closing connections instead of the servers, > leading HAProxy to run out of free ports to get connected on the > server side. > Actually, the "source" directive should help fixing your issue since > you would have 64K ports per client IP to get connected to servers > instead of relying only on HAProxy box local IP and its only 64K > ports. > That said, the source directive may be improved by using "clientip" > instead of "client". > More information here: > > http://cbonte.github.io/haproxy-dconv/snapshot/configuration-1.6.html#source%20%28Alphabetically%20sorted%20keywords%20reference%29 > > more about source port exhaustion (applied to mysql): > > http://blog.haproxy.com/2012/12/12/haproxy-high-mysql-request-rate-and-tcp-source-port-exhaustion/ > > Don't forget to set sysctl net.ipv4.ip_local_port_range > > Baptiste > > On Thu, Oct 22, 2015 at 1:56 PM, luca boncompagni <[email protected]> wrote: >> Hi to all, >> On my production server running on fedora 20 and haproxy 1.5.2: >> >> Linux prod-lb01.prod 3.15.10-200.fc20.x86_64 #1 SMP Thu Aug 14 15:39:24 UTC >> 2014 x86_64 x86_64 x86_64 GNU/Linux >> [root@prod-lb01 ~]# rpm -qa | grep hapro >> haproxy-debuginfo-1.5.2-1.fc20.x86_64 >> haproxy-1.5.2-1.fc20.x86_64 >> >> after disabling tcp_timestamp for securtiy reaseon >> (http://www.forensicswiki.org/wiki/TCP_timestamps): >> >> [root@prod-lb01 ~]# echo 0 > /proc/sys/net/ipv4/tcp_timestamps >> >> I get a a lot of "no free ports" in the log and client receives a connection >> reset : >> >> [root@prod-lb01 ~]# wc -l /var/log/haproxy/haproxy-20151021.log >> 841275 /var/log/haproxy/haproxy-20151021.log >> [root@prod-lb01 ~]# grep -c Connect /var/log/haproxy/haproxy-20151021.log >> 29091 >> [root@prod-lb01 ~]# grep Connect /var/log/haproxy/haproxy-20151021.log | >> grep -c 'Oct 21 14:57:11 ' >> 19 >> >> My configuration set the retries number to 18: >> >> defaults >> mode tcp >> log global >> option dontlognull >> option tcplog >> option redispatch >> timeout connect 10s >> timeout client 3600s >> timeout client-fin 60s >> timeout server 3600s >> #timeout server-fin 60s >> maxconn 20000 >> # Set retries needed with balance source to avoid connection errors on >> the client side >> # With: "check inter 10s fastinter 2s fall 3" and considering every >> retry waits 1 second: >> # set retries >= inter + fastinter * fall = 10 + 2 * 3 = 16 >> retries 18 >> default-server inter 10s fastinter 2s fall 3 >> >> frontend ssl >> bind 192.168.1.4:443 >> bind 192.168.2.10:443 >> default_backend ssl >> >> backend ssl >> balance source >> source 0.0.0.0 usesrc client >> option allbackups >> server web01 192.168.1.21:4443 check >> server web02 192.168.1.22:4443 check >> server web03 192.168.1.23:4443 check >> server web04 192.168.1.24:4443 check >> server sorry01 192.168.1.31:4443 backup check >> server sorry02 192.168.1.32:4443 backup check >> >> I upgraded to the fedora 21 and haproxy 1.5.14: >> >> Linux prod-lb02.prod 4.1.5-100.fc21.x86_64 #1 SMP Tue Aug 11 00:24:23 UTC >> 2015 x86_64 x86_64 x86_64 GNU/Linux >> [root@prod-lb02 ~]# rpm -qa | grep hapro >> haproxy-1.5.14-1.fc21.x86_64 >> >> and I get the same rate of errors. >> >> If I reenable the tcp timestamp: >> >> [root@prod-lb01 ~]# echo 0 > /proc/sys/net/ipv4/tcp_timestamps >> >> everithings works well in both vetsions of fedora. >> >> Do you have any idea about a resolution? >> >> Luca
