Hi Luca, It seems your clients are closing connections instead of the servers, leading HAProxy to run out of free ports to get connected on the server side. Actually, the "source" directive should help fixing your issue since you would have 64K ports per client IP to get connected to servers instead of relying only on HAProxy box local IP and its only 64K ports. That said, the source directive may be improved by using "clientip" instead of "client". More information here: http://cbonte.github.io/haproxy-dconv/snapshot/configuration-1.6.html#source%20%28Alphabetically%20sorted%20keywords%20reference%29
more about source port exhaustion (applied to mysql): http://blog.haproxy.com/2012/12/12/haproxy-high-mysql-request-rate-and-tcp-source-port-exhaustion/ Don't forget to set sysctl net.ipv4.ip_local_port_range Baptiste On Thu, Oct 22, 2015 at 1:56 PM, luca boncompagni <[email protected]> wrote: > Hi to all, > On my production server running on fedora 20 and haproxy 1.5.2: > > Linux prod-lb01.prod 3.15.10-200.fc20.x86_64 #1 SMP Thu Aug 14 15:39:24 UTC > 2014 x86_64 x86_64 x86_64 GNU/Linux > [root@prod-lb01 ~]# rpm -qa | grep hapro > haproxy-debuginfo-1.5.2-1.fc20.x86_64 > haproxy-1.5.2-1.fc20.x86_64 > > after disabling tcp_timestamp for securtiy reaseon > (http://www.forensicswiki.org/wiki/TCP_timestamps): > > [root@prod-lb01 ~]# echo 0 > /proc/sys/net/ipv4/tcp_timestamps > > I get a a lot of "no free ports" in the log and client receives a connection > reset : > > [root@prod-lb01 ~]# wc -l /var/log/haproxy/haproxy-20151021.log > 841275 /var/log/haproxy/haproxy-20151021.log > [root@prod-lb01 ~]# grep -c Connect /var/log/haproxy/haproxy-20151021.log > 29091 > [root@prod-lb01 ~]# grep Connect /var/log/haproxy/haproxy-20151021.log | > grep -c 'Oct 21 14:57:11 ' > 19 > > My configuration set the retries number to 18: > > defaults > mode tcp > log global > option dontlognull > option tcplog > option redispatch > timeout connect 10s > timeout client 3600s > timeout client-fin 60s > timeout server 3600s > #timeout server-fin 60s > maxconn 20000 > # Set retries needed with balance source to avoid connection errors on > the client side > # With: "check inter 10s fastinter 2s fall 3" and considering every > retry waits 1 second: > # set retries >= inter + fastinter * fall = 10 + 2 * 3 = 16 > retries 18 > default-server inter 10s fastinter 2s fall 3 > > frontend ssl > bind 192.168.1.4:443 > bind 192.168.2.10:443 > default_backend ssl > > backend ssl > balance source > source 0.0.0.0 usesrc client > option allbackups > server web01 192.168.1.21:4443 check > server web02 192.168.1.22:4443 check > server web03 192.168.1.23:4443 check > server web04 192.168.1.24:4443 check > server sorry01 192.168.1.31:4443 backup check > server sorry02 192.168.1.32:4443 backup check > > I upgraded to the fedora 21 and haproxy 1.5.14: > > Linux prod-lb02.prod 4.1.5-100.fc21.x86_64 #1 SMP Tue Aug 11 00:24:23 UTC > 2015 x86_64 x86_64 x86_64 GNU/Linux > [root@prod-lb02 ~]# rpm -qa | grep hapro > haproxy-1.5.14-1.fc21.x86_64 > > and I get the same rate of errors. > > If I reenable the tcp timestamp: > > [root@prod-lb01 ~]# echo 0 > /proc/sys/net/ipv4/tcp_timestamps > > everithings works well in both vetsions of fedora. > > Do you have any idea about a resolution? > > Luca
