I looked at my pcap file again. It looks really weird. My HAProxy gets the GET request, and sends the response. The the client resends the GET request, and there seems to be a lot of tcp_retransmission and dup ack packets. Here is a picture of one request to my haproxy:
http://i.imgur.com/r3oz6lz.png Any clue what would cause that problem? I tried to change the max_syn_backlog, and somaxconn values to both 10240/20480 and 262144/262144, neither seemes to have solved the problem. Conntrack is not loaded, I checked this a while ago, and I am not using anything that would load it. Here are the commands below. It also doesn't look like anything is being dropped. The interface I am using is bond1. root@haproxy2:~# dmesg | grep -i full [ 1.834160] usb 1-1.2: new full-speed USB device number 3 using ehci_hcd [ 69.515103] e1000e: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None [ 69.605840] bonding: bond0: link status definitely up for interface eth0, 1000 Mbps full duplex. [ 130.088812] e1000e: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: None [ 130.123654] bonding: bond1: link status definitely up for interface eth1, 1000 Mbps full duplex. root@haproxy2:~# netstat -i Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg bond0 1500 0 271295 0 0 0 639434 0 0 0 BMmRU bond1 1500 0 503194 0 0 0 393965 0 0 0 BMmRU eth0 1500 0 271295 0 0 0 639434 0 0 0 BMsRU eth1 1500 0 503195 0 0 0 393966 0 0 0 BMsRU eth2 1500 0 0 0 0 0 0 0 0 0 BMsU eth3 1500 0 0 0 0 0 0 0 0 0 BMsU lo 16436 0 722 0 0 0 722 0 0 0 LRU On Fri, Oct 11, 2013 at 1:35 AM, Willy Tarreau <[email protected]> wrote: > On Thu, Oct 10, 2013 at 03:57:37PM -0400, Andy M. wrote: > > Thanks for the response pelle, > > > > The haproxy box is running at about 10-15% cpu. > > > > Looking at the TCP Dump, it seems the client is doing 1 or more > > TCP_Retransmissions of the incoming request when there is a delay. > Anyone > > know why this would happen? > > When you say "the incoming request", you mean the SYN packet, right ? > If so, that means the backlog is full. > > > Here is the requested information: > > root@haproxy:~# sysctl net.core.somaxconn > > net.core.somaxconn = 128 > > This is far too low or 20k CPS. You fill it in 6 ms at 20k CPS. > Try increasing it to 1024 to see how things go. > > > root@haproxy:~# sysctl net.ipv4.tcp_max_syn_backlog > > net.ipv4.tcp_max_syn_backlog = 2048 > > In practice, I'm used to increase somaxconn to ~10000 and max_syn_backlog > to ~20000, but it's a bit more tricky as you don't want somaxconn to be > larger than haproxy's listen queue, otherwise the system will disable SYN > cookies. In practice, 1024 should already be OK (about 51ms to fill it). > > > I removed the tcp_tw_recycle, but this did not change anything > > OK. If things do not improve with the settings above, you'll have to > check if you're running netfilter and to see if your contrack table is > never full : > > # dmesg | grep -i full > > You also need to check if your network card or driver is not dropping > packets : > > # netstat -i > > Regards, > Willy > >
