I am trying to get closer to the switch, unfortunately my boxes are in a data center I do not have access too. I am working with them to try and debug this, so far it seems like in general the servers are re transmitting packets under the load I am testing at, not just packets through HAProxy, so I don't think this is a problem with HAProxy, but something in their network. If anything comes up I will update, but as of right now, I think the proxy is not that cause. Thanks again, you've all been a huge help.
On Fri, Oct 11, 2013 at 11:36 AM, Willy Tarreau <[email protected]> wrote: > On Fri, Oct 11, 2013 at 10:59:51AM -0400, Andy M. wrote: > > I looked at my pcap file again. It looks really weird. My HAProxy gets > > the GET request, and sends the response. The the client resends the GET > > request, and there seems to be a lot of tcp_retransmission and dup ack > > packets. Here is a picture of one request to my haproxy: > > > > http://i.imgur.com/r3oz6lz.png > > > > Any clue what would cause that problem? > > Yes, a typical packet loss between you and the client. > > > I tried to change the max_syn_backlog, and somaxconn values to both > > 10240/20480 and 262144/262144, neither seemes to have solved the problem. > > Here it's not the SYN backlog since it's the HTTP request that is > retransmitted. > It is possible that your network interface has a defect. It happened to me > once, > in a batch of 10 NICs, 3 had defective RAM chips which would randomly > corrupt > outgoing packets. Try to use another interface or another switch port. > > > Conntrack is not loaded, I checked this a while ago, and I am not using > > anything that would load it. Here are the commands below. It also > doesn't > > look like anything is being dropped. The interface I am using is bond1. > > Great, since you're using bonding, it's easy to switch to the other NIC and > see if it works better. > > From your stats, I'm assuming you're not running with both NICs attached to > the same bond in round robin. I was just checking, because doing so would > expose you to a high probability of disordering packets, which some > firewalls > generally don't accept and will block, causing the client to retransmit. > > I think now you need to sniff closer to the client to see where the packets > are lost. If you can make a span on your switch to check if it correctly > receives them, that will help you. > > Willy > >
