Hi David, On Thu, Feb 13, 2014 at 01:50:16PM +0000, David Harrold wrote: > Hi Willy > > Did some more investigation on the case where the application request is too > large to > fit within the initial SYN. > > Here is my test setup: > > Web clients ??> haproxy ??> long-thin-pipe ?> haproxy --?> web > servers > TFO Client > TFO Server > > Client sends an HTTP request larger than MSS, the client side haproxy uses > TFO and puts as much data > as possible within the initial SYN. When SYN ACK is returned, the remaining > request data is sent. > On closer inspection although the correct number of octets are sent, the > octets in the continuation packet are all NUL. > > E.g. Debug shows 1500 octets in the call to sendto() and a return value of > 1500. > Wireshark shows TFO sending 1420 octets in the SYN. After SYN ACK comes back, > 80 octets are sent in the next packet, > but these 80 octets are all NUL.
OK so that's clearly a bug below. > Looks like something broken in the TFO client, but would be good to see if > others can duplicate my results. > > I?m testing using VMware which I think emulates TCP offload by default, > wondering whether that could be the cause? Could be indeed, we've got a few issues with GRO/GSO in the past as well. I'll have to run some tests with your patch to see with different kernels if I can reproduce the same issue. It is also possible that it was a known old bug that's already fixed but not backported to some stable branches. > Regarding default values for the TFO backlog - I was concerned that if this > is maxconn then is > there a DoS vulnerability? Eg if a TFO client streams SYNs with random data > at you, each of these ties up > an haproxy connection for a while, starving other clients? But it's the same with real connections in practice, because even when the connection is accepted, we still need to parse it. This is also the reason for having a short http-request timeout. For example, if you support 100k concurrent connections on the frontend and wait for a request for 5 seconds, a client will have to send 20k conns/s to keep you full. In practice, even at this rate, you'll accept 100k extra conns in the backlog which will get served but will have to wait 0 to 5s on average. The worst thing to do is to reduce the accept rate, which lowers the bar for attackers. The higher the limit, the more information we have for dealing with pending data. One of the easy things we are already able to do is count the number of concurrent connections per source address for example. Something we cannot do if we refrain from accepting these connections. I also have some memories about the network stack defending itself when a SYN queue overflows, it would reject TFO or accelerate the regeneration of cookies, I don't remember exactly. Cheers, Willy
