Having one (or some small number) of slow readers isn't a problem. The problem comes up when some significant percentage of your requests are from slow readers. Those readers might be from the same client or distributed which seems pretty common from the attacks we see.
Would it be possible for haproxy to start dropping connections to slow readers if some percentage of them compared to capacity (of the frontend, backend, etc) were slow? If the slowness is due to network congestion, then those clients are already having connection issues anyway. -Bryan On Tue, Oct 2, 2012 at 11:13 PM, Willy Tarreau <[email protected]> wrote: > On Tue, Oct 02, 2012 at 04:12:33PM +0200, Baptiste wrote: > > Hi, > > > > Your timeouts are too long: > > timeout client 1020000 > > timeout server 1020000 > > > > This one sounds good: > > timeout http-request 6000 > > > > As far as I understand, your application is not compatible with > > slowloris protection :) > > It's not slowloris here, it's slow reading, which is the opposite. It's > a lot harder to protect against this on web servers precisely because of > today's applications behaviour. > > If you want to apply a timeout, you can't because the receiver will read > almost 1 byte every second. > > If you want to set a max transfer time (eg: 1 minute max), it will not > work : > - 1 minute is far too long for small objects > - 1 minute is far too short for large objects > > Ideally we need to consider the transfer rate : clients transfering too > slowly are building an attack. But this needs to be detected very quickly > (otherwise the attack works). And if you compute transfer rate over small > elements, you have to consider TCP retransmits which can add 3*2^N seconds > on losses. This means that it can be perfectly normal to have several > pauses of 3 or 9 seconds during a transfer if the uplink to the client > is congested. > > But if we allow even just 3 seconds for every send, then the attack still > works. > > I think that in practice we should consider the time it takes to transfer > the equivalent of two TCP segments (2896..2920 bytes over ethernet), > because > TCP acks every second segment. The idea could be that we rearm the client > timeout only after *that* many bytes have been transferred, or the buffer > is empty. At least we could still support slow clients (eg: 1kB/s) but not > extremely slow ones. But quite frankly, even at 1kB/s with a 1MB object, > it is possible to harm the server, the transfer will last 20 minutes. > > So what are we left with ? We want to support reasonably slow clients, > TCP retransmits over the internet and large objects ? There is a big > difference between a slow client and a slowread attack : the number of > concurrent connections from the same client. If your slowreader only > has one connection to your site, you don't mind. If it has 10, then it > becomes an issue. > > Fast clients such as proxies won't have too many connections even if they > have many users behind, because : > 1) they will merge similar requests > 2) they generally transfer much faster than the average client, which > significantly reduces the number of concurrent connections > > So it is perfectly possible to enforce the limit on the number of > concurrent > conns to stop such an attack, using a stick-table and a tcp-request rule. > The slow transfer will not be the key to detect the anomaly, but the fact > that the client needs many connections to succeed will be the key. > > Regards > Willy > > >
