On Tue, Oct 02, 2012 at 11:42:27PM -0700, Bryan Talbot wrote: > Having one (or some small number) of slow readers isn't a problem. The > problem comes up when some significant percentage of your requests are from > slow readers. Those readers might be from the same client or distributed > which seems pretty common from the attacks we see. > > Would it be possible for haproxy to start dropping connections to slow > readers if some percentage of them compared to capacity (of the frontend, > backend, etc) were slow? If the slowness is due to network congestion, > then those clients are already having connection issues anyway.
It's not necessarily true unfortunately. And if you want to be kind with your visitors, you'd better kill the fast connections because in one click they can get the same object again, than killing the ones who'll need another hour of download (and of connection usage on your side BTW). If you have some captures of such slow readers that are attacking you, I would be interested, I think they can be very useful. One significant issue to deal with is that the system's network buffers hide the slow ACKs to the application layer, so you can't easily detect that someone is reading one byte at a time, and distinguishing someone who does this from someone experiencing repeated retransmits (typically a smartphone) is very hard. What if your server is saturated serving smartphones ? You don't necessarily want to kill many transfers, it will only make things worse because they'll be there again after a retry. Probably that we should try to consider the transmission quality over the whole transfer, which might be similar to the transfer rate after all. The difference between the slow reader and other readers is that the slow reader reads *very* slowly in order to have its connection last as long as possible. The average send() size as well as the average time between two send() calls might be a good indication of what is happening. But even then, you see, the attack can change a bit. Imagine for a second that the attacker connects to your site, waits one second, sends "HEAD /favicon.ico", waits one second and does it again over the same connection. It's just one tiny request per second to keep the connection alive. Should we block them or let them pass ? Very hard to tell. This is a perfectly valid transfer pattern, but if abused, it becomes an attack. The difference is that the connection to the backend server will be very ephemeral with haproxy, but doing this with a normal server is as painful as reading one small segment at a time. Willy
