On Sat, May 31, 2014 at 09:22:12PM +0200, Jakov Sosic wrote: > On 05/27/2014 08:36 PM, Willy Tarreau wrote: > >I don't see why GoogleBot would see them since they should only affect > >the offending clients. > > > >>Is it by any chance possible that my ISP is somehow screwing up > >>connections? Because I see this kind of aborts/503s even from regular > >>clients fetching regular stuff? > > > >Could be possible, but that sounds really strange. You could easily check > >though, if you own a machine somewhere outside your ISP's network. Simply > >send a request from there to your site and sniff at both ends. You'll see > >if the trace matches or not. It could be possible that the ISP is running > >a misconfigured transparent proxy which systematically closes the request > >path after sending the request (as haproxy used to do with option > >forceclose > >in early version 1.1 12 years ago). Or maybe it's part of an IDS or > >anti-ddos > >mechanism that's automatically enabled when they run into trouble. > > I've talked to ISP technicians and what they told me is that company has > bandwith cap at XYZ Mbits, and once that limit is reached additional > packets are simply dropped. > > So, packets dropping at peaks seems promising as explanation of some of > the behaviour we have observed...
Wow, I didn't know there were still people doing that ugly thing. In general they're doing this with cheap switches with very short queues instead of routers with deep queues, and the marvellous news here is that such switches start to consider the limit reached during very short bursts of a few packets sometimes, resulting in the absolute impossibility to reach even near the capped bandwidth without already dropping a lot of packets. The traffic shaping we noted in the roadmap file a few years ago was aimed at exactly this problem. By having haproxy pace the traffic before it becomes packets, it will be possible to constantly stay below the threshold of such crappy equipments. But that was postponed for 1.6, maybe even later. Thanks for letting us know! Willy
