On Fri, Mar 6, 2009 at 12:53 PM, Willy Tarreau <[email protected]> wrote: > On Fri, Mar 06, 2009 at 11:49:39AM -0800, Michael Fortson wrote: >> Oops, looks like it's actually Gb -> Gb: >> http://pastie.org/409653 > > ah nice ! > >> Here's a netstat -s: >> http://pastie.org/409652 > > Oh there are interesting things there : > > - 513607 failed connection attempts > => let's assume it was for dead servers > > - 34784881 segments retransmited > => this is huge, maybe your outgoing bandwidth is limited > by the provider, causing lots of drops ? > > - 8325393 SYN cookies sent > => either you've been experiencing a SYN flood attack, or > one of your listening socket's backlog is extremely small > > - 1235433 times the listen queue of a socket overflowed > 1235433 SYNs to LISTEN sockets ignored > => up to 1.2 million times some client socket experienced > a drop, causing at least a 3 seconds delay to establish. > The errors your scripts detect certainly account for a small > part of those. > > - 2962458 times recovered from packet loss due to SACK data > => many losses, related to second point above. > > Could you post the output of "sysctl -a |grep ^net" ? I think that > your TCP syn backlog is very low. Your stats page indicate an average > of about 300 sessions/s over the last 24 hours. If your external > bandwidth is capped and causes drops, you can nearly saturate the > default backlog of 1024 with 300 sessions/s each taking 3s to > complete. If you're interested, the latest snapshot will report > the number of sess/s in the stats. > >> Haproxy and nginx are currently on the same box. Mongrels are all on a >> private network accessed through eth1 (public access is via eth0). > > OK. > >> stats page attached (backend "everything" is not currently in use; >> it'll be a use-when-full option for fast_mongrels once we upgrade to >> the next haproxy). > > According to the stats, your avg output bandwidth is around 10 Mbps. > Would this match your external link ? > > Regards, > Willy >
Thanks Willy -- here's the sysctl -a |grep ^net output: http://pastie.org/409735 Our outbound cap is 400 Mb
