On Fri, Mar 06, 2009 at 02:36:59PM -0800, Michael Fortson wrote: > On Fri, Mar 6, 2009 at 1:46 PM, Willy Tarreau <[email protected]> wrote: > > On Fri, Mar 06, 2009 at 01:00:38PM -0800, Michael Fortson wrote: > >> Thanks Willy -- here's the sysctl -a |grep ^net output: > >> http://pastie.org/409735 > > > > after a quick check, I see two major things : > > - net.ipv4.tcp_max_syn_backlog = 1024 > > => far too low, increase it to 10240 and check if it helps > > > > - net.netfilter.nf_conntrack_max = 265535 > > - net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120 > > => this proves that netfiler is indeed running on this machine > > and might be responsible for session drops. 265k sessions is > > very low for the large time_wait. It limits to about 2k > > sessions/s, including local connections on loopback, etc... > > > > You should then increase nf_conntrack_max and nf_conntrack_buckets > > to about nf_conntrack_max/16, and reduce nf_conntrack_tcp_timeout_time_wait > > to about 30 seconds. > > > >> Our outbound cap is 400 Mb > > > > OK so I think you're still far away from that. > > > > Regards, > > Willy > > > > > > Hmm; I did these (John is right, netfilter is down at the moment > because I dropped iptables to help troubleshoot this),
What did you unload precisely ? You don't need any iptables rules for the conntrack to take effect. > so I guess the > syn backlog is the only net change. No difference so far -- still > seeing regular 3s responses. > > It's weird, but I actually see better results testing mongrel than > nginx; haproxy => mongrel heartbeat is more reliable than the haproxy > => nginx request. mongrel is on another machine ? You might be running out of some resource on the local one making it difficult to reach accept(). Unfortunately I don't see what :-( Have you checked with "dmesg" that you don't have network stack errors or any type of warning ? Willy
