I'm running on Rackspace Cloud. I don't think I can run a custom kernel out there.
Michael Marano -- Senior Manager of Web Development Future US, Inc. desk: 650-238-2530 cell: 650-580-2132 twitter: @mmarano aim: michaelvicmarano skype: mmarano > From: Jonah Horowitz <[email protected]> > Date: Wed, 7 Oct 2009 15:18:29 -0700 > To: Michael Marano <[email protected]>, <[email protected]> > Cc: <[email protected]>, Mark Kramer <[email protected]> > Subject: RE: Kernel tuning recommendations > > I ended up just building a kernel without conntrack, module or otherwise. I'm > sure you could prevent conntrack from loading somehow, but this was easier > from my perspective. > > Jonah > > >> -----Original Message----- >> From: Michael Marano [mailto:[email protected]] >> Sent: Wednesday, October 07, 2009 3:03 PM >> To: [email protected] >> Cc: [email protected]; Mark Kramer >> Subject: Re: Kernel tuning recommendations >> >> I've made a handful of changes based up on Chris and Willy's >> suggestions, >> which I've included below. This avoids the nf_conntrack errors in the >> logs. >> >> I would like to skip nf_conntrack altogether. I've been digging around >> to >> try to learn how to do that, but I now admit I don't know how. I can't >> just >> drop the module, as it's currently in use. >> >> [mmar...@w1 w1]$ sudo modprobe -n -r nf_conntrack >> FATAL: Module nf_conntrack is in use. >> >> What do I need to change in my iptables rules to pave the way for >> removing >> this module. Once I've got that straight, how do I then disable the >> module. >> I'm happy to get an RTFM response if I'm just being stupid. Point me at >> the >> right M ;) >> >> Michael Marano >> >> >> ---- iptables rules script --------------- >> #!/bin/sh >> >> sudo /sbin/iptables -F >> sudo /sbin/iptables -A INPUT -i lo -j ACCEPT >> sudo /sbin/iptables -A INPUT -i ! lo -d 127.0.0.0/8 -j REJECT >> sudo /sbin/iptables -A INPUT -m state --state ESTABLISHED,RELATED -j >> ACCEPT >> sudo /sbin/iptables -A OUTPUT -j ACCEPT >> >> # don't track incoming or outgoing port 80 >> sudo /sbin/iptables -t raw -A PREROUTING -p tcp --dport 80 -j NOTRACK >> sudo /sbin/iptables -t raw -A PREROUTING -p tcp --dport 8080 -j NOTRACK >> sudo /sbin/iptables -t raw -A PREROUTING -p tcp --dport 81 -j NOTRACK >> >> # don't track traffic starting from the private ip >> sudo /sbin/iptables -t raw -A PREROUTING -p tcp -s 10.176.45.165 -j >> NOTRACK >> >> # these may not actually be useful, but I'm leaving them in. >> sudo /sbin/iptables -t raw -A OUTPUT -p tcp --sport 80 -j NOTRACK >> sudo /sbin/iptables -t raw -A OUTPUT -p tcp --sport 8080 -j NOTRACK >> sudo /sbin/iptables -t raw -A OUTPUT -p tcp --sport 81 -j NOTRACK >> >> sudo /sbin/iptables -A INPUT -p tcp -m state --state NEW --dport 22 -j >> ACCEPT >> sudo /sbin/iptables -A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT >> sudo /sbin/iptables -A INPUT -j REJECT >> sudo /sbin/iptables -A FORWARD -j REJECT >> ---- iptables rules script --------------- >> >> >> >> ---- additions to sysctl.conf --------------- >> # >> # TCP tuning >> # >> # from >> http://agiletesting.blogspot.com/2009/03/haproxy-and-apache- >> performance-tuni >> ng.html >> net.ipv4.tcp_tw_reuse = 1 >> net.ipv4.ip_local_port_range = 1024 65023 >> net.ipv4.tcp_max_syn_backlog = 10240 >> net.ipv4.tcp_max_tw_buckets = 400000 >> net.ipv4.tcp_max_orphans = 60000 >> net.ipv4.tcp_synack_retries = 3 >> net.core.somaxconn = 40000 >> >> # from >> http://serverfault.com/questions/11106/best-linux-network-tuning-tips >> net.ipv4.route.max_size = 262144 >> net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 18000 >> net.ipv4.neigh.default.gc_thresh1 = 1024 >> net.ipv4.neigh.default.gc_thresh2 = 2048 >> net.ipv4.neigh.default.gc_thresh3 = 4096 >> net.netfilter.nf_conntrack_max = 128000 >> net.netfilter.nf_conntrack_expect_max = 4096 >> >> # additions based on questions to the haproxy mailing list >> # http://www.mail-archive.com/[email protected]/msg01321.html >> net.ipv4.tcp_timestamps = 1 >> net.core.netdev_max_backlog = 40000 >> # these were all lower than the default values already set, so I left >> them >> out >> #net.ipv4.tcp_rmem = 4096 8192 16384 >> #net.ipv4.tcp_wmem = 4096 8192 16384 >> #net.ipv4.tcp_mem = 65536 98304 131072 >> >> ---- additions to sysctl.conf --------------- >> >> >> >>> From: <[email protected]> >>> Date: Wed, 07 Oct 2009 11:24:23 +0100 >>> To: Michael Marano <[email protected]> >>> Cc: <[email protected]> >>> Subject: Re: Kernel tuning recommendations >>> >>> Here is the adjusted IPv4 settings I use on my haproxy box - I picked >>> these up from around the web, and they seem to work for me, not that >>> they are in use on a particularly high volume site currently. >>> >>> Chris >>> >>> net.ipv4.tcp_tw_reuse = 1 >>> net.ipv4.ip_local_port_range = 1024 65023 >>> net.ipv4.tcp_max_syn_backlog = 10240 >>> net.ipv4.tcp_max_tw_buckets = 400000 >>> net.ipv4.tcp_max_orphans = 60000 >>> net.ipv4.tcp_synack_retries = 3 >>> net.ipv4.tcp_max_syn_backlog = 45000 >>> net.ipv4.tcp_timestamps = 1 >>> net.ipv4.tcp_rmem = 4096 8192 16384 >>> net.ipv4.tcp_wmem = 4096 8192 16384 >>> net.ipv4.tcp_mem = 65536 98304 131072 >>> net.core.somaxconn = 40000 >>> net.core.netdev_max_backlog = 40000 >>> >>> >>> >>> Quoting Michael Marano <[email protected]>: >>> >>>> Subsequent load tests proved me wrong. I¹m still getting the >> nf_conntrack >>>> messages. Perhaps I¹ve misconfigigured my iptables rules? >>>> >>>> >>>> # bits of /var/log/messages >>>> >>>> Oct 6 21:58:40 w1 kernel: [3718555.091684] printk: 2 messages >> suppressed. >>>> Oct 6 21:58:40 w1 kernel: [3718555.091705] nf_conntrack: table >> full, >>>> dropping packet. >>>> Oct 6 21:58:41 w1 kernel: [3718290.353966] device eth0 entered >> promiscuous >>>> mode >>>> Oct 6 21:58:43 w1 kernel: [3718558.070993] nf_conntrack: table >> full, >>>> dropping packet. >>>> Oct 6 21:58:44 w1 kernel: [3718559.097679] nf_conntrack: table >> full, >>>> dropping packet. >>>> >>>> >>>> I¹ve got this in a shell script: >>>> >>>> >>>> ---- >>>> #!/bin/sh >>>> >>>> sudo /sbin/iptables -F >>>> sudo /sbin/iptables -A INPUT -i lo -j ACCEPT >>>> sudo /sbin/iptables -A INPUT -i ! lo -d 127.0.0.0/8 -j REJECT >>>> sudo /sbin/iptables -A INPUT -m state --state ESTABLISHED,RELATED -j >> ACCEPT >>>> sudo /sbin/iptables -A OUTPUT -j ACCEPT >>>> >>>> # tell iptables to skip tracking on ports haproxy is monitoring >>>> sudo /sbin/iptables -t raw -A PREROUTING -p tcp --dport 80 -j >> NOTRACK >>>> sudo /sbin/iptables -t raw -A PREROUTING -p tcp --dport 8080 -j >> NOTRACK >>>> >>>> # ... Rules to allow stuff... >>>> >>>> sudo /sbin/iptables -A INPUT -j REJECT >>>> sudo /sbin/iptables -A FORWARD -j REJECT >>>> ------ >>>> >>>> But then when I list my tables, I¹m not seeing anything about the >> NOTRACK >>>> rules. >>>> >>>> ----- >>>> Chain INPUT (policy ACCEPT) >>>> target prot opt source destination >>>> ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 >>>> REJECT all -- 0.0.0.0/0 127.0.0.0/8 reject- >> with >>>> icmp-port-unreachable >>>> ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state >>>> RELATED,ESTABLISHED >>>> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp >> dpt:81 >>>> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp >> dpt:80 >>>> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp >> dpt:443 >>>> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp >> dpt:8080 >>>> >>>> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state >> NEW tcp >>>> dpt:22 >>>> ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmp >> type 8 >>>> REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject- >> with >>>> icmp-port-unreachable >>>> >>>> Chain FORWARD (policy ACCEPT) >>>> target prot opt source destination >>>> REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject- >> with >>>> icmp-port-unreachable >>>> >>>> Chain OUTPUT (policy ACCEPT) >>>> target prot opt source destination >>>> ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 >>>> >>>> Chain RH-Firewall-1-INPUT (0 references) >>>> target prot opt source destination >>>> >>>> ----- >>>> >>>> >>>> >>>> >>>> Michael Marano >>>> >>>> >>>> From: Michael Marano <[email protected]> >>>> Date: Tue, 06 Oct 2009 13:49:02 -0700 >>>> To: Stefan Johansson <[email protected]>, <[email protected]> >>>> Conversation: Kernel tuning recommendations >>>> Subject: Re: Kernel tuning recommendations >>>> >>>> Stefan, >>>> >>>> That seems to have eliminated any log messages in my staging >> environment >>>> under a load test. I think that will do the trick. Thanks for your >> help. >>>> >>>> Any general recommendations for sysctl settings would still be >> appreciated. >>>> This is the first time I¹ve had to tune the kernel settings so any >> guidance >>>> will help. >>>> >>>> Michael Marano >>>> >>>> >>>> From: Stefan Johansson <[email protected]> >>>> Date: Tue, 6 Oct 2009 22:27:49 +0200 >>>> To: Michael Marano <[email protected]>, <[email protected]> >>>> Subject: RE: Kernel tuning recommendations >>>> >>>> iptables -t raw -A PREROUTING -p tcp --dport 80 -j NOTRACK >>>> >>>> >>>> >>> >>> >>> >>> ---------------------------------------------------------------- >>> This message was sent using IMP, the Internet Messaging Program. >>> >> >
