I ended up just building a kernel without conntrack, module or otherwise. I'm sure you could prevent conntrack from loading somehow, but this was easier from my perspective.
Jonah > -----Original Message----- > From: Michael Marano [mailto:[email protected]] > Sent: Wednesday, October 07, 2009 3:03 PM > To: [email protected] > Cc: [email protected]; Mark Kramer > Subject: Re: Kernel tuning recommendations > > I've made a handful of changes based up on Chris and Willy's > suggestions, > which I've included below. This avoids the nf_conntrack errors in the > logs. > > I would like to skip nf_conntrack altogether. I've been digging around > to > try to learn how to do that, but I now admit I don't know how. I can't > just > drop the module, as it's currently in use. > > [mmar...@w1 w1]$ sudo modprobe -n -r nf_conntrack > FATAL: Module nf_conntrack is in use. > > What do I need to change in my iptables rules to pave the way for > removing > this module. Once I've got that straight, how do I then disable the > module. > I'm happy to get an RTFM response if I'm just being stupid. Point me at > the > right M ;) > > Michael Marano > > > ---- iptables rules script --------------- > #!/bin/sh > > sudo /sbin/iptables -F > sudo /sbin/iptables -A INPUT -i lo -j ACCEPT > sudo /sbin/iptables -A INPUT -i ! lo -d 127.0.0.0/8 -j REJECT > sudo /sbin/iptables -A INPUT -m state --state ESTABLISHED,RELATED -j > ACCEPT > sudo /sbin/iptables -A OUTPUT -j ACCEPT > > # don't track incoming or outgoing port 80 > sudo /sbin/iptables -t raw -A PREROUTING -p tcp --dport 80 -j NOTRACK > sudo /sbin/iptables -t raw -A PREROUTING -p tcp --dport 8080 -j NOTRACK > sudo /sbin/iptables -t raw -A PREROUTING -p tcp --dport 81 -j NOTRACK > > # don't track traffic starting from the private ip > sudo /sbin/iptables -t raw -A PREROUTING -p tcp -s 10.176.45.165 -j > NOTRACK > > # these may not actually be useful, but I'm leaving them in. > sudo /sbin/iptables -t raw -A OUTPUT -p tcp --sport 80 -j NOTRACK > sudo /sbin/iptables -t raw -A OUTPUT -p tcp --sport 8080 -j NOTRACK > sudo /sbin/iptables -t raw -A OUTPUT -p tcp --sport 81 -j NOTRACK > > sudo /sbin/iptables -A INPUT -p tcp -m state --state NEW --dport 22 -j > ACCEPT > sudo /sbin/iptables -A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT > sudo /sbin/iptables -A INPUT -j REJECT > sudo /sbin/iptables -A FORWARD -j REJECT > ---- iptables rules script --------------- > > > > ---- additions to sysctl.conf --------------- > # > # TCP tuning > # > # from > http://agiletesting.blogspot.com/2009/03/haproxy-and-apache- > performance-tuni > ng.html > net.ipv4.tcp_tw_reuse = 1 > net.ipv4.ip_local_port_range = 1024 65023 > net.ipv4.tcp_max_syn_backlog = 10240 > net.ipv4.tcp_max_tw_buckets = 400000 > net.ipv4.tcp_max_orphans = 60000 > net.ipv4.tcp_synack_retries = 3 > net.core.somaxconn = 40000 > > # from > http://serverfault.com/questions/11106/best-linux-network-tuning-tips > net.ipv4.route.max_size = 262144 > net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 18000 > net.ipv4.neigh.default.gc_thresh1 = 1024 > net.ipv4.neigh.default.gc_thresh2 = 2048 > net.ipv4.neigh.default.gc_thresh3 = 4096 > net.netfilter.nf_conntrack_max = 128000 > net.netfilter.nf_conntrack_expect_max = 4096 > > # additions based on questions to the haproxy mailing list > # http://www.mail-archive.com/[email protected]/msg01321.html > net.ipv4.tcp_timestamps = 1 > net.core.netdev_max_backlog = 40000 > # these were all lower than the default values already set, so I left > them > out > #net.ipv4.tcp_rmem = 4096 8192 16384 > #net.ipv4.tcp_wmem = 4096 8192 16384 > #net.ipv4.tcp_mem = 65536 98304 131072 > > ---- additions to sysctl.conf --------------- > > > > > From: <[email protected]> > > Date: Wed, 07 Oct 2009 11:24:23 +0100 > > To: Michael Marano <[email protected]> > > Cc: <[email protected]> > > Subject: Re: Kernel tuning recommendations > > > > Here is the adjusted IPv4 settings I use on my haproxy box - I picked > > these up from around the web, and they seem to work for me, not that > > they are in use on a particularly high volume site currently. > > > > Chris > > > > net.ipv4.tcp_tw_reuse = 1 > > net.ipv4.ip_local_port_range = 1024 65023 > > net.ipv4.tcp_max_syn_backlog = 10240 > > net.ipv4.tcp_max_tw_buckets = 400000 > > net.ipv4.tcp_max_orphans = 60000 > > net.ipv4.tcp_synack_retries = 3 > > net.ipv4.tcp_max_syn_backlog = 45000 > > net.ipv4.tcp_timestamps = 1 > > net.ipv4.tcp_rmem = 4096 8192 16384 > > net.ipv4.tcp_wmem = 4096 8192 16384 > > net.ipv4.tcp_mem = 65536 98304 131072 > > net.core.somaxconn = 40000 > > net.core.netdev_max_backlog = 40000 > > > > > > > > Quoting Michael Marano <[email protected]>: > > > >> Subsequent load tests proved me wrong. I¹m still getting the > nf_conntrack > >> messages. Perhaps I¹ve misconfigigured my iptables rules? > >> > >> > >> # bits of /var/log/messages > >> > >> Oct 6 21:58:40 w1 kernel: [3718555.091684] printk: 2 messages > suppressed. > >> Oct 6 21:58:40 w1 kernel: [3718555.091705] nf_conntrack: table > full, > >> dropping packet. > >> Oct 6 21:58:41 w1 kernel: [3718290.353966] device eth0 entered > promiscuous > >> mode > >> Oct 6 21:58:43 w1 kernel: [3718558.070993] nf_conntrack: table > full, > >> dropping packet. > >> Oct 6 21:58:44 w1 kernel: [3718559.097679] nf_conntrack: table > full, > >> dropping packet. > >> > >> > >> I¹ve got this in a shell script: > >> > >> > >> ---- > >> #!/bin/sh > >> > >> sudo /sbin/iptables -F > >> sudo /sbin/iptables -A INPUT -i lo -j ACCEPT > >> sudo /sbin/iptables -A INPUT -i ! lo -d 127.0.0.0/8 -j REJECT > >> sudo /sbin/iptables -A INPUT -m state --state ESTABLISHED,RELATED -j > ACCEPT > >> sudo /sbin/iptables -A OUTPUT -j ACCEPT > >> > >> # tell iptables to skip tracking on ports haproxy is monitoring > >> sudo /sbin/iptables -t raw -A PREROUTING -p tcp --dport 80 -j > NOTRACK > >> sudo /sbin/iptables -t raw -A PREROUTING -p tcp --dport 8080 -j > NOTRACK > >> > >> # ... Rules to allow stuff... > >> > >> sudo /sbin/iptables -A INPUT -j REJECT > >> sudo /sbin/iptables -A FORWARD -j REJECT > >> ------ > >> > >> But then when I list my tables, I¹m not seeing anything about the > NOTRACK > >> rules. > >> > >> ----- > >> Chain INPUT (policy ACCEPT) > >> target prot opt source destination > >> ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 > >> REJECT all -- 0.0.0.0/0 127.0.0.0/8 reject- > with > >> icmp-port-unreachable > >> ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state > >> RELATED,ESTABLISHED > >> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp > dpt:81 > >> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp > dpt:80 > >> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp > dpt:443 > >> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp > dpt:8080 > >> > >> ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state > NEW tcp > >> dpt:22 > >> ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmp > type 8 > >> REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject- > with > >> icmp-port-unreachable > >> > >> Chain FORWARD (policy ACCEPT) > >> target prot opt source destination > >> REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject- > with > >> icmp-port-unreachable > >> > >> Chain OUTPUT (policy ACCEPT) > >> target prot opt source destination > >> ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 > >> > >> Chain RH-Firewall-1-INPUT (0 references) > >> target prot opt source destination > >> > >> ----- > >> > >> > >> > >> > >> Michael Marano > >> > >> > >> From: Michael Marano <[email protected]> > >> Date: Tue, 06 Oct 2009 13:49:02 -0700 > >> To: Stefan Johansson <[email protected]>, <[email protected]> > >> Conversation: Kernel tuning recommendations > >> Subject: Re: Kernel tuning recommendations > >> > >> Stefan, > >> > >> That seems to have eliminated any log messages in my staging > environment > >> under a load test. I think that will do the trick. Thanks for your > help. > >> > >> Any general recommendations for sysctl settings would still be > appreciated. > >> This is the first time I¹ve had to tune the kernel settings so any > guidance > >> will help. > >> > >> Michael Marano > >> > >> > >> From: Stefan Johansson <[email protected]> > >> Date: Tue, 6 Oct 2009 22:27:49 +0200 > >> To: Michael Marano <[email protected]>, <[email protected]> > >> Subject: RE: Kernel tuning recommendations > >> > >> iptables -t raw -A PREROUTING -p tcp --dport 80 -j NOTRACK > >> > >> > >> > > > > > > > > ---------------------------------------------------------------- > > This message was sent using IMP, the Internet Messaging Program. > > >
