I think you are just running a stock Centos-Xen kernel, so a recompile
shouldn't be an issue, but its possibly overkill, depending on the
level of tuning you want. I'm running haproxy on a dedicated centos 5
box, and if you strip out the appropriate rules as directed by Willy,
then you should be able to remove the module and see an improvement in
performance.
Chris
Quoting Michael Marano <[email protected]>:
I'm running on Rackspace Cloud. I don't think I can run a custom kernel out
there.
Michael Marano
--
Senior Manager of Web Development
Future US, Inc.
desk: 650-238-2530
cell: 650-580-2132
twitter: @mmarano
aim: michaelvicmarano
skype: mmarano
From: Jonah Horowitz <[email protected]>
Date: Wed, 7 Oct 2009 15:18:29 -0700
To: Michael Marano <[email protected]>, <[email protected]>
Cc: <[email protected]>, Mark Kramer <[email protected]>
Subject: RE: Kernel tuning recommendations
I ended up just building a kernel without conntrack, module or
otherwise. I'm
sure you could prevent conntrack from loading somehow, but this was easier
from my perspective.
Jonah
-----Original Message-----
From: Michael Marano [mailto:[email protected]]
Sent: Wednesday, October 07, 2009 3:03 PM
To: [email protected]
Cc: [email protected]; Mark Kramer
Subject: Re: Kernel tuning recommendations
I've made a handful of changes based up on Chris and Willy's
suggestions,
which I've included below. This avoids the nf_conntrack errors in the
logs.
I would like to skip nf_conntrack altogether. I've been digging around
to
try to learn how to do that, but I now admit I don't know how. I can't
just
drop the module, as it's currently in use.
[mmar...@w1 w1]$ sudo modprobe -n -r nf_conntrack
FATAL: Module nf_conntrack is in use.
What do I need to change in my iptables rules to pave the way for
removing
this module. Once I've got that straight, how do I then disable the
module.
I'm happy to get an RTFM response if I'm just being stupid. Point me at
the
right M ;)
Michael Marano
---- iptables rules script ---------------
#!/bin/sh
sudo /sbin/iptables -F
sudo /sbin/iptables -A INPUT -i lo -j ACCEPT
sudo /sbin/iptables -A INPUT -i ! lo -d 127.0.0.0/8 -j REJECT
sudo /sbin/iptables -A INPUT -m state --state ESTABLISHED,RELATED -j
ACCEPT
sudo /sbin/iptables -A OUTPUT -j ACCEPT
# don't track incoming or outgoing port 80
sudo /sbin/iptables -t raw -A PREROUTING -p tcp --dport 80 -j NOTRACK
sudo /sbin/iptables -t raw -A PREROUTING -p tcp --dport 8080 -j NOTRACK
sudo /sbin/iptables -t raw -A PREROUTING -p tcp --dport 81 -j NOTRACK
# don't track traffic starting from the private ip
sudo /sbin/iptables -t raw -A PREROUTING -p tcp -s 10.176.45.165 -j
NOTRACK
# these may not actually be useful, but I'm leaving them in.
sudo /sbin/iptables -t raw -A OUTPUT -p tcp --sport 80 -j NOTRACK
sudo /sbin/iptables -t raw -A OUTPUT -p tcp --sport 8080 -j NOTRACK
sudo /sbin/iptables -t raw -A OUTPUT -p tcp --sport 81 -j NOTRACK
sudo /sbin/iptables -A INPUT -p tcp -m state --state NEW --dport 22 -j
ACCEPT
sudo /sbin/iptables -A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
sudo /sbin/iptables -A INPUT -j REJECT
sudo /sbin/iptables -A FORWARD -j REJECT
---- iptables rules script ---------------
---- additions to sysctl.conf ---------------
#
# TCP tuning
#
# from
http://agiletesting.blogspot.com/2009/03/haproxy-and-apache-
performance-tuni
ng.html
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 1024 65023
net.ipv4.tcp_max_syn_backlog = 10240
net.ipv4.tcp_max_tw_buckets = 400000
net.ipv4.tcp_max_orphans = 60000
net.ipv4.tcp_synack_retries = 3
net.core.somaxconn = 40000
# from
http://serverfault.com/questions/11106/best-linux-network-tuning-tips
net.ipv4.route.max_size = 262144
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 18000
net.ipv4.neigh.default.gc_thresh1 = 1024
net.ipv4.neigh.default.gc_thresh2 = 2048
net.ipv4.neigh.default.gc_thresh3 = 4096
net.netfilter.nf_conntrack_max = 128000
net.netfilter.nf_conntrack_expect_max = 4096
# additions based on questions to the haproxy mailing list
# http://www.mail-archive.com/[email protected]/msg01321.html
net.ipv4.tcp_timestamps = 1
net.core.netdev_max_backlog = 40000
# these were all lower than the default values already set, so I left
them
out
#net.ipv4.tcp_rmem = 4096 8192 16384
#net.ipv4.tcp_wmem = 4096 8192 16384
#net.ipv4.tcp_mem = 65536 98304 131072
---- additions to sysctl.conf ---------------
From: <[email protected]>
Date: Wed, 07 Oct 2009 11:24:23 +0100
To: Michael Marano <[email protected]>
Cc: <[email protected]>
Subject: Re: Kernel tuning recommendations
Here is the adjusted IPv4 settings I use on my haproxy box - I picked
these up from around the web, and they seem to work for me, not that
they are in use on a particularly high volume site currently.
Chris
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 1024 65023
net.ipv4.tcp_max_syn_backlog = 10240
net.ipv4.tcp_max_tw_buckets = 400000
net.ipv4.tcp_max_orphans = 60000
net.ipv4.tcp_synack_retries = 3
net.ipv4.tcp_max_syn_backlog = 45000
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_rmem = 4096 8192 16384
net.ipv4.tcp_wmem = 4096 8192 16384
net.ipv4.tcp_mem = 65536 98304 131072
net.core.somaxconn = 40000
net.core.netdev_max_backlog = 40000
Quoting Michael Marano <[email protected]>:
Subsequent load tests proved me wrong. I¹m still getting the
nf_conntrack
messages. Perhaps I¹ve misconfigigured my iptables rules?
# bits of /var/log/messages
Oct 6 21:58:40 w1 kernel: [3718555.091684] printk: 2 messages
suppressed.
Oct 6 21:58:40 w1 kernel: [3718555.091705] nf_conntrack: table
full,
dropping packet.
Oct 6 21:58:41 w1 kernel: [3718290.353966] device eth0 entered
promiscuous
mode
Oct 6 21:58:43 w1 kernel: [3718558.070993] nf_conntrack: table
full,
dropping packet.
Oct 6 21:58:44 w1 kernel: [3718559.097679] nf_conntrack: table
full,
dropping packet.
I¹ve got this in a shell script:
----
#!/bin/sh
sudo /sbin/iptables -F
sudo /sbin/iptables -A INPUT -i lo -j ACCEPT
sudo /sbin/iptables -A INPUT -i ! lo -d 127.0.0.0/8 -j REJECT
sudo /sbin/iptables -A INPUT -m state --state ESTABLISHED,RELATED -j
ACCEPT
sudo /sbin/iptables -A OUTPUT -j ACCEPT
# tell iptables to skip tracking on ports haproxy is monitoring
sudo /sbin/iptables -t raw -A PREROUTING -p tcp --dport 80 -j
NOTRACK
sudo /sbin/iptables -t raw -A PREROUTING -p tcp --dport 8080 -j
NOTRACK
# ... Rules to allow stuff...
sudo /sbin/iptables -A INPUT -j REJECT
sudo /sbin/iptables -A FORWARD -j REJECT
------
But then when I list my tables, I¹m not seeing anything about the
NOTRACK
rules.
-----
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
REJECT all -- 0.0.0.0/0 127.0.0.0/8 reject-
with
icmp-port-unreachable
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state
RELATED,ESTABLISHED
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp
dpt:81
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp
dpt:80
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp
dpt:443
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp
dpt:8080
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 state
NEW tcp
dpt:22
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0 icmp
type 8
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-
with
icmp-port-unreachable
Chain FORWARD (policy ACCEPT)
target prot opt source destination
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-
with
icmp-port-unreachable
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
Chain RH-Firewall-1-INPUT (0 references)
target prot opt source destination
-----
Michael Marano
From: Michael Marano <[email protected]>
Date: Tue, 06 Oct 2009 13:49:02 -0700
To: Stefan Johansson <[email protected]>, <[email protected]>
Conversation: Kernel tuning recommendations
Subject: Re: Kernel tuning recommendations
Stefan,
That seems to have eliminated any log messages in my staging
environment
under a load test. I think that will do the trick. Thanks for your
help.
Any general recommendations for sysctl settings would still be
appreciated.
This is the first time I¹ve had to tune the kernel settings so any
guidance
will help.
Michael Marano
From: Stefan Johansson <[email protected]>
Date: Tue, 6 Oct 2009 22:27:49 +0200
To: Michael Marano <[email protected]>, <[email protected]>
Subject: RE: Kernel tuning recommendations
iptables -t raw -A PREROUTING -p tcp --dport 80 -j NOTRACK
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
