Hello,
On Tue, Jul 13, 2010 at 12:56:21PM +0000, Angelo Höngens wrote:
> Hey, does anyone have an idea how iptables impacts network performance? (on
> CentOS 5.5 x64 for example).
There's no easy answer, there's rules processing and session processing.
See below.
> I've got haproxy running on quite some FreeBSD machines for quite a while
> now, and I'm very happy with it. We have quite some different setups
> (directly on the net, behind cisco firewall in dmz, with host firewall,
> without, etc). Now we're slowly moving from FreeBSD to CentOS, and by default
> iptables is enabled.
>
> On our FreeBSD machines that are directly connected to the net, we have a
> public interface with services only listening on port 80, and an internal
> interface for stats access, ssh and snmp. But we have some new machines on
> which we only want to use a single public interface. We'd use iptables to
> allow only trusted ip's to connect to management services.
>
> What are your real-life experiences? Do you have iptables enabled on your
> balancers? Normally I would do stresstests, but somehow my stresstests never
> simulate real-world behavior with a mix of tens of thousands of slow and fast
> clients, etc.
Iptables's rules processing is not much expensive. On todays machines, you can
evaluate a few million rules per second. So if you're running in stateless mode,
that means as many packet*rules per second. Let's say you have 10 rules in
average and you're running short sessions of 10 packets each, that leaves you
with a few tens of thousands of connections per second.
Once you're adding rules, you risk seeing those numbers suddenly drop for
various
reasons (rules that don't fit in CPU cache anymore, and risk of saturating a CPU
at a given packet rate). At that point you'll realize that it would be better to
evaluate your rules only on the first packet of each session and run stateful
using ip_conntrack (=nf_conntrack). This works very well provided you perform a
good deal of tuning that is not done by default. Among the things that need to
be tuned :
- max number of sessions : set it very very high. Around 100 times the max
connection rate (conn/s) for a normal firewall, or twice that on a proxy.
This means that at 10k conns/s, you need 2M sessions on a proxy.
- hash size : set it a few times below the max number of sessions. I tend to
use sessions/16 to sessions/4. The larger the hash size, the shorter the
session lookups, but the larger the memory. If the memory footprint is too
high, it will constantly trash your CPU caches, possibly resulting in a
lower performance.
- timewait timeout : shorten it to about 25-30 seconds, this will save you
a huge number of entries
- established timeout : check that it's not at the old default of 5 days ;-)
- ensure that you don't send the logs to the poor old sysklogd, install
syslog-ng and make it read /proc/kmsg itself !
Then write your rules to ensure that the most common packets are matched very
early. That means :
1) ESTABLISHED,RELATED => ACCEPT
2) send TCP SYN to a chain
3) check the rest
4) the SYN chain must be sorted from the most commonly used service to the
least common.
By applying these hints, you should be able to easily reach numbers as high
as 20k connections per second in a firewall mode, or 10k connections/s in
proxy mode (since the connections have to be created on both sides of the
proxy).
Iptables can be fast (even very fast) but since by default it performs very
poorly, I regularly recommend users to check for it and possibly disable it
if not needed. But with proper tuning it can be fast. That said, you must
keep in mind that by enabling iptables+conntrack on a machine where you
already have a proxy, you should expect to cust the max performance by 2 to
2.5x, which is not always acceptable in some environments!
Most often, there really is zero good reason for enabling a firewall on a
load balancer, because a high performance load balancer should not be
running other software, so only the ports which are configured will be
open and the firewall will be useless. But I know that in some situations,
you have to get that box to route/NAT etc at the same time as it proxies.
> By the way, some of our balancers do > 100 Mbit and > 2000 req/s by the way.
This goal is moderately low, with a quick tuning you can already reach it.
I've set up quite a few netfilter-based firewalls, some of which run at
about 10k connections/s on very cheap hardware and there's still some
margin. When the tuning is fine, the real problems only are logging and
rules maintenance.
Regards,
Willy
