Hey, all.
The situation:
As is always the case, it seems, I have to roll out an entire new server
farm with the network up, and with beta sites sending in data 24x7, and
without the possibility of simply taking some downtime to reconfigure and
rebuild my network, because heavens, That Would Be Too Easy. So we're
changing the tires on the bus while whipping down the highway at 75mph
with no brakes. I am not disgruntled...
The questions:
Is anyone here running ipfilter on a very high-volume web server farm
behind a load balancer? (By volume I'm talking about lots of very short
transactions, about 17-22 packets on average; the goal is 1B hits/week per
server.) What sort of tuning have you done to the OS and to ipfilter
itself? Are you using "keep state" on port 80?
The setup:
For testing, I'm currently running ipfilter 3.4.31 on a 4-way Sun 420R,
4GB, Solaris 7 (32-bit). Have tweaked various things with ndd and pushed
up tcp_conn_hash_size in /etc/system, plus reduced fr_tcpidletimeout from
5 days to 24 hours, but am considering cutting that back more.
IPSTATE_SIZE 64997
IPSTATE_MAX 45499
and am using "keep state" for now. With traffic around 2-2.5M hits/day
during this test phase:
# ipf -V
ipf: IP Filter: v3.4.31 (336)
Kernel: IP Filter: v3.4.31
Running: yes
Log Flags: 0 = none set
Default: block all, Logging: available
Active list: 1
# ipfstat -s
IP states added:
10453005 TCP
1744 UDP
0 ICMP
206468007 hits
73388876 misses
0 maximum
0 no memory
7246 bkts in use
7719 active
1743 expired
10445287 closed
# netstat -naf inet | awk '/\.80/ {print $NF}' | sort | uniq -c
6 CLOSE_WAIT
156 ESTABLISHED
39 FIN_WAIT_1
111 FIN_WAIT_2
9 LAST_ACK
2 LISTEN
18 SYN_RCVD
488 TIME_WAIT
Without analyzing things in depth (oh, for the "luxury" of time to be able
to actually do a proper job of this) it seems to me that scaling up the
traffic without turning down various timeouts is going to overwhelm the
state table. In some very simpleminded load tests last year this box
without ipfilter installed and no tuning (literally a clean OS install and
no tweaks) was easily handling over 2,000 hits/second, or about a billion
hits a week, which is what the Management quickly siezed upon as the basis
for the hardware budget. Hmmm.
So with two identical machines (at first) behind a Cisco LocalDirector, I
expect we can handle the traffic, but I'm wondering just how much pain is
going to be involved. Has anyone configured ipfilter to handle connection
rates that high? If you've bashed your head against this particular wall
and are willing to share your wisdom and war stories, I'd be mighty
grateful. :-)
I'm hoping that I can get some time this week to actually - gasp! - do
some further testing before going live with the new production farm, so
I'll report back if I find anything interesting. Comments and feedback
on- or off-list welcomed!
Cheers,
-- Chris
