Hi Chris,
Why using Solaris 7 and not Solaris 8 or 9 ?
Why using 32 bits and not 64 bits ? (Ipfilter work fine on 64bits)
Your traffic is around 2-2.5M hits/day,
What is the number in Mbs/s ?
Do you use a 10 or 100 or 1000 interface ? (Fiber or Ether)
hme/qfe ? others ?
# netstat -ni ???? (collisions/Errors)
If you disable Ipfilter on Solaris and of course active packet forwarding on
solaris,
do you have better perf ?
If you active Ipfilter and you don't use "keep state" rules, (stateless
rules)
Do you have better perf ?
Does it reduce the number of active states in 'ipfstat -s' ?
Your 'Ipfstat -s' indicate 7000-8000 rules,
this is not high value ...
What rules do you use ? (ipf.conf)
How many rules do you have ?
Do you use group/head in your conf ?
Do you use quick in your rules ?
Do you use 'keep frags' in your rules ?
Do you use Antispoof / ttl / tcp_option flags ... ?
Do you use nat ?
Do you have ipmon log message ??? (accept or block)
Do you monitor Solaris with SunPerf ?
If you start top, Do you see processes that are consumming cpu ?
run this: (and send me results) ipfstat -nhio
Regards.
Chris Lamb wrote:
> Hey, all.
>
> The situation:
> As is always the case, it seems, I have to roll out an entire new server
> farm with the network up, and with beta sites sending in data 24x7, and
> without the possibility of simply taking some downtime to reconfigure and
> rebuild my network, because heavens, That Would Be Too Easy. So we're
> changing the tires on the bus while whipping down the highway at 75mph
> with no brakes. I am not disgruntled...
>
> The questions:
> Is anyone here running ipfilter on a very high-volume web server farm
> behind a load balancer? (By volume I'm talking about lots of very short
> transactions, about 17-22 packets on average; the goal is 1B hits/week per
> server.) What sort of tuning have you done to the OS and to ipfilter
> itself? Are you using "keep state" on port 80?
>
> The setup:
> For testing, I'm currently running ipfilter 3.4.31 on a 4-way Sun 420R,
> 4GB, Solaris 7 (32-bit). Have tweaked various things with ndd and pushed
> up tcp_conn_hash_size in /etc/system, plus reduced fr_tcpidletimeout from
> 5 days to 24 hours, but am considering cutting that back more.
>
> IPSTATE_SIZE 64997
> IPSTATE_MAX 45499
>
> and am using "keep state" for now. With traffic around 2-2.5M hits/day
> during this test phase:
>
> # ipf -V
> ipf: IP Filter: v3.4.31 (336)
> Kernel: IP Filter: v3.4.31
> Running: yes
> Log Flags: 0 = none set
> Default: block all, Logging: available
> Active list: 1
>
> # ipfstat -s
> IP states added:
> 10453005 TCP
> 1744 UDP
> 0 ICMP
> 206468007 hits
> 73388876 misses
> 0 maximum
> 0 no memory
> 7246 bkts in use
> 7719 active
> 1743 expired
> 10445287 closed
>
> # netstat -naf inet | awk '/\.80/ {print $NF}' | sort | uniq -c
> 6 CLOSE_WAIT
> 156 ESTABLISHED
> 39 FIN_WAIT_1
> 111 FIN_WAIT_2
> 9 LAST_ACK
> 2 LISTEN
> 18 SYN_RCVD
> 488 TIME_WAIT
>
> Without analyzing things in depth (oh, for the "luxury" of time to be able
> to actually do a proper job of this) it seems to me that scaling up the
> traffic without turning down various timeouts is going to overwhelm the
> state table. In some very simpleminded load tests last year this box
> without ipfilter installed and no tuning (literally a clean OS install and
> no tweaks) was easily handling over 2,000 hits/second, or about a billion
> hits a week, which is what the Management quickly siezed upon as the basis
> for the hardware budget. Hmmm.
>
> So with two identical machines (at first) behind a Cisco LocalDirector, I
> expect we can handle the traffic, but I'm wondering just how much pain is
> going to be involved. Has anyone configured ipfilter to handle connection
> rates that high? If you've bashed your head against this particular wall
> and are willing to share your wisdom and war stories, I'd be mighty
> grateful. :-)
>
> I'm hoping that I can get some time this week to actually - gasp! - do
> some further testing before going live with the new production farm, so
> I'll report back if I find anything interesting. Comments and feedback
> on- or off-list welcomed!
>
> Cheers,
>
> -- Chris
