Okay, I've built a FreeBSD 4.8 box (4.8-RELEASE) to be a firewall for a
very large internet site. The box is a Celeron 466 (Pentium 3 based)
with 256MB of ram (yeah I know, thrifty aren't I?). I'm using a very
complex setup (as this was built to replace a PIX, and therefore it was
configured to be a direct drop in replacement (hence the extremely
complex configuration)), and have it working fine short of 1 problem.
Every 12 hours now I experience a kernel panic. I built a debug kernel
and here is the backtrace (the relevant part at least) of the crash.
#16 0xc0220a5f in malloc (size=140, type=0xc04410a0, flags=1)
at /usr/src/sys/kern/kern_malloc.c:243
#17 0xc01521b2 in nat_new (fin=0xc041e7bc, ip=0xc0f22820, np=0xc16a9200,
natsave=0x0, flags=1,
direction=1) at /usr/src/sys/contrib/ipfilter/netinet/ip_nat.c:1175
#18 0xc0153fee in ip_natin (ip=0xc0f22820, fin=0xc041e7bc)
at /usr/src/sys/contrib/ipfilter/netinet/ip_nat.c:2659
#19 0xc014d144 in fr_check (ip=0xc0f22820, hlen=20, ifp=0xc15bce00,
out=0, mp=0xc041e858)
at /usr/src/sys/contrib/ipfilter/netinet/fil.c:1017
#20 0xc027ca2e in ip_input (m=0xc0e3b800) at
/usr/src/sys/netinet/ip_input.c:455
#21 0xc027cfbb in ipintr () at /usr/src/sys/netinet/ip_input.c:948
#22 0xc038bf19 in swi_net_next ()
Everything above 16 is the kernel handling the trap and dumping to the
swap device. This is happening approximately every 12 hours (within
10-15 minutes to be exact), and is due to an attempt to access 0x1
(that's what the crash is, it's a line in the kernel code that says va =
kbp->kp_next and kbp->kp_next is 0x1 (kbp looks like a valid pointer at
least)). Here is what I see.
(kgdb)
$4 = (struct kmembuckets *) 0xc0481140
And clearly attempting to access it causes an error as well, as gdb
can't access 0x1 either.
Now for the configuration (I said this was an EXTREMELY busy box
right?). So this is the firewall and nat (NAT is a strong word, it's
primarily almost all 1 to 1 mappings (using BIMAP) for a large number of
VIP's on a load balancer). It averages 60Mbit/sec with peaks close to
90Mbit/sec.
The box has 2 Intel 10/100 cards in it (an external and an internal
interface). The kernel is built with the following changes from GENERIC.
28c28
< #makeoptions DEBUG=-g #Build kernel with gdb(1) debug symbols
---
> makeoptions DEBUG=-g #Build kernel with gdb(1) debug symbols
266a267,271
>
> # Changes
> options IPFILTER #ipfilter support
> options DEVICE_POLLING
> options HZ=1000
The system has also been tuned to handle the HUGE load this box is
getting. Here's the rest of the settings. My real ip subnet has been
changed to 1.2.3 to protect me from any of you curious people.
rc.conf
-------
defaultrouter="192.168.247.1"
gateway_enable="YES"
hostname="pix-replacement.you.wish"
ifconfig_fxp0="inet 1.2.3.1 netmask 255.255.255.0"
ifconfig_fxp0_alias0="inet 192.168.247.2 netmask 255.255.255.0"
ifconfig_fxp1="inet 10.10.2.141 netmask 255.255.255.0"
kern_securelevel_enable="NO"
linux_enable="YES"
nfs_reserved_port_only="YES"
sendmail_enable="NO"
sshd_enable="YES"
usbd_enable="NO"
named_enable="NO"
ipnat_enable="YES"
ipfilter_enable="YES"
dumpdev="/dev/ad0s1b"
ipf.rules
---------
#
# allow packets out
#
pass out quick on fxp0 proto tcp from 10.10.2.231 port = 80 to any
pass out quick on fxp0 proto tcp from 10.10.2.156 port = 80 to any
pass out quick on fxp0 proto tcp from 10.10.2.157 port = 80 to any
pass out quick on fxp0 proto tcp from 10.10.2.158 port = 80 to any
pass out quick on fxp0 proto tcp from 10.10.2.159 port = 80 to any
pass out quick on fxp0 proto tcp from 10.10.2.224 port = 80 to any
pass out quick on fxp0 proto tcp from 10.10.2.225 port = 80 to any
pass out quick on fxp0 proto tcp from 10.10.2.232 port = 80 to any
pass out quick on fxp0 proto tcp from 10.10.2.234 port = 80 to any
pass out quick on fxp0 proto tcp from 10.10.2.221 port = 80 to any
pass out quick on fxp0 proto tcp from 10.10.2.222 port = 80 to any
pass out quick on fxp0 proto tcp from 10.10.2.223 port = 80 to any
pass out quick on fxp0 proto tcp from 10.10.2.202 port = 80 to any
pass out quick on fxp0 proto tcp from 10.10.2.201 port = 80 to any
pass out quick on fxp0 proto tcp from 10.10.2.203 port = 80 to any
pass out quick on fxp0 proto tcp from 10.10.2.204 port = 80 to any
pass out quick on fxp0 proto tcp from 10.10.2.205 port = 80 to any
pass out quick on fxp0 proto tcp from 10.10.2.233 port = 80 to any
pass out quick on fxp0 proto tcp from 10.10.2.156 port = 443 to any
pass out quick on fxp0 proto tcp from 10.10.2.224 port = 443 to any
pass out quick on fxp0 proto tcp from 10.10.2.221 port = 443 to any
pass out quick on fxp0 proto tcp from 10.10.2.236 port = 21 to any
pass out quick on fxp0 proto tcp from 10.10.2.236 port = 20 to any
pass out quick on fxp0 proto tcp from 10.10.2.144 port = 22 to any
pass out quick on fxp0 proto tcp from 10.10.2.146 port = 22 to any
pass out quick on fxp0 proto tcp from 10.10.2.221 port = 22 to any
pass out quick on fxp0 proto tcp from 10.10.2.201 port = 22 to any
pass out quick on fxp0 proto tcp from 10.10.2.201 port = 53 to any
pass out quick on fxp0 proto udp from 10.10.2.201 port = 53 to any
pass out quick on fxp0 proto tcp from 10.10.2.210 port = 8080 to any
pass out quick on fxp0 proto tcp from 10.10.2.210 port = 33333 to any
pass out quick on fxp0 proto tcp from 10.10.2.210 port = 65001 to any
pass out quick on fxp0 proto tcp from 10.10.2.143 port = 8888 to any
pass out quick on fxp0 proto tcp from 10.10.2.143 port = 6188 to any
pass out quick on fxp0 proto tcp from 10.10.2.143 port = 6288 to any
#
# allow packets out with state
#
pass out quick on fxp0 proto tcp from 10.10.2.0/24 to any keep state
pass out quick on fxp0 proto udp from 10.10.2.0/24 to any keep state
pass out quick on fxp0 proto icmp from 10.10.2.0/24 to any keep state
pass out quick on fxp0 proto tcp from 1.2.3.0/24 to any keep state
pass out quick on fxp0 proto udp from 1.2.3.0/24 to any keep state
pass out quick on fxp0 proto icmp from 1.2.3.0/24 to any keep state
#
# block everything else out
#
block out quick on fxp0 all
#
# allow limited access
#
pass in quick on fxp0 proto icmp from any to any
pass in quick on fxp0 proto tcp from any to 10.10.2.231 port = 80
pass in quick on fxp0 proto tcp from any to 10.10.2.156 port = 80
pass in quick on fxp0 proto tcp from any to 10.10.2.157 port = 80
pass in quick on fxp0 proto tcp from any to 10.10.2.158 port = 80
pass in quick on fxp0 proto tcp from any to 10.10.2.159 port = 80
pass in quick on fxp0 proto tcp from any to 10.10.2.224 port = 80
pass in quick on fxp0 proto tcp from any to 10.10.2.225 port = 80
pass in quick on fxp0 proto tcp from any to 10.10.2.232 port = 80
pass in quick on fxp0 proto tcp from any to 10.10.2.234 port = 80
pass in quick on fxp0 proto tcp from any to 10.10.2.221 port = 80
pass in quick on fxp0 proto tcp from any to 10.10.2.222 port = 80
pass in quick on fxp0 proto tcp from any to 10.10.2.223 port = 80
pass in quick on fxp0 proto tcp from any to 10.10.2.202 port = 80
pass in quick on fxp0 proto tcp from any to 10.10.2.201 port = 80
pass in quick on fxp0 proto tcp from any to 10.10.2.203 port = 80
pass in quick on fxp0 proto tcp from any to 10.10.2.204 port = 80
pass in quick on fxp0 proto tcp from any to 10.10.2.205 port = 80
pass in quick on fxp0 proto tcp from any to 10.10.2.233 port = 80
pass in quick on fxp0 proto tcp from any to 10.10.2.156 port = 443
pass in quick on fxp0 proto tcp from any to 10.10.2.224 port = 443
pass in quick on fxp0 proto tcp from any to 10.10.2.221 port = 443
pass in quick on fxp0 proto tcp from any to 10.10.2.236 port = 21
pass in quick on fxp0 proto tcp from any to 10.10.2.236 port = 20
pass in quick on fxp0 proto tcp from any to 10.10.2.144 port = 22
pass in quick on fxp0 proto tcp from any to 10.10.2.146 port = 22
pass in quick on fxp0 proto tcp from any to 10.10.2.221 port = 22
pass in quick on fxp0 proto tcp from any to 10.10.2.201 port = 22
pass in quick on fxp0 proto tcp from any to 10.10.2.201 port = 53
pass in quick on fxp0 proto udp from any to 10.10.2.201 port = 53
pass in quick on fxp0 proto tcp from any to 10.10.2.210 port = 8080
pass in quick on fxp0 proto tcp from any to 10.10.2.207 port >= 49152
pass in quick on fxp0 proto tcp from any to 10.10.2.208 port >= 49152
pass in quick on fxp0 proto tcp from 1.2.3.0/24 to 10.10.2.210 port = 33333
pass in quick on fxp0 proto tcp from 1.2.3.0/24 to 10.10.2.210 port = 65001
pass in quick on fxp0 proto tcp from 1.2.3.70 to 10.10.2.143 port = 8888
pass in quick on fxp0 proto tcp from 1.2.3.0/24 to 10.10.2.143 port = 6188
pass in quick on fxp0 proto tcp from 1.2.3.0/24 to 10.10.2.143 port = 6288
pass in quick on fxp0 proto tcp from any to 1.2.3.1 port = 22
#
# block everything else inbound
#
block return-rst in quick on fxp0 proto tcp from any to any
block return-icmp-as-dest(port-unr) in quick on fxp0 proto udp from any to any
block in quick on fxp0 all
#
# allow localhost to work
#
pass in quick on lo0 all
pass out quick on lo0 all
ipnat.rules
-----------
map fxp0 10.10.2.0/24 -> 1.2.3.70/32 portmap tcp/udp 1025:65000
map fxp0 10.10.2.0/24 -> 1.2.3.70/32
bimap fxp0 10.10.2.231/32 -> 1.2.3.231/32 age 1
bimap fxp0 10.10.2.156/32 -> 1.2.3.156/32 age 1
bimap fxp0 10.10.2.157/32 -> 1.2.3.157/32 age 1
bimap fxp0 10.10.2.158/32 -> 1.2.3.158/32 age 1
bimap fxp0 10.10.2.159/32 -> 1.2.3.159/32 age 1
bimap fxp0 10.10.2.224/32 -> 1.2.3.224/32 age 1
bimap fxp0 10.10.2.225/32 -> 1.2.3.225/32 age 1
bimap fxp0 10.10.2.232/32 -> 1.2.3.232/32 age 1
bimap fxp0 10.10.2.234/32 -> 1.2.3.234/32 age 1
bimap fxp0 10.10.2.236/32 -> 1.2.3.236/32 age 1
bimap fxp0 10.10.2.146/32 -> 1.2.3.146/32 age 1
bimap fxp0 10.10.2.144/32 -> 1.2.3.229/32 age 1
bimap fxp0 10.10.2.221/32 -> 1.2.3.221/32 age 1
bimap fxp0 10.10.2.222/32 -> 1.2.3.222/32 age 1
bimap fxp0 10.10.2.223/32 -> 1.2.3.223/32 age 1
bimap fxp0 10.10.2.202/32 -> 1.2.3.202/32 age 1
bimap fxp0 10.10.2.201/32 -> 1.2.3.201/32 age 1
bimap fxp0 10.10.2.203/32 -> 1.2.3.203/32 age 1
bimap fxp0 10.10.2.204/32 -> 1.2.3.204/32 age 1
bimap fxp0 10.10.2.205/32 -> 1.2.3.205/32 age 1
bimap fxp0 10.10.2.207/32 -> 1.2.3.207/32 age 1
bimap fxp0 10.10.2.208/32 -> 1.2.3.208/32 age 1
bimap fxp0 10.10.2.210/32 -> 1.2.3.210/32 age 1
bimap fxp0 10.10.2.233/32 -> 1.2.3.233/32 age 1
bimap fxp0 10.10.2.143/32 -> 1.2.3.143/32 age 1
Yes I know the age 1 thing was just an attempt to shorten the time of
the bimap rules in the NAT table (why are they needed really????) which
doesn't work for TCP anyway.
rc.local
--------
#!/bin/sh
/usr/sbin/arp -S 1.2.3.231 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.156 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.157 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.158 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.159 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.224 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.225 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.232 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.234 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.236 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.146 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.229 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.221 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.222 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.223 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.202 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.201 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.203 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.204 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.205 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.207 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.208 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.210 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.233 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.143 00:50:08:00:6a:32 pub
/usr/sbin/arp -S 1.2.3.70 00:50:08:00:6a:32 pub
sysctl.conf
-----------
net.inet.ipf.fr_flags=0
net.inet.ipf.fr_pass=514
net.inet.ipf.fr_active= 0
net.inet.ipf.fr_tcpidletimeout=7200
net.inet.ipf.fr_tcpclosewait=30
net.inet.ipf.fr_tcphalfclosed=120
net.inet.ipf.fr_tcplastack=15
net.inet.ipf.fr_tcptimeout=30
net.inet.ipf.fr_tcpclosed=1
net.inet.ipf.fr_udptimeout=30
net.inet.ipf.fr_icmptimeout=30
net.inet.ipf.fr_defnatage=300
net.inet.ipf.fr_ipfrttl=120
net.inet.ipf.ipl_unreach=13
net.inet.ipf.ipl_inited=1
net.inet.ipf.fr_authsize=32
net.inet.ipf.fr_authused=0
net.inet.ipf.fr_defaultauthage=300
kern.polling.enable=1
kern.polling.user_frac=50
Still with me?
Okay, let me comment a bit further (if your curious how all the above
fits together). Okay, the rc.local (static arp entries) are used on all
the BIMAP'd and the NAT'd addresses. This is so that the packets will
actually run all the way out before coming back in (so for example let's
say your on machine 10.10.2.207 and you want to contact 1.2.3.208
(10.10.2.208 bimap'd), the above rules will allow this to work (and
10.10.2.208 thinks it's talking to 1.2.3.207), it took a lot of
tinkering to figure this out and get it working this way, while the PIX
I was replacing it just worked).
Anyway, I'm stuck here with a box that is rebooting like clock work (at
the 12 hour mark), and not quite sure what to do. Any thoughts?
Greg
--
Greg Rumple
[EMAIL PROTECTED]
