Hi,
I occasionally get 'no route to host' errors on clients on the internal network natted
to the external network. The errors in
the logs when debug is set to misc are as follows:
/bsd: pf: state insert failed: tree_ext_gwy lan: 192.168.1.250:43445 gwy:
#externalIP#:47566 ext: #externalHOST#:8080
This happens seemingly randomly, but under stress, the frequency increases. For
instance under normal conditions, I get maybe
1 out of 1000 to 2000 connections dropped/blocked. pflog0 and log files show no
blocks corresponding to these errors. This
seems to only happen through nat, I can flush the filter rules and still get the error.
When these errors happen, the memory counter shown in pfctl -si increases
accordingly. I increased nmbclusters to 8192 with
config on the running kernel. As soon as I get a chance I will reboot with a freshly
compiled kernel.
I also decreased timeout on tcp.closed to 50 from 90 and increased limit
states/frags(see below) after reading some suggestions
by Daniel Hartmeier in the archives. I really don't know C well enough to understand
fully what is happening with
tree_ext_gwy, so I am looking for some pointers or help. I'm really hoping its
something simple that I missed.
Thanks,
Daniel
Details:
I replaced IPs I didn't want to make public throughout this email, ie #externalIP# and
so on.
This is the setup:
OpenBSD 3.3 Generic i386 patched
IBM x330 1.4Ghz PIII 1Gig ram
3 interfaces, fxp0, fxp1 and em0
fxp0 192.168.1.xxx/24
fxp1 #externalIP#
em0 172.xx.xx.xx/30
remotenet is a table with all the networks coming from em0
Here are the rules:
ext_if="fxp1"
int_if="fxp0"
gig_if="em0"
external_addr = "#externalIP#"
internal_net = "192.168.1.0/24"
ftpports = "{ 55000 >< 57000 }"
table <remotenet> { a bunch of private networks }
set timeout { tcp.closing 900, tcp.finwait 45, tcp.closed 50 }
set limit { states 100000, frags 50000 }
set loginterface fxp0
set block-policy drop
scrub in all no-df random-id
nat on $ext_if from $internal_net to any -> #externalIP#
rdr on $int_if proto tcp from any to any port 21 -> 127.0.0.1 port 8021
block in log all
block out log all
pass quick on lo0 all
antispoof quick for $int_if inet
pass in quick on $int_if all keep state
pass out quick on $int_if all keep state
pass in on $gig_if from $gig_if:network to any keep state
pass out on $gig_if from any to $gig_if:network keep state
pass in on $gig_if from <remotenet> to any keep state
pass out on $gig_if from any to <remotenet> keep state
pass out log on $ext_if proto { tcp, udp, icmp } all keep state
pass in quick on $ext_if inet proto tcp from any port 20 to $ext_if port $ftpports
flags S/SA keep state
Output from pfctl -s info:
Interface Stats for fxp0 IPv4 IPv6
Bytes In 487075454 0
Bytes Out 1704986917 0
Packets In
Passed 2971613 0
Blocked 57 0
Packets Out
Passed 2789494 0
Blocked 0 0
State Table Total Rate
current entries 551
searches 89579085 639.0/s
inserts 496265 3.5/s
removals 495714 3.5/s
Counters
match 42601238 303.9/s
bad-offset 0 0.0/s
fragment 0 0.0/s
short 0 0.0/s
normalize 0 0.0/s
memory 110 0.0/s
netstat -m:
401 mbufs in use:
385 mbufs allocated to data
11 mbufs allocated to packet headers
5 mbufs allocated to socket names and addresses
385/408/1024 mbuf cluster pages in use (current/peak/max)
1844 Kbytes allocated to network (47% in use)
0 requests for memory denied
0 requests for memory delayed
0 calls to protocol drain routines
vmstat -m(trimmed):
Memory Totals: In Use Free Requests
1147K 246K 1920651
Memory resource pool statistics
Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
extentpl 20 179 0 166 1 0 1 1 0 inf 0
phpool 40 672 0 0 7 0 7 7 0 inf 0
pmappl 72 5655 0 5618 2 0 2 2 0 inf 1
vmsppl 212 5655 0 5618 4 0 4 4 0 inf 1
vmmpepl 88 163246 0 162510 25 0 25 25 0 inf 4
vmmpekpl 88 13691 0 13598 3 0 3 3 0 inf 0
aobjpl 52 1 0 0 1 0 1 1 0 inf 0
amappl 40 60269 0 59944 6 0 6 6 0 inf 1
bufpl 116 10 0 10 1 0 1 1 0 inf 1
mbpl 256 160495038 0 160494627 53 0 53 53 1 inf 27
mclpl 2048 48244518 0 48244133 408 0 408 408 4 1024 214
sockpl 200 1499 0 1462 4 0 4 4 0 inf 1
procpl 316 5662 0 5618 6 0 6 6 0 inf 1
zombiepl 72 5618 0 5618 1 0 1 1 0 inf 1
ucredpl 80 1385 0 1361 1 0 1 1 0 inf 0
pgrppl 24 2169 0 2140 1 0 1 1 0 inf 0
sessionpl 48 955 0 930 1 0 1 1 0 inf 0
pcredpl 24 5662 0 5618 1 0 1 1 0 inf 0
filepl 48 1384407 0 1384318 2 0 2 2 0 inf 0
fdescpl 320 5663 0 5618 6 0 6 6 0 inf 1
pipepl 76 5082 0 5074 1 0 1 1 0 inf 0
sigapl 308 5655 0 5618 5 0 5 5 0 inf 1
scxspl 128 360733 0 360731 1 0 1 1 0 inf 0
wdcspl 96 9 0 9 1 0 1 1 0 inf 1
vnodes 160 2621 0 0 105 0 105 105 0 inf 0
nchpl 64 1310 0 0 21 0 21 21 0 inf 0
ffsino 280 304265 0 301650 187 0 187 187 0 inf 0
semapl 64 1 0 0 1 0 1 1 0 inf 0
semupl 100 2 0 2 1 0 1 1 0 inf 1
pftrpl 60 1003914 0 1002736 66 0 66 66 0 inf 31
pfrulepl 484 306 0 288 3 0 3 3 0 inf 0
pfstatepl 144 501957 0 501368 79 0 79 79 0 3572 39
pfpooladdrpl 76 27 0 25 1 0 1 1 0 inf 0
pfrktable 196 32 0 31 1 0 1 1 0 inf 0
pfrkentry 152 368 0 345 1 0 1 1 0 inf 0
pffrent 16 124803 0 124803 2 0 2 2 0 198 2
pffrag 48 11170 0 11170 1 0 1 1 0 12 1
ipqepl 20 217 0 217 1 0 1 1 0 inf 1
tcpcbpl 344 526 0 503 4 0 4 4 0 inf 1
sackhlpl 20 10 0 10 1 0 1 1 0 inf 1
plimitpl 152 949 0 932 1 0 1 1 0 inf 0
kqeuepl 192 3 0 3 1 0 1 1 0 inf 1
knotepl 64 6 0 6 1 0 1 1 0 inf 1
In use 2430K, total allocated 4080K; utilization 59.6%
