Hi,
I have two pf firewalls, both running 3.6 (and stock 3.6 kernels), and
recently (about 4 days after enabling pfsync) one page faulted
after echoing some messages to the console. Here's what I saw:
Mar 12 00:46:32 fw1 /bsd: WARNING: mclpool limit reached; increase
kern.maxclusters
Mar 12 00:47:32 fw1 /bsd: WARNING: mclpool limit reached; increase
kern.maxclusters
Mar 12 00:49:32 fw1 last message repeated 2 times
Mar 12 00:59:33 fw1 last message repeated 10 times
Mar 12 01:00:33 fw1 /bsd: WARNING: mclpool limit reached; increase
kern.maxclusters
Mar 12 01:01:33 fw1 /bsd: WARNING: mclpool limit reached; increase
kern.maxclusters
Mar 12 01:03:33 fw1 last message repeated 2 times
kernel: page fault trap, code=0
Stopped at pf_frag_tree_RB_REMOVE+0x36: movl 0(%eax),%eax
ddb>
so I ran trace:
ddb> trace
pf_frag_tree_RB_REMOVE(d05cfeb0,ea902460,d06f1d9c,d01edbf0) at pf_frag_tree_RB_
REMOVE+0x36
pf_remove_fragment(ea902460,0,d06f0000,d01e957e,ea902460) at pf_remove_fragment
+0x1e
pf_free_fragment(ea902460,ffffffff,d06f1e2c,d029d009) at pf_free_fragment+0xcc
pf_purge_expired_fragments(30,d040968e,d0fb4500,d0682780,d05cfa2c) at pf_purge_
expired_fragments+0x52
pf_purge_timeout(d05cfa2c,20,d06f1e6c,d01e9933,d059b808,d04097bd,d06f1e70,0) at
pf_purge_timeout+0x21
softclock(58,d06f0010,10,10,d06f0000) at softclock+0x1d1
Bad frame pointer: 0xd06f1e6c
Here's ps:
PID PPID PGRP UID S FLAGS WAIT COMMAND
26973 1 26973 0 3 0x4086 ttyin getty
23917 32290 32290 83 3 0x184 poll ntpd
32290 1 32290 0 3 0x84 poll ntpd
31000 1 31000 0 3 0x4086 ttyin getty
9856 1 9856 0 3 0x4086 ttyin getty
24678 1 24678 0 3 0x4086 ttyin getty
11019 1 11019 0 3 0x4086 ttyin getty
31697 1 31697 0 3 0x4086 ttyin getty
30255 26537 26537 0 3 0x84 nanosleep apcupsd
6586 26537 26537 0 3 0x84 netcon apcupsd
14045 6555 6555 0 3 0x4086 piperd readproctitle
13840 6555 6555 0 3 0x4086 nanosleep svscan
17296 1 17296 0 3 0x84 select cron
26537 1 26537 0 3 0x84 wait apcupsd
6555 1 6555 0 3 0x4086 pause sh
22055 1 22055 0 3 0x84 select sshd
29249 17193 17193 74 3 0x184 bpf pflogd
17193 1 17193 0 3 0x84 netio pflogd
3591 30070 30070 73 2 0x184 syslogd
30070 1 30070 0 3 0x84 netio syslogd
11 0 0 0 3 0x100204 usbevt usb1
10 0 0 0 3 0x100204 usbtsk usbtask
9 0 0 0 3 0x100204 usbevt usb0
8 0 0 0 3 0x100204 kmalloc kmthread
7 0 0 0 3 0x100204 crypto_wa crypto
6 0 0 0 3 0x100204 aiodoned aiodoned
5 0 0 0 3 0x100204 syncer update
4 0 0 0 3 0x100204 cleaner cleaner
3 0 0 0 3 0x100204 reaper reaper
2 0 0 0 3 0x100204 pgdaemon pagedaemon
1 0 1 0 3 0x4084 wait init
* 0 -1 0 0 3 0x80204 scheduler swapper
What's weird is that only the backup (fw1) page faulted.
***** If I should simply bump kern.maxclusters to a higher value, what
value should I use? *****
It's currently at the default:
# sysctl kern.maxclusters
kern.maxclusters=6144
On both systems. Both systems have half a gig of ram, 2.8ghz xeon, and
ten gigE cards, with about 12 vlans + 6 phys ints used for non-vlan
traffic. I could easily bump the ram to a gig if needed. I've never
sized a pf firewall before, so suggestions would be lovely.
Here is dmesg on fw1:
ddb> dmesg
OpenBSD 3.6 (GENERIC) #59: Fri Sep 17 12:32:57 MDT 2004
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Xeon(TM) CPU 2.80GHz ("GenuineIntel" 686-class) 2.80 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,CNXT-ID
real mem = 536367104 (523796K)
avail mem = 482512896 (471204K)
using 4278 buffers containing 26923008 bytes (26292K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(00) BIOS, date 05/11/04, BIOS32 rev. 0 @
0xfdb54
pcibios0 at bios0: rev 2.1 @ 0xf0000/0x10000
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xf3630/336 (19 entries)
pcibios0: no compatible PCI ICU found: ICU vendor 0x8086 product 0x2480
pcibios0: Warning, unable to fix up PCI interrupt routing
pcibios0: PCI bus #4 is the last bus
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x5200! 0xe4000/0x4000!
cpu0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 vendor "Intel", unknown product 0x254c
rev 0x01
"Intel E7500 DRAM" rev 0x01 at pci0 dev 0 function 1 not configured
ppb0 at pci0 dev 3 function 0 "Intel E7500 MCH HI_C vppb 1" rev 0x01
pci1 at ppb0 bus 1
"Intel P64H2 IOxAPIC" rev 0x04 at pci1 dev 28 function 0 not configured
ppb1 at pci1 dev 29 function 0 "Intel 82870P2 P64H2 PCI-PCI" rev 0x04
pci2 at ppb1 bus 2
gdt0 at pci2 dev 8 function 0 "Intel GDT RAID" rev 0x00: irq 9 dpmem f8000000 2
-bus 1 cache device
gdt0: ver 222, cache on, strategy 2, writeback on, blksz 32
gdt0: raw feat 1 cache feat 101
scsibus0 at gdt0: 35 targets
sd0 at scsibus0 targ 0 lun 0: <ICP, Host drive #00, > SCSI2 0/direct fixed
sd0: 34695MB, 4423 cyl, 255 head, 63 sec, 512 bytes/sec, 71055495 sec total
scsibus1 at gdt0: 16 targets
scsibus2 at gdt0: 16 targets
em0 at pci2 dev 9 function 0 "Intel PRO/1000MT DP (82546EB)" rev 0x03: irq 9,
address: 00:04:23:ab:96:2e
em1 at pci2 dev 9 function 1 "Intel PRO/1000MT DP (82546EB)" rev 0x03: irq 9,
address: 00:04:23:ab:96:2f
em2 at pci2 dev 10 function 0 "Intel PRO/1000MT DP (82546EB)" rev 0x03: irq 9,
address: 00:04:23:ad:b1:f0
em3 at pci2 dev 10 function 1 "Intel PRO/1000MT DP (82546EB)" rev 0x03: irq 9,
address: 00:04:23:ad:b1:f1
"Intel P64H2 IOxAPIC" rev 0x04 at pci1 dev 30 function 0 not configured
ppb2 at pci1 dev 31 function 0 "Intel 82870P2 P64H2 PCI-PCI" rev 0x04
pci3 at ppb2 bus 3
em4 at pci3 dev 7 function 0 "Intel PRO/1000MT DP (82546EB)" rev 0x01: irq 9,
address: 00:0e:0c:5c:f5:10
em5 at pci3 dev 7 function 1 "Intel PRO/1000MT DP (82546EB)" rev 0x01: irq 9,
address: 00:0e:0c:5c:f5:11
em6 at pci3 dev 8 function 0 "Intel PRO/1000MT DP (82546EB)" rev 0x03: irq 9,
address: 00:04:23:ab:64:fa
em7 at pci3 dev 8 function 1 "Intel PRO/1000MT DP (82546EB)" rev 0x03: irq 9,
address: 00:04:23:ab:64:fb
em8 at pci3 dev 9 function 0 "Intel PRO/1000MT DP (82546EB)" rev 0x03: irq 9,
address: 00:04:23:ac:1b:72
em9 at pci3 dev 9 function 1 "Intel PRO/1000MT DP (82546EB)" rev 0x03: irq 9,
address: 00:04:23:ac:1b:73
"Intel E7500 MCH HI_C vppb 2" rev 0x01 at pci0 dev 3 function 1 not configured
uhci0 at pci0 dev 29 function 0 "Intel 82801CA/CAM USB" rev 0x02: irq 9
usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1 at pci0 dev 29 function 1 "Intel 82801CA/CAM USB" rev 0x02: irq 10
usb1 at uhci1: USB revision 1.0
uhub1 at usb1
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
ppb3 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0x42
pci4 at ppb3 bus 4
vga1 at pci4 dev 12 function 0 "ATI Rage XL" rev 0x27
wsdisplay0 at vga1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ichpcib0 at pci0 dev 31 function 0 "Intel 82801CA LPC" rev 0x02
pciide0 at pci0 dev 31 function 1 "Intel 82801CA IDE" rev 0x02: DMA, channel 0
configured to compatibility, channel 1 configured to compatibility
pciide0: channel 0 disabled (no drives)
atapiscsi0 at pciide0 channel 1 drive 0
scsibus3 at atapiscsi0: 2 targets
cd0 at scsibus3 targ 0 lun 0: <SAMSUNG, CD-ROM SN-124, N102> SCSI05/cdrom
removable
cd0(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 2
"Intel 82801CA/CAM SMBus" rev 0x02 at pci0 dev 31 function 3 not configured
isa0 at ichpcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
sysbeep0 at pcppi0
npx0 at isa0 port 0xf0/16: using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pccom0: console
pccom1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
biomask ffe5 netmask ffe5 ttymask ffe7
pctr: user-level cycle counter enabled
dkcsum: sd0 matched BIOS disk 80
root on sd0a
rootdev=0x400 rrootdev=0xd00 rawdev=0xd02
uplcom0 at uhub0 port 2
uplcom0: Prolific Technology Inc. USB-Serial Controller, rev 1.10/3.00,
addr 2
ucom0 at uplcom0
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
<3>WARNING: mclpool limit reached; increase kern.maxclusters
here is dmesg on fw0 (the primary, which didn't page fault):
# dmesg
OpenBSD 3.6 (GENERIC) #59: Fri Sep 17 12:32:57 MDT 2004
[EMAIL PROTECTED]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Xeon(TM) CPU 2.80GHz ("GenuineIntel" 686-class) 2.80 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,CNXT-ID
real mem = 536383488 (523812K)
avail mem = 482529280 (471220K)
using 4278 buffers containing 26923008 bytes (26292K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(00) BIOS, date 10/29/03, BIOS32 rev. 0 @ 0xfdb54
pcibios0 at bios0: rev 2.1 @ 0xf0000/0x10000
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xf3640/336 (19 entries)
pcibios0: no compatible PCI ICU found: ICU vendor 0x8086 product 0x2480
pcibios0: Warning, unable to fix up PCI interrupt routing
pcibios0: PCI bus #4 is the last bus
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x5200! 0xe4000/0x4000!
cpu0 at mainbus0
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 vendor "Intel", unknown product 0x254c rev 0x01
"Intel E7500 DRAM" rev 0x01 at pci0 dev 0 function 1 not configured
ppb0 at pci0 dev 3 function 0 "Intel E7500 MCH HI_C vppb 1" rev 0x01
pci1 at ppb0 bus 1
"Intel P64H2 IOxAPIC" rev 0x04 at pci1 dev 28 function 0 not configured
ppb1 at pci1 dev 29 function 0 "Intel 82870P2 P64H2 PCI-PCI" rev 0x04
pci2 at ppb1 bus 2
gdt0 at pci2 dev 8 function 0 "Intel GDT RAID" rev 0x00: irq 9 dpmem f8000000
2-bus 1 cache device
gdt0: ver 222, cache on, strategy 2, writeback on, blksz 32
gdt0: raw feat 1 cache feat 101
scsibus0 at gdt0: 35 targets
sd0 at scsibus0 targ 0 lun 0: <ICP, Host drive #00, > SCSI2 0/direct fixed
sd0: 34695MB, 4423 cyl, 255 head, 63 sec, 512 bytes/sec, 71055495 sec total
scsibus1 at gdt0: 16 targets
scsibus2 at gdt0: 16 targets
em0 at pci2 dev 9 function 0 "Intel PRO/1000MT DP (82546EB)" rev 0x03: irq 9,
address: 00:04:23:ab:a4:6c
em1 at pci2 dev 9 function 1 "Intel PRO/1000MT DP (82546EB)" rev 0x03: irq 9,
address: 00:04:23:ab:a4:6d
em2 at pci2 dev 10 function 0 "Intel PRO/1000MT DP (82546EB)" rev 0x03: irq 9,
address: 00:04:23:ac:1b:b4
em3 at pci2 dev 10 function 1 "Intel PRO/1000MT DP (82546EB)" rev 0x03: irq 9,
address: 00:04:23:ac:1b:b5
"Intel P64H2 IOxAPIC" rev 0x04 at pci1 dev 30 function 0 not configured
ppb2 at pci1 dev 31 function 0 "Intel 82870P2 P64H2 PCI-PCI" rev 0x04
pci3 at ppb2 bus 3
em4 at pci3 dev 7 function 0 "Intel PRO/1000MT DP (82546EB)" rev 0x01: irq 9,
address: 00:0e:0c:31:bf:30
em5 at pci3 dev 7 function 1 "Intel PRO/1000MT DP (82546EB)" rev 0x01: irq 9,
address: 00:0e:0c:31:bf:31
em6 at pci3 dev 8 function 0 "Intel PRO/1000MT DP (82546EB)" rev 0x03: irq 9,
address: 00:04:23:ab:36:e2
em7 at pci3 dev 8 function 1 "Intel PRO/1000MT DP (82546EB)" rev 0x03: irq 9,
address: 00:04:23:ab:36:e3
em8 at pci3 dev 9 function 0 "Intel PRO/1000MT DP (82546EB)" rev 0x03: irq 9,
address: 00:04:23:ab:61:b4
em9 at pci3 dev 9 function 1 "Intel PRO/1000MT DP (82546EB)" rev 0x03: irq 9,
address: 00:04:23:ab:61:b5
"Intel E7500 MCH HI_C vppb 2" rev 0x01 at pci0 dev 3 function 1 not configured
uhci0 at pci0 dev 29 function 0 "Intel 82801CA/CAM USB" rev 0x02: irq 9
usb0 at uhci0: USB revision 1.0
uhub0 at usb0
uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub0: 2 ports with 2 removable, self powered
uhci1 at pci0 dev 29 function 1 "Intel 82801CA/CAM USB" rev 0x02: irq 10
usb1 at uhci1: USB revision 1.0
uhub1 at usb1
uhub1: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
uhub1: 2 ports with 2 removable, self powered
ppb3 at pci0 dev 30 function 0 "Intel 82801BA AGP" rev 0x42
pci4 at ppb3 bus 4
vga1 at pci4 dev 12 function 0 "ATI Rage XL" rev 0x27
wsdisplay0 at vga1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ichpcib0 at pci0 dev 31 function 0 "Intel 82801CA LPC" rev 0x02
pciide0 at pci0 dev 31 function 1 "Intel 82801CA IDE" rev 0x02: DMA, channel 0
configured to compatibility, channel 1 configured to compatibility
pciide0: channel 0 disabled (no drives)
atapiscsi0 at pciide0 channel 1 drive 0
scsibus3 at atapiscsi0: 2 targets
cd0 at scsibus3 targ 0 lun 0: <SAMSUNG, CD-ROM SN-124, N102> SCSI0 5/cdrom
removable
cd0(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 2
"Intel 82801CA/CAM SMBus" rev 0x02 at pci0 dev 31 function 3 not configured
isa0 at ichpcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
sysbeep0 at pcppi0
npx0 at isa0 port 0xf0/16: using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pccom0: console
pccom1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
biomask ffe5 netmask ffe5 ttymask ffe7
pctr: user-level cycle counter enabled
dkcsum: sd0 matched BIOS disk 80
root on sd0a
rootdev=0x400 rrootdev=0xd00 rawdev=0xd02
uplcom0 at uhub0 port 2
uplcom0: Prolific Technology Inc. USB-Serial Controller, rev 1.10/3.00, addr 2
ucom0 at uplcom0
arp info overwritten for 10.1.1.113 by 00:03:ba:55:cc:bf on em2
arp info overwritten for 10.1.1.107 by 00:03:ba:8d:51:cf on em2
arp info overwritten for 10.1.1.113 by 00:03:ba:55:cc:bd on em2
arp info overwritten for 10.1.1.107 by 00:03:ba:8d:51:cd on em2
The arp info overwritten is generated when Sun's network multipathing
(IPMP) implementation kicks in.
# pfctl -s info
Status: Enabled for 0 days 19:19:24 Debug: Urgent
Hostid: 0xd264d6ad
State Table Total Rate
current entries 818
searches 124853290 1794.8/s
inserts 1136664 16.3/s
removals 1135846 16.3/s
Counters
match 28047720 403.2/s
bad-offset 0 0.0/s
fragment 0 0.0/s
short 0 0.0/s
normalize 0 0.0/s
memory 0 0.0/s
bad-timestamp 0 0.0/s
# pfctl -s all |wc -l
1137
is kind of long, so I will summarize pf.conf:
int_phys = "{ em0 em1 em2 em4 em5 em9 }"
int_vlan = "{ vlan2 vlan4 vlan6 vlan7 vlan8 vlan9 vlan12 vlan13 vlan32 vlan33 vl
an34 vlan35 vlan36 vlan37 vlan38 vlan39 }"
table <publicips> { 1.2.3.0/24 }
table <rfc1918> { 127.0.0.0/8 10.0.0.0/8 172.16.0.0/12 192.168.0.0/16 }
..
scrub in all
antispoof quick for $int_phys
antispoof quick for $int_vlan
antispoof quick for $external_ints
block in log quick on $external_ints from { <rfc1918> <publicips> }
block log all
pass on $int_phys proto carp keep state
pass on $int_vlan proto carp keep state
pass on em7 proto carp keep state
$pq on em4 proto pfsync
$pq on lo0 all
the remainder is just rules for things like http access, pings, ftp,
etc. etc., but a fair number of rules. Let me know if they are needed
to suggest remedies.
Thanks!
--
adam
