Can anyone suggest reasons for a firewall doing this? Happened a few
times to this machine, first on i386 GENERIC from May 3, now on Dec 9th.
[-- MARK -- Fri Jan 6 22:00:00 2012]
[-- MARK -- Fri Jan 6 23:00:00 2012]
[-- MARK -- Sat Jan 7 00:00:00 2012]
[-- MARK -- Sat Jan 7 01:00:00 2012]
[-- MARK -- Sat Jan 7 02:00:00 2012]
[-- MARK -- Sat Jan 7 03:00:00 2012]
[-- MARK -- Sat Jan 7 04:00:00 2012]
[-- MARK -- Sat Jan 7 05:00:00 2012]
panic: pmap_remove_ptes: unmanaged page marked PG_PVLIST, va = 0x56c5000, pa =
0xfffff000
Starting stack trace...
panic(d08d41b8,dc249e18,d08d7440,dc249e18,d14f944c) at panic+0x6a
panic(d08d7440,56c5000,fffff000,56cd000,0) at panic+0x6a
pmap_remove_ptes(d6d35ab0,d14f944c,ffc15b14,56c5000,56cd000) at
pmap_remove_ptes+0x159
pmap_do_remove(d6d35ab0,56c5000,56cd000,0,2) at pmap_do_remove+0xeb
pmap_remove(d6d35ab0,56c5000,56cd000,d03e64f0,d0add9f4) at pmap_remove+0x27
uvm_unmap_remove(d6a73928,1000,cfbfe000,dc249f3c,0) at uvm_unmap_remove+0x167
uvmspace_free(d6a73928,1,1,dc249f6c,d0203151) at uvmspace_free+0xe0
uvm_exit(d6a4bc28,d09b95a8,4,d08b076e,0) at uvm_exit+0x15
reaper(d6d3ad70) at reaper+0x8a
Bad frame pointer: 0xd0b93e68
End of stack trace.
syncing disks... 3 1 done
Running these daemons:
starting network daemons: sshd snmpd ospfd bgpd ifstated sendmail inetd sndiod.
starting local daemons: symon ftpsesame.
starting local daemons: sensorsd cron.
May also have been running isakmpd if it was carp master at the
time (this gets started/stopped by ifstated).
Full internet routing table (380K entries).
carp/pfsync partner (identical box+config+snapshot) stayed stable.
OpenBSD 5.0-current (GENERIC) #113: Fri Dec 9 11:44:41 MST 2011
dera...@i386.openbsd.org:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: AMD Opteron(tm) Processor 148 ("AuthenticAMD" 686-class, 1024KB L2 cache)
2.22 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW,SSE3,LAHF
real mem = 1072070656 (1022MB)
avail mem = 1044451328 (996MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 11/06/06, BIOS32 rev. 0 @ 0xfa780, SMBIOS
rev. 2.3 @ 0xf0000 (41 entries)
bios0: vendor Sun Microsystems version "1.1.3" date 11/06/2006
bios0: Sun Microsystems Sun Fire(TM) X2100
apm0 at bios0: Power Management spec V1.2 (slowidle)
acpi at bios0 function 0x0 not configured
pcibios0 at bios0: rev 3.0 @ 0xf0000/0xcc44
pcibios0: PCI IRQ Routing Table rev 1.0 @ 0xfcb20/288 (16 entries)
pcibios0: bad IRQ table checksum
pcibios0: PCI BIOS has 16 Interrupt Routing table entries
pcibios0: PCI Exclusive IRQs: 3 5 7 10 11
pcibios0: no compatible PCI ICU found
pcibios0: Warning, unable to fix up PCI interrupt routing
pcibios0: PCI bus #5 is the last bus
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x4000! 0xcc000/0x1600 0xce000/0x1800
ipmi at mainbus0 not configured
cpu0 at mainbus0: (uniprocessor)
cpu0: AMD erratum 89 present, BIOS upgrade may be required
cpu0: Cool'n'Quiet K8 2212 MHz: speeds: 2200 2000 1800 1000 MHz
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
"NVIDIA nForce4 DDR" rev 0xa3 at pci0 dev 0 function 0 not configured
pcib0 at pci0 dev 1 function 0 "NVIDIA nForce4 ISA" rev 0xa3
nviic0 at pci0 dev 1 function 1 "NVIDIA nForce4 SMBus" rev 0xa2
iic0 at nviic0
adt0 at iic0 addr 0x2e: sch5017 rev 0x89
spdmem0 at iic0 addr 0x50: 512MB DDR SDRAM ECC PC3200CL3.0
spdmem1 at iic0 addr 0x51: 512MB DDR SDRAM ECC PC3200CL3.0
iic1 at nviic0
adt1 at iic1 addr 0x2e: sch5017 rev 0x89
spdmem2 at iic1 addr 0x50: 512MB DDR SDRAM ECC PC3200CL3.0
spdmem3 at iic1 addr 0x51: 512MB DDR SDRAM ECC PC3200CL3.0
ohci0 at pci0 dev 2 function 0 "NVIDIA nForce4 USB" rev 0xa2: irq 10, version
1.0, legacy support
ehci0 at pci0 dev 2 function 1 "NVIDIA nForce4 USB" rev 0xa3: irq 11
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "NVIDIA EHCI root hub" rev 2.00/1.00 addr 1
pciide0 at pci0 dev 6 function 0 "NVIDIA nForce4 IDE" rev 0xf2: DMA, channel 0
configured to compatibility, channel 1 configured to compatibility
pciide0: channel 0 disabled (no drives)
pciide0: channel 1 disabled (no drives)
pciide1 at pci0 dev 7 function 0 "NVIDIA nForce4 SATA" rev 0xf3: DMA
pciide1: using irq 11 for native-PCI interrupt
wd0 at pciide1 channel 0 drive 0: <HITACHI HDS7280SASUN80G 0633MS3Z3M>
wd0: 16-sector PIO, LBA48, 76319MB, 156301488 sectors
wd0(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 6
pciide2 at pci0 dev 8 function 0 "NVIDIA nForce4 SATA" rev 0xf3: DMA
pciide2: using irq 10 for native-PCI interrupt
ppb0 at pci0 dev 9 function 0 "NVIDIA nForce4 PCI-PCI" rev 0xa2
pci1 at ppb0 bus 1
vga1 at pci1 dev 5 function 0 "ATI Rage XL" rev 0x27
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
nfe0 at pci0 dev 10 function 0 "NVIDIA CK804 LAN" rev 0xa3: irq 3, address
00:e0:81:5e:47:7e
eephy0 at nfe0 phy 1: 88E1111 Gigabit PHY, rev. 2
ppb1 at pci0 dev 11 function 0 "NVIDIA nForce4 PCIE" rev 0xa3
pci2 at ppb1 bus 2
ppb2 at pci0 dev 12 function 0 "NVIDIA nForce4 PCIE" rev 0xa3
pci3 at ppb2 bus 3
ppb3 at pci0 dev 13 function 0 "NVIDIA nForce4 PCIE" rev 0xa3
pci4 at ppb3 bus 4
bge0 at pci4 dev 0 function 0 "Broadcom BCM5721" rev 0x11, BCM5750 B1 (0x4101):
irq 5, address 00:e0:81:5e:47:7f
brgphy0 at bge0 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
ppb4 at pci0 dev 14 function 0 "NVIDIA nForce4 PCIE" rev 0xa3
pci5 at ppb4 bus 5
pchb0 at pci0 dev 24 function 0 "AMD AMD64 0Fh HyperTransport" rev 0x00
pchb1 at pci0 dev 24 function 1 "AMD AMD64 0Fh Address Map" rev 0x00
pchb2 at pci0 dev 24 function 2 "AMD AMD64 0Fh DRAM Cfg" rev 0x00
kate0 at pci0 dev 24 function 3 "AMD AMD64 0Fh Misc Cfg" rev 0x00
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
usb1 at ohci0: USB revision 1.0
uhub1 at usb1 "NVIDIA OHCI root hub" rev 1.00/1.00 addr 1
mtrr: Pentium Pro MTRR support
vscsi0 at root
scsibus0 at vscsi0: 256 targets
softraid0 at root
scsibus1 at softraid0: 256 targets
root on wd0a (8bc44f7e12e09ba9.a) swap on wd0b dump on wd0b
WARNING: / was not properly unmounted
# ps -xhklww -M bsd.2.core -N bsd.2
UID PID PPID CPU PRI NI VSZ RSS WCHAN STAT TT TIME COMMAND
0 10 0 0 -18 0 0 0 pgdaemo DK ?? 0:00.00
(pagedaemon)
0 11 0 25 -18 0 0 0 - RK ?? 0:26.00 (reaper)
0 12 0 0 -13 0 0 0 cleaner DK ?? 0:00.10 (cleaner)
0 13 0 0 18 0 0 0 - RK ?? 0:04.44 (update)
0 14 0 0 -18 0 0 0 aiodone DK ?? 0:00.00 (aiodoned)
0 0 0 0 -18 0 0 0 schedul DKs ?? 0:00.19 (swapper)
0 1 0 0 10 0 368 0 wait Is ?? 0:00.19 (init)
0 2 0 0 -18 0 0 0 kmalloc DK ?? 0:00.07 (kmthread)
0 3 0 36 -22 0 0 0 - DK ?? 9416:51.40 (idle0)
0 4 0 0 10 0 0 0 - RK ?? 41:23.19 (syswq)
0 5 0 0 10 0 0 0 - RK ?? 9:30.67 (apm0)
0 6 0 0 10 0 0 0 usbatsk DK ?? 0:00.00 (usbatsk)
0 7 0 0 10 0 0 0 usbtsk DK ?? 0:00.00 (usbtask)
0 8 0 0 10 0 0 0 - RK ?? 0:10.81 (pfpurge)
0 9 0 0 10 0 0 0 - RK ?? 4:06.27 (crypto)
0 17314 1 0 2 0 368 0 netio Is ?? 0:00.17 (syslogd)
0 17001 1 0 2 0 476 0 netio Is ?? 0:00.01 (pflogd)
0 18677 1 0 2 0 592 0 poll Is ?? 0:00.00 (ntpd)
0 9409 1 0 2 0 1392 0 netio Is ?? 0:00.14 (isakmpd)
0 9668 1 2 2 0 664 0 select Is ?? 0:03.77 (sshd)
0 12905 1 0 2 0 24720 0 - Rs ?? 3:40.04 (ospfd)
0 8981 1 0 2 0 25072 0 - Rs ?? 3:10.61 (bgpd)
0 27510 1 0 2 0 328 0 - Rs ?? 0:00.00 (ifstated)
0 14016 1 0 2 0 1312 0 - Rs ?? 0:08.32 (sendmail)
0 740 1 0 2 0 264 0 select Ss ?? 0:01.09 (inetd)
0 31967 1 0 10 0 536 0 nanosle Ss ?? 0:02.26 (sensorsd)
0 492 1 0 2 0 660 0 select Ss ?? 0:02.25 (cron)
0 23867 1 10 2 0 576 0 kqread Is ?? 0:00.00 (snmpd)
0 16430 26147 0 10 0 540 0 - R ?? 0:00.00
(newsyslog)
0 3965 492 0 -6 0 660 0 piperd S ?? 0:00.00 (cron)
0 26147 3965 0 18 0 532 0 pause Ss ?? 0:00.00 (sh)
0 19982 1 8 3 0 484 0 ttyin Is+ 00 0:00.00 (getty)
0 11636 1 8 3 0 292 0 ttyin Is+ C0 0:00.00 (getty)
0 3902 1 8 3 0 360 0 ttyin Is+ C1 0:00.00 (getty)
0 2958 1 8 3 0 452 0 ttyin Is+ C2 0:00.00 (getty)
0 24456 1 8 3 0 412 0 ttyin Is+ C3 0:00.00 (getty)
0 7884 1 8 3 0 380 0 ttyin Is+ C5 0:00.00 (getty)
FWIW this is how vmstat -m looks *at the moment* i.e. after restarting
in normal operations: (vmstat won't extract information from the core;
"vmstat: undefined symbols: _uvm_km_pages_free").
Memory statistics by bucket size
Size In Use Free Requests HighWater Couldfree
16 76005 1563 252727 1280 12
32 383718 282 420464 640 0
64 2747 69 308811 320 0
128 462 18 98194 160 0
256 238 1138 648571 80 261980
512 292 12 406385 40 0
1024 798 2 20927 20 0
2048 17 11 19731 10 0
4096 280 2 1072 5 0
8192 10 1 26579 5 0
16384 5 0 5 5 0
32768 6 0 7 5 0
65536 1 0 3 5 0
131072 1 0 1 5 0
262144 1 0 1 5 0
Memory usage type by bucket size
Size Type(s)
16 devbuf, pcb, routetbl, ifaddr, vnodes, UFS mount, dirhash,
ip_moptions, in_multi, exec, pfkey data, xform_data, UVM amap,
UVM aobj, USB, USB device, temp
32 devbuf, pcb, routetbl, ifaddr, sysctl, UFS mount, sem, dirhash, proc,
VFS cluster, in_multi, ether_multi, pfkey data, xform_data, VM swap,
UVM amap, USB, crypto data, temp
64 devbuf, pcb, routetbl, ifaddr, vnodes, sem, dirhash, NFS srvsock,
ip_moptions, in_multi, pfkey data, xform_data, UVM amap, USB,
USB device, IPsec creds, packet tags, NDP, temp
128 devbuf, routetbl, ifaddr, sysctl, iov, mount, vnodes, dirhash, ttys,
pfkey data, tdb, VM swap, UVM amap, USB, crypto data, NDP, temp
256 devbuf, routetbl, ifaddr, ioctlops, iov, shm, VM map, file desc,
NFS daemon, exec, pfkey data, tdb, newblk, UVM amap, USB, USB device,
IPsec creds, temp
512 devbuf, pcb, routetbl, ifaddr, ioctlops, iov, UFS mount, shm, dirhash,
file desc, proc, ttys, exec, UVM amap, crypto data, temp
1024 devbuf, ioctlops, iov, mount, ttys, exec, pfkey data, tdb, UVM amap,
UVM aobj, crypto data, IPsec creds, temp
2048 devbuf, ioctlops, iov, UFS mount, proc, pfkey data, VM swap, UVM amap,
temp
4096 devbuf, ifaddr, ioctlops, iov, UFS mount, pagedep, UVM amap, USB,
memdesc, temp
8192 devbuf, iov, UFS mount, MSDOSFS mount, ttys, temp
16384 UFS quota, UFS mount, ISOFS mount, inodedep, NTFS hash
32768 devbuf, UFS mount
65536 devbuf
131072 VM swap
262144 VM swap
Memory statistics by type Type Kern
Type InUse MemUse HighUse Limit Requests Limit Limit Size(s)
devbuf 2498 1964K 1964K 39322K 3110 0 0
16,32,64,128,256,512,1024,2048,4096,8192,32768,65536
pcb 70 8K 8K 39322K 1348 0 0 16,32,64,512
routetbl416710 12520K 12520K 39322K 1430141 0 0
16,32,64,128,256,512
ifaddr 151 34K 34K 39322K 157 0 0
16,32,64,128,256,512,4096
sysctl 2 1K 1K 39322K 2 0 0 32,128
ioctlops 0 0K 4K 39322K 5409 0 0
256,512,1024,2048,4096
iov 0 0K 8K 39322K 76722 0 0
128,256,512,1024,2048,4096,8192
mount 14 7K 7K 39322K 14 0 0 128,1024
vnodes 1459 92K 92K 39322K 1474 0 0 16,64,128
UFS quota 1 16K 16K 39322K 1 0 0 16384
UFS mount 25 90K 90K 39322K 25 0 0
16,32,512,2048,4096,8192,16384,32768
shm 2 1K 1K 39322K 2 0 0 256,512
VM map 2 1K 1K 39322K 2 0 0 256
sem 2 1K 1K 39322K 2 0 0 32,64
dirhash 36 7K 7K 39322K 42 0 0 16,32,64,128,512
file desc 3 1K 2K 39322K 4 0 0 256,512
proc 17 5K 5K 39322K 17 0 0 32,512,2048
VFS cluster 0 0K 1K 39322K 6 0 0 32
NFS srvsock 1 1K 1K 39322K 1 0 0 64
NFS daemon 1 1K 1K 39322K 1 0 0 256
ip_moptions 6 1K 1K 39322K 6 0 0 16,64
in_multi 170 7K 7K 39322K 170 0 0 16,32,64
ether_multi 84 3K 3K 39322K 84 0 0 32
ISOFS mount 1 16K 16K 39322K 1 0 0 16384
MSDOSFS mount 1 8K 8K 39322K 1 0 0 8192
ttys 414 282K 282K 39322K 414 0 0
128,512,1024,8192
exec 0 0K 2K 39322K 4323 0 0 16,256,512,1024
pfkey data 15 1K 4K 39322K 125 0 0
16,32,64,128,256,1024,2048
tdb 19 17K 18K 39322K 30 0 0 128,256,1024
xform_data 32 1K 3K 39322K 94856 0 0 16,32,64
pagedep 1 4K 4K 39322K 1 0 0 4096
inodedep 1 16K 16K 39322K 1 0 0 16384
newblk 1 1K 1K 39322K 1 0 0 256
VM swap 7 291K 291K 39322K 7 0 0
32,128,2048,131072,262144
UVM amap 42395 753K 1038K 39322K 269619 0 0
16,32,64,128,256,512,1024,2048,4096
UVM aobj 2 2K 2K 39322K 2 0 0 16,1024
USB 31 7K 7K 39322K 31 0 0
16,32,64,128,256,4096
USB device 7 1K 1K 39322K 7 0 0 16,64,256
memdesc 1 4K 4K 39322K 1 0 0 4096
crypto data 82 15K 15K 39322K 82 0 0 32,128,512,1024
IPsec creds 96 24K 24K 39322K 96 0 0 64,256,1024
packet tags 0 0K 2K 39322K 141982 0 0 64
NDP 33 3K 4K 39322K 43 0 0 64,128
temp 191 23K 45K 39322K 173152 0 0
16,32,64,128,256,512,1024,2048,4096,8192
NTFS hash 1 16K 16K 39322K 1 0 0 16384
Memory Totals: In Use Free Requests
16231K 371K 2203516
Memory resource pool statistics
Name Size Requests Fail InUse Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
extentpl 20 309 0 92 1 0 1 1 0 8 0
phpool 52 477 0 464 7 0 7 7 0 8 0
pmappl 72 4371 0 36 1 0 1 1 0 8 0
vmsppl 180 4371 0 36 3 0 3 3 0 8 1
vmmpepl 88 454218 0 42937 1163 6 1157 1157 0 183 0
vmmpekpl 88 16811 0 82 3 0 3 3 0 8 1
aobjpl 40 1 0 1 1 0 1 1 0 8 0
dma512 512 4 0 0 1 1 0 1 0 8 0
amappl 44 266443 0 42367 570 3 567 567 0 46 0
anonpl 12 418922 0 43846 187 2 185 185 0 48 24
bufpl 172 55902 0 2967 136 0 136 136 0 8 0
mbpl 256 5484525 0 216 91 0 91 91 1 384 75
mcl2k 2048 1817221 0 160 197 0 197 197 4 3072 115
sockpl 272 2443 0 169 13 1 12 12 0 8 0
procpl 344 4384 0 49 6 0 6 6 0 8 1
processpl 68 4384 0 49 2 0 2 2 0 8 1
zombiepl 72 4335 0 0 2 1 1 1 0 8 1
ucredpl 80 4978 0 33 1 0 1 1 0 8 0
pgrppl 24 809 0 26 1 0 1 1 0 8 0
sessionpl 48 687 0 25 1 0 1 1 0 8 0
pcredpl 20 4384 0 49 1 0 1 1 0 8 0
lockfpl 56 228 0 2 1 0 1 1 0 8 0
filepl 88 23684 0 226 6 0 6 6 0 8 0
fdescpl 300 4372 0 37 4 0 4 4 0 8 1
pipepl 72 4294 0 2 2 1 1 1 0 8 0
kqueuepl 192 23 0 6 1 0 1 1 0 8 0
knotepl 64 4995 0 31 1 0 1 1 0 8 0
sigapl 296 4371 0 36 4 0 4 4 0 8 1
wqtasks 20 9914 0 0 2 1 1 1 0 8 1
ifaddritempl 32 60 0 58 1 0 1 1 0 8 0
wdcspl 96 59252 0 0 2 1 1 1 0 8 1
pfiaddrpl 100 21 0 21 1 0 1 1 0 8 0
namei 1024 53325 0 0 4 1 3 3 0 8 3
vnodes 156 2005 0 2005 81 0 81 81 0 8 0
nchpl 88 3237 0 3093 69 0 69 69 0 8 0
ffsino 184 1992 0 1900 91 0 91 91 0 8 0
dino1pl 128 1992 0 1900 62 0 62 62 0 8 0
dirhash 1024 60 0 46 12 0 12 12 0 128 0
pfrulepl 1192 250 0 183 80 10 70 80 0 8 8
pfstatepl 220 167807 0 566 1629 1557 72 160 0 8 6
pfstatekeypl 72 167819 0 519 68 35 33 40 0 8 6
pfstateitempl 12 167473 0 519 7 0 7 7 0 8 2
pfruleitempl 12 96701 0 35 1 0 1 1 0 8 0
pfrktable 1252 52 0 26 15 0 15 15 0 8 0
pfrke_plain 92 165 0 165 4 0 4 4 0 8 0
pfosfpen 108 1400 0 700 30 11 19 19 0 8 0
pfosfp 28 820 0 410 3 0 3 3 0 8 0
pffrent 24 16 0 0 1 0 1 1 0 8 1
pffrag 80 8 0 0 1 0 1 1 0 20 1
cryptop 68 94824 0 0 1 0 1 1 0 8 1
cryptodesc 56 189648 0 0 1 0 1 1 0 8 1
rtentpl 116 399941 0 383424 11278 0 11278 11278 0 8 0
tcpcbpl 408 318 0 12 2 0 2 2 0 8 0
tcpqepl 16 11 0 0 1 0 1 1 0 8 1
synpl 192 372 0 0 1 0 1 1 0 8 1
plimitpl 148 675 0 22 1 0 1 1 0 8 0
inpcbpl 232 1108 0 112 8 1 7 7 0 8 0
pfsync 40 450 0 0 1 0 1 1 0 8 1
ipsec policy 216 16 0 16 1 0 1 1 0 8 0
In use 52232K, total allocated 57128K; utilization 91.4%
