Box running pf/ospf/bgp/relayd entered ddb with "Stopped at
cpu_idle_cycle+0xe: hlt".
Various of these seen in 'sh all pools':-
mcl9k: pool(0xd0a2df78:mcl9k): page inconsistency: page 0xd1cc6000; item
ordinal 0; addr 0xd1cc6018 (p 0xd1cc4000)
mcl9k: pool(0xd0a2df78:mcl9k): page inconsistency: page 0xd1cc2000; item
ordinal 0; addr 0xd1cc2014 (p 0xd1cc0000)
Various pieces of ddb output below followed by a dmesg.
Running i386 GENERIC from 9 Feb.
Anyone have ideas or requests for more info to collect if it recurs?
Thanks.
[-- MARK -- Tue Feb 21 07:00:00 2012 .. Tue Feb 21 18:00:00 2012 -- MARK --]
[-- Console down -- Tue Feb 21 18:05:52 2012]
[-- Console up -- Tue Feb 21 18:05:52 2012]
[-- MARK -- Tue Feb 21 19:00:00 2012 .. Tue Feb 21 21:00:00 2012 -- MARK --]
Stopped at cpu_idle_cycle+0xe: hlt
ddb> [read-only -- use ^E c ? for help]
[read-only -- use ^E c ? for help]
[read-only -- use ^E c ? for help]
[attach to reopen]
[bumped sthen@localhost]
ddb> show panic
the kernel did not panic
ddb> tr
cpu_idle_cycle(d0ae19a0) at cpu_idle_cycle+0xe
Bad frame pointer: 0xd0b97e28
ddb> ps
PID PPID PGRP UID S FLAGS WAIT COMMAND
11086 11860 11860 0 3 0x80 nanosleep newsyslog
11860 25649 11860 0 3 0x88 pause sh
25649 1142 1142 0 3 0x80 piperd cron
18671 1 18671 77 3 0x80 poll dhcpd
30544 1 30544 0 3 0x80 ttyin getty
14942 1 14942 0 3 0x80 ttyin getty
16786 1 16786 0 3 0x80 ttyin getty
16944 1 16944 0 3 0x80 ttyin getty
8025 1 8025 0 3 0x80 ttyin getty
26943 1 26943 0 3 0x80 ttyin getty
1142 1 1142 0 3 0x80 select cron
5690 1 5690 0 3 0x80 select sendmail
24136 1 24136 0 3 0x80 nanosleep sensorsd
3775 1 3775 535 3 0x80 nanosleep symon
5698 1 5698 99 3 0x80 poll sndiod
2672 1 2672 0 3 0x80 select inetd
5804 1 5804 71 3 0x80 kqread ftp-proxy
11338 3713 3713 89 3 0x80 kqread relayd
2489 3713 3713 89 3 0x80 kqread relayd
29211 3713 3713 89 3 0x80 kqread relayd
19896 3713 3713 89 3 0x80 kqread relayd
3713 8594 3713 89 3 0x80 kqread relayd
20059 8594 20059 89 3 0x80 kqread relayd
5952 8594 5952 89 3 0x80 kqread relayd
8594 1 8594 0 3 0x80 kqread relayd
30614 7638 7638 75 3 0x80 poll bgpd
28779 7638 7638 75 3 0x80 poll bgpd
7638 1 7638 0 3 0x80 poll bgpd
23530 12413 12413 85 3 0x80 kqread ospfd
4844 12413 12413 85 3 0x80 kqread ospfd
12413 1 12413 0 3 0x80 kqread ospfd
19399 21681 21681 91 3 0x80 kqread snmpd
21681 1 21681 0 3 0x80 kqread snmpd
29693 1 29693 0 3 0x80 select sshd
17309 11222 13566 83 3 0x80 poll ntpd
11222 13566 13566 83 3 0x80 poll ntpd
13566 1 13566 0 3 0x80 poll ntpd
7424 24405 24405 74 3 0x80 bpf pflogd
24405 1 24405 0 3 0x80 netio pflogd
5067 30685 30685 73 3 0x80 poll syslogd
30685 1 30685 0 3 0x80 netio syslogd
14 0 0 0 3 0x100200 aiodoned aiodoned
13 0 0 0 3 0x100200 syncer update
12 0 0 0 3 0x100200 cleaner cleaner
11 0 0 0 3 0x100200 reaper reaper
10 0 0 0 3 0x100200 pgdaemon pagedaemon
9 0 0 0 3 0x100200 bored crypto
8 0 0 0 3 0x100200 pftm pfpurge
7 0 0 0 3 0x100200 usbtsk usbtask
6 0 0 0 3 0x100200 usbatsk usbatsk
5 0 0 0 3 0x100200 acpi0 acpi0
4 0 0 0 3 0x100200 bored syswq
* 3 0 0 0 7 0x40100200 idle0
2 0 0 0 3 0x100200 kmalloc kmthread
1 0 1 0 3 0x80 wait init
0 -1 0 0 3 0x200 scheduler swapper
ddb> sh reg
ds 0x10
es 0x10
fs 0x20
gs 0
edi 0xd0ae19d4 cpu_info_primary+0x34
esi 0xd6d8b298 end+0x62a6c94
ebp 0xdc1b1f8c end+0xb6cd988
ebx 0
edx 0x1
ecx 0
eax 0
eip 0xd02008d2 cpu_idle_cycle+0xe
cs 0x8
eflags 0x246
esp 0xdc1b1f60 end+0xb6cd95c
ss 0x10
cpu_idle_cycle+0xe: hlt
ddb> sh uvmexp
Current UVM status:
pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12
255281 VM pages: 4744 active, 692 inactive, 0 wired, 232168 free
min 10% (25) anon, 10% (25) vnode, 5% (12) vtext
pages 0 anon, 0 vnode, 0 vtext
freemin=8509, free-target=11345, inactive-target=0, wired-max=85093
faults=76492848, traps=60500530, intrs=1089622887, ctxswitch=28674543 fpuswit
ch=793866
softint=1675263079, syscalls=215393823, swapins=0, swapouts=0, kmapent=50
fault counts:
noram=0, noanon=0, pgwait=0, pgrele=0
ok relocks(total)=27567(27567), anget(retries)=53422799(0), amapcopy=164175
45
neighbor anon/obj pg=2908271/47825150, gets(lock/unlock)=14004865/27567
cases: anon=47464331, anoncow=5958468, obj=12274522, prcopy=1730343, przero
=9065184
daemon and swap counts:
woke=0, revs=0, scans=0, obscans=0, anscans=0
busy=0, freed=0, reactivate=0, deactivate=0
pageouts=0, pending=0, nswget=0
nswapdev=1, nanon=0, nanonneeded=0 nfreeanon=0
swpages=66267, swpginuse=0, swpgonly=0 paging=0
kernel pointers:
objs(kern)=0xd0a04760
ddb> sh all pools
Name Size Requests Fail Releases Pgreq Pgrel Npage Hiwat Minpg Maxpg Idle
pfsync 40 85503 0 85503 1 0 1 1 0 8 1
inpcbpl 232 1120366 0 1120217 12 0 12 12 0 8 2
plimitpl 148 37635 0 37606 2 0 2 2 0 8 0
synpl 192 19651 0 19651 1 0 1 1 0 8 1
sackhlpl 20 679 0 679 1 0 1 1 0 8 1
tcpqepl 16 401617 0 401617 1 0 1 1 0 8 1
tcpcbpl 408 1081254 0 1081213 9 0 9 9 0 8 4
rtentpl 116 46189 0 45751 20 0 20 20 0 8 6
pffrag 80 4823 0 4823 1 0 1 1 0 20 1
pffrent 24 9730 0 9730 1 0 1 1 0 8 1
pfosfp 28 6150 0 5740 3 0 3 3 0 8 0
pfosfpen 108 10500 0 9800 173 154 19 19 0 8 0
pfrke_plain 92 5549 0 5065 41 20 21 26 0 8 8
pfrktable 1252 583 0 539 32 0 32 32 0 8 5
pfruleitempl 12 7679482 0 7678671 11 0 11 11 0 8 3
pfstateitempl 12 37705433 0 37703660 32 6 26 27 0 8 5
pfstatekeypl 72 38142525 0 38140752 32822 32726 96 163 0 8 8
pfstatepl 220 36635527 0 36633826 535680 535442 238 546 0 8 8
pfsnitempl 12 12898 0 12897 1 0 1 1 0 8 0
pfsrctrpl 124 6402 0 6401 1 0 1 1 0 8 0
pfrulepl 1192 10053 0 9002 3028 2637 391 726 0 8 8
dirhash 1024 9664 0 9445 98 0 98 98 0 128 42
dino1pl 128 1685640 0 1679725 191 0 191 191 0 8 0
ffsino 184 1685640 0 1679725 282 0 282 282 0 8 0
nchpl 88 1851502 0 1846704 107 0 107 107 0 8 0
vnodes 156 5926 0 0 238 0 238 238 0 8 0
namei 1024 5759590 0 5759590 3 0 3 3 0 8 3
scxspl 124 905304 0 905304 4 0 4 4 0 8 4
wdcspl 96 9 0 9 1 0 1 1 0 8 1
pfiaddrpl 100 11 0 10 1 0 1 1 0 8 0
ifaddritempl 32 191 0 4 2 0 2 2 0 8 0
wqtasks 20 194060 0 194060 1 0 1 1 0 8 1
sigapl 296 302227 0 302185 6 0 6 6 0 8 2
knotepl 64 5258371 0 5258262 3 0 3 3 0 8 1
kqueuepl 192 16 0 2 1 0 1 1 0 8 0
pipepl 72 380964 0 380961 1 0 1 1 0 8 0
fdescpl 300 302228 0 302185 6 0 6 6 0 8 2
filepl 88 3045329 0 3045064 9 0 9 9 0 8 2
lockfpl 56 3444 0 3442 1 0 1 1 0 8 0
pcredpl 20 302240 0 302185 1 0 1 1 0 8 0
sessionpl 48 52197 0 52173 1 0 1 1 0 8 0
pgrppl 24 52678 0 52651 1 0 1 1 0 8 0
ucredpl 80 245140 0 245106 2 0 2 2 0 8 1
zombiepl 72 302185 0 302185 1 0 1 1 0 8 1
processpl 88 302240 0 302185 2 0 2 2 0 8 0
procpl 332 302240 0 302185 8 0 8 8 0 8 3
sockpl 272 1260026 0 1259801 19 0 19 19 0 8 3
mcl9k 9216 51 0 17 42 0 42 42 8 6144 8
mcl9k: pool(0xd0a2df78:mcl9k): page inconsistency: page 0xd1cc6000; item
ordinal 0; addr 0xd1cc6018 (p 0xd1cc4000)
mcl9k: pool(0xd0a2df78:mcl9k): page inconsistency: page 0xd1cc2000; item
ordinal 0; addr 0xd1cc2014 (p 0xd1cc0000)
mcl9k: pool(0xd0a2df78:mcl9k): page inconsistency: page 0xd1cbe000; item
ordinal 0; addr 0xd1cbe010 (p 0xd1cbc000)
mcl9k: pool(0xd0a2df78:mcl9k): page inconsistency: page 0xd1cba000; item
ordinal 0; addr 0xd1cba00c (p 0xd1cb8000)
mcl9k: pool(0xd0a2df78:mcl9k): page inconsistency: page 0xd1cb6000; item
ordinal 0; addr 0xd1cb6008 (p 0xd1cb4000)
mcl9k: pool(0xd0a2df78:mcl9k): page inconsistency: page 0xd1cb2000; item
ordinal 0; addr 0xd1cb2004 (p 0xd1cb0000)
mcl2k 2048 1520482091 0 1520481971 158 0 158 158 4 3072 93
mbpl 256 62209509 0 62209271 37 0 37 37 1 384 21
bufpl 172 258607 0 255568 140 0 140 140 0 8 1
anonpl 12 16753995 0 16750498 18 0 18 18 0 48 3
amappl 44 13008470 0 13006867 37 0 37 37 0 46 14
dma512 512 3 0 3 1 0 1 1 0 8 1
dma256 256 8 0 8 1 0 1 1 0 8 1
dma64 64 88 0 88 1 0 1 1 0 8 1
dma16 16 8 0 7 1 0 1 1 0 8 0
aobjpl 40 1 0 0 1 0 1 1 0 8 0
vmmpekpl 88 1084818 0 1084735 4 0 4 4 0 8 2
vmmpepl 88 25488195 0 25485799 107 0 107 107 0 183 47
vmsppl 180 302227 0 302185 4 0 4 4 0 8 1
pmappl 72 302227 0 302185 2 0 2 2 0 8 1
extentpl 20 232 0 202 1 0 1 1 0 8 0
phpool 52 3488 0 2637 15 0 15 15 0 8 0
ddb> sh malloc
Type InUse MemUse HighUse Limit Requests Type Lim Kern Lim
devbuf 5504 3150K 3279K 39321K 201416 0 0
pcb 91 8K 10K 39321K 139677 0 0
routetbl 831 48K 70K 39321K 896657 0 0
ifaddr 403 82K 82K 39321K 434 0 0
sysctl 2 0K 0K 39321K 2 0 0
ioctlops 0 0K 4K 39321K 318794 0 0
iov 0 0K 0K 39321K 112936 0 0
mount 14 7K 7K 39321K 14 0 0
vnodes 1515 94K 94K 39321K 21009 0 0
UFS quota 1 16K 16K 39321K 1 0 0
UFS mount 25 79K 79K 39321K 25 0 0
shm 2 0K 0K 39321K 2 0 0
VM map 2 0K 0K 39321K 2 0 0
sem 2 0K 0K 39321K 2 0 0
dirhash 138 26K 43K 39321K 6267 0 0
ACPI 1425 85K 89K 39321K 5659 0 0
file 0 0K 2K 39321K 14 0 0
file desc 10 3K 3K 39321K 12 0 0
proc 20 5K 5K 39321K 20 0 0
VFS cluster 0 0K 0K 39321K 4357 0 0
NFS srvsock 1 0K 0K 39321K 1 0 0
NFS daemon 1 0K 0K 39321K 1 0 0
ip_moptions 23 1K 1K 39321K 27 0 0
in_multi 522 18K 18K 39321K 539 0 0
ether_multi 358 11K 11K 39321K 365 0 0
mrt 1 0K 0K 39321K 1 0 0
ISOFS mount 1 16K 16K 39321K 1 0 0
MSDOSFS mount 1 8K 8K 39321K 1 0 0
ttys 414 281K 281K 39321K 414 0 0
exec 0 0K 1K 39321K 296137 0 0
pfkey data 2 0K 0K 39321K 23 0 0
tdb 7 4K 5K 39321K 11 0 0
xform_data 4 0K 0K 39321K 4 0 0
pagedep 1 4K 4K 39321K 1 0 0
inodedep 1 16K 16K 39321K 1 0 0
newblk 1 0K 0K 39321K 1 0 0
VM swap 7 26K 26K 39321K 7 0 0
UVM amap 1642 139K 263K 39321K 13219569 0 0
UVM aobj 2 1K 1K 39321K 2 0 0
USB 12 1K 1K 39321K 12 0 0
USB device 3 0K 0K 39321K 3 0 0
memdesc 1 4K 4K 39321K 1 0 0
crypto data 1 1K 1K 39321K 1 0 0
packet tags 0 0K 0K 39321K 2802 0 0
ip6_options 0 0K 0K 39321K 2 0 0
NDP 90 8K 9K 39321K 114 0 0
temp 427 54K 66K 39321K 2663946 0 0
NTFS hash 1 16K 16K 39321K 1 0 0
OpenBSD 5.1 (GENERIC) #156: Thu Feb 9 13:28:45 MST 2012
[email protected]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Xeon(TM) CPU 2.80GHz ("GenuineIntel" 686-class) 2.79 GHz
cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,
CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,CNXT-ID,xTPR
real mem = 1073115136 (1023MB)
avail mem = 1045463040 (997MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 04/14/04, BIOS32 rev. 0 @ 0xffe90, SMBIO
S rev. 2.3 @ 0xfb320 (56 entries)
bios0: vendor Dell Computer Corporation version "A08" date 04/14/2004
bios0: Dell Computer Corporation PowerEdge 1750
acpi0 at bios0: rev 0
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP APIC SPCR
acpi0: wakeup devices RTC_(S5) PCI0(S5) PCI3(S5) PCI2(S5) PCI1(S5)
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 132MHz
cpu at mainbus0: not configured
ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 11, 16 pins
ioapic0: misconfigured as apic 0, remapped to apid 8
ioapic1 at mainbus0: apid 9 pa 0xfec01000, version 11, 16 pins
ioapic1: misconfigured as apic 0, remapped to apid 9
ioapic2 at mainbus0: apid 10 pa 0xfec02000, version 11, 16 pins
ioapic2: misconfigured as apic 0, remapped to apid 10
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 4 (PCI4)
acpiprt2 at acpi0: bus 3 (PCI3)
acpiprt3 at acpi0: bus 2 (PCI2)
acpiprt4 at acpi0: bus 1 (PCI1)
acpicpu0 at acpi0
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x1000 0xc9000/0x2200 0xcb800/0x1800 0
xcd000/0x1800 0xec000/0x4000!
ipmi at mainbus0 not configured
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "ServerWorks CNB20-HE Host (GC-LE)" rev 0x33
pchb1 at pci0 dev 0 function 1 "ServerWorks CNB20-HE Host (GC-LE)" rev 0x00
pci1 at pchb1 bus 1
pchb2 at pci0 dev 0 function 2 "ServerWorks CNB20-HE Host (GC-LE)" rev 0x00
pci2 at pchb2 bus 3
em0 at pci2 dev 6 function 0 "Intel PRO/1000MT (82546EB)" rev 0x01: apic 9 int
8, address 00:11:0a:62:92:68
em1 at pci2 dev 6 function 1 "Intel PRO/1000MT (82546EB)" rev 0x01: apic 9 int
9, address 00:11:0a:62:92:69
vga1 at pci0 dev 14 function 0 "ATI Rage XL" rev 0x27
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
piixpm0 at pci0 dev 15 function 0 "ServerWorks CSB5" rev 0x93: SMBus disabled
pciide0 at pci0 dev 15 function 1 "ServerWorks CSB5 IDE" rev 0x93: DMA
atapiscsi0 at pciide0 channel 1 drive 0
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <TEAC, CD-224E, K.9A> ATAPI 5/cdrom removable
cd0(pciide0:1:0): using PIO mode 4, DMA mode 2, Ultra-DMA mode 2
ohci0 at pci0 dev 15 function 2 "ServerWorks OSB4/CSB5 USB" rev 0x05: apic 8 i
nt 11, version 1.0, legacy support
pcib0 at pci0 dev 15 function 3 "ServerWorks CSB5 LPC" rev 0x00
pchb3 at pci0 dev 16 function 0 "ServerWorks CIOB-E" rev 0x12
pchb4 at pci0 dev 16 function 2 "ServerWorks CIOB-E" rev 0x12
pci3 at pchb4 bus 2
bge0 at pci3 dev 0 function 0 "Broadcom BCM5704C" rev 0x02, BCM5704 A2 (0x2002
): apic 9 int 0, address 00:0f:1f:67:54:a0
brgphy0 at bge0 phy 1: BCM5704 10/100/1000baseT PHY, rev. 0
bge1 at pci3 dev 0 function 1 "Broadcom BCM5704C" rev 0x02, BCM5704 A2 (0x2002
): apic 9 int 1, address 00:0f:1f:67:54:a2
brgphy1 at bge1 phy 1: BCM5704 10/100/1000baseT PHY, rev. 0
pchb5 at pci0 dev 17 function 0 "ServerWorks CIOB-X2 PCIX" rev 0x05
pchb6 at pci0 dev 17 function 2 "ServerWorks CIOB-X2 PCIX" rev 0x05
pci4 at pchb6 bus 4
ami0 at pci4 dev 3 function 0 "Dell PERC 4/Di Verde" rev 0x02: apic 9 int 2
ami0: Dell 14a, 32b, FW 412W, BIOS vH406, 128MB RAM
ami0: 2 channels, 0 FC loops, 1 logical drives
scsibus1 at ami0: 40 targets
sd0 at scsibus1 targ 0 lun 0: <AMI, Host drive #00, > SCSI2 0/direct fixed
sd0: 17280MB, 512 bytes/sector, 35389440 sectors
scsibus2 at ami0: 16 targets
safte0 at scsibus2 targ 6 lun 0: <PE/PV, 1x3 SCSI BP, 1.1> SCSI2 3/processor f
ixed
scsibus3 at ami0: 16 targets
usb0 at ohci0: USB revision 1.0
uhub0 at usb0 "ServerWorks OHCI root hub" rev 1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
mtrr: Pentium Pro MTRR support
vscsi0 at root
scsibus4 at vscsi0: 256 targets
softraid0 at root
scsibus5 at softraid0: 256 targets
root on sd0a (7232af9e27547fba.a) swap on sd0b dump on sd0b
Automatic boot in progress: starting file system checks.
