Hi all,
I have a server that I'm trying to get on-line that continues to drop
into ddb with a bad frame pointer crash. Right now it is only running
an hourly rsync from the server it is to replace on an hourly basis.
The server will run for no more than a week without this happening but
sometimes it will crash after only a day.
I filed a PR 6593 on Apr 26 on this. Right now it is running i386
-current from Apr 27, upgraded after I filed the PR.
The message I get at the ddb prompt is virtually identical to the one below.
If anyone has any ideas on how I can further debug this, I'd like to
know. I was hoping to use the upcoming 3 day weekend to make the final
switch to this new server but I can't see how that will happen if I
can't keep it running.
Trace, ps and dmesg below.
Thanks,
Jeff Ross
jross@mail:/home/jross $ sudo cu -l /dev/cua00
Password:
Connected
ddb{2}> show panic
the kernel did not panic
ddb{2}> trace
handle_workitem_freeblocks(d9d2ee78,e2ac1860,e0352f2c,d03e1c95) at
handle_worki
tem_freeblocks+0x2b
process_worklist_item(0,0,e0352f5c,d03e1eff,e0352f44) at
process_worklist_item+
0x171
softdep_process_worklist(0,28,d08c4bbb,0,d03d2ec3) at
softdep_process_worklist+
0x13f
sched_sync(daea43b8) at sched_sync+0xe5
Bad frame pointer: 0xd0ba9e88
ddb{2}> ps
PID PPID PGRP UID S FLAGS WAIT COMMAND
16869 14870 18539 0 3 0x2000080 nanosleep sleep
14870 18539 18539 0 3 0x2000080 pause sh
18539 16229 18539 0 3 0x2000080 pause sh
16229 17210 17210 0 3 0x2000080 piperd cron
5904 1 5904 0 3 0x2000080 ttyin getty
11510 1 11510 0 3 0x2000080 ttyin getty
23042 1 23042 0 3 0x2000080 ttyin getty
32176 1 32176 0 3 0x2000080 ttyin getty
14619 1 14619 0 3 0x2000080 ttyin getty
4932 1 4932 0 3 0x2000080 ttyin getty
21200 1 21200 0 3 0x2000080 poll wsmoused
17210 1 17210 0 3 0x2000080 select cron
25672 1 25672 0 3 0x2080080 nanosleep sensorsd
12854 1 12854 62 3 0x2000080 bpf spamlogd
9266 505 505 62 3 0x2000080 piperd spamd
14987 505 505 62 3 0x2000080 select spamd
505 1 505 62 3 0x2000080 nanosleep spamd
22372 1 22372 0 3 0x2000080 select inetd
18959 1 18959 0 3 0x2000080 select sshd
29919 0 0 0 3 0x2100280 nfsidl nfsio
19624 0 0 0 3 0x2100280 nfsidl nfsio
24472 0 0 0 3 0x2100280 nfsidl nfsio
8671 0 0 0 3 0x2100280 nfsidl nfsio
14655 16745 5437 83 3 0x2000080 poll ntpd
16745 5437 5437 83 3 0x2000080 poll ntpd
5437 1 5437 0 3 0x2000080 poll ntpd
20010 18817 18817 74 3 0x2000080 bpf pflogd
18817 1 18817 0 3 0x2000080 netio pflogd
22503 11789 22334 7794 3 0x2000080 piperd qmail-clean
2383 11789 22334 7795 3 0x2000080 select qmail-rspawn
16466 11789 22334 0 3 0x2000080 select qmail-lspawn
17980 9726 22334 7792 3 0x2000080 piperd multilog
28441 26724 22334 73 3 0x2000080 piperd multilog
5841 2589 22334 1001 3 0x2000080 poll dnscache
28883 21258 22334 1002 3 0x2000080 piperd multilog
11789 9610 22334 7796 3 0x2000080 select qmail-send
23511 2376 22334 73 3 0x2000080 netio socklog
11206 28517 22334 73 3 0x2000080 piperd multilog
26724 23627 22334 0 3 0x2000080 poll supervise
15592 23627 22334 0 3 0x2000080 poll supervise
28517 23627 22334 0 3 0x2000080 poll supervise
2376 23627 22334 0 3 0x2000080 poll supervise
9726 23627 22334 0 3 0x2000080 poll supervise
9610 23627 22334 0 3 0x2000080 poll supervise
21258 23627 22334 0 3 0x2000080 poll supervise
2589 23627 22334 0 3 0x2000080 poll supervise
5024 22334 22334 0 3 0x2000080 piperd readproctitle
23627 22334 22334 0 3 0x2000080 nanosleep svscan
22334 1 22334 0 3 0x2000080 pause sh
17 0 0 0 3 0x2100200 aiodoned aiodoned
* 16 0 0 0 7 0x2100200 update
15 0 0 0 3 0x2100200 cleaner cleaner
14 0 0 0 3 0x100200 reaper reaper
13 0 0 0 3 0x2100200 pgdaemon pagedaemon
12 0 0 0 3 0x2100200 bored crypto
11 0 0 0 3 0x2100200 pftm pfpurge
10 0 0 0 3 0x2100200 usbtsk usbtask
9 0 0 0 3 0x2100200 usbatsk usbatsk
8 0 0 0 3 0x2100200 acpi0 acpi0
7 0 0 0 7 0x40100200 idle3
6 0 0 0 3 0x40100200 idle2
5 0 0 0 7 0x40100200 idle1
4 0 0 0 3 0x2100200 bored syswq
3 0 0 0 7 0x40100200 idle0
2 0 0 0 3 0x2100200 kmalloc kmthread
1 0 1 0 3 0x2000080 wait init
0 -1 0 0 3 0x2080200 scheduler swapper
20563 15592 22334 503 5 0x2000 setuidgid
OpenBSD 4.9-current (GENERIC.MP) #100: Wed Apr 27 07:21:06 MDT 2011
[email protected]:/usr/src/sys/arch/i386/compile/GENERIC.MP
cpu0: Intel(R) Xeon(TM) CPU 2.40GHz ("GenuineIntel" 686-class) 2.41 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CF
LUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,CNXT-ID,xTPR
real mem = 4160253952 (3967MB)
avail mem = 4082049024 (3892MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 09/18/03, BIOS32 rev. 0 @ 0xf0010,
SMBIOS
rev. 2.3 @ 0xf84b0 (39 entries)
bios0: vendor American Megatrends Inc. version "080009" date 09/18/2003
bios0: Supermicro X5DPA
acpi0 at bios0: rev 2
acpi0: sleep states S0 S1 S5
acpi0: tables DSDT FACP APIC OEMB
acpi0: wakeup devices PS2K(S1) PS2M(S1) SMBS(S1) AUDI(S1) MODM(S1)
USB0(S1) USB1
(S1) USB2(S1) P0P1(S1) GLN1(S1) GLN2(S1) BT64(S1) PWRB(S1) SLPB(S1)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 133MHz
cpu1 at mainbus0: apid 6 (application processor)
cpu1: Intel(R) Xeon(TM) CPU 2.40GHz ("GenuineIntel" 686-class) 2.41 GHz
cpu1:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CF
LUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,CNXT-ID,xTPR
cpu2 at mainbus0: apid 1 (application processor)
cpu2: Intel(R) Xeon(TM) CPU 2.40GHz ("GenuineIntel" 686-class) 2.41 GHz
cpu2:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CF
LUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,CNXT-ID,xTPR
cpu3 at mainbus0: apid 7 (application processor)
cpu3: Intel(R) Xeon(TM) CPU 2.40GHz ("GenuineIntel" 686-class) 2.41 GHz
cpu3:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CF
LUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,CNXT-ID,xTPR
ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
ioapic1 at mainbus0: apid 9 pa 0xfec80000, version 20, 24 pins
ioapic2 at mainbus0: apid 10 pa 0xfec80400, version 20, 24 pins
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (P0P1)
acpiprt2 at acpi0: bus 3 (P2P3)
acpiprt3 at acpi0: bus 4 (BT64)
acpicpu0 at acpi0
acpicpu1 at acpi0acpibtn0 at acpi0: PWRB
acpibtn1 at acpi0: SLPB
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x1400 0xc9800/0x1000 0xca800/0x1000
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "Intel E7501 Host" rev 0x01
ppb0 at pci0 dev 2 function 0 "Intel E7500 PCI" rev 0x01
pci1 at ppb0 bus 2
"Intel 82870P2 IOxAPIC" rev 0x04 at pci1 dev 28 function 0 not configured
ppb1 at pci1 dev 29 function 0 "Intel 82870P2 PCIX-PCIX" rev 0x04
pci2 at ppb1 bus 4
ami0 at pci2 dev 2 function 0 "Symbios Logic MegaRAID" rev 0x01: apic 10
int 4
ami0: LSI 520, 64b/lhc, FW 1L26, BIOS vG112, 64MB RAM
ami0: 1 channels, 0 FC loops, 3 logical drives
scsibus0 at ami0: 40 targets
sd0 at scsibus0 targ 0 lun 0: <AMI, Host drive #00, > SCSI2 0/direct fixed
sd0: 35002MB, 512 bytes/sec, 71684096 sec total
sd1 at scsibus0 targ 1 lun 0: <AMI, Host drive #01, > SCSI2 0/direct fixed
sd1: 35002MB, 512 bytes/sec, 71684096 sec total
sd2 at scsibus0 targ 2 lun 0: <AMI, Host drive #02, > SCSI2 0/direct fixed
sd2: 70005MB, 512 bytes/sec, 143370240 sec total
scsibus1 at ami0: 16 targets
"Intel 82870P2 IOxAPIC" rev 0x04 at pci1 dev 30 function 0 not configured
ppb2 at pci1 dev 31 function 0 "Intel 82870P2 PCIX-PCIX" rev 0x04
pci3 at ppb2 bus 3
em0 at pci3 dev 1 function 0 "Intel PRO/1000MT (82541GI)" rev 0x00: apic
9 int 4
, address 00:30:48:74:2c:b8
em1 at pci3 dev 2 function 0 "Intel PRO/1000MT (82541GI)" rev 0x00: apic
9 int 5
, address 00:30:48:74:2c:b9
uhci0 at pci0 dev 29 function 0 "Intel 82801CA/CAM USB" rev 0x02: apic 8
int 16
uhci1 at pci0 dev 29 function 1 "Intel 82801CA/CAM USB" rev 0x02: apic 8
int 19
uhci2 at pci0 dev 29 function 2 "Intel 82801CA/CAM USB" rev 0x02: apic 8
int 18
ppb3 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x42
pci4 at ppb3 bus 1
vga1 at pci4 dev 4 function 0 "ATI Rage XL" rev 0x27
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ichpcib0 at pci0 dev 31 function 0 "Intel 82801CA LPC" rev 0x02
pciide0 at pci0 dev 31 function 1 "Intel 82801CA IDE" rev 0x02: DMA,
channel 0 c
onfigured to compatibility, channel 1 configured to compatibility
atapiscsi0 at pciide0 channel 0 drive 1
scsibus2 at atapiscsi0: 2 targetsichiic0 at pci0 dev 31 function 3
"Intel 82801CA/CAM SMBus" rev 0x02: apic 8 int
17
iic0 at ichiic0
admtemp0 at iic0 addr 0x18: Xeon
admtemp1 at iic0 addr 0x1a: Xeon
lm1 at iic0 addr 0x2d: W83627HF
spdmem0 at iic0 addr 0x51: 1GB DDR SDRAM registered ECC PC2100CL2.5
spdmem1 at iic0 addr 0x52: 1GB DDR SDRAM registered ECC PC2100CL2.5
spdmem2 at iic0 addr 0x55: 1GB DDR SDRAM registered ECC PC2100CL2.5
spdmem3 at iic0 addr 0x56: 1GB DDR SDRAM registered ECC PC2100CL2.5
usb0 at uhci0: USB revision 1.0
uhub0 at usb0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb1 at uhci1: USB revision 1.0
uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb2 at uhci2: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at ichpcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
wbsio0 at isa0 port 0x2e/2: W83627HF rev 0x17
lm2 at wbsio0 port 0x290/8: W83627HF
lm1 detached
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
mtrr: Pentium Pro MTRR support
umass0 at uhub0 port 1 configuration 1 interface 0 "Imation Nano Pro"
rev 2.00/1
.00 addr 2
umass0: using SCSI over Bulk-Only
scsibus3 at umass0: 2 targets, initiator 0
sd3 at scsibus3 targ 1 lun 0: <Imation, Nano Pro, PMAP> SCSI0 0/direct
removable
serial.071806381C036D10AFE9
sd3: 3822MB, 512 bytes/sec, 7827456 sec total
umass1 at uhub0 port 2 configuration 1 interface 0 "Imation Nano Pro"
rev 2.00/1
.00 addr 3
umass1: using SCSI over Bulk-Only
scsibus4 at umass1: 2 targets, initiator 0
sd4 at scsibus4 targ 1 lun 0: <Imation, Nano Pro, PMAP> SCSI0 0/direct
removable
serial.071806381A03C7141B3D
sd4: 3822MB, 512 bytes/sec, 7827456 sec total
vscsi0 at root
scsibus5 at vscsi0: 256 targets
softraid0 at root
root on sd3a swap on sd3b dump on sd3b
WARNING: / was not properly unmounted
