Hi, I got this weird hang on my server. I could ping it but ssh
and httpd were unresponsive.
bt trace and dmesg attached from serial console. Yay for serial
consoles on OpenBSD.
[halt sent]
Stopped at Debugger+0x5: leave
ddb{0}> ps
PID PPID PGRP UID S FLAGS WAIT COMMAND
18819 15214 18819 0 3 0x200000 inode sshd
9596 15214 9596 0 3 0x200000 inode sshd
26335 15214 26335 0 3 0x200000 inode sshd
11569 15214 11569 0 3 0x200000 inode sshd
30591 10750 10750 0 3 0x200000 inode master
13416 15214 13416 0 3 0x200000 inode sshd
10002 25204 26716 1000 3 0 inode sh
10951 25204 26716 1000 3 0 inode sh
12150 17962 17962 1000 3 0 inode sh
5231 17962 17962 1000 3 0 inode sh
12059 7433 12059 0 3 0 inode cron
7433 1522 1522 0 3 0x80 piperd cron
1216 29647 1216 1000 3 0x80 select ssh-agent
29647 19251 29647 1000 3 0x80 poll tmux
19251 22785 22785 1000 3 0 inode sshd
22785 15214 22785 0 3 0x80 poll sshd
6690 23658 6690 1000 3 0x80 select ssh-agent
23658 1 23658 1000 3 0x2080 ttyout tmux
8787 15214 8787 0 3 0 inode sshd
32356 32449 32449 507 3 0x80 nanosleep perl
32245 32449 32449 507 3 0x80 nanosleep perl
8031 9553 9553 67 3 0x80 kqread nginx
24243 9553 9553 67 3 0 inode nginx
19315 9553 9553 67 3 0 inode nginx
30796 9553 9553 67 3 0 inode nginx
14775 9553 9553 67 3 0 inode nginx
31193 19535 31193 1000 3 0x80 ttyin zsh
19535 18076 19535 1000 3 0x88 pause sh
19362 4815 4815 74 3 0x80 bpf pflogd
4815 1 4815 0 3 0x80 netio pflogd
5376 10750 10750 507 3 0x80 kqread qmgr
1619 32494 32494 67 3 0x80 poll php-fpm-5.3
23509 32494 32494 67 3 0x80 poll php-fpm-5.3
10761 32494 32494 67 3 0x80 poll php-fpm-5.3
32494 1 32494 0 3 0x80 poll php-fpm-5.3
23646 13534 13534 1000 3 0 inode mutt
13534 18076 13534 1000 3 0x88 pause sh
6713 1 6713 0 3 0x80 poll emacs-24.2.50
4585 1 4585 601 3 0x80 kqread unbound
18578 24276 24276 1000 3 0 inode rtorrent
24276 18076 24276 1000 3 0x88 pause sh
8042 6286 6286 67 3 0x80 netcon httpd
29190 6286 6286 67 3 0x80 netcon httpd
6727 6286 6286 67 3 0x80 netcon httpd
6638 6286 6286 67 3 0x80 netcon httpd
7280 6286 6286 67 3 0x80 netcon httpd
6286 1 6286 0 3 0x80 select httpd
9641 32013 32013 1000 4 0x8000000 emacsclient
32013 30562 32013 1000 4 0x8001088 sh
28064 1 28064 1000 3 0x80 poll emacs-24.2.50
24930 16156 16156 250 3 0 inode bitlbee
16156 1 16156 250 3 0x80 poll bitlbee
26754 3350 3350 1000 3 0x80 poll irssi
3350 18076 3350 1000 3 0x88 pause sh
18773 1 18773 539 3 0 inode clamd
12322 14129 12322 1000 3 0x80 ttyin zsh
30562 24979 30562 1000 3 0x80 ttyin zsh
19651 15004 19651 1000 3 0x80 ttyin zsh
6185 2796 6185 1000 3 0x80 ttyin zsh
14129 18076 14129 1000 3 0x88 pause sh
15004 18076 15004 1000 3 0x88 pause sh
24979 18076 24979 1000 3 0x88 pause sh
2796 18076 2796 1000 3 0x88 pause sh
18076 1 18076 1000 3 0 inode tmux
15038 9462 20303 1000 3 0x80 select mpc
10750 1 10750 0 3 0x80 kqread master
9553 1 9553 0 3 0x88 pause nginx
15147 31127 31127 70 3 0x80 select named
31127 1 31127 0 3 0x80 netio named
21619 1 21619 0 3 0 inode getty
9462 1 20303 1000 3 0x88 pause sh
14145 1 14145 0 3 0x80 select rsync
15214 1 15214 0 3 0x80 select sshd
17962 1 17962 1000 3 0x80 piperd sh
20302 30868 26716 1000 3 0x80 piperd sh
30403 30868 26716 1000 3 0x80 kqread tail
30868 1 26716 1000 3 0x88 pause sh
25204 1 26716 1000 3 0x80 piperd sh
22792 4269 2917 83 3 0x80 poll ntpd
4269 2917 2917 83 3 0x80 poll ntpd
2917 1 2917 0 3 0x80 poll ntpd
20765 26845 20765 0 3 0x80 poll syslog-ng
26845 1 17854 0 3 0x80 wait syslog-ng
1522 1 1522 0 3 0 inode cron
26641 1 26641 0 3 0x80 ttyin getty
9435 1 9435 0 3 0x80 ttyin getty
19690 1 19690 0 3 0x80 ttyin getty
9245 1 9245 0 3 0x80 ttyin getty
15782 1 15782 0 3 0x80 ttyin getty
32449 1 32449 507 3 0x80 wait perl
28704 1 28704 569 3 0x80 poll dkim-filter
32274 1 32274 560 3 0x80 poll mpd
12412 13936 8620 502 3 0x80 poll mysqld
13936 1 8620 0 3 0x88 pause sh
7106 1 7106 1001 3 0x80 poll openvpn
7503 8867 8867 62 3 0x80 piperd spamd
17788 1 17788 62 3 0x80 bpf spamlogd
8159 8867 8867 62 3 0x80 select spamd
8867 1 8867 62 3 0 inode spamd
31779 1 31779 0 3 0x80 netcon vsftpd
9864 1 9864 0 3 0x80 netcon vsftpd
18563 11213 11213 0 3 0x80 nfsd nfsd
2756 11213 11213 0 3 0x80 nfsd nfsd
20830 11213 11213 0 3 0x80 nfsd nfsd
20795 11213 11213 0 3 0x80 nfsd nfsd
11213 1 11213 0 3 0x80 netcon nfsd
23780 1 23780 0 3 0x80 select mountd
15095 1 15095 28 3 0x80 poll portmap
20 0 0 0 3 0x100200 aiodoned aiodoned
19 0 0 0 3 0x100200 syncer update
18 0 0 0 3 0x100200 cleaner cleaner
17 0 0 0 3 0x100200 reaper reaper
16 0 0 0 3 0x100200 pgdaemon pagedaemon
15 0 0 0 3 0x100200 bored crypto
14 0 0 0 3 0x100200 pftm pfpurge
13 0 0 0 3 0x100200 usbtsk usbtask
12 0 0 0 3 0x100200 usbatsk usbatsk
11 0 0 0 3 0x100200 acpi0 acpi0
10 0 0 0 7 0x40100200 idle7
9 0 0 0 7 0x40100200 idle6
8 0 0 0 7 0x40100200 idle5
7 0 0 0 7 0x40100200 idle4
6 0 0 0 7 0x40100200 idle3
5 0 0 0 7 0x40100200 idle2
4 0 0 0 7 0x40100200 idle1
3 0 0 0 3 0x100200 bored syswq
* 2 0 0 0 7 0x40100200 idle0
1 0 1 0 3 0x80 wait init
0 -1 0 0 3 0x200 scheduler swapper
3839 8787 8787 1000 5 0x2000 sshd
ddb{0}> trace
Debugger() at Debugger+0x5
comintr() at comintr+0x268
Xintr_ioapic_edge4() at Xintr_ioapic_edge4+0xe8
--- interrupt ---
Bad frame pointer: 0xffff800022151f10
end trace frame: 0xffff800022151f10, count: -3
cpu_idle_cycle+0x13:
ddb{0}> boot reboot
rebooting...
>> OpenBSD/amd64 BOOT 3.18ies Ltd.
boot> ghts Reserved
booting hd0a:/bsd: 5669864+1601484+935608+0+617568 [89+499848+323884]=0xd351b8
entry point at 0x1001e0 [7205c766, 34000004, 24448b12, 3cb8a304]
[ using 824664 bytes of bsd ELF symbol table ]or = 4
Copyright (c) 1982, 1986, 1989, 1991, 199333GHz
Intel(R)The Regents of the University of California. All rights reserved.
Copyright (c) 1995-2012 OpenBSD. All rights reserved. http://www.OpenBSD.org
OpenBSD 5.1 (GENERIC.MP) #207: Sun Feb 12 09:42:14 MST 2012
[email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 4292935680 (4094MB)
avail mem = 4164493312 (3971MB)
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.31 @ 0xdc010 (57 entries)
bios0: vendor HP version "O08" date 06/03/2009
bios0: HP ProLiant DL140 G3
acpi0 at bios0: rev 0
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP SPMI APIC MCFG BOOT SPCR SSDT
acpi0: wakeup devices BPD0(S5) BMF3(S5) P0P4(S5) P0P6(S5) PEX0(S5) PEX1(S5)
PEX2(S5) PEX3(S5) USB1(S5) USB2(S5) USB3(S5) EUSB(S5) PCIB(S5)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.81 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
cpu0: 4MB 64b/line 16-way L2 cache
cpu0: apic clock running at 332MHz
cpu1 at mainbus0: apid 4 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.51 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
cpu1: 4MB 64b/line 16-way L2 cache
cpu2 at mainbus0: apid 1 (application processor)
cpu2: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
cpu2: 4MB 64b/line 16-way L2 cache
cpu3 at mainbus0: apid 5 (application processor)
cpu3: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
cpu3: 4MB 64b/line 16-way L2 cache
cpu4 at mainbus0: apid 2 (application processor)
cpu4: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
cpu4:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
cpu4: 4MB 64b/line 16-way L2 cache
cpu5 at mainbus0: apid 6 (application processor)
cpu5: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
cpu5:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
cpu5: 4MB 64b/line 16-way L2 cache
cpu6 at mainbus0: apid 3 (application processor)
cpu6: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
cpu6:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
cpu6: 4MB 64b/line 16-way L2 cache
cpu7 at mainbus0: apid 7 (application processor)
cpu7: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
cpu7:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
cpu7: 4MB 64b/line 16-way L2 cache
ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
ioapic1 at mainbus0: apid 9 pa 0xfec80000, version 20, 24 pins
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-31
acpiprt0 at acpi0: bus 1 (P0P2)
acpiprt1 at acpi0: bus 2 (BMD0)
acpiprt2 at acpi0: bus 3 (BPD0)
acpiprt3 at acpi0: bus -1 (BPD1)
acpiprt4 at acpi0: bus -1 (BPD2)
acpiprt5 at acpi0: bus 11 (BMF3)
acpiprt6 at acpi0: bus 16 (P0P4)
acpiprt7 at acpi0: bus 18 (P0P6)
acpiprt8 at acpi0: bus 0 (PCI0)
acpiprt9 at acpi0: bus 30 (PEX0)
acpiprt10 at acpi0: bus 31 (PEX1)
acpiprt11 at acpi0: bus -1 (PEX2)
acpiprt12 at acpi0: bus -1 (PEX3)
acpiprt13 at acpi0: bus 32 (PCIB)
acpicpu0 at acpi0
acpicpu1 at acpi0
acpicpu2 at acpi0
acpicpu3 at acpi0
acpicpu4 at acpi0
acpicpu5 at acpi0
acpicpu6 at acpi0
acpicpu7 at acpi0
acpibtn0 at acpi0: PWRB
ipmi at mainbus0 not configured
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 5000X Host" rev 0x31
ppb0 at pci0 dev 2 function 0 "Intel 5000 PCIE x8" rev 0x31
pci1 at ppb0 bus 1
ppb1 at pci1 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
pci2 at ppb1 bus 2
ppb2 at pci2 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
pci3 at ppb2 bus 3
ppb3 at pci1 dev 0 function 3 "Intel 6321ESB PCIE-PCIX" rev 0x01
pci4 at ppb3 bus 11
mpi0 at pci4 dev 1 function 0 "Symbios Logic SAS1068" rev 0x01: msi
scsibus0 at mpi0: 112 targets
sd0 at scsibus0 targ 1 lun 0: <LSILOGIC, Logical Volume, 3000> SCSI2 0/direct
fixed naa.600508e000000000f37fc1e28b72c109
sd0: 1907348MB, 512 bytes/sector, 3906248704 sectors
ppb4 at pci0 dev 3 function 0 "Intel 5000 PCIE" rev 0x31
pci5 at ppb4 bus 12
ppb5 at pci0 dev 4 function 0 "Intel 5000 PCIE x16" rev 0x31: msi
pci6 at ppb5 bus 16
ppb6 at pci0 dev 5 function 0 "Intel 5000 PCIE" rev 0x31: msi
pci7 at ppb6 bus 17
ppb7 at pci0 dev 6 function 0 "Intel 5000 PCIE" rev 0x31: msi
pci8 at ppb7 bus 18
ppb8 at pci0 dev 7 function 0 "Intel 5000 PCIE" rev 0x31: msi
pci9 at ppb8 bus 19
pchb1 at pci0 dev 16 function 0 "Intel 5000 Error Reporting" rev 0x31
pchb2 at pci0 dev 16 function 1 "Intel 5000 Error Reporting" rev 0x31
pchb3 at pci0 dev 16 function 2 "Intel 5000 Error Reporting" rev 0x31
pchb4 at pci0 dev 17 function 0 "Intel 5000 Reserved" rev 0x31
pchb5 at pci0 dev 19 function 0 "Intel 5000 Reserved" rev 0x31
pchb6 at pci0 dev 21 function 0 "Intel 5000 FBD" rev 0x31
pchb7 at pci0 dev 22 function 0 "Intel 5000 FBD" rev 0x31
ppb9 at pci0 dev 28 function 0 "Intel 6321ESB PCIE" rev 0x09: msi
pci10 at ppb9 bus 30
bge0 at pci10 dev 0 function 0 "Broadcom BCM5721" rev 0x11, BCM5750 B1
(0x4101): apic 8 int 16, address 00:1a:4b:d4:1c:3e
brgphy0 at bge0 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
ppb10 at pci0 dev 28 function 1 "Intel 6321ESB PCIE" rev 0x09: msi
pci11 at ppb10 bus 31
bge1 at pci11 dev 0 function 0 "Broadcom BCM5721" rev 0x11, BCM5750 B1
(0x4101): apic 8 int 17, address 00:1a:4b:d4:1c:3f
brgphy1 at bge1 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
uhci0 at pci0 dev 29 function 0 "Intel 6321ESB USB" rev 0x09: apic 8 int 23
uhci1 at pci0 dev 29 function 1 "Intel 6321ESB USB" rev 0x09: apic 8 int 23
uhci2 at pci0 dev 29 function 2 "Intel 6321ESB USB" rev 0x09: apic 8 int 23
ehci0 at pci0 dev 29 function 7 "Intel 6321ESB USB" rev 0x09: apic 8 int 23
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb11 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xd9
pci12 at ppb11 bus 32
vga1 at pci12 dev 2 function 0 "Matrox MGA G200e (ServerEngines)" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 31 function 0 "Intel 6321ESB LPC" rev 0x09
pciide0 at pci0 dev 31 function 1 "Intel 6321ESB IDE" rev 0x09: DMA, channel 0
configured to compatibility, channel 1 configured to compatibility
pciide0: channel 0 disabled (no drives)
pciide0: channel 1 ignored (disabled)
ichiic0 at pci0 dev 31 function 3 "Intel 6321ESB SMBus" rev 0x09: apic 8 int 19
iic0 at ichiic0
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
mtrr: Pentium Pro MTRR support
uhidev0 at uhub3 port 1 configuration 1 interface 0 "ServerEngines SE USB
Device" rev 1.10/0.01 addr 2
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 modifier keys, 6 key codes
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
uhidev1 at uhub3 port 1 configuration 1 interface 1 "ServerEngines SE USB
Device" rev 1.10/0.01 addr 2
uhidev1: iclass 3/1
ums0 at uhidev1: 8 buttons, Z dir
wsmouse0 at ums0 mux 0
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root on sd0a (42c44955279fd28c.a) swap on sd0b dump on sd0b
WARNING: / was not properly unmounted
Automatic boot in progress: starting file system checks.
# Han
