On 2012/10/25 20:17, Han Boetes wrote:
> Hi, I got this weird hang on my server. I could ping it but ssh
> and httpd were unresponsive.
>
> bt trace and dmesg attached from serial console. Yay for serial
> consoles on OpenBSD.
>
> [halt sent]
> Stopped at Debugger+0x5: leave
> ddb{0}> ps
> PID PPID PGRP UID S FLAGS WAIT COMMAND
> 18819 15214 18819 0 3 0x200000 inode sshd
all processes wanting disk i/o stuck with waitchan inode - I've seen
that before as have a few others I think - I haven't hit it myself on
-current for months now though.
> 9596 15214 9596 0 3 0x200000 inode sshd
> 26335 15214 26335 0 3 0x200000 inode sshd
> 11569 15214 11569 0 3 0x200000 inode sshd
> 30591 10750 10750 0 3 0x200000 inode master
> 13416 15214 13416 0 3 0x200000 inode sshd
> 10002 25204 26716 1000 3 0 inode sh
> 10951 25204 26716 1000 3 0 inode sh
> 12150 17962 17962 1000 3 0 inode sh
> 5231 17962 17962 1000 3 0 inode sh
> 12059 7433 12059 0 3 0 inode cron
> 7433 1522 1522 0 3 0x80 piperd cron
> 1216 29647 1216 1000 3 0x80 select ssh-agent
> 29647 19251 29647 1000 3 0x80 poll tmux
> 19251 22785 22785 1000 3 0 inode sshd
> 22785 15214 22785 0 3 0x80 poll sshd
> 6690 23658 6690 1000 3 0x80 select ssh-agent
> 23658 1 23658 1000 3 0x2080 ttyout tmux
> 8787 15214 8787 0 3 0 inode sshd
> 32356 32449 32449 507 3 0x80 nanosleep perl
> 32245 32449 32449 507 3 0x80 nanosleep perl
> 8031 9553 9553 67 3 0x80 kqread nginx
> 24243 9553 9553 67 3 0 inode nginx
> 19315 9553 9553 67 3 0 inode nginx
> 30796 9553 9553 67 3 0 inode nginx
> 14775 9553 9553 67 3 0 inode nginx
> 31193 19535 31193 1000 3 0x80 ttyin zsh
> 19535 18076 19535 1000 3 0x88 pause sh
> 19362 4815 4815 74 3 0x80 bpf pflogd
> 4815 1 4815 0 3 0x80 netio pflogd
> 5376 10750 10750 507 3 0x80 kqread qmgr
> 1619 32494 32494 67 3 0x80 poll php-fpm-5.3
> 23509 32494 32494 67 3 0x80 poll php-fpm-5.3
> 10761 32494 32494 67 3 0x80 poll php-fpm-5.3
> 32494 1 32494 0 3 0x80 poll php-fpm-5.3
> 23646 13534 13534 1000 3 0 inode mutt
> 13534 18076 13534 1000 3 0x88 pause sh
> 6713 1 6713 0 3 0x80 poll emacs-24.2.50
> 4585 1 4585 601 3 0x80 kqread unbound
> 18578 24276 24276 1000 3 0 inode rtorrent
> 24276 18076 24276 1000 3 0x88 pause sh
> 8042 6286 6286 67 3 0x80 netcon httpd
> 29190 6286 6286 67 3 0x80 netcon httpd
> 6727 6286 6286 67 3 0x80 netcon httpd
> 6638 6286 6286 67 3 0x80 netcon httpd
> 7280 6286 6286 67 3 0x80 netcon httpd
> 6286 1 6286 0 3 0x80 select httpd
> 9641 32013 32013 1000 4 0x8000000 emacsclient
> 32013 30562 32013 1000 4 0x8001088 sh
> 28064 1 28064 1000 3 0x80 poll emacs-24.2.50
> 24930 16156 16156 250 3 0 inode bitlbee
> 16156 1 16156 250 3 0x80 poll bitlbee
> 26754 3350 3350 1000 3 0x80 poll irssi
> 3350 18076 3350 1000 3 0x88 pause sh
> 18773 1 18773 539 3 0 inode clamd
> 12322 14129 12322 1000 3 0x80 ttyin zsh
> 30562 24979 30562 1000 3 0x80 ttyin zsh
> 19651 15004 19651 1000 3 0x80 ttyin zsh
> 6185 2796 6185 1000 3 0x80 ttyin zsh
> 14129 18076 14129 1000 3 0x88 pause sh
> 15004 18076 15004 1000 3 0x88 pause sh
> 24979 18076 24979 1000 3 0x88 pause sh
> 2796 18076 2796 1000 3 0x88 pause sh
> 18076 1 18076 1000 3 0 inode tmux
> 15038 9462 20303 1000 3 0x80 select mpc
> 10750 1 10750 0 3 0x80 kqread master
> 9553 1 9553 0 3 0x88 pause nginx
> 15147 31127 31127 70 3 0x80 select named
> 31127 1 31127 0 3 0x80 netio named
> 21619 1 21619 0 3 0 inode getty
> 9462 1 20303 1000 3 0x88 pause sh
> 14145 1 14145 0 3 0x80 select rsync
> 15214 1 15214 0 3 0x80 select sshd
> 17962 1 17962 1000 3 0x80 piperd sh
> 20302 30868 26716 1000 3 0x80 piperd sh
> 30403 30868 26716 1000 3 0x80 kqread tail
> 30868 1 26716 1000 3 0x88 pause sh
> 25204 1 26716 1000 3 0x80 piperd sh
> 22792 4269 2917 83 3 0x80 poll ntpd
> 4269 2917 2917 83 3 0x80 poll ntpd
> 2917 1 2917 0 3 0x80 poll ntpd
> 20765 26845 20765 0 3 0x80 poll syslog-ng
> 26845 1 17854 0 3 0x80 wait syslog-ng
> 1522 1 1522 0 3 0 inode cron
> 26641 1 26641 0 3 0x80 ttyin getty
> 9435 1 9435 0 3 0x80 ttyin getty
> 19690 1 19690 0 3 0x80 ttyin getty
> 9245 1 9245 0 3 0x80 ttyin getty
> 15782 1 15782 0 3 0x80 ttyin getty
> 32449 1 32449 507 3 0x80 wait perl
> 28704 1 28704 569 3 0x80 poll dkim-filter
> 32274 1 32274 560 3 0x80 poll mpd
> 12412 13936 8620 502 3 0x80 poll mysqld
> 13936 1 8620 0 3 0x88 pause sh
> 7106 1 7106 1001 3 0x80 poll openvpn
> 7503 8867 8867 62 3 0x80 piperd spamd
> 17788 1 17788 62 3 0x80 bpf spamlogd
> 8159 8867 8867 62 3 0x80 select spamd
> 8867 1 8867 62 3 0 inode spamd
> 31779 1 31779 0 3 0x80 netcon vsftpd
> 9864 1 9864 0 3 0x80 netcon vsftpd
> 18563 11213 11213 0 3 0x80 nfsd nfsd
> 2756 11213 11213 0 3 0x80 nfsd nfsd
> 20830 11213 11213 0 3 0x80 nfsd nfsd
> 20795 11213 11213 0 3 0x80 nfsd nfsd
> 11213 1 11213 0 3 0x80 netcon nfsd
> 23780 1 23780 0 3 0x80 select mountd
> 15095 1 15095 28 3 0x80 poll portmap
> 20 0 0 0 3 0x100200 aiodoned aiodoned
> 19 0 0 0 3 0x100200 syncer update
> 18 0 0 0 3 0x100200 cleaner cleaner
> 17 0 0 0 3 0x100200 reaper reaper
> 16 0 0 0 3 0x100200 pgdaemon pagedaemon
> 15 0 0 0 3 0x100200 bored crypto
> 14 0 0 0 3 0x100200 pftm pfpurge
> 13 0 0 0 3 0x100200 usbtsk usbtask
> 12 0 0 0 3 0x100200 usbatsk usbatsk
> 11 0 0 0 3 0x100200 acpi0 acpi0
> 10 0 0 0 7 0x40100200 idle7
> 9 0 0 0 7 0x40100200 idle6
> 8 0 0 0 7 0x40100200 idle5
> 7 0 0 0 7 0x40100200 idle4
> 6 0 0 0 7 0x40100200 idle3
> 5 0 0 0 7 0x40100200 idle2
> 4 0 0 0 7 0x40100200 idle1
> 3 0 0 0 3 0x100200 bored syswq
> * 2 0 0 0 7 0x40100200 idle0
> 1 0 1 0 3 0x80 wait init
> 0 -1 0 0 3 0x200 scheduler swapper
> 3839 8787 8787 1000 5 0x2000 sshd
> ddb{0}> trace
> Debugger() at Debugger+0x5
> comintr() at comintr+0x268
> Xintr_ioapic_edge4() at Xintr_ioapic_edge4+0xe8
> --- interrupt ---
> Bad frame pointer: 0xffff800022151f10
> end trace frame: 0xffff800022151f10, count: -3
> cpu_idle_cycle+0x13:
> ddb{0}> boot reboot
> rebooting...
> >> OpenBSD/amd64 BOOT 3.18ies Ltd.
> boot> ghts Reserved
> booting hd0a:/bsd: 5669864+1601484+935608+0+617568 [89+499848+323884]=0xd351b8
> entry point at 0x1001e0 [7205c766, 34000004, 24448b12, 3cb8a304]
> [ using 824664 bytes of bsd ELF symbol table ]or = 4
> Copyright (c) 1982, 1986, 1989, 1991, 199333GHz
> Intel(R)The Regents of the University of California. All rights reserved.
> Copyright (c) 1995-2012 OpenBSD. All rights reserved. http://www.OpenBSD.org
>
> OpenBSD 5.1 (GENERIC.MP) #207: Sun Feb 12 09:42:14 MST 2012
> [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 4292935680 (4094MB)
> avail mem = 4164493312 (3971MB)
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.31 @ 0xdc010 (57 entries)
> bios0: vendor HP version "O08" date 06/03/2009
> bios0: HP ProLiant DL140 G3
> acpi0 at bios0: rev 0
> acpi0: sleep states S0 S4 S5
> acpi0: tables DSDT FACP SPMI APIC MCFG BOOT SPCR SSDT
> acpi0: wakeup devices BPD0(S5) BMF3(S5) P0P4(S5) P0P6(S5) PEX0(S5) PEX1(S5)
> PEX2(S5) PEX3(S5) USB1(S5) USB2(S5) USB3(S5) EUSB(S5) PCIB(S5)
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.81 MHz
> cpu0:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
> cpu0: 4MB 64b/line 16-way L2 cache
> cpu0: apic clock running at 332MHz
> cpu1 at mainbus0: apid 4 (application processor)
> cpu1: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.51 MHz
> cpu1:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
> cpu1: 4MB 64b/line 16-way L2 cache
> cpu2 at mainbus0: apid 1 (application processor)
> cpu2: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
> cpu2:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
> cpu2: 4MB 64b/line 16-way L2 cache
> cpu3 at mainbus0: apid 5 (application processor)
> cpu3: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
> cpu3:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
> cpu3: 4MB 64b/line 16-way L2 cache
> cpu4 at mainbus0: apid 2 (application processor)
> cpu4: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
> cpu4:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
> cpu4: 4MB 64b/line 16-way L2 cache
> cpu5 at mainbus0: apid 6 (application processor)
> cpu5: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
> cpu5:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
> cpu5: 4MB 64b/line 16-way L2 cache
> cpu6 at mainbus0: apid 3 (application processor)
> cpu6: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
> cpu6:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
> cpu6: 4MB 64b/line 16-way L2 cache
> cpu7 at mainbus0: apid 7 (application processor)
> cpu7: Intel(R) Xeon(R) CPU E5345 @ 2.33GHz, 2327.50 MHz
> cpu7:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,SSE3,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,NXE,LONG,LAHF
> cpu7: 4MB 64b/line 16-way L2 cache
> ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
> ioapic1 at mainbus0: apid 9 pa 0xfec80000, version 20, 24 pins
> acpimcfg0 at acpi0 addr 0xe0000000, bus 0-31
> acpiprt0 at acpi0: bus 1 (P0P2)
> acpiprt1 at acpi0: bus 2 (BMD0)
> acpiprt2 at acpi0: bus 3 (BPD0)
> acpiprt3 at acpi0: bus -1 (BPD1)
> acpiprt4 at acpi0: bus -1 (BPD2)
> acpiprt5 at acpi0: bus 11 (BMF3)
> acpiprt6 at acpi0: bus 16 (P0P4)
> acpiprt7 at acpi0: bus 18 (P0P6)
> acpiprt8 at acpi0: bus 0 (PCI0)
> acpiprt9 at acpi0: bus 30 (PEX0)
> acpiprt10 at acpi0: bus 31 (PEX1)
> acpiprt11 at acpi0: bus -1 (PEX2)
> acpiprt12 at acpi0: bus -1 (PEX3)
> acpiprt13 at acpi0: bus 32 (PCIB)
> acpicpu0 at acpi0
> acpicpu1 at acpi0
> acpicpu2 at acpi0
> acpicpu3 at acpi0
> acpicpu4 at acpi0
> acpicpu5 at acpi0
> acpicpu6 at acpi0
> acpicpu7 at acpi0
> acpibtn0 at acpi0: PWRB
> ipmi at mainbus0 not configured
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel 5000X Host" rev 0x31
> ppb0 at pci0 dev 2 function 0 "Intel 5000 PCIE x8" rev 0x31
> pci1 at ppb0 bus 1
> ppb1 at pci1 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
> pci2 at ppb1 bus 2
> ppb2 at pci2 dev 0 function 0 "Intel 6321ESB PCIE" rev 0x01
> pci3 at ppb2 bus 3
> ppb3 at pci1 dev 0 function 3 "Intel 6321ESB PCIE-PCIX" rev 0x01
> pci4 at ppb3 bus 11
> mpi0 at pci4 dev 1 function 0 "Symbios Logic SAS1068" rev 0x01: msi
> scsibus0 at mpi0: 112 targets
> sd0 at scsibus0 targ 1 lun 0: <LSILOGIC, Logical Volume, 3000> SCSI2 0/direct
> fixed naa.600508e000000000f37fc1e28b72c109
> sd0: 1907348MB, 512 bytes/sector, 3906248704 sectors
> ppb4 at pci0 dev 3 function 0 "Intel 5000 PCIE" rev 0x31
> pci5 at ppb4 bus 12
> ppb5 at pci0 dev 4 function 0 "Intel 5000 PCIE x16" rev 0x31: msi
> pci6 at ppb5 bus 16
> ppb6 at pci0 dev 5 function 0 "Intel 5000 PCIE" rev 0x31: msi
> pci7 at ppb6 bus 17
> ppb7 at pci0 dev 6 function 0 "Intel 5000 PCIE" rev 0x31: msi
> pci8 at ppb7 bus 18
> ppb8 at pci0 dev 7 function 0 "Intel 5000 PCIE" rev 0x31: msi
> pci9 at ppb8 bus 19
> pchb1 at pci0 dev 16 function 0 "Intel 5000 Error Reporting" rev 0x31
> pchb2 at pci0 dev 16 function 1 "Intel 5000 Error Reporting" rev 0x31
> pchb3 at pci0 dev 16 function 2 "Intel 5000 Error Reporting" rev 0x31
> pchb4 at pci0 dev 17 function 0 "Intel 5000 Reserved" rev 0x31
> pchb5 at pci0 dev 19 function 0 "Intel 5000 Reserved" rev 0x31
> pchb6 at pci0 dev 21 function 0 "Intel 5000 FBD" rev 0x31
> pchb7 at pci0 dev 22 function 0 "Intel 5000 FBD" rev 0x31
> ppb9 at pci0 dev 28 function 0 "Intel 6321ESB PCIE" rev 0x09: msi
> pci10 at ppb9 bus 30
> bge0 at pci10 dev 0 function 0 "Broadcom BCM5721" rev 0x11, BCM5750 B1
> (0x4101): apic 8 int 16, address 00:1a:4b:d4:1c:3e
> brgphy0 at bge0 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
> ppb10 at pci0 dev 28 function 1 "Intel 6321ESB PCIE" rev 0x09: msi
> pci11 at ppb10 bus 31
> bge1 at pci11 dev 0 function 0 "Broadcom BCM5721" rev 0x11, BCM5750 B1
> (0x4101): apic 8 int 17, address 00:1a:4b:d4:1c:3f
> brgphy1 at bge1 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
> uhci0 at pci0 dev 29 function 0 "Intel 6321ESB USB" rev 0x09: apic 8 int 23
> uhci1 at pci0 dev 29 function 1 "Intel 6321ESB USB" rev 0x09: apic 8 int 23
> uhci2 at pci0 dev 29 function 2 "Intel 6321ESB USB" rev 0x09: apic 8 int 23
> ehci0 at pci0 dev 29 function 7 "Intel 6321ESB USB" rev 0x09: apic 8 int 23
> usb0 at ehci0: USB revision 2.0
> uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> ppb11 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xd9
> pci12 at ppb11 bus 32
> vga1 at pci12 dev 2 function 0 "Matrox MGA G200e (ServerEngines)" rev 0x02
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> pcib0 at pci0 dev 31 function 0 "Intel 6321ESB LPC" rev 0x09
> pciide0 at pci0 dev 31 function 1 "Intel 6321ESB IDE" rev 0x09: DMA, channel
> 0 configured to compatibility, channel 1 configured to compatibility
> pciide0: channel 0 disabled (no drives)
> pciide0: channel 1 ignored (disabled)
> ichiic0 at pci0 dev 31 function 3 "Intel 6321ESB SMBus" rev 0x09: apic 8 int
> 19
> iic0 at ichiic0
> usb1 at uhci0: USB revision 1.0
> uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> usb2 at uhci1: USB revision 1.0
> uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> usb3 at uhci2: USB revision 1.0
> uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> isa0 at pcib0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> com0: console
> pckbc0 at isa0 port 0x60/5
> pckbd0 at pckbc0 (kbd slot)
> pckbc0: using irq 1 for kbd slot
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> mtrr: Pentium Pro MTRR support
> uhidev0 at uhub3 port 1 configuration 1 interface 0 "ServerEngines SE USB
> Device" rev 1.10/0.01 addr 2
> uhidev0: iclass 3/1
> ukbd0 at uhidev0: 8 modifier keys, 6 key codes
> wskbd1 at ukbd0 mux 1
> wskbd1: connecting to wsdisplay0
> uhidev1 at uhub3 port 1 configuration 1 interface 1 "ServerEngines SE USB
> Device" rev 1.10/0.01 addr 2
> uhidev1: iclass 3/1
> ums0 at uhidev1: 8 buttons, Z dir
> wsmouse0 at ums0 mux 0
> vscsi0 at root
> scsibus1 at vscsi0: 256 targets
> softraid0 at root
> scsibus2 at softraid0: 256 targets
> root on sd0a (42c44955279fd28c.a) swap on sd0b dump on sd0b
> WARNING: / was not properly unmounted
> Automatic boot in progress: starting file system checks.
>
>
>
>
> # Han
