snapshot from the 20th. can.t duplicate it on a uniprocessor. OpenBSD 5.8-beta (GENERIC.MP) #1171: Sun Jul 19 19:03:27 MDT 2015 [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
# while true ; do mount /dev/sd3a /a ; umount /dev/sd3a ; done $ while true ; do ls /a > /dev/null ; done After about 45 seconds, the machine will lockup It still responds to pings. amd64 in vmware, but also a physical machine will lockup takes longer with a sleep 1 between the mount/unmount, but still locks. copied part of /usr/src to /dev/sd3a to have some data # fsck /dev/sd3a ** /dev/rsd3a ** File system is clean; not checking # fsck -f /dev/sd3a ** /dev/rsd3a ** File system is already clean ** Last Mounted on /a ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 7367 files, 11076 used, 2051627 free (27 frags, 256450 blocks, 0.0% fragmentation) # in one terminal: vmstat 1 procs memory page disks traps cpu r b w avm fre flt re pi po fr sr cd0 sd0 int sys cs us sy id 0 1 0 44800 21681868 7954 0 0 0 0 0 0 24 405 4736 1269 2 10 88 0 1 0 44788 21681872 12996 0 0 0 0 0 0 36 609 7897 1938 0 10 90 0 1 0 44616 21682032 10883 0 0 0 0 0 0 31 542 6426 1649 0 11 89 0 1 0 44788 21681860 10495 0 0 0 0 0 0 30 510 6456 1525 1 7 92 0 1 0 44620 21682016 11717 0 0 0 0 0 0 34 579 6935 1777 0 11 89 2 0 0 44272 21682124 11457 0 0 0 0 0 0 31 540 6964 1668 1 12 87 0 1 0 44612 21682000 9588 0 0 0 0 0 0 28 467 5629 1482 1 11 88 0 1 0 44624 21681992 12365 0 0 0 0 0 0 34 584 7544 1864 0 9 91 0 1 0 44628 21681976 11401 0 0 0 0 0 0 34 566 6743 1680 1 9 90 0 1 0 44800 21681792 12536 0 0 0 0 0 0 36 601 7579 1843 0 12 88 0 1 0 44796 21681792 10082 0 0 0 0 0 0 30 502 5976 1508 0 12 88 1 1 0 44424 21682052 10389 0 0 0 0 0 0 28 483 6369 1458 0 5 95 2 0 0 44492 21682008 11125 0 0 0 0 0 0 34 569 6589 1777 1 10 89 0 1 0 44624 21681924 11808 0 0 0 0 0 0 31 548 7164 1621 1 12 87 0 1 0 44792 21681744 12438 0 0 0 0 0 0 38 624 7355 1924 0 12 88 0 1 0 44616 21681912 9201 0 0 0 0 0 0 24 420 5681 1340 2 6 92 0 0 0 44800 21681728 10332 0 0 0 0 0 0 32 514 6110 1581 0 6 94 0 0 0 44880 21681344 175 0 0 0 0 0 0 0 7 125 41 0 1 99 0 0 0 44880 21681344 7 0 0 0 0 0 0 0 5 37 33 0 0 100 0 0 0 44880 21681344 7 0 0 0 0 0 0 0 11 85 39 0 0 100 0 0 0 44880 21681348 7 0 0 0 0 0 0 0 4 37 24 0 0 100 0 0 0 44880 21681348 7 0 0 0 0 0 0 0 4 37 24 0 0 100 procs memory page disks traps cpu r b w avm fre flt re pi po fr sr cd0 sd0 int sys cs us sy id 1 0 0 44880 21681348 29 0 0 0 0 0 0 0 7 73 34 0 0 100 0 0 0 44880 21681348 7 0 0 0 0 0 0 0 6 41 28 0 0 100 0 0 0 44880 21681364 7 0 0 0 0 0 0 0 5 37 25 0 0 100 0 0 0 44880 21681368 7 0 0 0 0 0 0 0 6 37 27 0 0 100 in the second terminal # while true ; do mount /dev/sd3a /a ; umount /dev/sd3a ; done in the third terminal: $ while true ; do ls /a > /dev/null ; done You can see the lockup happen in the vmstat output when the fault column goes to .7.. No processes are startable when that happens, a running top will keep running, but cannot kill the ksh processes. top will stay running, but cannot kill the ksh PID USERNAME PRI NICE SIZE RES STATE WAIT TIME CPU COMMAND 28124 test 18 0 648K 748K sleep pause 0:03 0.54% ksh 10055 root 18 0 640K 740K sleep pause 0:01 0.34% ksh 1 root 10 0 464K 540K idle wait 0:01 0.00% init 13800 _x11 2 0 11M 15M idle select 0:01 0.00% Xorg 11954 root 10 0 264K 292K idle wait 0:00 0.00% mount the two ksh processes are in pause state. mount in wait. a looping ps -al to show wchan and state: 1000 12949 1924 36 14 0 340 352 vfslock D+ p5 0:00.00 ls /a 1000 17565 7901 1 18 0 644 732 pause Is p1 0:00.00 -ksh (ksh) 1000 28650 9250 0 18 0 640 732 pause Is p2 0:00.00 -ksh (ksh) 0 31431 28650 36 18 0 644 744 pause S p2 0:01.51 -ksh (ksh) 0 27668 31431 36 -5 0 244 232 biowait D+ p2 0:00.00 umount /dev/sd3a 1000 19565 6707 3 18 0 644 736 pause Is p3 0:00.00 -ksh (ksh) 1000 21718 24499 0 18 0 640 736 pause Ss p4 0:00.07 -ksh (ksh) 1000 1924 11677 36 18 0 640 732 pause Ss p5 0:00.33 -ksh (ksh) 1000 27159 1924 36 14 0 340 344 vfslock D+ p5 0:00.00 ls /a 1000 17565 7901 1 18 0 644 732 pause Is p1 0:00.00 -ksh (ksh) 1000 28650 9250 0 18 0 640 732 pause Is p2 0:00.00 -ksh (ksh) 0 31431 28650 36 18 0 644 744 pause S p2 0:01.52 -ksh (ksh) 0 22265 31431 36 -5 0 240 232 biowait D+ p2 0:00.00 umount /dev/sd3a 1000 19565 6707 3 18 0 644 736 pause Is p3 0:00.00 -ksh (ksh) 1000 21718 24499 0 18 0 640 736 pause Ss p4 0:00.07 -ksh (ksh) 1000 1924 11677 36 18 0 640 732 pause Ss p5 0:00.35 -ksh (ksh) 1000 19206 1924 36 14 0 340 344 vfslock D+ p5 0:00.00 ls /a 1000 17565 7901 1 18 0 644 732 pause Is p1 0:00.00 -ksh (ksh) 1000 28650 9250 0 18 0 640 732 pause Is p2 0:00.00 -ksh (ksh) 0 31431 28650 36 18 0 644 744 pause S p2 0:01.53 -ksh (ksh) 0 3936 31431 36 -5 0 244 240 biowait D+ p2 0:00.00 umount /dev/sd3a 1000 19565 6707 3 18 0 644 736 pause Is p3 0:00.00 -ksh (ksh) 1000 21718 24499 0 18 0 640 736 pause Ss p4 0:00.07 -ksh (ksh) 1000 1924 11677 36 18 0 640 732 pause Ss p5 0:00.36 -ksh (ksh) 1000 28177 1924 36 14 0 340 344 vfslock D+ p5 0:00.00 ls /a dmesg OpenBSD 5.8-beta (GENERIC.MP) #1171: Sun Jul 19 19:03:27 MDT 2015 [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP real mem = 23026663424 (21959MB) avail mem = 22324908032 (21290MB) mpath0 at root scsibus0 at mpath0: 256 targets mainbus0 at root bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xe0010 (364 entries) bios0: vendor Phoenix Technologies LTD version "6.00" date 04/14/2014 bios0: VMware, Inc. VMware Virtual Platform acpi0 at bios0: rev 2 acpi0: sleep states S0 S1 S4 S5 acpi0: tables DSDT FACP BOOT APIC MCFG SRAT HPET WAET acpi0: wakeup devices PCI0(S3) USB_(S1) P2P0(S3) S1F0(S3) S2F0(S3) S3F0(S3) S4F0(S3) S5F0(S3) S6F0(S3) S7F0(S3) S8F0(S3) S9F0(S3) S10F(S3) S11F(S3) S12F(S3) S13F(S3) [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz, 2399.74 MHz cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,XSAVE,AVX,HV,NXE,LONG,LAHF,PERF,ITSC,SENSOR,ARAT cpu0: 256KB 64b/line 8-way L2 cache cpu0: smt 0, core 0, package 0 mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges cpu0: apic clock running at 65MHz cpu1 at mainbus0: apid 1 (application processor) cpu1: Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz, 2399.95 MHz cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,AES,XSAVE,AVX,HV,NXE,LONG,LAHF,PERF,ITSC,SENSOR,ARAT cpu1: 256KB 64b/line 8-way L2 cache cpu1: smt 0, core 1, package 0 ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 11, 24 pins acpimcfg0 at acpi0 addr 0xf0000000, bus 0-127 acpihpet0 at acpi0: 14318179 Hz acpiprt0 at acpi0: bus 0 (PCI0) acpicpu0 at acpi0 acpiprt0 at acpi0: bus 0 (PCI0) acpicpu0 at acpi0 acpicpu1 at acpi0 acpibat0 at acpi0: BAT1 not present acpibat1 at acpi0: BAT2 not present acpiac0 at acpi0: AC unit online acpibtn0 at acpi0: SLPB acpibtn1 at acpi0: LID_ vmt0 at mainbus0 pci0 at mainbus0 bus 0 pchb0 at pci0 dev 0 function 0 "Intel 82443BX AGP" rev 0x01 ppb0 at pci0 dev 1 function 0 "Intel 82443BX AGP" rev 0x01 pci1 at ppb0 bus 1 pcib0 at pci0 dev 7 function 0 "Intel 82371AB PIIX4 ISA" rev 0x08 pciide0 at pci0 dev 7 function 1 "Intel 82371AB IDE" rev 0x01: DMA, channel 0 configured to compatibility, channel 1 configured to compatibility pciide0: channel 0 disabled (no drives) atapiscsi0 at pciide0 channel 1 drive 0 scsibus1 at atapiscsi0: 2 targets cd0 at scsibus1 targ 0 lun 0: <NECVMWar, VMware IDE CDR10, 1.00> ATAPI 5/cdrom removable cd0(pciide0:1:0): using PIO mode 4, Ultra-DMA mode 2 piixpm0 at pci0 dev 7 function 3 "Intel 82371AB Power" rev 0x08: SMBus disabled "VMware VMCI" rev 0x10 at pci0 dev 7 function 7 not configured vga1 at pci0 dev 15 function 0 "VMware SVGA II" rev 0x00 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation) mpi0 at pci0 dev 16 function 0 "Symbios Logic 53c1030" rev 0x01: apic 2 int 17 mpi0: 0, firmware 1.3.41.32 scsibus2 at mpi0: 16 targets, initiator 7 sd0 at scsibus2 targ 0 lun 0: <VMware, Virtual disk, 1.0> SCSI2 0/direct fixed sd0: 20480MB, 512 bytes/sector, 41943040 sectors sd1 at scsibus2 targ 1 lun 0: <VMware, Virtual disk, 1.0> SCSI2 0/direct fixed sd1: 4096MB, 512 bytes/sector, 8388608 sectors sd2 at scsibus2 targ 2 lun 0: <VMware, Virtual disk, 1.0> SCSI2 0/direct fixed sd2: 4096MB, 512 bytes/sector, 8388608 sectors sd3 at scsibus2 targ 3 lun 0: <VMware, Virtual disk, 1.0> SCSI2 0/direct fixed sd3: 4096MB, 512 bytes/sector, 8388608 sectors sd4 at scsibus2 targ 4 lun 0: <VMware, Virtual disk, 1.0> SCSI2 0/direct fixed sd4: 4096MB, 512 bytes/sector, 8388608 sectors mpi0: target 0 Sync at 160MHz width 16bit offset 127 QAS 1 DT 1 IU 1 mpi0: target 1 Sync at 160MHz width 16bit offset 127 QAS 1 DT 1 IU 1 mpi0: target 2 Sync at 160MHz width 16bit offset 127 QAS 1 DT 1 IU 1 mpi0: target 3 Sync at 160MHz width 16bit offset 127 QAS 1 DT 1 IU 1 mpi0: target 4 Sync at 160MHz width 16bit offset 127 QAS 1 DT 1 IU 1 ppb1 at pci0 dev 17 function 0 "VMware PCI" rev 0x02 pci2 at ppb1 bus 2 em0 at pci2 dev 0 function 0 "Intel 82545EM" rev 0x01: apic 2 int 18, address xx:xx:xx:xx:xx:xx ppb2 at pci0 dev 21 function 0 "VMware PCIE" rev 0x01 pci3 at ppb2 bus 3 ppb3 at pci0 dev 21 function 1 "VMware PCIE" rev 0x01 pci4 at ppb3 bus 4 ppb4 at pci0 dev 21 function 2 "VMware PCIE" rev 0x01 pci5 at ppb4 bus 5 ppb5 at pci0 dev 21 function 3 "VMware PCIE" rev 0x01 pci6 at ppb5 bus 6 ppb6 at pci0 dev 21 function 4 "VMware PCIE" rev 0x01 pci7 at ppb6 bus 7 ppb7 at pci0 dev 21 function 5 "VMware PCIE" rev 0x01 pci8 at ppb7 bus 8 ppb8 at pci0 dev 21 function 6 "VMware PCIE" rev 0x01 pci9 at ppb8 bus 9 ppb9 at pci0 dev 21 function 7 "VMware PCIE" rev 0x01 pci10 at ppb9 bus 10 ppb10 at pci0 dev 22 function 0 "VMware PCIE" rev 0x01 pci11 at ppb10 bus 11 ppb11 at pci0 dev 22 function 1 "VMware PCIE" rev 0x01 pci12 at ppb11 bus 12 ppb12 at pci0 dev 22 function 2 "VMware PCIE" rev 0x01 pci13 at ppb12 bus 13 ppb13 at pci0 dev 22 function 3 "VMware PCIE" rev 0x01 pci14 at ppb13 bus 14 ppb14 at pci0 dev 22 function 4 "VMware PCIE" rev 0x01 pci15 at ppb14 bus 15 ppb15 at pci0 dev 22 function 5 "VMware PCIE" rev 0x01 pci16 at ppb15 bus 16 ppb16 at pci0 dev 22 function 6 "VMware PCIE" rev 0x01 ppb16 at pci0 dev 22 function 6 "VMware PCIE" rev 0x01 pci17 at ppb16 bus 17 ppb17 at pci0 dev 22 function 7 "VMware PCIE" rev 0x01 pci18 at ppb17 bus 18 ppb18 at pci0 dev 23 function 0 "VMware PCIE" rev 0x01 pci19 at ppb18 bus 19 ppb19 at pci0 dev 23 function 1 "VMware PCIE" rev 0x01 pci20 at ppb19 bus 20 ppb20 at pci0 dev 23 function 2 "VMware PCIE" rev 0x01 pci21 at ppb20 bus 21 ppb21 at pci0 dev 23 function 3 "VMware PCIE" rev 0x01 pci22 at ppb21 bus 22 ppb22 at pci0 dev 23 function 4 "VMware PCIE" rev 0x01 pci23 at ppb22 bus 23 ppb23 at pci0 dev 23 function 5 "VMware PCIE" rev 0x01 pci24 at ppb23 bus 24 ppb24 at pci0 dev 23 function 6 "VMware PCIE" rev 0x01 pci25 at ppb24 bus 25 ppb25 at pci0 dev 23 function 7 "VMware PCIE" rev 0x01 pci26 at ppb25 bus 26 ppb26 at pci0 dev 24 function 0 "VMware PCIE" rev 0x01 pci27 at ppb26 bus 27 ppb27 at pci0 dev 24 function 1 "VMware PCIE" rev 0x01 pci28 at ppb27 bus 28 ppb28 at pci0 dev 24 function 2 "VMware PCIE" rev 0x01 pci29 at ppb28 bus 29 ppb29 at pci0 dev 24 function 3 "VMware PCIE" rev 0x01 pci30 at ppb29 bus 30 ppb30 at pci0 dev 24 function 4 "VMware PCIE" rev 0x01 pci31 at ppb30 bus 31 ppb31 at pci0 dev 24 function 5 "VMware PCIE" rev 0x01 pci32 at ppb31 bus 32 ppb32 at pci0 dev 24 function 6 "VMware PCIE" rev 0x01 pci33 at ppb32 bus 33 ppb33 at pci0 dev 24 function 7 "VMware PCIE" rev 0x01 pci34 at ppb33 bus 34 isa0 at pcib0 isadma0 at isa0 fdc0 at isa0 port 0x3f0/6 irq 6 drq 2 fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo pckbc0 at isa0 port 0x60/5 irq 1 irq 12 pckbd0 at pckbc0 (kbd slot) wskbd0 at pckbd0: console keyboard, using wsdisplay0 pms0 at pckbc0 (aux slot) wsmouse0 at pms0 mux 0 pcppi0 at isa0 port 0x61 spkr0 at pcppi0 lpt0 at isa0 port 0x378/4 irq 7 vscsi0 at root scsibus3 at vscsi0: 256 targets softraid0 at root scsibus4 at softraid0: 256 targets root on sd0a (xxxxxxxx.a) swap on sd0b dump on sd0b WARNING: / was not properly unmounted
