The rebuild has finished:
Volume Status Size Device
softraid0 0 Online 12002360033280 sd5 RAID5
0 Online 4000786726912 0:0.0 noencl <sd1a>
1 Online 4000786726912 0:1.0 noencl <sd2a>
2 Online 4000786726912 0:2.0 noencl <sd3a>
3 Online 4000786726912 0:3.0 noencl <sd4a>
I'm hoping for advise on how to proceed; I need to try to get as much
data off of this before things get worse. Or aren't they even bad at
all?
What bothers me is that there is a need to 'retrying read on block
52992'. IIRC I've seen this on dying HDDs but not on a softraid disk.
Does this mean the underlying softraid is broken?
Thanks in advance, Marcus
[email protected] (Marcus MERIGHI), 2016.01.28 (Thu) 10:58 (CET):
> Softraid RAID-5 on four 4TB HDDs.
> The four disks are in an external enclosure (JBOD), connected via USB.
> The array worked for about a month.
> Tested before loading (bioctl -O, bioctl -R).
> Copied roughly 6TB onto it while rebuilding was ongoing.
> Took about 3 weeks (via network).
> When the data was there I started another copy from the array to yet another
> external disk.
> While this was running the first kernel panic ocurred:
>
> sd5: retrying read on block 52992
> panic: softraid0: sd5: invalid volume state transition 1 -> 1
> Starting stack trace...
> panic() at panic+0x10b
> sr_raid5_set_vol_state() at sr_raid5_set_vol_state+0xe7
> sr_raid5_set_chunk_state() at sr_raid5_set_chunk_state+0xc7
> sr_ccb_done() at sr_ccb_done+0x76
> sr_raid5_intr() at sr_raid5_intr+0x3c
> sd_buf_done() at sd_buf_done+0x7b
> scsi_done() at scsi_done+0x1e
> usb_transfer_complete() at usb_transfer_complete+0x26c
> ehci_softintr() at ehci_softintr+0x3f
> softintr_dispatch() at softintr_dispatch+0x8b
> Xsoftnet() at Xsoftnet+0x1f
> --- interrupt ---
> end of kernel
> end trace frame: 0x1388, count: 246
> 0x8:
> End of stack trace.
> syncing disks... 80 48 10 1 1 1 1 1 1 [...] giving up
>
> After that I retried (reboot, fsck, mount) and when accessing the mount
> point there was another kernel panic. Silly me did not take another
> picture.
> Only after unpowering/powering the external enclosure and rebooting the
> machine the array was automagically assembled again, with one disk
> degraded. It is now rebuilding, 13% took about 14 hours.
>
> Thanks for reading, Marcus
>
> # bioctl softraid0
> Volume Status Size Device
> softraid0 0 Rebuild 12002360033280 sd5 RAID5 13% done
> 0 Rebuild 4000786726912 0:0.0 noencl <sd1a>
> 1 Online 4000786726912 0:1.0 noencl <sd2a>
> 2 Online 4000786726912 0:2.0 noencl <sd3a>
> 3 Online 4000786726912 0:3.0 noencl <sd4a>
> softraid0 1 Online 53691555840 sd6 CRYPTO
> 0 Online 53691555840 1:0.0 noencl <sd0k>
> softraid0 2 Online 53949354496 sd7 CRYPTO
> 0 Online 53949354496 2:0.0 noencl <sd0m>
>
> OpenBSD 5.8 (GENERIC.MP) #1236: Sun Aug 16 02:31:04 MDT 2015
> [email protected]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 4276822016 (4078MB)
> avail mem = 4143312896 (3951MB)
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.5 @ 0xcff9c000 (46 entries)
> bios0: vendor Dell Inc. version "1.4.3" date 06/05/2009
> bios0: Dell Inc. PowerEdge R200
> acpi0 at bios0: rev 2
> acpi0: sleep states S0 S4 S5
> acpi0: tables DSDT FACP APIC SPCR HPET MCFG WDAT SLIC ERST HEST BERT EINJ
> SSDT SSDT SSDT
> acpi0: wakeup devices PCI0(S5)
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Core(TM)2 Duo CPU E7300 @ 2.66GHz, 1600.27 MHz
> cpu0:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,NXE,LONG,LAHF,PERF,SENSOR
> cpu0: 3MB 64b/line 8-way L2 cache
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
> cpu0: apic clock running at 266MHz
> cpu0: mwait min=64, max=64, C-substates=0.2.2.2.2, IBE
> cpu1 at mainbus0: apid 1 (application processor)
> cpu1: Intel(R) Core(TM)2 Duo CPU E7300 @ 2.66GHz, 1600.06 MHz
> cpu1:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,DTES64,MWAIT,DS-CPL,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,NXE,LONG,LAHF,PERF,SENSOR
> cpu1: 3MB 64b/line 8-way L2 cache
> cpu1: smt 0, core 1, package 0
> ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
> ioapic0: misconfigured as apic 0, remapped to apid 2
> ioapic1 at mainbus0: apid 3 pa 0xfec10000, version 20, 24 pins
> ioapic1: misconfigured as apic 0, remapped to apid 3
> acpihpet0 at acpi0: 14318179 Hz
> acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpiprt1 at acpi0: bus 1 (PEX1)
> acpiprt2 at acpi0: bus 2 (SBE0)
> acpiprt3 at acpi0: bus 3 (PXHA)
> acpiprt4 at acpi0: bus 4 (SBE4)
> acpiprt5 at acpi0: bus 5 (SBE5)
> acpiprt6 at acpi0: bus 6 (COMP)
> acpicpu0 at acpi0: C1(@1 halt!), PSS
> acpicpu1 at acpi0: C1(@1 halt!), PSS
> ipmi at mainbus0 not configured
> cpu0: Enhanced SpeedStep 1600 MHz: speeds: 2667, 2400, 2133, 1867, 1600 MHz
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel 3200/3210 Host" rev 0x01
> ppb0 at pci0 dev 1 function 0 "Intel 3200/3210 PCIE" rev 0x01: msi
> pci1 at ppb0 bus 1
> mpi0 at pci1 dev 0 function 0 "Symbios Logic SAS1068E" rev 0x08: msi
> mpi0: SAS6IR, firmware 0.25.47.0
> scsibus1 at mpi0: 112 targets
> sd0 at scsibus1 targ 0 lun 0: <Dell, VIRTUAL DISK, 1028> SCSI3 0/direct fixed
> naa.600508e0000000006e54a90066b37109
> sd0: 152064MB, 512 bytes/sector, 311427072 sectors
> ppb1 at pci0 dev 28 function 0 "Intel 82801I PCIE" rev 0x02
> pci2 at ppb1 bus 2
> ppb2 at pci2 dev 0 function 0 "Intel 6702PXH PCIE-PCIX" rev 0x09
> pci3 at ppb2 bus 3
> ppb3 at pci0 dev 28 function 4 "Intel 82801I PCIE" rev 0x02
> pci4 at ppb3 bus 4
> bge0 at pci4 dev 0 function 0 "Broadcom BCM5721" rev 0x21, BCM5750 C1
> (0x4201): msi, address 00:25:64:3b:e9:34
> brgphy0 at bge0 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
> ppb4 at pci0 dev 28 function 5 "Intel 82801I PCIE" rev 0x02
> pci5 at ppb4 bus 5
> bge1 at pci5 dev 0 function 0 "Broadcom BCM5721" rev 0x21, BCM5750 C1
> (0x4201): msi, address 00:25:64:3b:e9:35
> brgphy1 at bge1 phy 1: BCM5750 10/100/1000baseT PHY, rev. 0
> uhci0 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x02: apic 2 int 21
> uhci1 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x02: apic 2 int 20
> uhci2 at pci0 dev 29 function 2 "Intel 82801I USB" rev 0x02: apic 2 int 21
> ehci0 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x02: apic 2 int 21
> usb0 at ehci0: USB revision 2.0
> uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> ppb5 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92
> pci6 at ppb5 bus 6
> radeondrm0 at pci6 dev 5 function 0 "ATI ES1000" rev 0x02
> drm0 at radeondrm0
> radeondrm0: apic 2 int 19
> pcib0 at pci0 dev 31 function 0 "Intel 82801IR LPC" rev 0x02
> pciide0 at pci0 dev 31 function 2 "Intel 82801I SATA" rev 0x02: DMA, channel
> 0 configured to native-PCI, channel 1 configured to native-PCI
> pciide0: using apic 2 int 23 for native-PCI interrupt
> atapiscsi0 at pciide0 channel 0 drive 1
> scsibus2 at atapiscsi0: 2 targets
> cd0 at scsibus2 targ 0 lun 0: <TSSTcorp, CDRWDVD TS-L463A, D550> ATAPI
> 5/cdrom removable
> cd0(pciide0:0:1): using PIO mode 4, Ultra-DMA mode 5
> usb1 at uhci0: USB revision 1.0
> uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> usb2 at uhci1: USB revision 1.0
> uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> usb3 at uhci2: USB revision 1.0
> uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> isa0 at pcib0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> pckbd0 at pckbc0 (kbd slot)
> wskbd0 at pckbd0: console keyboard
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> umass0 at uhub0 port 4 configuration 1 interface 0 "JMicron USB to ATA/ATAPI
> Bridge" rev 2.10/1.00 addr 2
> umass0: using SCSI over Bulk-Only
> scsibus3 at umass0: 2 targets, initiator 0
> sd1 at scsibus3 targ 1 lun 0: <WDC WD40, EFRX-68WT0N0, > SCSI3 0/direct fixed
> sd1: 3815447MB, 512 bytes/sector, 7814037168 sectors
> sd2 at scsibus3 targ 1 lun 1: <WDC WD40, EFRX-68WT0N0, > SCSI3 0/direct fixed
> sd2: 3815447MB, 512 bytes/sector, 7814037168 sectors
> sd3 at scsibus3 targ 1 lun 2: <WDC WD40, EFRX-68WT0N0, > SCSI3 0/direct fixed
> sd3: 3815447MB, 512 bytes/sector, 7814037168 sectors
> sd4 at scsibus3 targ 1 lun 3: <WDC WD40, EFRX-68WT0N0, > SCSI3 0/direct fixed
> sd4: 3815447MB, 512 bytes/sector, 7814037168 sectors
> uhub4 at uhub0 port 5 "Cypress Semiconductor USB2 Hub" rev 2.00/0.0b addr 3
> uhidev0 at uhub4 port 1 configuration 1 interface 0 "CPS OR600ELCDRM1U" rev
> 1.10/2.00 addr 4
> uhidev0: iclass 3/0, 44 report ids
> upd0 at uhidev0
> uhub5 at uhub4 port 2 "Terminus Technology USB 2.0 Hub" rev 2.00/1.11 addr 5
> umass1 at uhub5 port 1 configuration 1 interface 0 "Intenso USB 3.0 Device"
> rev 2.10/1.00 addr 6
> umass1: using SCSI over Bulk-Only
> scsibus4 at umass1: 2 targets, initiator 0
> sd5 at scsibus4 targ 1 lun 0: <Intenso, USB 3.0 Device, 0> SCSI4 0/direct
> fixed serial.174c55aa000000001F1F
> sd5: 2861588MB, 4096 bytes/sector, 732566646 sectors
> umass2 at uhub5 port 2 configuration 1 interface 0 "Intenso USB 3.0 Device"
> rev 2.10/1.00 addr 7
> umass2: using SCSI over Bulk-Only
> scsibus5 at umass2: 2 targets, initiator 0
> sd6 at scsibus5 targ 1 lun 0: <Intenso, USB 3.0 Device, 0> SCSI4 0/direct
> fixed serial.174c55aa0000000022DF
> sd6: 2861588MB, 4096 bytes/sector, 732566646 sectors
> umass3 at uhub5 port 3 configuration 1 interface 0 "Intenso USB 3.0 Device"
> rev 2.10/1.00 addr 8
> umass3: using SCSI over Bulk-Only
> scsibus6 at umass3: 2 targets, initiator 0
> sd7 at scsibus6 targ 1 lun 0: <Intenso, USB 3.0 Device, 0> SCSI4 0/direct
> fixed serial.174c55aa000000001EA0
> sd7: 2861588MB, 4096 bytes/sector, 732566646 sectors
> umass4 at uhub5 port 4 configuration 1 interface 0 "Intenso USB 3.0 Device"
> rev 2.10/1.00 addr 9
> umass4: using SCSI over Bulk-Only
> scsibus7 at umass4: 2 targets, initiator 0
> sd8 at scsibus7 targ 1 lun 0: <Intenso, USB 3.0 Device, 0> SCSI4 0/direct
> fixed serial.174c55aa000000004CC1
> sd8: 2861588MB, 4096 bytes/sector, 732566646 sectors
> uhidev1 at uhub2 port 1 configuration 1 interface 0 "Tangtop USB CAT5" rev
> 1.10/0.01 addr 2
> uhidev1: iclass 3/1
> ukbd0 at uhidev1: 8 variable keys, 6 key codes
> wskbd1 at ukbd0 mux 1
> uhidev2 at uhub2 port 1 configuration 1 interface 1 "Tangtop USB CAT5" rev
> 1.10/0.01 addr 2
> uhidev2: iclass 3/1
> ums0 at uhidev2: 3 buttons, Z dir
> wsmouse0 at ums0 mux 0
> vscsi0 at root
> scsibus8 at vscsi0: 256 targets
> softraid0 at root
> scsibus9 at softraid0: 256 targets
> softraid0: sd5 was not shutdown properly
> sd9 at scsibus9 targ 1 lun 0: <OPENBSD, SR RAID 5, 005> SCSI2 0/direct fixed
> sd9: 11446342MB, 512 bytes/sector, 23442109440 sectors
> softraid0: volume sd9 is roaming, it used to be sd5, updating metadata
> root on sd0a (4bc519b678dfdbe1.a) swap on sd0b dump on sd0b
> WARNING: / was not properly unmounted
> radeondrm0: 1024x768
> wsdisplay0 at radeondrm0 mux 1: console (std, vt100 emulation), using wskbd0
> wskbd1: connecting to wsdisplay0
> wsdisplay0: screen 1-5 added (std, vt100 emulation)
>
