I have been experimenting with a softraid mirror, using two cheap SATA disks. In general the performance is very good - except when rebuilding. A quick set of sums suggests that the problem is seek time.
The disks are 7200rpm, therefore one can hope for 120 seeks per second. "systat iostat" (while rebuilding) gives this (edited to fit in an email): DEVICE READ WRITE RTPS WTPS SEC wd0 3801088 3827302 58 59 1.0 wd1 0 3801088 0 58 0.0 sd0 0 0 0 0 0.0 Totals 3801088 7628390 58 117 1.0 When not rebuilding, the disks can stream data at ~100MB/s. Whereas while rebuilding the speed is closer to 3.8MB/s - which seems to be cause by the 64KB softraid rebuild blocksize and the seek time of the disk. (3.8MB/s = 64KB/block * 60block/s). Interestingly, even though I have a mirror, the rebuild process is writing to both disks during the rebuild. --- So... given that my raid will take ~5days to rebuild when I expect a few hours I have started to thing about how to 'fix' this problem. My instinct says that there are two ways to attack this problem. 1) I could modify /src/sys/dev/softraid.c so that it consumes much more kernel memory and rebuilds in blocks of (say) 1MB, rather than 64k. Which would work for anyone else having rebuild performance problems on any raid level. 2) I could, instead, modify /src/sys/dev/softraid_raid1.c so that it doesn't write back to source chunk when rebuilding. Obviously I want 'normal' IO to be written to all disks when rebuilding, so I'll need to be careful. --- My question is: has anyone else come across this issue before, and if so what did they do? Google fails me... so please don't say "google it".My second question is: which of the two ideas above has most merit? - and what should I do with the patch afterwards (assuming that it isn't nasty)?
---
Kernel is build by me from anoncvs dated 11-Apr-2010 17:30ish GMT:
OpenBSD 4.7-current (GENERIC) #0: Sun Apr 11 18:31:51 GMT 2010
[email protected]:/usr/src/sys/arch/i386/compile/GENERIC
cpu0: Intel(R) Pentium(R) 4 CPU 3.20GHz ("GenuineIntel" 686-class) 3.21 GHz
cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,CNXT-ID,xTPR
real mem = 2146988032 (2047MB) avail mem = 2071244800 (1975MB) mainbus0 at rootbios0 at mainbus0: AT/286+ BIOS, date 08/19/04, BIOS32 rev. 0 @ 0xfb060, SMBIOS rev. 2.2 @ 0xf0000 (34 entries)
bios0: vendor Phoenix Technologies, LTD version "6.00 PG" date 08/19/2004 bios0: Shuttle Inc FB71 acpi0 at bios0: rev 0 acpi0: tables DSDT FACP APICacpi0: wakeup devices HUB0(S5) UAR1(S5) UAR2(S5) USB0(S1) USB1(S1) USB2(S1) USB3(S1) USBE(S1) MODM(S5) PCI0(S5)
acpitimer0 at acpi0: 3579545 Hz, 24 bits acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat cpu0 at mainbus0: apid 0 (boot processor) cpu0: apic clock running at 200MHz cpu at mainbus0: not configured ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins ioapic0: misconfigured as apic 0, remapped to apid 2 acpiprt0 at acpi0: bus 0 (PCI0) acpiprt1 at acpi0: bus 2 (HUB0) acpicpu0 at acpi0 acpitz0 at acpi0: critical temperature 100 degC acpibtn0 at acpi0: PWRB bios0: ROM list: 0xc0000/0xde00 0xd0000/0x8000! pci0 at mainbus0 bus 0: configuration mode 1 (bios) pchb0 at pci0 dev 0 function 0 "Intel 82875P Host" rev 0x02 intelagp0 at pchb0 agp0 at intelagp0: aperture at 0xd0000000, size 0x10000000 ppb0 at pci0 dev 1 function 0 "Intel 82875P AGP" rev 0x02 pci1 at ppb0 bus 1 vga1 at pci1 dev 0 function 0 "NVIDIA GeForce 7600 GS" rev 0xa2 wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation) wsdisplay0: screen 1-5 added (80x25, vt100 emulation)uhci0 at pci0 dev 29 function 0 "Intel 82801EB/ER USB" rev 0x02: apic 2 int 16 (irq 5) uhci1 at pci0 dev 29 function 1 "Intel 82801EB/ER USB" rev 0x02: apic 2 int 19 (irq 10) uhci2 at pci0 dev 29 function 2 "Intel 82801EB/ER USB" rev 0x02: apic 2 int 18 (irq 11) uhci3 at pci0 dev 29 function 3 "Intel 82801EB/ER USB" rev 0x02: apic 2 int 16 (irq 5) ehci0 at pci0 dev 29 function 7 "Intel 82801EB/ER USB2" rev 0x02: apic 2 int 23 (irq 9)
usb0 at ehci0: USB revision 2.0 uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1 ppb1 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xc2 pci2 at ppb1 bus 2 "VIA VT6306 FireWire" rev 0x80 at pci2 dev 6 function 0 not configuredbge0 at pci2 dev 7 function 0 "Broadcom BCM5788" rev 0x03, BCM5705 A3 (0x3003): apic 2 int 19 (irq 10), address 00:30:1b:b2:ea:70
brgphy0 at bge0 phy 1: BCM5705 10/100/1000baseT PHY, rev. 2ste0 at pci2 dev 8 function 0 "D-Link Systems 550TX" rev 0x12: apic 2 int 20 (irq 12), address 00:05:5d:e0:7f:be ukphy0 at ste0 phy 0: Generic IEEE 802.3u media interface, rev. 0: OUI 0x000885, model 0x0023
ichpcib0 at pci0 dev 31 function 0 "Intel 82801EB/ER LPC" rev 0x02pciide0 at pci0 dev 31 function 1 "Intel 82801EB/ER IDE" rev 0x02: DMA, channel 0 configured to compatibility, channel 1 configured to compatibility
pciide0: channel 0 disabled (no drives) pciide0: channel 1 disabled (no drives)pciide1 at pci0 dev 31 function 2 "Intel 82801EB SATA" rev 0x02: DMA, channel 0 configured to native-PCI, channel 1 configured to native-PCI
pciide1: using apic 2 int 18 (irq 11) for native-PCI interrupt wd0 at pciide1 channel 0 drive 0: <Hitachi HDS722020ALA330> wd0: 16-sector PIO, LBA48, 1907729MB, 3907029168 sectors wd0(pciide1:0:0): using PIO mode 4, Ultra-DMA mode 6 wd1 at pciide1 channel 1 drive 0: <Hitachi HDS722020ALA330> wd1: 16-sector PIO, LBA48, 1907729MB, 3907029168 sectors wd1(pciide1:1:0): using PIO mode 4, Ultra-DMA mode 6ichiic0 at pci0 dev 31 function 3 "Intel 82801EB/ER SMBus" rev 0x02: apic 2 int 17 (irq 7)
iic0 at ichiic0 "it8712" at iic0 addr 0x2d not configurediic0: addr 0x2d 00=11 01=10 02=00 03=07 04=00 05=00 06=00 07=00 08=00 09=00 0a=0c 0b=64 0c=2f 0d=2b 0f=23 13=71 14=f7 15=81 16=00 17=81 18=2f 19=2f 1a=2f 1b=2f 1c=2f 1d=2f 1e=2f 1f=2f 20=5b 21=5e 22=cf 23=bd 24=be 25=40 26=a0 27=b7 28=cd 29=27 2a=2f 2b=31 2c=2f 2d=2f 2e=2f 2f=2f 48=2d 51=2a 52=7f 53=71 54=f7 58=90 59=65 5b=12 5c=80 5d=00 5e=00 5f=00 60=01 61=02 62=34 63=39 64=3e 65=05 66=08 67=0c 68=01 69=02 6a=34 6b=39 6c=3e 6d=05 6e=08 6f=0c 70=01 71=02 72=34 73=39 74=3e 75=7f 76=7f 77=7f 80=11 81=10 82=00 83=00 84=00 85=00 86=00 87=00 88=00 89=00 8a=0c 8b=64 8c=2f 8d=2b 8f=23 93=71 94=f7 95=81 96=00 97=81 98=2f 99=2f 9a=2f 9b=2f 9c=2f 9d=2f 9e=2f 9f=2f a0=5c a1=5c a2=cf a3=bb a4=bd a5=3f a6=a1 a7=b9 a8=cd a9=27 aa=2f ab=31 ac=2f ad=2f ae=2f af=2f c8=2d d1=2a d2=7f d3=71 d4=f7 d8=90 d9=65 db=12 dc=80 dd=00 de=00 df=00 e0=01 e1=02 e2=34 e3=39 e4=3e e5=05 e6=08 e7=0c e8=01 e9=02 ea=34 eb=39 ec=3e ed=05 ee=08 ef=0c f0=01 f1=02 f2=34 f3=39 f4=3e f5=7f f6=7f f7=7f words 00=11ff 01=10ff 02=00ff 03=00ff 04=00ff 05=00ff 06=00ff 07=00ff: it8712
spdmem0 at iic0 addr 0x50: 1GB DDR SDRAM non-parity PC3200CL3.0 spdmem1 at iic0 addr 0x52: 1GB DDR SDRAM non-parity PC3200CL3.0auich0 at pci0 dev 31 function 5 "Intel 82801EB/ER AC97" rev 0x02: apic 2 int 17 (irq 7), ICH5 AC97
ac97: codec id 0x414c4720 (Avance Logic ALC650) ac97: codec features 20 bit DAC, 18 bit ADC, Realtek 3D audio0 at auich0 usb1 at uhci0: USB revision 1.0 uhub1 at usb1 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb2 at uhci1: USB revision 1.0 uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb3 at uhci2: USB revision 1.0 uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1 usb4 at uhci3: USB revision 1.0 uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1 isa0 at ichpcib0 isadma0 at isa0 com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo pckbc0 at isa0 port 0x60/5 pckbd0 at pckbc0 (kbd slot) pckbc0: using irq 1 for kbd slot wskbd0 at pckbd0: console keyboard, using wsdisplay0 pcppi0 at isa0 port 0x61 midi0 at pcppi0: <PC speaker> spkr0 at pcppi0 it0 at isa0 port 0x2e/2: IT8712F rev 5, EC port 0x290 npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16 mtrr: Pentium Pro MTRR support vscsi0 at root scsibus0 at vscsi0: 256 targets softraid0 at root softraid0: sd0 was not shutdown properly scsibus1 at softraid0: 1 targets sd0 at scsibus1 targ 0 lun 0: <OPENBSD, SR RAID 1, 004> SCSI2 0/direct fixed sd0: 1904643MB, 512 bytes/sec, 3900709993 sec total root on wd0a swap on wd0b dump on wd0b
