Re: Experiences with OpenBSD RAID5
On 10/19/13 06:43, Scott McEachern wrote: List, I'm bringing you into the middle of an off-list conversation where I'm setting up a RAID10 array. Well, I'm using two RAID1 arrays as the drives for a RAID0 array. All relevant information follows. Any clue to why I'm ending up with an array 1/4 the size I'm expecting? On 10/18/13 23:16, Constantine A. Murenin wrote: No clue what you're talking about; I thought stacking works just fine since a few releases back. Are you sure it panic'ed with the partitions partitioned and specified correctly? Another question is whether you'd want to have a huge 6TB partition in OpenBSD -- generally something that's not advised. C. Hmm, I stand corrected. I must have done something wrong. Either way, I'm not quite getting the result I'd hoped for. Here's the details: - the 3TB drives in dmesg look like this: # dmesg|grep sd[0-9] sd0 at scsibus0 targ 0 lun 0: ATA, ST3000DM001-9YN1, CC4B SCSI3 0/direct fixed naa.5000c500525bf426 sd0: 2861588MB, 512 bytes/sector, 5860533168 sectors sd1 at scsibus0 targ 1 lun 0: ATA, ST3000DM001-9YN1, CC4B SCSI3 0/direct fixed naa.5000c5005265ff15 sd1: 2861588MB, 512 bytes/sector, 5860533168 sectors sd2 at scsibus0 targ 2 lun 0: ATA, ST3000DM001-9YN1, CC4B SCSI3 0/direct fixed naa.5000c5004a5baa2e sd2: 2861588MB, 512 bytes/sector, 5860533168 sectors sd3 at scsibus0 targ 3 lun 0: ATA, ST3000DM001-9YN1, CC4B SCSI3 0/direct fixed naa.5000c5004a6e56f1 sd3: 2861588MB, 512 bytes/sector, 5860533168 sectors sd4 at scsibus2 targ 0 lun 0: ATA, ST3000DM001-1CH1, CC43 SCSI3 0/direct fixed naa.5000c5004e455146 sd4: 2861588MB, 512 bytes/sector, 5860533168 sectors sd5 at scsibus2 targ 1 lun 0: ATA, ST3000DM001-1CH1, CC43 SCSI3 0/direct fixed naa.5000c5004e4a8141 sd5: 2861588MB, 512 bytes/sector, 5860533168 sectors [snip] The three RAID1 arrays are created from the above six HDDs like so: sd9 at scsibus4 targ 1 lun 0: OPENBSD, SR RAID 1, 005 SCSI2 0/direct fixed sd9: 2861588MB, 512 bytes/sector, 5860532576 sectors sd10 at scsibus4 targ 2 lun 0: OPENBSD, SR RAID 1, 005 SCSI2 0/direct fixed sd10: 2861588MB, 512 bytes/sector, 5860532576 sectors sd11 at scsibus4 targ 3 lun 0: OPENBSD, SR RAID 1, 005 SCSI2 0/direct fixed sd11: 2861588MB, 512 bytes/sector, 5860532576 sectors sd9 = sd0a + sd1a; sd10 = sd2a + sd3a; sd11 = sd4a + sd5a Observe: # bioctl -i sd9 Volume Status Size Device softraid0 0 Online 3000592678912 sd9 RAID1 0 Online 3000592678912 0:0.0 noencl sd1a 1 Online 3000592678912 0:1.0 noencl sd0a [ root@elminster:~ ] # bioctl -i sd10 Volume Status Size Device softraid0 1 Online 3000592678912 sd10RAID1 0 Online 3000592678912 1:0.0 noencl sd2a 1 Online 3000592678912 1:1.0 noencl sd3a [ root@elminster:~ ] # bioctl -i sd11 Volume Status Size Device softraid0 2 Online 3000592678912 sd11RAID1 0 Online 3000592678912 2:0.0 noencl sd4a 1 Online 3000592678912 2:1.0 noencl sd5a At this point, I have data on sd10, so I'll only use sd9 and sd11. Here are their (lightly snipped for brevity) disklabels: [ root@elminster:~ ] # disklabel -pg sd9 # /dev/rsd9c: label: SR RAID 1 duid: a7a8a62ef8e71b99 total sectors: 5860532576 # total bytes: 2794.5G boundstart: 0 boundend: 5860532576 16 partitions: #size offset fstype [fsize bsize cpg] a: 2794.5G0RAID c: 2794.5G0 unused [ root@elminster:~ ] # disklabel -pg sd11 # /dev/rsd11c: label: SR RAID 1 duid: 4b3e16399fbbbcf6 total sectors: 5860532576 # total bytes: 2794.5G boundstart: 0 boundend: 5860532576 16 partitions: #size offset fstype [fsize bsize cpg] a: 2794.5G0RAID c: 2794.5G0 unused As you can see above, all is looking good. sd10, which has data, was omitted. Now, the moment of truth... (I'm recreating this from memory..) # bioctl -c 0 -l /dev/sd9a,/dev/sd11a softraid0 I forget exactly what was said (it was one reboot ago), but I end up with this in dmesg: sd13 at scsibus4 targ 5 lun 0: OPENBSD, SR RAID 0, 005 SCSI2 0/direct fixed sd13: 1528871MB, 512 bytes/sector, 3131129344 sectors (BTW, I have a crypto volume in there as sd12, hence the jump from 11-13) Do you see the problem with the above? The disklabel makes it more obvious: # disklabel -pg sd13 # /dev/rsd13c: type: SCSI disk: SCSI disk label: SR RAID 0 duid: dcfed0a6c6b194e9 flags: bytes/sector: 512 sectors/track: 63 tracks/cylinder: 255 sectors/cylinder: 16065 cylinders: 194903 total sectors: 3131129344 # total bytes: 1493.0G boundstart: 0 boundend: 3131129344 drivedata: 0 16 partitions: #size offset fstype [fsize bsize cpg] a: 1493.0G0 4.2BSD 8192 655361 c: 1493.0G0 unused #
Re: Experiences with OpenBSD RAID5
On 10/19/13 13:10, Alexander Hall wrote: On 10/19/13 06:43, Scott McEachern wrote: List, I'm bringing you into the middle of an off-list conversation where I'm setting up a RAID10 array. Well, I'm using two RAID1 arrays as the drives for a RAID0 array. All relevant information follows. Any clue to why I'm ending up with an array 1/4 the size I'm expecting? On 10/18/13 23:16, Constantine A. Murenin wrote: No clue what you're talking about; I thought stacking works just fine since a few releases back. Are you sure it panic'ed with the partitions partitioned and specified correctly? Another question is whether you'd want to have a huge 6TB partition in OpenBSD -- generally something that's not advised. C. Hmm, I stand corrected. I must have done something wrong. Either way, I'm not quite getting the result I'd hoped for. Here's the details: - the 3TB drives in dmesg look like this: # dmesg|grep sd[0-9] sd0 at scsibus0 targ 0 lun 0: ATA, ST3000DM001-9YN1, CC4B SCSI3 0/direct fixed naa.5000c500525bf426 sd0: 2861588MB, 512 bytes/sector, 5860533168 sectors sd1 at scsibus0 targ 1 lun 0: ATA, ST3000DM001-9YN1, CC4B SCSI3 0/direct fixed naa.5000c5005265ff15 sd1: 2861588MB, 512 bytes/sector, 5860533168 sectors sd2 at scsibus0 targ 2 lun 0: ATA, ST3000DM001-9YN1, CC4B SCSI3 0/direct fixed naa.5000c5004a5baa2e sd2: 2861588MB, 512 bytes/sector, 5860533168 sectors sd3 at scsibus0 targ 3 lun 0: ATA, ST3000DM001-9YN1, CC4B SCSI3 0/direct fixed naa.5000c5004a6e56f1 sd3: 2861588MB, 512 bytes/sector, 5860533168 sectors sd4 at scsibus2 targ 0 lun 0: ATA, ST3000DM001-1CH1, CC43 SCSI3 0/direct fixed naa.5000c5004e455146 sd4: 2861588MB, 512 bytes/sector, 5860533168 sectors sd5 at scsibus2 targ 1 lun 0: ATA, ST3000DM001-1CH1, CC43 SCSI3 0/direct fixed naa.5000c5004e4a8141 sd5: 2861588MB, 512 bytes/sector, 5860533168 sectors [snip] The three RAID1 arrays are created from the above six HDDs like so: sd9 at scsibus4 targ 1 lun 0: OPENBSD, SR RAID 1, 005 SCSI2 0/direct fixed sd9: 2861588MB, 512 bytes/sector, 5860532576 sectors sd10 at scsibus4 targ 2 lun 0: OPENBSD, SR RAID 1, 005 SCSI2 0/direct fixed sd10: 2861588MB, 512 bytes/sector, 5860532576 sectors sd11 at scsibus4 targ 3 lun 0: OPENBSD, SR RAID 1, 005 SCSI2 0/direct fixed sd11: 2861588MB, 512 bytes/sector, 5860532576 sectors sd9 = sd0a + sd1a; sd10 = sd2a + sd3a; sd11 = sd4a + sd5a Observe: # bioctl -i sd9 Volume Status Size Device softraid0 0 Online 3000592678912 sd9 RAID1 0 Online 3000592678912 0:0.0 noencl sd1a 1 Online 3000592678912 0:1.0 noencl sd0a [ root@elminster:~ ] # bioctl -i sd10 Volume Status Size Device softraid0 1 Online 3000592678912 sd10RAID1 0 Online 3000592678912 1:0.0 noencl sd2a 1 Online 3000592678912 1:1.0 noencl sd3a [ root@elminster:~ ] # bioctl -i sd11 Volume Status Size Device softraid0 2 Online 3000592678912 sd11RAID1 0 Online 3000592678912 2:0.0 noencl sd4a 1 Online 3000592678912 2:1.0 noencl sd5a At this point, I have data on sd10, so I'll only use sd9 and sd11. Here are their (lightly snipped for brevity) disklabels: [ root@elminster:~ ] # disklabel -pg sd9 # /dev/rsd9c: label: SR RAID 1 duid: a7a8a62ef8e71b99 total sectors: 5860532576 # total bytes: 2794.5G boundstart: 0 boundend: 5860532576 16 partitions: #size offset fstype [fsize bsize cpg] a: 2794.5G0RAID c: 2794.5G0 unused [ root@elminster:~ ] # disklabel -pg sd11 # /dev/rsd11c: label: SR RAID 1 duid: 4b3e16399fbbbcf6 total sectors: 5860532576 # total bytes: 2794.5G boundstart: 0 boundend: 5860532576 16 partitions: #size offset fstype [fsize bsize cpg] a: 2794.5G0RAID c: 2794.5G0 unused As you can see above, all is looking good. sd10, which has data, was omitted. Now, the moment of truth... (I'm recreating this from memory..) # bioctl -c 0 -l /dev/sd9a,/dev/sd11a softraid0 I forget exactly what was said (it was one reboot ago), but I end up with this in dmesg: sd13 at scsibus4 targ 5 lun 0: OPENBSD, SR RAID 0, 005 SCSI2 0/direct fixed sd13: 1528871MB, 512 bytes/sector, 3131129344 sectors (BTW, I have a crypto volume in there as sd12, hence the jump from 11-13) Do you see the problem with the above? The disklabel makes it more obvious: # disklabel -pg sd13 # /dev/rsd13c: type: SCSI disk: SCSI disk label: SR RAID 0 duid: dcfed0a6c6b194e9 flags: bytes/sector: 512 sectors/track: 63 tracks/cylinder: 255 sectors/cylinder: 16065 cylinders: 194903 total sectors: 3131129344 # total bytes: 1493.0G boundstart: 0 boundend: 3131129344 drivedata: 0 16 partitions: #size offset fstype [fsize bsize cpg] a: 1493.0G0 4.2BSD 8192 655361 c:
Re: Experiences with OpenBSD RAID5
On 2013-10-18, Scott McEachern sc...@blackstaff.ca wrote: Circumstances change, and I might be able to redeploy those HDDs as a RAID5 array. This, at least in theory, would allow the 18TB total to be realized as 15TB as RAID5, gaining me 6TB. even if softraid would rebuild raid5, I'd worry about additional disk failures before/during rebuild for a volume of this sort of size.. (especially given that rebuilding is not automatic with softraid).
Re: Experiences with OpenBSD RAID5
On 10/18/13 07:31, Stuart Henderson wrote: On 2013-10-18, Scott McEachern sc...@blackstaff.ca wrote: Circumstances change, and I might be able to redeploy those HDDs as a RAID5 array. This, at least in theory, would allow the 18TB total to be realized as 15TB as RAID5, gaining me 6TB. even if softraid would rebuild raid5, I'd worry about additional disk failures before/during rebuild for a volume of this sort of size.. (especially given that rebuilding is not automatic with softraid). Follow-up: Thanks to all that replied publicly and privately, the information was most helpful. RAID5 can't rebuild, so that's a show stopper right there. However, now I understand why something I thought (at first) would be important has been left unwritten: RAID5 has its own lengthy set of problems. Like Stuart and others said, the potential for a secondary HDD failure causing a catastrophic failure to the entire volume is far greater than most people think. This link was given to me off-list, and it's worth the 60 seconds it takes to read: (It's short and to the point.) http://www.miracleas.com/BAARF/Why_RAID5_is_bad_news.pdf My primary goal with RAID is data integrity, with total capacity taking a back seat. As much as, in my case, 6TB seems like a rather large loss, the potential for RAID5 failure to gain that 6TB isn't worth it. Simply put, RAID1 (or even better, RAID10), is a superior course of action for data integrity. Assuming the numbers provided by CERN in that PDF are anywhere near accurate, it seems to me that using RAID5 is not only counter to the reason for RAID in the first place, but even reckless. Thanks again folks for the advice. I'm sticking to RAID1. -- Scott McEachern https://www.blackstaff.ca Beware the Four Horsemen of the Information Apocalypse: terrorists, drug dealers, kidnappers, and child pornographers. Seems like you can scare any public into allowing the government to do anything with those four. -- Bruce Schneier
Re: Experiences with OpenBSD RAID5
List, I'm bringing you into the middle of an off-list conversation where I'm setting up a RAID10 array. Well, I'm using two RAID1 arrays as the drives for a RAID0 array. All relevant information follows. Any clue to why I'm ending up with an array 1/4 the size I'm expecting? On 10/18/13 23:16, Constantine A. Murenin wrote: No clue what you're talking about; I thought stacking works just fine since a few releases back. Are you sure it panic'ed with the partitions partitioned and specified correctly? Another question is whether you'd want to have a huge 6TB partition in OpenBSD -- generally something that's not advised. C. Hmm, I stand corrected. I must have done something wrong. Either way, I'm not quite getting the result I'd hoped for. Here's the details: - the 3TB drives in dmesg look like this: # dmesg|grep sd[0-9] sd0 at scsibus0 targ 0 lun 0: ATA, ST3000DM001-9YN1, CC4B SCSI3 0/direct fixed naa.5000c500525bf426 sd0: 2861588MB, 512 bytes/sector, 5860533168 sectors sd1 at scsibus0 targ 1 lun 0: ATA, ST3000DM001-9YN1, CC4B SCSI3 0/direct fixed naa.5000c5005265ff15 sd1: 2861588MB, 512 bytes/sector, 5860533168 sectors sd2 at scsibus0 targ 2 lun 0: ATA, ST3000DM001-9YN1, CC4B SCSI3 0/direct fixed naa.5000c5004a5baa2e sd2: 2861588MB, 512 bytes/sector, 5860533168 sectors sd3 at scsibus0 targ 3 lun 0: ATA, ST3000DM001-9YN1, CC4B SCSI3 0/direct fixed naa.5000c5004a6e56f1 sd3: 2861588MB, 512 bytes/sector, 5860533168 sectors sd4 at scsibus2 targ 0 lun 0: ATA, ST3000DM001-1CH1, CC43 SCSI3 0/direct fixed naa.5000c5004e455146 sd4: 2861588MB, 512 bytes/sector, 5860533168 sectors sd5 at scsibus2 targ 1 lun 0: ATA, ST3000DM001-1CH1, CC43 SCSI3 0/direct fixed naa.5000c5004e4a8141 sd5: 2861588MB, 512 bytes/sector, 5860533168 sectors [snip] The three RAID1 arrays are created from the above six HDDs like so: sd9 at scsibus4 targ 1 lun 0: OPENBSD, SR RAID 1, 005 SCSI2 0/direct fixed sd9: 2861588MB, 512 bytes/sector, 5860532576 sectors sd10 at scsibus4 targ 2 lun 0: OPENBSD, SR RAID 1, 005 SCSI2 0/direct fixed sd10: 2861588MB, 512 bytes/sector, 5860532576 sectors sd11 at scsibus4 targ 3 lun 0: OPENBSD, SR RAID 1, 005 SCSI2 0/direct fixed sd11: 2861588MB, 512 bytes/sector, 5860532576 sectors sd9 = sd0a + sd1a; sd10 = sd2a + sd3a; sd11 = sd4a + sd5a Observe: # bioctl -i sd9 Volume Status Size Device softraid0 0 Online 3000592678912 sd9 RAID1 0 Online 3000592678912 0:0.0 noencl sd1a 1 Online 3000592678912 0:1.0 noencl sd0a [ root@elminster:~ ] # bioctl -i sd10 Volume Status Size Device softraid0 1 Online 3000592678912 sd10RAID1 0 Online 3000592678912 1:0.0 noencl sd2a 1 Online 3000592678912 1:1.0 noencl sd3a [ root@elminster:~ ] # bioctl -i sd11 Volume Status Size Device softraid0 2 Online 3000592678912 sd11RAID1 0 Online 3000592678912 2:0.0 noencl sd4a 1 Online 3000592678912 2:1.0 noencl sd5a At this point, I have data on sd10, so I'll only use sd9 and sd11. Here are their (lightly snipped for brevity) disklabels: [ root@elminster:~ ] # disklabel -pg sd9 # /dev/rsd9c: label: SR RAID 1 duid: a7a8a62ef8e71b99 total sectors: 5860532576 # total bytes: 2794.5G boundstart: 0 boundend: 5860532576 16 partitions: #size offset fstype [fsize bsize cpg] a: 2794.5G0RAID c: 2794.5G0 unused [ root@elminster:~ ] # disklabel -pg sd11 # /dev/rsd11c: label: SR RAID 1 duid: 4b3e16399fbbbcf6 total sectors: 5860532576 # total bytes: 2794.5G boundstart: 0 boundend: 5860532576 16 partitions: #size offset fstype [fsize bsize cpg] a: 2794.5G0RAID c: 2794.5G0 unused As you can see above, all is looking good. sd10, which has data, was omitted. Now, the moment of truth... (I'm recreating this from memory..) # bioctl -c 0 -l /dev/sd9a,/dev/sd11a softraid0 I forget exactly what was said (it was one reboot ago), but I end up with this in dmesg: sd13 at scsibus4 targ 5 lun 0: OPENBSD, SR RAID 0, 005 SCSI2 0/direct fixed sd13: 1528871MB, 512 bytes/sector, 3131129344 sectors (BTW, I have a crypto volume in there as sd12, hence the jump from 11-13) Do you see the problem with the above? The disklabel makes it more obvious: # disklabel -pg sd13 # /dev/rsd13c: type: SCSI disk: SCSI disk label: SR RAID 0 duid: dcfed0a6c6b194e9 flags: bytes/sector: 512 sectors/track: 63 tracks/cylinder: 255 sectors/cylinder: 16065 cylinders: 194903 total sectors: 3131129344 # total bytes: 1493.0G boundstart: 0 boundend: 3131129344 drivedata: 0 16 partitions: #size offset fstype [fsize bsize cpg] a: 1493.0G0 4.2BSD 8192 655361 c: 1493.0G0 unused # bioctl -i sd13 Volume
Experiences with OpenBSD RAID5
It wasn't planned this way, it just happened over time, but I ended up with a couple of machines with a few 3TB RAID1 volumes. As a result, around 18TB total (all HDDs are 3TB) is actually 9TB once RAID1'd. Circumstances change, and I might be able to redeploy those HDDs as a RAID5 array. This, at least in theory, would allow the 18TB total to be realized as 15TB as RAID5, gaining me 6TB. For now, I'm able to back up my RAID1 arrays to non-RAID HDDs. While this makes me nervous, I want to at least try creating a RAID5 setup to experiment on, with an eye to fully replacing RAID1 with RAID5. (6TB of lost capacity is nothing to sneeze at!) I realize the bioctl(8) man page says Use of the CRYPTO RAID 4/5 disciplines are currently considered experimental. However, I've been using RAID1 and CRYTPO in bootable and non-bootable configurations, including full disk encryption, quite happily for a while now. I've tested by removing drives, booting, rebuilding, etc., and I've never had a problem. So while the page may say experimental, I've found CRYPTO to be rock-solid thus far. I've never tried softraid's RAID5, so I have no idea if it truly is experimental, or solid like CRYPTO, or somewhere in between. So I ask my fellow list'ers for their experiences with OpenBSD's RAID5 implementation. (Yes, I'm running -current.) Does it work for you? Have you had any good experiences? Any bad experiences? Have you had to actually rebuild a RAID5 array? Can you add another drive to the array at a later time, or does the array need to be destroyed and rebuilt from scratch? Any questions I'm forgetting? I need to know about these things, from real-world use, before using RAID5 in the long-term. Also, in theory, with RAID5 you only lose one drive for the parity, hence my 18TB non-RAID = 15TB RAID5 math. Is this correct in practise with softraid? All stories are welcome, including private emails. Thanks, -- Scott McEachern https://www.blackstaff.ca Beware the Four Horsemen of the Information Apocalypse: terrorists, drug dealers, kidnappers, and child pornographers. Seems like you can scare any public into allowing the government to do anything with those four. -- Bruce Schneier
Re: Experiences with OpenBSD RAID5
On 10/17/13 20:07, Scott McEachern wrote: ... I realize the bioctl(8) man page says Use of the CRYPTO RAID 4/5 disciplines are currently considered experimental. ... I've never tried softraid's RAID5, so I have no idea if it truly is experimental, or solid like CRYPTO, or somewhere in between. So I ask my fellow list'ers for their experiences with OpenBSD's RAID5 implementation. (Yes, I'm running -current.) Does it work for you? with the exception of the fact there's no code to rebuild a failed disk, works great. that's a pretty big exception for most people. :) Have you had any good experiences? Any bad experiences? Have you had to actually rebuild a RAID5 array? you should be trying this stuff yourself. Doesn't matter if *I* can rebuild your array, it matters if YOU can. You don't have to practice on 3TB disks, you can practice on 10GB disks...though understanding the time required for 3TB disks would be wise. Can you add another drive to the array at a later time, or does the array need to be destroyed and rebuilt from scratch? we'll have to get rebuild working before we get restructure. :) Also, in theory, with RAID5 you only lose one drive for the parity, hence my 18TB non-RAID = 15TB RAID5 math. Is this correct in practise with softraid? other than a 3TB disk is closer to 2.75TB than 3TB, yeah the math works the same with softraid as it does with hw raid. Nick.
Re: Experiences with OpenBSD RAID5
On 10/17/13 20:57, Nick Holland wrote: with the exception of the fact there's no code to rebuild a failed disk, works great. that's a pretty big exception for most people. :) Hmm. That would present a problem. Let me make sure I'm absolutely clear here: A RAID5 array with four disks. I notice via bioctl(8) that one has failed. I pull the failed disk and replace it with my cold spare. I cannot use bioctl -R to incorporate the new disk into the array. Correct? (BTW, I know for a fact it works properly with RAID1.) So basically, if a drive fails, the RAID5 array is permanently borked until completely recreated with a new array, meanwhile I'll be trying to back up my data somewhere else. you should be trying this stuff yourself. Doesn't matter if *I* can rebuild your array, it matters if YOU can. You don't have to practice on 3TB disks, you can practice on 10GB disks...though understanding the time required for 3TB disks would be wise. As you've often advised in the past, test it yourself. I plan on it, just to see what happens. BTW, I tried it once with a 3TB RAID1, and I believe it took two days, but I could be wrong and is hardware-dependent. Either way, it takes a /long/-ass time. other than a 3TB disk is closer to 2.75TB than 3TB, yeah the math works the same with softraid as it does with hw raid. Nick. Yes, sorry for not making clear I realize that. I didn't want to do the usable space, GB/GiB, TB/TiB dance. You know what I mean.., hopefully. Either way, while I have the opportunity, I'm going to create some RAID5 arrays of varying sizes, pull (fail) disks, etc., and just see what happens for myself. Experimenting is fun! Unfortunately, if RAID5 cannot rebuild anything, it's about as useful as a leaky screwdriver and RAID1 remains the only viable option on OpenBSD. Damn. Thanks Nick, as always you're a gem of a resource. -- Scott McEachern https://www.blackstaff.ca Beware the Four Horsemen of the Information Apocalypse: terrorists, drug dealers, kidnappers, and child pornographers. Seems like you can scare any public into allowing the government to do anything with those four. -- Bruce Schneier
Re: Experiences with OpenBSD RAID5
On 10/17/13 21:34, Scott McEachern wrote: On 10/17/13 20:57, Nick Holland wrote: with the exception of the fact there's no code to rebuild a failed disk, works great. that's a pretty big exception for most people. :) Hmm. That would present a problem. Let me make sure I'm absolutely clear here: A RAID5 array with four disks. I notice via bioctl(8) that one has failed. I pull the failed disk and replace it with my cold spare. I cannot use bioctl -R to incorporate the new disk into the array. Correct? (BTW, I know for a fact it works properly with RAID1.) yep. works great for RAID1, not for RAID5. If you think about it, it's a pretty different process between the two. The RAID5 process is not yet written. So basically, if a drive fails, the RAID5 array is permanently borked until completely recreated with a new array, meanwhile I'll be trying to back up my data somewhere else. yep. basically, your RAID5 turns into RAID0 (with a performance hit), and you will need to dump/reload your data before it could be redundant again. Not that this might be without use. At my day job, I've got a machine set up that people use for analyzing large chunks of data. They copy it in, crunch it, then dump it. We are using RAID0 at the moment, but softraid RAID5 could have been used to give a little more cushion when the disk fails, so at least the current analysis could be continued. you should be trying this stuff yourself. Doesn't matter if *I* can rebuild your array, it matters if YOU can. You don't have to practice on 3TB disks, you can practice on 10GB disks...though understanding the time required for 3TB disks would be wise. As you've often advised in the past, test it yourself. hey, I'm redundant! :D I plan on it, just to see what happens. BTW, I tried it once with a 3TB RAID1, and I believe it took two days, but I could be wrong and is hardware-dependent. Either way, it takes a /long/-ass time. exactly. you don't want to promise replace the drive tonight and tomorrow all will be great if it takes two days. RAID5 should have a much longer rebuild time (RAID1: read once, write once. N disk RAID5: read N-1, write once). Also keep in mind that disk system performance during rebuild may be unacceptable for some applications. other than a 3TB disk is closer to 2.75TB than 3TB, yeah the math works the same with softraid as it does with hw raid. Yes, sorry for not making clear I realize that. I didn't want to do the usable space, GB/GiB, TB/TiB dance. You know what I mean.., hopefully. yes, I knew what you meant. I'm just pissed at disk vendors. it's been what, over 15 years they've been doing this? I'm still pissed. :) Either way, while I have the opportunity, I'm going to create some RAID5 arrays of varying sizes, pull (fail) disks, etc., and just see what happens for myself. Experimenting is fun! Unfortunately, if RAID5 cannot rebuild anything, it's about as useful as a leaky screwdriver and RAID1 remains the only viable option on OpenBSD. Damn. Thanks Nick, as always you're a gem of a resource. :) Nick.
