Re: [CentOS] RAID 5 setup?
Benjamin Franz wrote: Robert Heller wrote: I suspect that this is a simular case to what I did: I have a server with 4 drives. I have several (small) RAID1 partitions (/boot, /, /usr, /var, etc.) with 4 mirrors and one large RAID5 with three partitions and a hot spare (a LVM volumn group, containing /home and some other partitions). I would guess that the admin with the 8-way RAID1 for the OS probably also has a 6 or 8 disk RAID5 or RAID6 for the bulk of the disks Yup. 8 way RAID1 for the OS, 8 way RAID6 for the data. I was hoping when I setup the 8-way RAID1 for the OS that I would get really good read speeds since md is supposed to stripe reads from RAID1, but in practice the RAID6 completely kills it for read performance (~61 MB/sec from the RAID1 partition vs ~200 MB/sec from the RAID6 partition). What are you running? I think there was a patch that evened out the reads across all members as it would at times solely read from one and then another... ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
Robert Spangler wrote: On Thursday 25 March 2010 18:10, Robert Heller wrote: The prefered way to go would be RAID10 (RAID1 (mirror) + RAID0 (stripe)). Form pairs as RAID1, then strip the pairs. With 8 disks, this would 4 pairs, 1.5TB/pair = 1.5*4 = 6TB total. I am just starting to look into this RAID and I was wondering wouldn't RAID01 be better then RAID10? In a 4 disc system having the first two using stripping and then backing them up the second set with mirrors? My though is having D1 and D2 as the primary drives stripping and then having D3 backup D1 and D4 backup D2. And if enough room place a couple more drives in the system as hot standby's. Or am I looking at this all wrong? for all practical purposes its the same thing. if it was really stripe then mirror, a naive mirror handler would think it would have to remirror both drives when one half of one of the stripesets failed and was replaced. but in fact, the mirror handlres tend to be well aware of whats going on. mirror 0+1 aand stripe that with mirrored 2+3, and its really all the samethe native raid10 in newer mdraid is cleaner because you don't end up with extra partial volume metadevices... ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
On Mar 27, 2010, at 5:07 AM, John R Pierce pie...@hogranch.com wrote: Robert Spangler wrote: On Thursday 25 March 2010 18:10, Robert Heller wrote: The prefered way to go would be RAID10 (RAID1 (mirror) + RAID0 (stripe)). Form pairs as RAID1, then strip the pairs. With 8 disks, this would 4 pairs, 1.5TB/pair = 1.5*4 = 6TB total. I am just starting to look into this RAID and I was wondering wouldn't RAID01 be better then RAID10? In a 4 disc system having the first two using stripping and then backing them up the second set with mirrors? My though is having D1 and D2 as the primary drives stripping and then having D3 backup D1 and D4 backup D2. And if enough room place a couple more drives in the system as hot standby's. Or am I looking at this all wrong? for all practical purposes its the same thing. if it was really stripe then mirror, a naive mirror handler would think it would have to remirror both drives when one half of one of the stripesets failed and was replaced. but in fact, the mirror handlres tend to be well aware of whats going on. mirror 0+1 aand stripe that with mirrored 2+3, and its really all the samethe native raid10 in newer mdraid is cleaner because you don't end up with extra partial volume metadevices... RAID0+1 is never a good configuration because a single drive failure in a RAID0 stripe fails out the whole stripe, and with say an 4x2 RAID0+1, you are more likely to hit a disk failure with 4 drives in a RAID0 then 2 in a RAID1. That's why RAID1+0 came about. -Ross ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
Chan Chung Hang Christopher wrote: Benjamin Franz wrote: Yup. 8 way RAID1 for the OS, 8 way RAID6 for the data. I was hoping when I setup the 8-way RAID1 for the OS that I would get really good read speeds since md is supposed to stripe reads from RAID1, but in practice the RAID6 completely kills it for read performance (~61 MB/sec from the RAID1 partition vs ~200 MB/sec from the RAID6 partition). What are you running? I think there was a patch that evened out the reads across all members as it would at times solely read from one and then another... I'm running fully up-to-date CentOS 5.4, kernel 2.6.18-164.15.1. The test was done with a default run of bonnie++. Watching the disk I/O while it ran suggested that it was only using some of the disks at a time to read (but changing which ones every few seconds). 8 x RAID1 Version 1.96 --Sequential Output-- --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP pbox16.freerun. 16G 586 95 17263 5 13016 1 2263 97 60793 3 644.2 4 Latency 203ms 188s1521ms 14714us 557ms 1265ms Version 1.96 --Sequential Create-- Random Create pbox16 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 12418 16 + +++ 10999 12 27730 34 + +++ 29916 32 Latency 109us 638us 681us 200us 25us 39us 8 x RAID6 Version 1.96 --Sequential Output-- --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP pbox16.freerun. 16G 611 96 90060 15 55711 14 2407 96 200901 27 522.3 15 Latency 13157us 604ms1727ms 32420us 142ms 73131us Version 1.96 --Sequential Create-- Random Create pbox16 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 10298 16 + +++ 24208 32 30395 45 + +++ 32511 43 Latency 18504us 598us 613us 87us 25us 121us 1.96,1.96,pbox16,1,1269604576,16G,,611,96,90060,15,55711,14,2407,96,200901,27,522.3,15,16,10298,16,+,+++,24208,32,30395,45,+,+++,32511,43,13157us,604ms,1727ms,32420us,142ms,73131us,18504us,598us,613us,87us,25us,121us -- Benjamin Franz ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
On Saturday 27 March 2010 05:07, John R Pierce wrote: for all practical purposes its the same thing. if it was really stripe then mirror, a naive mirror handler would think it would have to remirror both drives when one half of one of the stripesets failed and was replaced. but in fact, the mirror handlres tend to be well aware of whats going on. mirror 0+1 aand stripe that with mirrored 2+3, and its really all the samethe native raid10 in newer mdraid is cleaner because you don't end up with extra partial volume metadevices... Thank you kindly for your reply. -- Regards Robert Linux User #296285 http://counter.li.org ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
On Saturday 27 March 2010 09:22, Ross Walker wrote: for all practical purposes its the same thing. if it was really stripe then mirror, a naive mirror handler would think it would have to remirror both drives when one half of one of the stripesets failed and was replaced. but in fact, the mirror handlres tend to be well aware of whats going on. mirror 0+1 aand stripe that with mirrored 2+3, and its really all the samethe native raid10 in newer mdraid is cleaner because you don't end up with extra partial volume metadevices... RAID0+1 is never a good configuration because a single drive failure in a RAID0 stripe fails out the whole stripe, and with say an 4x2 RAID0+1, you are more likely to hit a disk failure with 4 drives in a RAID0 then 2 in a RAID1. That's why RAID1+0 came about. Thnx for clearing this up. -- Regards Robert Linux User #296285 http://counter.li.org ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
On Mar 25, 2010, at 9:36 PM, Christopher Chan christopher.c...@bradbury.edu.hk wrote: On Friday, March 26, 2010 09:12 AM, John R Pierce wrote: Christopher Chan wrote: but with RAID 10, data is safe after many types of failures. Except for the case when a mirror dies after which the whole thing is toast but in theory you can survive up to four disks going down. if you have a 8 drive raid1+0, and a random drive fails, you can survive any other drive failing *except* the mirror of the failed one. so if a That's what I said right? I did not say when a mirror is broken... second drive fails, there's only a 1 in 7 chance that its the 'fatal' one. on a 4 drive raid10, its a 1 in 3 chance. meanwhile, a raid10 can rebuild from a hotspare in like an hour, if the system isn't busy, and a few hours in the background if its busy and active. yeah, I thought the raid10 module would be able to rebuild automatically from a hotspare and would therefore be better than using nested raid1+0. I better stop the nested raid1+0 thing...does ananconda support the raid10 module during install yet? I mean rather, is the raid10 module included in the installation initrd yet? No, not yet, but I always recommend setting up your data arrays manually so your intimately familiar with how they are constructed and the mdadm command usage is fresh in your head. Did you know with Neil's raid10 implementation you can store 3 copies of the data so ANY two drives can fail before you start playing Russian roulette! -Ross ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
Ross Walker wrote: No, not yet, but I always recommend setting up your data arrays manually so your intimately familiar with how they are constructed and the mdadm command usage is fresh in your head. Did you know with Neil's raid10 implementation you can store 3 copies of the data so ANY two drives can fail before you start playing Russian roulette! You can do that with RAID1+0, too. You can setup RAID1 with more than 2 drives. I have one system with an 8-way RAID1 for the OS. -- Benjamin Franz ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
Benjamin Franz wrote: Ross Walker wrote: No, not yet, but I always recommend setting up your data arrays manually so your intimately familiar with how they are constructed and the mdadm command usage is fresh in your head. Did you know with Neil's raid10 implementation you can store 3 copies of the data so ANY two drives can fail before you start playing Russian roulette! You can do that with RAID1+0, too. You can setup RAID1 with more than 2 drives. I have one system with an 8-way RAID1 for the OS. That's some serious redundancy dude! Good for reads too... ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
I have one system with an 8-way RAID1 for the OS. For real or is that a typo? Is that incase you go on holiday for a week and a drive-dies-a-day? -- Regards, James. http://www.jamesbensley.co.cc/ ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
At Fri, 26 Mar 2010 17:16:04 + CentOS mailing list centos@centos.org wrote: I have one system with an 8-way RAID1 for the OS. For real or is that a typo? Is that incase you go on holiday for a week and a drive-dies-a-day? I suspect that this is a simular case to what I did: I have a server with 4 drives. I have several (small) RAID1 partitions (/boot, /, /usr, /var, etc.) with 4 mirrors and one large RAID5 with three partitions and a hot spare (a LVM volumn group, containing /home and some other partitions). I would guess that the admin with the 8-way RAID1 for the OS probably also has a 6 or 8 disk RAID5 or RAID6 for the bulk of the disks. -- Robert Heller -- 978-544-6933 Deepwoods Software-- Download the Model Railroad System http://www.deepsoft.com/ -- Binaries for Linux and MS-Windows hel...@deepsoft.com -- http://www.deepsoft.com/ModelRailroadSystem/ ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
Robert Heller wrote: I suspect that this is a simular case to what I did: I have a server with 4 drives. I have several (small) RAID1 partitions (/boot, /, /usr, /var, etc.) with 4 mirrors and one large RAID5 with three partitions and a hot spare (a LVM volumn group, containing /home and some other partitions). I would guess that the admin with the 8-way RAID1 for the OS probably also has a 6 or 8 disk RAID5 or RAID6 for the bulk of the disks Yup. 8 way RAID1 for the OS, 8 way RAID6 for the data. I was hoping when I setup the 8-way RAID1 for the OS that I would get really good read speeds since md is supposed to stripe reads from RAID1, but in practice the RAID6 completely kills it for read performance (~61 MB/sec from the RAID1 partition vs ~200 MB/sec from the RAID6 partition). In a deeply ironic turn of events, one of the hard drives in that machine died in a way that freaked the hardware controller driver out and caused a kernel panic last week. The machine wouldn't finishing booting until I physically removed the bad drive. The RAID1 was fine afterwards, but the RAID6 had to be manually re-assembled (no corruption, it just wouldn't automatically start until I intervened). -- Benjamin Franz ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
On 3/26/2010 12:45 PM, Robert Heller wrote: At Fri, 26 Mar 2010 17:16:04 + CentOS mailing listcentos@centos.org wrote: I have one system with an 8-way RAID1 for the OS. For real or is that a typo? Is that incase you go on holiday for a week and a drive-dies-a-day? I suspect that this is a simular case to what I did: I have a server with 4 drives. I have several (small) RAID1 partitions (/boot, /, /usr, /var, etc.) with 4 mirrors and one large RAID5 with three partitions and a hot spare (a LVM volumn group, containing /home and some other partitions). I would guess that the admin with the 8-way RAID1 for the OS probably also has a 6 or 8 disk RAID5 or RAID6 for the bulk of the disks. If you are really paranoid you can split things up into raid1 mirrors of 2 drives each mounted into logical places. At the expense of having to mange the space in smaller chunks (and losing half to redundancy) you get the ability to control head contention among jobs and to recover data from any single disk after a problem. -- Les Mikesell lesmikes...@gmail.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
On Mar 26, 2010, at 10:25 AM, Benjamin Franz jfr...@freerun.com wrote: Ross Walker wrote: No, not yet, but I always recommend setting up your data arrays manually so your intimately familiar with how they are constructed and the mdadm command usage is fresh in your head. Did you know with Neil's raid10 implementation you can store 3 copies of the data so ANY two drives can fail before you start playing Russian roulette! You can do that with RAID1+0, too. You can setup RAID1 with more than 2 drives. I have one system with an 8-way RAID1 for the OS. Yes, but you can use a lot less drives to get the same IOPS with Neil's module (which technically isn't RAID10 at all but a whole new RAID level). -Ross ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
On Thursday 25 March 2010 18:10, Robert Heller wrote: The prefered way to go would be RAID10 (RAID1 (mirror) + RAID0 (stripe)). Form pairs as RAID1, then strip the pairs. With 8 disks, this would 4 pairs, 1.5TB/pair = 1.5*4 = 6TB total. I am just starting to look into this RAID and I was wondering wouldn't RAID01 be better then RAID10? In a 4 disc system having the first two using stripping and then backing them up the second set with mirrors? My though is having D1 and D2 as the primary drives stripping and then having D3 backup D1 and D4 backup D2. And if enough room place a couple more drives in the system as hot standby's. Or am I looking at this all wrong? -- Regards Robert Linux User #296285 http://counter.li.org ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
On 3/25/2010 2:24 PM, Slack-Moehrle wrote: Can anyone provide a tutorial or advice on how to configure a software RAID 5 from the command-line (since I did not install Gnome)? I have 8 x 1.5tb Drives. Make matching partitions on each disk with fdisk, setting the type to FD (raid autodetect), then 'mdadm create ...' to specify the options and start it. See the create section in 'man mdadm'. You'll need at least --raid-level= --raid-devices= --auto=yes. Then you'll probably want to add an entry in /etc/fstab to mount the new md device somewhere. -- Les Mikesell lesmikes...@gmail.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
At Thu, 25 Mar 2010 12:24:57 -0700 (PDT) CentOS mailing list centos@centos.org wrote: Can anyone provide a tutorial or advice on how to configure a software RAID 5 from the command-line (since I did not install Gnome)? I have 8 x 1.5tb Drives. mdadm --create /dev/md0 --level=5 --raid-devices=7 /dev/sd[abcdefg]1 The above will create a level 5 RAID named /dev/md0 of /dev/sda1 /dev/sdb1 /dev/sda1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1, with hot-spare /dev/sdg1 Note: RAID5 is not really recomended for such large disks. You run the risk of a complete data loss if one disk fails and the another disk fails during the rebuild. -Jason ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos -- Robert Heller -- 978-544-6933 Deepwoods Software-- Download the Model Railroad System http://www.deepsoft.com/ -- Binaries for Linux and MS-Windows hel...@deepsoft.com -- http://www.deepsoft.com/ModelRailroadSystem/ ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
I used this guide for my first RAID on an Ubuntu box, its very straight forward. Its all command line based so everything here I have used in CentOS (apart from the writer sets the RAID flag on his drives via the GParted GUI but this can be done via terminal); http://bfish.xaedalus.net/2006/11/software-raid-5-in-ubuntu-with-mdadm/ -- Regards, James. http://www.jamesbensley.co.cc/ ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
On Thu, Mar 25, 2010 at 3:36 PM, Robert Heller hel...@deepsoft.com wrote: At Thu, 25 Mar 2010 12:24:57 -0700 (PDT) CentOS mailing list centos@centos.org wrote: Can anyone provide a tutorial or advice on how to configure a software RAID 5 from the command-line (since I did not install Gnome)? I have 8 x 1.5tb Drives. mdadm --create /dev/md0 --level=5 --raid-devices=7 /dev/sd[abcdefg]1 The above will create a level 5 RAID named /dev/md0 of /dev/sda1 /dev/sdb1 /dev/sda1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1, with hot-spare /dev/sdg1 Note: RAID5 is not really recomended for such large disks. You run the risk of a complete data loss if one disk fails and the another disk fails during the rebuild. -Jason ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos -- Robert Heller -- 978-544-6933 Deepwoods Software -- Download the Model Railroad System http://www.deepsoft.com/ -- Binaries for Linux and MS-Windows hel...@deepsoft.com -- http://www.deepsoft.com/ModelRailroadSystem/ ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos Robert, Why is the size a factor here? Why would this be OK with smaller disks? How would you partition this instead? Thanks. Boris. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
On Thu, Mar 25, 2010 at 9:07 PM, Boris Epstein borepst...@gmail.com wrote: Note: RAID5 is not really recomended for such large disks. You run the risk of a complete data loss if one disk fails and the another disk fails during the rebuild. Why is the size a factor here? Why would this be OK with smaller disks? How would you partition this instead? As the disks get bigger, rebuild time also increases and the performance of the disks don't increase linearly with their storage. This means that when you are rebuilding a disk, the chances of one of your other disks failing becomes significantly large. Most suggest RAID6 these days as a minimum, mirroring and striping appears to be the most popular. -- Hakan (m1fcj) - http://www.hititgunesi.org ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
Am 25.03.2010 um 22:07 schrieb Boris Epstein: Robert, Why is the size a factor here? Why would this be OK with smaller disks? How would you partition this instead? Thanks. Boris. This has been discussed before. The root of the problem lies in the fact that when a disk fails, you have to read-out the data from the other disks to re-build the RAID. Reads from disks have a certain probability to contain an error. The larger the disk and the larger the array, the more probable it is to encounter such an error while rebuilding the RAID (and if that happens, you're RAID is just a piece of scrap-metal) http://www.google.com/search?q=the+end+of+raid RAID5 works OK-ish for a couple of 146GB SAS-disks. Rainer___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
On Thu, Mar 25, 2010 at 5:27 PM, Hakan Koseoglu ha...@koseoglu.org wrote: On Thu, Mar 25, 2010 at 9:07 PM, Boris Epstein borepst...@gmail.com wrote: Note: RAID5 is not really recomended for such large disks. You run the risk of a complete data loss if one disk fails and the another disk fails during the rebuild. Why is the size a factor here? Why would this be OK with smaller disks? How would you partition this instead? As the disks get bigger, rebuild time also increases and the performance of the disks don't increase linearly with their storage. This means that when you are rebuilding a disk, the chances of one of your other disks failing becomes significantly large. Most suggest RAID6 these days as a minimum, mirroring and striping appears to be the most popular. -- Hakan (m1fcj) - http://www.hititgunesi.org ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos Hakan, You surely do have a point there. However, it is still not all that likely that a disk will fail during the rebuild time in question (what are we talking? some hours max?) Boris. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
As the disks get bigger, rebuild time also increases and the performance of the disks don't increase linearly with their storage. This means that when you are rebuilding a disk, the chances of one of your other disks failing becomes significantly large. Most suggest RAID6 these days as a minimum, mirroring and striping appears to be the most popular. I looked up RAID6 and see the addition or a parity bit. http://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_6 RAID 10 is also something I looked at. Striped, then Mirrored So: 8 x 1.5tb = 12tb RAID 5 = 12tb - 1.5tb for parity data = 10.5tb space available RAID 10 = 4 x 1.5 = 6tb - 1.5tb for parity data = 4.5tb per stripe then mirror it. but with RAID 10, data is safe after many types of failures. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
On 3/25/2010 4:43 PM, Boris Epstein wrote: On Thu, Mar 25, 2010 at 5:27 PM, Hakan Koseogluha...@koseoglu.org wrote: On Thu, Mar 25, 2010 at 9:07 PM, Boris Epsteinborepst...@gmail.com wrote: Note: RAID5 is not really recomended for such large disks. You run the risk of a complete data loss if one disk fails and the another disk fails during the rebuild. Why is the size a factor here? Why would this be OK with smaller disks? How would you partition this instead? As the disks get bigger, rebuild time also increases and the performance of the disks don't increase linearly with their storage. This means that when you are rebuilding a disk, the chances of one of your other disks failing becomes significantly large. Most suggest RAID6 these days as a minimum, mirroring and striping appears to be the most popular. You surely do have a point there. However, it is still not all that likely that a disk will fail during the rebuild time in question (what are we talking? some hours max?) The common problem is that there are unused portions of the drives that go bad but are unnoticed for a long time. Then one fails badly enough to get kicked out of the raid. Then when you rebuild, you have to reconstruct parity for even the unused parts of the drive and you hit previously unnoticed bad spots in the process. I think the last Centos update added some sort of raid scan as a cron job that might detect bad spots earler, but I'm not sure what it actually does. -- Les Mikesell lesmikes...@gmail.com ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
At Thu, 25 Mar 2010 22:27:56 +0100 CentOS mailing list centos@centos.org wrote: Am 25.03.2010 um 22:07 schrieb Boris Epstein: Robert, Why is the size a factor here? Why would this be OK with smaller disks? How would you partition this instead? Thanks. Boris. This has been discussed before. The root of the problem lies in the fact that when a disk fails, you have to read-out the data from the other disks to re-build the RAID. Reads from disks have a certain probability to contain an error. The larger the disk and the larger the array, the more probable it is to encounter such an error while rebuilding the RAID (and if that happens, you're RAID is just a piece of scrap-metal) Or as was done recently at the Wendell Free Library, your disks become raw materials for an after school art project... :-) http://www.google.com/search?q=the+end+of+raid RAID5 works OK-ish for a couple of 146GB SAS-disks. More than a couple of disks for RAID5 -- at least 3 are needed for RAID5. Rainer MIME-Version: 1.0 ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos -- Robert Heller -- 978-544-6933 Deepwoods Software-- Download the Model Railroad System http://www.deepsoft.com/ -- Binaries for Linux and MS-Windows hel...@deepsoft.com -- http://www.deepsoft.com/ModelRailroadSystem/ ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
At Thu, 25 Mar 2010 17:07:47 -0400 CentOS mailing list centos@centos.org wrote: On Thu, Mar 25, 2010 at 3:36 PM, Robert Heller hel...@deepsoft.com wrote: At Thu, 25 Mar 2010 12:24:57 -0700 (PDT) CentOS mailing list centos@centos.org wrote: Can anyone provide a tutorial or advice on how to configure a software RAID 5 from the command-line (since I did not install Gnome)? I have 8 x 1.5tb Drives. mdadm --create /dev/md0 --level=5 --raid-devices=7 /dev/sd[abcdefg]1 The above will create a level 5 RAID named /dev/md0 of /dev/sda1 /dev/sdb1 /dev/sda1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1, with hot-spare /dev/sdg1 Note: RAID5 is not really recomended for such large disks.  You run the risk of a complete data loss if one disk fails and the another disk fails during the rebuild. -Jason ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos -- Robert Heller       -- 978-544-6933 Deepwoods Software     -- Download the Model Railroad System http://www.deepsoft.com/  -- Binaries for Linux and MS-Windows hel...@deepsoft.com    -- http://www.deepsoft.com/ModelRailroadSystem/ ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos Robert, Why is the size a factor here? Why would this be OK with smaller disks? How would you partition this instead? There was a thread some time back (a few weeks? Couple of months?) about how as disk size got so much larger, the error rate hasn't really gotten much better. With such large disks, the number of I/O operations needed to do a rebuild of a RAID 5 array is so large that one will be increasingly likely to hit an error, at which point all bets are off. (There are some papers talking about this -- I don't have the links, but I think they are in the list archives.) The prefered way to go would be RAID10 (RAID1 (mirror) + RAID0 (stripe)). Form pairs as RAID1, then strip the pairs. With 8 disks, this would 4 pairs, 1.5TB/pair = 1.5*4 = 6TB total. Thanks. Boris. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos -- Robert Heller -- 978-544-6933 Deepwoods Software-- Download the Model Railroad System http://www.deepsoft.com/ -- Binaries for Linux and MS-Windows hel...@deepsoft.com -- http://www.deepsoft.com/ModelRailroadSystem/ ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
Boris Epstein wrote: You surely do have a point there. However, it is still not all that likely that a disk will fail during the rebuild time in question (what are we talking? some hours max?) 8 disks is about the upper limit I'd suggest for a single raid group on any sort of system. rebuilding a 8x1.5TB raid5 could easily take a full day or more will you have an online hotspare? if not, then the rebuild time includes how long it takes you to realize there's a bad drive, procure and install the replacement, /AND/ the umpteen hours for the rebuild. personally, I prefer using raid10 or 1+0 (more or less the same thing), and for anything above a 2 disk mirror, I prefer to use a proper hardware raid controller... for 8+ disks, I'd likely be looking at external storage arrays such as the IBM DS3000 family. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
Slack-Moehrle wrote: RAID 10 = 4 x 1.5 = 6tb - 1.5tb for parity data = 4.5tb per stripe then mirror it. no -1.5 on that. you don't have parity when you are mirroring. 8x1.5TB raid10 is simply 4*1.5 = 6TB I'd still want hot spare. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
Robert Heller wrote:
At Thu, 25 Mar 2010 22:27:56 +0100 CentOS mailing list centos@centos.org
wrote:
[...]
The root of the problem lies in the fact that when a disk fails, you
have to read-out the data from the other disks to re-build the RAID.
Reads from disks have a certain probability to contain an error.
The larger the disk and the larger the array, the more probable it is
to encounter such an error while rebuilding the RAID (and if that
happens, you're RAID is just a piece of scrap-metal)
Or as was done recently at the Wendell Free Library, your disks become
raw materials for an after school art project... :-)
It depends on how redundant the array is. With enough
redundancy, one can rebuild even if more than one disc
fails. RAID is essentially indistinguishable from ECC.
If the number of errors (failed reads from discs) does
not exceed the correction ability of the code used
(usualy a Reed-Solomon BCH style code) then the reconstruction
can proceed.
Mike
--
p=p=%c%s%c;main(){printf(p,34,p,34);};main(){printf(p,34,p,34);}
Oppose globalization and One World Governments like the UN.
This message made from 100% recycled bits.
You have found the bank of Larn.
I speak only for myself, and I am unanimous in that!
___
CentOS mailing list
CentOS@centos.org
http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
On Friday, March 26, 2010 05:52 AM, Slack-Moehrle wrote: As the disks get bigger, rebuild time also increases and the performance of the disks don't increase linearly with their storage. This means that when you are rebuilding a disk, the chances of one of your other disks failing becomes significantly large. Most suggest RAID6 these days as a minimum, mirroring and striping appears to be the most popular. I looked up RAID6 and see the addition or a parity bit. http://en.wikipedia.org/wiki/Standard_RAID_levels#RAID_6 RAID6 allows you to survive failure of any two disks in the array at the cost of two disks of space. RAID 10 is also something I looked at. Striped, then Mirrored They recommend that you mirror and then stripe the mirrors. But that is probably old school now with Neil Brown's raid10 personality. Does anyone do nested raid1+0 setups anymore? but with RAID 10, data is safe after many types of failures. Except for the case when a mirror dies after which the whole thing is toast but in theory you can survive up to four disks going down. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
Christopher Chan wrote: but with RAID 10, data is safe after many types of failures. Except for the case when a mirror dies after which the whole thing is toast but in theory you can survive up to four disks going down. if you have a 8 drive raid1+0, and a random drive fails, you can survive any other drive failing *except* the mirror of the failed one. so if a second drive fails, there's only a 1 in 7 chance that its the 'fatal' one. on a 4 drive raid10, its a 1 in 3 chance. meanwhile, a raid10 can rebuild from a hotspare in like an hour, if the system isn't busy, and a few hours in the background if its busy and active. ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
Re: [CentOS] RAID 5 setup?
On Friday, March 26, 2010 09:12 AM, John R Pierce wrote: Christopher Chan wrote: but with RAID 10, data is safe after many types of failures. Except for the case when a mirror dies after which the whole thing is toast but in theory you can survive up to four disks going down. if you have a 8 drive raid1+0, and a random drive fails, you can survive any other drive failing *except* the mirror of the failed one. so if a That's what I said right? I did not say when a mirror is broken... second drive fails, there's only a 1 in 7 chance that its the 'fatal' one. on a 4 drive raid10, its a 1 in 3 chance. meanwhile, a raid10 can rebuild from a hotspare in like an hour, if the system isn't busy, and a few hours in the background if its busy and active. yeah, I thought the raid10 module would be able to rebuild automatically from a hotspare and would therefore be better than using nested raid1+0. I better stop the nested raid1+0 thing...does ananconda support the raid10 module during install yet? I mean rather, is the raid10 module included in the installation initrd yet? ___ CentOS mailing list CentOS@centos.org http://lists.centos.org/mailman/listinfo/centos
