Re: Help: very slow software RAID 5.
Dean S. Messing wrote: Bill Davidsen wrote: : Dean S. Messing wrote: : Again, I don't get these speeds. Seq. reads are about : 170% of the average of my three physical drives if I turn up : the look-ahead. Then random access reads drops to slightly less : than my slowest drive. : : As nearly as I can tell, Dean was talking about RAID-10 at that point (I : also suggested that) which you haven't tried. I was talking about the three drive RAID-5 on which I ran bonnie++ measurements. I have not (yet) tried RAID-10. : For small numbers of : drives, assume the read speed will be (N - 1) * S for large sequential : read, using RAID-10. Where S is the speed of a single drive. Random read : depends on so many things I can't begin to quantify them in anything : less than a full white paper, but for a single thread assume somewhere : around S and aggregate (N - 1) * S again. Writes depend a lot on system : tuning, stripe size, stripe_cache_size, chunk size, etc. Fortunately the : best way to boost write speed is to have lots of memory and let the : kernel buffer. How does one let the kernel buffer? (I have plenty of memory for most things.) I know about write-back vs. write-through to reduce the write asymmetry of RAID-5. Is this what you mean by a kernel buffer? Just by having adequate memory you will get kernel buffering (unless you use fsync or similar), and performance goes up if you increase your stripe_cache_size, although you hit diminishing returns on that somewhere between 8-32MB. : Finally, when you create your ext filesystem, think of: : - ext2 - no journal : - noatime mounts to avoid journal writes Please try this before you reach any conclusions. Doing measurements on a filesystem instead of raw raid arrays adds bottlenecks. : - manually make the journal file *large* to spread head motion over drives : - consider moving journal file to a dedicated device (that old 20GB : PATA drive?) : - use the ext3 stride tuning stuff (I'm quantifying that in the next : ten days). : : Or just make a RAID-10 far array and stop agonizing over this stuff, : there is no config which is best for everything, you must realize fast, : cheap, reliable - pick two is the design paradigm of RAID, and the more : you optimize for one usage pattern the more you impact some other. Dean - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html -- bill davidsen [EMAIL PROTECTED] CTO TMR Associates, Inc Doing interesting things with small computers since 1979 - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Help: very slow software RAID 5.
: Dean S. Messing wrote: : I have also discovered smartctl and have read that if the short smartctl : tests are run daily and the long test weekly that the chances of being : caught with my pants down are quite low, even in a two disk RAID-0 : config. What is your opinion? : : : There's a good paper on using smartctl to predict the health of disks, : and if you can't find it I probably have a copy somewhere, since I gave : a presentation on RAID issues which included it. But the basic premise : was that if you see errors of certain types, the drives are likely to : fail soon. It did *not* say that absent these warnings the drives were : unlikely to fail, un fact most drives which did fail did so without : warning. So for about 90% of the failures there is no warning. : : I had servers a few years ago, running 6TB/server, on lots of small fast : drives, and I concluded that the predictive value of SMART was so small : that it didn't justify looking at the reports. Take that as my opinion, : assume that drives fail without warning. From what you and another poster said (about the False Alarm rate of Smartctl) I'll put my trust in backups, alone. I agree: if it predicts such a low % of failures, there's no point to waste time reading the reports and having a false sense of security. : I'm getting around to replying to several things you have said in : various posts, so that people who are threading answers will be happy... I'll look forward to your comments, especially on my misconceptions. I've learned a great deal already. Dean - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Help: very slow software RAID 5.
Bill Davidsen wrote: : Dean S. Messing wrote: : Again, I don't get these speeds. Seq. reads are about : 170% of the average of my three physical drives if I turn up : the look-ahead. Then random access reads drops to slightly less : than my slowest drive. : : As nearly as I can tell, Dean was talking about RAID-10 at that point (I : also suggested that) which you haven't tried. I was talking about the three drive RAID-5 on which I ran bonnie++ measurements. I have not (yet) tried RAID-10. : For small numbers of : drives, assume the read speed will be (N - 1) * S for large sequential : read, using RAID-10. Where S is the speed of a single drive. Random read : depends on so many things I can't begin to quantify them in anything : less than a full white paper, but for a single thread assume somewhere : around S and aggregate (N - 1) * S again. Writes depend a lot on system : tuning, stripe size, stripe_cache_size, chunk size, etc. Fortunately the : best way to boost write speed is to have lots of memory and let the : kernel buffer. How does one let the kernel buffer? (I have plenty of memory for most things.) I know about write-back vs. write-through to reduce the write asymmetry of RAID-5. Is this what you mean by a kernel buffer? : Finally, when you create your ext filesystem, think of: : - ext2 - no journal : - noatime mounts to avoid journal writes : - manually make the journal file *large* to spread head motion over drives : - consider moving journal file to a dedicated device (that old 20GB : PATA drive?) : - use the ext3 stride tuning stuff (I'm quantifying that in the next : ten days). : : Or just make a RAID-10 far array and stop agonizing over this stuff, : there is no config which is best for everything, you must realize fast, : cheap, reliable - pick two is the design paradigm of RAID, and the more : you optimize for one usage pattern the more you impact some other. Dean - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Help: very slow software RAID 5.
Dean S. Messing wrote: I don't see how one would do incrementals. My backup system uses currently does a monthly full backup, a weekly level 3 (which saves everything that has changed since the last level 3 a week ago) and daily level 5's (which save everything that changed today). Rsync is fantastic tool for incremental backups. Everything that didn't change can be hardlinked to previous entry. And time of performing the backup is pretty much neglible. Essentially - you have equivalent of full backups at almost minimal time and space cost possible. - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Help: very slow software RAID 5.
Dean S. Messing wrote: I have also discovered smartctl and have read that if the short smartctl tests are run daily and the long test weekly that the chances of being caught with my pants down are quite low, even in a two disk RAID-0 config. What is your opinion? There's a good paper on using smartctl to predict the health of disks, and if you can't find it I probably have a copy somewhere, since I gave a presentation on RAID issues which included it. But the basic premise was that if you see errors of certain types, the drives are likely to fail soon. It did *not* say that absent these warnings the drives were unlikely to fail, un fact most drives which did fail did so without warning. So for about 90% of the failures there is no warning. I had servers a few years ago, running 6TB/server, on lots of small fast drives, and I concluded that the predictive value of SMART was so small that it didn't justify looking at the reports. Take that as my opinion, assume that drives fail without warning. I'm getting around to replying to several things you have said in various posts, so that people who are threading answers will be happy... -- bill davidsen [EMAIL PROTECTED] CTO TMR Associates, Inc Doing interesting things with small computers since 1979 - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Backups w/ rsync (was: Help: very slow software RAID 5.)
Michal Soltys writes: : Dean S. Messing wrote: : : I don't see how one would do incrementals. My backup system uses : currently does a monthly full backup, a weekly level 3 (which : saves everything that has changed since the last level 3 a week ago) and : daily level 5's (which save everything that changed today). : : : Rsync is fantastic tool for incremental backups. Everything that didn't : change can be hardlinked to previous entry. And time of performing the : backup is pretty much neglible. Essentially - you have equivalent of : full backups at almost minimal time and space cost possible. It has been some time since I read the rsync man page. I see that there is (among the bazillion and one switches) a --link-dest=DIR switch which I suppose does what you describe. I'll have to experiment with this and think things through. Thanks, Michal. Dean P.S. I changed the Subject: to reflect the new subject. Not sure if that starts a new thread or not. - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Help: very slow software RAID 5.
Dean S. Messing [EMAIL PROTECTED] writes: Michael Tokarev writes: : Dean S. Messing wrote: : [] : [] That's what : attracted me to RAID 0 --- which seems to have no downside EXCEPT : safety :-). : : So I'm not sure I'll ever figure out the right tuning. I'm at the : point of abandoning RAID entirely and just putting the three disks : together as a big LV and being done with it. (I don't have quite the : moxy to define a RAID 0 array underneath it. :-) : : Putting three disks together as a big LV - that's exactly what : linear md module. : It's almost as unsafe as raid0, but with : linear read/write speed equal to speed of single drive... I understand I only get the speed of a single drive was I was not aware of the safety factor. I had intended to use snapshotting off to a cheap USB drive each evening. Will that not keep me safe within a day's worth of data change? I only learned about snapshots yesterday. I'm utterly new to the disk array/LVM game. For that matter why not run a RAID-0 + LVM across two of the three drives and snapshot to the third? LVM is not the same as LVM. What I mean is that you still have choices left. One thing you have to think about though. An lvm volume group will not start cleanly with a disk missing but you can force it to start anyway. So a lost disk does not mean all data is lost. But it does mean that any logical volume with data on the missing disk will have serious data corruption. Also lvm can do raid0 itself. For each logical volume you create you can specify the number of stripes to use. So I would abandon all thoughts of raid0 and replace them with using lvm. Run one LV with 2 stripes on the first two disks and snapshot on the third. : Note also that the more drives you add to raid0-like config, : the more chances of failure you'll have - because raid0 fails : when ANY drive fails. Ditto - for certain extent - for linear : md module and for one big LV which is basically the same thing. I understand the probability increases for additional drives. : By the way, before abandoming R in RAID, I'd check whenever : the resulting speed with raid5 (after at least read-ahead tuning) : is acceptable, and use that if yes. My problem is not quite knowing what acceptable is. I bought a Dell Precision 490 with two relatively fast SATA II drives. With RAID 0 I attain speeds of nearly 140 MB/s (using 2 drives) for reads and writes and the system is very snappy for everything, from processing 4Kx2K video to building a 'locate' datebase, to searching my very large mail archives for technical info. When I see the speed loss of software RAID 5 (writes are at 55MB/s and random reads are at 54 MB/s) for everything but seq. reads (and that only if I increase read-ahead from 512 to 16384 to get read speeds of about 110 MB/s I lose heart, esp. since I don't know the other consequences of increasing read-ahead by so much. : If no, maybe raid10 over : the same 3 drives will give better results. Does RAID10 work on three drives? I though one needed 4 drives, with striping across a pair of mirrored pairs. I tested Raid10 and with far copies I got the full speed of all disks combined just like a raid0 would for reading and half speed for writing (as it has to write everything twice). I got pretty damn close to the theoretical limit it could get, which was surprising. Dean MfG Goswin - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Help: very slow software RAID 5.
Goswin von Brederlow writes: : Dean S. Messing writes: : Michael Tokarev writes: : : Dean S. Messing wrote: : : [] : : [] That's what : : attracted me to RAID 0 --- which seems to have no downside EXCEPT : : safety :-). : : : : So I'm not sure I'll ever figure out the right tuning. I'm at the : : point of abandoning RAID entirely and just putting the three disks : : together as a big LV and being done with it. (I don't have quite the : : moxy to define a RAID 0 array underneath it. :-) : : : : Putting three disks together as a big LV - that's exactly what : : linear md module. : : It's almost as unsafe as raid0, but with : : linear read/write speed equal to speed of single drive... : : I understand I only get the speed of a single drive but I was not : aware of the safety factor. I had intended to use snapshotting off : to a cheap USB drive each evening. Will that not keep me safe within a : day's worth of data change? I only learned about snapshots yesterday. : I'm utterly new to the disk array/LVM game. : : For that matter why not run a RAID-0 + LVM across two of the three drives : and snapshot to the third? : : LVM is not the same as LVM. What I mean is that you still have choices : left. Sorry, Goswin. Even though you gave your meaning, I still don't understand you here. (I must be dense this morning.) What does LVM is not the same as LVM mean? : : One thing you have to think about though. An lvm volume group will not : start cleanly with a disk missing but you can force it to start : anyway. So a lost disk does not mean all data is lost. But it does : mean that any logical volume with data on the missing disk will have : serious data corruption. If I am taking daily LVM snapshots will I not be able to reconstruct the file system as of the last snapshot? That's all I require. I have also discovered smartctl and have read that if the short smartctl tests are run daily and the long test weekly that the chances of being caught with my pants down are quite low, even in a two disk RAID-0 config. What is your opinion? : Also lvm can do raid0 itself. For each logical volume you create you : can specify the number of stripes to use. So I would abandon all : thoughts of raid0 and replace them with using lvm. : : Run one LV with 2 stripes on the first two disks and snapshot on the : third. Good idea. I waw aware of striped LV but did not think it would run nearly as fast as RAID-0. Do you think two LV stripes will equal RAID-0 for all kinds of read/write disk use? There would seem to be lots more than two RAID=0 stripes in the default case. (I do know enough to not run Striped LV with RAID-0 :-) snip : : I tested Raid10 and with far copies I got the full speed of all disks : combined just like a raid0 would for reading and half speed for : writing (as it has to write everything twice). I got pretty damn : close to the theoretical limit it could get, which was surprising. Very interesting! On three drives? When you said half speed for writes, did you mean half the RAID-0 read speed or half the physical device read speed? I hate the thought of half speed writes. Some of what I do requires more writing than reading---up-conversion of Full HD video to 4Kx2K video, for example. Given your test, I'll run some tests with a three device RAID-10 with far copies. But I would really like to know if I'm playing with fire putting my whole system on a RAID-0/non-striped LVM device (or striped LVM device w/o RAID) with daily snapshots, and good smartctl monitoring. Dean - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Help: very slow software RAID 5.
Dean S. Messing [EMAIL PROTECTED] writes: Goswin von Brederlow writes: : Dean S. Messing writes: : Michael Tokarev writes: : : Dean S. Messing wrote: : : [] : : [] That's what : : attracted me to RAID 0 --- which seems to have no downside EXCEPT : : safety :-). : : : : So I'm not sure I'll ever figure out the right tuning. I'm at the : : point of abandoning RAID entirely and just putting the three disks : : together as a big LV and being done with it. (I don't have quite the : : moxy to define a RAID 0 array underneath it. :-) : : : : Putting three disks together as a big LV - that's exactly what : : linear md module. : : It's almost as unsafe as raid0, but with : : linear read/write speed equal to speed of single drive... : : I understand I only get the speed of a single drive but I was not : aware of the safety factor. I had intended to use snapshotting off : to a cheap USB drive each evening. Will that not keep me safe within a : day's worth of data change? I only learned about snapshots yesterday. : I'm utterly new to the disk array/LVM game. : : For that matter why not run a RAID-0 + LVM across two of the three drives : and snapshot to the third? : : LVM is not the same as LVM. What I mean is that you still have choices : left. Sorry, Goswin. Even though you gave your meaning, I still don't understand you here. (I must be dense this morning.) What does LVM is not the same as LVM mean? The ultimate risk and speed of lvm depends on the striping and distribution of LVs accross the disks. : : One thing you have to think about though. An lvm volume group will not : start cleanly with a disk missing but you can force it to start : anyway. So a lost disk does not mean all data is lost. But it does : mean that any logical volume with data on the missing disk will have : serious data corruption. If I am taking daily LVM snapshots will I not be able to reconstruct the file system as of the last snapshot? That's all I require. A snapshot will only hold the differences between creation and now. So Not at all. What you would have to do is have the original on USB and work in a snapshot. But then there is no lvm command to commit a snapshot back to the original device to store the changes. I'm afraid you need to rsync the data to another disk or volume to make a backup. I have also discovered smartctl and have read that if the short smartctl tests are run daily and the long test weekly that the chances of being caught with my pants down are quite low, even in a two disk RAID-0 config. What is your opinion? Smart is a big fat lier. I have a disk with failure iminent that's been running for 3 years now. I have disks with 345638756348756 ECC errors. A disk that runs at 130° and gets warmer when I put a fan in front of it. and on and on and on. Smart certainly is no replacement of a backup. Raid5 is also no replacement for a backup. Imagine a faulty cable or a bug in your kernel that writes wrong data to the disks and corrupts your filesystem. The raid will be perfectly fine and could properly construct your data on a disk failure, the broken data. Big help. Raid basicaly only protects you from the downtime of having to restore the backup RIGHT NOW when a disk fails. Buys you the time to wait for the weekend to fix things or whatever. : Also lvm can do raid0 itself. For each logical volume you create you : can specify the number of stripes to use. So I would abandon all : thoughts of raid0 and replace them with using lvm. : : Run one LV with 2 stripes on the first two disks and snapshot on the : third. Good idea. I waw aware of striped LV but did not think it would run nearly as fast as RAID-0. Do you think two LV stripes will equal RAID-0 for all kinds of read/write disk use? There would seem to be lots more than two RAID=0 stripes in the default case. (I do know enough to not run Striped LV with RAID-0 :-) They are conceptually identicall and all else being the same they should behave the same. Beware though that lvm does not set the read ahead correctly (only the default size) while raid will set the read ahead to the sum of the disks read ahead (-1 or -2 disks for raid4/5/6). So by default all is not the same. So set the readahead to the same if you want to compare the two. snip : : I tested Raid10 and with far copies I got the full speed of all disks : combined just like a raid0 would for reading and half speed for : writing (as it has to write everything twice). I got pretty damn : close to the theoretical limit it could get, which was surprising. Very interesting! On three drives? When you said half speed for writes, did you mean half the RAID-0 read speed or half the physical device read speed? Both. The raid10 has to physically write every data block twice so the throughput of what you get as user is half of that the hardware has to do. So naturally you only get 50% of the
Re: Help: very slow software RAID 5.
Dean S. Messing [EMAIL PROTECTED] writes: Goswin von Brederlow writes: : Dean Mesing writes: : Goswin von Brederlow writes: : : LVM is not the same as LVM. What I mean is that you still have choices : : left. : : Sorry, Goswin. Even though you gave your meaning, I still don't : understand you here. (I must be dense this morning.) : What does LVM is not the same as LVM mean? : : The ultimate risk and speed of lvm depends on the striping and : distribution of LVs accross the disks. Even w/o LV striping, I don't know enough about LV organisation to try to recover from a serious disk crash. Even with out striping you can have 1GB data of an LV on disk1, then 1GB on disk2, then 1GB on disk3. When disk2 dies you loose 1GB in the middle of the fs and that is rather damaging. Now why would anyone split up an LV like that? Sounds seriously stupid, right? Well. Thinking about what happens over longer time when you resize LVs a few times. It will just use the next free space unless the LV is flaged continious. Allocations will fragment unless you take care not to (for example pvmove other LVs out of the way). : : : : One thing you have to think about though. An lvm volume group will not : : start cleanly with a disk missing but you can force it to start : : anyway. So a lost disk does not mean all data is lost. But it does : : mean that any logical volume with data on the missing disk will have : : serious data corruption. : : If I am taking daily LVM snapshots will I not be able to reconstruct : the file system as of the last snapshot? That's all I require. : : A snapshot will only hold the differences between creation and now. So : Not at all. : : What you would have to do is have the original on USB and work in a : snapshot. But then there is no lvm command to commit a snapshot back : to the original device to store the changes. : : I'm afraid you need to rsync the data to another disk or volume to : make a backup. Ok. I think I understand. The snapshot could not be used to restore to a master backup of the original since that backup in not in LV format. If I'm using an ext3 filesystem (which I plan to do) would Full and Incremental dumps to a cheap 'n big USB drive (using the dump/restore suite) not work? Probably. But why not rsync? It will copy all changes and the data on the USB disk will be accessible directly without restore. Very handy if you only need one file. : Smart certainly is no replacement of a backup. Raid5 is also no : replacement for a backup. I did not mean to imply I would forego backups. I've been using Unix for too long (26 years) to be that foolish. I simply thought that Smart would allow me to run RAID-0 or striped LV (and do backups!) with reduced risk of having an actual disk failure since I would be able to deal with a weak drive before it failed. Thanks for disabusing me of my fantasy. If it works right, and the numbers are probably obviously wrong if not, you can see the number of bad blocks. If that starts rising then you know the disk won't last long anymore. But when was the last time one of your disks died by bad blocks apearing? Mine always sieze up and won't spin up anymore or the heads won't seek anymore or the electronic dies. Never had a disk where the magnetization failed and more and more bad blocks appeared. : Beware though that lvm does not set the read ahead correctly (only the : default size) while raid will set the read ahead to the sum of the : disks read ahead (-1 or -2 disks for raid4/5/6). So by default all is : not the same. So set the readahead to the same if you want to compare : the two. Ok, good to know. However I'm not so sure RAID-4,5,6 actually sets the readahead correctly. My whole dilemma started when I saw how slowly RAID-5 was running on three drives---slower than the physical device speed of two of the three drives. Justin Piszcz suggested tweeking the parameters (in particular, readahead). Indeed, increasing read-ahead did increase seq. read speeds, but at a cost to random reads. And writes were still slow. For RAID-0, everything is faster, which makes the whole system snappy. Untuned I have this: # cat /proc/mdstat Personalities : [linear] [raid0] [raid1] [raid10] [raid6] [raid5] [raid4] md1 : active raid5 sdd2[3] sdc2[2] sdb2[1] sda2[0] 583062912 blocks level 5, 64k chunk, algorithm 2 [4/4] [] # blockdev --getra /dev/sda 256 # blockdev --getra /dev/md1 768 # blockdev --getra /dev/r/home 256 You see that the disk and LV is at the default of 256 blocks read ahead. But the raid it at (4-1)*256 == 768 blocks. You usualy can still raise those number a good bit. Esspecialy if you are working with large files and streaming access, like movies. :) : For writes you get 50% of 300% (slowest) disk speed. So you sill have : 150% speed total with raid10. With raid5 you have 200% (slowest) disk : speed for continious writes and maybe
Re: Help: very slow software RAID 5.
Dean S. Messing wrote: Michael Tokarev writes: : Dean S. Messing wrote: : [] : [] That's what : attracted me to RAID 0 --- which seems to have no downside EXCEPT : safety :-). : : So I'm not sure I'll ever figure out the right tuning. I'm at the : point of abandoning RAID entirely and just putting the three disks : together as a big LV and being done with it. (I don't have quite the : moxy to define a RAID 0 array underneath it. :-) : : Putting three disks together as a big LV - that's exactly what : linear md module. : It's almost as unsafe as raid0, but with : linear read/write speed equal to speed of single drive... I understand I only get the speed of a single drive was I was not aware of the safety factor. I had intended to use snapshotting off to a cheap USB drive each evening. Will that not keep me safe within a day's worth of data change? I only learned about snapshots yesterday. I'm utterly new to the disk array/LVM game. But your read speed need not be limited if you tune the readahead. There's also the question of how much transfer speed you actually *need*. If your application is CPU-bound faster will not be the same as runs in less time, and random access is limited by the seek speed of your drives, although some RAID tuning does apply to random writes. For that matter why not run a RAID-0 + LVM across two of the three drives and snapshot to the third? : Note also that the more drives you add to raid0-like config, : the more chances of failure you'll have - because raid0 fails : when ANY drive fails. Ditto - for certain extent - for linear : md module and for one big LV which is basically the same thing. I understand the probability increases for additional drives. : By the way, before abandoming R in RAID, I'd check whenever : the resulting speed with raid5 (after at least read-ahead tuning) : is acceptable, and use that if yes. My problem is not quite knowing what acceptable is. I bought a Dell Precision 490 with two relatively fast SATA II drives. With RAID 0 I attain speeds of nearly 140 MB/s (using 2 drives) for reads and writes and the system is very snappy for everything, from processing 4Kx2K video to building a 'locate' datebase, to searching my very large mail archives for technical info. When you process video and monitor the system with vmstat, do you see significant iowait time? No, neither do I, with a modest readahead I am totally CPU limited. If you are searching your mail database, if you just use a text tool which reads everything, that's pure sequential access. And unless you actually *use* the locate command, building that database is just a way to beat your disks (and it's more sequential than you would expect). You can turn it off and do your bit to avoid global warming. When I see the speed loss of software RAID 5 (writes are at 55MB/s and random reads are at 54 MB/s) for everything but seq. reads (and that only if I increase read-ahead from 512 to 16384 to get read speeds of about 110 MB/s I lose heart, esp. since I don't know the other consequences of increasing read-ahead by so much. Assuming that your have enough memory, there would be a small slowdown in random reading a lot of small records. You should know what your application would do, but that access is typical of looking things up in a database or processing small records, like a DNS or mail server. Numbers from bonnie or similar benchmarks are nice, but they show details of various performance area, and if you don't match what you do to what works best you make bad choices. In other words if your application can only read 10MB/s the benchmark is telling you your disk is fast enough to keep up with the CPU. : If no, maybe raid10 over : the same 3 drives will give better results. Does RAID10 work on three drives? I though one needed 4 drives, with striping across a pair of mirrored pairs. No, that's 0+1, RAID-10 works across any number of drives. Have you actually take 10-15 minutes to read man md and get the overview of how RAID works, or are you reading bits and pieces about individual features? -- bill davidsen [EMAIL PROTECTED] CTO TMR Associates, Inc Doing interesting things with small computers since 1979 - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Help: very slow software RAID 5.
Bill Davidsen wrote: Dean Messing wrote: : I understand I only get the speed of a single drive was I was not : aware of the safety factor. I had intended to use snapshotting off : to a cheap USB drive each evening. Will that not keep me safe within a : day's worth of data change? I only learned about snapshots yesterday. : I'm utterly new to the disk array/LVM game. : : : But your read speed need not be limited if you tune the readahead. : There's also the question of how much transfer speed you actually : *need*. If your application is CPU-bound faster will not be the same as : runs in less time, and random access is limited by the seek speed of : your drives, although some RAID tuning does apply to random writes. This is my main workstation. I do many things on it, not just an application. I live on the machine. It will run XP on top of VMware. It will transcode video. It will run map software, emacs, mail, mplayer, re-compile large applications like ImageMagick, Transcode, and Scilab, and much more. The video/image processing (whch can be random access) is just an example. Increasing readahead does increase the sequential read speed for RAID5. But the random read speed suffers and the write speed suffers a loss of 20% over single disk writes (according to the bonnie++ numbers). RAID0 on the other machine spoiled me, I'm afraid. Question: From what I've read, when reading in RAID5, parity is not read. Why then does read speed suffer so badly with the default Readahead parameter? : When you process video and monitor the system with vmstat, do you see : significant iowait time? No, neither do I, with a modest readahead I am : totally CPU limited. If you are searching your mail database, if you : just use a text tool which reads everything, that's pure sequential : access. And unless you actually *use* the locate command, building that : database is just a way to beat your disks (and it's more sequential than : you would expect). You can turn it off and do your bit to avoid global : warming. I have not used vmstat, but I am dealing with 4Kx2K 24bit uncompressed video frames. mplayer, for one, is quite disk i/o limited. So are several spatio-temporal adaptive interpolation algorithms we work on for upscaling video (part of my research). I can see this from the gkrellm disk meter both during video i/o and when swap space is getting used. Funny you should mention locate. I use it heavily to find stuff. I typically generate results on any of 4 different machines, and then `scp' the results around between the machines. So I rebuild the database relatively often. On another machine that I bought from Dell, configured with RAID0, everything is very snappy. Rebuilding the locate db simply flew. It is that uniform snappiness I was hoping (against hope?) to duplicate on this current workstation with a third drive and RAID5. : : If no, maybe raid10 over : : the same 3 drives will give better results. : : Does RAID10 work on three drives? I though one needed 4 drives, : with striping across a pair of mirrored pairs. : : No, that's 0+1, RAID-10 works across any number of drives. : : Have you actually take 10-15 minutes to read man md and get the : overview of how RAID works, or are you reading bits and pieces about : individual features? I confess I have not read the md man page. (I shall, right after this.) I have read the mdadm page pretty thoroughly. And I've read parts of lots of other stuff in the last few days. It has all uniformly said that RAID-10 is striped mirrors and requires 4 drives. One such example (which I just googled) is: http://www.pcguide.com/ref/hdd/perf/raid/levels/multLevel01-c.html http://www.pcguide.com/ref/hdd/perf/raid/levels/multXY-c.html I suppose Linux Software Raid is more general and allows 3 drives in a RAID-10 config. I'll find out in a few minutes. In my last note I asked: : : Putting three disks together as a big LV - that's exactly what : linear md module. : It's almost as unsafe as raid0, but with : linear read/write speed equal to speed of single drive... I understand I only get the speed of a single drive but I was not aware of the safety factor. I had intended to use snapshotting off to a cheap USB drive each evening. Will that not keep me safe within a day's worth of data change? I only learned about snapshots yesterday. I'm utterly new to the disk array/LVM game. snip For that matter why not run a RAID-0 + LVM across two of the three drives and snapshot to the third? What do you think about the RAID-0 + LVM plus daily snapshots? I am not running a server. In the (fairly remote) chance that I do have a RAID 0 failure, I can tolerate the couple of hours it will take to rebuild the file system and be back up and running (in ordinary non-RAID mode). Dean - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo
Re: Help: very slow software RAID 5.
This is to both Bill Davidsen and Michael Tokarev. I just realised in re-reading previous messages that I badly screwed up my attributions in my just-sent message. I attributed to Bill some technical remarks by Michael. I apologise to both of you! Dean - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Help: very slow software RAID 5.
Dean S. Messing wrote: Also (as I asked) what is the downside? From what I have read, random access reads will take a hit. Is this correct? Thanks very much for your help! Dean Besides bonnie++ you should probably check iozone. It will allow you to test very specific settings quite thoroughly. Although with current multi-gigabyte memory systems the test runs may take a bit time. http://www.iozone.org/ There's nice introduction to the progam there, along with some example graph results. - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Help: very slow software RAID 5.
Bill Davidsen wrote: : Dean S. Messing wrote: snip : Do you want to tune it to work well now or work well in the final : configuration? There is no magic tuning which is best for every use, if : there was it would be locked in and you couldn't change it. I want it to work well in the final config. I'm just now learning about benchmarking with tools like bonnie++. In my naivety, I thought that `hdparm -t' told all, at least for reads. : Aside: I have found RAID quite frustrating. With the original two : disks I was getting 120-130 MB/s in RAID 0. I would think that for : the investment of a 3rd drive I ought to get the modicum of : redundancey I expect and keep the speed (at least on reads) w/o : sacrificing anything. But it appears I actually lost something for my : investment. I'm back to the speed of single drives with the modicum : of redundancey that RAID 5 gives. Not a very good deal. : RAID-5 and RAID-1 performance are popular topic, reading the archives : may shed more light on that. So I'm seeing. I just finished wading through a long April 07 discussion on write-though vs. write-back for RAID 5. : After you get to LVM you can do read ahead : tuning on individual areas, which will allow you to do faster random : access on one part and faster sequential on another. *But* when you run : both types of access on the same physical device one or the other will : suffer, and with careful tuning both can be slow. This is why simply bumping up the read ahead parameter as has been suggested to me seems suspect. If this was the right fix, it seems that it would be getting set automatically by the default installation of mdadm. : When you get to the point where you know exactly what you are going to : do and how you are going to do it (layout) you can ask a better question : about tuning. Well (in my extreme naivete) I had hoped that I could (just) -- buy the extra SATA drive, -- configure RAID 5 on all three drives, -- have it present itself as a single device with the speed of RAID 0 (on two drives), and the safety net of RAID 5, -- install Fedora 7 on the array, -- use LVM to partition as I liked, -- and forget about it. Instead, this has turned into a many hour exercise in futility. This is my research machine (for signal/image processing) and the machine I live on. It does many different things. What I really need is more disk speed (but I can't afford very high speed drives). That's what attracted me to RAID 0 --- which seems to have no downside EXCEPT safety :-). So I'm not sure I'll ever figure out the right tuning. I'm at the point of abandoning RAID entirely and just putting the three disks together as a big LV and being done with it. (I don't have quite the moxy to define a RAID 0 array underneath it. :-) : PS: adding another drive and going to RAID-10 with far configuration : will give you speed and reliability, at the cost of capacity. Aren't : shoices fun? I don't know hat far configuration is, though I understand basically what RAID-10 is. Having that much wasted space is too costly, and besides the machine can't take but three drives internally. If I wished to add a 4th I'd need to buy a SATA controller. I had thought RAID 5 did exactly what I wanted. Unfortunately ... Which suggests a question for you, David. If I were to invest in a true hardware RAID SATA controller (is there such a thing) would RAID 5 across the three drives behave just like RAID 0 on two drives + 1 disk redundancy? In other words just abandon Linux software raid? At this point I would be willing to spring for such a card if it were not too expensive, and if I could find a slot to put it in. (My system is slot-challenged). Thanks for you remarks, David. I wish I had the time to learn how to do all this properly with multiple LV's, different read-aheads and write-through/write-back settings on different logical devices, but my head is swimming and my time is short. Dean - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Help: very slow software RAID 5.
Michal Soltys writes: : Dean S. Messing wrote: : : Also (as I asked) what is the downside? From what I have read, random : access reads will take a hit. Is this correct? : : Thanks very much for your help! : : Dean : : : Besides bonnie++ you should probably check iozone. It will allow you to test : very specific settings quite thoroughly. Although with current : multi-gigabyte memory systems the test runs may take a bit time. : : http://www.iozone.org/ : : There's nice introduction to the progam there, along with some example graph : results. Thanks very much, Michal. I'll have a look. Dean - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Help: very slow software RAID 5.
Dean S. Messing wrote: [] [] That's what attracted me to RAID 0 --- which seems to have no downside EXCEPT safety :-). So I'm not sure I'll ever figure out the right tuning. I'm at the point of abandoning RAID entirely and just putting the three disks together as a big LV and being done with it. (I don't have quite the moxy to define a RAID 0 array underneath it. :-) Putting three disks together as a big LV - that's exactly what linear md module. It's almost as unsafe as raid0, but with linear read/write speed equal to speed of single drive... Note also that the more drives you add to raid0-like config, the more chances of failure you'll have - because raid0 fails when ANY drive fails. Ditto - for certain extent - for linear md module and for one big LV which is basically the same thing. By the way, before abandoming R in RAID, I'd check whenever the resulting speed with raid5 (after at least read-ahead tuning) is acceptable, and use that if yes. If no, maybe raid10 over the same 3 drives will give better results. /mjt - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Help: very slow software RAID 5.
Michael Tokarev writes: : Dean S. Messing wrote: : [] : [] That's what : attracted me to RAID 0 --- which seems to have no downside EXCEPT : safety :-). : : So I'm not sure I'll ever figure out the right tuning. I'm at the : point of abandoning RAID entirely and just putting the three disks : together as a big LV and being done with it. (I don't have quite the : moxy to define a RAID 0 array underneath it. :-) : : Putting three disks together as a big LV - that's exactly what : linear md module. : It's almost as unsafe as raid0, but with : linear read/write speed equal to speed of single drive... I understand I only get the speed of a single drive was I was not aware of the safety factor. I had intended to use snapshotting off to a cheap USB drive each evening. Will that not keep me safe within a day's worth of data change? I only learned about snapshots yesterday. I'm utterly new to the disk array/LVM game. For that matter why not run a RAID-0 + LVM across two of the three drives and snapshot to the third? : Note also that the more drives you add to raid0-like config, : the more chances of failure you'll have - because raid0 fails : when ANY drive fails. Ditto - for certain extent - for linear : md module and for one big LV which is basically the same thing. I understand the probability increases for additional drives. : By the way, before abandoming R in RAID, I'd check whenever : the resulting speed with raid5 (after at least read-ahead tuning) : is acceptable, and use that if yes. My problem is not quite knowing what acceptable is. I bought a Dell Precision 490 with two relatively fast SATA II drives. With RAID 0 I attain speeds of nearly 140 MB/s (using 2 drives) for reads and writes and the system is very snappy for everything, from processing 4Kx2K video to building a 'locate' datebase, to searching my very large mail archives for technical info. When I see the speed loss of software RAID 5 (writes are at 55MB/s and random reads are at 54 MB/s) for everything but seq. reads (and that only if I increase read-ahead from 512 to 16384 to get read speeds of about 110 MB/s I lose heart, esp. since I don't know the other consequences of increasing read-ahead by so much. : If no, maybe raid10 over : the same 3 drives will give better results. Does RAID10 work on three drives? I though one needed 4 drives, with striping across a pair of mirrored pairs. Dean - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Help: very slow software RAID 5.
One of the 5-10 tuning settings: blockdev --getra /dev/md0 Try setting it to 4096,8192,16384,32768,65536 blockdev --setra 4096 /dev/md0 Also, with a 3-disk raid5 that is the worst performance you can get using only 3 disks, while with a 10 disk raid5 it'd be closer to 90%. Reads you should get good speeds, but for writes-- probably not. Then re-benchmark. Justin. On Tue, 18 Sep 2007, Dean S. Messing wrote: Justin Piszcz wrote: On Tue, 18 Sep 2007, Dean S. Messing wrote: : : : : I'm not getting nearly the read speed I expected : from a newly defined software RAID 5 array : across three disk partitions (on the 3 drives, : of course!). : : Would someone kindly point me straight? : : After defining the RAID 5 I did `hdparm -t /dev/md0' : and got the abysmal read speed of ~65MB/sec. : The individual device speeds are ~55, ~70, : and ~75 MB/sec. : : Shouldn't this array be running (at the slowest) : at about 55+70 = 125 MB/sec minus some overhead? : I defined a RAID0 on the ~55 and ~70 partitions : and got about 110 MB/sec. : : Shouldn't adding a 3rd (faster!) drive into the : array make the RAID 5 speed at least this fast? : : : Here are the details of my setup: : snip : Without tuning you will get very slow read speed. : : Read the mailing list, there are about 5-10 tunable options, for me, I get : 250 MiB/s no tuning (read/write), after tuning 464 MiB/s write and 622 : MiB/s read. : : Justin. Thanks Justin. 5-10 tunable options! Good grief. This sounds worse than regulating my 80 year old Grandfather Clock. (I'm quite a n00bie at this.) Are there any nasty system side effects to tuning these parameters? This is not a server I'm working with. It's my research desktop machine. I do lots and lots of different things on it. I started out with RAID 0, but after reading a lot I learned that this is Dangerous. So I bought a 3rd disk to do RAID 5. I intend to put LVM on top of the RAID 5 once I get it running at the speed it is supposed to, and then copy my entire linux system onto it. Are these tuned parameters going to mess other things up? Is there official documentation on this relative to RAID 5? I don't see much online. One thing I did learn is that if I use the --direct switch with `hdparm' I get much greater read speed. Goes from ~65 MB/s to ~120 MB/s. I have no idea what --direct does. Yes, I've read the man page. It says that it causes bypassing of the page cache. Ooookay. Alas using dd I find that the real read speed is still around ~65 MB/s. Sorry for these musings. I'm just uncomfortable trying to diddle with 5-10 system parameters without knowing what I'm doing. Any help or pointers to documentaion on tuning these for RAID-5 and what the tradeoffs are would be appreciated. Thanks. Dean - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Help: very slow software RAID 5.
Justin Piszcz wrote: : One of the 5-10 tuning settings: : : blockdev --getra /dev/md0 : : Try setting it to 4096,8192,16384,32768,65536 : : blockdev --setra 4096 /dev/md0 : : I discovered your January correspondence to the list about this. Yes, the read-ahead length makes a dramtic difference---for sequential data reading. However, in doing some further study on this parameter, I see that random access is going to suffer. Since I had intended to build an LVM on this RAID 5 array and put a full linux system on it, I'm not sure that large read-ahead values are a good idea. : Also, with a 3-disk raid5 that is the worst performance you can get using : only 3 disks I don't know why (for reads) I should suffer such bad performance. According to all I've read, the system is not even supposed to read the parity data on reads. So why do I not get near RAID 0 speeds w/o having to increase the Read-ahead value? : , while with a 10 disk raid5 it'd be closer to 90%. Reads you : should get good speeds, but for writes-- probably not. When you say reads you should get good speeds are you referring to the aforementioned 10 disk RAID 5 or my 3 disk one? : Then re-benchmark. Large Read-ahead nearly double the speed of 'hdparm -t'. (So does simply using the --direct flag. Can you explain this?) Also, in your opinion, is it wise to use such large read-aheads for a RAID 5 intended for the use to which I plan to put it? Aside: I have found RAID quite frustrating. With the original two disks I was getting 120-130 MB/s in RAID 0. I would think that for the investment of a 3rd drive I ought to get the modicum of redundancey I expect and keep the speed (at least on reads) w/o sacrificing anything. But it appears I actually lost something for my investment. I'm back to the speed of single drives with the modicum of redundancey that RAID 5 gives. Not a very good deal. Dean - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Help: very slow software RAID 5.
On Wed, 19 Sep 2007, Dean S. Messing wrote: Justin Piszcz wrote: : One of the 5-10 tuning settings: : : blockdev --getra /dev/md0 : : Try setting it to 4096,8192,16384,32768,65536 : : blockdev --setra 4096 /dev/md0 : : I discovered your January correspondence to the list about this. Yes, the read-ahead length makes a dramtic difference---for sequential data reading. However, in doing some further study on this parameter, I see that random access is going to suffer. Since I had intended to build an LVM on this RAID 5 array and put a full linux system on it, I'm not sure that large read-ahead values are a good idea. : Also, with a 3-disk raid5 that is the worst performance you can get using : only 3 disks I don't know why (for reads) I should suffer such bad performance. According to all I've read, the system is not even supposed to read the parity data on reads. So why do I not get near RAID 0 speeds w/o having to increase the Read-ahead value? : , while with a 10 disk raid5 it'd be closer to 90%. Reads you : should get good speeds, but for writes-- probably not. When you say reads you should get good speeds are you referring to the aforementioned 10 disk RAID 5 or my 3 disk one? : Then re-benchmark. Large Read-ahead nearly double the speed of 'hdparm -t'. (So does simply using the --direct flag. Can you explain this?) Also, in your opinion, is it wise to use such large read-aheads for a RAID 5 intended for the use to which I plan to put it? Aside: I have found RAID quite frustrating. With the original two disks I was getting 120-130 MB/s in RAID 0. I would think that for the investment of a 3rd drive I ought to get the modicum of redundancey I expect and keep the speed (at least on reads) w/o sacrificing anything. But it appears I actually lost something for my investment. I'm back to the speed of single drives with the modicum of redundancey that RAID 5 gives. Not a very good deal. Dean - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Did you try increasing the readahead and benchmarking? You can set it back to what it was when you are done. Justin. - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Help: very slow software RAID 5.
I'm not getting nearly the read speed I expected from a newly defined software RAID 5 array across three disk partitions (on the 3 drives, of course!). Would someone kindly point me straight? After defining the RAID 5 I did `hdparm -t /dev/md0' and got the abysmal read speed of ~65MB/sec. The individual device speeds are ~55, ~70, and ~75 MB/sec. Shouldn't this array be running (at the slowest) at about 55+70 = 125 MB/sec minus some overhead? I defined a RAID0 on the ~55 and ~70 partitions and got about 110 MB/sec. Shouldn't adding a 3rd (faster!) drive into the array make the RAID 5 speed at least this fast? Here are the details of my setup: Linux Fedora 7, kernel 2.6.22. # fdisk -l /dev/sda Disk /dev/sda: 160.0 GB, 1600 bytes 255 heads, 63 sectors/track, 19452 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 1 127 1020096 82 Linux swap / Solaris /dev/sda2 * 128 143 128520 83 Linux /dev/sda3 144 19452 155099542+ fd Linux raid autodetect # fdisk -l /dev/sdb Disk /dev/sdb: 160.0 GB, 1600 bytes 255 heads, 63 sectors/track, 19452 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdb1 * 1 127 1020096 82 Linux swap / Solaris /dev/sdb2 128 143 128520 83 Linux /dev/sdb3 144 19452 155099542+ fd Linux raid autodetect # fdisk -l /dev/sdc Disk /dev/sdc: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdc1 * 1 127 1020096 82 Linux swap / Solaris /dev/sdc2 128 19436 155099542+ fd Linux raid autodetect /dev/sdc3 19437 60801 332264362+ 8e Linux LVM The RAID 5 consists of sda3, sdb3, and sdc2. These partitions have these individual read speeds: # hdparm -t /dev/sda3 /dev/sdb3 /dev/sdc2 /dev/sda3: Timing buffered disk reads: 168 MB in 3.03 seconds = 55.39 MB/sec /dev/sdb3: Timing buffered disk reads: 216 MB in 3.03 seconds = 71.35 MB/sec /dev/sdc2: Timing buffered disk reads: 228 MB in 3.02 seconds = 75.49 MB/sec After defining RAID 5 with: mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sda3 /dev/sdb3 /dev/sdc2 and waiting the 50 minutes for /proc/mdstat to show it was finished, I did `hdparm -t /dev/md0' and got ~65MB/sec. Dean - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Help: very slow software RAID 5.
On Tue, 18 Sep 2007, Dean S. Messing wrote: I'm not getting nearly the read speed I expected from a newly defined software RAID 5 array across three disk partitions (on the 3 drives, of course!). Would someone kindly point me straight? After defining the RAID 5 I did `hdparm -t /dev/md0' and got the abysmal read speed of ~65MB/sec. The individual device speeds are ~55, ~70, and ~75 MB/sec. Shouldn't this array be running (at the slowest) at about 55+70 = 125 MB/sec minus some overhead? I defined a RAID0 on the ~55 and ~70 partitions and got about 110 MB/sec. Shouldn't adding a 3rd (faster!) drive into the array make the RAID 5 speed at least this fast? Here are the details of my setup: Linux Fedora 7, kernel 2.6.22. # fdisk -l /dev/sda Disk /dev/sda: 160.0 GB, 1600 bytes 255 heads, 63 sectors/track, 19452 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sda1 1 127 1020096 82 Linux swap / Solaris /dev/sda2 * 128 143 128520 83 Linux /dev/sda3 144 19452 155099542+ fd Linux raid autodetect # fdisk -l /dev/sdb Disk /dev/sdb: 160.0 GB, 1600 bytes 255 heads, 63 sectors/track, 19452 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdb1 * 1 127 1020096 82 Linux swap / Solaris /dev/sdb2 128 143 128520 83 Linux /dev/sdb3 144 19452 155099542+ fd Linux raid autodetect # fdisk -l /dev/sdc Disk /dev/sdc: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdc1 * 1 127 1020096 82 Linux swap / Solaris /dev/sdc2 128 19436 155099542+ fd Linux raid autodetect /dev/sdc3 19437 60801 332264362+ 8e Linux LVM The RAID 5 consists of sda3, sdb3, and sdc2. These partitions have these individual read speeds: # hdparm -t /dev/sda3 /dev/sdb3 /dev/sdc2 /dev/sda3: Timing buffered disk reads: 168 MB in 3.03 seconds = 55.39 MB/sec /dev/sdb3: Timing buffered disk reads: 216 MB in 3.03 seconds = 71.35 MB/sec /dev/sdc2: Timing buffered disk reads: 228 MB in 3.02 seconds = 75.49 MB/sec After defining RAID 5 with: mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 /dev/sda3 /dev/sdb3 /dev/sdc2 and waiting the 50 minutes for /proc/mdstat to show it was finished, I did `hdparm -t /dev/md0' and got ~65MB/sec. Dean - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Without tuning you will get very slow read speed. Read the mailing list, there are about 5-10 tunable options, for me, I get 250 MiB/s no tuning (read/write), after tuning 464 MiB/s write and 622 MiB/s read. Justin. - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Help: very slow software RAID 5.
Justin Piszcz wrote: On Tue, 18 Sep 2007, Dean S. Messing wrote: : : : : I'm not getting nearly the read speed I expected : from a newly defined software RAID 5 array : across three disk partitions (on the 3 drives, : of course!). : : Would someone kindly point me straight? : : After defining the RAID 5 I did `hdparm -t /dev/md0' : and got the abysmal read speed of ~65MB/sec. : The individual device speeds are ~55, ~70, : and ~75 MB/sec. : : Shouldn't this array be running (at the slowest) : at about 55+70 = 125 MB/sec minus some overhead? : I defined a RAID0 on the ~55 and ~70 partitions : and got about 110 MB/sec. : : Shouldn't adding a 3rd (faster!) drive into the : array make the RAID 5 speed at least this fast? : : : Here are the details of my setup: : snip : Without tuning you will get very slow read speed. : : Read the mailing list, there are about 5-10 tunable options, for me, I get : 250 MiB/s no tuning (read/write), after tuning 464 MiB/s write and 622 : MiB/s read. : : Justin. Thanks Justin. 5-10 tunable options! Good grief. This sounds worse than regulating my 80 year old Grandfather Clock. (I'm quite a n00bie at this.) Are there any nasty system side effects to tuning these parameters? This is not a server I'm working with. It's my research desktop machine. I do lots and lots of different things on it. I started out with RAID 0, but after reading a lot I learned that this is Dangerous. So I bought a 3rd disk to do RAID 5. I intend to put LVM on top of the RAID 5 once I get it running at the speed it is supposed to, and then copy my entire linux system onto it. Are these tuned parameters going to mess other things up? Is there official documentation on this relative to RAID 5? I don't see much online. One thing I did learn is that if I use the --direct switch with `hdparm' I get much greater read speed. Goes from ~65 MB/s to ~120 MB/s. I have no idea what --direct does. Yes, I've read the man page. It says that it causes bypassing of the page cache. Ooookay. Alas using dd I find that the real read speed is still around ~65 MB/s. Sorry for these musings. I'm just uncomfortable trying to diddle with 5-10 system parameters without knowing what I'm doing. Any help or pointers to documentaion on tuning these for RAID-5 and what the tradeoffs are would be appreciated. Thanks. Dean - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
