Re: which raid level gives maximum overall speed? (raid-10,f2 vs. raid-0)
Janek Kozicki wrote: writing on raid10 is supposed to be half the speed of reading. That's because it must write to both mirrors. I am not 100% certain about the following rules, but afaik any raid configuration has a theoretical[1] maximum read speed of the combined speed of all disks in the array and a maximum write speed equal to the combined speed of a disk-length of a stripe. By disk-length I mean how many disks are needed to reconstruct a single stripe - the rest of the writes are redundancy and are essentially non-accountable work. For raid5 it is N-1. For raid6 - N-2. For linux raid 10 it is N-C+1 where C is the number of chunk copies. So for -p n3 -n 5 we would get a maximum write speed of 3 x single drive speed. For raid1 the disk-length of a stripe is always 1. So the statement IMHO raid5 could perform good here, because in *continuous* write operation the blocks from other HDDs were just have been written, they stay in cache and can be used to calculate xor. So you could get close to almost raid-0 performance here. is quite incorrect. You will get close to raid-0 if you have many disks, but will never beat raid0, since once disk is always busy writing parity which is not part of the write request submitted to the mdX device in the first place. [1] Theoretical since any external factors (busy CPU, unsuitable elevator, random disk access, multiple raid levels on one physical device) would all contribute to take you further away from the maximums. - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: which raid level gives maximum overall speed? (raid-10,f2 vs. raid-0)
Bill Davidsen said: (by the date of Wed, 06 Feb 2008 13:16:14 -0500) Janek Kozicki wrote: Justin Piszcz said: (by the date of Tue, 5 Feb 2008 17:28:27 -0500 (EST)) writing on raid10 is supposed to be half the speed of reading. That's because it must write to both mirrors. ??? Are you assuming that write to mirrored copies are done sequentially rather than in parallel? Unless you have enough writes to saturate something the effective speed approaches the speed of a single drive. I just checked raid1 and raid5, writing 100MB with an fsync at the end. raid1 leveled off at 85% of a single drive after ~30MB. Hi, In above context I'm talking about raid10 (not about raid1, raid0, raid0+1, raid1+0, raid5 or raid6). Of course writes are done in parallel. When each chunk has two copies raid10 reads twice as fast as it writes. If each chunk has three copies, then writes are 1/3 speed of reading. If each chunk has number of copies equal to number of drives, then write speed drops down to that of a single drive - a 1/Nth of read speed. But it's all just a theory. I'd like to see more benchmarks :-) -- Janek Kozicki | - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: which raid level gives maximum overall speed? (raid-10,f2 vs. raid-0)
On Thu, Jan 31, 2008 at 02:55:07AM +0100, Keld Jørn Simonsen wrote: On Wed, Jan 30, 2008 at 11:36:39PM +0100, Janek Kozicki wrote: Keld Jørn Simonsen said: (by the date of Wed, 30 Jan 2008 23:00:07 +0100) All the raid10's will have double time for writing, and raid5 and raid6 will also have double or triple writing times, given that you can do striped writes on the raid0. For raid5 and raid6 I think this is even worse. My take is that for raid5 when you write something, you first read the chunk data involved, then you read the parity data, then you xor-subtract the data to be changed, and you xor-add the new data, and then write the new data chunk and the new parity chunk. In total 2 reads and 2 writes. The read/writes happen on the same chunks, so latency is minimized. But in essence it is still 4 IO operations, where it is only 2 writes on raid1/raid10, that is only half the speed for writing on raid5 compared to raid1/10. On raid6 this amounts to 6 IO operations, resulting in 1/3 of the writing speed of raid1/10. I note in passing that there is no difference between xor-subtract and xor-add. Also I assume that you can calculate the parities of both raid5 and raid6 given the old parities chunks and the old and new data chunk. If you have to calculate the new parities by reading all the component data chunks this is going to be really expensive, both in IO and CPU. For a 10 drive raid5 this would involve reading 9 data chunks, and making writes 5 times as expensive as raid1/10. best regards keld - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: which raid level gives maximum overall speed? (raid-10,f2 vs. raid-0)
On Tue, 5 Feb 2008, Keld Jørn Simonsen wrote: On Thu, Jan 31, 2008 at 02:55:07AM +0100, Keld Jørn Simonsen wrote: On Wed, Jan 30, 2008 at 11:36:39PM +0100, Janek Kozicki wrote: Keld Jørn Simonsen said: (by the date of Wed, 30 Jan 2008 23:00:07 +0100) All the raid10's will have double time for writing, and raid5 and raid6 will also have double or triple writing times, given that you can do striped writes on the raid0. For raid5 and raid6 I think this is even worse. My take is that for raid5 when you write something, you first read the chunk data involved, then you read the parity data, then you xor-subtract the data to be changed, and you xor-add the new data, and then write the new data chunk and the new parity chunk. In total 2 reads and 2 writes. The read/writes happen on the same chunks, so latency is minimized. But in essence it is still 4 IO operations, where it is only 2 writes on raid1/raid10, that is only half the speed for writing on raid5 compared to raid1/10. On raid6 this amounts to 6 IO operations, resulting in 1/3 of the writing speed of raid1/10. I note in passing that there is no difference between xor-subtract and xor-add. Also I assume that you can calculate the parities of both raid5 and raid6 given the old parities chunks and the old and new data chunk. If you have to calculate the new parities by reading all the component data chunks this is going to be really expensive, both in IO and CPU. For a 10 drive raid5 this would involve reading 9 data chunks, and making writes 5 times as expensive as raid1/10. best regards keld - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html On my benchmarks RAID5 gave the best overall speed with 10 raptors, although I did not play with the various offsets/etc as much as I have tweaked the RAID5. Justin.
Re: which raid level gives maximum overall speed? (raid-10,f2 vs. raid-0)
On Tue, Feb 05, 2008 at 11:54:27AM -0500, Justin Piszcz wrote: On Tue, 5 Feb 2008, Keld Jørn Simonsen wrote: On Thu, Jan 31, 2008 at 02:55:07AM +0100, Keld Jørn Simonsen wrote: On Wed, Jan 30, 2008 at 11:36:39PM +0100, Janek Kozicki wrote: Keld Jørn Simonsen said: (by the date of Wed, 30 Jan 2008 23:00:07 +0100) All the raid10's will have double time for writing, and raid5 and raid6 will also have double or triple writing times, given that you can do striped writes on the raid0. For raid5 and raid6 I think this is even worse. My take is that for raid5 when you write something, you first read the chunk data involved, then you read the parity data, then you xor-subtract the data to be changed, and you xor-add the new data, and then write the new data chunk and the new parity chunk. In total 2 reads and 2 writes. The read/writes happen on the same chunks, so latency is minimized. But in essence it is still 4 IO operations, where it is only 2 writes on raid1/raid10, that is only half the speed for writing on raid5 compared to raid1/10. On raid6 this amounts to 6 IO operations, resulting in 1/3 of the writing speed of raid1/10. I note in passing that there is no difference between xor-subtract and xor-add. Also I assume that you can calculate the parities of both raid5 and raid6 given the old parities chunks and the old and new data chunk. If you have to calculate the new parities by reading all the component data chunks this is going to be really expensive, both in IO and CPU. For a 10 drive raid5 this would involve reading 9 data chunks, and making writes 5 times as expensive as raid1/10. best regards keld - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html On my benchmarks RAID5 gave the best overall speed with 10 raptors, although I did not play with the various offsets/etc as much as I have tweaked the RAID5. Could you give some figures? best regards keld - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: which raid level gives maximum overall speed? (raid-10,f2 vs. raid-0)
On Tue, 5 Feb 2008, Keld Jørn Simonsen wrote: On Tue, Feb 05, 2008 at 11:54:27AM -0500, Justin Piszcz wrote: On Tue, 5 Feb 2008, Keld Jørn Simonsen wrote: On Thu, Jan 31, 2008 at 02:55:07AM +0100, Keld Jørn Simonsen wrote: On Wed, Jan 30, 2008 at 11:36:39PM +0100, Janek Kozicki wrote: Keld Jørn Simonsen said: (by the date of Wed, 30 Jan 2008 23:00:07 +0100) All the raid10's will have double time for writing, and raid5 and raid6 will also have double or triple writing times, given that you can do striped writes on the raid0. For raid5 and raid6 I think this is even worse. My take is that for raid5 when you write something, you first read the chunk data involved, then you read the parity data, then you xor-subtract the data to be changed, and you xor-add the new data, and then write the new data chunk and the new parity chunk. In total 2 reads and 2 writes. The read/writes happen on the same chunks, so latency is minimized. But in essence it is still 4 IO operations, where it is only 2 writes on raid1/raid10, that is only half the speed for writing on raid5 compared to raid1/10. On raid6 this amounts to 6 IO operations, resulting in 1/3 of the writing speed of raid1/10. I note in passing that there is no difference between xor-subtract and xor-add. Also I assume that you can calculate the parities of both raid5 and raid6 given the old parities chunks and the old and new data chunk. If you have to calculate the new parities by reading all the component data chunks this is going to be really expensive, both in IO and CPU. For a 10 drive raid5 this would involve reading 9 data chunks, and making writes 5 times as expensive as raid1/10. best regards keld - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html On my benchmarks RAID5 gave the best overall speed with 10 raptors, although I did not play with the various offsets/etc as much as I have tweaked the RAID5. Could you give some figures? I remember testing with bonnie++ and raid10 was about half the speed (200-265 MiB/s) as RAID5 (400-420 MiB/s) for sequential output, but input was closer to RAID5 speeds/did not seem affected (~550MiB/s). Justin.
Re: which raid level gives maximum overall speed? (raid-10,f2 vs. raid-0)
Justin Piszcz said: (by the date of Tue, 5 Feb 2008 17:28:27 -0500 (EST)) I remember testing with bonnie++ and raid10 was about half the speed (200-265 MiB/s) as RAID5 (400-420 MiB/s) for sequential output, writing on raid10 is supposed to be half the speed of reading. That's because it must write to both mirrors. IMHO raid5 could perform good here, because in *continuous* write operation the blocks from other HDDs were just have been written, they stay in cache and can be used to calculate xor. So you could get close to almost raid-0 performance here. Randomly scattered small-sized write operations will kill raid5 performance, for sure. Because corresponding blocks from few other drives must be read, to calculate parity correctly. I'm wondering how much raid5 performance would go down... Is there a bonnie++ test for that, or any other benchmark software for this? but input was closer to RAID5 speeds/did not seem affected (~550MiB/s). reading in raid5 and raid10 is supposed to be close to raid-0 speed. -- Janek Kozicki | - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: which raid level gives maximum overall speed? (raid-10,f2 vs. raid-0)
On Tue, Feb 05, 2008 at 05:28:27PM -0500, Justin Piszcz wrote: Could you give some figures? I remember testing with bonnie++ and raid10 was about half the speed (200-265 MiB/s) as RAID5 (400-420 MiB/s) for sequential output, but input was closer to RAID5 speeds/did not seem affected (~550MiB/s). Impressive. What levet of raid10 was involved? and what type of equipment, how many disks? Maybe the better output for raid5 could be due to some striping - AFAIK raid5 will be striping quite well, and writes almost equal to reading time indicates that the writes are striping too. best regards keld - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: which raid level gives maximum overall speed? (raid-10,f2 vs. raid-0)
On Tue, 5 Feb 2008, Keld Jørn Simonsen wrote: On Tue, Feb 05, 2008 at 05:28:27PM -0500, Justin Piszcz wrote: Could you give some figures? I remember testing with bonnie++ and raid10 was about half the speed (200-265 MiB/s) as RAID5 (400-420 MiB/s) for sequential output, but input was closer to RAID5 speeds/did not seem affected (~550MiB/s). Impressive. What levet of raid10 was involved? and what type of Like I said, it was baseline testing, so pretty much the default raid10 when you create it via mdadm, I did not mess with offsets, etc. equipment, how many disks? Ten 10,000rpm raptors. Maybe the better output for raid5 could be due to some striping - AFAIK raid5 will be striping quite well, and writes almost equal to reading time indicates that the writes are striping too. best regards keld
Re: which raid level gives maximum overall speed? (raid-10,f2 vs. raid-0)
Keld Jørn Simonsen said: (by the date of Thu, 31 Jan 2008 02:55:07 +0100) Given that you want maximum thruput for both reading and writing, I think there is only one way to go, that is raid0. All the raid10's will have double time for writing, and raid5 and raid6 will also have double or triple writing times, given that you can do striped writes on the raid0. For random and sequential writing in the normal case (no faulty disks) I would guess that all of the raid10's, the raid1 and raid5 are about equally fast, given the same amount of hardware. (raid5, raid6 a little slower given the unactive parity chunks). For random reading, raid0, raid1, raid10 should be equally fast, with raid5 a little slower, due to one of the disks virtually out of operation, as it is used for the XOR parity chunks. raid6 should be somewhat slower due to 2 non-operationable disks. raid10,f2 may have a slight edge due to virtually only using half the disk giving better average seek time, and using the faster outer disk halves. For sequential reading, raid0 and raid10,f2 should be equally fast. Possibly raid10,o2 comes quite close. My guess is that raid5 then is next, achieving striping rates, but with the loss of one parity drive, and then raid1 and raid10,n2 with equal performance. In degraded mode, I guess for random read/writes the difference is not big between any of the raid1, raid5 and raid10 layouts, while sequential reads will be especially bad for raid10,f2 approaching the random read rate, and others will enjoy the normal speed of the above filesystem (ext3, reiserfs, xfs etc). Wow! Thanks for detailed explanations. I was thinking that maybe raid10 on 4 drives could be faster than raid0. But now it's all logical for me. With 4 drives and raid10,f2 I could get an extra reading speed, but not the writing speed. Makes a lot of sense. Perhaps it should be added to linux-raid wiki? (and perhaps a FAQ there - isn't a question about speed a frequent one?) http://linux-raid.osdl.org/index.php/Main_Page Theory, theory theory. Show me some real figures. yes... that would be great if someone could spend some time benchmarking all possible configurations :-) thanks for your help! -- Janek Kozicki - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: which raid level gives maximum overall speed? (raid-10,f2 vs. raid-0)
Janek Kozicki wrote: Hello, Yes, I know that some levels give faster reading and slower writing, etc. I want to talk here about a typical workstation usage: compiling stuff (like kernel), editing openoffice docs, browsing web, reading email (email: I have a webdir format, and in boost mailing list directory I have 14000 files (posts), opening this directory takes circa 10 seconds in sylpheed). Moreover, opening .pdf files, more compiling of C++ stuff, etc... In other words, like most systems, more reads than writes. And while write can be (and usually are) cached and buffered, when you need the next bit of data the program waits for it, far more user visible. If this suggests tuning for acceptable write and max read speed, and setting the readahead higher than default, then you have reached the same conclusion as I did. I have a remote backup system configured (with rsnapshot), which does backups two times a day. So I'm not afraid to lose all my data due to disc failure. I want absolute speed. Currently I have Raid-0, because I was thinking that this one is fastest. But I also don't need twice the capacity. I could use Raid-1 as well, if it was faster. Due to recent discussion about Raid-10,f2 I'm getting worried that Raid-0 is not the fastest solution, but instead a Raid-10,f2 is faster. So how really is it, which level gives maximum overall speed? I would like to make a benchmark, but currently, technically, I'm not able to. I'll be able to do it next month, and then - as a result of this discussion - I will switch to other level and post here benchmark results. How does overall performance change with the number of available drives? Perhaps Raid-0 is best for 2 drives, while Raid-10 is best for 3, 4 and more drives? best regards -- Bill Davidsen [EMAIL PROTECTED] Woe unto the statesman who makes war without a reason that will still be valid when the war is over... Otto von Bismark - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: which raid level gives maximum overall speed? (raid-10,f2 vs. raid-0)
On Wed, Jan 30, 2008 at 07:21:33PM +0100, Janek Kozicki wrote: Hello, Yes, I know that some levels give faster reading and slower writing, etc. I want to talk here about a typical workstation usage: compiling stuff (like kernel), editing openoffice docs, browsing web, reading email (email: I have a webdir format, and in boost mailing list directory I have 14000 files (posts), opening this directory takes circa 10 seconds in sylpheed). Moreover, opening .pdf files, more compiling of C++ stuff, etc... I have a remote backup system configured (with rsnapshot), which does backups two times a day. So I'm not afraid to lose all my data due to disc failure. I want absolute speed. Currently I have Raid-0, because I was thinking that this one is fastest. But I also don't need twice the capacity. I could use Raid-1 as well, if it was faster. Due to recent discussion about Raid-10,f2 I'm getting worried that Raid-0 is not the fastest solution, but instead a Raid-10,f2 is faster. So how really is it, which level gives maximum overall speed? I would like to make a benchmark, but currently, technically, I'm not able to. I'll be able to do it next month, and then - as a result of this discussion - I will switch to other level and post here benchmark results. How does overall performance change with the number of available drives? Perhaps Raid-0 is best for 2 drives, while Raid-10 is best for 3, 4 and more drives? Teoretically, raid0 and raid10,f2 should be the same for reading, given the same size of the md partition, etc. For writing, raid10,f2 should be half the speed of raid0. This should go both for sequential and random read/writes. But I would like to have real test numbers. best regards keld - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: which raid level gives maximum overall speed? (raid-10,f2 vs. raid-0)
Keld Jørn Simonsen said: (by the date of Wed, 30 Jan 2008 23:00:07 +0100) Teoretically, raid0 and raid10,f2 should be the same for reading, given the same size of the md partition, etc. For writing, raid10,f2 should be half the speed of raid0. This should go both for sequential and random read/writes. But I would like to have real test numbers. Me too. Thanks. Are there any other raid levels that may count here? Raid-10 with some other options? -- Janek Kozicki | - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: which raid level gives maximum overall speed? (raid-10,f2 vs. raid-0)
On Wed, Jan 30, 2008 at 11:36:39PM +0100, Janek Kozicki wrote: Keld Jørn Simonsen said: (by the date of Wed, 30 Jan 2008 23:00:07 +0100) Teoretically, raid0 and raid10,f2 should be the same for reading, given the same size of the md partition, etc. For writing, raid10,f2 should be half the speed of raid0. This should go both for sequential and random read/writes. But I would like to have real test numbers. Me too. Thanks. Are there any other raid levels that may count here? Raid-10 with some other options? Given that you want maximum thruput for both reading and writing, I think there is only one way to go, that is raid0. All the raid10's will have double time for writing, and raid5 and raid6 will also have double or triple writing times, given that you can do striped writes on the raid0. For random and sequential writing in the normal case (no faulty disks) I would guess that all of the raid10's, the raid1 and raid5 are about equally fast, given the same amount of hardware. (raid5, raid6 a little slower given the unactive parity chunks). For random reading, raid0, raid1, raid10 should be equally fast, with raid5 a little slower, due to one of the disks virtually out of operation, as it is used for the XOR parity chunks. raid6 should be somewhat slower due to 2 non-operationable disks. raid10,f2 may have a slight edge due to virtually only using half the disk giving better average seek time, and using the faster outer disk halves. For sequential reading, raid0 and raid10,f2 should be equally fast. Possibly raid10,o2 comes quite close. My guess is that raid5 then is next, achieving striping rates, but with the loss of one parity drive, and then raid1 and raid10,n2 with equal performance. In degraded mode, I guess for random read/writes the difference is not big between any of the raid1, raid5 and raid10 layouts, while sequential reads will be especially bad for raid10,f2 approaching the random read rate, and others will enjoy the normal speed of the above filesystem (ext3, reiserfs, xfs etc). Theory, theory theory. Show me some real figures. Best regards Keld - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
