Re: RAID 5 Grow
Richard Scobie wrote: I will soon be adding another same sized drive to an existing 3 drive RAID 5 array. The machine is running Fedora Core 6 with kernel 2.6.20-1.2952.fc6 and mdadm 2.5.4, both of which are the latest available Fedora packages. Is anyone aware of any obvious bugs in either of these that will jeopardise this resize? I can compile and install later versions if required, but I'd rather leave the system standard if I can. I did a grow on RAID5 using an earlier FC6 setup, so I don't think you will have a problem. -- bill davidsen [EMAIL PROTECTED] CTO TMR Associates, Inc Doing interesting things with small computers since 1979 - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: stripe_cache_size and performance
Justin Piszcz wrote: I have found a 16MB stripe_cache_size results in optimal performance after testing many many values :) We have discussed this before, my experience has been that after 8 x stripe size the performance gains hit diminishing returns, particularly for typical write instead of big aligned blocks, possibly with O_DIRECT. I would suggest that as a target even on a low memory machine. Do your tests show similar? I was only able to test three and four drive setups using dedicated drives. -- bill davidsen [EMAIL PROTECTED] CTO TMR Associates, Inc Doing interesting things with small computers since 1979 - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: stripe_cache_size and performance
On Mon, 25 Jun 2007, Bill Davidsen wrote: Justin Piszcz wrote: I have found a 16MB stripe_cache_size results in optimal performance after testing many many values :) We have discussed this before, my experience has been that after 8 x stripe size the performance gains hit diminishing returns, particularly for typical write instead of big aligned blocks, possibly with O_DIRECT. I would suggest that as a target even on a low memory machine. Do your tests show similar? I was only able to test three and four drive setups using dedicated drives. -- bill davidsen [EMAIL PROTECTED] CTO TMR Associates, Inc Doing interesting things with small computers since 1979 I will re-benchmark with my current setup, however: Each of these are averaged over three runs: 128k_stripe: 69.2MB/s 256k_stripe: 105.3MB/s 512k_stripe: 142.0MB/s 1024k_stripe: 144.6MB/s 2048k_stripe: 208.3MB/s 4096k_stripe: 223.6MB/s 8192k_stripe: 226.0MB/s 16384k_stripe: 215.0MB/s This was with 6 sata disks. Justin. - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: stripe_cache_size and performance
On Mon, 25 Jun 2007, Justin Piszcz wrote: On Mon, 25 Jun 2007, Bill Davidsen wrote: Justin Piszcz wrote: I have found a 16MB stripe_cache_size results in optimal performance after testing many many values :) We have discussed this before, my experience has been that after 8 x stripe size the performance gains hit diminishing returns, particularly for typical write instead of big aligned blocks, possibly with O_DIRECT. I would suggest that as a target even on a low memory machine. Do your tests show similar? I was only able to test three and four drive setups using dedicated drives. -- bill davidsen [EMAIL PROTECTED] CTO TMR Associates, Inc Doing interesting things with small computers since 1979 I will re-benchmark with my current setup, however: Each of these are averaged over three runs: 128k_stripe: 69.2MB/s 256k_stripe: 105.3MB/s 512k_stripe: 142.0MB/s 1024k_stripe: 144.6MB/s 2048k_stripe: 208.3MB/s 4096k_stripe: 223.6MB/s 8192k_stripe: 226.0MB/s 16384k_stripe: 215.0MB/s This was with 6 sata disks. Justin. - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html I set it to 32 and my machine has several hung processes in D state, not good. I will start with 128k and up. Justin. - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: stripe_cache_size and performance [BUG with =64kb]
On Mon, 25 Jun 2007, Justin Piszcz wrote: On Mon, 25 Jun 2007, Justin Piszcz wrote: On Mon, 25 Jun 2007, Bill Davidsen wrote: Justin Piszcz wrote: I have found a 16MB stripe_cache_size results in optimal performance after testing many many values :) We have discussed this before, my experience has been that after 8 x stripe size the performance gains hit diminishing returns, particularly for typical write instead of big aligned blocks, possibly with O_DIRECT. I would suggest that as a target even on a low memory machine. Do your tests show similar? I was only able to test three and four drive setups using dedicated drives. -- bill davidsen [EMAIL PROTECTED] CTO TMR Associates, Inc Doing interesting things with small computers since 1979 - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Running test with 10 RAPTOR 150 hard drives, expect it to take awhile until I get the results, avg them etc. :) 128k,256k,512k,1024k,2048k,4096k,8192k,16384k Justin. Definitely a kernel bug, I set it to 64kb and it stayed in D-state until I ran alt-sysrq-b. Pretty nasty! Justin. Ack, help?? [ 64.032895] Starting XFS recovery on filesystem: md3 (logdev: internal) [ 66.210602] XFS: xlog_recover_process_data: bad clientid [ 66.210656] XFS: log mount/recovery failed: error 5 [ 66.210709] XFS: log mount failed After I ran 64kb it killed my RAID! - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: stripe_cache_size and performance [BUG with =64kb]
On Mon, 25 Jun 2007, Justin Piszcz wrote: On Mon, 25 Jun 2007, Justin Piszcz wrote: On Mon, 25 Jun 2007, Justin Piszcz wrote: On Mon, 25 Jun 2007, Bill Davidsen wrote: Justin Piszcz wrote: I have found a 16MB stripe_cache_size results in optimal performance after testing many many values :) We have discussed this before, my experience has been that after 8 x stripe size the performance gains hit diminishing returns, particularly for typical write instead of big aligned blocks, possibly with O_DIRECT. I would suggest that as a target even on a low memory machine. Do your tests show similar? I was only able to test three and four drive setups using dedicated drives. -- bill davidsen [EMAIL PROTECTED] CTO TMR Associates, Inc Doing interesting things with small computers since 1979 - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Running test with 10 RAPTOR 150 hard drives, expect it to take awhile until I get the results, avg them etc. :) 128k,256k,512k,1024k,2048k,4096k,8192k,16384k Justin. Definitely a kernel bug, I set it to 64kb and it stayed in D-state until I ran alt-sysrq-b. Pretty nasty! Justin. Ack, help?? [ 64.032895] Starting XFS recovery on filesystem: md3 (logdev: internal) [ 66.210602] XFS: xlog_recover_process_data: bad clientid [ 66.210656] XFS: log mount/recovery failed: error 5 [ 66.210709] XFS: log mount failed After I ran 64kb it killed my RAID! - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Yeah, I won't be trying that anymore :P p34:/r1# find lost+found/|wc 157 1573369 p34:/r1# du -sh lost+found/ 166Glost+found/ p34:/r1# - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: stripe_cache_size and performance
On Thu, 21 Jun 2007, Jon Nelson wrote: On Thu, 21 Jun 2007, Raz wrote: What is your raid configuration ? Please note that the stripe_cache_size is acting as a bottle neck in some cases. Well, that's kind of the point of my email. I'll try to restate things, as my question appears to have gotten lost. 1. I have a 3x component raid5, ~314G per component. Each component happens to be the 4th partition of a 320G SATA drive. Each drive can sustain approx. 70MB/s reads/writes. Except for the first drive, none of the other partitions are used for anything else at this time. The system is nominally quiescent during these tests. 2. The kernel is 2.6.18.8-0.3-default on x86_64 (openSUSE 10.2). 3. My best sustained write performance comes with a stripe_cache_size of 4096. Larger than that seems to reduce performance, although only very slightly. 4. At values below 4096, the absolute write performance is less than the best, but only marginally. 5. HOWEVER, at any value *above* 512 the 'check' performance is REALLY BAD. By 'check' performance I mean the value displayed by /proc/mdstat after I issue: echo check /sys/block/md0/md/sync_action When I say REALLY BAD I mean 3MB/s. 6. Here is a short incomplete table of stripe_cache_size to 'check' performance: 384 72-73MB/s 512 72-73MB/s 640 73-74MB/s 768.3-3.4MB/s And the performance stays bad as I increase the stripe_cache_size. 7. And now, the question: the best absolute 'write' performance comes with a stripe_cache_size value of 4096 (for my setup). However, any value of stripe_cache_size above 384 really, really hurts 'check' (and rebuild, one can assume) performance. Why? -- Jon Nelson [EMAIL PROTECTED] - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: stripe_cache_size and performance
7. And now, the question: the best absolute 'write' performance comes with a stripe_cache_size value of 4096 (for my setup). However, any value of stripe_cache_size above 384 really, really hurts 'check' (and rebuild, one can assume) performance. Why? Question: After performance goes bad does it go back up if you reduce the size back down to 384? -- Jon Nelson [EMAIL PROTECTED] Dan - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: stripe_cache_size and performance
On Mon, 25 Jun 2007, Jon Nelson wrote: On Thu, 21 Jun 2007, Jon Nelson wrote: On Thu, 21 Jun 2007, Raz wrote: What is your raid configuration ? Please note that the stripe_cache_size is acting as a bottle neck in some cases. Well, that's kind of the point of my email. I'll try to restate things, as my question appears to have gotten lost. 1. I have a 3x component raid5, ~314G per component. Each component happens to be the 4th partition of a 320G SATA drive. Each drive can sustain approx. 70MB/s reads/writes. Except for the first drive, none of the other partitions are used for anything else at this time. The system is nominally quiescent during these tests. 2. The kernel is 2.6.18.8-0.3-default on x86_64 (openSUSE 10.2). 3. My best sustained write performance comes with a stripe_cache_size of 4096. Larger than that seems to reduce performance, although only very slightly. 4. At values below 4096, the absolute write performance is less than the best, but only marginally. 5. HOWEVER, at any value *above* 512 the 'check' performance is REALLY BAD. By 'check' performance I mean the value displayed by /proc/mdstat after I issue: echo check /sys/block/md0/md/sync_action When I say REALLY BAD I mean 3MB/s. 6. Here is a short incomplete table of stripe_cache_size to 'check' performance: 384 72-73MB/s 512 72-73MB/s 640 73-74MB/s 768.3-3.4MB/s And the performance stays bad as I increase the stripe_cache_size. 7. And now, the question: the best absolute 'write' performance comes with a stripe_cache_size value of 4096 (for my setup). However, any value of stripe_cache_size above 384 really, really hurts 'check' (and rebuild, one can assume) performance. Why? -- Jon Nelson [EMAIL PROTECTED] - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html Neil has a patch for the bad speed. In the mean time, do this (or better to set it to 30, for instance): # Set minimum and maximum raid rebuild speed to 60MB/s. echo Setting minimum and maximum resync speed to 60 MiB/s... echo 6 /sys/block/md0/md/sync_speed_min echo 6 /sys/block/md0/md/sync_speed_max echo 6 /sys/block/md1/md/sync_speed_min echo 6 /sys/block/md1/md/sync_speed_max echo 6 /sys/block/md2/md/sync_speed_min echo 6 /sys/block/md2/md/sync_speed_max echo 6 /sys/block/md3/md/sync_speed_min echo 6 /sys/block/md3/md/sync_speed_max - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: stripe_cache_size and performance
On Mon, 25 Jun 2007, Dan Williams wrote: 7. And now, the question: the best absolute 'write' performance comes with a stripe_cache_size value of 4096 (for my setup). However, any value of stripe_cache_size above 384 really, really hurts 'check' (and rebuild, one can assume) performance. Why? Question: After performance goes bad does it go back up if you reduce the size back down to 384? Yes, and almost instantly. -- Jon Nelson [EMAIL PROTECTED] - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
