Re: Reading takes 100% precedence over writes for mdadm+raid5?
This is what dstat shows me copying lots of large files about (ext3), one file at a time. I've benchmarked the raid itself around 65-70 MB/s maximum actual write I/O so this 3-4MB/s stuff is pretty bad. I should note that ALL other I/O suffers horribly, even on other filesystems. What might the cause be? I should note: going larger in stripe_cache_size (384 and 512) performance stays the same, going smaller (128) performance *increases* and stays more steady to 10-13 MB/s. total-cpu-usage --dsk/sda-- --dsk/sdb-- --dsk/sdc-- --dsk/sdd-- -dsk/total- usr sys idl wai hiq siq| read writ: read writ: read writ: read writ: read writ 1 1 95 3 0 0| 12k 4261B: 106k 125k: 83k 110k: 83k 110k: 283k 348k 0 5 0 91 1 2| 0 0 :2384k 4744k:2612k 4412k:2336k 4804k:7332k 14M 0 4 0 91 1 3| 0 0 :2352k 4964k:2392k 4812k:2620k 4764k:7364k 14M 0 4 0 92 1 3| 0 0 :1068k 3524k:1336k 3184k:1360k 2912k:3764k 9620k 0 4 0 92 1 2| 0 0 :2304k 2612k:2128k 2484k:2332k 3028k:6764k 8124k 0 4 0 92 1 2| 0 0 :1584k 3428k:1252k 3992k:1592k 3416k:4428k 11M 0 3 0 93 0 2| 0 0 :1400k 2364k:1424k 2700k:1584k 2592k:4408k 7656k 0 4 0 93 1 2| 0 0 :1764k 3084k:1820k 2972k:1796k 2396k:5380k 8452k 0 4 0 92 2 3| 0 0 :1984k 3736k:1772k 4024k:1792k 4524k:5548k 12M 0 4 0 93 1 2| 0 0 :1852k 3860k:1840k 3408k:1696k 3648k:5388k 11M 0 4 0 93 0 2| 0 0 :1328k 2500k:1640k 2348k:1672k 2128k:4640k 6976k 0 4 0 92 0 4| 0 0 :1624k 3944k:2080k 3432k:1760k 3704k:5464k 11M 0 1 0 97 1 2| 0 0 :1480k 1340k: 976k 1564k:1268k 1488k:3724k 4392k 0 4 0 92 1 2| 0 0 :1320k 2676k:1608k 2548k: 968k 2572k:3896k 7796k 0 2 0 96 1 1| 0 0 :1856k 1808k:1752k 1988k:1752k 1600k:5360k 5396k 0 4 0 92 2 1| 0 0 :1360k 2560k:1240k 2788k:1580k 2940k:4180k 8288k 0 2 0 97 1 2| 0 0 :1928k 1456k:1628k 2080k:1488k 2308k:5044k 5844k 1 3 0 94 2 2| 0 0 :1432k 2156k:1320k 1840k: 936k 1072k:3688k 5068k 0 3 0 93 2 2| 0 0 :1760k 2164k:1440k 2384k:1276k 2972k:4476k 7520k 0 3 0 95 1 2| 0 0 :1088k 1064k: 896k 1424k:1152k 992k:3136k 3480k 0 0 0 96 0 2| 0 0 : 976k 888k: 632k 1120k:1016k 968k:2624k 2976k 0 2 0 94 1 2| 0 0 :1120k 1864k: 964k 1776k:1060k 1856k:3144k 5496k -- Jon - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Reading takes 100% precedence over writes for mdadm+raid5?
On Dec 6, 2007 1:06 AM, Justin Piszcz [EMAIL PROTECTED] wrote: On Wed, 5 Dec 2007, Jon Nelson wrote: I saw something really similar while moving some very large (300MB to 4GB) files. I was really surprised to see actual disk I/O (as measured by dstat) be really horrible. Any work-arounds, or just don't perform heavy reads the same time as writes? What kernel are you using? (Did I miss it in your OP?) The per-device write throttling in 2.6.24 should help significantly, have you tried the latest -rc and compared to your current kernel? -Dave - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Reading takes 100% precedence over writes for mdadm+raid5?
On Thu, 6 Dec 2007, David Rees wrote: On Dec 6, 2007 1:06 AM, Justin Piszcz [EMAIL PROTECTED] wrote: On Wed, 5 Dec 2007, Jon Nelson wrote: I saw something really similar while moving some very large (300MB to 4GB) files. I was really surprised to see actual disk I/O (as measured by dstat) be really horrible. Any work-arounds, or just don't perform heavy reads the same time as writes? What kernel are you using? (Did I miss it in your OP?) The per-device write throttling in 2.6.24 should help significantly, have you tried the latest -rc and compared to your current kernel? -Dave 2.6.23.9-- thanks will try out the latest -rc or wait for 2.6.24! Justin. - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Reading takes 100% precedence over writes for mdadm+raid5?
On 12/6/07, David Rees [EMAIL PROTECTED] wrote: On Dec 6, 2007 1:06 AM, Justin Piszcz [EMAIL PROTECTED] wrote: On Wed, 5 Dec 2007, Jon Nelson wrote: I saw something really similar while moving some very large (300MB to 4GB) files. I was really surprised to see actual disk I/O (as measured by dstat) be really horrible. Any work-arounds, or just don't perform heavy reads the same time as writes? What kernel are you using? (Did I miss it in your OP?) The per-device write throttling in 2.6.24 should help significantly, have you tried the latest -rc and compared to your current kernel? I was using 2.6.22.12 I think (openSUSE kernel). I can try using pretty much any kernel - I'm preparing to do an unrelated test using 2.6.24rc4 this weekend. If I remember I'll try to see what disk I/O looks like there. -- Jon - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Reading takes 100% precedence over writes for mdadm+raid5?
Justin Piszcz wrote: root 2206 1 4 Dec02 ?00:10:37 dd if /dev/zero of 1.out bs 1M root 2207 1 4 Dec02 ?00:10:38 dd if /dev/zero of 2.out bs 1M root 2208 1 4 Dec02 ?00:10:35 dd if /dev/zero of 3.out bs 1M root 2209 1 4 Dec02 ?00:10:45 dd if /dev/zero of 4.out bs 1M root 2210 1 4 Dec02 ?00:10:35 dd if /dev/zero of 5.out bs 1M root 2211 1 4 Dec02 ?00:10:35 dd if /dev/zero of 6.out bs 1M root 2212 1 4 Dec02 ?00:10:30 dd if /dev/zero of 7.out bs 1M root 2213 1 4 Dec02 ?00:10:42 dd if /dev/zero of 8.out bs 1M root 2214 1 4 Dec02 ?00:10:35 dd if /dev/zero of 9.out bs 1M root 2215 1 4 Dec02 ?00:10:37 dd if /dev/zero of 10.out bs 1M root 3080 24.6 0.0 10356 1672 ?D01:22 5:51 dd if /dev/md3 of /dev/null bs 1M Was curious if when running 10 DD's (which are writing to the RAID 5) fine, no issues, suddenly all go into D-state and let the read/give it 100% priority? Is this normal? I'm jumping back to the start of this thread, because after reading all the discussion I noticed that you are mixing apples and oranges here. Your write programs are going to files in the filesystem, and your read is going against the raw device. That may explain why you see something I haven't noticed doing all filesystem i/o. I am going to do a large rsync to another filesystem in the next two days, I will turn on some measurements when I do. But if you are just investigating this behavior, perhaps you could retry with a single read from a file rather than the device. [...snip...] -- Bill Davidsen [EMAIL PROTECTED] Woe unto the statesman who makes war without a reason that will still be valid when the war is over... Otto von Bismark - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Reading takes 100% precedence over writes for mdadm+raid5?
I saw something really similar while moving some very large (300MB to 4GB) files. I was really surprised to see actual disk I/O (as measured by dstat) be really horrible. -- Jon - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Reading takes 100% precedence over writes for mdadm+raid5?
root 2206 1 4 Dec02 ?00:10:37 dd if /dev/zero of 1.out bs 1M root 2207 1 4 Dec02 ?00:10:38 dd if /dev/zero of 2.out bs 1M root 2208 1 4 Dec02 ?00:10:35 dd if /dev/zero of 3.out bs 1M root 2209 1 4 Dec02 ?00:10:45 dd if /dev/zero of 4.out bs 1M root 2210 1 4 Dec02 ?00:10:35 dd if /dev/zero of 5.out bs 1M root 2211 1 4 Dec02 ?00:10:35 dd if /dev/zero of 6.out bs 1M root 2212 1 4 Dec02 ?00:10:30 dd if /dev/zero of 7.out bs 1M root 2213 1 4 Dec02 ?00:10:42 dd if /dev/zero of 8.out bs 1M root 2214 1 4 Dec02 ?00:10:35 dd if /dev/zero of 9.out bs 1M root 2215 1 4 Dec02 ?00:10:37 dd if /dev/zero of 10.out bs 1M root 3080 24.6 0.0 10356 1672 ?D01:22 5:51 dd if /dev/md3 of /dev/null bs 1M Was curious if when running 10 DD's (which are writing to the RAID 5) fine, no issues, suddenly all go into D-state and let the read/give it 100% priority? Is this normal? # du -sb . ; sleep 300; du -sb . 1115590287487 . 1115590287487 . Here my my raid5 config: # mdadm -D /dev/md3 /dev/md3: Version : 00.90.03 Creation Time : Sun Dec 2 12:15:20 2007 Raid Level : raid5 Array Size : 1465143296 (1397.27 GiB 1500.31 GB) Used Dev Size : 732571648 (698.63 GiB 750.15 GB) Raid Devices : 3 Total Devices : 3 Preferred Minor : 3 Persistence : Superblock is persistent Update Time : Sun Dec 2 22:00:54 2007 State : active Active Devices : 3 Working Devices : 3 Failed Devices : 0 Spare Devices : 0 Layout : left-symmetric Chunk Size : 1024K UUID : fea48e85:ddd2c33f:d19da839:74e9c858 (local to host box1) Events : 0.15 Number Major Minor RaidDevice State 0 8 330 active sync /dev/sdc1 1 8 491 active sync /dev/sdd1 2 8 652 active sync /dev/sde1 - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Reading takes 100% precedence over writes for mdadm+raid5?
On Sunday December 2, [EMAIL PROTECTED] wrote: Was curious if when running 10 DD's (which are writing to the RAID 5) fine, no issues, suddenly all go into D-state and let the read/give it 100% priority? So are you saying that the writes completely stalled while the read was progressing? How exactly did you measure that? What kernel version are you running. Is this normal? It shouldn't be. NeilBrown - To unsubscribe from this list: send the line unsubscribe linux-raid in the body of a message to [EMAIL PROTECTED] More majordomo info at http://vger.kernel.org/majordomo-info.html
