Re: One random read streaming is fast (~1200MB/s), but two or more are slower (~750MB/s)?
Hey Josef,
Were you able to try this multi-thread test on any more drives?
I did a test with 12, 6, 3, and 1 drive. And, it looks like I see the
multi-thread speed reduces, as the number of drives in the raid goes
up.
Like this:
- 50% speed reduction with 2 threads on 12 drives
- 25% speed reduction with 2 threads on 6 drives
- 10% speed reduction with 2 threads on 3 drives
- 5% speed reduction with 2 threads on 1 drive
I only have 12 slots on my HBA card, but I wonder if 24 drives would
reduce the speed to 25% with 2 threads?
Matt
make btrfs fs...
___
12 drives...
mkfs.btrfs -f -d raid6 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde
/dev/sdf /dev/sdg /dev/sdh /dev/sdi /dev/sdj /dev/sdk /dev/sdl
6 drives...
mkfs.btrfs -f -d raid6 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf
3 drives...
mkfs.btrfs -f -d raid5 /dev/sda /dev/sdb /dev/sdc
1 drive...
mkfs.btrfs -f /dev/sda
mount /dev/sda /tmp/btrfs_test/
___
make zero files...
___
kura1 ~ # for j in {1..2} ; do dd if=/dev/zero
of=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M count=1
conv=fdatasync done
___
===
btrfs raid6 on 12 drives with 2 threads = ~650MB/s
___
kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..2} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M
done
vm.drop_caches = 1
1048576 bytes (10 GB) copied, 31.0431 s, 338 MB/s
1048576 bytes (10 GB) copied, 31.2235 s, 336 MB/s
kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..2} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M
done
1048576 bytes (10 GB) copied, 29.869 s, 351 MB/s
1048576 bytes (10 GB) copied, 30.5561 s, 343 MB/s
___
btrfs raid6 on 12 drives with 1 thread = ~1100MB/s
___
kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..1} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M
done
1048576 bytes (10 GB) copied, 9.69881 s, 1.1 GB/s
kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..1} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M
done
1048576 bytes (10 GB) copied, 9.56475 s, 1.1 GB/s
___
==
btrfs raid6 on 6 drives with 2 thread = ~500MB/s
___
kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..2} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M
done
1048576 bytes (10 GB) copied, 41.3899 s, 253 MB/s
1048576 bytes (10 GB) copied, 41.6916 s, 252 MB/s
kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..2} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M
done
1048576 bytes (10 GB) copied, 40.3178 s, 260 MB/s
1048576 bytes (10 GB) copied, 41.4087 s, 253 MB/s
___
btrfs raid6 on 6 drives with 1 thread = ~600MB/s
___
kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..1} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M
done
1048576 bytes (10 GB) copied, 17.5686 s, 597 MB/s
kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..1} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M
done
1048576 bytes (10 GB) copied, 17.5396 s, 598 MB/s
___
==
btrfs raid5 on 3 drives with 2 thread = ~300MB/s
___
kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..2} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M
done
1048576 bytes (10 GB) copied, 67.636 s, 155 MB/s
1048576 bytes (10 GB) copied, 70.1783 s, 149 MB/s
kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..2} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M
done
1048576 bytes (10 GB) copied, 69.4945 s, 151 MB/s
1048576 bytes (10 GB) copied, 70.8279 s, 148 MB/s
___
btrfs raid5 on 3 drives with 1 thread = ~319MB/s
___
kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..1} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M
done
1048576 bytes (10 GB) copied, 32.8559 s, 319 MB/s
kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..1} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M
done
1048576 bytes (10 GB) copied, 32.8483 s, 319 MB/s
___
==
btrfs (no raid) on 1 drive with 2 thread = ~155MB/s
___
kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..2} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M
done
1048576 bytes (10 GB) copied, 134.982 s, 77.7 MB/s
1048576 bytes (10 GB) copied, 135.237 s, 77.5 MB/s
kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..2} ; do dd
of=/dev/null if=/tmp/btrfs_test/testfile_bs1m_size10GB_${j} bs=1M
done
1048576 bytes (10 GB) copied, 134.549 s, 77.9 MB/s
1048576 bytes (10 GB) copied, 135.293 s, 77.5 MB/s
___
btrfs (no raid) on 1 drive with 1 thread = ~162MB/s
___
kura1 btrfs_test # sysctl vm.drop_caches=1 ; for j in {1..1} ; do dd
of=/dev/null
Re: One random read streaming is fast (~1200MB/s), but two or more are slower (~750MB/s)?
Hey Josef, Thanks for looking into this further!That is about the same results that I was seeing, though I didn't test it with just one drive.. only with all 12 drives in my jbod. I will do a test with just one disk, and see if I also get the same results. Let me know if you also see the same results with multiple drives in your raid... Thanks, Matt On Thu, Apr 25, 2013 at 2:10 PM, Josef Bacik jba...@fusionio.com wrote: On Thu, Apr 25, 2013 at 03:01:18PM -0600, Matt Pursley wrote: Ok, awesome, let me know how it goes.. I don't have the raid formatted to btrfs right now, but I could probably do that in about 30 minutes or so. Huh so I'm getting the full bandwidth, 120 mb/s with one thread and 60 mb/s with two threads. These are just cheap sata drives tho, I'll try and dig up a box with 3 fusion cards for something a little closer to the speeds you are seeing and see if that makes a difference. Thanks, Josef -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: One random read streaming is fast (~1200MB/s), but two or more are slower (~750MB/s)?
Hey Josef, Were you able to look into this any further? It's still pretty reproducible on my machine... Thanks, Matt On Thu, Apr 18, 2013 at 2:58 PM, Josef Bacik jba...@fusionio.com wrote: This is strange, and I can't see any reason why this would happen. I'll try and reproduce next week when I'm back from LSF. Thanks, Josef -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: One random read streaming is fast (~1200MB/s), but two or more are slower (~750MB/s)?
On Thu, Apr 25, 2013 at 03:01:18PM -0600, Matt Pursley wrote: Ok, awesome, let me know how it goes.. I don't have the raid formatted to btrfs right now, but I could probably do that in about 30 minutes or so. Huh so I'm getting the full bandwidth, 120 mb/s with one thread and 60 mb/s with two threads. These are just cheap sata drives tho, I'll try and dig up a box with 3 fusion cards for something a little closer to the speeds you are seeing and see if that makes a difference. Thanks, Josef -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: One random read streaming is fast (~1200MB/s), but two or more are slower (~750MB/s)?
On Wed, Apr 17, 2013 at 07:50:09PM -0600, Matt Pursley wrote: Hey All, Here are the results of making and reading back a 13GB file on mdraid6 + ext4, mdraid6 + btrfs, and btrfsraid6 + btrfs. Seems to show that: 1) mdraid6 + ext4 can do ~1100 MB/s for these sequential reads with either one or two files at once. 2) btrfsraid6 + btrfs can do ~1100 MB/s for sequential reads with one file at a time, but only ~750 MB/s with two (or more). 3) mdraid6 + btrfs can only do ~750 MB/s for these sequential reads with either one or two files at once. So, seems like the speed drop is related more to the btrfs files system, then the experimental raid. Although it is interesting that btrfs can only do the full ~1100 MB/s with a single file on the btrfsraid6, but not mdraid6. Anyway, just some more info and reproducible results. I have also opened a ticket in bugzilla.kernel.org for this issue here... https://bugzilla.kernel.org/show_bug.cgi?id=56771 This is strange, and I can't see any reason why this would happen. I'll try and reproduce next week when I'm back from LSF. Thanks, Josef -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: One random read streaming is fast (~1200MB/s), but two or more are slower (~750MB/s)?
Matt Pursley wrote (ao): I have an LSI HBA card (LSI SAS 9207-8i) with 12 7200rpm SAS drives attached. When it's formated with mdraid6+ext4 I get about 1200MB/s for multiple streaming random reads with iozone. With btrfs in 3.9.0-rc4 I can also get about 1200MB/s, but only with one stream at a time. Just curious, is that btrfs on top of mdraid6, or is this experimental btrfs raid6 without md? -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: One random read streaming is fast (~1200MB/s), but two or more are slower (~750MB/s)?
On Tue, Apr 16, 2013 at 11:55 PM, Sander san...@humilis.net wrote: Matt Pursley wrote (ao): I have an LSI HBA card (LSI SAS 9207-8i) with 12 7200rpm SAS drives attached. When it's formated with mdraid6+ext4 I get about 1200MB/s for multiple streaming random reads with iozone. With btrfs in 3.9.0-rc4 I can also get about 1200MB/s, but only with one stream at a time. Just curious, is that btrfs on top of mdraid6, or is this experimental btrfs raid6 without md? This is the experimental btrfs raid6 without md. But, I did do a mdraid6 with btrfs test last night... and with that setup I only get the ~750MB/s result.. even with just one thread/stream... I will flip the system back to btrfsraid6+btrfs today to verify that I still get the full 1200MB/s with one stream/thread and ~750MB/s with two or more streams/threads with that setup... Thanks, Matt ___ mdraid6+btrfs_64GBRam_80files # sysctl vm.drop_caches=1 ; dd of=/dev/null if=/var/data/persist/testfile bs=640k vm.drop_caches = 1 2+0 records in 2+0 records out 1310720 bytes (13 GB) copied, 18.2109 s, 720 MB/s ___ -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
