Re: [zfs-discuss] ZFS configuration for a thumper
On Feb 4, 2008, at 5:10 PM, Marion Hakanson wrote: [EMAIL PROTECTED] said: FYI, you can use the '-c' option to compare results from various runs and have one single report to look at. That's a handy feature. I've added a couple of such comparisons: http://acc.ohsu.edu/~hakansom/thumper_bench.html Marion Your finding for random reads with or without NCQ match my findings: http://blogs.sun.com/erickustarz/entry/ncq_performance_analysis Disabling NCQ looks like a very tiny win for the multi-stream read case. I found a much bigger win, but i was doing RAID-0 instead of RAID-Z. eric ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS configuration for a thumper
[EMAIL PROTECTED] said: Your finding for random reads with or without NCQ match my findings: http:// blogs.sun.com/erickustarz/entry/ncq_performance_analysis Disabling NCQ looks like a very tiny win for the multi-stream read case. I found a much bigger win, but i was doing RAID-0 instead of RAID-Z. I didn't set out to do the with/without NCQ comparisons. Rather, my first runs of filebench and bonnie++ triggered a number of I/O errors and controller timeout/resets on several different drives, so I disabled NCQ based on bug 6587133's workaround suggestion. No more errors during subsequent testing, so we're running with NCQ disabled until a patch comes along. It was useful, however, to see what effect disabling NCQ had. I find filebench easier to use than bonnie++, mostly because filebench is automatically multithreaded, which is necessary to generate a heavy enough workload to exercise anything more than a few drives (esp. on machines like T2000's). The HTML output doesn't hurt, either. Regards, Marion ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS configuration for a thumper
There are *lots* of options for configuring a Thumper, what you choose really depends on the kind of performance you want. I found these sites incredibly helpful in working out what was best for us: http://blogs.sun.com/relling/entry/zfs_raid_recommendations_space_performance http://lindsay.at/blog/archive/2007/04/15/zfs-performance-models-for-a-streaming-server.html One point to remember is that read write performance are two different things, and are affected in different ways by the disk layout. You also have to consider how many hot spares you want, and whether you want to use a single or dual parity scheme. Based on the posts above I put together a table with estimated read write performance for several dual parity configurations. Bear in mind these are only estimated figures based on what I learnt in the threads above, but I believe they give a reasonable estimate of physical disk performance. Also bear in mind that while you gain performance by adding Raid sets, you also loose more disks and hence capacity. My full table shows hot spares, drives lost to raid parity and total capacity available, but I'm not going to try to replicate all that here. RAID-Z2 Disks per set.Sets.Write Performance.Read Performance.Read IOPS 46144175 232422150 143363225 114364300 9.5355375 7.6306450 6.7287525 *probably optimum with 6 controllers. 5.9279675 Dual Parity Mirrored 1531545...3375 *Our preferred configuration The dual parity mirrored set is very tempting for us, you loose a lot of space (only 7.5TB available instead of 22.5TB), but the Thumper is big cheap enough to get away with that, and for us, small random reads are likely to be what we need. However, all this is theoretical and we plan to test the extreme cases and our favoured configurations with real world data before we decide on a final layout. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS configuration for a thumper
Gaaah, no idea what happened to that. It looked ok in preview, but it seems the message board is adding odd characters to my text. Trying again: RAID-Z2 Disks per setSetsWrite PerformanceRead PerformanceRead IOPS 46...1...44...1...75 23...2...42...2...150 14...3...36...3...225 11...4...36...4...300 95...35...5...375 76...30...6...450 67...28...7...525 59...27...9...675 Dual Parity Mirrored 15...3...15...45..3375 This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS configuration for a thumper
[EMAIL PROTECTED] said: FYI, you can use the '-c' option to compare results from various runs and have one single report to look at. That's a handy feature. I've added a couple of such comparisons: http://acc.ohsu.edu/~hakansom/thumper_bench.html Marion ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS configuration for a thumper
On Feb 1, 2008, at 11:17 AM, Marion Hakanson wrote: [EMAIL PROTECTED] said: Depending on needs for space vs. performance, I'd probably pixk eithr 5*9 or 9*5, with 1 hot spare. [EMAIL PROTECTED] said: How you can check the speed (I'm totally newbie on Solaris) We're deploying a new Thumper w/750GB drives, and did space vs performance tests comparing raidz2 4*11 (2 spares, 24TB) with 7*6 (4 spares, 19TB). Here are our bonnie++ and filebench results: http://acc.ohsu.edu/~hakansom/thumper_bench.html Very cool that we're starting to see more filebench results. FYI, you can use the '-c' option to compare results from various runs and have one single report to look at. eric ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS configuration for a thumper
[EMAIL PROTECTED] said: Depending on needs for space vs. performance, I'd probably pixk eithr 5*9 or 9*5, with 1 hot spare. [EMAIL PROTECTED] said: How you can check the speed (I'm totally newbie on Solaris) We're deploying a new Thumper w/750GB drives, and did space vs performance tests comparing raidz2 4*11 (2 spares, 24TB) with 7*6 (4 spares, 19TB). Here are our bonnie++ and filebench results: http://acc.ohsu.edu/~hakansom/thumper_bench.html Regards, Marion ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS configuration for a thumper
Le 01/02/2008 à 11:17:14-0800, Marion Hakanson a écrit [EMAIL PROTECTED] said: Depending on needs for space vs. performance, I'd probably pixk eithr 5*9 or 9*5, with 1 hot spare. [EMAIL PROTECTED] said: How you can check the speed (I'm totally newbie on Solaris) We're deploying a new Thumper w/750GB drives, and did space vs performance tests comparing raidz2 4*11 (2 spares, 24TB) with 7*6 (4 spares, 19TB). Here are our bonnie++ and filebench results: http://acc.ohsu.edu/~hakansom/thumper_bench.html Lots of thanks for making this work. And let me to read it. Regards. -- Albert SHIH Observatoire de Paris Meudon SIO batiment 15 Heure local/Local time: Ven 1 fév 2008 23:03:59 CET ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] ZFS configuration for a thumper
Hi all I've a Sun X4500 with 48 disk of 750Go The server come with Solaris install on two disk. That's mean I've got 46 disk for ZFS. When I look the defautl configuration of the zpool zpool create -f zpool1 raidz c0t0d0 c1t0d0 c4t0d0 c6t0d0 c7t0d0 zpool add -f zpool1 raidz c0t1d0 c1t1d0 c4t1d0 c5t1d0 c6t1d0 c7t1d0 zpool add -f zpool1 raidz c0t2d0 c1t2d0 c4t2d0 c5t2d0 c6t2d0 c7t2d0 zpool add -f zpool1 raidz c0t3d0 c1t3d0 c4t3d0 c5t3d0 c6t3d0 c7t3d0 zpool add -f zpool1 raidz c0t4d0 c1t4d0 c4t4d0 c6t4d0 c7t4d0 zpool add -f zpool1 raidz c0t5d0 c1t5d0 c4t5d0 c5t5d0 c6t5d0 c7t5d0 zpool add -f zpool1 raidz c0t6d0 c1t6d0 c4t6d0 c5t6d0 c6t6d0 c7t6d0 zpool add -f zpool1 raidz c0t7d0 c1t7d0 c4t7d0 c5t7d0 c6t7d0 c7t7d0 that's mean there'are pool with 5 disk and other with 6 disk. When I want to do the same I've got this message : mismatched replication level: pool uses 5-way raidz and new vdev uses 6-way raidz I can force this with «-f» option. But what's that mean (sorry if the question is stupid). What's kind of pool you use with 46 disk ? (46=2*23 and 23 is prime number that's mean I can make raidz with 6 or 7 or any number of disk). Regards. -- Albert SHIH Observatoire de Paris Meudon SIO batiment 15 Heure local/Local time: Mer 30 jan 2008 16:36:49 CET ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS configuration for a thumper
Le 30/01/2008 à 11:01:35-0500, Kyle McDonald a écrit Albert Shih wrote: What's kind of pool you use with 46 disk ? (46=2*23 and 23 is prime number that's mean I can make raidz with 6 or 7 or any number of disk). Depending on needs for space vs. performance, I'd probably pixk eithr 5*9 or 9*5, with 1 hot spare. Thanks for the tips... How you can check the speed (I'm totally newbie on Solaris) I've use mkfile 10g for write and I've got same perf with 5*9 or 9*5. Have you some advice about tool like iozone ? Regards. -- Albert SHIH Observatoire de Paris Meudon SIO batiment 15 Heure local/Local time: Mer 30 jan 2008 17:10:55 CET ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS configuration for a thumper
On 1/30/08, Albert Shih [EMAIL PROTECTED] wrote: Thanks for the tips... How you can check the speed (I'm totally newbie on Solaris) I've use mkfile 10g for write and I've got same perf with 5*9 or 9*5. Have you some advice about tool like iozone ? Regards. -- Albert SHIH Observatoire de Paris Meudon SIO batiment 15 Heure local/Local time: Mer 30 jan 2008 17:10:55 CET ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss I'd take a look at bonnie++ http://www.sunfreeware.com/programlistintel10.html#bonnie++ --Tim ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS configuration for a thumper
Albert Shih wrote: What's kind of pool you use with 46 disk ? (46=2*23 and 23 is prime number that's mean I can make raidz with 6 or 7 or any number of disk). Depending on needs for space vs. performance, I'd probably pixk eithr 5*9 or 9*5, with 1 hot spare. -Kyle Regards. -- Albert SHIH Observatoire de Paris Meudon SIO batiment 15 Heure local/Local time: Mer 30 jan 2008 16:36:49 CET ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] ZFS configuration for a thumper
[EMAIL PROTECTED] said: I'd take a look at bonnie++ http://www.sunfreeware.com/programlistintel10.html#bonnie++ Also filebench: http://www.solarisinternals.com/wiki/index.php/FileBench You'll see the most difference between 5x9 and 9x5 in small random reads: http://blogs.sun.com/relling/entry/zfs_raid_recommendations_space_performance http://blogs.sun.com/relling/entry/raid_recommendations_space_vs_mttdl http://lindsay.at/blog/archive/2007/04/15/zfs-performance-models-for-a-streamin g-server.html Regards, Marion ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
