G'day, all, So, I've decided to migrate my home server from Linux+swRAID+LVM to Solaris+ZFS, because it seems to hold much better promise for data integrity, which is my primary concern.
However, naturally, I decided to do some benchmarks in the process, and I don't understand why the results are what they are. I though I had a reasonable understanding of ZFS, but now I'm not so sure. I've used bonnie++ and a variety of Linux RAID configs below to approximate equivalent ZFS configurations and compare. I do realise they're not exactly the same thing, but it seems to me they're reasonable comparisons and should return at least somewhat similar performance. I also realise bonnie++ is not an especially comprehensive or complex benchmark, but ultimately I don't really care about performance and this was only done out of curiosity. The executive summary is that ZFS write performance appears to be relatively awful (all the time), and it's read performance is relatively good most of the time (with "striping", "mirroring" and raidz[2]'s with fewer numbers of disks). Examples: * 8-disk RAID0 on Linux returns about 190MB/s write and 245MB/sec read, while a ZFS raidz using the same disks returns about 120MB/sec write, but 420MB/sec read. * 16-disk RAID10 on Linux returns 165MB/sec and 440MB/sec write and read, while a ZFS pool with 8 mirrored disks returns 140MB/sec write and 410MB/sec read. * 16-disk RAID6 on Linux returns 126MB/sec write, 162MB/sec read, while a 16-disk raidz2 returns 80MB/sec write and 142MB/sec read. The biggest problem I am having understanding "why is it so", is because I was under the impression with ZFS's CoW, etc, that writing (*especially* writes like this, to a raidz array) should be much faster than a regular old-fashioned RAID6 array. I certainly can't complain about the read speed, however - 400-odd MB/sec out of this old beastie is pretty impressive :). Help ? Have I missed something obvious or done something silly ? (Additionally, from the Linux perspective, why are reads so slow ?) PS: Hardware specs are Dual 2.8Ghz Xeon, 533Mhz bus, 2G RAM. Two SuperMicro 8-port SATA controllers - AOC-SAT2-MV8 - on separate PCI-X buses (Marvell 88SX6081 based, http://www.supermicro.com/products/accessories/addon/AoC-SAT2-MV8.cfm) 16 Seagate 250G drives, 8M cache, ST3250820AS Motherboard is a pull from an old Gateway dual Xeon server. PPS: Benchmark data (this will look awful with a proportional font): (With the Linux "RAID50" and "RAID60", I created two RAID5 and RAID6 arrays, respectively, then create a RAID0 array over the top of them. For "RAID10" I used the 'builtin' RAID10.) Single drive bonnie++ -d /mnt/ -s 8g -u0 -f Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP csmith-desktop 8G 42807 21 25278 7 65191 6 118.4 0 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ csmith-desktop,8G,,,42807,21,25278,7,,,65191,6,118.4,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++ zpool create internal c0t0d0 ./bonnie++ -d /internal/ -s 8g -u0 -f Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP nitrogen 8G 46916 18 27814 11 62103 8 212.5 0 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 15522 99 +++++ +++ 24489 99 10140 99 +++++ +++ 5196 99 nitrogen,8G,,,46916,18,27814,11,,,62103,8,212.5,0,16,15522,99,+++++,+++,24489,99,10140,99,+++++,+++,5196,99 8 disk RAID0, one controller, 128k stripe Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP csmith-desktop 8G 190853 95 119364 42 245770 28 421.4 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ csmith-desktop,8G,,,190853,95,119364,42,,,245770,28,421.4,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++ zpool create internal c0t0d0 zpool add internal c0t1d0 zpool add internal c0t2d0 zpool add internal c0t3d0 zpool add internal c0t4d0 zpool add internal c0t5d0 zpool add internal c0t6d0 zpool add internal c0t7d0 ./bonnie++ -d /internal/ -s 8g -u0 -f Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP nitrogen 8G 121711 47 100927 43 424900 75 617.4 3 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 14719 99 +++++ +++ 24421 99 14013 99 +++++ +++ 25612 99 nitrogen,8G,,,121711,47,100927,43,,,424900,75,617.4,3,16,14719,99,+++++,+++,24421,99,14013,99,+++++,+++,25612,99 16 disk RAID0, two controllers, 128k stripe Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP csmith-desktop 8G 189351 95 167948 65 442343 52 461.2 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ csmith-desktop,8G,,,189351,95,167948,65,,,442343,52,461.2,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++ zpool create internal c0t0d0 zpool add internal c0t1d0 zpool add internal c0t2d0 zpool add internal c0t3d0 zpool add internal c0t4d0 zpool add internal c0t5d0 zpool add internal c0t6d0 zpool add internal c0t7d0 zpool add internal c1t0d0 zpool add internal c1t1d0 zpool add internal c1t2d0 zpool add internal c1t3d0 zpool add internal c1t4d0 zpool add internal c1t5d0 zpool add internal c1t6d0 zpool add internal c1t7d0 ./bonnie++ -d /internal/ -s 8g -u0 -f Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP nitrogen 8G 130142 50 133316 57 412643 68 681.2 3 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 15778 99 +++++ +++ 24652 99 5835 99 +++++ +++ 22881 99 nitrogen,8G,,,130142,50,133316,57,,,412643,68,681.2,3,16,15778,99,+++++,+++,24652,99,5835,99,+++++,+++,22881,99 16 disk RAID10, two controllers, 128k stripe Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP csmith-desktop 8G 165585 93 107365 45 268066 34 420.7 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ csmith-desktop,8G,,,165585,93,107365,45,,,268066,34,420.7,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++ zpool create internal mirror c0t0d0 c1t0d0 zpool add internal mirror c0t1d0 c1t1d0 zpool add internal mirror c0t2d0 c1t2d0 zpool add internal mirror c0t3d0 c1t3d0 zpool add internal mirror c0t4d0 c1t4d0 zpool add internal mirror c0t5d0 c1t5d0 zpool add internal mirror c0t6d0 c1t6d0 zpool add internal mirror c0t7d0 c1t7d0 ./bonnie++ -d /internal/ -s 8g -u0 -f Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP nitrogen 8G 139750 54 120402 52 415646 73 702.6 3 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 14652 99 +++++ +++ 24324 99 14459 99 +++++ +++ 24347 99 nitrogen,8G,,,139750,54,120402,52,,,415646,73,702.6,3,16,14652,99,+++++,+++,24324,99,14459,99,+++++,+++,24347,99 8 disk RAID5, one controller, 128k stripe Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP csmith-desktop 8G 107288 67 54156 38 157472 47 299.2 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ csmith-desktop,8G,,,107288,67,54156,38,,,157472,47,299.2,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++ zpool create internal raidz1 c0t0d0 c0t1d0 c0t2d0 c0t3d0 c0t4d0 c0t5d0 c0t6d0 c0t7d0 ./bonnie++ -d /internal/ -s 8g -u0 -f Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP nitrogen 8G 80800 31 57461 28 216634 43 188.6 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 15646 99 +++++ +++ 18737 99 13388 99 +++++ +++ 25170 99 nitrogen,8G,,,80800,31,57461,28,,,216634,43,188.6,1,16,15646,99,+++++,+++,18737,99,13388,99,+++++,+++,25170,99 16 disk RAID50, two controllers, 128k stripe Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP csmith-desktop 8G 134082 89 52457 32 105118 30 456.3 2 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ csmith-desktop,8G,,,134082,89,52457,32,,,105118,30,456.3,2,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++ zpool add internal raidz1 c1t0d0 c1t1d0 c1t2d0 c1t3d0 c1t4d0 c1t5d0 c1t6d0 c1t7d0 ./bonnie++ -d /internal/ -s 8g -u0 -f Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP nitrogen 8G 82959 32 51422 33 193962 62 233.8 4 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 16458 99 +++++ +++ 16412 99 13135 99 +++++ +++ 11576 99 nitrogen,8G,,,82959,32,51422,33,,,193962,62,233.8,4,16,16458,99,+++++,+++,16412,99,13135,99,+++++,+++,11576,99 8 disk RAID6, one controller, 128k stripe Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP csmith-desktop 8G 96770 61 51418 34 148072 44 305.9 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ csmith-desktop,8G,,,96770,61,51418,34,,,148072,44,305.9,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++ zpool create internal raidz2 c0t0d0 c0t1d0 c0t2d0 c0t3d0 c0t4d0 c0t5d0 c0t6d0 c0t7d0 ./bonnie++ -d /internal/ -s 8g -u0 -f Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP nitrogen 8G 89015 34 58932 28 188166 36 178.3 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 15475 99 +++++ +++ 22184 99 9085 99 +++++ +++ 24124 99 nitrogen,8G,,,89015,34,58932,28,,,188166,36,178.3,1,16,15475,99,+++++,+++,22184,99,9085,99,+++++,+++,24124,99 16 disk RAID60, two controllers, 128k stripe Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP csmith-desktop 8G 131519 88 51273 31 102324 30 419.2 2 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ csmith-desktop,8G,,,131519,88,51273,31,,,102324,30,419.2,2,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++ zpool add internal raidz2 c1t0d0 c1t1d0 c1t2d0 c1t3d0 c1t4d0 c1t5d0 c1t6d0 c1t7d0 ./bonnie++ -d /internal/ -s 8g -u0 -f Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP nitrogen 8G 86688 33 46889 33 170449 58 254.2 5 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 16009 99 +++++ +++ 24150 99 12990 99 +++++ +++ 22813 99 nitrogen,8G,,,86688,33,46889,33,,,170449,58,254.2,5,16,16009,99,+++++,+++,24150,99,12990,99,+++++,+++,22813,99 16 disk RAID5, two controllers, 128k stripe Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP csmith-desktop 8G 135052 89 71118 56 167523 69 412.2 2 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ csmith-desktop,8G,,,135052,89,71118,56,,,167523,69,412.2,2,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++ zpool create internal raidz1 c0t0d0 c0t1d0 c0t2d0 c0t3d0 c0t4d0 c0t5d0 c0t6d0 c0t7d0 c1t0d0 c1t1d0 c1t2d0 c1t3d0 c1t4d0 c1t5d0 c1t6d0 c1t7d0 ./bonnie++ -d /internal/ -s 8g -u0 -f Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP nitrogen 8G 90557 34 44982 37 147654 69 209.4 8 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 15631 99 +++++ +++ 21228 99 10132 99 +++++ +++ 24391 99 nitrogen,8G,,,90557,34,44982,37,,,147654,69,209.4,8,16,15631,99,+++++,+++,21228,99,10132,99,+++++,+++,24391,99 16 disk RAID6, two controllers, 128k stripe Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP csmith-desktop 8G 126421 84 68968 55 162105 65 360.0 1 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ csmith-desktop,8G,,,126421,84,68968,55,,,162105,65,360.0,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++ zpool create internal raidz2 c0t0d0 c0t1d0 c0t2d0 c0t3d0 c0t4d0 c0t5d0 c0t6d0 c0t7d0 c1t0d0 c1t1d0 c1t2d0 c1t3d0 c1t4d0 c1t5d0 c1t6d0 c1t7d0 ./bonnie++ -d /internal/ -s 8g -u0 -f Version 1.03 ------Sequential Output------ --Sequential Input- --Random- -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP nitrogen 8G 80457 31 38875 35 142049 64 193.1 7 ------Sequential Create------ --------Random Create-------- -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 14426 99 +++++ +++ 21855 99 10068 99 +++++ +++ 24235 99 nitrogen,8G,,,80457,31,38875,35,,,142049,64,193.1,7,16,14426,99,+++++,+++,21855,99,10068,99,+++++,+++,24235,99 This message posted from opensolaris.org _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
