Just wanted to start a thread to publish some performance numbers here with a new test setup that I have been playign with and get some input as the performance I am seeing is very puzzling.
I am copy pasting most of this from a thread on another forum since we are kind of at a dead end over there, hoping the people that work with this on a daily basis can shed some light! Config: Q6600 P5Q-E 8GB RAM AOC-USAS-L8i HP Expander 750Watt PC&P 14x 1TB 7200.12's OpenSolaris 2009.06 & latest SVN (134 i think) These results are on 2009.06 with the 14 drives configured in different ways using dd for some simple write tests to compare speeds. 14 drive Raid10 - 316MB/s Write 7 drive Raidz2 - 318MB/s Write 14 drive Raid60 (two 7 disk Raidz2) - 376MB/s Write 13 Drive Stripe - 382MB/s Write the 7 drive raidz2 writes look about right to me as it is close ot the same as the speeds i have seen on my other test box (330MB/s write to a 7 disk raidz2) though the scaling on the other tests (14 drive raid60, and the stripe) do not seem even remotly proper to me. Based on 50MB/s write per disk the stripe should be pulling ~700MB/s I must be hitting a bottleneck of a wall here somewere and would like to track it down. Watching the box access LEDS during a long write is interesting as well... the data access/write is in little patches, drive access for 2-3 seconds, 1 second no acces, 2-3 seconds of access, etc. Now one of the drives is accessing wile the others are not wich is odd.... iostat doesent report any odd writes on that drive either... wich is kinda odd... Filebench results Got around to doing some more indept testing using Filebench for single stream read/write of 10/20GB files (just after some raw numbers here to make sure the disks are performing properly) And I am a bit mistifyed at my results... Hoping someone has some info to lend, or gently prod me in the right direction... [CODE] 7 drive Raidz2 - 10Gb Read IO Summary: 34297 ops, 566.0 ops/s, (566/0 r/w) 565.9mb/s, 2256us cpu/op, 1.7ms latency 732: 620.620: Shutting down processes 20GB Write IO Summary: 15246 ops, 245.3 ops/s, (0/245 r/w) 245.3mb/s, 5445us cpu/op, 4.1ms latency 761: 104.731: Shutting down processes [/CODE] Added another 7 drive Raidz2 to the same pool for basically a Raid60. (Logic would dictate that I/O performance should double, since you are doubling the total spindle count) [CODE] 2x 7 drive Raid60 - 10GB Read IO Summary: 40600 ops, 670.0 ops/s, (670/0 r/w) 669.9mb/s, 3124us cpu/op, 1.5ms latency 20GB Write IO Summary: 19860 ops, 308.0 ops/s, (0/308 r/w) 307.9mb/s, 5709us cpu/op, 3.2ms latency [/CODE] IOStat showes ALL drives being written to, and read from. After seeing these results I blew the pool away and setup a 14 drive stripe, results as follow; [CODE] 14 drive strip 10GB read IO Summary: 48335 ops, 797.6 ops/s, (798/0 r/w) 797.5mb/s, 1239us cpu/op, 1.2ms latency 1176: 133.002: Shutting down processes 14 drive stripe 20GB write IO Summary: 26573 ops, 432.2 ops/s, (0/432 r/w) 432.2mb/s, 2988us cpu/op, 2.3ms latency 1183: 81.070: Shutting down processes [/CODE] The read speeds are correct since the HP Expander is hardware limited to 800MB/s so I will settle for 797. But the write speeds are retardedly low... I mean each of these drives are capable of ATLEAST 50MB/s (LOW estimate) so with 14 it should have maxed the expander again... but limped along at 432MB/s. I did some testing with disabling ZIL completly (DO NOT DO THIS ON A SERVER WITH DATA ON IT!!!!) and noticed no improvement... wich also confused me. With a huge 14 drive stripe maxing the expander @ 800MB/s for reads, and writes should be a no brainer.... I cant seem to figure out were the bottleneck is. I will be doing some more testing tommorow and monitoring CPU/Kernal usage to see if something is leeching CPU clocks... but other than that I am at a complete loss... If anyone is interested I did some more read/write testing with some other configurations (14 drive raid10, 4x3 raidz2, 3x4 raidz2, 2x6 raidz2, 1x12 raidz2, etc.) I can post the results if someone is interested... But they showed the same kinda numbers as the setups listed. I am kinda fryed troubleshooting to many issues this week already >.< Batch two of testing; Well I have some odd numbers again... what a surprise :mad: 12 disk strip with no expander, HBA and 4 onboard. [CODE] 10Gb Read IO Summary: 23308 ops, 384.6 ops/s, (385/0 r/w) 384.6mb/s, 1206us cpu/op, 2.6ms latency IO Summary: 30464 ops, 502.7 ops/s, (503/0 r/w) 502.7mb/s, 1236us cpu/op, 2.0ms latency IO Summary: 42361 ops, 699.0 ops/s, (699/0 r/w) 698.9mb/s, 1237us cpu/op, 1.4ms latency [/CODE] [CODE] 20Gb write IO Summary: 22536 ops, 367.4 ops/s, (0/367 r/w) 367.4mb/s, 2964us cpu/op, 2.7ms latency IO Summary: 23541 ops, 383.4 ops/s, (0/383 r/w) 383.4mb/s, 2933us cpu/op, 2.6ms latency IO Summary: 22517 ops, 368.2 ops/s, (0/368 r/w) 368.2mb/s, 2899us cpu/op, 2.7ms latency [/CODE] 8 disk stripe HBA Only [CODE] 10GB Read IO Summary: 38464 ops, 634.7 ops/s, (635/0 r/w) 634.7mb/s, 1262us cpu/op, 1.6ms latency IO Summary: 42610 ops, 703.1 ops/s, (703/0 r/w) 703.1mb/s, 1235us cpu/op, 1.4ms latency IO Summary: 44781 ops, 739.0 ops/s, (739/0 r/w) 738.9mb/s, 1226us cpu/op, 1.3ms latency [/CODE] [CODE] 20Gb write IO Summary: 20135 ops, 328.1 ops/s, (0/328 r/w) 328.1mb/s, 4060us cpu/op, 2.9ms latency IO Summary: 19452 ops, 318.6 ops/s, (0/319 r/w) 318.6mb/s, 4281us cpu/op, 3.0ms latency IO Summary: 19449 ops, 317.9 ops/s, (0/318 r/w) 317.9mb/s, 3751us cpu/op, 3.0ms latency [/CODE] The first read/wright were done one after the other. Then the second two reads (for both sets of tests) were done back to back after destroyign the test file. When running the 12 disk test with the 4 extra drives running off the onboard ports I noticed some very odd LED activity... the 4 drives that were onboard would stop accessing every few seconds for a couple seconds... than about a second after the HBA controlled drives would stop... Im going to assume the onboard controller just sucks. Looking at the 8 drives off just the HBA the read speeds are acceptable... ~80-90MB/s per drive.... wich is the numbers I was expecting. The write speeds on the other hand have me baffled :( I think im going to post a copy of this thread over on the opensolaris forums because I must be missing/forgetting something major here :confused: Sorry if some of this is lacking flow like I said its mostly copy paste but all the critical information is in here. I will be connecting all the drives back up to the expander and testing under Server 2008 R2 tonight to make sure its not a hardware issue, but im pretty sure the issue lies in Opensolaris somewere. Thanks in advance. -- This message posted from opensolaris.org _______________________________________________ storage-discuss mailing list storage-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/storage-discuss
