Thanks for all the responses. I've been learning a lot about ZFS the past few weeks.
I ran some more benchmarks, and can answer some questions. So the way I am testing (for now) is that I have a single SSD as the main pool drive, under SmartOS. My goal is to have an all-SSD pool, without using an SLOG as I don't really have so much data that I need mechanical HDDs. I created a zone under SmartOS and installed iozone. The command line I am using is "iozone -s 1000m -r 4k -i 0 -i 1 -i 2 -o -O". I am only focusing on writes for the moment and ignoring reads. Compression is off for the zone. Thanks to Richard, I did some analysis using iosnoop and iopattern. To answer his question, these tools allowed me to see 8K random writes to the disk. I'm not sure why it writes 8K chunks to the disk at the moment, as iozone is writing 4K chunks. If anyone has an idea, I'd appreciate being told! It also writes 128K chunks periodically to the ZIL. Some benchmark results: Pool consisting of the Intel DC S3500 SSD: - iozone: write=3979 IOPS, rewrite=3903 IOPS, rand write=406 IOPS Same drive, disabled cache flushing: - iozone: write=9665 IOPS, rewrite=9289 IOPS, rand write=1691 IOPS Same drive, enabled cache flushing but disabled sync (strictly for testing, I know the dangers): - iozone: write=80897 IOPS, rewrite=73390 IOPS, rand write=7796 IOPS (of course this was almost all 128K sequential writes) Now, switching to the Seagate 600 Pro SSD instead of the Intel for the pool drive. Cache flushing enabled, sync standard: - Did not let complete. It is so slow, I estimated 22 hours to complete (Intel SSD took a few minutes). Showing less than 90 IOPS for "write" test. Seagate SSD, disabled cache flushing: - iozone: write=10631 IOPS, rewrite=10696 IOPS, rand write: 2364 IOPS Same drive, enabled cache flushing but disabled sync: - iozone: write=146784 IOPS, rewrite=69804 IOPS, rand write: 11414 IOPS Out of curiousity, I switched the pool drive to consist of a Toshiba 7200TB 2TB mechanical hard drive. Cache flushing enabled, sync standard: - Did not let complete. It is really slow, estimated 18 hours to complete. Showing less than 125 IOPS for "write" test. Same drive, disabled cache flushing: - iozone: write=4951 IOPS, rewrite=4653 IOPS, rand write=730 IOPS Same drive, enabled cache flushing but disabled sync: - iozone: write=48879 IOPS, rewrite=67295 IOPS, rand write=5679 IOPS I have more data if anyone is interested. So I'm noticing a few things: - The Intel drive writes 3-4 times slower with cache flushing enabled. I understand and appreciate the comments about not disabling cache flushing unless you have an intimate understanding of the hardware. So for me that is no. I would like to contact Intel and ask them about this, as I would think they could just NOOP the cache flush request but apparently they are not doing so. I'm afraid that I am too small of player to probably be able to talk to anyone there, but I can try. I was under the impression that this drive is one of the most popular new datacenter drives so I was hoping that someone who has purchased a large quantity has some more info. - Not surprising that sync disabled makes things really fast as it basically makes it so you don't need to actually write 4K blocks to disk but can group them together and write them more sequentially. - The slowness of the Seagate 600 Pro SSD really surprised me. This is one of the fastest drives on the market, and you can see that with the cache flushing disabled it outpaces the Intel SSD. It is supposed to be an "Enterprise" drive with power protection, so I thought it could ignore the flush commands, but that is clearly not the case. When doing 4K sync writes, it seems to be somewhere around 50 times slower than the Intel drive and it is actually slower than a mechnical hard drive, which really shocked me. These benchmarks are obviously all synthetic, but I wanted to get those done first so that I could better understand the drive's performance before moving on to more real-world, and therefore more complicated, benchmarks. Thanks, Nick ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
