On Jun 19, 2011, at 6:28 AM, Edward Ned Harvey wrote: >> From: Richard Elling [mailto:richard.ell...@gmail.com] >> Sent: Saturday, June 18, 2011 7:47 PM >> >> Actually, all of the data I've gathered recently shows that the number of >> IOPS does not significantly increase for HDDs running random workloads. >> However the response time does :-( > > Could you clarify what you mean by that?
Yes. I've been looking at what the value of zfs_vdev_max_pending should be. The old value was 35 (a guess, but a really bad guess) and the new value is 10 (another guess, but a better guess). I observe that data from a fast, modern HDD, for 1-10 threads (outstanding I/Os) the IOPS ranges from 309 to 333 IOPS. But as we add threads, the average response time increases from 2.3ms to 137ms. Since the whole idea is to get lower response time, and we know disks are not simple queues so there is no direct IOPS to response time relationship, maybe it is simply better to limit the number of outstanding I/Os. FWIW, I left disksort enabled (the default) > I was planning, in the near > future, to go run iozone on some system with, and without the disk cache > enabled according to format -e. If my hypothesis is right, it shouldn't > significantly affect the IOPS, which seems to be corroborated by your > message. iozone is a file system benchmark, won't tell you much about IOPS at the disk level. Be aware of all of the caching that goes on there. I used a simple vdbench test on the raw device: vary I/O size from 512 bytes to 128KB, vary threads from 1 to 10, full stroke, 4KB random, read and write. > > I was also planning to perform sequential throughput testing on two disks > simultaneously, with and without the disk cache enabled. If one disk is > actually able to hog the bus in an idle state, it should mean the total > combined throughput with cache disabled would be equal to a single disk. > (Which I highly doubt.) > > >> However the response time does [increase] :-( > > This comment seems to indicate that the drive queues up a whole bunch of > requests, and since the queue is large, each individual response time has > become large. It's not that physical actual performance has degraded with > the cache enabled, it's that the queue has become long. For async writes, > you don't really care how long the queue is, but if you have a mixture of > async writes and occasional sync writes... Then the queue gets long, and > when you sync, the sync operation will take a long time to complete. You > might actually benefit by disabling the disk cache. > > Richard, have I gotten the gist of what you're saying? I haven't formed an opinion yet, but I'm inclined towards wanting overall better latency. > > Incidentally, I have done extensive testing of enabling/disabling the HBA > writeback cache. I found that as long as you have a dedicated log device > for sync writes, your performance is significantly better by disabling the > HBA writeback. Something on order of 15% better. > Yes, I recall these tests. -- richard _______________________________________________ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
