On Tue, Mar 4, 2014 at 4:32 PM, Keith Wesolowski < [email protected]> wrote: > > It's also been my experience that most applications don't actually open > files O_DSYNC and proceed to do tiny writes to them. I've observed > MySQL, for instance, doing lots of 87-byte writes to its log file, but > it opens it without O_DSYNC and instead issues fsync(3c) calls when > required for transactional consistency. That pattern allows the OS to > aggregate small I/O and also to do writes before they have to be > committed if the device is otherwise idle. This pattern usually results > in much better apparent performance than writing O_DSYNC. Again, > whether this is relevant depends on your application. > > > - The slowness of the Seagate 600 Pro SSD really surprised me. This is > one > > of the fastest drives on the market, and you can see that with the cache > > flushing disabled it outpaces the Intel SSD. It is supposed to be an > > "Enterprise" drive with power protection, so I thought it could ignore > the > > flush commands, but that is clearly not the case. When doing 4K sync > > writes, it seems to be somewhere around 50 times slower than the Intel > > drive and it is actually slower than a mechnical hard drive, which really > > shocked me. > > It's an interesting result, for sure. Ask Seagate. FWIW, I tested > their Pulsar.2 SAS drive for slog use and found it pretty good, very > similar in fact to the STEC Mach16 SLC that we use today. I don't think > that's the same device, though, and we never seriously considered either > for primary storage. > > If you want to get further into this, you can look at using DTrace to > break down the slowness by attributes of the SCSI packets being sent by > the HBA. That might yield some interesting data; i.e., is it a > particular size, a particular LBA range, a particular SCSI command, etc. > If it's always SYNCHRONIZE CACHE that's slow, does it matter what > commands were issued prior to that? > > Thanks a lot Keith, I really appreciate your help. I've been benchmarking recently with mysql and sysbench as that is closer to our actual workload. I'm finding that, as you thought, with more threads going on, the Intel S3500 SSD's performance isn't hurt too much by adding cache flushing. Disabling cache flushing still speeds things up, but by less than 2x, so that seems decent. The Seagate on the other hand...
The performance hit on the Seagate 600 Pro is 3x to 10x depending on how many threads is run. I ran dtrace as you suggested to look at the SCSI commands and the SYNCHRONIZE CACHE commands are taking 10000us for the "fast" ones, and I saw some get up around 500000us, so it would seem that this drive is only capable of about 100 cache flushes per second, at most, which really is limiting its performance. This compares to the Intel S3500 taking a little over 100us for a SYNCHRONIZE CACHE command. One question I have for you or whoever can answer this, is: is there something special about MySQL's innodb_flush_log_at_trx_commit with ZFS? In the case of the Seagate drive, setting it to 2 obviously speeds up things significantly as the flush is only performed once per second, but my understanding is that you put the last second of transactions at risk by using any setting but 1, and I want to make sure there's nothing in ZFS to somehow mitigate this...I can't think of any reason that you could set it to 2 and be ACID safe, but I ask because all the default configurations for SmartOS's/Joyent's machines (Standard64, Percona, etc) have it innodb_flush_log_at_trx_commit = 2 by default, and I would be surprised that they would default to a not-safe setting. But perhaps I'm just not understanding something. Thanks, Nick ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
