Hi Nick, On Mar 7, 2014, at 12:21 PM, Nick <[email protected]> wrote:
> On Tue, Mar 4, 2014 at 4:32 PM, Keith Wesolowski > <[email protected]> wrote: > It's also been my experience that most applications don't actually open > files O_DSYNC and proceed to do tiny writes to them. I've observed > MySQL, for instance, doing lots of 87-byte writes to its log file, but > it opens it without O_DSYNC and instead issues fsync(3c) calls when > required for transactional consistency. That pattern allows the OS to > aggregate small I/O and also to do writes before they have to be > committed if the device is otherwise idle. This pattern usually results > in much better apparent performance than writing O_DSYNC. Again, > whether this is relevant depends on your application. > > > - The slowness of the Seagate 600 Pro SSD really surprised me. This is one > > of the fastest drives on the market, and you can see that with the cache > > flushing disabled it outpaces the Intel SSD. It is supposed to be an > > "Enterprise" drive with power protection, so I thought it could ignore the > > flush commands, but that is clearly not the case. When doing 4K sync > > writes, it seems to be somewhere around 50 times slower than the Intel > > drive and it is actually slower than a mechnical hard drive, which really > > shocked me. > > It's an interesting result, for sure. Ask Seagate. FWIW, I tested > their Pulsar.2 SAS drive for slog use and found it pretty good, very > similar in fact to the STEC Mach16 SLC that we use today. I don't think > that's the same device, though, and we never seriously considered either > for primary storage. > > If you want to get further into this, you can look at using DTrace to > break down the slowness by attributes of the SCSI packets being sent by > the HBA. That might yield some interesting data; i.e., is it a > particular size, a particular LBA range, a particular SCSI command, etc. > If it's always SYNCHRONIZE CACHE that's slow, does it matter what > commands were issued prior to that? > > > Thanks a lot Keith, I really appreciate your help. I've been benchmarking > recently with mysql and sysbench as that is closer to our actual workload. > I'm finding that, as you thought, with more threads going on, the Intel S3500 > SSD's performance isn't hurt too much by adding cache flushing. Disabling > cache flushing still speeds things up, but by less than 2x, so that seems > decent. The Seagate on the other hand... It so happens I've been looking at Seagate SSD performance today, too :-) > The performance hit on the Seagate 600 Pro is 3x to 10x depending on how many > threads is run. I ran dtrace as you suggested to look at the SCSI commands > and the SYNCHRONIZE CACHE commands are taking 10000us for the "fast" ones, > and I saw some get up around 500000us, so it would seem that this drive is > only capable of about 100 cache flushes per second, at most, which really is > limiting its performance. This compares to the Intel S3500 taking a little > over 100us for a SYNCHRONIZE CACHE command. I haven't done these measurements yet, but 100us is barely tolerable, more than 1ms becomes intolerable for slogs. -- richard > > One question I have for you or whoever can answer this, is: is there > something special about MySQL's innodb_flush_log_at_trx_commit with ZFS? In > the case of the Seagate drive, setting it to 2 obviously speeds up things > significantly as the flush is only performed once per second, but my > understanding is that you put the last second of transactions at risk by > using any setting but 1, and I want to make sure there's nothing in ZFS to > somehow mitigate this...I can't think of any reason that you could set it to > 2 and be ACID safe, but I ask because all the default configurations for > SmartOS's/Joyent's machines (Standard64, Percona, etc) have it > innodb_flush_log_at_trx_commit = 2 by default, and I would be surprised that > they would default to a not-safe setting. But perhaps I'm just not > understanding something. Thanks, > > Nick > > smartos-discuss | Archives | Modify Your Subscription -- [email protected] +1-760-896-4422 ------------------------------------------- smartos-discuss Archives: https://www.listbox.com/member/archive/184463/=now RSS Feed: https://www.listbox.com/member/archive/rss/184463/25769125-55cfbc00 Modify Your Subscription: https://www.listbox.com/member/?member_id=25769125&id_secret=25769125-7688e9fb Powered by Listbox: http://www.listbox.com
