Cross-posted to perf-discuss.
You can't change the write behavior of the app without
changing the app itself. The code would need to be modified
to issue fsync() calls on the file(s), or open the files for
synchronous writes (O_SYNC | O_DSYNC flags).
fsflush will run, by default, once per second, which is the
fastest rate it can be tuned for.
The actual size of the IO is driven by several factors. fsflush
is is writing 4kB or 8kB because the base page size on x86
is 4k, so fsflush is flushing 1 or 2 pages at a time.
You can use truss or dtrace to determine what size writes the
application is issuing;
#dtrace -n 'syscall::write:entry { @sizes = quantize(arg2); }'
I assume the application is issuing 32k writes, which would correspond
to what you observed with dtrace.
If you're looking to optimize write performance, and need to
ensure data is commited to disk, you could turn directio on
for the file system, which will bypass the page cache and
write straight to disk. This can really help, or it can really hurt,
depending on to what extent hitting in the page cache is
helping performance (like on the read side).
Getting to your specific issue, 6-8MB/sec does not sound like much,
but what if that's all the application is generating? In other words,
using a different load (dd), you observed much higher bandwidth
with a large, sequential write, so you know the underlying platform
is capable of delivering more bandwidth. At the application level,
the app is doing smaller, random writes.
What does the "iostat -zxnd 1" data look like? Are you getting
good latency? If the app is generating 32k IOs, and you're getting
6-8MB/sec (lets call it 7), that means your generating (7MB / 32k)
about 220 IOPS. Depending on what the underlying disk storage is,
this may be a very good number, but I need to know more about
the disk(s) the UFS is sitting on.
Thanks,
/jim
K Kelley wrote:
I'm troubleshooting an I/O performance problem with one of our applications that does a
lot of writing, generally blocks just over 32K, sequentially writing large files. It's a
Solaris 10 x86 system with UFS disk. We're often only seeing disk write throughput of
around 6-8MB/s, even when there is minimal read activity. Running iosnoop shows that
most of the physical writes are made by the actual app and average around 32KB. About
15% of the data, however, is done by fsflush and only 4 or 8KB at a time. The write
throughput for the fsflush writes is about 10% that of the app writes (using the
"DTIME" values and aggregating the results to get totals). CPU resources are
not a bottleneck.
If I turn off dopageflush the overall rate jumps to 18-20MB/s. However, this
would mean that the file data may not get flushed for a very long time, so is
not a suitable option for production environments.
If I do a dd (to create a file of 1GB, the amount of system memory), even with
a block size that matches our app, it does much larger block writes, often 1M,
the overall rate is around 60MB/s, and there were very few writes by fsflush.
Is there any way to cause more of the physical writes to be done by the app
rather than fsflush (and for that matter, what determines when fsflush does the
flushing rather than the app?)?
_______________________________________________
dtrace-discuss mailing list
dtrace-discuss@opensolaris.org
