It has been quite some time (about a year) since I did testing of
batch processing with my software (GraphicsMagick). In between time,
ZFS added write-throttling. I am using Solaris 10 with kernel
141415-03.
Quite a while back I complained that ZFS was periodically stalling the
writing process (which UFS did not do). The ZFS write-throttling
feature was supposed to avoid that. In my testing today I am still
seeing ZFS stall the writing process periodically. When the process
is stalled, there is a burst of disk activity, a burst of context
switching, and total CPU use drops to almost zero. Zpool iostat says
that read bandwidth is 15.8M and write bandwidth is 15.8M over a 60
second averaging interval. Since my drive array is good for writing
over 250MB/second, this is a very small write load and the array is
loafing.
My program uses the simple read->process->write approach. Each file
written (about 8MB/file) is written contiguously and written just
once. Data is read and written in 128K blocks. For this application
there is no value obtained by caching the file just written. From
what I am seeing, reading occurs as needed, but writes are being
batched up until the next ZFS synchronization cycle. During the ZFS
synchronization cycle it seems that processes are blocked from
writing. Since my system has a lot of memory and the ARC is capped at
10GB, quite a lot of data can be queued up to be written. The ARC is
currently running at its limit of 10GB.
If I tell my software to invoke fsync() before closing each written
file, then the stall goes away, but the program then needs to block so
there is less beneficial use of the CPU.
If this application stall annoys me, I am sure that it would really
annoy a user with mission-critical work which needs to get done on a
uniform basis.
If I run this little script then the application runs more smoothly
but I see evidence of many shorter stalls:
while true
do
sleep 3
sync
done
Is there a solution in the works for this problem?
Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
