We recently ran some tests trying different sync settings in PVFS2. We
ran into one pleasant surprise, although probably it is already obvious
to others. Here is the setup:
12 clients
4 servers
read/write test application, 100 MB operations, large files
fibre channel SAN storage
The test application is essentially the same as was used in the posting
regarding kernel buffer sizes, although with different parameters in
this environment.
At any rate, to get to the point:
with TroveSyncData=no (default settings): 173 MB/s
with TroveSyncData=yes: 194 MB/s
I think the issue is that if syncdata is turned off, then the buffer
cache tends to get very full before it starts writing. This bursty
behavior isn't doing the SAN any favors- it has a big cache on the back
end and probably performs better with sustained writes that don't put so
much sudden peak traffic on the HBA card.
There are probably more sophisticated variations of this kind of tuning
around (/proc vm settings, using direct io, etc.) but this is an easy
config file change to get an extra 12% throughput.
This setting is a little more unpredictable for local scsi disks- some
combinations of application and node go faster but some go slower.
Overall it seems better for our environment to just leave data syncing
on for both SAN and local disk, but your mileage may vary.
This is different from results that we have seen in the past (maybe a
year ago or so) for local disk- it used to be a big penalty to sync
every data operation. I'm not sure what exactly happened to change this
(new dbpf design? alt-aio? better kernels?) but I'm not complaining :)
-Phil
_______________________________________________
Pvfs2-developers mailing list
Pvfs2-developers@beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
