Hi Phil, First of all, great work! There are 2 other parameters that I thought could also make an impact a) Choice of file-system (this was investigated by Nathan last year, I think) as well as choice of journaling modes. b) In case of storage-spaces created directly on top of an IDE/SCSI disks or attached to local RAID controllers, there must be a way to enable hd parameters like write-caching/tcq at the disks (hdparm -W /dev/?,...), are they may be on by default?) (although that might conflict with the goal of stability of data on disks)
It is interesting that time is not as important a criterion as VM ratio (i.e. dirty_writeback_centisecs/dirty_expire_centisecs) for such write-intensive workloads.. > A. Is the AIO interface causing delays? > B. Is the linux kernel waiting too long to start writing out its > buffer cache? > C. Is the linux kernel disk scheduler appropriate for PVFS2? > > We can change this behavior by adjusting the /proc/sys/vm/dirty* files. > They are documented in the Documentation/filesystems/proc.txt file in > the linux kernel source. The only one that really ended up being > interesting for us (after trial and error) is the dirty_ratio file. The > explanation given in the documentation is: "Contains, as a percentage of > total system memory, the number of pages at which a process which is > generating disk writes will itself start writing out dirty data.". It > defaults to 40, but some of the results below show what happens when it > is set to 1. There is also a dirty_background_ratio file, which > controls when pdflush decides to write out data in the background. That > would seem to be the more desirable tweak, but it didn't have the effect > that dirty_ratio did for some reason. Could this because the pdflush daemon does not wake up regularly enough to start flushing things out? Or does pdflush wake up a) either if timeout passes (or) b) ratio is reached? Hey, one other thing which struck me. How much memory was there on this machine? Is this on an IA 32 machine running a kernel with CONFIG_HIGHMEM? Thanks! Murali _______________________________________________ Pvfs2-developers mailing list [email protected] http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
