On Wed, Sep 29, 2010 at 05:24:13PM +0100, Robert Horton wrote: > 1) Does anyone have any hints for improving the nfs performance under > these circumstances? I've tried using jumbo frames, different > filesystems, having the log device on an SSD and increasing the nfs > block size to 1MB, none of which have any significant effect.
The challenge with MPI-IO and NFS is achieving correct performance. NFS consistency semantics make it quite difficult, so we turn off the attribute cache. the MPI-IO library also locks around every I/O operation in an attempt to flush the client cache. Even those steps do not always work. > 2) Are there any reasonable alternatives to nfs in this situation? The > main possibilities seem to be: > > - PVFS or similar with a single IO server. Not sure what performance I > should expect from this though, and it's a lot more complex than nfs. You should expect pretty solid performance: First, your NFS server can go back to enabling caches for your other workloads Second, the MPI-IO library is pretty well tuned to PVFS. No extraneous locks. Third, if you decide one day you need a bit more performance, set up a PVFS volume with more servers. Presto-chango, clients will go faster without any changes. Fourth, is it really that much more complex? I'm myopic on this point, having worked with PVFS for a decade, but it's 90% userspace with a small kernel module. There's also a bunch of helpful people on the mailing lists, which would be where we should take any further PVFS discussions. > - Sharing a block device via iSCSI and using GFS, although this is also > going to be somewhat complex and I can't find any evidence that MPI-IO > will even work with GFS. I haven't used GFS in a decade. Back then, it only supported entire-file locks, making parallel access difficult. GFS has had more fine-grained locking, so it might work well. If you thought standing up PVFS was complicated, wait until you check out GFS (set up quorum, set up lock manager, set up kernel modules, etc). ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA _______________________________________________ Beowulf mailing list, [email protected] sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
