On Tue, Oct 26, 2010 at 05:39:09PM -0600, Dave Wade-Stein wrote: > As to MPI, we're both using openmpi 1.4.1. > > We're both using NFS file systems which are formatted as xfs. As I mentioned, > we had problems with ext3 filesystems, which were alleviated when we > reformatted as xfs. Unfortunately, that didn't work for the customer.
As the guy responsible for the MPI-IO library underneath HDF5, I can tell you that NFS is an awful awful choice for parallel I/O. The MPI-IO library will make a best effort to ensure correct behavior, but NFS consistency semantics are such that you really cannot guarantee correct behavior. the MPI-IO library (ROMIO) wraps each i/o operation in an "fcntl lock" in an effort to ensure that client-side data gets flushed. Those fcntl locks are advisory, but even so some times servicing those lock calls can take an inordinately long time. As a disclaimer, I'm closely affiliated with the PVFS project I'd suggest setting up PVFS: - it is both no-cost and open source, - it's fairly straightforward to build, install and configure - it requires only a small kernel module (and in fact you don't strictly need that for MPI-IO). - the MPI-IO library contains PVFS-specific optimizations. You could run a one-server PVFS instance on your NFS server. ==rob -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA _______________________________________________ Hdf-forum is for HDF software users discussion. [email protected] http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org
