Hi all,
In charge of ticket 1888 (see at
https://svn.open-mpi.org/trac/ompi/ticket/1888) ,
I have put the resulting code in bitbucket at:
http://bitbucket.org/devezep/new-romio-for-openmpi/
The work in this repo consisted in refreshing ROMIO to a newer
version: the one from the very last MPICH2 release (mpich2-1.3b1).
Testing:
1. runs fine except one minor error (see the explanation below) on
various FS.
2. runs fine with Lustre, but:
. had to add a small patch in romio/adio/ad_lustre_open.c
3. see below how to efficiently run with Lustre.
You are invited to test and send comments
Enjoy !
Pascal
======== The minor error ===================
The test error.c fails because OpenMPI does not handle correctly the
"two level" error functions of ROMIO:
error_code = MPIO_Err_create_code(MPI_SUCCESS, MPIR_ERR_RECOVERABLE,
myname, __LINE__, MPI_ERR_ARG,
"**iobaddisp", 0);
OpenMPI limits its view to MPI_ERR_ARG, but the real error is
"**iobaddisp".
========= How to test performances with Lustre ===================
1) Compile with Lustre ADIO driver. For this, add the flag
--with-io-romio-flags="--with-file-system=ufs+nfs+lustre" to
your configure command.
2) Of course, you should have a Lustre file system mounted on all the
nodes you will run on.
3) Take an application like coll_perf.c (in the test directory). In this
application, change the
three dimensions to 1000, that will create a file of 4 GB (big
files are required in order
to reach good performances with Lustre).
4) Put the highest possible striping_factor in the hint. For this, one
solution is :
- If your Lustre file system have 16 OST, create a hint file with the
following line:
striping_factor 16
- Export the path to this file in the variable ROMIO_HINTS:
export ROMIO_HINTS=my_directory/my_hints
If you do not specify the striping_factor, Lustre will set the default
value (often 2 only).
You can verify the striping_factor set by Lustre with the following
command:
lfs getstripe <file> (look at the value of lmm_stripe_count)
Note: The striping_factor is set once at file creation and cannot be
changed after.
5) Run your test, specifying a file located in the Lustre file system.
