the reason for potential performance issues on NFS are very different from
Lustre. Basically, depending on your use-case and the NFS configuration, you
have to enforce different locking policy to ensure correct output files. The
default value for chosen for ompio is the most conservative
Hi,
I'm trying to solve a memory leak since my new implementation of
communications based on MPI_AllToAllW and MPI_type_Create_SubArray
calls. Arrays of SubArray types are created/destroyed at each time step
and used for communications.
On my laptop the code runs fine (running for 15000
Thanks for all this suggestions. I'll try to create a small test
reproducing this behavior ans try the different parameters.
I do not use MPI I/O directly but parallel hdf5 which rely on MPI I/O .
NFS is the easiest way to share storage between nodes on a small
cluster. Beegfs or lustre require
Patrick,
based on George's idea, a simpler check is to retrieve the Fortran index
via the (standard) MPI_Type_c2() function
after you create a derived datatype.
If the index keeps growing forever even after you MPI_Type_free(), then
this clearly indicates a leak.
Unfortunately, this
Patrick,
I'm afraid there is no simple way to check this. The main reason being that
OMPI use handles for MPI objects, and these handles are not tracked by the
library, they are supposed to be provided by the user for each call. In
your case, as you already called MPI_Type_free on the datatype,
Hi George and Gilles,
Thanks George for your suggestion. Is it valuable for 4.05 and 3.1
OpenMPI Versions ? I will have a look today at these tables. May be
writing a small piece of code juste creating and freeing subarray datatype.
Thanks Gilles for suggesting disabling the interconnect. it is
Patrick,
In recent Open MPI releases, the default component for MPI-IO is ompio
(and no more romio)
unless the file is on a Lustre filesystem.
You can force romio with
mpirun --mca io ^ompio ...
Cheers,
Gilles
On 12/3/2020 4:20 PM, Patrick Bégou via users wrote:
Hi,
I'm using an
Patrick,
glad to hear you will upgrade Open MPI thanks to this workaround!
ompio has known performance issues on Lustre (this is why ROMIO is
still the default on this filesystem)
but I do not remember such performance issues have been reported on a
NFS filesystem.
Sharing a reproducer will be
Thanks Gilles,
this is the solution.
I will set OMPI_MCA_io=^ompio automatically when loading the parallel
hdf5 module on the cluster.
I was tracking this problem for several weeks but not looking in the
right direction (testing NFS server I/O, network bandwidth.)
I think we will now move
