Hi Bennet,
what is the issue you have with dlopen? and what options do you use
with mpi --bind?
i think the only change i make to my openmpi compile is to added
'--with-cuda=...' and '--with-pmi=...'
D
On 11/14/2017 10:01 AM, Bennet Fauber wrote:
> We are trying SLURM for the first time, and prior to this I've always
> built OMPI with Torque support. I was hoping that someone with more
> experience than I with both OMPI and SLURM might provide a bit of
> up-front advice.
>
> My situation is that we are running CentOS 7.3 (soon to be 7.4), we use
> Mellanox cards of several generations, but my systems team tells me the
> driver version is the same everywhere.
>
> 82:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0
> 5GT/s - IB QDR / 10GigE] (rev b0)
>
> 16:00.0 Network controller: Mellanox Technologies MT27500 Family
> [ConnectX-3]
>
> We have mixed NFSv3 shared directories and a Lustre filesystem (DDN).
>
> In the past, we had issues with using `dlopen` and we've had much grief
> with OMPI placing jobs on processors correctly, we think because we used
> cpusets at one point and use cgroups now and jobs share nodes with other
> jobs. My previous build options were
>
> export CONFIGURE_FLAGS='--disable-dlopen --enable-shared'
> export COMPILERS='CC=gcc CXX=g++ FC=gfortran F77=gfortran'
> export COMP_NAME='gcc-4.8.5'
> export PREFIX=/shared/nfs/directory
>
> ./configure \
> --prefix=${PREFIX} \
> --mandir=${PREFIX}/share/man \
> --with-tm \
> --with-verbs \
> $CONFIGURE_FLAGS \
> $COMPILERS
>
> Additionally, we have typically included the following lines in our
> $PREFIX/etc/openmpi-mca-params.conf
>
> orte_hetero_nodes=1
> hwloc_base_binding_policy=none
>
> Those may be there for purely historical reasons. So far as I know,
> there is no deterministic test recorded anywhere that would detect
> whether those are still needed.
>
> For this new resource manager, I am thinking that the compiler flags
> stay the same, but the configure be changed to
>
> ./configure \
> --prefix=${PREFIX} \
> --mandir=${PREFIX}/share/man \
> --with-slurm \
> --with-pmi=/usr/include/slurm \
> --with-verbs \
> $CONFIGURE_FLAGS \
> $COMPILERS
>
> I am curious, what file system support does --lustre-support enable?
>
> I will be installing three versions of OMPI to start: 1.10.7, 2.1.2,
> and 3.0.0. Are there changes to the configure line that are a priori
> known to be needed.
>
> There are references on the FAQ and other installation notes that lead
> me to believe they are a bit out of date, so I am asking preemptively
> here. Apologies if that is an incorrect assessment.
>
> Thanks, -- bennet
>
>
>
>
>
> _______________________________________________
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
>
--
David Lee Braun
Manager of Computational Facilities
for Dr Charles L. Brooks, III Ph.D.
930 N. University Ave
Chemistry 2006
(734) 615-1450
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
