Paul,
Try
# echo "ulimit -l unlimited" >/etc/sysconfig/slurm
on each of your compute nodes, and then restart slurm on the compute nodes.
FWIW, I explicitly set --with-pmi=<path to slurm> when I build OpenMPI.
Andy
On 06/17/2015 10:38 AM, Wiegand, Paul wrote:
Greetings
We have just started experimenting with Slurm, and I'm having trouble running
OpenMPI jobs over Slurm. I've read the Slurm and OpenMPI documentation in the
issue and *believe* I've done all I am supposed to, yet still I get a segfault
during the MPI_Init when using Slurm. I can run fine under Moab/Torque, and I
can run fine without a resource manager at all. Any help would be greatly
appreciated.
Slurm info:
Version 14.11.7
Install location /apps/slurm/slurm-14.11.7/
OpenMPI:
Version 1.8.3
Install location /apps/openmpi/openmpi-1.8.3-ic-2015-slurm-14.11/
Compiler:
Version Intel Composer 2015.1.33
Install location /apps/ic/ic-2015.1.133/
OS: CentOS 7
I have done the following, none of which has changed the outcome:
1. Built OpenMPI with and without the --with-slurm switch, explicitly pathing
it to the location and also letting config figure it out (it does so correctly).
2. Built OpenMPI using the --with-pmi switch, explicitly pathing it to Slurm's
PMI location and not, as well as using the --without-pmi and explicitly setting
the reserved ports.
3. Checked the registrable memory issue with the IB driver to confirm our
ib_mthca num_mtt and fmr_reserved_mtts are set such that registrable memory by
the IB driver is at least as much as physical memory.
4. Confirmed that the memlock ulimit is unlimited on all nodes (even the head
node).
5. Verified that all libs/bins are in path, and that ldd on the binary shows
the same locations for libs found whether running via Slurm or not.
6. Built and run a variety of different MPI programs, including those that I
did not write myself.
NOTE: I *can* run my MPI program under slurm correctly as root, but I see no
environmental or ulimit differences. So I'm assuming there's a permission
problem somewhere, but I can't find it. But I guess that this tells me that:
it's not the MPI build, it's not the Slurm build, and it's not the IB driver
memory issue.
Some traces showing a typical example of what happens appear below. Please let
me know if I can provide any more detail.
Thanks,
Paul.
---
$
$ echo -e "\n...Showing setup...\n"
...Showing setup...
$ module purge
$ module load openmpi-1.8.3-ic-2015-slurm-14.11
$ which mpicc
/apps/openmpi/openmpi-1.8.3-ic-2015-slurm-14.11/bin/mpicc
$ which mpirun
/apps/openmpi/openmpi-1.8.3-ic-2015-slurm-14.11/bin/mpirun
$ cat simple.c
#include <stdio.h>
#include <mpi.h>
int main(int argc, char *argv[]) {
MPI_Init(&argc, &argv);
MPI_Finalize();
printf("Done.\n");
}
$
$
$ echo -e "\n...Run without resource manager...\n"
...Run without resource manager...
$ mpicc -o simple simple.c
$ mpirun -host evc5 ./simple
Done.
$
$
$ echo -e "\n...Checking state info...\n"
...Checking state info...
$ ulimit -l
unlimited
$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
viz* up infinite 6 idle evc[5-9]
$
$
$ echo -e "\n...Run with slurm using srun...\n"
...Run with slurm using srun...
$ srun -n 1 ./simple
[evc5:19105] *** Process received signal ***
[evc5:19105] Signal: Segmentation fault (11)
[evc5:19105] Signal code: Address not mapped (1)
[evc5:19105] Failing at address: 0x30
[evc5:19105] [ 0] /lib64/libpthread.so.0(+0xf130)[0x2ae1b06ce130]
[evc5:19105] [ 1]
/apps/openmpi/openmpi-1.8.3-ic-2015-slurm-14.11/lib/openmpi/mca_btl_openib.so(+0x1fdd8)[0x2ae1b6464dd8]
[evc5:19105] [ 2]
/apps/openmpi/openmpi-1.8.3-ic-2015-slurm-14.11/lib/openmpi/mca_btl_openib.so(ompi_btl_openib_connect_base_select_for_local_port+0xd0)[0x2ae1b6463ae0]
[evc5:19105] [ 3]
/apps/openmpi/openmpi-1.8.3-ic-2015-slurm-14.11/lib/openmpi/mca_btl_openib.so(+0x109b8)[0x2ae1b64559b8]
[evc5:19105] [ 4]
/apps/openmpi/openmpi-1.8.3-ic-2015-slurm-14.11/lib/libmpi.so.1(mca_btl_base_select+0x171)[0x2ae1aff2d571]
[evc5:19105] [ 5]
/apps/openmpi/openmpi-1.8.3-ic-2015-slurm-14.11/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0xe)[0x2ae1b624297e]
[evc5:19105] [ 6]
/apps/openmpi/openmpi-1.8.3-ic-2015-slurm-14.11/lib/libmpi.so.1(mca_bml_base_init+0x74)[0x2ae1aff2cd64]
[evc5:19105] [ 7]
/apps/openmpi/openmpi-1.8.3-ic-2015-slurm-14.11/lib/openmpi/mca_pml_ob1.so(+0x5f06)[0x2ae1b7766f06]
[evc5:19105] [ 8]
/apps/openmpi/openmpi-1.8.3-ic-2015-slurm-14.11/lib/libmpi.so.1(mca_pml_base_select+0x449)[0x2ae1aff400e9]
[evc5:19105] [ 9]
/apps/openmpi/openmpi-1.8.3-ic-2015-slurm-14.11/lib/libmpi.so.1(ompi_mpi_init+0x699)[0x2ae1afeeb519]
[evc5:19105] [10]
/apps/openmpi/openmpi-1.8.3-ic-2015-slurm-14.11/lib/libmpi.so.1(MPI_Init+0xee)[0x2ae1aff0e44e]
[evc5:19105] [11] REDACTED/Projects/OpenMPI/./simple[0x401352]
[evc5:19105] [12] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2ae1b08fcaf5]
[evc5:19105] [13] REDACTED/Projects/OpenMPI/./simple[0x401249]
[evc5:19105] *** End of error message ***
srun: error: evc5: task 0: Segmentation fault (core dumped)
$
$
$ echo -e "\n...Run with slurm using salloc...\n"
...Run with slurm using salloc...
$ salloc -n 1
salloc: Granted job allocation 192
$ ulimit -l
unlimited
$ mpirun ./simple
[evc5:19184] *** Process received signal ***
[evc5:19184] Signal: Segmentation fault (11)
[evc5:19184] Signal code: Address not mapped (1)
[evc5:19184] Failing at address: 0x30
[evc5:19184] [ 0] /lib64/libpthread.so.0(+0xf130)[0x2b401b32c130]
[evc5:19184] [ 1]
/apps/openmpi/openmpi-1.8.3-ic-2015-slurm-14.11/lib/openmpi/mca_btl_openib.so(+0x1fdd8)[0x2b4020735dd8]
[evc5:19184] [ 2]
/apps/openmpi/openmpi-1.8.3-ic-2015-slurm-14.11/lib/openmpi/mca_btl_openib.so(ompi_btl_openib_connect_base_select_for_local_port+0xd0)[0x2b4020734ae0]
[evc5:19184] [ 3]
/apps/openmpi/openmpi-1.8.3-ic-2015-slurm-14.11/lib/openmpi/mca_btl_openib.so(+0x109b8)[0x2b40207269b8]
[evc5:19184] [ 4]
/apps/openmpi/openmpi-1.8.3-ic-2015-slurm-14.11/lib/libmpi.so.1(mca_btl_base_select+0x171)[0x2b401ab8b571]
[evc5:19184] [ 5]
/apps/openmpi/openmpi-1.8.3-ic-2015-slurm-14.11/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0xe)[0x2b402051397e]
[evc5:19184] [ 6]
/apps/openmpi/openmpi-1.8.3-ic-2015-slurm-14.11/lib/libmpi.so.1(mca_bml_base_init+0x74)[0x2b401ab8ad64]
[evc5:19184] [ 7]
/apps/openmpi/openmpi-1.8.3-ic-2015-slurm-14.11/lib/openmpi/mca_pml_ob1.so(+0x5f06)[0x2b4021a37f06]
[evc5:19184] [ 8]
/apps/openmpi/openmpi-1.8.3-ic-2015-slurm-14.11/lib/libmpi.so.1(mca_pml_base_select+0x449)[0x2b401ab9e0e9]
[evc5:19184] [ 9]
/apps/openmpi/openmpi-1.8.3-ic-2015-slurm-14.11/lib/libmpi.so.1(ompi_mpi_init+0x699)[0x2b401ab49519]
[evc5:19184] [10]
/apps/openmpi/openmpi-1.8.3-ic-2015-slurm-14.11/lib/libmpi.so.1(MPI_Init+0xee)[0x2b401ab6c44e]
[evc5:19184] [11] ./simple[0x401352]
[evc5:19184] [12] /lib64/libc.so.6(__libc_start_main+0xf5)[0x2b401b55aaf5]
[evc5:19184] [13] ./simple[0x401249]
[evc5:19184] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 0 with PID 19184 on node evc5 exited on signal
11 (Segmentation fault).
--------------------------------------------------------------------------
