Re: [OMPI devel] Is mpirun in slurm producing the correct "srun" cmd behind the scene?

2019-07-05 Thread Gilles Gouaillardet via devel
Lawrence,

My apologies for the incorrect link.
The patch is at https://github.com/open-mpi/ompi/pull/6672.patch

Cheers,

Gilles

On Sat, Jul 6, 2019 at 12:21 PM Gilles Gouaillardet
 wrote:
>
> Lawrence,
>
> this is a known issue (--cpu_bind optioned was removed from SLURM in
> favor of the --cpu-bind option) and the fix will be available in Open
> MPI 4.0.1
>
> Meanwhile, you can manually download and apply the patch at
> https://github.com/open-mpi/ompi/pull/6445.patch or use a nightly
> build of the v4.0.x branch.
>
> Cheers,
>
> Gilles
>
> On Sat, Jul 6, 2019 at 5:35 AM Sorrillo, Lawrence via devel
>  wrote:
> >
> > Hello,
> >
> >
> >
> > My attempt to run and troubleshoot my an OMPI job under a slurm allocation 
> > does not work as I would expect.
> >
> > The result below has led me to believe that under the hood, in this setup 
> > (SLURM with OMPI) that the correct srun options is not being used
> >
> > when I call, mpirun directly.
> >
> >
> >
> > Specifically the “cpu-bind=none” is breaking, but it also looks like the 
> > nodelist is incorrect.
> >
> >
> >
> > The job script.
> >
> >
> >
> >   1 #!/bin/bash
> >
> >   2
> >
> >   3 #SBATCH --job-name=mpi-hostname
> >
> >   4 #SBATCH --partition=dev
> >
> >   5 #SBATCH --account=Account1
> >
> >   6 #SBATCH --time=01:00:00
> >
> >   7 #SBATCH --nodes=2
> >
> >   8 #SBATCH --ntasks-per-node=1
> >
> >   9 #SBATCH --begin=now+10
> >
> > 10 #SBATCH --output="%x-%u-%N-%j.txt"  # 
> > jobName-userId-hostName-jobId.txt
> >
> > 11
> >
> > 12
> >
> > 13
> >
> > 14 # -- 
> > #
> >
> > 15 #module load DefApps
> >
> > 16 #module purge >/dev/null 2>&1
> >
> > 17 ##module load staging/slurm >/dev/null 2>&1
> >
> > 18 #module load  gcc/4.8.5 openmpi >/dev/null 2>&1
> >
> > 19 #module --ignore_cache spider openmpi/3.1.3 >/dev/null 2>&1
> >
> > 20 #
> >
> > 21 # -- 
> > #
> >
> > 22 #
> >
> > 23 MPI_RUN=$(which orterun)
> >
> > 24 if [[ -z "${MPI_RUN:+x}" ]]; then
> >
> > 25   echo "ERROR: Cannot find 'mpirun' executable..exiting"
> >
> > 26   exit 1
> >
> > 27 fi
> >
> > 28
> >
> > 29 echo
> >
> > 30 #CMD="orterun  -npernode 1 -np 2  /bin/hostname"
> >
> > 31 #CMD="srun /bin/hostname"
> >
> > 32 #CMD="srun -N2 -n2 --mpi=pmi2  /bin/hostname"
> >
> > 33 #MCMD="/sw/dev/openmpi401/bin/mpirun --bind-to-core  --report-bindings  
> > -mca btl openib,self -mca plm_base_verbose 10  /bin/hostname"
> >
> > 34 MCMD="/sw/dev/openmpi401/bin/mpirun  --report-bindings  -mca btl 
> > openib,self -mca plm_base_verbose 10  /bin/hostname"
> >
> > 35 echo "INFO: Executing the command: $MCMD"
> >
> > 36 $MCMD
> >
> > 37 sync
> >
> >
> >
> > Here is the output:
> >
> >
> >
> > 2 user1@node-login7g:~/git/slurm-jobs$ more 
> > mpi-hostname-user1-node513-835.txt
> >
> >   3
> >
> >   4 INFO: Executing the command: /sw/dev/openmpi401/bin/mpirun  
> > --report-bindings  -mca btl openib,self -mca plm_base_verbose 10  
> > /bin/hostname
> >
> >   5 [node513:32514] mca: base: components_register: registering framework 
> > plm components
> >
> >   6 [node513:32514] mca: base: components_register: found loaded component 
> > isolated
> >
> >   7 [node513:32514] mca: base: components_register: component isolated has 
> > no register or open function
> >
> >   8 [node513:32514] mca: base: components_register: found loaded component 
> > rsh
> >
> >   9 [node513:32514] mca: base: components_register: component rsh register 
> > function successful
> >
> > 10 [node513:32514] mca: base: components_register: found loaded component 
> > slurm
> >
> > 11 [node513:32514] mca: base: components_register: component slurm register 
> > function successful
> >
> > 12 [node513:32514] mca: base: components_register: found loaded component tm
> >
> > 13 [node513:32514] mca: base: components_register: component tm register 
> > function successful
> >
> > 14 [node513:32514] mca: base: components_open: opening plm components
> >
> > 15 [node513:32514] mca: base: components_open: found loaded component 
> > isolated
> >
> > 16 [node513:32514] mca: base: components_open: component isolated open 
> > function successful
> >
> > 17 [node513:32514] mca: base: components_open: found loaded component rsh
> >
> > 18 [node513:32514] mca: base: components_open: component rsh open function 
> > successful
> >
> > 19 [node513:32514] mca: base: components_open: found loaded component slurm
> >
> > 20 [node513:32514] mca: base: components_open: component slurm open 
> > function successful
> >
> > 21 [node513:32514] mca: base: components_open: found loaded component tm
> >
> > 22 [node513:32514] mca: base: components_open: component tm open function 
> > successful
> >
> > 23 [node513:32514] mca:base:select: Auto-selecting plm components
> >
> > 24 [node513:32514] mca:base:select:(  plm) Querying component [isolated]
> >
> > 25 [node513:32514] mca:base:select:(  plm) Query of component 

Re: [OMPI devel] Is mpirun in slurm producing the correct "srun" cmd behind the scene?

2019-07-05 Thread Gilles Gouaillardet via devel
Lawrence,

this is a known issue (--cpu_bind optioned was removed from SLURM in
favor of the --cpu-bind option) and the fix will be available in Open
MPI 4.0.1

Meanwhile, you can manually download and apply the patch at
https://github.com/open-mpi/ompi/pull/6445.patch or use a nightly
build of the v4.0.x branch.

Cheers,

Gilles

On Sat, Jul 6, 2019 at 5:35 AM Sorrillo, Lawrence via devel
 wrote:
>
> Hello,
>
>
>
> My attempt to run and troubleshoot my an OMPI job under a slurm allocation 
> does not work as I would expect.
>
> The result below has led me to believe that under the hood, in this setup 
> (SLURM with OMPI) that the correct srun options is not being used
>
> when I call, mpirun directly.
>
>
>
> Specifically the “cpu-bind=none” is breaking, but it also looks like the 
> nodelist is incorrect.
>
>
>
> The job script.
>
>
>
>   1 #!/bin/bash
>
>   2
>
>   3 #SBATCH --job-name=mpi-hostname
>
>   4 #SBATCH --partition=dev
>
>   5 #SBATCH --account=Account1
>
>   6 #SBATCH --time=01:00:00
>
>   7 #SBATCH --nodes=2
>
>   8 #SBATCH --ntasks-per-node=1
>
>   9 #SBATCH --begin=now+10
>
> 10 #SBATCH --output="%x-%u-%N-%j.txt"  # jobName-userId-hostName-jobId.txt
>
> 11
>
> 12
>
> 13
>
> 14 # -- #
>
> 15 #module load DefApps
>
> 16 #module purge >/dev/null 2>&1
>
> 17 ##module load staging/slurm >/dev/null 2>&1
>
> 18 #module load  gcc/4.8.5 openmpi >/dev/null 2>&1
>
> 19 #module --ignore_cache spider openmpi/3.1.3 >/dev/null 2>&1
>
> 20 #
>
> 21 # -- #
>
> 22 #
>
> 23 MPI_RUN=$(which orterun)
>
> 24 if [[ -z "${MPI_RUN:+x}" ]]; then
>
> 25   echo "ERROR: Cannot find 'mpirun' executable..exiting"
>
> 26   exit 1
>
> 27 fi
>
> 28
>
> 29 echo
>
> 30 #CMD="orterun  -npernode 1 -np 2  /bin/hostname"
>
> 31 #CMD="srun /bin/hostname"
>
> 32 #CMD="srun -N2 -n2 --mpi=pmi2  /bin/hostname"
>
> 33 #MCMD="/sw/dev/openmpi401/bin/mpirun --bind-to-core  --report-bindings  
> -mca btl openib,self -mca plm_base_verbose 10  /bin/hostname"
>
> 34 MCMD="/sw/dev/openmpi401/bin/mpirun  --report-bindings  -mca btl 
> openib,self -mca plm_base_verbose 10  /bin/hostname"
>
> 35 echo "INFO: Executing the command: $MCMD"
>
> 36 $MCMD
>
> 37 sync
>
>
>
> Here is the output:
>
>
>
> 2 user1@node-login7g:~/git/slurm-jobs$ more mpi-hostname-user1-node513-835.txt
>
>   3
>
>   4 INFO: Executing the command: /sw/dev/openmpi401/bin/mpirun  
> --report-bindings  -mca btl openib,self -mca plm_base_verbose 10  
> /bin/hostname
>
>   5 [node513:32514] mca: base: components_register: registering framework plm 
> components
>
>   6 [node513:32514] mca: base: components_register: found loaded component 
> isolated
>
>   7 [node513:32514] mca: base: components_register: component isolated has no 
> register or open function
>
>   8 [node513:32514] mca: base: components_register: found loaded component rsh
>
>   9 [node513:32514] mca: base: components_register: component rsh register 
> function successful
>
> 10 [node513:32514] mca: base: components_register: found loaded component 
> slurm
>
> 11 [node513:32514] mca: base: components_register: component slurm register 
> function successful
>
> 12 [node513:32514] mca: base: components_register: found loaded component tm
>
> 13 [node513:32514] mca: base: components_register: component tm register 
> function successful
>
> 14 [node513:32514] mca: base: components_open: opening plm components
>
> 15 [node513:32514] mca: base: components_open: found loaded component isolated
>
> 16 [node513:32514] mca: base: components_open: component isolated open 
> function successful
>
> 17 [node513:32514] mca: base: components_open: found loaded component rsh
>
> 18 [node513:32514] mca: base: components_open: component rsh open function 
> successful
>
> 19 [node513:32514] mca: base: components_open: found loaded component slurm
>
> 20 [node513:32514] mca: base: components_open: component slurm open function 
> successful
>
> 21 [node513:32514] mca: base: components_open: found loaded component tm
>
> 22 [node513:32514] mca: base: components_open: component tm open function 
> successful
>
> 23 [node513:32514] mca:base:select: Auto-selecting plm components
>
> 24 [node513:32514] mca:base:select:(  plm) Querying component [isolated]
>
> 25 [node513:32514] mca:base:select:(  plm) Query of component [isolated] set 
> priority to 0
>
> 26 [node513:32514] mca:base:select:(  plm) Querying component [rsh]
>
> 27 [node513:32514] mca:base:select:(  plm) Query of component [rsh] set 
> priority to 10
>
> 28 [node513:32514] mca:base:select:(  plm) Querying component [slurm]
>
> 29 [node513:32514] mca:base:select:(  plm) Query of component [slurm] set 
> priority to 75
>
> 30 [node513:32514] mca:base:select:(  plm) Querying component [tm]
>
> 31 [node513:32514] mca:base:select:(  plm) Selected component [slurm]
>
> 32 [node513:32514] mca: base: close: component isolated closed
>
> 

[OMPI devel] Is mpirun in slurm producing the correct "srun" cmd behind the scene?

2019-07-05 Thread Sorrillo, Lawrence via devel
Hello,

My attempt to run and troubleshoot my an OMPI job under a slurm allocation does 
not work as I would expect.
The result below has led me to believe that under the hood, in this setup 
(SLURM with OMPI) that the correct srun options is not being used
when I call, mpirun directly.

Specifically the “cpu-bind=none” is breaking, but it also looks like the 
nodelist is incorrect.

The job script.

  1 #!/bin/bash
  2
  3 #SBATCH --job-name=mpi-hostname
  4 #SBATCH --partition=dev
  5 #SBATCH --account=Account1
  6 #SBATCH --time=01:00:00
  7 #SBATCH --nodes=2
  8 #SBATCH --ntasks-per-node=1
  9 #SBATCH --begin=now+10
10 #SBATCH --output="%x-%u-%N-%j.txt"  # jobName-userId-hostName-jobId.txt
11
12
13
14 # -- #
15 #module load DefApps
16 #module purge >/dev/null 2>&1
17 ##module load staging/slurm >/dev/null 2>&1
18 #module load  gcc/4.8.5 openmpi >/dev/null 2>&1
19 #module --ignore_cache spider openmpi/3.1.3 >/dev/null 2>&1
20 #
21 # -- #
22 #
23 MPI_RUN=$(which orterun)
24 if [[ -z "${MPI_RUN:+x}" ]]; then
25   echo "ERROR: Cannot find 'mpirun' executable..exiting"
26   exit 1
27 fi
28
29 echo
30 #CMD="orterun  -npernode 1 -np 2  /bin/hostname"
31 #CMD="srun /bin/hostname"
32 #CMD="srun -N2 -n2 --mpi=pmi2  /bin/hostname"
33 #MCMD="/sw/dev/openmpi401/bin/mpirun --bind-to-core  --report-bindings  -mca 
btl openib,self -mca plm_base_verbose 10  /bin/hostname"
34 MCMD="/sw/dev/openmpi401/bin/mpirun  --report-bindings  -mca btl openib,self 
-mca plm_base_verbose 10  /bin/hostname"
35 echo "INFO: Executing the command: $MCMD"
36 $MCMD
37 sync

Here is the output:

2 user1@node-login7g:~/git/slurm-jobs$ more mpi-hostname-user1-node513-835.txt
  3
  4 INFO: Executing the command: /sw/dev/openmpi401/bin/mpirun  
--report-bindings  -mca btl openib,self -mca plm_base_verbose 10  /bin/hostname
  5 [node513:32514] mca: base: components_register: registering framework plm 
components
  6 [node513:32514] mca: base: components_register: found loaded component 
isolated
  7 [node513:32514] mca: base: components_register: component isolated has no 
register or open function
  8 [node513:32514] mca: base: components_register: found loaded component rsh
  9 [node513:32514] mca: base: components_register: component rsh register 
function successful
10 [node513:32514] mca: base: components_register: found loaded component slurm
11 [node513:32514] mca: base: components_register: component slurm register 
function successful
12 [node513:32514] mca: base: components_register: found loaded component tm
13 [node513:32514] mca: base: components_register: component tm register 
function successful
14 [node513:32514] mca: base: components_open: opening plm components
15 [node513:32514] mca: base: components_open: found loaded component isolated
16 [node513:32514] mca: base: components_open: component isolated open function 
successful
17 [node513:32514] mca: base: components_open: found loaded component rsh
18 [node513:32514] mca: base: components_open: component rsh open function 
successful
19 [node513:32514] mca: base: components_open: found loaded component slurm
20 [node513:32514] mca: base: components_open: component slurm open function 
successful
21 [node513:32514] mca: base: components_open: found loaded component tm
22 [node513:32514] mca: base: components_open: component tm open function 
successful
23 [node513:32514] mca:base:select: Auto-selecting plm components
24 [node513:32514] mca:base:select:(  plm) Querying component [isolated]
25 [node513:32514] mca:base:select:(  plm) Query of component [isolated] set 
priority to 0
26 [node513:32514] mca:base:select:(  plm) Querying component [rsh]
27 [node513:32514] mca:base:select:(  plm) Query of component [rsh] set 
priority to 10
28 [node513:32514] mca:base:select:(  plm) Querying component [slurm]
29 [node513:32514] mca:base:select:(  plm) Query of component [slurm] set 
priority to 75
30 [node513:32514] mca:base:select:(  plm) Querying component [tm]
31 [node513:32514] mca:base:select:(  plm) Selected component [slurm]
32 [node513:32514] mca: base: close: component isolated closed
33 [node513:32514] mca: base: close: unloading component isolated
34 [node513:32514] mca: base: close: component rsh closed
35 [node513:32514] mca: base: close: unloading component rsh
36 [node513:32514] mca: base: close: component tm closed
37 [node513:32514] mca: base: close: unloading component tm
38 [node513:32514] [[4367,0],0] plm:slurm: final top-level argv:
39 srun --ntasks-per-node=1 --kill-on-bad-exit --cpu_bind=none 
--nodes=1 --nodelist=node514 --ntasks=1 orted -mca orte_report_bindings "1" 
-mca ess "slurm" -mca ess_base_jobid "286195712" -mca ess_base_vpid "1" -mca 
ess_base_num_procs "2" -mca orte_node_regex "node[3:513-514]@0(2)" -mca 
orte_hnp_uri "286195712.0;tcp://172.30.146
40 10.38.146.45:43031" -mca btl "openib,self" -mca