Re: [OMPI users] Open MPI 4.0.3 outside as well as inside a SimpleFOAM container: step creation temporarily disabled, retrying Requested nodes are busy

Tue, 28 Feb 2023 20:43:09 -0800 

Rob,


Do you invoke mpirun from **inside** the container?
IIRC, mpirun is generally invoked from **outside** the container, could you try this if not already the case?
The error message is from SLURM, so this is really a SLURM vs singularity issue. 

What if you

srun -N 2 -n 2 hostname

instead of

mpirun ...


Cheers,


Gilles

On 3/1/2023 12:44 PM, Rob Kudyba via users wrote:
Singularity 3.5.3 on RHEL 7 cluster w/ OpenMPI 4.0.3 lives inside a SimpleFOAM version 10 container. I've confirmed the OpenMPI versions are the same. Perhaps this is a question for Singularity users as well but how can I troubleshoot why mpirun just returns step creation temporarily disabled, retrying Requested 

Singularity> mpirun -V
mpirun (Open MPI) 4.0.3
Report bugs to http://www.open-mpi.org/community/help/
Singularity> which mpirun
/usr/bin/mpirun
Singularity>

$ mpirun -V
mpirun (Open MPI) 4.0.3
mpirun -n 2 -mca plm_base_verbose 100 --mca ras_base_verbose 100 --mca rss_base_verbose 100 --mca rmaps_base_verbose 100  singularity exec openfoam simpleFoam -fileHandler uncollated -parallel | tee log.simpleFoam openfoam10/          openfoam10.sif openfoamtestfile.sh  openfoam_v2012.sif [myuser@node047 motorBike]$ mpirun -n 2 -mca plm_base_verbose 100 --mca ras_base_verbose 100 --mca rss_base_verbose 100 --mca rmaps_base_verbose 100  singularity exec openfoam   simpleFoam -fileHandler uncollated -parallel | tee log.simpleFoam openfoam10/          openfoam10.sif openfoamtestfile.sh  openfoam_v2012.sif [myuser@node047 motorBike]$ mpirun -n 2 -mca plm_base_verbose 100 --mca ras_base_verbose 100 --mca rss_base_verbose 100 --mca rmaps_base_verbose 100  singularity exec openfoam10.sif   simpleFoam  -parallel | tee log.simpleFoam [node047:11650] mca: base: components_register: registering framework plm components [node047:11650] mca: base: components_register: found loaded component slurm [node047:11650] mca: base: components_register: component slurm register function successful [node047:11650] mca: base: components_register: found loaded component isolated [node047:11650] mca: base: components_register: component isolated has no register or open function 
[node047:11650] mca: base: components_register: found loaded component rsh
[node047:11650] mca: base: components_register: component rsh register function successful 
[node047:11650] mca: base: components_open: opening plm components
[node047:11650] mca: base: components_open: found loaded component slurm
[node047:11650] mca: base: components_open: component slurm open function successful [node047:11650] mca: base: components_open: found loaded component isolated [node047:11650] mca: base: components_open: component isolated open function successful 
[node047:11650] mca: base: components_open: found loaded component rsh
[node047:11650] mca: base: components_open: component rsh open function successful 
[node047:11650] mca:base:select: Auto-selecting plm components
[node047:11650] mca:base:select:(  plm) Querying component [slurm]
[node047:11650] mca:base:select:(  plm) Query of component [slurm] set priority to 75 
[node047:11650] mca:base:select:(  plm) Querying component [isolated]
[node047:11650] mca:base:select:(  plm) Query of component [isolated] set priority to 0 
[node047:11650] mca:base:select:(  plm) Querying component [rsh]
[node047:11650] mca:base:select:(  plm) Query of component [rsh] set priority to 10 
[node047:11650] mca:base:select:(  plm) Selected component [slurm]
[node047:11650] mca: base: close: component isolated closed
[node047:11650] mca: base: close: unloading component isolated
[node047:11650] mca: base: close: component rsh closed
[node047:11650] mca: base: close: unloading component rsh
[node047:11650] mca: base: components_register: registering framework ras components [node047:11650] mca: base: components_register: found loaded component slurm [node047:11650] mca: base: components_register: component slurm register function successful [node047:11650] mca: base: components_register: found loaded component simulator [node047:11650] mca: base: components_register: component simulator register function successful 
[node047:11650] mca: base: components_open: opening ras components
[node047:11650] mca: base: components_open: found loaded component slurm
[node047:11650] mca: base: components_open: component slurm open function successful [node047:11650] mca: base: components_open: found loaded component simulator 
[node047:11650] mca:base:select: Auto-selecting ras components
[node047:11650] mca:base:select:(  ras) Querying component [slurm]
[node047:11650] mca:base:select:(  ras) Query of component [slurm] set priority to 50 
[node047:11650] mca:base:select:(  ras) Querying component [simulator]
[node047:11650] mca:base:select:(  ras) Selected component [slurm]
[node047:11650] mca: base: close: unloading component simulator
[node047:11650] mca: base: components_register: registering framework rmaps components 
[node047:11650] mca: base: components_register: found loaded component seq
[node047:11650] mca: base: components_register: component seq register function successful [node047:11650] mca: base: components_register: found loaded component rank_file [node047:11650] mca: base: components_register: component rank_file register function successful [node047:11650] mca: base: components_register: found loaded component resilient [node047:11650] mca: base: components_register: component resilient register function successful [node047:11650] mca: base: components_register: found loaded component mindist [node047:11650] mca: base: components_register: component mindist register function successful [node047:11650] mca: base: components_register: found loaded component round_robin [node047:11650] mca: base: components_register: component round_robin register function successful 
[node047:11650] mca: base: components_register: found loaded component ppr
[node047:11650] mca: base: components_register: component ppr register function successful [node047:11650] [[57513,0],0] rmaps:base set policy with NULL device NONNULL 
[node047:11650] mca: base: components_open: opening rmaps components
[node047:11650] mca: base: components_open: found loaded component seq
[node047:11650] mca: base: components_open: component seq open function successful [node047:11650] mca: base: components_open: found loaded component rank_file [node047:11650] mca: base: components_open: component rank_file open function successful [node047:11650] mca: base: components_open: found loaded component resilient [node047:11650] mca: base: components_open: component resilient open function successful 
[node047:11650] mca: base: components_open: found loaded component mindist
[node047:11650] mca: base: components_open: component mindist open function successful [node047:11650] mca: base: components_open: found loaded component round_robin [node047:11650] mca: base: components_open: component round_robin open function successful 
[node047:11650] mca: base: components_open: found loaded component ppr
[node047:11650] mca: base: components_open: component ppr open function successful 
[node047:11650] mca:rmaps:select: checking available component seq
[node047:11650] mca:rmaps:select: Querying component [seq]
[node047:11650] mca:rmaps:select: checking available component rank_file
[node047:11650] mca:rmaps:select: Querying component [rank_file]
[node047:11650] mca:rmaps:select: checking available component resilient
[node047:11650] mca:rmaps:select: Querying component [resilient]
[node047:11650] mca:rmaps:select: checking available component mindist
[node047:11650] mca:rmaps:select: Querying component [mindist]
[node047:11650] mca:rmaps:select: checking available component round_robin
[node047:11650] mca:rmaps:select: Querying component [round_robin]
[node047:11650] mca:rmaps:select: checking available component ppr
[node047:11650] mca:rmaps:select: Querying component [ppr]
[node047:11650] [[57513,0],0]: Final mapper priorities
[node047:11650] Mapper: ppr Priority: 90
[node047:11650] Mapper: seq Priority: 60
[node047:11650] Mapper: resilient Priority: 40
[node047:11650] Mapper: mindist Priority: 20
[node047:11650] Mapper: round_robin Priority: 10
[node047:11650] Mapper: rank_file Priority: 0
[node047:11650] [[57513,0],0] plm:slurm: final top-level argv:
srun --ntasks-per-node=1 --kill-on-bad-exit --nodes=1 --nodelist=node048 --ntasks=1 orted -mca ess "slurm" -mca ess_base_jobid "3769171968" -mca ess_base_vpid "1" -mca ess_base_num_procs "2" -mca orte_node_regex "t[3:47-48]@0(2)" -mca orte_hnp_uri "3769171968.0;tcp://10.x.x.47,10.x.x.47:50819" -mca plm_base_verbose "100" --mca ras_base_verbose "100" --mca rss_base_verbose "100" --mca rmaps_base_verbose "100" 

======================   ALLOCATED NODES ======================
node047: flags=0x11 slots=1 max_slots=0 slots_inuse=0 state=UP
node048: flags=0x10 slots=1 max_slots=0 slots_inuse=0 state=UP
=================================================================

My process:
myuser    11650  10965  0 22:28 pts/0    00:00:00 mpirun -n 2 -mca plm_base_verbose 100 --mca ras_base_verbose 100 --mca rss_base_verbose 100 --mca rmaps_base_verbose 100 singularity exec openfoam10.sif simpleFoam -parallel 

strace just hangs at:
strace: Process 11650 attached
restart_syscall(<... resuming interrupted poll ...>^Cstrace: Process 11650 detached 
 <detached ...>

With or without the --exclusive option all I get is:
srun: Job 12525169 step creation temporarily disabled, retrying (Requested nodes are busy) srun: Job 12525169 step creation temporarily disabled, retrying (Requested nodes are busy) 

Are the options not in the correct order?

Thanks,

Rob

Reply via email to