Re: [OMPI users] Subject: Clarification about mpirun behavior in Slurm jobs
Christopher, I do not think Open MPI explicitly asks SLURM which cores have been assigned on each node. So if you are planning to run multiple jobs on the same node, your best bet is probably to have SLURM use cpusets. Cheers, Gilles On Sat, Feb 24, 2024 at 7:25 AM Christopher Daley via users < users@lists.open-mpi.org> wrote: > Dear Support, > > I'm seeking clarification about the expected behavior of mpirun in Slurm > jobs. > > Our setup consists of using Slurm for resource allocation and OpenMPI > mpirun to launch MPI applications. We have found that when two Slurm jobs > have been allocated different cores on the same compute node that the MPI > ranks in Slurm job 1 map to the same cores as Slurm job 2. It appears that > OpenMPI mpirun is not considering the details of the Slurm allocation. We > get expected behavior when srun is employed as the MPI launcher instead of > mpirun, i.e. the MPI ranks in Slurm job 1 use different cores than the MPI > ranks in Slurm job 2. > > We have observed this with OpenMPI-4.1.6 and OpenMPI-5.0.2. Should we > expect that the mpirun in each job will only use the exact cores that were > allocated by Slurm? > > Thanks, > Chris >
Re: [OMPI users] Subject: Clarification about mpirun behavior in Slurm jobs
I had something like this happen on a test cluster of Raspberry Pis several years ago, and in my case, I was able to isolate it to being an MPI issue unrelated to SLURM. If you can run directly on the nodes, that might be a useful distinction for you to try to make. (Running "directly" might mean manually doing "miprun -n " inside an srun-dispatched shell, if you can''t bypass SLURM, which you shouldn't anyways.) In my case, on the four-core RPi, running e.g. two two-way jobs just oversubscribed the first two cores, whether run from SLURM or directly. I found a work-around, was to use the "--map-by socket" argument to mpirun. I don't think I ever figured it out -- it was a short-lifetime test cluster that I was using to explore SLURM config options. I also don't recall which version of OpenMPI it was, but I'd guess it was the one that's packaged for Debian/Raspbian 11, which is 4.1.0. -- A. On Fri, Feb 23, 2024 at 5:06 PM Christopher Daley via users < users@lists.open-mpi.org> wrote: > Dear Support, > > I'm seeking clarification about the expected behavior of mpirun in Slurm > jobs. > > Our setup consists of using Slurm for resource allocation and OpenMPI > mpirun to launch MPI applications. We have found that when two Slurm jobs > have been allocated different cores on the same compute node that the MPI > ranks in Slurm job 1 map to the same cores as Slurm job 2. It appears that > OpenMPI mpirun is not considering the details of the Slurm allocation. We > get expected behavior when srun is employed as the MPI launcher instead of > mpirun, i.e. the MPI ranks in Slurm job 1 use different cores than the MPI > ranks in Slurm job 2. > > We have observed this with OpenMPI-4.1.6 and OpenMPI-5.0.2. Should we > expect that the mpirun in each job will only use the exact cores that were > allocated by Slurm? > > Thanks, > Chris > -- Andrew Reid / andrew.ce.r...@gmail.com
[OMPI users] Subject: Clarification about mpirun behavior in Slurm jobs
Dear Support, I'm seeking clarification about the expected behavior of mpirun in Slurm jobs. Our setup consists of using Slurm for resource allocation and OpenMPI mpirun to launch MPI applications. We have found that when two Slurm jobs have been allocated different cores on the same compute node that the MPI ranks in Slurm job 1 map to the same cores as Slurm job 2. It appears that OpenMPI mpirun is not considering the details of the Slurm allocation. We get expected behavior when srun is employed as the MPI launcher instead of mpirun, i.e. the MPI ranks in Slurm job 1 use different cores than the MPI ranks in Slurm job 2. We have observed this with OpenMPI-4.1.6 and OpenMPI-5.0.2. Should we expect that the mpirun in each job will only use the exact cores that were allocated by Slurm? Thanks, Chris
