On Mon, 13 Jul 2015 03:46:12 -0700 "W. Miah" <[email protected]> wrote:
> > Hello, > > I would like to submit a hybrid MPI + OpenMP job and would like the > Slurm settings for it. I am using Intel MPI have the following > scenarios: Not sure if you're asking for slurm config (there's no special config to allow hybrid job). > 1. I would like one MPI rank per host and the OpenMP threads to be > spawned on the host; What I typically do is -n NUMBER_OF_RANKS_I_WANT and -c NUMBER_OF_OMP_THREADS_PER_RANK_I_WANT. Then in the batch script: export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK mpiexec.hydra -bootstrap slurm ./binary.x Intel mpi by default pins as widely as possible => works with many threads per rank. But you may have to play with more advanced pinning to get best performance (see KMP_AFFINITY, I_MPI_PIN_DOMAIN, ...). Also beware of running OpenMP across sockets in a NUMA system as this for many OpenMP codes is challenging performance wise... (that is, you probably want one rank per socket/NUMA zone, not one per node) /Peter > 2. I would like two MPI ranks per host and each MPI rank to be pinned > on each socket (I have two socket nodes). The OpenMP threads will then > be spawned on each socket.
