Thanks Moe, The document and advice helps and I can see options but what is best is still unclear so I'll ask two more questions.
1) Is using sbatch with scripts and srun within the scripts 'best practice' (or 'a' best practice)? Or are there sites that just use srun? Is there specific documentation on running srun nested in sbatch? The mc_support.html page only refers to srun options. 2) We currently have a few unrelated usage patterns where jobs request multiple nodes but only some of the cores (perhaps to match jobs that they used on our previous cluster configuration). How would you deal with that case where --exclusive is not necessarily appropriate? A big stick might be an option (and advice to use whole nodes) though the users are in different cities so it might have to be a virtual stick. Gareth BTW. --ntasks-per-node=1 was not needed in your advice as it was the default. However, in that case to use srun and use all the cores, extra options were needed. > -----Original Message----- > From: Moe Jette [mailto:[email protected]] > Sent: Tuesday, 3 March 2015 3:42 AM > To: slurm-dev > Subject: [slurm-dev] Re: mixing mpi and per node tasks > > > Use the "--exclusive" option to always get whole node allocations: > > $ sbatch --exclusive -N 3 my.bash > > I would use the "--ntasks-per-node=1" option to control the task count > per node: > > srun --ntasks-per-node=1 my.app > > I would also recommend this document: > http://slurm.schedmd.com/mc_support.html > > Quoting [email protected]: > > > We have a cluster with dual socket nodes with 10-core cpus (ht off) > > and we share nodes with SelectType=select/cons_res. Before (or > > after) running an MPI task, I'd like to run some pre (and post) > > processing tasks, one per node but am having trouble finding > > documentation for how to do this. I was expecting to submit a jobs > > with sbatch with --nodes=N --tasks-per-node=20 where N is an integer > > to get multiple whole nodes then run srun --tasks-per-node=1 for the > > per node tasks but this does not work (I get one task for each core). > > > > I'd also like any solution to work with hybrid mpi/openmp with one > > openmp task per node or per socket. > > > > Thanks, > > > > Gareth > > > -- > Morris "Moe" Jette > CTO, SchedMD LLC > Commercial Slurm Development and Support
