Slurm has an option --ntasks-per-node, so there are equivalents to > qsub -l nodes=8,ppn=4 > qsub -l nodes=4,ppn=8 > qsub -l nodes=2,ppn=16 like this sbatch -N8 --ntasks-per-node=4 sbatch -N4 --ntasks-per-node=8 sbatch -N2 --ntasks-per-node=16
You could use a job_submit plugin to map sbatch -n32 to _one_ of the above task distributions, but you would need to modify the Slurm code for it to use only one of these distributions. Quoting Ole Holm Nielsen <[email protected]>: > > We're currently running a Torque/Maui batch system on our cluster, but > we're evaluating whether to transition to Slurm. Like most other sites, > we have a heterogeneous cluster consisting of several generations of > hardware (all nodes are dual-socket with either 2, 4 or 8 cores/socket). > > One gripe we have about the Maui scheduler is its inability to > dynamically schedule jobs to different types of hardware while > maintaining an optimal layout of tasks. Our MPI jobs don't care how > they're distributed across nodes, but they should use entire nodes for > performance reasons. We do permit multiple jobs per node (shared nodes) > because there's a need for jobs using fewer cores than in an entire node. > > I've studied the Slurm examples in > http://www.schedmd.com/slurmdocs/cpu_management.html#Section4 but I > don't seem to find any mention of using heterogeneous hardware like ours. > > As an example, assume a user wants to submit a 32-task MPI job. With > Torque he would have to submit in one of these mutually exclusive ways > in order to achieve an optimal layout: > qsub -l nodes=8,ppn=4 > qsub -l nodes=4,ppn=8 > qsub -l nodes=2,ppn=16 > If he were to submit to just 32 nodes: > qsub -l nodes=32 > (which would be the simplest for users to understand), the job would get > scattered across irregular sets of nodes in stead of being packed into > an optimal set of nodes, so we unfortunately have to disallow this type > of usage. > > Question: Can Slurm be configured for heterogeneous hardware allowing > users to submit, for example, srun --ntasks=32 and get any one of the > optimal layouts discussed above? > > FYI, we have many different types of jobs in our cluster. Users are > allocating anywhere from 1 core to ~1000 cores per job. > > Thanks, > Ole > > -- > Ole Holm Nielsen > Department of Physics, Technical University of Denmark >
