Quoting Ole Holm Nielsen <[email protected]>: > > On 03-01-2013 18:34, Moe Jette wrote: >> >> Slurm has an option --ntasks-per-node, so there are equivalents to >>> qsub -l nodes=8,ppn=4 >>> qsub -l nodes=4,ppn=8 >>> qsub -l nodes=2,ppn=16 >> like this >> sbatch -N8 --ntasks-per-node=4 >> sbatch -N4 --ntasks-per-node=8 >> sbatch -N2 --ntasks-per-node=16 >> >> You could use a job_submit plugin to map sbatch -n32 to _one_ of the >> above task distributions, but you would need to modify the Slurm code >> for it to use only one of these distributions. > > Moe, thanks for the hint! I heard your impressive presentation at the SC'12
Thanks! > booth and wanted to investigate the option of switching our cluster to Slurm. > > Do I understand you correctly that the node layout would have to be decided > manually with the Slurm scheduler? I.e., Slurm has no way of > automatically and > dynamically mapping jobs to whatever kinds of nodes are free (with different > numbers of cores per node), just like the situation with Torque/Maui? Slurm will try to pack the jobs onto a minimal set of network switches, nodes, boards, and sockets. It also has a "Weight" that can be applied to each node to favor use of some nodes, say nodes with fewer cores or memory. The cluster can be very heterogeneous, with cpu types, memory sizes, GPUs, different operating systems, etc. However unless the job specifies --ntasks-per-node at submit time (or using a Slurm plugin), there is nothing to keep a job from being allocated 7 tasks on one node and 9 tasks on another node, which is what I understand that you want. It would be simple to write a Slurm plugin that would note the user did not explicitly specify --ntasks-per-node and pick a "reasonable" value for the job, but it would be difficult for that to dynamically change with respect to free resources without some Slurm code changes. > Thanks, > Ole > > >> Quoting Ole Holm Nielsen <[email protected]>: >> >>> >>> We're currently running a Torque/Maui batch system on our cluster, but >>> we're evaluating whether to transition to Slurm. Like most other sites, >>> we have a heterogeneous cluster consisting of several generations of >>> hardware (all nodes are dual-socket with either 2, 4 or 8 cores/socket). >>> >>> One gripe we have about the Maui scheduler is its inability to >>> dynamically schedule jobs to different types of hardware while >>> maintaining an optimal layout of tasks. Our MPI jobs don't care how >>> they're distributed across nodes, but they should use entire nodes for >>> performance reasons. We do permit multiple jobs per node (shared nodes) >>> because there's a need for jobs using fewer cores than in an entire node. >>> >>> I've studied the Slurm examples in >>> http://www.schedmd.com/slurmdocs/cpu_management.html#Section4 but I >>> don't seem to find any mention of using heterogeneous hardware like ours. >>> >>> As an example, assume a user wants to submit a 32-task MPI job. With >>> Torque he would have to submit in one of these mutually exclusive ways >>> in order to achieve an optimal layout: >>> qsub -l nodes=8,ppn=4 >>> qsub -l nodes=4,ppn=8 >>> qsub -l nodes=2,ppn=16 >>> If he were to submit to just 32 nodes: >>> qsub -l nodes=32 >>> (which would be the simplest for users to understand), the job would get >>> scattered across irregular sets of nodes in stead of being packed into >>> an optimal set of nodes, so we unfortunately have to disallow this type >>> of usage. >>> >>> Question: Can Slurm be configured for heterogeneous hardware allowing >>> users to submit, for example, srun --ntasks=32 and get any one of the >>> optimal layouts discussed above? >>> >>> FYI, we have many different types of jobs in our cluster. Users are >>> allocating anywhere from 1 core to ~1000 cores per job. >>> >>> Thanks, >>> Ole >>> >>> -- >>> Ole Holm Nielsen >>> Department of Physics, Technical University of Denmark >>> >> >
