On Jan 4, 2013, at 8:25 AM, Andy Wettstein <[email protected]> wrote:
> > On Thu, Jan 03, 2013 at 03:50:05PM -0700, Ole Holm Nielsen wrote: >> >> On 03-01-2013 18:34, Moe Jette wrote: >>> >>> Slurm has an option --ntasks-per-node, so there are equivalents to >>>> qsub -l nodes=8,ppn=4 >>>> qsub -l nodes=4,ppn=8 >>>> qsub -l nodes=2,ppn=16 >>> like this >>> sbatch -N8 --ntasks-per-node=4 >>> sbatch -N4 --ntasks-per-node=8 >>> sbatch -N2 --ntasks-per-node=16 >>> >>> You could use a job_submit plugin to map sbatch -n32 to _one_ of the >>> above task distributions, but you would need to modify the Slurm code >>> for it to use only one of these distributions. >> >> Moe, thanks for the hint! I heard your impressive presentation at the SC'12 >> booth and wanted to investigate the option of switching our cluster to Slurm. >> >> Do I understand you correctly that the node layout would have to be decided >> manually with the Slurm scheduler? I.e., Slurm has no way of automatically >> and >> dynamically mapping jobs to whatever kinds of nodes are free (with different >> numbers of cores per node), just like the situation with Torque/Maui? > > I don't know of a way to do this currently in slurm either, but I have > kind of thought that it may be possible to use the topology/tree plugin > for something like this. The topology/tree plugin allows you to group > nodes by user defined switches. This is a completely arbitrary mapping, > so you could say all 8 core nodes are connected to one switch, all 16 > core nodes connected to another, etc. Slurm currently tries to do > minimize the number of switches used by a job, but will still schedule > jobs across switches. If there was an option in the topology plugin to > not schedule across switches then I think it would work. I'm not sure > how feasible it would be to add an option like that. > > If this is possible it would by very similar to the parallel groups > functionality that SGE provides, which is actually a quite nice feature. FWIW: I believe we have a mapper in Open MPI that does what you want - i.e., it looks at the number of available cpus on each node, and maps the processes to maximize the number of procs co-located on nodes. In your described case, it would tend to favor the 16ppn nodes as that would provide the best MPI performance, then move to the 8ppn nodes, etc. I can look at porting it to slurm as a job_submit plugin if people would find it useful - the algorithm isn't too complex. I could easily tweak it to avoid unnecessarily subdividing nodes - e.g., if a job needs 4 tasks, then don't take part of a 16ppn node if a 4ppn one is available. > >> >> Thanks, >> Ole >> >> >>> Quoting Ole Holm Nielsen <[email protected]>: >>> >>>> >>>> We're currently running a Torque/Maui batch system on our cluster, but >>>> we're evaluating whether to transition to Slurm. Like most other sites, >>>> we have a heterogeneous cluster consisting of several generations of >>>> hardware (all nodes are dual-socket with either 2, 4 or 8 cores/socket). >>>> >>>> One gripe we have about the Maui scheduler is its inability to >>>> dynamically schedule jobs to different types of hardware while >>>> maintaining an optimal layout of tasks. Our MPI jobs don't care how >>>> they're distributed across nodes, but they should use entire nodes for >>>> performance reasons. We do permit multiple jobs per node (shared nodes) >>>> because there's a need for jobs using fewer cores than in an entire node. >>>> >>>> I've studied the Slurm examples in >>>> http://www.schedmd.com/slurmdocs/cpu_management.html#Section4 but I >>>> don't seem to find any mention of using heterogeneous hardware like ours. >>>> >>>> As an example, assume a user wants to submit a 32-task MPI job. With >>>> Torque he would have to submit in one of these mutually exclusive ways >>>> in order to achieve an optimal layout: >>>> qsub -l nodes=8,ppn=4 >>>> qsub -l nodes=4,ppn=8 >>>> qsub -l nodes=2,ppn=16 >>>> If he were to submit to just 32 nodes: >>>> qsub -l nodes=32 >>>> (which would be the simplest for users to understand), the job would get >>>> scattered across irregular sets of nodes in stead of being packed into >>>> an optimal set of nodes, so we unfortunately have to disallow this type >>>> of usage. >>>> >>>> Question: Can Slurm be configured for heterogeneous hardware allowing >>>> users to submit, for example, srun --ntasks=32 and get any one of the >>>> optimal layouts discussed above? >>>> >>>> FYI, we have many different types of jobs in our cluster. Users are >>>> allocating anywhere from 1 core to ~1000 cores per job. >>>> >>>> Thanks, >>>> Ole >>>> >>>> -- >>>> Ole Holm Nielsen >>>> Department of Physics, Technical University of Denmark >>>> >>> > > -- > andy wettstein > hpc system administrator > research computing center > university of chicago > 773.702.1104
