Andy, See inline below:
Quoting Andy Wettstein <[email protected]>:
>
> On Thu, Jan 03, 2013 at 03:50:05PM -0700, Ole Holm Nielsen wrote:
>>
>> On 03-01-2013 18:34, Moe Jette wrote:
>> >
>> > Slurm has an option --ntasks-per-node, so there are equivalents to
>> >> qsub -l nodes=8,ppn=4
>> >> qsub -l nodes=4,ppn=8
>> >> qsub -l nodes=2,ppn=16
>> > like this
>> > sbatch -N8 --ntasks-per-node=4
>> > sbatch -N4 --ntasks-per-node=8
>> > sbatch -N2 --ntasks-per-node=16
>> >
>> > You could use a job_submit plugin to map sbatch -n32 to _one_ of the
>> > above task distributions, but you would need to modify the Slurm code
>> > for it to use only one of these distributions.
>>
>> Moe, thanks for the hint! I heard your impressive presentation at the SC'12
>> booth and wanted to investigate the option of switching our cluster
>> to Slurm.
>>
>> Do I understand you correctly that the node layout would have to be decided
>> manually with the Slurm scheduler? I.e., Slurm has no way of
>> automatically and
>> dynamically mapping jobs to whatever kinds of nodes are free (with different
>> numbers of cores per node), just like the situation with Torque/Maui?
>
> I don't know of a way to do this currently in slurm either, but I have
> kind of thought that it may be possible to use the topology/tree plugin
> for something like this. The topology/tree plugin allows you to group
> nodes by user defined switches. This is a completely arbitrary mapping,
> so you could say all 8 core nodes are connected to one switch, all 16
> core nodes connected to another, etc. Slurm currently tries to do
> minimize the number of switches used by a job, but will still schedule
> jobs across switches. If there was an option in the topology plugin to
> not schedule across switches then I think it would work. I'm not sure
> how feasible it would be to add an option like that.
User's can specify how many switches they want their job to use and
how long to wait. This is from man sbatch:
--switches=<count>[@<max-time>]
When a tree topology is used, this defines the maximum count of
switches desired for the job allocation and optionally the maxi‐
mum time to wait for that number of switches. If SLURM finds an
allocation containing more switches than the count specified,
the job remains pending until it either finds an allocation with
desired switch count or the time limit expires. It there is no
switch count limit, there is no delay in starting the job.
Acceptable time formats include "minutes", "minutes:seconds",
"hours:minutes:seconds", "days-hours", "days-hours:minutes" and
"days-hours:minutes:seconds". The job's maximum time delay may
be limited by the system administrator using the SchedulerParam‐
eters configuration parameter with the max_switch_wait parameter
option. The default max-time is the max_switch_wait Scheduler‐
Parameter.
> If this is possible it would by very similar to the parallel groups
> functionality that SGE provides, which is actually a quite nice feature.
>
>>
>> Thanks,
>> Ole
>>
>>
>> > Quoting Ole Holm Nielsen <[email protected]>:
>> >
>> >>
>> >> We're currently running a Torque/Maui batch system on our cluster, but
>> >> we're evaluating whether to transition to Slurm. Like most other sites,
>> >> we have a heterogeneous cluster consisting of several generations of
>> >> hardware (all nodes are dual-socket with either 2, 4 or 8 cores/socket).
>> >>
>> >> One gripe we have about the Maui scheduler is its inability to
>> >> dynamically schedule jobs to different types of hardware while
>> >> maintaining an optimal layout of tasks. Our MPI jobs don't care how
>> >> they're distributed across nodes, but they should use entire nodes for
>> >> performance reasons. We do permit multiple jobs per node (shared nodes)
>> >> because there's a need for jobs using fewer cores than in an entire node.
>> >>
>> >> I've studied the Slurm examples in
>> >> http://www.schedmd.com/slurmdocs/cpu_management.html#Section4 but I
>> >> don't seem to find any mention of using heterogeneous hardware like ours.
>> >>
>> >> As an example, assume a user wants to submit a 32-task MPI job. With
>> >> Torque he would have to submit in one of these mutually exclusive ways
>> >> in order to achieve an optimal layout:
>> >> qsub -l nodes=8,ppn=4
>> >> qsub -l nodes=4,ppn=8
>> >> qsub -l nodes=2,ppn=16
>> >> If he were to submit to just 32 nodes:
>> >> qsub -l nodes=32
>> >> (which would be the simplest for users to understand), the job would get
>> >> scattered across irregular sets of nodes in stead of being packed into
>> >> an optimal set of nodes, so we unfortunately have to disallow this type
>> >> of usage.
>> >>
>> >> Question: Can Slurm be configured for heterogeneous hardware allowing
>> >> users to submit, for example, srun --ntasks=32 and get any one of the
>> >> optimal layouts discussed above?
>> >>
>> >> FYI, we have many different types of jobs in our cluster. Users are
>> >> allocating anywhere from 1 core to ~1000 cores per job.
>> >>
>> >> Thanks,
>> >> Ole
>> >>
>> >> --
>> >> Ole Holm Nielsen
>> >> Department of Physics, Technical University of Denmark
>> >>
>> >
>
> --
> andy wettstein
> hpc system administrator
> research computing center
> university of chicago
> 773.702.1104
