On 01/03/2013 04:26 PM, Ralph Castain wrote:
>
> Call me puzzled - are you sure you have Maui configured correctly? Typically,
> you configure it for exclusive node allocation, and then just ask it for a
> number of slots. It doesn't *require* you to specify nodes or ppn - those are
> options. Might be worth checking for advice on that mailing list.
We don't configure exclusive node allocation in Maui because we want to
cater for multiple single-CPU jobs per node.
Though I haven't tested it, I think that with Maui's exclusive node
allocation, if you submit with "qsub -l nodes=32" you would get 32 full
nodes with either 4, 8 or 16 cores per node in our cluster.
Thanks,
Ole
> On Jan 3, 2013, at 6:41 AM, Ole Holm Nielsen <[email protected]>
> wrote:
>
>>
>> We're currently running a Torque/Maui batch system on our cluster, but
>> we're evaluating whether to transition to Slurm. Like most other sites,
>> we have a heterogeneous cluster consisting of several generations of
>> hardware (all nodes are dual-socket with either 2, 4 or 8 cores/socket).
>>
>> One gripe we have about the Maui scheduler is its inability to
>> dynamically schedule jobs to different types of hardware while
>> maintaining an optimal layout of tasks. Our MPI jobs don't care how
>> they're distributed across nodes, but they should use entire nodes for
>> performance reasons. We do permit multiple jobs per node (shared nodes)
>> because there's a need for jobs using fewer cores than in an entire node.
>>
>> I've studied the Slurm examples in
>> http://www.schedmd.com/slurmdocs/cpu_management.html#Section4 but I
>> don't seem to find any mention of using heterogeneous hardware like ours.
>>
>> As an example, assume a user wants to submit a 32-task MPI job. With
>> Torque he would have to submit in one of these mutually exclusive ways
>> in order to achieve an optimal layout:
>> qsub -l nodes=8,ppn=4
>> qsub -l nodes=4,ppn=8
>> qsub -l nodes=2,ppn=16
>> If he were to submit to just 32 nodes:
>> qsub -l nodes=32
>> (which would be the simplest for users to understand), the job would get
>> scattered across irregular sets of nodes in stead of being packed into
>> an optimal set of nodes, so we unfortunately have to disallow this type
>> of usage.
>>
>> Question: Can Slurm be configured for heterogeneous hardware allowing
>> users to submit, for example, srun --ntasks=32 and get any one of the
>> optimal layouts discussed above?
>>
>> FYI, we have many different types of jobs in our cluster. Users are
>> allocating anywhere from 1 core to ~1000 cores per job.
>>
>> Thanks,
>> Ole
>>
>> --
>> Ole Holm Nielsen
>> Department of Physics, Technical University of Denmark
--
Ole Holm Nielsen
Department of Physics, Technical University of Denmark,
Building 307, DK-2800 Kongens Lyngby, Denmark
E-mail: [email protected]
Homepage: http://www.fysik.dtu.dk/~ohnielse/
Tel: (+45) 4525 3187 / Mobile (+45) 5180 1620 / Fax: (+45) 4593 2399
