[slurm-dev] Re: One CPU always reserved for one GPU

[Lachele Foley]

We do exactly that.  We use the CPUs as the consumable resource rather
than the GPUs for that reason.  We also limit memory use as needed.
You might want to see the configuration issues we ran into and solved
as recorded in the thread at the link below.

https://groups.google.com/forum/#!topic/slurm-devel/x6VaKfrdH5Y


On Tue, Mar 1, 2016 at 1:27 PM, John Desantis  wrote:
>
> Felix,
>
> Although I haven't run into a use-case like yours (yet), my initial
> thought was to use the flag "MaxCPUsPerNode" in your configuration:
>
> 'Maximum number of CPUs on any node available to all jobs from this
> partition.  This can be especially useful to schedule GPUs. For
> example  a  node can  be  associated  with  two Slurm partitions (e.g.
> "cpu" and "gpu") and the partition/queue "cpu" could be limited to
> only a subset of the node’s CPUs, insuring that one or more CPUs would
> be available to jobs in the "gpu" partition/queue.'
>
> HTH,
> John DeSantis
>
>
>
> 2016-03-01 9:05 GMT-05:00 Felix Willenborg 
> :
>> Hey folks,
>>
>> I'm kind of new to SLURM and we're setting it up in our work group with our
>> nodes. Our cluster contains per node 2 GPUs and 12 CPU cores.
>>
>> The GPUs are configured with gres like this :
>> Name=gpu_mem Count=6143
>> Name=gpu File=/dev/nvidia0
>> Name=gpu File=/dev/nvidia1
>> #Name=bandwidth count=4G
>> (Somehow the bandwith plugin isn't available in the repository slurm and I'm
>> getting error messages with that. That's why it's commented out. Is it even
>> necessary?)
>>
>> The nodes are defined like that in the slurm.conf :
>> [...]
>> NodeName=node01 NodeAddr=<...> CPUs=12 RealMemory=128740 Sockets=2
>> CoresPerSocket=6 ThreadsPerCore=1 State=UNKNOWN
>> Gres=gpu:3,gpu_mem:12287#,bandwidth:4G
>>
>>
>> We'd like to have a situation where one CPU is always available for one GPU
>> and only can allocated with one GPU, because we often had the situation that
>> reservations were made where all CPUs were allocated and we couldn't use the
>> GPUs anymore. I searched on the internet and didn't find any similiar cases
>> which could help me. The only thing I found was adding "CPUS=0,1" at the end
>> of every Name=gpu ... in gres.conf. Would this already do it? And if not,
>> what can I do? I've got the feeling that I could solve my problem with SLURM
>> in many ways. We're using SLURM version 14.11.8.
>>
>> Looking forward to some answers!
>>
>> Best wishes,
>> Felix Willenborg



-- 
:-) Lachele
Lachele Foley
CCRC/UGA
Athens, GA USA

[slurm-dev] Re: One CPU always reserved for one GPU

[John Desantis]

Felix,

Although I haven't run into a use-case like yours (yet), my initial
thought was to use the flag "MaxCPUsPerNode" in your configuration:

'Maximum number of CPUs on any node available to all jobs from this
partition.  This can be especially useful to schedule GPUs. For
example  a  node can  be  associated  with  two Slurm partitions (e.g.
"cpu" and "gpu") and the partition/queue "cpu" could be limited to
only a subset of the node’s CPUs, insuring that one or more CPUs would
be available to jobs in the "gpu" partition/queue.'

HTH,
John DeSantis



2016-03-01 9:05 GMT-05:00 Felix Willenborg :
> Hey folks,
>
> I'm kind of new to SLURM and we're setting it up in our work group with our
> nodes. Our cluster contains per node 2 GPUs and 12 CPU cores.
>
> The GPUs are configured with gres like this :
> Name=gpu_mem Count=6143
> Name=gpu File=/dev/nvidia0
> Name=gpu File=/dev/nvidia1
> #Name=bandwidth count=4G
> (Somehow the bandwith plugin isn't available in the repository slurm and I'm
> getting error messages with that. That's why it's commented out. Is it even
> necessary?)
>
> The nodes are defined like that in the slurm.conf :
> [...]
> NodeName=node01 NodeAddr=<...> CPUs=12 RealMemory=128740 Sockets=2
> CoresPerSocket=6 ThreadsPerCore=1 State=UNKNOWN
> Gres=gpu:3,gpu_mem:12287#,bandwidth:4G
>
>
> We'd like to have a situation where one CPU is always available for one GPU
> and only can allocated with one GPU, because we often had the situation that
> reservations were made where all CPUs were allocated and we couldn't use the
> GPUs anymore. I searched on the internet and didn't find any similiar cases
> which could help me. The only thing I found was adding "CPUS=0,1" at the end
> of every Name=gpu ... in gres.conf. Would this already do it? And if not,
> what can I do? I've got the feeling that I could solve my problem with SLURM
> in many ways. We're using SLURM version 14.11.8.
>
> Looking forward to some answers!
>
> Best wishes,
> Felix Willenborg

[slurm-dev] Re: Where should I report bugs/issues ?

[Tim Wickberg]

https://bugs.schedmd.com/ is the correct place to submit bugs.

For bug reports for systems without a support contract, we usually don't 
have time to respond - our work responding to customer-reported issues 
and developing future releases keeps us all quite busy here [1]. 
Continued Slurm development and support is made possible thanks to our 
customers, and we have to focus on their priorities first.


'Severity 6 - No Support Contract' is set automatically on these bugs to 
note that we won't generally be able to respond ourselves.


The source repository is maintained at https://github.com/SchedMD/slurm 
. The official Slurm source distribution is a tarball corresponding to 
the release tag (with a few adjustments to the META and slurm.spec files 
to denote the correct release version). As you noticed, I did turn off 
the Github issue tracker - it doesn't integrate with our internal workflow.


As I've begun to document under 'CONTRIBUTING.md', we do request that 
patches be submitted as an attachment to a bug. Patches submitted under 
the "Contributions" category in our Bugzilla are welcome from anyone, 
customer or not, and we try to provide at least some initial feedback on 
each submission. We highly value the Slurm development community and 
want to encourage additional contribution as best we can.


- Tim

[1] This may be a good time to mention that we're looking for additional 
systems programmers to help out in a support/development role. 
http://www.schedmd.com/#careers


--
Tim Wickberg
Director of Support, SchedMD LLC
Commercial Slurm Development and Support

On 03/01/2016 10:43 AM, Daniel Letai wrote:

Please disregard - slurm on github has "issues" disabled. My mistake.

On 03/01/2016 05:39 PM, Daniel Letai wrote:

Where should I report bugs/issues ? Hi,

I have recently opened a bug in https://bugs.schedmd.com/ but I have
now discovered github https://github.com/SchedMD/slurm also seems
quite active - should I have opened the issue in github?

Whats the correct/preferred way to report bugs/issues ?

[slurm-dev] Re: Where should I report bugs/issues ?

[Daniel Letai]

   body p { margin-bottom: 0cm; margin-top: 0pt; } 
 Please disregard - slurm on github has "issues" disabled. My
 mistake.
 
 On 03/01/2016 05:39 PM, Daniel Letai
   wrote:
   Where should I report bugs/issues ?
   body p { margin-bottom: 0cm; margin-top: 0pt; } 
   Hi,
   
   I have recently opened a bug in https://bugs.schedmd.com/
   but I have now discovered github  https://github.com/SchedMD/slurm
   also seems quite active - should I have opened the issue in
   github?
   
   Whats the correct/preferred way to report bugs/issues ?

[slurm-dev] Where should I report bugs/issues ?

[Daniel Letai]

   body p { margin-bottom: 0cm; margin-top: 0pt; } 
 Hi,
 
 I have recently opened a bug in https://bugs.schedmd.com/ but I have
 now discovered github  https://github.com/SchedMD/slurm also seems
 quite active - should I have opened the issue in github?
 
 Whats the correct/preferred way to report bugs/issues ?

[slurm-dev] Re: User education tools for fair share

[Paul Edmon]

We use the showq tool.  We are looking into fairshare plotting over 
time.  I've been doing it on our test cluster and seeing the evolution 
of fairshare overtime really gives you a good feel for what your usage 
is and what is going on.  We haven't implemented it yet though as we 
would need to set up a data repository for historic fairshare data.  
Still it is on our docket to do.


sshare is a great way of seeing things too, though it can be a bit much 
for your average user.


-Paul Edmon-

On 03/01/2016 04:52 AM, Chris Samuel wrote:

Hi Loris,

On Tue, 1 Mar 2016 12:29:12 AM Loris Bennett wrote:


To help the user understand their current fairshare/priority status, I
usually point them to 'sprio', generally in the following incantation:

sprio -l | sort -nk3

to get the jobs sorted by priority.

Thanks for that, our local version of "showq" already orders waiting jobs by
priority, so I'll direct users to that first, and then to sprio to get a
breakdown of priority.

All the best,
Chris

[slurm-dev] Re: One CPU always reserved for one GPU

[Barbara Krasovec]

For example you can try specifying CPUs with:

NodeName=node001,node002 Name=gpu File=/dev/nvidia0 CPUs=0-3
NodeName=node001,node002 Name=gpu File=/dev/nvidia1 CPUs=4-7

Cheers,
Barbara



On 03/01/2016 03:05 PM, Felix Willenborg wrote:

Hey folks,

I'm kind of new to SLURM and we're setting it up in our work group 
with our nodes. Our cluster contains per node 2 GPUs and 12 CPU cores.


The GPUs are configured with gres like this :
/Name=gpu_mem Count=6143//
//Name=gpu File=/dev/nvidia0 //
//Name=gpu File=/dev/nvidia1 //
//#Name=bandwidth count=4G//
//(Somehow the bandwith plugin isn't available in the repository slurm 
and I'm getting error messages with that. That's why it's commented 
out. Is it even necessary?)


/The nodes are defined like that in the slurm.conf :
/[...]//
//NodeName=node01 NodeAddr=<...> CPUs=12 RealMemory=128740 Sockets=2 
CoresPerSocket=6 ThreadsPerCore=1 State=UNKNOWN 
Gres=gpu:3,gpu_mem:12287#,bandwidth:4G/



We'd like to have a situation where one CPU is always available for 
one GPU and only can allocated with one GPU, because we often had the 
situation that reservations were made where all CPUs were allocated 
and we couldn't use the GPUs anymore. I searched on the internet and 
didn't find any similiar cases which could help me. The only thing I 
found was adding "CPUS=0,1" at the end of every Name=gpu ... in 
gres.conf. Would this already do it? And if not, what can I do? I've 
got the feeling that I could solve my problem with SLURM in many ways. 
We're using SLURM version 14.11.8.


Looking forward to some answers!

Best wishes,
Felix Willenborg

[slurm-dev] Re: scontrol reboot won't reboot reserved nodes?

[Ade Fewings]

Yes, we've seen this on occasion as well.  Seems to be something in interaction 
of node reboot and node state (drained, maint, etc.) - but I haven't quite 
managed to get my head round it yet.

Sorry can't be more help.

~~
Ade


From: Chris Samuel 
Sent: 01 March 2016 09:48
To: slurm-dev
Subject: [slurm-dev] Re: scontrol reboot won't reboot reserved nodes?

On Mon, 29 Feb 2016 11:50:18 PM Uwe Sauter wrote:

> Did you configure the RebootProgram parameter in slurm.conf and is that
> script working? Remember: this script is run on the compute node, therefore
> it must be available on the compute node and must be executable.

Yes, all our clusters are configured with:

RebootProgram=/sbin/reboot

It certainly works, as that's how I rebooted the compute nodes when
"scontrol reboot" wouldn't. ;-)

There's nothing in our syslog, slurmctld.log or slurmd.log's that mentions
anything related to the "scontrol reboot".

All the best,
Chris
--
 Christopher SamuelSenior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/  http://twitter.com/vlsci


   [HPC Wales - www.hpcwales.co.uk] 
[ISO 9001] 

[HPC Wales on Twitter] [HPC Wales on LinkedIn] 




The contents of this email and any files transmitted with it are confidential 
and intended solely for the named addressee only.  Unless you are the named 
addressee (or authorised to receive this on their behalf) you may not copy it 
or use it, or disclose it to anyone else.  If you have received this email in 
error, please notify the sender by email or telephone.  All emails sent by High 
Performance Computing Wales have been checked using an Anti-Virus system.  We 
would advise you to run your own virus check before opening any attachments 
received as we will not in any event accept any liability whatsoever, once an 
email and/or attachment is received.

High Performance Computing Wales is a private limited company incorporated in 
Wales on 8 March 2010 as company number 07181701.

Our registered office is at Ty Menai, Ffordd Penlan, Parc Menai Business Park, 
Bangor, Gwynedd. LL57 4HJ. UK.

High Performance Computing Wales is part funded by the European Regional 
Development Fund through the Welsh Government.

[slurm-dev] Re: scontrol reboot won't reboot reserved nodes?

[Uwe Sauter]

Hm, I'm not sure how I can help then but I do have a separate script configured 
as we have problems with Lustre not being quick
enough on a reboot. So I first unmount Lustre, remove Infiniband modules and 
then do a "reboot -f".

Am 01.03.2016 um 10:49 schrieb Chris Samuel:
> 
> On Mon, 29 Feb 2016 11:50:18 PM Uwe Sauter wrote:
> 
>> Did you configure the RebootProgram parameter in slurm.conf and is that
>> script working? Remember: this script is run on the compute node, therefore
>> it must be available on the compute node and must be executable.
> 
> Yes, all our clusters are configured with:
> 
> RebootProgram=/sbin/reboot
> 
> It certainly works, as that's how I rebooted the compute nodes when
> "scontrol reboot" wouldn't. ;-)
> 
> There's nothing in our syslog, slurmctld.log or slurmd.log's that mentions
> anything related to the "scontrol reboot".
> 
> All the best,
> Chris
> 

[slurm-dev] Re: User education tools for fair share

[Chris Samuel]

Hi Loris,

On Tue, 1 Mar 2016 12:29:12 AM Loris Bennett wrote:

> To help the user understand their current fairshare/priority status, I
> usually point them to 'sprio', generally in the following incantation:
> 
> sprio -l | sort -nk3
> 
> to get the jobs sorted by priority.

Thanks for that, our local version of "showq" already orders waiting jobs by 
priority, so I'll direct users to that first, and then to sprio to get a 
breakdown of priority.

All the best,
Chris
-- 
 Christopher SamuelSenior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/  http://twitter.com/vlsci

[slurm-dev] Re: scontrol reboot won't reboot reserved nodes?

[Chris Samuel]

On Mon, 29 Feb 2016 11:50:18 PM Uwe Sauter wrote:

> Did you configure the RebootProgram parameter in slurm.conf and is that
> script working? Remember: this script is run on the compute node, therefore
> it must be available on the compute node and must be executable.

Yes, all our clusters are configured with:

RebootProgram=/sbin/reboot

It certainly works, as that's how I rebooted the compute nodes when
"scontrol reboot" wouldn't. ;-)

There's nothing in our syslog, slurmctld.log or slurmd.log's that mentions
anything related to the "scontrol reboot".

All the best,
Chris
-- 
 Christopher SamuelSenior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/  http://twitter.com/vlsci

[slurm-dev] Re: User education tools for fair share

[Loris Bennett]

Hi Chris,

Christopher Samuel 
writes:

> Hi folks,
>
> We've just migrated to fairshare and one of the things we've been
> puzzling over is how to show users what their fairshare status is.
>
> With quotas it was pretty easy, we had a bar-graph showing how far
> through the quarter they were, and another bar-graph per project that
> showed the percentage of quota burnt so far this quarter.
>
> After 6 years of running like that it's hurting our heads to think
> differently about how to display it.
>
> It's also complicated as we are using Fair Tree (thanks Ryan et. al!)
> and so we think we should show users their priorities back up the tree.
>
> I'm even wondering if we should not worry about showing them that and
> instead just educate them about the priority of queued jobs instead.
>
> How do other sites handle this?
>
> All the best,
> Chris

We use fairshare without Fair Tree and with all users having the same
number of shares.  Occasionally we have users complaining about the
system being unfair, particularly when other users are able to profit
from backfill.  The problem is that users often just look at the number
of jobs someone is able to run, regardless of the resources being used.

To help the user understand their current fairshare/priority status, I
usually point them to 'sprio', generally in the following incantation:

sprio -l | sort -nk3

to get the jobs sorted by priority.

Cheers,

Loris

-- 
Dr. Loris Bennett (Mr.)
ZEDAT, Freie Universität Berlin Email loris.benn...@fu-berlin.de