[slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load

[Benjamin Redling]

re hi,

your script will occasionally fail because the number of fields in the
output of "uptime" is variable.
I was reminded by this one:
http://stackoverflow.com/questions/11735211/get-last-five-minutes-load-average-using-ksh-with-uptime

Even more a reason to use /proc...

Regards,
Benjamin

Am 21.03.2017 um 21:15 schrieb kesim:
> There is an error in the script. It could be:
> 
> scontrol update node=your_node_name WEIGHT=`echo 100*$(uptime | awk
> '{print $12}')/1 | bc`
> 
> 
> On Tue, Mar 21, 2017 at 8:41 PM, kesim  > wrote:
> 
> Dear SLURM Users,
> 
> My response here is for those who are trying to solve the simple
> problem of nodes ordering according to the CPU load. Actually,
> Markus was right and he gave me the idea (THANKS!!!)
> The solution is not pretty but it works and it has a lot of
> flexibility. Just put into crone a script:
>  
> #!/bin/sh
> scontrol update node=your_node_name WEIGHT=`echo 100*$(uptime | awk
> -F'[, ]' '{print $21}')/1 | bc`
> 
> Best Regards,
> 
> Ketiw
> 
> 
> 
> 
> On Mon, Mar 20, 2017 at 3:31 PM, Markus Koeberl
> > wrote:
> 
> 
> On Monday 20 March 2017 05:38:29 Christopher Samuel wrote:
> >
> > On 19/03/17 23:25, kesim wrote:
> >
> > > I have 11 nodes and declared 7 CPUs per node. My setup is
> such that all
> > > desktop belongs to group members who are using them mainly
> as graphics
> > > stations. Therefore from time to time an application is
> requesting high
> > > CPU usage.
> >
> > In this case I would suggest you carve off 3 cores via cgroups for
> > interactive users and give Slurm the other 7 to parcel out to
> jobs by
> > ensuring that Slurm starts within a cgroup dedicated to those
> 7 cores..
> >
> > This is similar to the "boot CPU set" concept that SGI came up
> with (at
> > least I've not come across people doing that before them).
> >
> > To be fair this is not really Slurm's problem to solve, Linux
> gives you
> > the tools to do this already, it's just that people don't
> realise that
> > you can use cgroups to do this.
> >
> > Your use case is valid, but it isn't really HPC, and you can't
> really
> > blame Slurm for not catering to this.  It can use cgroups to
> partition
> > cores to jobs precisely so it doesn't need to care what the
> load average
> > is - it knows the kernel is ensuring the cores the jobs want
> are not
> > being stomped on by other tasks.
> 
> You could additionally define a higher "Weight" value for a host
> if you know that the load is usually higher on it than on the
> others.
> 
> 
> regards
> Markus Köberl
> --
> Markus Koeberl
> Graz University of Technology
> Signal Processing and Speech Communication Laboratory
> E-mail: markus.koeb...@tugraz.at 
> 
> 
> 


-- 
FSU Jena | JULIELab.de/Staff/Benjamin+Redling.html
vox: +49 3641 9 44323 | fax: +49 3641 9 44321

[slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load

[Benjamin Redling]

Hi,

if you don't want to depend on the whitespaces in the output of "uptime"
(the number of fields depends on a locale) you can improve that via "awk
'{print $3}' /proc/loadavg" (for the 15min avg) -- it's always better to
avoid programmatically accessing output made for humans as long as possible.

Nice hack anyway!

Regards,
Benjamin


Am 21.03.2017 um 21:15 schrieb kesim:
> There is an error in the script. It could be:
> 
> scontrol update node=your_node_name WEIGHT=`echo 100*$(uptime | awk
> '{print $12}')/1 | bc`
> 
> 
> On Tue, Mar 21, 2017 at 8:41 PM, kesim  > wrote:
> 
> Dear SLURM Users,
> 
> My response here is for those who are trying to solve the simple
> problem of nodes ordering according to the CPU load. Actually,
> Markus was right and he gave me the idea (THANKS!!!)
> The solution is not pretty but it works and it has a lot of
> flexibility. Just put into crone a script:
>  
> #!/bin/sh
> scontrol update node=your_node_name WEIGHT=`echo 100*$(uptime | awk
> -F'[, ]' '{print $21}')/1 | bc`
> 
> Best Regards,
> 
> Ketiw
> 
> 
> 
> 
> On Mon, Mar 20, 2017 at 3:31 PM, Markus Koeberl
> > wrote:
> 
> 
> On Monday 20 March 2017 05:38:29 Christopher Samuel wrote:
> >
> > On 19/03/17 23:25, kesim wrote:
> >
> > > I have 11 nodes and declared 7 CPUs per node. My setup is
> such that all
> > > desktop belongs to group members who are using them mainly
> as graphics
> > > stations. Therefore from time to time an application is
> requesting high
> > > CPU usage.
> >
> > In this case I would suggest you carve off 3 cores via cgroups for
> > interactive users and give Slurm the other 7 to parcel out to
> jobs by
> > ensuring that Slurm starts within a cgroup dedicated to those
> 7 cores..
> >
> > This is similar to the "boot CPU set" concept that SGI came up
> with (at
> > least I've not come across people doing that before them).
> >
> > To be fair this is not really Slurm's problem to solve, Linux
> gives you
> > the tools to do this already, it's just that people don't
> realise that
> > you can use cgroups to do this.
> >
> > Your use case is valid, but it isn't really HPC, and you can't
> really
> > blame Slurm for not catering to this.  It can use cgroups to
> partition
> > cores to jobs precisely so it doesn't need to care what the
> load average
> > is - it knows the kernel is ensuring the cores the jobs want
> are not
> > being stomped on by other tasks.
> 
> You could additionally define a higher "Weight" value for a host
> if you know that the load is usually higher on it than on the
> others.
> 
> 
> regards
> Markus Köberl
> --
> Markus Koeberl
> Graz University of Technology
> Signal Processing and Speech Communication Laboratory
> E-mail: markus.koeb...@tugraz.at 
> 
> 
> 


-- 
FSU Jena | JULIELab.de/Staff/Benjamin+Redling.html
vox: +49 3641 9 44323 | fax: +49 3641 9 44321

[slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load

[kesim]

There is an error in the script. It could be:

scontrol update node=your_node_name WEIGHT=`echo 100*$(uptime | awk '{print
$12}')/1 | bc`


On Tue, Mar 21, 2017 at 8:41 PM, kesim  wrote:

> Dear SLURM Users,
>
> My response here is for those who are trying to solve the simple problem
> of nodes ordering according to the CPU load. Actually, Markus was right and
> he gave me the idea (THANKS!!!)
> The solution is not pretty but it works and it has a lot of flexibility.
> Just put into crone a script:
>
> #!/bin/sh
> scontrol update node=your_node_name WEIGHT=`echo 100*$(uptime | awk -F'[,
> ]' '{print $21}')/1 | bc`
>
> Best Regards,
>
> Ketiw
>
>
>
>
> On Mon, Mar 20, 2017 at 3:31 PM, Markus Koeberl 
> wrote:
>
>>
>> On Monday 20 March 2017 05:38:29 Christopher Samuel wrote:
>> >
>> > On 19/03/17 23:25, kesim wrote:
>> >
>> > > I have 11 nodes and declared 7 CPUs per node. My setup is such that
>> all
>> > > desktop belongs to group members who are using them mainly as graphics
>> > > stations. Therefore from time to time an application is requesting
>> high
>> > > CPU usage.
>> >
>> > In this case I would suggest you carve off 3 cores via cgroups for
>> > interactive users and give Slurm the other 7 to parcel out to jobs by
>> > ensuring that Slurm starts within a cgroup dedicated to those 7 cores..
>> >
>> > This is similar to the "boot CPU set" concept that SGI came up with (at
>> > least I've not come across people doing that before them).
>> >
>> > To be fair this is not really Slurm's problem to solve, Linux gives you
>> > the tools to do this already, it's just that people don't realise that
>> > you can use cgroups to do this.
>> >
>> > Your use case is valid, but it isn't really HPC, and you can't really
>> > blame Slurm for not catering to this.  It can use cgroups to partition
>> > cores to jobs precisely so it doesn't need to care what the load average
>> > is - it knows the kernel is ensuring the cores the jobs want are not
>> > being stomped on by other tasks.
>>
>> You could additionally define a higher "Weight" value for a host if you
>> know that the load is usually higher on it than on the others.
>>
>>
>> regards
>> Markus Köberl
>> --
>> Markus Koeberl
>> Graz University of Technology
>> Signal Processing and Speech Communication Laboratory
>> E-mail: markus.koeb...@tugraz.at
>>
>
>

[slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load

[kesim]

Dear SLURM Users,

My response here is for those who are trying to solve the simple problem of
nodes ordering according to the CPU load. Actually, Markus was right and he
gave me the idea (THANKS!!!)
The solution is not pretty but it works and it has a lot of flexibility.
Just put into crone a script:

#!/bin/sh
scontrol update node=your_node_name WEIGHT=`echo 100*$(uptime | awk -F'[,
]' '{print $21}')/1 | bc`

Best Regards,

Ketiw




On Mon, Mar 20, 2017 at 3:31 PM, Markus Koeberl 
wrote:

>
> On Monday 20 March 2017 05:38:29 Christopher Samuel wrote:
> >
> > On 19/03/17 23:25, kesim wrote:
> >
> > > I have 11 nodes and declared 7 CPUs per node. My setup is such that all
> > > desktop belongs to group members who are using them mainly as graphics
> > > stations. Therefore from time to time an application is requesting high
> > > CPU usage.
> >
> > In this case I would suggest you carve off 3 cores via cgroups for
> > interactive users and give Slurm the other 7 to parcel out to jobs by
> > ensuring that Slurm starts within a cgroup dedicated to those 7 cores..
> >
> > This is similar to the "boot CPU set" concept that SGI came up with (at
> > least I've not come across people doing that before them).
> >
> > To be fair this is not really Slurm's problem to solve, Linux gives you
> > the tools to do this already, it's just that people don't realise that
> > you can use cgroups to do this.
> >
> > Your use case is valid, but it isn't really HPC, and you can't really
> > blame Slurm for not catering to this.  It can use cgroups to partition
> > cores to jobs precisely so it doesn't need to care what the load average
> > is - it knows the kernel is ensuring the cores the jobs want are not
> > being stomped on by other tasks.
>
> You could additionally define a higher "Weight" value for a host if you
> know that the load is usually higher on it than on the others.
>
>
> regards
> Markus Köberl
> --
> Markus Koeberl
> Graz University of Technology
> Signal Processing and Speech Communication Laboratory
> E-mail: markus.koeb...@tugraz.at
>

[slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load

[Markus Koeberl]

On Monday 20 March 2017 05:38:29 Christopher Samuel wrote:
> 
> On 19/03/17 23:25, kesim wrote:
> 
> > I have 11 nodes and declared 7 CPUs per node. My setup is such that all
> > desktop belongs to group members who are using them mainly as graphics
> > stations. Therefore from time to time an application is requesting high
> > CPU usage.
> 
> In this case I would suggest you carve off 3 cores via cgroups for
> interactive users and give Slurm the other 7 to parcel out to jobs by
> ensuring that Slurm starts within a cgroup dedicated to those 7 cores..
> 
> This is similar to the "boot CPU set" concept that SGI came up with (at
> least I've not come across people doing that before them).
> 
> To be fair this is not really Slurm's problem to solve, Linux gives you
> the tools to do this already, it's just that people don't realise that
> you can use cgroups to do this.
> 
> Your use case is valid, but it isn't really HPC, and you can't really
> blame Slurm for not catering to this.  It can use cgroups to partition
> cores to jobs precisely so it doesn't need to care what the load average
> is - it knows the kernel is ensuring the cores the jobs want are not
> being stomped on by other tasks.

You could additionally define a higher "Weight" value for a host if you know 
that the load is usually higher on it than on the others. 


regards
Markus Köberl
-- 
Markus Koeberl
Graz University of Technology
Signal Processing and Speech Communication Laboratory
E-mail: markus.koeb...@tugraz.at

[slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load

[Christopher Samuel]

On 19/03/17 23:25, kesim wrote:

> I have 11 nodes and declared 7 CPUs per node. My setup is such that all
> desktop belongs to group members who are using them mainly as graphics
> stations. Therefore from time to time an application is requesting high
> CPU usage.

In this case I would suggest you carve off 3 cores via cgroups for
interactive users and give Slurm the other 7 to parcel out to jobs by
ensuring that Slurm starts within a cgroup dedicated to those 7 cores..

This is similar to the "boot CPU set" concept that SGI came up with (at
least I've not come across people doing that before them).

To be fair this is not really Slurm's problem to solve, Linux gives you
the tools to do this already, it's just that people don't realise that
you can use cgroups to do this.

Your use case is valid, but it isn't really HPC, and you can't really
blame Slurm for not catering to this.  It can use cgroups to partition
cores to jobs precisely so it doesn't need to care what the load average
is - it knows the kernel is ensuring the cores the jobs want are not
being stomped on by other tasks.

Best of luck!
Chris
-- 
 Christopher SamuelSenior Systems Administrator
 Melbourne Bioinformatics - The University of Melbourne
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545

[slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load

[kesim]

I have 11 nodes and declared 7 CPUs per node. My setup is such that all
desktop belongs to group members who are using them mainly as graphics
stations. Therefore from time to time an application is requesting high CPU
usage. Firefox can do it easily. We also have applications which were
compiled with intel MPI and the whole setup is mainly for them. I would
like my scheduler to fully fill nodes with tasks but starting from idling
nodes. Let say I looked a the CPU load of my nodes (sinfo -N -o '%N %O %C
will do that)  and since 2 nodes have load ~2 (usually it more or less
means that they use 2 processors at 100%) I want to use 73 instead 77
available processors and my simple minded understanding is that in the node
which have CPU load ~2 the two processors will be the last to allocate even
though they are technically available. This what a scheduler should do on
its own without my intervention. However sadly it is not what happens. If I
request 73 processors the scheduler does not take into account the real CPU
load and it is filling the nodes alphabetically. Since sinfo is aware of
the CPU load slurm should take it into account when filling nodes and this
is a serious bug that it is not doing that.
I use slurm 17.02.1-2 in the Ubutntu 16.04 environment.



On Sun, Mar 19, 2017 at 11:26 AM, TO_Webmaster <luftha...@gmail.com> wrote:

>
> Please remember that might lead to a huge waste of resources. Imagine
> you have a cluster with 10 nodes with 10 cores eachs. Then somebody
> submits 10 jobs requesting 1 core per job. If I understand you
> correctly, you would like to see one job per node then? Now imagine
> someone else submits 9 jobs requesting the nodes exclusively. Then
> none of these 9 jobs can start, because there is one job with one core
> on each node. If the former 10 jobs had been packed on one node, all
> of the latter 9 jobs could have started immediately.
>
> 2017-03-18 19:06 GMT+01:00 kesim <ketiw...@gmail.com>:
> > Dear John,
> >
> > Thank you for your answer. Obviously you are right that I could slurm up
> > everything and thus avoid the issue and your points are taken. However, I
> > still insist that it is a serious bug not to take into account the actual
> > CPU load when the scheduler submit a job regardless whose fault it is
> that a
> > non-slurm job is running. I would not suspect that from even simplest
> > scheduler and if I had such prior knowledge I would not invest so much
> time
> > and effort  to setup slurm.
> > Best regards,
> >
> > Ketiw
> >
> > On Sat, Mar 18, 2017 at 5:42 PM, John Hearns <john.hea...@xma.co.uk>
> wrote:
> >>
> >>
> >> Kesim,
> >>
> >> what you are saying is that Slurm schedukes tasks based on the number of
> >> allocated CPUs. Rather than the actual load factor on the server.
> >> As I recall Gridengine actually used the load factor.
> >>
> >> However you comment that "users run programs on the nodes" and "the
> slurm
> >> is aware about the load of non-slurm jobs"
> >> IMHO, in any well-run HPC setup any user running jobs without using the
> >> scheduler would have their fingers broken. or at least bruised using the
> >> clue stick.
> >>
> >> Seriously, three points:
> >>
> >> a) tell users to use 'salloc' and 'srun'  to run interactive jobs. They
> >> can easily open a Bash session on a compute node and do what they like.
> >> Under the Slurm scheduler.
> >>
> >> b) implement the pam-slurm PAMmodule. It is a few minutes work. This
> means
> >> your users cannot go behind the sluem scheduler and log into the nodes
> >>
> >> c) on Bright clusters which I configure, you have a healtcheck running
> >> which wans you when a user is detected as logging in withotu using Slurm
> >>
> >>
> >> Seriously again. You have implemented an HPC infrastructure, and have
> gone
> >> to the time and effort to implement a batch scheduling system.
> >> A batch scheduler can be adapted to let your users do their jobs,
> >> including interactive shell sessions and remote visualization sessions.
> >> Do not let the users ride roughshod over you.
> >>
> >> 
> >> From: kesim [ketiw...@gmail.com]
> >> Sent: 18 March 2017 16:16
> >> To: slurm-dev
> >> Subject: [slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load
> >>
> >> Unbelievable but it seems that nobody knows how to do that. It is
> >> astonishing that such sophisticated system fails with such simple
> problem.
> >> The slu

[slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load

[TO_Webmaster]

Please remember that might lead to a huge waste of resources. Imagine
you have a cluster with 10 nodes with 10 cores eachs. Then somebody
submits 10 jobs requesting 1 core per job. If I understand you
correctly, you would like to see one job per node then? Now imagine
someone else submits 9 jobs requesting the nodes exclusively. Then
none of these 9 jobs can start, because there is one job with one core
on each node. If the former 10 jobs had been packed on one node, all
of the latter 9 jobs could have started immediately.

2017-03-18 19:06 GMT+01:00 kesim <ketiw...@gmail.com>:
> Dear John,
>
> Thank you for your answer. Obviously you are right that I could slurm up
> everything and thus avoid the issue and your points are taken. However, I
> still insist that it is a serious bug not to take into account the actual
> CPU load when the scheduler submit a job regardless whose fault it is that a
> non-slurm job is running. I would not suspect that from even simplest
> scheduler and if I had such prior knowledge I would not invest so much time
> and effort  to setup slurm.
> Best regards,
>
> Ketiw
>
> On Sat, Mar 18, 2017 at 5:42 PM, John Hearns <john.hea...@xma.co.uk> wrote:
>>
>>
>> Kesim,
>>
>> what you are saying is that Slurm schedukes tasks based on the number of
>> allocated CPUs. Rather than the actual load factor on the server.
>> As I recall Gridengine actually used the load factor.
>>
>> However you comment that "users run programs on the nodes" and "the slurm
>> is aware about the load of non-slurm jobs"
>> IMHO, in any well-run HPC setup any user running jobs without using the
>> scheduler would have their fingers broken. or at least bruised using the
>> clue stick.
>>
>> Seriously, three points:
>>
>> a) tell users to use 'salloc' and 'srun'  to run interactive jobs. They
>> can easily open a Bash session on a compute node and do what they like.
>> Under the Slurm scheduler.
>>
>> b) implement the pam-slurm PAMmodule. It is a few minutes work. This means
>> your users cannot go behind the sluem scheduler and log into the nodes
>>
>> c) on Bright clusters which I configure, you have a healtcheck running
>> which wans you when a user is detected as logging in withotu using Slurm
>>
>>
>> Seriously again. You have implemented an HPC infrastructure, and have gone
>> to the time and effort to implement a batch scheduling system.
>> A batch scheduler can be adapted to let your users do their jobs,
>> including interactive shell sessions and remote visualization sessions.
>> Do not let the users ride roughshod over you.
>>
>> 
>> From: kesim [ketiw...@gmail.com]
>> Sent: 18 March 2017 16:16
>> To: slurm-dev
>> Subject: [slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load
>>
>> Unbelievable but it seems that nobody knows how to do that. It is
>> astonishing that such sophisticated system fails with such simple problem.
>> The slurm is aware about the cpu load of non-slurm jobs but it does not use
>> the info. My original understanding of LLN was apparently correct. I can
>> practically kill the CPUs on particular node with nonslurm tasks but slurm
>> will diligently submit 7 jobs to this node leaving other idling.  I consider
>> this as a serious bug of this program.
>>
>>
>> On Fri, Mar 17, 2017 at 10:32 AM, kesim
>> <ketiw...@gmail.com<mailto:ketiw...@gmail.com>> wrote:
>> Dear All,
>> Yesterday I did some tests and it seemed that the scheduling is following
>> CPU load but I was wrong.
>> My configuration is at the moment:
>> SelectType=select/cons_res
>> SelectTypeParameters=CR_CPU,CR_LLN
>>
>> Today I submitted 70 threaded jobs to the queue and here is the CPU_LOAD
>> info
>> node1 0.08  7/0/0/7
>> node20.01  7/0/0/7
>> node30.00  7/0/0/7
>> node42.97  7/0/0/7
>> node5   0.00  7/0/0/7
>> node6 0.01  7/0/0/7
>> node7  0.00  7/0/0/7
>> node8   0.05  7/0/0/7
>> node90.07  7/0/0/7
>> node100.38  7/0/0/7
>> node11 0.01  0/7/0/7
>> As you can see it allocated 7 CPUs on node 4 with CPU_LOAD 2.97 and 0 CPUs
>> on idling node11. Why such simple thing is not a default? What am I
>> missing???
>>
>> On Thu, Mar 16, 2017 at 7:53 PM, kesim
>> <ketiw...@gmail.com<mailto:ketiw...@gmail.com>> wrote:
>> Than you for great suggestion. It is working! Howev

[slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load

[John Hearns]

Kesim,
   Touche Sir. I agree with you.


From: kesim [ketiw...@gmail.com]
Sent: 18 March 2017 18:06
To: slurm-dev
Subject: [slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load

Dear John,

Thank you for your answer. Obviously you are right that I could slurm up 
everything and thus avoid the issue and your points are taken. However, I still 
insist that it is a serious bug not to take into account the actual CPU load 
when the scheduler submit a job regardless whose fault it is that a non-slurm 
job is running. I would not suspect that from even simplest scheduler and if I 
had such prior knowledge I would not invest so much time and effort  to setup 
slurm.
Best regards,

Ketiw

On Sat, Mar 18, 2017 at 5:42 PM, John Hearns 
<john.hea...@xma.co.uk<mailto:john.hea...@xma.co.uk>> wrote:

Kesim,

what you are saying is that Slurm schedukes tasks based on the number of 
allocated CPUs. Rather than the actual load factor on the server.
As I recall Gridengine actually used the load factor.

However you comment that "users run programs on the nodes" and "the slurm is 
aware about the load of non-slurm jobs"
IMHO, in any well-run HPC setup any user running jobs without using the 
scheduler would have their fingers broken. or at least bruised using the clue 
stick.

Seriously, three points:

a) tell users to use 'salloc' and 'srun'  to run interactive jobs. They can 
easily open a Bash session on a compute node and do what they like. Under the 
Slurm scheduler.

b) implement the pam-slurm PAMmodule. It is a few minutes work. This means your 
users cannot go behind the sluem scheduler and log into the nodes

c) on Bright clusters which I configure, you have a healtcheck running which 
wans you when a user is detected as logging in withotu using Slurm


Seriously again. You have implemented an HPC infrastructure, and have gone to 
the time and effort to implement a batch scheduling system.
A batch scheduler can be adapted to let your users do their jobs, including 
interactive shell sessions and remote visualization sessions.
Do not let the users ride roughshod over you.


From: kesim [ketiw...@gmail.com<mailto:ketiw...@gmail.com>]
Sent: 18 March 2017 16:16
To: slurm-dev
Subject: [slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load

Unbelievable but it seems that nobody knows how to do that. It is astonishing 
that such sophisticated system fails with such simple problem. The slurm is 
aware about the cpu load of non-slurm jobs but it does not use the info. My 
original understanding of LLN was apparently correct. I can practically kill 
the CPUs on particular node with nonslurm tasks but slurm will diligently 
submit 7 jobs to this node leaving other idling.  I consider this as a serious 
bug of this program.


On Fri, Mar 17, 2017 at 10:32 AM, kesim 
<ketiw...@gmail.com<mailto:ketiw...@gmail.com><mailto:ketiw...@gmail.com<mailto:ketiw...@gmail.com>>>
 wrote:
Dear All,
Yesterday I did some tests and it seemed that the scheduling is following CPU 
load but I was wrong.
My configuration is at the moment:
SelectType=select/cons_res
SelectTypeParameters=CR_CPU,CR_LLN

Today I submitted 70 threaded jobs to the queue and here is the CPU_LOAD info
node1 0.08  7/0/0/7
node20.01  7/0/0/7
node30.00  7/0/0/7
node42.97  7/0/0/7
node5   0.00  7/0/0/7
node6 0.01  7/0/0/7
node7  0.00  7/0/0/7
node8   0.05  7/0/0/7
node90.07  7/0/0/7
node100.38  7/0/0/7
node11 0.01  0/7/0/7
As you can see it allocated 7 CPUs on node 4 with CPU_LOAD 2.97 and 0 CPUs on 
idling node11. Why such simple thing is not a default? What am I missing???

On Thu, Mar 16, 2017 at 7:53 PM, kesim 
<ketiw...@gmail.com<mailto:ketiw...@gmail.com><mailto:ketiw...@gmail.com<mailto:ketiw...@gmail.com>>>
 wrote:
Than you for great suggestion. It is working! However the description of CR_LLN 
is misleading "Schedule resources to jobs on the least loaded nodes (based upon 
the number of idle CPUs)" Which I understood that if the two nodes has not 
fully allocated CPUs  the node with smaller number of allocated CPUs will take 
precedence. Therefore the bracketed comment should be removed from the 
description.

On Thu, Mar 16, 2017 at 6:24 PM, Paul Edmon 
<ped...@cfa.harvard.edu<mailto:ped...@cfa.harvard.edu><mailto:ped...@cfa.harvard.edu<mailto:ped...@cfa.harvard.edu>>>
 wrote:

You should look at LLN (least loaded nodes):

https://slurm.schedmd.com/slurm.conf.html

That should do what you want.

-Paul Edmon-

On 03/16/2017 12:54 PM, kesim wrote:

-- Forwarded message --
From: kesim 
<ketiw...@gmail.com<mailto:ketiw...@gmail.com><mailto:ketiw...@gmail.com<mailto

[slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load

[kesim]

Dear John,

Thank you for your answer. Obviously you are right that I could slurm up
everything and thus avoid the issue and your points are taken. However, I
still insist that it is a serious bug not to take into account the actual
CPU load when the scheduler submit a job regardless whose fault it is that
a non-slurm job is running. I would not suspect that from even simplest
scheduler and if I had such prior knowledge I would not invest so much time
and effort  to setup slurm.
Best regards,

Ketiw

On Sat, Mar 18, 2017 at 5:42 PM, John Hearns <john.hea...@xma.co.uk> wrote:

>
> Kesim,
>
> what you are saying is that Slurm schedukes tasks based on the number of
> allocated CPUs. Rather than the actual load factor on the server.
> As I recall Gridengine actually used the load factor.
>
> However you comment that "users run programs on the nodes" and "the slurm
> is aware about the load of non-slurm jobs"
> IMHO, in any well-run HPC setup any user running jobs without using the
> scheduler would have their fingers broken. or at least bruised using the
> clue stick.
>
> Seriously, three points:
>
> a) tell users to use 'salloc' and 'srun'  to run interactive jobs. They
> can easily open a Bash session on a compute node and do what they like.
> Under the Slurm scheduler.
>
> b) implement the pam-slurm PAMmodule. It is a few minutes work. This means
> your users cannot go behind the sluem scheduler and log into the nodes
>
> c) on Bright clusters which I configure, you have a healtcheck running
> which wans you when a user is detected as logging in withotu using Slurm
>
>
> Seriously again. You have implemented an HPC infrastructure, and have gone
> to the time and effort to implement a batch scheduling system.
> A batch scheduler can be adapted to let your users do their jobs,
> including interactive shell sessions and remote visualization sessions.
> Do not let the users ride roughshod over you.
>
> ________________________
> From: kesim [ketiw...@gmail.com]
> Sent: 18 March 2017 16:16
> To: slurm-dev
> Subject: [slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load
>
> Unbelievable but it seems that nobody knows how to do that. It is
> astonishing that such sophisticated system fails with such simple problem.
> The slurm is aware about the cpu load of non-slurm jobs but it does not use
> the info. My original understanding of LLN was apparently correct. I can
> practically kill the CPUs on particular node with nonslurm tasks but slurm
> will diligently submit 7 jobs to this node leaving other idling.  I
> consider this as a serious bug of this program.
>
>
> On Fri, Mar 17, 2017 at 10:32 AM, kesim <ketiw...@gmail.com<mailto:ket
> iw...@gmail.com>> wrote:
> Dear All,
> Yesterday I did some tests and it seemed that the scheduling is following
> CPU load but I was wrong.
> My configuration is at the moment:
> SelectType=select/cons_res
> SelectTypeParameters=CR_CPU,CR_LLN
>
> Today I submitted 70 threaded jobs to the queue and here is the CPU_LOAD
> info
> node1 0.08  7/0/0/7
> node20.01  7/0/0/7
> node30.00  7/0/0/7
> node42.97  7/0/0/7
> node5   0.00  7/0/0/7
> node6 0.01  7/0/0/7
> node7  0.00  7/0/0/7
> node8   0.05  7/0/0/7
> node90.07  7/0/0/7
> node100.38  7/0/0/7
> node11 0.01  0/7/0/7
> As you can see it allocated 7 CPUs on node 4 with CPU_LOAD 2.97 and 0 CPUs
> on idling node11. Why such simple thing is not a default? What am I
> missing???
>
> On Thu, Mar 16, 2017 at 7:53 PM, kesim <ketiw...@gmail.com<mailto:ket
> iw...@gmail.com>> wrote:
> Than you for great suggestion. It is working! However the description of
> CR_LLN is misleading "Schedule resources to jobs on the least loaded nodes
> (based upon the number of idle CPUs)" Which I understood that if the two
> nodes has not fully allocated CPUs  the node with smaller number of
> allocated CPUs will take precedence. Therefore the bracketed comment should
> be removed from the description.
>
> On Thu, Mar 16, 2017 at 6:24 PM, Paul Edmon <ped...@cfa.harvard.edu<
> mailto:ped...@cfa.harvard.edu>> wrote:
>
> You should look at LLN (least loaded nodes):
>
> https://slurm.schedmd.com/slurm.conf.html
>
> That should do what you want.
>
> -Paul Edmon-
>
> On 03/16/2017 12:54 PM, kesim wrote:
>
> -- Forwarded message --
> From: kesim <ketiw...@gmail.com<mailto:ketiw...@gmail.com>>
> Date: Thu, Mar 16, 2017 at 5:50 PM
> Subject: Scheduling jobs according to the CPU load
> To: sl

[slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load

[kesim]

Dear All,
Yesterday I did some tests and it seemed that the scheduling is following
CPU load but I was wrong.
My configuration is at the moment:
SelectType=select/cons_res
SelectTypeParameters=CR_CPU,CR_LLN

Today I submitted 70 threaded jobs to the queue and here is the CPU_LOAD
info
node1 0.08  7/0/0/7
node20.01  7/0/0/7
node30.00  7/0/0/7
node42.97  7/0/0/7
node5   0.00  7/0/0/7
node6 0.01  7/0/0/7
node7  0.00  7/0/0/7
node8   0.05  7/0/0/7
node90.07  7/0/0/7
node100.38  7/0/0/7
node11 0.01  0/7/0/7
As you can see it allocated 7 CPUs on node 4 with CPU_LOAD 2.97 and 0 CPUs
on idling node11. Why such simple thing is not a default? What am I
missing???

On Thu, Mar 16, 2017 at 7:53 PM, kesim  wrote:

> Than you for great suggestion. It is working! However the description of
> CR_LLN is misleading "Schedule resources to jobs on the least loaded nodes
> (based upon the number of idle CPUs)" Which I understood that if the two
> nodes has not fully allocated CPUs  the node with smaller number of
> allocated CPUs will take precedence. Therefore the bracketed comment should
> be removed from the description.
>
> On Thu, Mar 16, 2017 at 6:24 PM, Paul Edmon 
> wrote:
>
>> You should look at LLN (least loaded nodes):
>>
>> https://slurm.schedmd.com/slurm.conf.html
>>
>> That should do what you want.
>> -Paul Edmon-
>>
>> On 03/16/2017 12:54 PM, kesim wrote:
>>
>>
>> -- Forwarded message --
>> From: kesim 
>> Date: Thu, Mar 16, 2017 at 5:50 PM
>> Subject: Scheduling jobs according to the CPU load
>> To: slurm-dev@schedmd.com
>>
>>
>> Hi all,
>>
>> I am a new user and I created a small network of 11 nodes 7 CPUs per node
>> out of users desktops.
>> I configured slurm as:
>> SelectType=select/cons_res
>> SelectTypeParameters=CR_CPU
>> When I submit a task with srun -n70 task
>> It will fill 10 nodes with 7 tasks/node. However, I have no clue what is
>> the algorithm of choosing the nodes. Users run programs on the nodes and
>> some nodes are more busy than others. It seems logical that the scheduler
>> should submit the tasks to the less busy nodes but it is not the case.
>> In the sinfo -N -o '%N %O %C' I can see that the jobs are allocated to
>> the node11 with the load 2.06 leaving the node4 which is totally idling.
>> That somehow make no sense to me.
>> node1 0.00  7/0/0/7
>> node20.26  7/0/0/7
>> node3 0.54  7/0/0/7
>> node40.07  0/7/0/7
>> node5  0.00  7/0/0/7
>> node60.01  7/0/0/7
>> node7   0.00  7/0/0/7
>> node8   0.01  7/0/0/7
>> node90.06  7/0/0/7
>> node10  0.11  7/0/0/7
>> node11  2.06  7/0/0/7
>> How can I configure slurm to be able to fill the node with minimum load
>> first?
>>
>>
>>
>>
>

[slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load

[kesim]

Than you for great suggestion. It is working! However the description of
CR_LLN is misleading "Schedule resources to jobs on the least loaded nodes
(based upon the number of idle CPUs)" Which I understood that if the two
nodes has not fully allocated CPUs  the node with smaller number of
allocated CPUs will take precedence. Therefore the bracketed comment should
be removed from the description.

On Thu, Mar 16, 2017 at 6:24 PM, Paul Edmon  wrote:

> You should look at LLN (least loaded nodes):
>
> https://slurm.schedmd.com/slurm.conf.html
>
> That should do what you want.
> -Paul Edmon-
>
> On 03/16/2017 12:54 PM, kesim wrote:
>
>
> -- Forwarded message --
> From: kesim 
> Date: Thu, Mar 16, 2017 at 5:50 PM
> Subject: Scheduling jobs according to the CPU load
> To: slurm-dev@schedmd.com
>
>
> Hi all,
>
> I am a new user and I created a small network of 11 nodes 7 CPUs per node
> out of users desktops.
> I configured slurm as:
> SelectType=select/cons_res
> SelectTypeParameters=CR_CPU
> When I submit a task with srun -n70 task
> It will fill 10 nodes with 7 tasks/node. However, I have no clue what is
> the algorithm of choosing the nodes. Users run programs on the nodes and
> some nodes are more busy than others. It seems logical that the scheduler
> should submit the tasks to the less busy nodes but it is not the case.
> In the sinfo -N -o '%N %O %C' I can see that the jobs are allocated to the
> node11 with the load 2.06 leaving the node4 which is totally idling. That
> somehow make no sense to me.
> node1 0.00  7/0/0/7
> node20.26  7/0/0/7
> node3 0.54  7/0/0/7
> node40.07  0/7/0/7
> node5  0.00  7/0/0/7
> node60.01  7/0/0/7
> node7   0.00  7/0/0/7
> node8   0.01  7/0/0/7
> node90.06  7/0/0/7
> node10  0.11  7/0/0/7
> node11  2.06  7/0/0/7
> How can I configure slurm to be able to fill the node with minimum load
> first?
>
>
>
>

[slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load

[Paul Edmon]

You should look at LLN (least loaded nodes):

https://slurm.schedmd.com/slurm.conf.html

That should do what you want.

-Paul Edmon-

On 03/16/2017 12:54 PM, kesim wrote:

Fwd: Scheduling jobs according to the CPU load

-- Forwarded message --
From: *kesim* >
Date: Thu, Mar 16, 2017 at 5:50 PM
Subject: Scheduling jobs according to the CPU load
To: slurm-dev@schedmd.com 


Hi all,

I am a new user and I created a small network of 11 nodes 7 CPUs per 
node out of users desktops.

I configured slurm as:
SelectType=select/cons_res
SelectTypeParameters=CR_CPU
When I submit a task with srun -n70 task
It will fill 10 nodes with 7 tasks/node. However, I have no clue what 
is the algorithm of choosing the nodes. Users run programs on the 
nodes and some nodes are more busy than others. It seems logical that 
the scheduler should submit the tasks to the less busy nodes but it is 
not the case.
In the sinfo -N -o '%N %O %C' I can see that the jobs are allocated to 
the node11 with the load 2.06 leaving the node4 which is totally 
idling. That somehow make no sense to me.

node1 0.00  7/0/0/7
node20.26  7/0/0/7
node3 0.54  7/0/0/7
node40.07  0/7/0/7
node5  0.00  7/0/0/7
node60.01  7/0/0/7
node7   0.00  7/0/0/7
node8   0.01  7/0/0/7
node90.06  7/0/0/7
node10  0.11  7/0/0/7
node11  2.06  7/0/0/7
How can I configure slurm to be able to fill the node with minimum 
load first?