[slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load

2017-03-18 Thread John Hearns 

Kesim,
   Touche Sir. I agree with you.


From: kesim [ketiw...@gmail.com]
Sent: 18 March 2017 18:06
To: slurm-dev
Subject: [slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load

Dear John,

Thank you for your answer. Obviously you are right that I could slurm up 
everything and thus avoid the issue and your points are taken. However, I still 
insist that it is a serious bug not to take into account the actual CPU load 
when the scheduler submit a job regardless whose fault it is that a non-slurm 
job is running. I would not suspect that from even simplest scheduler and if I 
had such prior knowledge I would not invest so much time and effort  to setup 
slurm.
Best regards,

Ketiw

On Sat, Mar 18, 2017 at 5:42 PM, John Hearns 
> wrote:

Kesim,

what you are saying is that Slurm schedukes tasks based on the number of 
allocated CPUs. Rather than the actual load factor on the server.
As I recall Gridengine actually used the load factor.

However you comment that "users run programs on the nodes" and "the slurm is 
aware about the load of non-slurm jobs"
IMHO, in any well-run HPC setup any user running jobs without using the 
scheduler would have their fingers broken. or at least bruised using the clue 
stick.

Seriously, three points:

a) tell users to use 'salloc' and 'srun'  to run interactive jobs. They can 
easily open a Bash session on a compute node and do what they like. Under the 
Slurm scheduler.

b) implement the pam-slurm PAMmodule. It is a few minutes work. This means your 
users cannot go behind the sluem scheduler and log into the nodes

c) on Bright clusters which I configure, you have a healtcheck running which 
wans you when a user is detected as logging in withotu using Slurm


Seriously again. You have implemented an HPC infrastructure, and have gone to 
the time and effort to implement a batch scheduling system.
A batch scheduler can be adapted to let your users do their jobs, including 
interactive shell sessions and remote visualization sessions.
Do not let the users ride roughshod over you.


From: kesim [ketiw...@gmail.com]
Sent: 18 March 2017 16:16
To: slurm-dev
Subject: [slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load

Unbelievable but it seems that nobody knows how to do that. It is astonishing 
that such sophisticated system fails with such simple problem. The slurm is 
aware about the cpu load of non-slurm jobs but it does not use the info. My 
original understanding of LLN was apparently correct. I can practically kill 
the CPUs on particular node with nonslurm tasks but slurm will diligently 
submit 7 jobs to this node leaving other idling.  I consider this as a serious 
bug of this program.


On Fri, Mar 17, 2017 at 10:32 AM, kesim 
>>
 wrote:
Dear All,
Yesterday I did some tests and it seemed that the scheduling is following CPU 
load but I was wrong.
My configuration is at the moment:
SelectType=select/cons_res
SelectTypeParameters=CR_CPU,CR_LLN

Today I submitted 70 threaded jobs to the queue and here is the CPU_LOAD info
node1 0.08  7/0/0/7
node20.01  7/0/0/7
node30.00  7/0/0/7
node42.97  7/0/0/7
node5   0.00  7/0/0/7
node6 0.01  7/0/0/7
node7  0.00  7/0/0/7
node8   0.05  7/0/0/7
node90.07  7/0/0/7
node100.38  7/0/0/7
node11 0.01  0/7/0/7
As you can see it allocated 7 CPUs on node 4 with CPU_LOAD 2.97 and 0 CPUs on 
idling node11. Why such simple thing is not a default? What am I missing???

On Thu, Mar 16, 2017 at 7:53 PM, kesim 
>>
 wrote:
Than you for great suggestion. It is working! However the description of CR_LLN 
is misleading "Schedule resources to jobs on the least loaded nodes (based upon 
the number of idle CPUs)" Which I understood that if the two nodes has not 
fully allocated CPUs  the node with smaller number of allocated CPUs will take 
precedence. Therefore the bracketed comment should be removed from the 
description.

On Thu, Mar 16, 2017 at 6:24 PM, Paul Edmon 
>>
 wrote:

You should look at LLN (least loaded nodes):

https://slurm.schedmd.com/slurm.conf.html

That should do what you want.

-Paul Edmon-

On 03/16/2017 12:54 PM, kesim wrote:

-- Forwarded message --
From: kesim 
>>
Date: Thu, Mar 16, 2017 at 5:50 PM
Subject: Scheduling jobs according to the CPU load
To:

[slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load

2017-03-18 Thread kesim 
Dear John,

Thank you for your answer. Obviously you are right that I could slurm up
everything and thus avoid the issue and your points are taken. However, I
still insist that it is a serious bug not to take into account the actual
CPU load when the scheduler submit a job regardless whose fault it is that
a non-slurm job is running. I would not suspect that from even simplest
scheduler and if I had such prior knowledge I would not invest so much time
and effort  to setup slurm.
Best regards,

Ketiw

On Sat, Mar 18, 2017 at 5:42 PM, John Hearns  wrote:

>
> Kesim,
>
> what you are saying is that Slurm schedukes tasks based on the number of
> allocated CPUs. Rather than the actual load factor on the server.
> As I recall Gridengine actually used the load factor.
>
> However you comment that "users run programs on the nodes" and "the slurm
> is aware about the load of non-slurm jobs"
> IMHO, in any well-run HPC setup any user running jobs without using the
> scheduler would have their fingers broken. or at least bruised using the
> clue stick.
>
> Seriously, three points:
>
> a) tell users to use 'salloc' and 'srun'  to run interactive jobs. They
> can easily open a Bash session on a compute node and do what they like.
> Under the Slurm scheduler.
>
> b) implement the pam-slurm PAMmodule. It is a few minutes work. This means
> your users cannot go behind the sluem scheduler and log into the nodes
>
> c) on Bright clusters which I configure, you have a healtcheck running
> which wans you when a user is detected as logging in withotu using Slurm
>
>
> Seriously again. You have implemented an HPC infrastructure, and have gone
> to the time and effort to implement a batch scheduling system.
> A batch scheduler can be adapted to let your users do their jobs,
> including interactive shell sessions and remote visualization sessions.
> Do not let the users ride roughshod over you.
>
> 
> From: kesim [ketiw...@gmail.com]
> Sent: 18 March 2017 16:16
> To: slurm-dev
> Subject: [slurm-dev] Re: Fwd: Scheduling jobs according to the CPU load
>
> Unbelievable but it seems that nobody knows how to do that. It is
> astonishing that such sophisticated system fails with such simple problem.
> The slurm is aware about the cpu load of non-slurm jobs but it does not use
> the info. My original understanding of LLN was apparently correct. I can
> practically kill the CPUs on particular node with nonslurm tasks but slurm
> will diligently submit 7 jobs to this node leaving other idling.  I
> consider this as a serious bug of this program.
>
>
> On Fri, Mar 17, 2017 at 10:32 AM, kesim  iw...@gmail.com>> wrote:
> Dear All,
> Yesterday I did some tests and it seemed that the scheduling is following
> CPU load but I was wrong.
> My configuration is at the moment:
> SelectType=select/cons_res
> SelectTypeParameters=CR_CPU,CR_LLN
>
> Today I submitted 70 threaded jobs to the queue and here is the CPU_LOAD
> info
> node1 0.08  7/0/0/7
> node20.01  7/0/0/7
> node30.00  7/0/0/7
> node42.97  7/0/0/7
> node5   0.00  7/0/0/7
> node6 0.01  7/0/0/7
> node7  0.00  7/0/0/7
> node8   0.05  7/0/0/7
> node90.07  7/0/0/7
> node100.38  7/0/0/7
> node11 0.01  0/7/0/7
> As you can see it allocated 7 CPUs on node 4 with CPU_LOAD 2.97 and 0 CPUs
> on idling node11. Why such simple thing is not a default? What am I
> missing???
>
> On Thu, Mar 16, 2017 at 7:53 PM, kesim  iw...@gmail.com>> wrote:
> Than you for great suggestion. It is working! However the description of
> CR_LLN is misleading "Schedule resources to jobs on the least loaded nodes
> (based upon the number of idle CPUs)" Which I understood that if the two
> nodes has not fully allocated CPUs  the node with smaller number of
> allocated CPUs will take precedence. Therefore the bracketed comment should
> be removed from the description.
>
> On Thu, Mar 16, 2017 at 6:24 PM, Paul Edmon  mailto:ped...@cfa.harvard.edu>> wrote:
>
> You should look at LLN (least loaded nodes):
>
> https://slurm.schedmd.com/slurm.conf.html
>
> That should do what you want.
>
> -Paul Edmon-
>
> On 03/16/2017 12:54 PM, kesim wrote:
>
> -- Forwarded message --
> From: kesim >
> Date: Thu, Mar 16, 2017 at 5:50 PM
> Subject: Scheduling jobs according to the CPU load
> To: slurm-dev@schedmd.com
>
>
> Hi all,
>
> I am a new user and I created a small network of 11 nodes 7 CPUs per node
> out of users desktops.
> I configured slurm as:
> SelectType=select/cons_res
> SelectTypeParameters=CR_CPU
> When I submit a task with srun -n70 task
> It will fill 10 nodes with 7 tasks/node. However, I have no clue what is
> the algorithm of