[slurm-dev] Re: Account preferation

[Manuel Rodríguez Pascual]

yes, you can use backfill and multifactor together. Sorry if that was
not clear on my previous mail. Multifactor is in charge of sorting the
job queue, and backifll is in charge of executing them



2017-08-08 16:04 GMT+02:00 Dennis Tants :
>
> Hello Manuel,
>
> thank you very much for your insight.
>
> I have read about FairShare too, but I am not sure if/how I can use it
> for this specific example.
> But I will consider it!
> Or, can anyone confirm FairShare should work in this case?
>
>
> Just for clarification:
> Can't I use backfill and multifactor priority together?
> First goes backfill, depending on the job length and then it is sorted
> by priority?
> Atleast thats were my first thoughts about it :D
>
> Best regards,
> Dennis
>
> Am 08.08.2017 um 14:53 schrieb Manuel Rodríguez Pascual:
>> Hi Dennis,
>>
>> If I undersand you correctly, what you need is to use the Multifactor Plugin
>> https://slurm.schedmd.com/priority_multifactor.html
>>
>> In particular, I guess this is relevant for your installation:
>>
>> Note: Computing the fair-share factor requires the installation and
>> operation of the Slurm Accounting Database to provide the assigned
>> shares and the consumed, computing resources described below.
>>
>> and
>>
>> PriorityDecayHalfLife   This determines the contribution of historical
>> usage on the composite usage value. The larger the number, the longer
>> past usage affects fair-share. If set to 0 no decay will be applied.
>> This is helpful if you want to enforce hard time limits per
>> association. If set to 0 PriorityUsageResetPeriod must be set to some
>> interval. The unit is a time string (i.e. min, hr:min:00,
>> days-hr:min:00, or days-hr). The default value is 7-0 (7 days).
>>
>>
>> Also, note that backfill and multifactor are two different things:
>> Backfill tries to run small jobs if this does not affect to other ones
>> waiting on the queue;  Multifactor determines the order of the queue.
>> They are set with different parameters in slurm.conf
>>
>> SchedulerType=sched/backfill
>> PriorityType=priority/multifactor
>>
>>
>> I am not a slurm expert so I may probably be wrong on some
>> information, but I hope this can serve you as a small help on your
>> problem.
>>
>> Best,
>>
>>
>> Manuel
>>
>> 2017-08-08 14:37 GMT+02:00 Dennis Tants :
>>> Hey guys,
>>>
>>> I was asked to implement a walltime of 24 hours in our cluster (we only
>>> have 16 nodes, so no need until now).
>>> Furthermore, I need to prefer single accounts based on compute time. The
>>> example I was given is further below.
>>>
>>> Btw., I'm upgrading to SLURM 17.02.4.
>>>
>>> Example:
>>> If the partition is full, we should start to prefer people who haven't
>>> used much computing time the last month.
>>>
>>> I don't know where I should start with this request to be honest and if
>>> it is even possible (walltime is not the problem).
>>>
>>> First of, I would need to also implement accounting I guess (also, no
>>> need for until now^^)
>>> But how to continue? I wanted to implement backfill instead of FIFO, but
>>> would I also need multifactor priority?
>>>
>>> Here are my thoughts of being able to achieve this:
>>>
>>> 1. Could I do something like:
>>> A. People are getting points every month (which they can't actually
>>> empty within one month)
>>> B. Compare the points (last month) of all people when a queue appears.
>>>
>>> 2. Or something like (same but time based):
>>> A. Account the whole computing time for two months (and then start to
>>> delete old entries)
>>> B. While queue, compare computing time (last month) of all waiting users?
>>>
>>> I hope I made myself clear and that you can help me. Every hint is
>>> appreciated.
>>>
>>> Best regards,
>>> Dennis
>>>
>>> --
>>> Dennis Tants
>>> Auszubildender: Fachinformatiker für Systemintegration
>>>
>>> ZARM - Zentrum für angewandte Raumfahrttechnologie und Mikrogravitation
>>> ZARM - Center of Applied Space Technology and Microgravity
>>>
>>> Universität Bremen
>>> Am Fallturm
>>> 28359 Bremen, Germany
>>>
>>> Telefon: 0421 218 57940
>>> E-Mail: ta...@zarm.uni-bremen.de
>>>
>>> www.zarm.uni-bremen.de
>>>
>
> --
> Dennis Tants
> Auszubildender: Fachinformatiker für Systemintegration
>
> ZARM - Zentrum für angewandte Raumfahrttechnologie und Mikrogravitation
> ZARM - Center of Applied Space Technology and Microgravity
>
> Universität Bremen
> Am Fallturm
> 28359 Bremen, Germany
>
> Telefon: 0421 218 57940
> E-Mail: ta...@zarm.uni-bremen.de
>
> www.zarm.uni-bremen.de
>

[slurm-dev] Re: Account preferation

[Dennis Tants]

Hello Manuel,

thank you very much for your insight.

I have read about FairShare too, but I am not sure if/how I can use it
for this specific example.
But I will consider it!
Or, can anyone confirm FairShare should work in this case?


Just for clarification:
Can't I use backfill and multifactor priority together?
First goes backfill, depending on the job length and then it is sorted
by priority?
Atleast thats were my first thoughts about it :D

Best regards,
Dennis

Am 08.08.2017 um 14:53 schrieb Manuel Rodríguez Pascual:
> Hi Dennis,
>
> If I undersand you correctly, what you need is to use the Multifactor Plugin
> https://slurm.schedmd.com/priority_multifactor.html
>
> In particular, I guess this is relevant for your installation:
>
> Note: Computing the fair-share factor requires the installation and
> operation of the Slurm Accounting Database to provide the assigned
> shares and the consumed, computing resources described below.
>
> and
>
> PriorityDecayHalfLife   This determines the contribution of historical
> usage on the composite usage value. The larger the number, the longer
> past usage affects fair-share. If set to 0 no decay will be applied.
> This is helpful if you want to enforce hard time limits per
> association. If set to 0 PriorityUsageResetPeriod must be set to some
> interval. The unit is a time string (i.e. min, hr:min:00,
> days-hr:min:00, or days-hr). The default value is 7-0 (7 days).
>
>
> Also, note that backfill and multifactor are two different things:
> Backfill tries to run small jobs if this does not affect to other ones
> waiting on the queue;  Multifactor determines the order of the queue.
> They are set with different parameters in slurm.conf
>
> SchedulerType=sched/backfill
> PriorityType=priority/multifactor
>
>
> I am not a slurm expert so I may probably be wrong on some
> information, but I hope this can serve you as a small help on your
> problem.
>
> Best,
>
>
> Manuel
>
> 2017-08-08 14:37 GMT+02:00 Dennis Tants :
>> Hey guys,
>>
>> I was asked to implement a walltime of 24 hours in our cluster (we only
>> have 16 nodes, so no need until now).
>> Furthermore, I need to prefer single accounts based on compute time. The
>> example I was given is further below.
>>
>> Btw., I'm upgrading to SLURM 17.02.4.
>>
>> Example:
>> If the partition is full, we should start to prefer people who haven't
>> used much computing time the last month.
>>
>> I don't know where I should start with this request to be honest and if
>> it is even possible (walltime is not the problem).
>>
>> First of, I would need to also implement accounting I guess (also, no
>> need for until now^^)
>> But how to continue? I wanted to implement backfill instead of FIFO, but
>> would I also need multifactor priority?
>>
>> Here are my thoughts of being able to achieve this:
>>
>> 1. Could I do something like:
>> A. People are getting points every month (which they can't actually
>> empty within one month)
>> B. Compare the points (last month) of all people when a queue appears.
>>
>> 2. Or something like (same but time based):
>> A. Account the whole computing time for two months (and then start to
>> delete old entries)
>> B. While queue, compare computing time (last month) of all waiting users?
>>
>> I hope I made myself clear and that you can help me. Every hint is
>> appreciated.
>>
>> Best regards,
>> Dennis
>>
>> --
>> Dennis Tants
>> Auszubildender: Fachinformatiker für Systemintegration
>>
>> ZARM - Zentrum für angewandte Raumfahrttechnologie und Mikrogravitation
>> ZARM - Center of Applied Space Technology and Microgravity
>>
>> Universität Bremen
>> Am Fallturm
>> 28359 Bremen, Germany
>>
>> Telefon: 0421 218 57940
>> E-Mail: ta...@zarm.uni-bremen.de
>>
>> www.zarm.uni-bremen.de
>>

-- 
Dennis Tants
Auszubildender: Fachinformatiker für Systemintegration

ZARM - Zentrum für angewandte Raumfahrttechnologie und Mikrogravitation
ZARM - Center of Applied Space Technology and Microgravity

Universität Bremen
Am Fallturm
28359 Bremen, Germany

Telefon: 0421 218 57940
E-Mail: ta...@zarm.uni-bremen.de

www.zarm.uni-bremen.de

[slurm-dev] Re: Account preferation

[Manuel Rodríguez Pascual]

Hi Dennis,

If I undersand you correctly, what you need is to use the Multifactor Plugin
https://slurm.schedmd.com/priority_multifactor.html

In particular, I guess this is relevant for your installation:

Note: Computing the fair-share factor requires the installation and
operation of the Slurm Accounting Database to provide the assigned
shares and the consumed, computing resources described below.

and

PriorityDecayHalfLife   This determines the contribution of historical
usage on the composite usage value. The larger the number, the longer
past usage affects fair-share. If set to 0 no decay will be applied.
This is helpful if you want to enforce hard time limits per
association. If set to 0 PriorityUsageResetPeriod must be set to some
interval. The unit is a time string (i.e. min, hr:min:00,
days-hr:min:00, or days-hr). The default value is 7-0 (7 days).


Also, note that backfill and multifactor are two different things:
Backfill tries to run small jobs if this does not affect to other ones
waiting on the queue;  Multifactor determines the order of the queue.
They are set with different parameters in slurm.conf

SchedulerType=sched/backfill
PriorityType=priority/multifactor


I am not a slurm expert so I may probably be wrong on some
information, but I hope this can serve you as a small help on your
problem.

Best,


Manuel

2017-08-08 14:37 GMT+02:00 Dennis Tants :
>
> Hey guys,
>
> I was asked to implement a walltime of 24 hours in our cluster (we only
> have 16 nodes, so no need until now).
> Furthermore, I need to prefer single accounts based on compute time. The
> example I was given is further below.
>
> Btw., I'm upgrading to SLURM 17.02.4.
>
> Example:
> If the partition is full, we should start to prefer people who haven't
> used much computing time the last month.
>
> I don't know where I should start with this request to be honest and if
> it is even possible (walltime is not the problem).
>
> First of, I would need to also implement accounting I guess (also, no
> need for until now^^)
> But how to continue? I wanted to implement backfill instead of FIFO, but
> would I also need multifactor priority?
>
> Here are my thoughts of being able to achieve this:
>
> 1. Could I do something like:
> A. People are getting points every month (which they can't actually
> empty within one month)
> B. Compare the points (last month) of all people when a queue appears.
>
> 2. Or something like (same but time based):
> A. Account the whole computing time for two months (and then start to
> delete old entries)
> B. While queue, compare computing time (last month) of all waiting users?
>
> I hope I made myself clear and that you can help me. Every hint is
> appreciated.
>
> Best regards,
> Dennis
>
> --
> Dennis Tants
> Auszubildender: Fachinformatiker für Systemintegration
>
> ZARM - Zentrum für angewandte Raumfahrttechnologie und Mikrogravitation
> ZARM - Center of Applied Space Technology and Microgravity
>
> Universität Bremen
> Am Fallturm
> 28359 Bremen, Germany
>
> Telefon: 0421 218 57940
> E-Mail: ta...@zarm.uni-bremen.de
>
> www.zarm.uni-bremen.de
>