[slurm-dev] Re: resource usage, TRES and --exclusive option
Hello, I believe Robbert Eggermont answered my question. But thank you again anyway ! Cheers Christof On Thu, Sep 01, 2016 at 04:43:25PM -0700, Lachlan Musicman wrote: > On 1 September 2016 at 18:16, Christof Koehler < > christof.koeh...@bccms.uni-bremen.de> wrote: > > > Hello, > > > > On Wed, Aug 31, 2016 at 05:52:48PM -0700, Lachlan Musicman wrote: > > > I don't believe it's 100% necessary to use OverSubscribe Yes. > > > > > > We have > > > > > > select/cons_res and CR_CPU, on nodes that look like > > > > > > NodeName=stpr-res-compute[01-02] CPUs=40 RealMemory=385000 Sockets=2 > > > CoresPerSocket=10 ThreadsPerCore=2 State=UNKNOWN > > OK, I will keep this in mind. > > > > > > > > The CPUs aren't shared, but there's 40 of them per node, so that doesn't > > > really matter so much. > > I do not understand this. What part of a node is shared if not its > > processing resource ? Or may be there is a misunderstanding because my > > question was not well formulated. So I will rephrase it: > > > > What we are interested in is a situation where of 40 CPUs to one user 10 > > are allocated and the remaining 30 to for example one other user. So no > > actual oversubscription of CPUs but a sharing of the nodes CPUs between > > different users which get their exclusive cpusets. It has 40 CPUs as > > GRES and up to 40 can be allocated as exlusive resources to different > > users. I think cgroups would provide the necessary controls for this. > > > > > > Ok, I can't speak to GRES or cgroups - we have a simple set up. In *my* > explanation, a "node" is a "motherboard". IE, each motherboard has 40 cpus, > and those cpus can be shared as you would like them to be (10 to one user, > 30 to the other; or even 1 to 40 users, or 4 to 10 users). > > > > > > > > Now, the point we are not sure about is what happens if a user allocates > > 10 out of 40 and sets "--exclusive" (if possible). Is the usage of that > > user (job) actually computed with 40 CPUs as most people would expect ? > > As described before other systems appear to bill only 10 while 40 are in > > fact used up. > > > > > > > > Also, have you found this page? http://slurm.schedmd.com/mc_support.html > > > > That is certainly a very useful resource explaining the different > > concepts and how they relate to the terminology used by slurm. However, > > skimming through it there does not appear to be a direct connection to > > out usage computation question. > > > > > > Cheers > L. > > -- > The most dangerous phrase in the language is, "We've always done it this > way." > > - Grace Hopper -- Dr. rer. nat. Christof Köhler email: c.koeh...@bccms.uni-bremen.de Universitaet Bremen/ BCCMS phone: +49-(0)421-218-62334 Am Fallturm 1/ TAB/ Raum 3.12 fax: +49-(0)421-218-62770 28359 Bremen PGP: http://www.bccms.uni-bremen.de/cms/people/c_koehler/ signature.asc Description: Digital signature
[slurm-dev] Re: resource usage, TRES and --exclusive option
On 1 September 2016 at 18:16, Christof Koehler < christof.koeh...@bccms.uni-bremen.de> wrote: > Hello, > > On Wed, Aug 31, 2016 at 05:52:48PM -0700, Lachlan Musicman wrote: > > I don't believe it's 100% necessary to use OverSubscribe Yes. > > > > We have > > > > select/cons_res and CR_CPU, on nodes that look like > > > > NodeName=stpr-res-compute[01-02] CPUs=40 RealMemory=385000 Sockets=2 > > CoresPerSocket=10 ThreadsPerCore=2 State=UNKNOWN > OK, I will keep this in mind. > > > > > The CPUs aren't shared, but there's 40 of them per node, so that doesn't > > really matter so much. > I do not understand this. What part of a node is shared if not its > processing resource ? Or may be there is a misunderstanding because my > question was not well formulated. So I will rephrase it: > > What we are interested in is a situation where of 40 CPUs to one user 10 > are allocated and the remaining 30 to for example one other user. So no > actual oversubscription of CPUs but a sharing of the nodes CPUs between > different users which get their exclusive cpusets. It has 40 CPUs as > GRES and up to 40 can be allocated as exlusive resources to different > users. I think cgroups would provide the necessary controls for this. > > Ok, I can't speak to GRES or cgroups - we have a simple set up. In *my* explanation, a "node" is a "motherboard". IE, each motherboard has 40 cpus, and those cpus can be shared as you would like them to be (10 to one user, 30 to the other; or even 1 to 40 users, or 4 to 10 users). > Now, the point we are not sure about is what happens if a user allocates > 10 out of 40 and sets "--exclusive" (if possible). Is the usage of that > user (job) actually computed with 40 CPUs as most people would expect ? > As described before other systems appear to bill only 10 while 40 are in > fact used up. > > > > > Also, have you found this page? http://slurm.schedmd.com/mc_support.html > > That is certainly a very useful resource explaining the different > concepts and how they relate to the terminology used by slurm. However, > skimming through it there does not appear to be a direct connection to > out usage computation question. > > Cheers L. -- The most dangerous phrase in the language is, "We've always done it this way." - Grace Hopper
[slurm-dev] Re: resource usage, TRES and --exclusive option
Hello, On Thu, Sep 01, 2016 at 02:23:57AM -0700, Benjamin Redling wrote: > > You might want to (re)read > 1. http://slurm.schedmd.com/cons_res.html > 2. http://slurm.schedmd.com/cons_res_share.html > > [1] right at the beginning explains the default behaviour (exclusive mode) > > [2] should explain best that OverSubscribe (fka. Shared) is depended on > the "Selection Setting". > If you skim the column "Resulting Behavior" you should know really > fast what you aim for. > > We share nodes between jobs of different users (but: all one trusted > group, my colleagues -- really easy case) without any conflict and > neither use OverSubscribe nor cgroups (but cgroups is a ticket). > Rereading with your pointers I think I am starting to understand now. Is the following correct: Quoting [1]: "The only changes, users see when using the select/cons_res plugin, are that jobs can be co-scheduled on nodes when resources permit it." which taking [2] and manpage of sched.conf (OverSubscribe defaults to "NO") into account means that in your configuration only jobs from different partitions/queues can share a node. Now assume that there is only a single partition, which I think we would prefer ad hoc. Then: 1. The default (select/linear, OverSubscribe=NO) is always exclusive. So the job of course gets billed for the whole resources of a node. 2. With a single partition select/cons_res and OverSubscribe=YES must be simultaneously set so that jobs can share nodes. A user may allow other jobs on a node allocated to his job by explicitely setting "--shared", in which case only the actually allocated resources are billed to each different job on that node. If he does not share the full resources are billed to him. The option "--exclusive" is never needed. 4. Only if OverSubscribe=FORCE is set on the partition the argument "--exclusive" would make sense to prevent the default sharing of nodes. With "--exclusive" all resources of the node would be billed to the exclusive job automatically, right ? I would appreciate it if you could confirm so that I can introduce this into the local discussion we have. Thank you very much in advance ! Best Regards Christof -- Dr. rer. nat. Christof Köhler email: c.koeh...@bccms.uni-bremen.de Universitaet Bremen/ BCCMS phone: +49-(0)421-218-62334 Am Fallturm 1/ TAB/ Raum 3.12 fax: +49-(0)421-218-62770 28359 Bremen PGP: http://www.bccms.uni-bremen.de/cms/people/c_koehler/ signature.asc Description: Digital signature
[slurm-dev] Re: resource usage, TRES and --exclusive option
Hi On 09/01/2016 10:16, Christof Koehler wrote: > Now, the point we are not sure about is what happens if a user allocates > 10 out of 40 and sets "--exclusive" (if possible). Is the usage of that > user (job) actually computed with 40 CPUs as most people would expect ? > As described before other systems appear to bill only 10 while 40 are in > fact used up. >> Also, have you found this page? http://slurm.schedmd.com/mc_support.html > That is certainly a very useful resource explaining the different > concepts and how they relate to the terminology used by slurm. However, > skimming through it there does not appear to be a direct connection to > out usage computation question. You might want to (re)read 1. http://slurm.schedmd.com/cons_res.html 2. http://slurm.schedmd.com/cons_res_share.html [1] right at the beginning explains the default behaviour (exclusive mode) [2] should explain best that OverSubscribe (fka. Shared) is depended on the "Selection Setting". If you skim the column "Resulting Behavior" you should know really fast what you aim for. We share nodes between jobs of different users (but: all one trusted group, my colleagues -- really easy case) without any conflict and neither use OverSubscribe nor cgroups (but cgroups is a ticket). Regards, Benjamin -- FSU Jena | JULIELab.de/Staff/Benjamin+Redling.html vox: +49 3641 9 44323 | fax: +49 3641 9 44321
[slurm-dev] Re: resource usage, TRES and --exclusive option
Hello, On Wed, Aug 31, 2016 at 05:52:48PM -0700, Lachlan Musicman wrote: > I don't believe it's 100% necessary to use OverSubscribe Yes. > > We have > > select/cons_res and CR_CPU, on nodes that look like > > NodeName=stpr-res-compute[01-02] CPUs=40 RealMemory=385000 Sockets=2 > CoresPerSocket=10 ThreadsPerCore=2 State=UNKNOWN OK, I will keep this in mind. > > The CPUs aren't shared, but there's 40 of them per node, so that doesn't > really matter so much. I do not understand this. What part of a node is shared if not its processing resource ? Or may be there is a misunderstanding because my question was not well formulated. So I will rephrase it: What we are interested in is a situation where of 40 CPUs to one user 10 are allocated and the remaining 30 to for example one other user. So no actual oversubscription of CPUs but a sharing of the nodes CPUs between different users which get their exclusive cpusets. It has 40 CPUs as GRES and up to 40 can be allocated as exlusive resources to different users. I think cgroups would provide the necessary controls for this. Now, the point we are not sure about is what happens if a user allocates 10 out of 40 and sets "--exclusive" (if possible). Is the usage of that user (job) actually computed with 40 CPUs as most people would expect ? As described before other systems appear to bill only 10 while 40 are in fact used up. > > Also, have you found this page? http://slurm.schedmd.com/mc_support.html That is certainly a very useful resource explaining the different concepts and how they relate to the terminology used by slurm. However, skimming through it there does not appear to be a direct connection to out usage computation question. Best Regards Christof > > I only found last week, but made things a lot clearer in my head. > > > Cheers > L. -- Dr. rer. nat. Christof Köhler email: c.koeh...@bccms.uni-bremen.de Universitaet Bremen/ BCCMS phone: +49-(0)421-218-62334 Am Fallturm 1/ TAB/ Raum 3.12 fax: +49-(0)421-218-62770 28359 Bremen PGP: http://www.bccms.uni-bremen.de/cms/people/c_koehler/ signature.asc Description: Digital signature
[slurm-dev] Re: resource usage, TRES and --exclusive option
-- The most dangerous phrase in the language is, "We've always done it this way." - Grace Hopper On 31 August 2016 at 21:07, Christof Koehler < christof.koeh...@bccms.uni-bremen.de> wrote: > Hello everybody, > > If I understand the slurm documentation correctly the usual configuration > in slurm would be to allocate resources (i.e. whole nodes) exclusively > to a single job only. Setting select/cons_res (and subsequent > configuration of consumables) as well as OverSubscribe Yes would in > principle enable a user to decide whether to allow sharing of resources > by setting "--share/-s" or "--exclusive" as options to sbatch. > > Is this correct so far ? How is the resource usage calculated then ? I don't believe it's 100% necessary to use OverSubscribe Yes. We have select/cons_res and CR_CPU, on nodes that look like NodeName=stpr-res-compute[01-02] CPUs=40 RealMemory=385000 Sockets=2 CoresPerSocket=10 ThreadsPerCore=2 State=UNKNOWN Our users can share nodes. In this conf, there are 40 cpus/node, and resources are distributed as per each user's request. The CPUs aren't shared, but there's 40 of them per node, so that doesn't really matter so much. Also, have you found this page? http://slurm.schedmd.com/mc_support.html I only found last week, but made things a lot clearer in my head. Cheers L.
