[slurm-users] How to config slurm so that I can retrieve all of the history job logs.
Hi, I config slurm with the following log settings: werner@ubuntu-01:~$ scontrol show config | grep -i logfile SlurmctldLogFile= /var/log/slurm/slurmctld.log SlurmdLogFile = /var/log/slurm/slurmd.log SlurmSchedLogFile = /var/log/slurm/slurmsched.log But this still can let me retrieve all of the history job logs for the completed jobs: werner@ubuntu-01:~$ scontrol show jobs No jobs in the system Any hints for my problem? Regards -- Hongyi Zhao
Re: [slurm-users] [EXT] Re: Limit the number of GPUS per user per partition
Do you have other limits set? The QoS is hierarchical, and especially partition QoS can override other QoS. What's the output of sacctmgr show qos -p and scontrol show part Sean -- Sean Crosby | Senior DevOpsHPC Engineer and HPC Team Lead Research Computing Services | Business Services The University of Melbourne, Victoria 3010 Australia On Wed, 6 May 2020 at 23:44, Theis, Thomas wrote: > *UoM notice: External email. Be cautious of links, attachments, or > impersonation attempts.* > -- > > Still have the same issue when I updated the user and qos.. > > Command I am using. > > ‘sacctmgr modify qos normal set MaxTRESPerUser=gres/gpu=2’ > > I restarted the services. Unfortunately I am still have to saturate the > cluster with jobs. > > > > We have a cluster of 10 nodes each with 4 gpus, for a total of 40 gpus. > Each node is identical in the software, OS, SLURM. etc.. I am trying to > limit each user to only be able to use 2 out of 40 gpus across the entire > cluster or partition. A intended bottle neck so no one can saturate the > cluster.. > > > > I.E. desired outcome would be. Person A submits 100 jobs, 2 would run , > and 98 would be pending, 38 gpus would be idle. Once the 2 running are > finished, 2 more would run and 96 would be pending, still 38 gpus would be > idle.. > > > > > > > > *Thomas Theis* > > > > *From:* slurm-users *On Behalf Of > *Sean Crosby > *Sent:* Tuesday, May 5, 2020 6:48 PM > *To:* Slurm User Community List > *Subject:* Re: [slurm-users] [EXT] Re: Limit the number of GPUS per user > per partition > > > > External Email > > Hi Thomas, > > > > That value should be > > > > sacctmgr modify qos gpujobs set MaxTRESPerUser=gres/gpu=4 > > > > Sean > > > > -- > Sean Crosby | Senior DevOpsHPC Engineer and HPC Team Lead > Research Computing Services | Business Services > The University of Melbourne, Victoria 3010 Australia > > > > > > On Wed, 6 May 2020 at 04:53, Theis, Thomas > wrote: > > *UoM notice: External email. Be cautious of links, attachments, or > impersonation attempts.* > -- > > Hey Killian, > > > > I tried to limit the number of gpus a user can run on at a time by adding > MaxTRESPerUser = gres:gpu4 to both the user and the qos.. I restarted slurm > control daemon and unfortunately I am still able to run on all the gpus in > the partition. Any other ideas? > > > > *Thomas Theis* > > > > *From:* slurm-users *On Behalf Of > *Killian Murphy > *Sent:* Thursday, April 23, 2020 1:33 PM > *To:* Slurm User Community List > *Subject:* Re: [slurm-users] Limit the number of GPUS per user per > partition > > > > External Email > > Hi Thomas. > > > > We limit the maximum number of GPUs a user can have allocated in a > partition through the MaxTRESPerUser field of a QoS for GPU jobs, which is > set as the partition QoS on our GPU partition. I.E: > > > > We have a QOS `gpujobs` that sets MaxTRESPerUser => gres/gpu:4 to limit > total number of allocated GPUs to 4, and set the GPU partition QoS to the > `gpujobs` QoS. > > > > There is a section in the Slurm documentation on the 'Resource Limits' > page entitled 'QOS specific limits supported ( > https://slurm.schedmd.com/resource_limits.html) that details some care > needed when using this kind of limit setting with typed GRES. Although it > seems like you are trying to do something with generic GRES, it's worth a > read! > > > > Killian > > > > > > > > On Thu, 23 Apr 2020 at 18:19, Theis, Thomas > wrote: > > Hi everyone, > > First message, I am trying find a good way or multiple ways to limit the > usage of jobs per node or use of gpus per node, without blocking a user > from submitting them. > > > > Example. We have 10 nodes each with 4 gpus in a partition. We allow a team > of 6 people to submit jobs to any or all of the nodes. One job per gpu; > thus we can hold a total of 40 jobs concurrently in the partition. > > At the moment: each user usually submit 50- 100 jobs at once. Taking up > all gpus, and all other users have to wait in pending.. > > > > What I am trying to setup is allow all users to submit as many jobs as > they wish but only run on 1 out of the 4 gpus per node, or some number out > of the total 40 gpus across the entire partition. Using slurm 18.08.3.. > > > > This is roughly our slurm scripts. > > > > #SBATCH --job-name=Name # Job name > > #SBATCH --mem=5gb # Job memory request > > #SBATCH --ntasks=1 > > #SBATCH --gres=gpu:1 > > #SBATCH --partition=PART1 > > #SBATCH --time=200:00:00 # Time limit hrs:min:sec > > #SBATCH --output=job _%j.log # Standard output and error log > > #SBATCH --nodes=1 > > #SBATCH --qos=high > > > > srun -n1 --gres=gpu:1 --exclusive --export=ALL bash -c > "NV_GPU=$SLURM_JOB_GPUS nvidia-docker run --rm -e > SLURM_JOB_ID=$SLURM_JOB_ID -e SLURM_OUTPUT=$SLURM_OUTPUT --name > $SLURM_JOB_ID do_job.sh" > > > > *Thomas Theis* > > > > > > > -- > > Killian Murphy > > Research Software Engineer > > >
Re: [slurm-users] Do not upgrade mysql to 5.7.30!
Yeah, and I found the reason. Seems that (at least for the mysql procedure get_parent_limits) mySQL 5.7.30 returns NULL where mySQL 5.7.29 returned an empty string. Running mySQL < 5.7.30 is a bad idea, as there exist two remotely exploitable bugs with a CVSS score of 9.8! (see also https://www.cbronline.com/news/oracle-security-patches) So, this needs to get fixed soon. Best Marcus Am 06.05.2020 um 20:30 schrieb Dustin Lang: Hi, Ubuntu has made mysql 5.7.30 the default version. At least with Ubuntu 16.04, this causes severe problems with Slurm dbd (v 17.x, 18.x, and 19.x; not sure about 20). Reverting to mysql 5.7.29 seems to make everything work okay again. cheers, --dustin smime.p7s Description: S/MIME Cryptographic Signature
Re: [slurm-users] Job Step Resource Requests are Ignored
That's great! Thanks David! On Wed, May 6, 2020 at 11:35 AM David Braun wrote: > i'm not sure I understand the problem. If you want to make sure the > preamble and postamble run even if the main job doesn't run you can use '-d' > > from the man page > > -d, --dependency= > Defer the start of this job until the > specified dependencies have been satisfied completed. > is of the form >or > . All dependencies must be > satisfied if the "," separator is > used. Any dependency may be satisfied if the "?" > separator is used. Many jobs can share the same dependency and these jobs > may even belong to different > users. The value may be changed after job submission using > the scontrol command. Once a job dependency fails due to the termination > state of a preceding job, > the dependent job will never be run, even if the preceding > job is requeued and has a different termination state in a subsequent > execution. > > > for instance, create a job that contains this: > > preamble_id=`sbatch preamble.job` > main_id=`sbatch -d afterok:$preamble_id main.job` > sbatch -d afterany:$main_id postamble.job > > Best, > > D > > On Wed, May 6, 2020 at 2:19 PM Maria Semple wrote: > >> Hi Chris, >> >> I think my question isn't quite clear, but I'm also pretty confident the >> answer is no at this point. The idea is that the script is sort of like a >> template for running a job, and an end user can submit a custom job with >> their own desired resource requests which will end up filling in the >> template. I'm not in control of the Slurm cluster that will ultimately run >> the job, nor the details of the job itself. For example, template-job.sh >> might look like this: >> >> #!/bin/bash >> srun -c 1 --mem=1k echo "Preamble" >> srun -c --mem=m /bin/sh -c >> srun -c 1 --mem=1k echo "Postamble" >> >> My goal is that even if the user requests 10 CPUs when the cluster only >> has 4 available, the Preamble and Postamble steps will always run. But as I >> said, it seems like that's not possible since the maximum number of CPUs >> needs to be set on the sbatch allocation and the whole job would be >> rejected on the basis that too many CPUs were requested. Is that correct? >> >> On Tue, May 5, 2020, 11:13 PM Chris Samuel wrote: >> >>> On Tuesday, 5 May 2020 11:00:27 PM PDT Maria Semple wrote: >>> >>> > Is there no way to achieve what I want then? I'd like the first and >>> last job >>> > steps to always be able to run, even if the second step needs too many >>> > resources (based on the cluster). >>> >>> That should just work. >>> >>> #!/bin/bash >>> #SBATCH -c 2 >>> #SBATCH -n 1 >>> >>> srun -c 1 echo hello >>> srun -c 4 echo big wide >>> srun -c 1 echo world >>> >>> gives: >>> >>> hello >>> srun: Job step's --cpus-per-task value exceeds that of job (4 > 2). Job >>> step >>> may never run. >>> srun: error: Unable to create step for job 604659: More processors >>> requested >>> than permitted >>> world >>> >>> > As a side note, do you know why it's not even possible to restrict the >>> > number of resources a single step uses (i.e. set less CPUs than are >>> > available to the full job)? >>> >>> My suspicion is that you've not set up Slurm to use cgroups to restrict >>> the >>> resources a job can use to just those requested. >>> >>> https://slurm.schedmd.com/cgroups.html >>> >>> All the best, >>> Chris >>> -- >>> Chris Samuel : http://www.csamuel.org/ : Berkeley, CA, USA >>> >>> >>> >>> >>> -- Thanks, Maria
Re: [slurm-users] Job Step Resource Requests are Ignored
i'm not sure I understand the problem. If you want to make sure the preamble and postamble run even if the main job doesn't run you can use '-d' from the man page -d, --dependency= Defer the start of this job until the specified dependencies have been satisfied completed. is of the form or . All dependencies must be satisfied if the "," separator is used. Any dependency may be satisfied if the "?" separator is used. Many jobs can share the same dependency and these jobs may even belong to different users. The value may be changed after job submission using the scontrol command. Once a job dependency fails due to the termination state of a preceding job, the dependent job will never be run, even if the preceding job is requeued and has a different termination state in a subsequent execution. for instance, create a job that contains this: preamble_id=`sbatch preamble.job` main_id=`sbatch -d afterok:$preamble_id main.job` sbatch -d afterany:$main_id postamble.job Best, D On Wed, May 6, 2020 at 2:19 PM Maria Semple wrote: > Hi Chris, > > I think my question isn't quite clear, but I'm also pretty confident the > answer is no at this point. The idea is that the script is sort of like a > template for running a job, and an end user can submit a custom job with > their own desired resource requests which will end up filling in the > template. I'm not in control of the Slurm cluster that will ultimately run > the job, nor the details of the job itself. For example, template-job.sh > might look like this: > > #!/bin/bash > srun -c 1 --mem=1k echo "Preamble" > srun -c --mem=m /bin/sh -c > srun -c 1 --mem=1k echo "Postamble" > > My goal is that even if the user requests 10 CPUs when the cluster only > has 4 available, the Preamble and Postamble steps will always run. But as I > said, it seems like that's not possible since the maximum number of CPUs > needs to be set on the sbatch allocation and the whole job would be > rejected on the basis that too many CPUs were requested. Is that correct? > > On Tue, May 5, 2020, 11:13 PM Chris Samuel wrote: > >> On Tuesday, 5 May 2020 11:00:27 PM PDT Maria Semple wrote: >> >> > Is there no way to achieve what I want then? I'd like the first and >> last job >> > steps to always be able to run, even if the second step needs too many >> > resources (based on the cluster). >> >> That should just work. >> >> #!/bin/bash >> #SBATCH -c 2 >> #SBATCH -n 1 >> >> srun -c 1 echo hello >> srun -c 4 echo big wide >> srun -c 1 echo world >> >> gives: >> >> hello >> srun: Job step's --cpus-per-task value exceeds that of job (4 > 2). Job >> step >> may never run. >> srun: error: Unable to create step for job 604659: More processors >> requested >> than permitted >> world >> >> > As a side note, do you know why it's not even possible to restrict the >> > number of resources a single step uses (i.e. set less CPUs than are >> > available to the full job)? >> >> My suspicion is that you've not set up Slurm to use cgroups to restrict >> the >> resources a job can use to just those requested. >> >> https://slurm.schedmd.com/cgroups.html >> >> All the best, >> Chris >> -- >> Chris Samuel : http://www.csamuel.org/ : Berkeley, CA, USA >> >> >> >> >>
[slurm-users] Do not upgrade mysql to 5.7.30!
Hi, Ubuntu has made mysql 5.7.30 the default version. At least with Ubuntu 16.04, this causes severe problems with Slurm dbd (v 17.x, 18.x, and 19.x; not sure about 20). Reverting to mysql 5.7.29 seems to make everything work okay again. cheers, --dustin
Re: [slurm-users] Job Step Resource Requests are Ignored
Hi Chris, I think my question isn't quite clear, but I'm also pretty confident the answer is no at this point. The idea is that the script is sort of like a template for running a job, and an end user can submit a custom job with their own desired resource requests which will end up filling in the template. I'm not in control of the Slurm cluster that will ultimately run the job, nor the details of the job itself. For example, template-job.sh might look like this: #!/bin/bash srun -c 1 --mem=1k echo "Preamble" srun -c --mem=m /bin/sh -c srun -c 1 --mem=1k echo "Postamble" My goal is that even if the user requests 10 CPUs when the cluster only has 4 available, the Preamble and Postamble steps will always run. But as I said, it seems like that's not possible since the maximum number of CPUs needs to be set on the sbatch allocation and the whole job would be rejected on the basis that too many CPUs were requested. Is that correct? On Tue, May 5, 2020, 11:13 PM Chris Samuel wrote: > On Tuesday, 5 May 2020 11:00:27 PM PDT Maria Semple wrote: > > > Is there no way to achieve what I want then? I'd like the first and last > job > > steps to always be able to run, even if the second step needs too many > > resources (based on the cluster). > > That should just work. > > #!/bin/bash > #SBATCH -c 2 > #SBATCH -n 1 > > srun -c 1 echo hello > srun -c 4 echo big wide > srun -c 1 echo world > > gives: > > hello > srun: Job step's --cpus-per-task value exceeds that of job (4 > 2). Job > step > may never run. > srun: error: Unable to create step for job 604659: More processors > requested > than permitted > world > > > As a side note, do you know why it's not even possible to restrict the > > number of resources a single step uses (i.e. set less CPUs than are > > available to the full job)? > > My suspicion is that you've not set up Slurm to use cgroups to restrict > the > resources a job can use to just those requested. > > https://slurm.schedmd.com/cgroups.html > > All the best, > Chris > -- > Chris Samuel : http://www.csamuel.org/ : Berkeley, CA, USA > > > > >
Re: [slurm-users] how to restrict jobs
Is there no way to set or define a custom variable like at node level and you could use a per-node Feature for this, but a partition would also work.
Re: [slurm-users] how to restrict jobs
Is there no way to set or define a custom variable like at node level and then you pass the same variable in the job request so that it will land into those nodes only. Regards Navin On Wed, May 6, 2020, 21:04 Renfro, Michael wrote: > Ok, then regular license accounting won’t work. > > Somewhat tested, but should work or at least be a starting point. Given a > job number JOBID that’s already running with this license on one or more > nodes: > > sbatch -w $(scontrol show job JOBID | grep ' NodeList=' | cut -d= -f2) > -N 1 > > should start a one-node job on an available node being used by JOBID. Add > other parameters as required for cpus-per-task, time limits, or whatever > else is needed. If you start the larger jobs first, and let the later jobs > fill in on idle CPUs on those nodes, it should work. > > > On May 6, 2020, at 9:46 AM, navin srivastava > wrote: > > > > To explain with more details. > > > > job will be submitted based on core at any time but it will go to any > random nodes but limited to 4 Nodes only.(license having some intelligence > that it calculate the nodes and if it reached to 4 then it will not allow > any more nodes. yes it didn't depend on the no of core available on nodes. > > > > Case-1 if 4 jobs running with 4 cores each on 4 nodes [node1, node2, > node3 and node4] > > Again Fifth job assigned by SLURM with 4 cores on any one > node of node1, node2, node3 and node4 then license will be allowed. > > > > Case-2 if 4 jobs running with 4 cores each on 4 nodes [node1, node2, > node3 and node4] > > Again Fifth job assigned by SLURM on node5 with 4 cores > then license will not allowed [ license not found error came in this case] > > > > Regards > > Navin. > > > > > > On Wed, May 6, 2020 at 7:47 PM Renfro, Michael > wrote: > > To make sure I’m reading this correctly, you have a software license > that lets you run jobs on up to 4 nodes at once, regardless of how many > CPUs you use? That is, you could run any one of the following sets of jobs: > > > > - four 1-node jobs, > > - two 2-node jobs, > > - one 1-node and one 3-node job, > > - two 1-node and one 2-node jobs, > > - one 4-node job, > > > > simultaneously? And the license isn’t node-locked to specific nodes by > MAC address or anything similar? But if you try to run jobs beyond what > I’ve listed above, you run out of licenses, and you want those later jobs > to be held until licenses are freed up? > > > > If all of those questions have an answer of ‘yes’, I think you want the > remote license part of the https://slurm.schedmd.com/licenses.html, > something like: > > > > sacctmgr add resource name=software_name count=4 percentallowed=100 > server=flex_host servertype=flexlm type=license > > > > and submit jobs with a '-L software_name:N’ flag where N is the number > of nodes you want to run on. > > > > > On May 6, 2020, at 5:33 AM, navin srivastava > wrote: > > > > > > Thanks Micheal. > > > > > > Actually one application license are based on node and we have 4 Node > license( not a fix node). we have several nodes but when job lands on any 4 > random nodes it runs on those nodes only. After that it fails if it goes to > other nodes. > > > > > > can we define a custom variable and set it on the node level and when > user submit it will pass that variable and then job will and onto those > specific nodes? > > > i do not want to create a separate partition. > > > > > > is there any way to achieve this by any other method? > > > > > > Regards > > > Navin. > > > > > > > > > Regards > > > Navin. > > > > > > On Tue, May 5, 2020 at 7:46 PM Renfro, Michael > wrote: > > > Haven’t done it yet myself, but it’s on my todo list. > > > > > > But I’d assume that if you use the FlexLM or RLM parts of that > documentation, that Slurm would query the remote license server > periodically and hold the job until the necessary licenses were available. > > > > > > > On May 5, 2020, at 8:37 AM, navin srivastava > wrote: > > > > > > > > External Email Warning > > > > This email originated from outside the university. Please use > caution when opening attachments, clicking links, or responding to requests. > > > > Thanks Michael, > > > > > > > > yes i have gone through but the licenses are remote license and it > will be used by outside as well not only in slurm. > > > > so basically i am interested to know how we can update the database > dynamically to get the exact value at that point of time. > > > > i mean query the license server and update the database accordingly. > does slurm automatically updated the value based on usage? > > > > > > > > > > > > Regards > > > > Navin. > > > > > > > > > > > > On Tue, May 5, 2020 at 7:00 PM Renfro, Michael > wrote: > > > > Have you seen https://slurm.schedmd.com/licenses.html already? If > the software is just for use inside the cluster, one Licenses= line in > slurm.conf plus users submitting with the -L flag should suffice. Should be > able to set that license value is 4
Re: [slurm-users] how to restrict jobs
Ok, then regular license accounting won’t work. Somewhat tested, but should work or at least be a starting point. Given a job number JOBID that’s already running with this license on one or more nodes: sbatch -w $(scontrol show job JOBID | grep ' NodeList=' | cut -d= -f2) -N 1 should start a one-node job on an available node being used by JOBID. Add other parameters as required for cpus-per-task, time limits, or whatever else is needed. If you start the larger jobs first, and let the later jobs fill in on idle CPUs on those nodes, it should work. > On May 6, 2020, at 9:46 AM, navin srivastava wrote: > > To explain with more details. > > job will be submitted based on core at any time but it will go to any random > nodes but limited to 4 Nodes only.(license having some intelligence that it > calculate the nodes and if it reached to 4 then it will not allow any more > nodes. yes it didn't depend on the no of core available on nodes. > > Case-1 if 4 jobs running with 4 cores each on 4 nodes [node1, node2, node3 > and node4] > Again Fifth job assigned by SLURM with 4 cores on any one node > of node1, node2, node3 and node4 then license will be allowed. > > Case-2 if 4 jobs running with 4 cores each on 4 nodes [node1, node2, node3 > and node4] > Again Fifth job assigned by SLURM on node5 with 4 cores then > license will not allowed [ license not found error came in this case] > > Regards > Navin. > > > On Wed, May 6, 2020 at 7:47 PM Renfro, Michael wrote: > To make sure I’m reading this correctly, you have a software license that > lets you run jobs on up to 4 nodes at once, regardless of how many CPUs you > use? That is, you could run any one of the following sets of jobs: > > - four 1-node jobs, > - two 2-node jobs, > - one 1-node and one 3-node job, > - two 1-node and one 2-node jobs, > - one 4-node job, > > simultaneously? And the license isn’t node-locked to specific nodes by MAC > address or anything similar? But if you try to run jobs beyond what I’ve > listed above, you run out of licenses, and you want those later jobs to be > held until licenses are freed up? > > If all of those questions have an answer of ‘yes’, I think you want the > remote license part of the https://slurm.schedmd.com/licenses.html, something > like: > > sacctmgr add resource name=software_name count=4 percentallowed=100 > server=flex_host servertype=flexlm type=license > > and submit jobs with a '-L software_name:N’ flag where N is the number of > nodes you want to run on. > > > On May 6, 2020, at 5:33 AM, navin srivastava wrote: > > > > Thanks Micheal. > > > > Actually one application license are based on node and we have 4 Node > > license( not a fix node). we have several nodes but when job lands on any 4 > > random nodes it runs on those nodes only. After that it fails if it goes to > > other nodes. > > > > can we define a custom variable and set it on the node level and when user > > submit it will pass that variable and then job will and onto those specific > > nodes? > > i do not want to create a separate partition. > > > > is there any way to achieve this by any other method? > > > > Regards > > Navin. > > > > > > Regards > > Navin. > > > > On Tue, May 5, 2020 at 7:46 PM Renfro, Michael wrote: > > Haven’t done it yet myself, but it’s on my todo list. > > > > But I’d assume that if you use the FlexLM or RLM parts of that > > documentation, that Slurm would query the remote license server > > periodically and hold the job until the necessary licenses were available. > > > > > On May 5, 2020, at 8:37 AM, navin srivastava > > > wrote: > > > > > > External Email Warning > > > This email originated from outside the university. Please use caution > > > when opening attachments, clicking links, or responding to requests. > > > Thanks Michael, > > > > > > yes i have gone through but the licenses are remote license and it will > > > be used by outside as well not only in slurm. > > > so basically i am interested to know how we can update the database > > > dynamically to get the exact value at that point of time. > > > i mean query the license server and update the database accordingly. does > > > slurm automatically updated the value based on usage? > > > > > > > > > Regards > > > Navin. > > > > > > > > > On Tue, May 5, 2020 at 7:00 PM Renfro, Michael wrote: > > > Have you seen https://slurm.schedmd.com/licenses.html already? If the > > > software is just for use inside the cluster, one Licenses= line in > > > slurm.conf plus users submitting with the -L flag should suffice. Should > > > be able to set that license value is 4 if it’s licensed per node and you > > > can run up to 4 jobs simultaneously, or 4*NCPUS if it’s licensed per CPU, > > > or 1 if it’s a single license good for one run from 1-4 nodes. > > > > > > There are also options to query a FlexLM or RLM server for license > > > management. > > > > > > -- > >
Re: [slurm-users] how to restrict jobs
To explain with more details. job will be submitted based on core at any time but it will go to any random nodes but limited to 4 Nodes only.(license having some intelligence that it calculate the nodes and if it reached to 4 then it will not allow any more nodes. yes it didn't depend on the no of core available on nodes. Case-1 if 4 jobs running with 4 cores each on 4 nodes [node1, node2, node3 and node4] Again Fifth job assigned by SLURM with 4 cores on any one node of node1, node2, node3 and node4 then license will be allowed. Case-2 if 4 jobs running with 4 cores each on 4 nodes [node1, node2, node3 and node4] Again Fifth job assigned by SLURM on node5 with 4 cores then license will not allowed [ license not found error came in this case] Regards Navin. On Wed, May 6, 2020 at 7:47 PM Renfro, Michael wrote: > To make sure I’m reading this correctly, you have a software license that > lets you run jobs on up to 4 nodes at once, regardless of how many CPUs you > use? That is, you could run any one of the following sets of jobs: > > - four 1-node jobs, > - two 2-node jobs, > - one 1-node and one 3-node job, > - two 1-node and one 2-node jobs, > - one 4-node job, > > simultaneously? And the license isn’t node-locked to specific nodes by MAC > address or anything similar? But if you try to run jobs beyond what I’ve > listed above, you run out of licenses, and you want those later jobs to be > held until licenses are freed up? > > If all of those questions have an answer of ‘yes’, I think you want the > remote license part of the https://slurm.schedmd.com/licenses.html, > something like: > > sacctmgr add resource name=software_name count=4 percentallowed=100 > server=flex_host servertype=flexlm type=license > > and submit jobs with a '-L software_name:N’ flag where N is the number of > nodes you want to run on. > > > On May 6, 2020, at 5:33 AM, navin srivastava > wrote: > > > > Thanks Micheal. > > > > Actually one application license are based on node and we have 4 Node > license( not a fix node). we have several nodes but when job lands on any 4 > random nodes it runs on those nodes only. After that it fails if it goes to > other nodes. > > > > can we define a custom variable and set it on the node level and when > user submit it will pass that variable and then job will and onto those > specific nodes? > > i do not want to create a separate partition. > > > > is there any way to achieve this by any other method? > > > > Regards > > Navin. > > > > > > Regards > > Navin. > > > > On Tue, May 5, 2020 at 7:46 PM Renfro, Michael > wrote: > > Haven’t done it yet myself, but it’s on my todo list. > > > > But I’d assume that if you use the FlexLM or RLM parts of that > documentation, that Slurm would query the remote license server > periodically and hold the job until the necessary licenses were available. > > > > > On May 5, 2020, at 8:37 AM, navin srivastava > wrote: > > > > > > External Email Warning > > > This email originated from outside the university. Please use caution > when opening attachments, clicking links, or responding to requests. > > > Thanks Michael, > > > > > > yes i have gone through but the licenses are remote license and it > will be used by outside as well not only in slurm. > > > so basically i am interested to know how we can update the database > dynamically to get the exact value at that point of time. > > > i mean query the license server and update the database accordingly. > does slurm automatically updated the value based on usage? > > > > > > > > > Regards > > > Navin. > > > > > > > > > On Tue, May 5, 2020 at 7:00 PM Renfro, Michael > wrote: > > > Have you seen https://slurm.schedmd.com/licenses.html already? If the > software is just for use inside the cluster, one Licenses= line in > slurm.conf plus users submitting with the -L flag should suffice. Should be > able to set that license value is 4 if it’s licensed per node and you can > run up to 4 jobs simultaneously, or 4*NCPUS if it’s licensed per CPU, or 1 > if it’s a single license good for one run from 1-4 nodes. > > > > > > There are also options to query a FlexLM or RLM server for license > management. > > > > > > -- > > > Mike Renfro, PhD / HPC Systems Administrator, Information Technology > Services > > > 931 372-3601 / Tennessee Tech University > > > > > > > On May 5, 2020, at 7:54 AM, navin srivastava > wrote: > > > > > > > > Hi Team, > > > > > > > > we have an application whose licenses is limited .it scales upto 4 > nodes(~80 cores). > > > > so if 4 nodes are full, in 5th node job used to get fail. > > > > we want to put a restriction so that the application can't go for > the execution beyond the 4 nodes and fail it should be in queue state. > > > > i do not want to keep a separate partition to achieve this config.is > there a way to achieve this scenario using some dynamic resource which can > call the license variable on the fly and if it is reached it should keep
Re: [slurm-users] how to restrict jobs
To make sure I’m reading this correctly, you have a software license that lets you run jobs on up to 4 nodes at once, regardless of how many CPUs you use? That is, you could run any one of the following sets of jobs: - four 1-node jobs, - two 2-node jobs, - one 1-node and one 3-node job, - two 1-node and one 2-node jobs, - one 4-node job, simultaneously? And the license isn’t node-locked to specific nodes by MAC address or anything similar? But if you try to run jobs beyond what I’ve listed above, you run out of licenses, and you want those later jobs to be held until licenses are freed up? If all of those questions have an answer of ‘yes’, I think you want the remote license part of the https://slurm.schedmd.com/licenses.html, something like: sacctmgr add resource name=software_name count=4 percentallowed=100 server=flex_host servertype=flexlm type=license and submit jobs with a '-L software_name:N’ flag where N is the number of nodes you want to run on. > On May 6, 2020, at 5:33 AM, navin srivastava wrote: > > Thanks Micheal. > > Actually one application license are based on node and we have 4 Node > license( not a fix node). we have several nodes but when job lands on any 4 > random nodes it runs on those nodes only. After that it fails if it goes to > other nodes. > > can we define a custom variable and set it on the node level and when user > submit it will pass that variable and then job will and onto those specific > nodes? > i do not want to create a separate partition. > > is there any way to achieve this by any other method? > > Regards > Navin. > > > Regards > Navin. > > On Tue, May 5, 2020 at 7:46 PM Renfro, Michael wrote: > Haven’t done it yet myself, but it’s on my todo list. > > But I’d assume that if you use the FlexLM or RLM parts of that documentation, > that Slurm would query the remote license server periodically and hold the > job until the necessary licenses were available. > > > On May 5, 2020, at 8:37 AM, navin srivastava wrote: > > > > External Email Warning > > This email originated from outside the university. Please use caution when > > opening attachments, clicking links, or responding to requests. > > Thanks Michael, > > > > yes i have gone through but the licenses are remote license and it will be > > used by outside as well not only in slurm. > > so basically i am interested to know how we can update the database > > dynamically to get the exact value at that point of time. > > i mean query the license server and update the database accordingly. does > > slurm automatically updated the value based on usage? > > > > > > Regards > > Navin. > > > > > > On Tue, May 5, 2020 at 7:00 PM Renfro, Michael wrote: > > Have you seen https://slurm.schedmd.com/licenses.html already? If the > > software is just for use inside the cluster, one Licenses= line in > > slurm.conf plus users submitting with the -L flag should suffice. Should be > > able to set that license value is 4 if it’s licensed per node and you can > > run up to 4 jobs simultaneously, or 4*NCPUS if it’s licensed per CPU, or 1 > > if it’s a single license good for one run from 1-4 nodes. > > > > There are also options to query a FlexLM or RLM server for license > > management. > > > > -- > > Mike Renfro, PhD / HPC Systems Administrator, Information Technology > > Services > > 931 372-3601 / Tennessee Tech University > > > > > On May 5, 2020, at 7:54 AM, navin srivastava > > > wrote: > > > > > > Hi Team, > > > > > > we have an application whose licenses is limited .it scales upto 4 > > > nodes(~80 cores). > > > so if 4 nodes are full, in 5th node job used to get fail. > > > we want to put a restriction so that the application can't go for the > > > execution beyond the 4 nodes and fail it should be in queue state. > > > i do not want to keep a separate partition to achieve this config.is > > > there a way to achieve this scenario using some dynamic resource which > > > can call the license variable on the fly and if it is reached it should > > > keep the job in queue. > > > > > > Regards > > > Navin. > > > > > > > > > > > >
Re: [slurm-users] "sacctmgr add cluster" crashing slurmdbd
Sorry, forgot, we use by the way, slurm 18.08.7 I just saw, in an earlier coredump, that there is another (earlier) line involved: 2136: if (row2[ASSOC2_REQ_MTPJ][0]) the corresponding mysql response was: +-+--+--+--+--+---+---+---++---+-+--++ | @par_id | @mj | @mja | @mpt | @msj | @mwpj | @mtpj | @mtpn | @mtmpj | @mtrm | @def_qos_id | @qos | @delta_qos | +-+--+--+--+--+---+---+---++---+-+--++ | 990 | 800 | NULL | NULL | 1000 | 1440 | NULL | NULL | NULL | NULL |NULL | ,1, | NULL | +-+--+--+--+--+---+---+---++---+-+--++ 1 row in set (0.00 sec) So, here, @mtpj is NULL, in the other coredump, it was "1=8", so it was not NULL. But @mtpn was NULL, so it segfaulted in 2141:if (row2[ASSOC2_REQ_MTPN][0]) Could anyone with a not segfaulting slurmdbd please use the call directly in the database (it is a procedure generated by slurmdbd collecting the parent limits of an association) and report the result here? Best Marcus Am 06.05.2020 um 09:49 schrieb Ben Polman: On 06-05-2020 07:38, Chris Samuel wrote: We are experiencing exactly the same problem after mysql upgrade to 5.7.30, moving database to old mysql server running 5.6 solves the problem. Most likely downgrading mysql to 5.7.29 will work as well I have no clue which change in mysql-server is causing this best regards, Ben On Tuesday, 5 May 2020 3:21:45 PM PDT Dustin Lang wrote: Since this happens on a fresh new database, I just don't understand how I can get back to a basic functional state. This is exceedingly frustrating. I have to say that if you're seeing this with 17.11, 18.08 and 19.05 and this only started when your colleague upgraded MySQL then this sounds like MySQL is triggering this problem. We're running with MariaDB 10.x (from SLES15) without issues (our database is huge). All the best, Chris smime.p7s Description: S/MIME Cryptographic Signature
Re: [slurm-users] [EXT] Re: Limit the number of GPUS per user per partition
Still have the same issue when I updated the user and qos.. Command I am using. ‘sacctmgr modify qos normal set MaxTRESPerUser=gres/gpu=2’ I restarted the services. Unfortunately I am still have to saturate the cluster with jobs. We have a cluster of 10 nodes each with 4 gpus, for a total of 40 gpus. Each node is identical in the software, OS, SLURM. etc.. I am trying to limit each user to only be able to use 2 out of 40 gpus across the entire cluster or partition. A intended bottle neck so no one can saturate the cluster.. I.E. desired outcome would be. Person A submits 100 jobs, 2 would run , and 98 would be pending, 38 gpus would be idle. Once the 2 running are finished, 2 more would run and 96 would be pending, still 38 gpus would be idle.. Thomas Theis From: slurm-users On Behalf Of Sean Crosby Sent: Tuesday, May 5, 2020 6:48 PM To: Slurm User Community List Subject: Re: [slurm-users] [EXT] Re: Limit the number of GPUS per user per partition External Email Hi Thomas, That value should be sacctmgr modify qos gpujobs set MaxTRESPerUser=gres/gpu=4 Sean -- Sean Crosby | Senior DevOpsHPC Engineer and HPC Team Lead Research Computing Services | Business Services The University of Melbourne, Victoria 3010 Australia On Wed, 6 May 2020 at 04:53, Theis, Thomas mailto:thomas.th...@teledyne.com>> wrote: UoM notice: External email. Be cautious of links, attachments, or impersonation attempts. Hey Killian, I tried to limit the number of gpus a user can run on at a time by adding MaxTRESPerUser = gres:gpu4 to both the user and the qos.. I restarted slurm control daemon and unfortunately I am still able to run on all the gpus in the partition. Any other ideas? Thomas Theis From: slurm-users mailto:slurm-users-boun...@lists.schedmd.com>> On Behalf Of Killian Murphy Sent: Thursday, April 23, 2020 1:33 PM To: Slurm User Community List mailto:slurm-users@lists.schedmd.com>> Subject: Re: [slurm-users] Limit the number of GPUS per user per partition External Email Hi Thomas. We limit the maximum number of GPUs a user can have allocated in a partition through the MaxTRESPerUser field of a QoS for GPU jobs, which is set as the partition QoS on our GPU partition. I.E: We have a QOS `gpujobs` that sets MaxTRESPerUser => gres/gpu:4 to limit total number of allocated GPUs to 4, and set the GPU partition QoS to the `gpujobs` QoS. There is a section in the Slurm documentation on the 'Resource Limits' page entitled 'QOS specific limits supported (https://slurm.schedmd.com/resource_limits.html) that details some care needed when using this kind of limit setting with typed GRES. Although it seems like you are trying to do something with generic GRES, it's worth a read! Killian On Thu, 23 Apr 2020 at 18:19, Theis, Thomas mailto:thomas.th...@teledyne.com>> wrote: Hi everyone, First message, I am trying find a good way or multiple ways to limit the usage of jobs per node or use of gpus per node, without blocking a user from submitting them. Example. We have 10 nodes each with 4 gpus in a partition. We allow a team of 6 people to submit jobs to any or all of the nodes. One job per gpu; thus we can hold a total of 40 jobs concurrently in the partition. At the moment: each user usually submit 50- 100 jobs at once. Taking up all gpus, and all other users have to wait in pending.. What I am trying to setup is allow all users to submit as many jobs as they wish but only run on 1 out of the 4 gpus per node, or some number out of the total 40 gpus across the entire partition. Using slurm 18.08.3.. This is roughly our slurm scripts. #SBATCH --job-name=Name # Job name #SBATCH --mem=5gb # Job memory request #SBATCH --ntasks=1 #SBATCH --gres=gpu:1 #SBATCH --partition=PART1 #SBATCH --time=200:00:00 # Time limit hrs:min:sec #SBATCH --output=job _%j.log # Standard output and error log #SBATCH --nodes=1 #SBATCH --qos=high srun -n1 --gres=gpu:1 --exclusive --export=ALL bash -c "NV_GPU=$SLURM_JOB_GPUS nvidia-docker run --rm -e SLURM_JOB_ID=$SLURM_JOB_ID -e SLURM_OUTPUT=$SLURM_OUTPUT --name $SLURM_JOB_ID do_job.sh" Thomas Theis -- Killian Murphy Research Software Engineer Wolfson Atmospheric Chemistry Laboratories University of York Heslington York YO10 5DD +44 (0)1904 32 4753 e-mail disclaimer: http://www.york.ac.uk/docs/disclaimer/email.htm
Re: [slurm-users] "sacctmgr add cluster" crashing slurmdbd
Hi, same here :/
the segfault happens after the procedure call in mysql:
call get_parent_limits('assoc_table', 'rwth0515', 'rcc', 0); select
@par_id, @mj, @mja, @mpt, @msj, @mwpj, @mtpj, @mtpn, @mtmpj, @mtrm,
@def_qos_id, @qos, @delta_qos;
The mysql answer is:
+-+--+--+--+--+---+---+---++---+-+--++
| @par_id | @mj | @mja | @mpt | @msj | @mwpj | @mtpj | @mtpn | @mtmpj |
@mtrm | @def_qos_id | @qos | @delta_qos |
+-+--+--+--+--+---+---+---++---+-+--++
|5312 | 800 | NULL | NULL | 1000 | 1440 | 1=8 | NULL | NULL |
NULL |NULL | ,1, | NULL |
+-+--+--+--+--+---+---+---++---+-+--++
the segfault happens in as_mysql_assoc.c:
#0 0x2ae3dea6c05a in _cluster_get_assocs
(mysql_conn=mysql_conn@entry=0x2ae3f4000d70,
user=user@entry=0x2ae3e1feca90,
assoc_cond=assoc_cond@entry=0x2ae3f40009f0, cluster_name=0x63f110 "rcc",
fields=, sent_extra=,
is_admin=is_admin@entry=true, sent_list=sent_list@entry=0x6dc030) at
as_mysql_assoc.c:2141
2141if (row2[ASSOC2_REQ_MTPN][0])
hope that helps anyone.
Best
Marcus
Am 06.05.2020 um 09:49 schrieb Ben Polman:
On 06-05-2020 07:38, Chris Samuel wrote:
We are experiencing exactly the same problem after mysql upgrade to 5.7.30,
moving database to old mysql server running 5.6 solves the problem.
Most likely downgrading mysql to 5.7.29 will work as well
I have no clue which change in mysql-server is causing this
best regards,
Ben
On Tuesday, 5 May 2020 3:21:45 PM PDT Dustin Lang wrote:
Since this happens on a fresh new database, I just don't understand how I
can get back to a basic functional state. This is exceedingly frustrating.
I have to say that if you're seeing this with 17.11, 18.08 and 19.05 and this
only started when your colleague upgraded MySQL then this sounds like MySQL is
triggering this problem.
We're running with MariaDB 10.x (from SLES15) without issues (our database is
huge).
All the best,
Chris
smime.p7s
Description: S/MIME Cryptographic Signature
[slurm-users] Slurm memory units
Hi all. I'm probably making a rookie error here...which 'megabyte' (powers of 1000 or 1024) does the Slurm documentation refer to in, for example, the slurm.conf documentation for RealMemory and the sbatch documentation for `--mem`? Most of our nodes have the same physical memory configuration. From the output of `free -m` and `slurmd -C` on one of the nodes, we have 191668M (187G). Consequently, `RealMemory` for those nodes has been set to 191668 in slurm.conf. As a result, when a user requests memory above '187G' for node memory, Slurm reports to them that the requested node configuration is not available. Only...we have 191668MiB of system memory, not MB. `free -m --si` (use powers of 1000, not 1024) reports 192628MB of system memory (which, frustratingly indicates that the 'free' documentation is also not using the newer unit names). So it seems as though Slurm is working in powers of 1024, not powers of 1000. I'm probably just confused about the unit definitions, or there is some convention being applied here, but would appreciate some confirmation either way! Thanks. Killian -- Killian Murphy Research Software Engineer Wolfson Atmospheric Chemistry Laboratories University of York Heslington York YO10 5DD +44 (0)1904 32 4753 e-mail disclaimer: http://www.york.ac.uk/docs/disclaimer/email.htm
Re: [slurm-users] Slurm memory units
More investigation and your message has confirmed for me that it's all working in powers of 1024 (which is what I would expect, although the use of the word 'megabytes' in the doc is a little misleading, I think...). So, our nodes have 187GiB total memory, and we need to re-jig our user documentation to reflect that 187G is the requestable cap for running on the '192GB' nodes. On Wed, 6 May 2020 at 11:12, Peter Kjellström wrote: > On Wed, 6 May 2020 10:42:46 +0100 > Killian Murphy wrote: > > > Hi all. > > > > I'm probably making a rookie error here...which 'megabyte' (powers of > > 1000 or 1024) does the Slurm documentation refer to in, for example, > > the slurm.conf documentation for RealMemory and the sbatch > > documentation for `--mem`? > > > > Most of our nodes have the same physical memory configuration. From > > the output of `free -m` and `slurmd -C` on one of the nodes, we have > > 191668M (187G). Consequently, `RealMemory` for those nodes has been > > set to 191668 in slurm.conf. As a result, when a user requests memory > > above '187G' for node memory, Slurm reports to them that the > > requested node configuration is not available. > > Well yeah it's all 2-base. But it seems you have two problems 1) the > units 2) users expecting 192GiB out of your nodes but the actual > available memory is always lower (187G in your case). > > We see #2 also in that users know our thin nodes are 96G and some then > proceed to request 96G (which does not fit on the 96G nodes...). > > From our system: > $ LOCALINTERACTIVECOMMAND interactive --mem=3g -n 1 > ... > $ > cat /sys/fs/cgroup/memory/slurm/uid_x/job_y/memory.limit_in_bytes > 3221225472 # 3.0 GiB > > (this on slurm-18.08.8 with mem cgroups) > > /Peter > > > Only...we have 191668MiB of system memory, not MB. `free -m --si` (use > > powers of 1000, not 1024) reports 192628MB of system memory (which, > > frustratingly indicates that the 'free' documentation is also not > > using the newer unit names). So it seems as though Slurm is working > > in powers of 1024, not powers of 1000. > > > > I'm probably just confused about the unit definitions, or there is > > some convention being applied here, but would appreciate some > > confirmation either way! > > > > Thanks. > > > > Killian > > -- Killian Murphy Research Software Engineer Wolfson Atmospheric Chemistry Laboratories University of York Heslington York YO10 5DD +44 (0)1904 32 4753 e-mail disclaimer: http://www.york.ac.uk/docs/disclaimer/email.htm
Re: [slurm-users] how to restrict jobs
Thanks Micheal. Actually one application license are based on node and we have 4 Node license( not a fix node). we have several nodes but when job lands on any 4 random nodes it runs on those nodes only. After that it fails if it goes to other nodes. can we define a custom variable and set it on the node level and when user submit it will pass that variable and then job will and onto those specific nodes? i do not want to create a separate partition. is there any way to achieve this by any other method? Regards Navin. Regards Navin. On Tue, May 5, 2020 at 7:46 PM Renfro, Michael wrote: > Haven’t done it yet myself, but it’s on my todo list. > > But I’d assume that if you use the FlexLM or RLM parts of that > documentation, that Slurm would query the remote license server > periodically and hold the job until the necessary licenses were available. > > > On May 5, 2020, at 8:37 AM, navin srivastava > wrote: > > > > External Email Warning > > This email originated from outside the university. Please use caution > when opening attachments, clicking links, or responding to requests. > > Thanks Michael, > > > > yes i have gone through but the licenses are remote license and it will > be used by outside as well not only in slurm. > > so basically i am interested to know how we can update the database > dynamically to get the exact value at that point of time. > > i mean query the license server and update the database accordingly. > does slurm automatically updated the value based on usage? > > > > > > Regards > > Navin. > > > > > > On Tue, May 5, 2020 at 7:00 PM Renfro, Michael > wrote: > > Have you seen https://slurm.schedmd.com/licenses.html already? If the > software is just for use inside the cluster, one Licenses= line in > slurm.conf plus users submitting with the -L flag should suffice. Should be > able to set that license value is 4 if it’s licensed per node and you can > run up to 4 jobs simultaneously, or 4*NCPUS if it’s licensed per CPU, or 1 > if it’s a single license good for one run from 1-4 nodes. > > > > There are also options to query a FlexLM or RLM server for license > management. > > > > -- > > Mike Renfro, PhD / HPC Systems Administrator, Information Technology > Services > > 931 372-3601 / Tennessee Tech University > > > > > On May 5, 2020, at 7:54 AM, navin srivastava > wrote: > > > > > > Hi Team, > > > > > > we have an application whose licenses is limited .it scales upto 4 > nodes(~80 cores). > > > so if 4 nodes are full, in 5th node job used to get fail. > > > we want to put a restriction so that the application can't go for the > execution beyond the 4 nodes and fail it should be in queue state. > > > i do not want to keep a separate partition to achieve this config.is > there a way to achieve this scenario using some dynamic resource which can > call the license variable on the fly and if it is reached it should keep > the job in queue. > > > > > > Regards > > > Navin. > > > > > > > > > > > > >
Re: [slurm-users] Slurm memory units
On Wed, 6 May 2020 10:42:46 +0100 Killian Murphy wrote: > Hi all. > > I'm probably making a rookie error here...which 'megabyte' (powers of > 1000 or 1024) does the Slurm documentation refer to in, for example, > the slurm.conf documentation for RealMemory and the sbatch > documentation for `--mem`? > > Most of our nodes have the same physical memory configuration. From > the output of `free -m` and `slurmd -C` on one of the nodes, we have > 191668M (187G). Consequently, `RealMemory` for those nodes has been > set to 191668 in slurm.conf. As a result, when a user requests memory > above '187G' for node memory, Slurm reports to them that the > requested node configuration is not available. Well yeah it's all 2-base. But it seems you have two problems 1) the units 2) users expecting 192GiB out of your nodes but the actual available memory is always lower (187G in your case). We see #2 also in that users know our thin nodes are 96G and some then proceed to request 96G (which does not fit on the 96G nodes...). From our system: $ LOCALINTERACTIVECOMMAND interactive --mem=3g -n 1 ... $ cat /sys/fs/cgroup/memory/slurm/uid_x/job_y/memory.limit_in_bytes 3221225472 # 3.0 GiB (this on slurm-18.08.8 with mem cgroups) /Peter > Only...we have 191668MiB of system memory, not MB. `free -m --si` (use > powers of 1000, not 1024) reports 192628MB of system memory (which, > frustratingly indicates that the 'free' documentation is also not > using the newer unit names). So it seems as though Slurm is working > in powers of 1024, not powers of 1000. > > I'm probably just confused about the unit definitions, or there is > some convention being applied here, but would appreciate some > confirmation either way! > > Thanks. > > Killian
Re: [slurm-users] "sacctmgr add cluster" crashing slurmdbd
On 06-05-2020 07:38, Chris Samuel wrote: We are experiencing exactly the same problem after mysql upgrade to 5.7.30, moving database to old mysql server running 5.6 solves the problem. Most likely downgrading mysql to 5.7.29 will work as well I have no clue which change in mysql-server is causing this best regards, Ben > On Tuesday, 5 May 2020 3:21:45 PM PDT Dustin Lang wrote: > >> Since this happens on a fresh new database, I just don't understand how I >> can get back to a basic functional state. This is exceedingly frustrating. > I have to say that if you're seeing this with 17.11, 18.08 and 19.05 and this > only started when your colleague upgraded MySQL then this sounds like MySQL > is > triggering this problem. > > We're running with MariaDB 10.x (from SLES15) without issues (our database is > huge). > > All the best, > Chris -- - Dr. B.J.W. Polman, C, Radboud University Nijmegen. Heyendaalseweg 135, 6525 AJ Nijmegen, The Netherlands, Phone: +31-24-3653360 e-mail: ben.pol...@science.ru.nl
Re: [slurm-users] Job Step Resource Requests are Ignored
On Tuesday, 5 May 2020 11:00:27 PM PDT Maria Semple wrote: > Is there no way to achieve what I want then? I'd like the first and last job > steps to always be able to run, even if the second step needs too many > resources (based on the cluster). That should just work. #!/bin/bash #SBATCH -c 2 #SBATCH -n 1 srun -c 1 echo hello srun -c 4 echo big wide srun -c 1 echo world gives: hello srun: Job step's --cpus-per-task value exceeds that of job (4 > 2). Job step may never run. srun: error: Unable to create step for job 604659: More processors requested than permitted world > As a side note, do you know why it's not even possible to restrict the > number of resources a single step uses (i.e. set less CPUs than are > available to the full job)? My suspicion is that you've not set up Slurm to use cgroups to restrict the resources a job can use to just those requested. https://slurm.schedmd.com/cgroups.html All the best, Chris -- Chris Samuel : http://www.csamuel.org/ : Berkeley, CA, USA
Re: [slurm-users] Job Step Resource Requests are Ignored
Hi Chris, Thanks for the tip about the memory units, I'll double check that I'm using them. Is there no way to achieve what I want then? I'd like the first and last job steps to always be able to run, even if the second step needs too many resources (based on the cluster). As a side note, do you know why it's not even possible to restrict the number of resources a single step uses (i.e. set less CPUs than are available to the full job)? Thanks, Maria On Tue, May 5, 2020 at 10:27 PM Chris Samuel wrote: > On Tuesday, 5 May 2020 4:47:12 PM PDT Maria Semple wrote: > > > I'd like to set different resource limits for different steps of my job. > A > > sample script might look like this (e.g. job.sh): > > > > #!/bin/bash > > srun --cpus-per-task=1 --mem=1 echo "Starting..." > > srun --cpus-per-task=4 --mem=250 --exclusive > > srun --cpus-per-task=1 --mem=1 echo "Finished." > > > > Then I would run the script from the command line using the following > > command: sbatch --ntasks=1 job.sh. > > You shouldn't ask for more resources with "srun" than have been allocated > with > "sbatch" - so if you want the job to be able to use up to 4 cores at once > & > that amount of memory you'll need to use: > > sbatch -c 4 --mem=250 --ntasks=1 job.sh > > I'd also suggest using suffixes for memory to disambiguate the values. > > All the best, > Chris > -- > Chris Samuel : http://www.csamuel.org/ : Berkeley, CA, USA > > > > > -- Thanks, Maria
