Re: [slurm-users] how to restrict jobs

2020-05-07 Thread Daniel Letai
On 06/05/2020 20:44, Mark Hahn wrote: Is there no way to set or define a custom variable like at node level and you could use a per-node Feature for this, but a partition would also work. A bit of an ugly hack,

Re: [slurm-users] how to restrict jobs

2020-05-06 Thread Mark Hahn
Is there no way to set or define a custom variable like at node level and you could use a per-node Feature for this, but a partition would also work.

Re: [slurm-users] how to restrict jobs

2020-05-06 Thread navin srivastava
Is there no way to set or define a custom variable like at node level and then you pass the same variable in the job request so that it will land into those nodes only. Regards Navin On Wed, May 6, 2020, 21:04 Renfro, Michael wrote: > Ok, then regular license accounting won’t work. > >

Re: [slurm-users] how to restrict jobs

2020-05-06 Thread Renfro, Michael
Ok, then regular license accounting won’t work. Somewhat tested, but should work or at least be a starting point. Given a job number JOBID that’s already running with this license on one or more nodes: sbatch -w $(scontrol show job JOBID | grep ' NodeList=' | cut -d= -f2) -N 1 should start a

Re: [slurm-users] how to restrict jobs

2020-05-06 Thread navin srivastava
To explain with more details. job will be submitted based on core at any time but it will go to any random nodes but limited to 4 Nodes only.(license having some intelligence that it calculate the nodes and if it reached to 4 then it will not allow any more nodes. yes it didn't depend on the no

Re: [slurm-users] how to restrict jobs

2020-05-06 Thread Renfro, Michael
To make sure I’m reading this correctly, you have a software license that lets you run jobs on up to 4 nodes at once, regardless of how many CPUs you use? That is, you could run any one of the following sets of jobs: - four 1-node jobs, - two 2-node jobs, - one 1-node and one 3-node job, - two

Re: [slurm-users] how to restrict jobs

2020-05-06 Thread navin srivastava
Thanks Micheal. Actually one application license are based on node and we have 4 Node license( not a fix node). we have several nodes but when job lands on any 4 random nodes it runs on those nodes only. After that it fails if it goes to other nodes. can we define a custom variable and set it on

Re: [slurm-users] how to restrict jobs

2020-05-05 Thread Renfro, Michael
Haven’t done it yet myself, but it’s on my todo list. But I’d assume that if you use the FlexLM or RLM parts of that documentation, that Slurm would query the remote license server periodically and hold the job until the necessary licenses were available. > On May 5, 2020, at 8:37 AM, navin

Re: [slurm-users] how to restrict jobs

2020-05-05 Thread navin srivastava
Thanks Michael, yes i have gone through but the licenses are remote license and it will be used by outside as well not only in slurm. so basically i am interested to know how we can update the database dynamically to get the exact value at that point of time. i mean query the license server and

Re: [slurm-users] how to restrict jobs

2020-05-05 Thread Renfro, Michael
Have you seen https://slurm.schedmd.com/licenses.html already? If the software is just for use inside the cluster, one Licenses= line in slurm.conf plus users submitting with the -L flag should suffice. Should be able to set that license value is 4 if it’s licensed per node and you can run up

[slurm-users] how to restrict jobs

2020-05-05 Thread navin srivastava
Hi Team, we have an application whose licenses is limited .it scales upto 4 nodes(~80 cores). so if 4 nodes are full, in 5th node job used to get fail. we want to put a restriction so that the application can't go for the execution beyond the 4 nodes and fail it should be in queue state. i do not