Re: [slurm-users] not allocating jobs even resources are free

2020-04-24 Thread Brian W. Johanson
If you haven't looked at the man page for slurm.conf, it will answer most if not all your questions. https://slurm.schedmd.com/slurm.conf.html but I would depend on the the manual version that was distributed with the version you have installed as options do change. There is a ton of

Re: [slurm-users] not allocating jobs even resources are free

2020-04-24 Thread navin srivastava
Thanks Brian. I need to check the jobs order. Is there any way to define the default timeline of the job if user not specifying time limit. Also what does the meaning of fairtree in priorities in slurm.Conf file. The set of nodes are different in partitions.FIFO does not care for any

Re: [slurm-users] not allocating jobs even resources are free

2020-04-24 Thread Brian W. Johanson
Without seeing the jobs in your queue, I would expect the next job in FIFO order to be too large to fit in the current idle resources. Configure it to use the backfill scheduler: SchedulerType=sched/backfill   SchedulerType   Identifies  the type of scheduler to be used.  Note

Re: [slurm-users] not allocating jobs even resources are free

2020-04-24 Thread navin srivastava
In addition to the above when i see the sprio of both the jobs it says :- for normal queue jobs all jobs showing the same priority JOBID PARTITION PRIORITY FAIRSHARE 1291352 normal 15789 15789 for GPUsmall all jobs showing the same priority. JOBID PARTITION

[slurm-users] not allocating jobs even resources are free

2020-04-24 Thread navin srivastava
Hi Team, we are facing some issue in our environment. The resources are free but job is going into the QUEUE state but not running. i have attached the slurm.conf file here. scenario:- There are job only in the 2 partitions: 344 jobs are in PD state in normal partition and the node belongs

[slurm-users] Building Slurm on CentOS 7 with PMIx support

2020-04-24 Thread Prentice Bisbal
Slurm-users, Is there anything special needed to build Slurm 19.05.5 on CentOS 7 with PMIx support? We have the pmix and pmix-devel packages that come with CentOS 7 installed, and built the RPMs with the following rpmbuild command: rpmbuild -ta --with munge --with pam --with lua  --with pmix