Re: [slurm-users] not allocating jobs even resources are free

2020-05-04 Thread navin srivastava
Thanks Denial for detailed Description Regards Navin On Sun, May 3, 2020, 13:35 Daniel Letai wrote: > > On 29/04/2020 12:00:13, navin srivastava wrote: > > Thanks Daniel. > > All jobs went into run state so unable to provide the details but > definitely will reach out later if we see similar

Re: [slurm-users] not allocating jobs even resources are free

2020-05-03 Thread Daniel Letai
On 29/04/2020 12:00:13, navin srivastava wrote: Thanks Daniel.   All jobs went into run state so unable to provide the details but definitely will reach out later if we see similar issue. i

Re: [slurm-users] not allocating jobs even resources are free

2020-04-29 Thread Brian W. Johanson
Navin, Check out 'sprio', this will give show you how the job priority changes with the weight changes you are making. -b On 4/29/20 5:00 AM, navin srivastava wrote: Thanks Daniel. All jobs went into run state so unable to provide the details but definitely will reach out later if we see

Re: [slurm-users] not allocating jobs even resources are free

2020-04-29 Thread navin srivastava
Thanks Daniel. All jobs went into run state so unable to provide the details but definitely will reach out later if we see similar issue. i am more interested to understand the FIFO with Fair Tree.it will be good if anybody provide some insight on this combination and also if we will enable the

Re: [slurm-users] not allocating jobs even resources are free

2020-04-27 Thread Daniel Letai
Are you sure there are enough resources available? The node is in mixed state, so it's configured for both partitions - it's possible that earlier lower priority jobs are already running thus blocking the later jobs, especially since it's fifo. It would

Re: [slurm-users] not allocating jobs even resources are free

2020-04-26 Thread navin srivastava
Thanks Brian, As suggested i gone through document and what i understood that the fair tree leads to the Fairshare mechanism and based on that the job should be scheduling. so it mean job scheduling will be based on FIFO but priority will be decided on the Fairshare. i am not sure if both

Re: [slurm-users] not allocating jobs even resources are free

2020-04-24 Thread Brian W. Johanson
If you haven't looked at the man page for slurm.conf, it will answer most if not all your questions. https://slurm.schedmd.com/slurm.conf.html but I would depend on the the manual version that was distributed with the version you have installed as options do change. There is a ton of

Re: [slurm-users] not allocating jobs even resources are free

2020-04-24 Thread navin srivastava
Thanks Brian. I need to check the jobs order. Is there any way to define the default timeline of the job if user not specifying time limit. Also what does the meaning of fairtree in priorities in slurm.Conf file. The set of nodes are different in partitions.FIFO does not care for any

Re: [slurm-users] not allocating jobs even resources are free

2020-04-24 Thread Brian W. Johanson
Without seeing the jobs in your queue, I would expect the next job in FIFO order to be too large to fit in the current idle resources. Configure it to use the backfill scheduler: SchedulerType=sched/backfill   SchedulerType   Identifies  the type of scheduler to be used.  Note

Re: [slurm-users] not allocating jobs even resources are free

2020-04-24 Thread navin srivastava
In addition to the above when i see the sprio of both the jobs it says :- for normal queue jobs all jobs showing the same priority JOBID PARTITION PRIORITY FAIRSHARE 1291352 normal 15789 15789 for GPUsmall all jobs showing the same priority. JOBID PARTITION

[slurm-users] not allocating jobs even resources are free

2020-04-24 Thread navin srivastava
Hi Team, we are facing some issue in our environment. The resources are free but job is going into the QUEUE state but not running. i have attached the slurm.conf file here. scenario:- There are job only in the 2 partitions: 344 jobs are in PD state in normal partition and the node belongs