So this problem definitely seems to be introduced with slurm 2.6.5. I reverted to slurm 2.6.4 and an already submitted job that was not running with 2.6.5 started with no issues and no manual intervention after I restarted the slurmctld.
On Tue, Jan 14, 2014 at 06:27:54PM -0800, Andy Wettstein wrote: > > Hi, > > I'm seeing an strange issue with jobs requesting a specific reservation > not running even when the reserved nodes are idle. I have a 6 node > reservation dedicated to a single user. There are other nodes in the > partition and in use, but the reserved nodes are idle. The user submits > a few jobs using the correct reservation. Those jobs sit in the queue > with reason "Resources". If I manually raise the priority on those jobs, > the jobs start, so it seems like higher priority jobs in the partition > must be affecting the scheduling in some way. > > This is on slurm 2.6.5. We've had this reservation in place for some > time and I don't remember any issues like this before, so I don't know > if this is newly introduced bug or just an odd combination. > > Andy > > -- > andy wettstein > hpc system administrator > research computing center > university of chicago > 773.702.1104 -- andy wettstein hpc system administrator research computing center university of chicago 773.702.1104
