I've been seeing this type of behavior since upgrading to slurm 2.5 as well. I see things like this in the log:
[2013-04-30T02:38:37-05:00] _slurm_rpc_submit_batch_job JobId=4214940 usec=245998 [2013-04-30T02:39:07-05:00] error: cons_res: sync loop not progressing, holding job 4214940 Usually releasing the job will allow it to run but not always. I haven't been able to figure out why this is happening either. Nothing looks particulary strange about the submit script. We do have GPU GRES, but not on the partition where these are being submitted. On Mon, Apr 29, 2013 at 06:35:35AM -0700, Paul Edmon wrote: > Okay, I did see a con/res error associated with these jobs. So just > to be clear modifying the job priority will clear the Admin Hold > state? I found that resubmitting the job also works, but there is no > way to do that besides removing the job and having the user resubmit > it. > > -Paul Edmon- > > > On 4/29/2013 9:05 AM, Carles Fenoy wrote: > >Re: [slurm-dev] Re: JobHeldAdmin > >Hi, > > > >This can happen if you are using gpu GRES with cpus dedicated to > >some gpus. As Carl has said, check the slurmctld log for any error. > > > >Regards, > >Carles Fenoy > >Barcelona Supercomputing Center > > > > > >On Mon, Apr 29, 2013 at 3:01 PM, Carl Schmidtmann > ><[email protected] > ><mailto:[email protected]>> wrote: > > > > > > I am not sure of all the reasons a job might get to the Admin Hold > > state but I have seen it when a job tries to run but the resources > > it uses generate an error instead of running the job. Once they > > are in this state they will not be looked at by the scheduler > > because the job has its priority set to zero. Changing the > > priority to something greater than zero will put them back into > > the scheduling. However, if the error that caused them to fail > > still exists they will return to the Admin Hold state when they > > next try to run. > > > > Check the log files for the failure and possible reason. > > > > Carl > > > > ----- Original Message ----- > > > > > > What does this mean? I can't find any documentation and I have > > > several > > > jobs in our queues which are getting this Reason Waiting assigned > > > them > > > automatically by SLURM. The jobs themselves appear valid, but I > > > can't > > > force them to lose this state even though there are enough nodes for > > > the > > > job open. > > > > > > -Paul Edmon- > > > > > > > -- > > -- > > Carl Schmidtmann > > Center for Integrated Research Computing > > University of Rochester > > - Our Annual CIRC Poster Session is May 10, 2013 > > - Visit http://www.circ.rochester.edu/poster_submission.html > > > > > > > > > > > >-- > >-- > >Carles Fenoy > -- andy wettstein hpc system administrator research computing center university of chicago 773.702.1104
