When I've had jobs end up as JobHeldUser it was usually due to an issue launching the job on a compute node. So you'll want to check the slurmd.log on the compute node that was allocated the job that's now JobHeldUser.
- Trey ============================= Trey Dockendorf Systems Analyst I Texas A&M University Academy for Advanced Telecommunications and Learning Technologies Phone: (979)458-2396 Email: [email protected] Jabber: [email protected] On Fri, Jan 15, 2016 at 6:11 AM, Paweł Liskowski < [email protected]> wrote: > > Hello, > > Could anyone shed some light on why most of the jobs processed by the > slurm scheduler end up in the JobHeldUser state? The user is just running a > bunch of sbatch scripts and is not holding the jobs manually. If there are > resources available, some of the jobs are immediately run, but the rest is > not properly queued and just sits there with the JobHeldUser status. > > I am attaching part of the log below. Could you explain what exactly does > the state 0x8800 and 0x8000 mean. And what is reason 16? > > [2016-01-15T11:50:32.040] sched: Allocate JobId=2789 NodeList=lab-ci-9 > #CPUs=1 > [2016-01-15T11:50:32.053] _slurm_rpc_requeue: Processing RPC: > REQUEST_REQUEUE from uid=0 > [2016-01-15T11:50:32.053] debug: job_hold_requeue: job 2787 state 0x8800 > [2016-01-15T11:50:32.053] debug: job_hold_requeue: job 2787 state 0x8000 > reason 16 priority 0 > [2016-01-15T11:50:32.054] _slurm_rpc_requeue: 2787: usec=922 > [2016-01-15T11:50:32.056] requeue batch job 2787 > > Cheers=
