Trey, indeed the problems was on computer nodes. Rebooting them solved the issue. Thank you for hinting this!
Best, P > On 15 Jan 2016, at 16:42, Trey Dockendorf <[email protected]> wrote: > > When I've had jobs end up as JobHeldUser it was usually due to an issue > launching the job on a compute node. So you'll want to check the slurmd.log > on the compute node that was allocated the job that's now JobHeldUser. > > - Trey > > ============================= > > Trey Dockendorf > Systems Analyst I > Texas A&M University > Academy for Advanced Telecommunications and Learning Technologies > Phone: (979)458-2396 > Email: [email protected] <mailto:[email protected]> > Jabber: [email protected] <mailto:[email protected]> > On Fri, Jan 15, 2016 at 6:11 AM, Paweł Liskowski > <[email protected] <mailto:[email protected]>> > wrote: > > Hello, > > Could anyone shed some light on why most of the jobs processed by the slurm > scheduler end up in the JobHeldUser state? The user is just running a bunch > of sbatch scripts and is not holding the jobs manually. If there are > resources available, some of the jobs are immediately run, but the rest is > not properly queued and just sits there with the JobHeldUser status. > > I am attaching part of the log below. Could you explain what exactly does the > state 0x8800 and 0x8000 mean. And what is reason 16? > > [2016-01-15T11:50:32.040] sched: Allocate JobId=2789 NodeList=lab-ci-9 #CPUs=1 > [2016-01-15T11:50:32.053] _slurm_rpc_requeue: Processing RPC: REQUEST_REQUEUE > from uid=0 > [2016-01-15T11:50:32.053] debug: job_hold_requeue: job 2787 state 0x8800 > [2016-01-15T11:50:32.053] debug: job_hold_requeue: job 2787 state 0x8000 > reason 16 priority 0 > [2016-01-15T11:50:32.054] _slurm_rpc_requeue: 2787: usec=922 > [2016-01-15T11:50:32.056] requeue batch job 2787 > > Cheers= >
