We recently ran into an issue where a user was submitting job arrays of
0-500 one after another and would receive a message that the QOS limit had
been reached when "squeue --qos hepx --noheader |wc -l" would only print
~3000. The GrpSubmitJobs value for that QOS is 5000.
I looked at the code of how 'qos_ptr->usage->grp_used_submit_jobs' is
increased and decreased and noticed that when slurmctld starts the value is
set to 0 before reading in jobs from the state save location (is my
guess). For now we solved the issue by restarting slurmctld but I'm
interested in the cause.
I'm wondering if anyone has run into a situation that can cause SLURM to
think there are more jobs queued or running for a QOS than actually exist.
This is using SLURM 14.03.10 installed via RPM on CentOS 6.5.
$ sacctmgr show qos hepx format=Name,Flags,GrpSubmitJobs
Name Flags GrpSubmit
---------- -------------------- ---------
hepx DenyOnLimit 5000
Relevant config lines:
AccountingStorageEnforce=limits,qos
AccountingStorageType=accounting_storage/slurmdbd
Thanks
- Trey
=============================
Trey Dockendorf
Systems Analyst I
Texas A&M University
Academy for Advanced Telecommunications and Learning Technologies
Phone: (979)458-2396
Email: [email protected]
Jabber: [email protected]
