Hello, This sounds somewhat familiar to an oddity I reported a few months ago concerning the default QOS not being respected. I'll provide the URL to the thread below so that you don't have to search the list archives.
https://www.mail-archive.com/[email protected]/msg04167.html If I restart the slurmctld, the default QOS is respected and can be verified by squeue. I'd recommend trying that instead of the modifications to see if you get the expected results or not. I saw this on 14.03.6 and with 14.11.3 John DeSantis 2015-02-03 4:12 GMT-05:00 "Dr. Markus Stöhr" <[email protected]>: > > Dear all, > > we have found a small bug in slurm version 14.3 and 14.11. Everytime a new > user is added we see the following behaviour: When displaying the user > associations you can see the correct default qos, but it is not known when > submitting jobs. The newly created user always has to specify the default > qos in the job script. However, when modifying the account and resetting the > QOS and default QOS, the user needs no longer to specify the default qos. > See a the slurm commands, that we used, below. > > Thank you for your help! > > best regards > Markus > > > ==================================================================================== > #Create two qos: > sacctmgr -i add qos name=qos_test_a grpnodes=10 maxwall="3-0" > description='test_a' > sacctmgr -i add qos name=qos_test_b grpnodes=10 maxwall="3-0" > description='test_b' > > #add account with default qos: > sacctmgr add account test_def_qos QOS=qos_test_a,qos_test_b > DefaultQOS=qos_test_a > > #add partition with these qos to slurm.conf: > PartitionName=test_qos Default=NO MinNodes=1 AllowGroups=ALL Priority=1 > DisableRootJobs=NO RootOnly=NO Hidden=NO Shared=NO GraceTime=0 > PreemptMode=OFF ReqResv=NO AllowAccounts=test_def_qos > AllowQos=test_qos_a,test_qos_b > Nodes=n01-[010-012,045,094-096],n02-[001,006,013],n03-[025,066,071] > > > #add user to previously created account: > sacctmgr -i add user 70032 DefaultAccount=test_def_qos > > > #user associations: > [markus@l31 ~]$ sacctmgr show user `id -u` withassoc > format=user,defaultaccount,account,qos%40s,defaultqos%20s > User Def Acct Account QOS Def QOS > ---------- ---------- ---------- ---------------------------------------- > -------------------- > 70032 test_def_+ test_def_+ qos_test_a,qos_test_b qos_test_a > > > #JOB test1: > ------------------------------------------ > #!/bin/bash > #SBATCH -J jobname > #SBATCH -N 1 > #SBATCH --tasks-per-node=16 > #SBATCH --partition test_qos > sleep 600 > ------------------------------------------ > [markus@l31 slurm]$ sbatch job.sh > sbatch: error: Batch job submission failed: Job has invalid qos > > > #JOB test2 > [markus@l31 slurm]$ cat job.sh > ------------------------------------------ > #!/bin/bash > #SBATCH -J jobname > #SBATCH -N 1 > #SBATCH --tasks-per-node=16 > #SBATCH --partition test_qos > #SBATCH --qos qos_test_a > sleep 600 > ------------------------------------------ > [markus@l31 slurm]$ sbatch job.sh > Submitted batch job 23508 > > > #REDEFINE qos's > sacctmgr modify account test_def_qos set QOS=qos_test_a,qos_test_b > DefaultQOS=qos_test_a > Modified account associations... > C = vsc3 A = test_def_qos of root > C = vsc3 A = test_def_qos U = 70032 > Would you like to commit changes? (You have 30 seconds to decide) > (N/y): y > > > #JOB test3 > [markus@l31 slurm]$ cat job.sh > ------------------------------------------ > #!/bin/bash > #SBATCH -J jobname > #SBATCH -N 1 > #SBATCH --tasks-per-node=16 > #SBATCH --partition test_qos > > sleep 600 > ------------------------------------------ > [markus@l31 slurm]$ sbatch job.sh > Submitted batch job 23511 > > > -- > ===================================================== > Dr. Markus Stöhr > Zentraler Informatikdienst BOKU Wien / TU Wien > Wiedner Hauptstraße 8-10 > 1040 Wien > > Tel. +43-1-58801-420754 > Fax +43-1-58801-9420754 > > Email: [email protected] > =====================================================
