You don't appear to have any associations to the partitions. Seems like your associations aren't in place correctly.
On Fri, Jun 16, 2017 at 1:24 PM, Jagga Soorma <[email protected]> wrote: > > I don't see anything here: > > # sacctmgr show assoc partition=gred_sa > Cluster Account User Partition Share GrpJobs > GrpTRES GrpSubmit GrpWall GrpTRESMins MaxJobs MaxTRES > MaxTRESPerNode MaxSubmit MaxWall MaxTRESMins > QOS Def QOS GrpTRESRunMin > ---------- ---------- ---------- ---------- --------- ------- > ------------- --------- ----------- ------------- ------- > ------------- -------------- --------- ----------- ------------- > -------------------- --------- ------------- > > # sacctmgr list user | grep -i restst > user1 SA None > user2 SA None > user3 CIG None > user4 CIG None > > Here is what I have also tried without success: > > sacctmgr add qos gpu GrpTRES=GRES/gpu=8 Flags=OverPartQos > > PartitionName=gred Nodes=node[313-314] Default=NO MaxTime=INFINITE > State=UP DefMemPerCPU=2048 QOS=gpu > > sacctmgr modify account SA set qos=gpu > sacctmgr modify account CIG set qos=gpu > > Now when I submit a second job by user1 asking for 8gpu's I was hoping > it would go in the queue but it still runs: > > srun --gres=gpu:8 -p gred --pty bash > > Thanks again for your help. > > On Fri, Jun 16, 2017 at 10:08 AM, Barry Moore <[email protected]> wrote: > > Are there accounts with associations to that partition? > > > > sacctmgr show assoc partition=SA > > > > Sorry, I am not doing partition level management like this. I presumed it > > would be similar to how I set up association limits per account per > cluster. > > > > On Fri, Jun 16, 2017 at 12:57 PM, Jagga Soorma <[email protected]> > wrote: > >> > >> > >> So, looks like I am unable to set the grptres on the account. > >> Shouldn't this be set with qos instead? > >> > >> # sacctmgr modify account where partition=SA set GrpTRES=GRES/gpu=8 > >> Unknown option: GrpTRES=GRES/gpu=8 > >> Use keyword 'where' to modify condition > >> > >> Thanks. > >> > >> On Fri, Jun 16, 2017 at 9:52 AM, Jagga Soorma <[email protected]> > wrote: > >> > Ahh my apologies Barry. Looks like accidentally I did not choose the > >> > correct partition which was causing this error. It seems to do the > >> > right thing when I have picked the correct partition during sacctmgr > >> > add user. Will try the rest. > >> > > >> > Thanks again for your help with this! Much appreciated! > >> > > >> > On Fri, Jun 16, 2017 at 9:49 AM, Jagga Soorma <[email protected]> > wrote: > >> >> Thanks for your response Barry. But as soon as I add this user > using: > >> >> > >> >> sacctmgr add user user1 account=SA partition=partition2 > >> >> > >> >> They are also able to submit jobs to partiton1 which we don't want. > >> >> Does that maybe have something to do with qos? I want these users to > >> >> only be able to submit jobs to partition2 and not partiton1. > >> >> > >> >> # sacctmgr list qos > >> >> Name Priority GraceTime Preempt PreemptMode > >> >> Flags UsageThres UsageFactor GrpTRES > >> >> GrpTRESMins GrpTRESRunMin GrpJobs GrpSubmit GrpWall MaxTRES > >> >> MaxTRESPerNode MaxTRESMins MaxWall MaxTRESPU MaxJobsPU > >> >> MaxSubmitPU MaxTRESPA MaxJobsPA MaxSubmitPA MinTRES > >> >> ---------- ---------- ---------- ---------- ----------- > >> >> ---------------------------------------- ---------- ----------- > >> >> ------------- ------------- ------------- ------- --------- > >> >> ----------- ------------- -------------- ------------- ----------- > >> >> ------------- --------- ----------- ------------- --------- > >> >> ----------- ------------- > >> >> normal 0 00:00:00 cluster > >> >> 1.000000 > >> >> > >> >> # sacctmgr list user | grep -i user1 > >> >> user1 research None > >> >> > >> >> Thanks. > >> >> > >> >> On Fri, Jun 16, 2017 at 9:33 AM, Barry Moore <[email protected]> > >> >> wrote: > >> >>> Jagga, > >> >>> > >> >>> You got it. Something along the lines of: > >> >>> > >> >>> # Add users to accounts, partition > >> >>> sacctmgr add user <users> account=<account> partition=<whatever> > >> >>> > >> >>> # Set the association limits, I think you can do something similar > to > >> >>> gres/gpu=-1 for the super users (assuming they are in the same > >> >>> account?). > >> >>> sacctmgr modify account where partition=<partition> set > >> >>> grptres=gres/gpu=8 > >> >>> sacctmgr modify account where partition=<partition> account=root set > >> >>> grptres=gres/gpu=-1 > >> >>> > >> >>> This will work as long as your super users aren't under an account > you > >> >>> want > >> >>> to enforce. Maybe someone has a nicer solution for that. > >> >>> > >> >>> Hope that helps, > >> >>> > >> >>> Barry > >> >>> > >> >>> > >> >>> On Fri, Jun 16, 2017 at 12:18 PM, Jagga Soorma <[email protected]> > >> >>> wrote: > >> >>>> > >> >>>> > >> >>>> Hello, > >> >>>> > >> >>>> I have a new 17.02.2 slurm environment with only one partition and > >> >>>> now > >> >>>> have the following requirement that I need to implement: > >> >>>> > >> >>>> -- > >> >>>> Create a new partition (partition2) with 2 nodes. This new > partition > >> >>>> should be restricted, i.e. users cannot submit to it unless they > are > >> >>>> authorized. > >> >>>> > >> >>>> The two groups of users, each of whom should be authorized, are: > >> >>>> > >> >>>> group1: SA > >> >>>> users: user1, user2 > >> >>>> > >> >>>> group2: CIG > >> >>>> users: user3, user4, user5 > >> >>>> > >> >>>> Each group should be allowed no more than 8 gres/gpus at a time. > The > >> >>>> limit should be at the group level, not individual user. > >> >>>> > >> >>>> These users should not be able to submit jobs to the first > partition > >> >>>> (partition1). This is our first original partition on this > cluster. > >> >>>> > >> >>>> Also, please allow super users (userX, userY, userZ) to submit jobs > >> >>>> to > >> >>>> this partition without limits, for engineering/troubleshooting > >> >>>> purposes. > >> >>>> -- > >> >>>> > >> >>>> I am sure this is doable via associations in slurmdbd but was > >> >>>> wondering if that is the correct place to start. We have only > >> >>>> implemented simple accounts with slurmbdb so wanted to make sure I > do > >> >>>> this the right way and would appreciate any help with this. > >> >>>> > >> >>>> Thanks! > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> -- > >> >>> Barry E Moore II, PhD > >> >>> E-mail: [email protected] > >> >>> > >> >>> Assistant Research Professor > >> >>> Center for Simulation and Modeling > >> >>> University of Pittsburgh > >> >>> Pittsburgh, PA 15260 > > > > > > > > > > -- > > Barry E Moore II, PhD > > E-mail: [email protected] > > > > Assistant Research Professor > > Center for Simulation and Modeling > > University of Pittsburgh > > Pittsburgh, PA 15260 > -- Barry E Moore II, PhD E-mail: [email protected] Assistant Research Professor Center for Simulation and Modeling University of Pittsburgh Pittsburgh, PA 15260
