I am working in the BGQ simulator to test out new scheduling configurations and ran into what looks like a bug.
I created a QOS and tried to limit it to only running on 32 nodes. I tried setting the GrpNodes, MaxNodes and MaxNodesPerUser. None of these limits has the expected effect. In each case they apparently only see one node per job whether the job requests 1, 4 or 32 nodes. I currently have all 3 of the limits set to 32. I can start up to 32 jobs of any number of nodes and they will run concurrently. The 33rd and subsequent jobs will be held with the reason listed as QOSResourceLimit. I added some debug statements in slurmctld/acct_policy.c and found that it is indeed adding only one node per running job to the grp_used_nodes. When looking at the jobs using scontrol the jobs list the correct number of nodes but apparently the job_ptr->node_cnt always contains 1. I am guessing this has something to with the BGQ being handled as if each midplane is one node for some things but using a compute node as a node in other places. I am now digging for a better way to count the number of nodes currently in use by a person or group. Or figuring out how scontrol shows the correct node count but slurmctld doesn't see it when scheduling. Could this be a symptom of the simulator or do I need to fix this before moving into production? I saw this while working with 2.5.0 and it is the same in 2.5.1. Thanks, Carl -- Carl Schmidtmann Center for Integrated Research Computing University of Rochester
