Thanks for finding this Carl. This patch https://github.com/SchedMD/slurm/commit/b50e226908f0c35888c6d64efc4e3e7e6af90604 will fix the issue and others related to what you are noticing. It will be in 2.5.2.
Danny On 01/04/13 09:45, Carl Schmidtmann wrote: > I have a patch for this that works for a BGQ. I had to use the cpu_cnt in the > job record and divide by 16 to get the correct node count. The code I added > should continue to work even if the job_ptr->node_cnt is changed in later > releases to support sub-midplane reservations. Although at that point it > might not be necessary. > > To find the correct field and verify that my fix was working correctly I > added some debug statements to print various bits from the job_ptr and > qos_ptr. When I submit a patch should I leave in the debug statements or > remove them? > > Thanks, > Carl > > ----- Original Message ----- >> I am working in the BGQ simulator to test out new scheduling >> configurations and ran into what looks like a bug. >> >> I created a QOS and tried to limit it to only running on 32 nodes. I >> tried setting the GrpNodes, MaxNodes and MaxNodesPerUser. None of >> these limits has the expected effect. In each case they apparently >> only see one node per job whether the job requests 1, 4 or 32 nodes. >> I currently have all 3 of the limits set to 32. I can start up to 32 >> jobs of any number of nodes and they will run concurrently. The 33rd >> and subsequent jobs will be held with the reason listed as >> QOSResourceLimit. I added some debug statements in >> slurmctld/acct_policy.c and found that it is indeed adding only one >> node per running job to the grp_used_nodes. When looking at the jobs >> using scontrol the jobs list the correct number of nodes but >> apparently the job_ptr->node_cnt always contains 1. >> >> I am guessing this has something to with the BGQ being handled as if >> each midplane is one node for some things but using a compute node >> as a node in other places. >> >> I am now digging for a better way to count the number of nodes >> currently in use by a person or group. Or figuring out how scontrol >> shows the correct node count but slurmctld doesn't see it when >> scheduling. >> >> Could this be a symptom of the simulator or do I need to fix this >> before moving into production? >> >> I saw this while working with 2.5.0 and it is the same in 2.5.1. >> >> Thanks, >> Carl >> >> -- >> Carl Schmidtmann >> Center for Integrated Research Computing >> University of Rochester >>
