The two simulated nodes are being scheduled independently. They do not
consider GPUs allocated by the other node.
Quoting [email protected]:
I am now running two slurmds on one node with 4 GPUs, to simulate
two nodes with 3 GPUs. The configuration is as follows:
#slurm.conf
...
NodeName=bnode1,Gres=gpu:4
NodeName=foo1,NodeHostName=bnode1,Gres=gpu:3
NodeName=foo2,NodeHostName=bnode1,Gres=gpu:3
#gres.conf
NodeName=bnode1,Name=gpu,File=/dev/nvidia[0-3]
NodeName=foo1, Name=gpu,File=/dev/nvidia[0-2]
NodeName=foo1, Name=gpu,File=/dev/nvidia[1-3]
I expect that when I run a 3-card program on foo1, another 3-card
program should be queued for resources.
#srun -wfoo1 --gres=gpu:3 ./myprog1 & srun -wfoo2 --gres=gpu:3 ./myprog2
However the two programs are actually sharing GPU card[1-2].
Does anyone have an idea on this?
--
James Zhang At University of Science and Technology of China
15556980026
[email protected]
--
Morris "Moe" Jette
CTO, SchedMD LLC
