Hello all, I've been playing with the config of Slurm-14.11.3 a bit now, but after reading much of the documentation on gres.conf file and the slurm.conf file, I'm not seeing the expected behavior relative to the documentation.
I have the following in slurm.conf: -- GresTypes=gpu NodeName=loren[01-60] NodeAddr=192.168.1.[1-60] RealMemory=64387 Sockets=2 CoresPerSocket=10 ThreadsPerCore=1 State=UNKNOWN Feature=ib,gpu,k20x Gres=gpu:4 PartitionName=Loren Nodes=loren[01-60] State=UP Default=YES -- and the following in gres.conf -- NodeName=loren[01-60] Name=gpu Type=k20x File=/dev/nvidia[0-3] -- And it seems anytime I request more than 1 GPU (i.e., srun -N1 --gres=gpu:4 hostname), the job sits in pending state stating no node contains the correct resource. However, querying the node (scontrol show node loren01) shows that there are 4 GPUs configured for the node and looking at slurmd.log (at level 3) file on the node shows that there were 4 GPUs seen be slurmd. However, if I only request 1 GPU, the job runs immediately. Does anybody see an overly simple mistake I've made? Since all the compute nodes are identical, I can treat a single gres.conf file as an individual node gres.conf file using the following: -- Name=gpu Type=k20x File=/dev/nvidia0 Name=gpu Type=k20x File=/dev/nvidia1 Name=gpu Type=k20x File=/dev/nvidia2 Name=gpu Type=k20x File=/dev/nvidia3 -- And then I seem to be able to allocate all GPUs on the node without a problem. However, I can see this being a problem in a heterogeneous cluster with different GPU types or in a cluster which have GPU and non-GPU nodes. I should note that the installation of Slurm is on an NFS share. Any suggestions welcome :) Thanks, Jared
