I didn't know you could have node names in the gres.conf files, I thought you needed one per node, but anyway, in my setup the gres count is in the gres.conf file and not in the node specification in slurm.conf.
On Wed, 2015-01-14 at 14:13 -0800, Jared David Baker wrote: > Hello all, > > > > I’ve been playing with the config of Slurm-14.11.3 a bit now, but > after reading much of the documentation on gres.conf file and the > slurm.conf file, I’m not seeing the expected behavior relative to the > documentation. > > > > I have the following in slurm.conf: > > > > -- > > GresTypes=gpu > > NodeName=loren[01-60] NodeAddr=192.168.1.[1-60] RealMemory=64387 > Sockets=2 CoresPerSocket=10 ThreadsPerCore=1 State=UNKNOWN > Feature=ib,gpu,k20x Gres=gpu:4 > > PartitionName=Loren Nodes=loren[01-60] State=UP Default=YES > > -- > > > > and the following in gres.conf > > > > -- > > NodeName=loren[01-60] Name=gpu Type=k20x File=/dev/nvidia[0-3] > > -- > > > > And it seems anytime I request more than 1 GPU (i.e., srun -N1 > --gres=gpu:4 hostname), the job sits in pending state stating no node > contains the correct resource. However, querying the node (scontrol > show node loren01) shows that there are 4 GPUs configured for the node > and looking at slurmd.log (at level 3) file on the node shows that > there were 4 GPUs seen be slurmd. However, if I only request 1 GPU, > the job runs immediately. Does anybody see an overly simple mistake > I’ve made? Since all the compute nodes are identical, I can treat a > single gres.conf file as an individual node gres.conf file using the > following: > > > > -- > > Name=gpu Type=k20x File=/dev/nvidia0 > > Name=gpu Type=k20x File=/dev/nvidia1 > > Name=gpu Type=k20x File=/dev/nvidia2 > > Name=gpu Type=k20x File=/dev/nvidia3 > > -- > > > > And then I seem to be able to allocate all GPUs on the node without a > problem. However, I can see this being a problem in a heterogeneous > cluster with different GPU types or in a cluster which have GPU and > non-GPU nodes. I should note that the installation of Slurm is on an > NFS share. Any suggestions welcomeJ > > > > Thanks, > > > > Jared > > >
