I have a test cluster with three nodes with two gpus each one. In each node this installed cudatoolkit-3.2.16.It´s works fine with mpirun but with slurm I have the next error. what I missed?
[root@cluster bin]# srun -n5 -N 3 --gres=gpu:2 ejemplocudampi srun: error: Unable to allocate resources: Requested node configuration is not available This is the file /etc/slurm/slurm.conf NodeName=cluster NodeAddr=10.8.52.254 gres=gpu:2 GresTypes=gpu SelectType=select/cons_res This is the file /etc/slurm/gres.conf (this file is in each node) #Configuracion de gres en los nodos NodeName=compute-0-[0-2] Name=gpu File=/dev/nvidia[0-1] #Configuration of two GPUs Name=gpu File=/dev/nvidia0 Name=gpu File=/dev/nvidia1 This is the output of scontrol show node [root@cluster Desktop]# scontrol show node NodeName=cluster Arch=x86_64 CoresPerSocket=1 CPUAlloc=0 CPUErr=0 CPUTot=1 CPULoad=0.02 Features=(null) Gres=gpu:2 NodeAddr=10.8.52.254 NodeHostName=cluster Version=(null) OS=Linux RealMemory=3015 AllocMem=0 Sockets=1 Boards=1 State=IDLE ThreadsPerCore=1 TmpDisk=39803 Weight=1 BootTime=2015-09-23T12:05:37 SlurmdStartTime=2015-09-25T11:09:23 CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s NodeName=compute-0-0 Arch=x86_64 CoresPerSocket=1 CPUAlloc=0 CPUErr=0 CPUTot=8 CPULoad=0.00 Features=rack-0,8CPUs,nvidia Gres=gpu:4 NodeAddr=10.8.52.253 NodeHostName=compute-0-0 Version=(null) OS=Linux RealMemory=5968 AllocMem=0 Sockets=8 Boards=1 State=IDLE ThreadsPerCore=1 TmpDisk=447278 Weight=20488100 BootTime=2015-09-25T10:04:48 SlurmdStartTime=2015-09-25T10:07:59 CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s NodeName=compute-0-1 Arch=x86_64 CoresPerSocket=1 CPUAlloc=0 CPUErr=0 CPUTot=8 CPULoad=0.04 Features=rack-0,8CPUs,nvidia Gres=gpu:2 NodeAddr=10.8.52.252 NodeHostName=compute-0-1 Version=(null) OS=Linux RealMemory=5972 AllocMem=0 Sockets=8 Boards=1 State=IDLE ThreadsPerCore=1 TmpDisk=447278 Weight=20488101 BootTime=2015-09-15T10:09:34 SlurmdStartTime=2015-09-23T13:03:47 CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s NodeName=compute-0-2 Arch=x86_64 CoresPerSocket=1 CPUAlloc=0 CPUErr=0 CPUTot=8 CPULoad=0.08 Features=rack-0,8CPUs,nvidia Gres=gpu:2 NodeAddr=10.8.52.251 NodeHostName=compute-0-2 Version=(null) OS=Linux RealMemory=5972 AllocMem=0 Sockets=8 Boards=1 State=IDLE ThreadsPerCore=1 TmpDisk=447278 Weight=20488102 BootTime=2015-09-23T11:00:14 SlurmdStartTime=2015-09-23T13:04:45 CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s Please, any can help me? Thanks.
