[slurm-dev] job status after node reboot
Hello, is the problem, that jobs are in state running after a node crash, allready been fixed. We have a shared memory computer with only one node, where slurmdbd, slurmctld and slurmd is running on and ran into this problem. Sincerely yours, Danny Rotscher smime.p7s Description: S/MIME Cryptographic Signature
[slurm-dev] Error while loading shared libraries
Hello, When I throw a MPI job on slurm I get this error, Any can help me? thanks you [root@cluster bin]# srun -N4 numeroprimos /usr/bin/numeroprimos: error while loading shared libraries: libmpichcxx.so.1.2: cannot open shared object file: No such file or directory 50 Aniversario de la Cujae. Inaugurada por Fidel el 2 de diciembre de 1964 http://cujae.edu.cu
[slurm-dev] Re: Error while loading shared libraries
Do you have mpi installed on all the cluster nodes? On Wed, May 6, 2015 at 3:34 PM, Uwe Sauter uwe.sauter...@gmail.com wrote: Check the file permissions for libmpichcxx.so.1.2 as well as the permissions on the parent directories. Might be that you are not allowed to access the folder structure as the user you're running your application. Am 06.05.2015 um 07:57 schrieb Fany Pagés Díaz: Hello, WhenI throw a MPI job on slurm I get this error, Any can help me? thanks you [root@cluster bin]# srun -N4 numeroprimos /usr/bin/numeroprimos: error while loading shared libraries: libmpichcxx.so.1.2: cannot open shared object file: No such file or directory Untitled Document 50 Aniversario de la Cujae. Inaugurada por Fidel el 2 de diciembre de 1964 http://cujae.edu.cu http://cujae.edu.cu -- -- Carles Fenoy
[slurm-dev] Re: Error while loading shared libraries
This library is on a server and has all the permits, although in the nodes is not found. I nodes openmpi-1.5.4-2 is installed. De: Carlos Fenoy [mailto:mini...@gmail.com] Enviado el: miércoles, 06 de mayo de 2015 10:11 Para: slurm-dev Asunto: [slurm-dev] Re: Error while loading shared libraries Do you have mpi installed on all the cluster nodes? On Wed, May 6, 2015 at 3:34 PM, Uwe Sauter uwe.sauter...@gmail.com mailto:uwe.sauter...@gmail.com wrote: Check the file permissions for libmpichcxx.so.1.2 as well as the permissions on the parent directories. Might be that you are not allowed to access the folder structure as the user you're running your application. Am 06.05.2015 um 07:57 schrieb Fany Pagés Díaz: Hello, WhenI throw a MPI job on slurm I get this error, Any can help me? thanks you [root@cluster bin]# srun -N4 numeroprimos /usr/bin/numeroprimos: error while loading shared libraries: libmpichcxx.so.1.2: cannot open shared object file: No such file or directory Untitled Document 50 Aniversario de la Cujae. Inaugurada por Fidel el 2 de diciembre de 1964 http://cujae.edu.cu http://cujae.edu.cu -- -- Carles Fenoy http://smd-server.schedmd.local/cgi-bin/dada/mail.cgi/spacer_image/slurmdev/CAARb8mMfFGk9unEBX0JWKGTfC_-zf+_szxVD29aWWGe1ZsPZEA@mail/spacer.png 50 Aniversario de la Cujae. Inaugurada por Fidel el 2 de diciembre de 1964 http://cujae.edu.cu
[slurm-dev] Re: Error while loading shared libraries
Then you can not execute a binary compiled with MPICH in the compute nodes. If you want to execute on the compute nodes you have to compile the binary with openmpi On Wed, May 6, 2015 at 4:23 PM, Fany Pagés Díaz fpa...@udio.cujae.edu.cu wrote: This library is on a server and has all the permits, although in the nodes is not found. I nodes openmpi-1.5.4-2 is installed. *De:* Carlos Fenoy [mailto:mini...@gmail.com] *Enviado el:* miércoles, 06 de mayo de 2015 10:11 *Para:* slurm-dev *Asunto:* [slurm-dev] Re: Error while loading shared libraries Do you have mpi installed on all the cluster nodes? On Wed, May 6, 2015 at 3:34 PM, Uwe Sauter uwe.sauter...@gmail.com wrote: Check the file permissions for libmpichcxx.so.1.2 as well as the permissions on the parent directories. Might be that you are not allowed to access the folder structure as the user you're running your application. Am 06.05.2015 um 07:57 schrieb Fany Pagés Díaz: Hello, WhenI throw a MPI job on slurm I get this error, Any can help me? thanks you [root@cluster bin]# srun -N4 numeroprimos /usr/bin/numeroprimos: error while loading shared libraries: libmpichcxx.so.1.2: cannot open shared object file: No such file or directory Untitled Document 50 Aniversario de la Cujae. Inaugurada por Fidel el 2 de diciembre de 1964 http://cujae.edu.cu http://cujae.edu.cu -- -- Carles Fenoy 50 Aniversario de la Cujae. Inaugurada por Fidel el 2 de diciembre de 1964 http://cujae.edu.cu -- -- Carles Fenoy
[slurm-dev] Re: Error while loading shared libraries
I compiled this way mpic++ -openmp prime_number.mpi.C -o example, that's wrong? how it's made? -Original Message- From: Carlos Fenoy mini...@gmail.com To: slurm-dev slurm-dev@schedmd.com Date: Wed, 06 May 2015 07:27:31 -0700 Subject: [slurm-dev] Re: Error while loading shared libraries Then you can not execute a binary compiled with MPICH in the compute nodes. If you want to execute on the compute nodes you have to compile the binary with openmpi On Wed, May 6, 2015 at 4:23 PM, Fany Pag�s D�az fpa...@udio.cujae.edu.cu [mailto:fpa...@udio.cujae.edu.cu] wrote: Thislibrary is on a server and has all the permits, although in the nodes is not found. I nodes openmpi-1.5.4-2 is installed. De:Carlos Fenoy [mailto:mini...@gmail.com [mailto:mini...@gmail.com]] Enviado el: mi�rcoles, 06 de mayo de 2015 10:11 Para: slurm-dev Asunto: [slurm-dev] Re: Error while loading shared libraries Do you have mpi installed on all the cluster nodes? On Wed, May 6, 2015 at 3:34 PM, Uwe Sauter uwe.sauter...@gmail.com [mailto:uwe.sauter...@gmail.com] wrote: Check the file permissions for libmpichcxx.so.1.2 as well as the permissions on the parent directories. Might be that you are not allowed to access the folder structure as the user you're running your application. Am 06.05.2015 um 07:57 schrieb Fany Pag�s D�az: Hello, WhenI throw a MPI job on slurm I get this error, Any can help me? thanks you [root@cluster bin]# srun -N4 numeroprimos /usr/bin/numeroprimos: error while loading shared libraries: libmpichcxx.so.1.2: cannot open shared object file: No such file or directory Untitled Document 50 Aniversario de la Cujae. Inaugurada por Fidel el 2 de diciembre de 1964 http://cujae.edu.cu [http://cujae.edu.cu/] http://cujae.edu.cu [http://cujae.edu.cu/] -- -- Carles Fenoy 50 Aniversario de la Cujae. Inaugurada por Fidel el 2 de diciembre de 1964 http://cujae.edu.cu [http://cujae.edu.cu/] -- -- Carles Fenoy 50 Aniversario de la Cujae. Inaugurada por Fidel el 2 de diciembre de 1964 http://cujae.edu.cu
[slurm-dev] Re: Error while loading shared libraries
Check the file permissions for libmpichcxx.so.1.2 as well as the permissions on the parent directories. Might be that you are not allowed to access the folder structure as the user you're running your application. Am 06.05.2015 um 07:57 schrieb Fany Pagés Díaz: Hello, WhenI throw a MPI job on slurm I get this error, Any can help me? thanks you [root@cluster bin]# srun -N4 numeroprimos /usr/bin/numeroprimos: error while loading shared libraries: libmpichcxx.so.1.2: cannot open shared object file: No such file or directory Untitled Document 50 Aniversario de la Cujae. Inaugurada por Fidel el 2 de diciembre de 1964 http://cujae.edu.cu http://cujae.edu.cu
[slurm-dev] Job allocation for GPU jobs doesn't work using gpu plugin (node configuration not available)
Hello, currently I'm trying to set up SLURM on a gpu cluster with a small number of nodes (where smurf0[1-7] are the node names) using the gpu plugin to allocate jobs (requiring gpus). Unfortunately, when trying to run a gpu-job (any number of gpus; --gres=gpu:N), SLURM doesn't execute it, asserting unavailability of the requested configuration. I attached some logs and configuration text files in order to provide any information necessary to analyze this issue. Note: Cross posted here: http://serverfault.com/questions/685258 Example (using some test.sh which is echoing $CUDA_VISIBLE_DEVICES): srun -n1 --gres=gpu:1 test.sh -- srun: error: Unable to allocate resources: Requested node configuration is not available The slurmctld log for such calls shows: gres: gpu state for job X gres_cnt:1 node_cnt:1 type:(null) _pick_best_nodes: job X never runnable _slurm_rpc_allocate_resources: Requested node configuration is not available Jobs with any other type of configured generic resource complete successfully: srun -n1 --gres=gram:500 test.sh -- CUDA_VISIBLE_DEVICES=NoDevFiles The nodes and gres configuration in slurm.conf (which is attached as well) are like: GresTypes=gpu,ram,gram,scratch ... NodeName=smurf01 NodeAddr=192.168.1.101 Feature=intel,fermi Boards=1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 Gres=gpu:tesla:8,ram:48,gram:no_consume:6000,scratch:1300 NodeName=smurf02 NodeAddr=192.168.1.102 Feature=intel,fermi Boards=1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=1 Gres=gpu:tesla:8,ram:48,gram:no_consume:6000,scratch:1300 The respective gres.conf files are Name=gpu Count=8 Type=tesla File=/dev/nvidia[0-7] Name=ram Count=48 Name=gram Count=6000 Name=scratch Count=1300 The output of scontrol show node lists all the nodes with the correct gres configuration i.e.: NodeName=smurf01 Arch=x86_64 CoresPerSocket=6 CPUAlloc=0 CPUErr=0 CPUTot=24 CPULoad=0.01 Features=intel,fermi Gres=gpu:tesla:8,ram:48,gram:no_consume:6000,scratch:1300 ...etc. As far as I can tell, the slurmd daemon on the nodes recognizes the gpus (and other generic resources) correctly. My slurmd.log on node smurf01 says Gres Name = gpu Type = tesla Count = 8 ID = 7696487 File = /dev /nvidia[0 - 7] The log for slurmctld shows gres / gpu: state for smurf01 gres_cnt found : 8 configured : 8 avail : 8 alloc : 0 gres_bit_alloc : gres_used : (null) I can't figure out why the controller node states that jobs using --gres=gpu:N are never runnable and why the requested node configuration is not available. Any help is appreciated. Kind regards, Daniel Weber PS: If further information is required, don't hesitate to ask. # GENERAL ClusterName=egpc ControlMachine=wtch020 ControlAddr=192.168.1.1 AuthType=auth/munge CryptoType=crypto/munge CacheGroups=0 DisableRootJobs=YES MpiDefault=none Proctracktype=proctrack/cgroup # DAEMONS StateSaveLocation=/var/spool/slurmctld SlurmctldPidFile=/var/run/slurm/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurmd SlurmUser=slurm #SlurmdUser=root SwitchType=switch/none # SCRIPTS Prolog=/apps/.slurm/job_prolog.sh Epilog=/apps/.slurm/job_epilog.sh TaskProlog=/apps/.slurm/task_prolog.sh TaskEpilog=/apps/.slurm/task_epilog.sh # TIMERS AND LIMITS MaxJobCount=5000 ReturnToService=1 TaskPlugin=task/cgroup InactiveLimit=0 KillWait=30 MinJobAge=120 OverTimeLimit=60 SlurmctldTimeout=120 SlurmdTimeout=300 Waittime=0 # SCHEDULING #DefMemPerCPU=0 #MaxMemPerCPU=0 FastSchedule=1 #SchedulerRootFilter=1 #SchedulerTimeSlice=30 SchedulerType=sched/backfill SchedulerPort=7321 SelectType=select/linear #SelectTypeParameters= # JOB PRIORITY #PriorityFlags= PriorityType=priority/multifactor PriorityDecayHalfLife=7-0 PriorityFavorSmall=YES #PriorityCalcPeriod= #PriorityUsageResetPeriod= PriorityMaxAge=3-0 PriorityWeightAge=1 #PriorityWeightFairshare=0 PriorityWeightJobSize=1 #PriorityWeightQOS= #PriorityWeightPartition= # ACCOUNTING #AccountingStorageEnforce=0 AccountingStorageType=accounting_storage/slurmdbd AccountingStoragePort=6819 AccountingStorageUser=slurm AccountingStoreJobComment=YES JobAcctGatherFrequency=60 JobAcctGatherType=jobacct_gather/linux # LOGGING SlurmctldDebug=5 SlurmdDebug=5 DebugFlags=Gres SlurmctldLogFile=/var/log/slurm/ctld SlurmdLogFile=/var/log/slurmd #SlurmSchedLogFile= #SlurmSchedLogLevel= # COMPUTE NODES GresTypes=gpu,ram,gram,scratch NodeName=smurf01 NodeAddr=192.168.1.101 Feature=intel,fermi Boards=1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 Gres=gpu:tesla:8,ram:48,gram:no_consume:6000,scratch:1300 NodeName=smurf02 NodeAddr=192.168.1.102 Feature=intel,fermi Boards=1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=1 Gres=gpu:tesla:8,ram:48,gram:no_consume:6000,scratch:1300 NodeName=smurf03 NodeAddr=192.168.1.103 Feature=intel,fermi
[slurm-dev] slurmd on first node not responding and is running
Hello, I am running slum between two desktop computers for testing purposes and am having a problem where when I launch jobs from the controlnode (rousseau) I get that the jobs have been queued and are waiting for resources and that the required node is not available (down or drained). I’ve gone through all the steps in the trouble shooting guide and haven’t found a solution. I found that the slurred on the node (nucor-fusion) is running and the slurmctld is running on the head node, but that they seem to not be able to communicate. When I go into my slurmctld log I see, over and over again. My slum.conf is attached. Is this a problem with the configuration? or is there something else stopping the nodes from communicating with each other (with my munge keys or something else)? Thanks, Trevor slurm.conf Description: Binary data
[slurm-dev] Re: slurmd on first node not responding and is running
Quoting Trevor Gale tre...@snowhaven.com: Hello, I am running slum between two desktop computers for testing purposes and am having a problem where when I launch jobs from the controlnode (rousseau) I get that the jobs have been queued and are waiting for resources and that the required node is not available (down or drained). The issue is in Munge's configuration, which Slurm user for authentication. -- Morris Moe Jette CTO, SchedMD LLC Commercial Slurm Development and Support
[slurm-dev] Re: slurmd on first node not responding and is running
Likely inconsistent keys or clocks. On May 6, 2015 4:20:36 PM PDT, Trevor Gale tre...@snowhaven.com wrote: Do you happen to know what might be incorrect with munges configuration? On May 6, 2015, at 6:55 PM, Moe Jette je...@schedmd.com wrote: Quoting Trevor Gale tre...@snowhaven.com: Hello, I am running slum between two desktop computers for testing purposes and am having a problem where when I launch jobs from the controlnode (rousseau) I get that the jobs have been queued and are waiting for resources and that the required node is not available (down or drained). The issue is in Munge's configuration, which Slurm user for authentication. -- Morris Moe Jette CTO, SchedMD LLC Commercial Slurm Development and Support -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
[slurm-dev] Re: slurm-dev Re: Job allocation for GPU jobs doesn't work using gpu plugin (node configuration not available)
Hi John, DebugFlags=Gres has already been turned on. I raised the debug level to debug5, cleared the logfile, restarted daemons and tried to start a job with –gres=gpu:1. I attached the log files but I cannot find anything that points towards a reason why starting the jobs fails as long as gpu types are included. Regards Daniel Von: John Desantis [mailto:desan...@mail.usf.edu] Gesendet: Mittwoch, 6. Mai 2015 23:43 An: slurm-dev Betreff: [slurm-dev] Re: slurm-dev Re: Job allocation for GPU jobs doesn't work using gpu plugin (node configuration not available) Daniel, Ok, at this point I'd suggest enabling the DebugFlags=Gres in your slurm.conf and turning up the SlurmctldDebug level to debug. You could also change SlurmdDebug to a higher debug level as well. There may be some clues in the extra output. John DeSantis 2015-05-06 16:57 GMT-04:00 Daniel Weber daniel.we...@uni-wuerzburg.de mailto:daniel.we...@uni-wuerzburg.de : Hi John, I replaced all gres.conf files with the „global“ gres.conf file containing information about every node and restarted the controller daemon as well as the slave daemons. The problem persists. Regards Daniel Weber Von: John Desantis [mailto: mailto:desan...@mail.usf.edu desan...@mail.usf.edu] Gesendet: Mittwoch, 6. Mai 2015 22:34 An: slurm-dev Betreff: [slurm-dev] Re: slurm-dev Re: Job allocation for GPU jobs doesn't work using gpu plugin (node configuration not available) Daniel, Use the same gres.conf on all nodes in the cluster (including the controller), and then restart slurm and try again. John DeSantis On May 6, 2015 4:22 PM, Daniel Weber mailto:daniel.we...@uni-wuerzburg.de daniel.we...@uni-wuerzburg.de wrote: Hi John, I added the types into slurm.conf and the gres.conf files on the nodes again and included a gres.conf on the controller node - without any success. Slurm rejects jobs with --gres=gpu:1 or --gres=gpu:tesla:1. slurm.conf NodeName=smurf01 NodeAddr=192.168.1.101 Feature=intel,fermi Boards=1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 Gres=gpu:tesla:8,ram:48,gram:no_consume:6000,scratch:1300 ... gres.conf on controller NodeName=smurf01 Name=gpu Type=tesla File=/dev/nvidia0 Count=1 NodeName=smurf01 Name=gpu Type=tesla File=/dev/nvidia1 Count=1 NodeName=smurf01 Name=gpu Type=tesla File=/dev/nvidia2 Count=1 NodeName=smurf01 Name=gpu Type=tesla File=/dev/nvidia3 Count=1 NodeName=smurf01 Name=gpu Type=tesla File=/dev/nvidia4 Count=1 NodeName=smurf01 Name=gpu Type=tesla File=/dev/nvidia5 Count=1 NodeName=smurf01 Name=gpu Type=tesla File=/dev/nvidia6 Count=1 NodeName=smurf01 Name=gpu Type=tesla File=/dev/nvidia7 Count=1 NodeName=smurf01 Name=ram Count=48 NodeName=smurf01 Name=gram Count=6000 NodeName=smurf01 Name=scratch Count=1300 ... gres.conf on smurf01 Name=gpu Type=tesla File=/dev/nvidia0 Count=1 Name=gpu Type=tesla File=/dev/nvidia1 Count=1 Name=gpu Type=tesla File=/dev/nvidia2 Count=1 Name=gpu Type=tesla File=/dev/nvidia3 Count=1 Name=gpu Type=tesla File=/dev/nvidia4 Count=1 Name=gpu Type=tesla File=/dev/nvidia5 Count=1 Name=gpu Type=tesla File=/dev/nvidia6 Count=1 Name=gpu Type=tesla File=/dev/nvidia7 Count=1 Name=ram Count=48 Name=gram Count=6000 Name=scratch Count=1300 Regards Daniel -Ursprüngliche Nachricht- Von: John Desantis [mailto: mailto:desan...@mail.usf.edu desan...@mail.usf.edu] Gesendet: Mittwoch, 6. Mai 2015 21:33 An: slurm-dev Betreff: [slurm-dev] Re: slurm-dev Re: Job allocation for GPU jobs doesn't work using gpu plugin (node configuration not available) Daniel, I hit send without completing my message: # gres.conf NodeName=blah Name=gpu Type=Tesla-T10 File=/dev/nvidia[0-1] HTH. John DeSantis 2015-05-06 15:30 GMT-04:00 John Desantis mailto:desan...@mail.usf.edu desan...@mail.usf.edu: Daniel, You sparked an interest. I was able to get Gres Types working by: 1.) Ensuring that the type was defined in slurm.conf for the nodes in question; 2.) Ensuring that the global gres.conf respected the type. salloc -n 1 --gres=gpu:Tesla-T10:1 salloc: Pending job allocation 532507 salloc: job 532507 queued and waiting for resources # slurm.conf Nodename=blah CPUs=16 CoresPerSocket=4 Sockets=4 RealMemory=129055 Feature=ib_ddr,ib_ofa,sse,sse2,sse3,tpa,cpu_xeon,xeon_E7330,gpu_T10,ti tan,mem_128G Gres=gpu:Tesla-T10:2 Weight=1000 # gres.conf 2015-05-06 15:25 GMT-04:00 John Desantis mailto:desan...@mail.usf.edu desan...@mail.usf.edu: Daniel, I can handle that temporarily with node features instead but I'd prefer utilizing the gpu types. Guilty of reading your response too quickly... John DeSantis 2015-05-06 15:22 GMT-04:00 John Desantis mailto:desan...@mail.usf.edu desan...@mail.usf.edu: Daniel, Instead of defining the GPU type in our Gres configuration (global with hostnames, no count), we simply add a feature so that users can request a GPU (or GPU's) via Gres and the specific model via a constraint.
[slurm-dev] Re: slurm-dev Re: Job allocation for GPU jobs doesn't work using gpu plugin (node configuration not available)
Daniel, Ok, at this point I'd suggest enabling the DebugFlags=Gres in your slurm.conf and turning up the SlurmctldDebug level to debug. You could also change SlurmdDebug to a higher debug level as well. There may be some clues in the extra output. John DeSantis 2015-05-06 16:57 GMT-04:00 Daniel Weber daniel.we...@uni-wuerzburg.de: Hi John, I replaced all gres.conf files with the „global“ gres.conf file containing information about every node and restarted the controller daemon as well as the slave daemons. The problem persists. Regards Daniel Weber *Von:* John Desantis [mailto:desan...@mail.usf.edu] *Gesendet:* Mittwoch, 6. Mai 2015 22:34 *An:* slurm-dev *Betreff:* [slurm-dev] Re: slurm-dev Re: Job allocation for GPU jobs doesn't work using gpu plugin (node configuration not available) Daniel, Use the same gres.conf on all nodes in the cluster (including the controller), and then restart slurm and try again. John DeSantis On May 6, 2015 4:22 PM, Daniel Weber daniel.we...@uni-wuerzburg.de wrote: Hi John, I added the types into slurm.conf and the gres.conf files on the nodes again and included a gres.conf on the controller node - without any success. Slurm rejects jobs with --gres=gpu:1 or --gres=gpu:tesla:1. slurm.conf NodeName=smurf01 NodeAddr=192.168.1.101 Feature=intel,fermi Boards=1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 Gres=gpu:tesla:8,ram:48,gram:no_consume:6000,scratch:1300 ... gres.conf on controller NodeName=smurf01 Name=gpu Type=tesla File=/dev/nvidia0 Count=1 NodeName=smurf01 Name=gpu Type=tesla File=/dev/nvidia1 Count=1 NodeName=smurf01 Name=gpu Type=tesla File=/dev/nvidia2 Count=1 NodeName=smurf01 Name=gpu Type=tesla File=/dev/nvidia3 Count=1 NodeName=smurf01 Name=gpu Type=tesla File=/dev/nvidia4 Count=1 NodeName=smurf01 Name=gpu Type=tesla File=/dev/nvidia5 Count=1 NodeName=smurf01 Name=gpu Type=tesla File=/dev/nvidia6 Count=1 NodeName=smurf01 Name=gpu Type=tesla File=/dev/nvidia7 Count=1 NodeName=smurf01 Name=ram Count=48 NodeName=smurf01 Name=gram Count=6000 NodeName=smurf01 Name=scratch Count=1300 ... gres.conf on smurf01 Name=gpu Type=tesla File=/dev/nvidia0 Count=1 Name=gpu Type=tesla File=/dev/nvidia1 Count=1 Name=gpu Type=tesla File=/dev/nvidia2 Count=1 Name=gpu Type=tesla File=/dev/nvidia3 Count=1 Name=gpu Type=tesla File=/dev/nvidia4 Count=1 Name=gpu Type=tesla File=/dev/nvidia5 Count=1 Name=gpu Type=tesla File=/dev/nvidia6 Count=1 Name=gpu Type=tesla File=/dev/nvidia7 Count=1 Name=ram Count=48 Name=gram Count=6000 Name=scratch Count=1300 Regards Daniel -Ursprüngliche Nachricht- Von: John Desantis [mailto:desan...@mail.usf.edu] Gesendet: Mittwoch, 6. Mai 2015 21:33 An: slurm-dev Betreff: [slurm-dev] Re: slurm-dev Re: Job allocation for GPU jobs doesn't work using gpu plugin (node configuration not available) Daniel, I hit send without completing my message: # gres.conf NodeName=blah Name=gpu Type=Tesla-T10 File=/dev/nvidia[0-1] HTH. John DeSantis 2015-05-06 15:30 GMT-04:00 John Desantis desan...@mail.usf.edu: Daniel, You sparked an interest. I was able to get Gres Types working by: 1.) Ensuring that the type was defined in slurm.conf for the nodes in question; 2.) Ensuring that the global gres.conf respected the type. salloc -n 1 --gres=gpu:Tesla-T10:1 salloc: Pending job allocation 532507 salloc: job 532507 queued and waiting for resources # slurm.conf Nodename=blah CPUs=16 CoresPerSocket=4 Sockets=4 RealMemory=129055 Feature=ib_ddr,ib_ofa,sse,sse2,sse3,tpa,cpu_xeon,xeon_E7330,gpu_T10,ti tan,mem_128G Gres=gpu:Tesla-T10:2 Weight=1000 # gres.conf 2015-05-06 15:25 GMT-04:00 John Desantis desan...@mail.usf.edu: Daniel, I can handle that temporarily with node features instead but I'd prefer utilizing the gpu types. Guilty of reading your response too quickly... John DeSantis 2015-05-06 15:22 GMT-04:00 John Desantis desan...@mail.usf.edu: Daniel, Instead of defining the GPU type in our Gres configuration (global with hostnames, no count), we simply add a feature so that users can request a GPU (or GPU's) via Gres and the specific model via a constraint. This may help out the situation so that your users can request a specific GPU model. --srun --gres=gpu:1 -C gpu_k20 I didn't think of it at the time, but I remember running --gres=help when initially setting up GPU's to help rule out errors. I don't know if you ran that command or not, but it's worth a shot to verify that Gres types are being seen correctly on a node by the controller. I also wonder if using a cluster wide Gres definition (vs. only on nodes in question) would make a difference or not. John DeSantis 2015-05-06 15:12 GMT-04:00 Daniel Weber daniel.we...@uni-wuerzburg.de : Hi John, I already tried using Count=1 for each line as well as Count=8 for a single configuration
[slurm-dev] Re: slurm-dev Re: Job allocation for GPU jobs doesn't work using gpu plugin (node configuration not available)
Hi John, I already tried using Count=1 for each line as well as Count=8 for a single configuration line as well. I solved (or better circumvented) the problem by removing the Type=... specifications from the gres.conf files and from the slurm.conf. The jobs are running successfully without the possibility to request a certain GPU type. The generic resource examples on schedmd.com explicitly show the Type specifications on GPUs and I really would like to use them. I can handle that temporarily with node features instead but I'd prefer utilizing the gpu types. Thank you for your help (and the hint into the right direction). Kind regards Daniel -Ursprüngliche Nachricht- Von: John Desantis [mailto:desan...@mail.usf.edu] Gesendet: Mittwoch, 6. Mai 2015 18:16 An: slurm-dev Betreff: [slurm-dev] Re: slurm-dev Re: Job allocation for GPU jobs doesn't work using gpu plugin (node configuration not available) Daniel, What about a count? Try adding a count=1 after each of your GPU lines. John DeSantis 2015-05-06 11:54 GMT-04:00 Daniel Weber daniel.we...@uni-wuerzburg.de: The same problem occurs when using the grey type in the srun syntax (using i.e. --gres=gpu:tesla:1). Regards, Daniel -- Von: John Desantis [mailto:desan...@mail.usf.edu] Gesendet: Mittwoch, 6. Mai 2015 17:39 An: slurm-dev Betreff: [slurm-dev] Re: Job allocation for GPU jobs doesn't work using gpu plugin (node configuration not available) Daniel, We don't specify types in our Gres configuration, simply the resource. What happens if you update your srun syntax to: srun -n1 --gres=gpu:tesla:1 Does that dispatch the job? John DeSantis 2015-05-06 9:40 GMT-04:00 Daniel Weber daniel.we...@uni-wuerzburg.de: Hello, currently I'm trying to set up SLURM on a gpu cluster with a small number of nodes (where smurf0[1-7] are the node names) using the gpu plugin to allocate jobs (requiring gpus). Unfortunately, when trying to run a gpu-job (any number of gpus; --gres=gpu:N), SLURM doesn't execute it, asserting unavailability of the requested configuration. I attached some logs and configuration text files in order to provide any information necessary to analyze this issue. Note: Cross posted here: http://serverfault.com/questions/685258 Example (using some test.sh which is echoing $CUDA_VISIBLE_DEVICES): srun -n1 --gres=gpu:1 test.sh -- srun: error: Unable to allocate resources: Requested node configuration is not available The slurmctld log for such calls shows: gres: gpu state for job X gres_cnt:1 node_cnt:1 type:(null) _pick_best_nodes: job X never runnable _slurm_rpc_allocate_resources: Requested node configuration is not available Jobs with any other type of configured generic resource complete successfully: srun -n1 --gres=gram:500 test.sh -- CUDA_VISIBLE_DEVICES=NoDevFiles The nodes and gres configuration in slurm.conf (which is attached as well) are like: GresTypes=gpu,ram,gram,scratch ... NodeName=smurf01 NodeAddr=192.168.1.101 Feature=intel,fermi Boards=1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 Gres=gpu:tesla:8,ram:48,gram:no_consume:6000,scratch:1300 NodeName=smurf02 NodeAddr=192.168.1.102 Feature=intel,fermi Boards=1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=1 Gres=gpu:tesla:8,ram:48,gram:no_consume:6000,scratch:1300 The respective gres.conf files are Name=gpu Count=8 Type=tesla File=/dev/nvidia[0-7] Name=ram Count=48 Name=gram Count=6000 Name=scratch Count=1300 The output of scontrol show node lists all the nodes with the correct gres configuration i.e.: NodeName=smurf01 Arch=x86_64 CoresPerSocket=6 CPUAlloc=0 CPUErr=0 CPUTot=24 CPULoad=0.01 Features=intel,fermi Gres=gpu:tesla:8,ram:48,gram:no_consume:6000,scratch:1300 ...etc. As far as I can tell, the slurmd daemon on the nodes recognizes the gpus (and other generic resources) correctly. My slurmd.log on node smurf01 says Gres Name = gpu Type = tesla Count = 8 ID = 7696487 File = /dev /nvidia[0 - 7] The log for slurmctld shows gres / gpu: state for smurf01 gres_cnt found : 8 configured : 8 avail : 8 alloc : 0 gres_bit_alloc : gres_used : (null) I can't figure out why the controller node states that jobs using --gres=gpu:N are never runnable and why the requested node configuration is not available. Any help is appreciated. Kind regards, Daniel Weber PS: If further information is required, don't hesitate to ask.
[slurm-dev] Re: slurm-dev Re: Job allocation for GPU jobs doesn't work using gpu plugin (node configuration not available)
Daniel, I hit send without completing my message: # gres.conf NodeName=blah Name=gpu Type=Tesla-T10 File=/dev/nvidia[0-1] HTH. John DeSantis 2015-05-06 15:30 GMT-04:00 John Desantis desan...@mail.usf.edu: Daniel, You sparked an interest. I was able to get Gres Types working by: 1.) Ensuring that the type was defined in slurm.conf for the nodes in question; 2.) Ensuring that the global gres.conf respected the type. salloc -n 1 --gres=gpu:Tesla-T10:1 salloc: Pending job allocation 532507 salloc: job 532507 queued and waiting for resources # slurm.conf Nodename=blah CPUs=16 CoresPerSocket=4 Sockets=4 RealMemory=129055 Feature=ib_ddr,ib_ofa,sse,sse2,sse3,tpa,cpu_xeon,xeon_E7330,gpu_T10,titan,mem_128G Gres=gpu:Tesla-T10:2 Weight=1000 # gres.conf 2015-05-06 15:25 GMT-04:00 John Desantis desan...@mail.usf.edu: Daniel, I can handle that temporarily with node features instead but I'd prefer utilizing the gpu types. Guilty of reading your response too quickly... John DeSantis 2015-05-06 15:22 GMT-04:00 John Desantis desan...@mail.usf.edu: Daniel, Instead of defining the GPU type in our Gres configuration (global with hostnames, no count), we simply add a feature so that users can request a GPU (or GPU's) via Gres and the specific model via a constraint. This may help out the situation so that your users can request a specific GPU model. --srun --gres=gpu:1 -C gpu_k20 I didn't think of it at the time, but I remember running --gres=help when initially setting up GPU's to help rule out errors. I don't know if you ran that command or not, but it's worth a shot to verify that Gres types are being seen correctly on a node by the controller. I also wonder if using a cluster wide Gres definition (vs. only on nodes in question) would make a difference or not. John DeSantis 2015-05-06 15:12 GMT-04:00 Daniel Weber daniel.we...@uni-wuerzburg.de: Hi John, I already tried using Count=1 for each line as well as Count=8 for a single configuration line as well. I solved (or better circumvented) the problem by removing the Type=... specifications from the gres.conf files and from the slurm.conf. The jobs are running successfully without the possibility to request a certain GPU type. The generic resource examples on schedmd.com explicitly show the Type specifications on GPUs and I really would like to use them. I can handle that temporarily with node features instead but I'd prefer utilizing the gpu types. Thank you for your help (and the hint into the right direction). Kind regards Daniel -Ursprüngliche Nachricht- Von: John Desantis [mailto:desan...@mail.usf.edu] Gesendet: Mittwoch, 6. Mai 2015 18:16 An: slurm-dev Betreff: [slurm-dev] Re: slurm-dev Re: Job allocation for GPU jobs doesn't work using gpu plugin (node configuration not available) Daniel, What about a count? Try adding a count=1 after each of your GPU lines. John DeSantis 2015-05-06 11:54 GMT-04:00 Daniel Weber daniel.we...@uni-wuerzburg.de: The same problem occurs when using the grey type in the srun syntax (using i.e. --gres=gpu:tesla:1). Regards, Daniel -- Von: John Desantis [mailto:desan...@mail.usf.edu] Gesendet: Mittwoch, 6. Mai 2015 17:39 An: slurm-dev Betreff: [slurm-dev] Re: Job allocation for GPU jobs doesn't work using gpu plugin (node configuration not available) Daniel, We don't specify types in our Gres configuration, simply the resource. What happens if you update your srun syntax to: srun -n1 --gres=gpu:tesla:1 Does that dispatch the job? John DeSantis 2015-05-06 9:40 GMT-04:00 Daniel Weber daniel.we...@uni-wuerzburg.de: Hello, currently I'm trying to set up SLURM on a gpu cluster with a small number of nodes (where smurf0[1-7] are the node names) using the gpu plugin to allocate jobs (requiring gpus). Unfortunately, when trying to run a gpu-job (any number of gpus; --gres=gpu:N), SLURM doesn't execute it, asserting unavailability of the requested configuration. I attached some logs and configuration text files in order to provide any information necessary to analyze this issue. Note: Cross posted here: http://serverfault.com/questions/685258 Example (using some test.sh which is echoing $CUDA_VISIBLE_DEVICES): srun -n1 --gres=gpu:1 test.sh -- srun: error: Unable to allocate resources: Requested node configuration is not available The slurmctld log for such calls shows: gres: gpu state for job X gres_cnt:1 node_cnt:1 type:(null) _pick_best_nodes: job X never runnable _slurm_rpc_allocate_resources: Requested node configuration is not available Jobs with any other type of configured generic resource complete successfully: srun -n1 --gres=gram:500 test.sh -- CUDA_VISIBLE_DEVICES=NoDevFiles The nodes and gres configuration in slurm.conf (which is attached as well) are like: GresTypes=gpu,ram,gram,scratch ...
[slurm-dev] Slurm + openmpi + pmi2 + SLURM_MPI_TYPE environment variable
Hi - I have seen variants of this question asked, but haven't quite found what I'm looking for - so: We have slurm 14.11.5 installed on a cluster with multiple versions of openmpi compiled, with versions 1.8.4 (pmi2) support, and older versions (~1.6.5, w/o pmi2) For handling pmi2 launching with openmpi, we use module files, and upon loading, set the SLURM_MPI_TYPE variable to pmi2. Using salloc and srun will correctly launch processes through pmi2, presumably through correct propagation of the environment variable. Now I would like to accomplish the same thing in a batch script launched through sbatch. When I try this ala sbatch -N### -n## --export=ALL script script echo $SLURM_MPI_TYPE srun some openmpi-compiled executable I find that SLURM_MPI_TYPE, while defined in my launching environment, is undefined, and srun chokes on the 1-core launched / borked installation warning. I would like to utilize a batch script that can toggle between newer and older versions of openmpi, where pmi launching is detected by the appropriate enivornment variable, But haven't found a a way to get this to work. Should it? And if so, does anyone have a suggestion on how to do so? Thanks in advance! Peter - Peter Gordon ExxonMobil Research and Engineering Corporate Strategic Research 1545 Route 22 East Room LF372 Annandale, NJ 08801 phone: (908)-730-2546 fax: (908)-730-3344 -
[slurm-dev] Re: slurm-dev Re: Job allocation for GPU jobs doesn't work using gpu plugin (node configuration not available)
Daniel, I can handle that temporarily with node features instead but I'd prefer utilizing the gpu types. Guilty of reading your response too quickly... John DeSantis 2015-05-06 15:22 GMT-04:00 John Desantis desan...@mail.usf.edu: Daniel, Instead of defining the GPU type in our Gres configuration (global with hostnames, no count), we simply add a feature so that users can request a GPU (or GPU's) via Gres and the specific model via a constraint. This may help out the situation so that your users can request a specific GPU model. --srun --gres=gpu:1 -C gpu_k20 I didn't think of it at the time, but I remember running --gres=help when initially setting up GPU's to help rule out errors. I don't know if you ran that command or not, but it's worth a shot to verify that Gres types are being seen correctly on a node by the controller. I also wonder if using a cluster wide Gres definition (vs. only on nodes in question) would make a difference or not. John DeSantis 2015-05-06 15:12 GMT-04:00 Daniel Weber daniel.we...@uni-wuerzburg.de: Hi John, I already tried using Count=1 for each line as well as Count=8 for a single configuration line as well. I solved (or better circumvented) the problem by removing the Type=... specifications from the gres.conf files and from the slurm.conf. The jobs are running successfully without the possibility to request a certain GPU type. The generic resource examples on schedmd.com explicitly show the Type specifications on GPUs and I really would like to use them. I can handle that temporarily with node features instead but I'd prefer utilizing the gpu types. Thank you for your help (and the hint into the right direction). Kind regards Daniel -Ursprüngliche Nachricht- Von: John Desantis [mailto:desan...@mail.usf.edu] Gesendet: Mittwoch, 6. Mai 2015 18:16 An: slurm-dev Betreff: [slurm-dev] Re: slurm-dev Re: Job allocation for GPU jobs doesn't work using gpu plugin (node configuration not available) Daniel, What about a count? Try adding a count=1 after each of your GPU lines. John DeSantis 2015-05-06 11:54 GMT-04:00 Daniel Weber daniel.we...@uni-wuerzburg.de: The same problem occurs when using the grey type in the srun syntax (using i.e. --gres=gpu:tesla:1). Regards, Daniel -- Von: John Desantis [mailto:desan...@mail.usf.edu] Gesendet: Mittwoch, 6. Mai 2015 17:39 An: slurm-dev Betreff: [slurm-dev] Re: Job allocation for GPU jobs doesn't work using gpu plugin (node configuration not available) Daniel, We don't specify types in our Gres configuration, simply the resource. What happens if you update your srun syntax to: srun -n1 --gres=gpu:tesla:1 Does that dispatch the job? John DeSantis 2015-05-06 9:40 GMT-04:00 Daniel Weber daniel.we...@uni-wuerzburg.de: Hello, currently I'm trying to set up SLURM on a gpu cluster with a small number of nodes (where smurf0[1-7] are the node names) using the gpu plugin to allocate jobs (requiring gpus). Unfortunately, when trying to run a gpu-job (any number of gpus; --gres=gpu:N), SLURM doesn't execute it, asserting unavailability of the requested configuration. I attached some logs and configuration text files in order to provide any information necessary to analyze this issue. Note: Cross posted here: http://serverfault.com/questions/685258 Example (using some test.sh which is echoing $CUDA_VISIBLE_DEVICES): srun -n1 --gres=gpu:1 test.sh -- srun: error: Unable to allocate resources: Requested node configuration is not available The slurmctld log for such calls shows: gres: gpu state for job X gres_cnt:1 node_cnt:1 type:(null) _pick_best_nodes: job X never runnable _slurm_rpc_allocate_resources: Requested node configuration is not available Jobs with any other type of configured generic resource complete successfully: srun -n1 --gres=gram:500 test.sh -- CUDA_VISIBLE_DEVICES=NoDevFiles The nodes and gres configuration in slurm.conf (which is attached as well) are like: GresTypes=gpu,ram,gram,scratch ... NodeName=smurf01 NodeAddr=192.168.1.101 Feature=intel,fermi Boards=1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 Gres=gpu:tesla:8,ram:48,gram:no_consume:6000,scratch:1300 NodeName=smurf02 NodeAddr=192.168.1.102 Feature=intel,fermi Boards=1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=1 Gres=gpu:tesla:8,ram:48,gram:no_consume:6000,scratch:1300 The respective gres.conf files are Name=gpu Count=8 Type=tesla File=/dev/nvidia[0-7] Name=ram Count=48 Name=gram Count=6000 Name=scratch Count=1300 The output of scontrol show node lists all the nodes with the correct gres configuration i.e.: NodeName=smurf01 Arch=x86_64 CoresPerSocket=6 CPUAlloc=0 CPUErr=0 CPUTot=24 CPULoad=0.01 Features=intel,fermi Gres=gpu:tesla:8,ram:48,gram:no_consume:6000,scratch:1300 ...etc. As far as
[slurm-dev] Effect of QOS UsageFactor on Accounting
Slurm 14.11.5 w/ slurmdbd mysql It seems, both from testing (see below) and through examination of source, that QOS UsageFactor (typically qos_ptr-usage_factor in code) is NOT used to adjust actual usage but ONLY for adjusting items like the decay in the priority/multifactor plugin and (possibly) in modifying the behavior of backfill operations. The documentation[1] suggests that usage in the accounting database will be adjusted... UsageFactor Usage factor when running with this QOS (i.e. .5 would make it use only half the time as normal in accounting and 2 would make it use twice as much.) Is this a bug, documentation 'error' or simply a misunderstanding of the functionality? What we want is to be able to charge different amounts for different partitions and/or QOS's that provide access that differs from 'normal' access (ie. priority, resource, etc). Having the job usage adjusted automatically, as the AllocCPUs value is for Shared=EXCLUSIVE partitions is what we are after (ask for 1 cpu in EXCLUSIVE charged for ALL cpus automatically). Thanks, -- Trevor [1] - http://slurm.schedmd.com/qos.html#qos_other =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Example =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- $ egrep -i qos|gpu-shared /etc/slurm/slurm.conf PriorityWeightQOS=0 AccountingStorageEnforce=association,limits,qos PartitionName=gpu-shared Nodes=tux-1,tux-2 Default=NO Shared=FORCE:1 MaxTime=48:00:00 MaxNodes=2 MaxMemPerNode=122880 Priority=1000 AllowQOS=gpu-shared State=UP $ sacctmgr list assoc where user=tcooper format=user,qos User QOS -- tcoopergpu-shared,normal $ sacctmgr list qos where name=gpu-shared format=name,priority,flags,usagefactor,maxnodes,maxcpusperuser Name PriorityFlags UsageFactor MaxNodes MaxCPUsPU -- -- --- - gpu-shared 0 DenyOnLimit2.00124 NOTE: Here we want to charge a premium for GPU's $ sbatch -p gpu-shared --qos=gpu-shared --nodes=1 --ntasks-per-node=1 --gres=gpu:1 -t 00:10:00 sleep_for_walltime.run Submitted batch job 451721 NOTE: Job actually sleeps for walltime request less 5 seconds to prevent being timed out thus this job should have a walltime of 09:55 plus a second or two for prolog/epilog. $ scontrol show job 451721 JobId=451721 JobName=sleep_for_walltime UserId=tcooper(500) GroupId=tcooper(500) Priority=653 Nice=0 Account=tcooper QOS=gpu-shared JobState=RUNNING Reason=None Dependency=(null) Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0 RunTime=00:00:15 TimeLimit=00:10:00 TimeMin=N/A SubmitTime=2015-05-06T10:51:02 EligibleTime=2015-05-06T10:51:02 StartTime=2015-05-06T10:51:04 EndTime=2015-05-06T11:01:04 PreemptTime=None SuspendTime=None SecsPreSuspend=0 Partition=gpu-shared AllocNode:Sid=tux-ln1:18918 ReqNodeList=(null) ExcNodeList=(null) NodeList=tux-1 BatchHost=tux-1 NumNodes=1 NumCPUs=1 CPUs/Task=1 ReqB:S:C:T=0:0:*:* Socks/Node=* NtasksPerN:B:S:C=1:0:*:* CoreSpec=* MinCPUsNode=1 MinMemoryCPU=5G MinTmpDiskNode=0 Features=(null) Gres=gpu:1 Reservation=(null) Shared=OK Contiguous=0 Licenses=(null) Network=(null) Command=/home/tcooper/jobs/sleep_for_walltime.run WorkDir=/home/tcooper/jobs StdErr=/home/tcooper/jobs/sleep_for_walltime.451721.%N.err StdIn=/dev/null StdOut=/home/tcooper/jobs/sleep_for_walltime.451721.%N.out $ sacct --format=jobid,alloccpus,start,end,cputime,state,exitcode -j 451721 JobID AllocCPUS Start EndCPUTime State ExitCode -- --- --- -- -- 4517211 2015-05-06T10:51:04 2015-05-06T11:01:00 00:09:56 COMPLETED 0:0 451721.batch 1 2015-05-06T10:51:04 2015-05-06T11:01:00 00:09:56 COMPLETED 0:0 NOTE: Here I expect the CPUTime to be multiplied (ie. doubled) by the QOS-UsageFactor $ sreport cluster UserUtilizationByAccount start=2015-05-06T10:00:00 end=now format=login,account,used Cluster/User/Account Utilization 2015-05-06T10:00:00 - 2015-05-06T11:59:59 (7200 secs) Time reported in CPU Minutes Login Account Used - --- -- tcooper sys200 10 NOTE: Since the previous represents the 'actual' job runtime it's possible the QOS-UsageFactor could be applied in the rollup of usage. If that were the case I expect Used to be multiplied (ie. doubled) by the QOS-UsageFactor.
[slurm-dev] Re: slurm-dev Re: Job allocation for GPU jobs doesn't work using gpu plugin (node configuration not available)
Daniel, What about a count? Try adding a count=1 after each of your GPU lines. John DeSantis 2015-05-06 11:54 GMT-04:00 Daniel Weber daniel.we...@uni-wuerzburg.de: The same problem occurs when using the grey type in the srun syntax (using i.e. --gres=gpu:tesla:1). Regards, Daniel -- Von: John Desantis [mailto:desan...@mail.usf.edu] Gesendet: Mittwoch, 6. Mai 2015 17:39 An: slurm-dev Betreff: [slurm-dev] Re: Job allocation for GPU jobs doesn't work using gpu plugin (node configuration not available) Daniel, We don't specify types in our Gres configuration, simply the resource. What happens if you update your srun syntax to: srun -n1 --gres=gpu:tesla:1 Does that dispatch the job? John DeSantis 2015-05-06 9:40 GMT-04:00 Daniel Weber daniel.we...@uni-wuerzburg.de: Hello, currently I'm trying to set up SLURM on a gpu cluster with a small number of nodes (where smurf0[1-7] are the node names) using the gpu plugin to allocate jobs (requiring gpus). Unfortunately, when trying to run a gpu-job (any number of gpus; --gres=gpu:N), SLURM doesn't execute it, asserting unavailability of the requested configuration. I attached some logs and configuration text files in order to provide any information necessary to analyze this issue. Note: Cross posted here: http://serverfault.com/questions/685258 Example (using some test.sh which is echoing $CUDA_VISIBLE_DEVICES): srun -n1 --gres=gpu:1 test.sh -- srun: error: Unable to allocate resources: Requested node configuration is not available The slurmctld log for such calls shows: gres: gpu state for job X gres_cnt:1 node_cnt:1 type:(null) _pick_best_nodes: job X never runnable _slurm_rpc_allocate_resources: Requested node configuration is not available Jobs with any other type of configured generic resource complete successfully: srun -n1 --gres=gram:500 test.sh -- CUDA_VISIBLE_DEVICES=NoDevFiles The nodes and gres configuration in slurm.conf (which is attached as well) are like: GresTypes=gpu,ram,gram,scratch ... NodeName=smurf01 NodeAddr=192.168.1.101 Feature=intel,fermi Boards=1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 Gres=gpu:tesla:8,ram:48,gram:no_consume:6000,scratch:1300 NodeName=smurf02 NodeAddr=192.168.1.102 Feature=intel,fermi Boards=1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=1 Gres=gpu:tesla:8,ram:48,gram:no_consume:6000,scratch:1300 The respective gres.conf files are Name=gpu Count=8 Type=tesla File=/dev/nvidia[0-7] Name=ram Count=48 Name=gram Count=6000 Name=scratch Count=1300 The output of scontrol show node lists all the nodes with the correct gres configuration i.e.: NodeName=smurf01 Arch=x86_64 CoresPerSocket=6 CPUAlloc=0 CPUErr=0 CPUTot=24 CPULoad=0.01 Features=intel,fermi Gres=gpu:tesla:8,ram:48,gram:no_consume:6000,scratch:1300 ...etc. As far as I can tell, the slurmd daemon on the nodes recognizes the gpus (and other generic resources) correctly. My slurmd.log on node smurf01 says Gres Name = gpu Type = tesla Count = 8 ID = 7696487 File = /dev /nvidia[0 - 7] The log for slurmctld shows gres / gpu: state for smurf01 gres_cnt found : 8 configured : 8 avail : 8 alloc : 0 gres_bit_alloc : gres_used : (null) I can't figure out why the controller node states that jobs using --gres=gpu:N are never runnable and why the requested node configuration is not available. Any help is appreciated. Kind regards, Daniel Weber PS: If further information is required, don't hesitate to ask.
[slurm-dev] Re: slurmd on first node not responding and is running
I’ve disabled munge and now my two nodes see each other, but now I get an error that says “task launch for job 352 failed: Invalid job credential”. Any idea what might be causing this error? I’ve turned my SlurmdDebug up to 7 but the log file essentially says the exact same thing as the stdout when I try and submit a job. Thanks, Trevor On May 6, 2015, at 9:54 PM, Uwe Sauter uwe.sauter...@gmail.com wrote: Configure your SLURM host as NTP server and your compute clients as NTP clients, Am 06.05.2015 um 19:20 schrieb Trevor Gale: Ive made sure the keys are the same, how do you synchronize the clocks/tell if they're out of synch? On May 6, 2015, at 7:24 PM, Morris Jette je...@schedmd.com mailto:je...@schedmd.com wrote: Likely inconsistent keys or clocks. On May 6, 2015 4:20:36 PM PDT, Trevor Gale tre...@snowhaven.com mailto:tre...@snowhaven.com wrote: Do you happen to know what might be incorrect with munges configuration? On May 6, 2015, at 6:55 PM, Moe Jette je...@schedmd.com mailto:je...@schedmd.com wrote: Quoting Trevor Gale tre...@snowhaven.com mailto:tre...@snowhaven.com: Hello, I am running slum between two desktop computers for testing purposes and am having a problem where when I launch jobs from the controlnode (rousseau) I get that the jobs have been queued and are waiting for resources and that the required node is not available (down or drained). The issue is in Munge's configuration, which Slurm user for authentication. -- Morris Moe Jette CTO, SchedMD LLC Commercial Slurm Development and Support -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
[slurm-dev] Re: Job allocation for GPU jobs doesn't work using gpu plugin (node configuration not available)
Daniel, We don't specify types in our Gres configuration, simply the resource. What happens if you update your srun syntax to: srun -n1 --gres=gpu:tesla:1 Does that dispatch the job? John DeSantis 2015-05-06 9:40 GMT-04:00 Daniel Weber daniel.we...@uni-wuerzburg.de: Hello, currently I'm trying to set up SLURM on a gpu cluster with a small number of nodes (where smurf0[1-7] are the node names) using the gpu plugin to allocate jobs (requiring gpus). Unfortunately, when trying to run a gpu-job (any number of gpus; --gres=gpu:N), SLURM doesn't execute it, asserting unavailability of the requested configuration. I attached some logs and configuration text files in order to provide any information necessary to analyze this issue. Note: Cross posted here: http://serverfault.com/questions/685258 Example (using some test.sh which is echoing $CUDA_VISIBLE_DEVICES): srun -n1 --gres=gpu:1 test.sh -- srun: error: Unable to allocate resources: Requested node configuration is not available The slurmctld log for such calls shows: gres: gpu state for job X gres_cnt:1 node_cnt:1 type:(null) _pick_best_nodes: job X never runnable _slurm_rpc_allocate_resources: Requested node configuration is not available Jobs with any other type of configured generic resource complete successfully: srun -n1 --gres=gram:500 test.sh -- CUDA_VISIBLE_DEVICES=NoDevFiles The nodes and gres configuration in slurm.conf (which is attached as well) are like: GresTypes=gpu,ram,gram,scratch ... NodeName=smurf01 NodeAddr=192.168.1.101 Feature=intel,fermi Boards=1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 Gres=gpu:tesla:8,ram:48,gram:no_consume:6000,scratch:1300 NodeName=smurf02 NodeAddr=192.168.1.102 Feature=intel,fermi Boards=1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=1 Gres=gpu:tesla:8,ram:48,gram:no_consume:6000,scratch:1300 The respective gres.conf files are Name=gpu Count=8 Type=tesla File=/dev/nvidia[0-7] Name=ram Count=48 Name=gram Count=6000 Name=scratch Count=1300 The output of scontrol show node lists all the nodes with the correct gres configuration i.e.: NodeName=smurf01 Arch=x86_64 CoresPerSocket=6 CPUAlloc=0 CPUErr=0 CPUTot=24 CPULoad=0.01 Features=intel,fermi Gres=gpu:tesla:8,ram:48,gram:no_consume:6000,scratch:1300 ...etc. As far as I can tell, the slurmd daemon on the nodes recognizes the gpus (and other generic resources) correctly. My slurmd.log on node smurf01 says Gres Name = gpu Type = tesla Count = 8 ID = 7696487 File = /dev /nvidia[0 - 7] The log for slurmctld shows gres / gpu: state for smurf01 gres_cnt found : 8 configured : 8 avail : 8 alloc : 0 gres_bit_alloc : gres_used : (null) I can't figure out why the controller node states that jobs using --gres=gpu:N are never runnable and why the requested node configuration is not available. Any help is appreciated. Kind regards, Daniel Weber PS: If further information is required, don't hesitate to ask.
[slurm-dev] Re: Error while loading shared libraries
If you only have mpich, mpic++ is using mpich for the compilation. For it to use openmpi you should compile it in the compute nodes. On Wed, May 6, 2015 at 4:41 PM, Fany Pages Diaz fpa...@udio.cujae.edu.cu wrote: I compiled this way *mpic++ -openmp prime_number.mpi.C -o example*, that's wrong? how it's made? -Original Message- From: Carlos Fenoy mini...@gmail.com To: slurm-dev slurm-dev@schedmd.com Date: Wed, 06 May 2015 07:27:31 -0700 Subject: [slurm-dev] Re: Error while loading shared libraries Then you can not execute a binary compiled with MPICH in the compute nodes. If you want to execute on the compute nodes you have to compile the binary with openmpi On Wed, May 6, 2015 at 4:23 PM, Fany Pagés Díaz fpa...@udio.cujae.edu.cu wrote: This library is on a server and has all the permits, although in the nodes is not found. I nodes openmpi-1.5.4-2 is installed. * De:* Carlos Fenoy [mailto: mini...@gmail.com] *Enviado el:* miércoles, 06 de mayo de 2015 10:11 *Para:* slurm-dev *Asunto:* [slurm-dev] Re: Error while loading shared libraries Do you have mpi installed on all the cluster nodes? On Wed, May 6, 2015 at 3:34 PM, Uwe Sauter uwe.sauter...@gmail.com wrote: Check the file permissions for libmpichcxx.so.1.2 as well as the permissions on the parent directories. Might be that you are not allowed to access the folder structure as the user you're running your application. Am 06.05.2015 um 07:57 schrieb Fany Pagés Díaz: Hello, WhenI throw a MPI job on slurm I get this error, Any can help me? thanks you [root@cluster bin]# srun -N4 numeroprimos /usr/bin/numeroprimos: error while loading shared libraries: libmpichcxx.so.1.2: cannot open shared object file: No such file or directory Untitled Document 50 Aniversario de la Cujae. Inaugurada por Fidel el 2 de diciembre de 1964 http://cujae.edu.cu http://cujae.edu.cu -- -- Carles Fenoy 50 Aniversario de la Cujae. Inaugurada por Fidel el 2 de diciembre de 1964 http://cujae.edu.cu -- -- Carles Fenoy 50 Aniversario de la Cujae. Inaugurada por Fidel el 2 de diciembre de 1964 http://cujae.edu.cu -- -- Carles Fenoy
[slurm-dev] slurm-dev Re: Job allocation for GPU jobs doesn't work using gpu plugin (node configuration not available)
The same problem occurs when using the grey type in the srun syntax (using i.e. --gres=gpu:tesla:1). Regards, Daniel -- Von: John Desantis [mailto:desan...@mail.usf.edu] Gesendet: Mittwoch, 6. Mai 2015 17:39 An: slurm-dev Betreff: [slurm-dev] Re: Job allocation for GPU jobs doesn't work using gpu plugin (node configuration not available) Daniel, We don't specify types in our Gres configuration, simply the resource. What happens if you update your srun syntax to: srun -n1 --gres=gpu:tesla:1 Does that dispatch the job? John DeSantis 2015-05-06 9:40 GMT-04:00 Daniel Weber daniel.we...@uni-wuerzburg.de: Hello, currently I'm trying to set up SLURM on a gpu cluster with a small number of nodes (where smurf0[1-7] are the node names) using the gpu plugin to allocate jobs (requiring gpus). Unfortunately, when trying to run a gpu-job (any number of gpus; --gres=gpu:N), SLURM doesn't execute it, asserting unavailability of the requested configuration. I attached some logs and configuration text files in order to provide any information necessary to analyze this issue. Note: Cross posted here: http://serverfault.com/questions/685258 Example (using some test.sh which is echoing $CUDA_VISIBLE_DEVICES): srun -n1 --gres=gpu:1 test.sh -- srun: error: Unable to allocate resources: Requested node configuration is not available The slurmctld log for such calls shows: gres: gpu state for job X gres_cnt:1 node_cnt:1 type:(null) _pick_best_nodes: job X never runnable _slurm_rpc_allocate_resources: Requested node configuration is not available Jobs with any other type of configured generic resource complete successfully: srun -n1 --gres=gram:500 test.sh -- CUDA_VISIBLE_DEVICES=NoDevFiles The nodes and gres configuration in slurm.conf (which is attached as well) are like: GresTypes=gpu,ram,gram,scratch ... NodeName=smurf01 NodeAddr=192.168.1.101 Feature=intel,fermi Boards=1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=2 Gres=gpu:tesla:8,ram:48,gram:no_consume:6000,scratch:1300 NodeName=smurf02 NodeAddr=192.168.1.102 Feature=intel,fermi Boards=1 SocketsPerBoard=2 CoresPerSocket=6 ThreadsPerCore=1 Gres=gpu:tesla:8,ram:48,gram:no_consume:6000,scratch:1300 The respective gres.conf files are Name=gpu Count=8 Type=tesla File=/dev/nvidia[0-7] Name=ram Count=48 Name=gram Count=6000 Name=scratch Count=1300 The output of scontrol show node lists all the nodes with the correct gres configuration i.e.: NodeName=smurf01 Arch=x86_64 CoresPerSocket=6 CPUAlloc=0 CPUErr=0 CPUTot=24 CPULoad=0.01 Features=intel,fermi Gres=gpu:tesla:8,ram:48,gram:no_consume:6000,scratch:1300 ...etc. As far as I can tell, the slurmd daemon on the nodes recognizes the gpus (and other generic resources) correctly. My slurmd.log on node smurf01 says Gres Name = gpu Type = tesla Count = 8 ID = 7696487 File = /dev /nvidia[0 - 7] The log for slurmctld shows gres / gpu: state for smurf01 gres_cnt found : 8 configured : 8 avail : 8 alloc : 0 gres_bit_alloc : gres_used : (null) I can't figure out why the controller node states that jobs using --gres=gpu:N are never runnable and why the requested node configuration is not available. Any help is appreciated. Kind regards, Daniel Weber PS: If further information is required, don't hesitate to ask.
