Hi,I may have isolated the problem, but I don't know if I am doing anything wrong or there's a bug.
When I submit something like the job attached (job.sh), asking for a single GPU card, in a node and one process per core, and then have a look at its "description":
scontrol show job 4647 it reports (besides other information): NodeList=k20n002 BatchHost=k20n002 NumNodes=2 NumCPUs=2 CPUs/Task=1 ReqS:C:T=*:*:*Why does that say "NumNodes=2"? NumCPUs=2? I attach my slurm.conf, in case it is of interest.
If I submit too much of these jobs, I am able to hang SLURM and get: sbatch: error: slurm_receive_msg: Socket timed out on send/recv operationsbatch: error: Batch job submission failed: Socket timed out on send/recv operation
Slurm becomes then unresposive, but if I: pkill -9 slurmctld and then: service slurm start it comes back to life. Again, slurm-2.6.3 Any help would be welcome, Albert On 21/01/14 14:31, Albert Solernou wrote:
BTW, This was running slurm 2.6.3 without any patch. Albert On 21/01/14 14:28, Albert Solernou wrote:Hi, slurmctld stopped running without any apparent reason. I "service slurm start" but "squeue" still returns: slurm_load_jobs error: Unable to contact slurm controller (connect failure) Browsing the logs, I found four copies of the following message in the SlurmctldLogFile: [2014-01-21T14:20:27.692] gres_gresid_to_gresname--The gres_conf_list is NULL!!! Now the slurmctld is using 100% of a core and is absolutely unresponsive. I don't know any other solution than a "startclean", but this would mean that the jobs are deleted. Any solution, or idea on what could cause the problem? Best, Albert
-- --------------------------------- Dr. Albert Solernou Research Associate Oxford Supercomputing Centre, University of Oxford Tel: +44 (0)1865 610631 ---------------------------------
job.sh
Description: application/shellscript
# # Example slurm.conf file. Please run configurator.html # (in doc/html) to build a configuration file customized # for your environment. # # # slurm.conf file generated by configurator.html. # # See the slurm.conf man page for more information. # ClusterName=arcusgpu ControlMachine=arcus-gpu #ControlAddr= #BackupController= #BackupAddr= # SlurmUser=slurm #SlurmdUser=root SlurmctldPort=6817 SlurmdPort=6818 AuthType=auth/munge #JobCredentialPrivateKey= #JobCredentialPublicCertificate= StateSaveLocation=/var/spool/slurm/slurm.state SlurmdSpoolDir=/var/spool/slurm/slurmd SwitchType=switch/none MpiDefault=none # MpiParams=ports=12000-12999 SlurmctldPidFile=/var/spool/slurm/run/slurmctld.pid SlurmdPidFile=/var/spool/slurm/slurmd.pid # ProctrackType=proctrack/pgid ProctrackType=proctrack/linuxproc #PluginDir= # CacheGroups=0 CacheGroups=1 #FirstJobId= ReturnToService=0 RebootProgram=/usr/bin/reboot #MaxJobCount= #PlugStackConfig= #PropagatePrioProcess= PropagateResourceLimits=NONE #PropagateResourceLimitsExcept= #Prolog= #Epilog= #SrunProlog= #SrunEpilog= #TaskProlog= #TaskEpilog= #TaskPlugin= #TrackWCKey=no #TreeWidth=50 #TmpFS= #UsePAM= # # TIMERS SlurmctldTimeout=300 SlurmdTimeout=300 InactiveLimit=0 MinJobAge=300 KillWait=30 Waittime=0 # # SCHEDULING SchedulerType=sched/backfill #SchedulerAuth= #SchedulerPort= #SchedulerRootFilter= SelectType=select/cons_res SelectTypeParameters=CR_Core GresTypes=gpu FastSchedule=1 #PriorityType=priority/multifactor #PriorityDecayHalfLife=14-0 #PriorityUsageResetPeriod=14-0 #PriorityWeightFairshare=100000 #PriorityWeightAge=1000 #PriorityWeightPartition=10000 #PriorityWeightJobSize=1000 #PriorityMaxAge=1-0 # # LOGGING SlurmctldDebug=4 SlurmctldLogFile=/var/spool/slurm/ctld.log SlurmdDebug=4 SlurmdLogFile="/var/spool/slurm/%n.d.log" JobCompType=jobcomp/filetxt JobCompLoc=/var/spool/slurm/log # # ACCOUNTING JobAcctGatherType=jobacct_gather/linux #JobAcctGatherFrequency=30 # AccountingStorageType=accounting_storage/slurmdbd AccountingStorageHost=login1 AccountingStoragePort=6819 #AccountingStorageEnforce=associations #AccountingStorageHost= #AccountingStorageLoc= #AccountingStoragePass= #AccountingStorageUser= # # COMPUTE NODES NodeName=k20n00[1-8] Sockets=1 CoresPerSocket=6 ThreadsPerCore=2 Gres=gpu:2 State=UNKNOWN PartitionName=k20 Nodes=k20n00[1-8] Default=NO MaxTime=120:00:00 State=UP
