Hi everyone, I am trying to configure slurm to use openmpi for the first time. I am familiar with how to submit parallel jobs with openmpi on an existing cluster, but cannot for the life of me figure out how to get parallel jobs to submit correctly on my test virtual cluster. Here's some of system info
CentOS: 6.6 Slurm: 14.11.5 openmpi: 1.8.6 Virtual Cluster: 3 nodes: head, rd1, rd2. 2 total processors (1 each for rd1 and rd2) slurm has been working normally, and I compiled openmpi without any errors. However, when I use srun to try to run a parallel job, it just runs 1 process on each node separately, not as part of the same call. I don't know if I'm saying that right, but this output should explain what I mean: [mkg52@head mpi_jobs]$ mpiexec -np 2 ./test_mpi Process 0 on head out of 2 Process 1 on head out of 2 [mkg52@head mpi_jobs]$ salloc --ntasks=2 salloc: Granted job allocation 140 [mkg52@head mpi_jobs]$ mpiexec ./test_mpi Process 1 on rd2 out of 2 Process 0 on rd1 out of 2 [mkg52@head mpi_jobs]$ mpirun ./test_mpi Process 1 on rd2 out of 2 Process 0 on rd1 out of 2 [mkg52@head mpi_jobs]$ [mkg52@head mpi_jobs]$ srun ./test_mpi Process 0 on rd1 out of 1 Process 0 on rd2 out of 1 As you can see, srun doesn't really utilize mpi, it just runs it twice, once for each node. I just don't know what I did incorrectly. If anyone could give me some pointers, it would be greatly appreciated! Here's my slurm.conf: # slurm.conf file generated by configurator.html. # Put this file on all nodes of your cluster. # See the slurm.conf man page for more information. # #PluginDir=/usr/lib64/slurm ControlMachine=head ControlAddr=172.16.2.1 BackupController=head-bkup BackupAddr=172.16.2.2 # AuthType=auth/munge CacheGroups=0 #CheckpointType=checkpoint/none CryptoType=crypto/munge #DisableRootJobs=NO #EnforcePartLimits=NO #Epilog= #EpilogSlurmctld= #FirstJobId=1 #MaxJobId=999999 GresTypes=nic #GroupUpdateForce=0 #GroupUpdateTime=600 #JobCheckpointDir=/var/slurm/checkpoint #JobCredentialPrivateKey= #JobCredentialPublicCertificate= #JobFileAppend=0 #JobRequeue=1 #JobSubmitPlugins=job_submit/require_timelimit,lua JobSubmitPlugins=lua #KillOnBadExit=0 #LaunchType=launch/slurm #Licenses=foo*4,bar #MailProg=/bin/mail #MaxJobCount=5000 #MaxStepCount=40000 #MaxTasksPerNode=128 MpiDefault=openmpi MpiParams=ports=12000-12999 #PluginDir= #PlugStackConfig= #PrivateData=jobs #ProctrackType=proctrack/pgid ProctrackType=proctrack/cgroup #Prolog= #PrologSlurmctld= #PropagatePrioProcess=0 #PropagateResourceLimits= #PropagateResourceLimitsExcept= #RebootProgram= ReturnToService=1 #SallocDefaultCommand= SlurmctldPidFile=/var/run/slurm/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurm/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/spool/slurm/slurmd SlurmUser=slurm #SrunEpilog= #SrunProlog= StateSaveLocation=/var/spool/slurm SwitchType=switch/none #TaskEpilog= #TaskPlugin=task/affinity TaskPlugin=task/cgroup #TaskPluginParam= #TaskProlog= #TopologyPlugin=topology/tree #TmpFS=/tmp #TrackWCKey=no #TreeWidth= #UnkillableStepProgram= #UsePAM=0 # # # TIMERS #BatchStartTimeout=10 #CompleteWait=0 #EpilogMsgTime=2000 #GetEnvTimeout=2 #HealthCheckInterval=0 #HealthCheckProgram= InactiveLimit=0 KillWait=30 #MessageTimeout=10 #ResvOverRun=0 MinJobAge=300 #OverTimeLimit=0 SlurmctldTimeout=120 SlurmdTimeout=300 #UnkillableStepTimeout=60 #VSizeFactor=0 Waittime=0 # # # SCHEDULING #DefMemPerCPU=200 #MaxMemPerCPU=200 FastSchedule=1 #SchedulerRootFilter=1 #SchedulerTimeSlice=30 SchedulerType=sched/builtin SchedulerPort=7321 SelectType=select/cons_res SelectTypeParameters=CR_Core_Memory,CR_Pack_Nodes # # gtg for partition preempting #PreemptMode = SUSPEND,GANG #PreemptType = preempt/partition_prio PreemptType = preempt/qos PreemptMode = REQUEUE # # JOB PRIORITY #PriorityFlags= PriorityType=priority/multifactor PriorityDecayHalfLife=7-0 PriorityMaxAge=7-0 PriorityWeightAge=100 PriorityWeightFairshare=1000000 PriorityWeightJobSize=1000 PriorityWeightPartition=10000 PriorityWeightQOS=10000000 #PriorityCalcPeriod= #PriorityFavorSmall= #PriorityUsageResetPeriod= # # # LOGGING AND ACCOUNTING AccountingStorageEnforce=all AccountingStorageHost=head #AccountingStorageLoc= #AccountingStoragePass= AccountingStoragePort=6869 AccountingStorageType=accounting_storage/slurmdbd #AccountingStorageUser= AccountingStoreJobComment=YES ClusterName=monsoon-dev #DebugFlags= #JobCompHost= #JobCompLoc= #JobCompPass= #JobCompPort= #JobCompType=jobcomp/slurmdbd #JobCompUser= #JobAcctGatherFrequency=30 #JobAcctGatherType=jobacct_gather/slurmdbd SlurmctldDebug=debug3 #3 #SlurmctldLogFile= SlurmdDebug=debug3 #3 #SlurmdLogFile= #SlurmSchedLogFile= #SlurmSchedLogLevel= # # # POWER SAVE SUPPORT FOR IDLE NODES (optional) #SuspendProgram= #ResumeProgram= #SuspendTimeout= #ResumeTimeout= #ResumeRate= #SuspendExcNodes= #SuspendExcParts= #SuspendRate= #SuspendTime= # # # COMPUTE NODES #NodeName=rd[1] Gres=nic:1 CPUs=1 SocketsPerBoard=1 CoresPerSocket=1 ThreadsPerCore=1 State=UNKNOWN RealMemory=595 Weight=10 Feature=himem,debug,gpu NodeName=rd[1-8] CPUs=1 SocketsPerBoard=1 CoresPerSocket=1 ThreadsPerCore=1 State=UNKNOWN RealMemory=595 Weight=1 ## used for preemptable partition #PartitionName=debug Nodes=rd[1-2] Default=YES MaxTime=30:00 State=UP Priority=20 Shared=NO #PartitionName=all Nodes=rd[1-2] Default=NO MaxTime=04:00:00 State=UP Priority=10 Shared=NO PartitionName=debug Nodes=rd[1-8] Default=YES MaxTime=30:00 State=UP PartitionName=all Nodes=rd[1-8] Default=YES MaxTime=04:00:00 State=UP DefMemPerCPU=200 MaxMemPerCPU=200 PartitionName=himem Nodes=rd[1] Default=NO MaxTime=04:00:00 State=UP DefMemPerCPU=400 MaxMemPerCPU=400 PartitionName=long Nodes=rd[1] Default=NO MaxTime=04:00:00 State=UP DefMemPerCPU=200 MaxMemPerCPU=200 PartitionName=test Nodes=rd[1] Default=NO MaxTime=04:00:00 State=UP DefMemPerCPU=200 MaxMemPerCPU=200
