Hi All, I'm trying to get a job to run on 2 VMs that share the same physical host. I'm having some issues getting --switches to work. I chose --switches=1 and then using virtual switches to cluster the VMs together by physical host since it seemed the simplest way to do it. Perhaps there is a better way?
Here is the a simple reproducer [root@X ~]# salloc -N2 -p Virt --switches=1@3-00:00:00 salloc: Pending job allocation 307282 salloc: job 307282 queued and waiting for resources salloc: job 307282 has been allocated resources salloc: Granted job allocation 307282 [root@X ~]# squeue --state=R -p Virt JOBID PARTITION NAME USER STATE TIME NODES WCKEY NODELIST(REASON) PRIORITY 307282 Virt bash root RUNNIN 0:03 2 * sandy3-vm2,sandy4-vm2 0.99998585786670 As you can see I got two nodes on different physical hosts within ~60 seconds of making the request even though I set the maxtime to 3 days. I'm hoping I'm missing something, any help would be appreciated, thanks! We are using slurm 14.03.1-2 on all the nodes Here are my conf files >From slurm.conf ClusterName=X ControlMachine=X NodeName=sandy1-vm[1-2] RealMemory=1800 CoresPerSocket=8 Sockets=1 ThreadsPerCore=1 State=UNKNOWN Weight=20 Feature=intel,ib,sandy,virt,v1 NodeName=sandy2-vm[1-2] RealMemory=1800 CoresPerSocket=8 Sockets=1 ThreadsPerCore=1 State=UNKNOWN Weight=20 Feature=intel,ib,sandy,virt,v2 NodeName=sandy3-vm[1-2] RealMemory=1800 CoresPerSocket=8 Sockets=1 ThreadsPerCore=1 State=UNKNOWN Weight=20 Feature=intel,ib,sandy,virt,v3 NodeName=sandy4-vm[1-2] RealMemory=1800 CoresPerSocket=8 Sockets=1 ThreadsPerCore=1 State=UNKNOWN Weight=20 Feature=intel,ib,sandy,virt,v4 PartitionName=Virt Nodes=sandy[1-4]-vm[1-2] MaxTime=3-00 SwitchType=switch/none MpiDefault=none SlurmctldPort=6817 SlurmdPort=6818 AuthType=auth/munge SchedulerType=sched/backfill FirstJobId=1 FastSchedule=1 SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdLogFile=/var/log/slurm/slurmd.log SlurmctldPidFile=/var/run/slurmctld.pid SlurmdPidFile=/var/run/slurmd.pid SlurmdSpoolDir=/tmp/slurmd AccountingStorageType=accounting_storage/slurmdbd AccountingStorageHost=localhost JobAcctGatherType=jobacct_gather/linux TmpFS=/tmp InactiveLimit=1800 Epilog=/opt/slurm/epilog TrackWCKey=yes TaskProlog=/opt/slurm/prolog PrologSlurmctld=/usr/local/slurm/PrologSlurmctld JobRequeue=1 PlugStackConfig=/etc/slurm/plugstack.conf SlurmUser=slurm TaskPlugin=task/none HealthCheckInterval=3600 HealthCheckProgram=/home/slurm/healthcheck.sh DebugFlags=NO_CONF_HASH TopologyPlugin=topology/tree MaxStepCount=50000 SelectType=select/linear SlurmctldDebug=2 SchedulerParameters=max_switch_wait=30-00:00:00, >From topology.conf SwitchName=v1 Nodes=sandy1-vm[1-2] SwitchName=v2 Nodes=sandy2-vm[1-2] SwitchName=v3 Nodes=sandy3-vm[1-2] SwitchName=v4 Nodes=sandy4-vm[1-2] .. SwitchName=Virt Switches=v[1-8] -Mark
