Hi Moe, I've been using the topology.conf file to separate nodes that aren't reachable via the same IB network for some time. If I try to allocate a node on two different networks I get
salloc: error: Job submit/allocate failed: Requested node configuration is not available This lead me to believe that the topology support was enabled. If it's disabled, is there an easy way for me to maintain the same SwitchType behavior but also be able to take advantage of the --switches option? Or use some other method (like constraints) to get 1 pair of VMs out of the 4 we have to be on the same physical host, instead of on different physical hosts. Thanks! -Mark On Wed, Sep 2, 2015 at 5:08 PM, Moe Jette <[email protected]> wrote: > > You completely disabled network topology support: > >> SwitchType=switch/none >> > > Quoting Mark Arnold <[email protected]>: > >> Hi All, >> >> I'm trying to get a job to run on 2 VMs that share the same physical host. >> I'm having some issues getting --switches to work. I chose --switches=1 >> and >> then using virtual switches to cluster the VMs together by physical host >> since it seemed the simplest way to do it. Perhaps there is a better way? >> >> Here is the a simple reproducer >> >> [root@X ~]# salloc -N2 -p Virt --switches=1@3-00:00:00 >> salloc: Pending job allocation 307282 >> salloc: job 307282 queued and waiting for resources >> salloc: job 307282 has been allocated resources >> salloc: Granted job allocation 307282 >> >> >> [root@X ~]# squeue --state=R -p Virt >> JOBID PARTITION NAME USER STATE TIME NODES WCKEY >> NODELIST(REASON) PRIORITY >> 307282 Virt bash root RUNNIN 0:03 2 * >> sandy3-vm2,sandy4-vm2 >> 0.99998585786670 >> >> >> As you can see I got two nodes on different physical hosts within ~60 >> seconds of making the request even though I set the maxtime to 3 days. >> >> I'm hoping I'm missing something, any help would be appreciated, thanks! >> >> We are using slurm 14.03.1-2 on all the nodes >> >> Here are my conf files >> >> From slurm.conf >> >> ClusterName=X >> ControlMachine=X >> NodeName=sandy1-vm[1-2] RealMemory=1800 CoresPerSocket=8 Sockets=1 >> ThreadsPerCore=1 State=UNKNOWN Weight=20 Feature=intel,ib,sandy,virt,v1 >> NodeName=sandy2-vm[1-2] RealMemory=1800 CoresPerSocket=8 Sockets=1 >> ThreadsPerCore=1 State=UNKNOWN Weight=20 Feature=intel,ib,sandy,virt,v2 >> NodeName=sandy3-vm[1-2] RealMemory=1800 CoresPerSocket=8 Sockets=1 >> ThreadsPerCore=1 State=UNKNOWN Weight=20 Feature=intel,ib,sandy,virt,v3 >> NodeName=sandy4-vm[1-2] RealMemory=1800 CoresPerSocket=8 Sockets=1 >> ThreadsPerCore=1 State=UNKNOWN Weight=20 Feature=intel,ib,sandy,virt,v4 >> >> PartitionName=Virt Nodes=sandy[1-4]-vm[1-2] MaxTime=3-00 >> >> SwitchType=switch/none >> MpiDefault=none >> SlurmctldPort=6817 >> SlurmdPort=6818 >> AuthType=auth/munge >> SchedulerType=sched/backfill >> FirstJobId=1 >> FastSchedule=1 >> SlurmctldLogFile=/var/log/slurm/slurmctld.log >> SlurmdLogFile=/var/log/slurm/slurmd.log >> SlurmctldPidFile=/var/run/slurmctld.pid >> SlurmdPidFile=/var/run/slurmd.pid >> SlurmdSpoolDir=/tmp/slurmd >> AccountingStorageType=accounting_storage/slurmdbd >> AccountingStorageHost=localhost >> JobAcctGatherType=jobacct_gather/linux >> TmpFS=/tmp >> InactiveLimit=1800 >> Epilog=/opt/slurm/epilog >> TrackWCKey=yes >> TaskProlog=/opt/slurm/prolog >> PrologSlurmctld=/usr/local/slurm/PrologSlurmctld >> JobRequeue=1 >> PlugStackConfig=/etc/slurm/plugstack.conf >> SlurmUser=slurm >> TaskPlugin=task/none >> HealthCheckInterval=3600 >> HealthCheckProgram=/home/slurm/healthcheck.sh >> DebugFlags=NO_CONF_HASH >> TopologyPlugin=topology/tree >> MaxStepCount=50000 >> SelectType=select/linear >> SlurmctldDebug=2 >> SchedulerParameters=max_switch_wait=30-00:00:00, >> >> From topology.conf >> >> SwitchName=v1 Nodes=sandy1-vm[1-2] >> SwitchName=v2 Nodes=sandy2-vm[1-2] >> SwitchName=v3 Nodes=sandy3-vm[1-2] >> SwitchName=v4 Nodes=sandy4-vm[1-2] >> .. >> SwitchName=Virt Switches=v[1-8] >> >> -Mark >> > > > -- > Morris "Moe" Jette > CTO, SchedMD LLC > Commercial Slurm Development and Support > =============================================================== > Slurm User Group Meeting, 15-16 September 2015, Washington D.C. > http://slurm.schedmd.com/slurm_ug_agenda.html >
