I already replied but it seemed to have gotten lost. Also, the slurm-dev list is probably a better place to send these emails vs slurm-announce. I didn't realise the first time around that the mail was sent to slurm-announce :)
Hi Herbert, Everything appears to be working as expected. I think there may be some confusion about the purpose of slurmd/slurmctld. Slurmd is responsible for execution of jobs on a given node. Slurmctld is responsible for scheduling jobs across the entire cluster and coordinating with the slurmds. Implicit high availability of slurmd is provided by the other slurmds running in the cluster. A given node's slurmd isn't highly available because it doesn't make sense to be. Your jobs don't run when you stop slurmd on the master because they explicitly request node1 for execution. Try not requesting a specific node name in the job and things should work as you expect. Best, Aaron Sent from my iPhone On Mar 16, 2012, at 11:41 AM, Herbert Jung <[email protected]> wrote: > Hi All, > > We have 2 servers being setup/configured for using slurm (1 master and 1 > backup controller) but for some reason we don’t get the failover to work and > we cant figure out why. > When I stop the slurmctld daemon on the master controller, the backup slurm > server takes over and all the jobs which are in the queue are being triggert > via the backup controller, but when I shutdown the slurmd daemon on the > master controller the backup controller wont take over > > > installed slurm Version 2.3.3 > > daemon running on master controller > slurm 15096 1 0 Mar15 ? 00:00:02 /usr/local/sbin/slurmctld > root 15105 1 0 Mar15 ? 00:00:00 /usr/local/sbin/slurmd > munge 2286 1 0 Mar08 ? 00:00:26 /usr/sbin/munged > > daemon running on backup controller > slurm 313 1 0 Mar14 ? 00:00:00 /usr/local/sbin/slurmdbd > slurm 13192 1 0 Mar15 ? 00:00:01 /usr/local/sbin/slurmctld > root 13202 1 0 Mar15 ? 00:00:00 /usr/local/sbin/slurmd > munge 19842 1 0 Mar08 ? 00:00:26 /usr/sbin/munged > mysqld > > the slurm.state folder is mounted on both servers, user slurm has r-w > permissions > > example > I just triggert a few jobs on the master controller > sbatch --begin=now+20 -n4 -w "host[1,2]" -o /tmp/my.stdout /tmp/my.script > sbatch --begin=now+40 -n4 -w "host[1,2]" -o /tmp/my.stdout /tmp/my.script > . > > > squeue says > JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) > 281 debug my.scrip root PD 0:00 2 (BeginTime) > 282 debug my.scrip root PD 0:00 2 (BeginTime) > 283 debug my.scrip root PD 0:00 2 (BeginTime) > 284 debug my.scrip root PD 0:00 2 (BeginTime) > 285 debug my.scrip root PD 0:00 2 (BeginTime) > 286 debug my.scrip root PD 0:00 2 (BeginTime) > > everything looks good so far > > . > . > current squeue status > > JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) > 283 debug my.scrip root PD 0:00 2 (BeginTime) > 284 debug my.scrip root PD 0:00 2 (BeginTime) > 285 debug my.scrip root PD 0:00 2 (BeginTime) > 286 debug my.scrip root PD 0:00 2 (BeginTime) > > next I am going to shutdown slurmd/slurmctld on the master controller > > squeue status short after > > JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) > 283 debug my.scrip root PD 0:00 2 (ReqNodeNotAvail) > 284 debug my.scrip root PD 0:00 2 (ReqNodeNotAvail) > 285 debug my.scrip root PD 0:00 2 (ReqNodeNotAvail) > 286 debug my.scrip root PD 0:00 2 (ReqNodeNotAvail) > > > I have reconfigured the srm.conf files several times, but nothing seems to > work > > > any help would be very much appreciated > > I have attached the slurmctld and slurmd logs from the master and backup > controller, as also a snapshot of our current config file and the output of > scontrol-show-config > > > Thanks a lot > Herbert > > > > P Please consider the environment before printing this email… > > > > > ******************************************************* > This message (including any files transmitted with it) may contain > confidential and/or proprietary information, is the property of Interactive > Data Corporation and/or its subsidiaries, and is directed only to the > addressee(s). If you are not the designated recipient or have reason to > believe you received this message in error, please delete this message from > your system and notify the sender immediately. An unintended recipient's > disclosure, copying, distribution, or use of this message or any attachments > is prohibited and may be unlawful. > ******************************************************* > <slurmctld.log-backup-controller> > <slurmctld.log-master-controller> > <slurmd.log-backup-controller> > <slurmd.log-master-controller> > <scontrol-show-config> > <slurm-conf-master-backup>
