Hi, Thanks for your response.
I have the ResumeTimeout set to 6 minutes, I can try increasing it some more. My resume script powers up the node and then exits in advance while the node is still booting. Is perhaps the resume script supposed to wait until the node is powered up before exiting? Here is a snippet from the power saving settings in slurm.conf: SuspendTime=60 SuspendProgram=/var/tmp/wax/power_save/waxsuspend.sh SuspendTimeout=30 ResumeProgram=/var/tmp/wax/power_save/waxresume.sh ResumeTimeout=360 Kind regards, - Ramon. And the rest: ControlMachine=gb-r7n17 AuthType=auth/munge CacheGroups=0 CryptoType=crypto/munge EnforcePartLimits=YES JobCheckpointDir=/var/lib/slurm-llnl/checkpoint JobRequeue=1 MailProg=/usr/bin/mail MpiDefault=none ProctrackType=proctrack/pgid ReturnToService=1 SlurmctldPidFile=/var/run/slurm-llnl/slurmctld.pid SlurmctldPort=6817 SlurmdPidFile=/var/run/slurm-llnl/slurmd.pid SlurmdPort=6818 SlurmdSpoolDir=/var/lib/slurm-llnl/slurmd StateSaveLocation=/var/lib/slurm-llnl/slurmctld SwitchType=switch/none TaskPlugin=task/affinity TaskPluginParam=Sched TmpFs=/scratch InactiveLimit=0 KillWait=30 MinJobAge=300 SlurmctldTimeout=300 SlurmdTimeout=360 Waittime=0 FastSchedule=1 SchedulerType=sched/backfill SchedulerPort=7321 SelectType=select/cons_res SelectTypeParameters=CR_Core AccountingStorageLoc=/var/log/slurm/accounting.log AccountingStorageType=accounting_storage/none ClusterName=cluster DebugFlags=NO_CONF_HASH JobCompLoc=/var/log/slurm/jobcomp.log JobCompType=jobcomp/filetxt JobAcctGatherFrequency=30 JobAcctGatherType=jobacct_gather/none SlurmctldDebug=3 SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdDebug=3 SlurmdLogFile=/var/log/slurm/slurmd.log NodeName=gb-r7n[18,20-32] Procs=8 PartitionName=batch Nodes=gb-r7n[18,20-32] Default=YES MaxTime=120 State=UP R. Bastiaans, B.ICT * Senior Systems Programmer * Operations, Support and Development SARA Science Park 140 PO Box 94613 1098 XG Amsterdam NL 1090 GP Amsterdam NL P.+31 (0)20 592 3000 F.+31 (0)20 668 3167 On 13 dec. 2012, at 22:59, Danny Auble <[email protected]> wrote: > > You might want to up the ResumeTimeout in your slurm.conf. By default > it is 60 seconds, and since your nodes appear to take around 3 minutes > (180 seconds) to boot this is where the problem appears to arise from. > I would up it to 300 if I were you. Of course if this isn't the case > and you already have it set to a higher number let us know. > > Danny > > On 12/13/2012 07:39 AM, Ramon Bastiaans wrote: >> Hi, >> >> I am experimenting with SLURM to evaluate if it is something we could use on >> our cluster. I am running SLURM 2.5.0 >> >> I am experiencing some weird behavior with the power saving feature. >> >> When all nodes are power saved (switched off) and I restart slurmctld, it >> powers up / resumes all nodes and then complains that the nodes unexpectedly >> rebooted and switches all nodes down. >> >> This seems weird to me. I tried starting slurmctld with -R to fully recover >> node state, but it still does this. >> >> Am I doing something wrong? >> >> Any help would be appreciated. >> >> Kind regards, >> - Ramon. >> >> >> === >> >> # sinfo >> PARTITION AVAIL TIMELIMIT NODES STATE NODELIST >> batch* up 2:00:00 14 idle~ gb-r7n[18,20-32] >> >> # scontrol show nodes | grep State >> State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 >> State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 >> State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 >> State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 >> State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 >> State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 >> State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 >> State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 >> State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 >> State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 >> State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 >> State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 >> State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 >> State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 >> >> # tail -1 /var/log/slurm/slurmctld.log >> [2012-12-13T16:11:54+01:00] Power save mode: 14 nodes >> >> # tail -f slurmctld.log & >> # tail -f power_save.log & >> >> >> # /etc/init.d/slurm-llnl restart >> Stopping slurm central management daemon: slurmctld >> [2012-12-13T16:22:48+01:00] Terminate signal (SIGINT or SIGTERM) received >> . >> [2012-12-13T16:22:49+01:00] Saving all slurm state >> slurmctld is stopped >> Starting slurm central management daemon: slurmctld >> [2012-12-13T16:22:49+01:00] slurmctld version 2.5.0 started on cluster >> cluster >> . >> [2012-12-13T16:22:49+01:00] Recovered state of 14 nodes >> [2012-12-13T16:22:49+01:00] Recovered state of 1 partitions >> [2012-12-13T16:22:49+01:00] Recovered information about 0 jobs >> [2012-12-13T16:22:49+01:00] cons_res: select_p_node_init >> [2012-12-13T16:22:49+01:00] cons_res: preparing for 1 partitions >> [2012-12-13T16:22:49+01:00] Recovered state of 0 reservations >> [2012-12-13T16:22:49+01:00] read_slurm_conf: backup_controller not specified. >> [2012-12-13T16:22:49+01:00] cons_res: select_p_reconfigure >> [2012-12-13T16:22:49+01:00] cons_res: select_p_node_init >> [2012-12-13T16:22:49+01:00] cons_res: preparing for 1 partitions >> [2012-12-13T16:22:49+01:00] Running as primary controller >> [2012-12-13T16:22:50+01:00] Power save mode: 14 nodes >> Thu Dec 13 16:22:50 CET 2012 Resume invoked >> /var/tmp/wax/power_save/waxresume.sh gb-r7n[18,20-32] >> Thu Dec 13 16:22:50 CET 2012 host gb-r7n18 drac dr-r7n18 >> Thu Dec 13 16:22:52 CET 2012 host gb-r7n20 drac dr-r7n20 >> Thu Dec 13 16:22:54 CET 2012 host gb-r7n21 drac dr-r7n21 >> Thu Dec 13 16:22:57 CET 2012 host gb-r7n22 drac dr-r7n22 >> Thu Dec 13 16:22:59 CET 2012 host gb-r7n23 drac dr-r7n23 >> Thu Dec 13 16:23:01 CET 2012 host gb-r7n24 drac dr-r7n24 >> Thu Dec 13 16:23:04 CET 2012 host gb-r7n25 drac dr-r7n25 >> Thu Dec 13 16:23:06 CET 2012 host gb-r7n26 drac dr-r7n26 >> Thu Dec 13 16:23:08 CET 2012 host gb-r7n27 drac dr-r7n27 >> Thu Dec 13 16:23:11 CET 2012 host gb-r7n28 drac dr-r7n28 >> Thu Dec 13 16:23:13 CET 2012 host gb-r7n29 drac dr-r7n29 >> Thu Dec 13 16:23:15 CET 2012 host gb-r7n30 drac dr-r7n30 >> Thu Dec 13 16:23:18 CET 2012 host gb-r7n31 drac dr-r7n31 >> Thu Dec 13 16:23:20 CET 2012 host gb-r7n32 drac dr-r7n32 >> [2012-12-13T16:25:14+01:00] Node gb-r7n21 unexpectedly rebooted >> [2012-12-13T16:25:26+01:00] Node gb-r7n22 unexpectedly rebooted >> [2012-12-13T16:25:33+01:00] Node gb-r7n29 unexpectedly rebooted >> [2012-12-13T16:25:37+01:00] Node gb-r7n27 unexpectedly rebooted >> [2012-12-13T16:25:41+01:00] Node gb-r7n28 unexpectedly rebooted >> [2012-12-13T16:25:42+01:00] Node gb-r7n31 unexpectedly rebooted >> [2012-12-13T16:25:46+01:00] Node gb-r7n25 unexpectedly rebooted >> [2012-12-13T16:25:48+01:00] Node gb-r7n20 unexpectedly rebooted >> [2012-12-13T16:25:56+01:00] Node gb-r7n32 unexpectedly rebooted >> [2012-12-13T16:26:16+01:00] Node gb-r7n23 unexpectedly rebooted >> [2012-12-13T16:26:24+01:00] Node gb-r7n18 unexpectedly rebooted >> [2012-12-13T16:26:29+01:00] Node gb-r7n24 unexpectedly rebooted >> [2012-12-13T16:26:40+01:00] Node gb-r7n26 unexpectedly rebooted >> [2012-12-13T16:26:42+01:00] Node gb-r7n30 unexpectedly rebooted >> >> # sinfo >> PARTITION AVAIL TIMELIMIT NODES STATE NODELIST >> batch* up 2:00:00 14 down gb-r7n[18,20-32] >> >> >> >> R. Bastiaans, B.ICT >> * Senior Systems Programmer >> * Operations, Support and Development >> >> SARA >> Science Park 140 PO Box 94613 >> 1098 XG Amsterdam NL 1090 GP Amsterdam NL >> P.+31 (0)20 592 3000 F.+31 (0)20 668 3167 >> >> >>
smime.p7s
Description: S/MIME cryptographic signature
