You might want to up the ResumeTimeout in your slurm.conf. By default it is 60 seconds, and since your nodes appear to take around 3 minutes (180 seconds) to boot this is where the problem appears to arise from. I would up it to 300 if I were you. Of course if this isn't the case and you already have it set to a higher number let us know.
Danny On 12/13/2012 07:39 AM, Ramon Bastiaans wrote: > Hi, > > I am experimenting with SLURM to evaluate if it is something we could use on > our cluster. I am running SLURM 2.5.0 > > I am experiencing some weird behavior with the power saving feature. > > When all nodes are power saved (switched off) and I restart slurmctld, it > powers up / resumes all nodes and then complains that the nodes unexpectedly > rebooted and switches all nodes down. > > This seems weird to me. I tried starting slurmctld with -R to fully recover > node state, but it still does this. > > Am I doing something wrong? > > Any help would be appreciated. > > Kind regards, > - Ramon. > > > === > > # sinfo > PARTITION AVAIL TIMELIMIT NODES STATE NODELIST > batch* up 2:00:00 14 idle~ gb-r7n[18,20-32] > > # scontrol show nodes | grep State > State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 > State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 > State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 > State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 > State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 > State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 > State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 > State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 > State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 > State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 > State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 > State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 > State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 > State=IDLE+POWER ThreadsPerCore=1 TmpDisk=0 Weight=1 > > # tail -1 /var/log/slurm/slurmctld.log > [2012-12-13T16:11:54+01:00] Power save mode: 14 nodes > > # tail -f slurmctld.log & > # tail -f power_save.log & > > > # /etc/init.d/slurm-llnl restart > Stopping slurm central management daemon: slurmctld > [2012-12-13T16:22:48+01:00] Terminate signal (SIGINT or SIGTERM) received > . > [2012-12-13T16:22:49+01:00] Saving all slurm state > slurmctld is stopped > Starting slurm central management daemon: slurmctld > [2012-12-13T16:22:49+01:00] slurmctld version 2.5.0 started on cluster cluster > . > [2012-12-13T16:22:49+01:00] Recovered state of 14 nodes > [2012-12-13T16:22:49+01:00] Recovered state of 1 partitions > [2012-12-13T16:22:49+01:00] Recovered information about 0 jobs > [2012-12-13T16:22:49+01:00] cons_res: select_p_node_init > [2012-12-13T16:22:49+01:00] cons_res: preparing for 1 partitions > [2012-12-13T16:22:49+01:00] Recovered state of 0 reservations > [2012-12-13T16:22:49+01:00] read_slurm_conf: backup_controller not specified. > [2012-12-13T16:22:49+01:00] cons_res: select_p_reconfigure > [2012-12-13T16:22:49+01:00] cons_res: select_p_node_init > [2012-12-13T16:22:49+01:00] cons_res: preparing for 1 partitions > [2012-12-13T16:22:49+01:00] Running as primary controller > [2012-12-13T16:22:50+01:00] Power save mode: 14 nodes > Thu Dec 13 16:22:50 CET 2012 Resume invoked > /var/tmp/wax/power_save/waxresume.sh gb-r7n[18,20-32] > Thu Dec 13 16:22:50 CET 2012 host gb-r7n18 drac dr-r7n18 > Thu Dec 13 16:22:52 CET 2012 host gb-r7n20 drac dr-r7n20 > Thu Dec 13 16:22:54 CET 2012 host gb-r7n21 drac dr-r7n21 > Thu Dec 13 16:22:57 CET 2012 host gb-r7n22 drac dr-r7n22 > Thu Dec 13 16:22:59 CET 2012 host gb-r7n23 drac dr-r7n23 > Thu Dec 13 16:23:01 CET 2012 host gb-r7n24 drac dr-r7n24 > Thu Dec 13 16:23:04 CET 2012 host gb-r7n25 drac dr-r7n25 > Thu Dec 13 16:23:06 CET 2012 host gb-r7n26 drac dr-r7n26 > Thu Dec 13 16:23:08 CET 2012 host gb-r7n27 drac dr-r7n27 > Thu Dec 13 16:23:11 CET 2012 host gb-r7n28 drac dr-r7n28 > Thu Dec 13 16:23:13 CET 2012 host gb-r7n29 drac dr-r7n29 > Thu Dec 13 16:23:15 CET 2012 host gb-r7n30 drac dr-r7n30 > Thu Dec 13 16:23:18 CET 2012 host gb-r7n31 drac dr-r7n31 > Thu Dec 13 16:23:20 CET 2012 host gb-r7n32 drac dr-r7n32 > [2012-12-13T16:25:14+01:00] Node gb-r7n21 unexpectedly rebooted > [2012-12-13T16:25:26+01:00] Node gb-r7n22 unexpectedly rebooted > [2012-12-13T16:25:33+01:00] Node gb-r7n29 unexpectedly rebooted > [2012-12-13T16:25:37+01:00] Node gb-r7n27 unexpectedly rebooted > [2012-12-13T16:25:41+01:00] Node gb-r7n28 unexpectedly rebooted > [2012-12-13T16:25:42+01:00] Node gb-r7n31 unexpectedly rebooted > [2012-12-13T16:25:46+01:00] Node gb-r7n25 unexpectedly rebooted > [2012-12-13T16:25:48+01:00] Node gb-r7n20 unexpectedly rebooted > [2012-12-13T16:25:56+01:00] Node gb-r7n32 unexpectedly rebooted > [2012-12-13T16:26:16+01:00] Node gb-r7n23 unexpectedly rebooted > [2012-12-13T16:26:24+01:00] Node gb-r7n18 unexpectedly rebooted > [2012-12-13T16:26:29+01:00] Node gb-r7n24 unexpectedly rebooted > [2012-12-13T16:26:40+01:00] Node gb-r7n26 unexpectedly rebooted > [2012-12-13T16:26:42+01:00] Node gb-r7n30 unexpectedly rebooted > > # sinfo > PARTITION AVAIL TIMELIMIT NODES STATE NODELIST > batch* up 2:00:00 14 down gb-r7n[18,20-32] > > > > R. Bastiaans, B.ICT > * Senior Systems Programmer > * Operations, Support and Development > > SARA > Science Park 140 PO Box 94613 > 1098 XG Amsterdam NL 1090 GP Amsterdam NL > P.+31 (0)20 592 3000 F.+31 (0)20 668 3167 > > >
