Still, for maintenance of the primary controller, you should call "scontrol takeover" because Slurm commands, especially "srun", do not work during these 60 seconds otherwise.
2017-01-31 19:12 GMT+01:00 Andrus, Brian Contractor <[email protected]>: > Aha! That was just far too large for me. Set it down to 60 seconds and things > seem happier (along with the users). > > Thanks! > > Brian > > -----Original Message----- > From: TO_Webmaster [mailto:[email protected]] > Sent: Tuesday, January 31, 2017 12:26 AM > To: slurm-dev <[email protected]> > Subject: [slurm-dev] Re: Backup controller not responding to requests > > > What is the output of > > scontrol show config | grep SlurmctldTimeout > > ? > > 2017-01-31 6:57 GMT+01:00 Andrus, Brian Contractor <[email protected]>: >> Yes, if I do scontrol takeover, it successfully goes to the backup. >> >> >> Brian Andrus >> ITACS/Research Computing >> Naval Postgraduate School >> Monterey, California >> voice: 831-656-6238 >> >> >> >> -----Original Message----- >> From: TO_Webmaster [mailto:[email protected]] >> Sent: Monday, January 30, 2017 11:02 AM >> To: slurm-dev <[email protected]> >> Subject: [slurm-dev] Re: Backup controller not responding to requests >> >> >> Does it work if you use "scontrol takeover" to shut down the primary >> controller and switch immediately to the backup controller? >> >> 2017-01-30 19:41 GMT+01:00 Andrus, Brian Contractor <[email protected]>: >>> Paddy, >>> >>> I will enable those and try. The backup controller does have access to the >>> directory and it is the same version as the master. >>> >>> Not seeing much more in the logs.. >>> The backup controller ends with a ping of the master and then just >>> sits. I restart the master and the backup starts saying "Invalid RPC". When >>> the master comes back up, it says it is ignoring the RPC: REQUEST_CONTROL >>> So, for some reason, it seems the backup will not promote itself... >>> >>> ---------------------- >>> [2017-01-30T10:30:21.321] debug3: Success. >>> [2017-01-30T10:30:21.322] trigger pulled for SLURMCTLD event 16384 >>> successful [2017-01-30T10:30:27.323] debug3: pinging slurmctld at >>> 10.1.1.127 [2017-01-30T10:31:55.814] error: Invalid RPC received 2009 >>> while in standby mode [2017-01-30T10:32:04.839] debug3: Ignoring RPC: >>> REQUEST_CONTROL [2017-01-30T10:32:06.133] error: Invalid RPC received >>> 2009 while in standby mode [2017-01-30T10:32:07.338] debug3: pinging >>> slurmctld at 10.1.1.127 [2017-01-30T10:32:07.339] debug2: >>> slurm_connect failed: Connection refused [2017-01-30T10:32:07.339] >>> debug2: Error connecting slurm stream socket at 10.1.1.127:6817: >>> Connection refused [2017-01-30T10:32:07.339] error: >>> _ping_controller/slurm_send_node_msg error: Connection refused >>> [2017-01-30T10:33:47.351] debug3: pinging slurmctld at 10.1.1.127 >>> [2017-01-30T10:35:27.366] debug3: pinging slurmctld at 10.1.1.127 >>> [2017-01-30T10:35:33.758] debug3: Ignoring RPC: REQUEST_CONTROL >>> --------------------- >>> >>> >>> Brian Andrus >>> ITACS/Research Computing >>> Naval Postgraduate School >>> Monterey, California >>> voice: 831-656-6238 >>> >>> >>> >>> -----Original Message----- >>> From: Paddy Doyle [mailto:[email protected]] >>> Sent: Monday, January 30, 2017 9:48 AM >>> To: slurm-dev <[email protected]> >>> Subject: [slurm-dev] Re: Backup controller not responding to requests >>> >>> >>> Hi Brian, >>> >>> You could turn up the SlurmctldDebug and SlurmdDebug values in slurm.conf >>> to get it to be more verbose. >>> >>> As a wild guess, perhaps your backup control doesn't have access to the >>> StateSaveLocation directory? >>> >>> Or another possibility could be it's running a different version of slurm. >>> >>> Paddy >>> >>> On Mon, Jan 30, 2017 at 08:21:59AM -0800, Andrus, Brian Contractor wrote: >>> >>>> All, >>>> >>>> I have configured a backup slurmctld system and it appears to work at >>>> first, but not in practice. >>>> In particular, when I start it, it says it is running in background mode: >>>> [2017-01-25T14:23:37.648] slurmctld version 16.05.6 started on >>>> cluster hamming [2017-01-25T14:23:37.650] slurmctld running in >>>> background mode >>>> >>>> But if I stop the primary daemon, it does not take over. I keep getting >>>> Invalid RPC errors (random snippets): >>>> [2017-01-25T15:50:37.664] error: Invalid RPC received 2007 while in >>>> standby mode [2017-01-25T15:53:50.495] error: Invalid RPC received >>>> 5018 while in standby mode [2017-01-25T15:59:36.847] error: Invalid >>>> RPC received 2007 while in standby mode [2017-01-25T15:59:37.499] >>>> error: Invalid RPC received 2007 while in standby mode >>>> [2017-01-25T15:59:38.923] error: Invalid RPC received 2007 while in >>>> standby mode [2017-01-25T15:59:38.985] error: Invalid RPC received >>>> 2007 while in standby mode [2017-01-25T15:59:39.246] error: Invalid >>>> RPC received 2007 while in standby mode [2017-01-25T15:59:39.293] >>>> error: Invalid RPC received 2009 while in standby mode >>>> [2017-01-25T15:59:39.522] error: Invalid RPC received 5018 while in >>>> standby mode [2017-01-25T15:59:43.839] error: Invalid RPC received >>>> 2009 while in standby mode [2017-01-25T15:59:43.930] error: Invalid >>>> RPC received 2009 while in standby mode [2017-01-25T16:19:47.215] >>>> error: Invalid RPC received 6012 while in standby mode >>>> [2017-01-25T16:19:48.238] error: Invalid RPC received 6012 while in >>>> standby mode >>>> >>>> And on any client running 'sinfo' for instance, it merely hangs. >>>> The interfaces for both slurmctld controllers are in the 'trusted' >>>> firewall group and there is no filtering between them. >>>> Is there something I am missing to make the backup controller 'kick in' >>>> and start responding to requests? >>>> >>>> >>>> Brian Andrus >>>> ITACS/Research Computing >>>> Naval Postgraduate School >>>> Monterey, California >>>> voice: 831-656-6238 >>>> >>> >>> -- >>> Paddy Doyle >>> Trinity Centre for High Performance Computing, Lloyd Building, Trinity >>> College Dublin, Dublin 2, Ireland. >>> Phone: +353-1-896-3725 >>> http://www.tchpc.tcd.ie/
