Hi, have a look at the slurmd.log on the none responding hosts. Does it show any relevant information?
/David On Tue, Feb 19, 2013 at 2:46 PM, Sefa Arslan <[email protected]>wrote: > > Hello, > > After a few minutes I restart the SLURM service on the worker node, the > controller says the worker node is not responding, and set as DOWN. The > related log of the controller: > > tail -f /var/log/slurm/slurmctld.log|grep lufer121 > > [2013-02-19T13:34:37] node lufer121 returned to service > [2013-02-19T13:34:37] debug2: _slurm_rpc_node_registration complete for > lufer121 usec=231 > [2013-02-19T13:36:24] debug3: Tree sending to lufer93 along with > lufer121,mercan[27-52,54-63,65-71,73-77,89] > [2013-02-19T13:36:24] agent/is_node_resp: node:lufer121 rpc:1008 : Can't > find an address, check slurm.conf > [2013-02-19T13:36:25] error: Nodes lufer121 not responding > [2013-02-19T13:38:26] error: Nodes lufer121 not responding > [2013-02-19T13:40:25] error: Nodes lufer121 not responding > [2013-02-19T13:42:13] error: Nodes lufer121 not responding, setting DOWN > > > I controlled if I have some problem on network, but everything seem to > be OK. Even if the slurmctld log says the worker is does not responding, > I can ping/or make ssh or telnet to the 6818th port of to the worker. > > I have this problem on only a few worker. The others are working quite > well. > > Note: slurm version is 2.4.1-1, no firewall is working.. All nodes has > dns records on the name server.. > > > Do you have any suggestion? >
