->[2013-02-19T17:38:59] agent/is_node_resp: node:lufer121 rpc:1008 : Can't find an address, check slurm.conf
This is the error. The controller either does not know the machine, there are some problems with its address or communication (name resolution?). Is lufer121 talking to the controller using the correct IP? /David On Tue, Feb 19, 2013 at 5:26 PM, Sefa Arslan <[email protected]>wrote: > ** > There is not much information on worker's log.. > > [2013-02-19T17:34:11] slurmd version 2.4.1 started > [2013-02-19T17:34:11] debug3: finished daemonize > [2013-02-19T17:34:11] debug3: Trying to load plugin > /usr/lib64/slurm/switch_none.so > [2013-02-19T17:34:11] switch NONE plugin loaded > [2013-02-19T17:34:11] debug3: Success. > [2013-02-19T17:34:11] debug3: successfully opened slurm listen port *:6818 > [2013-02-19T17:34:11] slurmd started on Tue 19 Feb 2013 17:34:11 +0200 > [2013-02-19T17:34:11] CPUs=8 Sockets=2 Cores=4 Threads=1 Memory=48253 > TmpDisk=922082 Uptime=14516 > [2013-02-19T17:35:19] debug3: in the service_connection > [2013-02-19T17:35:19] debug2: got this type of message 1001 > [2013-02-19T17:35:19] debug3: CPUs=8 Sockets=2 Cores=4 Threads=1 > Memory=48253 TmpDisk=922082 Uptime=14584 > > log ends here until i restart the slurmd again.. > > > controller's log > [2013-02-19T17:34:31] debug2: _slurm_rpc_node_registration complete for > lufer121 usec=120 > [2013-02-19T17:35:39] debug3: Tree sending to lufer121 > [2013-02-19T17:35:39] debug2: _slurm_rpc_node_registration complete for > lufer121 usec=5160 > [2013-02-19T17:35:42] debug2: node_did_resp lufer121 > [2013-02-19T17:38:59] agent/is_node_resp: node:lufer121 rpc:1008 : Can't > find an address, check slurm.conf > [2013-02-19T17:39:00] error: Nodes lufer121 not responding > [2013-02-19T17:40:40] error: Nodes lufer121 not responding > [2013-02-19T17:42:20] error: Nodes lufer121 not responding > [2013-02-19T17:43:59] error: Nodes lufer121 not responding, setting DOWN > > > > > > On 02/19/2013 04:50 PM, David Bigagli wrote: > > Hi, have a look at the slurmd.log on the none responding hosts. Does it > show any relevant information? > > /David > > > On Tue, Feb 19, 2013 at 2:46 PM, Sefa Arslan > <[email protected]>wrote: > >> >> Hello, >> >> After a few minutes I restart the SLURM service on the worker node, the >> controller says the worker node is not responding, and set as DOWN. The >> related log of the controller: >> >> tail -f /var/log/slurm/slurmctld.log|grep lufer121 >> >> [2013-02-19T13:34:37] node lufer121 returned to service >> [2013-02-19T13:34:37] debug2: _slurm_rpc_node_registration complete for >> lufer121 usec=231 >> [2013-02-19T13:36:24] debug3: Tree sending to lufer93 along with >> lufer121,mercan[27-52,54-63,65-71,73-77,89] >> [2013-02-19T13:36:24] agent/is_node_resp: node:lufer121 rpc:1008 : Can't >> find an address, check slurm.conf >> [2013-02-19T13:36:25] error: Nodes lufer121 not responding >> [2013-02-19T13:38:26] error: Nodes lufer121 not responding >> [2013-02-19T13:40:25] error: Nodes lufer121 not responding >> [2013-02-19T13:42:13] error: Nodes lufer121 not responding, setting DOWN >> >> >> I controlled if I have some problem on network, but everything seem to >> be OK. Even if the slurmctld log says the worker is does not responding, >> I can ping/or make ssh or telnet to the 6818th port of to the worker. >> >> I have this problem on only a few worker. The others are working quite >> well. >> >> Note: slurm version is 2.4.1-1, no firewall is working.. All nodes has >> dns records on the name server.. >> >> >> Do you have any suggestion? >> > >
