There is not much information on worker's log..
[2013-02-19T17:34:11] slurmd version 2.4.1 started
[2013-02-19T17:34:11] debug3: finished daemonize
[2013-02-19T17:34:11] debug3: Trying to load plugin
/usr/lib64/slurm/switch_none.so
[2013-02-19T17:34:11] switch NONE plugin loaded
[2013-02-19T17:34:11] debug3: Success.
[2013-02-19T17:34:11] debug3: successfully opened slurm listen port *:6818
[2013-02-19T17:34:11] slurmd started on Tue 19 Feb 2013 17:34:11 +0200
[2013-02-19T17:34:11] CPUs=8 Sockets=2 Cores=4 Threads=1 Memory=48253
TmpDisk=922082 Uptime=14516
[2013-02-19T17:35:19] debug3: in the service_connection
[2013-02-19T17:35:19] debug2: got this type of message 1001
[2013-02-19T17:35:19] debug3: CPUs=8 Sockets=2 Cores=4 Threads=1
Memory=48253 TmpDisk=922082 Uptime=14584
log ends here until i restart the slurmd again..
controller's log
[2013-02-19T17:34:31] debug2: _slurm_rpc_node_registration complete for
lufer121 usec=120
[2013-02-19T17:35:39] debug3: Tree sending to lufer121
[2013-02-19T17:35:39] debug2: _slurm_rpc_node_registration complete for
lufer121 usec=5160
[2013-02-19T17:35:42] debug2: node_did_resp lufer121
[2013-02-19T17:38:59] agent/is_node_resp: node:lufer121 rpc:1008 : Can't
find an address, check slurm.conf
[2013-02-19T17:39:00] error: Nodes lufer121 not responding
[2013-02-19T17:40:40] error: Nodes lufer121 not responding
[2013-02-19T17:42:20] error: Nodes lufer121 not responding
[2013-02-19T17:43:59] error: Nodes lufer121 not responding, setting DOWN
On 02/19/2013 04:50 PM, David Bigagli wrote:
Hi, have a look at the slurmd.log on the none responding hosts. Does
it show any relevant information?
/David
On Tue, Feb 19, 2013 at 2:46 PM, Sefa Arslan
<[email protected] <mailto:[email protected]>> wrote:
Hello,
After a few minutes I restart the SLURM service on the worker
node, the
controller says the worker node is not responding, and set as
DOWN. The
related log of the controller:
tail -f /var/log/slurm/slurmctld.log|grep lufer121
[2013-02-19T13:34:37] node lufer121 returned to service
[2013-02-19T13:34:37] debug2: _slurm_rpc_node_registration
complete for
lufer121 usec=231
[2013-02-19T13:36:24] debug3: Tree sending to lufer93 along with
lufer121,mercan[27-52,54-63,65-71,73-77,89]
[2013-02-19T13:36:24] agent/is_node_resp: node:lufer121 rpc:1008 :
Can't
find an address, check slurm.conf
[2013-02-19T13:36:25] error: Nodes lufer121 not responding
[2013-02-19T13:38:26] error: Nodes lufer121 not responding
[2013-02-19T13:40:25] error: Nodes lufer121 not responding
[2013-02-19T13:42:13] error: Nodes lufer121 not responding,
setting DOWN
I controlled if I have some problem on network, but everything seem to
be OK. Even if the slurmctld log says the worker is does not
responding,
I can ping/or make ssh or telnet to the 6818th port of to the
worker.
I have this problem on only a few worker. The others are working quite
well.
Note: slurm version is 2.4.1-1, no firewall is working.. All nodes has
dns records on the name server..
Do you have any suggestion?
