I am able to make ssh , telnet (to the slurmd port 6818) and ping
lufer121 from the controller although the node seem to be down according
to the sview and slurmctld log.
Also I can send and run jobs on the node via srun or sbatch using
--nodelist before the the node is marked as down.
The "Can't find an address, check slurm.conf" message is a little bit
confusing but if there were a problem related with host name or IP or
slurmd.conf, performing the above actions would not be possible I thing..
I couldn't find a way to understand what the problem really is. As I
said, I have this problem on only a few nodes. All the nodes are
completely the same as harware and operating system, all the nodes are
set from the same system image of a golden client. Only difference is
their hostnames and IP address. One point: The problematic nodes are
added to the slurm much later after the cluster start serving.
On 02/19/2013 06:50 PM, Sefa Arslan wrote:
There is not much information on worker's log..
[2013-02-19T17:34:11] slurmd version 2.4.1 started
[2013-02-19T17:34:11] debug3: finished daemonize
[2013-02-19T17:34:11] debug3: Trying to load plugin
/usr/lib64/slurm/switch_none.so
[2013-02-19T17:34:11] switch NONE plugin loaded
[2013-02-19T17:34:11] debug3: Success.
[2013-02-19T17:34:11] debug3: successfully opened slurm listen port *:6818
[2013-02-19T17:34:11] slurmd started on Tue 19 Feb 2013 17:34:11 +0200
[2013-02-19T17:34:11] CPUs=8 Sockets=2 Cores=4 Threads=1 Memory=48253
TmpDisk=922082 Uptime=14516
[2013-02-19T17:35:19] debug3: in the service_connection
[2013-02-19T17:35:19] debug2: got this type of message 1001
[2013-02-19T17:35:19] debug3: CPUs=8 Sockets=2 Cores=4 Threads=1
Memory=48253 TmpDisk=922082 Uptime=14584
log ends here until i restart the slurmd again..
controller's log
[2013-02-19T17:34:31] debug2: _slurm_rpc_node_registration complete
for lufer121 usec=120
[2013-02-19T17:35:39] debug3: Tree sending to lufer121
[2013-02-19T17:35:39] debug2: _slurm_rpc_node_registration complete
for lufer121 usec=5160
[2013-02-19T17:35:42] debug2: node_did_resp lufer121
[2013-02-19T17:38:59] agent/is_node_resp: node:lufer121 rpc:1008 :
Can't find an address, check slurm.conf
[2013-02-19T17:39:00] error: Nodes lufer121 not responding
[2013-02-19T17:40:40] error: Nodes lufer121 not responding
[2013-02-19T17:42:20] error: Nodes lufer121 not responding
[2013-02-19T17:43:59] error: Nodes lufer121 not responding, setting DOWN
On 02/19/2013 04:50 PM, David Bigagli wrote:
Hi, have a look at the slurmd.log on the none responding hosts. Does
it show any relevant information?
/David
On Tue, Feb 19, 2013 at 2:46 PM, Sefa Arslan
<[email protected] <mailto:[email protected]>> wrote:
Hello,
After a few minutes I restart the SLURM service on the worker
node, the
controller says the worker node is not responding, and set as
DOWN. The
related log of the controller:
tail -f /var/log/slurm/slurmctld.log|grep lufer121
[2013-02-19T13:34:37] node lufer121 returned to service
[2013-02-19T13:34:37] debug2: _slurm_rpc_node_registration
complete for
lufer121 usec=231
[2013-02-19T13:36:24] debug3: Tree sending to lufer93 along with
lufer121,mercan[27-52,54-63,65-71,73-77,89]
[2013-02-19T13:36:24] agent/is_node_resp: node:lufer121 rpc:1008
: Can't
find an address, check slurm.conf
[2013-02-19T13:36:25] error: Nodes lufer121 not responding
[2013-02-19T13:38:26] error: Nodes lufer121 not responding
[2013-02-19T13:40:25] error: Nodes lufer121 not responding
[2013-02-19T13:42:13] error: Nodes lufer121 not responding,
setting DOWN
I controlled if I have some problem on network, but everything
seem to
be OK. Even if the slurmctld log says the worker is does not
responding,
I can ping/or make ssh or telnet to the 6818th port of to the
worker.
I have this problem on only a few worker. The others are working
quite
well.
Note: slurm version is 2.4.1-1, no firewall is working.. All
nodes has
dns records on the name server..
Do you have any suggestion?
