We have to make sure all daemons run on the same configuration. If not slurmctld will not like it:
slurmctld: error: Node zsys appears to have a different slurm.conf than the slurmctld. This could cause issues with communication and functionality. Please review both files and make sure they are the same. If this is expected ignore, and set DebugFlags=NO_CONF_HASH in your slurm.conf. /David On Thu, Feb 21, 2013 at 10:16 AM, Marcin Stolarek <[email protected] > wrote: > > > 2013/2/20 David Bigagli <[email protected]> > >> Hi, do you share your slurm.conf or each node has its own? All nodes >> need to share the very same configuration. After you add some nodes to your >> cluster configuration you should restart the entire cluster, 'service slurm >> restart' on all nodes. >> > Is this really necessary to restart also slurmd daemons? > > I understand that slurmctld have to be restarted to create new bitmaps, > but why should I restart all slurmd daemons? > > cheers, > marcin > > > /David > > > On Wed, Feb 20, 2013 at 8:41 AM, Sefa Arslan > <[email protected]>wrote: > >> I am able to make ssh , telnet (to the slurmd port 6818) and ping >> lufer121 from the controller although the node seem to be down according to >> the sview and slurmctld log. >> >> Also I can send and run jobs on the node via srun or sbatch using >> --nodelist before the the node is marked as down. >> >> The "Can't find an address, check slurm.conf" message is a little bit >> confusing but if there were a problem related with host name or IP or >> slurmd.conf, performing the above actions would not be possible I thing.. >> >> I couldn't find a way to understand what the problem really is. As I >> said, I have this problem on only a few nodes. All the nodes are completely >> the same as harware and operating system, all the nodes are set from the >> same system image of a golden client. Only difference is their hostnames >> and IP address. One point: The problematic nodes are added to the slurm >> much later after the cluster start serving. >> >> >> >> On 02/19/2013 06:50 PM, Sefa Arslan wrote: >> >> There is not much information on worker's log.. >> >> [2013-02-19T17:34:11] slurmd version 2.4.1 started >> [2013-02-19T17:34:11] debug3: finished daemonize >> [2013-02-19T17:34:11] debug3: Trying to load plugin >> /usr/lib64/slurm/switch_none.so >> [2013-02-19T17:34:11] switch NONE plugin loaded >> [2013-02-19T17:34:11] debug3: Success. >> [2013-02-19T17:34:11] debug3: successfully opened slurm listen port *:6818 >> [2013-02-19T17:34:11] slurmd started on Tue 19 Feb 2013 17:34:11 +0200 >> [2013-02-19T17:34:11] CPUs=8 Sockets=2 Cores=4 Threads=1 Memory=48253 >> TmpDisk=922082 Uptime=14516 >> [2013-02-19T17:35:19] debug3: in the service_connection >> [2013-02-19T17:35:19] debug2: got this type of message 1001 >> [2013-02-19T17:35:19] debug3: CPUs=8 Sockets=2 Cores=4 Threads=1 >> Memory=48253 TmpDisk=922082 Uptime=14584 >> >> log ends here until i restart the slurmd again.. >> >> >> controller's log >> [2013-02-19T17:34:31] debug2: _slurm_rpc_node_registration complete for >> lufer121 usec=120 >> [2013-02-19T17:35:39] debug3: Tree sending to lufer121 >> [2013-02-19T17:35:39] debug2: _slurm_rpc_node_registration complete for >> lufer121 usec=5160 >> [2013-02-19T17:35:42] debug2: node_did_resp lufer121 >> [2013-02-19T17:38:59] agent/is_node_resp: node:lufer121 rpc:1008 : Can't >> find an address, check slurm.conf >> [2013-02-19T17:39:00] error: Nodes lufer121 not responding >> [2013-02-19T17:40:40] error: Nodes lufer121 not responding >> [2013-02-19T17:42:20] error: Nodes lufer121 not responding >> [2013-02-19T17:43:59] error: Nodes lufer121 not responding, setting DOWN >> >> >> >> >> On 02/19/2013 04:50 PM, David Bigagli wrote: >> >> Hi, have a look at the slurmd.log on the none responding hosts. Does it >> show any relevant information? >> >> /David >> >> >> On Tue, Feb 19, 2013 at 2:46 PM, Sefa Arslan >> <[email protected]>wrote: >> >>> >>> Hello, >>> >>> After a few minutes I restart the SLURM service on the worker node, the >>> controller says the worker node is not responding, and set as DOWN. The >>> related log of the controller: >>> >>> tail -f /var/log/slurm/slurmctld.log|grep lufer121 >>> >>> [2013-02-19T13:34:37] node lufer121 returned to service >>> [2013-02-19T13:34:37] debug2: _slurm_rpc_node_registration complete for >>> lufer121 usec=231 >>> [2013-02-19T13:36:24] debug3: Tree sending to lufer93 along with >>> lufer121,mercan[27-52,54-63,65-71,73-77,89] >>> [2013-02-19T13:36:24] agent/is_node_resp: node:lufer121 rpc:1008 : Can't >>> find an address, check slurm.conf >>> [2013-02-19T13:36:25] error: Nodes lufer121 not responding >>> [2013-02-19T13:38:26] error: Nodes lufer121 not responding >>> [2013-02-19T13:40:25] error: Nodes lufer121 not responding >>> [2013-02-19T13:42:13] error: Nodes lufer121 not responding, setting DOWN >>> >>> >>> I controlled if I have some problem on network, but everything seem to >>> be OK. Even if the slurmctld log says the worker is does not responding, >>> I can ping/or make ssh or telnet to the 6818th port of to the worker. >>> >>> I have this problem on only a few worker. The others are working quite >>> well. >>> >>> Note: slurm version is 2.4.1-1, no firewall is working.. All nodes has >>> dns records on the name server.. >>> >>> >>> Do you have any suggestion? >>> >> >> > >
