Hi Brandon, My guess is that this isn't a SLURM problem since AFAIK SLURM uses the libc routines for name resolution. You might benefit from enabling nscd on your compute nodes which should cache the name requests and mask anything strange going on at the DNS level.
Out of curiosity, does it affect multiple nodes at the same time or is it sporadic? -Aaron Sent from my iPhone > On Nov 22, 2014, at 8:05 PM, Brandon Barker <[email protected]> > wrote: > > Hello, > > I am having trouble with slurm daemons sometimes not resolving node names, > despite nslookup/dig being able to resolve the hostname without trouble. I've > attached the slurm.conf used on both the slurm controller and the nodes > running slurm daemons. The node running slurmctld also runs dnsmasq for DNS > resolution (as mentioned above, this works when trying nslookup from the > nodes running slurmd). > > If I kill slurmd, and bring it back up on the worker node, the end of > slurmd.log looks like this: > > [root@rcsmall7 ~]# tail /var/log/slurmd.log > [2014-11-19T13:33:43.354] error: Unable to establish control machine address > [2014-11-19T13:33:43.354] error: Unable to register: Resource temporarily > unavailable > [2014-11-19T13:33:44.359] error: Unable to resolve "head": Host name lookup > failure > [2014-11-19T13:33:44.359] error: Unable to establish control machine address > [2014-11-19T13:33:44.359] error: Unable to register: Resource temporarily > unavailable > [2014-11-19T13:33:45.360] Slurmd shutdown completing > [2014-11-19T13:33:58.066] CPU frequency setting not configured for this node > [2014-11-19T13:33:58.070] slurmd version 14.03.9 started > [2014-11-19T13:33:58.071] slurmd started on Wed, 19 Nov 2014 13:33:58 -0500 > [2014-11-19T13:33:58.071] CPUs=1 Boards=1 Sockets=1 Cores=1 Threads=1 > Memory=3953 TmpDisk=9950 Uptime=5768 > > The error messages about host name lookup occur repeatedly and highly > frequently, as you can probably see from the timestamps. So it seems like > slurmd is somehow getting in to a bad state with regard to hostname. If you > can suggestion some possible causes, I may be able to investigate further. > > Thank you! > > <slurm.conf>
