Who owns the munge directory and key? Is it the right uid/gid? Is the munge daemon running?
-- Sean Crosby | Senior DevOpsHPC Engineer and HPC Team Lead Research Computing Services | Business Services The University of Melbourne, Victoria 3010 Australia On Thu, 16 Apr 2020 at 04:57, Dean Schulze <dean.w.schu...@gmail.com> wrote: > *UoM notice: External email. Be cautious of links, attachments, or > impersonation attempts.* > ------------------------------ > /etc/munge is 700 > /etc/munge/munge.key is 400 > > > > On Wed, Apr 15, 2020 at 12:11 PM Riebs, Andy <andy.ri...@hpe.com> wrote: > >> Two trivial things to check: >> >> 1. Permissions on /etc/munge and /etc/munge.key >> >> 2. Is munged running on the problem node? >> >> >> >> Andy >> >> >> >> *From:* slurm-users [mailto:slurm-users-boun...@lists.schedmd.com] *On >> Behalf Of *Dean Schulze >> *Sent:* Wednesday, April 15, 2020 1:57 PM >> *To:* Slurm User Community List <slurm-users@lists.schedmd.com> >> *Subject:* [slurm-users] Munge decode failing on new node >> >> >> >> I've installed two new nodes onto my slurm cluster. One node works, but >> the other one complains about an invalid credential for munge. I've >> verified that the munge.key is the same as on all other nodes with >> >> >> sudo cksum /etc/munge/munge.key >> >> >> >> I recopied a munge.key from a node that works. I've verified that munge >> uid and gid are the same on the nodes. The time is in sync on all nodes. >> >> >> >> Here is what is in the slurmd.log: >> >> >> >> error: Unable to register: Unable to contact slurm controller (connect >> failure) >> error: Munge decode failed: Invalid credential >> ENCODED: Wed Dec 31 17:00:00 1969 >> DECODED: Wed Dec 31 17:00:00 1969 >> error: authentication: Invalid authentication credential >> error: slurm_receive_msg_and_forward: Protocol authentication error >> error: service_connection: slurm_receive_msg: Protocol authentication >> error >> error: Unable to register: Unable to contact slurm controller (connect >> failure) >> >> >> >> I've checked in the munged.log and all it says is >> >> >> >> Invalid credential >> >> >> >> Thanks for your help >> >