Disregard -- solved. There was another "rogue" slurmd on the network that had a different munge key and had the same control machine in its slurm.conf.
Found it with netstat, looking for connections to port 6817. Would be useful if the logs listed the invalid node IP address in this situation, to make that easier to track down. Allan > This seems like a FAQ, but I think I've eliminated the usual causes. > > Slurm control node logs the following, repeated once per second: > > [2016-05-10T23:49:10.184] error: Munge decode failed: Invalid credential > [2016-05-10T23:49:10.184] error: slurm_receive_msg: > MESSAGE_NODE_REGISTRATION_STATUS has authentication error: Invalid credential > [2016-05-10T23:49:10.184] error: slurm_receive_msg: Protocol authentication > error > [2016-05-10T23:49:10.194] error: slurm_receive_msg: Protocol authentication > error > > [...]
