Looking at the DRMAA code it appears false was returned from calling slurm_load_job( &job_info, fsd_atoi(self->job_id), SHOW_ALL), which triggered the error output and stack dump. Haven't looked at the code for slurm_load_job to see if it's doing anything different. I'm using 14.03.08, FYI.
On Mon, Oct 27, 2014 at 12:00 PM, <[email protected]> wrote: > > Slurm already has connect retry logic (10 times with 0.1 sec between > retries). DRMAA should need no changes unless it directly accesses munge. > > Has anyone else seen this problem? > > > Quoting E V <[email protected]>: > >> Had 2 jobs die yesterday morning with a slurm_load_jobs error: >> Protocol authentication error from inside DRMAA, and this interesting >> message in the log: >> >> If munged is up, restart with --num-threads=10 >> error: Munge encode failed: Unable to access >> "/var/run/munge/munge.socket.2": No such file or directory >> error: authentication: Munged communication error >> >> The slurmctl.log has this >> error: slurm_receive_msg: Zero Bytes were transmitted or received >> >> right about the same time. >> >> Digging deeper it appears that the jobs state's were changing in the >> slurmctl just as the munge daemon got restarted for a logrotate. I >> changed logrotate to rotate munge.log based on size instead of daily, >> which may fix the problem, but feels more like a work around. Any >> other suggestions? I'd be nice to have some sort of retry in the code, >> but not really sure if it'd be in the slurmctl or the DRMAA code. > > > > -- > Morris "Moe" Jette > CTO, SchedMD LLC
