I've just installed 2.6.9 and it seems that slurmctld is increasing it's RAM usage steadily... caught it at 5GB this morning. Restarting seems to free it all up and it goes back down to a more reasonable amount... but it's back up to 1.5GB, so I think there's a problem.
I saw this in the list of changes for Slurm 14.03.4: -- select/cons_res plugin: Fix memory leak related to job preemption. We do lots of preemption, which makes me think this is it. Otherwise I can't see a good reason for this happening. Would 2.6.9 be affected? Any other ideas? Thanks Michael
