[slurm-users] Re: Integrating Slurm with WekaIO

2024-04-19 Thread Robert Kudyba via slurm-users
On Bright it's set in a few places: grep -r -i SLURM_CONF /etc /etc/systemd/system/slurmctld.service.d/99-cmd.conf:Environment=SLURM_CONF=/cm/shared/apps/slurm/var/etc/slurm/slurm.conf /etc/systemd/system/slurmdbd.service.d/99-cmd.conf:Environment=SLURM_CONF=/cm/shared/apps/slurm/var/etc/slurm/slur

[slurm-users] Re: Integrating Slurm with WekaIO

2024-04-19 Thread Brian Andrus via slurm-users
I would double-check where you are setting SLURM_CONF then. It is acting as if it is not set (typo maybe?) It should be in /etc/defaults/slurmd (but could be /etc/sysconfig/slurmd). Also check what the final, actual command being run to start it is. If anyone has changed the .service file or a

[slurm-users] Re: Integrating Slurm with WekaIO

2024-04-19 Thread Jeffrey Layton via slurm-users
I like it, however, it was working before without a slurm.conf in /etc/slurm. Plus the environment variable SLURM_CONF is pointing to the correct slurm.conf file (the one in /cm/...). Wouldn't Slurm pick up that one? Thanks! Jeff On Fri, Apr 19, 2024 at 1:11 PM Brian Andrus via slurm-users < s

[slurm-users] Re: Integrating Slurm with WekaIO

2024-04-19 Thread Robert Kudyba via slurm-users
> > Simple solution: put a copy of slurm.conf in /etc/slurm/ on the node(s). > For Bright slurm.conf is in /cm/shared/apps/slurm/var/etc/slurm including on all nodes. Make sure on the compute nodes $SLURM_CONF resolves to the correct path. > On 4/19/2024 9:56 AM, Jeffrey Layton via slurm-users w

[slurm-users] Re: Integrating Slurm with WekaIO

2024-04-19 Thread Brian Andrus via slurm-users
This is because you have no slurm.conf in /etc/slurm, so it it is trying 'configless' which queries DNS to find out where to get the config. It is failing because you do not have DNS configured to tell nodes where to ask about the config. Simple solution: put a copy of slurm.conf in /etc/slurm