HA for slurmctld is not multidatacenter HA but rather a traditional HA
setup where you have two server heads off of one storage brick
(connected by SAS cables or other fast interconnect). Multidatacenter
HA has issues with keeping things in sync due to latency and IOPs (as
noted below).
So the HA setup for slurmctld will protect you from the server hosting
the slurmctld getting hosed, not the entire rack going down or the
datacenter going down.
-Paul Edmon-
On 10/24/2022 4:14 AM, Ole Holm Nielsen wrote:
On 10/24/22 09:57, Diego Zuccato wrote:
Il 24/10/2022 09:32, Ole Holm Nielsen ha scritto:
> It is definitely a BAD idea to store Slurm StateSaveLocation on a
slow
> NFS directory! SchedMD recommends to use local NVME or SSD disks
> because there will be many IOPS to this file system!
IIUC it does have to be shared between controllers, right?
Possibly use NVME-backed (or even better NVDIMM-backed) NFS share. Or
replica-3 Gluster volume with NVDIMMs for the bricks, for the
paranoid :)
IOPS is the key parameter! Local NVME or SSD should beat any
networked storage. The original question refers to having
StateSaveLocation on a standard (slow) NFS drive, AFAICT.
I don't know how many people prefer using 2 slurmctld hosts (primary
and backup)? I certainly don't do that. Slurm does have a
configurable SlurmctldTimeout parameter so that you can reboot the
server quickly when needed.
It would be nice if people with experience in HA storage for slurmctld
could comment.
/Ole
