Hi Loris!
We also have HA setup where we use the internal HA mechanism of slurm
for the slurmctld daemons (primary and backup controllers), but for
slurmdbd daemon we use pscd HA tools (Centos 7) with resources in
active/passive mode: one slurmdbd instance, one mariadb instance and a
shared fs that is mounted only on one master node at a time.
I would do such action, like slurm major upgrade, only during offline
maintenance... In our case that means that jobs are in hold state by the
admins and the production partitions are set in inactive mode, users
cannot login or submit new jobs... During this period we may bring down
slurmdbd, slurmctld or both for a while.
In this setup we can upgrade/update Slurm without worries during offline
maintenance. I wouldn't try to mix Slurm with different running versions
at any time. If you can not bring the cluster down then I cannot tell
you a safe scenario (ok a general idea is to bring the backup
controllers down, do the updates on the primary and when everything is
ok the primary then upgrade and start the backup controllers). And I
would be really careful with both the slurm's mysql DB and the shared
state between the main controllers. In any case follow the online
documentation for upgrading.
Cheers
Chrysovalantis Paschoulas
PS: Always upgrade and start slurmdbd and let it do the updates in the
database and then start slurmctld. And when everything is OK with the
controllers then you can upgrade Slurm also on the compute nodes.
PS2: Don't forget to read the man pages + changelog and change correctly
slurm.conf and slurmdbd.conf after an upgrade.
On 10/16/2015 03:12 PM, Loris Bennett wrote:
"Loris Bennett" <[email protected]>
writes:
"Loris Bennett" <[email protected]>
writes:
Hi,
I have an HA configuration and want to update from 14.11.8 to 14.11.9.
On the page
http://slurm.schedmd.com/quickstart_admin.html
an upgrade order is given, but it is not clear to me whether there is
anything particular to take into account in the HA case.
Should each step regarding the slurmdbd and slurmctld be carried out
first on the primary controller and then on the secondary controller, or
should the complete stack on the primary controller be updated and then
that on the secondary controller?
Cheers,
Loris
PS There is a typo in the section 'Upgrades' on the page given above:
"If the Slurm daemon's are down"
should be
"If the Slurm daemons are down"
Does no one have any recommendations for updating an HA setup?
I'm surprised there have been no responses to this. Is no one using the
HA setup, or is it a non-issue, because I just need to update one server
as described in the documentation and then the other?
Cheers,
Loris
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Forschungszentrum Juelich GmbH
52425 Juelich
Sitz der Gesellschaft: Juelich
Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498
Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher
Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender),
Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt,
Prof. Dr. Sebastian M. Schmidt
------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
