Hi Chrysovalantis, Chrysovalantis Paschoulas <[email protected]> writes:
> Hi Loris! > > We also have HA setup where we use the internal HA mechanism of slurm > for the slurmctld daemons (primary and backup controllers), but for > slurmdbd daemon we use pscd HA tools (Centos 7) with resources in > active/passive mode: one slurmdbd instance, one mariadb instance and a > shared fs that is mounted only on one master node at a time. We are also planning to move to a setup including Pacemaker and Corosync. How do you keep the slurm.conf for the two Slurm controllers and the compute nodes in sync? > I would do such action, like slurm major upgrade, only during offline > maintenance... In our case that means that jobs are in hold state by > the admins and the production partitions are set in inactive mode, > users cannot login or submit new jobs... During this period we may > bring down slurmdbd, slurmctld or both for a while. We're are planning only to go from 14.11.8 to 14.11.9, so, in terms of the versions, a down-time shouldn't be necessary. > In this setup we can upgrade/update Slurm without worries during > offline maintenance. I wouldn't try to mix Slurm with different > running versions at any time. If you can not bring the cluster down > then I cannot tell you a safe scenario (ok a general idea is to bring > the backup controllers down, do the updates on the primary and when > everything is ok the primary then upgrade and start the backup > controllers). And I would be really careful with both the slurm's > mysql DB and the shared state between the main controllers. In any > case follow the online documentation for upgrading. That's my problem: the online documentation doesn't mention HA in connection with upgrade. However, your suggestion of stopping the backup controller makes sense, because then I'll have the non-HA situation for the primary controller, the situation which is described in the documentation. I guess the point to be aware of is that you have to disable HA in order to perform the update. > Cheers > Chrysovalantis Paschoulas Thanks for the response, Loris > PS: Always upgrade and start slurmdbd and let it do the updates in the > database and then start slurmctld. And when everything is OK with the > controllers then you can upgrade Slurm also on the compute nodes. > > PS2: Don't forget to read the man pages + changelog and change correctly > slurm.conf and slurmdbd.conf after an upgrade. > > On 10/16/2015 03:12 PM, Loris Bennett wrote: >> "Loris Bennett" <[email protected]> >> writes: >> >>> "Loris Bennett" <[email protected]> >>> writes: >>> >>>> Hi, >>>> >>>> I have an HA configuration and want to update from 14.11.8 to 14.11.9. >>>> On the page >>>> >>>> http://slurm.schedmd.com/quickstart_admin.html >>>> >>>> an upgrade order is given, but it is not clear to me whether there is >>>> anything particular to take into account in the HA case. >>>> >>>> Should each step regarding the slurmdbd and slurmctld be carried >>>> out first on the primary controller and then on the secondary >>>> controller, or should the complete stack on the primary controller >>>> be updated and then that on the secondary controller? >>>> >>>> Cheers, >>>> >>>> Loris >>>> >>>> PS There is a typo in the section 'Upgrades' on the page given above: >>>> >>>> "If the Slurm daemon's are down" >>>> >>>> should be >>>> >>>> "If the Slurm daemons are down" >>> Does no one have any recommendations for updating an HA setup? >> I'm surprised there have been no responses to this. Is no one using the >> HA setup, or is it a non-issue, because I just need to update one server >> as described in the documentation and then the other? >> >> Cheers, >> >> Loris >> > > > > ----------------------------------------------------------------------------- > ----------------------------------------------------------------------------- > Forschungszentrum Juelich GmbH > 52425 Juelich > Sitz der Gesellschaft: Juelich > Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 > Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher > Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), > Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, > Prof. Dr. Sebastian M. Schmidt > ----------------------------------------------------------------------------- > ----------------------------------------------------------------------------- > -- This signature is currently under construction.
