On 04/09/2018 10:54 PM, Roberts, John E. wrote:
The documentation is a little unclear to me, so I was wondering how do a 
complete backup and restore of Slurm for testing and/or disaster recovery.

I'm looking to upgrade Slurm from 16.05.10 to the latest and I'm not sure all 
of what should go. I stood up some VMs to test this upgrade and most things 
looked good after running through it. I imported the mysql DB from our slurmdbd 
instance, installed all of the relevant packages and copied over configs. 
Everything looked ok between the slurmctld node, DB node and test compute 
nodes. Accounts and associations all looked right and I could run jobs.

Some things, however, that seemed to be missing were any kind of usage data. Using Slurm bank, all 
accounts were reset to 0 (rawusage), but they all had the correct limits. Job history was also 
gone. After reading into it a bit farther and testing some more, it seemed all of this was stored 
in the "StateSaveLocation". Should this also not be in the DB? Is there a way to get this 
into the DB? Is this related to the field I saw "JobCompType"? This specifically didn't 
seem clear to me if it needed to be set if we were using slurmdbd.

With that said, for this and disaster recovery, do we need to save copies of 
anything besides the DB, config files and state saves?

John, perhaps you may find some useful information in my Slurm Wiki, for example, the page about Slurm upgrading: https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#upgrading-slurm

The database backup/restore may be useful as well: https://wiki.fysik.dtu.dk/niflheim/Slurm_database#backup-and-restore-of-database

Before major upgrades I've made dry-run test upgrades as documented in https://wiki.fysik.dtu.dk/niflheim/Slurm_installation#make-a-dry-run-database-upgrade


Reply via email to