I didn't time our recent upgrade from 14.03.10 to 15.08.6 but it took around 30 minutes to an hour for slurmdbd to perform all the database changes. We have around 10 million jobs in the database.
I personally setup a test instance of SLURM, including slurmdbd and then do something like 'mysqldump --opt slurmdbd | mysql slurmdbd_dev' to put my production database into my test instance and then test the upgrade on the test cluster. My production and test environments share the same MySQL server but use different MySQL accounts to access the databases. - Trey ============================= Trey Dockendorf Systems Analyst I Texas A&M University Academy for Advanced Telecommunications and Learning Technologies Phone: (979)458-2396 Email: [email protected] Jabber: [email protected] On Tue, Jan 12, 2016 at 4:05 PM, Andrew E. Bruno <[email protected]> wrote: > > We're planning an upgrade from 14.11.10 to 15.08.6 and in the past the > slurmdbd upgrades can take a while (~20-30 minutes). Curious if there's > any way to gauge what we can expect this time? To give an idea, we have > upwards of ~4.5 million records in our job tables. We frequently get > these warnings in our logs: > > Warning: Note very large processing time from hourly_rollup > ... > > Also, any "best practices" for upgrading the slurmdbd other than: > > yum update; systemctl restart slurmdbd > > (this usually hangs until the database migrations are complete) > > Thanks in advance for any pointers. > > Cheers, > > --Andrew >
