Thanks guys. Very helpful. Appreciate the responses. Cheers,
--Andrew On Wed, Jan 13, 2016 at 12:38:08PM -0800, John Desantis wrote: > Andrew, > > Our database has roughly 3.4 million rows in the entire schema (including a > view, but we also purge job records after 6 months). > > After the slurmdbd was upgraded, it took ~9 minutes (manifested by the > changes being performed in the log) before the daemon was active again. > > HTH, > John DeSantis > > 2016-01-13 13:13 GMT-05:00 Trey Dockendorf <[email protected]>: > > > I didn't time our recent upgrade from 14.03.10 to 15.08.6 but it took > > around 30 minutes to an hour for slurmdbd to perform all the database > > changes. We have around 10 million jobs in the database. > > > > I personally setup a test instance of SLURM, including slurmdbd and then > > do something like 'mysqldump --opt slurmdbd | mysql slurmdbd_dev' to put my > > production database into my test instance and then test the upgrade on the > > test cluster. My production and test environments share the same MySQL > > server but use different MySQL accounts to access the databases. > > > > - Trey > > > > ============================= > > > > Trey Dockendorf > > Systems Analyst I > > Texas A&M University > > Academy for Advanced Telecommunications and Learning Technologies > > Phone: (979)458-2396 > > Email: [email protected] > > Jabber: [email protected] > > > > On Tue, Jan 12, 2016 at 4:05 PM, Andrew E. Bruno <[email protected]> > > wrote: > > > >> > >> We're planning an upgrade from 14.11.10 to 15.08.6 and in the past the > >> slurmdbd upgrades can take a while (~20-30 minutes). Curious if there's > >> any way to gauge what we can expect this time? To give an idea, we have > >> upwards of ~4.5 million records in our job tables. We frequently get > >> these warnings in our logs: > >> > >> Warning: Note very large processing time from hourly_rollup > >> ... > >> > >> Also, any "best practices" for upgrading the slurmdbd other than: > >> > >> yum update; systemctl restart slurmdbd > >> > >> (this usually hangs until the database migrations are complete) > >> > >> Thanks in advance for any pointers. > >> > >> Cheers, > >> > >> --Andrew > >> > > > >
