[slurm-dev] Re: slurmdbd upgrade

[Trey Dockendorf]

I didn't time our recent upgrade from 14.03.10 to 15.08.6 but it took
around 30 minutes to an hour for slurmdbd to perform all the database
changes.  We have around 10 million jobs in the database.

I personally setup a test instance of SLURM, including slurmdbd and then do
something like 'mysqldump --opt slurmdbd | mysql slurmdbd_dev' to put my
production database into my test instance and then test the upgrade on the
test cluster.  My production and test environments share the same MySQL
server but use different MySQL accounts to access the databases.

- Trey

=

Trey Dockendorf
Systems Analyst I
Texas A University
Academy for Advanced Telecommunications and Learning Technologies
Phone: (979)458-2396
Email: treyd...@tamu.edu
Jabber: treyd...@tamu.edu

On Tue, Jan 12, 2016 at 4:05 PM, Andrew E. Bruno 
wrote:

>
> We're planning an upgrade from 14.11.10 to 15.08.6 and in the past the
> slurmdbd upgrades can take a while (~20-30 minutes). Curious if there's
> any way to gauge what we can expect this time? To give an idea, we have
> upwards of ~4.5 million records in our job tables. We frequently get
> these warnings in our logs:
>
>  Warning: Note very large processing time from hourly_rollup
>  ...
>
> Also, any "best practices" for upgrading the slurmdbd other than:
>
>   yum update; systemctl restart slurmdbd
>
> (this usually hangs until the database migrations are complete)
>
> Thanks in advance for any pointers.
>
> Cheers,
>
> --Andrew
>

[slurm-dev] Re: slurmdbd upgrade

[John Desantis]

Andrew,

Our database has roughly 3.4 million rows in the entire schema (including a
view, but we also purge job records after 6 months).

After the slurmdbd was upgraded, it took ~9 minutes (manifested by the
changes being performed in the log) before the daemon was active again.

HTH,
John DeSantis

2016-01-13 13:13 GMT-05:00 Trey Dockendorf :

> I didn't time our recent upgrade from 14.03.10 to 15.08.6 but it took
> around 30 minutes to an hour for slurmdbd to perform all the database
> changes.  We have around 10 million jobs in the database.
>
> I personally setup a test instance of SLURM, including slurmdbd and then
> do something like 'mysqldump --opt slurmdbd | mysql slurmdbd_dev' to put my
> production database into my test instance and then test the upgrade on the
> test cluster.  My production and test environments share the same MySQL
> server but use different MySQL accounts to access the databases.
>
> - Trey
>
> =
>
> Trey Dockendorf
> Systems Analyst I
> Texas A University
> Academy for Advanced Telecommunications and Learning Technologies
> Phone: (979)458-2396
> Email: treyd...@tamu.edu
> Jabber: treyd...@tamu.edu
>
> On Tue, Jan 12, 2016 at 4:05 PM, Andrew E. Bruno 
> wrote:
>
>>
>> We're planning an upgrade from 14.11.10 to 15.08.6 and in the past the
>> slurmdbd upgrades can take a while (~20-30 minutes). Curious if there's
>> any way to gauge what we can expect this time? To give an idea, we have
>> upwards of ~4.5 million records in our job tables. We frequently get
>> these warnings in our logs:
>>
>>  Warning: Note very large processing time from hourly_rollup
>>  ...
>>
>> Also, any "best practices" for upgrading the slurmdbd other than:
>>
>>   yum update; systemctl restart slurmdbd
>>
>> (this usually hangs until the database migrations are complete)
>>
>> Thanks in advance for any pointers.
>>
>> Cheers,
>>
>> --Andrew
>>
>
>

[slurm-dev] Re: slurmdbd upgrade

[Andrew E. Bruno]

Thanks guys. Very helpful. Appreciate the responses.

Cheers,

--Andrew


On Wed, Jan 13, 2016 at 12:38:08PM -0800, John Desantis wrote:
> Andrew,
> 
> Our database has roughly 3.4 million rows in the entire schema (including a
> view, but we also purge job records after 6 months).
> 
> After the slurmdbd was upgraded, it took ~9 minutes (manifested by the
> changes being performed in the log) before the daemon was active again.
> 
> HTH,
> John DeSantis
> 
> 2016-01-13 13:13 GMT-05:00 Trey Dockendorf :
> 
> > I didn't time our recent upgrade from 14.03.10 to 15.08.6 but it took
> > around 30 minutes to an hour for slurmdbd to perform all the database
> > changes.  We have around 10 million jobs in the database.
> >
> > I personally setup a test instance of SLURM, including slurmdbd and then
> > do something like 'mysqldump --opt slurmdbd | mysql slurmdbd_dev' to put my
> > production database into my test instance and then test the upgrade on the
> > test cluster.  My production and test environments share the same MySQL
> > server but use different MySQL accounts to access the databases.
> >
> > - Trey
> >
> > =
> >
> > Trey Dockendorf
> > Systems Analyst I
> > Texas A University
> > Academy for Advanced Telecommunications and Learning Technologies
> > Phone: (979)458-2396
> > Email: treyd...@tamu.edu
> > Jabber: treyd...@tamu.edu
> >
> > On Tue, Jan 12, 2016 at 4:05 PM, Andrew E. Bruno 
> > wrote:
> >
> >>
> >> We're planning an upgrade from 14.11.10 to 15.08.6 and in the past the
> >> slurmdbd upgrades can take a while (~20-30 minutes). Curious if there's
> >> any way to gauge what we can expect this time? To give an idea, we have
> >> upwards of ~4.5 million records in our job tables. We frequently get
> >> these warnings in our logs:
> >>
> >>  Warning: Note very large processing time from hourly_rollup
> >>  ...
> >>
> >> Also, any "best practices" for upgrading the slurmdbd other than:
> >>
> >>   yum update; systemctl restart slurmdbd
> >>
> >> (this usually hangs until the database migrations are complete)
> >>
> >> Thanks in advance for any pointers.
> >>
> >> Cheers,
> >>
> >> --Andrew
> >>
> >
> >

[slurm-dev] Re: slurmdbd upgrade

[Paul Edmon]

That sounds about right.  We have about the same order of magnitude and 
our last major upgrade took about an hour for the DB to  update itself.


-Paul Edmon-

On 1/12/2016 5:04 PM, Andrew E. Bruno wrote:

We're planning an upgrade from 14.11.10 to 15.08.6 and in the past the
slurmdbd upgrades can take a while (~20-30 minutes). Curious if there's
any way to gauge what we can expect this time? To give an idea, we have
upwards of ~4.5 million records in our job tables. We frequently get
these warnings in our logs:

  Warning: Note very large processing time from hourly_rollup
  ...

Also, any "best practices" for upgrading the slurmdbd other than:

   yum update; systemctl restart slurmdbd

(this usually hangs until the database migrations are complete)

Thanks in advance for any pointers.

Cheers,

--Andrew