Public bug reported:
There are three scenarios during a rolling upgrade process where we
could prevent operators from doing the "wrong thing" (doing things out
of order):
1) Operators running code from the next release before `keystone-manage
db_sync --expand` has been run: If you run the next release before
--expand is run, then you'll surely end up with fatal query errors as
columns and tables won't exist that the app thinks should exist.
2) (the scary one) Operators running code from the next release before
`keystone-manage db_sync --migrate` has been run: If you run the next
release before --migrate is run, then any number of different types of
failures are possible due to unpopulated columns & tables, including a
risk of data loss as the new release tries to update records that it
perceives to be unpopulated, which might propagate to the legacy schema
during UPDATE operations, for example.
3) Operators running code from the previous release after `keystone-
manage db_sync --contract` has been run: As in case (1), this may result
in fatal query errors, but also presents a risk of introducing data
inconsistency, as the legacy schema might not have a "full
understanding" of the new schema, as would be the case with additive
schema changes. The legacy application would no longer have triggers to
rely on, so consequences would mostly be dependent on the default values
of columns, constraints, etc.
The second case worries me, as it's the most likely scenario where
operators might not realize what's going on until it's too late.
To prevent all of these scenarios, I think the application should check
at startup to ensure that the expand and data migration repositories
both match a minimum value (specifically, the most recent migration in
the application's respective repositories).
Doing the same sort of check at startup for the contract repo would be
more difficult, as it'd be entirely dependent on when you last upgraded
(whether it be last stable/* release or master at any point), so I'd
like to leave that out of scope here.
** Affects: keystone
Importance: Medium
Status: New
** Tags: upgrades
** Summary changed:
- Prevent keystone from serving requests when schema is not up to date
+ Prevent keystone from serving requests when schema or data migrations are not
up to date
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Identity (keystone).
https://bugs.launchpad.net/bugs/1623117
Title:
Prevent keystone from serving requests when schema or data migrations
are not up to date
Status in OpenStack Identity (keystone):
New
Bug description:
There are three scenarios during a rolling upgrade process where we
could prevent operators from doing the "wrong thing" (doing things out
of order):
1) Operators running code from the next release before `keystone-
manage db_sync --expand` has been run: If you run the next release
before --expand is run, then you'll surely end up with fatal query
errors as columns and tables won't exist that the app thinks should
exist.
2) (the scary one) Operators running code from the next release before
`keystone-manage db_sync --migrate` has been run: If you run the next
release before --migrate is run, then any number of different types of
failures are possible due to unpopulated columns & tables, including a
risk of data loss as the new release tries to update records that it
perceives to be unpopulated, which might propagate to the legacy
schema during UPDATE operations, for example.
3) Operators running code from the previous release after `keystone-
manage db_sync --contract` has been run: As in case (1), this may
result in fatal query errors, but also presents a risk of introducing
data inconsistency, as the legacy schema might not have a "full
understanding" of the new schema, as would be the case with additive
schema changes. The legacy application would no longer have triggers
to rely on, so consequences would mostly be dependent on the default
values of columns, constraints, etc.
The second case worries me, as it's the most likely scenario where
operators might not realize what's going on until it's too late.
To prevent all of these scenarios, I think the application should
check at startup to ensure that the expand and data migration
repositories both match a minimum value (specifically, the most recent
migration in the application's respective repositories).
Doing the same sort of check at startup for the contract repo would be
more difficult, as it'd be entirely dependent on when you last
upgraded (whether it be last stable/* release or master at any point),
so I'd like to leave that out of scope here.
To manage notifications about this bug go to:
https://bugs.launchpad.net/keystone/+bug/1623117/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
