[Yahoo-eng-team] [Bug 1647800] Re: keystone-manage bootstrap isn't completely idempotent
** Changed in: keystone/mitaka Assignee: Steve Martinelli (stevemar) => Lance Bragstad (lbragstad) ** Changed in: keystone/mitaka Status: Fix Committed => Fix Released ** Changed in: keystone/newton Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1647800 Title: keystone-manage bootstrap isn't completely idempotent Status in OpenStack Identity (keystone): Fix Released Status in OpenStack Identity (keystone) mitaka series: Fix Released Status in OpenStack Identity (keystone) newton series: Fix Released Bug description: The keystone-manage bootstrap command was designed to be idempotent. Most everything in the bootstrap command is wrapped with a try/except to handle cases where specific entities already exist (i.e. there is already an admin project or an admin user from a previous bootstrap run). This is important because bootstrap handles the creation of administrator-like things in order to "bootstrap" a deployment. If bootstrap wasn't idempotent, the side-effect of running it multiple times would be catastrophic. During an upgrade scenario, using OpenStack Ansible's rolling upgrade support [0], from stable/newton to master, I noticed a very specific case where bootstrap was not idempotent. Even if the admin user passed to bootstrap already exists, the command will still attempt to update it's password [1], even if the admin password hasn't changed. It does the same thing with the user's enabled property. This somehow creates a revocation event to be stored for that specific user [2]. As a result, all tokens for the user specified in the bootstrap command will be invalid once the upgrade happens, since OpenStack Ansible relies on `keystone-manage bootstrap` during the upgrade. This only affects the bootstrap user, but it can be considered a service interruption since it is being done during an upgrade. We could look into only updating the user's password, or enabled field, if and only if they have changed. In that case, a revocation event *should* be persisted since the bootstrap command is changing something about the account. In the case where there is no change in password or enabled status, tokens should still be able to be validated across releases. I have documented the upgrade procedure and process in a separate repository [3] [0] https://review.openstack.org/#/c/384269/ [1] https://github.com/openstack/keystone/blob/1c60b1539cf63bba79711e237df496dfa094b2c5/keystone/cmd/cli.py#L226-L232 [2] http://cdn.pasteraw.com/9gz9964mwufyw3f98rv1mv1hqxezpis [3] https://github.com/lbragstad/keystone-performance-upgrade To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1647800/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1647800] Re: keystone-manage bootstrap isn't completely idempotent
Reviewed: https://review.openstack.org/408719 Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=90f2f96e69b8bfd5058628b50c9f0083e3f293e9 Submitter: Jenkins Branch:master commit 90f2f96e69b8bfd5058628b50c9f0083e3f293e9 Author: Lance Bragstad Date: Thu Dec 8 17:01:22 2016 + Make bootstrap idempotent when it needs to be This commit makes `keystone-manage bootstrap` completely idempotent when configuration values or environment variables haven't changed between runs. If they have changed, then `bootstrap` shouldn't be as idempotent becuase it's changing the state of the deployment. This commit addresses these issues and adds tests to ensure the proper behavior is tested. Change-Id: I053b27e881f5bb67db1ace01e6d06aead10b1e47 Closes-Bug: 1647800 ** Changed in: keystone Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1647800 Title: keystone-manage bootstrap isn't completely idempotent Status in OpenStack Identity (keystone): Fix Released Status in OpenStack Identity (keystone) mitaka series: In Progress Status in OpenStack Identity (keystone) newton series: In Progress Bug description: The keystone-manage bootstrap command was designed to be idempotent. Most everything in the bootstrap command is wrapped with a try/except to handle cases where specific entities already exist (i.e. there is already an admin project or an admin user from a previous bootstrap run). This is important because bootstrap handles the creation of administrator-like things in order to "bootstrap" a deployment. If bootstrap wasn't idempotent, the side-effect of running it multiple times would be catastrophic. During an upgrade scenario, using OpenStack Ansible's rolling upgrade support [0], from stable/newton to master, I noticed a very specific case where bootstrap was not idempotent. Even if the admin user passed to bootstrap already exists, the command will still attempt to update it's password [1], even if the admin password hasn't changed. It does the same thing with the user's enabled property. This somehow creates a revocation event to be stored for that specific user [2]. As a result, all tokens for the user specified in the bootstrap command will be invalid once the upgrade happens, since OpenStack Ansible relies on `keystone-manage bootstrap` during the upgrade. This only affects the bootstrap user, but it can be considered a service interruption since it is being done during an upgrade. We could look into only updating the user's password, or enabled field, if and only if they have changed. In that case, a revocation event *should* be persisted since the bootstrap command is changing something about the account. In the case where there is no change in password or enabled status, tokens should still be able to be validated across releases. I have documented the upgrade procedure and process in a separate repository [3] [0] https://review.openstack.org/#/c/384269/ [1] https://github.com/openstack/keystone/blob/1c60b1539cf63bba79711e237df496dfa094b2c5/keystone/cmd/cli.py#L226-L232 [2] http://cdn.pasteraw.com/9gz9964mwufyw3f98rv1mv1hqxezpis [3] https://github.com/lbragstad/keystone-performance-upgrade To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1647800/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1647800] Re: keystone-manage bootstrap isn't completely idempotent
Marking this as Medium in mitaka since we didn't support zero-downtime upgrades then, but this is still an unexpected behavior of bootstrap that would potentially affect an upgrade process. ** Also affects: keystone/mitaka Importance: Undecided Status: New ** Changed in: keystone/mitaka Importance: Undecided => High ** Changed in: keystone/newton Importance: Undecided => High ** Changed in: keystone/mitaka Importance: High => Medium ** Changed in: keystone/mitaka Status: New => Confirmed ** Changed in: keystone/newton Status: New => Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1647800 Title: keystone-manage bootstrap isn't completely idempotent Status in OpenStack Identity (keystone): Confirmed Status in OpenStack Identity (keystone) mitaka series: Confirmed Status in OpenStack Identity (keystone) newton series: Confirmed Bug description: The keystone-manage bootstrap command was designed to be idempotent. Most everything in the bootstrap command is wrapped with a try/except to handle cases where specific entities already exist (i.e. there is already an admin project or an admin user from a previous bootstrap run). This is important because bootstrap handles the creation of administrator-like things in order to "bootstrap" a deployment. If bootstrap wasn't idempotent, the side-effect of running it multiple times would be catastrophic. During an upgrade scenario, using OpenStack Ansible's rolling upgrade support [0], from stable/newton to master, I noticed a very specific case where bootstrap was not idempotent. Even if the admin user passed to bootstrap already exists, the command will still attempt to update it's password [1], even if the admin password hasn't changed. It does the same thing with the user's enabled property. This somehow creates a revocation event to be stored for that specific user [2]. As a result, all tokens for the user specified in the bootstrap command will be invalid once the upgrade happens, since OpenStack Ansible relies on `keystone-manage bootstrap` during the upgrade. This only affects the bootstrap user, but it can be considered a service interruption since it is being done during an upgrade. We could look into only updating the user's password, or enabled field, if and only if they have changed. In that case, a revocation event *should* be persisted since the bootstrap command is changing something about the account. In the case where there is no change in password or enabled status, tokens should still be able to be validated across releases. I have documented the upgrade procedure and process in a separate repository [3] [0] https://review.openstack.org/#/c/384269/ [1] https://github.com/openstack/keystone/blob/1c60b1539cf63bba79711e237df496dfa094b2c5/keystone/cmd/cli.py#L226-L232 [2] http://cdn.pasteraw.com/9gz9964mwufyw3f98rv1mv1hqxezpis [3] https://github.com/lbragstad/keystone-performance-upgrade To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1647800/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1647800] Re: keystone-manage bootstrap isn't completely idempotent
Marking this as High because the consequence is perceivable downtime during a zero-downtime upgrade. ** Also affects: keystone/newton Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1647800 Title: keystone-manage bootstrap isn't completely idempotent Status in OpenStack Identity (keystone): Confirmed Status in OpenStack Identity (keystone) mitaka series: Confirmed Status in OpenStack Identity (keystone) newton series: Confirmed Bug description: The keystone-manage bootstrap command was designed to be idempotent. Most everything in the bootstrap command is wrapped with a try/except to handle cases where specific entities already exist (i.e. there is already an admin project or an admin user from a previous bootstrap run). This is important because bootstrap handles the creation of administrator-like things in order to "bootstrap" a deployment. If bootstrap wasn't idempotent, the side-effect of running it multiple times would be catastrophic. During an upgrade scenario, using OpenStack Ansible's rolling upgrade support [0], from stable/newton to master, I noticed a very specific case where bootstrap was not idempotent. Even if the admin user passed to bootstrap already exists, the command will still attempt to update it's password [1], even if the admin password hasn't changed. It does the same thing with the user's enabled property. This somehow creates a revocation event to be stored for that specific user [2]. As a result, all tokens for the user specified in the bootstrap command will be invalid once the upgrade happens, since OpenStack Ansible relies on `keystone-manage bootstrap` during the upgrade. This only affects the bootstrap user, but it can be considered a service interruption since it is being done during an upgrade. We could look into only updating the user's password, or enabled field, if and only if they have changed. In that case, a revocation event *should* be persisted since the bootstrap command is changing something about the account. In the case where there is no change in password or enabled status, tokens should still be able to be validated across releases. I have documented the upgrade procedure and process in a separate repository [3] [0] https://review.openstack.org/#/c/384269/ [1] https://github.com/openstack/keystone/blob/1c60b1539cf63bba79711e237df496dfa094b2c5/keystone/cmd/cli.py#L226-L232 [2] http://cdn.pasteraw.com/9gz9964mwufyw3f98rv1mv1hqxezpis [3] https://github.com/lbragstad/keystone-performance-upgrade To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1647800/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp