Reviewed: https://review.openstack.org/413495 Committed: https://git.openstack.org/cgit/openstack/keystone/commit/?id=5b7c9a66f0aed860ea0776d4c5b42710d88fcb5f Submitter: Jenkins Branch: master
commit 5b7c9a66f0aed860ea0776d4c5b42710d88fcb5f Author: johnlinp <[email protected]> Date: Wed Dec 21 15:17:01 2016 +0800 Handle disk write failure when doing Fernet key rotation _create_new_key() is broke down into 2 parts: 1. _create_tmp_new_key() 2. _become_valid_new_key() This can avoid empty Fernet keys when the write to the staged key fails. The _become_valid_new_key() is called only after a successful call to _create_tmp_new_key(). Change-Id: Iaf33e2b291f13b9eb9464ef345a8664a634121ff Closes-Bug: #1642457 Signed-off-by: John Lin <[email protected]> ** Changed in: keystone Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1642457 Title: Fernet rotate doesn't prevent rotation when disk is full Status in OpenStack Identity (keystone): Fix Released Bug description: When the root partition on any control node is full, the Fernet key on all control nodes will be empty. This will cause Keystone cannot auth anyone (500 Internal Server Error). Is that caused by Fernet key rotation? When I check the files in /etc/keystone/fernet-keys: root@control1:/etc/keystone/fernet-keys# ll total 40 drwxr-s--- 2 keystone keystone 4096 Nov 17 00:00 ./ drwxr-xr-x 5 keystone keystone 4096 Nov 10 11:24 ../ -rw------- 1 keystone keystone 0 Nov 17 00:00 0 -rw------- 1 keystone keystone 44 Nov 16 13:57 10 -rw------- 1 keystone keystone 44 Nov 9 00:00 3 -rw------- 1 keystone keystone 44 Nov 10 00:00 4 -rw------- 1 keystone keystone 44 Nov 11 00:00 5 -rw------- 1 keystone keystone 44 Nov 12 00:00 6 -rw------- 1 keystone keystone 44 Nov 13 00:00 7 -rw------- 1 keystone keystone 44 Nov 14 00:00 8 -rw------- 1 keystone keystone 44 Nov 15 00:00 9 Here is some of the Keystone logs when the master Fernet token is empty. [req-37cfe30f-5ff0-4d28-a187-066bf8031ad4 - - - - -] Fernet key must be 32 url-safe base64-encoded bytes. Traceback (most recent call last): File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/common/wsgi.py", line 249, in __call__ result = method(context, **params) File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/auth/controllers.py", line 416, in authenticate_for_token parent_audit_id=token_audit_id) File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/common/manager.py", line 124, in wrapped __ret_val = __f(*args, **kwargs) File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/provider.py", line 388, in issue_v3_token parent_audit_id) File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/fernet/core.py", line 44, in issue_v3_token *args, **kwargs) File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/common.py", line 623, in issue_v3_token token_id = self._get_token_id(token_data) File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/fernet/core.py", line 201, in _get_token_id access_token_id=access_token_id File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/fernet/token_formatters.py", line 165, in create_token token = self.pack(serialized_payload) File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/fernet/token_formatters.py", line 75, in pack return self.crypto.encrypt(payload).rstrip(b'=').decode('utf-8') File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/keystone/token/providers/fernet/token_formatters.py", line 64, in crypto fernet_instances = [fernet.Fernet(key) for key in keys] File "/openstack/venvs/keystone-mitaka/lib/python2.7/site-packages/cryptography/fernet.py", line 37, in __init__ "Fernet key must be 32 url-safe base64-encoded bytes." ValueError: Fernet key must be 32 url-safe base64-encoded bytes. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1642457/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
