Public bug reported: With a recent bump of the upper constraint for the msgpack library to version v1.0 [1], the *-grenade-multinode gate jobs fail with the following trace.
The data format was changed in msgpack v1.0 [2], so during a rolling upgrade, tokens cached from the N-1 release (msgpack < v1.0) will fail to unpack with the new format in msgpack v1.0. Aug 11 14:45:47.321414 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239fb20e-9ca7-476c-ba19-4c6707192097 None None] Validating token access rules against request {{(pid=29023) validate_allowed_request /usr/local/lib/python3.6/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 11 14:45:47.325427 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] RBAC: auth_context: {'token': <TokenModel (audit_id=XBmEG0BZRNCTDw5-ANYeHw, audit_chain_id=['XBmEG0BZRNCTDw5-ANYeHw']) at 0x7f6d17d4d0f0>, 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60203c873e874a11ae4e8ee43f6ed7cc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cdb7b38e6fc84c20b1aa05d012d599f7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=29023) fill_context /opt/stack/new/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 11 14:45:47.325958 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] REQUEST_METHOD: `GET` {{(pid=29023) log_request_info /opt/stack/new/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 11 14:45:47.326214 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] SCRIPT_NAME: `/identity` {{(pid=29023) log_request_info /opt/stack/new/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 11 14:45:47.326462 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=29023) log_request_info /opt/stack/new/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 11 14:45:47.327842 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: DEBUG keystone.common.fernet_utils [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=29023) load_keys /opt/stack/new/keystone/keystone/common/fernet_utils.py:289}} Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: CRITICAL keystone [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] Unhandled error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in position 3: invalid start byte Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone Traceback (most recent call last): Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2464, in __call__ Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.wsgi_app(environ, start_response) Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/werkzeug/middleware/proxy_fix.py", line 169, in __call__ Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.app(environ, start_response) Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 129, in __call__ Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 193, in call_func Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.func(req, *args, **kwargs) Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/oslo_middleware/base.py", line 124, in __call__ Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone response = req.get_response(self.application) Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1314, in send Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone application, catch_exc_info=False) Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1278, in call_application Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone app_iter = application(self.environ, start_response) Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 143, in __call__ Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return resp(environ, start_response) Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 129, in __call__ Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 193, in call_func Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.func(req, *args, **kwargs) Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/oslo_middleware/base.py", line 124, in __call__ Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone response = req.get_response(self.application) Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1314, in send Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone application, catch_exc_info=False) Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1278, in call_application Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone app_iter = application(self.environ, start_response) Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 129, in __call__ Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 193, in call_func Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.func(req, *args, **kwargs) Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/osprofiler/web.py", line 112, in __call__ Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return request.get_response(self.application) Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1314, in send Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone application, catch_exc_info=False) Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1278, in call_application Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone app_iter = application(self.environ, start_response) Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 129, in __call__ Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 193, in call_func Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.func(req, *args, **kwargs) Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/oslo_middleware/request_id.py", line 58, in __call__ Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone response = req.get_response(self.application) Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1314, in send Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone application, catch_exc_info=False) Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1278, in call_application Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone app_iter = application(self.environ, start_response) Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/server/flask/request_processing/middleware/url_normalize.py", line 38, in __call__ Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.app(environ, start_response) Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 129, in __call__ Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 193, in call_func Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.func(req, *args, **kwargs) Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/keystonemiddleware/auth_token/__init__.py", line 341, in __call__ Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone response = req.get_response(self._app) Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1314, in send Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone application, catch_exc_info=False) Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1278, in call_application Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone app_iter = application(self.environ, start_response) Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/werkzeug/middleware/dispatcher.py", line 66, in __call__ Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return app(environ, start_response) Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2450, in wsgi_app Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone response = self.handle_exception(e) Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return original_handler(e) Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return original_handler(e) Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return original_handler(e) Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone [Previous line repeated 27 more times] Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1867, in handle_exception Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone reraise(exc_type, exc_value, tb) Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 38, in reraise Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone raise value.with_traceback(tb) Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2447, in wsgi_app Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone response = self.full_dispatch_request() Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1952, in full_dispatch_request Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone rv = self.handle_user_exception(e) Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return original_handler(e) Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return original_handler(e) Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return original_handler(e) Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone [Previous line repeated 27 more times] Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1821, in handle_user_exception Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone reraise(exc_type, exc_value, tb) Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 38, in reraise Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone raise value.with_traceback(tb) Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1950, in full_dispatch_request Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone rv = self.dispatch_request() Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1936, in dispatch_request Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.view_functions[rule.endpoint](**req.view_args) Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 468, in wrapper Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone resp = resource(*args, **kwargs) Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/views.py", line 89, in view Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.dispatch_request(*args, **kwargs) Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 583, in dispatch_request Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone resp = meth(*args, **kwargs) Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/api/auth.py", line 285, in get Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone ENFORCER.enforce_call(action='identity:validate_token') Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/common/rbac_enforcer/enforcer.py", line 414, in enforce_call Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone subj_token_target_data = cls._extract_subject_token_target_data() Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/common/rbac_enforcer/enforcer.py", line 256, in _extract_subject_token_target_data Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone access_rules_support=access_rules_support Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/common/manager.py", line 115, in wrapped Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone __ret_val = __f(*args, **kwargs) Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/token/provider.py", line 145, in validate_token Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone token = self._validate_token(token_id) Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "<decorator-gen-26>", line 2, in _validate_token Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 1357, in get_or_create_for_user_func Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone key, user_func, timeout, should_cache_fn, (arg, kw) Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 959, in get_or_create Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone async_creator, Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self._enter() Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/dogpile/lock.py", line 94, in _enter Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone generated = self._enter_create(value, createdtime) Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.creator() Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 913, in gen_value Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone *creator_args[0], **creator_args[1] Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/token/provider.py", line 158, in _validate_token Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone expires_at) = self.driver.validate_token(token_id) Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/token/providers/fernet/core.py", line 96, in validate_token Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.token_formatter.validate_token(token_id) Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/token/token_formatters.py", line 173, in validate_token Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone versioned_payload = msgpack.unpackb(serialized_payload) Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in position 3: invalid start byte Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone [1] https://review.opendev.org/#/c/745437/2/upper-constraints.txt@373 [2] https://github.com/msgpack/msgpack-python/blob/v1.0.0/README.md#major-breaking-changes-in-msgpack-10 ** Affects: keystone Importance: Undecided Assignee: melanie witt (melwitt) Status: In Progress -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1891244 Title: *-grenade-multinode jobs fail with UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in position 3: invalid start byte Status in OpenStack Identity (keystone): In Progress Bug description: With a recent bump of the upper constraint for the msgpack library to version v1.0 [1], the *-grenade-multinode gate jobs fail with the following trace. The data format was changed in msgpack v1.0 [2], so during a rolling upgrade, tokens cached from the N-1 release (msgpack < v1.0) will fail to unpack with the new format in msgpack v1.0. Aug 11 14:45:47.321414 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-239fb20e-9ca7-476c-ba19-4c6707192097 None None] Validating token access rules against request {{(pid=29023) validate_allowed_request /usr/local/lib/python3.6/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Aug 11 14:45:47.325427 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] RBAC: auth_context: {'token': <TokenModel (audit_id=XBmEG0BZRNCTDw5-ANYeHw, audit_chain_id=['XBmEG0BZRNCTDw5-ANYeHw']) at 0x7f6d17d4d0f0>, 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '60203c873e874a11ae4e8ee43f6ed7cc', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'cdb7b38e6fc84c20b1aa05d012d599f7', 'project_domain_id': 'default', 'roles': ['member', 'service', 'admin', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=29023) fill_context /opt/stack/new/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Aug 11 14:45:47.325958 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] REQUEST_METHOD: `GET` {{(pid=29023) log_request_info /opt/stack/new/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Aug 11 14:45:47.326214 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] SCRIPT_NAME: `/identity` {{(pid=29023) log_request_info /opt/stack/new/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Aug 11 14:45:47.326462 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=29023) log_request_info /opt/stack/new/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Aug 11 14:45:47.327842 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: DEBUG keystone.common.fernet_utils [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=29023) load_keys /opt/stack/new/keystone/keystone/common/fernet_utils.py:289}} Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: CRITICAL keystone [None req-e87762ae-3f2a-4a39-9cd9-1c9c901767e8 service placement] Unhandled error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in position 3: invalid start byte Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone Traceback (most recent call last): Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2464, in __call__ Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.wsgi_app(environ, start_response) Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/werkzeug/middleware/proxy_fix.py", line 169, in __call__ Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.app(environ, start_response) Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 129, in __call__ Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 193, in call_func Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.func(req, *args, **kwargs) Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/oslo_middleware/base.py", line 124, in __call__ Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone response = req.get_response(self.application) Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1314, in send Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone application, catch_exc_info=False) Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1278, in call_application Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone app_iter = application(self.environ, start_response) Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 143, in __call__ Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return resp(environ, start_response) Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 129, in __call__ Aug 11 14:45:47.335522 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 193, in call_func Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.func(req, *args, **kwargs) Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/oslo_middleware/base.py", line 124, in __call__ Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone response = req.get_response(self.application) Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1314, in send Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone application, catch_exc_info=False) Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1278, in call_application Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone app_iter = application(self.environ, start_response) Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 129, in __call__ Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 193, in call_func Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.func(req, *args, **kwargs) Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/osprofiler/web.py", line 112, in __call__ Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return request.get_response(self.application) Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1314, in send Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone application, catch_exc_info=False) Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1278, in call_application Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone app_iter = application(self.environ, start_response) Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 129, in __call__ Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 11 14:45:47.336802 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 193, in call_func Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.func(req, *args, **kwargs) Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/oslo_middleware/request_id.py", line 58, in __call__ Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone response = req.get_response(self.application) Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1314, in send Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone application, catch_exc_info=False) Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1278, in call_application Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone app_iter = application(self.environ, start_response) Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/server/flask/request_processing/middleware/url_normalize.py", line 38, in __call__ Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.app(environ, start_response) Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 129, in __call__ Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone resp = self.call_func(req, *args, **kw) Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/dec.py", line 193, in call_func Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.func(req, *args, **kwargs) Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/keystonemiddleware/auth_token/__init__.py", line 341, in __call__ Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone response = req.get_response(self._app) Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1314, in send Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone application, catch_exc_info=False) Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/webob/request.py", line 1278, in call_application Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone app_iter = application(self.environ, start_response) Aug 11 14:45:47.338008 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/werkzeug/middleware/dispatcher.py", line 66, in __call__ Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return app(environ, start_response) Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2450, in wsgi_app Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone response = self.handle_exception(e) Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return original_handler(e) Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return original_handler(e) Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return original_handler(e) Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone [Previous line repeated 27 more times] Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1867, in handle_exception Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone reraise(exc_type, exc_value, tb) Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 38, in reraise Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone raise value.with_traceback(tb) Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2447, in wsgi_app Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone response = self.full_dispatch_request() Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1952, in full_dispatch_request Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone rv = self.handle_user_exception(e) Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return original_handler(e) Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router Aug 11 14:45:47.340661 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return original_handler(e) Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 272, in error_router Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return original_handler(e) Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone [Previous line repeated 27 more times] Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1821, in handle_user_exception Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone reraise(exc_type, exc_value, tb) Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 38, in reraise Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone raise value.with_traceback(tb) Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1950, in full_dispatch_request Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone rv = self.dispatch_request() Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1936, in dispatch_request Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.view_functions[rule.endpoint](**req.view_args) Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 468, in wrapper Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone resp = resource(*args, **kwargs) Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask/views.py", line 89, in view Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.dispatch_request(*args, **kwargs) Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/flask_restful/__init__.py", line 583, in dispatch_request Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone resp = meth(*args, **kwargs) Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/api/auth.py", line 285, in get Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone ENFORCER.enforce_call(action='identity:validate_token') Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/common/rbac_enforcer/enforcer.py", line 414, in enforce_call Aug 11 14:45:47.342100 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone subj_token_target_data = cls._extract_subject_token_target_data() Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/common/rbac_enforcer/enforcer.py", line 256, in _extract_subject_token_target_data Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone access_rules_support=access_rules_support Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/common/manager.py", line 115, in wrapped Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone __ret_val = __f(*args, **kwargs) Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/token/provider.py", line 145, in validate_token Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone token = self._validate_token(token_id) Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "<decorator-gen-26>", line 2, in _validate_token Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 1357, in get_or_create_for_user_func Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone key, user_func, timeout, should_cache_fn, (arg, kw) Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 959, in get_or_create Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone async_creator, Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/dogpile/lock.py", line 185, in __enter__ Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self._enter() Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/dogpile/lock.py", line 94, in _enter Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone generated = self._enter_create(value, createdtime) Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/dogpile/lock.py", line 178, in _enter_create Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.creator() Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/usr/local/lib/python3.6/dist-packages/dogpile/cache/region.py", line 913, in gen_value Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone *creator_args[0], **creator_args[1] Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/token/provider.py", line 158, in _validate_token Aug 11 14:45:47.343355 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone expires_at) = self.driver.validate_token(token_id) Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/token/providers/fernet/core.py", line 96, in validate_token Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone return self.token_formatter.validate_token(token_id) Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "/opt/stack/new/keystone/keystone/token/token_formatters.py", line 173, in validate_token Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone versioned_payload = msgpack.unpackb(serialized_payload) Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone File "msgpack/_unpacker.pyx", line 195, in msgpack._cmsgpack.unpackb Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone UnicodeDecodeError: 'utf-8' codec can't decode byte 0x87 in position 3: invalid start byte Aug 11 14:45:47.344721 ubuntu-bionic-rax-iad-0019208786 [email protected][29022]: ERROR keystone [1] https://review.opendev.org/#/c/745437/2/upper-constraints.txt@373 [2] https://github.com/msgpack/msgpack-python/blob/v1.0.0/README.md#major-breaking-changes-in-msgpack-10 To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1891244/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
