** Description changed: + [Impact] + octavia lb's listener can't be delete when there is no secret container which is already deleted. + + + [Test Case] + + 1. deploy and setup octavia env + 2. run below script + - https://pastebin.ubuntu.com/p/xPWKnPqFhw/ + 3. check if there is listener-1 is there or not ( in this case, there is ) + + [Regression] + + This fix needs restarting octavia services. so downtime will be there + shortly. + + [Others] + + After patching with https://review.opendev.org/#/c/691693/, + + I Was able to see the same error , but listener is actually deleted. + + + [Original Description] + There seems to be a fault condition in amphora builds that can lead to not being able to delete a failed loadbalancer/listener. If the barbican container for a tls terminated endpoint listener is deleted before the listener/amphora is deleted, the query to update to the listener's tls certificates fails during the delete_pools flow. It exhibits the following in the client: DELETE call to None for https://octavia.mysite:9876/v2.0/lbaas/listeners/4f46a6a3-a756-41b2-9148-c060bf28e621 used request id req-8146cc8c-334a-4155-9fa6-489d4cf1ecbf Request returned failure status: 500 Not Found: Not Found. Sorry but your container is in another castle. (HTTP 500) (Request-ID: req-8146cc8c-334a-4155-9fa6-489d4cf1ecbf) Traceback (most recent call last): - File "/usr/lib/python3/dist-packages/octaviaclient/api/v2/octavia.py", line 29, in wrapper - response = func(*args, **kwargs) - File "/usr/lib/python3/dist-packages/octaviaclient/api/v2/octavia.py", line 204, in listener_delete - response = self.delete(url) - File "/usr/lib/python3/dist-packages/osc_lib/api/api.py", line 185, in delete - return self._request('DELETE', url, **params) - File "/usr/lib/python3/dist-packages/osc_lib/api/api.py", line 141, in _request - return session.request(url, method, **kwargs) - File "/usr/lib/python3/dist-packages/osc_lib/session.py", line 40, in request - resp = super(TimingSession, self).request(url, method, **kwargs) - File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 737, in request - raise exceptions.from_response(resp, method, url) + File "/usr/lib/python3/dist-packages/octaviaclient/api/v2/octavia.py", line 29, in wrapper + response = func(*args, **kwargs) + File "/usr/lib/python3/dist-packages/octaviaclient/api/v2/octavia.py", line 204, in listener_delete + response = self.delete(url) + File "/usr/lib/python3/dist-packages/osc_lib/api/api.py", line 185, in delete + return self._request('DELETE', url, **params) + File "/usr/lib/python3/dist-packages/osc_lib/api/api.py", line 141, in _request + return session.request(url, method, **kwargs) + File "/usr/lib/python3/dist-packages/osc_lib/session.py", line 40, in request + resp = super(TimingSession, self).request(url, method, **kwargs) + File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 737, in request + raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500) (Request-ID: req-8146cc8c-334a-4155-9fa6-489d4cf1ecbf) During handling of the above exception, another exception occurred: Traceback (most recent call last): - File "/usr/lib/python3/dist-packages/cliff/app.py", line 400, in run_subcommand - result = cmd.run(parsed_args) - File "/usr/lib/python3/dist-packages/osc_lib/command/command.py", line 41, in run - return super(Command, self).run(parsed_args) - File "/usr/lib/python3/dist-packages/cliff/command.py", line 184, in run - return_code = self.take_action(parsed_args) or 0 - File "/usr/lib/python3/dist-packages/octaviaclient/osc/v2/listener.py", line 152, in take_action - listener_id=listener_id) - File "/usr/lib/python3/dist-packages/octaviaclient/api/v2/octavia.py", line 38, in wrapper - request_id=e.request_id) + File "/usr/lib/python3/dist-packages/cliff/app.py", line 400, in run_subcommand + result = cmd.run(parsed_args) + File "/usr/lib/python3/dist-packages/osc_lib/command/command.py", line 41, in run + return super(Command, self).run(parsed_args) + File "/usr/lib/python3/dist-packages/cliff/command.py", line 184, in run + return_code = self.take_action(parsed_args) or 0 + File "/usr/lib/python3/dist-packages/octaviaclient/osc/v2/listener.py", line 152, in take_action + listener_id=listener_id) + File "/usr/lib/python3/dist-packages/octaviaclient/api/v2/octavia.py", line 38, in wrapper + request_id=e.request_id) octaviaclient.api.v2.octavia.OctaviaClientException: Not Found: Not Found. Sorry but your container is in another castle. (HTTP 500) (Request-ID: req-8146cc8c-334a-4155-9fa6-489d4cf1ecbf) clean_up DeleteListener: Not Found: Not Found. Sorry but your container is in another castle. (HTTP 500) (Request-ID: req-8146cc8c-334a-4155-9fa6-489d4cf1ecbf) Traceback (most recent call last): - File "/usr/lib/python3/dist-packages/octaviaclient/api/v2/octavia.py", line 29, in wrapper - response = func(*args, **kwargs) - File "/usr/lib/python3/dist-packages/octaviaclient/api/v2/octavia.py", line 204, in listener_delete - response = self.delete(url) - File "/usr/lib/python3/dist-packages/osc_lib/api/api.py", line 185, in delete - return self._request('DELETE', url, **params) - File "/usr/lib/python3/dist-packages/osc_lib/api/api.py", line 141, in _request - return session.request(url, method, **kwargs) - File "/usr/lib/python3/dist-packages/osc_lib/session.py", line 40, in request - resp = super(TimingSession, self).request(url, method, **kwargs) - File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 737, in request - raise exceptions.from_response(resp, method, url) + File "/usr/lib/python3/dist-packages/octaviaclient/api/v2/octavia.py", line 29, in wrapper + response = func(*args, **kwargs) + File "/usr/lib/python3/dist-packages/octaviaclient/api/v2/octavia.py", line 204, in listener_delete + response = self.delete(url) + File "/usr/lib/python3/dist-packages/osc_lib/api/api.py", line 185, in delete + return self._request('DELETE', url, **params) + File "/usr/lib/python3/dist-packages/osc_lib/api/api.py", line 141, in _request + return session.request(url, method, **kwargs) + File "/usr/lib/python3/dist-packages/osc_lib/session.py", line 40, in request + resp = super(TimingSession, self).request(url, method, **kwargs) + File "/usr/lib/python3/dist-packages/keystoneauth1/session.py", line 737, in request + raise exceptions.from_response(resp, method, url) keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500) (Request-ID: req-8146cc8c-334a-4155-9fa6-489d4cf1ecbf) During handling of the above exception, another exception occurred: Traceback (most recent call last): - File "/usr/lib/python3/dist-packages/osc_lib/shell.py", line 134, in run - ret_val = super(OpenStackShell, self).run(argv) - File "/usr/lib/python3/dist-packages/cliff/app.py", line 279, in run - result = self.run_subcommand(remainder) - File "/usr/lib/python3/dist-packages/osc_lib/shell.py", line 169, in run_subcommand - ret_value = super(OpenStackShell, self).run_subcommand(argv) - File "/usr/lib/python3/dist-packages/cliff/app.py", line 400, in run_subcommand - result = cmd.run(parsed_args) - File "/usr/lib/python3/dist-packages/osc_lib/command/command.py", line 41, in run - return super(Command, self).run(parsed_args) - File "/usr/lib/python3/dist-packages/cliff/command.py", line 184, in run - return_code = self.take_action(parsed_args) or 0 - File "/usr/lib/python3/dist-packages/octaviaclient/osc/v2/listener.py", line 152, in take_action - listener_id=listener_id) - File "/usr/lib/python3/dist-packages/octaviaclient/api/v2/octavia.py", line 38, in wrapper - request_id=e.request_id) + File "/usr/lib/python3/dist-packages/osc_lib/shell.py", line 134, in run + ret_val = super(OpenStackShell, self).run(argv) + File "/usr/lib/python3/dist-packages/cliff/app.py", line 279, in run + result = self.run_subcommand(remainder) + File "/usr/lib/python3/dist-packages/osc_lib/shell.py", line 169, in run_subcommand + ret_value = super(OpenStackShell, self).run_subcommand(argv) + File "/usr/lib/python3/dist-packages/cliff/app.py", line 400, in run_subcommand + result = cmd.run(parsed_args) + File "/usr/lib/python3/dist-packages/osc_lib/command/command.py", line 41, in run + return super(Command, self).run(parsed_args) + File "/usr/lib/python3/dist-packages/cliff/command.py", line 184, in run + return_code = self.take_action(parsed_args) or 0 + File "/usr/lib/python3/dist-packages/octaviaclient/osc/v2/listener.py", line 152, in take_action + listener_id=listener_id) + File "/usr/lib/python3/dist-packages/octaviaclient/api/v2/octavia.py", line 38, in wrapper + request_id=e.request_id) octaviaclient.api.v2.octavia.OctaviaClientException: Not Found: Not Found. Sorry but your container is in another castle. (HTTP 500) (Request-ID: req-8146cc8c-334a-4155-9fa6-489d4cf1ecbf) - The server side log shows: 2019-11-05 14:41:48.288 2020577 WARNING octavia.controller.worker.controller_worker [req-046fe879-63c2-4804-9732-9985faf380e6 - ef8b2568c694461499c074c641a57a14 - - -] Task 'octavia.controller.worker.tasks.amphora_driver_tasks.ListenersUpdate' (1e77205e-486a-432b-b236-8e806c3c2b7e) transitioned into state 'FAILURE' from state 'RUNNING' 5 predecessors (most recent first): - Atom 'octavia.controller.worker.tasks.model_tasks.DeleteModelObject' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'object': <octavia.common.data_models.Pool object at 0x7fd217b21978>}, 'provides': None} - |__Atom 'octavia.controller.worker.tasks.database_tasks.CountPoolChildrenForQuota' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'pool': <octavia.common.data_models.Pool object at 0x7fd217b21978>}, 'provides': {'HM': 1, 'member': 3}} - |__Atom 'octavia.controller.worker.tasks.database_tasks.MarkPoolPendingDeleteInDB' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'pool': <octavia.common.data_models.Pool object at 0x7fd217b21978>}, 'provides': None} - |__Atom 'octavia.controller.worker.tasks.lifecycle_tasks.PoolToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'pool': <octavia.common.data_models.Pool object at 0x7fd217b21978>, 'listeners': [<octavia.common.data_models.Listener object at 0x7fd217b7f400>], 'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7fd216dc6550>}, 'provides': None} - |__Flow 'octavia-delete-pool-flow': barbicanclient.exceptions.HTTPClientError: Not Found: Not Found. Sorry but your container is in another castle. + Atom 'octavia.controller.worker.tasks.model_tasks.DeleteModelObject' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'object': <octavia.common.data_models.Pool object at 0x7fd217b21978>}, 'provides': None} + |__Atom 'octavia.controller.worker.tasks.database_tasks.CountPoolChildrenForQuota' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'pool': <octavia.common.data_models.Pool object at 0x7fd217b21978>}, 'provides': {'HM': 1, 'member': 3}} + |__Atom 'octavia.controller.worker.tasks.database_tasks.MarkPoolPendingDeleteInDB' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'pool': <octavia.common.data_models.Pool object at 0x7fd217b21978>}, 'provides': None} + |__Atom 'octavia.controller.worker.tasks.lifecycle_tasks.PoolToErrorOnRevertTask' {'intention': 'EXECUTE', 'state': 'SUCCESS', 'requires': {'pool': <octavia.common.data_models.Pool object at 0x7fd217b21978>, 'listeners': [<octavia.common.data_models.Listener object at 0x7fd217b7f400>], 'loadbalancer': <octavia.common.data_models.LoadBalancer object at 0x7fd216dc6550>}, 'provides': None} + |__Flow 'octavia-delete-pool-flow': barbicanclient.exceptions.HTTPClientError: Not Found: Not Found. Sorry but your container is in another castle. 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker Traceback (most recent call last): 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/octavia/certificates/manager/barbican.py", line 114, in get_cert 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker return pkcs12.PKCS12Cert(cert_secret.payload) 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/v1/secrets.py", line 193, in payload 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker self._fetch_payload() 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/v1/secrets.py", line 261, in _fetch_payload 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker if not self.payload_content_type and not self.content_types: 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/v1/secrets.py", line 184, in payload_content_type 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker if not self._payload_content_type and self.content_types: 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/v1/secrets.py", line 34, in wrapper 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker self._fill_lazy_properties() 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/v1/secrets.py", line 414, in _fill_lazy_properties 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker result = self._api.get(self._secret_ref) 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/client.py", line 70, in get 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker return super(_HTTPClient, self).get(*args, **kwargs).json() 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 375, in get 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker return self.request(url, 'GET', **kwargs) 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/client.py", line 63, in request 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker self._check_status_code(resp) 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/client.py", line 107, in _check_status_code 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker status 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker barbicanclient.exceptions.HTTPClientError: Not Found: Not Found. Sorry but your container is in another castle. 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker During handling of the above exception, another exception occurred: 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker Traceback (most recent call last): 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker result = task.execute(**arguments) 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/octavia/controller/worker/tasks/amphora_driver_tasks.py", line 78, in execute 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker self.amphora_driver.update(listener, loadbalancer.vip) 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 162, in update 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker certs = self._process_tls_certificates(listener) 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/octavia/amphorae/drivers/haproxy/rest_api_driver.py", line 284, in _process_tls_certificates 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker self.cert_manager, listener) 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/octavia/common/tls_utils/cert_parser.py", line 353, in load_certificates_data 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker check_only=True)) 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/octavia/certificates/manager/barbican.py", line 122, in get_cert 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker check_only=check_only, service_name=service_name 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/octavia/certificates/manager/barbican_legacy.py", line 160, in get_cert 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker LOG.error('Error getting cert %s: %s', cert_ref, str(e)) 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker self.force_reraise() 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker six.reraise(self.type_, self.value, self.tb) 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker raise value 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/octavia/certificates/manager/barbican_legacy.py", line 138, in get_cert 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker container_ref=cert_ref 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/v1/containers.py", line 540, in get 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker response = self._api.get(container_ref) 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/client.py", line 70, in get 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker return super(_HTTPClient, self).get(*args, **kwargs).json() 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/keystoneauth1/adapter.py", line 375, in get 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker return self.request(url, 'GET', **kwargs) 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/client.py", line 63, in request 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker self._check_status_code(resp) 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker File "/usr/lib/python3/dist-packages/barbicanclient/client.py", line 107, in _check_status_code 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker status 2019-11-05 14:41:48.288 2020577 ERROR octavia.controller.worker.controller_worker barbicanclient.exceptions.HTTPClientError: Not Found: Not Found. Sorry but your container is in another castle.
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1852599 Title: Cannot delete TLS Terminated listener if container has been deleted in barbican To manage notifications about this bug go to: https://bugs.launchpad.net/charm-octavia/+bug/1852599/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
