Reviewed: https://review.opendev.org/c/openstack/neutron/+/881459 Committed: https://opendev.org/openstack/neutron/commit/dd184c5c10e3311ceaffd2cde8d66f76aadaebac Submitter: "Zuul (22348)" Branch: master
commit dd184c5c10e3311ceaffd2cde8d66f76aadaebac Author: Rodolfo Alonso Hernandez <[email protected]> Date: Sat Apr 22 04:09:18 2023 +0200 Fix Loki tempest jobs This patch fixes the issues reported in the "neutron-ovn-tempest-with-uwsgi-loki" job: * The missing retry decorators in some ``TapPlugin`` methods. * Added a session flush in ``OVNL3RouterPlugin.create_router_precommit``. In case the router creation fails before the creation of the initial ``OVNRevisionNumbers`` register, the transaction is retried. That will prevent from trying to retrieve a router standard attribute that doesn't exist, due to a previous error. Closes-Bug: #2017131 Change-Id: Idbf7ea6018146b67387f2ff99e281e32170ba039 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/2017131 Title: neutron-loki job fails tests with ORM session: SQL execution without transaction in progress Status in neutron: Fix Released Bug description: Job(neutron-ovn-tempest-with-uwsgi-loki) running neutron-loki service[1] fails tempest tests randomly with Tracebacks like below:- ORM session: SQL execution without transaction in progress, traceback: Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/eventlet/greenthread.py", line 221, in main Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: result = function(*args, **kwargs) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/eventlet/wsgi.py", line 837, in process_request Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: proto.__init__(conn_state, self) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/eventlet/wsgi.py", line 350, in __init__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: self.handle() Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/eventlet/wsgi.py", line 383, in handle Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: self.handle_one_request() Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/eventlet/wsgi.py", line 459, in handle_one_request Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: self.handle_one_response() Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/eventlet/wsgi.py", line 569, in handle_one_response Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: result = self.application(self.environ, start_response) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/paste/urlmap.py", line 216, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: return app(environ, start_response) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: resp = self.call_func(req, *args, **kw) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: return self.func(req, *args, **kwargs) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/oslo_middleware/base.py", line 124, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: response = req.get_response(self.application) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1313, in send Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: status, headers, app_iter = self.call_application( Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1278, in call_application Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: app_iter = application(self.environ, start_response) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: resp = self.call_func(req, *args, **kw) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: return self.func(req, *args, **kwargs) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/oslo_middleware/base.py", line 124, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: response = req.get_response(self.application) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1313, in send Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: status, headers, app_iter = self.call_application( Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1278, in call_application Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: app_iter = application(self.environ, start_response) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: resp = self.call_func(req, *args, **kw) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: return self.func(req, *args, **kwargs) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/oslo_middleware/request_id.py", line 58, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: response = req.get_response(self.application) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1313, in send Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: status, headers, app_iter = self.call_application( Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1278, in call_application Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: app_iter = application(self.environ, start_response) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: resp = self.call_func(req, *args, **kw) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: return self.func(req, *args, **kwargs) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/oslo_middleware/catch_errors.py", line 40, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: response = req.get_response(self.application) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1313, in send Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: status, headers, app_iter = self.call_application( Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1278, in call_application Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: app_iter = application(self.environ, start_response) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: resp = self.call_func(req, *args, **kw) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: return self.func(req, *args, **kwargs) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/osprofiler/web.py", line 111, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: return request.get_response(self.application) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1313, in send Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: status, headers, app_iter = self.call_application( Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1278, in call_application Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: app_iter = application(self.environ, start_response) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: resp = self.call_func(req, *args, **kw) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: return self.func(req, *args, **kwargs) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/keystonemiddleware/auth_token/__init__.py", line 340, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: response = req.get_response(self._app) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1313, in send Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: status, headers, app_iter = self.call_application( Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/request.py", line 1278, in call_application Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: app_iter = application(self.environ, start_response) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 143, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: return resp(environ, start_response) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 143, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: return resp(environ, start_response) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/routes/middleware.py", line 153, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: response = self.app(environ, start_response) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 143, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: return resp(environ, start_response) Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 129, in __call__ Apr 17 12:12:44.386611 np0033759257 neutron-server[57996]: resp = self.call_func(req, *args, **kw) Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/webob/dec.py", line 193, in call_func Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: return self.func(req, *args, **kwargs) Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/opt/stack/neutron/neutron/api/v2/resource.py", line 98, in resource Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: result = method(request=request, **args) Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/opt/stack/neutron/neutron/api/v2/base.py", line 438, in create Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: return self._create(request, body, **kwargs) Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 138, in wrapped Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: return f(*args, **kwargs) Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/oslo_db/api.py", line 142, in wrapper Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: return f(*args, **kwargs) Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 184, in wrapped Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: return f(*dup_args, **dup_kwargs) Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/opt/stack/neutron/neutron/api/v2/base.py", line 563, in _create Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: obj = do_create(body) Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/opt/stack/neutron/neutron/api/v2/base.py", line 537, in do_create Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: return obj_creator(request.context, **kwargs) Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/opt/stack/neutron/neutron/services/ovn_l3/plugin.py", line 181, in create_router Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: router = super(OVNL3RouterPlugin, self).create_router(context, router) Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/opt/stack/neutron/neutron/db/l3_db.py", line 2182, in create_router Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: router_dict = super(L3_NAT_db_mixin, self).create_router(context, Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 226, in wrapped Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: return f_with_retry(*args, **kwargs, Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 138, in wrapped Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: return f(*args, **kwargs) Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/oslo_db/api.py", line 142, in wrapper Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: return f(*args, **kwargs) Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 184, in wrapped Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: return f(*dup_args, **dup_kwargs) Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/opt/stack/neutron/neutron/db/l3_db.py", line 296, in create_router Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: router_db, _unused = db_utils.safe_creation(context, create, Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/opt/stack/neutron/neutron/db/_utils.py", line 67, in safe_creation Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: obj = create_fn() Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/opt/stack/neutron/neutron/db/l3_db.py", line 255, in _create_router_db Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: registry.publish(resources.ROUTER, events.PRECOMMIT_CREATE, self, Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/neutron_lib/callbacks/registry.py", line 54, in publish Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: _get_callback_manager().publish(resource, event, trigger, payload=payload) Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/utils.py", line 104, in _wrapped Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: return function(*args, **kwargs) Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/neutron_lib/callbacks/manager.py", line 150, in publish Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: errors = self._notify_loop(resource, event, trigger, payload) Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/neutron_lib/callbacks/manager.py", line 181, in _notify_loop Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: callback(resource, event, trigger, payload=payload) Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/opt/stack/neutron/neutron/services/ovn_l3/plugin.py", line 176, in create_router_precommit Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: db_rev.create_initial_revision( Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 226, in wrapped Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: return f_with_retry(*args, **kwargs, Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 138, in wrapped Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: return f(*args, **kwargs) Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/oslo_db/api.py", line 142, in wrapper Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: return f(*args, **kwargs) Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/neutron_lib/db/api.py", line 184, in wrapped Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: return f(*dup_args, **dup_kwargs) Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py", line 112, in create_initial_revision Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: std_attr_id = std_attr_id or _get_standard_attr_id( Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/opt/stack/neutron/neutron/db/ovn_revision_numbers_db.py", line 98, in _get_standard_attr_id Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: id=resource_uuid).one() Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/query.py", line 2869, in one Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: return self._iter().one() Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/query.py", line 2907, in _iter Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: result = self.session.execute( Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: File "/usr/local/lib/python3.10/dist-packages/sqlalchemy/orm/session.py", line 1693, in execute Apr 17 12:12:44.402712 np0033759257 neutron-server[57996]: result = fn(orm_exec_state) The job is not running from last couple of releases due to missconfig caused in [2], the issue noticed when the job was reenabled[3], also looking previous build history it was not green since 08-12-2020[4]. The issue is seen in ovs job too[5] The bug is to track the issue fix. [1] https://docs.openstack.org/neutron/latest//contributor/testing/db_transient_failure_injection.html [2] https://review.opendev.org/c/openstack/neutron/+/797051 [3] https://review.opendev.org/c/openstack/neutron/+/880586 [4] https://zuul.openstack.org/builds?job_name=neutron-tempest-with-uwsgi-loki&result=SUCCESS&skip=0 [5] https://zuul.opendev.org/t/openstack/build/78c1dc4eff424b0c8bdc8b6d1f7dcfc4 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/2017131/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
