Public bug reported: With the current master branch of neutron and neutron-vpnaas requests to create VPN ipsec site connections fail with a 500 Internal Server Error. I noticed that in a devstack with vpnaas enabled.
The automated tests (neutron-tempest-plugin-vpnaas) also started to fail. See https://zuul.opendev.org/t/openstack/build/7f680cca45544cec9e3418bda7399324 In the neutron-server log there is an exception (see below). This seems to be related to neutron commit 6b529ea3c559ed1a016e2aa2c30769429192d528 "[sqlalchemy-20] Use session.transaction information to decide if active" https://review.opendev.org/c/openstack/neutron/+/843256 Probably the neutron-vpnaas code needs to be adapted to avoid the error of remove_reservation inside a transaction. Exception log: Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource [None req-6d1d249c-3dba-4be4-996d-fa6b5f1fa718 tempest-VPNaaSTestJSON-706399164 tempest-VPNaaSTestJSON-706399164-project] create failed: No details.: RuntimeError: Method <function remove_reservation at 0x7f8df5b5f940> cannot be called within a transaction. Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource Traceback (most recent call last): Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/resource.py", line 98, in resource Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource result = method(request=request, **args) Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 438, in create Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource return self._create(request, body, **kwargs) Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 142, in wrapped Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True) Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource self.force_reraise() Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource raise self.value Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 138, in wrapped Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource return f(*args, **kwargs) Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 154, in wrapper Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource ectxt.value = e.inner_exc Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource self.force_reraise() Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource raise self.value Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142, in wrapper Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource return f(*args, **kwargs) Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 190, in wrapped Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource context_reference.session.rollback() Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource self.force_reraise() Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource raise self.value Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 184, in wrapped Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs) Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 563, in _create Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource return notify({self._resource: self._view(request.context, Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 504, in notify Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource quota.QUOTAS.commit_reservation( Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/quota/__init__.py", line 103, in commit_reservation Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource self.get_driver().commit_reservation(context, reservation_id) Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/quota/driver.py", line 271, in commit_reservation Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource quota_api.remove_reservation(context, reservation_id, Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/common/utils.py", line 700, in inner Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource raise RuntimeError(_("Method %s cannot be called within a " Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource RuntimeError: Method <function remove_reservation at 0x7f8df5b5f940> cannot be called within a transaction. Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource Jun 12 08:46:58.912701 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: INFO neutron.wsgi [None req-6d1d249c-3dba-4be4-996d-fa6b5f1fa718 tempest-VPNaaSTestJSON-706399164 tempest-VPNaaSTestJSON-706399164-project] 149.202.162.36,149.202.162.36 "POST /v2.0/vpn/ipsec-site-connections HTTP/1.1" status: 500 len: 368 time: 0.2223299 ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1978571 Title: VPNaaS: creating ipsec site connection fails (in master) Status in neutron: New Bug description: With the current master branch of neutron and neutron-vpnaas requests to create VPN ipsec site connections fail with a 500 Internal Server Error. I noticed that in a devstack with vpnaas enabled. The automated tests (neutron-tempest-plugin-vpnaas) also started to fail. See https://zuul.opendev.org/t/openstack/build/7f680cca45544cec9e3418bda7399324 In the neutron-server log there is an exception (see below). This seems to be related to neutron commit 6b529ea3c559ed1a016e2aa2c30769429192d528 "[sqlalchemy-20] Use session.transaction information to decide if active" https://review.opendev.org/c/openstack/neutron/+/843256 Probably the neutron-vpnaas code needs to be adapted to avoid the error of remove_reservation inside a transaction. Exception log: Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource [None req-6d1d249c-3dba-4be4-996d-fa6b5f1fa718 tempest-VPNaaSTestJSON-706399164 tempest-VPNaaSTestJSON-706399164-project] create failed: No details.: RuntimeError: Method <function remove_reservation at 0x7f8df5b5f940> cannot be called within a transaction. Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource Traceback (most recent call last): Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/resource.py", line 98, in resource Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource result = method(request=request, **args) Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 438, in create Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource return self._create(request, body, **kwargs) Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 142, in wrapped Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource setattr(e, '_RETRY_EXCEEDED', True) Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource self.force_reraise() Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource raise self.value Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 138, in wrapped Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource return f(*args, **kwargs) Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 154, in wrapper Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource ectxt.value = e.inner_exc Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource self.force_reraise() Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource raise self.value Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_db/api.py", line 142, in wrapper Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource return f(*args, **kwargs) Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 190, in wrapped Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource context_reference.session.rollback() Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource self.force_reraise() Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource raise self.value Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/usr/local/lib/python3.8/dist-packages/neutron_lib/db/api.py", line 184, in wrapped Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource return f(*dup_args, **dup_kwargs) Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 563, in _create Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource return notify({self._resource: self._view(request.context, Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/api/v2/base.py", line 504, in notify Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource quota.QUOTAS.commit_reservation( Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/quota/__init__.py", line 103, in commit_reservation Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource self.get_driver().commit_reservation(context, reservation_id) Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/db/quota/driver.py", line 271, in commit_reservation Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource quota_api.remove_reservation(context, reservation_id, Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource File "/opt/stack/neutron/neutron/common/utils.py", line 700, in inner Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource raise RuntimeError(_("Method %s cannot be called within a " Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource RuntimeError: Method <function remove_reservation at 0x7f8df5b5f940> cannot be called within a transaction. Jun 12 08:46:58.908642 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: ERROR neutron.api.v2.resource Jun 12 08:46:58.912701 ubuntu-focal-ovh-gra1-0029982028 neutron-server[74088]: INFO neutron.wsgi [None req-6d1d249c-3dba-4be4-996d-fa6b5f1fa718 tempest-VPNaaSTestJSON-706399164 tempest-VPNaaSTestJSON-706399164-project] 149.202.162.36,149.202.162.36 "POST /v2.0/vpn/ipsec-site-connections HTTP/1.1" status: 500 len: 368 time: 0.2223299 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1978571/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
