Public bug reported:
Neutron multiple workers are enabled as follows in neutron.conf:
- api_workers=3
- rpc_workers=3
The following were configured:
- 20 tenants
- Each tenant had 5 tenant networks
- For each network, one VM at each Compute nodes (2) for a total of 10 VMs
- Total 100 VLANs/200 VMs
A script which did the following at tenant-1:
- Delete all 10 VMs
- For each network, delete its router interface
- Delete the subnet
- Delete the network
- Re-create the network, subnet and router interface
- For each network, launch 2 VMs (one at each Compute node)
- Repeat steps 1 – 6
Intermittently the following delete port error is encountered:
2015-07-06 16:17:51.903 43190 DEBUG neutron.plugins.ml2.plugin
[req-f18af2a1-0047-4301-9fa1-01632fa5b2b8 None] Calling delete_port for
fcf17b5d-235c-466b-b54b-ce80acca7359 owned by network:dhcp delete_p
ort /usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py:1076
2015-07-06 16:17:51.904 43216 ERROR oslo.db.sqlalchemy.exc_filters
[req-cbb23fa8-5043-405c-a569-fcfdc912555a ] DB exception wrapped.
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters Traceback
(most recent call last):
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 781, in
fetchall
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters l =
self.process_rows(self._fetchall_impl())
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 750, in
_fetchall_impl
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters
self._non_result()
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 755, in
_non_result
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters "This
result object does not return rows. "
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters
ResourceClosedError: This result object does not return rows. It has been
closed automatically.
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters
2015-07-06 16:17:51.906 43216 DEBUG neutron.openstack.common.lockutils
[req-cbb23fa8-5043-405c-a569-fcfdc912555a ] Releasing semaphore "db-access"
lock /usr/lib/python2.7/site-packages/neutron/openstack
/common/lockutils.py:238
2015-07-06 16:17:51.906 43216 ERROR neutron.api.v2.resource
[req-cbb23fa8-5043-405c-a569-fcfdc912555a None] delete failed
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource Traceback (most
recent call last):
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib/python2.7/site-packages/neutron/api/v2/resource.py", line 81, in
resource
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource result =
method(request=request, **args)
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 476, in delete
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource
obj_deleter(request.context, id, **kwargs)
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 680, in
delete_network
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource continue
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line
82, in __exit__
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource
six.reraise(self.type_, self.value, self.tb)
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 640, in
delete_network
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource
with_lockmode('update').all())
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2300, in all
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource return
list(self)
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/loading.py", line 66, in
instances
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource fetch =
cursor.fetchall()
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 787, in
fetchall
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource self.cursor,
self.context)
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1156, in
_handle_dbapi_exception
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource
util.raise_from_cause(newraise, exc_info)
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in
raise_from_cause
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource
reraise(type(exception), exception, tb=exc_tb)
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 781, in
fetchall
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource l =
self.process_rows(self._fetchall_impl())
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 750, in
_fetchall_impl
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource
self._non_result()
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 755, in
_non_result
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource "This result
object does not return rows. "
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource DBError: This
result object does not return rows. It has been closed automatically.
As a result, the network is not deleted, and hence duplicate network when the
network is created again with the same name.
** Affects: neutron
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1472347
Title:
With multiple Neutron api/rpc workers enabled, intermittent failure
deleting dhcp_port
Status in OpenStack Neutron (virtual network service):
New
Bug description:
Neutron multiple workers are enabled as follows in neutron.conf:
- api_workers=3
- rpc_workers=3
The following were configured:
- 20 tenants
- Each tenant had 5 tenant networks
- For each network, one VM at each Compute nodes (2) for a total of 10 VMs
- Total 100 VLANs/200 VMs
A script which did the following at tenant-1:
- Delete all 10 VMs
- For each network, delete its router interface
- Delete the subnet
- Delete the network
- Re-create the network, subnet and router interface
- For each network, launch 2 VMs (one at each Compute node)
- Repeat steps 1 – 6
Intermittently the following delete port error is encountered:
2015-07-06 16:17:51.903 43190 DEBUG neutron.plugins.ml2.plugin
[req-f18af2a1-0047-4301-9fa1-01632fa5b2b8 None] Calling delete_port for
fcf17b5d-235c-466b-b54b-ce80acca7359 owned by network:dhcp delete_p
ort /usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py:1076
2015-07-06 16:17:51.904 43216 ERROR oslo.db.sqlalchemy.exc_filters
[req-cbb23fa8-5043-405c-a569-fcfdc912555a ] DB exception wrapped.
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters Traceback
(most recent call last):
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 781, in
fetchall
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters l =
self.process_rows(self._fetchall_impl())
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 750, in
_fetchall_impl
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters
self._non_result()
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 755, in
_non_result
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters "This
result object does not return rows. "
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters
ResourceClosedError: This result object does not return rows. It has been
closed automatically.
2015-07-06 16:17:51.904 43216 TRACE oslo.db.sqlalchemy.exc_filters
2015-07-06 16:17:51.906 43216 DEBUG neutron.openstack.common.lockutils
[req-cbb23fa8-5043-405c-a569-fcfdc912555a ] Releasing semaphore "db-access"
lock /usr/lib/python2.7/site-packages/neutron/openstack
/common/lockutils.py:238
2015-07-06 16:17:51.906 43216 ERROR neutron.api.v2.resource
[req-cbb23fa8-5043-405c-a569-fcfdc912555a None] delete failed
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource Traceback (most
recent call last):
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib/python2.7/site-packages/neutron/api/v2/resource.py", line 81, in
resource
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource result =
method(request=request, **args)
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 476, in delete
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource
obj_deleter(request.context, id, **kwargs)
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 680, in
delete_network
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource continue
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib/python2.7/site-packages/neutron/openstack/common/excutils.py", line
82, in __exit__
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource
six.reraise(self.type_, self.value, self.tb)
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 640, in
delete_network
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource
with_lockmode('update').all())
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2300, in all
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource return
list(self)
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/loading.py", line 66, in
instances
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource fetch =
cursor.fetchall()
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 787, in
fetchall
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource self.cursor,
self.context)
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1156, in
_handle_dbapi_exception
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource
util.raise_from_cause(newraise, exc_info)
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in
raise_from_cause
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource
reraise(type(exception), exception, tb=exc_tb)
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 781, in
fetchall
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource l =
self.process_rows(self._fetchall_impl())
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 750, in
_fetchall_impl
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource
self._non_result()
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/result.py", line 755, in
_non_result
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource "This result
object does not return rows. "
2015-07-06 16:17:51.906 43216 TRACE neutron.api.v2.resource DBError: This
result object does not return rows. It has been closed automatically.
As a result, the network is not deleted, and hence duplicate network when the
network is created again with the same name.
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1472347/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
