Public bug reported:
I have multiple controllers and each controllers are running neutron-
server. When I start the neutron-server, some of the controllers
occasionally crash with DBDeadlock exceptions:
2016-08-27 00:32:11.937 193309 CRITICAL neutron [-] DBDeadlock:
(_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get
lock; try restarting transaction') [SQL: u'INSERT INTO ml2_vlan_allocations
(physical_network, vlan_id, allocated) VALUES (%s, %s, %s)'] [parameters:
(('bond0', 1, 0), ('bond0', 2, 0), ('bond0', 3, 0), ('bond0', 4, 0), ('bond0',
5, 0), ('bond0', 6, 0), ('bond0', 7, 0), ('bond0', 8, 0) ... displaying 10 of
4095 total bound parameter sets ... ('bond0', 4094, 0), ('bond0', 4095, 0))]
2016-08-27 00:32:11.937 193309 ERROR neutron Traceback (most recent call last):
2016-08-27 00:32:11.937 193309 ERROR neutron File "/usr/bin/neutron-server",
line 10, in <module>
2016-08-27 00:32:11.937 193309 ERROR neutron sys.exit(main_wsgi_eventlet())
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/cmd/eventlet/server/__init__.py",
line 19, in main_wsgi_eventlet
2016-08-27 00:32:11.937 193309 ERROR neutron wsgi_eventlet.main()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/server/wsgi_eventlet.py", line 50, in
main
2016-08-27 00:32:11.937 193309 ERROR neutron
server.boot_server(_eventlet_wsgi_server)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/server/__init__.py", line 35, in
boot_server
2016-08-27 00:32:11.937 193309 ERROR neutron server_func()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/server/wsgi_eventlet.py", line 27, in
_eventlet_wsgi_server
2016-08-27 00:32:11.937 193309 ERROR neutron neutron_api =
service.serve_wsgi(service.NeutronApiService)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/service.py", line 117, in serve_wsgi
2016-08-27 00:32:11.937 193309 ERROR neutron
LOG.exception(_LE('Unrecoverable error: please check log '
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-08-27 00:32:11.937 193309 ERROR neutron six.reraise(self.type_,
self.value, self.tb)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/service.py", line 114, in serve_wsgi
2016-08-27 00:32:11.937 193309 ERROR neutron service.start()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/service.py", line 87, in start
2016-08-27 00:32:11.937 193309 ERROR neutron self.wsgi_app =
_run_wsgi(self.app_name)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/service.py", line 225, in _run_wsgi
2016-08-27 00:32:11.937 193309 ERROR neutron app =
config.load_paste_app(app_name)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/common/config.py", line 246, in
load_paste_app
2016-08-27 00:32:11.937 193309 ERROR neutron app =
deploy.loadapp("config:%s" % config_path, name=app_name)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in
loadapp
2016-08-27 00:32:11.937 193309 ERROR neutron return loadobj(APP, uri,
name=name, **kw)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in
loadobj
2016-08-27 00:32:11.937 193309 ERROR neutron return context.create()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-08-27 00:32:11.937 193309 ERROR neutron return
self.object_type.invoke(self)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-08-27 00:32:11.937 193309 ERROR neutron **context.local_conf)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2016-08-27 00:32:11.937 193309 ERROR neutron val = callable(*args, **kw)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/urlmap.py", line 25, in urlmap_factory
2016-08-27 00:32:11.937 193309 ERROR neutron app = loader.get_app(app_name,
global_conf=global_conf)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in
get_app
2016-08-27 00:32:11.937 193309 ERROR neutron name=name,
global_conf=global_conf).create()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-08-27 00:32:11.937 193309 ERROR neutron return
self.object_type.invoke(self)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-08-27 00:32:11.937 193309 ERROR neutron **context.local_conf)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2016-08-27 00:32:11.937 193309 ERROR neutron val = callable(*args, **kw)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/auth.py", line 71, in pipeline_factory
2016-08-27 00:32:11.937 193309 ERROR neutron app =
loader.get_app(pipeline[-1])
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in
get_app
2016-08-27 00:32:11.937 193309 ERROR neutron name=name,
global_conf=global_conf).create()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-08-27 00:32:11.937 193309 ERROR neutron return
self.object_type.invoke(self)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2016-08-27 00:32:11.937 193309 ERROR neutron return
fix_call(context.object, context.global_conf, **context.local_conf)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2016-08-27 00:32:11.937 193309 ERROR neutron val = callable(*args, **kw)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 81, in factory
2016-08-27 00:32:11.937 193309 ERROR neutron return cls(**local_config)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 85, in
__init__
2016-08-27 00:32:11.937 193309 ERROR neutron plugin =
manager.NeutronManager.get_plugin()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/manager.py", line 248, in get_plugin
2016-08-27 00:32:11.937 193309 ERROR neutron return
weakref.proxy(cls.get_instance().plugin)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/manager.py", line 242, in get_instance
2016-08-27 00:32:11.937 193309 ERROR neutron cls._create_instance()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 254, in
inner
2016-08-27 00:32:11.937 193309 ERROR neutron return f(*args, **kwargs)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/manager.py", line 228, in
_create_instance
2016-08-27 00:32:11.937 193309 ERROR neutron cls._instance = cls()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/manager.py", line 121, in __init__
2016-08-27 00:32:11.937 193309 ERROR neutron plugin_provider)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/manager.py", line 161, in
_get_plugin_instance
2016-08-27 00:32:11.937 193309 ERROR neutron return plugin_class()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/quota/resource_registry.py", line
121, in wrapper
2016-08-27 00:32:11.937 193309 ERROR neutron return f(*args, **kwargs)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 146, in
__init__
2016-08-27 00:32:11.937 193309 ERROR neutron self.type_manager.initialize()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 182,
in initialize
2016-08-27 00:32:11.937 193309 ERROR neutron driver.obj.initialize()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_vlan.py",
line 167, in initialize
2016-08-27 00:32:11.937 193309 ERROR neutron self._sync_vlan_allocations()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_vlan.py",
line 161, in _sync_vlan_allocations
2016-08-27 00:32:11.937 193309 ERROR neutron session.delete(alloc)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 490, in
__exit__
2016-08-27 00:32:11.937 193309 ERROR neutron self.rollback()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60,
in __exit__
2016-08-27 00:32:11.937 193309 ERROR neutron compat.reraise(exc_type,
exc_value, exc_tb)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 487, in
__exit__
2016-08-27 00:32:11.937 193309 ERROR neutron self.commit()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 392, in
commit
2016-08-27 00:32:11.937 193309 ERROR neutron self._prepare_impl()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 372, in
_prepare_impl
2016-08-27 00:32:11.937 193309 ERROR neutron self.session.flush()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2004, in
flush
2016-08-27 00:32:11.937 193309 ERROR neutron self._flush(objects)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2122, in
_flush
2016-08-27 00:32:11.937 193309 ERROR neutron
transaction.rollback(_capture_exception=True)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60,
in __exit__
2016-08-27 00:32:11.937 193309 ERROR neutron compat.reraise(exc_type,
exc_value, exc_tb)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2086, in
_flush
2016-08-27 00:32:11.937 193309 ERROR neutron flush_context.execute()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in
execute
2016-08-27 00:32:11.937 193309 ERROR neutron rec.execute(self)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in
execute
2016-08-27 00:32:11.937 193309 ERROR neutron uow
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 174,
in save_obj
2016-08-27 00:32:11.937 193309 ERROR neutron mapper, table, insert)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 748,
in _emit_insert_statements
2016-08-27 00:32:11.937 193309 ERROR neutron execute(statement, multiparams)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in
execute
2016-08-27 00:32:11.937 193309 ERROR neutron return meth(self, multiparams,
params)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in
_execute_on_connection
2016-08-27 00:32:11.937 193309 ERROR neutron return
connection._execute_clauseelement(self, multiparams, params)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in
_execute_clauseelement
2016-08-27 00:32:11.937 193309 ERROR neutron compiled_sql, distilled_params
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in
_execute_context
2016-08-27 00:32:11.937 193309 ERROR neutron context)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in
_handle_dbapi_exception
2016-08-27 00:32:11.937 193309 ERROR neutron
util.raise_from_cause(newraise, exc_info)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in
raise_from_cause
2016-08-27 00:32:11.937 193309 ERROR neutron reraise(type(exception),
exception, tb=exc_tb)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1116, in
_execute_context
2016-08-27 00:32:11.937 193309 ERROR neutron context)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line
95, in do_executemany
2016-08-27 00:32:11.937 193309 ERROR neutron rowcount =
cursor.executemany(statement, parameters)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 206, in
executemany
2016-08-27 00:32:11.937 193309 ERROR neutron r = r + self.execute(query, a)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
2016-08-27 00:32:11.937 193309 ERROR neutron self.errorhandler(self, exc,
value)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in
defaulterrorhandler
2016-08-27 00:32:11.937 193309 ERROR neutron raise errorclass, errorvalue
2016-08-27 00:32:11.937 193309 ERROR neutron DBDeadlock:
(_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get
lock; try restarting transaction') [SQL: u'INSERT INTO ml2_vlan_allocations
(physical_network, vlan_id, allocated) VALUES (%s, %s, %s)'] [parameters:
(('bond0', 1, 0), ('bond0', 2, 0), ('bond0', 3, 0), ('bond0', 4, 0), ('bond0',
5, 0), ('bond0', 6, 0), ('bond0', 7, 0), ('bond0', 8, 0) ... displaying 10 of
4095 total bound parameter sets ... ('bond0', 4094, 0), ('bond0', 4095, 0))]
The problem occurs more oftern under this following situation:
1. network_vlan_ranges is large (My condition is from 1 to 4095)
2. slow controller
** Affects: neutron
Importance: Undecided
Assignee: Kahou Lei (kahou82)
Status: New
** Changed in: neutron
Assignee: (unassigned) => Kahou Lei (kahou82)
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1617499
Title:
DBDeadlock occurs when multiple neutron server starts at the same time
Status in neutron:
New
Bug description:
I have multiple controllers and each controllers are running neutron-
server. When I start the neutron-server, some of the controllers
occasionally crash with DBDeadlock exceptions:
2016-08-27 00:32:11.937 193309 CRITICAL neutron [-] DBDeadlock:
(_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get
lock; try restarting transaction') [SQL: u'INSERT INTO ml2_vlan_allocations
(physical_network, vlan_id, allocated) VALUES (%s, %s, %s)'] [parameters:
(('bond0', 1, 0), ('bond0', 2, 0), ('bond0', 3, 0), ('bond0', 4, 0), ('bond0',
5, 0), ('bond0', 6, 0), ('bond0', 7, 0), ('bond0', 8, 0) ... displaying 10 of
4095 total bound parameter sets ... ('bond0', 4094, 0), ('bond0', 4095, 0))]
2016-08-27 00:32:11.937 193309 ERROR neutron Traceback (most recent call
last):
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/bin/neutron-server", line 10, in <module>
2016-08-27 00:32:11.937 193309 ERROR neutron
sys.exit(main_wsgi_eventlet())
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/cmd/eventlet/server/__init__.py",
line 19, in main_wsgi_eventlet
2016-08-27 00:32:11.937 193309 ERROR neutron wsgi_eventlet.main()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/server/wsgi_eventlet.py", line 50, in
main
2016-08-27 00:32:11.937 193309 ERROR neutron
server.boot_server(_eventlet_wsgi_server)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/server/__init__.py", line 35, in
boot_server
2016-08-27 00:32:11.937 193309 ERROR neutron server_func()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/server/wsgi_eventlet.py", line 27, in
_eventlet_wsgi_server
2016-08-27 00:32:11.937 193309 ERROR neutron neutron_api =
service.serve_wsgi(service.NeutronApiService)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/service.py", line 117, in serve_wsgi
2016-08-27 00:32:11.937 193309 ERROR neutron
LOG.exception(_LE('Unrecoverable error: please check log '
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-08-27 00:32:11.937 193309 ERROR neutron six.reraise(self.type_,
self.value, self.tb)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/service.py", line 114, in serve_wsgi
2016-08-27 00:32:11.937 193309 ERROR neutron service.start()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/service.py", line 87, in start
2016-08-27 00:32:11.937 193309 ERROR neutron self.wsgi_app =
_run_wsgi(self.app_name)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/service.py", line 225, in _run_wsgi
2016-08-27 00:32:11.937 193309 ERROR neutron app =
config.load_paste_app(app_name)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/common/config.py", line 246, in
load_paste_app
2016-08-27 00:32:11.937 193309 ERROR neutron app =
deploy.loadapp("config:%s" % config_path, name=app_name)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 247, in
loadapp
2016-08-27 00:32:11.937 193309 ERROR neutron return loadobj(APP, uri,
name=name, **kw)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 272, in
loadobj
2016-08-27 00:32:11.937 193309 ERROR neutron return context.create()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-08-27 00:32:11.937 193309 ERROR neutron return
self.object_type.invoke(self)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-08-27 00:32:11.937 193309 ERROR neutron **context.local_conf)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2016-08-27 00:32:11.937 193309 ERROR neutron val = callable(*args, **kw)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/urlmap.py", line 25, in urlmap_factory
2016-08-27 00:32:11.937 193309 ERROR neutron app =
loader.get_app(app_name, global_conf=global_conf)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in
get_app
2016-08-27 00:32:11.937 193309 ERROR neutron name=name,
global_conf=global_conf).create()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-08-27 00:32:11.937 193309 ERROR neutron return
self.object_type.invoke(self)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 144, in invoke
2016-08-27 00:32:11.937 193309 ERROR neutron **context.local_conf)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2016-08-27 00:32:11.937 193309 ERROR neutron val = callable(*args, **kw)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/auth.py", line 71, in pipeline_factory
2016-08-27 00:32:11.937 193309 ERROR neutron app =
loader.get_app(pipeline[-1])
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 350, in
get_app
2016-08-27 00:32:11.937 193309 ERROR neutron name=name,
global_conf=global_conf).create()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 710, in create
2016-08-27 00:32:11.937 193309 ERROR neutron return
self.object_type.invoke(self)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/loadwsgi.py", line 146, in invoke
2016-08-27 00:32:11.937 193309 ERROR neutron return
fix_call(context.object, context.global_conf, **context.local_conf)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/paste/deploy/util.py", line 55, in fix_call
2016-08-27 00:32:11.937 193309 ERROR neutron val = callable(*args, **kw)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 81, in factory
2016-08-27 00:32:11.937 193309 ERROR neutron return cls(**local_config)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/api/v2/router.py", line 85, in
__init__
2016-08-27 00:32:11.937 193309 ERROR neutron plugin =
manager.NeutronManager.get_plugin()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/manager.py", line 248, in get_plugin
2016-08-27 00:32:11.937 193309 ERROR neutron return
weakref.proxy(cls.get_instance().plugin)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/manager.py", line 242, in get_instance
2016-08-27 00:32:11.937 193309 ERROR neutron cls._create_instance()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 254, in
inner
2016-08-27 00:32:11.937 193309 ERROR neutron return f(*args, **kwargs)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/manager.py", line 228, in
_create_instance
2016-08-27 00:32:11.937 193309 ERROR neutron cls._instance = cls()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/manager.py", line 121, in __init__
2016-08-27 00:32:11.937 193309 ERROR neutron plugin_provider)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/manager.py", line 161, in
_get_plugin_instance
2016-08-27 00:32:11.937 193309 ERROR neutron return plugin_class()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/quota/resource_registry.py", line
121, in wrapper
2016-08-27 00:32:11.937 193309 ERROR neutron return f(*args, **kwargs)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py", line 146, in
__init__
2016-08-27 00:32:11.937 193309 ERROR neutron
self.type_manager.initialize()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/managers.py", line 182,
in initialize
2016-08-27 00:32:11.937 193309 ERROR neutron driver.obj.initialize()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_vlan.py",
line 167, in initialize
2016-08-27 00:32:11.937 193309 ERROR neutron self._sync_vlan_allocations()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/type_vlan.py",
line 161, in _sync_vlan_allocations
2016-08-27 00:32:11.937 193309 ERROR neutron session.delete(alloc)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 490, in
__exit__
2016-08-27 00:32:11.937 193309 ERROR neutron self.rollback()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60,
in __exit__
2016-08-27 00:32:11.937 193309 ERROR neutron compat.reraise(exc_type,
exc_value, exc_tb)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 487, in
__exit__
2016-08-27 00:32:11.937 193309 ERROR neutron self.commit()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 392, in
commit
2016-08-27 00:32:11.937 193309 ERROR neutron self._prepare_impl()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 372, in
_prepare_impl
2016-08-27 00:32:11.937 193309 ERROR neutron self.session.flush()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2004, in
flush
2016-08-27 00:32:11.937 193309 ERROR neutron self._flush(objects)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2122, in
_flush
2016-08-27 00:32:11.937 193309 ERROR neutron
transaction.rollback(_capture_exception=True)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60,
in __exit__
2016-08-27 00:32:11.937 193309 ERROR neutron compat.reraise(exc_type,
exc_value, exc_tb)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 2086, in
_flush
2016-08-27 00:32:11.937 193309 ERROR neutron flush_context.execute()
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 373, in
execute
2016-08-27 00:32:11.937 193309 ERROR neutron rec.execute(self)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/unitofwork.py", line 532, in
execute
2016-08-27 00:32:11.937 193309 ERROR neutron uow
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 174,
in save_obj
2016-08-27 00:32:11.937 193309 ERROR neutron mapper, table, insert)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/orm/persistence.py", line 748,
in _emit_insert_statements
2016-08-27 00:32:11.937 193309 ERROR neutron execute(statement,
multiparams)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 914, in
execute
2016-08-27 00:32:11.937 193309 ERROR neutron return meth(self,
multiparams, params)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 323, in
_execute_on_connection
2016-08-27 00:32:11.937 193309 ERROR neutron return
connection._execute_clauseelement(self, multiparams, params)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1010, in
_execute_clauseelement
2016-08-27 00:32:11.937 193309 ERROR neutron compiled_sql,
distilled_params
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in
_execute_context
2016-08-27 00:32:11.937 193309 ERROR neutron context)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1337, in
_handle_dbapi_exception
2016-08-27 00:32:11.937 193309 ERROR neutron
util.raise_from_cause(newraise, exc_info)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in
raise_from_cause
2016-08-27 00:32:11.937 193309 ERROR neutron reraise(type(exception),
exception, tb=exc_tb)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1116, in
_execute_context
2016-08-27 00:32:11.937 193309 ERROR neutron context)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/sqlalchemy/dialects/mysql/mysqldb.py", line
95, in do_executemany
2016-08-27 00:32:11.937 193309 ERROR neutron rowcount =
cursor.executemany(statement, parameters)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 206, in
executemany
2016-08-27 00:32:11.937 193309 ERROR neutron r = r + self.execute(query,
a)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/MySQLdb/cursors.py", line 174, in execute
2016-08-27 00:32:11.937 193309 ERROR neutron self.errorhandler(self, exc,
value)
2016-08-27 00:32:11.937 193309 ERROR neutron File
"/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 36, in
defaulterrorhandler
2016-08-27 00:32:11.937 193309 ERROR neutron raise errorclass, errorvalue
2016-08-27 00:32:11.937 193309 ERROR neutron DBDeadlock:
(_mysql_exceptions.OperationalError) (1213, 'Deadlock found when trying to get
lock; try restarting transaction') [SQL: u'INSERT INTO ml2_vlan_allocations
(physical_network, vlan_id, allocated) VALUES (%s, %s, %s)'] [parameters:
(('bond0', 1, 0), ('bond0', 2, 0), ('bond0', 3, 0), ('bond0', 4, 0), ('bond0',
5, 0), ('bond0', 6, 0), ('bond0', 7, 0), ('bond0', 8, 0) ... displaying 10 of
4095 total bound parameter sets ... ('bond0', 4094, 0), ('bond0', 4095, 0))]
The problem occurs more oftern under this following situation:
1. network_vlan_ranges is large (My condition is from 1 to 4095)
2. slow controller
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1617499/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
