** No longer affects: nova
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1394351
Title:
deadlock when delete port
Status in OpenStack Neutron (virtual network service):
Confirmed
Bug description:
netdemoid=$(neutron net-list | awk '{if($4=="'demo-net'"){print $2;}}')
subnetdemoid=$(neutron subnet-list | awk '{if($4=="'demo-subnet'"){print
$2;}}')
exnetid=$(neutron net-list | awk '{if($4=="'ext-net'"){print $2;}}')
for i in `seq 1 10`; do
#boot vm, and create floating ip
nova boot --image cirros --flavor m1.tiny --nic net-id=$netdemoid
cirrosdemo${i}
cirrosdemoid[i]=$(nova list | awk '{if($4=="'cirrosdemo${i}'"){print
$2;}}')
output=$(neutron floatingip-create $exnetid)
echo $output
floatipid[i]=$(echo "$output" | awk '{if($2=="id"){print $4;}}')
floatip[i]=$(echo "$output" | awk '{if($2=="floating_ip_address"){print
$4;}}')a
done
# Setup router
neutron router-gateway-set $routerdemoid $exnetid
neutron router-interface-add demo-router $subnetdemoid
#wait for VM to be running
sleep 30
for i in `seq 1 10`; do
cirrosfix=$(nova list | awk '{if($4=="'cirrosdemo${i}'"){print $12;}}')
cirrosfixip=${cirrosfix#*=}
output=$(neutron port-list | grep ${cirrosfixip})
echo $output
portid=$(echo "$output" | awk '{print $2;}')
neutron floatingip-associate --fixed-ip-address $cirrosfixip
${floatipid[i]} $portid
neutron floatingip-delete ${floatipid[i]}
nova delete ${cirrosdemoid[i]}
done
With several tries, I have one instance in ERROR state:
2014-11-19 19:41:02.670 8659 DEBUG neutron.context
[req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 None] Arguments dropped when creating
context: {u'project_name': None, u'tenant': None} __init__
/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/context.py:83
2014-11-19 19:41:02.671 8659 DEBUG neutron.plugins.ml2.rpc
[req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 None] Device
498e7a54-22dd-4e5b-a8db-d6bffb8edd25 details requested by agent
ovs-agent-overcloud-controller0-d5wwhbhhtlmp with host
overcloud-controller0-d5wwhbhhtlmp get_device_details
/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py:90
2014-11-19 19:41:02.707 8659 DEBUG neutron.openstack.common.lockutils
[req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 None] Got semaphore "db-access" lock
/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/openstack/common/lockutils.py:168
2014-11-19 19:41:04.061 8658 ERROR oslo.messaging.rpc.dispatcher
[req-4303cd41-c87c-44aa-b78a-549fb914ac9c ] Exception during message handling:
(OperationalError) (1213, 'Deadlock found when trying to get lock; try
restarting transaction') None None
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher Traceback
(most recent call last):
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py",
line 134, in _dispatch_and_reply
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher
incoming.message))
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py",
line 177, in _dispatch
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher return
self._do_dispatch(endpoint, method, ctxt, args)
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py",
line 123, in _do_dispatch
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher result =
getattr(endpoint, method)(ctxt, **new_args)
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/db/agents_db.py",
line 220, in report_state
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher
self.plugin.create_or_update_agent(context, agent_state)
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/db/agents_db.py",
line 180, in create_or_update_agent
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher return
self._create_or_update_agent(context, agent)
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/db/agents_db.py",
line 174, in _create_or_update_agent
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher
greenthread.sleep(0)
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 470, in __exit__
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher
self.rollback()
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py",
line 60, in __exit__
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher
compat.reraise(exc_type, exc_value, exc_tb)
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 467, in __exit__
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher
self.commit()
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 381, in commit
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher
t[1].commit()
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1334, in commit
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher
self._do_commit()
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1365, in _do_commit
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher
self.connection._commit_impl()
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 527, in _commit_impl
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher
self._handle_dbapi_exception(e, None, None, None, None)
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1160, in _handle_dbapi_exception
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher exc_info
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py",
line 199, in raise_from_cause
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher
reraise(type(exception), exception, tb=exc_tb)
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 525, in _commit_impl
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher
self.engine.dialect.do_commit(self.connection)
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py",
line 2251, in do_commit
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher
dbapi_connection.commit()
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher
OperationalError: (OperationalError) (1213, 'Deadlock found when trying to get
lock; try restarting transaction') None None
2014-11-19 19:41:04.061 8658 TRACE oslo.messaging.rpc.dispatcher
2014-11-19 19:41:04.071 8659 ERROR oslo.messaging.rpc.dispatcher
[req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 ] Exception during message handling:
(OperationalError) (1213, 'Deadlock found when trying to get lock; try
restarting transaction') None None
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher Traceback
(most recent call last):
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py",
line 134, in _dispatch_and_reply
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher
incoming.message))
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py",
line 177, in _dispatch
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher return
self._do_dispatch(endpoint, method, ctxt, args)
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py",
line 123, in _do_dispatch
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher result =
getattr(endpoint, method)(ctxt, **new_args)
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py",
line 145, in get_devices_details_list
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher for
device in kwargs.pop('devices', [])
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py",
line 122, in get_device_details
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher host)
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py",
line 1146, in update_port_status
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher updated
= True
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher
self.gen.next()
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.7/contextlib.py", line 121, in nested
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher if
exit(*exc):
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 470, in __exit__
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher
self.rollback()
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py",
line 60, in __exit__
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher
compat.reraise(exc_type, exc_value, exc_tb)
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 467, in __exit__
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher
self.commit()
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 381, in commit
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher
t[1].commit()
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1334, in commit
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher
self._do_commit()
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1365, in _do_commit
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher
self.connection._commit_impl()
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 527, in _commit_impl
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher
self._handle_dbapi_exception(e, None, None, None, None)
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1160, in _handle_dbapi_exception
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher exc_info
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py",
line 199, in raise_from_cause
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher
reraise(type(exception), exception, tb=exc_tb)
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 525, in _commit_impl
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher
self.engine.dialect.do_commit(self.connection)
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py",
line 2251, in do_commit
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher
dbapi_connection.commit()
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher
OperationalError: (OperationalError) (1213, 'Deadlock found when trying to get
lock; try restarting transaction') None None
2014-11-19 19:41:04.071 8659 TRACE oslo.messaging.rpc.dispatcher
2014-11-19 19:41:04.084 8659 ERROR oslo.messaging._drivers.common
[req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 ] Returning exception
(OperationalError) (1213, 'Deadlock found when trying to get lock; try
restarting transaction') None None to caller
2014-11-19 19:41:04.084 8659 ERROR oslo.messaging._drivers.common
[req-3ff9aed1-e5fb-4388-b26d-e35bb7fc25f7 ] ['Traceback (most recent call
last):\n', ' File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py",
line 134, in _dispatch_and_reply\n incoming.message))\n', ' File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py",
line 177, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt,
args)\n', ' File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py",
line 123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt,
**new_args)\n', ' File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py",
line 145, in get_devices_details_list\n for device in
kwargs.pop(\'devices\', [])\n', ' File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/rpc.py",
line 12
2, in get_device_details\n host)\n', ' File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/neutron/plugins/ml2/plugin.py",
line 1146, in update_port_status\n updated = True\n', ' File
"/usr/lib/python2.7/contextlib.py", line 24, in __exit__\n
self.gen.next()\n', ' File "/usr/lib/python2.7/contextlib.py", line 121, in
nested\n if exit(*exc):\n', ' File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 470, in __exit__\n self.rollback()\n', ' File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py",
line 60, in __exit__\n compat.reraise(exc_type, exc_value, exc_tb)\n', '
File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 467, in __exit__\n self.commit()\n', ' File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py",
line 381, in commit\n t[1].commit()\n', ' File "/opt/
stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1334, in commit\n self._do_commit()\n', ' File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1365, in _do_commit\n self.connection._commit_impl()\n', ' File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 527, in _commit_impl\n self._handle_dbapi_exception(e, None, None,
None, None)\n', ' File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 1160, in _handle_dbapi_exception\n exc_info\n', ' File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py",
line 199, in raise_from_cause\n reraise(type(exception), exception,
tb=exc_tb)\n', ' File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py",
line 525, in _commit_impl\n self.engine.dialect.do_commit(self.connection)
\n', ' File
"/opt/stack/venvs/openstack/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py",
line 2251, in do_commit\n dbapi_connection.commit()\n', "OperationalError:
(OperationalError) (1213, 'Deadlock found when trying to get lock; try
restarting transaction') None None\n"]
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1394351/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : [email protected]
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
