Hi, all.
I hit an error about ovs agents state report after deployed openstack by chef.
I check the code in agents_db. Found there are some greenthread.sleep(0) in
_create_or_update_agent (
https://github.com/openstack/neutron/commit/21bf6f7e4945bc7e8c303273ad10c28b6cfc8b08
). I'm not sure why we need those sleep here. But I think this can let the
user hit DBDuplicateEntry excetpion(there already have an retry to avoid this
error, but it still not safe, see
https://github.com/openstack/neutron/blob/ebc3ccfdd070fb69274069057c7c1cd8f37b508a/neutron/db/agents_db.py#L198
). This error is not easy to reproduce. I think if we make several rpc calls
for create_or_update_agent at the same time before this agents has already been
stored in db, we can encounter this error again.
Remove those sleep can avoid this issue, but seems those sleep is used to fix
some bugs. Need someone's help to fix this issue.
2014-11-10 23:14:36.050 31989 ERROR oslo.messaging.rpc.dispatcher
[req-e07a7ed7-d22f-40e9-b299-aba6e06f9470 ] Exception during message handling:
(IntegrityError) ibm_db_dbi::IntegrityError: Statement Execute Failed:
[IBM][CLI Driver][DB2/LINUXX8664] SQL0803N One or more values in the INSERT
statement, UPDATE statement, or foreign key update caused by a DELETE statement
are not valid because the primary key, unique constraint or unique index
identified by "2" constrains table "NEUTRON.AGENTS" from having duplicate
values for the index key. SQLSTATE=23505 SQLCODE=-803 'INSERT INTO agents (id,
agent_type, "binary", topic, host, admin_state_up, created_at, started_at,
heartbeat_timestamp, description, configurations) VALUES (?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?)' ('5028b520-1a90-46c2-9e97-b1cb80d6ebe0', 'Open vSwitch agent',
'neutron-openvswitch-agent', 'N/A', 'testhl12.sce.ibm.com', '1',
datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), datetime.datetime(2014, 11,
11, 4, 14, 35, 580590), datetime.datetime(2014, 11, 11, 4, 14, 35, 580590),
None, '{"arp_responder_enabled": false, "tunneling_ip": "10.11.1.12",
"devices": 0, "l2_population": false, "tunnel_types": ["gre", "vxlan"],
"enable_distributed_routing": false, "bridge_mappings": {"default":
"br-eth1"}}')
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher Traceback
(most recent call last):
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134,
in _dispatch_and_reply
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher
incoming.message))
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 177,
in _dispatch
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher return
self._do_dispatch(endpoint, method, ctxt, args)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 123,
in _do_dispatch
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher result =
getattr(endpoint, method)(ctxt, **new_args)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 237, in
report_state
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher
self.plugin.create_or_update_agent(context, agent_state)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 214, in
create_or_update_agent
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher return
self._create_or_update_agent(context, agent)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/neutron/openstack/common/excutils.py", line
82, in __exit__
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 197, in
create_or_update_agent
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher return
self._create_or_update_agent(context, agent)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 191, in
_create_or_update_agent
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher
greenthread.sleep(0)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 447, in
__exit__
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher
self.rollback()
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/sqlalchemy/util/langhelpers.py", line 58,
in __exit__
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher
compat.reraise(exc_type, exc_value, exc_tb)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 444, in
__exit__
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher self.commit()
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 354, in
commit
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher
self._prepare_impl()
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 334, in
_prepare_impl
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher
self.session.flush()
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1818, in
flush
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher
self._flush(objects)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1936, in
_flush
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher
transaction.rollback(_capture_exception=True)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/sqlalchemy/util/langhelpers.py", line 58,
in __exit__
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher
compat.reraise(exc_type, exc_value, exc_tb)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1900, in
_flush
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher
flush_context.execute()
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 372, in
execute
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher
rec.execute(self)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 525, in
execute
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher uow
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 64, in
save_obj
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher table,
insert)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 569,
in _emit_insert_statements
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher
execute(statement, params)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 662, in
execute
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher params)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 761, in
_execute_clauseelement
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher compiled_sql,
distilled_params
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 874, in
_execute_context
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher context)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/compat/handle_error.py",
line 125, in _handle_dbapi_exception
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher
six.reraise(type(newraise), newraise, sys.exc_info()[2])
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/compat/handle_error.py",
line 102, in _handle_dbapi_exception
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher per_fn =
fn(ctx)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/exc_filters.py", line 323,
in handler
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher
context.is_disconnect)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/exc_filters.py", line 223,
in _db2_dupe_key_error
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher raise
exception.DBDuplicateEntry([], integrity_error)
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher
DBDuplicateEntry: (IntegrityError) ibm_db_dbi::IntegrityError: Statement
Execute Failed: [IBM][CLI Driver][DB2/LINUXX8664] SQL0803N One or more values
in the INSERT statement, UPDATE statement, or foreign key update caused by a
DELETE statement are not valid because the primary key, unique constraint or
unique index identified by "2" constrains table "NEUTRON.AGENTS" from having
duplicate values for the index key. SQLSTATE=23505 SQLCODE=-803 'INSERT INTO
agents (id, agent_type, "binary", topic, host, admin_state_up, created_at,
started_at, heartbeat_timestamp, description, configurations) VALUES (?, ?, ?,
?, ?, ?, ?, ?, ?, ?, ?)' ('5028b520-1a90-46c2-9e97-b1cb80d6ebe0', 'Open vSwitch
agent', 'neutron-openvswitch-agent', 'N/A', 'testhl12.sce.ibm.com', '1',
datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), datetime.datetime(2014, 11,
11, 4, 14, 35, 580590), datetime.datetime(2014, 11, 11, 4, 14, 35, 580590),
None, '{"arp_responder_enabled": false, "tunneling_ip": "10.11.1.12",
"devices": 0, "l2_population": false, "tunnel_types": ["gre", "vxlan"],
"enable_distributed_routing": false, "bridge_mappings": {"default":
"br-eth1"}}')
2014-11-10 23:14:36.050 31989 TRACE oslo.messaging.rpc.dispatcher
2014-11-10 23:14:36.064 31989 ERROR oslo.messaging._drivers.common
[req-e07a7ed7-d22f-40e9-b299-aba6e06f9470 ] Returning exception
(IntegrityError) ibm_db_dbi::IntegrityError: Statement Execute Failed:
[IBM][CLI Driver][DB2/LINUXX8664] SQL0803N One or more values in the INSERT
statement, UPDATE statement, or foreign key update caused by a DELETE statement
are not valid because the primary key, unique constraint or unique index
identified by "2" constrains table "NEUTRON.AGENTS" from having duplicate
values for the index key. SQLSTATE=23505 SQLCODE=-803 'INSERT INTO agents (id,
agent_type, "binary", topic, host, admin_state_up, created_at, started_at,
heartbeat_timestamp, description, configurations) VALUES (?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?)' ('5028b520-1a90-46c2-9e97-b1cb80d6ebe0', 'Open vSwitch agent',
'neutron-openvswitch-agent', 'N/A', 'testhl12.sce.ibm.com', '1',
datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), datetime.datetime(2014, 11,
11, 4, 14, 35, 580590), datetime.datetime(2014, 11, 11, 4, 14, 35, 580590),
None, '{"arp_responder_enabled": false, "tunneling_ip": "10.11.1.12",
"devices": 0, "l2_population": false, "tunnel_types": ["gre", "vxlan"],
"enable_distributed_routing": false, "bridge_mappings": {"default":
"br-eth1"}}') to caller
2014-11-10 23:14:36.064 31989 ERROR oslo.messaging._drivers.common
[req-e07a7ed7-d22f-40e9-b299-aba6e06f9470 ] ['Traceback (most recent call
last):\n', ' File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134,
in _dispatch_and_reply\n incoming.message))\n', ' File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 177,
in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', '
File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line
123, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n',
' File "/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 237, in
report_state\n self.plugin.create_or_update_agent(context, agent_state)\n', '
File "/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 214, in
create_or_update_agent\n return self._create_or_update_agent(context,
agent)\n', ' File
"/usr/lib/python2.6/site-packages/neutron/openstack/common/excutils.py", line
82, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File
"/usr/lib/python2.6/site-packages/neutron/db/agents_db.py", line 197, in
create_or_update_agent\n return self._create_or_update_agent(context,
agent)\n', ' File "/usr/lib/python2.6/site-packages/neutron/db/agents_db.py",
line 191, in _create_or_update_agent\n greenthread.sleep(0)\n', ' File
"/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 447, in
__exit__\n self.rollback()\n', ' File
"/usr/lib64/python2.6/site-packages/sqlalchemy/util/langhelpers.py", line 58,
in __exit__\n compat.reraise(exc_type, exc_value, exc_tb)\n', ' File
"/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 444, in
__exit__\n self.commit()\n', ' File
"/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 354, in
commit\n self._prepare_impl()\n', ' File
"/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 334, in
_prepare_impl\n self.session.flush()\n', ' File
"/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1818, in
flush\n self._flush(objects)\n', ' File
"/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1936, in
_flush\n transaction.rollback(_capture_exception=True)\n', ' File
"/usr/lib64/python2.6/site-packages/sqlalchemy/util/langhelpers.py", line 58,
in __exit__\n compat.reraise(exc_type, exc_value, exc_tb)\n', ' File
"/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 1900, in
_flush\n flush_context.execute()\n', ' File
"/usr/lib64/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 372, in
execute\n rec.execute(self)\n', ' File
"/usr/lib64/python2.6/site-packages/sqlalchemy/orm/unitofwork.py", line 525, in
execute\n uow\n', ' File
"/usr/lib64/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 64, in
save_obj\n table, insert)\n', ' File
"/usr/lib64/python2.6/site-packages/sqlalchemy/orm/persistence.py", line 569,
in _emit_insert_statements\n execute(statement, params)\n', ' File
"/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 662, in
execute\n params)\n', ' File
"/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 761, in
_execute_clauseelement\n compiled_sql, distilled_params\n', ' File
"/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 874, in
_execute_context\n context)\n', ' File
"/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/compat/handle_error.py",
line 125, in _handle_dbapi_exception\n six.reraise(type(newraise), newraise,
sys.exc_info()[2])\n', ' File
"/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/compat/handle_error.py",
line 102, in _handle_dbapi_exception\n per_fn = fn(ctx)\n', ' File
"/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/exc_filters.py", line 323,
in handler\n context.is_disconnect)\n', ' File
"/usr/lib/python2.6/site-packages/oslo/db/sqlalchemy/exc_filters.py", line 223,
in _db2_dupe_key_error\n raise exception.DBDuplicateEntry([],
integrity_error)\n', 'DBDuplicateEntry: (IntegrityError)
ibm_db_dbi::IntegrityError: Statement Execute Failed: [IBM][CLI
Driver][DB2/LINUXX8664] SQL0803N One or more values in the INSERT statement,
UPDATE statement, or foreign key update caused by a DELETE statement are not
valid because the primary key, unique constraint or unique index identified by
"2" constrains table "NEUTRON.AGENTS" from having duplicate values for the
index key. SQLSTATE=23505 SQLCODE=-803 \'INSERT INTO agents (id, agent_type,
"binary", topic, host, admin_state_up, created_at, started_at,
heartbeat_timestamp, description, configurations) VALUES (?, ?, ?, ?, ?, ?, ?,
?, ?, ?, ?)\' (\'5028b520-1a90-46c2-9e97-b1cb80d6ebe0\', \'Open vSwitch
agent\', \'neutron-openvswitch-agent\', \'N/A\', \'testhl12.sce.ibm.com\',
\'1\', datetime.datetime(2014, 11, 11, 4, 14, 35, 580590),
datetime.datetime(2014, 11, 11, 4, 14, 35, 580590), datetime.datetime(2014, 11,
11, 4, 14, 35, 580590), None, \'{"arp_responder_enabled": false,
"tunneling_ip": "10.11.1.12", "devices": 0, "l2_population": false,
"tunnel_types": ["gre", "vxlan"], "enable_distributed_routing": false,
"bridge_mappings": {"default": "br-eth1"}}\')\n']
[email protected]
_______________________________________________
OpenStack-dev mailing list
[email protected]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
