Reviewed: https://review.openstack.org/507856
Committed:
https://git.openstack.org/cgit/openstack/neutron/commit/?id=7910c3a1c090ed7a46a32b371ecf73157ddba88f
Submitter: Jenkins
Branch: master
commit 7910c3a1c090ed7a46a32b371ecf73157ddba88f
Author: venkata anil <anilvenk...@redhat.com>
Date: Wed Sep 27 13:27:48 2017 +0000
Refactor DVR HA migarations DB operations
when a router is migrated from/to HA, router interface device_owner
in DB is updated in the AFTER_UPDATE event handler. Because of this
we see StaleDataError Trace as described in the bug report.
Instead, we follow DVR approach and update it in PRECOMMIT_UPDATE
handler, which only does DB changes. Here we update device_owner only
between legacy and HA router modes as migration involving DVR is
already handled in DVR code.
Also update distributed flag after all DB operations. Otherwise existing
code may affect router unbinding during retry as distributed flag
already updated before retry.
Testing is covered with scenario tempest test i.e
DVR HA migrations tests from gate job
"legacy-tempest-dsvm-neutron-dvr-multinode-scenario".
Closes-Bug: 1715163
Change-Id: I6e5783c44c77ec07371921e900ed45efd031ebdb
** Changed in: neutron
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1715163
Title:
StaleDataError in _reconfigure_ha_resources while running tempest
tests
Status in neutron:
Fix Released
Bug description:
When running tempest-NetworkMigrationFromHA tempest test for
gate-tempest-dsvm-neutron-dvr-multinode-scenario-ubuntu-xenial-nv on CI, the
following Trace is seen resulting in test failure.
(note: Link to q-svc log file [1])
Sep 05 10:00:10.799230 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager [None
req-f381e785-9e70-478c-ac95-f95e0d5d01a5
tempest-NetworkMigrationFromHA-629661935
tempest-NetworkMigrationFromHA-629661935] Error during notification for
neutron.services.l3_router.l3_router_plugin.L3RouterPlugin._reconfigure_ha_resources--9223372036852786466
router, after_update: StaleDataError: UPDATE statement on table
'standardattributes' expected to update 1 row(s); 0 were matched.
Sep 05 10:00:10.799496 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager Traceback (most
recent call last):
Sep 05 10:00:10.799687 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager File
"/usr/local/lib/python2.7/dist-packages/neutron_lib/callbacks/manager.py", line
171, in _notify_loop
Sep 05 10:00:10.799859 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager
callback(resource, event, trigger, **kwargs)
Sep 05 10:00:10.800009 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager File
"/opt/stack/new/neutron/neutron/db/l3_hamode_db.py", line 486, in
_reconfigure_ha_resources
Sep 05 10:00:10.800150 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager
self.schedule_router(context, router_id)
Sep 05 10:00:10.800284 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager File
"/opt/stack/new/neutron/neutron/db/l3_agentschedulers_db.py", line 469, in
schedule_router
Sep 05 10:00:10.800422 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager self, context,
router, candidates=candidates)
Sep 05 10:00:10.800561 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager File
"/opt/stack/new/neutron/neutron/scheduler/l3_agent_scheduler.py", line 54, in
schedule
Sep 05 10:00:10.800703 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager plugin, context,
router_id, candidates=candidates)
Sep 05 10:00:10.800842 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager File
"/opt/stack/new/neutron/neutron/scheduler/l3_agent_scheduler.py", line 233, in
_schedule_router
Sep 05 10:00:10.801071 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager if not
plugin.router_supports_scheduling(context, router_id):
Sep 05 10:00:10.801242 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager File
"/opt/stack/new/neutron/neutron/services/l3_router/l3_router_plugin.py", line
128, in router_supports_scheduling
Sep 05 10:00:10.801413 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager return
self.l3_driver_controller.uses_scheduler(context, router_id)
Sep 05 10:00:10.801716 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager File
"/opt/stack/new/neutron/neutron/services/l3_router/service_providers/driver_controller.py",
line 195, in uses_scheduler
Sep 05 10:00:10.801866 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager return
(self.get_provider_for_router(context, router_id).
Sep 05 10:00:10.802022 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager File
"/opt/stack/new/neutron/neutron/services/l3_router/service_providers/driver_controller.py",
line 149, in get_provider_for_router
Sep 05 10:00:10.802202 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager context,
[router_id]).get(router_id)
Sep 05 10:00:10.802356 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager File
"/opt/stack/new/neutron/neutron/db/servicetype_db.py", line 73, in
get_provider_names_by_resource_ids
Sep 05 10:00:10.802488 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager context,
resource_id=resource_ids)
Sep 05 10:00:10.802625 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager File
"/opt/stack/new/neutron/neutron/objects/base.py", line 463, in get_objects
Sep 05 10:00:10.802784 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager with
context.session.begin(subtransactions=True):
Sep 05 10:00:10.802932 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 824,
in begin
Sep 05 10:00:10.803106 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager self,
nested=nested)
Sep 05 10:00:10.803252 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 218,
in __init__
Sep 05 10:00:10.803437 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager
self._take_snapshot()
Sep 05 10:00:10.803575 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 327,
in _take_snapshot
Sep 05 10:00:10.803713 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager
self.session.flush()
Sep 05 10:00:10.803864 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2171,
in flush
Sep 05 10:00:10.804017 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager
self._flush(objects)
Sep 05 10:00:10.804170 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2291,
in _flush
Sep 05 10:00:10.804302 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager
transaction.rollback(_capture_exception=True)
Sep 05 10:00:10.804441 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line
66, in __exit__
Sep 05 10:00:10.804589 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager
compat.reraise(exc_type, exc_value, exc_tb)
Sep 05 10:00:10.804731 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2255,
in _flush
Sep 05 10:00:10.804996 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager
flush_context.execute()
Sep 05 10:00:10.805177 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line
389, in execute
Sep 05 10:00:10.805323 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager rec.execute(self)
Sep 05 10:00:10.805451 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line
548, in execute
Sep 05 10:00:10.805612 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager uow
Sep 05 10:00:10.805762 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line
177, in save_obj
Sep 05 10:00:10.805893 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager mapper, table,
update)
Sep 05 10:00:10.806023 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager File
"/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line
760, in _emit_update_statements
Sep 05 10:00:10.806153 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager
(table.description, len(records), rows))
Sep 05 10:00:10.806357 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager StaleDataError:
UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0
were matched.
Sep 05 10:00:10.806500 ubuntu-xenial-2-node-rax-iad-10771875
neutron-server[30374]: ERROR neutron_lib.callbacks.manager
[1]
http://logs.openstack.org/84/500384/3/check/gate-tempest-dsvm-neutron-dvr-multinode-scenario-ubuntu-xenial-nv/66eea59/logs/screen-q-svc.txt.gz
To manage notifications about this bug go to:
https://bugs.launchpad.net/neutron/+bug/1715163/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
