Public bug reported: When running tempest-NetworkMigrationFromHA tempest test for gate-tempest-dsvm-neutron-dvr-multinode-scenario-ubuntu-xenial-nv on CI, the following Trace is seen resulting in test failure. (note: Link to q-svc log file [1])
Sep 05 10:00:10.799230 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager [None req-f381e785-9e70-478c-ac95-f95e0d5d01a5 tempest-NetworkMigrationFromHA-629661935 tempest-NetworkMigrationFromHA-629661935] Error during notification for neutron.services.l3_router.l3_router_plugin.L3RouterPlugin._reconfigure_ha_resources--9223372036852786466 router, after_update: StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched. Sep 05 10:00:10.799496 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last): Sep 05 10:00:10.799687 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/neutron_lib/callbacks/manager.py", line 171, in _notify_loop Sep 05 10:00:10.799859 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) Sep 05 10:00:10.800009 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/db/l3_hamode_db.py", line 486, in _reconfigure_ha_resources Sep 05 10:00:10.800150 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self.schedule_router(context, router_id) Sep 05 10:00:10.800284 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/db/l3_agentschedulers_db.py", line 469, in schedule_router Sep 05 10:00:10.800422 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self, context, router, candidates=candidates) Sep 05 10:00:10.800561 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/scheduler/l3_agent_scheduler.py", line 54, in schedule Sep 05 10:00:10.800703 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager plugin, context, router_id, candidates=candidates) Sep 05 10:00:10.800842 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/scheduler/l3_agent_scheduler.py", line 233, in _schedule_router Sep 05 10:00:10.801071 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager if not plugin.router_supports_scheduling(context, router_id): Sep 05 10:00:10.801242 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/services/l3_router/l3_router_plugin.py", line 128, in router_supports_scheduling Sep 05 10:00:10.801413 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager return self.l3_driver_controller.uses_scheduler(context, router_id) Sep 05 10:00:10.801716 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/services/l3_router/service_providers/driver_controller.py", line 195, in uses_scheduler Sep 05 10:00:10.801866 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager return (self.get_provider_for_router(context, router_id). Sep 05 10:00:10.802022 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/services/l3_router/service_providers/driver_controller.py", line 149, in get_provider_for_router Sep 05 10:00:10.802202 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager context, [router_id]).get(router_id) Sep 05 10:00:10.802356 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/db/servicetype_db.py", line 73, in get_provider_names_by_resource_ids Sep 05 10:00:10.802488 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager context, resource_id=resource_ids) Sep 05 10:00:10.802625 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/objects/base.py", line 463, in get_objects Sep 05 10:00:10.802784 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager with context.session.begin(subtransactions=True): Sep 05 10:00:10.802932 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 824, in begin Sep 05 10:00:10.803106 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self, nested=nested) Sep 05 10:00:10.803252 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 218, in __init__ Sep 05 10:00:10.803437 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self._take_snapshot() Sep 05 10:00:10.803575 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 327, in _take_snapshot Sep 05 10:00:10.803713 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self.session.flush() Sep 05 10:00:10.803864 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2171, in flush Sep 05 10:00:10.804017 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self._flush(objects) Sep 05 10:00:10.804170 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2291, in _flush Sep 05 10:00:10.804302 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager transaction.rollback(_capture_exception=True) Sep 05 10:00:10.804441 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ Sep 05 10:00:10.804589 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager compat.reraise(exc_type, exc_value, exc_tb) Sep 05 10:00:10.804731 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2255, in _flush Sep 05 10:00:10.804996 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager flush_context.execute() Sep 05 10:00:10.805177 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute Sep 05 10:00:10.805323 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager rec.execute(self) Sep 05 10:00:10.805451 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute Sep 05 10:00:10.805612 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager uow Sep 05 10:00:10.805762 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 177, in save_obj Sep 05 10:00:10.805893 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager mapper, table, update) Sep 05 10:00:10.806023 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 760, in _emit_update_statements Sep 05 10:00:10.806153 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager (table.description, len(records), rows)) Sep 05 10:00:10.806357 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched. Sep 05 10:00:10.806500 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager [1] http://logs.openstack.org/84/500384/3/check/gate-tempest-dsvm-neutron-dvr-multinode-scenario-ubuntu-xenial-nv/66eea59/logs/screen-q-svc.txt.gz ** Affects: neutron Importance: Undecided Assignee: venkata anil (anil-venkata) Status: New ** Changed in: neutron Assignee: (unassigned) => venkata anil (anil-venkata) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1715163 Title: StaleDataError in _reconfigure_ha_resources while running tempest tests Status in neutron: New Bug description: When running tempest-NetworkMigrationFromHA tempest test for gate-tempest-dsvm-neutron-dvr-multinode-scenario-ubuntu-xenial-nv on CI, the following Trace is seen resulting in test failure. (note: Link to q-svc log file [1]) Sep 05 10:00:10.799230 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager [None req-f381e785-9e70-478c-ac95-f95e0d5d01a5 tempest-NetworkMigrationFromHA-629661935 tempest-NetworkMigrationFromHA-629661935] Error during notification for neutron.services.l3_router.l3_router_plugin.L3RouterPlugin._reconfigure_ha_resources--9223372036852786466 router, after_update: StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched. Sep 05 10:00:10.799496 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager Traceback (most recent call last): Sep 05 10:00:10.799687 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/neutron_lib/callbacks/manager.py", line 171, in _notify_loop Sep 05 10:00:10.799859 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager callback(resource, event, trigger, **kwargs) Sep 05 10:00:10.800009 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/db/l3_hamode_db.py", line 486, in _reconfigure_ha_resources Sep 05 10:00:10.800150 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self.schedule_router(context, router_id) Sep 05 10:00:10.800284 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/db/l3_agentschedulers_db.py", line 469, in schedule_router Sep 05 10:00:10.800422 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self, context, router, candidates=candidates) Sep 05 10:00:10.800561 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/scheduler/l3_agent_scheduler.py", line 54, in schedule Sep 05 10:00:10.800703 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager plugin, context, router_id, candidates=candidates) Sep 05 10:00:10.800842 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/scheduler/l3_agent_scheduler.py", line 233, in _schedule_router Sep 05 10:00:10.801071 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager if not plugin.router_supports_scheduling(context, router_id): Sep 05 10:00:10.801242 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/services/l3_router/l3_router_plugin.py", line 128, in router_supports_scheduling Sep 05 10:00:10.801413 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager return self.l3_driver_controller.uses_scheduler(context, router_id) Sep 05 10:00:10.801716 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/services/l3_router/service_providers/driver_controller.py", line 195, in uses_scheduler Sep 05 10:00:10.801866 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager return (self.get_provider_for_router(context, router_id). Sep 05 10:00:10.802022 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/services/l3_router/service_providers/driver_controller.py", line 149, in get_provider_for_router Sep 05 10:00:10.802202 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager context, [router_id]).get(router_id) Sep 05 10:00:10.802356 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/db/servicetype_db.py", line 73, in get_provider_names_by_resource_ids Sep 05 10:00:10.802488 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager context, resource_id=resource_ids) Sep 05 10:00:10.802625 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/opt/stack/new/neutron/neutron/objects/base.py", line 463, in get_objects Sep 05 10:00:10.802784 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager with context.session.begin(subtransactions=True): Sep 05 10:00:10.802932 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 824, in begin Sep 05 10:00:10.803106 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self, nested=nested) Sep 05 10:00:10.803252 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 218, in __init__ Sep 05 10:00:10.803437 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self._take_snapshot() Sep 05 10:00:10.803575 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 327, in _take_snapshot Sep 05 10:00:10.803713 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self.session.flush() Sep 05 10:00:10.803864 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2171, in flush Sep 05 10:00:10.804017 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager self._flush(objects) Sep 05 10:00:10.804170 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2291, in _flush Sep 05 10:00:10.804302 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager transaction.rollback(_capture_exception=True) Sep 05 10:00:10.804441 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__ Sep 05 10:00:10.804589 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager compat.reraise(exc_type, exc_value, exc_tb) Sep 05 10:00:10.804731 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2255, in _flush Sep 05 10:00:10.804996 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager flush_context.execute() Sep 05 10:00:10.805177 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute Sep 05 10:00:10.805323 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager rec.execute(self) Sep 05 10:00:10.805451 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute Sep 05 10:00:10.805612 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager uow Sep 05 10:00:10.805762 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 177, in save_obj Sep 05 10:00:10.805893 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager mapper, table, update) Sep 05 10:00:10.806023 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 760, in _emit_update_statements Sep 05 10:00:10.806153 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager (table.description, len(records), rows)) Sep 05 10:00:10.806357 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager StaleDataError: UPDATE statement on table 'standardattributes' expected to update 1 row(s); 0 were matched. Sep 05 10:00:10.806500 ubuntu-xenial-2-node-rax-iad-10771875 neutron-server[30374]: ERROR neutron_lib.callbacks.manager [1] http://logs.openstack.org/84/500384/3/check/gate-tempest-dsvm-neutron-dvr-multinode-scenario-ubuntu-xenial-nv/66eea59/logs/screen-q-svc.txt.gz To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1715163/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
