Reviewed: https://review.openstack.org/605164 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=ff03b157b930de23e5912802cbfbc86889c869c2 Submitter: Zuul Branch: master
commit ff03b157b930de23e5912802cbfbc86889c869c2 Author: Jack Ding <[email protected]> Date: Tue Sep 25 13:20:25 2018 -0400 Handle missing marker during online data migration During upgrade the instance used by the request spec marker could be deleted and purged between sessions. This would cause the database online data migration to fail as the marker instance couldn't be found. Fix by handling the MarkerNotFound exception and re-trying without the marker. This will go through all the instances and reset the marker when done. Closes-Bug: #1793419 Change-Id: If96e3d038346f16cc93209bccf3db028bacfe59b Signed-off-by: Jack Ding <[email protected]> ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1793419 Title: database online data migration fail due to missing request spec marker Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) pike series: Triaged Status in OpenStack Compute (nova) queens series: Triaged Status in OpenStack Compute (nova) rocky series: Triaged Bug description: Description =========== During upgrade we run nova online migration that goes through the list of instances and creates a request spec record in the db if one does not exist. As the online migrations are batched, the request spec migration leaves a marker record in the request_specs table to indicate the last instance uuid that was processed. It continues processing starting from that instances on the next batch. In our upgrade test, we hit a scenario where the marker instance from the online migration that was run during the Mitaka->Newton upgrade had been deleted and purged from the db by time we ran the Newton->Pike upgrade. This caused the online migration to fail as the marker instance couldn't be found. Steps to reproduce ================== - run data online migration on installed Newton load. nova-manage db online_data_migrations - delete the instance referenced by the marker (instance_uuid 00000000-0000-0000-0000-000000000000) - purge db: nova-manage db purge - upgrade to Pike. Expected result =============== Upgrade successful with no exceptions. Actual result ============= Exceptions occur during upgrade with missing marker an upgrade failed. Error attempting to run <function migrate_instances_add_request_spec at 0x5151050> 14 rows matched query service_uuids_online_data_migration, 14 migrated 13 rows matched query migrate_quota_limits_to_api_db, 13 migrated Error attempting to run <function migrate_instances_add_request_spec at 0x5151050> +---------------------------------------------+--------------+-----------+ | Migration | Total Needed | Completed | +---------------------------------------------+--------------+-----------+ | delete_build_requests_with_no_instance_uuid | 0 | 0 | | migrate_aggregate_reset_autoincrement | 0 | 0 | | migrate_aggregates | 0 | 0 | | migrate_flavor_reset_autoincrement | 0 | 0 | | migrate_flavors | 0 | 0 | | migrate_instance_groups_to_api_db | 0 | 0 | | migrate_instance_keypairs | 0 | 0 | | migrate_instances_add_request_spec | 0 | 0 | | migrate_keypairs_to_api_db | 0 | 0 | | migrate_quota_classes_to_api_db | 0 | 0 | | migrate_quota_limits_to_api_db | 0 | 0 | | service_uuids_online_data_migration | 0 | 0 | +---------------------------------------------+--------------+-----------+ To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1793419/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
