[Yahoo-eng-team] [Bug 1794364] Re: 'nova-manage db online_data_migrations' count fail
Reviewed: https://review.openstack.org/611463 Committed: https://git.openstack.org/cgit/openstack/cinder/commit/?id=d47486d317e8cdb1cdab50d73f6484289ab082d4 Submitter: Zuul Branch:master commit d47486d317e8cdb1cdab50d73f6484289ab082d4 Author: imacdonn Date: Wed Oct 17 22:48:18 2018 + cinder-manage online_data_migrations fixes Addresses some issues with this command: 1) When used without the --max-count option, the summary table will always show zero migrations run, because it only accounts for the last batch, and the loop only exits when the last batch does no work. 2) "remaining" counts cannot be accurate, given the way migrations are implemented, because the "found" count refers to the number of rows that exist in the database, not the number that still need the migration applied. 3) In the case where no migrations are successful, but some raise exceptions, the command was exiting with status zero, which usually indicates "success". This can cause issues that cause migration failures to go unnoticed, especially when automated. 4) When exceptions do occur, a minimally useful message is output, and no detail about the exception is available to the user. The exception detail should be logged. 5) Inaccuracies in the documentation - "--max_number" should be "--max-count", and stale references to the "--ignore_state" option, which was removed in [1] The solution for (3) introduces a new exit status, 2. See release note for details. These changes are aligned with equivalents [2][3] for the nova-manage command, except for the calculation of "Total Needed" - nova seems to interpret the "found" count differently/inconsistently. [1] https://review.openstack.org/510201 [2] https://review.openstack.org/605828 [3] https://review.openstack.org/608091 Change-Id: I878480eb2359625cde839b073230844acc645cba Closes-Bug: #1794364 Closes-Bug: #1796192 ** Changed in: cinder Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1794364 Title: 'nova-manage db online_data_migrations' count fail Status in Cinder: Fix Released Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) ocata series: Fix Committed Status in OpenStack Compute (nova) pike series: Fix Committed Status in OpenStack Compute (nova) queens series: Fix Committed Status in OpenStack Compute (nova) rocky series: Fix Committed Bug description: 'nova-manage db online_data_migrations' attempts to display summary counts of migrations "Needed" and "Completed" in a pretty table at the end, but fails to accumulate the totals between successive invocations of _run_migration(), and ends up reporting zeroes. # nova-manage db online_data_migrations /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:332: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported exception.NotSupportedWarning Running batches of 50 until complete /usr/lib/python2.7/site-packages/pymysql/cursors.py:166: Warning: (3090, u"Changing sql mode 'NO_AUTO_CREATE_USER' is deprecated. It will be removed in a future release.") result = self._query(query) 2 rows matched query migrate_instances_add_request_spec, 0 migrated 13 rows matched query migrate_quota_limits_to_api_db, 13 migrated 37 rows matched query populate_uuids, 37 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50
[Yahoo-eng-team] [Bug 1794364] Re: 'nova-manage db online_data_migrations' count fail
** Also affects: cinder Importance: Undecided Status: New ** Changed in: cinder Assignee: (unassigned) => iain MacDonnell (imacdonn) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1794364 Title: 'nova-manage db online_data_migrations' count fail Status in Cinder: New Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) ocata series: Fix Committed Status in OpenStack Compute (nova) pike series: Fix Committed Status in OpenStack Compute (nova) queens series: Fix Committed Status in OpenStack Compute (nova) rocky series: Fix Committed Bug description: 'nova-manage db online_data_migrations' attempts to display summary counts of migrations "Needed" and "Completed" in a pretty table at the end, but fails to accumulate the totals between successive invocations of _run_migration(), and ends up reporting zeroes. # nova-manage db online_data_migrations /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:332: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported exception.NotSupportedWarning Running batches of 50 until complete /usr/lib/python2.7/site-packages/pymysql/cursors.py:166: Warning: (3090, u"Changing sql mode 'NO_AUTO_CREATE_USER' is deprecated. It will be removed in a future release.") result = self._query(query) 2 rows matched query migrate_instances_add_request_spec, 0 migrated 13 rows matched query migrate_quota_limits_to_api_db, 13 migrated 37 rows matched query populate_uuids, 37 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 21 rows matched query populate_uuids, 21 migrated +-+--+---+ | Migration | Total Needed | Completed | +-+--+---+ | delete_build_requests_with_no_instance_uuid | 0 | 0 | |migrate_aggregate_reset_autoincrement| 0 | 0 | | migrate_aggregates | 0 | 0 | | migrate_instance_groups_to_api_db | 0 | 0 | | migrate_instances_add_request_spec | 0 | 0 | | migrate_keypairs_to_api_db | 0 | 0 | | migrate_quota_classes_to_api_db | 0 | 0 | |migrate_quota_limits_to_api_db |
[Yahoo-eng-team] [Bug 1794364] Re: 'nova-manage db online_data_migrations' count fail
Reviewed: https://review.openstack.org/605329 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=c4c6dc736e084f2d919b9bcf93d26df87e0341c9 Submitter: Zuul Branch:master commit c4c6dc736e084f2d919b9bcf93d26df87e0341c9 Author: imacdonn Date: Wed Sep 26 04:30:49 2018 + nova-manage - fix online_data_migrations counts When running online_data_migrations in batches, totals were not being accumulated - rather the counts each batch would clobber those from the previous one, and the last batch would run no migrations, so the totals were reported as zero. Change-Id: Ib616f2efb69baa16e18601d27b747220bbefeb16 Closes-Bug: #1794364 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1794364 Title: 'nova-manage db online_data_migrations' count fail Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) ocata series: In Progress Status in OpenStack Compute (nova) pike series: In Progress Status in OpenStack Compute (nova) queens series: In Progress Status in OpenStack Compute (nova) rocky series: In Progress Bug description: 'nova-manage db online_data_migrations' attempts to display summary counts of migrations "Needed" and "Completed" in a pretty table at the end, but fails to accumulate the totals between successive invocations of _run_migration(), and ends up reporting zeroes. # nova-manage db online_data_migrations /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:332: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported exception.NotSupportedWarning Running batches of 50 until complete /usr/lib/python2.7/site-packages/pymysql/cursors.py:166: Warning: (3090, u"Changing sql mode 'NO_AUTO_CREATE_USER' is deprecated. It will be removed in a future release.") result = self._query(query) 2 rows matched query migrate_instances_add_request_spec, 0 migrated 13 rows matched query migrate_quota_limits_to_api_db, 13 migrated 37 rows matched query populate_uuids, 37 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 21 rows matched query populate_uuids, 21 migrated +-+--+---+ | Migration | Total Needed | Completed | +-+--+---+ |
[Yahoo-eng-team] [Bug 1794364] Re: 'nova-manage db online_data_migrations' count fail
(3:05:42 PM) mriedem: that populate_uuids migration was added in queens, and looking at a grenade run from queens, we should see bdms getting migrated to have a uuid, and i'm not seeing any results in the table for a grenade run on queens (3:05:42 PM) mriedem: http://logs.openstack.org/48/604448/1/check/neutron-grenade/19d8166/logs/grenade.sh.txt.gz (3:05:46 PM) mriedem: so yeah, likely busted ** Changed in: nova Status: New => Confirmed ** Changed in: nova Importance: Undecided => High ** Also affects: nova/ocata Importance: Undecided Status: New ** Also affects: nova/pike Importance: Undecided Status: New ** Also affects: nova/rocky Importance: Undecided Status: New ** Also affects: nova/queens Importance: Undecided Status: New ** Changed in: nova/pike Status: New => Confirmed ** Changed in: nova/queens Status: New => Confirmed ** Tags added: upgrade ** Changed in: nova/pike Importance: Undecided => High ** Changed in: nova/queens Importance: Undecided => High ** Changed in: nova/ocata Status: New => Confirmed ** Changed in: nova/rocky Status: New => Confirmed ** Changed in: nova/ocata Importance: Undecided => High ** Changed in: nova/rocky Importance: Undecided => High -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1794364 Title: 'nova-manage db online_data_migrations' count fail Status in OpenStack Compute (nova): Confirmed Status in OpenStack Compute (nova) ocata series: Confirmed Status in OpenStack Compute (nova) pike series: Confirmed Status in OpenStack Compute (nova) queens series: Confirmed Status in OpenStack Compute (nova) rocky series: Confirmed Bug description: 'nova-manage db online_data_migrations' attempts to display summary counts of migrations "Needed" and "Completed" in a pretty table at the end, but fails to accumulate the totals between successive invocations of _run_migration(), and ends up reporting zeroes. # nova-manage db online_data_migrations /usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py:332: NotSupportedWarning: Configuration option(s) ['use_tpool'] not supported exception.NotSupportedWarning Running batches of 50 until complete /usr/lib/python2.7/site-packages/pymysql/cursors.py:166: Warning: (3090, u"Changing sql mode 'NO_AUTO_CREATE_USER' is deprecated. It will be removed in a future release.") result = self._query(query) 2 rows matched query migrate_instances_add_request_spec, 0 migrated 13 rows matched query migrate_quota_limits_to_api_db, 13 migrated 37 rows matched query populate_uuids, 37 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query populate_uuids, 50 migrated 50 rows matched query
