** Also affects: nova/ocata Importance: Undecided Status: New ** Also affects: nova/newton Importance: Undecided Status: New
** Changed in: nova Importance: Undecided => High ** Changed in: nova/newton Importance: Undecided => High ** Changed in: nova/ocata Importance: Undecided => High -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1680918 Title: Nova upgrade fails if PCI devices of type-PF or type-PCI are present in the database Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) newton series: New Status in OpenStack Compute (nova) ocata series: New Bug description: Description =========== If a Nova DB is upgraded (migrated) while containing PCI devices with device type 'type-PF' or 'type-PCI', a validation error similar to this will be thrown: "ValidationError: There are still 2 unmigrated records in the pci_devices table. Migration cannot continue until all records have been migrated." The error is generated by the 330_enforce_mitaka_online_migrations.py upgrade script. The PCI device migration validation will fail if any PCI device entries without a populated parent_addr are found. However, the parent_addr really only applies to PCI device entries of 'type-VF' (ie. SRIOV virtual functions) This is an example of what the pci_devices table looks like with SRIOV enabled PCI devices if the appropriate entries are whitelisted in nova.conf: MariaDB [nova]> select * from pci_devices; +---------------------+---------------------+------------+---------+----+-----------------+--------------+------------+-----------+----------+------------------+-----------------+-----------+------------+---------------+------------+-----------+--------------+ | created_at | updated_at | deleted_at | deleted | id | compute_node_id | address | product_id | vendor_id | dev_type | dev_id | label | status | extra_info | instance_uuid | request_id | numa_node | parent_addr | +---------------------+---------------------+------------+---------+----+-----------------+--------------+------------+-----------+----------+------------------+-----------------+-----------+------------+---------------+------------+-----------+--------------+ | 2017-04-06 21:01:21 | 2017-04-06 21:53:13 | NULL | 0 | 1 | 1 | 0000:05:10.1 | 10ed | 8086 | type-VF | pci_0000_05_10_1 | label_8086_10ed | available | {} | NULL | NULL | 0 | 0000:05:00.1 | | 2017-04-06 21:01:21 | 2017-04-06 21:53:13 | NULL | 0 | 2 | 1 | 0000:05:10.3 | 10ed | 8086 | type-VF | pci_0000_05_10_3 | label_8086_10ed | available | {} | NULL | NULL | 0 | 0000:05:00.1 | | 2017-04-06 21:01:21 | 2017-04-06 21:53:13 | NULL | 0 | 3 | 1 | 0000:05:10.5 | 10ed | 8086 | type-VF | pci_0000_05_10_5 | label_8086_10ed | available | {} | NULL | NULL | 0 | 0000:05:00.1 | | 2017-04-06 21:01:21 | 2017-04-06 21:53:13 | NULL | 0 | 4 | 1 | 0000:05:10.7 | 10ed | 8086 | type-VF | pci_0000_05_10_7 | label_8086_10ed | available | {} | NULL | NULL | 0 | 0000:05:00.1 | | 2017-04-06 21:53:13 | NULL | NULL | 0 | 5 | 1 | 0000:05:00.0 | 10fb | 8086 | type-PF | pci_0000_05_00_0 | label_8086_10fb | available | {} | NULL | NULL | 0 | NULL | | 2017-04-06 21:53:13 | NULL | NULL | 0 | 6 | 1 | 0000:05:00.1 | 10fb | 8086 | type-PF | pci_0000_05_00_1 | label_8086_10fb | available | {} | NULL | NULL | 0 | NULL | +---------------------+---------------------+------------+---------+----+-----------------+--------------+------------+-----------+----------+------------------+-----------------+-----------+------------+---------------+------------+-----------+--------------+ 6 rows in set (0.00 sec) I think the upgrade script should be checking the PciDevice dev_type field for 'type-VF' when validating the parent_addr. Steps to reproduce ================== 1. Install a Mitaka control node and edit the nova.conf file to include 1 or more PCI devices in the pci_passthrough_whitelist. ie: pci_passthrough_whitelist = {"vendor_id": "8086", "product_id":"10fb"} 2. Install a second Newton or newer control node and edit the nova.conf to point to the SQL database of the Mitaka node. ie: [database] connection = mysql+pymysql://root:supersecret@<MITAKA CONTROLLER NODE>/nova?charset=utf8 [api_database] connection = mysql+pymysql://root:supersecret@<MITAKA CONTROLLER NODE>/nova_api?charset=utf8 3. From the new control node, issue the following command: nova-manage db sync Expected result =============== Database migration/upgrade should succeed Actual result ============= A ValidationError, similar to: "ValidationError: There are still <N> unmigrated records in the pci_devices table. Migration cannot continue until all records have been migrated." Environment =========== 1. Exact version of OpenStack you are running. See the following Mitaka (old node) Newton, Ocata (new node) 2. Which hypervisor did you use? libvirt + kvm 2. Which storage type did you use? lvm 3. Which networking type did you use? Neutron, OVS Logs & Configs ============== See attached To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1680918/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp