Reviewed: https://review.opendev.org/c/openstack/nova/+/853611 Committed: https://opendev.org/openstack/nova/commit/2b447b7236f95752d00ebcee8c32cfef4850cf5d Submitter: "Zuul (22348)" Branch: master
commit 2b447b7236f95752d00ebcee8c32cfef4850cf5d Author: Balazs Gibizer <[email protected]> Date: Wed Aug 17 18:19:15 2022 +0200 Trigger reschedule if PCI consumption fail on compute The PciPassthroughFilter logic checks each InstancePCIRequest individually against the available PCI pools of a given host and given boot request. So it is possible that the scheduler accepts a host that has a single PCI device available even if two devices are requested for a single instance via two separate PCI aliases. Then the PCI claim on the compute detects this but does not stop the boot just logs an ERROR. This results in the instance booted without any PCI device. This patch does two things: 1) changes the PCI claim to fail with an exception and trigger a re-schedule instead of just logging an ERROR. 2) change the PciDeviceStats.support_requests that is called during scheduling to not just filter pools for individual requests but also consume the request from the pool within the scope of a single boot request. The fix in #2) would not be enough alone as two parallel scheduling request could race for a single device on the same host. #1) is the ultimate place where we consume devices under a compute global lock so we need the fix there too. Closes-Bug: #1986838 Change-Id: Iea477be57ae4e95dfc03acc9368f31d4be895343 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1986838 Title: Booting with two identical PCI aliases on a host with a single matching dev succeeds but the instance will have no PCI allocations Status in OpenStack Compute (nova): Fix Released Bug description: Detected during reading the code. Reproduction 1) configure a host with a single PCI passthrough device 2) configure two PCI aliases (a1, a2) with different names but each matching the above device 3) boot an instance with 'pci_passthrough:alias': 'a1:1,a2:1' flavor extra_spec. Expected result The instance fails to schedule Actual result The instance schedules to the host but has no PCI allocations The nova scheduler logs: Selected host: compute1 failed to consume from instance. Error: PCI device request [InstancePCIRequest(alias_name='a1',count=1,is_new=<?>,numa_policy='legacy',request_id=None,requester_id=<?>,spec=[{product_id='1533',vendor_id='8086'}]), InstancePCIRequest(alias_name='a2',count=1,is_new=<?>,numa_policy='legacy',request_id=None,requester_id=<?>,spec=[{product_id='1533',vendor_id='8086'}])] failed The nova compute logs: Failed to allocate PCI devices for instance. Unassigning devices back to pools. This should not happen, since the scheduler should have accurate information, and allocation during claims is controlled via a hold on the compute node semaphore. I think the root cause of the fault is that the PciDeviceStats.support_requests() [1] call matches each InstancePCIRequest object independently to the available PCI pools and does not update the status of the pools locally. I will push a functional reproduction test shortly. [1] https://github.com/openstack/nova/blob/69bc4c38d1c5b98fcbbe8b16a7dfeb654e3b8173/nova/pci/stats.py#L645 To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1986838/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
