Reviewed: https://review.openstack.org/607650 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=035708c37d587e4c5ede7fe80270bdbff98016ac Submitter: Zuul Branch: master
commit 035708c37d587e4c5ede7fe80270bdbff98016ac Author: Matt Riedemann <[email protected]> Date: Wed Oct 3 12:54:53 2018 -0400 Handle IndexError in _populate_neutron_binding_profile This fixes the code that was blindly pop'ing an entry from an empty list of PCI devices claimed by the instance. It's not exactly clear how we can get into this situation, presumably there was a failure in the actual PCI device claim logic in the ResourceTracker - maybe related to the configured PCI passthrough whitelist. Regardless, we should handle the empty PCI device list in this method and raise an appropriate exception to fail the build on this host. Change-Id: I401bb74cf6e17c2b72cc62bf8ec03ec58238c44a Closes-Bug: #1795064 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1795064 Title: SR-IOV error IndexError: pop from empty list Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) ocata series: Confirmed Status in OpenStack Compute (nova) pike series: Confirmed Status in OpenStack Compute (nova) queens series: Confirmed Status in OpenStack Compute (nova) rocky series: Confirmed Bug description: I am building SR-IOV support in compute node on Queens i have following NIC card and VF enabled [root@ostack-compute-63 ~]# lspci -nn | grep -i eth 03:00.0 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet [14e4:168e] (rev 10) 03:00.1 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet [14e4:168e] (rev 10) 03:01.0 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af] 03:01.1 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af] 03:01.2 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af] 03:01.3 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af] 03:01.4 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af] 03:01.5 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af] 03:01.6 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM57810 10 Gigabit Ethernet Virtual Function [14e4:16af] I have setup everything according official documents and so far everything looks good. I have created neutron-port and when i trying to launch instance i got following error on compute node. 2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager [req-095c3f53-a558-4178-84ee-cf79bf7f3c7c eebe97b4bc714b8f814af8a44d08c2a4 2927a06cf30f4f7e938fdda2cc05aed2 - default default] Instance failed network setup after 1 attempt(s): IndexError: pop from empty list 2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager Traceback (most recent call last): 2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/compute/manager.py", line 1398, in _allocate_network_async 2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager bind_host_id=bind_host_id) 2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 954, in allocate_for_instance 2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager bind_host_id, available_macs, requested_ports_dict) 2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1087, in _update_ports_for_instance 2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager vif.destroy() 2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager self.force_reraise() 2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager six.reraise(self.type_, self.value, self.tb) 2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1042, in _update_ports_for_instance 2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager bind_host_id=bind_host_id) 2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1192, in _populate_neutron_extension_values 2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager port_req_body) 2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1138, in _populate_neutron_binding_profile 2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager instance, pci_request_id).pop() 2018-09-28 14:41:53.584 11957 ERROR nova.compute.manager IndexError: pop from empty list ------------- also every 60 second i am getting following error --------------- 2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call 2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager rv = execute(f, *args, **kwargs) 2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute 2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager six.reraise(c, e, tb) 2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker 2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager rv = meth(*args, **kwargs) 2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager File "/openstack/venvs/nova-17.0.8/lib/python2.7/site-packages/libvirt.py", line 4232, in nodeDeviceLookupByName 2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager if ret is None:raise libvirtError('virNodeDeviceLookupByName() failed', conn=self) 2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager libvirtError: Node device not found: no node device with matching name 'net_enp3s1f4_00_00_00_00_00_00' 2018-09-28 16:22:30.646 28663 ERROR nova.compute.manager also i was reading this but it didn't help: http://lists.openstack.org/pipermail/openstack/2018-January/045982.html To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1795064/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
