[Yahoo-eng-team] [Bug 1782696] [NEW] nova migration a vm with pci devices to a host don't have pci available

[Yongli He]

Public bug reported:

on nova master, multinodes with 2 compute nodes available. only one of
nova compute had pci devices available.

1. booting vm with pci reqested, assume it's name: pci-vm
2. nova migrate pci-vm
3. the pci-vm been schedule to a host don't have pci devices available.

we see exception on nova compute(the host don't have pci devs and which
nova schedule the pci-vm to):

2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server 
[req-71f0427e-4bd3-4014-8807-179e6e718066 b7d4ff5b9c0b4739865cce43a8de1c64 
f0584549d87143dfbf235199c653f33c - default default] Exception during message 
handling: PciRequestAliasNotDefined: PCI alias intelpci is not defined
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server Traceback (most 
recent call last):
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
 line 163, in _process_incoming
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     res = 
self.dispatcher.dispatch(message)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
 line 265, in dispatch
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return 
self._do_dispatch(endpoint, method, ctxt, args)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
 line 194, in _do_dispatch
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     result = 
func(ctxt, **new_args)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/exception_wrapper.py",
 line 79, in wrapped
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     function_name, 
call_dict, binary, tb)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", 
line 220, in __exit__
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     
self.force_reraise()
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", 
line 196, in force_reraise
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     
six.reraise(self.type_, self.value, self.tb)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/exception_wrapper.py",
 line 69, in wrapped
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return f(self, 
context, *args, **kw)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py",
 line 185, in decorated_function
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     "Error: %s", e, 
instance=instance)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", 
line 220, in __exit__
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     
self.force_reraise()
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", 
line 196, in force_reraise
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     
six.reraise(self.type_, self.value, self.tb)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py",
 line 155, in decorated_function
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return 
function(self, context, *args, **kwargs)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/utils.py", 
line 1090, in decorated_function
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return 
function(self, context, *args, **kwargs)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py",
 line 213, in decorated_function
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     
kwargs['instance'], e, sys.exc_info())
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", 
line 220, in __exit__
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     
self.force_reraise()
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", 
line 196, in force_reraise
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     
six.reraise(self.type_, self.value, self.tb)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py",
 line 201, in decorated_function
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return 
function(self, context, *args, **kwargs)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py",
 line 4181, in prep_resize
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     
filter_properties, host_list)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py",
 line 4245, in _reschedule_resize_or_reraise
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     
six.reraise(*exc_info)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py",
 line 4160, in prep_resize
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     node, migration, 
clean_shutdown)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py",
 line 4120, in _prep_resize
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     limits=limits) as 
claim:
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_concurrency/lockutils.py",
 line 274, in inner
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return f(*args, 
**kwargs)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/resource_tracker.py",
 line 254, in resize_claim
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     limits=limits)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/resource_tracker.py",
 line 310, in _move_claim
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     new_instance_type)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/pci/request.py", 
line 219, in get_pci_requests_from_flavor
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     
flavor['extra_specs']['pci_passthrough:alias'])
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/pci/request.py", 
line 157, in _translate_alias_to_requests
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     raise 
exception.PciRequestAliasNotDefined(alias=name)
2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server 
PciRequestAliasNotDefined: PCI alias intelpci is not defined

** Affects: nova
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1782696

Title:
  nova migration a vm with pci devices to a host don't have pci
  available

Status in OpenStack Compute (nova):
  New

Bug description:
  on nova master, multinodes with 2 compute nodes available. only one of
  nova compute had pci devices available.

  1. booting vm with pci reqested, assume it's name: pci-vm
  2. nova migrate pci-vm
  3. the pci-vm been schedule to a host don't have pci devices available.

  we see exception on nova compute(the host don't have pci devs and
  which nova schedule the pci-vm to):

  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server 
[req-71f0427e-4bd3-4014-8807-179e6e718066 b7d4ff5b9c0b4739865cce43a8de1c64 
f0584549d87143dfbf235199c653f33c - default default] Exception during message 
handling: PciRequestAliasNotDefined: PCI alias intelpci is not defined
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server Traceback (most 
recent call last):
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py",
 line 163, in _process_incoming
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     res = 
self.dispatcher.dispatch(message)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
 line 265, in dispatch
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return 
self._do_dispatch(endpoint, method, ctxt, args)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py",
 line 194, in _do_dispatch
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     result = 
func(ctxt, **new_args)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/exception_wrapper.py",
 line 79, in wrapped
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     function_name, 
call_dict, binary, tb)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", 
line 220, in __exit__
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     
self.force_reraise()
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", 
line 196, in force_reraise
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     
six.reraise(self.type_, self.value, self.tb)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/exception_wrapper.py",
 line 69, in wrapped
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return f(self, 
context, *args, **kw)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py",
 line 185, in decorated_function
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     "Error: %s", e, 
instance=instance)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", 
line 220, in __exit__
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     
self.force_reraise()
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", 
line 196, in force_reraise
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     
six.reraise(self.type_, self.value, self.tb)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py",
 line 155, in decorated_function
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return 
function(self, context, *args, **kwargs)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/utils.py", 
line 1090, in decorated_function
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return 
function(self, context, *args, **kwargs)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py",
 line 213, in decorated_function
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     
kwargs['instance'], e, sys.exc_info())
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", 
line 220, in __exit__
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     
self.force_reraise()
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", 
line 196, in force_reraise
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     
six.reraise(self.type_, self.value, self.tb)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py",
 line 201, in decorated_function
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return 
function(self, context, *args, **kwargs)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py",
 line 4181, in prep_resize
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     
filter_properties, host_list)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py",
 line 4245, in _reschedule_resize_or_reraise
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     
six.reraise(*exc_info)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py",
 line 4160, in prep_resize
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     node, 
migration, clean_shutdown)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/manager.py",
 line 4120, in _prep_resize
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     limits=limits) 
as claim:
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_concurrency/lockutils.py",
 line 274, in inner
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     return f(*args, 
**kwargs)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/resource_tracker.py",
 line 254, in resize_claim
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     limits=limits)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/compute/resource_tracker.py",
 line 310, in _move_claim
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     
new_instance_type)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/pci/request.py", 
line 219, in get_pci_requests_from_flavor
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     
flavor['extra_specs']['pci_passthrough:alias'])
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server   File 
"/var/lib/kolla/venv/local/lib/python2.7/site-packages/nova/pci/request.py", 
line 157, in _translate_alias_to_requests
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server     raise 
exception.PciRequestAliasNotDefined(alias=name)
  2018-07-19 22:04:15.950 7 ERROR oslo_messaging.rpc.server 
PciRequestAliasNotDefined: PCI alias intelpci is not defined

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1782696/+subscriptions

-- 
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to     : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help   : https://help.launchpad.net/ListHelp