Public bug reported:
When doing a concurrent spawn for VMs(30). Noticed some VM spawn
failures due to error below.
The reason is that during prebuild_instance, it will list all instances
of existing. But if concurrent process for spawning is happening, it is
possible that certain VM is just in process of creation in VCenter, and
the vsphere SDK only return partial object for that VM. So there is no
propSet for that VM created at the phase.
Should add check for these cases.
2014-09-14 03:53:31.311 2680 ERROR oslo.messaging.rpc.dispatcher [-] Exception
during message handling: ObjectContent instance has no attribute 'propSet'
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher Traceback
(most recent call last):
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134,
in _dispatch_and_reply
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
incoming.message))
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 177,
in _dispatch
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher return
self._do_dispatch(endpoint, method, ctxt, args)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 123,
in _do_dispatch
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher result =
getattr(endpoint, method)(ctxt, **new_args)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/server.py", line 139, in
inner
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher return
func(*args, **kwargs)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/exception.py", line 88, in wrapped
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher payload)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68,
in __exit__
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/exception.py", line 71, in wrapped
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher return
f(self, context, *args, **kw)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 282, in
decorated_function
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher pass
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68,
in __exit__
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 268, in
decorated_function
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher return
function(self, context, *args, **kwargs)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 335, in
decorated_function
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
function(self, context, *args, **kwargs)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 311, in
decorated_function
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher e,
sys.exc_info())
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68,
in __exit__
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 298, in
decorated_function
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher return
function(self, context, *args, **kwargs)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2140, in
run_instance
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
do_run_instance()
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line
249, in inner
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher return
f(*args, **kwargs)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2139, in
do_run_instance
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
legacy_bdm_in_spec)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1236, in
_run_instance
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
notify("error", fault=e) # notify that build failed
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68,
in __exit__
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1206, in
_run_instance
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
self._prebuild_instance(context, instance)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1239, in
_prebuild_instance
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
self._check_instance_exists(context, instance)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1504, in
_check_instance_exists
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher if
self.driver.instance_exists(instance['name']):
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/virt/driver.py", line 204, in
instance_exists
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher return
instance_id in self.list_instances()
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 604, in
list_instances
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
instances.extend(vmops.list_instances())
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 2067, in
list_instances
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
lst_vm_names = self._get_valid_vms_from_retrieve_result(vms)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 1851, in
_get_valid_vms_from_retrieve_result
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher for prop
in vm.propSet:
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
AttributeError: ObjectContent instance has no attribute 'propSet'
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
** Affects: nova
Importance: Undecided
Status: New
** Tags: vmware
** Tags added: vmware
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1372708
Title:
VMWare: vm spawn failure due to no attribute 'propSet' in concurrent
case
Status in OpenStack Compute (Nova):
New
Bug description:
When doing a concurrent spawn for VMs(30). Noticed some VM spawn
failures due to error below.
The reason is that during prebuild_instance, it will list all
instances of existing. But if concurrent process for spawning is
happening, it is possible that certain VM is just in process of
creation in VCenter, and the vsphere SDK only return partial object
for that VM. So there is no propSet for that VM created at the phase.
Should add check for these cases.
2014-09-14 03:53:31.311 2680 ERROR oslo.messaging.rpc.dispatcher [-]
Exception during message handling: ObjectContent instance has no attribute
'propSet'
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher Traceback
(most recent call last):
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 134,
in _dispatch_and_reply
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
incoming.message))
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 177,
in _dispatch
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher return
self._do_dispatch(endpoint, method, ctxt, args)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py", line 123,
in _do_dispatch
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher result =
getattr(endpoint, method)(ctxt, **new_args)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/server.py", line 139, in
inner
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher return
func(*args, **kwargs)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/exception.py", line 88, in wrapped
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher payload)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68,
in __exit__
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/exception.py", line 71, in wrapped
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher return
f(self, context, *args, **kw)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 282, in
decorated_function
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher pass
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68,
in __exit__
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 268, in
decorated_function
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher return
function(self, context, *args, **kwargs)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 335, in
decorated_function
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
function(self, context, *args, **kwargs)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 311, in
decorated_function
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher e,
sys.exc_info())
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68,
in __exit__
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 298, in
decorated_function
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher return
function(self, context, *args, **kwargs)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2140, in
run_instance
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
do_run_instance()
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line
249, in inner
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher return
f(*args, **kwargs)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2139, in
do_run_instance
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
legacy_bdm_in_spec)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1236, in
_run_instance
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
notify("error", fault=e) # notify that build failed
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68,
in __exit__
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1206, in
_run_instance
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
self._prebuild_instance(context, instance)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1239, in
_prebuild_instance
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
self._check_instance_exists(context, instance)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1504, in
_check_instance_exists
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher if
self.driver.instance_exists(instance['name']):
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/virt/driver.py", line 204, in
instance_exists
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher return
instance_id in self.list_instances()
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/driver.py", line 604, in
list_instances
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
instances.extend(vmops.list_instances())
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 2067, in
list_instances
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
lst_vm_names = self._get_valid_vms_from_retrieve_result(vms)
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/virt/vmwareapi/vmops.py", line 1851, in
_get_valid_vms_from_retrieve_result
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher for prop
in vm.propSet:
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
AttributeError: ObjectContent instance has no attribute 'propSet'
2014-09-14 03:53:31.311 2680 TRACE oslo.messaging.rpc.dispatcher
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1372708/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
