** Also affects: nova/kilo
Importance: Undecided
Status: New
** Changed in: nova/kilo
Status: New => In Progress
** Changed in: nova/kilo
Importance: Undecided => High
** Changed in: nova/kilo
Assignee: (unassigned) => Dan Smith (danms)
** Tags removed: kilo-backport-potential
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1446082
Title:
Instance without "extra" data crashes nova-compute
Status in OpenStack Compute (Nova):
Fix Committed
Status in OpenStack Compute (nova) kilo series:
In Progress
Bug description:
I'm upgrading from Icehouse to Kilo. I have a single instance that was
created in Icehouse. After the upgrade, nova-compute crashes because
it's looking for instance "extra" data that is not there.
To fix this, we need to check if there is any "extra" data for the
instance before trying to read properties such as "numa_topology".
# dpkg -l | grep nova
ii nova-common 1:2015.1~rc1-0ubuntu1~cloud0
all OpenStack Compute - common files
ii nova-compute 1:2015.1~rc1-0ubuntu1~cloud0
all OpenStack Compute - compute node base
ii nova-compute-kvm 1:2015.1~rc1-0ubuntu1~cloud0
all OpenStack Compute - compute node (KVM)
ii nova-compute-libvirt 1:2015.1~rc1-0ubuntu1~cloud0
all OpenStack Compute - compute node libvirt support
ii python-nova 1:2015.1~rc1-0ubuntu1~cloud0
all OpenStack Compute Python libraries
ii python-novaclient 1:2.22.0-0ubuntu1~cloud0
all client library for OpenStack Compute API
nova-compute.log:
2015-04-20 17:35:09.214 15508 DEBUG oslo_concurrency.lockutils
[req-43d3110a-cac7-425e-842c-f725bda91c10 - - - - -] Lock "compute_resources"
acquired by "_update_available_resource" :: waited 0.000s inner
/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:444
2015-04-20 17:35:09.299 15508 DEBUG oslo_concurrency.lockutils
[req-43d3110a-cac7-425e-842c-f725bda91c10 - - - - -] Lock "compute_resources"
released by "_update_available_resource" :: held 0.085s inner
/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:456
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 457, in
fire_timers
timer()
File "/usr/lib/python2.7/dist-packages/eventlet/hubs/timer.py", line 58, in
__call__
cb(*args, **kw)
File "/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214,
in main
result = function(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/openstack/common/service.py",
line 497, in run_service
service.start()
File "/usr/lib/python2.7/dist-packages/nova/service.py", line 183, in start
self.manager.pre_start_hook()
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1287,
in pre_start_hook
self.update_available_resource(nova.context.get_admin_context())
File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6236,
in update_available_resource
rt.update_available_resource(context)
File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py",
line 402, in update_available_resource
self._update_available_resource(context, resources)
File "/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line
445, in inner
return f(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py",
line 436, in _update_available_resource
'numa_topology'])
File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 163, in
wrapper
result = fn(cls, context, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line
1152, in get_by_host_and_node
expected_attrs)
File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line
1068, in _make_instance_list
expected_attrs=expected_attrs)
File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 501,
in _from_db_object
db_inst.get('extra').get('numa_topology'))
AttributeError: 'NoneType' object has no attribute 'get'
2015-04-20 17:35:09.301 15508 ERROR nova.openstack.common.threadgroup
[req-12483464-12a6-4b74-a671-bc6bb943b265 - - - - -] 'NoneType' object has no
attribute 'get'
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup
Traceback (most recent call last):
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line
145, in wait
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup
x.wait()
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/threadgroup.py", line
47, in wait
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup
return self.thread.wait()
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup File
"/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 175, in wait
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup
return self._exit_event.wait()
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup File
"/usr/lib/python2.7/dist-packages/eventlet/event.py", line 121, in wait
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup
return hubs.get_hub().switch()
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup File
"/usr/lib/python2.7/dist-packages/eventlet/hubs/hub.py", line 294, in switch
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup
return self.greenlet.switch()
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup File
"/usr/lib/python2.7/dist-packages/eventlet/greenthread.py", line 214, in main
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup
result = function(*args, **kwargs)
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup File
"/usr/lib/python2.7/dist-packages/nova/openstack/common/service.py", line 497,
in run_service
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup
service.start()
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup File
"/usr/lib/python2.7/dist-packages/nova/service.py", line 183, in start
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup
self.manager.pre_start_hook()
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1287, in
pre_start_hook
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup
self.update_available_resource(nova.context.get_admin_context())
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup File
"/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6236, in
update_available_resource
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup
rt.update_available_resource(context)
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup File
"/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 402,
in update_available_resource
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup
self._update_available_resource(context, resources)
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup File
"/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py", line 445, in
inner
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup
return f(*args, **kwargs)
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup File
"/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 436,
in _update_available_resource
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup
'numa_topology'])
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup File
"/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 163, in wrapper
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup
result = fn(cls, context, *args, **kwargs)
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup File
"/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1152, in
get_by_host_and_node
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup
expected_attrs)
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup File
"/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 1068, in
_make_instance_list
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup
expected_attrs=expected_attrs)
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup File
"/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 501, in
_from_db_object
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup
db_inst.get('extra').get('numa_topology'))
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup
AttributeError: 'NoneType' object has no attribute 'get'
2015-04-20 17:35:09.301 15508 TRACE nova.openstack.common.threadgroup
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1446082/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
