Hi, We've been seeing an issue in one of our (semi-)production systems running Icehouse on CentOS 6.5 in which VMs fail to spawn due to a "corrupted image". Nova compute log shows this:
2015-09-16 18:38:17.439 8279 ERROR nova.compute.manager [req-a77819b1-7137-4c1d-9465-3f6d55c64c4c ****** 27076b1b035c4f278a581c3d938360a7] [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] Instance failed to spawn 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] Traceback (most recent call last): 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1737, in _spawn 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] block_device_info) 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2287, in spawn 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] admin_pass=admin_password) 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2656, in _create_image 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] project_id=instance['project_id']) 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 192, in cache 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] *args, **kwargs) 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 383, in create_image 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] prepare_template(target=base, max_size=size, *args, **kwargs) 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] File "/usr/lib/python2.6/site-packages/nova/openstack/common/lockutils.py", line 249, in inner 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] return f(*args, **kwargs) 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/imagebackend.py", line 182, in fetch_func_sync 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] fetch_func(target=target, *args, **kwargs) 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/utils.py", line 653, in fetch_image 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] max_size=max_size) 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] File "/usr/lib/python2.6/site-packages/nova/virt/images.py", line 78, in fetch_to_raw 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] max_size=max_size) 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] File "/usr/lib/python2.6/site-packages/nova/virt/images.py", line 72, in fetch 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] image_service.download(context, image_id, dst_path=path) 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] File "/usr/lib/python2.6/site-packages/nova/image/glance.py", line 342, in download 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] for chunk in image_chunks: 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] File "/usr/lib/python2.6/site-packages/glanceclient/common/http.py", line 504, in __iter__ 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] 'expected %s' % (md5sum, self._checksum)) 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] IOError: [Errno 32] Corrupted image. Checksum was 43525c5261f518e96831b93ffa530afe expected 60589b89b66722a4e41c1607a4385882 2015-09-16 18:38:17.439 8279 TRACE nova.compute.manager [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] 2015-09-16 18:38:17.441 8279 DEBUG nova.compute.claims [req-a77819b1-7137-4c1d-9465-3f6d55c64c4c ****** 27076b1b035c4f278a581c3d938360a7] [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] Aborting claim: [Claim: 4096 MB memory, 40 GB disk, 2 VCPUS] abort /usr/lib/python2.6/site-packages/nova/compute/claims.py:113 2015-09-16 18:38:17.473 8279 DEBUG nova.compute.utils [req-a77819b1-7137-4c1d-9465-3f6d55c64c4c ****** 27076b1b035c4f278a581c3d938360a7] [instance: 3cd644f2-c392-4306-8463-4fe2708d31e1] [Errno 32] Corrupted image. Checksum was 43525c5261f518e96831b93ffa530afe expected 60589b89b66722a4e41c1607a4385882 notify_about_instance_usage /usr/lib/python2.6/site-packages/nova/compute/utils.py:336 This bug looks related: https://bugs.launchpad.net/nova/+bug/1350766 However, after a few patches were proposed to the above, it seems that the bug was classified as a qemu issue: * https://bugs.launchpad.net/qemu/+bug/1368815 * https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1292234 It's not clear, however, if these bugs are what exactly what we're seeing and I could not replicate the problem with the manual steps described in them. Also, I couldn't figure out what versions of qemu have the fix and if the particular version of qemu-img is broken or not. This is the package we have installed: Name : qemu-img Arch : x86_64 Epoch : 2 Version : 0.12.1.2 Release : 2.415.el6_5.14 Size : 969 k Repo : installed >From repo : C65-Updates We're using the default ext3 filesystems. Any help in solving the mystery here would be hugely appreciated. Thanks, -Sergio
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : [email protected] Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
