Hi again,
I've dug deeper into this... those errors appear because the tgtd
configuration on /etc/cinder/volumes/ on the block server hosting the
instance's volume is erased during the live migration; "tgtadm --lld
iscsi --mode target --op show" does not show the volume any more - on
ANY of the block servers.
Basically the volume is crippled (but shown as ok with cinder list).
This seems like a bug, but I'll let smarter people decide ;-)
Let me explain...
Let's say we have an instance running on compute1 with all storage on a
volume which resides on block1. From the CLI we run nova live-migration
towards compute2. What happens, is the instance config gets moved to
compute2, the iSCSI target is discovered from compute2 (which is why
/dev/disk/by/path shows the volume as existing) BUT after this, the
iSCSI target definition on block1:/etc/cinder/volumes/ is deleted (and
tgtd reloaded) - as such, compute2 cannot work properly and fdisk -l on
compute2 does not show the disk device corresponding to the iSCSI volume.
I've circumvented it manually by making a backup of the volume
definition file from /etc/cinder/volumes/ and restoring it immediately
after the deletion then reloading tgtd, and the migration succeeded.
Not a real solution, but until this gets fixed....
Best regards,
Andrei
On 05/22/2014 12:10 PM, [email protected] wrote:
Hello,
I'm trying to live-migrate a volume-based instance and it fails. I
have successfully done this in Havana, but Icehouse either works
differently or I'm missing something.
I'm using Centos6.5 with KVM, local firewalls and selinux are disabled
on all nodes.
I have a controller, 3 compute nodes (compute1 to compute3) and 3
block storage nodes (block1 to block3).
I'm creating an instance using "boot-from-image-creates-new-volume";
the instance gets created on compute2 using a volume created on block2
and works fine, although I can see
2014-05-21 16:31:48.487 1462 WARNING nova.virt.libvirt.driver
[req-e53e2353-2073-4a09-8d3b-94298a982e2d
38f06b6b30b241e0a40b6a3578e166e4 b33491f24b6b459291fe072067aac478]
[instance: a20aa279-4701-47c7-9be1-ae5846a00d91] File injection into a
boot from volume instance is not supported
I have made the modifications to libvirtd.conf, nova.conf and
/etc/sysconfig/libvirtd suggested here:
http://kimizhang.wordpress.com/2013/08/26/openstack-vm-live-migration/
From the CLI, I do
nova live-migration a20aa279-4701-47c7-9be1-ae5846a00d91
compute3.gbif.clu
With havana this used to work; with Icehouse, the instance ends up on
compute3, with status 'Active' but power state is 'Paused'. No errors
in the compute or libvirtd logs of source compute node, but on
destination (compute3) comput.log says:
2014-05-21 17:00:37.135 1467 WARNING nova.compute.manager [-]
[instance: a20aa279-4701-47c7-9be1-ae5846a00d91] Instance is paused
unexpectedly. Ignore.
Instance stopped with 'nova stop a20aa279-4701-47c7-9be1-ae5846a00d91'.
Trying to start with 'instance start' results in the following erros
in libvirtd.log on compute3:
2014-05-22 09:53:19.396+0000: 1443: error : virFileOpenAs:1093 :
failed to open file
'/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1':
No such device or address
2014-05-22 09:53:19.396+0000: 1443: error :
virStorageFileGetMetadataRecurse:939 : cannot open file
'/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1':
No such device or address
with corresponding errors in compute3's compute.log (see below).
The thing is, on compute3, we have:
[root@compute3 nova]# ls -alh
/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1
lrwxrwxrwx 1 root root 9 May 22 11:49
/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1
-> ../../sdb
so the symlink exists....
'Regular' migration works on the installation with this type of volume
based instances.
I was wondering if anyone could shed any light on this.
Thanks!
Best regards,
Andrei
Compute.log full errors:
2014-05-22 11:53:17.668 1467 ERROR glanceclient.common.http [-]
Request returned failure status.
2014-05-22 11:53:17.668 1467 WARNING nova.compute.utils
[req-5c09ada2-2072-4858-91a7-063b3d6224ed
38f06b6b30b241e0a40b6a3578e166e4 b33491f24b6b459291fe072067aac478]
[instance: a20aa279-4701-47c7-9be1-ae5846a00d91] Can't access image :
Image could not be found.
2014-05-22 11:53:19.398 1467 ERROR nova.virt.libvirt.driver
[req-5c09ada2-2072-4858-91a7-063b3d6224ed
38f06b6b30b241e0a40b6a3578e166e4 b33491f24b6b459291fe072067aac478] An
error occurred while trying to launch a defined domain with xml:
<domain type='kvm'>
<name>instance-00000015</name>
<uuid>a20aa279-4701-47c7-9be1-ae5846a00d91</uuid>
<memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory>
<vcpu placement='static'>1</vcpu>
<sysinfo type='smbios'>
<system>
<entry name='manufacturer'>Red Hat Inc.</entry>
<entry name='product'>OpenStack Nova</entry>
<entry name='version'>2014.1-2.el6</entry>
<entry name='serial'>96ccf268-2a3e-11dc-a7d0-000ea68f7399</entry>
<entry name='uuid'>a20aa279-4701-47c7-9be1-ae5846a00d91</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='rhel6.5.0'>hvm</type>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-model'>
<model fallback='allow'/>
</cpu>
<clock offset='utc'>
<timer name='pit' tickpolicy='delay'/>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source
dev='/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1'/>
<target dev='vda' bus='virtio'/>
<serial>b897a227-e7ca-4179-a17d-3e00700557a4</serial>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
</disk>
<controller type='usb' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x2'/>
</controller>
<interface type='bridge'>
<mac address='fa:16:3e:78:0f:fa'/>
<source bridge='br100'/>
<model type='virtio'/>
<filterref filter='nova-instance-instance-00000015-fa163e780ffa'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
</interface>
<serial type='file'>
<source
path='/var/lib/nova/instances/a20aa279-4701-47c7-9be1-ae5846a00d91/console.log'/>
<target port='0'/>
</serial>
<serial type='pty'>
<target port='1'/>
</serial>
<console type='file'>
<source
path='/var/lib/nova/instances/a20aa279-4701-47c7-9be1-ae5846a00d91/console.log'/>
<target type='serial' port='0'/>
</console>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0'
keymap='en-us'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='cirrus' vram='9216' heads='1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
</video>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x0'/>
</memballoon>
</devices>
</domain>
2014-05-22 11:53:19.896 1467 ERROR oslo.messaging.rpc.dispatcher [-]
Exception during message handling: cannot open file
'/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1':
No such device or address
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
Traceback (most recent call last):
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
line 133, in _dispatch_and_reply
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
incoming.message))
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
line 176, in _dispatch
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
return self._do_dispatch(endpoint, method, ctxt, args)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
line 122, in _do_dispatch
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
result = getattr(endpoint, method)(ctxt, **new_args)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/exception.py", line 88, in wrapped
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher payload)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/exception.py", line 71, in wrapped
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
return f(self, context, *args, **kw)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 280,
in decorated_function
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher pass
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 266,
in decorated_function
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
return function(self, context, *args, **kwargs)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 333,
in decorated_function
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
function(self, context, *args, **kwargs)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 309,
in decorated_function
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher e,
sys.exc_info())
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 296,
in decorated_function
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
return function(self, context, *args, **kwargs)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2300,
in start_instance
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
self._power_on(context, instance)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2288,
in _power_on
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
block_device_info)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
2128, in power_on
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
self._hard_reboot(context, instance, network_info, block_device_info)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
2092, in _hard_reboot
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
vifs_already_plugged=True)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
3644, in _create_domain_and_network
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
power_on=power_on)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
3547, in _create_domain
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
domain.XMLDesc(0))
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
six.reraise(self.type_, self.value, self.tb)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
3542, in _create_domain
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
domain.createWithFlags(launch_flags)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 179, in doit
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 139, in
proxy_call
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher rv =
execute(f,*args,**kwargs)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 77, in tworker
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher rv =
meth(*args,**kwargs)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher File
"/usr/lib64/python2.6/site-packages/libvirt.py", line 708, in
createWithFlags
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher if
ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed',
dom=self)
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
libvirtError: cannot open file
'/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1':
No such device or address
2014-05-22 11:53:19.896 1467 TRACE oslo.messaging.rpc.dispatcher
2014-05-22 11:53:19.898 1467 ERROR oslo.messaging._drivers.common [-]
Returning exception cannot open file
'/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1':
No such device or address to caller
2014-05-22 11:53:19.898 1467 ERROR oslo.messaging._drivers.common [-]
['Traceback (most recent call last):\n', ' File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
line 133, in _dispatch_and_reply\n incoming.message))\n', ' File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
line 176, in _dispatch\n return self._do_dispatch(endpoint, method,
ctxt, args)\n', ' File
"/usr/lib/python2.6/site-packages/oslo/messaging/rpc/dispatcher.py",
line 122, in _do_dispatch\n result = getattr(endpoint,
method)(ctxt, **new_args)\n', ' File
"/usr/lib/python2.6/site-packages/nova/exception.py", line 88, in
wrapped\n payload)\n', ' File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__\n six.reraise(self.type_, self.value,
self.tb)\n', ' File
"/usr/lib/python2.6/site-packages/nova/exception.py", line 71, in
wrapped\n return f(self, context, *args, **kw)\n', ' File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 280,
in decorated_function\n pass\n', ' File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__\n six.reraise(self.type_, self.value,
self.tb)\n', ' File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 266,
in decorated_function\n return function(self, context, *args,
**kwargs)\n', ' File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 333,
in decorated_function\n function(self, context, *args,
**kwargs)\n', ' File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 309,
in decorated_function\n e, sys.exc_info())\n', ' File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__\n six.reraise(self.type_, self.value,
self.tb)\n', ' File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 296,
in decorated_function\n return function(self, context, *args,
**kwargs)\n', ' File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2300,
in start_instance\n self._power_on(context, instance)\n', ' File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 2288,
in _power_on\n block_device_info)\n', ' File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
2128, in power_on\n self._hard_reboot(context, instance,
network_info, block_device_info)\n', ' File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
2092, in _hard_reboot\n vifs_already_plugged=True)\n', ' File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
3644, in _create_domain_and_network\n power_on=power_on)\n', ' File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
3547, in _create_domain\n domain.XMLDesc(0))\n', ' File
"/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py",
line 68, in __exit__\n six.reraise(self.type_, self.value,
self.tb)\n', ' File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line
3542, in _create_domain\n domain.createWithFlags(launch_flags)\n', '
File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 179,
in doit\n result = proxy_call(self._autowrap, f, *args,
**kwargs)\n', ' File
"/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 139, in
proxy_call\n rv = execute(f,*args,**kwargs)\n', ' File
"/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 77, in
tworker\n rv = meth(*args,**kwargs)\n', ' File
"/usr/lib64/python2.6/site-packages/libvirt.py", line 708, in
createWithFlags\n if ret == -1: raise libvirtError
(\'virDomainCreateWithFlags() failed\', dom=self)\n', "libvirtError:
cannot open file
'/dev/disk/by-path/ip-10.0.33.22:3260-iscsi-iqn.2010-10.org.openstack:volume-b897a227-e7ca-4179-a17d-3e00700557a4-lun-1':
No such device or address\n"]
_______________________________________________
Mailing list:
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : [email protected]
Unsubscribe :
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
_______________________________________________
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : [email protected]
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
