I am doing a bit of R&D around this for a client at the moment. I am 
semi-successful in getting live migrations to different storage pools to work. 
The method I’m using is as follows – this does not take into account any 
efficiency optimisation around the disk transfer (which is next on my list). 
The below should answer your question Eric about moving to a different location 
– and I am also working with your steps to see where I can improve the 
following. Keep in mind all of this is external to CloudStack – although 
CloudStack picks up the destination KVM host automatically it does not update 
the volume tables etc., neither does it do any housekeeping.

1) Ensure the same network bridges are up on source and destination – these are 
found with:

[root@kvm1 ~]# virsh dumpxml 9 | grep source
      <source bridge='breth1-725'/>
      <source path='/dev/pts/3'/>
      <source path='/dev/pts/3'/>

So from this make sure breth1-725 is up on the destionation host (do it the 
hard way or cheat and spin up a VM from same account and network on that host)

2) Find size of source disk and create stub disk in destination (this part can 
be made more efficient to speed up disk transfer – by doing similar things to 
what Eric is doing):

[root@kvm1 ~]# qemu-img info 
file format: qcow2
virtual size: 8.0G (8589934592 bytes)
disk size: 32M
cluster_size: 65536
backing file: 


[root@kvm3 50848ff7-c6aa-3fdd-b487-27899bf2129c]# qemu-img create -f qcow2 
d0ab5dd5-e3dd-47ac-a326-5ce3d47d194d 8G
Formatting 'd0ab5dd5-e3dd-47ac-a326-5ce3d47d194d', fmt=qcow2 size=8589934592 
encryption=off cluster_size=65536
[root@kvm3 50848ff7-c6aa-3fdd-b487-27899bf2129c]# qemu-img info 
image: d0ab5dd5-e3dd-47ac-a326-5ce3d47d194d
file format: qcow2
virtual size: 8.0G (8589934592 bytes)
disk size: 448K
cluster_size: 65536

3) Rewrite the new VM XML file for the destination with:
a) New disk location, in this case this is just a new path (Eric – this answers 
your question)
b) Different IP addresses for VNC – in this case to
and carry out migration.

[root@kvm1 ~]# virsh dumpxml 9 | sed -e 
| sed -e 's/' > /root/i-2-14-VM.xml

[root@kvm1 ~]# virsh migrate --live --persistent --copy-storage-all --xml 
/root/i-2-14-VM.xml i-2-14-VM qemu+tcp:// --verbose 
Migration: [ 25 %]

4) Once complete delete the source file. This can be done with extra switches 
on the virsh migrate command if need be. 
= = = 

In the simplest tests this works – destination VM remains online and has 
storage in new location – but it’s not persistent – sometimes the destination 
VM ends up in a paused state, and I’m working on how to get around this. I also 
noted virsh migrate has a  migrate-setmaxdowntime which I think can be useful 

Dag Sonstebo
Cloud Architect

On 01/02/2018, 20:30, "Andrija Panic" <> wrote:

    Actually,  we have this feature (we call this internally
    online-storage-migration) to migrate volume from CEPH/NFS to SolidFire
    (thanks to Mike Tutkowski)
    There is libvirt mechanism, where basically you start another PAUSED VM on
    another host (same name and same XML file, except the storage volumes are
    pointing to new storage, different paths, etc and maybe VNC listening
    address needs to be changed or so) and then you issue on original host/VM
    the live migrate command with few parameters... the libvirt will
    transaprently handle the copy data process from Soruce to New volumes, and
    after migration the VM will be alive (with new XML since have new volumes)
    on new host, while the original VM on original host is destroyed....
    (I can send you manual for this, that is realted to SF, but idea is the
    same and you can exercies this on i.e. 2 NFS volumes on 2 different
    This mechanism doesn't exist in ACS in general (AFAIK), except for when
    migrating to SolidFire.
    Perhaps community/DEV can help extend Mike's code to do same work on
    different storage types...

53 Chandos Place, Covent Garden, London  WC2N 4HSUK

On 19 January 2018 at 18:45, Eric Green <> wrote:
    > KVM is able to live migrate entire virtual machines complete with local
    > volumes (see 'man virsh') but does require nbd (Network Block Device) to 
    > installed on the destination host to do so. It may need installation of
    > later libvirt / qemu packages from OpenStack repositories on Centos 6, I'm
    > not sure, but just works on Centos 7. In any event, I have used this
    > functionality to move virtual machines between virtualization hosts on my
    > home network. It works.
    > What is missing is the ability to live-migrate a disk from one shared
    > storage to another. The functionality built into virsh live-migrates the
    > volume ***to the exact same location on the new host***, so obviously is
    > useless for migrating the disk to a new location on shared storage. I
    > looked everywhere for the ability of KVM to live migrate a disk from point
    > A to point B all by itself, and found no such thing. libvirt/qemu has the
    > raw capabilities needed to do this, but it is not currently exposed as a
    > single API via the qemu console or virsh. It can be emulated via scripting
    > however:
    > 1. Pause virtual machine
    > 2. Do qcow2 snapshot.
    > 3. Detach base disk, attach qcow2 snapshot
    > 4. unpause virtual machine
    > 5. copy qcow2 base file to new location
    > 6. pause virtual machine
    > 7. detach snapshot
    > 8. unsnapshot qcow2 snapshot at its new location.
    > 9. attach new base at new location.
    > 10. unpause virtual machine.
    > Thing is, if that entire process is not built into the underlying
    > kvm/qemu/libvirt infrastructure as tested functionality with a defined 
    > there's no guarantee that it will work seamlessly and will continue 
    > with the next release of the underlying infrastructure. This is using
    > multiple different tools to manipulate the qcow2 file and attach/detach
    > base disks to the running (but paused) kvm domain, and would have to be
    > tested against all variations of those tools on all supported Cloudstack
    > KVM host platforms. The test matrix looks pretty grim.
    > By contrast, the migrate-with-local-storage process is built into virsh
    > and is tested by the distribution vendor and the set of tools provided 
    > the distribution is guaranteed to work with the virsh / libvirt/ qemu
    > distributed by the distribution vendor. That makes the test matrix for
    > move-with-local-storage look a lot simpler -- "is this functionality
    > supported by that version of virsh on that distribution? Yes? Enable it.
    > No? Don't enable it."
    > I'd love to have live migration of disks on shared storage with Cloudstack
    > KVM, but not at the expense of reliability. Shutting down a virtual 
    > in order to migrate one of its disks from one shared datastore to another
    > is not ideal, but at least it's guaranteed reliable.
    > > On Jan 19, 2018, at 04:54, Rafael Weingärtner <
    >> wrote:
    > >
    > > Hey Marc,
    > > It is very interesting that you are going to pick this up for KVM. I am
    > > working in a related issue for XenServer [1].
    > > If you can confirm that KVM is able to live migrate local volumes to
    > other
    > > local storage or shared storage I could make the feature I am working on
    > > available to KVM as well.
    > >
    > >
    > > [1]
    > >
    > > On Thu, Jan 18, 2018 at 11:35 AM, Marc-Aurèle Brothier <
    > > wrote:
    > >
    > >> There's a PR waiting to be fixed about live migration with local volume
    > for
    > >> KVM. So it will come at some point. I'm the one who made this PR but 
    > >> not using the upstream release so it's hard for me to debug the 
    > >> You can add yourself to the PR to get notify when things are moving on
    > it.
    > >>
    > >>
    > >>
    > >> On Wed, Jan 17, 2018 at 10:56 AM, Eric Green <>
    > >> wrote:
    > >>
    > >>> Theoretically on Centos 7 as the host KVM OS it could be done with a
    > >>> couple of pauses and the snapshotting mechanism built into qcow2, but
    > >> there
    > >>> is no simple way to do it directly via virsh, the libvirtd/qemu 
    > >>> program that is used to manage virtualization. It's not as with
    > issuing a
    > >>> simple vmotion 'migrate volume' call in Vmware.
    > >>>
    > >>> I scripted out how it would work without that direct support in
    > >>> libvirt/virsh and after looking at all the points where things could 
    > >>> wrong, honestly, I think we need to wait until there is support in
    > >>> libvirt/virsh to do this. virsh clearly has the capability internally
    > to
    > >> do
    > >>> live migration of storage, since it does this for live domain 
    > >> of
    > >>> local storage between machines when migrating KVM domains from one 
    > >> to
    > >>> another, but that capability is not currently exposed in a way
    > Cloudstack
    > >>> could use, at least not on Centos 7.
    > >>>
    > >>>
    > >>>> On Jan 17, 2018, at 01:05, Piotr Pisz <> wrote:
    > >>>>
    > >>>> Hello,
    > >>>>
    > >>>> Is there a chance that one day it will be possible to migrate volume
    > >>> (root disk) of a live VM in KVM between storage pools (in CloudStack)?
    > >>>> Like a storage vMotion in Vmware.
    > >>>>
    > >>>> Best regards,
    > >>>> Piotr
    > >>>>
    > >>>
    > >>>
    > >>
    > >
    > >
    > >
    > > --
    > > Rafael Weingärtner
    Andrija Panić

Reply via email to