prashanthr2 opened a new issue, #14720:
URL: https://github.com/apache/cloudstack/issues/14720

   ### problem
   
   Live migration with storage (migrateVirtualMachineWithVolume) between NFS 
primary storage pools fails for any VM deployed from a direct-download 
template, whatever the root disk’s provisioning type (THIN, SPARSE or FAT).
   
   On NFS, every direct download stores the template under a new random file 
name, so the same template has a different install_path on each pool:
   
   ```
   mysql> select id,unique_name,name,uuid,direct_download,url from vm_template 
where id=390\G
   id: 390
   unique_name: 390-2-fb622993-cd7c-3cda-8b06-8fe101c12419
   name: Ubuntu_direct
   uuid: 416766f3-8c5c-4c30-bdd4-681b095e1cc2
   direct_download: 1
   
   
   
   mysql> select id,pool_id,template_id,created,local_path,install_path,state 
from template_spool_ref where template_id=390\G
            
   id: 37
   pool_id: 2
   template_id: 390
   created: 2026-11-16 14:54:59
   install_path: d7e40725-ef5d-4620-b616-e12549be3de8
   state: Ready
             
   id: 38
   pool_id: 1
   template_id: 390
   created: 2026-11-16 15:01:50
   install_path: 514da2b1-bc37-47dc-8b3c-7772f7427952
   state: Ready
   
   ```
   
   When migrating VM 361 (root volume 399, on pool 2) to pool 1, the management 
server (StorageSystemDataMotionStrategy.createLinkedCloneMigrationOptions) 
sends the destination pool’s (pool 1) template file name as srcBackingFilePath, 
with copySrcTemplate=false:
   ```
   CreateObjectCommand ... "volumeId":"401", "poolId":"1", 
"provisioningType":"THIN",
   
"migrationOptions":{"srcPoolUuid":"68f38850-a051-3ff1-9865-6a24a2ba2864","srcPoolType":"NetworkFilesystem",
   "srcPoolClusterId":"1","type":"LinkedClone","scopeType":"CLUSTER",
   
"srcBackingFilePath":"514da2b1-bc37-47dc-8b3c-7772f7427952","copySrcTemplate":"false","timeout":"10800"}
   ```
   
   srcPoolUuid is pool 2 (68f38850…), but 514da2b1… is the template’s file name 
on pool 1. On pool 2 the file is d7e40725….
   
   The agent (KVMStorageProcessor.createLinkedCloneVolume) always runs 
srcPool.getPhysicalDisk(srcBackingFilePath) on the source pool, even when 
copySrcTemplate=false, so the lookup fails:
   ```
   DEBUG [kvm.storage.LibvirtStorageAdaptor] Could not find volume <dest 
template name>: Storage volume not found: no storage vol with matching name 
'<dest template name>'
   DEBUG [kvm.storage.LibvirtStorageAdaptor] Refreshing storage pool <source 
pool uuid>
   DEBUG [kvm.storage.LibvirtStoragePool] volume: <dest template name> not 
exist on storage pool
   DEBUG [kvm.storage.KVMStorageProcessor] Failed to create volume: 
com.cloud.utils.exception.CloudRuntimeException: Can't find volume:<dest 
template name>
           at 
com.cloud.hypervisor.kvm.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStoragePool.java:179)
           at 
com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.createLinkedCloneVolume(KVMStorageProcessor.java:1709)
           at 
com.cloud.hypervisor.kvm.storage.KVMStorageProcessor.createVolume(KVMStorageProcessor.java:1753)
   ```
   
   Sending the source pool’s name instead wouldn’t fix it. With 
copySrcTemplate=false, the agent then looks up the same name on the destination 
pool. A single srcBackingFilePath is used for both pools, which only works when 
the template has the same file name on both.
   
   Before the migration, copyTemplateToTargetFilesystemStorageIfNeeded stages a 
direct-download template on the destination pool through a new direct download, 
which also gets a new random name. So the names always differ for these VMs.
   
   
   **Related problems in the same flow:**
   
   **1. LinkedClone is chosen for disks with no backing file.** 
   decideMigrationTypeAndCopyTemplateIfNeeded chooses LinkedClone based only on 
template_spool_ref and volumes.template_id. SPARSE and FAT root disks are full 
copies (qemu-img convert in LibvirtStorageAdaptor.createDiskFromTemplate) with 
no backing file, but they’re still migrated as LinkedClone. That also sets 
migrateNonSharedInc, so only the top layer is copied onto a new disk layered on 
the template. A SPARSE VM from the same template in the lab has no backing file:
   ```
   <source 
file='/mnt/68f38850-a051-3ff1-9865-6a24a2ba2864/fc49e52d-ef71-4a5f-965e-944dfde1ca67'
 index='2'/>
   <backingStore/>
   
   # qemu-img info -U --backing-chain 
/mnt/68f38850-.../fc49e52d-ef71-4a5f-965e-944dfde1ca67
   file format: qcow2
   virtual size: 100 GiB (107374182400 bytes)
   disk size: 3.16 GiB
   (no backing file)
   ```
   
   In the production occurrence (provisioning type SPARSE, root disk confirmed 
to have no backing file), the migration was also sent as "type":"LinkedClone".
   
   **2. The failed CreateObjectAnswer is ignored.** 
   The management server logs Received: { Ans: ... { CreateObjectAnswer } } 
with result=false and continues, sending ModifyTargetsCommand for the volume 
that was never created. That command took about 5 minutes to fail:
   ```
   10:28:50,908 Sending { Cmd ... ModifyTargetsCommand ... "iqn":"<new volume 
uuid>" ...
   10:33:54,937 Details from executing class 
com.cloud.agent.api.ModifyTargetsCommand: 
com.cloud.utils.exception.CloudRuntimeException: Can't find volume:<new volume 
uuid>
   ```
   
   **3. A ClassCastException hides the real error:**
   ```
   ERROR [o.a.c.e.o.VolumeOrchestrator] Failed to migrate VM [VM instance 
{...}] along with its volumes due to
   [java.lang.ClassCastException: class com.cloud.agent.api.Answer cannot be 
cast to class com.cloud.agent.api.ModifyTargetsAnswer]
   ```
   
   **4. The new volume row is left in Migrating after the failure:**
   `| 708 | <new volume uuid> | <new volume uuid> | 7 | 205 | Migrating |`
   
   ### versions
   
   4.22.1.1 
   
   ### The steps to reproduce the bug
   
   1. Register a QCOW2 template with directdownload=true, hypervisor KVM.
   2. Create two NFS primary storage pools (pool 1 and pool 2) in the same 
cluster.
   3. Deploy a VM from the template with its root disk on pool 2. The template 
is downloaded to pool 2 under a random name (d7e40725…).
   - With a THIN offering, the root disk is layered on the template.
   - With a SPARSE offering, the root disk is a full copy with no backing file 
(also reproduces the problem).
   
   4. Live-migrate the VM with its root volume to pool 1 using 
migrateVirtualMachineWithVolume (host 2 / pool 2 → host 1 / pool 1).
   5. The template is downloaded to pool 1 under another random name 
(514da2b1…). Check with:`select 
id,pool_id,template_id,created,install_path,state from template_spool_ref where 
template_id=<template id>;`
   7. In the agent log, CreateObjectCommand has "type":"LinkedClone", 
"copySrcTemplate":"false" and "srcBackingFilePath":"<pool 1 name>", followed by 
Can't find volume:<pool 1 name> on pool 2.
   8. The management server reports the ClassCastException, and the new volume 
row stays in Migrating.
   
   ### What to do about it?
   
   **Fixes:**
   
   1. Agent: in KVMStorageProcessor.createLinkedCloneVolume, only look up 
srcTemplate on the source pool when copySrcTemplate=true. Or the management 
server should send the source and destination template file names in separate 
fields.
   2. Management server: in decideMigrationTypeAndCopyTemplateIfNeeded, choose 
LinkedClone or FullClone based on whether the volume actually has a backing 
file, or at least on its provisioning type, since SPARSE and FAT disks never 
have one.
   3. Management server: stop the migration when CreateObjectAnswer reports 
result=false, and handle a plain Answer coming back from ModifyTargetsCommand, 
so the real error is reported.
   4. Management server: clean up the destination volume row when the migration 
fails.
   
   
   **Workarounds until fixed:**
   
   1. Offline migration (any VM; confirmed): stop the VM, migrate the volume 
with migrateVolume, then start the VM. This avoids the live linked-clone code 
path. Requires downtime.
   2. Hard link (VMs whose root disk is layered on the template, e.g. THIN): 
give the source pool’s template file the destination’s file name, so the 
agent’s lookup succeeds:
   ```
   # confirm both copies are identical first
   qemu-img compare /mnt/<src-pool-uuid>/<src install_path> 
/mnt/<dest-pool-uuid>/<dest install_path>
   # e.g. ln /mnt/68f38850-.../d7e40725-... /mnt/68f38850-.../514da2b1-...
   ln /mnt/<src-pool-uuid>/<src install_path> /mnt/<src-pool-uuid>/<dest 
install_path>
   ```
   Retry the live migration, then remove the extra name with rm 
/mnt/<src-pool-uuid>/<dest install_path>. Do not use this for disks without a 
backing file (see workaround 3): the migration would copy only the disk’s 
written data onto a new disk layered on the template.
   
   3. Force a full-clone migration (SPARSE/FAT disks with no backing file): 
back up the database, then:
   `UPDATE volumes SET template_id = NULL WHERE id = <root volume id>;`
   
   Retry the live migration. decideMigrationTypeAndCopyTemplateIfNeeded then 
chooses FullClone (empty disk on the destination, full copy by libvirt). 
Afterwards, set template_id back to the template id on the new volume row (the 
one that now carries the volume’s uuid). Validate this in a test environment 
first.
   
   4. Clean up after failed attempts: remove or mark as expunged any 
destination volume rows left in Migrating (back up the database first) before 
retrying, so they don’t interfere with later operations.
   
   5. To avoid the problem for new VMs: use templates that are not 
direct-download for VMs that may need live migration with storage, so the 
template isn’t downloaded separately to each pool under a different name.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to