calvix opened a new pull request, #13922:
URL: https://github.com/apache/cloudstack/pull/13922
## Description
`restoreBackup` of a KVM instance with **more than one data disk** can write
a
backed up disk into a **different volume** than the one it was taken from.
This happens only on a second or further restore of a VM from a backup that was
taken. ie: create backup of VM, restore 1, then restore 2 again.
- **equal-sized data disks** → the volume contents are silently exchanged,
and the API returns `success: true`, but the appearance of the bug is not 100%
because the enumeration can randomly be the same as before. I recommend trying
instances with 5 or 7 data disks for testing where occurrence is very likely.
- **differently sized data disks** → the smaller image is written over the
larger volume, the restore then
fails converting the larger image into the smaller volume, and returns an
**error after the data is already
destroyed**, with no rollback
- the **ROOT disk is never affected** (always re-attached at device id 0, so
it sorts first in both lists)
- `createVMFromBackup` and `restoreVolumeFromBackupAndAttachToVM` are
**not** affected
Reproduced on 4.22.1.0 with KVM + Ceph/RBD primary storage and the NAS
backup provider.
At first, this can be a very silent error, especially if the VM uses mount
by LABEL/UUID, as the VM will mount the data to the right mount paths, but
volume A now contains the data of volume B ( including LABELs) and volume B
contains data for volume A.
## Logs
### case 1 - different disk sizes
The logs show output showing this bug on the VM, with 2 data volumes
`diska`, `diskb` - all stored on `Ceph primary storage` and backed up via
backup and recovery plugin to `cephfs` storage. `diskcheck` is a simple bash
util that prints useful data from the disks on a remote machine and connects
via ssh.
```
#: diskcheck 10.3.77.212
DEVICE SIZE LABEL MOUNT FS-SIZE FILE
vdb 1G DISKA /mnt/diska 974M DISKA=DISKA
vdc 2G DISKB /mnt/diskb 2.0G DISKB=DISKB
#: cmk -p stable list volumes
virtualmachineid=3d602d8a-c00c-4f49-a432-8c8f82237b3d filter=name,deviceid,size
{
"count": 3,
"volume": [
{
"deviceid": 0,
"name": "ROOT-959",
"size": 630159872
},
{
"deviceid": 1,
"name": "demo-disk-a",
"size": 1073741824
},
{
"deviceid": 2,
"name": "demo-disk-b",
"size": 2147483648
}
]
}
#: cmk -p stable stop virtualmachine id=3d602d8a-c00c-4f49-a432-8c8f82237b3d
{
...... redacted
}
#: cmk -p stable restore backup id=557b755b-6426-4daa-9c96-871281a4112f
{
"success": true
}
# Below we can see the deviceID changed for both data disks
#: cmk -p stable list volumes
virtualmachineid=3d602d8a-c00c-4f49-a432-8c8f82237b3d filter=name,deviceid
{
"count": 3,
"volume": [
{
"deviceid": 0,
"name": "ROOT-959"
},
{
"deviceid": 4,
"name": "demo-disk-a"
},
{
"deviceid": 1,
"name": "demo-disk-b"
}
]
}
#: cmk -p stable start virtualmachine
id=3d602d8a-c00c-4f49-a432-8c8f82237b3d
{
...
}
# Here we can see that disk ordering changed - this will basically break
any further restore
#: diskcheck 10.3.77.212
DEVICE SIZE LABEL MOUNT FS-SIZE FILE
vdb 2G DISKB /mnt/diskb 2.0G DISKB=DISKB
vdc 1G DISKA /mnt/diska 974M DISKA=DISKA
#: cmk -p stable stop virtualmachine id=3d602d8a-c00c-4f49-a432-8c8f82237b3d
{
...
}
# The second restore fails because it cannot restore 2GB disk into 1 GB
volume
#: cmk -p stable restore backup id=557b755b-6426-4daa-9c96-871281a4112f
{
"account": "admin",
"accountid": "65fed4ca-48b6-11f1-a321-bc24119cb31e",
"cmd": "org.apache.cloudstack.api.command.user.backup.RestoreBackupCmd",
"completed": "2026-08-19T11:35:55+0200",
"created": "2026-08-19T11:35:44+0200",
"domainid": "176895fd-48b6-11f1-a321-bc24119cb31e",
"domainpath": "ROOT",
"jobid": "3c7c45dc-68f1-4152-8276-a1f4cf03678f",
"jobprocstatus": 0,
"jobresult": {
"errorcode": 530,
"errortext": "Error restoring VM from backup
[{\"externalId\":\"i-2-959-VM\\/2026.08.19.11.28.52\",\"name\":\"demo\",\"uuid\":\"557b755b-6426-4daa-9c96-871281a4112f\",\"vmId\":959}]."
},
"jobresultcode": 530,
"jobresulttype": "object",
"jobstatus": 2,
"userid": "65ff40f2-48b6-11f1-a321-bc24119cb31e"
}
```
If the disk sizes are the same, the second restore works but disk can be
swapped depending on the scenario how disk are renumbered.
```
```
## Root cause
`NASBackupProvider.restoreVMBackup` builds two lists the backed up volumes
and the instance's current
volumes, each sorted by device id, and
`LibvirtRestoreBackupCommandWrapper.restoreVolumesOfExistingVM`
consumes them index by index:
`BackupManagerImpl.restoreBackup` - `importRestoredVM` -
`KVMGuru.importVirtualMachineFromBackup`
re-attaches every data disk with `getNextAvailableDeviceId()`. The volumes
are **never detached first**, so
their current ids still count as "in use" and the helper can never return
the id a volume already holds —
every data disk is shifted by one slot on **every** restore.
A restore is therefore only correct while the instance's device ids still
match the ones recorded in the
backup. The first restore satisfies that and then breaks it for the next one.
## Behaviour, restore by restore
The first restore is always correct. It is the restore itself that can plant
the fault, so the damage only
appears on second restore.
### Restore #1 — succeeds, but renumbers the disks
The device ids recorded in the backup still match the instance, so the two
sorted lists line up and every
backup lands in its own volume. Verified on a 2-data-disk instance by
reading the volumes directly from
primary storage — each still held its own filesystem.
But on completion `KVMGuru.importVirtualMachineFromBackup` re-attaches the
data disks with
`getNextAvailableDeviceId()`:
t
```
before restore #1: ROOT=0 demo-disk-a=1 demo-disk-b=2
after restore #1: ROOT=0 ddemo-disk-a=4 emo-disk-b=1 <--
renumbered
after restore #2: ROOT=0 demo-disk-a=2 demo-disk-b=4 <--
renumbered again
```
After the first restore, diskA gets id 4 because ids 1 and 2 are taken; id 3
is cdrom, so the next free id is 4. diskb gets id 1 because the id was just
freed.
### Restore #2 — the damage, and what it looks like depends on the disk sizes
Sorted by device id the instance is now `[ROOT, demo-disk-b, demo-disk-a]`
while the backup is
`[ROOT, demo-disk-a, demo-disk-b]`, so index 1 and 2 point at each other's
volumes.
**Equal-sized data disks (1 GB + 1 GB) — silent corruption.** Both images
fit their new targets, so
`qemu-img` succeeds and the API returns `success: true`. The volume contents
are simply exchanged:
```
demo-disk-a contains ext4 label: DISKB <-- swapped
demo-disk-b contains ext4 label: DISKA
```
A guest that mounts by `LABEL=`/`UUID=` still mounts everything at the right
paths, because the labels
travel with the filesystems — so nothing looks wrong from inside the
instance, while CloudStack's
volume-to-content mapping is now wrong for every per-volume operation
(detach, delete, snapshot, resize,
restore single volume).
Differently sized data disks (1 GB + 2 GB), data loss, reported as a
failure.
The loop processes the disks in order and only aborts on the second pairing,
so the first one has already been written.
The API returns:
```
"errorcode": 530,
"errortext": "Error restoring VM from backup
[{\"externalId\":\"i-2-959-VM/2026.08.19.11.28.52\", ...}]"
```
and the agent log shows the real cause — the 2 GB backup being written into
the 1 GB volume:
```
qemu-img convert -n -O raw --image-opts
driver=qcow2,file.filename=.../datadisk.806fa24b-a2da-45b1-9bd4-5fd51d0cc78b.qcow2
<- backup of disk B (2 GB)
rbd:cloudstack/da2029ad-8517-4f40-af03-bd5f3ec5d959
<- volume of disk A (1 GB)
encountered the error: [qemu-img: output file is smaller than input]
```
State afterwards — **the restore is reported as failed, but one volume is
already gone, with no rollback**:
```
demo-disk-a (1 GiB) ext4-label = DISKA <- untouched, the write into it
failed
demo-disk-b (2 GiB) ext4-label = DISKA <- OVERWRITTEN, its own data
now exists only in the backup
```
Both volumes still report `Ready` in CloudStack. An operator seeing error
530 would reasonably assume
nothing happened.
## How to reproduce
Instance with two data disks, a NAS backup offering, KVM primary storage.
Use two different sizes to get
the loud failure, if you want to reproduce the disk swap, I recommend using
more data disks, as there is a chance that reumbering accdientaly get right and
the bug does not occur. So trying like 5 or 7 data disks should most likely
show the bug.
```bash
# 1. baseline: device ids 1 and 2
cmk list volumes virtualmachineid=$VM filter=name,deviceid
# ROOT deviceid=0 | demo-disk-a deviceid=1 | demo-disk-b deviceid=2
# 2. label the filesystems so they can be told apart
# mkfs.ext4 -L DISKA /dev/vdb ; mkfs.ext4 -L DISKB /dev/vdc
cmk create backup virtualmachineid=$VM name=demo
cmk stop virtualmachine id=$VM
# 3. FIRST restore - correct, but renumbers the disks
cmk restore backup id=$B
cmk list volumes virtualmachineid=$VM filter=name,deviceid
# ROOT deviceid=0 | demo-disk-b deviceid=1 | demo-disk-a deviceid=4
<-- misaligned from here on
# 4. SECOND restore - silent swap (equal sizes) or error 530 after
destroying a volume (different sizes)
cmk restore backup id=$B
# if you have the same sizes, you could try from UI detach diskA ( after
either turning off teh safety config or disabling backups) and see that in OS
the different disk vanishes.
```
## Types of changes
- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to change)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
- [ ] build/CI
## How Has This Been Tested?
**Unit tests** — three added, all of which fail without the patch:
| | with the fix | without the fix |
|---|---|---|
| `KVMGuruTest` | 30 tests, 0 failures | 30 tests, **1 error** |
| `LibvirtRestoreBackupCommandWrapperTest` | 13 tests, 0 failures | 13
tests, **1 failure, 1 error** |
Every pre-existing test in both classes passes either way.
-
`LibvirtRestoreBackupCommandWrapperTest.testRestoreOfExistingVmMapsBackupsToVolumesByUuid`
— the
instance's volumes arrive in a different order than the backed up volumes;
each backup must still be
written into the volume it was taken from. Also asserts with `never()`
that neither data disk's backup is
written into the other one.
-
`LibvirtRestoreBackupCommandWrapperTest.testRestoreOfExistingVmFailsWhenBackedUpVolumeIsNoLongerAttached`
-
`KVMGuruTest.testImportVirtualMachineFromBackupReinstatesRecordedDeviceIds` — a
data disk recorded on
device id 5 must be re-attached on device id 5, and
`getNextAvailableDeviceId()` must not be consulted.
**Live testing** — a patched build was deployed to a KVM + Ceph/RBD zone
(management server and KVM
agents). The same 2-data-disk instance was restored **three times** in place:
- device ids stayed `0,1,2` on every restore (unpatched: `1,2` → `4,1` after
the first)
- both volumes kept their own filesystem, verified with `rbd export` and
from inside the guest
- a 5-data-disk instance backed up and restored with all checksums intact
- `createVMFromBackup` verified unaffected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
