abh1sar opened a new issue, #14691: URL: https://github.com/apache/cloudstack/issues/14691
### problem A volume snapshot of a running VM does three steps in KVMStorageProcessor: snapshotCreateXML(DISK_ONLY) moves the disk onto a new overlay /mnt/<pool>/<random uuid> (instant). qemu-img convert copies the frozen base to the snapshot location (minutes to hours, depends on volume size). virsh blockcommit --active --wait --pivot merges the overlay back into the base. If step 3 never starts, the VM keeps running on the overlay while volumes.path still points to the base. The next stop/start builds the domain from volumes.path, so every write since the snapshot started is silently lost (it only exists in the orphan overlay). After a start the base is written again, so the overlay can no longer simply be committed back. There are two ways step 3 never starts. Both were reproduced: A. The agent JVM goes away during the convert. SIGKILL / OOM kill of the agent. The agent’s own System.exit(1) in Agent.java (ReadyCommand with details) after a failed reconnect. Reproduced by making the management server fail the reconnect with Unable to acquire lock on host while the convert was running. SIGTERM to the JVM only. Snapshot ends in Error. A partial copy is left in <pool>/snapshots/<overlay uuid>. (systemctl stop/restart cloudstack-agent does not trigger it: the convert is killed with the JVM and the merge still runs.) B. virsh blockcommit fails before the job starts, and the failure is reported as success. Script.runScript() returns null when execute() returns an error, so mergeSnapshotIntoBaseFileWithoutEvents() gets null for any non-zero exit and logs Successfully merged snapshot. The snapshot ends BackedUp. Reproduced with: a NAS backup (backup-begin) started during the convert and still running at merge time: Operation not supported: disk 'vdb' already in active block job; the domain job held by another operation for more than 30 s: Timed out during operation: cannot acquire state change lock. NAS backups do not go through the VM work queue and do not check for volumes in Snapshotting, so nothing prevents this overlap. STEPS TO REPRODUCE Running VM with a data disk on NFS, write some data so the convert takes a while. Take a volume snapshot of the data disk. As soon as virsh domblklist shows the disk on a new bare-UUID file, either: kill -9 the agent JVM; or start a backup: virsh backup-begin <vm> (push or pull mode) and keep it running until the merge. Write data in the guest, then stop and start the VM from CloudStack. EXPECTED RESULTS The merge either completes, or the snapshot fails and the VM is not left on an overlay that CloudStack does not track. ACTUAL RESULTS The VM runs on /mnt/<pool>/<uuid> (backing file = the volume). Case B logs: WARN Process [N] for command [/bin/bash -c virsh blockcommit <vm> vdb --base ... --pivot ] encountered the error: [error: Operation not supported: disk 'vdb' already in active block job]. DEBUG Successfully merged snapshot [<overlay uuid>] into VM [<vm>] ... Later snapshots of the volume fail with VM [...] has no disk with path [<base>]. After stop/start the guest reads the data from the time the snapshot started. PROPOSED FIX Detect merge failures. Make Script.runScript() (or at least the merge call) return an error on a non-zero exit. After the merge, check that the domain disk source is the base path again. If the disk has another block job, wait and retry the commit. If it still fails, fail the snapshot with a clear error instead of BackedUp. Recover after agent death. Before creating the overlay, write a small marker (vm, disk, base, overlay) next to the snapshot copy. On agent start, for each marker whose domain is still on the overlay, run virsh blockcommit <vm> <disk> --active --pivot --wait and remove the marker. Do not exit with commands in progress. On a ReadyCommand with details, reconnect (as isExitOnFailures()=false already does) instead of System.exit(1), at least while commands are in progress. Serialize NAS backups with volume snapshots of the same VM (VM work queue or refuse while a volume is Snapshotting). ### versions The versions of ACS, hypervisors, storage, network etc.. ### The steps to reproduce the bug 1. 2. 3. ... ### What to do about it? _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
