This is an automated email from the ASF dual-hosted git repository.
weizhouapache pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/cloudstack.git
The following commit(s) were added to refs/heads/main by this push:
new 9f281c48517 NAS backup: resume paused VM on backup failure and fix
missing exit (#12822)
9f281c48517 is described below
commit 9f281c48517d89495109f36c29635e039e20bd07
Author: James Peru Mmbono <[email protected]>
AuthorDate: Tue Jul 7 14:04:50 2026 +0300
NAS backup: resume paused VM on backup failure and fix missing exit (#12822)
* NAS backup: resume paused VM on backup failure and fix missing exit
When a NAS backup job fails (e.g. due to backup storage being full or
I/O errors), the VM may remain indefinitely paused because:
1. The cleanup() function never checks or resumes the VM's paused state
that was set by virsh backup-begin during the push backup operation.
2. The 'Failed' case in the backup job monitoring loop calls cleanup()
but lacks an 'exit' statement, causing an infinite loop where the
script repeatedly detects the failed job and calls cleanup().
3. Similarly, backup_stopped_vm() calls cleanup() on qemu-img convert
failure but does not exit, allowing the loop to continue with
subsequent disks despite the failure.
This fix:
- Adds VM state detection and resume to cleanup(), ensuring the VM is
always resumed if found in a paused state during error handling
- Adds missing 'exit 1' after cleanup() in the Failed backup job case
to prevent the infinite monitoring loop
- Adds missing 'exit 1' after cleanup() in backup_stopped_vm() on
qemu-img convert failure
Co-Authored-By: Claude Opus 4.6 <[email protected]>
* ci: retrigger workflow (flaky/stale shards)
* nasbackup.sh: keep cleanup() best-effort if domstate fails
With set -eo pipefail, a non-zero virsh domstate (libvirt unavailable or the
domain gone) in the vm_state assignment would abort cleanup() before the
rm/umount/rmdir. Append '|| true' so cleanup always runs to completion.
Addresses the review suggestion (Copilot, endorsed by @abh1sar and
@weizhouapache).
Signed-off-by: James Peru <[email protected]>
---------
Signed-off-by: James Peru <[email protected]>
Co-authored-by: Claude Opus 4.6 <[email protected]>
Co-authored-by: jmsperu <[email protected]>
---
scripts/vm/hypervisor/kvm/nasbackup.sh | 17 ++++++++++++++++-
1 file changed, 16 insertions(+), 1 deletion(-)
diff --git a/scripts/vm/hypervisor/kvm/nasbackup.sh
b/scripts/vm/hypervisor/kvm/nasbackup.sh
index 441312f35e8..e7ad3657cf0 100755
--- a/scripts/vm/hypervisor/kvm/nasbackup.sh
+++ b/scripts/vm/hypervisor/kvm/nasbackup.sh
@@ -167,7 +167,8 @@ backup_running_vm() {
break ;;
Failed)
echo "Virsh backup job failed"
- cleanup ;;
+ cleanup
+ exit 1 ;;
esac
sleep 5
done
@@ -221,6 +222,7 @@ backup_stopped_vm() {
if ! qemu-img convert -O qcow2 "$disk" "$output" > "$logFile" 2> >(cat
>&2); then
echo "qemu-img convert failed for $disk $output"
cleanup
+ exit 1
fi
name="datadisk"
done
@@ -265,6 +267,19 @@ mount_operation() {
cleanup() {
local status=0
+ # Resume the VM if it was paused during backup to prevent it from
+ # remaining indefinitely paused when the backup job fails (e.g. due
+ # to storage full or I/O errors on the backup target)
+ local vm_state
+ vm_state=$(virsh -c qemu:///system domstate "$VM" 2>/dev/null || true)
+ if [[ "$vm_state" == "paused" ]]; then
+ log -ne "Resuming paused VM $VM during backup cleanup"
+ if ! virsh -c qemu:///system resume "$VM" > /dev/null 2>&1; then
+ echo "Failed to resume VM $VM"
+ status=1
+ fi
+ fi
+
rm -rf "$dest" || { echo "Failed to delete $dest"; status=1; }
umount "$mount_point" || { echo "Failed to unmount $mount_point"; status=1; }
rmdir "$mount_point" || { echo "Failed to remove mount point $mount_point";
status=1; }
