Hanarion opened a new issue, #12122:
URL: https://github.com/apache/cloudstack/issues/12122

   ### problem
   
   It seems that in my current conditions, the backups are failing because of a 
CIFS warning that is failing the Backup even if the backup was indeed successful
   
   ```
   2025-11-24 11:05:32,938 ERROR [o.a.c.b.NASBackupProvider] 
(API-Job-Executor-11:[ctx-673f730c, job-3216, ctx-2492b04d]) (logid:2a7df0d3) 
Failed to take backup for VM i-12-606-VM: Job type:         Completed   
   Operation:        Backup      
   Time elapsed:     29235        ms
   File processed:   23.000 GiB
   File remaining:   0.000 B
   File total:       23.000 GiB
   
   2769033951
   2025-11-24 11:05:32,946 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] 
(API-Job-Executor-11:[ctx-673f730c, job-3216]) (logid:2a7df0d3) Complete async 
job-3216, jobStatus: FAILED, resultCode: 530, result: 
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
 to create VM backup"}
   ```
   
   I tried downgrading from smb 3.1.1 to 3.0, but it didn't fixed the issue, 
here is my full mount options :
   `//XXXX.XXXX/xxxxxx on /tmp/csbackup.YDH0M type cifs 
(rw,relatime,vers=3.0,cache=strict,upcall_target=app,username=xxxxxxx,uid=0,noforceuid,gid=0,noforcegid,addr=78.46.12.119,file_mode=0755,dir_mode=0755,soft,nounix,serverino,mapposix,nobrl,reparse=nfs,nativesocket,symlink=native,rsize=4194304,wsize=4194304,bsize=1048576,retrans=1,echo_interval=60,actimeo=1,closetimeo=1)`
   
   
   ### versions
   
   - CloudStack 4.22.0.0
   ```
   [root@compute01 ~]# uname -a
   Linux compute01 5.14.0-570.17.1.el9_6.x86_64 #1 SMP PREEMPT_DYNAMIC Fri May 
23 22:47:01 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
   ```
   ```
   [root@compute01 ~]# cat /etc/os-release 
   NAME="Rocky Linux"
   VERSION="9.6 (Blue Onyx)"
   ```
   ```
   [root@compute01 ~]# ls -lah 
/tmp/csbackup.YDH0M/i-12-606-VM/2025.11.24.11.04.56/
   total 1.4G
   drwxr-xr-x. 2 root root    0 Nov 24 11:04 .
   drwxr-xr-x. 2 root root    0 Nov 24 11:03 ..
   -rwxr-xr-x. 1 root root 2.8M Nov 24 11:03 
datadisk.96c4484b-2b51-46bd-9476-c7c9e0cedb87.qcow2
   -rwxr-xr-x. 1 root root 5.7K Nov 24 11:04 domain-config.xml
   -rwxr-xr-x. 1 root root  299 Nov 24 11:05 domblklist.xml
   -rwxr-xr-x. 1 root root  166 Nov 24 11:04 domiflist.xml
   -rwxr-xr-x. 1 root root  620 Nov 24 11:04 dominfo.xml
   -rwxr-xr-x. 1 root root 2.6G Nov 24 11:04 
root.98ee2a4e-8c87-4a2e-8b61-aac596fffebd.qcow2
   ```
   
   ### The steps to reproduce the bug
   
   1. Configure the NAS Backup plugin
   2. Add a CIFS (SMB) backup repository
   3. Try to do multiple backups (Sometimes it fails, sometimes it doesnt)
   
   
   ### What to do about it?
   
   
https://github.com/apache/cloudstack/blob/8171d9568c941877e0290f77051ed45f31a73544/scripts/vm/hypervisor/kvm/nasbackup.sh#L151
   
   It seems that the CIFS mount doesn't really treat `sync` as it should in my 
current setup, and stays busy even after syncing :
   ```
   [root@compute01 ~]# /usr/bin/bash 
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/nasbackup.sh -o backup 
-v i-12-606-VM -t cifs -s '//XXXXXXXX.xxxxxxxx/xxxxxxxx' -m 
'vers=3.0,username=xxxxxxx,password=xxxxxxxx' -p 
'i-12-606-VM/2025.11.24.12.02.09' -q false -d ''
   Job type:         Completed   
   Operation:        Backup      
   Time elapsed:     27079        ms
   File processed:   23.000 GiB
   File remaining:   0.000 B
   File total:       23.000 GiB
   
   2769033951
   umount: /tmp/csbackup.rnf14: target is busy.
   
   ```
   
   We could avoid this issue by checking if the mountpoint is still busy, for 
example with fuser (more available than lsof) :
   ```bash
     # Print statistics
     virsh -c qemu:///system domjobinfo $VM --completed
     du -sb $dest | cut -f1
   
     elapsed=0
     while fuser -m "$mount_point" >/dev/null 2>&1; do
       if (( elapsed >= 10 )); then
         echo "Timeout for unmounting reached: still busy"
         exit 1
       fi
       sleep 1
       elapsed=$((elapsed + 1))
     done
   
     umount $mount_point
     rmdir $mount_point
   ```
   
   Also when i checked the code for the script, it seems that the 
backup_stopped_vm is not umounting and removing the mount point, i don't know 
if that's on purpose.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to