draggeta opened a new issue, #13883:
URL: https://github.com/apache/cloudstack/issues/13883
### problem
When backing up up larger VMs (the ones we've tested have 2+ TB of storage),
scheduled and manual backups eventually fail to NFS storage. The files are
written until some sort of timeout seems to happen.
The files are available on the NFS storage but every backup they stop after
a certain period of time. A side effect of this is that cloudstack doesn't
apply retention to the backups in this case so they keep filling up storage.
Smaller servers (tested until 600 GB backup correctly. There is no firewall
in the path. The hypervisors and NFS storage are in the same subnet.
```
2026-08-15 06:04:06,169 DEBUG [o.a.c.b.BackupManagerImpl]
(BackupPollTask:[ctx-f0772149]) (logid:34dc33a0) Scheduling 1 backup for VM (VM
instance
{"id":1083,"instanceName":"i-10-1083-VM","state":"Running","type":"User","uuid":"cac7bcdc-2ab2-43b4-81ce-64a5e5d0bf59"})
for backup schedule (BackupSchedule
{"id":6,"schedule":"00:06:7","scheduleType":2,"vmId":1083}) at
[2026-08-15T06:00:00+0000].
2026-08-15 06:04:06,607 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:[ctx-98b8e7e5, job-5367, ctx-8cb83fa7]) (logid:b96ba19b)
Seq 10-4578471971175552676: Sending { Cmd , MgmtId: 90520740805077, via:
10(nc-lab03), Ver: v1, Flags: 100111,
[{"org.apache.cloudstack.backup.TakeBackupCommand":{"vmName":"i-10-1083-VM","backupPath":"i-10-1083-VM/2026.08.15.06.04.06","backupRepoType":"nfs","backupRepoAddress":"[fd33:9999:1::21]:/mnt/POOL01/Backup/cs-backup","wait":"0","bypassHostMaintenance":"false"}}]
}
2026-08-15 06:08:39,558 DEBUG [o.a.c.b.B.BackupSyncTask]
(BackgroundTaskPollManager-6:[ctx-9f798002]) (logid:1d647320) Trying to sync
backups of VM [VM instance
{"id":1083,"instanceName":"i-10-1083-VM","state":"Running","type":"User","uuid":"cac7bcdc-2ab2-43b4-81ce-64a5e5d0bf59"}]
using backup provider [nas].
2026-08-15 06:13:40,517 DEBUG [o.a.c.b.B.BackupSyncTask]
(BackgroundTaskPollManager-5:[ctx-55244db9]) (logid:55fc3c21) Trying to sync
backups of VM [VM instance
{"id":1083,"instanceName":"i-10-1083-VM","state":"Running","type":"User","uuid":"cac7bcdc-2ab2-43b4-81ce-64a5e5d0bf59"}]
using backup provider [nas].
2026-08-15 07:04:06,609 WARN [c.c.a.m.ClusteredAgentAttache]
(API-Job-Executor-30:[ctx-98b8e7e5, job-5367, ctx-8cb83fa7]) (logid:b96ba19b)
Seq 10-4578471971175552676: Timed out on Seq 10-4578471971175552676: { Cmd ,
MgmtId: 90520740805077, via: 10(nc-lab03), Ver: v1, Flags: 100111,
[{"org.apache.cloudstack.backup.TakeBackupCommand":{"vmName":"i-10-1083-VM","backupPath":"i-10-1083-VM/2026.08.15.06.04.06","backupRepoType":"nfs","backupRepoAddress":"[fd33:9999:1::21]:/mnt/POOL01/Backup/cs-backup","wait":"0","bypassHostMaintenance":"false"}}]
}
2026-08-15 07:04:06,609 ERROR [o.a.c.b.NASBackupProvider]
(API-Job-Executor-30:[ctx-98b8e7e5, job-5367, ctx-8cb83fa7]) (logid:b96ba19b)
Operation to initiate backup timed out for VM i-10-1083-VM
```
### versions
Ubuntu 24.04.4
10G networking over IPv6
Truenas NFS storage
Primary storage is Ceph
Other backup solutions for other hypervisors to the same storage have no
issues with long running backups.
### The steps to reproduce the bug
1. Schedule or run backup
2. Runs for a while
3. Fails, no backups show in UI, data files seen on NFS storage
### What to do about it?
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
