nxsbi opened a new issue, #9430:
URL: https://github.com/apache/cloudstack/issues/9430
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and main branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete
the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
~~~
UI
~~~
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on main
branch.
-->
~~~
4.19
~~~
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network,
advanced networking, etc. N/A otherwise
-->
XCP-NG
##### OS / ENVIRONMENT
<!--
Information about the environment if relevant, N/A otherwise
-->
NA
##### SUMMARY
<!-- Explain the problem/feature briefly -->
After adding new secondary store, I moved (balanced) from existing secondary
store to new secondary store.
The move completed successfully. However, new snapshots are all failing for
the snapshots that moved over to the new secondary store.
In logs, It appears Cloudstack is trying to mount the old NFS server's
snapshot folder path.
However the cloud.snapshot_store_ref shows the correct new path for all
snapshots (on new NFS server).
Note that the new NFS share is from Truenas, while old one was NFS share
from CentOS 7 linux - not that it should make any difference.
##### STEPS TO REPRODUCE
<!--
For bugs, show exactly how to reproduce the problem, using a minimal
test-case. Use Screenshots if accurate.
For new features, show how the feature would be used.
-->
<!-- Paste example playbooks or commands between quotes below -->
~~~
Have existing recurring snapshots (Daily). Add a new secondary storage, move
to balance. The volumes that moved to new storage fail
~~~
<!-- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS
<!-- What did you expect to happen when running the steps above? -->
~~~
Snapshots continue to function
~~~
##### ACTUAL RESULTS
<!-- What actually happened? -->
<!-- Paste verbatim command output between quotes below -->
~~~
Snapshot job fails. Below is a sample error
```
2024-07-22 09:38:55,499 DEBUG [c.c.s.r.StorageSubsystemCommandHandlerBase]
(DirectAgent-476:ctx-2f182f84) (logid:66082af1) Executing command CopyCommand:
[{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"360da96e-e989-40b1-ac6d-0b7d45019d9e","volume":{"uuid":"ff1a290f-5bbc-4477-be99-9b7b83d09fed","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"0dbbc58c-fc4c-31a6-9787-5ce215ade6f9","name":"Primary
Storage
CargoHold3","id":13,"poolType":"NetworkFilesystem","host":"XX.XX.XX.31","path":"/mnt/data/vmdata3/prod/primary","port":2049,"url":"NetworkFilesystem://XX.XX.XX.31/mnt/data/vmdata3/prod/primary/?ROLE=Primary&STOREUUID=0dbbc58c-fc4c-31a6-9787-5ce215ade6f9","isManaged":false}},"name":"DATA-999","size":536870912000,"path":"371ca470-c808-4cec-99b8-fd43bdbc73fd","volumeId":424,"vmName":"i-100-176-VM","accountId":10,"format":"VHD","provisioningType":"THIN","poolId":13,"id":424,"deviceId":7,"cacheMode":"NONE","hyper
visorType":"XenServer","directDownload":false,"deployAsIs":false,"followRedirects":true},"parentSnapshotPath":"687d708d-2dec-45ac-a20c-cc697c32b82a","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"0dbbc58c-fc4c-31a6-9787-5ce215ade6f9","name":"Primary
Storage
CargoHold3","id":13,"poolType":"NetworkFilesystem","host":"XX.XX.XX.31","path":"/mnt/data/vmdata3/prod/primary","port":2049,"url":"NetworkFilesystem://XX.XX.XX.31/mnt/data/vmdata3/prod/primary/?ROLE=Primary&STOREUUID=0dbbc58c-fc4c-31a6-9787-5ce215ade6f9","isManaged":false}},"vmName":"i-100-176-VM","name":"VMTest_DATA-999_20240722163619","hypervisorType":"XenServer","id":28689,"quiescevm":false,"parents":["687d708d-2dec-45ac-a20c-cc697c32b82a"],"physicalSize":0,"accountId":10,"followRedirects":false}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/10/424","volume":{"uuid":"ff1a290f-5bbc-4477-be99-9b7b83d09fed","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack
.storage.to.PrimaryDataStoreTO":{"uuid":"0dbbc58c-fc4c-31a6-9787-5ce215ade6f9","name":"Primary
Storage
CargoHold3","id":13,"poolType":"NetworkFilesystem","host":"XX.XX.XX.31","path":"/mnt/data/vmdata3/prod/primary","port":2049,"url":"NetworkFilesystem://XX.XX.XX.31/mnt/data/vmdata3/prod/primary/?ROLE=Primary&STOREUUID=0dbbc58c-fc4c-31a6-9787-5ce215ade6f9","isManaged":false}},"name":"DATA-999","size":536870912000,"path":"371ca470-c808-4cec-99b8-fd43bdbc73fd","volumeId":424,"vmName":"i-100-176-VM","accountId":10,"format":"VHD","provisioningType":"THIN","poolId":13,"id":424,"deviceId":7,"cacheMode":"NONE","hypervisorType":"XenServer","directDownload":false,"deployAsIs":false,"followRedirects":true},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://XX.XX.XX.31/mnt/data/vmdata3/prod/secondary","_role":"Image"}},"vmName":"i-100-176-VM","name":"VMTest_DATA-999_20240722163619","hypervisorType":"XenServer","id":28689,"quiescevm":false,"physicalSize":0,"accountId":10,"followRedirects
":false}},"executeInSequence":true,"options":{"snapshot.backup.to.secondary":"true","fullSnapshot":"false"},"options2":{},"wait":43200,"bypassHostMaintenance":false}].
2024-07-22 09:38:56,598 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-476:ctx-2f182f84) (logid:66082af1) No file SR found for path:
/var/cloud_mount/5926ce99-6aec-3e14-9737-3cf5a2e1c3e1/snapshots/10/424
2024-07-22 09:38:56,599 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-476:ctx-2f182f84) (logid:66082af1) Creating file SR for path
[/var/cloud_mount/5926ce99-6aec-3e14-9737-3cf5a2e1c3e1/snapshots/10/424] on
host [327366b9-a80c-496c-9fd2-1547c4c4465a]
2024-07-22 09:38:57,391 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-476:ctx-2f182f84) (logid:66082af1) Unpluging PBD
[f4a04899-a9e9-99ad-06d4-c7b9443afb11] of SR
[f6063a1e-bfc8-3bb0-b62b-9932e5edfaf0] as it is not working properly.
2024-07-22 09:38:57,411 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-476:ctx-2f182f84) (logid:66082af1) Forgetting SR
[f6063a1e-bfc8-3bb0-b62b-9932e5edfaf0] as it is not working properly.
2024-07-22 09:38:57,472 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-476:ctx-2f182f84) (logid:66082af1) Could not create file SR
[/var/cloud_mount/5926ce99-6aec-3e14-9737-3cf5a2e1c3e1/snapshots/10/424] on
host [327366b9-a80c-496c-9fd2-1547c4c4465a].
2024-07-22 09:38:57,474 DEBUG [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-476:ctx-2f182f84) (logid:66082af1) Exception in backupsnapshot
stage due to com.cloud.utils.exception.CloudRuntimeException: Could not
retrieve an already used file SR for path
[/var/cloud_mount/5926ce99-6aec-3e14-9737-3cf5a2e1c3e1/snapshots/10/424] or
create a new file SR on host [327366b9-a80c-496c-9fd2-1547c4c4465a]
2024-07-22 09:38:57,474 WARN [c.c.h.x.r.XenServerStorageProcessor]
(DirectAgent-476:ctx-2f182f84) (logid:66082af1) BackupSnapshot Failed due to
Exception in backupsnapshot stage due to
com.cloud.utils.exception.CloudRuntimeException: Could not retrieve an already
used file SR for path
[/var/cloud_mount/5926ce99-6aec-3e14-9737-3cf5a2e1c3e1/snapshots/10/424] or
create a new file SR on host [327366b9-a80c-496c-9fd2-1547c4c4465a]
2024-07-22 09:38:58,438 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-476:ctx-2f182f84) (logid:66082af1) Seq 11-6261129381952313782:
Response Received:
2024-07-22 09:38:58,439 DEBUG [c.c.a.t.Request]
(DirectAgent-476:ctx-2f182f84) (logid:66082af1) Seq 11-6261129381952313782:
Processing: { Ans: , MgmtId: 196482732657985, via: 11(xc-host100), Ver: v1,
Flags: 110,
[{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":"false","details":"BackupSnapshot
Failed due to Exception in backupsnapshot stage due to
com.cloud.utils.exception.CloudRuntimeException: Could not retrieve an already
used file SR for path
[/var/cloud_mount/5926ce99-6aec-3e14-9737-3cf5a2e1c3e1/snapshots/10/424] or
create a new file SR on host
[327366b9-a80c-496c-9fd2-1547c4c4465a]","wait":"0","bypassHostMaintenance":"false"}}]
}
2024-07-22 09:38:58,439 DEBUG [c.c.a.m.AgentAttache]
(DirectAgent-476:ctx-2f182f84) (logid:66082af1) Seq 11-6261129381952313782: No
more commands found
2024-07-22 09:38:58,439 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-152:ctx-f87ab687 job-71705/job-71706 ctx-8efd6070)
(logid:66082af1) Seq 11-6261129381952313782: Received: { Ans: , MgmtId:
196482732657985, via: 11(xc-host100), Ver: v1, Flags: 110, { CopyCmdAnswer } }
2024-07-22 09:38:58,552 DEBUG [c.c.s.s.SnapshotManagerImpl]
(Work-Job-Executor-152:ctx-f87ab687 job-71705/job-71706 ctx-8efd6070)
(logid:66082af1) Failed to create snapshotBackupSnapshot Failed due to
Exception in backupsnapshot stage due to
com.cloud.utils.exception.CloudRuntimeException: Could not retrieve an already
used file SR for path
[/var/cloud_mount/5926ce99-6aec-3e14-9737-3cf5a2e1c3e1/snapshots/10/424] or
create a new file SR on host [327366b9-a80c-496c-9fd2-1547c4c4465a]
2024-07-22 09:38:58,553 DEBUG [c.c.r.ResourceLimitManagerImpl]
(Work-Job-Executor-152:ctx-f87ab687 job-71705/job-71706 ctx-8efd6070)
(logid:66082af1) Updating resource Type = snapshot count for Account = 10
Operation = decreasing Amount = 1
2024-07-22 09:38:58,618 DEBUG [c.c.r.ResourceLimitManagerImpl]
(Work-Job-Executor-152:ctx-f87ab687 job-71705/job-71706 ctx-8efd6070)
(logid:66082af1) Updating resource Type = secondary_storage count for Account =
10 Operation = decreasing Amount = (500.00 GB) 536870912000
2024-07-22 09:38:58,656 ERROR [o.a.c.s.v.VolumeServiceImpl]
(Work-Job-Executor-152:ctx-f87ab687 job-71705/job-71706 ctx-8efd6070)
(logid:66082af1) Take snapshot: 424 failed
2024-07-22 09:38:58,661 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-152:ctx-f87ab687 job-71705/job-71706 ctx-8efd6070)
(logid:66082af1) Invocation exception, caused by:
com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to
Exception in backupsnapshot stage due to
com.cloud.utils.exception.CloudRuntimeException: Could not retrieve an already
used file SR for path
[/var/cloud_mount/5926ce99-6aec-3e14-9737-3cf5a2e1c3e1/snapshots/10/424] or
create a new file SR on host [327366b9-a80c-496c-9fd2-1547c4c4465a]
2024-07-22 09:38:58,661 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-152:ctx-f87ab687 job-71705/job-71706 ctx-8efd6070)
(logid:66082af1) Rethrow exception
com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to
Exception in backupsnapshot stage due to
com.cloud.utils.exception.CloudRuntimeException: Could not retrieve an already
used file SR for path
[/var/cloud_mount/5926ce99-6aec-3e14-9737-3cf5a2e1c3e1/snapshots/10/424] or
create a new file SR on host [327366b9-a80c-496c-9fd2-1547c4c4465a]
2024-07-22 09:38:58,661 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-152:ctx-f87ab687 job-71705/job-71706) (logid:66082af1) Done
with run of VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 176, job
origin: 71705
```
~~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
