GitHub user top-secrett edited a comment on the discussion: Multiple secondary storage in one zone; problem with volume snapshots
@top-secrett ok, for now I have 2 secstores. New snapshot of new VM was placed on second secstore. First:  Second:  Why all new snapshots are placed on only one storage even if its capacity is almost full? ~~~ 2024-10-08 09:35:56,237 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-109:ctx-42ac2243 job-75567) (logid:c49c9ce3) Executing AsyncJobVO: {id:75567, userId: 63, accountId: 60, instanceType: Snapshot, instanceId: 1170, cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, cmdInfo: {"quiescevm":"false","httpmethod":"GET","ctxAccountId":"60","uuid":"9a38809f-72e5-4436-998b-c8e67a066e2b","cmdEventType":"SNAPSHOT.CREATE","asyncBackup":"false","response":"json","ctxUserId":"63","volumeId":"2bb3eeac-0c70-4f1f-84b0-21bb7c449c3f","name":"test-karasev","ctxStartEventId":"1821908","id":"1170","ctxDetails":"{\"interface com.cloud.storage.Volume\":\"2bb3eeac-0c70-4f1f-84b0-21bb7c449c3f\",\"interface com.cloud.storage.Snapshot\":\"9a38809f-72e5-4436-998b-c8e67a066e2b\"}"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 108597816265214, completeMsid: null, lastUpdated: null, lastPolled: null, created: null, removed: null} 2024-10-08 09:35:56,261 DEBUG [c.c.a.ApiServlet] (qtp501107890-757360:ctx-7422ae44) (logid:8eac6395) ===START=== 0:0:0:0:0:0:0:1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:35:56,285 DEBUG [c.c.a.ApiServlet] (qtp501107890-757360:ctx-7422ae44 ctx-64db583d) (logid:8eac6395) ===END=== 0:0:0:0:0:0:0:1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:35:56,298 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-109:ctx-42ac2243 job-75567 ctx-29a24e5f) (logid:c49c9ce3) Sync job-75568 execution on object VmWorkJobQueue.5492 2024-10-08 09:35:58,082 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568) (logid:c49c9ce3) Executing AsyncJobVO: {id:75568, userId: 63, accountId: 60, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot, cmdInfo: rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8CAAdaAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20vY2xvdWQvc3RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMamF2YS9sYW5nL0xvbmc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAkwAB3pvbmVJZHN0ABBMamF2YS91dGlsL0xpc3Q7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAADwAAAAAAAAAPwAAAAAAABV0dAAUVm9sdW1lQXBpU2VydmljZUltcGwAAHBzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAAAAAAAABzcQB-AAgAAAAAAAAEknNxAH4ACAAAAAAAABg7cA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 108597816265214, completeMsid: null, lastUpdated: null, lastPolled: null, created: Tue Oct 08 09:35:56 MSK 2024, removed: null} 2024-10-08 09:35:58,082 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568) (logid:c49c9ce3) Run VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 5492, job origin: 75567 2024-10-08 09:35:58,085 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) Execute VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":6203,"policyId":0,"snapshotId":1170,"quiesceVm":false,"asyncBackup":false,"userId":63,"accountId":60,"vmId":5492,"handlerName":"VolumeApiServiceImpl"} 2024-10-08 09:35:58,098 DEBUG [o.a.c.s.s.StorPoolSnapshotStrategy] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) StorpoolSnapshotStrategy.canHandle: snapshot=test-karasev, uuid=9a38809f-72e5-4436-998b-c8e67a066e2b, op=TAKE 2024-10-08 09:35:58,138 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) Taking snapshot of org.apache.cloudstack.storage.snapshot.SnapshotObject@3a2fbae5 2024-10-08 09:35:58,176 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) Taking snapshot of org.apache.cloudstack.storage.snapshot.SnapshotObject@3a2fbae5 and encryption required is false 2024-10-08 09:35:58,176 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) getCommandHostDelegation: class org.apache.cloudstack.storage.command.CreateObjectCommand 2024-10-08 09:35:58,177 DEBUG [c.c.h.XenServerGuru] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) We are returning the default host to execute commands because the command is not of Copy type. 2024-10-08 09:35:58,220 DEBUG [c.c.a.t.Request] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) Seq 77352-1081145385545647332: Sending { Cmd , MgmtId: 108597816265214, via: 77352(xcp-sr247-u0910.aisrp.local), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"2bb3eeac-0c70-4f1f-84b0-21bb7c449c3f","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":"54","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":"0","url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":"false"}},"name":"ROOT-5492","size":"(10.00 GB) 10737418240","path":"cd4d9886-55ed-49a7-a086-5edef8f77067","volumeId":"6203","vmName":"i-60-5492-VM","accountId":"60","format":"VHD","provisioningType":"THIN","poolId":"54","id":"6203"," deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":"54","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":"0","url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":"false"}},"vmName":"i-60-5492-VM","name":"test-karasev","hypervisorType":"XenServer","id":"1170","quiescevm":"false","physicalSize":"0","accountId":"60","followRedirects":"false"}},"wait":"0","bypassHostMaintenance":"false"}}] } 2024-10-08 09:35:58,221 DEBUG [c.c.a.t.Request] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) Seq 77352-1081145385545647332: Executing: { Cmd , MgmtId: 108597816265214, via: 77352(xcp-sr247-u0910.aisrp.local), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"2bb3eeac-0c70-4f1f-84b0-21bb7c449c3f","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":"54","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":"0","url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":"false"}},"name":"ROOT-5492","size":"(10.00 GB) 10737418240","path":"cd4d9886-55ed-49a7-a086-5edef8f77067","volumeId":"6203","vmName":"i-60-5492-VM","accountId":"60","format":"VHD","provisioningType":"THIN","poolId":"54","id":"6203 ","deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":"54","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":"0","url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":"false"}},"vmName":"i-60-5492-VM","name":"test-karasev","hypervisorType":"XenServer","id":"1170","quiescevm":"false","physicalSize":"0","accountId":"60","followRedirects":"false"}},"wait":"0","bypassHostMaintenance":"false"}}] } 2024-10-08 09:35:58,222 DEBUG [c.c.s.r.StorageSubsystemCommandHandlerBase] (DirectAgent-31:ctx-8affc615) (logid:c49c9ce3) Executing command CreateObjectCommand: [{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"2bb3eeac-0c70-4f1f-84b0-21bb7c449c3f","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":54,"poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":0,"url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":false}},"name":"ROOT-5492","size":10737418240,"path":"cd4d9886-55ed-49a7-a086-5edef8f77067","volumeId":6203,"vmName":"i-60-5492-VM","accountId":60,"format":"VHD","provisioningType":"THIN","poolId":54,"id":6203,"deviceId":0,"cacheMode":"NONE","hypervisorType":"XenServer","directDownload":false,"deployAsIs":false,"followRedirects":true},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid ":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":54,"poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":0,"url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":false}},"vmName":"i-60-5492-VM","name":"test-karasev","hypervisorType":"XenServer","id":1170,"quiescevm":false,"physicalSize":0,"accountId":60,"followRedirects":false}},"wait":0,"bypassHostMaintenance":false}]. 2024-10-08 09:35:59,315 DEBUG [c.c.a.ApiServlet] (qtp501107890-923975:ctx-3ad490b5) (logid:a7956111) ===START=== 0:0:0:0:0:0:0:1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:35:59,338 DEBUG [c.c.a.ApiServlet] (qtp501107890-923975:ctx-3ad490b5 ctx-b7640e87) (logid:a7956111) ===END=== 0:0:0:0:0:0:0:1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:02,353 DEBUG [c.c.a.ApiServlet] (qtp501107890-920932:ctx-072feeae) (logid:02659f8f) ===START=== 0:0:0:0:0:0:0:1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:02,378 DEBUG [c.c.a.ApiServlet] (qtp501107890-920932:ctx-072feeae ctx-b2058784) (logid:02659f8f) ===END=== 0:0:0:0:0:0:0:1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:05,308 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-31:ctx-8affc615) (logid:c49c9ce3) Seq 77352-1081145385545647332: Response Received: 2024-10-08 09:36:05,308 DEBUG [c.c.a.t.Request] (DirectAgent-31:ctx-8affc615) (logid:c49c9ce3) Seq 77352-1081145385545647332: Processing: { Ans: , MgmtId: 108597816265214, via: 77352(xcp-sr247-u0910.aisrp.local), Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"488f6825-bba8-419e-90ee-53996277abb8","id":"0","quiescevm":"false","physicalSize":"0","accountId":"0","followRedirects":"false"}},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2024-10-08 09:36:05,308 DEBUG [c.c.a.t.Request] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) Seq 77352-1081145385545647332: Received: { Ans: , MgmtId: 108597816265214, via: 77352(xcp-sr247-u0910.aisrp.local), Ver: v1, Flags: 10, { CreateObjectAnswer } } 2024-10-08 09:36:05,381 DEBUG [o.a.c.s.h.HeuristicRuleHelper] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) No heuristic rules found for zone with ID [7] and heuristic type [SNAPSHOT]. Returning null. 2024-10-08 09:36:05,385 DEBUG [c.c.s.StatsCollector] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) Verifying image storage [10]. Capacity: total=[2 TB], used=[1 TB], threshold=[89.99999761581421%]. 2024-10-08 09:36:05,397 DEBUG [c.c.a.ApiServlet] (qtp501107890-922834:ctx-a232aee6) (logid:dc1d5623) ===START=== 127.0.0.1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:05,426 DEBUG [c.c.a.ApiServlet] (qtp501107890-922834:ctx-a232aee6 ctx-49c349e5) (logid:dc1d5623) ===END=== 127.0.0.1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:05,433 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) copyAsync inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT 2024-10-08 09:36:05,558 ERROR [o.a.c.s.e.DefaultEndPointSelector] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) IR24 select BACKUPSNAPSHOT from primary to secondary 1170 dest=1170 2024-10-08 09:36:05,565 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) getCommandHostDelegation: class org.apache.cloudstack.storage.command.CopyCommand 2024-10-08 09:36:05,566 DEBUG [c.c.h.XenServerGuru] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) We are returning the default host to execute commands because the source and destination objects are not NFS type. 2024-10-08 09:36:05,570 DEBUG [c.c.a.t.Request] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) Seq 77357-4446460207098250533: Sending { Cmd , MgmtId: 108597816265214, via: 77357(xcp-sr247-u0708.aisrp.local), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"488f6825-bba8-419e-90ee-53996277abb8","volume":{"uuid":"2bb3eeac-0c70-4f1f-84b0-21bb7c449c3f","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":"54","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":"0","url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":"false"}},"name":"ROOT-5492","size":"(10.00 GB) 10737418240","path":"cd4d9886-55ed-49a7-a086-5edef8f77067","volumeId":"6203","vmName":"i-60-5492-VM","accountId":"60","format":"VHD","provisioningT ype":"THIN","poolId":"54","id":"6203","deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":"54","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":"0","url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":"false"}},"vmName":"i-60-5492-VM","name":"test-karasev","hypervisorType":"XenServer","id":"1170","quiescevm":"false","physicalSize":"0","accountId":"60","followRedirects":"false"}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/60/6203","volume":{"uuid":"2bb3eeac-0c70-4f1f-84b0-21bb7c449c3f","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":"54","poolType":"PreSetup","host":"localhost","path":"/DM7100 _LUN4","port":"0","url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":"false"}},"name":"ROOT-5492","size":"(10.00 GB) 10737418240","path":"cd4d9886-55ed-49a7-a086-5edef8f77067","volumeId":"6203","vmName":"i-60-5492-VM","accountId":"60","format":"VHD","provisioningType":"THIN","poolId":"54","id":"6203","deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.69.105.243/var/NFS3","_role":"Image"}},"vmName":"i-60-5492-VM","name":"test-karasev","hypervisorType":"XenServer","id":"1170","quiescevm":"false","physicalSize":"0","accountId":"60","followRedirects":"false"}},"executeInSequence":"true","options":{"snapshot.backup.to.secondary":"true","fullSnapshot":"true"},"options2":{},"wait":"21600","bypassHostMaintenance":"false"}}] } 2024-10-08 09:36:05,572 DEBUG [c.c.a.t.Request] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) Seq 77357-4446460207098250533: Executing: { Cmd , MgmtId: 108597816265214, via: 77357(xcp-sr247-u0708.aisrp.local), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"488f6825-bba8-419e-90ee-53996277abb8","volume":{"uuid":"2bb3eeac-0c70-4f1f-84b0-21bb7c449c3f","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":"54","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":"0","url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":"false"}},"name":"ROOT-5492","size":"(10.00 GB) 10737418240","path":"cd4d9886-55ed-49a7-a086-5edef8f77067","volumeId":"6203","vmName":"i-60-5492-VM","accountId":"60","format":"VHD","provisioni ngType":"THIN","poolId":"54","id":"6203","deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":"54","poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":"0","url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":"false"}},"vmName":"i-60-5492-VM","name":"test-karasev","hypervisorType":"XenServer","id":"1170","quiescevm":"false","physicalSize":"0","accountId":"60","followRedirects":"false"}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/60/6203","volume":{"uuid":"2bb3eeac-0c70-4f1f-84b0-21bb7c449c3f","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":"54","poolType":"PreSetup","host":"localhost","path":"/DM7 100_LUN4","port":"0","url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":"false"}},"name":"ROOT-5492","size":"(10.00 GB) 10737418240","path":"cd4d9886-55ed-49a7-a086-5edef8f77067","volumeId":"6203","vmName":"i-60-5492-VM","accountId":"60","format":"VHD","provisioningType":"THIN","poolId":"54","id":"6203","deviceId":"0","cacheMode":"NONE","hypervisorType":"XenServer","directDownload":"false","deployAsIs":"false","followRedirects":"true"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.69.105.243/var/NFS3","_role":"Image"}},"vmName":"i-60-5492-VM","name":"test-karasev","hypervisorType":"XenServer","id":"1170","quiescevm":"false","physicalSize":"0","accountId":"60","followRedirects":"false"}},"executeInSequence":"true","options":{"snapshot.backup.to.secondary":"true","fullSnapshot":"true"},"options2":{},"wait":"21600","bypassHostMaintenance":"false"}}] } 2024-10-08 09:36:05,572 DEBUG [c.c.s.r.StorageSubsystemCommandHandlerBase] (DirectAgent-219:ctx-11fc864d) (logid:c49c9ce3) Executing command CopyCommand: [{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"488f6825-bba8-419e-90ee-53996277abb8","volume":{"uuid":"2bb3eeac-0c70-4f1f-84b0-21bb7c449c3f","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":54,"poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":0,"url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":false}},"name":"ROOT-5492","size":10737418240,"path":"cd4d9886-55ed-49a7-a086-5edef8f77067","volumeId":6203,"vmName":"i-60-5492-VM","accountId":60,"format":"VHD","provisioningType":"THIN","poolId":54,"id":6203,"deviceId":0,"cacheMode":"NONE","hypervisorType":"XenServer","directDownload":false,"deployAsIs":false,"followRedirects":true},"dataStore":{"org.apache.cloudsta ck.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":54,"poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":0,"url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":false}},"vmName":"i-60-5492-VM","name":"test-karasev","hypervisorType":"XenServer","id":1170,"quiescevm":false,"physicalSize":0,"accountId":60,"followRedirects":false}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/60/6203","volume":{"uuid":"2bb3eeac-0c70-4f1f-84b0-21bb7c449c3f","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"DM7100_LUN4","name":"cluster5-dm7100_lun4","id":54,"poolType":"PreSetup","host":"localhost","path":"/DM7100_LUN4","port":0,"url":"PreSetup://localhost/DM7100_LUN4/?ROLE=Primary&STOREUUID=DM7100_LUN4","isManaged":false}},"name":"ROOT-5492","size":10737418240,"path":"cd4d9886-55ed-49a7-a086-5edef8f77067","volumeId":6203,"vmName":"i -60-5492-VM","accountId":60,"format":"VHD","provisioningType":"THIN","poolId":54,"id":6203,"deviceId":0,"cacheMode":"NONE","hypervisorType":"XenServer","directDownload":false,"deployAsIs":false,"followRedirects":true},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.69.105.243/var/NFS3","_role":"Image"}},"vmName":"i-60-5492-VM","name":"test-karasev","hypervisorType":"XenServer","id":1170,"quiescevm":false,"physicalSize":0,"accountId":60,"followRedirects":false}},"executeInSequence":true,"options":{"snapshot.backup.to.secondary":"true","fullSnapshot":"true"},"options2":{},"wait":21600,"bypassHostMaintenance":false}]. 2024-10-08 09:36:07,732 DEBUG [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-219:ctx-11fc864d) (logid:c49c9ce3) No file SR found for path: /var/cloud_mount/38f03c70-7d69-3505-85fd-a108790bfde3/snapshots/60/6203 2024-10-08 09:36:07,732 DEBUG [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-219:ctx-11fc864d) (logid:c49c9ce3) Creating file SR for path [/var/cloud_mount/38f03c70-7d69-3505-85fd-a108790bfde3/snapshots/60/6203] on host [32ed12ed-6b7c-4274-92a4-807b32075733] 2024-10-08 09:36:08,448 DEBUG [c.c.a.ApiServlet] (qtp501107890-757360:ctx-4021b50a) (logid:57b35f1c) ===START=== 127.0.0.1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:08,471 DEBUG [c.c.a.ApiServlet] (qtp501107890-757360:ctx-4021b50a ctx-40e17e8a) (logid:57b35f1c) ===END=== 127.0.0.1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:11,486 DEBUG [c.c.a.ApiServlet] (qtp501107890-920932:ctx-2fd102d2) (logid:c39505de) ===START=== 0:0:0:0:0:0:0:1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:11,512 DEBUG [c.c.a.ApiServlet] (qtp501107890-920932:ctx-2fd102d2 ctx-e4be5e15) (logid:c39505de) ===END=== 0:0:0:0:0:0:0:1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:14,539 DEBUG [c.c.a.ApiServlet] (qtp501107890-922834:ctx-7df66d9f) (logid:311c29c3) ===START=== 0:0:0:0:0:0:0:1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:14,564 DEBUG [c.c.a.ApiServlet] (qtp501107890-922834:ctx-7df66d9f ctx-4808f0aa) (logid:311c29c3) ===END=== 0:0:0:0:0:0:0:1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:17,586 DEBUG [c.c.a.ApiServlet] (qtp501107890-926241:ctx-ef4e8e3c) (logid:8cd14435) ===START=== 0:0:0:0:0:0:0:1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:17,608 DEBUG [c.c.a.ApiServlet] (qtp501107890-926241:ctx-ef4e8e3c ctx-09238735) (logid:8cd14435) ===END=== 0:0:0:0:0:0:0:1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:20,621 DEBUG [c.c.a.ApiServlet] (qtp501107890-926242:ctx-bc3d1597) (logid:1113a64b) ===START=== 127.0.0.1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:20,644 DEBUG [c.c.a.ApiServlet] (qtp501107890-926242:ctx-bc3d1597 ctx-cc7870e5) (logid:1113a64b) ===END=== 127.0.0.1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:23,664 DEBUG [c.c.a.ApiServlet] (qtp501107890-922834:ctx-d5db471b) (logid:4e29dc0f) ===START=== 127.0.0.1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:23,686 DEBUG [c.c.a.ApiServlet] (qtp501107890-922834:ctx-d5db471b ctx-5bd913bf) (logid:4e29dc0f) ===END=== 127.0.0.1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:26,699 DEBUG [c.c.a.ApiServlet] (qtp501107890-926241:ctx-151bdf90) (logid:f6adbae1) ===START=== 0:0:0:0:0:0:0:1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:26,727 DEBUG [c.c.a.ApiServlet] (qtp501107890-926241:ctx-151bdf90 ctx-be666c59) (logid:f6adbae1) ===END=== 0:0:0:0:0:0:0:1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:29,752 DEBUG [c.c.a.ApiServlet] (qtp501107890-757360:ctx-e68b35d0) (logid:8bb6246d) ===START=== 0:0:0:0:0:0:0:1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:29,773 DEBUG [c.c.a.ApiServlet] (qtp501107890-757360:ctx-e68b35d0 ctx-67af04ec) (logid:8bb6246d) ===END=== 0:0:0:0:0:0:0:1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:32,787 DEBUG [c.c.a.ApiServlet] (qtp501107890-922834:ctx-4c729b88) (logid:3a88ff96) ===START=== 0:0:0:0:0:0:0:1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:32,811 DEBUG [c.c.a.ApiServlet] (qtp501107890-922834:ctx-4c729b88 ctx-ebe31ca1) (logid:3a88ff96) ===END=== 0:0:0:0:0:0:0:1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:33,817 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-219:ctx-11fc864d) (logid:c49c9ce3) Host 10.69.105.27 OpaqueRef:5f9f4d9a-eb89-4da1-88c8-e3c71ca2b2af: Removing SR 2024-10-08 09:36:33,831 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-219:ctx-11fc864d) (logid:c49c9ce3) Host 10.69.105.27 OpaqueRef:7b2cf5a2-e6dd-4a66-ae7a-ebb1aface613: Unplugging pbd 2024-10-08 09:36:34,272 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-219:ctx-11fc864d) (logid:c49c9ce3) Host 10.69.105.27 OpaqueRef:5f9f4d9a-eb89-4da1-88c8-e3c71ca2b2af: Forgetting 2024-10-08 09:36:34,279 DEBUG [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-219:ctx-11fc864d) (logid:c49c9ce3) Successfully destroyed snapshot on volume: cd4d9886-55ed-49a7-a086-5edef8f77067 execept this current snapshot 488f6825-bba8-419e-90ee-53996277abb8 2024-10-08 09:36:34,279 INFO [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-219:ctx-11fc864d) (logid:c49c9ce3) New snapshot details: SnapshotTO[datastore=null|volume=null|pathsnapshots/60/6203/9d6af389-3e09-49cc-a12c-4df6d06e1779.vhd] 2024-10-08 09:36:34,279 INFO [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-219:ctx-11fc864d) (logid:c49c9ce3) New snapshot physical utilization: (4.48 GB) 4809777664 2024-10-08 09:36:34,279 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-219:ctx-11fc864d) (logid:c49c9ce3) Seq 77357-4446460207098250533: Response Received: 2024-10-08 09:36:34,280 DEBUG [c.c.a.t.Request] (DirectAgent-219:ctx-11fc864d) (logid:c49c9ce3) Seq 77357-4446460207098250533: Processing: { Ans: , MgmtId: 108597816265214, via: 77357(xcp-sr247-u0708.aisrp.local), Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/60/6203/9d6af389-3e09-49cc-a12c-4df6d06e1779.vhd","id":"0","quiescevm":"false","physicalSize":"4809777664","accountId":"0","followRedirects":"false"}},"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2024-10-08 09:36:34,280 DEBUG [c.c.a.m.AgentAttache] (DirectAgent-219:ctx-11fc864d) (logid:c49c9ce3) Seq 77357-4446460207098250533: No more commands found 2024-10-08 09:36:34,280 DEBUG [c.c.a.t.Request] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) Seq 77357-4446460207098250533: Received: { Ans: , MgmtId: 108597816265214, via: 77357(xcp-sr247-u0708.aisrp.local), Ver: v1, Flags: 110, { CopyCmdAnswer } } 2024-10-08 09:36:34,327 DEBUG [c.c.r.ResourceLimitManagerImpl] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) Updating resource Type = secondary_storage count for Account = 60 Operation = decreasing Amount = (5.52 GB) 5927640576 2024-10-08 09:36:34,336 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) Done executing VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":6203,"policyId":0,"snapshotId":1170,"quiesceVm":false,"asyncBackup":false,"userId":63,"accountId":60,"vmId":5492,"handlerName":"VolumeApiServiceImpl"} 2024-10-08 09:36:34,337 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) Complete async job-75568, jobStatus: SUCCEEDED, resultCode: 0, result: rO0ABXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAEkg 2024-10-08 09:36:34,338 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) Publish async job-75568 complete on message bus 2024-10-08 09:36:34,338 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) Wake up jobs related to job-75568 2024-10-08 09:36:34,338 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) Update db status for job-75568 2024-10-08 09:36:34,339 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568 ctx-7f4fab53) (logid:c49c9ce3) Wake up jobs joined with job-75568 and disjoin all subjobs created from job- 75568 2024-10-08 09:36:34,348 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568) (logid:c49c9ce3) Done with run of VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 5492, job origin: 75567 2024-10-08 09:36:34,348 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568) (logid:c49c9ce3) Done executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-75568 2024-10-08 09:36:34,349 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-11:ctx-b410a074 job-75567/job-75568) (logid:c49c9ce3) Remove job-75568 from job monitoring 2024-10-08 09:36:34,395 DEBUG [o.a.c.s.s.StorPoolSnapshotStrategy] (API-Job-Executor-109:ctx-42ac2243 job-75567 ctx-29a24e5f) (logid:c49c9ce3) StorpoolSnapshotStrategy.canHandle: snapshot=test-karasev, uuid=9a38809f-72e5-4436-998b-c8e67a066e2b, op=REVERT 2024-10-08 09:36:34,401 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-109:ctx-42ac2243 job-75567 ctx-29a24e5f) (logid:c49c9ce3) Complete async job-75567, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.SnapshotResponse/snapshot/{"id":"9a38809f-72e5-4436-998b-c8e67a066e2b","account":"dkarasev","domainid":"35e53f94-4224-11ec-8697-62c4e64309fe","domain":"ROOT","snapshottype":"MANUAL","volumeid":"2bb3eeac-0c70-4f1f-84b0-21bb7c449c3f","volumename":"ROOT-5492","volumetype":"ROOT","created":"2024-10-08T09:35:56+0300","name":"test-karasev","intervaltype":"MANUAL","state":"BackedUp","physicalsize":"4809777664","zoneid":"9505445a-7164-4675-876d-1a2fee1f7e46","zonename":"xcp-zone-02","revertable":"false","ostypeid":"c7dbb910-46b3-11ec-afc7-62c4e64309fe","osdisplayname":"AlmaLinux 8.3","virtualsize":"10737418240","tags":[],"hasannotations":"false"} 2024-10-08 09:36:34,402 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-109:ctx-42ac2243 job-75567 ctx-29a24e5f) (logid:c49c9ce3) Publish async job-75567 complete on message bus 2024-10-08 09:36:34,402 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-109:ctx-42ac2243 job-75567 ctx-29a24e5f) (logid:c49c9ce3) Wake up jobs related to job-75567 2024-10-08 09:36:34,402 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-109:ctx-42ac2243 job-75567 ctx-29a24e5f) (logid:c49c9ce3) Update db status for job-75567 2024-10-08 09:36:34,403 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-109:ctx-42ac2243 job-75567 ctx-29a24e5f) (logid:c49c9ce3) Wake up jobs joined with job-75567 and disjoin all subjobs created from job- 75567 2024-10-08 09:36:34,408 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-109:ctx-42ac2243 job-75567) (logid:c49c9ce3) Done executing org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-75567 2024-10-08 09:36:34,408 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-109:ctx-42ac2243 job-75567) (logid:c49c9ce3) Remove job-75567 from job monitoring 2024-10-08 09:36:35,824 DEBUG [c.c.a.ApiServlet] (qtp501107890-920932:ctx-cd5b6739) (logid:a075da24) ===START=== 127.0.0.1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json 2024-10-08 09:36:35,845 DEBUG [c.c.a.ApiServlet] (qtp501107890-920932:ctx-cd5b6739 ctx-220a40e9) (logid:a075da24) ===END=== 127.0.0.1 -- GET jobId=c49c9ce3-4117-4c72-9572-716ae35d8f3b&command=queryAsyncJobResult&response=json ~~~ GitHub link: https://github.com/apache/cloudstack/discussions/9754#discussioncomment-10876176 ---- This is an automatically sent email for users@cloudstack.apache.org. To unsubscribe, please send an email to: users-unsubscr...@cloudstack.apache.org
