andrijapanicsb edited a comment on issue #4018: Snapshots GC from DB - needs refractoring and fixing snapshot_store_ref garbage URL: https://github.com/apache/cloudstack/issues/4018#issuecomment-611628944 XenServer removes previous snapshots from the "primary storage" when a new one is created - it does NOT delete it really, but deletes the PRIMARY row in the snap_store_ref table, thus the garbage is left on Primary Store while the reference is deleted from the snap_store_ref - and later GC is not picking it up as the row/reference is not there.... Stupid implementation by all means: [root@ref-trl-774-k-M7-apanic-mgmt1 ~]# grep 301a4ad6 /var/log/cloudstack/management/management-server.log | grep -v "queryAsyncJobResult" 2020-04-09 16:29:45,717 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-56cd28cc job-592) (logid:301a4ad6) Executing AsyncJobVO {id:592, userId: 2, accountId: 2, instanceType: Snapshot, instanceId: 85, cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, cmdInfo: {"quiescevm":"false","httpmethod":"GET","ctxAccountId":"2","uuid":"ba2df407-1772-4668-9327-026af717dbce","cmdEventType":"SNAPSHOT.CREATE","asyncBackup":"false","response":"json","ctxUserId":"2","volumeId":"05c88236-1aa6-44c3-9bf9-bad4edfc34e5","name":"snap4","ctxStartEventId":"1274","id":"85","ctxDetails":"{\"interface com.cloud.storage.Snapshot\":\"ba2df407-1772-4668-9327-026af717dbce\",\"interface com.cloud.storage.Volume\":\"05c88236-1aa6-44c3-9bf9-bad4edfc34e5\"}","_":"1586449031328"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 32988805007466, completeMsid: null, lastUpdated: null, lastPolled: null, created: null, removed: null} 2020-04-09 16:29:45,743 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-56cd28cc job-592 ctx-eabcd3e5) (logid:301a4ad6) Sync job-593 execution on object VmWorkJobQueue.22 2020-04-09 16:29:47,735 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593) (logid:301a4ad6) Executing AsyncJobVO {id:593, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot, cmdInfo: rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8CAAZaAAthc3luY0JhY2t1cFoACXF1aWVzY2VWbUwADGxvY2F0aW9uVHlwZXQAKUxjb20vY2xvdWQvc3RvcmFnZS9TbmFwc2hvdCRMb2NhdGlvblR5cGU7TAAIcG9saWN5SWR0ABBMamF2YS9sYW5nL0xvbmc7TAAKc25hcHNob3RJZHEAfgACTAAIdm9sdW1lSWRxAH4AAnhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAFnQAFFZvbHVtZUFwaVNlcnZpY2VJbXBsAABwc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAAAAAc3EAfgAHAAAAAAAAAFVzcQB-AAcAAAAAAAAANg, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 32988805007466, completeMsid: null, lastUpdated: null, lastPolled: null, created: Thu Apr 09 16:29:45 UTC 2020, removed: null} 2020-04-09 16:29:47,736 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593) (logid:301a4ad6) Run VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 22, job origin: 592 2020-04-09 16:29:47,737 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593 ctx-34f207e8) (logid:301a4ad6) Execute VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":54,"policyId":0,"snapshotId":85,"quiesceVm":false,"asyncBackup":false,"userId":2,"accountId":2,"vmId":22,"handlerName":"VolumeApiServiceImpl"} 2020-04-09 16:29:47,775 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593 ctx-34f207e8) (logid:301a4ad6) getCommandHostDelegation: class org.apache.cloudstack.storage.command.CreateObjectCommand 2020-04-09 16:29:47,775 DEBUG [c.c.h.XenServerGuru] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593 ctx-34f207e8) (logid:301a4ad6) We are returning the default host to execute commands because the command is not of Copy type. 2020-04-09 16:29:47,776 DEBUG [c.c.a.t.Request] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593 ctx-34f207e8) (logid:301a4ad6) Seq 2-5707749577738682460: Sending { Cmd , MgmtId: 32988805007466, via: 2(ref-trl-774-k-M7-apanic-xs1), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"05c88236-1aa6-44c3-9bf9-bad4edfc34e5","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"39bf30d6-f959-359d-b974-b8ebd2d8727f","id":5,"poolType":"NetworkFilesystem","host":"10.2.0.16","path":"/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1","port":2049,"url":"NetworkFilesystem://10.2.0.16/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1/?ROLE=Primary&STOREUUID=39bf30d6-f959-359d-b974-b8ebd2d8727f","isManaged":false}},"name":"xen-data-1","size":1073741824,"path":"dd772879-f006-41ec-b09e-db5699f7a7fb","volumeId":54,"vmName":"i-2-22-VM","accountId":2,"format":"VHD","provisioningType":"THIN","id":54,"deviceId":1,"hypervisorType":"XenServer"},"parentSnapshotPath":"ffda9432-b5c2-41db-ac36-2321b8ed92b3","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"39bf30d6-f959-359d-b974-b8ebd2d8727f","id":5,"poolType":"NetworkFilesystem","host":"10.2.0.16","path":"/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1","port":2049,"url":"NetworkFilesystem://10.2.0.16/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1/?ROLE=Primary&STOREUUID=39bf30d6-f959-359d-b974-b8ebd2d8727f","isManaged":false}},"vmName":"i-2-22-VM","name":"snap4","hypervisorType":"XenServer","id":85,"quiescevm":false,"parents":["ffda9432-b5c2-41db-ac36-2321b8ed92b3"],"physicalSize":0}},"wait":0}}] } 2020-04-09 16:29:47,776 DEBUG [c.c.a.t.Request] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593 ctx-34f207e8) (logid:301a4ad6) Seq 2-5707749577738682460: Executing: { Cmd , MgmtId: 32988805007466, via: 2(ref-trl-774-k-M7-apanic-xs1), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"05c88236-1aa6-44c3-9bf9-bad4edfc34e5","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"39bf30d6-f959-359d-b974-b8ebd2d8727f","id":5,"poolType":"NetworkFilesystem","host":"10.2.0.16","path":"/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1","port":2049,"url":"NetworkFilesystem://10.2.0.16/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1/?ROLE=Primary&STOREUUID=39bf30d6-f959-359d-b974-b8ebd2d8727f","isManaged":false}},"name":"xen-data-1","size":1073741824,"path":"dd772879-f006-41ec-b09e-db5699f7a7fb","volumeId":54,"vmName":"i-2-22-VM","accountId":2,"format":"VHD","provisioningType":"THIN","id":54,"deviceId":1,"hypervisorType":"XenServer"},"parentSnapshotPath":"ffda9432-b5c2-41db-ac36-2321b8ed92b3","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"39bf30d6-f959-359d-b974-b8ebd2d8727f","id":5,"poolType":"NetworkFilesystem","host":"10.2.0.16","path":"/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1","port":2049,"url":"NetworkFilesystem://10.2.0.16/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1/?ROLE=Primary&STOREUUID=39bf30d6-f959-359d-b974-b8ebd2d8727f","isManaged":false}},"vmName":"i-2-22-VM","name":"snap4","hypervisorType":"XenServer","id":85,"quiescevm":false,"parents":["ffda9432-b5c2-41db-ac36-2321b8ed92b3"],"physicalSize":0}},"wait":0}}] } 2020-04-09 16:29:50,193 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-39:ctx-5870584c) (logid:301a4ad6) Seq 2-5707749577738682460: Response Received: 2020-04-09 16:29:50,193 DEBUG [c.c.a.t.Request] (DirectAgent-39:ctx-5870584c) (logid:301a4ad6) Seq 2-5707749577738682460: Processing: { Ans: , MgmtId: 32988805007466, via: 2(ref-trl-774-k-M7-apanic-xs1), Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"ec235cfb-3e6e-40aa-9515-e6943b8e2241","id":0,"quiescevm":false,"physicalSize":0}},"result":true,"wait":0}}] } 2020-04-09 16:29:50,193 DEBUG [c.c.a.t.Request] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593 ctx-34f207e8) (logid:301a4ad6) Seq 2-5707749577738682460: Received: { Ans: , MgmtId: 32988805007466, via: 2(ref-trl-774-k-M7-apanic-xs1), Ver: v1, Flags: 10, { CreateObjectAnswer } } 2020-04-09 16:29:50,254 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593 ctx-34f207e8) (logid:301a4ad6) copyAsync inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT 2020-04-09 16:29:50,323 ERROR [o.a.c.s.e.DefaultEndPointSelector] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593 ctx-34f207e8) (logid:301a4ad6) IR24 select BACKUPSNAPSHOT from primary to secondary 85 dest=85 2020-04-09 16:29:50,326 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593 ctx-34f207e8) (logid:301a4ad6) getCommandHostDelegation: class org.apache.cloudstack.storage.command.CopyCommand 2020-04-09 16:29:50,326 DEBUG [c.c.h.XenServerGuru] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593 ctx-34f207e8) (logid:301a4ad6) We are returning the default host to execute commands because the source and destination objects are not NFS type. 2020-04-09 16:29:50,327 DEBUG [c.c.a.t.Request] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593 ctx-34f207e8) (logid:301a4ad6) Seq 2-5707749577738682461: Sending { Cmd , MgmtId: 32988805007466, via: 2(ref-trl-774-k-M7-apanic-xs1), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"ec235cfb-3e6e-40aa-9515-e6943b8e2241","volume":{"uuid":"05c88236-1aa6-44c3-9bf9-bad4edfc34e5","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"39bf30d6-f959-359d-b974-b8ebd2d8727f","id":5,"poolType":"NetworkFilesystem","host":"10.2.0.16","path":"/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1","port":2049,"url":"NetworkFilesystem://10.2.0.16/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1/?ROLE=Primary&STOREUUID=39bf30d6-f959-359d-b974-b8ebd2d8727f","isManaged":false}},"name":"xen-data-1","size":1073741824,"path":"dd772879-f006-41ec-b09e-db5699f7a7fb","volumeId":54,"vmName":"i-2-22-VM","accountId":2,"format":"VHD","provisioningType":"THIN","id":54,"deviceId":1,"hypervisorType":"XenServer"},"parentSnapshotPath":"ffda9432-b5c2-41db-ac36-2321b8ed92b3","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"39bf30d6-f959-359d-b974-b8ebd2d8727f","id":5,"poolType":"NetworkFilesystem","host":"10.2.0.16","path":"/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1","port":2049,"url":"NetworkFilesystem://10.2.0.16/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1/?ROLE=Primary&STOREUUID=39bf30d6-f959-359d-b974-b8ebd2d8727f","isManaged":false}},"vmName":"i-2-22-VM","name":"snap4","hypervisorType":"XenServer","id":85,"quiescevm":false,"parents":["ffda9432-b5c2-41db-ac36-2321b8ed92b3"],"physicalSize":0}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/54","volume":{"uuid":"05c88236-1aa6-44c3-9bf9-bad4edfc34e5","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"39bf30d6-f959-359d-b974-b8ebd2d8727f","id":5,"poolType":"NetworkFilesystem","host":"10.2.0.16","path":"/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1","port":2049,"url":"NetworkFilesystem://10.2.0.16/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1/?ROLE=Primary&STOREUUID=39bf30d6-f959-359d-b974-b8ebd2d8727f","isManaged":false}},"name":"xen-data-1","size":1073741824,"path":"dd772879-f006-41ec-b09e-db5699f7a7fb","volumeId":54,"vmName":"i-2-22-VM","accountId":2,"format":"VHD","provisioningType":"THIN","id":54,"deviceId":1,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshots/2/54/a585fa13-201e-40ef-b586-11822133287e.vhd","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"NFS://10.2.0.16/acs/secondary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-sec1","_role":"Image"}},"vmName":"i-2-22-VM","name":"snap4","hypervisorType":"XenServer","id":85,"quiescevm":false,"parents":["snapshots/2/54/14c13d66-53f4-4003-a284-cf266fe1ee3c.vhd","snapshots/2/54/151ca037-2b00-44a3-abb2-b212962b51be.vhd","snapshots/2/54/a585fa13-201e-40ef-b586-11822133287e.vhd"],"physicalSize":0}},"executeInSequence":true,"options":{"fullSnapshot":"true"},"options2":{},"wait":21600}}] } 2020-04-09 16:29:50,328 DEBUG [c.c.a.t.Request] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593 ctx-34f207e8) (logid:301a4ad6) Seq 2-5707749577738682461: Executing: { Cmd , MgmtId: 32988805007466, via: 2(ref-trl-774-k-M7-apanic-xs1), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"ec235cfb-3e6e-40aa-9515-e6943b8e2241","volume":{"uuid":"05c88236-1aa6-44c3-9bf9-bad4edfc34e5","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"39bf30d6-f959-359d-b974-b8ebd2d8727f","id":5,"poolType":"NetworkFilesystem","host":"10.2.0.16","path":"/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1","port":2049,"url":"NetworkFilesystem://10.2.0.16/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1/?ROLE=Primary&STOREUUID=39bf30d6-f959-359d-b974-b8ebd2d8727f","isManaged":false}},"name":"xen-data-1","size":1073741824,"path":"dd772879-f006-41ec-b09e-db5699f7a7fb","volumeId":54,"vmName":"i-2-22-VM","accountId":2,"format":"VHD","provisioningType":"THIN","id":54,"deviceId":1,"hypervisorType":"XenServer"},"parentSnapshotPath":"ffda9432-b5c2-41db-ac36-2321b8ed92b3","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"39bf30d6-f959-359d-b974-b8ebd2d8727f","id":5,"poolType":"NetworkFilesystem","host":"10.2.0.16","path":"/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1","port":2049,"url":"NetworkFilesystem://10.2.0.16/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1/?ROLE=Primary&STOREUUID=39bf30d6-f959-359d-b974-b8ebd2d8727f","isManaged":false}},"vmName":"i-2-22-VM","name":"snap4","hypervisorType":"XenServer","id":85,"quiescevm":false,"parents":["ffda9432-b5c2-41db-ac36-2321b8ed92b3"],"physicalSize":0}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/54","volume":{"uuid":"05c88236-1aa6-44c3-9bf9-bad4edfc34e5","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"39bf30d6-f959-359d-b974-b8ebd2d8727f","id":5,"poolType":"NetworkFilesystem","host":"10.2.0.16","path":"/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1","port":2049,"url":"NetworkFilesystem://10.2.0.16/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1/?ROLE=Primary&STOREUUID=39bf30d6-f959-359d-b974-b8ebd2d8727f","isManaged":false}},"name":"xen-data-1","size":1073741824,"path":"dd772879-f006-41ec-b09e-db5699f7a7fb","volumeId":54,"vmName":"i-2-22-VM","accountId":2,"format":"VHD","provisioningType":"THIN","id":54,"deviceId":1,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshots/2/54/a585fa13-201e-40ef-b586-11822133287e.vhd","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"NFS://10.2.0.16/acs/secondary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-sec1","_role":"Image"}},"vmName":"i-2-22-VM","name":"snap4","hypervisorType":"XenServer","id":85,"quiescevm":false,"parents":["snapshots/2/54/14c13d66-53f4-4003-a284-cf266fe1ee3c.vhd","snapshots/2/54/151ca037-2b00-44a3-abb2-b212962b51be.vhd","snapshots/2/54/a585fa13-201e-40ef-b586-11822133287e.vhd"],"physicalSize":0}},"executeInSequence":true,"options":{"fullSnapshot":"true"},"options2":{},"wait":21600}}] } 2020-04-09 16:29:52,641 DEBUG [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-78:ctx-24a25b86) (logid:301a4ad6) No file SR found for path: /var/cloud_mount/569d499d-d7f7-31d4-b124-baf0ad97e161/snapshots/2/54 2020-04-09 16:29:52,641 DEBUG [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-78:ctx-24a25b86) (logid:301a4ad6) Creating file SR for path [/var/cloud_mount/569d499d-d7f7-31d4-b124-baf0ad97e161/snapshots/2/54] on host [fb4b5939-8702-46b7-bb02-7f43ab6bb9b8] 2020-04-09 16:30:04,285 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-78:ctx-24a25b86) (logid:301a4ad6) Host 10.2.3.240 OpaqueRef:3d230637-af2f-e354-2db8-e0fccbe62d2b: Removing SR 2020-04-09 16:30:04,325 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-78:ctx-24a25b86) (logid:301a4ad6) Host 10.2.3.240 OpaqueRef:d3f62ef8-cbbf-3d41-f4bd-c8fb4252adaa: Unplugging pbd 2020-04-09 16:30:04,651 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-78:ctx-24a25b86) (logid:301a4ad6) Host 10.2.3.240 OpaqueRef:3d230637-af2f-e354-2db8-e0fccbe62d2b: Forgetting **2020-04-09 16:30:04,667 DEBUG [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-78:ctx-24a25b86) (logid:301a4ad6) Successfully destroyed snapshot on volume: dd772879-f006-41ec-b09e-db5699f7a7fb execept this current snapshot ec235cfb-3e6e-40aa-9515-e6943b8e2241 2020-04-09 16:30:04,667 INFO [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-78:ctx-24a25b86) (logid:301a4ad6) New snapshot details: SnapshotTO[datastore=null|volume=null|pathsnapshots/2/54/d4e824b5-f9ac-469f-b94f-dbdf39e8b682.vhd]** 2020-04-09 16:30:04,667 INFO [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-78:ctx-24a25b86) (logid:301a4ad6) New snapshot physical utilization: 67244544 2020-04-09 16:30:04,668 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-78:ctx-24a25b86) (logid:301a4ad6) Seq 2-5707749577738682461: Response Received: 2020-04-09 16:30:04,668 DEBUG [c.c.a.t.Request] (DirectAgent-78:ctx-24a25b86) (logid:301a4ad6) Seq 2-5707749577738682461: Processing: { Ans: , MgmtId: 32988805007466, via: 2(ref-trl-774-k-M7-apanic-xs1), Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/54/d4e824b5-f9ac-469f-b94f-dbdf39e8b682.vhd","id":0,"quiescevm":false,"physicalSize":67244544}},"result":true,"wait":0}}] } 2020-04-09 16:30:04,668 DEBUG [c.c.a.m.AgentAttache] (DirectAgent-78:ctx-24a25b86) (logid:301a4ad6) Seq 2-5707749577738682461: No more commands found 2020-04-09 16:30:04,668 DEBUG [c.c.a.t.Request] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593 ctx-34f207e8) (logid:301a4ad6) Seq 2-5707749577738682461: Received: { Ans: , MgmtId: 32988805007466, via: 2(ref-trl-774-k-M7-apanic-xs1), Ver: v1, Flags: 110, { CopyCmdAnswer } } 2020-04-09 16:30:04,690 DEBUG [c.c.r.ResourceLimitManagerImpl] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593 ctx-34f207e8) (logid:301a4ad6) Updating resource Type = secondary_storage count for Account = 2 Operation = decreasing Amount = 1006497280 2020-04-09 16:30:04,694 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593 ctx-34f207e8) (logid:301a4ad6) Done executing VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":54,"policyId":0,"snapshotId":85,"quiesceVm":false,"asyncBackup":false,"userId":2,"accountId":2,"vmId":22,"handlerName":"VolumeApiServiceImpl"} 2020-04-09 16:30:04,694 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593 ctx-34f207e8) (logid:301a4ad6) Complete async job-593, jobStatus: SUCCEEDED, resultCode: 0, result: rO0ABXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAVQ 2020-04-09 16:30:04,695 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593 ctx-34f207e8) (logid:301a4ad6) Publish async job-593 complete on message bus 2020-04-09 16:30:04,695 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593 ctx-34f207e8) (logid:301a4ad6) Wake up jobs related to job-593 2020-04-09 16:30:04,695 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593 ctx-34f207e8) (logid:301a4ad6) Update db status for job-593 2020-04-09 16:30:04,696 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593 ctx-34f207e8) (logid:301a4ad6) Wake up jobs joined with job-593 and disjoin all subjobs created from job- 593 2020-04-09 16:30:04,702 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593) (logid:301a4ad6) Done with run of VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 22, job origin: 592 2020-04-09 16:30:04,702 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593) (logid:301a4ad6) Done executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-593 2020-04-09 16:30:04,705 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593) (logid:301a4ad6) Remove job-593 from job monitoring 2020-04-09 16:30:04,720 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-56cd28cc job-592 ctx-eabcd3e5) (logid:301a4ad6) Complete async job-592, jobStatus: SUCCEEDED, resultCode: 0, result: org.apache.cloudstack.api.response.SnapshotResponse/snapshot/{"id":"ba2df407-1772-4668-9327-026af717dbce","account":"admin","domainid":"887f0da6-72a9-11ea-9abc-1e00ce01086a","domain":"ROOT","snapshottype":"MANUAL","volumeid":"05c88236-1aa6-44c3-9bf9-bad4edfc34e5","volumename":"xen-data-1","volumetype":"DATADISK","created":"2020-04-09T16:29:45+0000","name":"snap4","intervaltype":"MANUAL","state":"BackedUp","physicalsize":67244544,"zoneid":"e5d78fcd-deb6-4316-9190-136bc788b265","revertable":false,"virtualsize":1073741824,"tags":[]} 2020-04-09 16:30:04,721 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-56cd28cc job-592 ctx-eabcd3e5) (logid:301a4ad6) Publish async job-592 complete on message bus 2020-04-09 16:30:04,721 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-56cd28cc job-592 ctx-eabcd3e5) (logid:301a4ad6) Wake up jobs related to job-592 2020-04-09 16:30:04,721 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-56cd28cc job-592 ctx-eabcd3e5) (logid:301a4ad6) Update db status for job-592 2020-04-09 16:30:04,721 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-56cd28cc job-592 ctx-eabcd3e5) (logid:301a4ad6) Wake up jobs joined with job-592 and disjoin all subjobs created from job- 592 2020-04-09 16:30:04,725 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-56cd28cc job-592) (logid:301a4ad6) Done executing org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-592 2020-04-09 16:30:04,725 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-12:ctx-56cd28cc job-592) (logid:301a4ad6) Remove job-592 from job monitoring From XS snapshots, we can create template/volume: - In both cases (creating a template of a volume) - the existing folder holding the snaphosts on the Secondary Storage is mounted as a new SRs on XS host and template is created (again inside a new SR from a new folder where the template is put - previously created on the Sec Stor). For creating a volume out of a snap, just the existing Secondary Storage snap folder is mounted and whole image copied back to Primary Storage - i.e. we NEVER read original snap from the PRIMARY storage - confirmed by manually deleteing all snaps of a volume and being able to create tempalte/volume from some snap. - i.e. we should really delete all older snaps (as it seems) when we create a new one, from the Primary Storage - and keep all snaps on Secondary Storage (like so far)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
