andrijapanicsb edited a comment on issue #4018: Snapshots GC from DB and XenServer Primary Storage garbage URL: https://github.com/apache/cloudstack/issues/4018#issuecomment-611628944 Observation from logs: XenServer removes **previous** snapshots from the "primary storage" when a new one is created - it does NOT delete it really, but deletes the PRIMARY row in the snap_store_ref table, thus the garbage is left on Primary Store while the reference is deleted from the snap_store_ref - and later GC is not picking it up as the row/reference is not there.... Stupid implementation by all means: [root@ref-trl-774-k-M7-apanic-mgmt1 ~]# grep 301a4ad6 /var/log/cloudstack/management/management-server.log | grep -v "queryAsyncJobResult" 2020-04-09 16:29:45,717 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-56cd28cc job-592) (logid:301a4ad6) Executing AsyncJobVO {id:592, userId: 2, accountId: 2, instanceType: Snapshot, instanceId: 85, cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, cmdInfo: {"quiescevm":"false","httpmethod":"GET","ctxAccountId":"2","uuid":"ba2df407-1772-4668-9327-026af717dbce","cmdEventType":"SNAPSHOT.CREATE","asyncBackup":"false","response":"json","ctxUserId":"2","volumeId":"05c88236-1aa6-44c3-9bf9-bad4edfc34e5","name":"snap4","ctxStartEventId":"1274","id":"85","ctxDetails":"{\"interface com.cloud.storage.Snapshot\":\"ba2df407-1772-4668-9327-026af717dbce\",\"interface com.cloud.storage.Volume\":\"05c88236-1aa6-44c3-9bf9-bad4edfc34e5\"}","_":"1586449031328"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 32988805007466, completeMsid: null, lastUpdated: null, lastPolled: null, created: null, removed: null} 2020-04-09 16:29:45,743 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-12:ctx-56cd28cc job-592 ctx-eabcd3e5) (logid:301a4ad6) Sync job-593 execution on object VmWorkJobQueue.22 ... ... 2020-04-09 16:29:50,328 DEBUG [c.c.a.t.Request] (Work-Job-Executor-8:ctx-b9c690a2 job-592/job-593 ctx-34f207e8) (logid:301a4ad6) Seq 2-5707749577738682461: Executing: { Cmd , MgmtId: 32988805007466, via: 2(ref-trl-774-k-M7-apanic-xs1), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"ec235cfb-3e6e-40aa-9515-e6943b8e2241","volume":{"uuid":"05c88236-1aa6-44c3-9bf9-bad4edfc34e5","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"39bf30d6-f959-359d-b974-b8ebd2d8727f","id":5,"poolType":"NetworkFilesystem","host":"10.2.0.16","path":"/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1","port":2049,"url":"NetworkFilesystem://10.2.0.16/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1/?ROLE=Primary&STOREUUID=39bf30d6-f959-359d-b974-b8ebd2d8727f","isManaged":false}},"name":"xen-data-1","size":1073741824,"path":"dd772879-f006-41ec-b09e-db5699f7a7fb","volumeId":54,"vmName":"i-2-22-VM","accountId":2,"format":"VHD","provisioningType":"THIN","id":54,"deviceId":1,"hypervisorType":"XenServer"},"parentSnapshotPath":"ffda9432-b5c2-41db-ac36-2321b8ed92b3","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"39bf30d6-f959-359d-b974-b8ebd2d8727f","id":5,"poolType":"NetworkFilesystem","host":"10.2.0.16","path":"/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1","port":2049,"url":"NetworkFilesystem://10.2.0.16/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1/?ROLE=Primary&STOREUUID=39bf30d6-f959-359d-b974-b8ebd2d8727f","isManaged":false}},"vmName":"i-2-22-VM","name":"snap4","hypervisorType":"XenServer","id":85,"quiescevm":false,"parents":["ffda9432-b5c2-41db-ac36-2321b8ed92b3"],"physicalSize":0}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/54","volume":{"uuid":"05c88236-1aa6-44c3-9bf9-bad4edfc34e5","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"39bf30d6-f959-359d-b974-b8ebd2d8727f","id":5,"poolType":"NetworkFilesystem","host":"10.2.0.16","path":"/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1","port":2049,"url":"NetworkFilesystem://10.2.0.16/acs/primary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-xs-pri1/?ROLE=Primary&STOREUUID=39bf30d6-f959-359d-b974-b8ebd2d8727f","isManaged":false}}"name":"xen-data-1","size":1073741824,"path":"dd772879-f006-41ec-b09e-db5699f7a7fb","volumeId":54,"vmName":"i-2-22-VM","accountId":2,"format":"VHD","provisioningType":"THIN","id":54,"deviceId":1,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshots/2/54/a585fa13-201e-40ef-b586-11822133287e.vhd","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"NFS://10.2.0.16/acs/secondary/ref-trl-774-k-M7-apanic/ref-trl-774-k-M7-apanic-sec1","_role":"Image"}},"vmName":"i-2-22-VM","name":"snap4","hypervisorType":"XenServer","id":85,"quiescevm":false,"parents":["snapshots/2/54/14c13d66-53f4-4003-a284-cf266fe1ee3c.vhd","snapshots/2/54/151ca037-2b00-44a3-abb2-b212962b51be.vhd","snapshots/2/54/a585fa13-201e-40ef-b586-11822133287e.vhd"],"physicalSize":0}},"executeInSequence":true,"options": ... **{"fullSnapshot":"true"},"options2":{},"wait":21600}}] }** 2020-04-09 16:29:52,641 DEBUG [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-78:ctx-24a25b86) (logid:301a4ad6) No file SR found for path: /var/cloud_mount/569d499d-d7f7-31d4-b124-baf0ad97e161/snapshots/2/54 2020-04-09 16:29:52,641 DEBUG [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-78:ctx-24a25b86) (logid:301a4ad6) Creating file SR for path [/var/cloud_mount/569d499d-d7f7-31d4-b124-baf0ad97e161/snapshots/2/54] on host [fb4b5939-8702-46b7-bb02-7f43ab6bb9b8] 2020-04-09 16:30:04,285 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-78:ctx-24a25b86) (logid:301a4ad6) Host 10.2.3.240 OpaqueRef:3d230637-af2f-e354-2db8-e0fccbe62d2b: Removing SR 2020-04-09 16:30:04,325 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-78:ctx-24a25b86) (logid:301a4ad6) Host 10.2.3.240 OpaqueRef:d3f62ef8-cbbf-3d41-f4bd-c8fb4252adaa: Unplugging pbd 2020-04-09 16:30:04,651 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-78:ctx-24a25b86) (logid:301a4ad6) Host 10.2.3.240 OpaqueRef:3d230637-af2f-e354-2db8-e0fccbe62d2b: Forgetting **2020-04-09 16:30:04,667 DEBUG [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-78:ctx-24a25b86) (logid:301a4ad6) Successfully destroyed snapshot on volume: dd772879-f006-41ec-b09e-db5699f7a7fb execept this current snapshot ec235cfb-3e6e-40aa-9515-e6943b8e2241 2020-04-09 16:30:04,667 INFO [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-78:ctx-24a25b86) (logid:301a4ad6) New snapshot details: SnapshotTO[datastore=null|volume=null|pathsnapshots/2/54/d4e824b5-f9ac-469f-b94f-dbdf39e8b682.vhd]** ... ... From XS snapshots, we can create template/volume: - In both cases (creating a template of a volume) - the existing folder holding the snaphosts on the Secondary Storage is mounted as a new SRs on XS host and template is created (again inside a new SR from a new folder where the template is put - previously created on the Sec Stor). For creating a volume out of a snap, just the existing Secondary Storage snap folder is mounted and whole image copied back to Primary Storage - i.e. we NEVER read original snap from the PRIMARY storage - confirmed by manually deleting all snaps of a volume and being able to create tempalte/volume from some snap. - i.e. we should really delete all older snaps (as it seems) when we create a new one, from the Primary Storage - and keep all snaps on Secondary Storage (like so far)
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
