andrijapanicsb edited a comment on issue #3649: Garbage snapshots are left on Primary and Secondary URL: https://github.com/apache/cloudstack/pull/3649#issuecomment-557715680 Alright - so far tested only KVM (centos 7) with NFS: **While backup to secondary = true**, results as follows: - snap created on Prim Stor, copied to Sec Store, snap deleted from Prim Store (exists only on Sec Store) - revert from snap is possible (image moved back and overwritten on Prim Store, as FULL clone) - When deleting snap, it's instantly removed from Sec Stor NFS, DB records are changed to "destroyed" state for both snap reference on Prim Store and on Sec Store but no DB records removed so far. - CleanUp thread finds snaps in snapshos_store_ref, and removes them from the snapshots_store_ref table - but it only removes the ones that are pointing to the Sec Store, not the ones that are pointing to PrimStore (though they are marked as deleted, but rows physically present in the table) In general, looking good, except that those records for snap reference on Prim Store are not deleted from the snapshots_store_ref table - not sure what was the original behaviour here, years ago. **While backup to secondary = false**, results as follows: - snap created and stays only on Prim Store, all good. - **can't delete the snap** - getting error _"Failed to delete snapshot:java.util.MissingFormatArgumentException: Format specifier '%d'"_ - cant revert volume from snap (seems to require to be on secondary storage, API confirms this if you try to revert it via API) - what is the use of this setup then? I'm certainly missing something or perhaps idea is that we have identical behaviour as many years ago, just this time we don't copy it over to Sec Store @GabrielBrascher (you can only create a template or a volume out of it - can't restore it) Here is a more detailed error when deleting a snap that exists only on Prim Store: > 2019-11-22 22:18:42,983 DEBUG [c.c.a.ApiServlet] (qtp504527234-18:ctx-12265ddd ctx-75ecde1c) (logid:83e43335) ===END=== 10.1.0.1 -- GET command=deleteSnapshot&id=73120b09-10d7-48e0-bddb-42b442bbd79e&response=json&_=1574460913233 > 2019-11-22 22:18:43,030 DEBUG [c.c.s.s.SnapshotManagerImpl] (API-Job-Executor-3:ctx-2cf618ce job-75 ctx-39866784) (logid:6a5a4b68) Failed to delete snapshot: 7:java.util.MissingFormatArgumentException: Format specifier '%d' > 2019-11-22 22:18:43,038 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-3:ctx-2cf618ce job-75) (logid:6a5a4b68) Unexpected exception while executing org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd > com.cloud.utils.exception.CloudRuntimeException: Failed to delete snapshot:java.util.MissingFormatArgumentException: Format specifier '%d' > at com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(SnapshotManagerImpl.java:611) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) > at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) > at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) > at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) > at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) > at com.sun.proxy.$Proxy199.deleteSnapshot(Unknown Source) > at org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd.execute(DeleteSnapshotCmd.java:103) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156) > at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) > at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:583) > at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:531) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > 2019-11-22 22:18:43,035 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-1:ctx-3d310368 job-102) (logid:ed551d42) Complete async job-102, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to delete snapshot:java.util.MissingFormatArgumentException: Format specifier \u0027%d\u0027"} @GabrielBrascher I'm wondering, is the behaviour SAME in your tests with Ceph (you obviously do NOT have issues deleting the snap from the Ceph only ( backup to Sec Store = false) Current behaviour that I see (snap removed from Prim when it's copied over to Sec) is the new in this PR - i.e. previously snap was left on Prim Store even though it was copied over to Sec Store.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
