Hi,
Overtime, there are many volume snapshot are failed to remove from UI or API,
the snapshot entries are changed to “Destroying” states from snapshot_store_ref
but never cleanup by storage GC, and the snapshot entries remains in UI, do you
know any safety way to clear them properly.
Appreciated for your help.
Regards,
Levin
2024-03-12 01:33:04,827 WARN [o.a.c.s.s.DefaultSnapshotStrategy]
(API-Job-Executor-55:ctx-51c04882 job-100407 ctx-76acc975) (logid:80e4d8b4)
Unable do delete snapshot Snapshot
{"uuid":"cce84201-44d7-4b15-a6f4-2cf6cfa9ec47","name":"redmine-1_ROOT-2058_20220507080822","volumeId":2439}
on Primary {uuid: "5db4c6c5-cc09-4f99-af42-c2256f76748c", name: "sg1"} due to
[Failed to remove snapshot 22759382-0986-4771-ab83-573b78eb7383]. The reference
will be marked as 'Destroying' for future garbage collecting.
com.cloud.utils.exception.CloudRuntimeException: Failed to remove snapshot
22759382-0986-4771-ab83-573b78eb7383
at
org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.deleteSnapshot(SnapshotServiceImpl.java:541)
at
org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.deleteSnapshotInPrimaryStorage(DefaultSnapshotStrategy.java:392)
at
org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.deleteSnapshotInfo(DefaultSnapshotStrategy.java:375)
at
org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.deleteSnapshotInfos(DefaultSnapshotStrategy.java:340)
at
org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.destroySnapshotEntriesAndFiles(DefaultSnapshotStrategy.java:312)
at
org.apache.cloudstack.storage.snapshot.DefaultSnapshotStrategy.deleteSnapshot(DefaultSnapshotStrategy.java:304)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(SnapshotManagerImpl.java:714)
at jdk.internal.reflect.GeneratedMethodAccessor3402.invoke(Unknown Source)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy224.deleteSnapshot(Unknown Source)
at
org.apache.cloudstack.api.command.user.snapshot.DeleteSnapshotCmd.execute(DeleteSnapshotCmd.java:106)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
