Thanks for the responses all. The "Removed Field" for the snapshots with the
status of "BackedUp" is NULL.
I combed the logs and found the exception below. It was successfully deleting
snapshots before that log entry, then errored on the "Allocated" snapshot and
stopped any further deletions. I'm not sure what allocated mean, but will
start researching.
2016-04-20 00:44:38,830 DEBUG [c.c.s.s.SnapshotManagerImpl]
(Work-Job-Executor-1:ctx-954cbd99 job-7396/job-7401 ctx-d0261c6c)
(logid:bafb5d42) post process snapshot failed
com.cloud.utils.exception.CloudRuntimeException: Failed to delete
snapshot:com.cloud.exception.InvalidParameterValueException: Can't delete
snapshotshot 1351 due to it is in Allocated Status
at
com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshot(SnapshotManagerImpl.java:478)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.postCreateRecurringSnapshotForPolicy(SnapshotManagerImpl.java:420)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.postCreateSnapshot(SnapshotManagerImpl.java:399)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1010)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy191.takeSnapshot(Unknown Source)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1591)
at
com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2107)
at
com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2899)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2907)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy196.handleVmWorkJob(Unknown Source)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
-----Original Message-----
From: Anshul Gangwar [mailto:[email protected]]
Sent: Wednesday, April 20, 2016 12:05 AM
To: [email protected]
Subject: Re: Bug in Snapshot Retention?
Are you getting any exception in logs after the completion of snapshot for this
volume?
If so try changing “job.cancel.threshold” global setting to appropriate
timeout. This will make sure that job is not completed before the snapshot
process is finished and will also make sure that cleanup process is not
failing. This setting will affect all the jobs.
Regards,
Anshul
> On 20-Apr-2016, at 6:24 AM, Sean Lair <[email protected]> wrote:
>
> Hi all,
>
> I'm running Cloudstack 4.8 on XenServer 6.5. I have one volume that I'm
> taking snapshots of, it is set to keep a total of 29 snapshots, but I have
> close to 100 snapshots in the state of "BackedUp". Am I misinterpreting the
> scheduled snapshot screen or am I running into a bug? Please see the output
> below for more detail:
>
> [cid:[email protected]]
>
> MariaDB [cloud]> select count(status) from snapshots where volume_id = 71 and
> status = 'backedup';
> +---------------+
> | count(status) |
> +---------------+
> | 98 |
> +---------------+
> 1 row in set (0.00 sec)
>
> Thanks
> Sean
>
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the
property of Accelerite, a Persistent Systems business. It is intended only for
the use of the individual or entity to which it is addressed. If you are not
the intended recipient, you are not authorized to read, retain, copy, print,
distribute or use this message. If you have received this communication in
error, please notify the sender and delete all copies of this message.
Accelerite, a Persistent Systems business does not accept any liability for
virus infected mails.
