manasaveloori created CLOUDSTACK-7263:
-----------------------------------------
Summary: Account Clean up is failing when there are snapshots in
allocated state.
Key: CLOUDSTACK-7263
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7263
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Snapshot, Storage Controller
Affects Versions: 4.5.0
Reporter: manasaveloori
Priority: Critical
Fix For: 4.5.0
Steps:
1. Created an account "test" with a user "testuser"
2. Deployed a VM with data disk.
3. Created snapshots of both root and data disks.
4. Scheduled hourly,daily,weekly,monthly snapshots of both data and root
volumes.
5. While the snapshot is in backingup state deleted the account "test"
Observation:
1. Account "test" got deleted but clean up of account failed as one of the
snapshot is in "backingup" state.
2. Now changed the "account.cleanup.interval"=300sec
3. After 300secs...Account cleanup is triggered and at the same time scheduled
snapshot creation is triggered.....
Snapshot creation failed with NPE and left in Allocated state
2014-08-06 12:51:42,597 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-11:ctx-0e534263 job-279) Add job-279 into job monitoring
2014-08-06 12:51:42,597 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-11:ctx-0e534263 job-279) Executing AsyncJobVO {id:279,
userId: 1, accountId: 5, instanceType: Snapshot, instanceId: 52, cmd:
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, cmdInfo:
{"id":"52","ctxUserId":"1","volumeid":"24","ctxAccountId":"5","ctxStartEventId":"1","policyid":"15"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 6876007760021, completeMsid: null, lastUpdated: null,
lastPolled: null, created: null}
2014-08-06 12:51:42,600 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-11:ctx-0e534263 job-279) Unexpected exception while executing
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd
java.lang.NullPointerException
at org.apache.cloudstack.context.CallContext.<init>(CallContext.java:82)
at
org.apache.cloudstack.context.CallContext.register(CallContext.java:156)
at
org.apache.cloudstack.context.CallContext.register(CallContext.java:143)
at
org.apache.cloudstack.context.CallContext.register(CallContext.java:180)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:100)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-08-06 12:51:42,606 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-11:ctx-0e534263 job-279) Complete async job-279, jobStatus:
FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530}
2014-08-06 12:51:42,614 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-11:ctx-0e534263 job-279) Done executing
org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-279
4. Now as the snapshot state is left in "allocated state...everytime cleanup is
triggered it is failed with below exception and account never gets cleaned up:
2014-08-06 12:51:42,580 WARN [c.c.u.AccountManagerImpl]
(AccountChecker-1:ctx-eb492f52) Failed to cleanup account
Acct[deb3b748-63ca-4566-8c34-a8bf6685bf11-acct] due to
com.cloud.exception.InvalidParameterValueException: Can't delete snapshotshot
51 due to it is in Allocated Status
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.deleteSnapshot(XenserverSnapshotStrategy.java:213)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshotDirsForAccount(SnapshotManagerImpl.java:623)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy179.deleteSnapshotDirsForAccount(Unknown Source)
at
com.cloud.user.AccountManagerImpl.cleanupAccount(AccountManagerImpl.java:708)
at
com.cloud.user.AccountManagerImpl$AccountCleanupTask.runInContext(AccountManagerImpl.java:1675)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-08-06 12:51:42,582 INFO [c.c.u.AccountManagerImpl]
(AccountChecker-1:ctx-eb492f52) Cleanup for account 5 is needed.
Attaching the Ms logs
--
This message was sent by Atlassian JIRA
(v6.2#6252)
