Sailaja Mada created CLOUDSTACK-3876:
----------------------------------------
Summary: Runtime Exceptions while deleting account which has
snapshots created from volume
Key: CLOUDSTACK-3876
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3876
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Storage Controller
Affects Versions: 4.2.0
Reporter: Sailaja Mada
Steps:
1. Configure AdvZone with Xenserver
2. Create new account
3. Deploy VM
4. Add new volume and attach to this instance
5. Create snapshots to this volume ( Total4 )
6. Delete the account
7. Account got deleted with no issues but there were Runtime exceptions in the
log while deleting the snapshots of DATA volume.
2013-07-27 14:24:55,363 DEBUG [cloud.api.ApiServlet] (catalina-exec-2:null)
===START=== 10.104.255.13 -- GET
command=deleteAccount&response=json&sessionkey=C%2FrzjdA6B1wbWrgmfiasGDl6A6Q%3D&id=7cf7f1c8-b1f2-4754-9049-10506322e037&_=1374915066313
2013-07-27 14:24:55,398 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-2:null) submit async job-45 = [
e87af4d1-54e2-46b4-bbc9-3b020bb364ba ], details: AsyncJobVO {id:45, userId: 2,
accountId: 2, sessionKey: null, instanceType: Account, instanceId: null, cmd:
org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd,
cmdOriginator: null, cmdInfo:
{"response":"json","id":"7cf7f1c8-b1f2-4754-9049-10506322e037","sessionkey":"C/rzjdA6B1wbWrgmfiasGDl6A6Q\u003d","cmdEventType":"ACCOUNT.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1374915066313","ctxAccountId":"2","ctxStartEventId":"163"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 226870599129537,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2013-07-27 14:24:55,401 DEBUG [cloud.api.ApiServlet] (catalina-exec-2:null)
===END=== 10.104.255.13 -- GET
command=deleteAccount&response=json&sessionkey=C%2FrzjdA6B1wbWrgmfiasGDl6A6Q%3D&id=7cf7f1c8-b1f2-4754-9049-10506322e037&_=1374915066313
2013-07-27 14:24:55,403 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Executing
org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd for job-45 = [
e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]
2013-07-27 14:24:55,434 DEBUG [cloud.user.AccountManagerImpl]
(Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Removed
account 4
2013-07-27 14:24:55,459 DEBUG [agent.transport.Request] (Job-Executor-34:job-45
= [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq 5-1160774981: Sending { Cmd ,
MgmtId: 226870599129537, via: 5, Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/abhinav/xen-sec-old","_role":"Image"}},"directory":"snapshots/4/16","wait":0}}]
}
2013-07-27 14:24:55,551 DEBUG [agent.transport.Request]
(AgentManager-Handler-2:null) Seq 5-1160774981: Processing: { Ans: , MgmtId:
226870599129537, via: 5, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2013-07-27 14:24:55,551 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-2:null) Seq 5-1160774981: No more commands found
2013-07-27 14:24:55,551 DEBUG [agent.transport.Request] (Job-Executor-34:job-45
= [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq 5-1160774981: Received: { Ans:
, MgmtId: 226870599129537, via: 5, Ver: v1, Flags: 110, { Answer } }
2013-07-27 14:24:55,551 DEBUG [storage.snapshot.SnapshotManagerImpl]
(Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Deleted all
snapshots for volume: 16 under account: 4
2013-07-27 14:24:55,567 DEBUG [storage.snapshot.XenserverSnapshotStrategy]
(Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) delete
snapshot chain for snapshot: 4
2013-07-27 14:24:55,569 DEBUG [storage.snapshot.XenserverSnapshotStrategy]
(Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Snapshot: 4
doesn't have children, so it's ok to delete it and its parents
2013-07-27 14:24:55,616 DEBUG [agent.transport.Request] (Job-Executor-34:job-45
= [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq 5-1160774982: Sending { Cmd ,
MgmtId: 226870599129537, via: 5, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/16/6671a6a2-565f-47ec-96cf-f804665c600d","volume":{"uuid":"bb9c0beb-ba86-4d03-9cbe-66dad892cd57","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c6bb64ae-9919-3068-97a9-fe8ca71acbe0","id":200,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/abhinav/xen-pri1","port":2049}},"name":"ROOT-11","size":21474836480,"path":"12bb3e53-db3b-4d47-ab01-a4d8bbd5bbc3","volumeId":16,"vmName":"i-4-11-VM","accountId":4,"format":"VHD","id":16},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/abhinav/xen-sec-old","_role":"Image"}},"vmName":"i-4-11-VM","name":"instance1_ROOT-11_20130727072147","hypervisorType":"XenServer","id":4}},"wait":0}}]
}
2013-07-27 14:24:55,641 DEBUG [agent.transport.Request]
(AgentManager-Handler-4:null) Seq 5-1160774982: Processing: { Ans: , MgmtId:
226870599129537, via: 5, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"snapshot directory 16
doesn't exist","wait":0}}] }
2013-07-27 14:24:55,642 DEBUG [agent.transport.Request] (Job-Executor-34:job-45
= [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq 5-1160774982: Received: { Ans:
, MgmtId: 226870599129537, via: 5, Ver: v1, Flags: 10, { Answer } }
2013-07-27 14:24:55,642 DEBUG [storage.snapshot.SnapshotServiceImpl]
(Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) delete
snapshot failedsnapshot directory 16 doesn't exist
2013-07-27 14:24:55,673 DEBUG [storage.snapshot.XenserverSnapshotStrategy]
(Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Failed to
delete snapshot:
com.cloud.utils.exception.CloudRuntimeException: snapshot directory 16 doesn't
exist
at
org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.deleteSnapshot(SnapshotServiceImpl.java:375)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.deleteSnapshotChain(XenserverSnapshotStrategy.java:143)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.deleteSnapshot(XenserverSnapshotStrategy.java:187)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshotDirsForAccount(SnapshotManagerImpl.java:689)
at
com.cloud.user.AccountManagerImpl.cleanupAccount(AccountManagerImpl.java:581)
at
com.cloud.user.AccountManagerImpl.deleteAccount(AccountManagerImpl.java:544)
at
com.cloud.user.AccountManagerImpl.deleteUserAccount(AccountManagerImpl.java:1273)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.region.RegionManagerImpl.deleteUserAccount(RegionManagerImpl.java:177)
at
org.apache.cloudstack.region.RegionServiceImpl.deleteUserAccount(RegionServiceImpl.java:118)
at
org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd.execute(DeleteAccountCmd.java:100)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
2013-07-27 14:24:55,736 DEBUG [agent.transport.Request] (Job-Executor-34:job-45
= [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq 5-1160774983: Sending { Cmd ,
MgmtId: 226870599129537, via: 5, Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/abhinav/xen-sec-old","_role":"Image"}},"directory":"snapshots/4/18","wait":0}}]
}
2013-07-27 14:24:55,866 DEBUG [agent.transport.Request]
(AgentManager-Handler-3:null) Seq 5-1160774983: Processing: { Ans: , MgmtId:
226870599129537, via: 5, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2013-07-27 14:24:55,866 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-3:null) Seq 5-1160774983: No more commands found
2013-07-27 14:24:55,866 DEBUG [agent.transport.Request] (Job-Executor-34:job-45
= [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq 5-1160774983: Received: { Ans:
, MgmtId: 226870599129537, via: 5, Ver: v1, Flags: 110, { Answer } }
2013-07-27 14:24:55,866 DEBUG [storage.snapshot.SnapshotManagerImpl]
(Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Deleted all
snapshots for volume: 18 under account: 4
2013-07-27 14:24:55,882 DEBUG [storage.snapshot.XenserverSnapshotStrategy]
(Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) delete
snapshot chain for snapshot: 8
2013-07-27 14:24:55,885 DEBUG [storage.snapshot.XenserverSnapshotStrategy]
(Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) the
snapshot has child, can't delete it on the storage
2013-07-27 14:24:55,944 DEBUG [storage.snapshot.XenserverSnapshotStrategy]
(Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) delete
snapshot chain for snapshot: 9
2013-07-27 14:24:55,947 DEBUG [storage.snapshot.XenserverSnapshotStrategy]
(Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) the
snapshot has child, can't delete it on the storage
2013-07-27 14:24:56,011 DEBUG [storage.snapshot.XenserverSnapshotStrategy]
(Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) delete
snapshot chain for snapshot: 10
2013-07-27 14:24:56,014 DEBUG [storage.snapshot.XenserverSnapshotStrategy]
(Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) the
snapshot has child, can't delete it on the storage
2013-07-27 14:24:56,075 DEBUG [storage.snapshot.XenserverSnapshotStrategy]
(Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) delete
snapshot chain for snapshot: 11
2013-07-27 14:24:56,077 DEBUG [storage.snapshot.XenserverSnapshotStrategy]
(Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Snapshot:
11 doesn't have children, so it's ok to delete it and its parents
2013-07-27 14:24:56,129 DEBUG [agent.transport.Request] (Job-Executor-34:job-45
= [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq 5-1160774984: Sending { Cmd ,
MgmtId: 226870599129537, via: 5, Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/18/c44a1894-0142-444a-9c75-868358893a6a","volume":{"uuid":"f4e0e5fa-1a98-4dbb-b7b4-10e4ddf28465","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"c6bb64ae-9919-3068-97a9-fe8ca71acbe0","id":200,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/abhinav/xen-pri1","port":2049}},"name":"Volumenew1","size":5368709120,"path":"a06f2cde-31f2-41e3-bdaa-82d224996856","volumeId":18,"vmName":"i-4-11-VM","accountId":4,"format":"VHD","id":18},"parentSnapshotPath":"snapshots/4/18/c44a1894-0142-444a-9c75-868358893a6a","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/abhinav/xen-sec-old","_role":"Image"}},"vmName":"i-4-11-VM","name":"instance1_Volumenew1_20130727085422","hypervisorType":"XenServer","id":11}},"wait":0}}]
}
2013-07-27 14:24:56,174 DEBUG [agent.transport.Request]
(AgentManager-Handler-7:null) Seq 5-1160774984: Processing: { Ans: , MgmtId:
226870599129537, via: 5, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"snapshot directory 18
doesn't exist","wait":0}}] }
2013-07-27 14:24:56,175 DEBUG [agent.transport.Request] (Job-Executor-34:job-45
= [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq 5-1160774984: Received: { Ans:
, MgmtId: 226870599129537, via: 5, Ver: v1, Flags: 10, { Answer } }
2013-07-27 14:24:56,175 DEBUG [storage.snapshot.SnapshotServiceImpl]
(Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) delete
snapshot failedsnapshot directory 18 doesn't exist
2013-07-27 14:24:56,192 DEBUG [storage.snapshot.XenserverSnapshotStrategy]
(Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Failed to
delete snapshot:
com.cloud.utils.exception.CloudRuntimeException: snapshot directory 18 doesn't
exist
at
org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.deleteSnapshot(SnapshotServiceImpl.java:375)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.deleteSnapshotChain(XenserverSnapshotStrategy.java:143)
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.deleteSnapshot(XenserverSnapshotStrategy.java:187)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshotDirsForAccount(SnapshotManagerImpl.java:689)
at
com.cloud.user.AccountManagerImpl.cleanupAccount(AccountManagerImpl.java:581)
at
com.cloud.user.AccountManagerImpl.deleteAccount(AccountManagerImpl.java:544)
at
com.cloud.user.AccountManagerImpl.deleteUserAccount(AccountManagerImpl.java:1273)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.region.RegionManagerImpl.deleteUserAccount(RegionManagerImpl.java:177)
at
org.apache.cloudstack.region.RegionServiceImpl.deleteUserAccount(RegionServiceImpl.java:118)
at
org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd.execute(DeleteAccountCmd.java:100)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
2013-07-27 14:24:56,234 DEBUG [cloud.user.AccountManagerImpl]
(Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ])
Successfully deleted snapshots directories for all volumes under account 4
across all zones
2013-07-27 14:24:56,238 DEBUG [cloud.user.AccountManagerImpl]
(Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Expunging #
of vms (accountId=4): 1
2013-07-27 14:24:56,261 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-34:job-45 = [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) VM state
transitted from :Running to Stopping with event: StopRequestedvm's original
host id: 1 new host id: 1 host id before state transition: 1
2013-07-27 14:24:56,265 DEBUG [agent.transport.Request] (Job-Executor-34:job-45
= [ e87af4d1-54e2-46b4-bbc9-3b020bb364ba ]) Seq 1-1592922337: Sending { Cmd ,
MgmtId: 226870599129537, via: 1, Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"i-4-11-VM","wait":0}}]
}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
