[
https://issues.apache.org/jira/browse/CLOUDSTACK-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sanjay Tripathi resolved CLOUDSTACK-7799.
-----------------------------------------
Resolution: Not a Problem
> [Accounts] Account deletion failed to remove all the entities related to it
> in case of failure in deleting one entity
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-7799
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7799
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.5.0
> Environment: Latest build from 4.5 with commit [root@BPKxDmS ~]#
> cloudstack-sccs
> 385c4f673dfbd1fd326e539625e2c06db4cdc27d
> Reporter: Sanjeev N
> Assignee: Sanjay Tripathi
> Priority: Critical
> Fix For: 4.5.0
>
> Attachments: cloud.dmp, management-server.rar
>
>
> [Critical][Accounts] Account deletion failed to remove all the entities
> related to it in case of failure in deleting one entity
> Steps to Reproduce:
> ================
> 1.Bring up CS in advanced zone with one vmware cluster
> 2.Create a user account
> 3.With the user account deploy few (4-5) vms
> 4.Take snapshots on all of the vms root disks
> 5.Deploy another vm and simulate snapshot failure operation(In my case I
> tried snapshot operation with quiesce option set to true) so that snapshot
> will be in "Allocated" state
> 6.Delete this user account
> Expected Behavior:
> ==============
> Whatever may the state of the snapshot account deletion shall clean all the
> objects related that account
> Actual Behavior:
> =============
> Account deletion started deleting snapshots. Since one of the snapshots is in
> Allocated state it failed to delete that snapshot and the Account clean up
> job ended there. So vms and networks related to the accounts are still there
> but the account was deleted.
> Here is the log snippet:
> =================
> 2014-10-28 15:16:38,822 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-5:ctx-1c16bc72) ===START=== 10.252.193.8 -- GET
> command=deleteAccount&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&id=83f26103-9d83-4e4e-8bcc-c282c2acb498&_=1414470667367
> 2014-10-28 15:16:38,974 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (catalina-exec-5:ctx-1c16bc72 ctx-51140d28) submit async job-53, details:
> AsyncJobVO {id:53, userId: 2, accountId: 2, instanceType: Account,
> instanceId: null, cmd:
> org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd, cmdInfo:
> {"response":"json","id":"83f26103-9d83-4e4e-8bcc-c282c2acb498","sessionkey":"EpI1EuP5ZFF0VKispQyNb9AxWF4\u003d","ctxDetails":"{\"com.cloud.user.Account\":\"83f26103-9d83-4e4e-8bcc-c282c2acb498\"}","cmdEventType":"ACCOUNT.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1414470667367","uuid":"83f26103-9d83-4e4e-8bcc-c282c2acb498","ctxAccountId":"2","ctxStartEventId":"118"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
> null, initMsid: 6637838401571, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: null}
> 2014-10-28 15:16:38,978 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-5:ctx-1c16bc72 ctx-51140d28) ===END=== 10.252.193.8 -- GET
> command=deleteAccount&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&id=83f26103-9d83-4e4e-8bcc-c282c2acb498&_=1414470667367
> 2014-10-28 15:16:38,996 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-30:ctx-0b9151e2 job-53) Add job-53 into job monitoring
> 2014-10-28 15:16:38,996 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53) Executing AsyncJobVO {id:53,
> userId: 2, accountId: 2, instanceType: Account, instanceId: null, cmd:
> org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd, cmdInfo:
> {"response":"json","id":"83f26103-9d83-4e4e-8bcc-c282c2acb498","sessionkey":"EpI1EuP5ZFF0VKispQyNb9AxWF4\u003d","ctxDetails":"{\"com.cloud.user.Account\":\"83f26103-9d83-4e4e-8bcc-c282c2acb498\"}","cmdEventType":"ACCOUNT.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1414470667367","uuid":"83f26103-9d83-4e4e-8bcc-c282c2acb498","ctxAccountId":"2","ctxStartEventId":"118"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
> null, initMsid: 6637838401571, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: null}
> 2014-10-28 15:16:39,153 DEBUG [c.c.u.AccountManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Removed account 4
> 2014-10-28 15:16:39,287 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292948: Sending { Cmd , MgmtId: 6637838401571, via:
> 4(s-2-QA), Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/5","wait":0}}]
> }
> 2014-10-28 15:16:40,556 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-12:null) Seq 4-4574812796478292948: Processing: { Ans:
> , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
> 2014-10-28 15:16:40,556 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292948: Received: { Ans: , MgmtId: 6637838401571, via: 4,
> Ver: v1, Flags: 110, { Answer } }
> 2014-10-28 15:16:40,557 DEBUG [c.c.s.s.SnapshotManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots
> for volume: 5 under account: 4
> 2014-10-28 15:16:40,561 DEBUG [c.c.a.m.AgentAttache]
> (AgentManager-Handler-12:null) Seq 4-4574812796478292948: No more commands
> found
> 2014-10-28 15:16:40,628 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain
> for snapshot: 1
> 2014-10-28 15:16:40,637 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 1 doesn't
> have children, so it's ok to delete it and its parents
> 2014-10-28 15:16:40,713 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292949: Sending { Cmd , MgmtId: 6637838401571, via:
> 4(s-2-QA), Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/5/a2b6fc7f-c086-4494-aa89-d271f0b96d36/a2b6fc7f-c086-4494-aa89-d271f0b96d36","volume":{"uuid":"280c4f37-87ff-4b62-b1f5-5c610e90531d","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-5","size":2147483648,"path":"ROOT-5","volumeId":5,"vmName":"i-4-5-QA","accountId":4,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e55fe7db3847af9]
> i-4-5-QA/ROOT-5.vmdk\",\"[adc6929588b23db89e55fe7db3847af9]
> 0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":5,"deviceId":0,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-5-QA","name":"v1_ROOT-5_20141027213127","hypervisorType":"VMware","id":1,"quiescevm":false,"physicalSize":0}},"wait":0}}]
> }
> 2014-10-28 15:16:40,768 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-17:null) Seq 4-4574812796478292949: Processing: { Ans:
> , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory
> a2b6fc7f-c086-4494-aa89-d271f0b96d36 doesn't exist","wait":0}}] }
> 2014-10-28 15:16:40,768 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292949: Received: { Ans: , MgmtId: 6637838401571, via: 4,
> Ver: v1, Flags: 10, { Answer } }
> 2014-10-28 15:16:40,874 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292950: Sending { Cmd , MgmtId: 6637838401571, via:
> 4(s-2-QA), Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/7","wait":0}}]
> }
> 2014-10-28 15:16:42,048 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-6:ctx-452a9eca) ===START=== 10.252.193.8 -- GET
> command=queryAsyncJobResult&jobId=2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=1414470670595
> 2014-10-28 15:16:42,100 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-6:ctx-452a9eca ctx-9e36e958) ===END=== 10.252.193.8 -- GET
> command=queryAsyncJobResult&jobId=2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=1414470670595
> 2014-10-28 15:16:42,267 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-15:null) Seq 4-4574812796478292950: Processing: { Ans:
> , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
> 2014-10-28 15:16:42,268 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292950: Received: { Ans: , MgmtId: 6637838401571, via: 4,
> Ver: v1, Flags: 110, { Answer } }
> 2014-10-28 15:16:42,268 DEBUG [c.c.s.s.SnapshotManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots
> for volume: 7 under account: 4
> 2014-10-28 15:16:42,271 DEBUG [c.c.a.m.AgentAttache]
> (AgentManager-Handler-15:null) Seq 4-4574812796478292950: No more commands
> found
> 2014-10-28 15:16:42,311 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain
> for snapshot: 2
> 2014-10-28 15:16:42,314 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 2 doesn't
> have children, so it's ok to delete it and its parents
> 2014-10-28 15:16:42,347 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292951: Sending { Cmd , MgmtId: 6637838401571, via:
> 4(s-2-QA), Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/7/ecb858aa-1656-4594-996a-869f1695bce2/ecb858aa-1656-4594-996a-869f1695bce2","volume":{"uuid":"5b639b64-a539-4651-ac4e-f06b69f37d97","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-7","size":2147483648,"path":"ROOT-7","volumeId":7,"vmName":"i-4-7-QA","accountId":4,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e55fe7db3847af9]
> i-4-7-QA/ROOT-7.vmdk\",\"[adc6929588b23db89e55fe7db3847af9]
> 0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":7,"deviceId":0,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-7-QA","name":"v2_ROOT-7_20141027213212","hypervisorType":"VMware","id":2,"quiescevm":false,"physicalSize":0}},"wait":0}}]
> }
> 2014-10-28 15:16:42,359 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-16:null) Seq 4-4574812796478292951: Processing: { Ans:
> , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory
> ecb858aa-1656-4594-996a-869f1695bce2 doesn't exist","wait":0}}] }
> 2014-10-28 15:16:42,360 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292951: Received: { Ans: , MgmtId: 6637838401571, via: 4,
> Ver: v1, Flags: 10, { Answer } }
> 2014-10-28 15:16:42,434 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292952: Sending { Cmd , MgmtId: 6637838401571, via:
> 4(s-2-QA), Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/8","wait":0}}]
> }
> 2014-10-28 15:16:43,338 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-19:null) Seq 4-4574812796478292952: Processing: { Ans:
> , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
> 2014-10-28 15:16:43,339 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292952: Received: { Ans: , MgmtId: 6637838401571, via: 4,
> Ver: v1, Flags: 110, { Answer } }
> 2014-10-28 15:16:43,339 DEBUG [c.c.s.s.SnapshotManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots
> for volume: 8 under account: 4
> 2014-10-28 15:16:43,342 DEBUG [c.c.a.m.AgentAttache]
> (AgentManager-Handler-19:null) Seq 4-4574812796478292952: No more commands
> found
> 2014-10-28 15:16:43,354 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain
> for snapshot: 3
> 2014-10-28 15:16:43,356 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 3 doesn't
> have children, so it's ok to delete it and its parents
> 2014-10-28 15:16:43,389 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292953: Sending { Cmd , MgmtId: 6637838401571, via:
> 4(s-2-QA), Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/8/657a763c-4fc6-49e0-8e7b-cb1ad2471a41/657a763c-4fc6-49e0-8e7b-cb1ad2471a41","volume":{"uuid":"1aebff64-f4c6-45bb-add1-249d54eca270","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-8","size":2147483648,"path":"ROOT-8","volumeId":8,"vmName":"i-4-8-QA","accountId":4,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e55fe7db3847af9]
> i-4-8-QA/ROOT-8.vmdk\",\"[adc6929588b23db89e55fe7db3847af9]
> 0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":8,"deviceId":0,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-8-QA","name":"v3_ROOT-8_20141027213237","hypervisorType":"VMware","id":3,"quiescevm":false,"physicalSize":0}},"wait":0}}]
> }
> 2014-10-28 15:16:43,400 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-20:null) Seq 4-4574812796478292953: Processing: { Ans:
> , MgmtId: 6637838401571, via: 4, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory
> 657a763c-4fc6-49e0-8e7b-cb1ad2471a41 doesn't exist","wait":0}}] }
> 2014-10-28 15:16:43,400 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292953: Received: { Ans: , MgmtId: 6637838401571, via: 4,
> Ver: v1, Flags: 10, { Answer } }
> 2014-10-28 15:16:43,483 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292954: Sending { Cmd , MgmtId: 6637838401571, via:
> 4(s-2-QA), Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/9","wait":0}}]
> }
> 2014-10-28 15:16:43,758 DEBUG [c.c.s.StatsCollector]
> (StatsCollector-4:ctx-6ecc0a29) AutoScaling Monitor is running...
> 2014-10-28 15:16:44,555 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:null)
> Seq 4-4574812796478292954: Processing: { Ans: , MgmtId: 6637838401571, via:
> 4, Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
> 2014-10-28 15:16:44,556 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292954: Received: { Ans: , MgmtId: 6637838401571, via: 4,
> Ver: v1, Flags: 110, { Answer } }
> 2014-10-28 15:16:44,556 DEBUG [c.c.s.s.SnapshotManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots
> for volume: 9 under account: 4
> 2014-10-28 15:16:44,559 DEBUG [c.c.a.m.AgentAttache]
> (AgentManager-Handler-2:null) Seq 4-4574812796478292954: No more commands
> found
> 2014-10-28 15:16:44,579 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain
> for snapshot: 4
> 2014-10-28 15:16:44,582 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 4 doesn't
> have children, so it's ok to delete it and its parents
> 2014-10-28 15:16:44,616 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292955: Sending { Cmd , MgmtId: 6637838401571, via:
> 4(s-2-QA), Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/9/e7d15b83-06be-4596-9572-09478415e776/e7d15b83-06be-4596-9572-09478415e776","volume":{"uuid":"e88d9c15-e81b-44fd-99be-4d14f3d19c06","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-9","size":2147483648,"path":"ROOT-9","volumeId":9,"vmName":"i-4-9-QA","accountId":4,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e55fe7db3847af9]
> i-4-9-QA/ROOT-9.vmdk\",\"[adc6929588b23db89e55fe7db3847af9]
> 0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":9,"deviceId":0,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-9-QA","name":"v4_ROOT-9_20141027213330","hypervisorType":"VMware","id":4,"quiescevm":false,"physicalSize":0}},"wait":0}}]
> }
> 2014-10-28 15:16:44,628 DEBUG [c.c.a.t.Request] (AgentManager-Handler-1:null)
> Seq 4-4574812796478292955: Processing: { Ans: , MgmtId: 6637838401571, via:
> 4, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory
> e7d15b83-06be-4596-9572-09478415e776 doesn't exist","wait":0}}] }
> 2014-10-28 15:16:44,628 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292955: Received: { Ans: , MgmtId: 6637838401571, via: 4,
> Ver: v1, Flags: 10, { Answer } }
> 2014-10-28 15:16:44,692 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292956: Sending { Cmd , MgmtId: 6637838401571, via:
> 4(s-2-QA), Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/10","wait":0}}]
> }
> 2014-10-28 15:16:44,805 DEBUG [c.c.a.t.Request] (AgentManager-Handler-3:null)
> Seq 4-4574812796478292956: Processing: { Ans: , MgmtId: 6637838401571, via:
> 4, Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
> 2014-10-28 15:16:44,805 DEBUG [c.c.a.m.AgentAttache]
> (AgentManager-Handler-3:null) Seq 4-4574812796478292956: No more commands
> found
> 2014-10-28 15:16:44,805 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292956: Received: { Ans: , MgmtId: 6637838401571, via: 4,
> Ver: v1, Flags: 110, { Answer } }
> 2014-10-28 15:16:44,805 DEBUG [c.c.s.s.SnapshotManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots
> for volume: 10 under account: 4
> 2014-10-28 15:16:44,820 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain
> for snapshot: 5
> 2014-10-28 15:16:44,822 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 5 doesn't
> have children, so it's ok to delete it and its parents
> 2014-10-28 15:16:44,849 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292957: Sending { Cmd , MgmtId: 6637838401571, via:
> 4(s-2-QA), Ver: v1, Flags: 100011,
> [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/10/1c4b5c4a-b14e-4e42-9a0e-698215db758a/1c4b5c4a-b14e-4e42-9a0e-698215db758a","volume":{"uuid":"cb40dae3-6c47-40b1-8f07-f7ef6d095995","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"Data-10","size":5368709120,"path":"9bd9c800f78545db87fc805e9f3fec1d","volumeId":10,"accountId":4,"format":"OVA","provisioningType":"THIN","id":10,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"name":"detached_Data-10_20141027214535","hypervisorType":"VMware","id":5,"quiescevm":false,"physicalSize":0}},"wait":0}}]
> }
> 2014-10-28 15:16:44,860 DEBUG [c.c.a.t.Request] (AgentManager-Handler-5:null)
> Seq 4-4574812796478292957: Processing: { Ans: , MgmtId: 6637838401571, via:
> 4, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory
> 1c4b5c4a-b14e-4e42-9a0e-698215db758a doesn't exist","wait":0}}] }
> 2014-10-28 15:16:44,860 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292957: Received: { Ans: , MgmtId: 6637838401571, via: 4,
> Ver: v1, Flags: 10, { Answer } }
> 2014-10-28 15:16:44,928 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292958: Sending { Cmd , MgmtId: 6637838401571, via:
> 4(s-2-QA), Ver: v1, Flags: 100111,
> [{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/11","wait":0}}]
> }
> 2014-10-28 15:16:44,975 DEBUG [c.c.a.t.Request] (AgentManager-Handler-4:null)
> Seq 4-4574812796478292958: Processing: { Ans: , MgmtId: 6637838401571, via:
> 4, Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory
> 11 doesn't exist","wait":0}}] }
> 2014-10-28 15:16:44,975 DEBUG [c.c.a.m.AgentAttache]
> (AgentManager-Handler-4:null) Seq 4-4574812796478292958: No more commands
> found
> 2014-10-28 15:16:44,975 DEBUG [c.c.a.t.Request]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
> 4-4574812796478292958: Received: { Ans: , MgmtId: 6637838401571, via: 4,
> Ver: v1, Flags: 110, { Answer } }
> 2014-10-28 15:16:44,975 DEBUG [c.c.s.s.SnapshotManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots
> for volume: 11 under account: 4
> 2014-10-28 15:16:44,985 WARN [c.c.u.AccountManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Failed to cleanup
> account Acct[83f26103-9d83-4e4e-8bcc-c282c2acb498-acc] due to
> com.cloud.exception.InvalidParameterValueException: Can't delete snapshotshot
> 6 due to it is in Allocated Status
> at
> org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.deleteSnapshot(XenserverSnapshotStrategy.java:213)
> at
> com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshotDirsForAccount(SnapshotManagerImpl.java:623)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy180.deleteSnapshotDirsForAccount(Unknown Source)
> at
> com.cloud.user.AccountManagerImpl.cleanupAccount(AccountManagerImpl.java:706)
> at
> com.cloud.user.AccountManagerImpl.deleteAccount(AccountManagerImpl.java:665)
> at
> com.cloud.user.AccountManagerImpl.deleteUserAccount(AccountManagerImpl.java:1444)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy103.deleteUserAccount(Unknown Source)
> at
> org.apache.cloudstack.region.RegionManagerImpl.deleteUserAccount(RegionManagerImpl.java:187)
> at
> org.apache.cloudstack.region.RegionServiceImpl.deleteUserAccount(RegionServiceImpl.java:121)
> at
> org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd.execute(DeleteAccountCmd.java:104)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:513)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:470)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2014-10-28 15:16:45,008 INFO [c.c.u.AccountManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Cleanup for account 4
> is needed.
> 2014-10-28 15:16:45,047 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-24:ctx-bcb44ca9) ===START=== 10.252.193.8 -- GET
> command=queryAsyncJobResult&jobId=2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=1414470673593
> 2014-10-28 15:16:45,082 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-24:ctx-bcb44ca9 ctx-82b3efc5) ===END=== 10.252.193.8 -- GET
> command=queryAsyncJobResult&jobId=2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=1414470673593
> 2014-10-28 15:16:45,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Complete async job-53,
> jobStatus: SUCCEEDED, resultCode: 0, result:
> org.apache.cloudstack.api.response.SuccessResponse/null/{"success":true}
> 2014-10-28 15:16:45,232 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Publish async job-53
> complete on message bus
> 2014-10-28 15:16:45,233 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Wake up jobs related
> to job- 53
> 2014-10-28 15:16:45,233 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Update db status for
> job- 53
> 2014-10-28 15:16:45,236 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Wake up jobs joined
> with job- 53 and disjoin all subjobs created from job- 53
> 2014-10-28 15:16:45,241 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-30:ctx-0b9151e2 job-53) Done executing
> org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd for job-53
> 2014-10-28 15:16:45,248 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-30:ctx-0b9151e2 job-53) Remove job-53 from job monitoring
> Even the account cleanup thread could not clean any of the remained artifacts
> like vms,networks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
