Sanjeev N created CLOUDSTACK-7799:
-------------------------------------
Summary: [Critical][Accounts] Account deletion failed to remove
all the entities related to it in case of failure in deleting one entity
Key: CLOUDSTACK-7799
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7799
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.5.0
Environment: Latest build from 4.5 with commit [root@BPKxDmS ~]#
cloudstack-sccs
385c4f673dfbd1fd326e539625e2c06db4cdc27d
Reporter: Sanjeev N
Priority: Critical
Fix For: 4.5.0
[Critical][Accounts] Account deletion failed to remove all the entities related
to it in case of failure in deleting one entity
Steps to Reproduce:
================
1.Bring up CS in advanced zone with one vmware cluster
2.Create a user account
3.With the user account deploy few (4-5) vms
4.Take snapshots on all of the vms root disks
5.Deploy another vm and simulate snapshot failure operation(In my case I tried
snapshot operation with quiesce option set to true) so that snapshot will be in
"Allocated" state
6.Delete this user account
Expected Behavior:
==============
Whatever may the state of the snapshot account deletion shall clean all the
objects related that account
Actual Behavior:
=============
Account deletion started deleting snapshots. Since one of the snapshots is in
Allocated state it failed to delete that snapshot and the Account clean up job
ended there. So vms and networks related to the accounts are still there but
the account was deleted.
Here is the log snippet:
=================
2014-10-28 15:16:38,822 DEBUG [c.c.a.ApiServlet] (catalina-exec-5:ctx-1c16bc72)
===START=== 10.252.193.8 -- GET
command=deleteAccount&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&id=83f26103-9d83-4e4e-8bcc-c282c2acb498&_=1414470667367
2014-10-28 15:16:38,974 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(catalina-exec-5:ctx-1c16bc72 ctx-51140d28) submit async job-53, details:
AsyncJobVO {id:53, userId: 2, accountId: 2, instanceType: Account, instanceId:
null, cmd: org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd,
cmdInfo:
{"response":"json","id":"83f26103-9d83-4e4e-8bcc-c282c2acb498","sessionkey":"EpI1EuP5ZFF0VKispQyNb9AxWF4\u003d","ctxDetails":"{\"com.cloud.user.Account\":\"83f26103-9d83-4e4e-8bcc-c282c2acb498\"}","cmdEventType":"ACCOUNT.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1414470667367","uuid":"83f26103-9d83-4e4e-8bcc-c282c2acb498","ctxAccountId":"2","ctxStartEventId":"118"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 6637838401571, completeMsid: null, lastUpdated: null,
lastPolled: null, created: null}
2014-10-28 15:16:38,978 DEBUG [c.c.a.ApiServlet] (catalina-exec-5:ctx-1c16bc72
ctx-51140d28) ===END=== 10.252.193.8 -- GET
command=deleteAccount&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&id=83f26103-9d83-4e4e-8bcc-c282c2acb498&_=1414470667367
2014-10-28 15:16:38,996 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-30:ctx-0b9151e2 job-53) Add job-53 into job monitoring
2014-10-28 15:16:38,996 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-30:ctx-0b9151e2 job-53) Executing AsyncJobVO {id:53, userId:
2, accountId: 2, instanceType: Account, instanceId: null, cmd:
org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd, cmdInfo:
{"response":"json","id":"83f26103-9d83-4e4e-8bcc-c282c2acb498","sessionkey":"EpI1EuP5ZFF0VKispQyNb9AxWF4\u003d","ctxDetails":"{\"com.cloud.user.Account\":\"83f26103-9d83-4e4e-8bcc-c282c2acb498\"}","cmdEventType":"ACCOUNT.DELETE","ctxUserId":"2","httpmethod":"GET","_":"1414470667367","uuid":"83f26103-9d83-4e4e-8bcc-c282c2acb498","ctxAccountId":"2","ctxStartEventId":"118"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 6637838401571, completeMsid: null, lastUpdated: null,
lastPolled: null, created: null}
2014-10-28 15:16:39,153 DEBUG [c.c.u.AccountManagerImpl]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Removed account 4
2014-10-28 15:16:39,287 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
4-4574812796478292948: Sending { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA),
Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/5","wait":0}}]
}
2014-10-28 15:16:40,556 DEBUG [c.c.a.t.Request] (AgentManager-Handler-12:null)
Seq 4-4574812796478292948: Processing: { Ans: , MgmtId: 6637838401571, via: 4,
Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2014-10-28 15:16:40,556 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
4-4574812796478292948: Received: { Ans: , MgmtId: 6637838401571, via: 4, Ver:
v1, Flags: 110, { Answer } }
2014-10-28 15:16:40,557 DEBUG [c.c.s.s.SnapshotManagerImpl]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots
for volume: 5 under account: 4
2014-10-28 15:16:40,561 DEBUG [c.c.a.m.AgentAttache]
(AgentManager-Handler-12:null) Seq 4-4574812796478292948: No more commands found
2014-10-28 15:16:40,628 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain
for snapshot: 1
2014-10-28 15:16:40,637 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 1 doesn't have
children, so it's ok to delete it and its parents
2014-10-28 15:16:40,713 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
4-4574812796478292949: Sending { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA),
Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/5/a2b6fc7f-c086-4494-aa89-d271f0b96d36/a2b6fc7f-c086-4494-aa89-d271f0b96d36","volume":{"uuid":"280c4f37-87ff-4b62-b1f5-5c610e90531d","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-5","size":2147483648,"path":"ROOT-5","volumeId":5,"vmName":"i-4-5-QA","accountId":4,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e55fe7db3847af9]
i-4-5-QA/ROOT-5.vmdk\",\"[adc6929588b23db89e55fe7db3847af9]
0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":5,"deviceId":0,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-5-QA","name":"v1_ROOT-5_20141027213127","hypervisorType":"VMware","id":1,"quiescevm":false,"physicalSize":0}},"wait":0}}]
}
2014-10-28 15:16:40,768 DEBUG [c.c.a.t.Request] (AgentManager-Handler-17:null)
Seq 4-4574812796478292949: Processing: { Ans: , MgmtId: 6637838401571, via: 4,
Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory
a2b6fc7f-c086-4494-aa89-d271f0b96d36 doesn't exist","wait":0}}] }
2014-10-28 15:16:40,768 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
4-4574812796478292949: Received: { Ans: , MgmtId: 6637838401571, via: 4, Ver:
v1, Flags: 10, { Answer } }
2014-10-28 15:16:40,874 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
4-4574812796478292950: Sending { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA),
Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/7","wait":0}}]
}
2014-10-28 15:16:42,048 DEBUG [c.c.a.ApiServlet] (catalina-exec-6:ctx-452a9eca)
===START=== 10.252.193.8 -- GET
command=queryAsyncJobResult&jobId=2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=1414470670595
2014-10-28 15:16:42,100 DEBUG [c.c.a.ApiServlet] (catalina-exec-6:ctx-452a9eca
ctx-9e36e958) ===END=== 10.252.193.8 -- GET
command=queryAsyncJobResult&jobId=2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=1414470670595
2014-10-28 15:16:42,267 DEBUG [c.c.a.t.Request] (AgentManager-Handler-15:null)
Seq 4-4574812796478292950: Processing: { Ans: , MgmtId: 6637838401571, via: 4,
Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2014-10-28 15:16:42,268 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
4-4574812796478292950: Received: { Ans: , MgmtId: 6637838401571, via: 4, Ver:
v1, Flags: 110, { Answer } }
2014-10-28 15:16:42,268 DEBUG [c.c.s.s.SnapshotManagerImpl]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots
for volume: 7 under account: 4
2014-10-28 15:16:42,271 DEBUG [c.c.a.m.AgentAttache]
(AgentManager-Handler-15:null) Seq 4-4574812796478292950: No more commands found
2014-10-28 15:16:42,311 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain
for snapshot: 2
2014-10-28 15:16:42,314 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 2 doesn't have
children, so it's ok to delete it and its parents
2014-10-28 15:16:42,347 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
4-4574812796478292951: Sending { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA),
Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/7/ecb858aa-1656-4594-996a-869f1695bce2/ecb858aa-1656-4594-996a-869f1695bce2","volume":{"uuid":"5b639b64-a539-4651-ac4e-f06b69f37d97","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-7","size":2147483648,"path":"ROOT-7","volumeId":7,"vmName":"i-4-7-QA","accountId":4,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e55fe7db3847af9]
i-4-7-QA/ROOT-7.vmdk\",\"[adc6929588b23db89e55fe7db3847af9]
0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":7,"deviceId":0,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-7-QA","name":"v2_ROOT-7_20141027213212","hypervisorType":"VMware","id":2,"quiescevm":false,"physicalSize":0}},"wait":0}}]
}
2014-10-28 15:16:42,359 DEBUG [c.c.a.t.Request] (AgentManager-Handler-16:null)
Seq 4-4574812796478292951: Processing: { Ans: , MgmtId: 6637838401571, via: 4,
Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory
ecb858aa-1656-4594-996a-869f1695bce2 doesn't exist","wait":0}}] }
2014-10-28 15:16:42,360 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
4-4574812796478292951: Received: { Ans: , MgmtId: 6637838401571, via: 4, Ver:
v1, Flags: 10, { Answer } }
2014-10-28 15:16:42,434 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
4-4574812796478292952: Sending { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA),
Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/8","wait":0}}]
}
2014-10-28 15:16:43,338 DEBUG [c.c.a.t.Request] (AgentManager-Handler-19:null)
Seq 4-4574812796478292952: Processing: { Ans: , MgmtId: 6637838401571, via: 4,
Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2014-10-28 15:16:43,339 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
4-4574812796478292952: Received: { Ans: , MgmtId: 6637838401571, via: 4, Ver:
v1, Flags: 110, { Answer } }
2014-10-28 15:16:43,339 DEBUG [c.c.s.s.SnapshotManagerImpl]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots
for volume: 8 under account: 4
2014-10-28 15:16:43,342 DEBUG [c.c.a.m.AgentAttache]
(AgentManager-Handler-19:null) Seq 4-4574812796478292952: No more commands found
2014-10-28 15:16:43,354 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain
for snapshot: 3
2014-10-28 15:16:43,356 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 3 doesn't have
children, so it's ok to delete it and its parents
2014-10-28 15:16:43,389 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
4-4574812796478292953: Sending { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA),
Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/8/657a763c-4fc6-49e0-8e7b-cb1ad2471a41/657a763c-4fc6-49e0-8e7b-cb1ad2471a41","volume":{"uuid":"1aebff64-f4c6-45bb-add1-249d54eca270","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-8","size":2147483648,"path":"ROOT-8","volumeId":8,"vmName":"i-4-8-QA","accountId":4,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e55fe7db3847af9]
i-4-8-QA/ROOT-8.vmdk\",\"[adc6929588b23db89e55fe7db3847af9]
0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":8,"deviceId":0,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-8-QA","name":"v3_ROOT-8_20141027213237","hypervisorType":"VMware","id":3,"quiescevm":false,"physicalSize":0}},"wait":0}}]
}
2014-10-28 15:16:43,400 DEBUG [c.c.a.t.Request] (AgentManager-Handler-20:null)
Seq 4-4574812796478292953: Processing: { Ans: , MgmtId: 6637838401571, via: 4,
Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory
657a763c-4fc6-49e0-8e7b-cb1ad2471a41 doesn't exist","wait":0}}] }
2014-10-28 15:16:43,400 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
4-4574812796478292953: Received: { Ans: , MgmtId: 6637838401571, via: 4, Ver:
v1, Flags: 10, { Answer } }
2014-10-28 15:16:43,483 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
4-4574812796478292954: Sending { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA),
Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/9","wait":0}}]
}
2014-10-28 15:16:43,758 DEBUG [c.c.s.StatsCollector]
(StatsCollector-4:ctx-6ecc0a29) AutoScaling Monitor is running...
2014-10-28 15:16:44,555 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:null)
Seq 4-4574812796478292954: Processing: { Ans: , MgmtId: 6637838401571, via: 4,
Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2014-10-28 15:16:44,556 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
4-4574812796478292954: Received: { Ans: , MgmtId: 6637838401571, via: 4, Ver:
v1, Flags: 110, { Answer } }
2014-10-28 15:16:44,556 DEBUG [c.c.s.s.SnapshotManagerImpl]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots
for volume: 9 under account: 4
2014-10-28 15:16:44,559 DEBUG [c.c.a.m.AgentAttache]
(AgentManager-Handler-2:null) Seq 4-4574812796478292954: No more commands found
2014-10-28 15:16:44,579 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain
for snapshot: 4
2014-10-28 15:16:44,582 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 4 doesn't have
children, so it's ok to delete it and its parents
2014-10-28 15:16:44,616 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
4-4574812796478292955: Sending { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA),
Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/9/e7d15b83-06be-4596-9572-09478415e776/e7d15b83-06be-4596-9572-09478415e776","volume":{"uuid":"e88d9c15-e81b-44fd-99be-4d14f3d19c06","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"ROOT-9","size":2147483648,"path":"ROOT-9","volumeId":9,"vmName":"i-4-9-QA","accountId":4,"chainInfo":"{\"diskDeviceBusName\":\"ide0:1\",\"diskChain\":[\"[adc6929588b23db89e55fe7db3847af9]
i-4-9-QA/ROOT-9.vmdk\",\"[adc6929588b23db89e55fe7db3847af9]
0bc482b12d62345e9fca3cbce554086c/0bc482b12d62345e9fca3cbce554086c.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":9,"deviceId":0,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"vmName":"i-4-9-QA","name":"v4_ROOT-9_20141027213330","hypervisorType":"VMware","id":4,"quiescevm":false,"physicalSize":0}},"wait":0}}]
}
2014-10-28 15:16:44,628 DEBUG [c.c.a.t.Request] (AgentManager-Handler-1:null)
Seq 4-4574812796478292955: Processing: { Ans: , MgmtId: 6637838401571, via: 4,
Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory
e7d15b83-06be-4596-9572-09478415e776 doesn't exist","wait":0}}] }
2014-10-28 15:16:44,628 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
4-4574812796478292955: Received: { Ans: , MgmtId: 6637838401571, via: 4, Ver:
v1, Flags: 10, { Answer } }
2014-10-28 15:16:44,692 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
4-4574812796478292956: Sending { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA),
Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/10","wait":0}}]
}
2014-10-28 15:16:44,805 DEBUG [c.c.a.t.Request] (AgentManager-Handler-3:null)
Seq 4-4574812796478292956: Processing: { Ans: , MgmtId: 6637838401571, via: 4,
Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2014-10-28 15:16:44,805 DEBUG [c.c.a.m.AgentAttache]
(AgentManager-Handler-3:null) Seq 4-4574812796478292956: No more commands found
2014-10-28 15:16:44,805 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
4-4574812796478292956: Received: { Ans: , MgmtId: 6637838401571, via: 4, Ver:
v1, Flags: 110, { Answer } }
2014-10-28 15:16:44,805 DEBUG [c.c.s.s.SnapshotManagerImpl]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots
for volume: 10 under account: 4
2014-10-28 15:16:44,820 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) delete snapshot chain
for snapshot: 5
2014-10-28 15:16:44,822 DEBUG [o.a.c.s.s.XenserverSnapshotStrategy]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Snapshot: 5 doesn't have
children, so it's ok to delete it and its parents
2014-10-28 15:16:44,849 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
4-4574812796478292957: Sending { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA),
Ver: v1, Flags: 100011,
[{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/10/1c4b5c4a-b14e-4e42-9a0e-698215db758a/1c4b5c4a-b14e-4e42-9a0e-698215db758a","volume":{"uuid":"cb40dae3-6c47-40b1-8f07-f7ef6d095995","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"adc69295-88b2-3db8-9e55-fe7db3847af9","id":1,"poolType":"NetworkFilesystem","host":"10.147.28.7","path":"/export/home/sanjeev/pri_cluster","port":2049,"url":"NetworkFilesystem://10.147.28.7/export/home/sanjeev/pri_cluster/?ROLE=Primary&STOREUUID=adc69295-88b2-3db8-9e55-fe7db3847af9"}},"name":"Data-10","size":5368709120,"path":"9bd9c800f78545db87fc805e9f3fec1d","volumeId":10,"accountId":4,"format":"OVA","provisioningType":"THIN","id":10,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"name":"detached_Data-10_20141027214535","hypervisorType":"VMware","id":5,"quiescevm":false,"physicalSize":0}},"wait":0}}]
}
2014-10-28 15:16:44,860 DEBUG [c.c.a.t.Request] (AgentManager-Handler-5:null)
Seq 4-4574812796478292957: Processing: { Ans: , MgmtId: 6637838401571, via: 4,
Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory
1c4b5c4a-b14e-4e42-9a0e-698215db758a doesn't exist","wait":0}}] }
2014-10-28 15:16:44,860 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
4-4574812796478292957: Received: { Ans: , MgmtId: 6637838401571, via: 4, Ver:
v1, Flags: 10, { Answer } }
2014-10-28 15:16:44,928 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
4-4574812796478292958: Sending { Cmd , MgmtId: 6637838401571, via: 4(s-2-QA),
Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.DeleteSnapshotsDirCommand":{"store":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.147.28.7/export/home/sanjeev/sec_45","_role":"Image"}},"directory":"snapshots/4/11","wait":0}}]
}
2014-10-28 15:16:44,975 DEBUG [c.c.a.t.Request] (AgentManager-Handler-4:null)
Seq 4-4574812796478292958: Processing: { Ans: , MgmtId: 6637838401571, via: 4,
Ver: v1, Flags: 110,
[{"com.cloud.agent.api.Answer":{"result":true,"details":"snapshot directory 11
doesn't exist","wait":0}}] }
2014-10-28 15:16:44,975 DEBUG [c.c.a.m.AgentAttache]
(AgentManager-Handler-4:null) Seq 4-4574812796478292958: No more commands found
2014-10-28 15:16:44,975 DEBUG [c.c.a.t.Request]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Seq
4-4574812796478292958: Received: { Ans: , MgmtId: 6637838401571, via: 4, Ver:
v1, Flags: 110, { Answer } }
2014-10-28 15:16:44,975 DEBUG [c.c.s.s.SnapshotManagerImpl]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Deleted all snapshots
for volume: 11 under account: 4
2014-10-28 15:16:44,985 WARN [c.c.u.AccountManagerImpl]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Failed to cleanup
account Acct[83f26103-9d83-4e4e-8bcc-c282c2acb498-acc] due to
com.cloud.exception.InvalidParameterValueException: Can't delete snapshotshot 6
due to it is in Allocated Status
at
org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.deleteSnapshot(XenserverSnapshotStrategy.java:213)
at
com.cloud.storage.snapshot.SnapshotManagerImpl.deleteSnapshotDirsForAccount(SnapshotManagerImpl.java:623)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy180.deleteSnapshotDirsForAccount(Unknown Source)
at
com.cloud.user.AccountManagerImpl.cleanupAccount(AccountManagerImpl.java:706)
at
com.cloud.user.AccountManagerImpl.deleteAccount(AccountManagerImpl.java:665)
at
com.cloud.user.AccountManagerImpl.deleteUserAccount(AccountManagerImpl.java:1444)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy103.deleteUserAccount(Unknown Source)
at
org.apache.cloudstack.region.RegionManagerImpl.deleteUserAccount(RegionManagerImpl.java:187)
at
org.apache.cloudstack.region.RegionServiceImpl.deleteUserAccount(RegionServiceImpl.java:121)
at
org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd.execute(DeleteAccountCmd.java:104)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:513)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:470)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2014-10-28 15:16:45,008 INFO [c.c.u.AccountManagerImpl]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Cleanup for account 4 is
needed.
2014-10-28 15:16:45,047 DEBUG [c.c.a.ApiServlet]
(catalina-exec-24:ctx-bcb44ca9) ===START=== 10.252.193.8 -- GET
command=queryAsyncJobResult&jobId=2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=1414470673593
2014-10-28 15:16:45,082 DEBUG [c.c.a.ApiServlet] (catalina-exec-24:ctx-bcb44ca9
ctx-82b3efc5) ===END=== 10.252.193.8 -- GET
command=queryAsyncJobResult&jobId=2c5b0fa0-ef16-4a61-bd64-7cae6acfdaa4&response=json&sessionkey=EpI1EuP5ZFF0VKispQyNb9AxWF4%3D&_=1414470673593
2014-10-28 15:16:45,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Complete async job-53,
jobStatus: SUCCEEDED, resultCode: 0, result:
org.apache.cloudstack.api.response.SuccessResponse/null/{"success":true}
2014-10-28 15:16:45,232 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Publish async job-53
complete on message bus
2014-10-28 15:16:45,233 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Wake up jobs related to
job- 53
2014-10-28 15:16:45,233 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Update db status for
job- 53
2014-10-28 15:16:45,236 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-30:ctx-0b9151e2 job-53 ctx-3bc74f24) Wake up jobs joined with
job- 53 and disjoin all subjobs created from job- 53
2014-10-28 15:16:45,241 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-30:ctx-0b9151e2 job-53) Done executing
org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd for job-53
2014-10-28 15:16:45,248 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-30:ctx-0b9151e2 job-53) Remove job-53 from job monitoring
Even the account cleanup thread could not clean any of the remained artifacts
like vms,networks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
