Thomas created CLOUDSTACK-9286:
----------------------------------
Summary: Delete Domain not working: Failed to clean up domain
resources and sub domains, delete failed on domain
Key: CLOUDSTACK-9286
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-9286
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Automation
Affects Versions: 4.7.0
Environment: Two CentOS7 MGMT Servers
XenServer Hypervisors
Reporter: Thomas
If you disable an account of a domain and then try to delete that domain it
fails with following error:
--
2016-02-15 16:22:04,921 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-25:ctx-da3cbbe6 job-1635) (logid:7354b28c) Executing
AsyncJobVO {id:1635, userId: 18, accountId: 2, instanceType: None, instanceId:
null, cmd: org.apache.cloudstack.api.command.admin.domain.DeleteDomainCmd,
cmdInfo:
{"cleanup":"true","response":"json","ctxUserId":"18","httpmethod":"GET","ctxStartEventId":"4107","id":"dd40f585-73d6-4710-a4b7-7dd74e910675","ctxDetails":"{\"interface
com.cloud.domain.Domain\":\"dd40f585-73d6-4710-a4b7-7dd74e910675\"}","ctxAccountId":"2","uuid":"dd40f585-73d6-4710-a4b7-7dd74e910675","cmdEventType":"DOMAIN.DELETE","_":"1455549724913"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 22095312942840, completeMsid: null, lastUpdated: null,
lastPolled: null, created: null}
2016-02-15 16:22:04,921 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(catalina-exec-11:ctx-5ea86460 ctx-69c37a64) (logid:ee5245e3) submit async
job-1635, details: AsyncJobVO {id:1635, userId: 18, accountId: 2, instanceType:
None, instanceId: null, cmd:
org.apache.cloudstack.api.command.admin.domain.DeleteDomainCmd, cmdInfo:
{"cleanup":"true","response":"json","ctxUserId":"18","httpmethod":"GET","ctxStartEventId":"4107","id":"dd40f585-73d6-4710-a4b7-7dd74e910675","ctxDetails":"{\"interface
com.cloud.domain.Domain\":\"dd40f585-73d6-4710-a4b7-7dd74e910675\"}","ctxAccountId":"2","uuid":"dd40f585-73d6-4710-a4b7-7dd74e910675","cmdEventType":"DOMAIN.DELETE","_":"1455549724913"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 22095312942840, completeMsid: null, lastUpdated: null,
lastPolled: null, created: null}
2016-02-15 16:22:04,922 DEBUG [c.c.a.ApiServlet] (catalina-exec-11:ctx-5ea86460
ctx-69c37a64) (logid:ee5245e3) ===END=== 212.123.98.2 -- GET
command=deleteDomain&id=dd40f585-73d6-4710-a4b7-7dd74e910675&cleanup=true&response=json&_=1455549724913
2016-02-15 16:22:04,937 DEBUG [c.c.u.AccountManagerImpl]
(API-Job-Executor-25:ctx-da3cbbe6 job-1635 ctx-633df0cf) (logid:7354b28c)
Access granted to Acct[c1022585-b874-11e5-b4fa-1418774e8f7a-admin] to
Domain:97/xxxxx/xxxxxxxxxxx/ by AffinityGroupAccessChecker
2016-02-15 16:22:04,937 DEBUG [c.c.u.DomainManagerImpl]
(API-Job-Executor-25:ctx-da3cbbe6 job-1635 ctx-633df0cf) (logid:7354b28c)
Marking domain id=97 as Inactive before actually deleting it
2016-02-15 16:22:04,940 DEBUG [c.c.u.DomainManagerImpl]
(API-Job-Executor-25:ctx-da3cbbe6 job-1635 ctx-633df0cf) (logid:7354b28c)
Cleaning up domain id=97
2016-02-15 16:22:04,949 DEBUG [c.c.u.DomainManagerImpl]
(API-Job-Executor-25:ctx-da3cbbe6 job-1635 ctx-633df0cf) (logid:7354b28c)
Deleting networks for domain id=97
2016-02-15 16:22:04,952 DEBUG [c.c.u.DomainManagerImpl]
(API-Job-Executor-25:ctx-da3cbbe6 job-1635 ctx-633df0cf) (logid:7354b28c) Can't
delete the domain yet because it has 1accounts that need a cleanup
2016-02-15 16:22:04,952 ERROR [c.c.u.DomainManagerImpl]
(API-Job-Executor-25:ctx-da3cbbe6 job-1635 ctx-633df0cf) (logid:7354b28c)
Exception deleting domain with id 97
com.cloud.utils.exception.CloudRuntimeException: Failed to clean up domain
resources and sub domains, delete failed on domain 5002000011 (id: 97).
at
com.cloud.user.DomainManagerImpl.deleteDomain(DomainManagerImpl.java:268)
at
com.cloud.user.DomainManagerImpl.deleteDomain(DomainManagerImpl.java:251)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy128.deleteDomain(Unknown Source)
at
org.apache.cloudstack.region.RegionManagerImpl.deleteDomain(RegionManagerImpl.java:240)
at
org.apache.cloudstack.region.RegionServiceImpl.deleteDomain(RegionServiceImpl.java:167)
at
org.apache.cloudstack.api.command.admin.domain.DeleteDomainCmd.execute(DeleteDomainCmd.java:103)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-02-15 16:22:04,953 DEBUG [c.c.u.DomainManagerImpl]
(API-Job-Executor-25:ctx-da3cbbe6 job-1635 ctx-633df0cf) (logid:7354b28c)
Changing domain id=97 state back to Active because it can't be removed due to
resources referencing to it
2016-02-15 16:22:04,962 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-25:ctx-da3cbbe6 job-1635) (logid:7354b28c) Unexpected
exception while executing
org.apache.cloudstack.api.command.admin.domain.DeleteDomainCmd
com.cloud.utils.exception.CloudRuntimeException: Failed to clean up domain
resources and sub domains, delete failed on domain 5002000011 (id: 97).
at
com.cloud.user.DomainManagerImpl.deleteDomain(DomainManagerImpl.java:268)
at
com.cloud.user.DomainManagerImpl.deleteDomain(DomainManagerImpl.java:251)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy128.deleteDomain(Unknown Source)
at
org.apache.cloudstack.region.RegionManagerImpl.deleteDomain(RegionManagerImpl.java:240)
at
org.apache.cloudstack.region.RegionServiceImpl.deleteDomain(RegionServiceImpl.java:167)
at
org.apache.cloudstack.api.command.admin.domain.DeleteDomainCmd.execute(DeleteDomainCmd.java:103)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-02-15 16:22:04,962 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-25:ctx-da3cbbe6 job-1635) (logid:7354b28c) Complete async
job-1635, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to clean up domain resources and sub domains, delete failed on domain
5002000011 (id: 97)."}
------
The account is not showing up in webgui anymore, so you cant enable the account
via GUI.
As workaround you can enable the account via mysql and then it will cleanup and
remove successfully:
--
update account set state='enabled' where id="77" limit 1;
--
=> Cleanup successfully!
I would expect, that cloudstack automation will do this for me or it`s cleanup
process will not rely on an enabled account.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
