[jira] (CLOUDSTACK-9764) Delete domain failure due to Account Cleanup task

[Nicolas Vazquez (JIRA)]

Title: Message Title

Nicolas Vazquez created an issue   CloudStack / CLOUDSTACK-9764 Delete domain failure due to Account Cleanup task Issue Type: Bug Affects Versions: 4.10.0.0 Assignee: Nicolas Vazquez Created: 30/Jan/17 19:20 Fix Versions: 4.10.0.0 Priority: Major Reporter: Nicolas Vazquez Security Level: Public (Anyone can view this level - this is the default.) It was noticed in production environments that deleteDomain task failed for domains with multiple accounts and resources. Examining logs it was found out that if Account Cleanup Task got executed after domain (and all of its subchilds) got marked as Inactive; and before delete domain task finishes, it produces a failure. AccountCleanupTask gets executed every account.cleanup.interval seconds looking for: Removed accounts Disabled accounts Inactive domains As deleteDomain marks domain to delete (and its subchilds) as Inactive before deleting them, when AccountCleanupTask is executed, it removes marked domains Example account.cleanup.interval = 100 2017-01-26 06:07:03,621 DEBUG [cloud.api.ApiServlet] (catalina-exec-8:ctx-50cfa3b6 ctx-92ad5b38) ===END=== 10.39.251.17 -- GET command=deleteDomain&id=1910a3dc-6fa6-457b-ab3a-602b0cfb6686&cleanup=true&response=json&_=1485439623475 ... // Domain and its subchilds marked as Inactive 2017-01-26 06:07:03,640 DEBUG [cloud.user.DomainManagerImpl] (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Marking domain id=27 as Inactive before actually deleting it 2017-01-26 06:07:03,646 DEBUG [cloud.user.DomainManagerImpl] (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain id=27 2017-01-26 06:07:03,670 DEBUG [cloud.user.DomainManagerImpl] (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain id=28 2017-01-26 06:07:03,685 DEBUG [cloud.user.DomainManagerImpl] (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Cleaning up domain id=29 ... //AccountCleanupTask removes Inactive domains 2017-01-26 06:07:44,285 INFO [cloud.user.AccountManagerImpl] (AccountChecker-1:ctx-b8a01824) Found 0 removed accounts to cleanup 2017-01-26 06:07:44,287 INFO [cloud.user.AccountManagerImpl] (AccountChecker-1:ctx-b8a01824) Found 0 disabled accounts to cleanup 2017-01-26 06:07:44,289 INFO [cloud.user.AccountManagerImpl] (AccountChecker-1:ctx-b8a01824) Found 3 inactive domains to cleanup 2017-01-26 06:07:44,292 DEBUG [cloud.user.AccountManagerImpl] (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=27 2017-01-26 06:07:44,297 DEBUG [db.Transaction.Transaction] (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name = AccountChecker-1; called by -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62 2017-01-26 06:07:44,301 DEBUG [cloud.user.AccountManagerImpl] (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=28 2017-01-26 06:07:44,304 DEBUG [db.Transaction.Transaction] (AccountChecker-1:ctx-b8a01824) Rolling back the transaction: Time = 2 Name = AccountChecker-1; called by -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172-JdkDynamicAopProxy.invoke:204-$Proxy63.remove:-1-DomainManagerImpl.removeDomain:248-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62 2017-01-26 06:07:44,307 DEBUG [cloud.user.AccountManagerImpl] (AccountChecker-1:ctx-b8a01824) Removing inactive domain id=29 2017-01-26 06:07:44,319 INFO [cloud.user.AccountManagerImpl] (AccountChecker-1:ctx-b8a01824) Found 0 disabled projects to cleanup ... // Failure due to domain is already removed 2017-01-26 06:07:46,369 WARN [cloud.user.AccountManagerImpl] (API-Job-Executor-29:ctx-23415942 job-7165 ctx-fe3d13d6) Failed to cleanup account Acct[6a6e63ad-d89b-4a53-b3ae-1c06ea3d1899-ac2] due to com.cloud.exception.InvalidParameterValueException: Please specify a valid domain ID. at com.cloud.resourcelimit.ResourceLimitManagerImpl.recalculateResourceCount(ResourceLimitManagerImpl.java:752) at com.cloud.vm.UserVmManagerImpl.expunge(UserVmManagerImpl.java:2053) ... } Add Comment   This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)