Chandan Purushothama created CLOUDSTACK-5730:
------------------------------------------------
Summary: DirectAgentAttache's send command fails due to
ArrayIndexOutOfBoundsException; Unable to cleanup hypervisor's datastructures
as part of Delete Account Job
Key: CLOUDSTACK-5730
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5730
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.3.0
Reporter: Chandan Purushothama
Priority: Critical
Fix For: 4.3.0
=======================================================
ArrayIndexOutOfBoundsException during DeleteAccount Job Execution:
=======================================================
2014-01-03 07:39:16,752 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Destroying vm
VM[User|QA-88cfa2c8-e8ef-4133-8cf1-002ea3cb0c7b]
2014-01-03 07:39:16,752 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Cleaning up NICS
2014-01-03 07:39:16,752 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Cleaning network for vm: 44
2014-01-03 07:39:16,763 DEBUG [c.c.n.NetworkModelImpl]
(Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Service SecurityGroup is not
supported in the network id=229
2014-01-03 07:39:16,780 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Removed nic id=81
2014-01-03 07:39:16,782 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Revoving nic secondary ip entry ...
2014-01-03 07:39:16,783 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Cleaning up hypervisor data
structures (ex. SRs in XenServer) for managed storage
2014-01-03 07:39:16,796 DEBUG [c.c.a.ApiServlet] (catalina-exec-5:ctx-f044cb4e)
===START=== 10.147.38.149 -- GET
signature=shTK1s%2FnQv84C1gwQYEUo%2F5WFW4%3D&apiKey=SyITk_W_BpbLs73aYJqNRBu9Sir1d7k6hqoKVUmFCMSVG-vshSHHKOdj_nVVTOQHqyvfR3ZSXhDzL2QUquSdJA&command=queryAsyncJobResult&response=json&jobid=fb908e1c-cdf8-4ae9-8c0d-09116061c3f7
2014-01-03 07:39:16,800 DEBUG [c.c.a.t.Request] (Job-Executor-72:ctx-d82a5609
ctx-1472cd68) Seq 1-678298558: Sending { Cmd , MgmtId: 6631563722783, via:
1(Rack1Pod1Host15), Ver: v1, Flags: 100011, [] }
2014-01-03 07:39:16,801 DEBUG [c.c.a.t.Request] (Job-Executor-72:ctx-d82a5609
ctx-1472cd68) Seq 1-678298558: Executing: { Cmd , MgmtId: 6631563722783, via:
1(Rack1Pod1Host15), Ver: v1, Flags: 100011, [] }
2014-01-03 07:39:16,801 WARN [c.c.a.m.AgentAttache]
(Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Seq 1-678298558: Unable to send due
to
java.lang.ArrayIndexOutOfBoundsException: 0
at
com.cloud.agent.manager.DirectAgentAttache.send(DirectAgentAttache.java:100)
at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:374)
at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:398)
at
com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:394)
at
com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:920)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:497)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:448)
at com.cloud.vm.UserVmManagerImpl.expunge(UserVmManagerImpl.java:1704)
at sun.reflect.GeneratedMethodAccessor429.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy169.expunge(Unknown Source)
at
com.cloud.user.AccountManagerImpl.cleanupAccount(AccountManagerImpl.java:629)
at
com.cloud.user.AccountManagerImpl.deleteAccount(AccountManagerImpl.java:561)
at
com.cloud.user.AccountManagerImpl.deleteUserAccount(AccountManagerImpl.java:1308)
at sun.reflect.GeneratedMethodAccessor478.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy82.deleteUserAccount(Unknown Source)
at
org.apache.cloudstack.region.RegionManagerImpl.deleteUserAccount(RegionManagerImpl.java:193)
at
org.apache.cloudstack.region.RegionServiceImpl.deleteUserAccount(RegionServiceImpl.java:118)
at
org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd.execute(DeleteAccountCmd.java:101)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
at
com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
at
com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:522)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2014-01-03 07:39:16,836 DEBUG [c.c.a.m.AgentAttache]
(Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Seq 1-678298558: Cancelling.
2014-01-03 07:39:16,837 WARN [c.c.v.UserVmManagerImpl]
(Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Unable to expunge
VM[User|QA-88cfa2c8-e8ef-4133-8cf1-002ea3cb0c7b]
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
unreachable: Host 1: Problem due to other exception 0
at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:389)
at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:398)
at
com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:394)
at
com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:920)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:497)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceExpunge(VirtualMachineManagerImpl.java:448)
at com.cloud.vm.UserVmManagerImpl.expunge(UserVmManagerImpl.java:1704)
at sun.reflect.GeneratedMethodAccessor429.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy169.expunge(Unknown Source)
at
com.cloud.user.AccountManagerImpl.cleanupAccount(AccountManagerImpl.java:629)
at
com.cloud.user.AccountManagerImpl.deleteAccount(AccountManagerImpl.java:561)
at
com.cloud.user.AccountManagerImpl.deleteUserAccount(AccountManagerImpl.java:1308)
at sun.reflect.GeneratedMethodAccessor478.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy82.deleteUserAccount(Unknown Source)
at
org.apache.cloudstack.region.RegionManagerImpl.deleteUserAccount(RegionManagerImpl.java:193)
at
org.apache.cloudstack.region.RegionServiceImpl.deleteUserAccount(RegionServiceImpl.java:118)
at
org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd.execute(DeleteAccountCmd.java:101)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
at
com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
at
com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:522)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2014-01-03 07:39:16,839 ERROR [c.c.u.AccountManagerImpl]
(Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Unable to expunge vm: 44
2014-01-03 07:39:16,869 INFO [c.c.u.AccountManagerImpl]
(Job-Executor-72:ctx-d82a5609 ctx-1472cd68) deleteAccount: Deleted 1 network
groups for account 26
2014-01-03 07:39:16,870 INFO [c.c.u.AccountManagerImpl]
(Job-Executor-72:ctx-d82a5609 ctx-1472cd68) deleteAccount: Deleted 0 affinity
groups for account 26
2014-01-03 07:39:16,871 DEBUG [c.c.u.AccountManagerImpl]
(Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Deleting networks for account 26
2014-01-03 07:39:16,884 DEBUG [c.c.a.ApiServlet] (catalina-exec-5:ctx-f044cb4e
ctx-e5d3f936 ctx-efc2b479) ===END=== 10.147.38.149 -- GET
signature=shTK1s%2FnQv84C1gwQYEUo%2F5WFW4%3D&apiKey=SyITk_W_BpbLs73aYJqNRBu9Sir1d7k6hqoKVUmFCMSVG-vshSHHKOdj_nVVTOQHqyvfR3ZSXhDzL2QUquSdJA&command=queryAsyncJobResult&response=json&jobid=fb908e1c-cdf8-4ae9-8c0d-09116061c3f7
2014-01-03 07:39:16,894 WARN [o.a.c.e.o.NetworkOrchestrator]
(Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Can't delete the network, not all
user vms are expunged. Vm VM[User|QA-03d9e9fa-081a-4175-9bf2-2508034b6e17] is
in Expunging state
2014-01-03 07:39:16,894 WARN [c.c.u.AccountManagerImpl]
(Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Unable to destroy network
Ntwk[229|Guest|8] as a part of account id=26 cleanup.
2014-01-03 07:39:16,895 DEBUG [c.c.u.AccountManagerImpl]
(Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Deleting vpcs for account 26
2014-01-03 07:39:16,897 DEBUG [c.c.u.AccountManagerImpl]
(Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Deleting site-to-site VPN customer
gateways for account 26
2014-01-03 07:39:16,908 INFO [c.c.u.AccountManagerImpl]
(Job-Executor-72:ctx-d82a5609 ctx-1472cd68) deleteAccount: Released 0 dedicated
guest vlan ranges from account 26
2014-01-03 07:39:17,034 INFO [c.c.u.AccountManagerImpl]
(Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Cleanup for account 26 is needed.
2014-01-03 07:39:17,048 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-72:ctx-d82a5609 ctx-1472cd68) Complete async job-359, jobStatus:
SUCCEEDED, resultCode: 0, result:
org.apache.cloudstack.api.response.SuccessResponse/null/{"success":true}
2014-01-03 07:39:17,056 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Job-Executor-72:ctx-d82a5609) Done executing
org.apache.cloudstack.api.command.admin.account.DeleteAccountCmd for job-359
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
