Sailaja Mada created CLOUDSTACK-6684:
----------------------------------------
Summary: Error instances are not getting expunged within the
specified expunge interval
Key: CLOUDSTACK-6684
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-6684
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.4.0
Reporter: Sailaja Mada
Priority: Critical
Attachments: expungelogs.zip
Steps:
1. Configure Adv zone using Xenserver 6.2.5
2. Deployed windows 2012 R2 and window 8 instances
3. Set the expunge interval to 10 seconds
3. Windows 2012 R2 instances failed to deploy due to another bug
https://issues.apache.org/jira/browse/CLOUDSTACK-6661
These failed instances are in Error state.
Expunge thread triggered to expunge instances. It identified the instances to
expunge but failed to do with Runtime exceptions,
2014-05-15 15:23:03,552 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-14:null) SeqA 2-1697: Processing Seq 2-1697: { Cmd ,
MgmtId: -1, via: 2, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n
\"connections\": [\n {\n \"id\": 10,\n \"clientInfo\": \"\",\n
\"host\": \"10.102.192.10\",\n \"port\": -1,\n \"tag\":
\"3884c408-9b63-4f04-ab63-4626e0cbd9af\",\n \"createTime\":
1400166406368,\n \"lastUsedTime\": 1400166620142\n }\n
]\n}","wait":0}}] }
2014-05-15 15:23:03,556 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-14:null) SeqA 2-1697: Sending Seq 2-1697: { Ans: ,
MgmtId: 55355881446856, via: 2, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2014-05-15 15:23:06,528 INFO [c.c.v.UserVmManagerImpl]
(UserVm-Scavenger-1:ctx-8a506c20) Found 2 vms to expunge.
2014-05-15 15:23:06,531 WARN [c.c.v.UserVmManagerImpl]
(UserVm-Scavenger-1:ctx-8a506c20) Unable to expunge VM[User|i-3-7-VM]
com.cloud.utils.exception.CloudRuntimeException: Please destroy vm with
specified vmId before expunge
at com.cloud.vm.UserVmManagerImpl.expungeVm(UserVmManagerImpl.java:3620)
at
com.cloud.vm.UserVmManagerImpl$ExpungeTask.runInContext(UserVmManagerImpl.java:1801)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-05-15 15:23:06,534 WARN [c.c.v.UserVmManagerImpl]
(UserVm-Scavenger-1:ctx-8a506c20) Unable to expunge VM[User|i-2-10-VM]
com.cloud.utils.exception.CloudRuntimeException: Please destroy vm with
specified vmId before expunge
at com.cloud.vm.UserVmManagerImpl.expungeVm(UserVmManagerImpl.java:3620)
at
com.cloud.vm.UserVmManagerImpl$ExpungeTask.runInContext(UserVmManagerImpl.java:1801)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-05-15 15:23:08,270 DEBUG [c.c.s.StatsCollector]
(StatsCollector-3:ctx-ac43b5b4) AutoScaling Monitor is running...
2014-05-15 15:23:13,554 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-12:null) SeqA 2-1698: Processing Seq 2-1698: { Cmd ,
MgmtId: -1, via: 2, Ver: v1, Flags: 11,
[{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n
\"connections\": [\n {\n \"id\": 10,\n \"clientInfo\": \"\",\n
\"host\": \"10.102.192.10\",\n \"port\": -1,\n \"tag\":
\"3884c408-9b63-4f04-ab63-4626e0cbd9af\",\n \"createTime\":
1400166406368,\n \"lastUsedTime\": 1400166620142\n }\n
]\n}","wait":0}}] }
2014-05-15 15:23:13,587 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-12:null) SeqA 2-1698: Sending Seq 2-1698: { Ans: ,
MgmtId: 55355881446856, via: 2, Ver: v1, Flags: 100010,
[{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2014-05-15 15:23:16,539 INFO [c.c.v.UserVmManagerImpl]
(UserVm-Scavenger-1:ctx-172411bd) Found 2 vms to expunge.
2014-05-15 15:23:16,542 WARN [c.c.v.UserVmManagerImpl]
(UserVm-Scavenger-1:ctx-172411bd) Unable to expunge VM[User|i-3-7-VM]
com.cloud.utils.exception.CloudRuntimeException: Please destroy vm with
specified vmId before expunge
at com.cloud.vm.UserVmManagerImpl.expungeVm(UserVmManagerImpl.java:3620)
at
com.cloud.vm.UserVmManagerImpl$ExpungeTask.runInContext(UserVmManagerImpl.java:1801)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-05-15 15:23:16,545 WARN [c.c.v.UserVmManagerImpl]
(UserVm-Scavenger-1:ctx-172411bd) Unable to expunge VM[User|i-2-10-VM]
com.cloud.utils.exception.CloudRuntimeException: Please destroy vm with
specified vmId before expunge
at com.cloud.vm.UserVmManagerImpl.expungeVm(UserVmManagerImpl.java:3620)
at
com.cloud.vm.UserVmManagerImpl$ExpungeTask.runInContext(UserVmManagerImpl.java:1801)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-05-15 15:23:22,805 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentManager-Handler-11:null) Ping from 2
--
This message was sent by Atlassian JIRA
(v6.2#6252)
