manasaveloori created CLOUDSTACK-5792:
-----------------------------------------
Summary: All the VMs are shown as running even after the host is
put in maintenance.NPE in logs
Key: CLOUDSTACK-5792
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5792
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Hypervisor Controller
Affects Versions: 4.3.0
Environment: upgraded from 2.2.16 to 4.3
Reporter: manasaveloori
Priority: Critical
Fix For: 4.3.0
Attachments: management-server.rar
Steps:
Deploy CS 2.2 X.16 using Xen5.6 sp2 HV.
2. Add the External firewall SRX to CS.
3. Set the GC parameter firewall.rule.ui.enabled to "true."
4. Now acquire the IP and configure firewall and PF rules.
5. Upgrade the CS to 4.3.
6. Stop and start all the System VMs and router VMs so that the new template is
upgraded.
7. Now perform Network restart on which the firwall and PF rules are configured.
Here there is a issue CLOUDSTACK-5747
8.Enable maintenance mode on host.---CLOUDSTACK-5706
9. Now unmanage the cluster.
Observation:
Even after the host is in maintenance CS shows the VMs as running and
continuously seeing the following NPE in logs:
2014-01-06 21:15:40,767 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-80:ctx-531f0e5c) Seq 1-1135935494: Executing request
2014-01-06 21:15:40,934 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-80:ctx-531f0e5c) The VM is now missing marking it as Stopped
i-2-11-VM
2014-01-06 21:15:40,934 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-80:ctx-531f0e5c) The VM is now missing marking it as Stopped r-6-VM
2014-01-06 21:15:40,935 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-80:ctx-531f0e5c) The VM is now missing marking it as Stopped
s-10-VM
2014-01-06 21:15:40,935 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-80:ctx-531f0e5c) The VM is now missing marking it as Stopped
r-12-VM
2014-01-06 21:15:40,935 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-80:ctx-531f0e5c) The VM is now missing marking it as Stopped
r-13-VM
2014-01-06 21:15:40,936 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-80:ctx-531f0e5c) The VM is now missing marking it as Stopped r-4-VM
2014-01-06 21:15:40,936 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-80:ctx-531f0e5c) The VM is now missing marking it as Stopped v-7-VM
2014-01-06 21:15:40,936 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-80:ctx-531f0e5c) Seq 1-1135935494: Response Received:
2014-01-06 21:15:40,938 DEBUG [c.c.a.t.Request] (DirectAgent-80:ctx-531f0e5c)
Seq 1-1135935494: Processing: { Ans: , MgmtId: 7588401905746, via: 1, Ver: v1,
Flags: 10,
[{"com.cloud.agent.api.ClusterSyncAnswer":{"_clusterId":1,"_newStates":{"i-2-11-VM":{"t":"aef4c88e-59df-46df-8441-4985a6481ee8","u":"Stopped"},"r-6-VM":{"t":"aef4c88e-59df-46df-8441-4985a6481ee8","u":"Stopped"},"s-10-VM":{"t":"aef4c88e-59df-46df-8441-4985a6481ee8","u":"Stopped"},"r-12-VM":{"t":"aef4c88e-59df-46df-8441-4985a6481ee8","u":"Stopped"},"r-13-VM":{"t":"aef4c88e-59df-46df-8441-4985a6481ee8","u":"Stopped"},"r-4-VM":{"t":"aef4c88e-59df-46df-8441-4985a6481ee8","u":"Stopped"},"v-7-VM":{"t":"aef4c88e-59df-46df-8441-4985a6481ee8","u":"Stopped"}},"_isExecuted":false,"result":true,"wait":0}}]
}
2014-01-06 21:15:40,938 DEBUG [c.c.a.m.AgentAttache]
(DirectAgent-80:ctx-531f0e5c) Seq 1-1135935494: Unable to find listener.
2014-01-06 21:15:40,965 DEBUG [c.c.v.VirtualMachineManagerImpl]
(DirectAgent-80:ctx-531f0e5c) VM r-4-VM: cs state = Running and realState =
Stopped
2014-01-06 21:15:40,965 DEBUG [c.c.v.VirtualMachineManagerImpl]
(DirectAgent-80:ctx-531f0e5c) VM r-4-VM: cs state = Running and realState =
Stopped
2014-01-06 21:15:40,965 DEBUG [c.c.h.HighAvailabilityManagerImpl]
(DirectAgent-80:ctx-531f0e5c) VM does not require investigation so I'm marking
it as Stopped: VM[DomainRouter|r-4-VM]
2014-01-06 21:15:40,965 WARN [c.c.a.m.DirectAgentAttache]
(DirectAgent-80:ctx-531f0e5c) Seq 1-1135935494: Exception caught
java.lang.NullPointerException
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:1268)
at
com.cloud.ha.HighAvailabilityManagerImpl.scheduleRestart(HighAvailabilityManagerImpl.java:346)
at
com.cloud.vm.VirtualMachineManagerImpl.compareState(VirtualMachineManagerImpl.java:2719)
at
com.cloud.vm.VirtualMachineManagerImpl.deltaSync(VirtualMachineManagerImpl.java:2353)
at
com.cloud.vm.VirtualMachineManagerImpl.processAnswers(VirtualMachineManagerImpl.java:2830)
at
com.cloud.agent.manager.AgentManagerImpl.notifyAnswersToMonitors(AgentManagerImpl.java:301)
at com.cloud.agent.manager.AgentAttache.processAnswers(AgentAttache.java:306)
at
com.cloud.agent.manager.ClusteredDirectAgentAttache.processAnswers(ClusteredDirectAgentAttache.java:65)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:242)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
2014-01-06 21:15:45,667 INFO [c.c.a.m.AgentManagerImpl]
(AgentMonitor-1:ctx-c19a3a0d) Found the following agents behind on ping: [7, 10]
2014-01-06 21:15:45,670 DEBUG [c.c.h.Status] (AgentMonitor-1:ctx-c19a3a0d) Ping
timeout for host 7, do invstigation
2014-01-06 21:15:45,672 INFO [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-14:ctx-e3a1c19a) Investigating why host 7 has disconnected with
event PingTimeout
2014-01-06 21:15:45,673 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-14:ctx-e3a1c19a) checking if agent (7) is alive
Attaching the MS logs:
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
