[
https://issues.apache.org/jira/browse/CLOUDSTACK-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chandan Purushothama updated CLOUDSTACK-7725:
---------------------------------------------
Description:
There is something still wrong with the Agents in System VMs on HyperV Setup.
The System VM agents got disconnected due to Ping Issues and VMs got shut down.
I found this observation when I noticed that all the SSVM Test cases failed.
Someone has to look at what is happening between management server and agents
communication. I don’t think it is anything wrong with the Setup based on the
success of other test suites.
=======================================
Disconnection Information Log is as shown below:
=======================================
2014-10-14 19:21:50,378 INFO [c.c.a.m.AgentManagerImpl]
(AgentMonitor-1:ctx-72fa526b) Found the following agents behind on ping: [3, 4]
2014-10-14 19:21:50,380 WARN [c.c.a.m.AgentManagerImpl]
(AgentMonitor-1:ctx-72fa526b) Disconnect agent for CPVM/SSVM due to physical
connection close. host: 3
2014-10-14 19:21:50,381 INFO [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Host 3 is disconnecting with event PingTimeout
2014-10-14 19:21:50,382 WARN [c.c.a.m.AgentManagerImpl]
(AgentMonitor-1:ctx-72fa526b) Disconnect agent for CPVM/SSVM due to physical
connection close. host: 4
2014-10-14 19:21:50,383 INFO [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-6:ctx-f560f275) Host 4 is disconnecting with event PingTimeout
2014-10-14 19:21:50,384 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) The next status of agent 3 is Alert, current
status is Up
2014-10-14 19:21:50,384 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Deregistering link for 3 with state Alert
2014-10-14 19:21:50,384 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Remove Agent : 3
2014-10-14 19:21:50,384 DEBUG [c.c.a.m.ConnectedAgentAttache]
(AgentTaskPool-5:ctx-693d2497) Processing Disconnect.
2014-10-14 19:21:50,384 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
com.cloud.hypervisor.xenserver.discoverer.XcpServerDiscoverer
2014-10-14 19:21:50,384 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
com.cloud.hypervisor.hyperv.discoverer.HypervServerDiscoverer
2014-10-14 19:21:50,384 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
com.cloud.storage.secondary.SecondaryStorageListener
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-6:ctx-f560f275) The next status of agent 4 is Alert, current
status is Up
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-6:ctx-f560f275) Deregistering link for 4 with state Alert
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-6:ctx-f560f275) Remove Agent : 4
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.ConnectedAgentAttache]
(AgentTaskPool-6:ctx-f560f275) Processing Disconnect.
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
com.cloud.vm.ClusteredVirtualMachineManagerImpl
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
com.cloud.storage.listener.StoragePoolMonitor
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
com.cloud.hypervisor.vmware.manager.VmwareManagerImpl
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
com.cloud.deploy.DeploymentPlanningManagerImpl
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-6:ctx-f560f275) Sending Disconnect to listener:
com.cloud.hypervisor.xenserver.discoverer.XcpServerDiscoverer
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-6:ctx-f560f275) Sending Disconnect to listener:
com.cloud.hypervisor.hyperv.discoverer.HypervServerDiscoverer
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-6:ctx-f560f275) Sending Disconnect to listener:
com.cloud.storage.secondary.SecondaryStorageListener
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
com.cloud.network.security.SecurityGroupListener
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
com.cloud.network.SshKeysDistriMonitor
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl
2014-10-14 19:21:50,386 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
com.cloud.consoleproxy.ConsoleProxyListener
2014-10-14 19:21:50,387 INFO [c.c.c.ConsoleProxyManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Console proxy agent disconnected, proxy: v-1-VM
2014-10-14 19:21:50,391 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(AgentTaskPool-5:ctx-693d2497) Job is executed without a context, setup psudo
job for the executing thread
2014-10-14 19:21:50,397 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(AgentTaskPool-6:ctx-f560f275) Job is executed without a context, setup psudo
job for the executing thread
Thank you,
Chandan.
was:
There is something still wrong with the Agents in System VMs on HyperV Setup.
The System VM agents got disconnected due to Ping Issues and VMs got shut down.
I found this observation when I noticed that all the SSVM Test cases failed.
Someone has to look at what is happening between management server and agents
communication. I don’t think it is anything wrong with the Setup based on the
success of other test suites.
Disconnection Information Log is as shown below:
2014-10-14 19:21:50,378 INFO [c.c.a.m.AgentManagerImpl]
(AgentMonitor-1:ctx-72fa526b) Found the following agents behind on ping: [3, 4]
2014-10-14 19:21:50,380 WARN [c.c.a.m.AgentManagerImpl]
(AgentMonitor-1:ctx-72fa526b) Disconnect agent for CPVM/SSVM due to physical
connection close. host: 3
2014-10-14 19:21:50,381 INFO [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Host 3 is disconnecting with event PingTimeout
2014-10-14 19:21:50,382 WARN [c.c.a.m.AgentManagerImpl]
(AgentMonitor-1:ctx-72fa526b) Disconnect agent for CPVM/SSVM due to physical
connection close. host: 4
2014-10-14 19:21:50,383 INFO [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-6:ctx-f560f275) Host 4 is disconnecting with event PingTimeout
2014-10-14 19:21:50,384 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) The next status of agent 3 is Alert, current
status is Up
2014-10-14 19:21:50,384 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Deregistering link for 3 with state Alert
2014-10-14 19:21:50,384 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Remove Agent : 3
2014-10-14 19:21:50,384 DEBUG [c.c.a.m.ConnectedAgentAttache]
(AgentTaskPool-5:ctx-693d2497) Processing Disconnect.
2014-10-14 19:21:50,384 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
com.cloud.hypervisor.xenserver.discoverer.XcpServerDiscoverer
2014-10-14 19:21:50,384 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
com.cloud.hypervisor.hyperv.discoverer.HypervServerDiscoverer
2014-10-14 19:21:50,384 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
com.cloud.storage.secondary.SecondaryStorageListener
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-6:ctx-f560f275) The next status of agent 4 is Alert, current
status is Up
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-6:ctx-f560f275) Deregistering link for 4 with state Alert
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-6:ctx-f560f275) Remove Agent : 4
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.ConnectedAgentAttache]
(AgentTaskPool-6:ctx-f560f275) Processing Disconnect.
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
com.cloud.vm.ClusteredVirtualMachineManagerImpl
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
com.cloud.storage.listener.StoragePoolMonitor
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
com.cloud.hypervisor.vmware.manager.VmwareManagerImpl
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
com.cloud.deploy.DeploymentPlanningManagerImpl
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-6:ctx-f560f275) Sending Disconnect to listener:
com.cloud.hypervisor.xenserver.discoverer.XcpServerDiscoverer
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-6:ctx-f560f275) Sending Disconnect to listener:
com.cloud.hypervisor.hyperv.discoverer.HypervServerDiscoverer
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-6:ctx-f560f275) Sending Disconnect to listener:
com.cloud.storage.secondary.SecondaryStorageListener
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
com.cloud.network.security.SecurityGroupListener
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
com.cloud.network.SshKeysDistriMonitor
2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl
2014-10-14 19:21:50,386 DEBUG [c.c.a.m.AgentManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
com.cloud.consoleproxy.ConsoleProxyListener
2014-10-14 19:21:50,387 INFO [c.c.c.ConsoleProxyManagerImpl]
(AgentTaskPool-5:ctx-693d2497) Console proxy agent disconnected, proxy: v-1-VM
2014-10-14 19:21:50,391 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(AgentTaskPool-5:ctx-693d2497) Job is executed without a context, setup psudo
job for the executing thread
2014-10-14 19:21:50,397 WARN [o.a.c.f.j.AsyncJobExecutionContext]
(AgentTaskPool-6:ctx-f560f275) Job is executed without a context, setup psudo
job for the executing thread
Thank you,
Chandan.
> [Automation][HyperV] After sometime System VM 'agents' get disconnected due
> to Ping Issues and get shut down
> ------------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-7725
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7725
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Automation
> Affects Versions: 4.5.0
> Reporter: Chandan Purushothama
> Priority: Blocker
> Fix For: 4.5.0
>
>
> There is something still wrong with the Agents in System VMs on HyperV Setup.
> The System VM agents got disconnected due to Ping Issues and VMs got shut
> down. I found this observation when I noticed that all the SSVM Test cases
> failed.
> Someone has to look at what is happening between management server and agents
> communication. I don’t think it is anything wrong with the Setup based on the
> success of other test suites.
> =======================================
> Disconnection Information Log is as shown below:
> =======================================
> 2014-10-14 19:21:50,378 INFO [c.c.a.m.AgentManagerImpl]
> (AgentMonitor-1:ctx-72fa526b) Found the following agents behind on ping: [3,
> 4]
> 2014-10-14 19:21:50,380 WARN [c.c.a.m.AgentManagerImpl]
> (AgentMonitor-1:ctx-72fa526b) Disconnect agent for CPVM/SSVM due to physical
> connection close. host: 3
> 2014-10-14 19:21:50,381 INFO [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-5:ctx-693d2497) Host 3 is disconnecting with event PingTimeout
> 2014-10-14 19:21:50,382 WARN [c.c.a.m.AgentManagerImpl]
> (AgentMonitor-1:ctx-72fa526b) Disconnect agent for CPVM/SSVM due to physical
> connection close. host: 4
> 2014-10-14 19:21:50,383 INFO [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-6:ctx-f560f275) Host 4 is disconnecting with event PingTimeout
> 2014-10-14 19:21:50,384 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-5:ctx-693d2497) The next status of agent 3 is Alert, current
> status is Up
> 2014-10-14 19:21:50,384 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-5:ctx-693d2497) Deregistering link for 3 with state Alert
> 2014-10-14 19:21:50,384 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-5:ctx-693d2497) Remove Agent : 3
> 2014-10-14 19:21:50,384 DEBUG [c.c.a.m.ConnectedAgentAttache]
> (AgentTaskPool-5:ctx-693d2497) Processing Disconnect.
> 2014-10-14 19:21:50,384 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
> com.cloud.hypervisor.xenserver.discoverer.XcpServerDiscoverer
> 2014-10-14 19:21:50,384 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
> com.cloud.hypervisor.hyperv.discoverer.HypervServerDiscoverer
> 2014-10-14 19:21:50,384 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
> com.cloud.storage.secondary.SecondaryStorageListener
> 2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-6:ctx-f560f275) The next status of agent 4 is Alert, current
> status is Up
> 2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-6:ctx-f560f275) Deregistering link for 4 with state Alert
> 2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-6:ctx-f560f275) Remove Agent : 4
> 2014-10-14 19:21:50,385 DEBUG [c.c.a.m.ConnectedAgentAttache]
> (AgentTaskPool-6:ctx-f560f275) Processing Disconnect.
> 2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
> com.cloud.vm.ClusteredVirtualMachineManagerImpl
> 2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator
> 2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
> com.cloud.storage.listener.StoragePoolMonitor
> 2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
> com.cloud.hypervisor.vmware.manager.VmwareManagerImpl
> 2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
> com.cloud.deploy.DeploymentPlanningManagerImpl
> 2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-6:ctx-f560f275) Sending Disconnect to listener:
> com.cloud.hypervisor.xenserver.discoverer.XcpServerDiscoverer
> 2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-6:ctx-f560f275) Sending Disconnect to listener:
> com.cloud.hypervisor.hyperv.discoverer.HypervServerDiscoverer
> 2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-6:ctx-f560f275) Sending Disconnect to listener:
> com.cloud.storage.secondary.SecondaryStorageListener
> 2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
> com.cloud.network.security.SecurityGroupListener
> 2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
> com.cloud.network.SshKeysDistriMonitor
> 2014-10-14 19:21:50,385 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl
> 2014-10-14 19:21:50,386 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentTaskPool-5:ctx-693d2497) Sending Disconnect to listener:
> com.cloud.consoleproxy.ConsoleProxyListener
> 2014-10-14 19:21:50,387 INFO [c.c.c.ConsoleProxyManagerImpl]
> (AgentTaskPool-5:ctx-693d2497) Console proxy agent disconnected, proxy: v-1-VM
> 2014-10-14 19:21:50,391 WARN [o.a.c.f.j.AsyncJobExecutionContext]
> (AgentTaskPool-5:ctx-693d2497) Job is executed without a context, setup psudo
> job for the executing thread
> 2014-10-14 19:21:50,397 WARN [o.a.c.f.j.AsyncJobExecutionContext]
> (AgentTaskPool-6:ctx-f560f275) Job is executed without a context, setup psudo
> job for the executing thread
> Thank you,
> Chandan.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
