[
https://issues.apache.org/jira/browse/CLOUDSTACK-3421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gerard Lynch updated CLOUDSTACK-3421:
-------------------------------------
Attachment: catalina_management-server.zip
Attached our management server catalina.out and management-server.log files.
Let me know if you require anything further.
> When hypervisor is down, no HA occurs with log output "Agent state cannot be
> determined, do nothing"
> ----------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-3421
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3421
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: KVM, Management Server
> Affects Versions: 4.1.0
> Environment: CentOS 6.4 minimal install
> Libvirt, KVM/Qemu
> CloudStack 4.1
> GlusterFS 3.2, replicated+distributed as primary storage via Shared Mount
> Point
> 3 physical servers
> * 1 management server, running NFS secondary storage
> ** 1 nic, management+storage
> * 2 hypervisor nodes, running glusterfs-server
> ** 4x nic, management+storage, public, guest, gluster peering
> * Advanced zone
> * KVM
> * 4 networks:
> eth0: cloudbr0: management+secondary storage,
> eth2: cloudbr1: public
> eth3: cloudbr2: guest
> eth1: gluster peering
> * Shared Mount Point
> * custom network offering with redundant routers enabled
> * global settings tweaked to increase speed of identifying down state
> ** ping.interval: 10sec
> Reporter: Gerard Lynch
> Priority: Critical
> Fix For: 4.1.1, 4.2.0, Future
>
> Attachments: catalina_management-server.zip
>
>
> We wanted to test CloudStack's HA capabilities by simulating outages to find
> out how long it would take to recover. One of the tests was simulating loss
> of a hypervisor node by shutting it down. When we tested this, we found
> that CloudStack failed to bring up any of the VMs (System or Instance), which
> were on the down node, until the node was powered back up and reconnected.
> In the logs, we see repeating occurances of:
> INFO [utils.exception.CSExceptionErrorCode] (AgentTaskPool-11:) Could not
> find exception: com.cloud.exception.OperationTimedoutException in error code
> list for exceptions
> INFO [utils.exception.CSExceptionErrorCode] (AgentTaskPool-10:) Could not
> find exception: com.cloud.exception.OperationTimedoutException in error code
> list for exceptions
> WARN [agent.manager.AgentAttache] (AgentTaskPool-11:) Seq 14-660013135:
> Timed out on Seq 14-660013135: { Cmd , MgmtId: 93515041483, via: 14, Ver:
> v1, Flags: 100011, [{"CheckHealthCommand":{"wait":50}}] }
> WARN [agent.manager.AgentAttache] (AgentTaskPool-10:) Seq 15-1097531400:
> Timed out on Seq 15-1097531400: { Cmd , MgmtId: 93515041483, via: 15, Ver:
> v1, Flags: 100011, [{"CheckHealthCommand":{"wait":50}}] }
> WARN [agent.manager.AgentManagerImpl] (AgentTaskPool-11:) Operation timed
> out: Commands 660013135 to Host 14 timed out after 100
> WARN [agent.manager.AgentManagerImpl] (AgentTaskPool-10:) Operation timed
> out: Commands 1097531400 to Host 15 timed out after 100
> WARN [agent.manager.AgentManagerImpl] (AgentTaskPool-11:) Agent state cannot
> be determined, do nothing
> WARN [agent.manager.AgentManagerImpl] (AgentTaskPool-10:) Agent state cannot
> be determined, do nothing
> To reproduce:
> 1. Build the environment as detailed above
> 2. Register an ISO
> 3. Create a new guest network using the custom network offering (that offers
> redundant routers)
> 3. Provision an instance
> 4. Ensure the system VMs and instance are on the first hypervisor node
> 5. Shutdown the first hypervisor node (or pull the plug)
> Expected result:
> All system VMs and instance(s) should be brought up on the 2nd hypervisor
> node.
> Actual result:
> We see the first hypervisor node marked "disconnected."
> All System VMs and the Instance are still marked "Running", however ping to
> any of them fails.
> Ping to the redundant router on the 2nd hypervisor node is still working.
> We see in the logs
> "INFO [utils.exception.CSExceptionErrorCode] (AgentTaskPool-11:) Could not
> find exception: com.cloud.exception.OperationTimedoutException in error code
> list for exceptions"
> Followed by
> "Agent state cannot be determined, do nothing"
> Searching for "Cloudstack Agent state cannot be determined, do nothing" lead
> to: CLOUDSTACK-803 - https://reviews.apache.org/r/8853/
> Which caused me some concern, because if I read the logic in the ticket
> correctly... The management server will not perform any HA actions if it's
> unable to determine the state of a hypervisor node. In the scenario above,
> it's not a loss of connectivity, but an actual outage on the hypervisor... so
> I'd rather like HA to occur. Split brain is a concern, but I think that
> something along the lines of "if hypervisor can't see management or gateway,
> stop instances)" is more relevant than "do nothing"
> I'm hoping this is something really obvious and simple to resolve, because
> otherwise this is a pretty serious issue as currently any accidental
> shutdown, or hardware fault will cause a continuous outage requiring manual
> action to resolve.
> Thanks
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
