Struggling to understand what does stopping a single mgmt server has to do
with ANY HA anywhere in the ACS setup...? Am I missing something, or did
you forget to share more info?
Cheers,
On Tue, 17 Nov 2020 at 12:27, Piotr Pisz <pi...@piszki.pl> wrote:
> Hi Users!
>
> I have a question asking for an explanation of the HA mechanism which
> behaves quite strangely
>
> We have three mgmt servers, IP 10.89.12.101,102,103
> And NFS Pool (sys2.cenagis.local) on floating IP via haproxy and
> nfs-ganesha.
> We make a simple test, power off 10.89.12.101, and we saw strange
> infromation in cloudstack-agent logs, NFS checks are going via cs mgmt IP
> and not via floating IP (sys2.cenagis.local), why?
>
> Do we have a logical error somewhere?
>
> Best regards,
> Piotr
>
> 2020-11-17 09:56:37,140 DEBUG [kvm.resource.KVMHAChecker]
> (pool-72086-thread-1:null) (logid:0b27330b) Executing while with timeout :
> 360000
> 2020-11-17 09:56:37,148 DEBUG [kvm.resource.KVMHAChecker]
> (pool-72086-thread-1:null) (logid:0b27330b) Execution is successful.
> 2020-11-17 09:56:37,149 DEBUG [kvm.resource.KVMHAChecker]
> (pool-72086-thread-1:null) (logid:0b27330b) KVMHAChecker pool:
> sys2.cenagis.local
> 2020-11-17 09:56:37,149 DEBUG [kvm.resource.KVMHAChecker]
> (pool-72086-thread-1:null) (logid:0b27330b) KVMHAChecker result: null
> 2020-11-17 09:56:37,149 DEBUG [kvm.resource.KVMHAChecker]
> (pool-72086-thread-1:null) (logid:0b27330b) KVMHAChecker parser: =====>
> ALIVE <=====
> 2020-11-17 09:56:37,149 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-3:null) (logid:0b27330b) Seq 25-6380193297100515623:
> { Ans: , MgmtId: 176206389408022, via: 25, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.Answer":{"result":false,"details":"Heart is
> beating...","wait":0}}] }
> 2020-11-17 09:56:39,048 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:null) (logid:246f60a3) Request:Seq
> 25-8191766247210225151: { Cmd , MgmtId: 176206389407982, via: 25, Ver: v1,
> Flags: 100011,
> [{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"27135d91-2859-3df8-9464-77f12c886598-LibvirtComputingResource","privateNetwork":{"ip":"10.89.12.101","netmask":"255.255.255.0","mac":"6a:fd:73:b6:d0:46","isSecurityGroupEnabled":false},"storageNetwork1":{"ip":"10.89.12.101","netmask":"255.255.255.0","mac":"6a:fd:73:b6:d0:46","isSecurityGroupEnabled":false}},"wait":20}}]
> }
> 2020-11-17 09:56:39,048 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:null) (logid:246f60a3) Processing command:
> com.cloud.agent.api.CheckOnHostCommand
> 2020-11-17 09:56:39,049 DEBUG [kvm.resource.KVMHAChecker]
> (pool-72087-thread-1:null) (logid:246f60a3) Executing:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i
> sys2.cenagis.local -p /HA -m /mnt/9cc142eb-ff79-3ea4-957d-9a45b987604e -h
> 10.89.12.101 -r -t 60
> 2020-11-17 09:56:39,049 DEBUG [kvm.resource.KVMHAChecker]
> (pool-72087-thread-1:null) (logid:246f60a3) Executing while with timeout :
> 360000
> 2020-11-17 09:56:39,058 DEBUG [kvm.resource.KVMHAChecker]
> (pool-72087-thread-1:null) (logid:246f60a3) Execution is successful.
> 2020-11-17 09:56:39,058 DEBUG [kvm.resource.KVMHAChecker]
> (pool-72087-thread-1:null) (logid:246f60a3) KVMHAChecker pool:
> sys2.cenagis.local
> 2020-11-17 09:56:39,058 DEBUG [kvm.resource.KVMHAChecker]
> (pool-72087-thread-1:null) (logid:246f60a3) KVMHAChecker result: null
> 2020-11-17 09:56:39,058 DEBUG [kvm.resource.KVMHAChecker]
> (pool-72087-thread-1:null) (logid:246f60a3) KVMHAChecker parser: =====>
> DEAD <======
> 2020-11-17 09:56:39,058 DEBUG [kvm.resource.KVMHAChecker]
> (pool-72087-thread-1:null) (logid:246f60a3) read heartbeat failed:
> 2020-11-17 09:56:39,058 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-5:null) (logid:246f60a3) Seq 25-8191766247210225151:
> { Ans: , MgmtId: 176206389407982, via: 25, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
> 2020-11-17 09:56:52,834 DEBUG [kvm.resource.LibvirtConnection]
> (Thread-11215:null) (logid:) Looking for libvirtd connection at:
> qemu:///system
> 2020-11-17 09:56:52,834 DEBUG [kvm.storage.IscsiStorageCleanupMonitor]
> (Thread-11215:null) (logid:) found 0 domains
> 2020-11-17 09:56:58,860 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-1:null) (logid:4a826d3a) Request:Seq
> 25-6380193297100515624: { Cmd , MgmtId: 176206389408022, via: 25, Ver: v1,
> Flags: 100011,
> [{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"de2a757a-4d20-39c1-97be-2eb0096c32fb-LibvirtComputingResource","privateNetwork":{"ip":"10.89.12.103","netmask":"255.255.255.0","mac":"7e:c2:06:20:06:48","isSecurityGroupEnabled":false},"storageNetwork1":{"ip":"10.89.12.103","netmask":"255.255.255.0","mac":"7e:c2:06:20:06:48","isSecurityGroupEnabled":false}},"wait":20}}]
> }
> 2020-11-17 09:56:58,860 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-1:null) (logid:4a826d3a) Processing command:
> com.cloud.agent.api.CheckOnHostCommand
> 2020-11-17 09:56:58,861 DEBUG [kvm.resource.KVMHAChecker]
> (pool-72088-thread-1:null) (logid:4a826d3a) Executing:
> /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i
> sys2.cenagis.local -p /HA -m /mnt/9cc142eb-ff79-3ea4-957d-9a45b987604e -h
> 10.89.12.103 -r -t 60
> 2020-11-17 09:56:58,861 DEBUG [kvm.resource.KVMHAChecker]
> (pool-72088-thread-1:null) (logid:4a826d3a) Executing while with timeout :
> 360000
> 2020-11-17 09:56:58,870 DEBUG [kvm.resource.KVMHAChecker]
> (pool-72088-thread-1:null) (logid:4a826d3a) Execution is successful.
> 2020-11-17 09:56:58,870 DEBUG [kvm.resource.KVMHAChecker]
> (pool-72088-thread-1:null) (logid:4a826d3a) KVMHAChecker pool:
> sys2.cenagis.local
> 2020-11-17 09:56:58,870 DEBUG [kvm.resource.KVMHAChecker]
> (pool-72088-thread-1:null) (logid:4a826d3a) KVMHAChecker result: null
> 2020-11-17 09:56:58,870 DEBUG [kvm.resource.KVMHAChecker]
> (pool-72088-thread-1:null) (logid:4a826d3a) KVMHAChecker parser: =====>
> ALIVE <=====
> 2020-11-17 09:56:58,870 DEBUG [cloud.agent.Agent]
> (agentRequest-Handler-1:null) (logid:4a826d3a) Seq 25-6380193297100515624:
> { Ans: , MgmtId: 176206389408022, via: 25, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.Answer":{"result":false,"details":"Heart is
> beating...","wait":0}}] }
>
>
--
Andrija Panić
