Hi Users!
I have a question asking for an explanation of the HA mechanism which behaves
quite strangely
We have three mgmt servers, IP 10.89.12.101,102,103
And NFS Pool (sys2.cenagis.local) on floating IP via haproxy and nfs-ganesha.
We make a simple test, power off 10.89.12.101, and we saw strange infromation
in cloudstack-agent logs, NFS checks are going via cs mgmt IP and not via
floating IP (sys2.cenagis.local), why?
Do we have a logical error somewhere?
Best regards,
Piotr
2020-11-17 09:56:37,140 DEBUG [kvm.resource.KVMHAChecker]
(pool-72086-thread-1:null) (logid:0b27330b) Executing while with timeout :
360000
2020-11-17 09:56:37,148 DEBUG [kvm.resource.KVMHAChecker]
(pool-72086-thread-1:null) (logid:0b27330b) Execution is successful.
2020-11-17 09:56:37,149 DEBUG [kvm.resource.KVMHAChecker]
(pool-72086-thread-1:null) (logid:0b27330b) KVMHAChecker pool:
sys2.cenagis.local
2020-11-17 09:56:37,149 DEBUG [kvm.resource.KVMHAChecker]
(pool-72086-thread-1:null) (logid:0b27330b) KVMHAChecker result: null
2020-11-17 09:56:37,149 DEBUG [kvm.resource.KVMHAChecker]
(pool-72086-thread-1:null) (logid:0b27330b) KVMHAChecker parser: =====> ALIVE
<=====
2020-11-17 09:56:37,149 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null)
(logid:0b27330b) Seq 25-6380193297100515623: { Ans: , MgmtId: 176206389408022,
via: 25, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"Heart is
beating...","wait":0}}] }
2020-11-17 09:56:39,048 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null)
(logid:246f60a3) Request:Seq 25-8191766247210225151: { Cmd , MgmtId:
176206389407982, via: 25, Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"27135d91-2859-3df8-9464-77f12c886598-LibvirtComputingResource","privateNetwork":{"ip":"10.89.12.101","netmask":"255.255.255.0","mac":"6a:fd:73:b6:d0:46","isSecurityGroupEnabled":false},"storageNetwork1":{"ip":"10.89.12.101","netmask":"255.255.255.0","mac":"6a:fd:73:b6:d0:46","isSecurityGroupEnabled":false}},"wait":20}}]
}
2020-11-17 09:56:39,048 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null)
(logid:246f60a3) Processing command: com.cloud.agent.api.CheckOnHostCommand
2020-11-17 09:56:39,049 DEBUG [kvm.resource.KVMHAChecker]
(pool-72087-thread-1:null) (logid:246f60a3) Executing:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i
sys2.cenagis.local -p /HA -m /mnt/9cc142eb-ff79-3ea4-957d-9a45b987604e -h
10.89.12.101 -r -t 60
2020-11-17 09:56:39,049 DEBUG [kvm.resource.KVMHAChecker]
(pool-72087-thread-1:null) (logid:246f60a3) Executing while with timeout :
360000
2020-11-17 09:56:39,058 DEBUG [kvm.resource.KVMHAChecker]
(pool-72087-thread-1:null) (logid:246f60a3) Execution is successful.
2020-11-17 09:56:39,058 DEBUG [kvm.resource.KVMHAChecker]
(pool-72087-thread-1:null) (logid:246f60a3) KVMHAChecker pool:
sys2.cenagis.local
2020-11-17 09:56:39,058 DEBUG [kvm.resource.KVMHAChecker]
(pool-72087-thread-1:null) (logid:246f60a3) KVMHAChecker result: null
2020-11-17 09:56:39,058 DEBUG [kvm.resource.KVMHAChecker]
(pool-72087-thread-1:null) (logid:246f60a3) KVMHAChecker parser: =====> DEAD
<======
2020-11-17 09:56:39,058 DEBUG [kvm.resource.KVMHAChecker]
(pool-72087-thread-1:null) (logid:246f60a3) read heartbeat failed:
2020-11-17 09:56:39,058 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null)
(logid:246f60a3) Seq 25-8191766247210225151: { Ans: , MgmtId: 176206389407982,
via: 25, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2020-11-17 09:56:52,834 DEBUG [kvm.resource.LibvirtConnection]
(Thread-11215:null) (logid:) Looking for libvirtd connection at: qemu:///system
2020-11-17 09:56:52,834 DEBUG [kvm.storage.IscsiStorageCleanupMonitor]
(Thread-11215:null) (logid:) found 0 domains
2020-11-17 09:56:58,860 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null)
(logid:4a826d3a) Request:Seq 25-6380193297100515624: { Cmd , MgmtId:
176206389408022, via: 25, Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.CheckOnHostCommand":{"host":{"guid":"de2a757a-4d20-39c1-97be-2eb0096c32fb-LibvirtComputingResource","privateNetwork":{"ip":"10.89.12.103","netmask":"255.255.255.0","mac":"7e:c2:06:20:06:48","isSecurityGroupEnabled":false},"storageNetwork1":{"ip":"10.89.12.103","netmask":"255.255.255.0","mac":"7e:c2:06:20:06:48","isSecurityGroupEnabled":false}},"wait":20}}]
}
2020-11-17 09:56:58,860 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null)
(logid:4a826d3a) Processing command: com.cloud.agent.api.CheckOnHostCommand
2020-11-17 09:56:58,861 DEBUG [kvm.resource.KVMHAChecker]
(pool-72088-thread-1:null) (logid:4a826d3a) Executing:
/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i
sys2.cenagis.local -p /HA -m /mnt/9cc142eb-ff79-3ea4-957d-9a45b987604e -h
10.89.12.103 -r -t 60
2020-11-17 09:56:58,861 DEBUG [kvm.resource.KVMHAChecker]
(pool-72088-thread-1:null) (logid:4a826d3a) Executing while with timeout :
360000
2020-11-17 09:56:58,870 DEBUG [kvm.resource.KVMHAChecker]
(pool-72088-thread-1:null) (logid:4a826d3a) Execution is successful.
2020-11-17 09:56:58,870 DEBUG [kvm.resource.KVMHAChecker]
(pool-72088-thread-1:null) (logid:4a826d3a) KVMHAChecker pool:
sys2.cenagis.local
2020-11-17 09:56:58,870 DEBUG [kvm.resource.KVMHAChecker]
(pool-72088-thread-1:null) (logid:4a826d3a) KVMHAChecker result: null
2020-11-17 09:56:58,870 DEBUG [kvm.resource.KVMHAChecker]
(pool-72088-thread-1:null) (logid:4a826d3a) KVMHAChecker parser: =====> ALIVE
<=====
2020-11-17 09:56:58,870 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null)
(logid:4a826d3a) Seq 25-6380193297100515624: { Ans: , MgmtId: 176206389408022,
via: 25, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.Answer":{"result":false,"details":"Heart is
beating...","wait":0}}] }
