Hi experts, I have configured two VMs in the same hypervisor as hot-standby mode HA. I believe they are successfully communicating with each other with heart beat packets, as we can see the primary VM kea-1 has successfully received the “ha-heartbeat” from the standby VM kea-2 in green logs. But the red logs indicate that the ha_hooks think the HA heartbeat communications failed due to “no route”. But this is a LAN network, and there’s indeed route installed as we can see below and we can ping the 69 ip.
[yxiao322@kea_home1 ~]$ ip route 192.168.100.0/24 dev ens18 proto kernel scope link src 192.168.100.197 metric 100 <<< This should be the route to 192.168.100.69 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown [root@kea_home1 yxiao322]# ping 192.168.100.69 PING 192.168.100.69 (192.168.100.69) 56(84) bytes of data. 64 bytes from 192.168.100.69: icmp_seq=1 ttl=64 time=0.342 ms 64 bytes from 192.168.100.69: icmp_seq=2 ttl=64 time=0.302 ms 64 bytes from 192.168.100.69: icmp_seq=3 ttl=64 time=0.236 ms ^Z [1]+ Stopped ping 192.168.100.69 And if I stop the kea service on primary, then we can see standby server will complain “communication with kea_home1 is interrupted”. And as soon as I start kea service again on primary, then the database began sync again. Thus, I believe there’s indeed communications and syncs between primary and standby VMs. But for some reason, if I shut the kea service on primary, then the standby won’t distribute DHCP leases even after I waited for a long time. Did I miss something here? Primary logs: Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.198 INFO [kea-dhcp6.commands/1224.139988007651072] COMMAND_RECEIVED Received command 'ha-heartbeat' Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.627 WARN [kea-dhcp6.ha-hooks/1224.139988049614592] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home2 (http://192.168.100.69:8000/): No route to host Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.627 WARN [kea-dhcp6.ha-hooks/1224.139988049614592] HA_COMMUNICATION_INTERRUPTED communication with kea_home2 is interrupted Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.199 INFO [kea-dhcp6.commands/1224.139988016043776] COMMAND_RECEIVED Received command 'ha-heartbeat' Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.627 WARN [kea-dhcp6.ha-hooks/1224.139988032829184] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home2 (http://192.168.100.69:8000/): No route to host Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.627 WARN [kea-dhcp6.ha-hooks/1224.139988032829184] HA_COMMUNICATION_INTERRUPTED communication with kea_home2 is interrupted Standby logs: Mar 25 10:10:24 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:24.129 WARN [kea-dhcp6.ha-hooks/2836.139717198915328] HA_COMMUNICATION_INTERRUPTED communication with kea_home1 is interrupted Mar 25 10:10:25 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:25.130 WARN [kea-dhcp6.ha-hooks/2836.139717207308032] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home1 (http://192.168.100.197:8000/): Connection refused Mar 25 10:10:25 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:25.130 WARN [kea-dhcp6.ha-hooks/2836.139717207308032] HA_COMMUNICATION_INTERRUPTED communication with kea_home1 is interrupted Mar 25 10:10:26 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:26.132 WARN [kea-dhcp6.ha-hooks/2836.139717190522624] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home1 (http://192.168.100.197:8000/): Connection refused Mar 25 10:10:26 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:26.132 WARN [kea-dhcp6.ha-hooks/2836.139717190522624] HA_COMMUNICATION_INTERRUPTED communication with kea_home1 is interrupted Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.136 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_STATE_TRANSITION server transitions from PARTNER-DOWN to WAITING state, partner state is PARTNER-DOWN Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner while in WAITING state Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LOCAL_DHCP_DISABLE local DHCP service is disabled while the kea_home2 is in the WAITING state Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_STATE_TRANSITION server transitions from WAITING to SYNCING state, partner state is PARTNER-DOWN Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner while in SYNCING state Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_SYNC_START starting lease database synchronization with kea_home1 Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.140 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASES_SYNC_LEASE_PAGE_RECEIVED received 3 leases from kea_home1 Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.141 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_SYNC_SUCCESSFUL lease database synchronization with kea_home1 completed successfully in 4.350 ms Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.142 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_STATE_TRANSITION server transitions from SYNCING to READY state, partner state is PARTNER-DOWN Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.142 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner while in READY state Primary Config: # HA related hooks configuration "hooks-libraries": [{ "library": "/usr/lib64/kea/hooks/libdhcp_lease_cmds.so", "parameters": { } }, { "library": "/usr/lib64/kea/hooks/libdhcp_ha.so", "parameters": { "high-availability": [{ "this-server-name": "kea_home1", "mode": "hot-standby", "heartbeat-delay": 200, "max-response-delay": 600, "max-ack-delay": 800, "max-unacked-clients": 5, "send-lease-updates": true, "sync-leases": true, "sync-page-limit": 200, "sync-timeout": 600, "delayed-updates-limit": 0, "peers": [{ "name": "kea_home1", "url": http://192.168.100.197:8000/, "role": "primary", "auto-failover": true }, { "name": "kea_home2", "url": http://192.168.100.69:8000/, "role": "standby", "auto-failover": true }] }] } }], "Control-agent": { "http-host": "192.168.100.197", // If enabling HA and multi-threading, the 8000 port is used by the HA // hook library http listener. When using HA hook library with // multi-threading to function, make sure the port used by dedicated // listener is different (e.g. 8001) than the one used by CA. Note // the commands should still be sent via CA. The dedicated listener // is specifically for HA updates only. "http-port": 8001, Standby config: # HA related hooks configuration "hooks-libraries": [{ "library": "/usr/lib64/kea/hooks/libdhcp_lease_cmds.so", "parameters": { } }, { "library": "/usr/lib64/kea/hooks/libdhcp_ha.so", "parameters": { "high-availability": [{ "this-server-name": "kea_home2", "mode": "hot-standby", "heartbeat-delay": 200, "max-response-delay": 600, "max-ack-delay": 800, "max-unacked-clients": 5, "delayed-updates-limit": 0, "send-lease-updates": true, "sync-leases": true, "sync-page-limit": 200, "sync-timeout": 600, "peers": [{ "name": "kea_home1", "url": http://192.168.100.197:8000/, "role": "primary", "auto-failover": true }, { "name": "kea_home2", "url": http://192.168.100.69:8000/, "role": "standby", "auto-failover": true }] }] } }], "Control-agent": { "http-host": "192.168.100.69", // If enabling HA and multi-threading, the 8000 port is used by the HA // hook library http listener. When using HA hook library with // multi-threading to function, make sure the port used by dedicated // listener is different (e.g. 8001) than the one used by CA. Note // the commands should still be sent via CA. The dedicated listener // is specifically for HA updates only. "http-port": 8001, Best Regards, Yu
-- ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users. Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users
