Hi Darren, I am using 2.4.1.
Best Regards, Yu From: Kea-users <kea-users-boun...@lists.isc.org> on behalf of Darren Ankney <darren.ank...@gmail.com> Date: Tuesday, March 26, 2024 at 6:03 AM To: Kea user's list <kea-users@lists.isc.org> Subject: [EXTERNAL] Re: [Kea-users] kea-2.4.1 // Strange HA related errors Hello Yu, What version of Kea are you running on these servers, out of curiosity? Thank you, Darren Ankney On Mon, Mar 25, 2024 at 10:55 AM Xiao, Yu (CCI-Atlanta) via Kea-users <kea-users@lists.isc.org> wrote: > > Hi experts, > > > > I have configured two VMs in the same hypervisor as hot-standby mode HA. I > believe they are successfully communicating with each other with heart beat > packets, as we can see the primary VM kea-1 has successfully received the > “ha-heartbeat” from the standby VM kea-2 in green logs. But the red logs > indicate that the ha_hooks think the HA heartbeat communications failed due > to “no route”. But this is a LAN network, and there’s indeed route installed > as we can see below and we can ping the 69 ip. > > > > [yxiao322@kea_home1 ~]$ ip route > > 192.168.100.0/24 dev ens18 proto kernel scope link src 192.168.100.197 metric > 100 <<< This should be the route to 192.168.100.69 > > 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown > > > > [root@kea_home1 yxiao322]# ping 192.168.100.69 > > PING 192.168.100.69 (192.168.100.69) 56(84) bytes of data. > > 64 bytes from 192.168.100.69: icmp_seq=1 ttl=64 time=0.342 ms > > 64 bytes from 192.168.100.69: icmp_seq=2 ttl=64 time=0.302 ms > > 64 bytes from 192.168.100.69: icmp_seq=3 ttl=64 time=0.236 ms > > ^Z > > [1]+ Stopped ping 192.168.100.69 > > > > And if I stop the kea service on primary, then we can see standby server will > complain “communication with kea_home1 is interrupted”. And as soon as I > start kea service again on primary, then the database began sync again. Thus, > I believe there’s indeed communications and syncs between primary and standby > VMs. But for some reason, if I shut the kea service on primary, then the > standby won’t distribute DHCP leases even after I waited for a long time. Did > I miss something here? > > > > > > Primary logs: > > > > Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.198 INFO > [kea-dhcp6.commands/1224.139988007651072] COMMAND_RECEIVED Received command > 'ha-heartbeat' > > Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.627 WARN > [kea-dhcp6.ha-hooks/1224.139988049614592] HA_HEARTBEAT_COMMUNICATIONS_FAILED > failed to send heartbeat to kea_home2 > (https://urldefense.com/v3/__http://192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$ > ): No route to host > > Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.627 WARN > [kea-dhcp6.ha-hooks/1224.139988049614592] HA_COMMUNICATION_INTERRUPTED > communication with kea_home2 is interrupted > > Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.199 INFO > [kea-dhcp6.commands/1224.139988016043776] COMMAND_RECEIVED Received command > 'ha-heartbeat' > > Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.627 WARN > [kea-dhcp6.ha-hooks/1224.139988032829184] HA_HEARTBEAT_COMMUNICATIONS_FAILED > failed to send heartbeat to kea_home2 > (https://urldefense.com/v3/__http://192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$ > ): No route to host > > Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.627 WARN > [kea-dhcp6.ha-hooks/1224.139988032829184] HA_COMMUNICATION_INTERRUPTED > communication with kea_home2 is interrupted > > > > Standby logs: > > > > Mar 25 10:10:24 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:24.129 WARN > [kea-dhcp6.ha-hooks/2836.139717198915328] HA_COMMUNICATION_INTERRUPTED > communication with kea_home1 is interrupted > > Mar 25 10:10:25 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:25.130 WARN > [kea-dhcp6.ha-hooks/2836.139717207308032] HA_HEARTBEAT_COMMUNICATIONS_FAILED > failed to send heartbeat to kea_home1 > (https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$ > ): Connection refused > > Mar 25 10:10:25 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:25.130 WARN > [kea-dhcp6.ha-hooks/2836.139717207308032] HA_COMMUNICATION_INTERRUPTED > communication with kea_home1 is interrupted > > Mar 25 10:10:26 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:26.132 WARN > [kea-dhcp6.ha-hooks/2836.139717190522624] HA_HEARTBEAT_COMMUNICATIONS_FAILED > failed to send heartbeat to kea_home1 > (https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$ > ): Connection refused > > Mar 25 10:10:26 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:26.132 WARN > [kea-dhcp6.ha-hooks/2836.139717190522624] HA_COMMUNICATION_INTERRUPTED > communication with kea_home1 is interrupted > > Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.136 INFO > [kea-dhcp6.ha-hooks/2836.139717215700736] HA_STATE_TRANSITION server > transitions from PARTNER-DOWN to WAITING state, partner state is PARTNER-DOWN > > Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO > [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASE_UPDATES_DISABLED lease > updates will not be sent to the partner while in WAITING state > > Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO > [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LOCAL_DHCP_DISABLE local DHCP > service is disabled while the kea_home2 is in the WAITING state > > Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO > [kea-dhcp6.ha-hooks/2836.139717215700736] HA_STATE_TRANSITION server > transitions from WAITING to SYNCING state, partner state is PARTNER-DOWN > > Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO > [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASE_UPDATES_DISABLED lease > updates will not be sent to the partner while in SYNCING state > > Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO > [kea-dhcp6.ha-hooks/2836.139717215700736] HA_SYNC_START starting lease > database synchronization with kea_home1 > > Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.140 INFO > [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASES_SYNC_LEASE_PAGE_RECEIVED > received 3 leases from kea_home1 > > Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.141 INFO > [kea-dhcp6.ha-hooks/2836.139717215700736] HA_SYNC_SUCCESSFUL lease database > synchronization with kea_home1 completed successfully in 4.350 ms > > Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.142 INFO > [kea-dhcp6.ha-hooks/2836.139717215700736] HA_STATE_TRANSITION server > transitions from SYNCING to READY state, partner state is PARTNER-DOWN > > Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.142 INFO > [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASE_UPDATES_DISABLED lease > updates will not be sent to the partner while in READY state > > > > Primary Config: > > > > # HA related hooks configuration > > "hooks-libraries": [{ > > "library": "/usr/lib64/kea/hooks/libdhcp_lease_cmds.so", > > "parameters": { } > > }, { > > "library": "/usr/lib64/kea/hooks/libdhcp_ha.so", > > "parameters": { > > "high-availability": [{ > > "this-server-name": "kea_home1", > > "mode": "hot-standby", > > "heartbeat-delay": 200, > > "max-response-delay": 600, > > "max-ack-delay": 800, > > "max-unacked-clients": 5, > > "send-lease-updates": true, > > "sync-leases": true, > > "sync-page-limit": 200, > > "sync-timeout": 600, > > "delayed-updates-limit": 0, > > "peers": [{ > > "name": "kea_home1", > > "url": > https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$<https://urldefense.com/v3/__http:/192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$> > , > > "role": "primary", > > "auto-failover": true > > }, { > > "name": "kea_home2", > > "url": > https://urldefense.com/v3/__http://192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$<https://urldefense.com/v3/__http:/192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$> > , > > "role": "standby", > > "auto-failover": true > > }] > > }] > > } > > }], > > > > > > "Control-agent": { > > "http-host": "192.168.100.197", > > // If enabling HA and multi-threading, the 8000 port is used by the HA > > // hook library http listener. When using HA hook library with > > // multi-threading to function, make sure the port used by dedicated > > // listener is different (e.g. 8001) than the one used by CA. Note > > // the commands should still be sent via CA. The dedicated listener > > // is specifically for HA updates only. > > "http-port": 8001, > > > > > > Standby config: > > > > # HA related hooks configuration > > "hooks-libraries": [{ > > "library": "/usr/lib64/kea/hooks/libdhcp_lease_cmds.so", > > "parameters": { } > > }, { > > "library": "/usr/lib64/kea/hooks/libdhcp_ha.so", > > "parameters": { > > "high-availability": [{ > > "this-server-name": "kea_home2", > > "mode": "hot-standby", > > "heartbeat-delay": 200, > > "max-response-delay": 600, > > "max-ack-delay": 800, > > "max-unacked-clients": 5, > > "delayed-updates-limit": 0, > > "send-lease-updates": true, > > "sync-leases": true, > > "sync-page-limit": 200, > > "sync-timeout": 600, > > "peers": [{ > > "name": "kea_home1", > > "url": > https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$<https://urldefense.com/v3/__http:/192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$> > , > > "role": "primary", > > "auto-failover": true > > }, { > > "name": "kea_home2", > > "url": > https://urldefense.com/v3/__http://192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$<https://urldefense.com/v3/__http:/192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$> > , > > "role": "standby", > > "auto-failover": true > > }] > > }] > > } > > }], > > > > "Control-agent": { > > "http-host": "192.168.100.69", > > // If enabling HA and multi-threading, the 8000 port is used by the HA > > // hook library http listener. When using HA hook library with > > // multi-threading to function, make sure the port used by dedicated > > // listener is different (e.g. 8001) than the one used by CA. Note > > // the commands should still be sent via CA. The dedicated listener > > // is specifically for HA updates only. > > "http-port": 8001, > > > > > > Best Regards, > > Yu > > > > -- > ISC funds the development of this software with paid support subscriptions. > Contact us at > https://urldefense.com/v3/__https://www.isc.org/contact/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTl3O_IGO$<https://urldefense.com/v3/__https:/www.isc.org/contact/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTl3O_IGO$> > for more information. > > To unsubscribe visit > https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$> > . > > Kea-users mailing list > Kea-users@lists.isc.org > https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$> -- ISC funds the development of this software with paid support subscriptions. Contact us at https://urldefense.com/v3/__https://www.isc.org/contact/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTl3O_IGO$<https://urldefense.com/v3/__https:/www.isc.org/contact/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTl3O_IGO$> for more information. To unsubscribe visit https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$> . Kea-users mailing list Kea-users@lists.isc.org https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTp7W6GPF$>
-- ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users. Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users
