[Kea-users] kea-2.4.1 // Strange HA related errors
Hi experts, I have configured two VMs in the same hypervisor as hot-standby mode HA. I believe they are successfully communicating with each other with heart beat packets, as we can see the primary VM kea-1 has successfully received the “ha-heartbeat” from the standby VM kea-2 in green logs. But the red logs indicate that the ha_hooks think the HA heartbeat communications failed due to “no route”. But this is a LAN network, and there’s indeed route installed as we can see below and we can ping the 69 ip. [yxiao322@kea_home1 ~]$ ip route 192.168.100.0/24 dev ens18 proto kernel scope link src 192.168.100.197 metric 100 <<< This should be the route to 192.168.100.69 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown [root@kea_home1 yxiao322]# ping 192.168.100.69 PING 192.168.100.69 (192.168.100.69) 56(84) bytes of data. 64 bytes from 192.168.100.69: icmp_seq=1 ttl=64 time=0.342 ms 64 bytes from 192.168.100.69: icmp_seq=2 ttl=64 time=0.302 ms 64 bytes from 192.168.100.69: icmp_seq=3 ttl=64 time=0.236 ms ^Z [1]+ Stopped ping 192.168.100.69 And if I stop the kea service on primary, then we can see standby server will complain “communication with kea_home1 is interrupted”. And as soon as I start kea service again on primary, then the database began sync again. Thus, I believe there’s indeed communications and syncs between primary and standby VMs. But for some reason, if I shut the kea service on primary, then the standby won’t distribute DHCP leases even after I waited for a long time. Did I miss something here? Primary logs: Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.198 INFO [kea-dhcp6.commands/1224.139988007651072] COMMAND_RECEIVED Received command 'ha-heartbeat' Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.627 WARN [kea-dhcp6.ha-hooks/1224.139988049614592] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home2 (http://192.168.100.69:8000/): No route to host Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.627 WARN [kea-dhcp6.ha-hooks/1224.139988049614592] HA_COMMUNICATION_INTERRUPTED communication with kea_home2 is interrupted Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.199 INFO [kea-dhcp6.commands/1224.139988016043776] COMMAND_RECEIVED Received command 'ha-heartbeat' Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.627 WARN [kea-dhcp6.ha-hooks/1224.139988032829184] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home2 (http://192.168.100.69:8000/): No route to host Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.627 WARN [kea-dhcp6.ha-hooks/1224.139988032829184] HA_COMMUNICATION_INTERRUPTED communication with kea_home2 is interrupted Standby logs: Mar 25 10:10:24 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:24.129 WARN [kea-dhcp6.ha-hooks/2836.139717198915328] HA_COMMUNICATION_INTERRUPTED communication with kea_home1 is interrupted Mar 25 10:10:25 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:25.130 WARN [kea-dhcp6.ha-hooks/2836.139717207308032] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home1 (http://192.168.100.197:8000/): Connection refused Mar 25 10:10:25 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:25.130 WARN [kea-dhcp6.ha-hooks/2836.139717207308032] HA_COMMUNICATION_INTERRUPTED communication with kea_home1 is interrupted Mar 25 10:10:26 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:26.132 WARN [kea-dhcp6.ha-hooks/2836.139717190522624] HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home1 (http://192.168.100.197:8000/): Connection refused Mar 25 10:10:26 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:26.132 WARN [kea-dhcp6.ha-hooks/2836.139717190522624] HA_COMMUNICATION_INTERRUPTED communication with kea_home1 is interrupted Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.136 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_STATE_TRANSITION server transitions from PARTNER-DOWN to WAITING state, partner state is PARTNER-DOWN Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner while in WAITING state Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LOCAL_DHCP_DISABLE local DHCP service is disabled while the kea_home2 is in the WAITING state Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_STATE_TRANSITION server transitions from WAITING to SYNCING state, partner state is PARTNER-DOWN Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:27.137 INFO [kea-dhcp6.ha-hooks/2836.139717215700736] HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the partner while in SYNCING state Mar 25 10:10:27 kea_home2 kea-dhcp6[2836]: 2024-03-
Re: [Kea-users] unable to start Kea with HA - 'Invalid argument'
Found the note about the accepted URL syntax: The ``url`` schema can be ``http`` or ``https``, but since Kea version 1.9.6 the ``https`` schema requires a TLS setup. The hostname part must be an IPv4 address or an IPv6 address between square brackets, e.g. ``http://[2001:db8::1]:8080/``. Names are not accepted. About the name in TLS certificates it depends on the crypto backend so either OpenSSL or Botan and for OpenSSL the version too. Here are the notes about creating the crypto material (i.e. certificates) for tests (src/lib/asiolink/testutils/ca/doc.txt): Some critical details: - recent versions of OpenSSL requires at least 2038 bit RSA - certificate version should be 3 (enforced by Botan for leaves), if openssl creates a version 1 add an extension - RSA allows a simpler format than PKCS#8 for RSA private keys but Botan and other algorithms require PKCS#8 - some tools check the alternate subject name of the server so put a correct value in it The last point should answer to your question about what name to use in certificates. There were some discussions about self-signed certificates too: usually they are not accepted for end-entity certificates. Thanks Francis Dupont -- ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users. Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users
Re: [Kea-users] unable to start Kea with HA - 'Invalid argument'
Lexi Winter: > thanks. i assume this is a bug in the example configuration, then. actually, i just checked the example config and it does use an IP address, not a hostname, so not a bug. but i'm still curious about the correct way to configure TLS. regards, lexi. signature.asc Description: PGP signature -- ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users. Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users
Re: [Kea-users] unable to start Kea with HA - 'Invalid argument'
Kea does not support names in URL for many reasons explained in tickets asking for this. Note that IPv6 addresses in URL follow a specific not so trivial syntax and I can't find an example in the doc... Creating a ticket for this. Thanks Francis Dupont PS: https://gitlab.isc.org/isc-projects/kea/-/issues/2775#note_359268 for a long answer by Tomek about DNS resolution in Kea. -- ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users. Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users
Re: [Kea-users] unable to start Kea with HA - 'Invalid argument'
Yordanov, Damyan via Kea-users: > Kea’s HA hook library still does not support URLs, unfortunately. S. > also > https://www.mail-archive.com/kea-users@lists.isc.org/msg03105.html. thanks. i assume this is a bug in the example configuration, then. but if hostnames aren't supported, i'm unsure how to configure TLS - how do i tell the HA library what name to expect in the remote system's TLS certificate? does it use the name from the peer 'name' option? thanks, lexi. signature.asc Description: PGP signature -- ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users. Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users
Re: [Kea-users] unable to start Kea with HA - 'Invalid argument'
Hi,
Kea’s HA hook library still does not support URLs, unfortunately. S. also
https://www.mail-archive.com/kea-users@lists.isc.org/msg03105.html.
Best,
Damyan
On 25. Mar 2024, at 11:12, Lexi Winter wrote:
[You don't often get email from l...@le-fay.org. Learn why this is important at
https://aka.ms/LearnAboutSenderIdentification ]
hello,
i'm trying to configure Kea (DHCPv4, for now) in a two-server HA setup.
i've configured kea-dhcp4.conf based on the example at [0], but trying
to start it on the first server produces the error:
2024-03-25 10:03:26.811 INFO [kea-dhcp4.hooks/52504.0x23d31c212000]
HOOKS_LIBRARY_LOADED hooks library
/usr/local/lib/kea/hooks/libdhcp_lease_cmds.so successfully loaded
2024-03-25 10:03:26.820 ERROR [kea-dhcp4.ha-hooks/52504.0x23d31c212000]
HA_CONFIGURATION_FAILED failed to configure High Availability hooks library:
bad url 'http://dhcp-1.svc.eden.le-fay.org:8000/': Failed to convert string to
address 'dhcp-1.svc.eden.le-fay.org': Invalid argument for server dhcp-1
i'm not sure what 'Invalid argument' means here. the hostname does have
an IPv4 address:
# getent hosts dhcp-1.svc.eden.le-fay.org
2001:8b0:aab5:d100::f dhcp-1.svc.eden.le-fay.org
10.254.1.7dhcp-1.svc.eden.le-fay.org
this is the complete HA configuration from kea-dhcp4.conf:
"hooks-libraries": [
{
"library":
"/usr/local/lib/kea/hooks/libdhcp_lease_cmds.so",
"parameters": { }
},
{
"library": "/usr/local/lib/kea/hooks/libdhcp_ha.so",
"parameters": {
"high-availability": [
{
"this-server-name": "dhcp-1",
"mode": "load-balancing",
"heartbeat-delay": 1000,
"max-response-delay": 1,
"max-ack-delay": 5000,
"max-unacked-clients": 5,
"sync-timeout": 6,
"multi-threading": {
"enable-multi-threading": true,
"http-dedicated-listener": true,
"http-listener-threads":
0,
"http-client-threads": 0
},
"peers": [
{
"name": "dhcp-1",
"url":
"http://dhcp-1.svc.eden.le-fay.org:8000/";,
"trust-anchor":
"/usr/local/etc/kea/tls/root.pem",
"cert-file":
"/usr/local/etc/kea/tls/fullchain.pem",
"key-file":
"/usr/local/etc/kea/tls/privkey.pem",
"require-client-certs": true,
"role": "primary"
},
{
"name": "dhcp-2",
"url":
"http://dhcp-2.svc.eden.le-fay.org:8000/";,
"trust-anchor":
"/usr/local/etc/kea/tls/root.pem",
"cert-file":
"/usr/local/etc/kea/tls/fullchain.pem",
"key-file":
"/usr/local/etc/kea/tls/privkey.pem",
"require-client-certs": true,
"role":
"secondary"
}
]
}
]
}
}
],
i'm using Kea 2.4.1 on FreeBSD/arm64 15.0. can anyone point out what
i'm doing wrong here?
thanks, lexi.
[0]
https://github.com/isc-projects/kea/blob/master/doc/examples/kea4/ha-load-balancing-server1-mt-with-tls.json
--
ISC funds the development of this software with paid support subscriptions.
Contact us at https://www.isc.org/contact/ for more information.
To unsubscribe visit h
[Kea-users] unable to start Kea with HA - 'Invalid argument'
hello,
i'm trying to configure Kea (DHCPv4, for now) in a two-server HA setup.
i've configured kea-dhcp4.conf based on the example at [0], but trying
to start it on the first server produces the error:
2024-03-25 10:03:26.811 INFO [kea-dhcp4.hooks/52504.0x23d31c212000]
HOOKS_LIBRARY_LOADED hooks library
/usr/local/lib/kea/hooks/libdhcp_lease_cmds.so successfully loaded
2024-03-25 10:03:26.820 ERROR [kea-dhcp4.ha-hooks/52504.0x23d31c212000]
HA_CONFIGURATION_FAILED failed to configure High Availability hooks library:
bad url 'http://dhcp-1.svc.eden.le-fay.org:8000/': Failed to convert string to
address 'dhcp-1.svc.eden.le-fay.org': Invalid argument for server dhcp-1
i'm not sure what 'Invalid argument' means here. the hostname does have
an IPv4 address:
# getent hosts dhcp-1.svc.eden.le-fay.org
2001:8b0:aab5:d100::f dhcp-1.svc.eden.le-fay.org
10.254.1.7dhcp-1.svc.eden.le-fay.org
this is the complete HA configuration from kea-dhcp4.conf:
"hooks-libraries": [
{
"library":
"/usr/local/lib/kea/hooks/libdhcp_lease_cmds.so",
"parameters": { }
},
{
"library": "/usr/local/lib/kea/hooks/libdhcp_ha.so",
"parameters": {
"high-availability": [
{
"this-server-name": "dhcp-1",
"mode": "load-balancing",
"heartbeat-delay": 1000,
"max-response-delay": 1,
"max-ack-delay": 5000,
"max-unacked-clients": 5,
"sync-timeout": 6,
"multi-threading": {
"enable-multi-threading": true,
"http-dedicated-listener": true,
"http-listener-threads": 0,
"http-client-threads": 0
},
"peers": [
{
"name":
"dhcp-1",
"url":
"http://dhcp-1.svc.eden.le-fay.org:8000/";,
"trust-anchor":
"/usr/local/etc/kea/tls/root.pem",
"cert-file":
"/usr/local/etc/kea/tls/fullchain.pem",
"key-file":
"/usr/local/etc/kea/tls/privkey.pem",
"require-client-certs": true,
"role":
"primary"
},
{
"name":
"dhcp-2",
"url":
"http://dhcp-2.svc.eden.le-fay.org:8000/";,
"trust-anchor":
"/usr/local/etc/kea/tls/root.pem",
"cert-file":
"/usr/local/etc/kea/tls/fullchain.pem",
"key-file":
"/usr/local/etc/kea/tls/privkey.pem",
"require-client-certs": true,
"role":
"secondary"
}
]
}
]
}
}
],
i'm using Kea 2.4.1 on FreeBSD/arm64 15.0. can anyone point out what
i'm doing wrong here?
thanks, lexi.
[0]
https://github.com/isc-projects/kea/blob/master/doc/examples/kea4/ha-load-balancing-server1-mt-with-tls.json
signature.asc
Description: PGP signature
--
ISC funds the development of this software with paid support subscriptions.
Contact us at https://www.isc.org/contact/ for more information.
To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.
Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users
