Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
Thank you, Darren! Best Regards, Yu From: Kea-users on behalf of Darren Ankney Date: Friday, April 5, 2024 at 3:17 PM To: Kea user's list Subject: Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors Hi Yu, That is correct. You could set one of them to maintenance mode to perform some maintenance. There would be no way to do this if the server were down, however, and so you wouldn't be able to prevent it resuming service in the event of an outage. Thank you, Darren Ankney On Fri, Apr 5, 2024 at 11:56 AM Xiao, Yu (CCI-Atlanta) via Kea-users wrote: > > Hi Darren, > > > > But imagine you have a maintenance, and you don’t want the primary revert > back until the end of maintenance window. I found this API > “ha-maintenance-start” seems can be used for such situation. > > > > > > > > Best Regards, > > Yu > > > > > > From: Kea-users on behalf of Darren Ankney > > Date: Friday, April 5, 2024 at 5:40 AM > To: Kea user's list > Subject: Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors > > Hi Yu, > > > > I have another question, I re-enabled the primary kea-01 again, after that, > > I noticed Kea-01 seems to have retaken the role of the primary again > > automatically. Is there a way we can keep the standby continue serve as > > temporary primary instead of revert automatically? > > Not that I am aware of. I think the configuration designated primary > will take over if all of the check boxes are ticked. > > > Also, is there a convenient way to check both hosts which one is serving > > the DHCP lease now instead of checking the details of the debug logs? I > > don’t remember see those details from the 2.4.1 documentation. > > You can check with the API: > https://urldefense.com/v3/__https://kea.readthedocs.io/en/kea-2.4.1/api.html*status-get__;Iw!!Hit2Ag!2xl9qbmkix1hSKUZDE9pxZvhL3AjsqTjNtXpsoyCPxsUTJ_6pI5KG-tkxPD2Kw3AvHq5eDXMy-OEbXhoZu8g$<https://urldefense.com/v3/__https:/kea.readthedocs.io/en/kea-2.4.1/api.html*status-get__;Iw!!Hit2Ag!2xl9qbmkix1hSKUZDE9pxZvhL3AjsqTjNtXpsoyCPxsUTJ_6pI5KG-tkxPD2Kw3AvHq5eDXMy-OEbXhoZu8g$> > > Thank you, > Darren Ankney > -- > ISC funds the development of this software with paid support subscriptions. > Contact us at > https://urldefense.com/v3/__https://www.isc.org/contact/__;!!Hit2Ag!2xl9qbmkix1hSKUZDE9pxZvhL3AjsqTjNtXpsoyCPxsUTJ_6pI5KG-tkxPD2Kw3AvHq5eDXMy-OEbXm6YOo0$<https://urldefense.com/v3/__https:/www.isc.org/contact/__;!!Hit2Ag!2xl9qbmkix1hSKUZDE9pxZvhL3AjsqTjNtXpsoyCPxsUTJ_6pI5KG-tkxPD2Kw3AvHq5eDXMy-OEbXm6YOo0$> > for more information. > > To unsubscribe visit > https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!2xl9qbmkix1hSKUZDE9pxZvhL3AjsqTjNtXpsoyCPxsUTJ_6pI5KG-tkxPD2Kw3AvHq5eDXMy-OEbYNJaLhi$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!2xl9qbmkix1hSKUZDE9pxZvhL3AjsqTjNtXpsoyCPxsUTJ_6pI5KG-tkxPD2Kw3AvHq5eDXMy-OEbYNJaLhi$> > . > > Kea-users mailing list > Kea-users@lists.isc.org > https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!2xl9qbmkix1hSKUZDE9pxZvhL3AjsqTjNtXpsoyCPxsUTJ_6pI5KG-tkxPD2Kw3AvHq5eDXMy-OEbYNJaLhi$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!2xl9qbmkix1hSKUZDE9pxZvhL3AjsqTjNtXpsoyCPxsUTJ_6pI5KG-tkxPD2Kw3AvHq5eDXMy-OEbYNJaLhi$> > > -- > ISC funds the development of this software with paid support subscriptions. > Contact us at > https://urldefense.com/v3/__https://www.isc.org/contact/__;!!Hit2Ag!wvQfwO6CRGpPWHjUJ2HffGP4G6AENAYOKM2EwPd4xLy-OQSYjxr6tMEXUQrnagzPgEgGimSasXEh2EjHTylS$<https://urldefense.com/v3/__https:/www.isc.org/contact/__;!!Hit2Ag!wvQfwO6CRGpPWHjUJ2HffGP4G6AENAYOKM2EwPd4xLy-OQSYjxr6tMEXUQrnagzPgEgGimSasXEh2EjHTylS$> > for more information. > > To unsubscribe visit > https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!wvQfwO6CRGpPWHjUJ2HffGP4G6AENAYOKM2EwPd4xLy-OQSYjxr6tMEXUQrnagzPgEgGimSasXEh2OvVL-Wv$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!wvQfwO6CRGpPWHjUJ2HffGP4G6AENAYOKM2EwPd4xLy-OQSYjxr6tMEXUQrnagzPgEgGimSasXEh2OvVL-Wv$> > . > > Kea-users mailing list > Kea-users@lists.isc.org > https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!wvQfwO6CRGpPWHjUJ2HffGP4G6AENAYOKM2EwPd4xLy-OQSYjxr6tMEXUQrnagzPgEgGimSasXEh2OvVL-Wv$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!wvQfwO6CRGpPWHjUJ2HffGP4G6AENAYOKM2EwPd4xLy-OQSYjxr6tMEXUQrnagzPgEgGimSasXEh2OvVL-Wv$> -- ISC funds the development of this software with paid support subscriptions. Contact us at https://urldef
Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
Hi Yu, That is correct. You could set one of them to maintenance mode to perform some maintenance. There would be no way to do this if the server were down, however, and so you wouldn't be able to prevent it resuming service in the event of an outage. Thank you, Darren Ankney On Fri, Apr 5, 2024 at 11:56 AM Xiao, Yu (CCI-Atlanta) via Kea-users wrote: > > Hi Darren, > > > > But imagine you have a maintenance, and you don’t want the primary revert > back until the end of maintenance window. I found this API > “ha-maintenance-start” seems can be used for such situation. > > > > > > > > Best Regards, > > Yu > > > > > > From: Kea-users on behalf of Darren Ankney > > Date: Friday, April 5, 2024 at 5:40 AM > To: Kea user's list > Subject: Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors > > Hi Yu, > > > > I have another question, I re-enabled the primary kea-01 again, after that, > > I noticed Kea-01 seems to have retaken the role of the primary again > > automatically. Is there a way we can keep the standby continue serve as > > temporary primary instead of revert automatically? > > Not that I am aware of. I think the configuration designated primary > will take over if all of the check boxes are ticked. > > > Also, is there a convenient way to check both hosts which one is serving > > the DHCP lease now instead of checking the details of the debug logs? I > > don’t remember see those details from the 2.4.1 documentation. > > You can check with the API: > https://urldefense.com/v3/__https://kea.readthedocs.io/en/kea-2.4.1/api.html*status-get__;Iw!!Hit2Ag!2xl9qbmkix1hSKUZDE9pxZvhL3AjsqTjNtXpsoyCPxsUTJ_6pI5KG-tkxPD2Kw3AvHq5eDXMy-OEbXhoZu8g$ > > Thank you, > Darren Ankney > -- > ISC funds the development of this software with paid support subscriptions. > Contact us at > https://urldefense.com/v3/__https://www.isc.org/contact/__;!!Hit2Ag!2xl9qbmkix1hSKUZDE9pxZvhL3AjsqTjNtXpsoyCPxsUTJ_6pI5KG-tkxPD2Kw3AvHq5eDXMy-OEbXm6YOo0$ > for more information. > > To unsubscribe visit > https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!2xl9qbmkix1hSKUZDE9pxZvhL3AjsqTjNtXpsoyCPxsUTJ_6pI5KG-tkxPD2Kw3AvHq5eDXMy-OEbYNJaLhi$ > . > > Kea-users mailing list > Kea-users@lists.isc.org > https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!2xl9qbmkix1hSKUZDE9pxZvhL3AjsqTjNtXpsoyCPxsUTJ_6pI5KG-tkxPD2Kw3AvHq5eDXMy-OEbYNJaLhi$ > > -- > ISC funds the development of this software with paid support subscriptions. > Contact us at https://www.isc.org/contact/ for more information. > > To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users. > > Kea-users mailing list > Kea-users@lists.isc.org > https://lists.isc.org/mailman/listinfo/kea-users -- ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users. Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users
Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
Hi Darren, But imagine you have a maintenance, and you don’t want the primary revert back until the end of maintenance window. I found this API “ha-maintenance-start” seems can be used for such situation. Best Regards, Yu From: Kea-users on behalf of Darren Ankney Date: Friday, April 5, 2024 at 5:40 AM To: Kea user's list Subject: Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors Hi Yu, > I have another question, I re-enabled the primary kea-01 again, after that, I > noticed Kea-01 seems to have retaken the role of the primary again > automatically. Is there a way we can keep the standby continue serve as > temporary primary instead of revert automatically? Not that I am aware of. I think the configuration designated primary will take over if all of the check boxes are ticked. > Also, is there a convenient way to check both hosts which one is serving the > DHCP lease now instead of checking the details of the debug logs? I don’t > remember see those details from the 2.4.1 documentation. You can check with the API: https://urldefense.com/v3/__https://kea.readthedocs.io/en/kea-2.4.1/api.html*status-get__;Iw!!Hit2Ag!2xl9qbmkix1hSKUZDE9pxZvhL3AjsqTjNtXpsoyCPxsUTJ_6pI5KG-tkxPD2Kw3AvHq5eDXMy-OEbXhoZu8g$<https://urldefense.com/v3/__https:/kea.readthedocs.io/en/kea-2.4.1/api.html*status-get__;Iw!!Hit2Ag!2xl9qbmkix1hSKUZDE9pxZvhL3AjsqTjNtXpsoyCPxsUTJ_6pI5KG-tkxPD2Kw3AvHq5eDXMy-OEbXhoZu8g$> Thank you, Darren Ankney -- ISC funds the development of this software with paid support subscriptions. Contact us at https://urldefense.com/v3/__https://www.isc.org/contact/__;!!Hit2Ag!2xl9qbmkix1hSKUZDE9pxZvhL3AjsqTjNtXpsoyCPxsUTJ_6pI5KG-tkxPD2Kw3AvHq5eDXMy-OEbXm6YOo0$<https://urldefense.com/v3/__https:/www.isc.org/contact/__;!!Hit2Ag!2xl9qbmkix1hSKUZDE9pxZvhL3AjsqTjNtXpsoyCPxsUTJ_6pI5KG-tkxPD2Kw3AvHq5eDXMy-OEbXm6YOo0$> for more information. To unsubscribe visit https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!2xl9qbmkix1hSKUZDE9pxZvhL3AjsqTjNtXpsoyCPxsUTJ_6pI5KG-tkxPD2Kw3AvHq5eDXMy-OEbYNJaLhi$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!2xl9qbmkix1hSKUZDE9pxZvhL3AjsqTjNtXpsoyCPxsUTJ_6pI5KG-tkxPD2Kw3AvHq5eDXMy-OEbYNJaLhi$> . Kea-users mailing list Kea-users@lists.isc.org https://urldefense.com/v3/__https://lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!2xl9qbmkix1hSKUZDE9pxZvhL3AjsqTjNtXpsoyCPxsUTJ_6pI5KG-tkxPD2Kw3AvHq5eDXMy-OEbYNJaLhi$<https://urldefense.com/v3/__https:/lists.isc.org/mailman/listinfo/kea-users__;!!Hit2Ag!2xl9qbmkix1hSKUZDE9pxZvhL3AjsqTjNtXpsoyCPxsUTJ_6pI5KG-tkxPD2Kw3AvHq5eDXMy-OEbYNJaLhi$> -- ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users. Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users
Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
Hi Yu, > I have another question, I re-enabled the primary kea-01 again, after that, I > noticed Kea-01 seems to have retaken the role of the primary again > automatically. Is there a way we can keep the standby continue serve as > temporary primary instead of revert automatically? Not that I am aware of. I think the configuration designated primary will take over if all of the check boxes are ticked. > Also, is there a convenient way to check both hosts which one is serving the > DHCP lease now instead of checking the details of the debug logs? I don’t > remember see those details from the 2.4.1 documentation. You can check with the API: https://kea.readthedocs.io/en/kea-2.4.1/api.html#status-get Thank you, Darren Ankney -- ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users. Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users
Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
Hi Yu, Basically you are not waiting for clients to go down. Clients will attempt to renew at lease half-life. "max-unacked-clients" is a way of confirming that the other server is actually down. In certain scenarios it would be possible that the two servers couldn't see each other but COULD see the clients. If they were both answering the clients, that could lead to duplicate IP address assignment and the like. Thank you, Darren Ankney On Wed, Apr 3, 2024 at 5:32 PM Xiao, Yu (CCI-Atlanta) wrote: > > Hi Darren, > > > > Thank you for your information. I adjusted the parameter and now it is > working. But I have a question, why should we need this "max-unacked-clients" > parameter? Because imagine we have 1k devices in this network, and they all > have their IP assigned by dhcp_1, then suddenly dhcp_1 went down, but at this > moment, all devices still have their assigned IPs. Then if we design dhcp_2 > to become primary after losing sync time, we may be able to bring dhcp_2 to > primary before T2, so there won’t be large number of devices offline. But if > we wait until offline devices observed, then it could be too late and many > devices already offline. > > > > > > > > Best Regards, > > Yu > > > > > > From: Kea-users on behalf of Xiao, Yu > (CCI-Atlanta) via Kea-users > Date: Wednesday, April 3, 2024 at 9:12 AM > To: Darren Ankney , Kea user's list > > Cc: Xiao, Yu (CCI-Atlanta) > Subject: Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors > > Hi Darren, > > > > Thank you for the explanation. Let me investigate further. > > > > > > Best Regards, > > Yu > > > > > > From: Darren Ankney > Date: Wednesday, April 3, 2024 at 9:06 AM > To: Kea user's list > Cc: Xiao, Yu (CCI-Atlanta) > Subject: Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors > > Hi Yu, > > Look in your logs for messages containing > HA_COMMUNICATION_INTERRUPTED_CLIENT4_UNACKED. These messages indicate > how many clients are still needed before the failover will occur. > Your settings specify: > > "max-ack-delay": 300, > "max-unacked-clients": 5, > > max-ack-delay is specified in milliseconds so pretty much every client > that reaches one in the "elapsed time" option in DHCPv6 will meet the > criteria of an "unacked" client. Once the sixth client shows up, > failover should occur. That log message should repeat with each new > client being tracked giving statistics about how many more are needed. > > Thank you, > Darren Ankney > > > On Tue, Apr 2, 2024 at 12:39 PM Xiao, Yu (CCI-Atlanta) via Kea-users > wrote: > > > > Hi experts, > > > > > > > > I have tested further in my lab and found the answers to my questions. HA > > works when I don’t enable the control agent. As for the debug log, I think > > there’s a potential document bug for 2.4.1 document. Cause you can’t assign > > the log to /var/log folder, you need to assign it to /tmp/ folder. > > > > > > > > I found a problem on myside on firewall, after I fixed it. Now seems the HA > > is working. But still having one problem. When I used “systemctl stop > > kea-dhcp6” on the primary, for example, kea01, then the standby DHCPv6 > > server kea03 won’t process the DHCP packets from clients. I waited for more > > time, but the behaviors won’t change. The standby server just drop those > > DHCP packets from clients due to “dropping query to be processed by > > another server”. > > > > > > > > According to Kea documentation, this happens when: > > > > > > > > “%1: dropping query to be processed by another server > > > > > > > > This debug message is issued when the received DHCPv6 query is dropped by > > this server because it should be served by another server. This is the case > > when the remote server was designated to process the packet as a result of > > load balancing or because it is a primary server in the hot standby > > configuration. The argument provides client identification information > > retrieved from the query.” > > > > > > > > But now since the primary service is down, so the standby server should > > become the primary one, right? But seems not the case. Also from the debug > > logs, I can’t see the current status of the remaining server, technically > > we should be able to see its status changed from standby to primary, right? > > But I only see the role information from the logs before I shut down the > > kea service
Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
Hi Darren,
Thank you for the explanation. Let me investigate further.
Best Regards,
Yu
From: Darren Ankney
Date: Wednesday, April 3, 2024 at 9:06 AM
To: Kea user's list
Cc: Xiao, Yu (CCI-Atlanta)
Subject: Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
Hi Yu,
Look in your logs for messages containing
HA_COMMUNICATION_INTERRUPTED_CLIENT4_UNACKED. These messages indicate
how many clients are still needed before the failover will occur.
Your settings specify:
"max-ack-delay": 300,
"max-unacked-clients": 5,
max-ack-delay is specified in milliseconds so pretty much every client
that reaches one in the "elapsed time" option in DHCPv6 will meet the
criteria of an "unacked" client. Once the sixth client shows up,
failover should occur. That log message should repeat with each new
client being tracked giving statistics about how many more are needed.
Thank you,
Darren Ankney
On Tue, Apr 2, 2024 at 12:39 PM Xiao, Yu (CCI-Atlanta) via Kea-users
wrote:
>
> Hi experts,
>
>
>
> I have tested further in my lab and found the answers to my questions. HA
> works when I don’t enable the control agent. As for the debug log, I think
> there’s a potential document bug for 2.4.1 document. Cause you can’t assign
> the log to /var/log folder, you need to assign it to /tmp/ folder.
>
>
>
> I found a problem on myside on firewall, after I fixed it. Now seems the HA
> is working. But still having one problem. When I used “systemctl stop
> kea-dhcp6” on the primary, for example, kea01, then the standby DHCPv6 server
> kea03 won’t process the DHCP packets from clients. I waited for more time,
> but the behaviors won’t change. The standby server just drop those DHCP
> packets from clients due to “dropping query to be processed by another
> server”.
>
>
>
> According to Kea documentation, this happens when:
>
>
>
> “%1: dropping query to be processed by another server
>
>
>
> This debug message is issued when the received DHCPv6 query is dropped by
> this server because it should be served by another server. This is the case
> when the remote server was designated to process the packet as a result of
> load balancing or because it is a primary server in the hot standby
> configuration. The argument provides client identification information
> retrieved from the query.”
>
>
>
> But now since the primary service is down, so the standby server should
> become the primary one, right? But seems not the case. Also from the debug
> logs, I can’t see the current status of the remaining server, technically we
> should be able to see its status changed from standby to primary, right? But
> I only see the role information from the logs before I shut down the kea
> service on primary server.
>
>
>
> Config:
>
> # HA related hooks configuration
>
> "hooks-libraries": [{
>
> "library": "/usr/lib64/kea/hooks/libdhcp_lease_cmds.so",
>
> "parameters": { }
>
> }, {
>
> "library": "/usr/lib64/kea/hooks/libdhcp_ha.so",
>
> "parameters": {
>
> "high-availability": [{
>
> "this-server-name": "kea_home3",
>
> "mode": "hot-standby",
>
> "heartbeat-delay": 100,
>
> "max-response-delay": 200,
>
> "max-ack-delay": 300,
>
> "max-unacked-clients": 5,
>
> "peers": [{
>
> "name": "kea_home1",
>
> "url":
> https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvY8Qy6Ik$<https://urldefense.com/v3/__http:/192.168.100.197:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvY8Qy6Ik$>
> ,
>
> "role": "primary",
>
> "auto-failover": true
>
> }, {
>
> "name": "kea_home3",
>
> "url":
> https://urldefense.com/v3/__http://192.168.100.70:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvU0f5KAX$<https://urldefense.com/v3/__http:/192.168.100.70:8000/__;!!Hit2Ag!yteZar1VMg4V2QncUrW9qHJR12Fao76aOPAzxE4bRwaXFdrveqPfJanYcIlzufpSJtigEMc_AtWEvU0f5KAX$>
> ,
>
> "role": "standby",
>
> "aut
Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
903754057472]
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home1
> (http://192.168.100.197:8000/): Connection refused
>
> 2024-04-02 12:03:41.696 WARN [kea-dhcp6.ha-hooks/15314.139903754057472]
> HA_COMMUNICATION_INTERRUPTED communication with kea_home1 is interrupted
>
> 2024-04-02 12:03:41.773 DEBUG [kea-dhcp6.packets/15314.139903918055424]
> DHCP6_BUFFER_RECEIVED received buffer from fe80::c40b:ebff:fed1:7298:546 to
> ff02::1:2:0 over interface ens18
>
> 2024-04-02 12:03:41.773 DEBUG [kea-dhcp6.callouts/15314.139903787628288]
> HOOKS_CALLOUTS_BEGIN begin all callouts for hook buffer6_receive
>
> 2024-04-02 12:03:41.773 DEBUG [kea-dhcp6.ha-hooks/15314.139903787628288]
> HA_BUFFER6_RECEIVE_NOT_FOR_US
> duid=[00:04:a3:35:01:e3:85:15:e1:76:3e:47:e3:b0:c5:f8:55:10], tid=0x65a570:
> dropping query to be processed by another server
>
> 2024-04-02 12:03:41.773 DEBUG [kea-dhcp6.callouts/15314.139903787628288]
> HOOKS_CALLOUT_CALLED hooks library with index 2 has called a callout on hook
> buffer6_receive that has address 0x7f3de3bc3060 (callout duration: 0.161 ms)
>
> 2024-04-02 12:03:41.773 DEBUG [kea-dhcp6.callouts/15314.139903787628288]
> HOOKS_CALLOUTS_COMPLETE completed callouts for hook buffer6_receive (total
> callouts duration: 0.161 ms)
>
> 2024-04-02 12:03:41.773 DEBUG [kea-dhcp6.hooks/15314.139903787628288]
> DHCP6_HOOK_BUFFER_RCVD_DROP received buffer from fe80::c40b:ebff:fed1:7298 to
> ff02::1:2 over interface ens18 was dropped because a callout set the drop flag
>
> 2024-04-02 12:03:42.774 DEBUG [kea-dhcp6.http/15314.139903918055424]
> HTTP_CLIENT_REQUEST_SEND sending HTTP request POST / HTTP/1.1 to
> http://192.168.100.197:8000/
>
> 2024-04-02 12:03:42.774 DEBUG [kea-dhcp6.http/15314.139903918055424]
> HTTP_CLIENT_REQUEST_SEND_DETAILS detailed information about request sent to
> http://192.168.100.197:8000/:
>
> POST / HTTP/1.1
>
> Host: 192.168.100.197
>
> Content-Length: 53
>
> Content-Type: application/json
>
>
>
> { "command": "ha-heartbeat", "service": [ "dhcp6" ] }
>
> 2024-04-02 12:03:42.775 DEBUG [kea-dhcp6.http/15314.139903728879360]
> HTTP_BAD_SERVER_RESPONSE_RECEIVED bad response received when communicating
> with http://192.168.100.197:8000/: Connection refused
>
> 2024-04-02 12:03:42.775 WARN [kea-dhcp6.ha-hooks/15314.139903728879360]
> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home1
> (http://192.168.100.197:8000/): Connection refused
>
> 2024-04-02 12:03:42.775 WARN [kea-dhcp6.ha-hooks/15314.139903728879360]
> HA_COMMUNICATION_INTERRUPTED communication with kea_home1 is interrupted
>
> 2024-04-02 12:03:43.776 DEBUG [kea-dhcp6.http/15314.139903918055424]
> HTTP_CLIENT_REQUEST_SEND sending HTTP request POST / HTTP/1.1 to
> http://192.168.100.197:8000/
>
> 2024-04-02 12:03:43.776 DEBUG [kea-dhcp6.http/15314.139903918055424]
> HTTP_CLIENT_REQUEST_SEND_DETAILS detailed information about request sent to
> http://192.168.100.197:8000/:
>
> POST / HTTP/1.1
>
> Host: 192.168.100.197
>
> Content-Length: 53
>
> Content-Type: application/json
>
>
>
> Server status from debug logs:
>
>
>
> 2024-04-02 11:50:32.503 DEBUG [kea-dhcp6.dhcp6/15314.139903918055424]
> DHCP6_CONFIG_START DHCPv6 server is processing the following configuration: {
> "control-socket": { "socket-name": "/tmp/kea6-ctrl-socket", "socket-type":
> "unix" }, "hooks-libraries": [ { "library":
> "/usr/lib64/kea/hooks/libdhcp_lease_cmds.so", "parameters": { } }, {
> "library": "/usr/lib64/kea/hooks/libdhcp_ha.so", "parameters": {
> "high-availability": [ { "heartbeat-delay": 100, "max-ack-delay": 300,
> "max-response-delay": 200, "max-unacked-clients": 5, "mode": "hot-standby",
> "peers": [ { "auto-failover": true, "name": "kea_home1", "role": "primary",
> "url": http://192.168.100.197:8000/ }, { "auto-failover": true, "name":
> "kea_home3", "role": "standby", "url": http://192.168.100.70:8000/ } ],
> "this-server-name": "kea_home3" } ] } } ], "interfaces-config": {
> "interfaces": [ "ens18" ], "service-sockets-require-all": true },
> "lease-database": { "name": "/var/lib/kea/dhcp6.leases", "persist": true,
> "type": "memfile" }, "loggers": [ { &
Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
ebff:fed1:7298 to
ff02::1:2 over interface ens18 was dropped because a callout set the drop flag
2024-04-02 12:03:42.774 DEBUG [kea-dhcp6.http/15314.139903918055424]
HTTP_CLIENT_REQUEST_SEND sending HTTP request POST / HTTP/1.1 to
http://192.168.100.197:8000/
2024-04-02 12:03:42.774 DEBUG [kea-dhcp6.http/15314.139903918055424]
HTTP_CLIENT_REQUEST_SEND_DETAILS detailed information about request sent to
http://192.168.100.197:8000/:
POST / HTTP/1.1
Host: 192.168.100.197
Content-Length: 53
Content-Type: application/json
{ "command": "ha-heartbeat", "service": [ "dhcp6" ] }
2024-04-02 12:03:42.775 DEBUG [kea-dhcp6.http/15314.139903728879360]
HTTP_BAD_SERVER_RESPONSE_RECEIVED bad response received when communicating with
http://192.168.100.197:8000/: Connection refused
2024-04-02 12:03:42.775 WARN [kea-dhcp6.ha-hooks/15314.139903728879360]
HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home1
(http://192.168.100.197:8000/): Connection refused
2024-04-02 12:03:42.775 WARN [kea-dhcp6.ha-hooks/15314.139903728879360]
HA_COMMUNICATION_INTERRUPTED communication with kea_home1 is interrupted
2024-04-02 12:03:43.776 DEBUG [kea-dhcp6.http/15314.139903918055424]
HTTP_CLIENT_REQUEST_SEND sending HTTP request POST / HTTP/1.1 to
http://192.168.100.197:8000/
2024-04-02 12:03:43.776 DEBUG [kea-dhcp6.http/15314.139903918055424]
HTTP_CLIENT_REQUEST_SEND_DETAILS detailed information about request sent to
http://192.168.100.197:8000/:
POST / HTTP/1.1
Host: 192.168.100.197
Content-Length: 53
Content-Type: application/json
Server status from debug logs:
2024-04-02 11:50:32.503 DEBUG [kea-dhcp6.dhcp6/15314.139903918055424]
DHCP6_CONFIG_START DHCPv6 server is processing the following configuration: {
"control-socket": { "socket-name": "/tmp/kea6-ctrl-socket", "socket-type":
"unix" }, "hooks-libraries": [ { "library":
"/usr/lib64/kea/hooks/libdhcp_lease_cmds.so", "parameters": { } }, {
"library": "/usr/lib64/kea/hooks/libdhcp_ha.so", "parameters": {
"high-availability": [ { "heartbeat-delay": 100, "max-ack-delay": 300,
"max-response-delay": 200, "max-unacked-clients": 5, "mode": "hot-standby",
"peers": [ { "auto-failover": true, "name": "kea_home1", "role": "primary",
"url": http://192.168.100.197:8000/ }, { "auto-failover": true, "name":
"kea_home3", "role": "standby", "url": http://192.168.100.70:8000/ } ],
"this-server-name": "kea_home3" } ] } } ], "interfaces-config": { "interfaces":
[ "ens18" ], "service-sockets-require-all": true }, "lease-database": { "name":
"/var/lib/kea/dhcp6.leases", "persist": true, "type": "memfile" }, "loggers": [
{ "debuglevel": 99, "name": "kea-dhcp6", "output_options": [ { "maxsize":
4048, "maxver": 8, "output": "/tmp/kea-debug.log" } ], "severity": "DEBUG"
} ], "preferred-lifetime": 300, "rebind-timer": 200, "renew-timer": 100,
"subnet6": [ { "id": 1, "interface": "ens18", "pools": [ { "pool":
"fd74:5656:15e2:10::200-fd74:5656:15e2:10::" } ], "subnet":
"fd74:5656:15e2:10::/64" } ], "valid-lifetime": 400 }
Best Regards,
Yu
From: Kea-users on behalf of Xiao, Yu
(CCI-Atlanta) via Kea-users
Date: Monday, April 1, 2024 at 5:18 PM
To: Kea user's list
Cc: Xiao, Yu (CCI-Atlanta)
Subject: Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
Hi Darren/experts,
I have changed to 8000 port, but still the HA is not working. I have the
following questions:
1. Do I need to enable kea-ctrl-agent to make HA work? Or they are separate?
2. I configured the logger related configuration, but I don’t see the log
file has been created, did I miss something? Here’s my config:
"loggers": [
{
"name": "kea-dhcp6",
"output_options": [
{
"output": "/var/log/kea-debug.log",
"maxver": 8,
"maxsize": 204800,
"flush": true,
"pattern": "%d{%j %H:%M:%S.%q} %c %m\n"
}
],
"severity": "DEBUG",
"debuglevel": 99
}
]
[root@kea_home3 kea]# systemctl status kea-dhcp6.service
● kea-dhcp6.service - Kea DHCPv6 S
Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
Hi Darren/experts,
I have changed to 8000 port, but still the HA is not working. I have the
following questions:
1. Do I need to enable kea-ctrl-agent to make HA work? Or they are separate?
2. I configured the logger related configuration, but I don’t see the log
file has been created, did I miss something? Here’s my config:
"loggers": [
{
"name": "kea-dhcp6",
"output_options": [
{
"output": "/var/log/kea-debug.log",
"maxver": 8,
"maxsize": 204800,
"flush": true,
"pattern": "%d{%j %H:%M:%S.%q} %c %m\n"
}
],
"severity": "DEBUG",
"debuglevel": 99
}
]
[root@kea_home3 kea]# systemctl status kea-dhcp6.service
● kea-dhcp6.service - Kea DHCPv6 Service
Loaded: loaded (/usr/lib/systemd/system/kea-dhcp6.service; enabled; vendor
preset: disabled)
Active: active (running) since Mon 2024-04-01 17:08:48 EDT; 4min 30s ago
Docs: man:kea-dhcp6(8)
Main PID: 4085 (kea-dhcp6)
Tasks: 13 (limit: 10999)
Memory: 4.3M
CGroup: /system.slice/kea-dhcp6.service
└─4085 /usr/sbin/kea-dhcp6 -c /etc/kea/kea-dhcp6.conf
Here’s a kea folder but no kea-debug.log as I defined:
[root@kea_home3 kea]# ll /var/log/ |grep kea
drwxr-x---. 2 keakea 6 Nov 23 14:06 kea
Best Regards,
Yu
From: Kea-users on behalf of Xiao, Yu
(CCI-Atlanta) via Kea-users
Date: Friday, March 29, 2024 at 10:47 AM
To: Kea user's list
Cc: Xiao, Yu (CCI-Atlanta)
Subject: Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
Thank you for the advice, Darren. Let me test on them.
Best Regards,
Yu
From: Kea-users on behalf of Darren Ankney
Date: Friday, March 29, 2024 at 6:58 AM
To: Kea user's list
Subject: Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
Hi Yu,
You will need to enable multi-threading dedicated listener for HA as
described here:
https://urldefense.com/v3/__https://kea.readthedocs.io/en/kea-2.4.1/arm/hooks.html*multi-threaded-configuration-ha-mt__;Iw!!Hit2Ag!zoh5xPnvNfIYnDVa8fKOaBy5jIIQIMWT5qUv1AwVKSJJWw6Sv4JGdCWpcT1787WpOLtTS--cCCJ1garUIhR_$<https://urldefense.com/v3/__https:/kea.readthedocs.io/en/kea-2.4.1/arm/hooks.html*multi-threaded-configuration-ha-mt__;Iw!!Hit2Ag!zoh5xPnvNfIYnDVa8fKOaBy5jIIQIMWT5qUv1AwVKSJJWw6Sv4JGdCWpcT1787WpOLtTS--cCCJ1garUIhR_$>
or alter your "url" in each in the peers blocks to port 8001 so that
they communicate via the kea-ctrl-agent. I think that should fix it.
Thank you,
Darren Ankney
On Tue, Mar 26, 2024 at 7:58 AM Xiao, Yu (CCI-Atlanta) via Kea-users
wrote:
>
> Hi Darren,
>
>
>
> I am using 2.4.1.
>
>
>
>
>
> Best Regards,
>
> Yu
>
>
>
>
>
> From: Kea-users on behalf of Darren Ankney
>
> Date: Tuesday, March 26, 2024 at 6:03 AM
> To: Kea user's list
> Subject: [EXTERNAL] Re: [Kea-users] kea-2.4.1 // Strange HA related errors
>
> Hello Yu,
>
> What version of Kea are you running on these servers, out of curiosity?
>
> Thank you,
> Darren Ankney
>
> On Mon, Mar 25, 2024 at 10:55 AM Xiao, Yu (CCI-Atlanta) via Kea-users
> wrote:
> >
> > Hi experts,
> >
> >
> >
> > I have configured two VMs in the same hypervisor as hot-standby mode HA. I
> > believe they are successfully communicating with each other with heart beat
> > packets, as we can see the primary VM kea-1 has successfully received the
> > “ha-heartbeat” from the standby VM kea-2 in green logs. But the red logs
> > indicate that the ha_hooks think the HA heartbeat communications failed due
> > to “no route”. But this is a LAN network, and there’s indeed route
> > installed as we can see below and we can ping the 69 ip.
> >
> >
> >
> > [yxiao322@kea_home1 ~]$ ip route
> >
> > 192.168.100.0/24 dev ens18 proto kernel scope link src 192.168.100.197
> > metric 100 <<< This should be the route to 192.168.100.69
> >
> > 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1
> > linkdown
> >
> >
> >
> > [root@kea_home1 yxiao322]# ping 192.168.100.69
> >
> > PING 192.168.100.69 (192.168.100.69) 56(84) bytes of data.
> >
> > 64 bytes from 192.168.100.69: icmp_seq=1 ttl=64 time=0.342 ms
> >
> > 64 bytes from 192.168.100.69: icmp_seq=2 ttl=64 time=0.302 ms
> >
> > 64 bytes from 192.168.100.69: icmp_seq=3 ttl=64 time=0.236 ms
> >
> > ^Z
> >
> > [1]+ Stopped ping 192.168.100.69
> >
> >
> >
Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
Thank you for the advice, Darren. Let me test on them. Best Regards, Yu From: Kea-users on behalf of Darren Ankney Date: Friday, March 29, 2024 at 6:58 AM To: Kea user's list Subject: Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors Hi Yu, You will need to enable multi-threading dedicated listener for HA as described here: https://urldefense.com/v3/__https://kea.readthedocs.io/en/kea-2.4.1/arm/hooks.html*multi-threaded-configuration-ha-mt__;Iw!!Hit2Ag!zoh5xPnvNfIYnDVa8fKOaBy5jIIQIMWT5qUv1AwVKSJJWw6Sv4JGdCWpcT1787WpOLtTS--cCCJ1garUIhR_$<https://urldefense.com/v3/__https:/kea.readthedocs.io/en/kea-2.4.1/arm/hooks.html*multi-threaded-configuration-ha-mt__;Iw!!Hit2Ag!zoh5xPnvNfIYnDVa8fKOaBy5jIIQIMWT5qUv1AwVKSJJWw6Sv4JGdCWpcT1787WpOLtTS--cCCJ1garUIhR_$> or alter your "url" in each in the peers blocks to port 8001 so that they communicate via the kea-ctrl-agent. I think that should fix it. Thank you, Darren Ankney On Tue, Mar 26, 2024 at 7:58 AM Xiao, Yu (CCI-Atlanta) via Kea-users wrote: > > Hi Darren, > > > > I am using 2.4.1. > > > > > > Best Regards, > > Yu > > > > > > From: Kea-users on behalf of Darren Ankney > > Date: Tuesday, March 26, 2024 at 6:03 AM > To: Kea user's list > Subject: [EXTERNAL] Re: [Kea-users] kea-2.4.1 // Strange HA related errors > > Hello Yu, > > What version of Kea are you running on these servers, out of curiosity? > > Thank you, > Darren Ankney > > On Mon, Mar 25, 2024 at 10:55 AM Xiao, Yu (CCI-Atlanta) via Kea-users > wrote: > > > > Hi experts, > > > > > > > > I have configured two VMs in the same hypervisor as hot-standby mode HA. I > > believe they are successfully communicating with each other with heart beat > > packets, as we can see the primary VM kea-1 has successfully received the > > “ha-heartbeat” from the standby VM kea-2 in green logs. But the red logs > > indicate that the ha_hooks think the HA heartbeat communications failed due > > to “no route”. But this is a LAN network, and there’s indeed route > > installed as we can see below and we can ping the 69 ip. > > > > > > > > [yxiao322@kea_home1 ~]$ ip route > > > > 192.168.100.0/24 dev ens18 proto kernel scope link src 192.168.100.197 > > metric 100 <<< This should be the route to 192.168.100.69 > > > > 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 > > linkdown > > > > > > > > [root@kea_home1 yxiao322]# ping 192.168.100.69 > > > > PING 192.168.100.69 (192.168.100.69) 56(84) bytes of data. > > > > 64 bytes from 192.168.100.69: icmp_seq=1 ttl=64 time=0.342 ms > > > > 64 bytes from 192.168.100.69: icmp_seq=2 ttl=64 time=0.302 ms > > > > 64 bytes from 192.168.100.69: icmp_seq=3 ttl=64 time=0.236 ms > > > > ^Z > > > > [1]+ Stopped ping 192.168.100.69 > > > > > > > > And if I stop the kea service on primary, then we can see standby server > > will complain “communication with kea_home1 is interrupted”. And as soon as > > I start kea service again on primary, then the database began sync again. > > Thus, I believe there’s indeed communications and syncs between primary and > > standby VMs. But for some reason, if I shut the kea service on primary, > > then the standby won’t distribute DHCP leases even after I waited for a > > long time. Did I miss something here? > > > > > > > > > > > > Primary logs: > > > > > > > > Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.198 INFO > > [kea-dhcp6.commands/1224.139988007651072] COMMAND_RECEIVED Received command > > 'ha-heartbeat' > > > > Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.627 WARN > > [kea-dhcp6.ha-hooks/1224.139988049614592] > > HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home2 > > (https://urldefense.com/v3/__http://192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$ > > ): No route to host > > > > Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.627 WARN > > [kea-dhcp6.ha-hooks/1224.139988049614592] HA_COMMUNICATION_INTERRUPTED > > communication with kea_home2 is interrupted > > > > Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.199 INFO > > [kea-dhcp6.commands/1224.139988016043776] COMMAND_RECEIVED Received command > > 'ha-heartbeat' > > > > Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.627 WARN > > [kea-dhcp6.ha-hooks/
Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
Hi Yu, You will need to enable multi-threading dedicated listener for HA as described here: https://kea.readthedocs.io/en/kea-2.4.1/arm/hooks.html#multi-threaded-configuration-ha-mt or alter your "url" in each in the peers blocks to port 8001 so that they communicate via the kea-ctrl-agent. I think that should fix it. Thank you, Darren Ankney On Tue, Mar 26, 2024 at 7:58 AM Xiao, Yu (CCI-Atlanta) via Kea-users wrote: > > Hi Darren, > > > > I am using 2.4.1. > > > > > > Best Regards, > > Yu > > > > > > From: Kea-users on behalf of Darren Ankney > > Date: Tuesday, March 26, 2024 at 6:03 AM > To: Kea user's list > Subject: [EXTERNAL] Re: [Kea-users] kea-2.4.1 // Strange HA related errors > > Hello Yu, > > What version of Kea are you running on these servers, out of curiosity? > > Thank you, > Darren Ankney > > On Mon, Mar 25, 2024 at 10:55 AM Xiao, Yu (CCI-Atlanta) via Kea-users > wrote: > > > > Hi experts, > > > > > > > > I have configured two VMs in the same hypervisor as hot-standby mode HA. I > > believe they are successfully communicating with each other with heart beat > > packets, as we can see the primary VM kea-1 has successfully received the > > “ha-heartbeat” from the standby VM kea-2 in green logs. But the red logs > > indicate that the ha_hooks think the HA heartbeat communications failed due > > to “no route”. But this is a LAN network, and there’s indeed route > > installed as we can see below and we can ping the 69 ip. > > > > > > > > [yxiao322@kea_home1 ~]$ ip route > > > > 192.168.100.0/24 dev ens18 proto kernel scope link src 192.168.100.197 > > metric 100 <<< This should be the route to 192.168.100.69 > > > > 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 > > linkdown > > > > > > > > [root@kea_home1 yxiao322]# ping 192.168.100.69 > > > > PING 192.168.100.69 (192.168.100.69) 56(84) bytes of data. > > > > 64 bytes from 192.168.100.69: icmp_seq=1 ttl=64 time=0.342 ms > > > > 64 bytes from 192.168.100.69: icmp_seq=2 ttl=64 time=0.302 ms > > > > 64 bytes from 192.168.100.69: icmp_seq=3 ttl=64 time=0.236 ms > > > > ^Z > > > > [1]+ Stopped ping 192.168.100.69 > > > > > > > > And if I stop the kea service on primary, then we can see standby server > > will complain “communication with kea_home1 is interrupted”. And as soon as > > I start kea service again on primary, then the database began sync again. > > Thus, I believe there’s indeed communications and syncs between primary and > > standby VMs. But for some reason, if I shut the kea service on primary, > > then the standby won’t distribute DHCP leases even after I waited for a > > long time. Did I miss something here? > > > > > > > > > > > > Primary logs: > > > > > > > > Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.198 INFO > > [kea-dhcp6.commands/1224.139988007651072] COMMAND_RECEIVED Received command > > 'ha-heartbeat' > > > > Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.627 WARN > > [kea-dhcp6.ha-hooks/1224.139988049614592] > > HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home2 > > (https://urldefense.com/v3/__http://192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$ > > ): No route to host > > > > Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.627 WARN > > [kea-dhcp6.ha-hooks/1224.139988049614592] HA_COMMUNICATION_INTERRUPTED > > communication with kea_home2 is interrupted > > > > Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.199 INFO > > [kea-dhcp6.commands/1224.139988016043776] COMMAND_RECEIVED Received command > > 'ha-heartbeat' > > > > Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.627 WARN > > [kea-dhcp6.ha-hooks/1224.139988032829184] > > HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home2 > > (https://urldefense.com/v3/__http://192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$ > > ): No route to host > > > > Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.627 WARN > > [kea-dhcp6.ha-hooks/1224.139988032829184] HA_COMMUNICATION_INTERRUPTED > > communication with kea_home2 is interrupted > > > > > > > > Standby logs: > > > > > > > > Mar 25 10:10:24 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:24.129 WARN > > [kea-dhcp6.ha-hooks/2836.139717198915328] HA_COMMUNICATION_INTERRUPTED > > communication with kea_home1 is interrupted > > > > Mar 25 10:10:25 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:25.130 WARN > > [kea-dhcp6.ha-hooks/2836.139717207308032] > > HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to kea_home1 > > (https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$ > > ): Connection refused > > > > Mar 25 10:10:25 kea_home2 kea-dhcp6[2836]: 2024-03-25
Re: [Kea-users] [EXTERNAL] Re: kea-2.4.1 // Strange HA related errors
Hi Darren, I am using 2.4.1. Best Regards, Yu From: Kea-users on behalf of Darren Ankney Date: Tuesday, March 26, 2024 at 6:03 AM To: Kea user's list Subject: [EXTERNAL] Re: [Kea-users] kea-2.4.1 // Strange HA related errors Hello Yu, What version of Kea are you running on these servers, out of curiosity? Thank you, Darren Ankney On Mon, Mar 25, 2024 at 10:55 AM Xiao, Yu (CCI-Atlanta) via Kea-users wrote: > > Hi experts, > > > > I have configured two VMs in the same hypervisor as hot-standby mode HA. I > believe they are successfully communicating with each other with heart beat > packets, as we can see the primary VM kea-1 has successfully received the > “ha-heartbeat” from the standby VM kea-2 in green logs. But the red logs > indicate that the ha_hooks think the HA heartbeat communications failed due > to “no route”. But this is a LAN network, and there’s indeed route installed > as we can see below and we can ping the 69 ip. > > > > [yxiao322@kea_home1 ~]$ ip route > > 192.168.100.0/24 dev ens18 proto kernel scope link src 192.168.100.197 metric > 100 <<< This should be the route to 192.168.100.69 > > 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown > > > > [root@kea_home1 yxiao322]# ping 192.168.100.69 > > PING 192.168.100.69 (192.168.100.69) 56(84) bytes of data. > > 64 bytes from 192.168.100.69: icmp_seq=1 ttl=64 time=0.342 ms > > 64 bytes from 192.168.100.69: icmp_seq=2 ttl=64 time=0.302 ms > > 64 bytes from 192.168.100.69: icmp_seq=3 ttl=64 time=0.236 ms > > ^Z > > [1]+ Stopped ping 192.168.100.69 > > > > And if I stop the kea service on primary, then we can see standby server will > complain “communication with kea_home1 is interrupted”. And as soon as I > start kea service again on primary, then the database began sync again. Thus, > I believe there’s indeed communications and syncs between primary and standby > VMs. But for some reason, if I shut the kea service on primary, then the > standby won’t distribute DHCP leases even after I waited for a long time. Did > I miss something here? > > > > > > Primary logs: > > > > Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.198 INFO > [kea-dhcp6.commands/1224.139988007651072] COMMAND_RECEIVED Received command > 'ha-heartbeat' > > Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.627 WARN > [kea-dhcp6.ha-hooks/1224.139988049614592] HA_HEARTBEAT_COMMUNICATIONS_FAILED > failed to send heartbeat to kea_home2 > (https://urldefense.com/v3/__http://192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$ > ): No route to host > > Mar 25 10:35:37 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:37.627 WARN > [kea-dhcp6.ha-hooks/1224.139988049614592] HA_COMMUNICATION_INTERRUPTED > communication with kea_home2 is interrupted > > Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.199 INFO > [kea-dhcp6.commands/1224.139988016043776] COMMAND_RECEIVED Received command > 'ha-heartbeat' > > Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.627 WARN > [kea-dhcp6.ha-hooks/1224.139988032829184] HA_HEARTBEAT_COMMUNICATIONS_FAILED > failed to send heartbeat to kea_home2 > (https://urldefense.com/v3/__http://192.168.100.69:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTi1Q4B1j$ > ): No route to host > > Mar 25 10:35:38 kea_home1 kea-dhcp6[1224]: 2024-03-25 10:35:38.627 WARN > [kea-dhcp6.ha-hooks/1224.139988032829184] HA_COMMUNICATION_INTERRUPTED > communication with kea_home2 is interrupted > > > > Standby logs: > > > > Mar 25 10:10:24 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:24.129 WARN > [kea-dhcp6.ha-hooks/2836.139717198915328] HA_COMMUNICATION_INTERRUPTED > communication with kea_home1 is interrupted > > Mar 25 10:10:25 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:25.130 WARN > [kea-dhcp6.ha-hooks/2836.139717207308032] HA_HEARTBEAT_COMMUNICATIONS_FAILED > failed to send heartbeat to kea_home1 > (https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$ > ): Connection refused > > Mar 25 10:10:25 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:25.130 WARN > [kea-dhcp6.ha-hooks/2836.139717207308032] HA_COMMUNICATION_INTERRUPTED > communication with kea_home1 is interrupted > > Mar 25 10:10:26 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:26.132 WARN > [kea-dhcp6.ha-hooks/2836.139717190522624] HA_HEARTBEAT_COMMUNICATIONS_FAILED > failed to send heartbeat to kea_home1 > (https://urldefense.com/v3/__http://192.168.100.197:8000/__;!!Hit2Ag!xbrgKugiAG9Ig2_84VjCtBW6x-DpB9Aox3x1n_oCHrMu1W-ZDDMlKiorN7OVohqT2UUHSGoJ3ePqTiEJ4bVb$ > ): Connection refused > > Mar 25 10:10:26 kea_home2 kea-dhcp6[2836]: 2024-03-25 10:10:26.132 WARN > [kea-dhcp6.ha-hooks/2836.139717190522624] HA_COMMUNICATION_INTERRUPTED > communication with
