Hi Sébastien, I do not have any familiarity with HAPROXY. However, it seems there might be a problem between HAPROXY <-> Mariadb from your description of the behavior. This may not have anything to do with Kea. Are you able to test with some other software package to see if the same problem exists? As far as what can be done in Kea to mitigate this issue, you could setup retries which you've already done.
As far as the issue (https://gitlab.isc.org/isc-projects/kea/-/issues/1746) you mentioned is concerned. This seems to be describing adding support to Kea to connect to multiple database servers. It does not seem to pertain to HAPROXY. If Kea supported this, then you wouldn't need HAPROXY as Kea would be able to be configured with all of your database IP addresses directly and would somehow choose one to connect to. I think the comment you mention specifically is a workaround where you run one Kea server per database node. If the database node fails, that Kea server would also fail. That is a workable strategy if you were interested in it. Of course, the HA hook would not be usable if you have more than two servers. The result of not using the HA hook would be that all servers would try to answer each client. You would also want to run the random lease allocator to avoid collisions. Thank you, Darren Ankney On Thu, Jan 9, 2025 at 4:51 AM Sébastien Nauer <sna...@hotcity.lu> wrote: > > Hello Daren, > > Thank you for your comments. > > We use MariaDB v11.4.3 which seems to be fully compatible with KEA. > > In MariaDB logs, we only see such entries : > > 2025-01-09 9:06:50 8232949 [Warning] Aborted connection 8232949 to db: > > 'kea' user: 'kea_user' host: '1.2.3.4' real ip: '5.6.7.8' (Got an error > > reading communication packets) > > As we don't see such errors by connecting directly a KEA node to a MariaDB > node, it seems that the issue only occurs when using HAPROXY in front of the > GALERA Cluster. > > Also, according the following link, it seems that we should have to configure > 4 KEA instances (as we use 4 MariaDB instances which each KEA instance being > connected to one MariaDB node) > ???https://gitlab.isc.org/isc-projects/kea/-/issues/1746 > > For troubleshooting this issue, we currently configured HAPROXY with only 1 > MariaDB node and source IP stickiness, the aims being : > * to persist the KEA database connection with the same MariaDB node > * to be able to connect KEA to another MariaDB node if the current node fails > > We can confirm that the error occurs in such a simple situation (with only > one database node) and we don't understand why KEA generates > DATABASE_MYSQL_FATAL_ERROR messages. > > Could you confirm if we can use HAPROXY in such a manner ? > Are there specific settings to activate in KEA and/or HAPROXY ? > Or finaly, is it better not to use HAPROXY in such a scenario ? > > Any advice for configuring this kind of KEA cluster will be appreciated. > Kind regards. > > Le mer. 8 janv. 2025 à 22:59, Darren Ankney <darren.ank...@gmail.com> a écrit > : >> >> Hi Sébastien, >> >> The frequent disconnections could be anything. Is there anything in the >> MySQL logs? Also, is this MySQL or MariaDB? MySQL 8 has some performance >> issues that affect Kea pretty badly (See here: >> https://kea.readthedocs.io/en/kea-2.6.1/arm/admin.html#mysql-5-7-vs-mysql-8-vs-mariadb-10-and-11) >> >> If you haven't already found them, there are a couple of articles about >> MySQL clusters and Kea: >> https://kb.isc.org/docs/getting-started-with-galera-or-percona-for-kea >> https://kb.isc.org/docs/experimental-mysql-clusters-for-kea (this one might >> not be relevant) >> >> Thank you, >> Darren Ankney >> >> On Wed, Jan 8, 2025 at 4:51 AM Sébastien Nauer via Kea-users >> <kea-users@lists.isc.org> wrote: >>> >>> Hello, >>> >>> We are experiencing issues with configuring a KEA cluster in >>> high-availability mode with a GALERA Cluster database backend. >>> >>> As we'd like to use the database cluster for managing leases >>> synchronization : >>> >>> we configured KEA High Availability mode like this for disabling lease sync >>> & update : >>> >>> "high-availability": [ >>> { >>> "this-server-name": "kea01", >>> "mode": "load-balancing", >>> "heartbeat-delay": 10000, >>> "max-response-delay": 10000, >>> "max-ack-delay": 5000, >>> "max-unacked-clients": 5, >>> "send-lease-updates": false, >>> "sync-leases": false, >>> "sync-page-limit": 10000, >>> "sync-timeout": 30000, >>> "multi-threading": { >>> "enable-multi-threading": true, >>> "http-dedicated-listener": true, >>> "http-listener-threads": 0, >>> "http-client-threads": 0 >>> }, >>> "peers": [ >>> { >>> "name": "kea0&", >>> "url": "http://1.2.3.4:8080/";, >>> "role": "primary", >>> "auto-failover": true >>> }, >>> { >>> "name": "kea04", >>> "url": "http://5.6.7.8:8080/";, >>> "role": "secondary", >>> "auto-failover": true >>> } >>> ] >>> } >>> ] >>> >>> we configured the lease database like this for connecting to the database >>> cluster : >>> >>> "lease-database": { >>> "type": "mysql", >>> "name": "kea", >>> "user": "KEA_USER", >>> "password": "KEA_PASSWORD", >>> "host": "1.1.1.1", >>> "max-reconnect-tries" : 10, >>> "reconnect-wait-time" : 30000 >>> }, >>> >>> we tried to connect to the database backend : >>> >>> through a VIP with load balancing >>> through HAPROXY with / without load balancing >>> >>> When we start KEA service and the servers start, KEA logs show a lot of >>> DATABASE_MYSQL_FATAL_ERROR messages like this : >>> >>> 2025-01-07 16:18:42.483 ERROR [kea-dhcp4.database/4349.0x34cf8b308500] >>> DATABASE_MYSQL_FATAL_ERROR Unrecoverable MySQL error occurred: unable to >>> execute for <SELECT h.host_id, h.dhcp_identifier, h.dhcp_identifier_type, >>> h.dhcp4_subnet_id, h.dhcp6_subnet_id, h.ipv4_address, h.hostname, >>> h.dhcp4_client_classes, h.dhcp6_client_classes, h.user_context, >>> h.dhcp4_next_server, h.dhcp4_server_hostname, h.dhcp4_boot_file_name, >>> h.auth_key, o.option_id, o.code, o.value, o.formatted_value, o.space, >>> o.persistent, o.cancelled, o.user_context FROM hosts AS h LEFT JOIN >>> dhcp4_options AS o ON h.host_id = o.host_id WHERE h.dhcp4_subnet_id = ? AND >>> h.dhcp_identifier_type = ? AND h.dhcp_identifier = ? ORDER BY h.host_id, >>> o.option_id>, reason: Server has gone away (error code: 2006). >>> 2025-01-07 16:18:43.533 INFO [kea-dhcp4.dhcp4/4349.0x34cf8aa12000] >>> DHCP4_DB_RECONNECT_LOST_CONNECTION database connection lost. >>> 2025-01-07 16:18:43.533 INFO [kea-dhcp4.hosts/4349.0x34cf8aa12000] >>> DHCPSRV_MYSQL_HOST_DB opening MySQL hosts database: host=10.194.50.12 >>> max-reconnect-tries=10 name=prod_kea password=***** >>> reconnect-wait-time=30000 type=mysql universe=4 user=keaCluster >>> 2025-01-07 16:18:43.586 INFO [kea-dhcp4.dhcp4/4349.0x34cf8aa12000] >>> DHCP4_DB_RECONNECT_SUCCEEDED database connection recovered. >>> >>> As we noticed that the database connection is frequently lost for no >>> apparent reason we suspect that there is an issue with using persistent >>> database connections through a VIP with load balancing (maybe related to >>> multithreading also) >>> >>> Has anyone successfully configured such a setup ? How ? >>> What do you suggest for avoiding this error message and configuring a high >>> availability DHCP cluster with a high availability database cluster ? >>> >>> Any feedback from the community will be appreciated ! >>> Kind regards >>> >>> >>> >>> -- >>> >>> >>> >>> Sébastien NAUER| Technical Manager >>> 11, Avenue Guillaume | L-1651 Luxembourg >>> Phone: (+352) 2663 2660 | Fax: (+352) 2663 2665 >>> Facebook | Twitter >>> -- >>> ISC funds the development of this software with paid support subscriptions. >>> Contact us at https://www.isc.org/contact/ for more information. >>> >>> To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users. >>> >>> Kea-users mailing list >>> Kea-users@lists.isc.org >>> https://lists.isc.org/mailman/listinfo/kea-users > > > > -- > > > > Sébastien NAUER| Technical Manager > 11, Avenue Guillaume | L-1651 Luxembourg > Phone: (+352) 2663 2660 | Fax: (+352) 2663 2665 > Facebook | Twitter -- ISC funds the development of this software with paid support subscriptions. Contact us at https://www.isc.org/contact/ for more information. To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users. Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users
