I'll be excited to here the result!
On Thu, Dec 28, 2023 at 2:43 PM CS <cs.temp.m...@gmail.com> wrote:
>
> Me too, especially SELinux. But not in this case. apparmor is disabled as far
> as I know.
>
> I'm going to clone server1 into a server3 and see if the new server binds to
> the port then and if one and three will HA effectively. Bit of a sledgehammer
> approach but I'm at a loss too.
>
> CS, cs.temp.m...@gmail.com
>
>
> On Thu, 28 Dec 2023 at 11:36, Darren Ankney <darren.ank...@gmail.com> wrote:
>>
>> Hello,
>>
>> At some point we got off list. Putting us back on the list. Hope
>> that is OK. Whatever the ultimate resolution is, the information may
>> help someone in the future.
>>
>> I am at a loss. Have you checked for and/or temporarily disabled
>> appArmor or SELinux to make sure that they are not the problem?
>> Either of them might filter one application's access to the network
>> but not another application even on the same network / port
>> combination. I have bitten more than once by both of them.
>>
>> Thank you,
>>
>> Darren Ankney
>>
>> On Thu, Dec 28, 2023 at 2:28 PM CS <cs.temp.m...@gmail.com> wrote:
>> >
>> > Unfortunately just ip address obscured.
>> > =/
>> > Thanks
>> > CS, cs.temp.m...@gmail.com
>> >
>> >
>> > On Thu, 28 Dec 2023 at 11:25, Darren Ankney <darren.ank...@gmail.com>
>> > wrote:
>> >>
>> >> Hello,
>> >>
>> >> Is the "url" that is obscured a name or IP address?
>> >>
>> >> Thank you,
>> >>
>> >> Darren Ankney
>> >>
>> >> On Thu, Dec 28, 2023 at 2:07 PM CS <cs.temp.m...@gmail.com> wrote:
>> >>>
>> >>> At this moment? You wrote it =)
>> >>>
>> >>> {
>> >>> "Dhcp4": {
>> >>> "hooks-libraries": [
>> >>> {
>> >>> "library": "/xxx/libdhcp_lease_cmds.so"
>> >>> },
>> >>> {
>> >>> "library" : "/xxx/libdhcp_ha.so",
>> >>> "parameters": {
>> >>> "high-availability": [
>> >>> {
>> >>> "this-server-name": "server2",
>> >>> "mode": "load-balancing",
>> >>> "auto-failover": true,
>> >>> "peers": [
>> >>> {
>> >>> "name": "server1",
>> >>> "url": "http://xxx:8001";,
>> >>> "role": "primary"
>> >>> },
>> >>> {
>> >>> "name": "server2",
>> >>> "url": "http://xxx:8001";,
>> >>> "role": "secondary"
>> >>> }
>> >>> ]
>> >>> }
>> >>> ]
>> >>> }
>> >>> }
>> >>> ]
>> >>> }
>> >>> }
>> >>>
>> >>>
>> >>> CS, cs.temp.m...@gmail.com
>> >>>
>> >>>
>> >>> On Thu, 28 Dec 2023 at 10:34, Darren Ankney <darren.ank...@gmail.com>
>> >>> wrote:
>> >>>>
>> >>>> Hi,
>> >>>>
>> >>>> Could you share your hooks configuration from server2?
>> >>>>
>> >>>> Thank you,
>> >>>>
>> >>>> Darren Ankney
>> >>>>
>> >>>> On Thu, Dec 28, 2023 at 1:29 PM CS <cs.temp.m...@gmail.com> wrote:
>> >>>>>
>> >>>>> > there could either be a routing problem or a firewall problem
>> >>>>> Those things it aint. I setup a tiny webserver with openssl on port
>> >>>>> 8001 and I can telnet back and forth to each server just fine.
>> >>>>>
>> >>>>> Kea isn't setting up what it is supposed to and isn't logging a
>> >>>>> failure of any sort:
>> >>>>> server1$ sudo netstat -lnp | grep 8001
>> >>>>> tcp 0 0 10.111.45.45:8001 0.0.0.0:*
>> >>>>> LISTEN 1534068/kea-dhcp4
>> >>>>>
>> >>>>> server2$ sudo netstat -lnp | grep 8001
>> >>>>> server2:$ systemctl status isc-kea-dhcp4-server.service
>> >>>>> ● isc-kea-dhcp4-server.service - Kea IPv4 DHCP daemon
>> >>>>> Loaded: loaded (/lib/systemd/system/isc-kea-dhcp4-server.service;
>> >>>>> enabled; vendor preset: enabled)
>> >>>>> Active: active (running) since Thu 2023-12-28 18:22:55 UTC; 3min
>> >>>>> 24s ago
>> >>>>> Docs: man:kea-dhcp4(8)
>> >>>>> Main PID: 1466785 (kea-dhcp4)
>> >>>>> Tasks: 5 (limit: 19052)
>> >>>>> Memory: 2.8M
>> >>>>> CPU: 48ms
>> >>>>> CGroup: /system.slice/isc-kea-dhcp4-server.service
>> >>>>> └─1466785 /usr/sbin/kea-dhcp4 -c /etc/kea/kea-dhcp4.conf
>> >>>>> server2$ tail -n10 /var/log/kea/kea-dhcp4.log
>> >>>>> 2023-12-28 18:22:55.427 INFO [kea-dhcp4.ha-hooks/1466785]
>> >>>>> HA_SERVICE_STARTED started high availability service in load-balancing
>> >>>>> mode as primary server
>> >>>>> 2023-12-28 18:22:55.427 WARN [kea-dhcp4.dhcp4/1466785]
>> >>>>> DHCP4_MULTI_THREADING_INFO enabled: no, number of threads: 0, queue
>> >>>>> size: 0
>> >>>>> 2023-12-28 18:22:55.427 INFO [kea-dhcp4.dhcp4/1466785] DHCP4_STARTED
>> >>>>> Kea DHCPv4 server version 2.2.0 started
>> >>>>> 2023-12-28 18:23:06.441 INFO [kea-dhcp4.ha-hooks/1466785]
>> >>>>> HA_STATE_TRANSITION server transitions from WAITING to SYNCING state,
>> >>>>> partner state is WAITING
>> >>>>> 2023-12-28 18:23:06.442 INFO [kea-dhcp4.ha-hooks/1466785]
>> >>>>> HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the
>> >>>>> partner while in SYNCING state
>> >>>>> 2023-12-28 18:23:06.442 INFO [kea-dhcp4.ha-hooks/1466785]
>> >>>>> HA_SYNC_START starting lease database synchronization with server1
>> >>>>> 2023-12-28 18:23:06.444 INFO [kea-dhcp4.ha-hooks/1466785]
>> >>>>> HA_LEASES_SYNC_LEASE_PAGE_RECEIVED received 0 leases from server1
>> >>>>> 2023-12-28 18:23:06.446 INFO [kea-dhcp4.ha-hooks/1466785]
>> >>>>> HA_SYNC_SUCCESSFUL lease database synchronization with server1
>> >>>>> completed successfully in 3.534 ms
>> >>>>> 2023-12-28 18:23:06.446 INFO [kea-dhcp4.ha-hooks/1466785]
>> >>>>> HA_STATE_TRANSITION server transitions from SYNCING to READY state,
>> >>>>> partner state is WAITING
>> >>>>> 2023-12-28 18:23:06.446 INFO [kea-dhcp4.ha-hooks/1466785]
>> >>>>> HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the
>> >>>>> partner while in READY state
>> >>>>>
>> >>>>>
>> >>>>> On Thu, Dec 28, 2023, 03:12 Darren Ankney <darren.ank...@gmail.com>
>> >>>>> wrote:
>> >>>>>>
>> >>>>>> Hi,
>> >>>>>>
>> >>>>>> My advice had nothing to do with micetro as I am not familiar with it
>> >>>>>> at all. Micetro shouldn't have anything to do with HA functionality,
>> >>>>>> however. If one server can reach the other on port 8001 but not the
>> >>>>>> reverse, there could either be a routing problem or a firewall
>> >>>>>> problem.
>> >>>>>>
>> >>>>>> Thank you,
>> >>>>>>
>> >>>>>> Darren Ankney
>> >>>>>>
>> >>>>>> On Wed, Dec 27, 2023 at 10:50 PM CS <cs.temp.m...@gmail.com> wrote:
>> >>>>>>>
>> >>>>>>> Thanks for your input Thijs, tho I find your assessment confusing.
>> >>>>>>> It certainly seems to affirm the observed behavior with my full
>> >>>>>>> configuration but it flies against Darren's advice and my read of:
>> >>>>>>>
>> >>>>>>> // If enabling HA and multi-threading, the 8000 port is used
>> >>>>>>> by the HA
>> >>>>>>> // hook library http listener. When using HA hook library
>> >>>>>>> with
>> >>>>>>> // multi-threading to function, make sure the port used by
>> >>>>>>> dedicated
>> >>>>>>> // listener is different (e.g. 8001) than the one used by
>> >>>>>>> CA. Note
>> >>>>>>> // the commands should still be sent via CA. The dedicated
>> >>>>>>> listener
>> >>>>>>> // is specifically for HA updates only.
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> >Micetro is not using port 8000 :
>> >>>>>>> >https://docs.menandmice.com/en/10.5/guides/implementation/firewall_ports/
>> >>>>>>> Doesn't this indicate it is? And as far as I can see in Micetro this
>> >>>>>>> cannot be changed. So at a minimum the CA must be on port 8000
>> >>>>>>>
>> >>>>>>> >This part is also important:
>> >>>>>>> >https://docs.menandmice.com/en/10.5/guides/user-manual/dhcp_kea/#add-kea-hooks
>> >>>>>>> A good point, however I suspect my deployment is failing before any
>> >>>>>>> problem with this hook library.
>> >>>>>>>
>> >>>>>>> CS, cs.temp.m...@gmail.com
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On Wed, 27 Dec 2023 at 15:51, Thijs Blok <blokth...@gmail.com> wrote:
>> >>>>>>>>
>> >>>>>>>> Hi,
>> >>>>>>>>
>> >>>>>>>> Your control agent needs to run on the same port as the HA
>> >>>>>>>> configuration parameters.
>> >>>>>>>> Please make sure you don't listen on the localhost only which is
>> >>>>>>>> the default for the control agent.
>> >>>>>>>>
>> >>>>>>>> "Control-agent": {
>> >>>>>>>>
>> >>>>>>>> "http-host": "x.x.x.x",
>> >>>>>>>>
>> >>>>>>>> "http-port": 8000,
>> >>>>>>>>
>> >>>>>>>> ....
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> Micetro is not using port 8000 :
>> >>>>>>>> https://docs.menandmice.com/en/10.5/guides/implementation/firewall_ports/
>> >>>>>>>>
>> >>>>>>>> This part is also important:
>> >>>>>>>> https://docs.menandmice.com/en/10.5/guides/user-manual/dhcp_kea/#add-kea-hooks
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> Op wo 27 dec 2023 om 23:47 schreef CS <cs.temp.m...@gmail.com>:
>> >>>>>>>>>
>> >>>>>>>>> telneting around it appears server1 can reach 8001 on itself but
>> >>>>>>>>> not 2. server 2 can telnet 8001 on server 1 but not itself.
>> >>>>>>>>> I see nothing logged that indicates a conflict tho and both of
>> >>>>>>>>> these besides being on different subnets these servers are
>> >>>>>>>>> virtually identical and firewall free.
>> >>>>>>>>>
>> >>>>>>>>> CS, cs.temp.m...@gmail.com
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> On Wed, 27 Dec 2023 at 14:38, CS <cs.temp.m...@gmail.com> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>> They made it through. With minor changes (ip addresses, library
>> >>>>>>>>>> locations, and logging) the files are accepted and daemons are
>> >>>>>>>>>> running but... no dice.
>> >>>>>>>>>>
>> >>>>>>>>>> server1$ tail -f /var/log/kea/kea-dhcp4.log
>> >>>>>>>>>> 2023-12-27 22:24:48.486 INFO [kea-dhcp4.dhcpsrv/1495687]
>> >>>>>>>>>> DHCPSRV_MEMFILE_LEASE_FILE_LOAD loading leases from file
>> >>>>>>>>>> /var/lib/kea/kea-leases4.csv
>> >>>>>>>>>> 2023-12-27 22:24:48.486 INFO [kea-dhcp4.dhcpsrv/1495687]
>> >>>>>>>>>> DHCPSRV_MEMFILE_EXTRACT_EXTENDED_INFO4 extracting extended info
>> >>>>>>>>>> saw 0 leases, extended info sanity checks modified 0 / updated 0
>> >>>>>>>>>> leases and 0 leases have relay or remote id
>> >>>>>>>>>> 2023-12-27 22:24:48.486 INFO [kea-dhcp4.dhcpsrv/1495687]
>> >>>>>>>>>> DHCPSRV_MEMFILE_LFC_SETUP setting up the Lease File Cleanup
>> >>>>>>>>>> interval to 3600 sec
>> >>>>>>>>>> 2023-12-27 22:24:48.486 WARN [kea-dhcp4.dhcpsrv/1495687]
>> >>>>>>>>>> DHCPSRV_NO_SOCKETS_OPEN no interface configured to listen to DHCP
>> >>>>>>>>>> traffic
>> >>>>>>>>>> 2023-12-27 22:24:48.486 INFO [kea-dhcp4.ha-hooks/1495687]
>> >>>>>>>>>> HA_LOCAL_DHCP_DISABLE local DHCP service is disabled while the
>> >>>>>>>>>> server1 is in the WAITING state
>> >>>>>>>>>> 2023-12-27 22:24:48.486 INFO [kea-dhcp4.ha-hooks/1495687]
>> >>>>>>>>>> HA_SERVICE_STARTED started high availability service in
>> >>>>>>>>>> load-balancing mode as primary server
>> >>>>>>>>>> 2023-12-27 22:24:48.487 WARN [kea-dhcp4.dhcp4/1495687]
>> >>>>>>>>>> DHCP4_MULTI_THREADING_INFO enabled: yes, number of threads: 2,
>> >>>>>>>>>> queue size: 64
>> >>>>>>>>>> 2023-12-27 22:24:48.487 INFO [kea-dhcp4.dhcp4/1495687]
>> >>>>>>>>>> DHCP4_STARTED Kea DHCPv4 server version 2.4.0 started
>> >>>>>>>>>> 2023-12-27 22:24:58.498 WARN [kea-dhcp4.ha-hooks/1495687]
>> >>>>>>>>>> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to
>> >>>>>>>>>> server2 (http://xxx:8001): Connection refused
>> >>>>>>>>>> 2023-12-27 22:25:08.510 WARN [kea-dhcp4.ha-hooks/1495687]
>> >>>>>>>>>> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to
>> >>>>>>>>>> server2 (http://xxx:8001): Connection refused
>> >>>>>>>>>> 2023-12-27 22:25:18.519 WARN [kea-dhcp4.ha-hooks/1495687]
>> >>>>>>>>>> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to
>> >>>>>>>>>> server2 (http://xxx5:8001): Connection refused
>> >>>>>>>>>> 2023-12-27 22:25:28.531 WARN [kea-dhcp4.ha-hooks/1495687]
>> >>>>>>>>>> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to
>> >>>>>>>>>> server2 (http://xxx:8001): Connection refused
>> >>>>>>>>>> 2023-12-27 22:25:37.660 INFO [kea-dhcp4.commands/1495687]
>> >>>>>>>>>> COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >>>>>>>>>> 2023-12-27 22:25:38.535 WARN [kea-dhcp4.ha-hooks/1495687]
>> >>>>>>>>>> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to
>> >>>>>>>>>> server2 (http://xxx:8001): Connection refused
>> >>>>>>>>>> 2023-12-27 22:25:47.674 INFO [kea-dhcp4.commands/1495687]
>> >>>>>>>>>> COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >>>>>>>>>> 2023-12-27 22:25:48.546 WARN [kea-dhcp4.ha-hooks/1495687]
>> >>>>>>>>>> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to
>> >>>>>>>>>> server2 (http://xxx:8001): Connection refused
>> >>>>>>>>>> 2023-12-27 22:25:48.546 WARN [kea-dhcp4.ha-hooks/1495687]
>> >>>>>>>>>> HA_COMMUNICATION_INTERRUPTED communication with server2 is
>> >>>>>>>>>> interrupted
>> >>>>>>>>>> 2023-12-27 22:25:48.546 INFO [kea-dhcp4.ha-hooks/1495687]
>> >>>>>>>>>> HA_STATE_TRANSITION server transitions from WAITING to
>> >>>>>>>>>> PARTNER-DOWN state, partner state is UNDEFINED
>> >>>>>>>>>> 2023-12-27 22:25:48.547 INFO [kea-dhcp4.ha-hooks/1495687]
>> >>>>>>>>>> HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the
>> >>>>>>>>>> partner while in PARTNER-DOWN state
>> >>>>>>>>>> 2023-12-27 22:25:48.547 INFO [kea-dhcp4.ha-hooks/1495687]
>> >>>>>>>>>> HA_LOCAL_DHCP_ENABLE local DHCP service is enabled while the
>> >>>>>>>>>> server1 is in the PARTNER-DOWN state
>> >>>>>>>>>> 2023-12-27 22:25:57.687 INFO [kea-dhcp4.commands/1495687]
>> >>>>>>>>>> COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >>>>>>>>>> 2023-12-27 22:25:57.690 INFO [kea-dhcp4.commands/1495687]
>> >>>>>>>>>> COMMAND_RECEIVED Received command 'dhcp-disable'
>> >>>>>>>>>> 2023-12-27 22:25:57.691 INFO [kea-dhcp4.commands/1495687]
>> >>>>>>>>>> COMMAND_RECEIVED Received command 'lease4-get-page'
>> >>>>>>>>>> 2023-12-27 22:25:57.693 INFO [kea-dhcp4.commands/1495687]
>> >>>>>>>>>> COMMAND_RECEIVED Received command 'ha-sync-complete-notify'
>> >>>>>>>>>> 2023-12-27 22:25:58.557 WARN [kea-dhcp4.ha-hooks/1495687]
>> >>>>>>>>>> HA_HEARTBEAT_COMMUNICATIONS_FAILED failed to send heartbeat to
>> >>>>>>>>>> server2 (http://xxx:8001): Connection refused
>> >>>>>>>>>> 2023-12-27 22:25:58.557 WARN [kea-dhcp4.ha-hooks/1495687]
>> >>>>>>>>>> HA_COMMUNICATION_INTERRUPTED communication with server2 is
>> >>>>>>>>>> interrupted
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> server2$ tail -f /var/log/kea/kea-dhcp4.log
>> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcp4/1434251]
>> >>>>>>>>>> DHCP4_CONFIG_COMPLETE DHCPv4 server has completed configuration:
>> >>>>>>>>>> no IPv4 subnets!; DDNS: disabled
>> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcpsrv/1434251]
>> >>>>>>>>>> DHCPSRV_MEMFILE_DB opening memory file lease database:
>> >>>>>>>>>> type=memfile universe=4
>> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcpsrv/1434251]
>> >>>>>>>>>> DHCPSRV_MEMFILE_LEASE_FILE_LOAD loading leases from file
>> >>>>>>>>>> /var/lib/kea/kea-leases4.csv.2
>> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcpsrv/1434251]
>> >>>>>>>>>> DHCPSRV_MEMFILE_LEASE_FILE_LOAD loading leases from file
>> >>>>>>>>>> /var/lib/kea/kea-leases4.csv
>> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcpsrv/1434251]
>> >>>>>>>>>> DHCPSRV_MEMFILE_LFC_SETUP setting up the Lease File Cleanup
>> >>>>>>>>>> interval to 3600 sec
>> >>>>>>>>>> 2023-12-27 22:25:26.650 WARN [kea-dhcp4.dhcpsrv/1434251]
>> >>>>>>>>>> DHCPSRV_NO_SOCKETS_OPEN no interface configured to listen to DHCP
>> >>>>>>>>>> traffic
>> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.ha-hooks/1434251]
>> >>>>>>>>>> HA_LOCAL_DHCP_DISABLE local DHCP service is disabled while the
>> >>>>>>>>>> server2 is in the WAITING state
>> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.ha-hooks/1434251]
>> >>>>>>>>>> HA_SERVICE_STARTED started high availability service in
>> >>>>>>>>>> load-balancing mode as secondary server
>> >>>>>>>>>> 2023-12-27 22:25:26.650 WARN [kea-dhcp4.dhcp4/1434251]
>> >>>>>>>>>> DHCP4_MULTI_THREADING_INFO enabled: no, number of threads: 0,
>> >>>>>>>>>> queue size: 0
>> >>>>>>>>>> 2023-12-27 22:25:26.650 INFO [kea-dhcp4.dhcp4/1434251]
>> >>>>>>>>>> DHCP4_STARTED Kea DHCPv4 server version 2.2.0 started
>> >>>>>>>>>> 2023-12-27 22:25:57.690 INFO [kea-dhcp4.ha-hooks/1434251]
>> >>>>>>>>>> HA_STATE_TRANSITION server transitions from WAITING to SYNCING
>> >>>>>>>>>> state, partner state is PARTNER-DOWN
>> >>>>>>>>>> 2023-12-27 22:25:57.690 INFO [kea-dhcp4.ha-hooks/1434251]
>> >>>>>>>>>> HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the
>> >>>>>>>>>> partner while in SYNCING state
>> >>>>>>>>>> 2023-12-27 22:25:57.690 INFO [kea-dhcp4.ha-hooks/1434251]
>> >>>>>>>>>> HA_SYNC_START starting lease database synchronization with server1
>> >>>>>>>>>> 2023-12-27 22:25:57.693 INFO [kea-dhcp4.ha-hooks/1434251]
>> >>>>>>>>>> HA_LEASES_SYNC_LEASE_PAGE_RECEIVED received 0 leases from server1
>> >>>>>>>>>> 2023-12-27 22:25:57.695 INFO [kea-dhcp4.ha-hooks/1434251]
>> >>>>>>>>>> HA_SYNC_SUCCESSFUL lease database synchronization with server1
>> >>>>>>>>>> completed successfully in 3.877 ms
>> >>>>>>>>>> 2023-12-27 22:25:57.695 INFO [kea-dhcp4.ha-hooks/1434251]
>> >>>>>>>>>> HA_STATE_TRANSITION server transitions from SYNCING to READY
>> >>>>>>>>>> state, partner state is PARTNER-DOWN
>> >>>>>>>>>> 2023-12-27 22:25:57.695 INFO [kea-dhcp4.ha-hooks/1434251]
>> >>>>>>>>>> HA_LEASE_UPDATES_DISABLED lease updates will not be sent to the
>> >>>>>>>>>> partner while in READY state
>> >>>>>>>>>>
>> >>>>>>>>>> CS, cs.temp.m...@gmail.com
>> >>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>> On Wed, 27 Dec 2023 at 11:22, Darren Ankney
>> >>>>>>>>>> <darren.ank...@gmail.com> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>> Hi,
>> >>>>>>>>>>>
>> >>>>>>>>>>> See attached four files:
>> >>>>>>>>>>>
>> >>>>>>>>>>> ca-server1.json (config for kea-ctrl-agent)
>> >>>>>>>>>>> ca-server2.json (config for kea-ctrl-agent)
>> >>>>>>>>>>> dhcp4-server1.json (config for kea-dhcp4)
>> >>>>>>>>>>> dhcp4-server2.json (config for kea-dhcp4)
>> >>>>>>>>>>>
>> >>>>>>>>>>> These files use port 8000 for kea-ctrl-agent and 8001 for
>> >>>>>>>>>>> kea-dhcp4 on
>> >>>>>>>>>>> version 2.4.0. They don't do anything other than setup the
>> >>>>>>>>>>> heartbeats
>> >>>>>>>>>>> for HA (in fact you don't even need to run the control agent as
>> >>>>>>>>>>> Kea is
>> >>>>>>>>>>> communicating directly). Heartbeats are sent back and forth on
>> >>>>>>>>>>> port
>> >>>>>>>>>>> 8001 as expected. Can you give these a try and see if they work
>> >>>>>>>>>>> (in
>> >>>>>>>>>>> testing of course, they won't serve any clients)? I'm genuinely
>> >>>>>>>>>>> curious if they work. Yours should work (unless there is some
>> >>>>>>>>>>> problem
>> >>>>>>>>>>> with the certificates or something). I didn't notice any reason
>> >>>>>>>>>>> why
>> >>>>>>>>>>> they wouldn't.
>> >>>>>>>>>>>
>> >>>>>>>>>>> Thank you,
>> >>>>>>>>>>>
>> >>>>>>>>>>> Darren Ankney
>> >>>>>>>>>>>
>> >>>>>>>>>>> PS: I'm not sure if these attachments will make it through to
>> >>>>>>>>>>> the list.
>> >>>>>>>>>>>
>> >>>>>>>>>>> On Wed, Dec 27, 2023 at 7:16 AM CS <cs.temp.m...@gmail.com>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>> >
>> >>>>>>>>>>> > Kea 2.4.0
>> >>>>>>>>>>> >
>> >>>>>>>>>>> > On Wed, Dec 27, 2023, 03:18 Darren Ankney
>> >>>>>>>>>>> > <darren.ank...@gmail.com> wrote:
>> >>>>>>>>>>> >>
>> >>>>>>>>>>> >> Hi,
>> >>>>>>>>>>> >>
>> >>>>>>>>>>> >> If I may ask, what version of Kea are you using? Some
>> >>>>>>>>>>> >> defaults have
>> >>>>>>>>>>> >> changed across versions.
>> >>>>>>>>>>> >>
>> >>>>>>>>>>> >> Thank you,
>> >>>>>>>>>>> >>
>> >>>>>>>>>>> >> Darren Ankney
>> >>>>>>>>>>> >>
>> >>>>>>>>>>> >> On Tue, Dec 26, 2023 at 4:31 PM CS <cs.temp.m...@gmail.com>
>> >>>>>>>>>>> >> wrote:
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > >Please describe what you mean by "it doesn't work".
>> >>>>>>>>>>> >> > I mean I get a pretty useless error: "Unable to connect to
>> >>>>>>>>>>> >> > Kea Control Agent."
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > > it might be be best to ask Men & Mice about "micetro" and
>> >>>>>>>>>>> >> > > how best to set things
>> >>>>>>>>>>> >> > I will at some point, when I find a resource with them. But
>> >>>>>>>>>>> >> > there are two players in this and since kea isn't behaving
>> >>>>>>>>>>> >> > as expected like you, I and the docs said. I'm starting
>> >>>>>>>>>>> >> > here.
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > >It actually SHOULDN'T work
>> >>>>>>>>>>> >> > That's my read on it too. But here's proof. The CA config
>> >>>>>>>>>>> >> > for one server. It matches for the other server except
>> >>>>>>>>>>> >> > certs and ip addresses obv.
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > "Control-agent": {
>> >>>>>>>>>>> >> > "http-host": "xxx.xx1.xxx.xxx",
>> >>>>>>>>>>> >> > "trust-anchor": "Certificate_Autority.pem",
>> >>>>>>>>>>> >> > "cert-file": "ca1_cert.pem",
>> >>>>>>>>>>> >> > "key-file": "ca1_key.pem",
>> >>>>>>>>>>> >> > "cert-required": true,
>> >>>>>>>>>>> >> > "http-port": 8000,
>> >>>>>>>>>>> >> > "authentication": {
>> >>>>>>>>>>> >> > "type": "basic",
>> >>>>>>>>>>> >> > "realm": "kea-control-agent",
>> >>>>>>>>>>> >> > "clients": [{
>> >>>>>>>>>>> >> > "user": "baduser",
>> >>>>>>>>>>> >> > "password": "badpassword",
>> >>>>>>>>>>> >> > }]
>> >>>>>>>>>>> >> > },
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > And the dhcp4 config, likewise only the small differences
>> >>>>>>>>>>> >> > between the two servers
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > "hooks-libraries": [{
>> >>>>>>>>>>> >> > "library":
>> >>>>>>>>>>> >> > "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_lease_cmds.so",
>> >>>>>>>>>>> >> > "parameters": {}
>> >>>>>>>>>>> >> > },{
>> >>>>>>>>>>> >> > "library" :
>> >>>>>>>>>>> >> > "/usr/lib/x86_64-linux-gnu/kea/hooks/libdhcp_ha.so",
>> >>>>>>>>>>> >> > "parameters": {
>> >>>>>>>>>>> >> > "high-availability": [{
>> >>>>>>>>>>> >> > "this-server-name":
>> >>>>>>>>>>> >> > "server1.org.org",
>> >>>>>>>>>>> >> > "mode":
>> >>>>>>>>>>> >> > "load-balancing",
>> >>>>>>>>>>> >> > "heartbeat-delay":
>> >>>>>>>>>>> >> > 10000,
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > "max-response-delay": 60000,
>> >>>>>>>>>>> >> > "max-ack-delay":
>> >>>>>>>>>>> >> > 5000,
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > "max-unacked-clients": 0,
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > "require-client-certs": true,
>> >>>>>>>>>>> >> > "trust-anchor":
>> >>>>>>>>>>> >> > "Certificate_Autority.pem",
>> >>>>>>>>>>> >> > "auto-failover":
>> >>>>>>>>>>> >> > true,
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > "peers": [{
>> >>>>>>>>>>> >> > "name":
>> >>>>>>>>>>> >> > "server1.org.org",
>> >>>>>>>>>>> >> > "url":
>> >>>>>>>>>>> >> > "http://xxx.xx1.xxx.xxx:8000/";,
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > "cert-file": "dhcp1_cert.pem",
>> >>>>>>>>>>> >> > "key-file":
>> >>>>>>>>>>> >> > "dhcp1_key.pem",
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > "basic-auth-user": "baduser",
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > "basic-auth-password": "badpassword",
>> >>>>>>>>>>> >> > "role":
>> >>>>>>>>>>> >> > "primary",
>> >>>>>>>>>>> >> > },{
>> >>>>>>>>>>> >> > "name":
>> >>>>>>>>>>> >> > "server2.org.org",
>> >>>>>>>>>>> >> > "url":
>> >>>>>>>>>>> >> > "http://xxx.xx2.xxx.xxx:8000/";,
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > "cert-file": "dhcp2_cert.pem",
>> >>>>>>>>>>> >> > "key-file":
>> >>>>>>>>>>> >> > "dhcp2_key.pem",
>> >>>>>>>>>>> >> > "role":
>> >>>>>>>>>>> >> > "secondary",
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > "basic-auth-user": "baduser",
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > "basic-auth-password": "badpassword",
>> >>>>>>>>>>> >> > }]
>> >>>>>>>>>>> >> > }]
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > low and behold it runs. The same nature of daemon status
>> >>>>>>>>>>> >> > and logs on the other server.
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > $ sudo systemctl restart isc-kea-ctrl-agent.service
>> >>>>>>>>>>> >> > isc-kea-dhcp4-server.service
>> >>>>>>>>>>> >> > $ sudo systemctl status isc-kea-ctrl-agent.service
>> >>>>>>>>>>> >> > isc-kea-dhcp4-server.service
>> >>>>>>>>>>> >> > ● isc-kea-ctrl-agent.service - Kea Control Agent
>> >>>>>>>>>>> >> > Loaded: loaded
>> >>>>>>>>>>> >> > (/lib/systemd/system/isc-kea-ctrl-agent.service; enabled;
>> >>>>>>>>>>> >> > vendor preset: enabled)
>> >>>>>>>>>>> >> > Active: active (running) since Tue 2023-12-26 20:57:29
>> >>>>>>>>>>> >> > UTC; 11s ago
>> >>>>>>>>>>> >> > Docs: man:kea-ctrl-agent(8)
>> >>>>>>>>>>> >> > Main PID: 1393724 (kea-ctrl-agent)
>> >>>>>>>>>>> >> > Tasks: 5 (limit: 19052)
>> >>>>>>>>>>> >> > Memory: 2.5M
>> >>>>>>>>>>> >> > CPU: 26ms
>> >>>>>>>>>>> >> > CGroup: /system.slice/isc-kea-ctrl-agent.service
>> >>>>>>>>>>> >> > └─1393724 /usr/sbin/kea-ctrl-agent -c
>> >>>>>>>>>>> >> > /etc/kea/kea-ctrl-agent.conf
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > Dec 26 20:57:29 kea1 systemd[1]: Started Kea Control Agent.
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > ● isc-kea-dhcp4-server.service - Kea DHCPv4 Service
>> >>>>>>>>>>> >> > Loaded: loaded
>> >>>>>>>>>>> >> > (/lib/systemd/system/isc-kea-dhcp4-server.service; enabled;
>> >>>>>>>>>>> >> > vendor preset: enabled)
>> >>>>>>>>>>> >> > Active: active (running) since Tue 2023-12-26 20:57:29
>> >>>>>>>>>>> >> > UTC; 11s ago
>> >>>>>>>>>>> >> > Docs: man:kea-dhcp4(8)
>> >>>>>>>>>>> >> > Main PID: 1393730 (kea-dhcp4)
>> >>>>>>>>>>> >> > Tasks: 9 (limit: 19052)
>> >>>>>>>>>>> >> > Memory: 4.5M
>> >>>>>>>>>>> >> > CPU: 96ms
>> >>>>>>>>>>> >> > CGroup: /system.slice/isc-kea-dhcp4-server.service
>> >>>>>>>>>>> >> > └─1393730 /usr/sbin/kea-dhcp4 -c
>> >>>>>>>>>>> >> > /etc/kea/kea-dhcp4.conf
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > Dec 26 20:57:29 kea1 systemd[1]:
>> >>>>>>>>>>> >> > isc-kea-dhcp4-server.service: Deactivated successfully.
>> >>>>>>>>>>> >> > Dec 26 20:57:29 kea1 systemd[1]: Stopped Kea DHCPv4 Service.
>> >>>>>>>>>>> >> > Dec 26 20:57:29 kea1 systemd[1]:
>> >>>>>>>>>>> >> > isc-kea-dhcp4-server.service: Consumed 1min 28.504s CPU
>> >>>>>>>>>>> >> > time.
>> >>>>>>>>>>> >> > Dec 26 20:57:29 kea1 systemd[1]: Started Kea DHCPv4 Service.
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > $ tail -n10 /var/log/kea/kea-ctrl-agent.log
>> >>>>>>>>>>> >> > 2023-12-26 20:59:53.827 INFO
>> >>>>>>>>>>> >> > [kea-ctrl-agent.ctrl-agent/1393724]
>> >>>>>>>>>>> >> > CTRL_AGENT_COMMAND_RECEIVED command ha-heartbeat received
>> >>>>>>>>>>> >> > from remote address xxx.xxx2.xxx.xxx
>> >>>>>>>>>>> >> > 2023-12-26 20:59:53.828 INFO
>> >>>>>>>>>>> >> > [kea-ctrl-agent.ctrl-agent/1393724]
>> >>>>>>>>>>> >> > CTRL_AGENT_COMMAND_FORWARDED command ha-heartbeat
>> >>>>>>>>>>> >> > successfully forwarded to the service dhcp4 from remote
>> >>>>>>>>>>> >> > address xxx.xx2.xxx.xxx
>> >>>>>>>>>>> >> > 2023-12-26 21:00:03.843 INFO [kea-ctrl-agent.auth/1393724]
>> >>>>>>>>>>> >> > HTTP_CLIENT_REQUEST_AUTHORIZED received HTTP request
>> >>>>>>>>>>> >> > authorized for 'baduser'
>> >>>>>>>>>>> >> > 2023-12-26 21:00:03.843 INFO
>> >>>>>>>>>>> >> > [kea-ctrl-agent.commands/1393724] COMMAND_RECEIVED Received
>> >>>>>>>>>>> >> > command 'ha-heartbeat'
>> >>>>>>>>>>> >> > 2023-12-26 21:00:03.843 INFO
>> >>>>>>>>>>> >> > [kea-ctrl-agent.ctrl-agent/1393724]
>> >>>>>>>>>>> >> > CTRL_AGENT_COMMAND_RECEIVED command ha-heartbeat received
>> >>>>>>>>>>> >> > from remote address xxx.xxx2.xxx.xxx
>> >>>>>>>>>>> >> > 2023-12-26 21:00:03.844 INFO
>> >>>>>>>>>>> >> > [kea-ctrl-agent.ctrl-agent/1393724]
>> >>>>>>>>>>> >> > CTRL_AGENT_COMMAND_FORWARDED command ha-heartbeat
>> >>>>>>>>>>> >> > successfully forwarded to the service dhcp4 from remote
>> >>>>>>>>>>> >> > address xxx.xxx2.xxx.xxx
>> >>>>>>>>>>> >> > 2023-12-26 21:00:13.859 INFO [kea-ctrl-agent.auth/1393724]
>> >>>>>>>>>>> >> > HTTP_CLIENT_REQUEST_AUTHORIZED received HTTP request
>> >>>>>>>>>>> >> > authorized for 'baduser'
>> >>>>>>>>>>> >> > 2023-12-26 21:00:13.859 INFO
>> >>>>>>>>>>> >> > [kea-ctrl-agent.commands/1393724] COMMAND_RECEIVED Received
>> >>>>>>>>>>> >> > command 'ha-heartbeat'
>> >>>>>>>>>>> >> > 2023-12-26 21:00:13.859 INFO
>> >>>>>>>>>>> >> > [kea-ctrl-agent.ctrl-agent/1393724]
>> >>>>>>>>>>> >> > CTRL_AGENT_COMMAND_RECEIVED command ha-heartbeat received
>> >>>>>>>>>>> >> > from remote address xxx.xxx2.xxx.xxx
>> >>>>>>>>>>> >> > 2023-12-26 21:00:13.860 INFO
>> >>>>>>>>>>> >> > [kea-ctrl-agent.ctrl-agent/1393724]
>> >>>>>>>>>>> >> > CTRL_AGENT_COMMAND_FORWARDED command ha-heartbeat
>> >>>>>>>>>>> >> > successfully forwarded to the service dhcp4 from remote
>> >>>>>>>>>>> >> > address xxx.xxx2.xxx.xxx
>> >>>>>>>>>>> >> > $ tail -n10 /var/log/kea/kea-dhcp4.log
>> >>>>>>>>>>> >> > 2023-12-26 20:58:53.728 INFO [kea-dhcp4.commands/1393730]
>> >>>>>>>>>>> >> > COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >>>>>>>>>>> >> > 2023-12-26 20:59:03.745 INFO [kea-dhcp4.commands/1393730]
>> >>>>>>>>>>> >> > COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >>>>>>>>>>> >> > 2023-12-26 20:59:13.762 INFO [kea-dhcp4.commands/1393730]
>> >>>>>>>>>>> >> > COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >>>>>>>>>>> >> > 2023-12-26 20:59:23.777 INFO [kea-dhcp4.commands/1393730]
>> >>>>>>>>>>> >> > COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >>>>>>>>>>> >> > 2023-12-26 20:59:33.793 INFO [kea-dhcp4.commands/1393730]
>> >>>>>>>>>>> >> > COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >>>>>>>>>>> >> > 2023-12-26 20:59:43.811 INFO [kea-dhcp4.commands/1393730]
>> >>>>>>>>>>> >> > COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >>>>>>>>>>> >> > 2023-12-26 20:59:53.827 INFO [kea-dhcp4.commands/1393730]
>> >>>>>>>>>>> >> > COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >>>>>>>>>>> >> > 2023-12-26 21:00:03.844 INFO [kea-dhcp4.commands/1393730]
>> >>>>>>>>>>> >> > COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >>>>>>>>>>> >> > 2023-12-26 21:00:13.859 INFO [kea-dhcp4.commands/1393730]
>> >>>>>>>>>>> >> > COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >>>>>>>>>>> >> > 2023-12-26 21:00:23.875 INFO [kea-dhcp4.commands/1393730]
>> >>>>>>>>>>> >> > COMMAND_RECEIVED Received command 'ha-heartbeat'
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > And changing the CA or the server HA paramersts to port
>> >>>>>>>>>>> >> > 8001 without changing the other (and the other server
>> >>>>>>>>>>> >> > results in "connection refused" logs. It obv wants the CA
>> >>>>>>>>>>> >> > port to match the HA parameters port despite what we and
>> >>>>>>>>>>> >> > the documentation suggests...
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > CS, cs.temp.m...@gmail.com
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> >
>> >>>>>>>>>>> >> > On Mon, 25 Dec 2023 at 02:45, Darren Ankney
>> >>>>>>>>>>> >> > <darren.ank...@gmail.com> wrote:
>> >>>>>>>>>>> >> >>
>> >>>>>>>>>>> >> >> Hi,
>> >>>>>>>>>>> >> >>
>> >>>>>>>>>>> >> >> It actually SHOULDN'T work to set your control agent and
>> >>>>>>>>>>> >> >> multi-threaded HA listener to the same port as only one of
>> >>>>>>>>>>> >> >> the
>> >>>>>>>>>>> >> >> applications should be able to setup a listener on that
>> >>>>>>>>>>> >> >> port. Please
>> >>>>>>>>>>> >> >> describe what you mean by "it doesn't work". I'm thinking
>> >>>>>>>>>>> >> >> it might be
>> >>>>>>>>>>> >> >> be best to ask Men & Mice about "micetro" and how best to
>> >>>>>>>>>>> >> >> set things
>> >>>>>>>>>>> >> >> up there.
>> >>>>>>>>>>> >> >>
>> >>>>>>>>>>> >> >> Thank you,
>> >>>>>>>>>>> >> >>
>> >>>>>>>>>>> >> >> Darren Ankney
>> >>>>>>>>>>> >> >>
>> >>>>>>>>>>> >> >> On Thu, Dec 21, 2023 at 6:47 PM CS
>> >>>>>>>>>>> >> >> <cs.temp.m...@gmail.com> wrote:
>> >>>>>>>>>>> >> >> >
>> >>>>>>>>>>> >> >> > Hi all,
>> >>>>>>>>>>> >> >> > Moving on from my failure to start and logging issues
>> >>>>>>>>>>> >> >> > (thank you for your help btw!) I now don't have my
>> >>>>>>>>>>> >> >> > heartbeat/control_agent working correctly.
>> >>>>>>>>>>> >> >> >
>> >>>>>>>>>>> >> >> > It works fine so long as I set the ports of my control
>> >>>>>>>>>>> >> >> > agents and ha hook parameters to be the same (IE 8000 or
>> >>>>>>>>>>> >> >> > 8001)
>> >>>>>>>>>>> >> >> >
>> >>>>>>>>>>> >> >> > However I am unable to tie the tiny cluster into
>> >>>>>>>>>>> >> >> > micetro, probably because the CA port is occupied with
>> >>>>>>>>>>> >> >> > HA heartbeats?
>> >>>>>>>>>>> >> >> >
>> >>>>>>>>>>> >> >> > Looking to these examples:
>> >>>>>>>>>>> >> >> > https://github.com/isc-projects/kea/tree/master/doc/examples/template-ha-mt-tls
>> >>>>>>>>>>> >> >> >
>> >>>>>>>>>>> >> >> > Documentation points out
>> >>>>>>>>>>> >> >> > //This specifies the port CA will listen on.
>> >>>>>>>>>>> >> >> > // If enabling HA and multi-threading, the 8000
>> >>>>>>>>>>> >> >> > port is used by the HA
>> >>>>>>>>>>> >> >> > // hook library http listener. When using HA
>> >>>>>>>>>>> >> >> > hook library with
>> >>>>>>>>>>> >> >> > // multi-threading to function, make sure the
>> >>>>>>>>>>> >> >> > port used by dedicated
>> >>>>>>>>>>> >> >> > // listener is different (e.g. 8001) than the
>> >>>>>>>>>>> >> >> > one used by CA. Note
>> >>>>>>>>>>> >> >> > // the commands should still be sent via CA. The
>> >>>>>>>>>>> >> >> > dedicated listener
>> >>>>>>>>>>> >> >> > // is specifically for HA updates only.
>> >>>>>>>>>>> >> >> >
>> >>>>>>>>>>> >> >> > However, how to have a dedicated port for HA and a
>> >>>>>>>>>>> >> >> > different one for CA escapes me.
>> >>>>>>>>>>> >> >> >
>> >>>>>>>>>>> >> >> > CS, cs.temp.m...@gmail.com
>> >>>>>>>>>>> >> >> > --
>> >>>>>>>>>>> >> >> > ISC funds the development of this software with paid
>> >>>>>>>>>>> >> >> > support subscriptions. Contact us at
>> >>>>>>>>>>> >> >> > https://www.isc.org/contact/ for more information.
>> >>>>>>>>>>> >> >> >
>> >>>>>>>>>>> >> >> > To unsubscribe visit
>> >>>>>>>>>>> >> >> > https://lists.isc.org/mailman/listinfo/kea-users.
>> >>>>>>>>>>> >> >> >
>> >>>>>>>>>>> >> >> > Kea-users mailing list
>> >>>>>>>>>>> >> >> > Kea-users@lists.isc.org
>> >>>>>>>>>>> >> >> > https://lists.isc.org/mailman/listinfo/kea-users
>> >>>>>>>>>>> >> >> --
>> >>>>>>>>>>> >> >> ISC funds the development of this software with paid
>> >>>>>>>>>>> >> >> support subscriptions. Contact us at
>> >>>>>>>>>>> >> >> https://www.isc.org/contact/ for more information.
>> >>>>>>>>>>> >> >>
>> >>>>>>>>>>> >> >> To unsubscribe visit
>> >>>>>>>>>>> >> >> https://lists.isc.org/mailman/listinfo/kea-users.
>> >>>>>>>>>>> >> >>
>> >>>>>>>>>>> >> >> Kea-users mailing list
>> >>>>>>>>>>> >> >> Kea-users@lists.isc.org
>> >>>>>>>>>>> >> >> https://lists.isc.org/mailman/listinfo/kea-users
>> >>>>>>>>>
>> >>>>>>>>> --
>> >>>>>>>>> ISC funds the development of this software with paid support
>> >>>>>>>>> subscriptions. Contact us at https://www.isc.org/contact/ for more
>> >>>>>>>>> information.
>> >>>>>>>>>
>> >>>>>>>>> To unsubscribe visit
>> >>>>>>>>> https://lists.isc.org/mailman/listinfo/kea-users.
>> >>>>>>>>>
>> >>>>>>>>> Kea-users mailing list
>> >>>>>>>>> Kea-users@lists.isc.org
>> >>>>>>>>> https://lists.isc.org/mailman/listinfo/kea-users
>> >>>>>>
>> >>>>>> --
>> >>>>>> ISC funds the development of this software with paid support
>> >>>>>> subscriptions. Contact us at https://www.isc.org/contact/ for more
>> >>>>>> information.
>> >>>>>>
>> >>>>>> To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.
>> >>>>>>
>> >>>>>> Kea-users mailing list
>> >>>>>> Kea-users@lists.isc.org
>> >>>>>> https://lists.isc.org/mailman/listinfo/kea-users
--
ISC funds the development of this software with paid support subscriptions.
Contact us at https://www.isc.org/contact/ for more information.
To unsubscribe visit https://lists.isc.org/mailman/listinfo/kea-users.
Kea-users mailing list
Kea-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/kea-users
