Re: [Kea-users] HA based on galera
In our case, our MySQL server is basically DBaaS. There was a Meetup presentation about it (https://www.meetup.com/la-mysql/events/228737163/). If you want, I can try to track it down -- the actual MySQL step isn't something I had to engineer myself -- so I was able to take HA for granted. In our use case, different scopes are handled by different individual Kea servers (as the network is naturally partitioned), so avoiding write conflicts is easy, but ultimately that's not the complicated part of our configuration. cheers, Klaus On Wed, Mar 14, 2018 at 10:10 AM, Jason Guy wrote: > Hey Klaus, > > Can you elaborate a bit on your design? I am in the middle of testing Kea > 1.3 with a 3 node mysql setup, and have not seen any issues yet. I > researched both MySQL-Cluster and MySQL Group Replication in MySQL5.7. I > settled on first testing with group replication, since it did not require > an extra management server arbitrating the writes. > > The group replication HA allows all devices to be a master, and uses GTIDs > to prevent 2 nodes from writing the same row. So my curiosity here is if I > am missing something in the design, and how I would easily stress test > this, to see if the SQL group can handle the distributed writes from the > Kea server. > > Thanks, > Jason > > On Tue, Mar 13, 2018 at 8:17 PM, Klaus Steden > wrote: > >> >> We have HA-MySQL behind our Kea servers, but the scopes themselves are >> sharded to avoid this kind of write contention. In our case, we've got a >> network partition that serves as an effective bright line between who's >> responsible for what. >> >> cheers >> Klaus >> >> On Tue, Mar 13, 2018 at 4:18 AM, wrote: >> >>> Hey, pal, that's my thread )) >>> >>> if you installed boost and configure can't find it - use ./configure >>> --help >>> there should be option --with-boost-source or smth like that >>> >>> >>> >>> On 2018-03-13 15:10, Tobi Obadiah wrote: >>> Hello there, Am just getting started with Kea, on my attempt to install kea on Ubuntu16.04 desktop I keep getting the "Missing required header files" errors I have to manage to fix some of them but wonder if there is a standard procedure to get this fix. Though I have boost installed "boost/shared_ptr.hpp" is reported missing. I appreciate your help. On Tue, Mar 13, 2018 at 11:37 AM, Dominik Epple wrote: Hi, > > it might help if you pick a dedicated galera write node, i.e. both > of your kea servers write to the same galera node at each point in > time. You can still configure that in a HA fashion by some > active-passive loadbalancer or a failover IP or something. > > The other option would be that the kea code gets adjusted for > running "galera-aware" but I am no expert on what exactly needs to > happen there. But if it can't be fixed on application level, it > probably can be fixed on system level by picking a dedicated write > node. > > Dominik > > 2018-03-13 11:28 GMT+01:00 : > > Hi guys, >> >> I'm running 2 kea dhcp servers and connect them to galera-mysql >> cluster for HA. >> Everything is ok, but sometimes i have this message in logs: >> >> 2018-03-13 14:14:45.036 ERROR [kea-dhcp4.alloc-engine/11864] >> ALLOC_ENGINE_V4_ALLOC_ERROR [hwtype=1 b8:53:ac:xx:xx:xx], >> cid=[01:b8:53:ac:xx:xx:xx], tid=0xa56d03a4: error during attempt >> to allocate an IPv4 address: unable to execute for > address, hwaddr, client_id, valid_lifetime, expire, subnet_id, >> fqdn_fwd, fqdn_rev, hostname, state FROM lease4 WHERE client_id = >> ? AND subnet_id = ?>, reason: Deadlock found when trying to get >> lock; try restarting transaction (error code 1213) >> >> the network is relatively small - not more than 50-100 requests >> per second >> >> How can I get rid of these messages? Cause after getting this >> error, server sends NACK to client >> ___ >> Kea-users mailing list >> Kea-users@lists.isc.org >> https://lists.isc.org/mailman/listinfo/kea-users [1] >> > > ___ > Kea-users mailing list > Kea-users@lists.isc.org > https://lists.isc.org/mailman/listinfo/kea-users [1] > Links: -- [1] https://lists.isc.org/mailman/listinfo/kea-users >>> ___ >>> Kea-users mailing list >>> Kea-users@lists.isc.org >>> https://lists.isc.org/mailman/listinfo/kea-users >>> >> >> >> ___ >> Kea-users mailing list >> Kea-users@lists.isc.org >> https://lists.isc.org/mailman/listinfo/kea-users >> >> > ___ Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users
Re: [Kea-users] HA based on galera
Hey Klaus, Can you elaborate a bit on your design? I am in the middle of testing Kea 1.3 with a 3 node mysql setup, and have not seen any issues yet. I researched both MySQL-Cluster and MySQL Group Replication in MySQL5.7. I settled on first testing with group replication, since it did not require an extra management server arbitrating the writes. The group replication HA allows all devices to be a master, and uses GTIDs to prevent 2 nodes from writing the same row. So my curiosity here is if I am missing something in the design, and how I would easily stress test this, to see if the SQL group can handle the distributed writes from the Kea server. Thanks, Jason On Tue, Mar 13, 2018 at 8:17 PM, Klaus Steden wrote: > > We have HA-MySQL behind our Kea servers, but the scopes themselves are > sharded to avoid this kind of write contention. In our case, we've got a > network partition that serves as an effective bright line between who's > responsible for what. > > cheers > Klaus > > On Tue, Mar 13, 2018 at 4:18 AM, wrote: > >> Hey, pal, that's my thread )) >> >> if you installed boost and configure can't find it - use ./configure >> --help >> there should be option --with-boost-source or smth like that >> >> >> >> On 2018-03-13 15:10, Tobi Obadiah wrote: >> >>> Hello there, >>> >>> Am just getting started with Kea, on my attempt to install kea on >>> Ubuntu16.04 desktop I keep getting the "Missing required header >>> files" errors >>> I have to manage to fix some of them but wonder if there is a standard >>> procedure to get this fix. Though I have boost installed >>> "boost/shared_ptr.hpp" >>> is reported missing. >>> >>> I appreciate your help. >>> >>> On Tue, Mar 13, 2018 at 11:37 AM, Dominik Epple >>> wrote: >>> >>> Hi, it might help if you pick a dedicated galera write node, i.e. both of your kea servers write to the same galera node at each point in time. You can still configure that in a HA fashion by some active-passive loadbalancer or a failover IP or something. The other option would be that the kea code gets adjusted for running "galera-aware" but I am no expert on what exactly needs to happen there. But if it can't be fixed on application level, it probably can be fixed on system level by picking a dedicated write node. Dominik 2018-03-13 11:28 GMT+01:00 : Hi guys, > > I'm running 2 kea dhcp servers and connect them to galera-mysql > cluster for HA. > Everything is ok, but sometimes i have this message in logs: > > 2018-03-13 14:14:45.036 ERROR [kea-dhcp4.alloc-engine/11864] > ALLOC_ENGINE_V4_ALLOC_ERROR [hwtype=1 b8:53:ac:xx:xx:xx], > cid=[01:b8:53:ac:xx:xx:xx], tid=0xa56d03a4: error during attempt > to allocate an IPv4 address: unable to execute for address, hwaddr, client_id, valid_lifetime, expire, subnet_id, > fqdn_fwd, fqdn_rev, hostname, state FROM lease4 WHERE client_id = > ? AND subnet_id = ?>, reason: Deadlock found when trying to get > lock; try restarting transaction (error code 1213) > > the network is relatively small - not more than 50-100 requests > per second > > How can I get rid of these messages? Cause after getting this > error, server sends NACK to client > ___ > Kea-users mailing list > Kea-users@lists.isc.org > https://lists.isc.org/mailman/listinfo/kea-users [1] > ___ Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users [1] >>> >>> >>> >>> Links: >>> -- >>> [1] https://lists.isc.org/mailman/listinfo/kea-users >>> >> ___ >> Kea-users mailing list >> Kea-users@lists.isc.org >> https://lists.isc.org/mailman/listinfo/kea-users >> > > > ___ > Kea-users mailing list > Kea-users@lists.isc.org > https://lists.isc.org/mailman/listinfo/kea-users > > ___ Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users
Re: [Kea-users] HA based on galera
On Tue, 13 Mar 2018, l...@ezelink.com wrote: I'm running 2 kea dhcp servers and connect them to galera-mysql cluster for HA. Everything is ok, but sometimes i have this message in logs: 2018-03-13 14:14:45.036 ERROR [kea-dhcp4.alloc-engine/11864] ALLOC_ENGINE_V4_ALLOC_ERROR [hwtype=1 b8:53:ac:xx:xx:xx], cid=[01:b8:53:ac:xx:xx:xx], tid=0xa56d03a4: error during attempt to allocate an IPv4 address: unable to execute for valid_lifetime, expire, subnet_id, fqdn_fwd, fqdn_rev, hostname, state FROM lease4 WHERE client_id = ? AND subnet_id = ?>, reason: Deadlock found when trying to get lock; try restarting transaction (error code 1213) the network is relatively small - not more than 50-100 requests per second How can I get rid of these messages? Cause after getting this error, server sends NACK to client You could always work around it with a hook that throws NAKs on the floor. Owen ___ Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users
Re: [Kea-users] HA based on galera
We have HA-MySQL behind our Kea servers, but the scopes themselves are sharded to avoid this kind of write contention. In our case, we've got a network partition that serves as an effective bright line between who's responsible for what. cheers Klaus On Tue, Mar 13, 2018 at 4:18 AM, wrote: > Hey, pal, that's my thread )) > > if you installed boost and configure can't find it - use ./configure --help > there should be option --with-boost-source or smth like that > > > > On 2018-03-13 15:10, Tobi Obadiah wrote: > >> Hello there, >> >> Am just getting started with Kea, on my attempt to install kea on >> Ubuntu16.04 desktop I keep getting the "Missing required header >> files" errors >> I have to manage to fix some of them but wonder if there is a standard >> procedure to get this fix. Though I have boost installed >> "boost/shared_ptr.hpp" >> is reported missing. >> >> I appreciate your help. >> >> On Tue, Mar 13, 2018 at 11:37 AM, Dominik Epple >> wrote: >> >> Hi, >>> >>> it might help if you pick a dedicated galera write node, i.e. both >>> of your kea servers write to the same galera node at each point in >>> time. You can still configure that in a HA fashion by some >>> active-passive loadbalancer or a failover IP or something. >>> >>> The other option would be that the kea code gets adjusted for >>> running "galera-aware" but I am no expert on what exactly needs to >>> happen there. But if it can't be fixed on application level, it >>> probably can be fixed on system level by picking a dedicated write >>> node. >>> >>> Dominik >>> >>> 2018-03-13 11:28 GMT+01:00 : >>> >>> Hi guys, I'm running 2 kea dhcp servers and connect them to galera-mysql cluster for HA. Everything is ok, but sometimes i have this message in logs: 2018-03-13 14:14:45.036 ERROR [kea-dhcp4.alloc-engine/11864] ALLOC_ENGINE_V4_ALLOC_ERROR [hwtype=1 b8:53:ac:xx:xx:xx], cid=[01:b8:53:ac:xx:xx:xx], tid=0xa56d03a4: error during attempt to allocate an IPv4 address: unable to execute for >>> address, hwaddr, client_id, valid_lifetime, expire, subnet_id, fqdn_fwd, fqdn_rev, hostname, state FROM lease4 WHERE client_id = ? AND subnet_id = ?>, reason: Deadlock found when trying to get lock; try restarting transaction (error code 1213) the network is relatively small - not more than 50-100 requests per second How can I get rid of these messages? Cause after getting this error, server sends NACK to client ___ Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users [1] >>> >>> ___ >>> Kea-users mailing list >>> Kea-users@lists.isc.org >>> https://lists.isc.org/mailman/listinfo/kea-users [1] >>> >> >> >> >> Links: >> -- >> [1] https://lists.isc.org/mailman/listinfo/kea-users >> > ___ > Kea-users mailing list > Kea-users@lists.isc.org > https://lists.isc.org/mailman/listinfo/kea-users > ___ Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users
Re: [Kea-users] HA based on galera
Hey, pal, that's my thread )) if you installed boost and configure can't find it - use ./configure --help there should be option --with-boost-source or smth like that On 2018-03-13 15:10, Tobi Obadiah wrote: Hello there, Am just getting started with Kea, on my attempt to install kea on Ubuntu16.04 desktop I keep getting the "Missing required header files" errors I have to manage to fix some of them but wonder if there is a standard procedure to get this fix. Though I have boost installed "boost/shared_ptr.hpp" is reported missing. I appreciate your help. On Tue, Mar 13, 2018 at 11:37 AM, Dominik Epple wrote: Hi, it might help if you pick a dedicated galera write node, i.e. both of your kea servers write to the same galera node at each point in time. You can still configure that in a HA fashion by some active-passive loadbalancer or a failover IP or something. The other option would be that the kea code gets adjusted for running "galera-aware" but I am no expert on what exactly needs to happen there. But if it can't be fixed on application level, it probably can be fixed on system level by picking a dedicated write node. Dominik 2018-03-13 11:28 GMT+01:00 : Hi guys, I'm running 2 kea dhcp servers and connect them to galera-mysql cluster for HA. Everything is ok, but sometimes i have this message in logs: 2018-03-13 14:14:45.036 ERROR [kea-dhcp4.alloc-engine/11864] ALLOC_ENGINE_V4_ALLOC_ERROR [hwtype=1 b8:53:ac:xx:xx:xx], cid=[01:b8:53:ac:xx:xx:xx], tid=0xa56d03a4: error during attempt to allocate an IPv4 address: unable to execute for , reason: Deadlock found when trying to get lock; try restarting transaction (error code 1213) the network is relatively small - not more than 50-100 requests per second How can I get rid of these messages? Cause after getting this error, server sends NACK to client ___ Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users [1] ___ Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users [1] Links: -- [1] https://lists.isc.org/mailman/listinfo/kea-users ___ Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users
Re: [Kea-users] HA based on galera
I am running a similar configuration with other weirdness. I was told that it is currently an unsupported configuration until version 1.4 On Tue, Mar 13, 2018 at 7:11 AM wrote: > Thanks, Dominik, > > I don't think the dedicated write node will help, cause mysql reports > deadlock. > that's probably sequential locks which are not properly adjusted. > > On 2018-03-13 14:37, Dominik Epple wrote: > > Hi, > > > > it might help if you pick a dedicated galera write node, i.e. both of > > your kea servers write to the same galera node at each point in time. > > You can still configure that in a HA fashion by some active-passive > > loadbalancer or a failover IP or something. > > > > The other option would be that the kea code gets adjusted for running > > "galera-aware" but I am no expert on what exactly needs to happen > > there. But if it can't be fixed on application level, it probably can > > be fixed on system level by picking a dedicated write node. > > > > Dominik > > > > 2018-03-13 11:28 GMT+01:00 : > > > >> Hi guys, > >> > >> I'm running 2 kea dhcp servers and connect them to galera-mysql > >> cluster for HA. > >> Everything is ok, but sometimes i have this message in logs: > >> > >> 2018-03-13 14:14:45.036 ERROR [kea-dhcp4.alloc-engine/11864] > >> ALLOC_ENGINE_V4_ALLOC_ERROR [hwtype=1 b8:53:ac:xx:xx:xx], > >> cid=[01:b8:53:ac:xx:xx:xx], tid=0xa56d03a4: error during attempt to > >> allocate an IPv4 address: unable to execute for >> hwaddr, client_id, valid_lifetime, expire, subnet_id, fqdn_fwd, > >> fqdn_rev, hostname, state FROM lease4 WHERE client_id = ? AND > >> subnet_id = ?>, reason: Deadlock found when trying to get lock; try > >> restarting transaction (error code 1213) > >> > >> the network is relatively small - not more than 50-100 requests per > >> second > >> > >> How can I get rid of these messages? Cause after getting this error, > >> server sends NACK to client > >> ___ > >> Kea-users mailing list > >> Kea-users@lists.isc.org > >> https://lists.isc.org/mailman/listinfo/kea-users [1] > > > > > > > > Links: > > -- > > [1] https://lists.isc.org/mailman/listinfo/kea-users > ___ > Kea-users mailing list > Kea-users@lists.isc.org > https://lists.isc.org/mailman/listinfo/kea-users > -- Munroe Sollog Senior Network Engineer mun...@lehigh.edu ___ Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users
Re: [Kea-users] HA based on galera
Thanks, Dominik, I don't think the dedicated write node will help, cause mysql reports deadlock. that's probably sequential locks which are not properly adjusted. On 2018-03-13 14:37, Dominik Epple wrote: Hi, it might help if you pick a dedicated galera write node, i.e. both of your kea servers write to the same galera node at each point in time. You can still configure that in a HA fashion by some active-passive loadbalancer or a failover IP or something. The other option would be that the kea code gets adjusted for running "galera-aware" but I am no expert on what exactly needs to happen there. But if it can't be fixed on application level, it probably can be fixed on system level by picking a dedicated write node. Dominik 2018-03-13 11:28 GMT+01:00 : Hi guys, I'm running 2 kea dhcp servers and connect them to galera-mysql cluster for HA. Everything is ok, but sometimes i have this message in logs: 2018-03-13 14:14:45.036 ERROR [kea-dhcp4.alloc-engine/11864] ALLOC_ENGINE_V4_ALLOC_ERROR [hwtype=1 b8:53:ac:xx:xx:xx], cid=[01:b8:53:ac:xx:xx:xx], tid=0xa56d03a4: error during attempt to allocate an IPv4 address: unable to execute for , reason: Deadlock found when trying to get lock; try restarting transaction (error code 1213) the network is relatively small - not more than 50-100 requests per second How can I get rid of these messages? Cause after getting this error, server sends NACK to client ___ Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users [1] Links: -- [1] https://lists.isc.org/mailman/listinfo/kea-users ___ Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users
Re: [Kea-users] HA based on galera
Hello there, Am just getting started with Kea, on my attempt to install kea on Ubuntu16.04 desktop I keep getting the "Missing required header files" errors I have to manage to fix some of them but wonder if there is a standard procedure to get this fix. Though I have boost installed "boost/shared_ptr.hpp" is reported missing. I appreciate your help. On Tue, Mar 13, 2018 at 11:37 AM, Dominik Epple wrote: > Hi, > > it might help if you pick a dedicated galera write node, i.e. both of your > kea servers write to the same galera node at each point in time. You can > still configure that in a HA fashion by some active-passive loadbalancer or > a failover IP or something. > > The other option would be that the kea code gets adjusted for running > "galera-aware" but I am no expert on what exactly needs to happen there. > But if it can't be fixed on application level, it probably can be fixed on > system level by picking a dedicated write node. > > Dominik > > 2018-03-13 11:28 GMT+01:00 : > >> Hi guys, >> >> I'm running 2 kea dhcp servers and connect them to galera-mysql cluster >> for HA. >> Everything is ok, but sometimes i have this message in logs: >> >> 2018-03-13 14:14:45.036 ERROR [kea-dhcp4.alloc-engine/11864] >> ALLOC_ENGINE_V4_ALLOC_ERROR [hwtype=1 b8:53:ac:xx:xx:xx], >> cid=[01:b8:53:ac:xx:xx:xx], tid=0xa56d03a4: error during attempt to >> allocate an IPv4 address: unable to execute for > client_id, valid_lifetime, expire, subnet_id, fqdn_fwd, fqdn_rev, hostname, >> state FROM lease4 WHERE client_id = ? AND subnet_id = ?>, reason: Deadlock >> found when trying to get lock; try restarting transaction (error code 1213) >> >> the network is relatively small - not more than 50-100 requests per second >> >> How can I get rid of these messages? Cause after getting this error, >> server sends NACK to client >> ___ >> Kea-users mailing list >> Kea-users@lists.isc.org >> https://lists.isc.org/mailman/listinfo/kea-users >> > > > ___ > Kea-users mailing list > Kea-users@lists.isc.org > https://lists.isc.org/mailman/listinfo/kea-users > > ___ Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users
Re: [Kea-users] HA based on galera
Hi, it might help if you pick a dedicated galera write node, i.e. both of your kea servers write to the same galera node at each point in time. You can still configure that in a HA fashion by some active-passive loadbalancer or a failover IP or something. The other option would be that the kea code gets adjusted for running "galera-aware" but I am no expert on what exactly needs to happen there. But if it can't be fixed on application level, it probably can be fixed on system level by picking a dedicated write node. Dominik 2018-03-13 11:28 GMT+01:00 : > Hi guys, > > I'm running 2 kea dhcp servers and connect them to galera-mysql cluster > for HA. > Everything is ok, but sometimes i have this message in logs: > > 2018-03-13 14:14:45.036 ERROR [kea-dhcp4.alloc-engine/11864] > ALLOC_ENGINE_V4_ALLOC_ERROR [hwtype=1 b8:53:ac:xx:xx:xx], > cid=[01:b8:53:ac:xx:xx:xx], tid=0xa56d03a4: error during attempt to > allocate an IPv4 address: unable to execute for client_id, valid_lifetime, expire, subnet_id, fqdn_fwd, fqdn_rev, hostname, > state FROM lease4 WHERE client_id = ? AND subnet_id = ?>, reason: Deadlock > found when trying to get lock; try restarting transaction (error code 1213) > > the network is relatively small - not more than 50-100 requests per second > > How can I get rid of these messages? Cause after getting this error, > server sends NACK to client > ___ > Kea-users mailing list > Kea-users@lists.isc.org > https://lists.isc.org/mailman/listinfo/kea-users > ___ Kea-users mailing list Kea-users@lists.isc.org https://lists.isc.org/mailman/listinfo/kea-users