Re: haproxy backend server template service discovery questions
On Sat, 6 Apr 2024 at 20:17, Илья Шипицин wrote: > > Consul template is something done by consul itself, after that haproxy.conf > is rendered > > Do you mean "how haproxy deals with rendered template"? > He doesn't use that method of discovery, he uses DNS resolvers so haproxy gets the SRV records with the list of servers of backends. Cheers, Pavlos
Re: haproxy backend server template service discovery questions
On Fri, 5 Apr 2024 at 15:00, Andrii Ustymenko wrote: > > Dear list! > > My name is Andrii. I work for Adyen. We are using haproxy as our main > software loadbalancer at quite large scale. > Af of now our main use-case for backends routing based on > server-template and dynamic dns from consul as service discovery. Below > is the example of simple backend configuration: > > ``` > backend example >balance roundrobin >server-template application 10 _application._tcp.consul resolvers > someresolvers init-addr last,libc,none resolve-opts allow-dup-ip > resolve-prefer ipv4 check ssl verify none > ``` > > and in global configuration > > ``` > resolvers someresolvers >nameserver ns1 10.10.10.10:53 >nameserver ns2 10.10.10.11:53 > ``` > > As we see haproxy will create internal table for backends with some > be_id and be_name=application and allocate 10 records for each server > with se_id from 1 to 10. Then those records get populated and updated > with the data from resolvers. > I would like to understand couple of things with regards to this > structure and how it works, which I could not figure out myself from the > source code: > > 1) In tcpdump for dns queries we see that haproxy asynchronously polls > all the nameservers simultaneously. For instance: > > ``` > 11:06:17.587798 eth2 Out ifindex 4 aa:aa:aa:aa:aa:aa ethertype IPv4 > (0x0800), length 108: 10.10.10.50.24050 > 10.10.10.10.53: 34307+ [1au] > SRV? _application._tcp.consul. (60) > 11:06:17.587802 eth2 Out ifindex 4 aa:aa:aa:aa:aa:aa ethertype IPv4 > (0x0800), length 108: 10.10.10.50.63155 > 10.10.10.11.53: 34307+ [1au] > SRV? _application._tcp.consul. (60) > 11:06:17.588097 eth2 In ifindex 4 ff:ff:ff:ff:ff:ff ethertype IPv4 > (0x0800), length 205: 10.10.10.10.53 > 10.10.10.50.24050: 2194 2/0/1 SRV > 0a5099e5.addr.consul.:25340 1 1, SRV 0a509934.addr.consul.:26010 1 1 (157) > 11:06:17.588097 eth2 In ifindex 4 ff:ff:ff:ff:ff:ff ethertype IPv4 > (0x0800), length 205: 10.10.10.11.53 > 10.10.10.50.63155: 2194 2/0/1 SRV > 0a5099e5.addr.consul.:25340 1 1, SRV 0a509934.addr.consul.:26010 1 1 (157) > ``` > > Both nameservers reply with the same response. But what if they are out > of sync? Let's say one says: server1, server2 and the second one says > server2, server3? So far testing this locally - I see sometimes the > reply overrides the table, but sometimes it seems to just gets merged > with the rest. > Most service discovery systems employ "eventually consistency". Therefore, some instances of the DNS servers in front of that system may have "stale" data. The instance usually becomes consistent with milliseconds or seconds; it depends on the topology. I don't think haproxy performs any kind of merging, I believe it uses the 1st valid response. > 2) Each entry from SRV reply will be placed into the table under > specific se_id. Most of the times that placement won't change. So, for > the example above the most likely 0a5099e5.addr.consul. and > 0a509934.addr.consul. will have se_id 1 and 2 respectively. However > sometimes we have the following scenario: > > 1. We admistratively disable the server (drain traffic) with the next > command: > > ``` > echo "set server example/application1 state maint" | nc -U > /var/lib/haproxy/stats > ``` > > the MAINT flag will be added to the record with se_id 1 > > 2. Instance of application goes down and gets de-registered from consul, > so also evicted from srv replies and out of discovery of haproxy. > > 3. Instance of application goes up and gets registered by consul and > discovered by haproxy, but haproxy allocates different se_id for it. > Haproxy healthchecks will control the traffic to it in this case. > > 4. We will still have se_id 1 with MAINT flag and application instance > dns record placed into different se_id. > > The problem comes that any new discovered record which get placed into > se_id 1 will never be active until either command: > > ``` > echo "set server example/application1 state ready" | nc -U > /var/lib/haproxy/stats > ``` > > executed or haproxy gets reloaded without state file. With this pattern > we basically have persistent "records pollution" due to operations made > directly with control socket. > This situation could lead to minor incidents where most newly registered servers are assigned to se_ids that were previously in maintenance mode. So, you end up with a backend that has, let's say, 75% of servers in maintenance mode. > I am not sure is there anything to do about this. Maybe, if haproxy > could cache the state not only of se_id but also associated record with > that and then if that gets changed - re-schedule healtchecks. Or instead > of integer ids use some hashed ids based on dns/ip-addresses of > discovered records, in this case binding will happen exactly in the same > slot. > Another approach could be haproxy resetting the `srv_admin_state ` field for IDs for which an entry was de-registered. My 2 cents, Pavlos
Re: [ANNOUNCE] haproxy-3.0-dev7
сб, 6 апр. 2024 г. в 17:53, Willy Tarreau : > Hi, > > HAProxy 3.0-dev7 was released on 2024/04/06. It added 73 new commits > after version 3.0-dev6. > > Among the changes that stand out in this version, here's what I'm seeing: > > - improvements to the CLI internal API so that the various keyword > handlers now have their own buffers. This might possibly uncover > a few long-lasting bugs but over time will improve the reliability > and avoid the occasional bugs with connections never closing or > spinning loops. > > - we no longer depend on libsystemd. Not only this will avoid pulling > in tons of questionable dependencies, this also allows to enable > USE_SYSTEMD by default (it's only done on linux-glibc though), thus > reducing config combinations. > > - log load-balancing internals were simplified. The very first version > (never merged) didn't rely on backends, thus used to implement its > own servers and load-balancing. It was finally remapped to backends > and real servers, but the LB algorithms had remained specific, with > some exceptions at various places in the setup code to handle them. > Now the backends have switched to regular LB algorithms, which not > only helps for code maintenance, but also exposes all table-based > algorithms to the log backends with support for weights, and also > exposed the "sticky" algorithm to TCP and HTTP backends. It's one of > these changes which remove code while adding features :-) > > - Linux capabilities are now properly checked so that haproxy won't > complain about permissions for example when used in transparent mode, > if capabilities are sufficient. In addition, file-system capabilities > set on the binary are also supported now. > > - stick-tables are now sharded over multiple tree heads each with their > own locks. This significantly reduces locking contention on systems > with many threads (gains of ~6x measured on a 80-thread systems). In > addition, the locking could be reduced even with low thread counts, > particulary when using peers, where the performance could be doubled. > > - cookies are now permitted for dynamically added servers. The only > reason they were not previously was that it required to audit the > whole cookie initialization/release code to figure whether it had > corner cases or not. With that audit now done, the cookies could > be allowed. In addition, dynamic cookies were supported a bit by > accident with a small defect (one had to set the address again to > index the server), and are now properly supported. > > - the "enabled" keyword used to be silently ignored when adding a > dynamic server. Now it's properly rejected to avoid confusing > scripts. We don't know yet if it will be supported later or not, > so better stay safe. > > - the key used by consistent hash to map to a server used to always > be the server's id (either explicit or implicit, position-based). > Now the "hash-key" directive will also allow to use the server's > address or address+port for this. The benefit is that multiple LBs > with servers in a different order will still send the same hashes > to the same servers. > > - a new "guid" keyword was added for servers, listeners and proxies. > The purpose will be to make it possible for external APIs to assign > a globally unique object identifier to each of them in stats dumps > or CLI accesses, and to later reliably recognize a server upon > reloads. For now the identifier is not exploited. > I have a question about the UUID version. it is not specified. Is it UUID version 6 ? > > - QUIC now supports the HyStart++ (RFC9406) alternative to slowstart > with the Cubic algorithm. It's supposed to show better recovery > patterns. More testing is needed before enabling it by default. > > - a few bug fixes (truncated responses when splicing, QUIC crashes > on strict-alignment platforms, redispatch 0 didn't work, more OCSP > update fixes, proper reporting of too big CLI payload, etc). > > - some build fixes, code cleanups, CI updates, doc updates, and > cleanups of regtests. > > I think that's all. It's currently up and running on haproxy.org. I'd > suspect that with the many stable updates yesterday, we may see less > test reports on 3.0-dev7, but please don't forget to test it if you > can, that helps a lot ;-) > > Please find the usual URLs below : >Site index : https://www.haproxy.org/ >Documentation: https://docs.haproxy.org/ >Wiki : https://github.com/haproxy/wiki/wiki >Discourse: https://discourse.haproxy.org/ >Slack channel: https://slack.haproxy.org/ >Issue tracker: https://github.com/haproxy/haproxy/issues >Sources : https://www.haproxy.org/download/3.0/src/ >Git repository : https://git.haproxy.org/git/haproxy.git/
