Re: DNS resolution every second - v2.0.10
On Thu, Nov 28, 2019 at 2:17 PM Julien Pivotto wrote: > On 28 Nov 11:02, Baptiste wrote: > > On Thu, Nov 28, 2019 at 10:56 AM Julien Pivotto > > wrote: > > > > > On 28 Nov 10:38, Baptiste wrote: > > > > 'hold valid' still prevents HAProxy from changing the status of the > > > server > > > > in current Valid status to an other status for that period of time. > > > > Imagine your server is UP, DNS is valid, then your server returns NX > for > > > 2 > > > > minutes, then the status of the server won't change. If NX is > returned > > > for > > > > more than 5 minutes (as stated in your config), then it will change. > > > > > > > > Baptiste > > > > > > That is really great. Does it mean that with > > > > > > hold valid 1h > > > timeout resolve 30s > > > > > > we can have: > > > 1h of DNS downtime without impact on haproxy > > > > > > but if DNS is up, any change will be picked after 30 seconds? > > > > > > > > yep exactly! > > Previous behavior was wrong (using hold valid as timeout resolve). > > hold > Defines during which the last name resolution should be kept > based on last resolution > > So ... I guess the documentation is not clear here. > Would you mind clarifying it? I read it as: > > host valid 300s > > define a period of 300s during which the last name resolution should be > kept based > on last valid resolution > > I understand: if we get a valid resolution, we keep the last name > resolution for 300s. > > > > > > Baptiste > > -- > (o-Julien Pivotto > //\Open-Source Consultant > V_/_ Inuits - https://www.inuits.eu Actually, it's the status pointed to by "hold" of the latest resolution which is kept. I'll update the documentation to make it clearer. Baptiste
Re: DNS resolution every second - v2.0.10
On 28 Nov 11:02, Baptiste wrote: > On Thu, Nov 28, 2019 at 10:56 AM Julien Pivotto > wrote: > > > On 28 Nov 10:38, Baptiste wrote: > > > 'hold valid' still prevents HAProxy from changing the status of the > > server > > > in current Valid status to an other status for that period of time. > > > Imagine your server is UP, DNS is valid, then your server returns NX for > > 2 > > > minutes, then the status of the server won't change. If NX is returned > > for > > > more than 5 minutes (as stated in your config), then it will change. > > > > > > Baptiste > > > > That is really great. Does it mean that with > > > > hold valid 1h > > timeout resolve 30s > > > > we can have: > > 1h of DNS downtime without impact on haproxy > > > > but if DNS is up, any change will be picked after 30 seconds? > > > > > yep exactly! > Previous behavior was wrong (using hold valid as timeout resolve). hold Defines during which the last name resolution should be kept based on last resolution So ... I guess the documentation is not clear here. Would you mind clarifying it? I read it as: host valid 300s define a period of 300s during which the last name resolution should be kept based on last valid resolution I understand: if we get a valid resolution, we keep the last name resolution for 300s. > > Baptiste -- (o-Julien Pivotto //\Open-Source Consultant V_/_ Inuits - https://www.inuits.eu signature.asc Description: PGP signature
Re: DNS resolution every second - v2.0.10
Hello, On Thu, Nov 28, 2019 at 10:35 AM Baptiste wrote: > > @Willy, since 1.8 (I think), the DNS task is autonomous and not triggered by > the check anymore. > > Second, HAProxy never ever follows up TTLs. > > Third, I "fixed" a bug in 2.0.10 which triggers this change of behavior. > Basically, "timeout resolve" which is supposed to be the interval between 2 > DNS resolutions was not applied when the response was valid. > (f50e1ac4442be41ed8b9b7372310d1d068b85b33) For clarification and future reference, this bug fix is in: 2.1: 2.1.0 (first release) 2.0: 2.0.9 1.9: 1.9.13 1.8: 1.8.23 Lukas
Re: DNS resolution every second - v2.0.10
On Thu, Nov 28, 2019 at 10:56 AM Julien Pivotto wrote: > On 28 Nov 10:38, Baptiste wrote: > > 'hold valid' still prevents HAProxy from changing the status of the > server > > in current Valid status to an other status for that period of time. > > Imagine your server is UP, DNS is valid, then your server returns NX for > 2 > > minutes, then the status of the server won't change. If NX is returned > for > > more than 5 minutes (as stated in your config), then it will change. > > > > Baptiste > > That is really great. Does it mean that with > > hold valid 1h > timeout resolve 30s > > we can have: > 1h of DNS downtime without impact on haproxy > > but if DNS is up, any change will be picked after 30 seconds? > > yep exactly! Previous behavior was wrong (using hold valid as timeout resolve). Baptiste
Re: DNS resolution every second - v2.0.10
On 28 Nov 10:38, Baptiste wrote: > 'hold valid' still prevents HAProxy from changing the status of the server > in current Valid status to an other status for that period of time. > Imagine your server is UP, DNS is valid, then your server returns NX for 2 > minutes, then the status of the server won't change. If NX is returned for > more than 5 minutes (as stated in your config), then it will change. > > Baptiste That is really great. Does it mean that with hold valid 1h timeout resolve 30s we can have: 1h of DNS downtime without impact on haproxy but if DNS is up, any change will be picked after 30 seconds? -- (o-Julien Pivotto //\Open-Source Consultant V_/_ Inuits - https://www.inuits.eu signature.asc Description: PGP signature
Re: DNS resolution every second - v2.0.10
'hold valid' still prevents HAProxy from changing the status of the server in current Valid status to an other status for that period of time. Imagine your server is UP, DNS is valid, then your server returns NX for 2 minutes, then the status of the server won't change. If NX is returned for more than 5 minutes (as stated in your config), then it will change. Baptiste
Re: DNS resolution every second - v2.0.10
@Willy, since 1.8 (I think), the DNS task is autonomous and not triggered by the check anymore. Second, HAProxy never ever follows up TTLs. Third, I "fixed" a bug in 2.0.10 which triggers this change of behavior. Basically, "timeout resolve" which is supposed to be the interval between 2 DNS resolutions was not applied when the response was valid. (f50e1ac4442be41ed8b9b7372310d1d068b85b33) So to recover from previous behavior, just increase this value, which is by default 1s. Baptiste
Re: DNS resolution every second - v2.0.10
Hi! > If it bothers you (I don't really see why), you can increase the "inter" > value on your servers to check them less often and as such refresh their > address less often. You can configure "hold valid " to configure internal caching (it should be 10 seconds by default though): I post here a full configuration that produces the behaviour (sorry, it is long). In my tests the DNS is contacted every second even if: . I set "hold valid 5m" . I disable the "check" for the servers As I would expect, the resolution correctly happens only once for each server name even if the names are repeated in several backends. This is not a big problem! Thank you anyway .marcoc --- global nbproc 1 log /var/lib/haproxy/dev/log local0 user haproxy group haproxy stats socket /run/haproxy/admin.sock mode 660 level admin spread-checks 4 hard-stop-after 2d daemon ssl-server-verify none ssl-default-bind-options ssl-min-ver TLSv1.2 ssl-default-bind-ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256 ssl-default-bind-ciphersuites TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256 ssl-default-server-options ssl-min-ver TLSv1.2 ssl-default-server-ciphers ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256 defaults log global retries 3 option redispatch option splice-auto balance roundrobin timeout connect 2s timeout client 2m timeout server 10m timeout check 4s timeout client-fin 20s timeout tunnel 1h timeout http-request 5s compression algo gzip compression type text/html text/css text/javascript text/xml text/plain image/bmp image/x-icon image/svg+xml application/rss+xml application/javascript application/x-javascript application/xml application/xhtml+xml application/x-font application/x-font-truetype application/x-font-ttf application/x-font-otf application/x-font-opentype application/vnd.ms-fontobject application/x-amf application/x-shockwave-flash font/ttf font/otf font/opentype mode http option httplog default-server inter 30s fastinter 15s downinter 60s fall 5 rise 3 resolvers systemd resolve-prefer ipv4 init-addr libc,none resolvers systemd nameserver local 127.0.0.53:53 hold valid 5m frontend tesi-http bind 10.64.44.112:80 name HTTP bind 10.64.44.112:443 name HTTPS ssl crt /etc/ssl/private/tesitest.uno.due curves X25519:P-256 alpn h2,http/1.1 timeout client 10m option forwardfor stick-table type ip size 1000 expire 10m store http_req_rate(10s),http_err_rate(10s) http-request track-sc0 src http-request deny deny_status 429 if { sc_http_err_rate(0) gt 20 } http-request deny deny_status 429 if { sc_http_req_rate(0) gt 200 } http-request redirect scheme https code 301 if !{ ssl_fc } http-response set-header Strict-Transport-Security max-age=1600;\ includeSubDomains http-response replace-header Set-Cookie (.*) \1;\ Secure http-response replace-header Location http://(.+) https://\1 acl tesitest hdr(host) tesitest.uno.due acl fb path_beg /fb acl tec path_beg /tec use_backend prefb-http if tesipre fb use_backend pretec-http if tesipre tec use_backend preportal-http if tesipre backend preportal-http option httpchk HEAD / cookie prs insert indirect nocache httponly maxidle 4h secure server vxws142a-82 vxws142a-tesife.uno.due:82 weight 2 cookie a check server vxws142b-82 vxws142b-tesife.uno.due:82 weight 2 cookie b check backend prefb-http option httpchk HEAD /fb/client/util/checkIstance.cfm cookie fbrs insert indirect nocache httponly maxidle 4h secure server vxws142a-80 vxws142a-tesife.uno.due:80 weight 2 cookie a check server vxws142b-80 vxws142b-tesife.uno.due:80 weight 2 cookie b check backend pretec-http option httpchk HEAD /tec/client/util/checkIstance.cfm cookie tecrs insert indirect nocache httponly maxidle 4h secure server vxws142a-83 vxws142a-tesife.uno.due:83 weight 2 cookie a check server vxws142b-83 vxws142b-tesife.uno.due:83 weight 2 cookie b check listen stats-http bind 10.64.44.112:8443 name HTTPS ssl crt /etc/ssl/private/tesitest.uno.due-full.pem curves X25519:P-256 alpn h2,http/1.1 stats enable stats show-legends stats show-node stats refresh 60s stats uri /haproxystats ---
Re: DNS resolution every second - v2.0.10
Hi Lukas, On Wed, Nov 27, 2019 at 11:53:03PM +0100, Lukas Tribus wrote: > > If it bothers you (I don't really see why), you can increase the "inter" > > value on your servers to check them less often and as such refresh their > > address less often. > > You can configure "hold valid " to configure internal caching > (it should be 10 seconds by default though): > > https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#5.3.2-hold Ah thanks, I had vague memories about something around this but I didn't remember if it was just a discussion or something actually implemented :-) Willy
Re: DNS resolution every second - v2.0.10
Hello, On Wed, Nov 27, 2019 at 10:25 PM Willy Tarreau wrote: > > Hi Marco, > > On Wed, Nov 27, 2019 at 08:38:03AM +0100, Marco Corte wrote: > > Hello! > > > > I see a strange behaviour of the DNS resolution on version 2.0.9 and 2.0.10, > > but I do not know since when this happens. > > > > On Ubuntu 18.04, I set up haproxy to use the local DNS service provided by > > systemd. > > Actually I see that haproxy tries to resolve the names every second. > > The resolution is successful, the TTL in the answer is 1800s or longer. > > It's the interval of your server checks. The resolution is sent at the same > frequency as the checks. The reason is that in environments using dynamic > server addresses, you definitely need to check your address at the same > frequency you care for the server's availability. In addition, in some > fast-recycling environments (AWS used to be like this, I don't know if > it's still the case), when you shut down your server, it was possible that > its IP address was reassigned so fast that the server didn't even have the > time to appear dead, and the traffic could continue to be delivered to > someone else's server! > > If it bothers you (I don't really see why), you can increase the "inter" > value on your servers to check them less often and as such refresh their > address less often. You can configure "hold valid " to configure internal caching (it should be 10 seconds by default though): https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#5.3.2-hold Lukas
Re: DNS resolution every second - v2.0.10
Hi Marco, On Wed, Nov 27, 2019 at 08:38:03AM +0100, Marco Corte wrote: > Hello! > > I see a strange behaviour of the DNS resolution on version 2.0.9 and 2.0.10, > but I do not know since when this happens. > > On Ubuntu 18.04, I set up haproxy to use the local DNS service provided by > systemd. > Actually I see that haproxy tries to resolve the names every second. > The resolution is successful, the TTL in the answer is 1800s or longer. It's the interval of your server checks. The resolution is sent at the same frequency as the checks. The reason is that in environments using dynamic server addresses, you definitely need to check your address at the same frequency you care for the server's availability. In addition, in some fast-recycling environments (AWS used to be like this, I don't know if it's still the case), when you shut down your server, it was possible that its IP address was reassigned so fast that the server didn't even have the time to appear dead, and the traffic could continue to be delivered to someone else's server! If it bothers you (I don't really see why), you can increase the "inter" value on your servers to check them less often and as such refresh their address less often. Hoping this helps, Willy