Re: DNS resolution every second - v2.0.10
On Thu, Nov 28, 2019 at 2:17 PM Julien Pivotto wrote: > On 28 Nov 11:02, Baptiste wrote: > > On Thu, Nov 28, 2019 at 10:56 AM Julien Pivotto > > wrote: > > > > > On 28 Nov 10:38, Baptiste wrote: > > > > 'hold valid' still prevents HAProxy from changing the status of the > > > server > > > > in current Valid status to an other status for that period of time. > > > > Imagine your server is UP, DNS is valid, then your server returns NX > for > > > 2 > > > > minutes, then the status of the server won't change. If NX is > returned > > > for > > > > more than 5 minutes (as stated in your config), then it will change. > > > > > > > > Baptiste > > > > > > That is really great. Does it mean that with > > > > > > hold valid 1h > > > timeout resolve 30s > > > > > > we can have: > > > 1h of DNS downtime without impact on haproxy > > > > > > but if DNS is up, any change will be picked after 30 seconds? > > > > > > > > yep exactly! > > Previous behavior was wrong (using hold valid as timeout resolve). > > hold > Defines during which the last name resolution should be kept > based on last resolution > > So ... I guess the documentation is not clear here. > Would you mind clarifying it? I read it as: > > host valid 300s > > define a period of 300s during which the last name resolution should be > kept based > on last valid resolution > > I understand: if we get a valid resolution, we keep the last name > resolution for 300s. > > > > > > Baptiste > > -- > (o-Julien Pivotto > //\Open-Source Consultant > V_/_ Inuits - https://www.inuits.eu Actually, it's the status pointed to by "hold" of the latest resolution which is kept. I'll update the documentation to make it clearer. Baptiste
Re: DNS resolution every second - v2.0.10
On 28 Nov 11:02, Baptiste wrote: > On Thu, Nov 28, 2019 at 10:56 AM Julien Pivotto > wrote: > > > On 28 Nov 10:38, Baptiste wrote: > > > 'hold valid' still prevents HAProxy from changing the status of the > > server > > > in current Valid status to an other status for that period of time. > > > Imagine your server is UP, DNS is valid, then your server returns NX for > > 2 > > > minutes, then the status of the server won't change. If NX is returned > > for > > > more than 5 minutes (as stated in your config), then it will change. > > > > > > Baptiste > > > > That is really great. Does it mean that with > > > > hold valid 1h > > timeout resolve 30s > > > > we can have: > > 1h of DNS downtime without impact on haproxy > > > > but if DNS is up, any change will be picked after 30 seconds? > > > > > yep exactly! > Previous behavior was wrong (using hold valid as timeout resolve). hold Defines during which the last name resolution should be kept based on last resolution So ... I guess the documentation is not clear here. Would you mind clarifying it? I read it as: host valid 300s define a period of 300s during which the last name resolution should be kept based on last valid resolution I understand: if we get a valid resolution, we keep the last name resolution for 300s. > > Baptiste -- (o-Julien Pivotto //\Open-Source Consultant V_/_ Inuits - https://www.inuits.eu signature.asc Description: PGP signature
Re: DNS resolution every second - v2.0.10
Hello, On Thu, Nov 28, 2019 at 10:35 AM Baptiste wrote: > > @Willy, since 1.8 (I think), the DNS task is autonomous and not triggered by > the check anymore. > > Second, HAProxy never ever follows up TTLs. > > Third, I "fixed" a bug in 2.0.10 which triggers this change of behavior. > Basically, "timeout resolve" which is supposed to be the interval between 2 > DNS resolutions was not applied when the response was valid. > (f50e1ac4442be41ed8b9b7372310d1d068b85b33) For clarification and future reference, this bug fix is in: 2.1: 2.1.0 (first release) 2.0: 2.0.9 1.9: 1.9.13 1.8: 1.8.23 Lukas
Re: DNS resolution every second - v2.0.10
On Thu, Nov 28, 2019 at 10:56 AM Julien Pivotto wrote: > On 28 Nov 10:38, Baptiste wrote: > > 'hold valid' still prevents HAProxy from changing the status of the > server > > in current Valid status to an other status for that period of time. > > Imagine your server is UP, DNS is valid, then your server returns NX for > 2 > > minutes, then the status of the server won't change. If NX is returned > for > > more than 5 minutes (as stated in your config), then it will change. > > > > Baptiste > > That is really great. Does it mean that with > > hold valid 1h > timeout resolve 30s > > we can have: > 1h of DNS downtime without impact on haproxy > > but if DNS is up, any change will be picked after 30 seconds? > > yep exactly! Previous behavior was wrong (using hold valid as timeout resolve). Baptiste
Re: DNS resolution every second - v2.0.10
On 28 Nov 10:38, Baptiste wrote: > 'hold valid' still prevents HAProxy from changing the status of the server > in current Valid status to an other status for that period of time. > Imagine your server is UP, DNS is valid, then your server returns NX for 2 > minutes, then the status of the server won't change. If NX is returned for > more than 5 minutes (as stated in your config), then it will change. > > Baptiste That is really great. Does it mean that with hold valid 1h timeout resolve 30s we can have: 1h of DNS downtime without impact on haproxy but if DNS is up, any change will be picked after 30 seconds? -- (o-Julien Pivotto //\Open-Source Consultant V_/_ Inuits - https://www.inuits.eu signature.asc Description: PGP signature
Re: DNS resolution every second - v2.0.10
'hold valid' still prevents HAProxy from changing the status of the server in current Valid status to an other status for that period of time. Imagine your server is UP, DNS is valid, then your server returns NX for 2 minutes, then the status of the server won't change. If NX is returned for more than 5 minutes (as stated in your config), then it will change. Baptiste
Re: DNS resolution every second - v2.0.10
@Willy, since 1.8 (I think), the DNS task is autonomous and not triggered by the check anymore. Second, HAProxy never ever follows up TTLs. Third, I "fixed" a bug in 2.0.10 which triggers this change of behavior. Basically, "timeout resolve" which is supposed to be the interval between 2 DNS resolutions was not applied when the response was valid. (f50e1ac4442be41ed8b9b7372310d1d068b85b33) So to recover from previous behavior, just increase this value, which is by default 1s. Baptiste
Re: DNS resolution every second - v2.0.10
Hi!
> If it bothers you (I don't really see why), you can increase the "inter"
> value on your servers to check them less often and as such refresh their
> address less often.
You can configure "hold valid " to configure internal caching
(it should be 10 seconds by default though):
I post here a full configuration that produces the behaviour (sorry, it
is long).
In my tests the DNS is contacted every second even if:
. I set "hold valid 5m"
. I disable the "check" for the servers
As I would expect, the resolution correctly happens only once for each
server name even if the names are repeated in several backends.
This is not a big problem!
Thank you anyway
.marcoc
---
global
nbproc 1
log /var/lib/haproxy/dev/log local0
user haproxy
group haproxy
stats socket /run/haproxy/admin.sock mode 660 level admin
spread-checks 4
hard-stop-after 2d
daemon
ssl-server-verify none
ssl-default-bind-options ssl-min-ver TLSv1.2
ssl-default-bind-ciphers
ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256
ssl-default-bind-ciphersuites
TLS_AES_128_GCM_SHA256:TLS_CHACHA20_POLY1305_SHA256
ssl-default-server-options ssl-min-ver TLSv1.2
ssl-default-server-ciphers
ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256
defaults
log global
retries 3
option redispatch
option splice-auto
balance roundrobin
timeout connect 2s
timeout client 2m
timeout server 10m
timeout check 4s
timeout client-fin 20s
timeout tunnel 1h
timeout http-request 5s
compression algo gzip
compression type text/html text/css text/javascript text/xml
text/plain image/bmp image/x-icon image/svg+xml application/rss+xml
application/javascript application/x-javascript application/xml
application/xhtml+xml application/x-font application/x-font-truetype
application/x-font-ttf application/x-font-otf
application/x-font-opentype application/vnd.ms-fontobject
application/x-amf application/x-shockwave-flash font/ttf font/otf
font/opentype
mode http
option httplog
default-server inter 30s fastinter 15s downinter 60s fall 5 rise 3
resolvers systemd resolve-prefer ipv4 init-addr libc,none
resolvers systemd
nameserver local 127.0.0.53:53
hold valid 5m
frontend tesi-http
bind 10.64.44.112:80 name HTTP
bind 10.64.44.112:443 name HTTPS ssl crt
/etc/ssl/private/tesitest.uno.due curves X25519:P-256 alpn h2,http/1.1
timeout client 10m
option forwardfor
stick-table type ip size 1000 expire 10m store
http_req_rate(10s),http_err_rate(10s)
http-request track-sc0 src
http-request deny deny_status 429 if { sc_http_err_rate(0) gt 20 }
http-request deny deny_status 429 if { sc_http_req_rate(0) gt 200 }
http-request redirect scheme https code 301 if !{ ssl_fc }
http-response set-header Strict-Transport-Security max-age=1600;\
includeSubDomains
http-response replace-header Set-Cookie (.*) \1;\ Secure
http-response replace-header Location http://(.+) https://\1
acl tesitest hdr(host) tesitest.uno.due
acl fb path_beg /fb
acl tec path_beg /tec
use_backend prefb-http if tesipre fb
use_backend pretec-http if tesipre tec
use_backend preportal-http if tesipre
backend preportal-http
option httpchk HEAD /
cookie prs insert indirect nocache httponly maxidle 4h secure
server vxws142a-82 vxws142a-tesife.uno.due:82 weight 2 cookie a check
server vxws142b-82 vxws142b-tesife.uno.due:82 weight 2 cookie b check
backend prefb-http
option httpchk HEAD /fb/client/util/checkIstance.cfm
cookie fbrs insert indirect nocache httponly maxidle 4h secure
server vxws142a-80 vxws142a-tesife.uno.due:80 weight 2 cookie a check
server vxws142b-80 vxws142b-tesife.uno.due:80 weight 2 cookie b check
backend pretec-http
option httpchk HEAD /tec/client/util/checkIstance.cfm
cookie tecrs insert indirect nocache httponly maxidle 4h secure
server vxws142a-83 vxws142a-tesife.uno.due:83 weight 2 cookie a check
server vxws142b-83 vxws142b-tesife.uno.due:83 weight 2 cookie b check
listen stats-http
bind 10.64.44.112:8443 name HTTPS ssl crt
/etc/ssl/private/tesitest.uno.due-full.pem curves X25519:P-256 alpn
h2,http/1.1
stats enable
stats show-legends
stats show-node
stats refresh 60s
stats uri /haproxystats
---
Re: DNS resolution every second - v2.0.10
Hi Lukas, On Wed, Nov 27, 2019 at 11:53:03PM +0100, Lukas Tribus wrote: > > If it bothers you (I don't really see why), you can increase the "inter" > > value on your servers to check them less often and as such refresh their > > address less often. > > You can configure "hold valid " to configure internal caching > (it should be 10 seconds by default though): > > https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#5.3.2-hold Ah thanks, I had vague memories about something around this but I didn't remember if it was just a discussion or something actually implemented :-) Willy
Re: DNS resolution every second - v2.0.10
Hello, On Wed, Nov 27, 2019 at 10:25 PM Willy Tarreau wrote: > > Hi Marco, > > On Wed, Nov 27, 2019 at 08:38:03AM +0100, Marco Corte wrote: > > Hello! > > > > I see a strange behaviour of the DNS resolution on version 2.0.9 and 2.0.10, > > but I do not know since when this happens. > > > > On Ubuntu 18.04, I set up haproxy to use the local DNS service provided by > > systemd. > > Actually I see that haproxy tries to resolve the names every second. > > The resolution is successful, the TTL in the answer is 1800s or longer. > > It's the interval of your server checks. The resolution is sent at the same > frequency as the checks. The reason is that in environments using dynamic > server addresses, you definitely need to check your address at the same > frequency you care for the server's availability. In addition, in some > fast-recycling environments (AWS used to be like this, I don't know if > it's still the case), when you shut down your server, it was possible that > its IP address was reassigned so fast that the server didn't even have the > time to appear dead, and the traffic could continue to be delivered to > someone else's server! > > If it bothers you (I don't really see why), you can increase the "inter" > value on your servers to check them less often and as such refresh their > address less often. You can configure "hold valid " to configure internal caching (it should be 10 seconds by default though): https://cbonte.github.io/haproxy-dconv/2.0/configuration.html#5.3.2-hold Lukas
Re: DNS resolution every second - v2.0.10
Hi Marco, On Wed, Nov 27, 2019 at 08:38:03AM +0100, Marco Corte wrote: > Hello! > > I see a strange behaviour of the DNS resolution on version 2.0.9 and 2.0.10, > but I do not know since when this happens. > > On Ubuntu 18.04, I set up haproxy to use the local DNS service provided by > systemd. > Actually I see that haproxy tries to resolve the names every second. > The resolution is successful, the TTL in the answer is 1800s or longer. It's the interval of your server checks. The resolution is sent at the same frequency as the checks. The reason is that in environments using dynamic server addresses, you definitely need to check your address at the same frequency you care for the server's availability. In addition, in some fast-recycling environments (AWS used to be like this, I don't know if it's still the case), when you shut down your server, it was possible that its IP address was reassigned so fast that the server didn't even have the time to appear dead, and the traffic could continue to be delivered to someone else's server! If it bothers you (I don't really see why), you can increase the "inter" value on your servers to check them less often and as such refresh their address less often. Hoping this helps, Willy
