Re: http-request do-resolve Woes
On Wed, Oct 30, 2019 at 4:48 PM David Birdsong wrote: > > On Wed, Oct 30, 2019 at 11:39 AM Baptiste wrote: > >> Thanks! >>> >>> It had that feel to it...seemed like a cache lock timeout and/or somehow >>> tied to the request interval. >>> >>> >> I think I know where to fix this behavior in the code. I will work on the >> "how to fix it" later tonight. >> In the meantime, you can apply the workaround below. This is doable >> because the DNS cache is per resolvers section: >> 1. create a second dummy DNS section: >> resolvers main_resolver_do-resolve >> nameserver dns1 8.8.8.8:53 >> >> which is a copy of the first one with a different name. >> >> 2. reference this new resolvers section in your do-resolve action: >> http-request do-resolve(txn.myip,main_resolver_do-resolve,ipv4) >> hdr(Host),lower >> >> And you should be good until I fix it and it's backported. >> > > Awesome, thanks! > > Quick question: should I pay attention to timer: Tr as a proxy for both > request received and DNS latency? I'm guessing that the capture and > dns-resolve cause delays in haproxy fully reading the request in, is that > right? > Capture should not take generate any delays. do-resolve does :) And yes, from what I saw, it is reported in HAProxy's Tr, so yes, you are correct. Baptiste
Re: http-request do-resolve Woes
On Wed, Oct 30, 2019 at 11:39 AM Baptiste wrote: > Thanks! >> >> It had that feel to it...seemed like a cache lock timeout and/or somehow >> tied to the request interval. >> >> > I think I know where to fix this behavior in the code. I will work on the > "how to fix it" later tonight. > In the meantime, you can apply the workaround below. This is doable > because the DNS cache is per resolvers section: > 1. create a second dummy DNS section: > resolvers main_resolver_do-resolve > nameserver dns1 8.8.8.8:53 > > which is a copy of the first one with a different name. > > 2. reference this new resolvers section in your do-resolve action: > http-request do-resolve(txn.myip,main_resolver_do-resolve,ipv4) > hdr(Host),lower > > And you should be good until I fix it and it's backported. > Awesome, thanks! Quick question: should I pay attention to timer: Tr as a proxy for both request received and DNS latency? I'm guessing that the capture and dns-resolve cause delays in haproxy fully reading the request in, is that right? > > Baptiste >
Re: http-request do-resolve Woes
> > Thanks! > > It had that feel to it...seemed like a cache lock timeout and/or somehow > tied to the request interval. > > I think I know where to fix this behavior in the code. I will work on the "how to fix it" later tonight. In the meantime, you can apply the workaround below. This is doable because the DNS cache is per resolvers section: 1. create a second dummy DNS section: resolvers main_resolver_do-resolve nameserver dns1 8.8.8.8:53 which is a copy of the first one with a different name. 2. reference this new resolvers section in your do-resolve action: http-request do-resolve(txn.myip,main_resolver_do-resolve,ipv4) hdr(Host),lower And you should be good until I fix it and it's backported. Baptiste
Re: http-request do-resolve Woes
On Wed, Oct 30, 2019, 9:58 AM Baptiste wrote: > > > On Tue, Oct 29, 2019 at 8:18 PM David Birdsong > wrote: > >> I should have put the haproxy version in the mail too: >> >> haproxy 2.0.8 >> >> On Tue, Oct 29, 2019 at 3:07 PM David Birdsong >> wrote: >> >>> I've narrowed down a behavior that I think might be a bug, but is >>> definitely not ideal. >>> >>> This minimal configuration copies header: X-Host into Host and performs >>> a dynamic DNS query against that field name, stores the output in a txn >>> var, and then uses a backend whic sets the dest ip to that txn var. >>> >>> For any requests with an X-Host header that matches a name already >>> tracked by DNS in a backend, I see that haproxy spends 4-9 seconds reading >>> the request from the client while any X-Host values which are not currently >>> tracked by a backend show haproxy spending 1ms reading in the request from >>> the client (normal.) >>> >>> unnamed, fast: curl -v -H "X-Host: google.com" http://127.0.0.1:8080/foo >>> >>> named, very slow: curl -v -H "X-Host: mixpanel.com" >>> http://127.0.0.1:8080/foo >>> >>> Config: >>> https://gist.github.com/davidbirdsong/1c3ec695fdbab10f64783437ffab901c >>> haproxy -vv >>> https://gist.github.com/davidbirdsong/d4c1c71e715d8461ad73a4891caca6f1 >>> >>> cat /etc/lsb-release >>> DISTRIB_ID=Ubuntu >>> DISTRIB_RELEASE=16.04 >>> DISTRIB_CODENAME=xenial >>> DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS" >>> >>> >>> david@david-VirtualBox:~/tls_demo/CA$ uname -a >>> Linux david-VirtualBox 4.15.0-65-generic #74~16.04.1-Ubuntu SMP Wed Sep >>> 18 09:51:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux >>> >>> > Hi David, > > I confirm I can reproduce the issue and from my first quick look, it is > related to DNS code in HAProxy. > Basically, there is a cache of the valid responses and from what I > observed, your do-resolve session is registered to the resolution and > instead of pulling info from the cache, it's waiting until the next request > is sent and gets updated with the next response. > Thanks! It had that feel to it...seemed like a cache lock timeout and/or somehow tied to the request interval. > Let me fix this. > > Baptiste >
Re: http-request do-resolve Woes
On Tue, Oct 29, 2019 at 8:18 PM David Birdsong wrote: > I should have put the haproxy version in the mail too: > > haproxy 2.0.8 > > On Tue, Oct 29, 2019 at 3:07 PM David Birdsong > wrote: > >> I've narrowed down a behavior that I think might be a bug, but is >> definitely not ideal. >> >> This minimal configuration copies header: X-Host into Host and performs a >> dynamic DNS query against that field name, stores the output in a txn var, >> and then uses a backend whic sets the dest ip to that txn var. >> >> For any requests with an X-Host header that matches a name already >> tracked by DNS in a backend, I see that haproxy spends 4-9 seconds reading >> the request from the client while any X-Host values which are not currently >> tracked by a backend show haproxy spending 1ms reading in the request from >> the client (normal.) >> >> unnamed, fast: curl -v -H "X-Host: google.com" http://127.0.0.1:8080/foo >> >> named, very slow: curl -v -H "X-Host: mixpanel.com" >> http://127.0.0.1:8080/foo >> >> Config: >> https://gist.github.com/davidbirdsong/1c3ec695fdbab10f64783437ffab901c >> haproxy -vv >> https://gist.github.com/davidbirdsong/d4c1c71e715d8461ad73a4891caca6f1 >> >> cat /etc/lsb-release >> DISTRIB_ID=Ubuntu >> DISTRIB_RELEASE=16.04 >> DISTRIB_CODENAME=xenial >> DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS" >> >> >> david@david-VirtualBox:~/tls_demo/CA$ uname -a >> Linux david-VirtualBox 4.15.0-65-generic #74~16.04.1-Ubuntu SMP Wed Sep >> 18 09:51:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux >> >> Hi David, I confirm I can reproduce the issue and from my first quick look, it is related to DNS code in HAProxy. Basically, there is a cache of the valid responses and from what I observed, your do-resolve session is registered to the resolution and instead of pulling info from the cache, it's waiting until the next request is sent and gets updated with the next response. Let me fix this. Baptiste
Re: http-request do-resolve Woes
Hi, On Tue, Oct 29, David Birdsong wrote: > I've narrowed down a behavior that I think might be a bug, but is > definitely not ideal. > > This minimal configuration copies header: X-Host into Host and performs a > dynamic DNS query against that field name, stores the output in a txn var, > and then uses a backend whic sets the dest ip to that txn var. > > For any requests with an X-Host header that matches a name already tracked > by DNS in a backend, I see that haproxy spends 4-9 seconds reading the > request from the client while any X-Host values which are not currently > tracked by a backend show haproxy spending 1ms reading in the request from > the client (normal.) > > unnamed, fast: curl -v -H "X-Host: google.com" http://127.0.0.1:8080/foo > > named, very slow: curl -v -H "X-Host: mixpanel.com" > http://127.0.0.1:8080/foo > > Config: > https://gist.github.com/davidbirdsong/1c3ec695fdbab10f64783437ffab901c > haproxy -vv > https://gist.github.com/davidbirdsong/d4c1c71e715d8461ad73a4891caca6f1 I tested this on latest 2.1dev3 snapshot. What happens if you add timeouts to your main_resolver resolvers: hold valid 15s For me increasing hold valid makes be_named requests take even longer and if I add timeout client(to defaults) < hold valid then (be_named) requests fail with: cR-- status -Jarno -- Jarno Huuskonen
Re: http-request do-resolve Woes
I should have put the haproxy version in the mail too: haproxy 2.0.8 On Tue, Oct 29, 2019 at 3:07 PM David Birdsong wrote: > I've narrowed down a behavior that I think might be a bug, but is > definitely not ideal. > > This minimal configuration copies header: X-Host into Host and performs a > dynamic DNS query against that field name, stores the output in a txn var, > and then uses a backend whic sets the dest ip to that txn var. > > For any requests with an X-Host header that matches a name already tracked > by DNS in a backend, I see that haproxy spends 4-9 seconds reading the > request from the client while any X-Host values which are not currently > tracked by a backend show haproxy spending 1ms reading in the request from > the client (normal.) > > unnamed, fast: curl -v -H "X-Host: google.com" http://127.0.0.1:8080/foo > > named, very slow: curl -v -H "X-Host: mixpanel.com" > http://127.0.0.1:8080/foo > > Config: > https://gist.github.com/davidbirdsong/1c3ec695fdbab10f64783437ffab901c > haproxy -vv > https://gist.github.com/davidbirdsong/d4c1c71e715d8461ad73a4891caca6f1 > > cat /etc/lsb-release > DISTRIB_ID=Ubuntu > DISTRIB_RELEASE=16.04 > DISTRIB_CODENAME=xenial > DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS" > > > david@david-VirtualBox:~/tls_demo/CA$ uname -a > Linux david-VirtualBox 4.15.0-65-generic #74~16.04.1-Ubuntu SMP Wed Sep 18 > 09:51:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux > >
http-request do-resolve Woes
I've narrowed down a behavior that I think might be a bug, but is definitely not ideal. This minimal configuration copies header: X-Host into Host and performs a dynamic DNS query against that field name, stores the output in a txn var, and then uses a backend whic sets the dest ip to that txn var. For any requests with an X-Host header that matches a name already tracked by DNS in a backend, I see that haproxy spends 4-9 seconds reading the request from the client while any X-Host values which are not currently tracked by a backend show haproxy spending 1ms reading in the request from the client (normal.) unnamed, fast: curl -v -H "X-Host: google.com" http://127.0.0.1:8080/foo named, very slow: curl -v -H "X-Host: mixpanel.com" http://127.0.0.1:8080/foo Config: https://gist.github.com/davidbirdsong/1c3ec695fdbab10f64783437ffab901c haproxy -vv https://gist.github.com/davidbirdsong/d4c1c71e715d8461ad73a4891caca6f1 cat /etc/lsb-release DISTRIB_ID=Ubuntu DISTRIB_RELEASE=16.04 DISTRIB_CODENAME=xenial DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS" david@david-VirtualBox:~/tls_demo/CA$ uname -a Linux david-VirtualBox 4.15.0-65-generic #74~16.04.1-Ubuntu SMP Wed Sep 18 09:51:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
