Re: [dnsdist] dnsdist performance

[Remi Gacogne]

Hi Klaus,

On 8/13/19 9:31 PM, Klaus Darilion wrote:
> My results were:
> 1 VM with 4vCPUs. The VM runs NSD and PDNS/PGSQL. DNSDIST forwards some
> zones to NSD, others to PDNS/PGSQL
> 
> PDNS with cached responses: 40.000 q/s
> PDNS random labels: 9.000 q/s
> NSD: 65.000 q/s
> 
> 1. dnsdist with single "listen" and each backend added only once
> dnsdist with cached responses: 40.000 q/s
> dnsdist random labels to PDNS: 7.300 q/s
> dnsdist random labels to NSD: 40.000 q/s
> 
> 2. dnsdist with 3 "listen" and each backend added 3 times
> dnsdist with cached responses: 100.000 q/s
> dnsdist random labels to PDNS: 7.300 q/s
> dnsdist random labels to NSD: 60.000 q/s

That's interesting, thank you! I assume the "dnsdist with cached
responses" lines are with dnsdist's packet cache enabled? If so I'd be
interested in knowing the results without any packet cache in dnsdist
but with the packet cache in PDNS enabled.

>> If you increase the number of threads you'll need to use ring buffers
>> sharding to limit contention, by the way.
> 
> Any config recommendation? I currently use a single Cache for both
> backends with 1 shard:
> 
> cache = newPacketCache(50, 10, 10, 60, 60, true, 1, true, 10)
> getPool(""):setCache(cache)
> getPool("static"):setCache(cache)

I'd advise the use of more shards since it will pretty much get rid of
the contention between the threads, although it might not matter that
much at this level of QPS. A value like 50 should be enough for 6
threads but in some setups I have used values up to 1000 shards. The
sweet spot depends on your exact setup.

Note that this is the number of shards in the packet cache, not the ring
buffers. Even if you don't use the packet cache it would be a good idea
to shard the ring buffers as soon as you use several threads, with:

setRingBuffersSize(1, 16)

which keeps the default value for the number of entries to keep in
memory but use 16 shards instead of 1, reducing lock contention there as
well.

Best regards,
-- 
Remi Gacogne
PowerDNS.COM BV - https://www.powerdns.com/



signature.asc
Description: OpenPGP digital signature
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] DelayAction with dnsdist 1.4.0-rc1

[Remi Gacogne]

Hi Brian,

On 8/13/19 7:28 PM, Brian Sullivan wrote:
> I am running dnsdist 1.4.0-rc1 and I am seeing something odd. As a test
> I have the following rule configured. 
> 
> addAction(MaxQPSIPRule(2, 32, 48), DelayAction(500))
> 
> and I run 100 queries at 10 per second and my statistics from my client
> are as follows:
> 
> Statistics for data/input/input1: min = 52.35 ave = 59.13 max = 131.05
> 
> The following is a snippit from the webserver
> 
> Screen Shot 2019-08-13 at 1.27.56 PM.png
> 
> Any ideas on what is going on?

It's hard to say without knowing exactly how the client sends the
queries, in particular whether it waits for an answer before sending the
next query. Perhaps a network capture would help?

Best regards,
-- 
Remi Gacogne
PowerDNS.COM BV - https://www.powerdns.com/



signature.asc
Description: OpenPGP digital signature
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] Define out-of-band IP to reach carbon server

[frank+pdns]

Hi Leo,

By quickly glancing at the code, I don’t think there’s a way to set the source 
ip right now. However, you can probably solve this using the ip routing (and/or 
firewalling) table(s) on the node itself: set the source ip for all connections 
to that particular destination ip.

Regards,

Frank Louwers
Certified PowerDNS Consultant @ Kiwazo.be

> On 13 Aug 2019, at 17:40, Leo Vandewoestijne  wrote:
> 
> Hi,
> 
> 
> I'm running dnsdist inside a virtualized host, which has anycasted IP's and 
> out-of-band IP's.
> Using nc I can reach my carbon server BUT have to define the source IP.
> 
> Now I wish to have dnsdist sent data to metronome.
> So I assumed using "setLocal" would define the main IP, and so set the 
> out-of-band IP.
> Whatever I do; put it before or after addLocal, or use addLocal (first and 
> last), dnsdist can't reach the carbon server.
> 
> What information am I missing to get more success?
> 
> 
> -- 
> 
> Met vriendelijke groet,
> With kind regards,
> 
> 
> Leo Vandewoestijne
> <***@dns.company>
> 
> ___
> dnsdist mailing list
> dnsdist@mailman.powerdns.com
> https://mailman.powerdns.com/mailman/listinfo/dnsdist

Frank Louwers
PowerDNS Certified Consultant @ Kiwazo.be




___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] Feature Request?

[Otto Moerbeek]

Hi,

Submitting an issue to https://github.com/PowerDNS/pdns would be a
first step. 

But it does not hurt to discuss the feature here. Maybe there's a
solution to your problem possible without a new feature.

-Otto


On Tue, Aug 13, 2019 at 10:49:59AM -0400, Brian Sullivan wrote:

> Hi,
> 
> I'd like to know where I should submit a feature request and what that
> process might be?
> 
> Thanks,
> brian
> 
> -- 
> 
> 
> 
> Brian M. Sullivan
> Senior Staff Security Intelligence Engineer
> bsulli...@lookout.com |  www.lookout.com

> ___
> dnsdist mailing list
> dnsdist@mailman.powerdns.com
> https://mailman.powerdns.com/mailman/listinfo/dnsdist

___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] Feature Request?

[Remi Gacogne]

Hi Brian,

On 8/13/19 4:49 PM, Brian Sullivan wrote:
> I'd like to know where I should submit a feature request and what that
> process might be?

The usual process is to open a new feature request on our Github
repository [1], filling the corresponding parts of the template.
It provides the opportunity for other users to chime in and if the
feature makes sense to use it's usually added to the relevant milestone.


[1]: https://github.com/PowerDNS/pdns/issues/new
-- 
Remi Gacogne
PowerDNS.COM BV - https://www.powerdns.com/



signature.asc
Description: OpenPGP digital signature
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] Define out-of-band IP to reach carbon server

[Remi Gacogne]

Hi,

On 8/14/19 9:47 AM, frank+p...@tembo.be wrote:
> By quickly glancing at the code, I don’t think there’s a way to set the
> source ip right now. However, you can probably solve this using the ip
> routing (and/or firewalling) table(s) on the node itself: set the source
> ip for all connections to that particular destination ip.

Yes, Frank is right.
Please open a new feature request on Github to be able to specify the
source IP used to reach the carbon server if you think it might be
useful to others.

Best regards,
-- 
Remi Gacogne
PowerDNS.COM BV - https://www.powerdns.com/



signature.asc
Description: OpenPGP digital signature
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] metronome on FreeBSD - rc.d script

[Remi Gacogne]

Hi Leo,

On 8/13/19 5:40 PM, Leo Vandewoestijne wrote:
> This message is not dnsdist specifically, but about metronome.
> Still I though this channel would be the most appropriate.
> 
> I just wrote an rc.d script for the metronome port in FreeBSD.
> 
> If you happen to be interrested in both, then please review
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=239819

Thanks! If you don't mind it might be useful to submit a pull request to
the metronome repository instead [1] to add it in, for example, a
contrib/ directory?

[1]: https://github.com/ahupowerdns/metronome

Best regards,
-- 
Remi Gacogne
PowerDNS.COM BV - https://www.powerdns.com/



signature.asc
Description: OpenPGP digital signature
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

[dnsdist] debian 9 and dnsdist 1.1.0 drops too

[Lichtnau Frank]

Hi,

I setup debain 9 (Kernel 4.9.0-9) and install package dnsdist 1.1.0 from debian 
repository.

There I got the same problem like 
https://mailman.powerdns.com/pipermail/dnsdist/2019-August/000627.html.

After a while drops starts and count high from this moment.

restart dnsdist solve the problem till next.

I use 2 debian-machines as a cluster, connected via keepalived (master/backup) 
with a virtuell IP Address.
On this IP-Address listen dnsdist.

~Frank
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] Define out-of-band IP to reach carbon server

[Leo Vandewoestijne]

In reply to frank+p...@tembo.be

> Hi Leo,
> 
> By quickly glancing at the code, I don???t think there???s a way to set the 
> source ip right now. However, you can probably solve this using the ip 
> routing (and/or firewalling) table(s) on the node itself: set the source ip 
> for all connections to that particular destination ip.
> 
> Regards,
> 
> Frank Louwers
> Certified PowerDNS Consultant @ Kiwazo.be
>

Why I didn't thought of that? That did the trick!

Thanks.

--

Met vriendelijke groet,
With kind regards,


Leo Vandewoestijne
<***@dns.company>

___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] DNSDIST 1.3.3-3 from standard debian buster

[Remi Gacogne]

Hi Frank,

On 8/13/19 4:59 PM, Lichtnau Frank wrote:
> I think, it has to be with 'high latencies'.
> 
> I have: 
> - 1 pool (winmls) for windows-ad-dns-queries
> - 1 pool (mls) for rest of our internal domain
> - and a dns-forwarder (with 3 listener) for external dns-queries.
> 
> The pools work fine with latencies of 0.3 - 0.8 The single dns-forwarder has 
> latencies of 40 - 56. And there I have the drops.
> 
> For testing I reconfigure the external dns-queries over the pool(mls). 
> And than I have the drops in this pool.
> 
> I would try to install dnsdist 1.3.3 in debian 9, but it works not, because 
> some packet-dependencies was not given.
> And the  dnsdist-packet in debian 9 was to old.
> 
> Your tool is importent for me, because it helps me to capture queer manner of 
> our windows machines. 
> If the dns-server is gone, windows don't switch to the second dns-server in 
> his given list of dns-servers. 
> 
> BTW, I would build now a tool as workaround for checking dnsdist frequency.
> if the quote between queries and drops too bad or grow up I restart the 
> daemon.

So now I'm confused, are you experiencing the same issue reported by
Chris where dnsdist stops responding to every single query sent over UDP
after a while, or are you just experiencing some queries being dropped?

Queries being dropped happens for one of two reasons, either the backend
did not respond fast enough, for example because it is overloaded, or
dnsdist is struggling to keep up with the responses and could not
process them fast enough.
It is quite easy to see the difference between the two cases because in
the second one dnsdist will be CPU bound, which is easy to spot in top,
metronome or grafana. If the backend is not responding fast enough, then
you need to investigate the backend or eventually the network.

> I check your API api/v1/servers/localhost and see, that the value from
Column "Drops" are given in field=reused.
>
> Why ist he name reused and what means reused in this context?

I agree it's not clear but that's the same thing. Historically dnsdist
did not actively discover timeouts, ie a backend not responding, but
would notice later when picking up a state (a slot in the table dnsdist
uses to keep track of queries sent to backends) to forward a new query
that the state was still marked as "in use", meaning that the response
never came through. We will then "reuse" that state, and so the
corresponding metric is named "reused" even though nowadays we usually
notice the timeout by regularly scanning the table.

Best regards,
-- 
Remi Gacogne
PowerDNS.COM BV - https://www.powerdns.com/



signature.asc
Description: OpenPGP digital signature
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] DNSDIST 1.3.3-3 from standard debian buster

[Remi Gacogne]

Hi Chris,

On 8/14/19 2:58 AM, Chris wrote:
> For this issue I have not been able to make any progress yet. I have
> asked my colleagues for help as I am a network admin by trade, something
> they found that may be potentially related is this kernel bug:
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=202997
> 
> One of my colleagues also deployed a base install of Debian and did some
> testing with iperf; apparently he could reproduce the issue without
> involving PowerDNS at all. From the above link he has adjusted a few
> things but as of yet the problem hasn't been resolved.

That's very interesting, thank you! I'm a bit surprised to see the bug
entry marked as RESOLVED INVALID since clearly the sockets did not lock
up that way in earlier kernels under the same conditions.

Did you try reducing the sysctl net.core.{r,w}mem_{default,max} values
on your system to see if the issue remains?

I have been directed to this patch as another possible lead:

https://git.kernel.org/pub/scm/linux/kernel/git/davem/net.git/commit/?id=891584f48a9084ba462f10da4c6bb28b6181b543

Do you know if your system receives a lot of fragmented UDP datagrams?

> I am wondering if there is a plan for official dnsdist 1.4 packages on
> Debian Stretch? I am going to need to use that for now as Buster isn't
> suitable for production in my environment yet.

We already provide 1.4.0 packages for Stretch at
https://repo.powerdns.com unless I'm missing something?

Best regards,
-- 
Remi Gacogne
PowerDNS.COM BV - https://www.powerdns.com/



signature.asc
Description: OpenPGP digital signature
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] metronome on FreeBSD - rc.d script

[Leo Vandewoestijne]

Hi,

> Thanks! If you don't mind it might be useful to submit a pull request to
> the metronome repository instead [1] to add it in, for example, a
> contrib/ directory?
>
> [1]: https://github.com/ahupowerdns/metronome
>

The patch is against the ports (the OS's packaging system), not dnsdist itself.


I'm trying to learn to understand the whole dataformat, and see if I can add or 
improve.
For example I saw I'm not the only one who also wish to use this for monitoring 
the system.

I'm also playing with grafena6 but I find it all a hairy beast.
Storing data is relative simple, but displaying I find complex.

If I cannot contribute on that, then at least I will write another howto 
(already made notes).

Last but not least; my compliments on this "toy", to me it's very useful.


-- 

Met vriendelijke groet,
With kind regards,


Leo Vandewoestijne
<***@dns.company>

___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] dnsdist performance

[Remi Gacogne]

Hi Gentian,

On 8/12/19 10:20 AM, Gentian Bajraktari wrote:
> Just did, its showing on metronome: Gentian-DNSDIST-TEST

Thank you, I see it! Unfortunately it looks like your test was quite
short and metronome reduces the precision after a few hours so I'm not
sure I see the whole picture, but it looks like the results are terrible.
I see that the packet cache is enabled but you reach a very low number
of QPS for a very high system CPU usage. It does not really make sense
to me, could you make sure that the CPUs assigned to your VMs are
dedicated to it, or at the very least are not over committed? It
certainly looks that way.

Best regards,

Remi

> On Mon, 12 Aug 2019 at 10:16, Remi Gacogne  > wrote:
> 
> Hi Gentian,
> 
> On 8/12/19 10:12 AM, Gentian Bajraktari wrote:
> > I have attached results from resperf ( latest compiled, 2.3.1) and the
> > configuration file (dnsdist.conf)
> 
> Would you consider sending metrics to our public metronome server [1]
> so we understand what's the limiting factor during your tests? You only
> need to add a line like this one to your dnsdist configuration:
> 
> carbonServer("37.252.122.50:2003 ",
> "put-the-name-you-want-to-display-here", 5)
> 
> This will only send some metrics without any private information and is
> very useful to understand where the bottleneck may be.
> 
> [1]: https://metronome1.powerdns.com/
> 
> Best regards,
> 
> Remi
> 
> > On Sat, 10 Aug 2019 at 22:30, Klaus Darilion
> >  
>  >> wrote:
> >
> >     Am 08.08.2019 um 10:20 schrieb Gentian Bajraktari:
> >     > Dear Dnsdist community,
> >     >
> >     > we are trying to setup dnsdist as a loadbalancer for 2 bind
> recursive
> >     > cache only servers,
> >     >
> >     > we have done a very simple setup:
> >     > addLocal("DNSDIST_IP")
> >     > newServer({address="BIND_IP1"})
> >     > newServer({address=" BIND_IP2"})
> >     > setServerPolicy(roundrobin)
> >     >
> >     > but when we test with resperf (from dnsperf tool), the
> results are
> >     very
> >     > bad for DNSDIST, around 5-15K QPS , while when we test
> directly to
> >     one
> >     > of BIND ip addresses the QPS goes up to 50-60Qps.
> >     >
> >     > we have tried both rpm install of dnsdist, compiling from
> source and
> >     > even comiling the concur version but all with similar
> results, ie
> >     when
> >     > resperf tests go through dnsdist the results are much lower in
> >     terms of qps.
> >     >
> >     > OS is Centos7 in all servers, tweaked all settings for file
> >     limits, udp
> >     > connections etc.
> >     >
> >     > any thoughts on what we can do? maybe resperf is not the
> tool to test
> >     > this? do you have some suggetions on how we can test and
> make sure we
> >     > are doing the right thing?
> >
> >     I had similar results. Starting 4 listening threads and 4
> receivers
> >     threads (by adding the same backend 4 times) boosted my
> performance -
> >     almost linear.
> >
> >     If it still does not work your should post your config (and
> first test
> >     without any rules to avoid performance issues because of rules)
> >
> >     regards
> >     Klaus
> >     ___
> >     dnsdist mailing list
> >     dnsdist@mailman.powerdns.com
> 
>  >
> >     https://mailman.powerdns.com/mailman/listinfo/dnsdist
> >
> >
> > ___
> > dnsdist mailing list
> > dnsdist@mailman.powerdns.com 
> > https://mailman.powerdns.com/mailman/listinfo/dnsdist
> >
> 
> 
> -- 
> Remi Gacogne
> PowerDNS.COM BV - https://www.powerdns.com/
> 
> ___
> dnsdist mailing list
> dnsdist@mailman.powerdns.com 
> https://mailman.powerdns.com/mailman/listinfo/dnsdist
> 


-- 
Remi Gacogne
PowerDNS.COM BV - https://www.powerdns.com/



signature.asc
Description: OpenPGP digital signature
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] Feature Request?

[Remi Gacogne]

On 8/14/19 4:21 PM, Brian Sullivan wrote:
> We have a requirement to not allow negotiation of TLS version to go
> below 1.2. Is there a way to configure TLS so that it will only
> negotiate version 1.2 or above? 

It looks like we don't support that explicitly.. You could probably work
something around by tuning the allowed ciphers, but I guess an option to
to select the TLS versions allowed, like ssl_protocols in nginx or
SSLProtocol in Apache HTTPD, would make sense as well.

Best regards,
-- 
Remi Gacogne
PowerDNS.COM BV - https://www.powerdns.com/



signature.asc
Description: OpenPGP digital signature
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] Feature Request?

[Brian Sullivan]

Thanks! Before I do that let me make sure it doesn't already exist in some
form.

We have a requirement to not allow negotiation of TLS version to go below
1.2. Is there a way to configure TLS so that it will only negotiate version
1.2 or above?

Regards,
brian

On Wed, Aug 14, 2019 at 4:20 AM Remi Gacogne 
wrote:

> Hi Brian,
>
> On 8/13/19 4:49 PM, Brian Sullivan wrote:
> > I'd like to know where I should submit a feature request and what that
> > process might be?
>
> The usual process is to open a new feature request on our Github
> repository [1], filling the corresponding parts of the template.
> It provides the opportunity for other users to chime in and if the
> feature makes sense to use it's usually added to the relevant milestone.
>
>
> [1]: https://github.com/PowerDNS/pdns/issues/new
> --
> Remi Gacogne
> PowerDNS.COM BV - https://www.powerdns.com/
>
> ___
> dnsdist mailing list
> dnsdist@mailman.powerdns.com
> https://mailman.powerdns.com/mailman/listinfo/dnsdist
>


-- 



Brian M. Sullivan
Senior Staff Security Intelligence Engineer
bsulli...@lookout.com |  www.lookout.com
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist