Re: [dnsdist] dnsdist performance

[Klaus Darilion]

Hi Remi!

Am 14.08.2019 um 10:39 schrieb Remi Gacogne:
> Hi Klaus,
> 
> On 8/13/19 9:31 PM, Klaus Darilion wrote:
>> My results were:
>> 1 VM with 4vCPUs. The VM runs NSD and PDNS/PGSQL. DNSDIST forwards some
>> zones to NSD, others to PDNS/PGSQL
>>
>> PDNS with cached responses: 40.000 q/s
>> PDNS random labels: 9.000 q/s
>> NSD: 65.000 q/s
>>
>> 1. dnsdist with single "listen" and each backend added only once
>> dnsdist with cached responses: 40.000 q/s
>> dnsdist random labels to PDNS: 7.300 q/s
>> dnsdist random labels to NSD: 40.000 q/s
>>
>> 2. dnsdist with 3 "listen" and each backend added 3 times
>> dnsdist with cached responses: 100.000 q/s
>> dnsdist random labels to PDNS: 7.300 q/s
>> dnsdist random labels to NSD: 60.000 q/s
> 
> That's interesting, thank you! I assume the "dnsdist with cached
> responses" lines are with dnsdist's packet cache enabled? If so I'd be
> interested in knowing the results without any packet cache in dnsdist
> but with the packet cache in PDNS enabled.

First I have to say that my q/s values fluctuate quite strong, +-3000
q/s. So, these values show more a trend than exact results.

Yes, "dnsdist with cached responses" means that dnsdist has a cache and
I query always the same domain.

Now, I disabled the dnsdist-cache and tested again:

dnsdist w/o cache random labels to PDNS: still around 7300 q/s
dnsdist w/o cache static label to PDNS (cached responses): still around
4000 q/s.

Hence, dnsdist without cache has no impact when the results are coming
from PDNS packet cache, but seems to have impact when the results are
not cached by PDNS.

>>> If you increase the number of threads you'll need to use ring buffers
>>> sharding to limit contention, by the way.
>>
>> Any config recommendation? I currently use a single Cache for both
>> backends with 1 shard:
>>
>> cache = newPacketCache(50, 10, 10, 60, 60, true, 1, true, 10)
>> getPool(""):setCache(cache)
>> getPool("static"):setCache(cache)
> 
> I'd advise the use of more shards since it will pretty much get rid of
> the contention between the threads, although it might not matter that
> much at this level of QPS. A value like 50 should be enough for 6
> threads but in some setups I have used values up to 1000 shards. The
> sweet spot depends on your exact setup.
> 
> Note that this is the number of shards in the packet cache, not the ring
> buffers. Even if you don't use the packet cache it would be a good idea
> to shard the ring buffers as soon as you use several threads, with:
> 
> setRingBuffersSize(1, 16)
> 
> which keeps the default value for the number of entries to keep in
> memory but use 16 shards instead of 1, reducing lock contention there as
> well.

I guess in my setup, where I only have one backend, the backend is the
bottleneck, not the dnsdist cache. Anyway, I just tested with:

setRingBuffersSize(1, 16)
cache = newPacketCache(50, 10, 10, 60, 60, true, 50, true, 10)

instead of
cache = newPacketCache(50, 10, 10, 60, 60, true, 1, true, 10)

and do not see a difference (which is more than my q/s jitter)

Thanks
Klaus



___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] dnsdist performance

[Remi Gacogne]

Hi Gentian,

On 8/12/19 10:20 AM, Gentian Bajraktari wrote:
> Just did, its showing on metronome: Gentian-DNSDIST-TEST

Thank you, I see it! Unfortunately it looks like your test was quite
short and metronome reduces the precision after a few hours so I'm not
sure I see the whole picture, but it looks like the results are terrible.
I see that the packet cache is enabled but you reach a very low number
of QPS for a very high system CPU usage. It does not really make sense
to me, could you make sure that the CPUs assigned to your VMs are
dedicated to it, or at the very least are not over committed? It
certainly looks that way.

Best regards,

Remi

> On Mon, 12 Aug 2019 at 10:16, Remi Gacogne  > wrote:
> 
> Hi Gentian,
> 
> On 8/12/19 10:12 AM, Gentian Bajraktari wrote:
> > I have attached results from resperf ( latest compiled, 2.3.1) and the
> > configuration file (dnsdist.conf)
> 
> Would you consider sending metrics to our public metronome server [1]
> so we understand what's the limiting factor during your tests? You only
> need to add a line like this one to your dnsdist configuration:
> 
> carbonServer("37.252.122.50:2003 ",
> "put-the-name-you-want-to-display-here", 5)
> 
> This will only send some metrics without any private information and is
> very useful to understand where the bottleneck may be.
> 
> [1]: https://metronome1.powerdns.com/
> 
> Best regards,
> 
> Remi
> 
> > On Sat, 10 Aug 2019 at 22:30, Klaus Darilion
> >  
>  >> wrote:
> >
> >     Am 08.08.2019 um 10:20 schrieb Gentian Bajraktari:
> >     > Dear Dnsdist community,
> >     >
> >     > we are trying to setup dnsdist as a loadbalancer for 2 bind
> recursive
> >     > cache only servers,
> >     >
> >     > we have done a very simple setup:
> >     > addLocal("DNSDIST_IP")
> >     > newServer({address="BIND_IP1"})
> >     > newServer({address=" BIND_IP2"})
> >     > setServerPolicy(roundrobin)
> >     >
> >     > but when we test with resperf (from dnsperf tool), the
> results are
> >     very
> >     > bad for DNSDIST, around 5-15K QPS , while when we test
> directly to
> >     one
> >     > of BIND ip addresses the QPS goes up to 50-60Qps.
> >     >
> >     > we have tried both rpm install of dnsdist, compiling from
> source and
> >     > even comiling the concur version but all with similar
> results, ie
> >     when
> >     > resperf tests go through dnsdist the results are much lower in
> >     terms of qps.
> >     >
> >     > OS is Centos7 in all servers, tweaked all settings for file
> >     limits, udp
> >     > connections etc.
> >     >
> >     > any thoughts on what we can do? maybe resperf is not the
> tool to test
> >     > this? do you have some suggetions on how we can test and
> make sure we
> >     > are doing the right thing?
> >
> >     I had similar results. Starting 4 listening threads and 4
> receivers
> >     threads (by adding the same backend 4 times) boosted my
> performance -
> >     almost linear.
> >
> >     If it still does not work your should post your config (and
> first test
> >     without any rules to avoid performance issues because of rules)
> >
> >     regards
> >     Klaus
> >     ___
> >     dnsdist mailing list
> >     dnsdist@mailman.powerdns.com
> 
>  >
> >     https://mailman.powerdns.com/mailman/listinfo/dnsdist
> >
> >
> > ___
> > dnsdist mailing list
> > dnsdist@mailman.powerdns.com 
> > https://mailman.powerdns.com/mailman/listinfo/dnsdist
> >
> 
> 
> -- 
> Remi Gacogne
> PowerDNS.COM BV - https://www.powerdns.com/
> 
> ___
> dnsdist mailing list
> dnsdist@mailman.powerdns.com 
> https://mailman.powerdns.com/mailman/listinfo/dnsdist
> 


-- 
Remi Gacogne
PowerDNS.COM BV - https://www.powerdns.com/



signature.asc
Description: OpenPGP digital signature
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] dnsdist performance

[Remi Gacogne]

Hi Klaus,

On 8/13/19 9:31 PM, Klaus Darilion wrote:
> My results were:
> 1 VM with 4vCPUs. The VM runs NSD and PDNS/PGSQL. DNSDIST forwards some
> zones to NSD, others to PDNS/PGSQL
> 
> PDNS with cached responses: 40.000 q/s
> PDNS random labels: 9.000 q/s
> NSD: 65.000 q/s
> 
> 1. dnsdist with single "listen" and each backend added only once
> dnsdist with cached responses: 40.000 q/s
> dnsdist random labels to PDNS: 7.300 q/s
> dnsdist random labels to NSD: 40.000 q/s
> 
> 2. dnsdist with 3 "listen" and each backend added 3 times
> dnsdist with cached responses: 100.000 q/s
> dnsdist random labels to PDNS: 7.300 q/s
> dnsdist random labels to NSD: 60.000 q/s

That's interesting, thank you! I assume the "dnsdist with cached
responses" lines are with dnsdist's packet cache enabled? If so I'd be
interested in knowing the results without any packet cache in dnsdist
but with the packet cache in PDNS enabled.

>> If you increase the number of threads you'll need to use ring buffers
>> sharding to limit contention, by the way.
> 
> Any config recommendation? I currently use a single Cache for both
> backends with 1 shard:
> 
> cache = newPacketCache(50, 10, 10, 60, 60, true, 1, true, 10)
> getPool(""):setCache(cache)
> getPool("static"):setCache(cache)

I'd advise the use of more shards since it will pretty much get rid of
the contention between the threads, although it might not matter that
much at this level of QPS. A value like 50 should be enough for 6
threads but in some setups I have used values up to 1000 shards. The
sweet spot depends on your exact setup.

Note that this is the number of shards in the packet cache, not the ring
buffers. Even if you don't use the packet cache it would be a good idea
to shard the ring buffers as soon as you use several threads, with:

setRingBuffersSize(1, 16)

which keeps the default value for the number of entries to keep in
memory but use 16 shards instead of 1, reducing lock contention there as
well.

Best regards,
-- 
Remi Gacogne
PowerDNS.COM BV - https://www.powerdns.com/



signature.asc
Description: OpenPGP digital signature
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] dnsdist performance

[Klaus Darilion]

Hi Remi!

Am 11.08.2019 um 18:26 schrieb Remi Gacogne:
> Hi Klaus,
> 
> On 8/10/19 10:30 PM, Klaus Darilion wrote:
>> I had similar results. Starting 4 listening threads and 4 receivers
>> threads (by adding the same backend 4 times) boosted my performance -
>> almost linear.
> 
> Similar in terms of QPS as well? I just tested master on Arch using
> calidns and dumresp as a responder, and I get 55k+ QPS on my 10 year-old
> CPU using no tuning, simply:
> 
> ./dnsdist -C /dev/null -l 127.0.0.1:53 127.0.0.1:54
> 
> So one listening thread and one receiver thread.
> 
> A quick glance at a perf recording shows that more than 76% of the CPU
> time is spent in syscalls, so I'm pretty sure disabling the
> meltdown/spectre mitigations would do a big difference, but it's already
> pretty far from 5-15K QPS.

My results were:
1 VM with 4vCPUs. The VM runs NSD and PDNS/PGSQL. DNSDIST forwards some
zones to NSD, others to PDNS/PGSQL

PDNS with cached responses: 40.000 q/s
PDNS random labels: 9.000 q/s
NSD: 65.000 q/s

1. dnsdist with single "listen" and each backend added only once
dnsdist with cached responses: 40.000 q/s
dnsdist random labels to PDNS: 7.300 q/s
dnsdist random labels to NSD: 40.000 q/s

2. dnsdist with 3 "listen" and each backend added 3 times
dnsdist with cached responses: 100.000 q/s
dnsdist random labels to PDNS: 7.300 q/s
dnsdist random labels to NSD: 60.000 q/s

> If you increase the number of threads you'll need to use ring buffers
> sharding to limit contention, by the way.

Any config recommendation? I currently use a single Cache for both
backends with 1 shard:

cache = newPacketCache(50, 10, 10, 60, 60, true, 1, true, 10)
getPool(""):setCache(cache)
getPool("static"):setCache(cache)


thanks
Klaus
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] dnsdist performance

[Gentian Bajraktari]

Just did, its showing on metronome: Gentian-DNSDIST-TEST

On Mon, 12 Aug 2019 at 10:16, Remi Gacogne 
wrote:

> Hi Gentian,
>
> On 8/12/19 10:12 AM, Gentian Bajraktari wrote:
> > I have attached results from resperf ( latest compiled, 2.3.1) and the
> > configuration file (dnsdist.conf)
>
> Would you consider sending metrics to our public metronome server [1]
> so we understand what's the limiting factor during your tests? You only
> need to add a line like this one to your dnsdist configuration:
>
> carbonServer("37.252.122.50:2003",
> "put-the-name-you-want-to-display-here", 5)
>
> This will only send some metrics without any private information and is
> very useful to understand where the bottleneck may be.
>
> [1]: https://metronome1.powerdns.com/
>
> Best regards,
>
> Remi
>
> > On Sat, 10 Aug 2019 at 22:30, Klaus Darilion
> > mailto:klaus.mailingli...@pernau.at>>
> wrote:
> >
> > Am 08.08.2019 um 10:20 schrieb Gentian Bajraktari:
> > > Dear Dnsdist community,
> > >
> > > we are trying to setup dnsdist as a loadbalancer for 2 bind
> recursive
> > > cache only servers,
> > >
> > > we have done a very simple setup:
> > > addLocal("DNSDIST_IP")
> > > newServer({address="BIND_IP1"})
> > > newServer({address=" BIND_IP2"})
> > > setServerPolicy(roundrobin)
> > >
> > > but when we test with resperf (from dnsperf tool), the results are
> > very
> > > bad for DNSDIST, around 5-15K QPS , while when we test directly to
> > one
> > > of BIND ip addresses the QPS goes up to 50-60Qps.
> > >
> > > we have tried both rpm install of dnsdist, compiling from source
> and
> > > even comiling the concur version but all with similar results, ie
> > when
> > > resperf tests go through dnsdist the results are much lower in
> > terms of qps.
> > >
> > > OS is Centos7 in all servers, tweaked all settings for file
> > limits, udp
> > > connections etc.
> > >
> > > any thoughts on what we can do? maybe resperf is not the tool to
> test
> > > this? do you have some suggetions on how we can test and make sure
> we
> > > are doing the right thing?
> >
> > I had similar results. Starting 4 listening threads and 4 receivers
> > threads (by adding the same backend 4 times) boosted my performance -
> > almost linear.
> >
> > If it still does not work your should post your config (and first
> test
> > without any rules to avoid performance issues because of rules)
> >
> > regards
> > Klaus
> > ___
> > dnsdist mailing list
> > dnsdist@mailman.powerdns.com 
> > https://mailman.powerdns.com/mailman/listinfo/dnsdist
> >
> >
> > ___
> > dnsdist mailing list
> > dnsdist@mailman.powerdns.com
> > https://mailman.powerdns.com/mailman/listinfo/dnsdist
> >
>
>
> --
> Remi Gacogne
> PowerDNS.COM BV - https://www.powerdns.com/
>
> ___
> dnsdist mailing list
> dnsdist@mailman.powerdns.com
> https://mailman.powerdns.com/mailman/listinfo/dnsdist
>
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] dnsdist performance

[Remi Gacogne]

Hi Gentian,

On 8/12/19 10:12 AM, Gentian Bajraktari wrote:
> I have attached results from resperf ( latest compiled, 2.3.1) and the
> configuration file (dnsdist.conf)

Would you consider sending metrics to our public metronome server [1]
so we understand what's the limiting factor during your tests? You only
need to add a line like this one to your dnsdist configuration:

carbonServer("37.252.122.50:2003",
"put-the-name-you-want-to-display-here", 5)

This will only send some metrics without any private information and is
very useful to understand where the bottleneck may be.

[1]: https://metronome1.powerdns.com/

Best regards,

Remi

> On Sat, 10 Aug 2019 at 22:30, Klaus Darilion
> mailto:klaus.mailingli...@pernau.at>> wrote:
> 
> Am 08.08.2019 um 10:20 schrieb Gentian Bajraktari:
> > Dear Dnsdist community,
> >
> > we are trying to setup dnsdist as a loadbalancer for 2 bind recursive
> > cache only servers,
> >
> > we have done a very simple setup:
> > addLocal("DNSDIST_IP")
> > newServer({address="BIND_IP1"})
> > newServer({address=" BIND_IP2"})
> > setServerPolicy(roundrobin)
> >
> > but when we test with resperf (from dnsperf tool), the results are
> very
> > bad for DNSDIST, around 5-15K QPS , while when we test directly to
> one
> > of BIND ip addresses the QPS goes up to 50-60Qps.
> >
> > we have tried both rpm install of dnsdist, compiling from source and
> > even comiling the concur version but all with similar results, ie
> when
> > resperf tests go through dnsdist the results are much lower in
> terms of qps.
> >
> > OS is Centos7 in all servers, tweaked all settings for file
> limits, udp
> > connections etc.
> >
> > any thoughts on what we can do? maybe resperf is not the tool to test
> > this? do you have some suggetions on how we can test and make sure we
> > are doing the right thing?
> 
> I had similar results. Starting 4 listening threads and 4 receivers
> threads (by adding the same backend 4 times) boosted my performance -
> almost linear.
> 
> If it still does not work your should post your config (and first test
> without any rules to avoid performance issues because of rules)
> 
> regards
> Klaus
> ___
> dnsdist mailing list
> dnsdist@mailman.powerdns.com 
> https://mailman.powerdns.com/mailman/listinfo/dnsdist
> 
> 
> ___
> dnsdist mailing list
> dnsdist@mailman.powerdns.com
> https://mailman.powerdns.com/mailman/listinfo/dnsdist
> 


-- 
Remi Gacogne
PowerDNS.COM BV - https://www.powerdns.com/



signature.asc
Description: OpenPGP digital signature
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] dnsdist performance

[Gentian Bajraktari]

I have attached results from resperf ( latest compiled, 2.3.1) and the
configuration file (dnsdist.conf)



On Sat, 10 Aug 2019 at 22:30, Klaus Darilion 
wrote:

> Am 08.08.2019 um 10:20 schrieb Gentian Bajraktari:
> > Dear Dnsdist community,
> >
> > we are trying to setup dnsdist as a loadbalancer for 2 bind recursive
> > cache only servers,
> >
> > we have done a very simple setup:
> > addLocal("DNSDIST_IP")
> > newServer({address="BIND_IP1"})
> > newServer({address=" BIND_IP2"})
> > setServerPolicy(roundrobin)
> >
> > but when we test with resperf (from dnsperf tool), the results are very
> > bad for DNSDIST, around 5-15K QPS , while when we test directly to one
> > of BIND ip addresses the QPS goes up to 50-60Qps.
> >
> > we have tried both rpm install of dnsdist, compiling from source and
> > even comiling the concur version but all with similar results, ie when
> > resperf tests go through dnsdist the results are much lower in terms of
> qps.
> >
> > OS is Centos7 in all servers, tweaked all settings for file limits, udp
> > connections etc.
> >
> > any thoughts on what we can do? maybe resperf is not the tool to test
> > this? do you have some suggetions on how we can test and make sure we
> > are doing the right thing?
>
> I had similar results. Starting 4 listening threads and 4 receivers
> threads (by adding the same backend 4 times) boosted my performance -
> almost linear.
>
> If it still does not work your should post your config (and first test
> without any rules to avoid performance issues because of rules)
>
> regards
> Klaus
> ___
> dnsdist mailing list
> dnsdist@mailman.powerdns.com
> https://mailman.powerdns.com/mailman/listinfo/dnsdist
>

#/usr/local/bin/resperf -s DNSDIST_IP -d queryfile-example-current -C 1 -m 
20 -q 20
DNS Resolution Performance Testing Tool
Version 2.3.1

[Status] Command line: resperf -s DNSDIST_IP -d queryfile-example-current -C 
1 -m 20 -q 20
[Status] Sending
[Status] Reached 20 outstanding queries
[Status] Waiting for more responses
[Status] Testing complete

Statistics:

  Queries sent: 490163
  Queries completed:373610
  Queries lost: 116553
  Response codes:   NOERROR 276271 (73.95%), SERVFAIL 3765 (1.01%), 
NXDOMAIN 93574 (25.05%)
  Run time (s): 62.149292
  Maximum throughput:   42032.00 qps
  Lost at that point:   0.78%


# /usr/local/bin/resperf -s BIND2_IP -d queryfile-example-current -C 1 -m 
20 -q 20
DNS Resolution Performance Testing Tool
Version 2.3.1

[Status] Command line: resperf -s BIND2_IP -d queryfile-example-current -C 
1 -m 20 -q 20
[Status] Sending
[Status] Reached 20 outstanding queries
[Status] Waiting for more responses
[Status] Testing complete

Statistics:

  Queries sent: 516511
  Queries completed:379883
  Queries lost: 136628
  Response codes:   NOERROR 279796 (73.65%), SERVFAIL 5373 (1.41%), 
NXDOMAIN 94714 (24.93%)
  Run time (s): 62.604170
  Maximum throughput:   44166.00 qps
  Lost at that point:   0.00%


dnsdist.conf
Description: Binary data
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] dnsdist performance

[Gentian Bajraktari]

Hi Klaus


i have tried with adding listeners as well as adding same Bind server
again, but results are the same,
its very strange, i get better results hitting directly one Bind server
rather then Dnsdist with 8 listerners and 8 servers ( 4 x Bind1 and 4x
Bind2), and i have no other logic/routing/cache configured on dnsdist,
just the listeners and servers.

what are you using for load testing? resperf?

rg,

Gentian


On Sat, 10 Aug 2019 at 22:30, Klaus Darilion 
wrote:

> Am 08.08.2019 um 10:20 schrieb Gentian Bajraktari:
> > Dear Dnsdist community,
> >
> > we are trying to setup dnsdist as a loadbalancer for 2 bind recursive
> > cache only servers,
> >
> > we have done a very simple setup:
> > addLocal("DNSDIST_IP")
> > newServer({address="BIND_IP1"})
> > newServer({address=" BIND_IP2"})
> > setServerPolicy(roundrobin)
> >
> > but when we test with resperf (from dnsperf tool), the results are very
> > bad for DNSDIST, around 5-15K QPS , while when we test directly to one
> > of BIND ip addresses the QPS goes up to 50-60Qps.
> >
> > we have tried both rpm install of dnsdist, compiling from source and
> > even comiling the concur version but all with similar results, ie when
> > resperf tests go through dnsdist the results are much lower in terms of
> qps.
> >
> > OS is Centos7 in all servers, tweaked all settings for file limits, udp
> > connections etc.
> >
> > any thoughts on what we can do? maybe resperf is not the tool to test
> > this? do you have some suggetions on how we can test and make sure we
> > are doing the right thing?
>
> I had similar results. Starting 4 listening threads and 4 receivers
> threads (by adding the same backend 4 times) boosted my performance -
> almost linear.
>
> If it still does not work your should post your config (and first test
> without any rules to avoid performance issues because of rules)
>
> regards
> Klaus
> ___
> dnsdist mailing list
> dnsdist@mailman.powerdns.com
> https://mailman.powerdns.com/mailman/listinfo/dnsdist
>
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] dnsdist performance

[Remi Gacogne]

Hi Klaus,

On 8/10/19 10:30 PM, Klaus Darilion wrote:
> I had similar results. Starting 4 listening threads and 4 receivers
> threads (by adding the same backend 4 times) boosted my performance -
> almost linear.

Similar in terms of QPS as well? I just tested master on Arch using
calidns and dumresp as a responder, and I get 55k+ QPS on my 10 year-old
CPU using no tuning, simply:

./dnsdist -C /dev/null -l 127.0.0.1:53 127.0.0.1:54

So one listening thread and one receiver thread.

A quick glance at a perf recording shows that more than 76% of the CPU
time is spent in syscalls, so I'm pretty sure disabling the
meltdown/spectre mitigations would do a big difference, but it's already
pretty far from 5-15K QPS.

If you increase the number of threads you'll need to use ring buffers
sharding to limit contention, by the way.

Best regards
-- 
Remi Gacogne
PowerDNS.COM BV - https://www.powerdns.com/



signature.asc
Description: OpenPGP digital signature
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] dnsdist performance

[Klaus Darilion]

Am 08.08.2019 um 10:20 schrieb Gentian Bajraktari:

Dear Dnsdist community,

we are trying to setup dnsdist as a loadbalancer for 2 bind recursive 
cache only servers,


we have done a very simple setup:
addLocal("DNSDIST_IP")
newServer({address="BIND_IP1"})
newServer({address=" BIND_IP2"})
setServerPolicy(roundrobin)

but when we test with resperf (from dnsperf tool), the results are very 
bad for DNSDIST, around 5-15K QPS , while when we test directly to one 
of BIND ip addresses the QPS goes up to 50-60Qps.


we have tried both rpm install of dnsdist, compiling from source and 
even comiling the concur version but all with similar results, ie when 
resperf tests go through dnsdist the results are much lower in terms of qps.


OS is Centos7 in all servers, tweaked all settings for file limits, udp 
connections etc.


any thoughts on what we can do? maybe resperf is not the tool to test 
this? do you have some suggetions on how we can test and make sure we 
are doing the right thing?


I had similar results. Starting 4 listening threads and 4 receivers 
threads (by adding the same backend 4 times) boosted my performance - 
almost linear.


If it still does not work your should post your config (and first test 
without any rules to avoid performance issues because of rules)


regards
Klaus
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] dnsdist performance

[Gentian Bajraktari]

Hi Remi,

yes all servers are on VMware, 8 cpu, 8G Ram. OS is Centos ( Linux release
7.6.1810 (Core) 3.10.0-957.27.2.el7.x86_64)

have disabled selinux and all firewall/iptables,

performance on Bind (BIND 9.14.4 (Stable Release)) directly  is 50-60K QPS
sorry for the typo,

i have tried setMaxUDPOutstanding(65535) as well

as well as i added 8 Local server using reusePort=true ( and also confirmed
with netstat -ln there were 8 running threads)

i also added 8 new servers with combination of both BIND servers or only 1
server. tried different Server policies, leastoutstanding/roundrobin etc.

For the dnsdist version these are 3 versions i have tried ( i have them all
on same machine, playing with systemd service file on which one to activate)

Compiled concur version:
 /usr/local/dnsdist/bin/dnsdist -V
dnsdist 0.0.dnsdistconcur.g (Lua 5.1.4)
Enabled features: systemd

Repo version from centos/epel:
/usr/bin/dnsdist -V
dnsdist 1.3.3 (Lua 5.1.4 [LuaJIT 2.0.4])
Enabled features: dns-over-tls(openssl) dnscrypt libsodium protobuf
recvmmsg/sendmmsg systemd

Compiled main master version:
/usr/local/bin/dnsdist -V
dnsdist 0.0.17315.0.master.g32ffe8cde3 (Lua 5.1.4)
Enabled features: ipcipher recvmmsg/sendmmsg snmp


I also tried having BIND and DNSDIST on same machine ( listening different
interfaces) and also on separate machines,
but no much effect. in all cases Dnsdist has very low QPS compared to
directly testing Bind.

some tweaks, besides removing any security/firewall:

 sysctl -w net.ipv4.conf.all.forwarding=1
 sysctl -w net.core.rmem_max=33554432
 sysctl -w net.core.wmem_max=33554432
 sysctl -w net.core.rmem_default=16777216
sysctl -w net.core.wmem_default=16777216
ulimit -n 16000
fs.file-max = 7

more /etc/security/limits.conf | grep dnsdist
dnsdist soft nofile 65535
dnsdist soft nofile 65535



thanks

On Thu, 8 Aug 2019 at 10:52, Remi Gacogne  wrote:

> Hi Gentian,
>
> On 8/8/19 10:20 AM, Gentian Bajraktari wrote:
> > but when we test with resperf (from dnsperf tool), the results are very
> > bad for DNSDIST, around 5-15K QPS , while when we test directly to one
> > of BIND ip addresses the QPS goes up to 50-60Qps.
>
> Those are very low numbers, for dnsdist but also for Bind (assuming you
> mean 50-60k qps), on what hardware are you testing this? Is this a VM?
>
> > we have tried both rpm install of dnsdist, compiling from source and
> > even comiling the concur version but all with similar results, ie when
> > resperf tests go through dnsdist the results are much lower in terms of
> qps.
>
> Knowing the exact version of dnsdist would help a lot.
>
> Did you read the tuning guide [1]? At the very least you should set
> setMaxUDPOutstanding(65535). You are also using a single thread for
> receiving UDP queries which might not be what you want, although you
> should get much more than 15k qps even with a single thread.
>
> > OS is Centos7 in all servers, tweaked all settings for file limits, udp
> > connections etc.
>
> Did you disable any firewall? Remove any connection tracking modules? It
> would be very useful to know exactly what you tweaked.
>
> [1]: https://dnsdist.org/advanced/tuning.html
>
> Best regards,
> --
> Remi Gacogne
> PowerDNS.COM BV - https://www.powerdns.com/
>
> ___
> dnsdist mailing list
> dnsdist@mailman.powerdns.com
> https://mailman.powerdns.com/mailman/listinfo/dnsdist
>
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] dnsdist performance

[Remi Gacogne]

Please keep replying to the mailing-list instead of replying to me
directly, as your answers might help others. I have not included your
message because I'm not sure what you expected to share publicly, but we
only do non-public support for customers [1].

Would you consider sending metrics to our public metronome server [2]
so we understand what's the limiting factor during your tests? You only
need to add a line like this one to your dnsdist configuration:

carbonServer("37.252.122.50:2003",
"put-the-name-you-want-to-display-here", 5)

This will only send some metrics without any private information and is
very useful to understand where the bottleneck may be.

[1]:
https://blog.powerdns.com/2016/01/18/open-source-support-out-in-the-open/
[2]: https://metronome1.powerdns.com/

Best regards,

Remi

> On Thu, 8 Aug 2019 at 10:52, Remi Gacogne  > wrote:
> 
> Hi Gentian,
> 
> On 8/8/19 10:20 AM, Gentian Bajraktari wrote:
> > but when we test with resperf (from dnsperf tool), the results are
> very
> > bad for DNSDIST, around 5-15K QPS , while when we test directly to one
> > of BIND ip addresses the QPS goes up to 50-60Qps.
> 
> Those are very low numbers, for dnsdist but also for Bind (assuming you
> mean 50-60k qps), on what hardware are you testing this? Is this a VM?
> 
> > we have tried both rpm install of dnsdist, compiling from source and
> > even comiling the concur version but all with similar results, ie when
> > resperf tests go through dnsdist the results are much lower in
> terms of qps.
> 
> Knowing the exact version of dnsdist would help a lot.
> 
> Did you read the tuning guide [1]? At the very least you should set
> setMaxUDPOutstanding(65535). You are also using a single thread for
> receiving UDP queries which might not be what you want, although you
> should get much more than 15k qps even with a single thread.
> 
> > OS is Centos7 in all servers, tweaked all settings for file
> limits, udp
> > connections etc.
> 
> Did you disable any firewall? Remove any connection tracking modules? It
> would be very useful to know exactly what you tweaked.
> 
> [1]: https://dnsdist.org/advanced/tuning.html
> 
> Best regards,
> -- 
> Remi Gacogne
> PowerDNS.COM BV - https://www.powerdns.com/
> 
> ___
> dnsdist mailing list
> dnsdist@mailman.powerdns.com 
> https://mailman.powerdns.com/mailman/listinfo/dnsdist
> 


-- 
Remi Gacogne
PowerDNS.COM BV - https://www.powerdns.com/



signature.asc
Description: OpenPGP digital signature
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] dnsdist performance

[Remi Gacogne]

Hi Gentian,

On 8/8/19 10:20 AM, Gentian Bajraktari wrote:
> but when we test with resperf (from dnsperf tool), the results are very
> bad for DNSDIST, around 5-15K QPS , while when we test directly to one
> of BIND ip addresses the QPS goes up to 50-60Qps.

Those are very low numbers, for dnsdist but also for Bind (assuming you
mean 50-60k qps), on what hardware are you testing this? Is this a VM?

> we have tried both rpm install of dnsdist, compiling from source and
> even comiling the concur version but all with similar results, ie when
> resperf tests go through dnsdist the results are much lower in terms of qps.

Knowing the exact version of dnsdist would help a lot.

Did you read the tuning guide [1]? At the very least you should set
setMaxUDPOutstanding(65535). You are also using a single thread for
receiving UDP queries which might not be what you want, although you
should get much more than 15k qps even with a single thread.

> OS is Centos7 in all servers, tweaked all settings for file limits, udp
> connections etc.

Did you disable any firewall? Remove any connection tracking modules? It
would be very useful to know exactly what you tweaked.

[1]: https://dnsdist.org/advanced/tuning.html

Best regards,
-- 
Remi Gacogne
PowerDNS.COM BV - https://www.powerdns.com/



signature.asc
Description: OpenPGP digital signature
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] dnsdist performance

[Remi Gacogne]

Hi Nico,

On 06/19/2018 10:42 PM, Nico wrote:
> We upgraded to 1.3, everything fine.
> Redid our test with the following results:
> # procs   # listeners kq/s
> 
> % no error
> 
> 1 6   340 94%
> 1 9   380 98%
> 1 15  400 100%
> 1 15  500 99%
> 1 25  600 98%
> 3 6   500 99%
> 
> 
> Above 550 Kqps we were over 85% of interface usage and were many network
> errors.

Thanks a lot for these numbers! It does look a lot better than last time.

> One important thing we noticed is that in our production configuration
> we have ~60 drop rules :
> addAction('example.com ', DropAction())
> 
> Whit these rules the performance dropped almost 30%
> We are moving those rules to the backend resolvers now, because it
> doesn't make sense
> to have this drop on performance when almost all our queries are cached
> (98% hit rate)

Did you consider grouping all those 60 rules into a single one using
SuffixMatchNodeRule()? The cost should be much lighter.


Best regards,
-- 
Remi Gacogne
PowerDNS.COM BV - https://www.powerdns.com/



signature.asc
Description: OpenPGP digital signature
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] dnsdist performance

[Nico]

Just a follow-up on this.

We upgraded to 1.3, everything fine.
Redid our test with the following results:
# procs # listenerskq/s
%
no error

1 6 340 94%
1 9 380 98%
1 15 400 100%
1 15 500 99%
1 25 600 98%
3 6 500 99%
Above 550 Kqps we were over 85% of interface usage and were many network
errors.

One important thing we noticed is that in our production configuration we
have ~60 drop rules :
addAction('example.com', DropAction())

Whit these rules the performance dropped almost 30%
We are moving those rules to the backend resolvers now, because it doesn't
make sense
to have this drop on performance when almost all our queries are cached
(98% hit rate)

Thanks again for this great software!

Nico


On Thu, Apr 6, 2017 at 6:33 PM Nico  wrote:

> Hi Remi,
> Yes, the new version was almost 30% better in the full config test. Great!
>
> > So quite a noticeable gain but it looks like lock contention is still an
> > issue. I would like to understand why, if you don't mind answering a few
> > questions.
> >
> > - You mentioned having 32 cores, are they real cores or is it with
> > hyper-threading? Intel reports [1] only 8 real cores for the E5-2660, so
> > you should probably stick with at most 8 total threads per CPU
> > (listeners mostly in your case).
> You are right, this is with HT.
> CPU(s):32
> Thread(s) per core:2
> Core(s) per socket:8
> Socket(s): 2
> NUMA node(s):  2
> Model name:Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
> CPU MHz:   2194.783
> L1d cache: 32K
> L1i cache: 32K
> L2 cache:  256K
> L3 cache:  20480K
>
> Regarding the number of listeners, we run the tests with different amounts
> of listeners:
> 1,2,4,8,12,
> 1 listener was the worse, 120 Kqps,
> the other configs were more or less the same oscillating from 165 to 175
> being the
> 2 and 8 listeners configs the more stable..
>
>
> > - I'd be interested in the results of the dumpStats() and
> > cache:printStats() commands during your test, as well as a perf top,
> > ideally with a vanilla dnsdist and a dnsdist-concur.
> See attached file
>
> > - The cache cleaning algo might be a bit aggressive by default, you can
> > tweak it with:
> > setCacheCleaningDelay(30)
> > setCacheCleaningPercentage(20)
> Done, but no impact. (because of our test set)
>
> > - Exporting carbon data to our public metronome instance would be great
> > too, as it would immediately make a lot of metrics available to us. You
> > can do that with: carbonServer('37.252.122.50  ', '', 30)
> unfortunately can't do that. This is on a closed net.
> We have our own carbon-graphite and check the stats there.
> can send you any additional info you like to get.
>
> > - Tuning the network buffer might also help:
> > net.core.rmem_max=33554432
> > net.core.wmem_max=33554432
> > net.core.rmem_default=16777216
> > net.core.wmem_default=16777216
> Already done with very similar values.
> Also tried kernel.sched_migration_cost_ns, but with no visible impact.
>
>
> > - Would you consider upgrading your kernel? There has been a lot of
> > improvements since 3.10.0, and we noticed huge performance increases in
> > the past just by upgrading to a 4.x one.
> I would like to do that, but we are required to use redhat
> We've done some tests on a small core2 with 4 cores whith 4.9 and
> we obtained almost the same results as in the "big one".
> This was a surprise.
> Trying to find a way (if security approves) to update redhat kernel.
>
>
> > Oh and if you didn't already, would you mind setting
> > setMaxUDPOutstanding() to 65535? Even at a 99% cache hit ratio, that
> > leaves quite a few requests going to the backend so we better be sure we
> > don't mess up these. The cache in dnsdist tries very hard not to degrade
> > performance, so we prefer skipping the cache and passing the query to a
> > backend rather than waiting for a cache lock, for example.
> Already done, also no difference.
> The queries we are sending are ~50 continously repeating.
>
> Will keep testing. But I think this is all we can get by now.
> The optimum config now seems to be 3 processes with 6 or 8 listeners each.
> Will have to do some workarounds on the stats (aggregation rules on
> graphite?) and
> service control scripts.
>
> Thanks again!
>
>
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] dnsdist performance

[Nico]

Hi Remi,
Yes, the new version was almost 30% better in the full config test. Great!

> So quite a noticeable gain but it looks like lock contention is still an
> issue. I would like to understand why, if you don't mind answering a few
> questions.
>
> - You mentioned having 32 cores, are they real cores or is it with
> hyper-threading? Intel reports [1] only 8 real cores for the E5-2660, so
> you should probably stick with at most 8 total threads per CPU
> (listeners mostly in your case).
You are right, this is with HT.
CPU(s):32
Thread(s) per core:2
Core(s) per socket:8
Socket(s): 2
NUMA node(s):  2
Model name:Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
CPU MHz:   2194.783
L1d cache: 32K
L1i cache: 32K
L2 cache:  256K
L3 cache:  20480K

Regarding the number of listeners, we run the tests with different amounts
of listeners:
1,2,4,8,12,
1 listener was the worse, 120 Kqps,
the other configs were more or less the same oscillating from 165 to 175
being the
2 and 8 listeners configs the more stable..


> - I'd be interested in the results of the dumpStats() and
> cache:printStats() commands during your test, as well as a perf top,
> ideally with a vanilla dnsdist and a dnsdist-concur.
See attached file

> - The cache cleaning algo might be a bit aggressive by default, you can
> tweak it with:
> setCacheCleaningDelay(30)
> setCacheCleaningPercentage(20)
Done, but no impact. (because of our test set)

> - Exporting carbon data to our public metronome instance would be great
> too, as it would immediately make a lot of metrics available to us. You
> can do that with: carbonServer('37.252.122.50  ', '', 30)
unfortunately can't do that. This is on a closed net.
We have our own carbon-graphite and check the stats there.
can send you any additional info you like to get.

> - Tuning the network buffer might also help:
> net.core.rmem_max=33554432
> net.core.wmem_max=33554432
> net.core.rmem_default=16777216
> net.core.wmem_default=16777216
Already done with very similar values.
Also tried kernel.sched_migration_cost_ns, but with no visible impact.


> - Would you consider upgrading your kernel? There has been a lot of
> improvements since 3.10.0, and we noticed huge performance increases in
> the past just by upgrading to a 4.x one.
I would like to do that, but we are required to use redhat
We've done some tests on a small core2 with 4 cores whith 4.9 and
we obtained almost the same results as in the "big one".
This was a surprise.
Trying to find a way (if security approves) to update redhat kernel.


> Oh and if you didn't already, would you mind setting
> setMaxUDPOutstanding() to 65535? Even at a 99% cache hit ratio, that
> leaves quite a few requests going to the backend so we better be sure we
> don't mess up these. The cache in dnsdist tries very hard not to degrade
> performance, so we prefer skipping the cache and passing the query to a
> backend rather than waiting for a cache lock, for example.
Already done, also no difference.
The queries we are sending are ~50 continously repeating.

Will keep testing. But I think this is all we can get by now.
The optimum config now seems to be 3 processes with 6 or 8 listeners each.
Will have to do some workarounds on the stats (aggregation rules on
graphite?) and
service control scripts.

Thanks again!
RPM
8 listeners
receiving constat 132 kqps traffic
answering ~96% of queries
> dumpStats()
acl-drops 0latency1-10   28
block-filter  0latency10-50  13
cache-hits 14010959latency100-1000   41
cache-misses344latency50-100  1
cpu-sys-msec 147910no-policy  0
cpu-user-msec122819noncompliant-queries   0
downstream-send-errors0noncompliant-responses 0
downstream-timeouts   0queries 14011303
dyn-block-nmg-size0rdqueries   14011303
dyn-blocked   0real-memory-usage  261877760
empty-queries 0responses344
fd-usage 63rule-drop  0
latency-avg1000.0  rule-nxdomain  0
latency-avg1000   0.0  rule-refused   0
latency-avg1  0.1  self-answered  0
latency-avg1000.4  servfail-responses 0
latency-slow  0trunc-failures 0
latency0-1 14011220uptime   247

> cache:printStats()
Entries: 54/10
Hits: 14010959
Misses: 282
Deferred inserts: 15
Deferred lookups: 62
Lookup Collisi

Re: [dnsdist] dnsdist performance

[Remi Gacogne]

Hi Nico,

On 04/05/2017 12:24 AM, Nico wrote:
> Test with rule REFUSED:
> dnsdist from rpm: ~450 Kqps
> dnsdist-concur: ~530kqps
> 17% improvement

I guess that's the move from CLOCK_MONOTONIC_RAW to CLOCK_MONOTONIC,
quite impressive.

> Test with answer from cache:
> 1 dnsdist from rpm 12 listeners: ~130 kqps
> 3 dnsdist from rpm 6 listeners each: ~370kqps
> 1 dnsdist-concur 12 listeners ~170 kqps
> 1 dnsdist-concur 20 listeners  ~170 kqps
> 3 dnsdist-concur 6 listeners each ~450 kqps

So quite a noticeable gain but it looks like lock contention is still an
issue. I would like to understand why, if you don't mind answering a few
questions.

- You mentioned having 32 cores, are they real cores or is it with
hyper-threading? Intel reports [1] only 8 real cores for the E5-2660, so
you should probably stick with at most 8 total threads per CPU
(listeners mostly in your case).

- I'd be interested in the results of the dumpStats() and
cache:printStats() commands during your test, as well as a perf top,
ideally with a vanilla dnsdist and a dnsdist-concur.

- The cache cleaning algo might be a bit aggressive by default, you can
tweak it with:
setCacheCleaningDelay(30)
setCacheCleaningPercentage(20)

- Exporting carbon data to our public metronome instance would be great
too, as it would immediately make a lot of metrics available to us. You
can do that with: carbonServer('37.252.122.50  ', '', 30)

- Tuning the network buffer might also help:
net.core.rmem_max=33554432
net.core.wmem_max=33554432
net.core.rmem_default=16777216
net.core.wmem_default=16777216

- Would you consider upgrading your kernel? There has been a lot of
improvements since 3.10.0, and we noticed huge performance increases in
the past just by upgrading to a 4.x one.

Oh and if you didn't already, would you mind setting
setMaxUDPOutstanding() to 65535? Even at a 99% cache hit ratio, that
leaves quite a few requests going to the backend so we better be sure we
don't mess up these. The cache in dnsdist tries very hard not to degrade
performance, so we prefer skipping the cache and passing the query to a
backend rather than waiting for a cache lock, for example.

Thanks,

Remi

[1]:
https://ark.intel.com/products/64584/Intel-Xeon-Processor-E5-2660-20M-Cache-2_20-GHz-8_00-GTs-Intel-QPI




signature.asc
Description: OpenPGP digital signature
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] dnsdist performance

[Nico]

We re-run the tests with concur.

The result were better:

Test with rule drop:
no noticeable change,  around 650kqps, guess this is network or stack limit

Test with rule REFUSED:
dnsdist from rpm: ~450 Kqps
dnsdist-concur: ~530kqps
17% improvement

Test with answer from cache:
1 dnsdist from rpm 12 listeners: ~130 kqps
3 dnsdist from rpm 6 listeners each: ~370kqps
1 dnsdist-concur 12 listeners ~170 kqps
1 dnsdist-concur 20 listeners  ~170 kqps
3 dnsdist-concur 6 listeners each ~450 kqps


Hope this help.
Thanks !


On Mon, Apr 3, 2017 at 6:03 PM, bert hubert 
wrote:

> On Mon, Apr 03, 2017 at 05:44:06PM -0300, Nico wrote:
> > Hi!
> >
> > Thanks for answering!.
> > I'm not pretending to be anonymous.
> > Mi name is Nicolas Baumgarten and I do infrastructure support .
>
> Hi, welcome! Thanks.
>
> > One dnsdist process, whith 12 listeners, small cache, and all cached
> queries
> > can't get more than 120/140 kqps on our config.
> >
> > Now we just tried to run 3 instances of dnsdist, with almost the same
> > config and 6 listeners each,
> > and we obtained 360/380 kqps, that's a linear increment.
>
> Indeed - the goal of the dnsdist-concur branch is to get a single dnsdist
> to
> be (nearly) as fasst as 6 dnsdsists on the same machine.
>
> > I'm happy to share my results, what else I can send you?
>
> Please let us know what your results are with dnsdist-concur, we have one
> measurement that is very good, but more would be welcome.
>
> Bert
>
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] dnsdist performance

[bert hubert]

On Mon, Apr 03, 2017 at 05:44:06PM -0300, Nico wrote:
> Hi!
> 
> Thanks for answering!.
> I'm not pretending to be anonymous.
> Mi name is Nicolas Baumgarten and I do infrastructure support .

Hi, welcome! Thanks.

> One dnsdist process, whith 12 listeners, small cache, and all cached queries
> can't get more than 120/140 kqps on our config.
> 
> Now we just tried to run 3 instances of dnsdist, with almost the same
> config and 6 listeners each,
> and we obtained 360/380 kqps, that's a linear increment.

Indeed - the goal of the dnsdist-concur branch is to get a single dnsdist to
be (nearly) as fasst as 6 dnsdsists on the same machine.

> I'm happy to share my results, what else I can send you?

Please let us know what your results are with dnsdist-concur, we have one
measurement that is very good, but more would be welcome.

Bert
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] dnsdist performance

[Nico]

Hi!

Thanks for answering!.
I'm not pretending to be anonymous.
Mi name is Nicolas Baumgarten and I do infrastructure support .

I really appreciate your work. And hope to help.

Will try your suggestion with the version from repository.

Our results are those which I've send in the initial mail.

I'm trying to set up a baseline for maximum performance with cache,
and without any additional rules to be able to compare the impact of
future configuration changes.

One dnsdist process, whith 12 listeners, small cache, and all cached queries
can't get more than 120/140 kqps on our config.

Now we just tried to run 3 instances of dnsdist, with almost the same
config and 6 listeners each,
and we obtained 360/380 kqps, that's a linear increment.

I'm happy to share my results, what else I can send you?

Thanks!


On Mon, Apr 3, 2017 at 5:00 PM, bert hubert 
wrote:

> Hello "Nico Mail",
>
> Thank you for contacting us anonymously. My name is Bert Hubert and we work
> very hard to provide fast software for you, for free.
>
> Could you try this branch:
>
> git clone -b dnsdist-concur https://github.com/ahupowerdns/pdns.git
> cd pdns/pdns/dnsdistdist
> autoreconf -i
> ./configure
> make -j4 -k dnsdist
> ./dnsdist -C /your/dnsdist.conf
>
> Before you compile, try
> $ apt-get install g++ libboost-all-dev libtool make pkg-config
> libmysqlclient-dev libssl-dev liblua-dev
>
> Or the equivalent.
>
> Could you let us know what performance you measure?
>
> We've recently been optimizing the dnsdist cache for SMP use and timings,
> and we measure around 500kqps now.  Would love to know your results.
>
> Thanks.
>
> Bert
>
>
> On Mon, Apr 03, 2017 at 04:30:47PM -0300, Nico wrote:
> > Hi!
> > We found dnsdist very useful and interesting and using it in a couple of
> > configurations.
> > Now we are testing dnsdist to find the best tuning for our setup,
> > and we find the performance is not what we expected (based on what we saw
> > in presentations)
> >
> > We can't get more than 120/150 kqps
> > We don't think it's network related, and doesn't seem to be interface or
> > udp stack.
> >
> > If we test whith only
> >
> > addAction(AllRule(),DropAction())
> > We get ~600kqps
> >
> > with some answer
> > addAction(AllRule(),RCodeAction(dnsdist.REFUSED))
> > we get ~450kqps
> >
> > but when we enable cache, and real queries/answers we get stuck at
> ~120kqps.
> > (numbers and test details below.)
> >
> > The queries are a repeated sequence, so we have 100% cache hit.
> >
> > Is this a real limit? or there is something else we can tune?
> >
> > Thanks!
> >
> > 
> > Our config:
> > dnsdist-1.1.0 from rpm (also compiled, no difference)
> > Centos 7 kernel 3.10.0
> > 32 cores  CPU E5-2660/ 32GB ram
> > 1gb card.
> >
> > we use resperf and drool to test dns.
> > nuutcp and iperf3 to test network  (reporting 900mbs udp and 900Kpps udp
> > 150bytes pkts)
> > and netdata and graphite to see usage/errors/qps, etc.
> >
> > The tests.
> > Always 12 listeners (addLocal)
> > ===
> > addAction(AllRule(),DropAction())
> > drool
> > sending ~ 900kqps
> > dnsdist web reports: ~650kqps
> > udp.inerrrors  small
> > =
> > addAction(AllRule(),RCodeAction(dnsdist.REFUSED))
> > drool
> > sending ~460Kqps
> > dnsdist report ~450kqps
> > udp.inerrrors  ~0 whith sporadic high lost peak
> >
> > multiple resperf:
> > Sending 650 Kqps
> > dnsdist reports ~400kqps
> > Udp.inerrors ~250 kpps
> >
> > sending 300 kqps,  reports ~300kqps, lost responses: negligible
> > udp.inerrors ~0
> > Recv-Q ~0 Send-Q 0-5000
> > 
> > Full config:
> > 12 addLocal, small cache, many backend servers (almost not used)
> >
> > cache = newPacketCache(1000, 86400, 0, 60, 60)
> > getPool("dns-dist1"):setCache(cache)
> > addPoolRule({"0.0.0.0/0"}, "dns-dist1")
> >
> > setServerPolicy(firstAvailable)
> >
> > Seding more than 140kqps Recv-Q grows, and queries get lost.
> > DNSDIST web interface and graphite counters report between 120kqps and
> > 140kqps
> > never more.
> >
> > Netstat output
> > Active Internet connections (servers and established)
> > Proto Recv-Q Send-Q Local Address   Foreign Address State
> > udp   14006016   1536 dns-dist1-e:domain0.0.0.0:*
> > udp   16487424768 dns-dist1-e:domain0.0.0.0:*
> > udp   16766208   3328 dns-dist1-e:domain0.0.0.0:*
> > udp   14918400   9472 dns-dist1-e:domain0.0.0.0:*
> > udp   16750848   2048 dns-dist1-e:domain0.0.0.0:*
> > udp   13962240768 dns-dist1-e:domain0.0.0.0:*
> > udp   16637184  0 dns-dist1-e:domain0.0.0.0:*
> > udp   12026880   2048 dns-dist1-e:domain0.0.0.0:*
> > udp   15539712768 dns-dist1-e:domain0.0.0.0:*
> > udp   16772352  0 dns-dist1-e:domain0.0.0.0:*
> > udp   10476288768 dns-dist1-e:domain0.0.0.0:*
> > udp   15005184  0 dns-d

Re: [dnsdist] dnsdist performance

[bert hubert]

Hello "Nico Mail",

Thank you for contacting us anonymously. My name is Bert Hubert and we work
very hard to provide fast software for you, for free. 

Could you try this branch:

git clone -b dnsdist-concur https://github.com/ahupowerdns/pdns.git 
cd pdns/pdns/dnsdistdist
autoreconf -i
./configure
make -j4 -k dnsdist
./dnsdist -C /your/dnsdist.conf

Before you compile, try 
$ apt-get install g++ libboost-all-dev libtool make pkg-config 
libmysqlclient-dev libssl-dev liblua-dev

Or the equivalent.

Could you let us know what performance you measure?

We've recently been optimizing the dnsdist cache for SMP use and timings,
and we measure around 500kqps now.  Would love to know your results.

Thanks.

Bert


On Mon, Apr 03, 2017 at 04:30:47PM -0300, Nico wrote:
> Hi!
> We found dnsdist very useful and interesting and using it in a couple of
> configurations.
> Now we are testing dnsdist to find the best tuning for our setup,
> and we find the performance is not what we expected (based on what we saw
> in presentations)
> 
> We can't get more than 120/150 kqps
> We don't think it's network related, and doesn't seem to be interface or
> udp stack.
> 
> If we test whith only
> 
> addAction(AllRule(),DropAction())
> We get ~600kqps
> 
> with some answer
> addAction(AllRule(),RCodeAction(dnsdist.REFUSED))
> we get ~450kqps
> 
> but when we enable cache, and real queries/answers we get stuck at ~120kqps.
> (numbers and test details below.)
> 
> The queries are a repeated sequence, so we have 100% cache hit.
> 
> Is this a real limit? or there is something else we can tune?
> 
> Thanks!
> 
> 
> Our config:
> dnsdist-1.1.0 from rpm (also compiled, no difference)
> Centos 7 kernel 3.10.0
> 32 cores  CPU E5-2660/ 32GB ram
> 1gb card.
> 
> we use resperf and drool to test dns.
> nuutcp and iperf3 to test network  (reporting 900mbs udp and 900Kpps udp
> 150bytes pkts)
> and netdata and graphite to see usage/errors/qps, etc.
> 
> The tests.
> Always 12 listeners (addLocal)
> ===
> addAction(AllRule(),DropAction())
> drool
> sending ~ 900kqps
> dnsdist web reports: ~650kqps
> udp.inerrrors  small
> =
> addAction(AllRule(),RCodeAction(dnsdist.REFUSED))
> drool
> sending ~460Kqps
> dnsdist report ~450kqps
> udp.inerrrors  ~0 whith sporadic high lost peak
> 
> multiple resperf:
> Sending 650 Kqps
> dnsdist reports ~400kqps
> Udp.inerrors ~250 kpps
> 
> sending 300 kqps,  reports ~300kqps, lost responses: negligible
> udp.inerrors ~0
> Recv-Q ~0 Send-Q 0-5000
> 
> Full config:
> 12 addLocal, small cache, many backend servers (almost not used)
> 
> cache = newPacketCache(1000, 86400, 0, 60, 60)
> getPool("dns-dist1"):setCache(cache)
> addPoolRule({"0.0.0.0/0"}, "dns-dist1")
> 
> setServerPolicy(firstAvailable)
> 
> Seding more than 140kqps Recv-Q grows, and queries get lost.
> DNSDIST web interface and graphite counters report between 120kqps and
> 140kqps
> never more.
> 
> Netstat output
> Active Internet connections (servers and established)
> Proto Recv-Q Send-Q Local Address   Foreign Address State
> udp   14006016   1536 dns-dist1-e:domain0.0.0.0:*
> udp   16487424768 dns-dist1-e:domain0.0.0.0:*
> udp   16766208   3328 dns-dist1-e:domain0.0.0.0:*
> udp   14918400   9472 dns-dist1-e:domain0.0.0.0:*
> udp   16750848   2048 dns-dist1-e:domain0.0.0.0:*
> udp   13962240768 dns-dist1-e:domain0.0.0.0:*
> udp   16637184  0 dns-dist1-e:domain0.0.0.0:*
> udp   12026880   2048 dns-dist1-e:domain0.0.0.0:*
> udp   15539712768 dns-dist1-e:domain0.0.0.0:*
> udp   16772352  0 dns-dist1-e:domain0.0.0.0:*
> udp   10476288768 dns-dist1-e:domain0.0.0.0:*
> udp   15005184  0 dns-dist1-e:domain0.0.0.0:*

> ___
> dnsdist mailing list
> dnsdist@mailman.powerdns.com
> https://mailman.powerdns.com/mailman/listinfo/dnsdist

___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] dnsdist performance

[Nico]

Hi, thanks for you fast reply,

but  i think that setMaxUDPOutstanding relates to queries pending from
backend servers to dnsdist.
My tests are almost all serviced from cache.

On Mon, Apr 3, 2017 at 4:42 PM, Aleš Rygl  wrote:

>
> > We can't get more than 120/150 kqps
> > We don't think it's network related, and doesn't seem to be interface or
> > udp stack.
>
> Hi,
>
> I was struggling with some packet loss some time ago. There was a packet
> loss
> in the output of netstat -su command.
>
> Finally I was advised to increase to use setMaxUDPOutstanding(65535) which
> worked as a magic.
>
> BR
>
> Ales
> ___
> dnsdist mailing list
> dnsdist@mailman.powerdns.com
> https://mailman.powerdns.com/mailman/listinfo/dnsdist
>
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] dnsdist performance

[Aleš Rygl]

 
> We can't get more than 120/150 kqps
> We don't think it's network related, and doesn't seem to be interface or
> udp stack.

Hi, 

I was struggling with some packet loss some time ago. There was a packet loss 
in the output of netstat -su command.

Finally I was advised to increase to use setMaxUDPOutstanding(65535) which 
worked as a magic.

BR

Ales
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist