Re: [dnsdist] dnsdist tuning for high qps on nxdomain ddos

[Klaus Darilion via dnsdist]

Hi Jasper!

Not that I can help you that much with dnsdist, but I want to share some things 
we have done….

I found some measurements from 2022 on a VM with 8 vCPUs.
Dnsdist with PowerDns/postgresql Backend and random queries: 20k qps
Dnsdist (with PowerDns/postgresql Backend) and hot dnsdist cache: 150k qps
Knot and random queries: 575k qps

So I think, if you do not need the dnsdist features you might be better using a 
faster nameserver for all your zones on the public facing name servers. 250K 
zones is doable with Knot and Co. We still use PowerDNS for zone provisioning 
(API) and we still use dnsdist+PowerDNS as public facing nameservers. But for 
customer which have random subdomain attacks 24x7 we use Knot as public facing 
nameserver (which get its zone via AXFR from a local PowerDNS). Of course this 
is more management overhead but solved our random subdomain attack problems.

You might be interested in my talk at DNS-OARC [1].

It was quite some work until it was running smoothly, but we now serve several 
million zones from Knot. Some things are not that easy any more and checking if 
all zones are in sync is cumbersome [2]. You might also consider, like we do, 
using 2 setups, one with dnsdist+powerdns for “normal” zones and using only 
Knot (or NSD/Bind) for “exposed” zones.

But on the other hand: If you manage to tune dnsdist please let us know 


Regards
Klaus

[1] https://indico.dns-oarc.net/event/47/contributions/1008/
https://www.youtube.com/watch?v=8UnM7_uGDv0

[2] https://indico.dns-oarc.net/event/47/contributions/1017/
https://www.youtube.com/watch?v=jgtODGv7X4Y


Von: dnsdist  Im Auftrag von Jasper 
Aikema via dnsdist
Gesendet: Montag, 6. Mai 2024 16:02
An: dnsdist@mailman.powerdns.com
Betreff: Re: [dnsdist] dnsdist tuning for high qps on nxdomain ddos

> 200k QPS is fairly low based on what you describe. Would you mind
> sharing the whole configuration (redacting passwords and keys, of
> course), and telling us a bit more about the hardware dnsdist is running on?

The server is a virtual server (Ubuntu 22.04) on our vmware platform with 16GB 
of memory and 8 cores (Intel Xeon 4214R @2.4Ghz). I have pasted the new config 
at the bottom of this message.

> 6 times the amount of cores is probably not a good idea. I usually
> advise to make it so that the number of threads is roughly equivalent to
> the number of cores that are dedicated to dnsdist, so in your case the
> number of addLocal + the number of newServer + the number of TCP workers
> should ideally match the number of cores you have. If you need to
> overcommit the cores a bit that's fine, but keep it to something like
> twice the number of cores you have, not 10 times.

> I'm pretty sure this does not make sense, I would first go with the
> default until you see TCP/DoT connections are not processed correctly.

I did overcommit / try to tune, because I was getting a high number of 
udp-in-errors and also a high number of Drops in showServers().
If those issues are gone, I agree there should be no reason to overcommit.

> When you say it doesn't work for NXDomain, I'm assuming you mean it
> doesn't solve the problem of random sub-domains attacks, not that a
> NXDomain is not properly cached/accounted?

Yes. That is indeed what I meant, the responses are getting cached, but that is 
exactly why nxdomains attacks are working. They request a lot of random 
sub-domains and caching doesnt help making it more responsive.

> I expect lowering the number of threads will reduce the context switches
> a lot. If you are still not getting good QPS numbers, I would suggest
> checking if disabling the rules help, to figure out the bottleneck. You
> might also want to take a look with "perf top -p "
> during the high load to see where the CPU time is spent.

I have updated the config and lowered the threads. But now I get a high number 
of udp-in-errors. The perf top command gives:

Samples: 80K of event 'cpu-clock:pppH', 4000 Hz, Event count (approx.): 
15028605853 lost: 0/0 drop: 0/0
Overhead  Shared Object   Symbol
   4.78%  [kernel][k] __lock_text_start
   2.29%  [kernel][k] 
copy_user_generic_unrolled
   2.29%  [kernel][k] 
copy_from_kernel_nofault
   1.86%  [nf_conntrack]  [k] 
__nf_conntrack_find_get
   1.81%  [kernel][k] __fget_files
   1.42%  [kernel][k] _raw_spin_lock
   1.39%  [vmxnet3]   [k] 
vmxnet3_poll_rx_only
   1.34%  [kernel][k] 
finish_task_switch.isra.0
   1.32%  [nf_tables] [k] nft_do_chain
   1.23%  libc.so.6   [.] cfree
   1.08%  [kernel][k] 
__siphash_unaligned
   1.07% 

Re: [dnsdist] [EXT] AW: Suggestions for rules to block abusive traffic

[Klaus Darilion via dnsdist]

Hi Remi!

Thanks for the details. 

> > Blocking all queries to the attacked domain prevents collateral
> damage, but causes a DoS to the attacked domain and makes the customer
> of the attacked domain unhappy.
> 
> I fully agree, and we are working on having smarter mitigations in
> dnsdist to only drops/truncate/route to a different pool queries that
> are very likely to be part of a PRSD/enumeration attack.

Do you already have ideas how to implement that? I have thought a lot about an 
algorithm to block only "bad" queries bad have not found a method yet.

For authoritative nameservers, meanwhile I think it would be better to just 
load the attacked zone completely into dnsdist or pdns-cache (or something 
similar to aggressive caching). Because I think just answering (mostly 
NXDOMAIN) may be faster then deciding if a query is bad or good.

Regards
Klaus

___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] Suggestions for rules to block abusive traffic

[Klaus Darilion via dnsdist]

> -Ursprüngliche Nachricht-
> Von: dnsdist  Im Auftrag von
> Remi Gacogne via dnsdist
> Gesendet: Montag, 8. Januar 2024 17:51
> An: dnsdist@mailman.powerdns.com
> Betreff: Re: [dnsdist] Suggestions for rules to block abusive traffic
> 
> Hi Dan,
> 
> On 08/01/2024 17:28, Dan McCombs via dnsdist wrote:
> >   In our case we are affected as we use Pdns + DB backend as backend.
> >
> > Yep, that's exactly our case as well - our legacy Pdns + mysql backends
> > don't handle this very well. Longer term we intend to move away from
> > that, but finding some improvements in the meantime for handling these
> > floods would be helpful. I'll let you know if we come up with anything
> > interesting!
> 
> This is unfortunately a common issue indeed these days. It is possible
> to use dnsdist to detect and mitigate these attacks to a certain extent,
> using the StatNode API along with DynBlockRulesGroup:setSuffixMatchRule
> [1] or the FFI equivalent for better performance. It requires writing a
> bit of Lua code and some tuning on top of dnsdist, but all the building
> blocks are there already. We have implemented this for several customers
> and they are happy with the results.

Hi Remi!
How does this work in detail? Does your implementation block only the queries 
for .example.com or also "normal" queries like www.example.com or 
example.com MX? Or do you explicitly allow common subdomains before blocking 
everything else?

Blocking all queries to the attacked domain prevents collateral damage, but 
causes a DoS to the attacked domain and makes the customer of the attacked 
domain unhappy.

Regards
Klaus
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] Suggestions for rules to block abusive traffic

[Klaus Darilion via dnsdist]

Von: Dan McCombs 
Gesendet: Montag, 8. Januar 2024 17:28
An: Klaus Darilion 
Cc: dnsdist@mailman.powerdns.com
Betreff: Re: [dnsdist] Suggestions for rules to block abusive traffic

Hi Klaus!

 In our case we are affected as we use Pdns + DB backend as backend.

Yep, that's exactly our case as well - our legacy Pdns + mysql backends don't 
handle this very well. Longer term we intend to move away from that, but 
finding some improvements in the meantime for handling these floods would be 
helpful. I'll let you know if we come up with anything interesting!

If you use PDNS make sure to use at least version 4.5 and use 
https://doc.powerdns.com/authoritative/settings.html#zone-cache-refresh-interval
 and 
https://doc.powerdns.com/authoritative/settings.html#setting-consistent-backends=yes
 (this saves plenty of DB queries). Further, the DB server must have enough RAM 
to have the database in RAM (i.e. in the linux file buffers).

Further you might be interested in 
https://indico.dns-oarc.net/event/47/contributions/1008/ and 
https://indico.dns-oarc.net/event/47/contributions/1017/ if you plan to use 
another name server. Another very fresh option would be PDNS + lmdb backend and 
https://doc.powerdns.com/lightningstream/ for replication.

For dnsdist there are probably other guys with more know.

Regards
Klaus
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] Suggestions for rules to block abusive traffic

[Klaus Darilion via dnsdist]

Hi Dan!

This is a known issue and we have not found a simple solution in dnsdist. And 
obviously it is only a problem if the backend is slow. In our case we are 
affected as we use Pdns + DB backend as backend.

  1.  Use a fast name server as additional backend (we used NSD) and 
dynamically provision targeted zones (and all subzones) on the faster backend 
and redirect the zone to the fast backend (dnsdist rule). Out detection is 
based on “dsc” statistics collector.
  2.  Use a fast nameserver instead of dnsdist + slow backend (we use Knot for 
customers that are constantly under attack)

These two methods helped us, but of course add additional operations work to 
implement and operate it.

If you find a simple dnsdist based solution to filter these random queries I 
would be interested too ;-)

Regards
Klaus

Von: dnsdist  Im Auftrag von Dan McCombs 
via dnsdist
Gesendet: Freitag, 29. Dezember 2023 20:11
An: dnsdist@mailman.powerdns.com
Betreff: [dnsdist] Suggestions for rules to block abusive traffic

Hi all,

I'm wondering if anyone has suggestions of reasonable ways to handle this type 
of abusive traffic with dnsdist.

We've had on and off attacks recently targeting legitimate domains delegated to 
our authoritative service flooding queries for random subdomains of varying 
length and characters/words. i.e. 12345.example.com, 
fred.example.com, 
abc178371jd.example.com, where 
example.com is a different domain we're authoritative for 
each attack.

The dnsdist nodes can handle the traffic, but breaking cache and going through 
to our backends is having more of an impact.

We have thousands of domains, so it doesn't seem reasonable to apply individual 
rate limits to them all, but if there is a straight forward way to do something 
like that I'd be happy to hear it. The source addresses are well known public 
resolvers that we shouldn't rate limit either.

I'm wondering if there's any way to detect and apply a rule dynamically to 
respond to queries for one of these domains without affecting the source IP 
address entirely, and not require us to manually add a rule for each domain as 
it occurs.

Any ideas would be appreciated.

Take care,

-Dan

[https://digitaloceanspace.nyc3.digitaloceanspaces.com/do-sig_files/do-email_signature.png]

Dan McCombs
Senior Engineer I - DNS
dmcco...@digitalocean.com
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] Backend Questions

[Klaus Darilion via dnsdist]

> > Shouldn't newServer(...healthCheckMode='UP') also work? In my case it
> does not work.
> > I have set healthCheckMode='UP' but:
> > showServers show status as "up" whereas after setUp() the status is "UP".
> And it still does helathchecks and status goes "down" if the backend is down.
> >
> > What is wrong? The docs? Am I misinterpreting the docs? Bug?
> 
> Which version are yor running? healthCheckMode wil only be available
> in 1.8.0, which is not released yet. See
> https://dnsdist.org/reference/config.html?highlight=newserver#newServer

Oh I missed that comment about changes in 1.8.

Thanks
Klaus
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] Backend Questions

[Klaus Darilion via dnsdist]

(resent to the list)

Hi Remi!

> On 07/10/2022 10:53, Klaus Darilion via dnsdist wrote:
> 
> > We use dnsdist with 1 single backend server (PDNS). So if this backend
> > is overloaded, dnsdist will detect the backend as DOWN. Hence, the only
> > server for this backend pool down. How will dnsdist behave if all
> > servers for a backend pool are down? Will it stop senden queries to the
> > backend, or will it still send queries to the DOWN server as there is no
> > UP server available?
> 
> All of the built-in load balancing policies will stop forwarding queries
> when all the servers in the selected pool are down. It would be possible
> to write a custom load-balancing in Lua that does not do that, of
> course, but I don't think that's what you want in that case.
> 
> > So it may be useful to disable healthchecks completely. How can this be
> > done?
> 
> You can force a server in the "up" state using the 'setUp()' method, see
> [1].

Shouldn't newServer(...healthCheckMode='UP') also work? In my case it does not 
work. 
I have set healthCheckMode='UP' but:
showServers show status as "up" whereas after setUp() the status is "UP". And 
it still does helathchecks and status goes "down" if the backend is down.

What is wrong? The docs? Am I misinterpreting the docs? Bug?

Thanks
Klaus
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

[dnsdist] Backend Questions

[Klaus Darilion via dnsdist]

Hello!

We use dnsdist with 1 single backend server (PDNS). So if this backend is 
overloaded, dnsdist will detect the backend as DOWN. Hence, the only server for 
this backend pool down. How will dnsdist behave if all servers for a backend 
pool are down? Will it stop senden queries to the backend, or will it still 
send queries to the DOWN server as there is no UP server available?

So it may be useful to disable healthchecks completely. How can this be done?


My current config is a few years old tested with dnsdist 1.3. These days, 
dnsdist was faster when I added the listen port multiple times, and also add 
the single backend server multiple times, to have more receiver threads. For 
example:
-- Open the same socket multiple times. This allows better load distribution
-- over all cores. Note: 1. setLocal(), dann addLocal()!
setLocal("192.174.68.96:53", { reusePort=true, tcpFastOpenSize=100 })
addLocal("192.174.68.96:53", { reusePort=true, tcpFastOpenSize=100 })

-- Define the Backend Server Pools. Define them multiple times to have multiple 
receiver threads
-- handling the responses from the Backend.
newServer{address='127.0.0.1:14001',name='pdns_1'}   -- this is the 
PowerDNS server
newServer{address='127.0.0.1:14001',name='pdns_2'}   -- this is the 
PowerDNS server


Is it still (dnsdist 1.6/1.7) useful/necessary to add listenSockets and 
Backendserver multiple times to improve performance?

Thanks
Klaus


--
Klaus Darilion, Head of Operations
nic.at GmbH, Jakob-Haringer-Straße 8/V
5020 Salzburg, Austria
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] dnsdist[29321]: Marking downstream IP:53 as 'down'

[Klaus Darilion via dnsdist]

Indeed that might be a problem. We use (ferm syntax):
table raw {
# Wir wollen NOTRACK fuer eingehende DNS Anfragen und die dazugehoerigen
# ausgehenden Antworten. Ausgehende DNS Anfragen sollen weiter getrackt
# werden damit die dazugehoerige Antwort rein darf.
chain PREROUTING {
proto (udp tcp) dport 53 NOTRACK;
}
chain OUTPUT {
proto (udp tcp) sport 53 NOTRACK;
}
}
Same for IPv4 and IPv6 in our case.

regards
Klaus





Von: dnsdist  Im Auftrag von Rasto 
Rickardt via dnsdist
Gesendet: Donnerstag, 24. März 2022 11:36
An: dnsdist@mailman.powerdns.com
Betreff: Re: [dnsdist] dnsdist[29321]: Marking downstream IP:53 as 'down'

Hello Rais,
i noticed that you are increasing nf_conntrack_max. I am not sure how the 
backend servers are connected,
but i suggest not to use connection tracking/NAT at all. You can use for 
example dedicated interface for backend
management and other one to connect to dnsdist.
r.
On 24/03/2022 11:11, Rais Ahmed via dnsdist wrote:
Hi,

Thanks for the guidance...!

We are testing with multiple scenarios, with/without kernel tuning. We observed 
UDP packets errors on both backend servers (not a single UDP error on dnsdist 
LB server).

Tested with resperf 15K QPS
resperf -s 192.168.0.1 -R -d queryfile-example-10million-201202 -C 100 -c 300 
-r 0 -m 15000 -q 20

Backend 1: 192.168.1.1 (without Kernel tuning):
netstat -su
IcmpMsg:
InType3: 2229
InType8: 6
InType11: 194
OutType0: 6
OutType3: 762
Udp:
1634847 packets received
843 packets to unknown port received.
193891 packet receive errors
1859642 packets sent
193891 receive buffer errors
0 send buffer errors
UdpLite:
IpExt:
InOctets: 580762744
OutOctets: 237368675
InNoECTPkts: 1995692
InECT0Pkts: 27

Backend 2: 192.168.1.2 (with Kernel Tuning):
netstat -su
IcmpMsg:
InType3: 19177
InType8: 5802
InType11: 2645
OutType0: 5802
OutType3: 5122
Udp:
10798358 packets received
6846 packets to unknown port received.
4815377 packet receive errors
11949871 packets sent
4815377 receive buffer errors
0 send buffer errors
UdpLite:
IpExt:
InNoRoutes: 11
InOctets: 3312682950
OutOctets: 1741771756
InNoECTPkts: 16355120
InECT1Pkts: 72
InECT0Pkts: 92
InCEPkts: 4

Kernel Tuning configured in /etc/rc.local

ethtool -L eth0 combined 16
echo 52428800 > /proc/sys/net/netfilter/nf_conntrack_max
sysctl -w net.core.rmem_max=33554432
sysctl -w net.core.wmem_max=33554432
sysctl -w net.core.rmem_default=16777216
sysctl -w net.core.wmem_default=16777216
sysctl -w net.core.netdev_max_backlog=65536
sysctl -w net.core.somaxconn=1024
ulimit -n 16000

Network config/ specs are same on all three servers, are we doing something 
wrong?


Regards,
Rais 

-Original Message-
From: Klaus Darilion mailto:klaus.daril...@nic.at 
Sent: Thursday, March 24, 2022 12:38 PM
To: Rais Ahmed mailto:rais.ah...@tes.com.pk; mailto:dnsdist@mailman.powerdns.com
Subject: AW: [dnsdist] dnsdist[29321]: Marking downstream IP:53 as 'down'

Have you tested how many Qps your Backend is capably to handle? First test your 
Backend performance to know how much qps a single backend can handle. I guess 
500k qps might be difficult to achieve with bind. If you need more performance 
switch the Backend to NSD or Knot.

regards
Klaus

-Ursprüngliche Nachricht-
Von: dnsdist mailto:dnsdist-boun...@mailman.powerdns.com Im Auftrag von 
Rais Ahmed via dnsdist
Gesendet: Mittwoch, 23. März 2022 22:02
An: mailto:dnsdist@mailman.powerdns.com
Betreff: [dnsdist] dnsdist[29321]: Marking downstream IP:53 as 'down'

Hi,
Thanks for reply...!

We have configured setMaxUDPOutstanding(65535) and still we are seeing 
backend down, logs are showing frequently as below.

Timeout while waiting for the health check response from backend
192.168.1.1:53
Timeout while waiting for the health check response from backend
192.168.1.2:53

Please have a look at below dnsdist configuration and help us to find 
misconfiguration (16 Listeners & 8+8 backends added as per vCPUs 
available
(2 Socket x 8 Cores):

controlSocket('127.0.0.1:5199')
setKey("")

 Listen addresses
addLocal('192.168.0.1:53', { reusePort=true }) 
addLocal('192.168.0.1:53', { reusePort=true }) 
addLocal('192.168.0.1:53', { reusePort=true }) 
addLocal('192.168.0.1:53', { reusePort=true }) 
addLocal('192.168.0.1:53', { reusePort=true }) 
addLocal('192.168.0.1:53', { reusePort=true }) 
addLocal('192.168.0.1:53', { reusePort=true }) 
addLocal('192.168.0.1:53', { reusePort=true }) 
addLocal('192.168.0.1:53', { reusePort=true }) 
addLocal('192.168.0.1:53', { reusePort=true }) 
addLocal('192.168.0.1:53', { reusePort=true }) 
addLocal('192.168.0.1:53', { reusePort=true }) 
addLocal('192.168.0.1:53', { reusePort=true }) 
addLocal('192.168.0.1:53', { reusePort=true }) 
addLocal('192.168.0.1:53', { reusePort=true }) 
addLocal('192.168.0.1:53', { reusePort=true })

---

Re: [dnsdist] dnsdist[29321]: Marking downstream IP:53 as 'down'

[Klaus Darilion via dnsdist]

Have you tested how many Qps your Backend is capably to handle? First test your 
Backend performance to know how much qps a single backend can handle. I guess 
500k qps might be difficult to achieve with bind. If you need more performance 
switch the Backend to NSD or Knot.

regards
Klaus

> -Ursprüngliche Nachricht-
> Von: dnsdist  Im Auftrag von
> Rais Ahmed via dnsdist
> Gesendet: Mittwoch, 23. März 2022 22:02
> An: dnsdist@mailman.powerdns.com
> Betreff: [dnsdist] dnsdist[29321]: Marking downstream IP:53 as 'down'
> 
> Hi,
> Thanks for reply...!
> 
> We have configured setMaxUDPOutstanding(65535) and still we are seeing
> backend down, logs are showing frequently as below.
> 
> Timeout while waiting for the health check response from backend
> 192.168.1.1:53
> Timeout while waiting for the health check response from backend
> 192.168.1.2:53
> 
> Please have a look at below dnsdist configuration and help us to find
> misconfiguration (16 Listeners & 8+8 backends added as per vCPUs available
> (2 Socket x 8 Cores):
> 
> controlSocket('127.0.0.1:5199')
> setKey("")
> 
>  Listen addresses
> addLocal('192.168.0.1:53', { reusePort=true })
> addLocal('192.168.0.1:53', { reusePort=true })
> addLocal('192.168.0.1:53', { reusePort=true })
> addLocal('192.168.0.1:53', { reusePort=true })
> addLocal('192.168.0.1:53', { reusePort=true })
> addLocal('192.168.0.1:53', { reusePort=true })
> addLocal('192.168.0.1:53', { reusePort=true })
> addLocal('192.168.0.1:53', { reusePort=true })
> addLocal('192.168.0.1:53', { reusePort=true })
> addLocal('192.168.0.1:53', { reusePort=true })
> addLocal('192.168.0.1:53', { reusePort=true })
> addLocal('192.168.0.1:53', { reusePort=true })
> addLocal('192.168.0.1:53', { reusePort=true })
> addLocal('192.168.0.1:53', { reusePort=true })
> addLocal('192.168.0.1:53', { reusePort=true })
> addLocal('192.168.0.1:53', { reusePort=true })
> 
>  Back-end server
> newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5,
> weight=4, qps=4, order=1})
> newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5,
> weight=4, qps=4, order=2})
> newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5,
> weight=4, qps=4, order=3})
> newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5,
> weight=4, qps=4, order=4})
> newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5,
> weight=4, qps=4, order=5})
> newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5,
> weight=4, qps=4, order=6})
> newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5,
> weight=4, qps=4, order=7})
> newServer({address='192.168.1.1', maxCheckFailures=3, checkInterval=5,
> weight=4, qps=4, order=8})
> newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5,
> weight=4, qps=4, order=9})
> newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5,
> weight=4, qps=4, order=10})
> newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5,
> weight=4, qps=4, order=11})
> newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5,
> weight=4, qps=4, order=12})
> newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5,
> weight=4, qps=4, order=13})
> newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5,
> weight=4, qps=4, order=14})
> newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5,
> weight=4, qps=4, order=15})
> newServer({address='192.168.1.2', maxCheckFailures=3, checkInterval=5,
> weight=4, qps=4, order=16})
> 
> setMaxUDPOutstanding(65535)
> 
>  Server Load Balancing Policy
> setServerPolicy(leastOutstanding)
> 
>  Web-server
> webserver('192.168.0.1:8083')
> setWebserverConfig({acl='192.168.0.0/24', password='Secret'})
> 
>  Customers Policy
> customerACLs={'192.168.1.0/24'}
> setACL(customerACLs)
> 
> pc = newPacketCache(30, {maxTTL=86400, minTTL=0,
> temporaryFailureTTL=60, staleTTL=60, dontAge=false})
> getPool(""):setCache(pc)
> 
> setVerboseHealthChecks(true)
> 
> Servers Specs are as below:
> Dnsdist LB Server Specs: 16 vCPUs, 16 GB RAM, Virtio NIC (10G) with 16
> Multiqueues.
> Backend bind9 servers Specs: 16 vCPUs, 16GM RAM, Virtio NIC (10G) with 16
> Multiqueues.
> 
> We are trying to handle 500K qps (will increase hardware specs, If required)
> or with above specs atleast 100K qps.
> 
> 
> Regards,
> Rais
> 
> -Original Message-
> From: dnsdist  On Behalf Of
> dnsdist-requ...@mailman.powerdns.com
> Sent: Wednesday, March 23, 2022 5:00 PM
> To: dnsdist@mailman.powerdns.com
> Subject: dnsdist Digest, Vol 79, Issue 3
> 
> Send dnsdist mailing list submissions to
>   dnsdist@mailman.powerdns.com
> 
> To subscribe or unsubscribe via the World Wide Web, visit
>   https://mailman.powerdns.com/mailman/listinfo/dnsdist
> or, via email, send a message with subject or body 'help' to
>   

[dnsdist] XDP/eBPF blocking (was dnsdist 1.7.0 released)

[Klaus Darilion via dnsdist]

Hi!
> Pierre Grié from Nameshield contributed an XDP program to reply to
> blocked UDP queries with a truncated response directly from the kernel,
> in a similar way to what we were already doing using eBPF socket
> filters. This version adds support for eBPF pinned maps, allowing
> dnsdist to populate the maps using our dynamic blocking mechanism, and
> letting the external XDP program do the actual blocking or response.

How does this work in detail? If example.com is on these lists (filtering or 
truncate response), will it block also www.example.com (and other subdomains) 
or only exactly the name on the list?

Thanks
Klaus
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] dnsdist performance

[Klaus Darilion]

Hi Remi!

Am 11.08.2019 um 18:26 schrieb Remi Gacogne:
> Hi Klaus,
> 
> On 8/10/19 10:30 PM, Klaus Darilion wrote:
>> I had similar results. Starting 4 listening threads and 4 receivers
>> threads (by adding the same backend 4 times) boosted my performance -
>> almost linear.
> 
> Similar in terms of QPS as well? I just tested master on Arch using
> calidns and dumresp as a responder, and I get 55k+ QPS on my 10 year-old
> CPU using no tuning, simply:
> 
> ./dnsdist -C /dev/null -l 127.0.0.1:53 127.0.0.1:54
> 
> So one listening thread and one receiver thread.
> 
> A quick glance at a perf recording shows that more than 76% of the CPU
> time is spent in syscalls, so I'm pretty sure disabling the
> meltdown/spectre mitigations would do a big difference, but it's already
> pretty far from 5-15K QPS.

My results were:
1 VM with 4vCPUs. The VM runs NSD and PDNS/PGSQL. DNSDIST forwards some
zones to NSD, others to PDNS/PGSQL

PDNS with cached responses: 40.000 q/s
PDNS random labels: 9.000 q/s
NSD: 65.000 q/s

1. dnsdist with single "listen" and each backend added only once
dnsdist with cached responses: 40.000 q/s
dnsdist random labels to PDNS: 7.300 q/s
dnsdist random labels to NSD: 40.000 q/s

2. dnsdist with 3 "listen" and each backend added 3 times
dnsdist with cached responses: 100.000 q/s
dnsdist random labels to PDNS: 7.300 q/s
dnsdist random labels to NSD: 60.000 q/s

> If you increase the number of threads you'll need to use ring buffers
> sharding to limit contention, by the way.

Any config recommendation? I currently use a single Cache for both
backends with 1 shard:

cache = newPacketCache(50, 10, 10, 60, 60, true, 1, true, 10)
getPool(""):setCache(cache)
getPool("static"):setCache(cache)


thanks
Klaus
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] pool selection without implicit accept

[Klaus Darilion]

Am 12.07.2019 um 16:08 schrieb Remi Gacogne:
> 
> 
> On 7/12/19 2:52 PM, Klaus Darilion wrote:
>> That does not work. At the moment, if I want to add another domain (ie
>> rate1.com) to the "static" pool I have to check first, if the domain is
>> rate limited or not, and if yes, use a andAction construct to achieve my
>> goal. So, yes it would be possible, the complex to manage.
> 
> You are right, while Pieter's solution would work it increases the
> complexity a bit. Since it's not the first time I would like to be able
> to apply an existing action without stopping the processing, I'm
> considering implementing a ContinueAction() action that would accept an
> existing action as parameter, execute it and return None, regardless of
> the value returned by the existing action.
> 
> That would make it possible to do something like that in a generic way:
> 
> addAction({ 'rate1.com','a.aa' }, ContinueAction(PoolAction("static")))
> 
> I don't think it would be too complicated to implement and since it
> would be self-contained we could even consider it for 1.4.0.
> 
> Any thoughts?

Sounds like a useful hack.

Although I think the otherway round would be better, ie:

addAction({ 'rate1.com','a.aa' }, PoolAction("static") && AllowAction())

with default "continue" for PoolAction, but I guess for such changes it
it a few years to late.

Thanks
Klaus
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] pool selection without implicit accept

[Klaus Darilion]

Hi Peter!

My current config is:

-- Send requests for zones without dynamic features to pool "static" (NSD)
-- For other zones forwarded to default pool (PDNS)
addAction({ 'a.aa' }, PoolAction("static"))


-- Rate limiting for domains
addAction(QNameRule("rate1.com"), AllowAction())
addAction(QNameRule("www.rate1.com"), AllowAction())
addAction(AndRule({makeRule("rate1.com"),NotRule(MaxQPSRule(10))}),
DropAction())

addAction(QNameRule("rate2.com"), AllowAction())
addAction(QNameRule("www.rate2.com"), AllowAction())
addAction(AndRule({makeRule("rate2.com"),NotRule(MaxQPSRule(10))}),
DropAction())




Now I need to add rate1.com to the "static" pool. I would like to have
my rate filtering logic independent of my pool logic and just use:
addAction({ 'rate1.com','a.aa' }, PoolAction("static"))

That does not work. At the moment, if I want to add another domain (ie
rate1.com) to the "static" pool I have to check first, if the domain is
rate limited or not, and if yes, use a andAction construct to achieve my
goal. So, yes it would be possible, the complex to manage.

regards
Klaus

PS: This is not a high priority thing, but I think it might be useful
also for other functions to not imply a final action but continue with
the rules




Am 12.07.2019 um 12:58 schrieb Pieter Lexis:
> Hi Klaus,
> 
> On 7/12/19 10:34 AM, Klaus Darilion wrote:
>> I have a ruleset with severals whitelist (AllowAction) and ratelimit
>> (MaxQPSRule+DropAction).
>>
>> Now, independent of these rules I would like to use different backend
>> pools. But now I have a problem as PoolAction() immediately forwards the
>> request and my blacklist/whitelist rules are not handled anymore.
>>
>> Moving the pool selection after the black/whitelist also does not work
>> as the whiteliste sends immediately to the default pool.
>>
>> Hence, I suggest a PoolAction() without implicit AllowAction, for
>> example PoolActionSetOnly() or PoolActionContinue() to set the pool for
>> a request but continue in the rules processing.
> 
> Perhaps the AndRule[1] operator could help here?
> 
> ```
> rule1 = MaxQPSRule(...)
> rule2 = SomeOtherRule()
> 
> addAction(AndRule{rule1, rule2}, PoolAction(...))
> ```
> 
> We use this in our own setups:
> 
> ```
> allowed_axfr_addresses_rule = makeRule({'192.0.2.1', '2001:DB8::1'})
> -- Deny AXFR from anything but the allowed addresses
> addAction(
>   AndRule({
> OrRule({
>   QTypeRule(DNSQType.AXFR),
>   QTypeRule(DNSQType.IXFR),
> }),
> NotRule(allowed_axfr_addresses_rule)
>   }),
>   RCodeAction(DNSRCode.REFUSED)
> )
> 
> ```
> 
> This way, with the rules correctly ordered, you should be able to
> achieve your goal. If you could provide the mailinglist with your
> current config and a description of what exactly you're trying to do, we
> could be able to judge if what you're asking for is indeed a new feature
> that could be implemented.
> 
> Cheers,
> 
> Pieter
> 
> 1 - https://dnsdist.org/rules-actions.html#AndRule
> 
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] identical metrics

[Klaus Darilion]

Hi Remi!

Am 12.07.2019 um 10:27 schrieb Remi Gacogne:
> Hi Klaus!
> 
> On 7/12/19 10:23 AM, Klaus Darilion wrote:
>> I think this may cause problems with later visualization tools. I think
>> it would be good to add an optional "name" parameter (as with newServer).
> 
> Yes, I agree the current situation could be improved, I have a few ideas
> in mind for the next refactoring of this code.
> In the meantime this exact issue should have been fixed by [1] on master.
> 
> [1]: https://github.com/PowerDNS/pdns/pull/7934

Will this rename the metrics to be unique or will it generate a single
metric with the sum of all submetrics?

Thanks
Klaus
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

[dnsdist] pool selection without implicit accept

[Klaus Darilion]

Hello!

I have a ruleset with severals whitelist (AllowAction) and ratelimit
(MaxQPSRule+DropAction).

Now, independent of these rules I would like to use different backend
pools. But now I have a problem as PoolAction() immediately forwards the
request and my blacklist/whitelist rules are not handled anymore.

Moving the pool selection after the black/whitelist also does not work
as the whiteliste sends immediately to the default pool.

Hence, I suggest a PoolAction() without implicit AllowAction, for
example PoolActionSetOnly() or PoolActionContinue() to set the pool for
a request but continue in the rules processing.

regards
Klaus
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

[dnsdist] identical metrics

[Klaus Darilion]

Hi!

For performance reasons (it helps a lot) I have 3 listeners:
setLocal("0.0.0.0:53", { doTCP=true, reusePort=true, tcpFastOpenSize=100 })
addLocal("0.0.0.0:53", { doTCP=true, reusePort=true, tcpFastOpenSize=100 })
addLocal("0.0.0.0:53", { doTCP=true, reusePort=true, tcpFastOpenSize=100 })


/metrics then uses the same metric name:
dnsdist_frontend_queries{frontend="0.0.0.0:53",proto="udp"} 3459
dnsdist_frontend_queries{frontend="0.0.0.0:53",proto="udp"} 3516
dnsdist_frontend_queries{frontend="0.0.0.0:53",proto="udp"} 3422

I think this may cause problems with later visualization tools. I think
it would be good to add an optional "name" parameter (as with newServer).

regards
Klaus
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

Re: [dnsdist] Number of Actions

[Klaus Darilion]

Hello Bert!

Am 15.04.2019 um 21:23 schrieb bert hubert:
> On Mon, Apr 15, 2019 at 08:39:30PM +0200, Klaus Darilion wrote:
>> Hello!
>>
>> Is there a max number of actions, before there might be performance
>> problems?
> 
> Yes. The design goal is not to have hundreds of rules, but to have fewer,
> more powerful rules.
> 
> For example, in your case, you can do:
> addAction({"lots", "of", "domains"}, DropAction())
> 
> This is then fast, much faster than three separate rules.
> 
> You can also create a SuffixMatchNode and fill it programatically and then
> use a SuffixMatchNodeRule(smn) on it.
> 
> Finally for your case, which I spotted on another list, you may want to look
> at https://dnsdist.org/guides/dynblocks.html which can be a lot of fun.

Do you have an example? The examples on the page you referenced do not
deal with query-domains. Filtering based on response code does not help
as the request already hit the authoritative server. Filtering on client
IP would also filter good queries coming from this client.

Thanks
Klaus


___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist

[dnsdist] Number of Actions

[Klaus Darilion]

Hello!

Is there a max number of actions, before there might be performance
problems?

During a random subdomain attack I would like to "whitelist" all real
subdomains and then rate limit the rest, ie:

-- Allow the following lables without limiting
addAction(QNameRule("www.example.com"), AllowAction())
addAction(QNameRule("mail.example.com"), AllowAction())
... (around 500 more lables)

-- Ratelimiting for all other labels
addAction(AndRule({makeRule("example.com"),NotRule(MaxQPSRule(10))}),
DropAction())

May I experience performance problems when all my queries have to pass
through 500 actions first?


Thanks
Klaus
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist