Re: [Pdns-users] DDoS attack with random A requests causes SQL backend overload
On Fri, Jul 16, 2021 at 12:21 PM Peter van Dijk via Pdns-users < pdns-users@mailman.powerdns.com> wrote: > On Fri, 2021-07-16 at 12:08 +0200, Thomas Mieslinger via Pdns-users > wrote: > > Suggestions from older threads (Klaus Darrilon): > > - Put that zone in a more efficent Backend (he suggested lmdb) > > Good idea. > > > - Put that zone in a more efficent Software (he suggested nsd) and use > > dnsdist to route the traffic to the alternate Software > > Also a good idea. > > Not to mention, you could use dnsdist (fronting a localhost pdns on an alternate port, bound to 127.0.0.1) to match that base domain name and apply a rate limit against it sending queries for that domain to the backend. It's a couple of extra lines of Lua. I've not actually used the rate limiting so I don't have a handy example but it'd be something like: -- Match on .mydomain.com and rate limit it addAction( RegexRule( "\\.mydomain\\.com$" ), QPSAction( your_desired_rate_limit ) ) -- Fall through for everything else addAction( AllRule(), PoolAction( "your_backend_pool" ) ) Lua regexes are a little weird. Definitely note the double escapes in the regex. It doesn't treat the period as an escaped period in RegexRule if you only have one "\" (and you'll want to tear out your hair when it matches itsmydomain.com and blahblahmydomain.com). I'm assuming that the domain under attack is one that you care about, vs a purely garbage domain, in which case you could just block it at the dnsdist level too. > Very old suggestion: > > - Use a firewall uint32 match to lock out queries to the attacked zone. > > Should work, bit more work to manage. > > > Crazy idea: > > - enable DNSSec on that zone > > - setup pdns recursor or similar add delegate the zone to it > > - pdns-recursor should now be able to efficiently calculate the > > NXDOMAINs based on NSEC/NSEC3 information > > Recursor can do that, but it cannot serve the zone to the world. It is > not an Authoritative server. > > So, sadly, this suggestion does not work. > > Kind regards, > -- > Peter van Dijk > PowerDNS.COM BV - https://www.powerdns.com/ > > ___ > Pdns-users mailing list > Pdns-users@mailman.powerdns.com > https://mailman.powerdns.com/mailman/listinfo/pdns-users > ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] DDoS attack with random A requests causes SQL backend overload
On Fri, 2021-07-16 at 12:08 +0200, Thomas Mieslinger via Pdns-users wrote: > Suggestions from older threads (Klaus Darrilon): > - Put that zone in a more efficent Backend (he suggested lmdb) Good idea. > - Put that zone in a more efficent Software (he suggested nsd) and use > dnsdist to route the traffic to the alternate Software Also a good idea. > Very old suggestion: > - Use a firewall uint32 match to lock out queries to the attacked zone. Should work, bit more work to manage. > Crazy idea: > - enable DNSSec on that zone > - setup pdns recursor or similar add delegate the zone to it > - pdns-recursor should now be able to efficiently calculate the > NXDOMAINs based on NSEC/NSEC3 information Recursor can do that, but it cannot serve the zone to the world. It is not an Authoritative server. So, sadly, this suggestion does not work. Kind regards, -- Peter van Dijk PowerDNS.COM BV - https://www.powerdns.com/ ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] DDoS attack with random A requests causes SQL backend overload
Suggestions from older threads (Klaus Darrilon): - Put that zone in a more efficent Backend (he suggested lmdb) - Put that zone in a more efficent Software (he suggested nsd) and use dnsdist to route the traffic to the alternate Software Very old suggestion: - Use a firewall uint32 match to lock out queries to the attacked zone. Crazy idea: - enable DNSSec on that zone - setup pdns recursor or similar add delegate the zone to it - pdns-recursor should now be able to efficiently calculate the NXDOMAINs based on NSEC/NSEC3 information Cheers Am 16.07.21 um 11:33 schrieb David Porter via Pdns-users: Hello, We have received a DDoS attack on our powerdns infrastructure. The DNS requests were all non-existing records in 1 single zone. Eg: ghz2.mydomain.com cdzx.mydomain.ocom hh3r.mydomain.com The result was that the SQL backend was overloaded with these queries and caused some of our servers not to respond to legitimate queries. See here an example from the SQL log: 2021-07-13T14:50:43.459635Z 3061 Reset stmt 2021-07-13T14:50:43.463172Z 3059 Execute SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and name='gzh1.mydomain.com' and domain_id=1280 2021-07-13T14:50:43.463989Z 3059 Reset stmt 2021-07-13T14:50:43.468001Z 3060 Execute SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and name='cdzx.mydomain.com' and domain_id=1280 2021-07-13T14:50:43.468822Z 3060 Reset stmt 2021-07-13T14:50:43.471102Z 3061 Execute SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and name='cvqi.mydomain.com' and domain_id=1280 2021-07-13T14:50:43.472178Z 3061 Reset stmt 2021-07-13T14:50:43.474985Z 3059 Execute SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and name='hh3r.mydomain.com' and domain_id=1280 2021-07-13T14:50:43.475371Z 3059 Reset stmt 2021-07-13T14:50:43.478971Z 3060 Execute SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and name='9jv9.mydomain.com' and domain_id=1280 2021-07-13T14:50:43.479399Z 3060 Reset stmt 2021-07-13T14:50:43.483063Z 3061 Execute SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and name='boxl.mydomain.com' and domain_id=1280 2021-07-13T14:50:43.483457Z 3061 Reset stmt The new zone cache feature is only caching the "domains" table, it's not caching the each record in the backend. Is there any way how we can ensure that powerdns is caching a complete zone in case we are encountering a random generated dns attack on our authorative DNS servers? Thank you, David ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/pdns-users ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/pdns-users
[Pdns-users] DDoS attack with random A requests causes SQL backend overload
Hello, We have received a DDoS attack on our powerdns infrastructure. The DNS requests were all non-existing records in 1 single zone. Eg: ghz2.mydomain.com cdzx.mydomain.ocom hh3r.mydomain.com The result was that the SQL backend was overloaded with these queries and caused some of our servers not to respond to legitimate queries. See here an example from the SQL log: 2021-07-13T14:50:43.459635Z 3061 Reset stmt 2021-07-13T14:50:43.463172Z 3059 Execute SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and name='gzh1.mydomain.com' and domain_id=1280 2021-07-13T14:50:43.463989Z 3059 Reset stmt 2021-07-13T14:50:43.468001Z 3060 Execute SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and name='cdzx.mydomain.com' and domain_id=1280 2021-07-13T14:50:43.468822Z 3060 Reset stmt 2021-07-13T14:50:43.471102Z 3061 Execute SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and name='cvqi.mydomain.com' and domain_id=1280 2021-07-13T14:50:43.472178Z 3061 Reset stmt 2021-07-13T14:50:43.474985Z 3059 Execute SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and name='hh3r.mydomain.com' and domain_id=1280 2021-07-13T14:50:43.475371Z 3059 Reset stmt 2021-07-13T14:50:43.478971Z 3060 Execute SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and name='9jv9.mydomain.com' and domain_id=1280 2021-07-13T14:50:43.479399Z 3060 Reset stmt 2021-07-13T14:50:43.483063Z 3061 Execute SELECT content,ttl,prio,type,domain_id,disabled,name,auth FROM records WHERE disabled=0 and name='boxl.mydomain.com' and domain_id=1280 2021-07-13T14:50:43.483457Z 3061 Reset stmt The new zone cache feature is only caching the "domains" table, it's not caching the each record in the backend. Is there any way how we can ensure that powerdns is caching a complete zone in case we are encountering a random generated dns attack on our authorative DNS servers? Thank you, David ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/pdns-users