Re: [Pdns-users] PDNS Authoritative Server DDOS Protection
Hi, Is there any update here? Could anyone help me on this ? BR, Hamed Haghshenas -Original Message- From: Hamed Haghshenas [mailto:haghshe...@chavoosh.com] Sent: Saturday, July 21, 2018 10:38 AM To: 'pdns-users@mailman.powerdns.com' Subject: RE: [Pdns-users] PDNS Authoritative Server DDOS Protection Hi Bert, Thanks for your solution, I use it same as below: local dbr = dynBlockRulesGroup() dbr:setQueryRate(3, 10, "Exceeded query rate", 60) dbr:setRCodeRate(dnsdist.NXDOMAIN, 3, 10, "Exceeded NXD rate", 60) dbr:setRCodeRate(dnsdist.SERVFAIL, 3, 10, "Exceeded ServFail rate", 60) dbr:setQTypeRate(dnsdist.ANY, 3, 10, "Exceeded ANY rate", 60) dbr:setResponseByteRate(5000, 10, "Exceeded resp BW rate", 60) function maintenance() dbr:apply() end For attacks build by Mausezahn with small Src Address subnet, worked fine and blocked every /32 subnet that reach the query rate . but when use big SRC subnet like /20 it can't manage the queries and CPU rate increase . could you please let me know is there any way to force Dyn blocked function check /24 subnet instead of /32 and, for every /24 SRC subnet, if query rate exceed then block /24 subnet . for example for 10.10.10.0/24, if query rate exist 10 for 10s then block 10.10.10.0/24. BR, Hamed Haghshenas -Original Message- From: Pdns-users [mailto:pdns-users-boun...@mailman.powerdns.com] On Behalf Of bert hubert Sent: Tuesday, July 17, 2018 3:49 PM To: pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] PDNS Authoritative Server DDOS Protection On Tue, Jul 17, 2018 at 03:24:22PM +0430, Hamed Haghshenas wrote: > Could you please let me know how handle these large DDOS attacks? Hi Hamed, Please take a look at https://dnsdist.org/guides/dynblocks.html#dynblockrulesgroup This is specifically meant for the case of many different IP addresses attacking you. Good luck! ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/pdns-users ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Performance issues
> You do around 441 DNS queries/second. A single database connection
> would therefore not be able to support your needs (given DNSSEC).
We are currently running with 3 receiver-threads and 3
distributor-threads. We have tried every combination of 1, 3 and 4 of
each of these parameters without much effect. Running on a single
receiver-thread increases the latency with several milliseconds,
however, the problem seems harder to trigger. Having set the number of
signing-threads to 4 has increased overall performance, but did not have
any effect on the issue.
> Can you enable graphs as described on https://blog.powerdns.com/2014/
> 12/11/powerdns-graphing-as-a-service/ ? This will allow us to see what
> is going on.
They are added as `pdns.wns{1..3}`. It takes some time to get back to
steady-state behavior.
> dnsgram may also have written a file for you with dropped queries, can
> you check if anything shows up there?
The list contains mostly between 6 and 20 retries for somewhere around
16000 domains. Some other domains are between 60 and 120 retries, mostly
and one ANY. It does not hint in the direction of a specific domain
or query type.
On 24/07/2018 14:46, bert hubert wrote:
> On Tue, Jul 24, 2018 at 02:22:08PM +0200, Martijn Reening wrote:
>> We are running PowerDNS 4.1.3 and have tested against MySQL 5.1.73 en
>> PostgreSQL 10.4. It runs on CentOS 6.9, tested with both kernel versions
>> `2.6.32-696.20.1.el6.x86_64` and `4.15.13-x86_64-linode106`.
> Thanks!
>
>>1.32192 millisecond/lookup
>>Retrieved 31554 records, did 1 queries which should have no match
>>Packet cache reports: 0 hits (should be 0) and 0 misses
>
> You do around 441 DNS queries/second. A single database connection would
> therefore not be able to support your needs (given DNSSEC).
>
> Can you enable graphs as described on
> https://blog.powerdns.com/2014/12/11/powerdns-graphing-as-a-service/
> ? This will allow us to see what is going on.
>
>>99.43% of questions answered within 32.00 msec (3.52%)
>>99.52% of questions answered within 64.00 msec (0.09%)
>>99.65% of questions answered within 256.00 msec (0.12%)
>>99.86% of questions answered within 1024.00 msec (0.21%)
>
> So when it works, it is great, it appears.
>
> dnsgram may also have written a file for you with dropped queries, can you
> check if anything shows up there?
>
> Bert
> ___
> Pdns-users mailing list
> Pdns-users@mailman.powerdns.com
> https://mailman.powerdns.com/mailman/listinfo/pdns-users
>
--
Regards,
Martijn Reening
___
Pdns-users mailing list
Pdns-users@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Performance issues
On Tue, Jul 24, 2018 at 02:22:08PM +0200, Martijn Reening wrote: > We are running PowerDNS 4.1.3 and have tested against MySQL 5.1.73 en > PostgreSQL 10.4. It runs on CentOS 6.9, tested with both kernel versions > `2.6.32-696.20.1.el6.x86_64` and `4.15.13-x86_64-linode106`. Thanks! >1.32192 millisecond/lookup >Retrieved 31554 records, did 1 queries which should have no match >Packet cache reports: 0 hits (should be 0) and 0 misses You do around 441 DNS queries/second. A single database connection would therefore not be able to support your needs (given DNSSEC). Can you enable graphs as described on https://blog.powerdns.com/2014/12/11/powerdns-graphing-as-a-service/ ? This will allow us to see what is going on. >99.43% of questions answered within 32.00 msec (3.52%) >99.52% of questions answered within 64.00 msec (0.09%) >99.65% of questions answered within 256.00 msec (0.12%) >99.86% of questions answered within 1024.00 msec (0.21%) So when it works, it is great, it appears. dnsgram may also have written a file for you with dropped queries, can you check if anything shows up there? Bert ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Performance issues
Hello Bert, We are running PowerDNS 4.1.3 and have tested against MySQL 5.1.73 en PostgreSQL 10.4. It runs on CentOS 6.9, tested with both kernel versions `2.6.32-696.20.1.el6.x86_64` and `4.15.13-x86_64-linode106`. Running bench-db for all domains reports: 1.32192 millisecond/lookup Retrieved 31554 records, did 1 queries which should have no match Packet cache reports: 0 hits (should be 0) and 0 misses In the output of dnsgram are many lines like this, especially during the performance drops: 2018-07-05 16:12:12: Auth server dropped too many questions (683 vs 151), diff: 532 I could put the output files somewhere online if needed. dnsscope produces the following output: PCAP contained 948599 correct packets, 10 runts, 0 oversize, 0 non-UDP. Timespan: 0.310556 hours 0 non-DNS UDP, 0 dns decoding errors, 2 bogus packets Ignored fragment packets: 0 Dropped DNS packets based on recursion-desired filter: 0 DNS IPv4: 761357 packets, IPv6: 187242 packets Questions: 518087, answers: 430510 492392 (95.04% of all) queries did not request recursion 21131 answers had recursion desired bit set, but recursion available=0 (for 92 remotes) 87277 queries went unanswered, of which 128 were answered on exact retransmit 71 responses could not be matched to questions 439883 questions requested EDNS processing, do=1: 412804, ad=1: 9, cd=1: 319760 RcodeCount No Error 318934(74.1%) Server Failure 422 (0.1%) Non-Existent domain 14897 (3.5%) Query Refused96257 (22.4%) 0.19% of questions answered within 50 usec (0.19%) 23.38% of questions answered within 100 usec (23.19%) 42.11% of questions answered within 200 usec (18.73%) 48.09% of questions answered within 300 usec (5.98%) 51.58% of questions answered within 400 usec (3.49%) 57.36% of questions answered within 800 usec (5.78%) 59.76% of questions answered within 1000 usec (2.41%) 70.02% of questions answered within 2.00 msec (10.26%) 85.34% of questions answered within 4.00 msec (15.31%) 95.91% of questions answered within 8.00 msec (10.57%) 99.43% of questions answered within 32.00 msec (3.52%) 99.52% of questions answered within 64.00 msec (0.09%) 99.65% of questions answered within 256.00 msec (0.12%) 99.86% of questions answered within 1024.00 msec (0.21%) 127 responses (0.03%) older than 2 seconds Average non-late response time: 4702.69 usec Saw questions from 29140 distinct remotes, answers to 1 Saw 29139 unique remotes asking questions, but not getting RA answers On 24/07/2018 13:58, bert hubert wrote: > On Tue, Jul 24, 2018 at 01:54:53PM +0200, Martijn Reening wrote: >> We have run out of ideas where to look and what to tune. Perhaps anyone >> here could help us further? > > could you tell us what database you run, what version of PowerDNS, what the > output is of 'pdnsutil bench-db', what operating system? Thanks. > > Also, can you start running a tcpdump and feeding it through dnsscope and > dnsgram? > > Also some sample domain names we can query would be nice, maybe we can see a > pattern. > > Thanks. > > > Bert > ___ > Pdns-users mailing list > Pdns-users@mailman.powerdns.com > https://mailman.powerdns.com/mailman/listinfo/pdns-users > -- Regards, Martijn Reening ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Performance issues
On Tue, Jul 24, 2018 at 01:54:53PM +0200, Martijn Reening wrote: > We have run out of ideas where to look and what to tune. Perhaps anyone > here could help us further? could you tell us what database you run, what version of PowerDNS, what the output is of 'pdnsutil bench-db', what operating system? Thanks. Also, can you start running a tcpdump and feeding it through dnsscope and dnsgram? Also some sample domain names we can query would be nice, maybe we can see a pattern. Thanks. Bert ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/pdns-users
[Pdns-users] Performance issues
Hello everyone, We are seeing very vague issues with our PowerDNS setup where certain sequences of requests can cause full queues and dropped queries. Under normal circumstances, the server can handle more than 10 kqueries/sec, but when the bug is triggered, performance drops to 300-400 queries/sec. There seems to be a correlation with the amount of domains that are queried which return REFUSED. DNSSEC makes it easier to trigger this problem, but disabling it does not make it disappear. It is reproducible with both MySQL and PostgreSQL used as a backend, so it doesn't seem to be an issue with the database. The schemas and indices are correct. Our conclusions so far are that there is some kind of bottleneck between PowerDNS and the database, but either are fast enough under normal operation. The slow requests are cached, because a second run within the TTL does not cause slowness. We have already tried tuning these variables, and while it does seem to help, it only masks the real issue. We have run out of ideas where to look and what to tune. Perhaps anyone here could help us further? -- Regards, Martijn Reening ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Supporting ALIAS Records with DNSSEC specially with powerDNS-Admin as managment tool
Hello I think you should use DNAME, like CNAME but for domain. Regards, Baptiste Dupont De : Pdns-users [mailto:pdns-users-boun...@mailman.powerdns.com] De la part de Mohamad F. Barham Envoyé : mardi 24 juillet 2018 09:28 À : Brian Candler; pdns-users@mailman.powerdns.com Objet : Re: [Pdns-users] Supporting ALIAS Records with DNSSEC specially with powerDNS-Admin as managment tool Thanks Brian for your fast response,,, you got exactly what I need (mirror entire domain to another), is there a way to do that? _ From: Brian Candler Sent: Tuesday, July 24, 2018 10:05:41 AM To: Mohamad F. Barham; pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] Supporting ALIAS Records with DNSSEC specially with powerDNS-Admin as managment tool On 24/07/2018 07:21, Mohamad F. Barham wrote: > and I Editted powerDNS-Admin to Support ALIAS, > Now What I want is to make these domains ALIASES > (bzu.ps,birzeit.edu.ps, birzeit.ps) for this domain (birzeit.edu) > where shoud I insert the ALIAS records? In which format? I think you've misunderstood what ALIAS does. ALIAS does not mirror an entire domain. ALIAS is pseudo resource-record, which expands to the dynamic lookup of another resource record. It's typically used at the apex of a zone, where you might like to insert a CNAME but you're not allowed to: CNAME cannot exist alongside any other resource type, including NS and SOA. ; not allowed @NSns1.mydomain.com. @NSns2.mydomain.com. @CNAMEweb1.cloudhost.com. ; solution @NSns1.mydomain.com. @NSns2.mydomain.com. @ALIASweb1.cloudhost.com. When someone looks up an A record for mydomain.com, they will get the A record(s) for web1.cloudhost.com, which are dynamically looked up (via the configured resolver) but returned as if they were stored in the authoritative server. See: https://doc.powerdns.com/authoritative/guides/alias.html HTH, Brian. ~~ The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The University is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt. ~~ _ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Supporting ALIAS Records with DNSSEC specially with powerDNS-Admin as managment tool
Thanks Brian for your fast response,,, you got exactly what I need (mirror entire domain to another), is there a way to do that? From: Brian Candler Sent: Tuesday, July 24, 2018 10:05:41 AM To: Mohamad F. Barham; pdns-users@mailman.powerdns.com Subject: Re: [Pdns-users] Supporting ALIAS Records with DNSSEC specially with powerDNS-Admin as managment tool On 24/07/2018 07:21, Mohamad F. Barham wrote: > and I Editted powerDNS-Admin to Support ALIAS, > Now What I want is to make these domains ALIASES > (bzu.ps,birzeit.edu.ps, birzeit.ps) for this domain (birzeit.edu) > where shoud I insert the ALIAS records? In which format? I think you've misunderstood what ALIAS does. ALIAS does not mirror an entire domain. ALIAS is pseudo resource-record, which expands to the dynamic lookup of another resource record. It's typically used at the apex of a zone, where you might like to insert a CNAME but you're not allowed to: CNAME cannot exist alongside any other resource type, including NS and SOA. ; not allowed @NSns1.mydomain.com. @NSns2.mydomain.com. @CNAMEweb1.cloudhost.com. ; solution @NSns1.mydomain.com. @NSns2.mydomain.com. @ALIASweb1.cloudhost.com. When someone looks up an A record for mydomain.com, they will get the A record(s) for web1.cloudhost.com, which are dynamically looked up (via the configured resolver) but returned as if they were stored in the authoritative server. See: https://doc.powerdns.com/authoritative/guides/alias.html HTH, Brian. ~~ The information contained in this communication is intended solely for the use of the individual or entity to whom it is addressed and others authorized to receive it. It may contain confidential or legally privileged information. If you are not the intended recipient you are hereby notified that any disclosure, copying, distribution or taking any action in reliance on the contents of this information is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by responding to this email and then delete it from your system. The University is neither liable for the proper and complete transmission of the information contained in this communication nor for any delay in its receipt. ~~ ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/pdns-users
Re: [Pdns-users] Supporting ALIAS Records with DNSSEC specially with powerDNS-Admin as managment tool
On 24/07/2018 07:21, Mohamad F. Barham wrote: and I Editted powerDNS-Admin to Support ALIAS, Now What I want is to make these domains ALIASES (bzu.ps,birzeit.edu.ps, birzeit.ps) for this domain (birzeit.edu) where shoud I insert the ALIAS records? In which format? I think you've misunderstood what ALIAS does. ALIAS does not mirror an entire domain. ALIAS is pseudo resource-record, which expands to the dynamic lookup of another resource record. It's typically used at the apex of a zone, where you might like to insert a CNAME but you're not allowed to: CNAME cannot exist alongside any other resource type, including NS and SOA. ; not allowed @ NS ns1.mydomain.com. @ NS ns2.mydomain.com. @ CNAME web1.cloudhost.com. ; solution @ NS ns1.mydomain.com. @ NS ns2.mydomain.com. @ ALIAS web1.cloudhost.com. When someone looks up an A record for mydomain.com, they will get the A record(s) for web1.cloudhost.com, which are dynamically looked up (via the configured resolver) but returned as if they were stored in the authoritative server. See: https://doc.powerdns.com/authoritative/guides/alias.html HTH, Brian. ___ Pdns-users mailing list Pdns-users@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/pdns-users
