On 2016-10-28 2:11 AM, Alejandro Adroher Mellado wrote:
Thanks David,
Yes we have dnsdist in front of the recursor, and some scripts detecting those
SERVFAILS and redirecting all those queries to the abuse pool(dnsdist) in which
we have QPS restrictions. (all of that trying to deal with the SERVFAILs
automatically )
But the fact is when I'm having throttled-outqueries my kernel logs keeps
logging lots of those messages like
Oct 28 09:54:22 rec01 kernel: [22546440.225357] net_ratelimit: 28 callbacks
suppressed
Oct 28 09:54:27 rec01 kernel: [22546445.394524] net_ratelimit: 35 callbacks
suppressed
Oct 28 09:54:33 rec01 kernel: [22546450.915227] net_ratelimit: 30 callbacks
suppressed
Oct 28 09:54:38 rec01 kernel: [22546456.054420] net_ratelimit: 22 callbacks
suppressed
Is this "trusty" ubuntu? Is it logging to the journal or do you use
syslog or rsyslog? I suspect one of these is limiting the amount of logs
it's writing, but I can't say I've seen that exact message before.
Seems both terms related, because If I restart my recursor, those kernel
messages stops for a while.
dnsdist 1.1.0~beta1-1pdns.trusty
pdns-recursor 3.7.3-1
IIRC log-common-errors was enabled by default on 3.7.x - which will
produce a lot of log entries already for these abusive queries. As a
second note, 4.x handles these much better. We used to have queries that
would spawn 50 recursive lookups (the max) but now with 4.x they stop
much quicker.
I'll try enabling full logging for the recursor ... I need information in deep.
Thanks a lot family. Any help will be much appreciated.
Ale.
-----Original Message-----
From: Pdns-users [mailto:[email protected]] On Behalf Of
David
Sent: martes, 11 de octubre de 2016 17:11
To: [email protected]
Subject: Re: [Pdns-users] throttled-outqueries
On 2016-10-11 9:05 AM, Alejandro Adroher Mellado wrote:
Hi all,t
I'm interested in how can I investigate in deep the throttled out queries.
We have an open recursor due to business needs and during the last
weeks we are answering a lot of SERVFAIL for random queries like
web-127.com ..... (all of the forwarded to our abuse pool server)
Your resolver is likely under subdomain reflection attacks. If you look at
"rec_control current-queries" you will probably see lookups similar to
sajasljkdasjdkl.web-127.com sdjksdjlk.web-127.com
etc. Pretty much anything that shows up on:
https://twitter.com/dnsstream will be likely to hit your server as well.
You can either block abusive clients making these queries, or you can filter
them from answering on your recursor. If all you have is powerdns and nothing
like dnsdist in front you can setup forward zones for these pointing to
yourself so that your recursor will respond with a servfail right away instead
of doing the work trying to resolve the name. This will be a never ending cat
and mouse game, though.
From last service reboot I have this stats: (Recursor v 3.7.3)
throttle-entries 390
throttled-out 344055
throttled-outqueries 344055
We received 2.291.374 of SERVFAILS on last 10 days.
The server performance it's fine!
But, we are receiving on syslog a lot of entries like : kernel:
[21099878.651281] net_ratelimit: 153 callbacks suppressed We cannot be sure that
both (SERVFAIL & kernel net_ratelimit) are related.
Have anyone previous experience on this case?
Thanks a lot.
Alejandro
_______________________________________________
Pdns-users mailing list
[email protected]
https://mailman.powerdns.com/mailman/listinfo/pdns-users
_______________________________________________
Pdns-users mailing list
[email protected]
https://mailman.powerdns.com/mailman/listinfo/pdns-users
_______________________________________________
Pdns-users mailing list
[email protected]
https://mailman.powerdns.com/mailman/listinfo/pdns-users
