Re: [dnsdist] Dnsdist dynamic backend selection between AUTH and RECURSOR
Hello Bernd, * Bernd KLAUS via dnsdist [230107 11:01]: > Regarding: > „ My first suggestion would be to not need to do the name based > forwarding by separating the incoming recurosr and auth traffic on ip > address or port“ > > So i should forward all querys to the recursor? I believe the best practice is to have a dedicated IP for auth services, and another dedicated IP for recursive. I'd expect Otto's suggestions to be that ^. Best, -- Chris Hofstaedtler / Deduktiva GmbH (FN 418592 b, HG Wien) www.deduktiva.com / +43 1 353 1707 ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] dnstap unix socket not created
Hi Derrick, * Derrick Lim via dnsdist [200511 06:23]: > Hi, > > I'm trying to use dnstap logging, but the unix socket doesn't seem to > be created. > I've looked through the previous messages here on the mailing list and > checked the systemd private directory as well the specified directory, > but there doesn't seem to be anything there. > > What am I missing? It is expected that the dnstap reader creates the socket - it acts as the "server" from a dnstap PoV. dnsdist will connect to the socket once it exists. Your friendly OS vendor should have packages fstrm_capture, which can act as the dnstap server. JP Mens has a nice intro to dnstap in general: https://jpmens.net/2017/09/11/dns-query-response-logging-with-dnstap/ Chris -- Chris Hofstaedtler / Deduktiva GmbH (FN 418592 b, HG Wien) www.deduktiva.com / +43 1 353 1707 ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] rules for flood protection?
Hi,
On 18/4/20 11:26 pm, Marco Marzo via dnsdist wrote:
> I haven't found much "done" around to protect against popular attacks.
Which attacks? I presume you are only talking about attacks at the DNS
protocol level, but for starters:
* Filter traffic destined to the externally accessible IP's bound to
dnsdist as far upstream as possible; if you are on the receiving end of
a DDoS attack it's better to have the traffic dropped outright before
even making it to the server especially for unused services. Low ports
can be blocked with stateless ACL's on network hardware usually at line
rate. Higher ports may require a stateful firewall though but very often
I see attacks that are only destined to low ports (<1024).
* Make sure that your method of HA/load balancing will not be killed by
a DDoS attack. For this reason I prefer to use ECMP to distribute the
traffic to the IP's used for DNS services. If you are using a load
balancer make sure that it can handle a large amount of
sessions/connections (until the attack may be blocked by other methods).
No point having rules in dnsdist to help block attacks if the traffic
never makes it there in the first place.
* If using iptables on the dnsdist servers, skip stateful processing for
DNS traffic.
* Make use of SO_REUSEPORT if the dnsdist server has the resources to
handle large amounts of traffic. Send some traffic floods to the server
to benchmark the difference with different numbers of listeners to find
the optimum number for your server. You may also find that you will get
better performance by running multiple dnsdist instances, each with
multiple listeners (they can listen to the same IP('s) using
SO_REUSEPORT as well).
* Use the dnsdist packet cache. Also perform benchmarking to find the
optimum settings for your setup (number of shards).
* Craft any rules configured with dnsdist carefully. Adding rules does
have a significant impact. As an example you may have 3 rules each with
a regex - if you can combine those rules into a single regex it may be a
bit more complex from a management point of view but you should find the
additional capacity is worth it.
* Use rules to outright block traffic before processing other rules. As
an example, if you don't require AXFR/IXFR/Notify/Update support
(running a recursor? running authoritative that use other methods to
replicate the zones such as MySQL replication?) that can be blocked
outright either by dropping or returning a REFUSED response. A rule to
do this may look like:
addAction(OrRule({QTypeRule(DNSQType.AXFR), QTypeRule(DNSQType.IXFR),
OpcodeRule(DNSOpcode.Notify), OpcodeRule(DNSOpcode.Update)}),
RCodeAction(DNSRCode.REFUSED))
If you do need support for those query types and the IP's sending those
requests are known, add a nmg and filter based on that nmg so that all
other sources are blocked.
* Make use of dynamic blocks, as an example:
local dbr = dynBlockRulesGroup()
--- Create a dynamic block rule for overall queries/second allowed
dbr:setQueryRate(
The number of queries/second to rate limit at
150,
Set the measurement period over the last 10 seconds
10,
Log the action to syslog
"Exceeded query rate limit",
Add the block for 120 seconds
120
)
--- Create a dynamic block rule to block queries that have resulted in a
NXDOMAIN response
dbr:setRCodeRate(
Match the NXDOMAIN response
DNSRCode.NXDOMAIN,
The number of queries/second to rate limit at
75,
Set the measurement period over the last 10 seconds
10,
Log the action to syslog
"Exceeded NXD response rate",
Add the block for 120 seconds
120
)
--- No dynamic block rule configured for SERVFAIL queries/second from hosts
--- Create a dynamic block rule to block inbound queries/second from
hosts of the ANY type
dbr:setQTypeRate(
Match the ANY query type
DNSQType.ANY,
The number of queries/second to rate limit at
50,
Set the measurement period over the last 10 seconds
10,
Log the action to syslog
"Exceeded ANY rate",
Add the block for 120 seconds
120
)
--- These IP ranges will never be blocked
dbr:excludeRange({
RFC1918 allocations
"10.0.0.0/8",
"172.16.0.0/12",
"192.168.0.0/16"
})
--- Apply the dynamic block rules
function maintenance()
dbr:apply()
end
* Add rules to manage requests of the ANY type.
* In addition to the dynamic blocks, use rules to limit the number of
queries that may be sent from a source. You can whitelist your own
prefixes to ensure that they do not get triggered. A rule can be added
to force clients sending a certain range of QPS to TCP and then if
exceeding that limit drop.
That's just a few things that are important in my mind, of course there
are general recommendations like making sure the number of servers
actually running dnsdist is sized appropriately, the number of backends
is sized appropriately and so on.
Thanks
___
Re: [dnsdist] dnsdist Action dependant on source IP and queried domain
Hi,
On 27/2/20 4:13 pm, Jochen Demmer via dnsdist wrote:
addAction(RegexRule(".internal\\.domain\\.net$"), PoolAction("privatezone"))
But this would need a second selector which would be this NMG thing. How
can I combine that?
As with the previous reply from Jacob Bunk, I would suggest this is the
wrong way to go about things. Ideally you would be running a separate
dnsdist instance for authoritative zones from the recursor (they can be
on the same host with no problems).
Anyway, to answer your question you may combine rules by using "AndRule"
and "OrRule". As an example:
-- Create NMG for source IP's
trustedNMG = newNMG()
trustedNMG:addMask("10.0.0.0/8")
trustedNMG:addMask("172.16.0.0/12")
trustedNMG:addMask("192.168.0.0/16")
trustedNMG:addMask("fd12::/16")
addAction(AndRule({NetmaskGroupRule(trustedNMG),RegexRule(".internal\\.domain\\.net$")}),
PoolAction("privatezone"))
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] SNMP Support
* Brian Sullivan [191122 17:18]: > Hi, > > I am trying to configure the snmpAgent in dnsdist and get the following log > message. > > NET SNMP support is required to use snmpAgent() > > I am running dnsdist 1.4.0-rc5. I assume that NET SNMP support is not > provided in this release. What do I need to do to get NET SNMP support? I believe on some platforms net-snmp is incompatible with the modern libssl that dnsdist needs. If you are on such a platform, you'll need to upgrade the OS first. Chris -- Chris Hofstaedtler / Deduktiva GmbH (FN 418592 b, HG Wien) www.deduktiva.com / +43 1 353 1707 ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] dnsdist 1.4 and Debian Buster
Hi all, As an update to this, I have not been able to figure out or fix the issue as of yet, so I have had to continue using Stretch for now which seems to be working fine. The problem appears to be kernel related, if I use the kernel from Stretch backports the problem occurs. I guess that's not much of a surprise since the Stretch backports kernel is the same release as Buster. I have left one of my servers on Buster in a broken state for testing sometime in future if I get time. If I do find out a fix I'll post a follow up. Thanks ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] DNSDIST 1.3.3-3 from standard debian buster
Hi Frank, On 12/08/2019 10:27 pm, Lichtnau Frank wrote: Hi, I can confirm that we have the same problems under debian buster like Chris call “dnsdist 1.4 and Debian buster”. https://mailman.powerdns.com/pipermail/dnsdist/2019-August/000601.html Thanks for the feedback, this is very helpful to me. For this issue I have not been able to make any progress yet. I have asked my colleagues for help as I am a network admin by trade, something they found that may be potentially related is this kernel bug: https://bugzilla.kernel.org/show_bug.cgi?id=202997 One of my colleagues also deployed a base install of Debian and did some testing with iperf; apparently he could reproduce the issue without involving PowerDNS at all. From the above link he has adjusted a few things but as of yet the problem hasn't been resolved. I am wondering if there is a plan for official dnsdist 1.4 packages on Debian Stretch? I am going to need to use that for now as Buster isn't suitable for production in my environment yet. Thanks ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] dnsdist 1.4 and Debian Buster
Hi Winfried, On 9/08/2019 3:50 pm, ab...@t-ipnet.net wrote: Hi Chris, Maybe I missed that in this thread, but did you try with turning off connection tracking or rising conntrack kernel table size? dmesg might you show wether connection tracking limit was exceeded. Winfried Thanks for the suggestion. By default I raise the conntrack table size as normally this server is using iptables with stateful rules to allow management. I also tried with no iptables rules and the conntrack module not loaded. Thanks ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] dnsdist 1.4 and Debian Buster
Hi Remi, I deployed a new copy of a PowerDNS authoritative server on Debian Buster and ran into a similar problem but with a slight twist. As with dnsdist I use multiple instances of PowerDNS which use different SQL DB's. As with dnsdist, after a period of time I stopped being able to make UDP queries but TCP queries worked fine. The built in web server also works (I guess because its TCP). This happened to all instances on the server at the same time (even an instance which only gets health check queries from a few dnsdist servers). I wanted to see if I could see anything different comparing a working instance with a not working instance on the same server so I restarted one of the instances. When I restart that one instance all started working again as expected. With that in mind it sounds like some sort of limit gets hit. I do raise 'LimitNOFILE' and 'TasksMax' settings in a systemd service.d file for each instance already. As with dnsdist I couldn't find anything in the system logs indicating why. The auth servers have the same configuration and server setup as I was running on Debian Stretch - I deploy a minimal install with puppet installed and it will deploy the rest. On 8/08/2019 9:15 pm, Remi Gacogne wrote: Be careful that dig (the 9.14.4 I have here at least) uses TCP by default for ANY queries so you might need a +notcp to actually test UDP. Thanks, I double checked and it is using UDP for those queries. I'll have to keep digging to see if I can find out why, as of now I don't understand why its happening like this. Thanks ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] dnsdist 1.4 and Debian Buster
Hi Remi, On 8/08/2019 4:46 pm, Remi Gacogne wrote: That's actually one of the most readable configuration I have seen in a while, don't worry ;-) Good to know, still got some work to do on it to make it more friendly though. That's very weird, I don't see anything unusual in your configuration, the backtrace seems to indicate that all threads are working as expected, and I even see some UDP queries being received and forwarded in the strace (albeit very few, you can spot them easily by looking for "recvmsg resumed" with grep). That is strange. When the issue occurs it will receive minimal traffic except from the health checking service that controls the IP's being announced with BGP. I just noticed there is even more strange behavior. I restarted the dnsdist instance and sent traffic for it to reproduce the issue. While it was working I made a 'ANY' query for google.com. One the issue occured I could still send that query and get an answer (both with UDP and TCP). Queries for things that were not in the cache I guess is what stopped working. Would you mind providing a 'lsof -n -p ' while it's stuck? The lsof output is available here: https://gbe0.com/dnsdist/dnsdist_lsof.txt.gz Would you by any chance be able to do a strace when it's stuck, while at the same time sending a few UDP queries to it, ideally with an easily recognizable qname like "why-is-dnsdist-not-responding.to.this." ? The stack trace is available here: https://gbe0.com/dnsdist/dnsdist_strace2.txt.gz During the stack trace I performed 4 requests (in order) - UDP A request for why-is-dnsdist-not-responding.to.this. (not working) - TCP A request for why-is-dnsdist-not-responding.to.this. (working) - UDP ANY request for google.com (working) - UDP A request for google.com (not working) Do you collect some metrics via prometheus? I don't see a carbon export, you might want to send some metrics to our public metronome server [1] for a while, just from one box, we might some spot something there. I'll configure this shortly to the public metronome server. Also, apart from Debian being upgraded from Stretch to Buster and dnsdist from 1.3.x to 1.4.0-beta2, did anything else change in your setup? To be clear, I actually installed a new copy of Debian, I didn't upgrade the existing stretch install. The dnsdist configuration changed slightly: - I originally wrote a lua function for load balancing. Now I am using poolAvailable with rules so I can use a built in method. - The rules were tidied up a bit, previously each dnsdist instance had left over rules that were no longer required - The cache sizes were adjusted Thanks ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] dnsdist 1.4 and Debian Buster
Hi Remi, On 7/08/2019 10:07 pm, Remi Gacogne wrote: I wouldn't mind having a look at your configuration either, since I can see in the metrics that you have some rules. I have put a copy of the configs here, the two 'conf-d' named files live in /etc/dnsdist/dnsdist.conf.d: https://gbe0.com/dnsdist/dnsdist.txt https://gbe0.com/dnsdist/conf-d-global.txt https://gbe0.com/dnsdist/conf-d-nmg.txt Note in the above I replaced my Windows domain with mydomain.com. In the configuration I use PoolAvailable rules with the aim of keeping queries to the backends local if possible. Sorry if its a bit hard to read, I created a puppet module which takes the data from a more human friendly yaml file. Also if it helps, I don't think I mentioned it before but the query load I have on different instances varies, there are a couple of instances that get basically no queries as they are for IP's that are not allocated yet. Those instances don't have a problem at the same time, they continue to work as normal. Thanks ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] dnsdist 1.4 and Debian Buster
Hi Remi, On 7/08/2019 10:07 pm, Remi Gacogne wrote: Are you sure about that? In the strace I see some of the metrics in HTTP exchanges (you might want to change your HTTP password, by the way, sorry about that), and it looks like the counters of received UDP queries in the frontend do increase a bit? I double checked, it definitely doesn't increase the counters from UDP queries from the console using dumpStats. I do see some counters increasing (related to FD's, CPU usage, memory usage) but not query related counters. For the password it is temporary until this works (it is firewalled for outside), thanks for the reminder. I also see some healthchecks failing in the strace, sometimes with a timeout and sometimes even with a connection refused. Several backends in the metrics appear to be down. Is that expected? Yeah, some backends I have taken offline as they are getting OS upgrades I have a couple ideas to know a bit more about what's happening: - running dnsdist in verbose mode, it will log a lot but if you can reproduce the issue in less than a minute it might be bearable ; I will give this a try in the morning. - having a look at the network traffic with tcpdump, for example, to see if the queries are forwarded to the backends and whether these respond. With a tcpdump I can see the queries coming in to dnsdist but they do not get forwarded to the backends at all. The only queries going out to the backends that I can see are for the health checks. I wouldn't mind having a look at your configuration either, since I can see in the metrics that you have some rules. I will also supply this tomorrow. Thanks ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] dnsdist 1.4 and Debian Buster
Hi Remi, On 7/08/2019 5:57 pm, Remi Gacogne wrote: Thanks for reporting this. As far as I can remember it's the first time I hear about this kind of issue. Stracing (strace -f -p ) the process for a few seconds when the problem occurs might be a good start, attaching a debugger (gdb -p ) then getting a backtrace of all threads (thread apply all bt full) would be even better. A strace and backtrace is available here: https://gbe0.com/dnsdist/dnsdist_bt.txt.gz https://gbe0.com/dnsdist/dnsdist_strace.txt.gz Would you also be able to check whether you have some metrics still increasing when the problem arises using the console and 'showBinds()', 'showServers()' and 'dumpStats()'? When the issue occurs the metrics do not change if I send UDP queries; if I sent TCP queries the metrics increase as expected. It's not clear to me from your message whether going back to 1.3.x fixes the issue, could you clarify that? Sorry, I did not try 1.3.x on Debian Buster, I would need to build packages for that as there is a conflict that prevents installation of the Stretch package. The only related issue I can think of involves the network environment being altered under dnsdist's feet, see [1]. The network config shouldn't be changing (IPv6 is used and the link local addresses are stable too). Thanks ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
[dnsdist] dnsdist 1.4 and Debian Buster
Hi all, I have started replacing my dnsdist servers that are running Debian Stretch with the updated Debian Buster release. I came across an issue where dnsdist is running for a period of time it will stop answering any UDP queries until it is restarted. Sending queries either externally to the server or from the server itself results in a timeout. The server is running on bare metal and is receiving around 900-1200 queries/second. There is multiple dnsdist instances running for different purposes, there is 8 instances total. Each instance is bound to a different set of IP's, no IP's are shared between dnsdist instances. For better performance I am using reusePort with 4 listeners normally, for this issue I tried removing the additional listeners and removing the reusePort setting but it still occured. With reusePort enabled the issue occurs after a short period of time usually between 30 and 40 seconds from the period queries start getting sent to it. Without reusePort enabled and a single listen thread the issue seems to occur after around 4-5 hours, when it occurs the symptom seems the same. Once the issue occurs it doesn't seem to recover itself (I waited for around 30 hours) until the dnsdist service is restarted. The dnsdist console and API/web interface continues to work. I can also see dnsdist sending out the health checking queries to the backends and it correctly marks them as up or down. TCP queries also work fine. I had a look through the servers logs but couldn't find anything that could be related to this problem in syslog/messages/dmesg. I do raise the open file limit and the tasksmax limits applied by systemd. The configuration I am using has been migrated from the older 1.3 dnsdist release with a few changes to the configuration to make use of newer features as the config was built a few years ago. I also gave the current master release a go but that also had the same problem. I am wondering if anyone else has had the same issue? If the configuration I am using would be useful I can upload a copy of that. Thanks ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] pool selection without implicit accept
* Remi Gacogne [190712 16:08]:
> That would make it possible to do something like that in a generic way:
>
> addAction({ 'rate1.com','a.aa' }, ContinueAction(PoolAction("static")))
>
> I don't think it would be too complicated to implement and since it
> would be self-contained we could even consider it for 1.4.0.
>
> Any thoughts?
Sounds good to me.
At a future time maybe all currently terminating actions could be
reviewed and possibly be turned into SetXxxAction (ex SetPoolAction)
which then would imply "Continue". Personally I like the iptables design
where there are only very few terminating actions ("targets" in
iptables lingo), and almost everything else implicitly continues.
Writing this in a very vague way as I haven't even done a
preliminary analysis what this would entail...
--
Chris Hofstaedtler / Deduktiva GmbH (FN 418592 b, HG Wien)
www.deduktiva.com / +43 1 353 1707
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] dnsdist package from master repo (2019-05-09)
* Christoph [190513 20:42]: > Hi, > > Remi asked us to test a fix for the DoH segfault crashes by updating via > the master repo > https://github.com/PowerDNS/pdns/issues/7810#issuecomment-491804119 > > but we are having issues getting dnsdist installed from > that repo on Debian Buster > https://github.com/PowerDNS/pdns/issues/7810#issuecomment-491880809 For this specific fix you could also try Debians dnsdist from experimental, version 1.4.0~alpha2-2, which has the #7814 PR patched in. -- Chris Hofstaedtler / Deduktiva GmbH (FN 418592 b, HG Wien) www.deduktiva.com / +43 1 353 1707 ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] dnsdist package from master repo (2019-05-09)
* Christoph [190513 20:42]: > The following packages have unmet dependencies: > dnsdist : Depends: libprotobuf17 but it is not going to be installed > Depends: libre2-5 (>= 20131024+dfsg) but it is not going to be installed > Depends: libsnmp30 (>= 5.7.3+dfsg) but it is not going to be installed > Depends: libstdc++6 (>= 6) but it is not going to be installed > E: Unable to correct problems, you have held broken packages. > > Is anyone else using today's master builds? In the buster development cycle there was a gcc issue that affected a number of C++ programs, including dnsdist (and pdns-recursor). This was "solved" in Debian by having libstdc++6 break the older versions, but that clearly breaks the very reasonable special versioning of the repo.powerdns.com master-branch packages. This is tracked as https://github.com/PowerDNS/pdns/issues/7781 Cheers, Chris -- Chris Hofstaedtler / Deduktiva GmbH (FN 418592 b, HG Wien) www.deduktiva.com / +43 1 353 1707 ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] dnsdist and protobuf support for DnstapLogAction
* Ambauen Daniel (ID NET) [180628 12:11]: > Hello Daniel, > > Thank you for your explanation and clarification about the two dnstap modes. > Do I understand you correct, if I will use the dnstab tools from > http://dnstap.info/Source/, > then I have to build dnsdist with FSTRM support? Yes. The dnstap-over-raw option is a dnsdist extension which is not supported by any of the official tools. -- Chris Hofstaedtler / Deduktiva GmbH (FN 418592 b, HG Wien) www.deduktiva.com / +43 1 353 1707 ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] dnsdist single point of failure
Hi, On 16/10/2017 12:08 PM, long...@viettel.com.vn wrote: Now I want to use dnsdist to distribute DNS request onto those 2 DNS servers. Wouldn’t that make dnsdist the new SPoF? For my situation, I have set up the following: Each location has 3 dnsdist servers. For recursive DNS I have 2 IPv4 IP's and 2 IPv6 IP's bound on loopback on the dnsdist servers. Each dnsdist server runs ExaBGP with a health check and announces the 4 IP's bound to loopback to the route servers which are then distributed into the network. The same 4 IP's are announced from each location (anycast), there is IPSEC tunnels with BGP running over those between the locations as well. The route servers have BGP add path enabled as well as our other network devices that talk BGP to the route servers, so the route servers advertise the 3 available paths locally. This takes care of balancing the load with ECMP between the 3 paths as well as a bonus (no stateful load balancer to worry about). This removes any single point of failure, infact all 3 dnsdist servers can be entirely offline in a location and routing will take care of making the traffic be directed to the next closest available location transparently. ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] server policy to failover to another pool
Hi On 5/10/2017 5:05 AM, Justin Valentini wrote: Hi, I'm trying to create a Lua server policy to use the least outstanding policy on one pool but send traffic to another pool if none of the servers in the first pool are up. I don't have a solution for the way that you want to do this specifically, but I have a lua policy you can use and with some changes to your setup this can be achieved. 1. Use the orderedLeastOutstanding policy from here: https://github.com/sysadminblog/dnsdist-configs/blob/master/orderedLeastOutstanding.lua 2. Only use the primary pool, add your primary servers as order 1 and the secondary servers as order 2. Queries will only be directed to the servers with the order "1" normally, if they are all marked as down then they will be directed to the servers with the order "2". ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] Round-robin inside preferred pool with another pool as backup
Hi, - How would we achieve the load balancing behavior above? Will we have to write our own load balancing code in Lua, or is there some existing configurations we can use? I wrote a lua script to do this already: https://github.com/sysadminblog/dnsdist-configs/blob/master/orderedLeastOutstanding.lua So with the above script, as an example you have the following setup: AZ 1: server1 server2 server3 AZ 2: server4 server5 server6 Using that script, as an example you could do this: On the dnsdist instance(s) in AZ 1, you would add the server1/2/3 backends with a priority of 1. You would add server4/5/6 with a priority of 2. On the dnsdist instance(s) in AZ 2 you would add the server4/5/6 backends with a priority of 1, server1/2/3 backends with a priority of 2. Normal behavior would result in the requests being load balanced to the servers with a priority of 1, if those three servers are down then the requests would then be load balanced to the servers with a priority of 2. ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] Multiple IP Addresses - Different Pools
On 19/04/2017 12:17 PM, Daniel Oakes wrote:
How do I get requests to IP1 to poolA and IP2 to poolB?
Hi,
If IP1 is 1.1.1.1 and IP2 is 1.1.1.2:
poolA = newNMG()
poolA:addMask("1.1.1.1/32")
poolB = newNMG()
poolB:addMask("1.1.1.2/32")
addAction(NetmaskGroupRule(poolA, false), PoolAction("poolA"))
addAction(NetmaskGroupRule(poolB, false), PoolAction("poolB"))
Obvioulsy if you have more IP's for poolA or poolB you can add further
IP's as needed, eg.:
poolA = newNMG()
poolA:addMask("1.1.1.1/32")
poolA:addMask("2001:DB8::1/128")
poolA:addMask("1.1.1.5/32")
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] leastOutstanding - With priority
Hi, On 5/01/2017 10:15 PM, Remi Gacogne wrote: You should be able to write a custom load-balancing policy in Lua quite easily, the Lua policy function gets the list of servers in the selected pool, and from the server object you can get its status (up or down), order and weight. You can easily select all the available servers of order 1 and pick one at random. If there is no such server available, pick one of order 10. It looks like this is what I have to do. I am new to Lua, so there is more than likely a more efficient way to do this than I have done but it looks like I have it working now. If anyone else is interested in doing it I put the code on GitHub, feel free to use it as a template/example or hack away at it: https://github.com/sysadminblog/dnsdist-configs/blob/master/orderedLeastOutstanding.lua If anyone has a look and has any comments on what could be done better I am happy to learn :) ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
Re: [dnsdist] leastOutstanding - With priority
On 5/01/2017 5:51 PM, Daniel Stirnimann wrote: Hi Chris, - Is it possible to tweak the load balancing policy to take the order into account to do this instead? Why not firstAvailable which takes order into account? . dns1 - active dns2 - standby unless dns1 offline or QPS limit reached dns-slave - standby unless both dns1 and dns2 are offline or they reached their QPS limit Daniel Hi Daniel, Ideally, I want dns1 and dns2 to take queries always if available and never the dns-slave server unless there is no other option as it will be serving old data. It would be nice if I could do it in a distributed fashion (distributing queries to all up DNS servers EXCEPT the slave) due to the number of incoming queries which is nearing the performance levels of a single DNS server on occasion. I am planning to add an extra pair of servers to the pool so it would be nice if the queries were distributed over all 4 of them. I also plan to add other DNS servers from different regions into the same pool with a cost of between the dns1/dns2 servers and dns-slave so that they can also be used in the case of dns1/dns2 being down. Thanks ___ dnsdist mailing list dnsdist@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/dnsdist
[dnsdist] leastOutstanding - With priority
Hello,
I am using dnsdist now with great success, except for one thing that I
am trying to figure out if its possible to do within dnsdist.
My pools have 3 backend servers configured. I am using the server policy
"leastOutstanding". Before I moved to dnsdist my DNS servers were setup
like this:
dns1 - Active
dns2 - Active
dns-slave - Standby
The standby server has lagged MySQL replication, so that if there is a
failure with the DB records (eg. someone runs a badly formed query that
updates all records/deletes all records) it gives us time to stop
replication, failover everything (I was using BGP to manage this, the
routes were withdrawn if certain things went missing from the active DNS
servers and the dns-slave route with the higher metric would take over)
to the slave server which doesn't have the changes yet and give us a bit
of time to recover everything without causing a large outage. I found
this was the easiest way for us to scale out our PowerDNS instances and
recover from potential issues like this quickly (~3M authoratative zones).
What I was hoping to achieve is to be able to keep this same setup with
dnsdist. It would be nice if there was a way to use the order parameter
like this:
newServer({address="192.168.90.1:5356", name="dns1.au-internal",
pool="recursor", checkType="A", checkName="a.root-servers.net.",
mustResolve=true, tcpRecvTimeout=10, tcpSendTimeout=10, retries=5,
useClientSubnet=true, order=1})
newServer({address="192.168.90.2:5356", name="dns2.au-internal",
pool="recursor", checkType="A", checkName="a.root-servers.net.",
mustResolve=true, tcpRecvTimeout=10, tcpSendTimeout=10, retries=5,
useClientSubnet=true, order=1})
newServer({address="192.168.90.3:5356", name="dns-slave.au-internal",
pool="recursor", checkType="A", checkName="b.root-servers.net.",
mustResolve=true, tcpRecvTimeout=10, tcpSendTimeout=10, retries=5,
useClientSubnet=true, order=10})
And make all queries only use the servers dns1.au-internal and
dns2.au-internal UNLESS they are both not working, then use the higher
ordered server.
I can do this other ways (outside of dnsdist) but it would be nice if
its all handled in dnsdist, so I was hoping one of these would be possible:
- Is it possible to direct all queries to a different pool if there are
no available servers in another pool? Keep in mind I cannot use a
default pool as I direct queries to the appropriate pool based on the
destination address (so that I only have to run a single instance of
dnsdist for my various recursor/authoratative servers).
- Is it possible to tweak the load balancing policy to take the order
into account to do this instead?
I had a play with the lua scripting but couldn't find a way to keep both
the leastOutstanding policy + failover working.
Thanks
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist
[dnsdist] Handling auth and recursive queries
Hi all,
I am looking at using dnsdist in my environment.
Currently my environment in each location is:
3 physical servers
Each physical server runs 3 different PowerDNS auth instances (one is
for our own domains, one is for customers with shared hosting and one is
for customers with only dns hosting).
Each physical server also runs a PowerDNS recursor instance.
I currently manage distributing traffic using BGP - load balancing is
taken care of by equal cost multipath and failover is provided by
running a health check to ensure the servers are up and withdrawing
routes if they are down.
This all works great but I would like to start using dnsdist. The setup
I am planning for is:
3 physical servers running esxi
Each physical server running 3 VM's - dnsdist, PowerDNS auth and
PowerDNS recursor
dnsdist has the public IP's on it, use BGP to manage load
balancing/failover for traffic destined to dnsdist.
dnsdist to manage the load balancing/failover for traffic destined to
PowerDNS auth and caching.
What I would like to to do is run an instance of dnsdist with a config
like this:
newServer({address="10.254.1.10", pool="dnscache"})
newServer({address="10.254.1.11", pool="dnscache"})
newServer({address="10.254.1.12", pool="dnscache"})
newServer({address="10.254.1.20", pool="dnsauth-internal"})
newServer({address="10.254.1.21", pool="dnsauth-internal"})
newServer({address="10.254.1.22", pool="dnsauth-internal"})
newServer({address="10.254.1.30", pool="dnsauth-shared"})
newServer({address="10.254.1.31", pool="dnsauth-shared"})
newServer({address="10.254.1.32", pool="dnsauth-shared"})
newServer({address="10.254.1.40", pool="dnsauth-dnshosting"})
newServer({address="10.254.1.41", pool="dnsauth-dnshosting"})
newServer({address="10.254.1.42", pool="dnsauth-dnshosting"})
The part I am stuck on is it does not appear to be possible to direct
queries to certain IP's to certain pools. As an example, my caching
resolver IP's are 10.254.1.1, 10.254.1.2. I use addLocal like this:
addLocal("10.254.1.1:53")
addLocal("10.254.1.2:53")
I want to direct all queries destined to those two IP's to the dnscache
pool. I want to do the same thing for the other "addLocal" IP's as well,
queries to the IP's for our own domains should go to pool
dnsauth-internal, queries for the IP's for shared hosting should go to
pool dnsauth-shared etc. Is this possible with dnsdist? I can see how I
can do it based on filtering the domains but at the scale I am using
this it isn't really possible for me, the dnsauth-shared pool for
example has over 2M domains, dnsauth-dnshosting has over 4M domains and
there are very frequent changes to the domains for these.
If not my other option is running 4 seperate dnsdist instances, but I
wanted to try and avoid that if possible to keep things simple.
Thanks
___
dnsdist mailing list
dnsdist@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/dnsdist
