Re: [dns-operations] differ
>> it occurred to me that it migh tme wise to have a rancid like >> (https://shrubbery.net/rancid/) equivalent for critical domains. >> i.e. to git record changes and warn of radical diffs. >> >> is there any foss tooling in this space? > > Assuming there isn't - yet...- What would you want a tool like this > to do ? Would a simple diff (e.g.: number of deleted lines > X, > assuming one is working with files) be too vague ? Would you want the > granularity to be RRsets ? Have you looked at https://dotat.at/prog/nsdiff/ ? Steinar Haug, AS2116 ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] G root servers unreachable via ICMP(v6)
> DNS speaking, I can query G root servers; at least, that's the most > important. > > However, from several sites, either on IPv4 or IPv6, I cannot ping(6) > them. Is it by design, or it's an issue? > > Side question: even if it was by design, is it a good practice to > completely restrict ICMP(v6)? As others have pointed out, we don't *know* if they completely restrict ICMP(v6). All we can say is that they restrict ICMP Echo Request. Speaking purely for myself - I run a number of name servers, some recursive, some authoritative. The function of these name servers is to answer DNS queries, *not* to answer ICMP(v6) Echo requests. I happen to allow ICMP(v6) Echo requests to these name servers (so they will reply to ping and traceroute) - however, this type of traffic is heavily rate limited (again, because the purpose of these servers is to handle DNS traffic). I feel absolutely zero moral obligation to allow unlimited ping. Steinar Haug, AS 2116 ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
[dns-operations] Resolvers seeing repeated bursts of identical queries
We are receiving a significant amount of query bursts on our resolvers with the following characteristics: - A client IP doing a burst of queries for the same name repeatedly, very quickly. - The query is typically an A query. - A burst often has 50 - 100 queries for the same name within a few milliseconds. - All the queries within one burst have the same DNS query ID (but different IP id and source port number). - The same client IP producing such bursts of identical queries also sends regular queries (one query per name, DNS query IDs vary). Example of (part of) query burst - in this case the client sends bursts of 84 queries within less than 1 ms: 09:24:56.593259 IP 194.19.79.131.58089 > 193.75.75.193.53: 24781+ A? www.jointraining.com. (38) 09:24:56.593283 IP 194.19.79.131.38426 > 193.75.75.193.53: 24781+ A? www.jointraining.com. (38) 09:24:56.593307 IP 194.19.79.131.56931 > 193.75.75.193.53: 24781+ A? www.jointraining.com. (38) 09:24:56.593346 IP 194.19.79.131.42976 > 193.75.75.193.53: 24781+ A? www.jointraining.com. (38) 09:24:56.593350 IP 194.19.79.131.11638 > 193.75.75.193.53: 24781+ A? www.jointraining.com. (38) 09:24:56.593366 IP 194.19.79.131.22476 > 193.75.75.193.53: 24781+ A? www.jointraining.com. (38) ... 09:24:56.594364 IP 194.19.79.131.41548 > 193.75.75.193.53: 24781+ A? www.jointraining.com. (38) followed by another burst of 84 queries in around 1.1 ms: 09:24:56.594416 IP 194.19.79.131.38426 > 193.75.75.193.53: 28221+ A? www.facebook.com. (34) 09:24:56.594475 IP 194.19.79.131.42976 > 193.75.75.193.53: 28221+ A? www.facebook.com. (34) 09:24:56.594501 IP 194.19.79.131.58089 > 193.75.75.193.53: 28221+ A? www.facebook.com. (34) 09:24:56.594560 IP 194.19.79.131.14419 > 193.75.75.193.53: 28221+ A? www.facebook.com. (34) 09:24:56.594561 IP 194.19.79.131.56931 > 193.75.75.193.53: 28221+ A? www.facebook.com. (34) 09:24:56.594562 IP 194.19.79.131.18576 > 193.75.75.193.53: 28221+ A? www.facebook.com. (34) ... 09:24:56.595596 IP 194.19.79.131.41232 > 193.75.75.193.53: 28221+ A? www.facebook.com. (34) I *suspect* the bursts and the regular queries are actually produced by different clients on the inside of a firewall with NAT - but note I don't *know* this is the case. Does anybody know of software / applications that would produce such query bursts? Note that I don't believe the query bursts are caused by L2 loops or similar, because - These problems have lasted for weeks - And they occur for several different (unrelated) customers Steinar Haug, AS2116 ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] Netgear time-g.netgear.com + time-f.netgear.com - flooding....
> > 14:23:58.147601 IP customer.32769 > 212.60.63.246.53: 17710+ A? > time-g.netgear.com. (36) > 14:23:58.147603 IP customer.32769 > 212.60.61.246.53: 17710+ A? > time-g.netgear.com. (36) > 14:23:58.147613 IP customer.32769 > 212.60.63.246.53: 17710+ A? > time-g.netgear.com. (36) > 14:23:58.147613 IP customer.32769 > 212.60.61.246.53: 17710+ A? > time-g.netgear.com. (36) > 14:23:58.147616 IP customer.32769 > 212.60.63.246.53: 17710+ A? > time-g.netgear.com. (36) > 14:23:58.147617 IP customer.32769 > 212.60.61.246.53: 17710+ A? > time-g.netgear.com. (36) > 14:23:58.147618 IP customer.32769 > 212.60.63.246.53: 17710+ A? > time-g.netgear.com. (36) > ... > * Has anybody seen similar situations in their recursives? (and what could > you do about it) We've seen it many times. Haven't normally followed up with customer (not enough of a problem to be worth while). > * Is this a on-device (netgear) issue or is this part of some kind of DoS > attempt? For us it looks like a Netgear issue, not an organized DoS attempt. Steinar Haug, AS2116 ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] How widely implemented are different DNSSEC algorithms?
>> Are there any published numbers estimating how well the various DNSSEC >> algorithms are supported in DNS caches and client software? > > Off the top of my head: https://dnsthought.nlnetlabs.nl/#ecdsa256 > > but 13 in particular feels quite deployed in zones - I know about .cz > but I'm sure there are others. Also by far the most common algorithm for .no with more than 70% of all DNSSEC domains. Steinar Haug, AS2116 ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] DNS Flag Day 2020 will become effective on 2020-10-01
> this isn't a flag day and shouldn't be called that. it cheapens the > term. > > 1232 is a cargo-cult number. we must not revere as holy those things > which fall out of the sky. > > there is a right way to deprecate fragmentation. it would not involve > adding config complexity. > > there is a right way to reach consensus. it's an RFC draft, not a > github repo for the initiated. > > in the testing referenced by the "flagday2020" web page, there was no > significant difference in loss between 1200 and 1400. there will be a > significant difference in truncation and tcp retry. So - as an operator of several recursive name servers and one of the authoritative .no name servers: Are there suitable scripts I can use to analyze data sources (log files, pcap files etc) and get actual *numbers* for truncation and TCP retry? Developing this myself is possible but is *way* down on the priority list. Which means that unless I have a good reason to do otherwise, I will simply follow the defaults of the DNS software providers we use. Which currently says 1232. Steinar Haug, AS2116 ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] s3.amazonaws.com problem?
>> Is anyone else experiencing resolution issues for names in this domain? We >> are seeing a lot of queries of the form: >> CNAME? bvusfvyq.s3.amazonaws.com >> (the label before ".s3" looks random) and a lot of SERVFAIL responses. >> >> Any clues would be appreciated. > > If you believe The Register (I know, I know...) Amazon’s DNS infrastructure > is/was the target of a DDoS attack. > https://www.theregister.co.uk/2019/10/22/aws_dns_ddos/ And Amazon *claims* the problems are resolved: https://status.aws.amazon.com/ "[RESOLVED] Intermittent DNS Resolution Errors Between 10:30 AM and 6:30 PM PDT, we experienced intermittent errors with resolution of some AWS DNS names. Beginning at 5:16 PM, a very small number of specific DNS names experienced a higher error rate. These issues have been resolved." However, the SERVFAIL statistics from our resolvers indicate that the problem is very much ongoing still. Steinar Haug, AS2116 ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] glitch on [ip6|in-addr].arpa?
> No, even small responses receive no answers from the IPv6 addresses > of the C and F roots. Both of the below time out even though I'm > not setting the "DO" bit: > > $ dig -6 +norecur -t soa arpa. @2001:500:2f::f > $ dig -6 +norecur -t soa arpa. @2001:500:2::c > > Looks like an outage from my vantage point. But not universal. Both of these queries work fine from my vantage point in Oslo, Norway. Traceroute shows 2001:500:2f::f as local, while 2001:500:2::c seems to be in Hamburg. Steinar Haug, AS2116 ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] Best Resources for Deep Dive Understanding of DNS
I am seeing a lot of them (9,997) with Transaction ID of 0x04d2. This seems to be something odd (but again I still need to learn a lot more about the decisions implementations make with their queries) but it gives me a feeling of a hard coded request. ... I believe this may be a hard coded query from TP Link routers (only supposition at this point) but it seems logical. We use mostly TP Link routers around the network and behind the 321 query IP Address is a cluster of them and a hand check of the addresses in the list indicates they are TP Link devices as well. I will try set our reference router up in the lab and run a test against it to confirm. Hard coded query ID indeed - 0x04d2 = 1234 :-) Seeing quite a bit of it here too, interspersed with queries for www.tp-link.com with the same query ID - seems to support your theory. Steinar Haug, AS 2116 ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
Re: [dns-operations] Botnets, botnets everywhere
Our current thinking (based on evidence from some of our customers, and also from Nominum's analysis presented at the Warsaw DNS-OARC workship earlier this year) that the majority of these recent query spates are intended as an attack on the domain (e.g. feile.com) or the nameserver hosting it. Once overwhelmed with query traffic, the DNS servers cease responding, or only respond sporadically. Being responsible for the recursive name servers for a large Norwegian ISP, I see these attacks on a more or less daily basis, mostly due to CPEs with DNS proxies open towards the Internet. I am reasonably sure that the attack is on the authoritative name servers, and not on the domains as such. This conclusion is based on the following (which is obviously not *proof*): - Some of the domains have only been registered a few days before an attack starts. - There are obvious similarities in the non-random part of many of these domain names which seems to indicate that they are *generated*, e.g. www.6644qq.com www.6644se.com www.6655pp.com www.6655qq.com www.667788.com www.6688hh.com www.6688pp.com or dafa888567.com dafa888678.com dafa888789.com dafa888cg.com dafa888vd.com Steinar Haug, Nethelp consulting, sth...@nethelp.no ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
Re: [dns-operations] Botnets, botnets everywhere
Although the attack could be done with a botnet or by reflecting traffic off end-user equipment, many of the attacks I have seen involve source IP spoofing. I deduce this by noting that a fairly large percentage of the traffic comes from blocks of IPs that are not currently routed on the open Internet. What I see is typically a mix of obviously spoofed (non-routed) and routed addresses. The routed addresses may also be spoofed - but this is much harder to determine. E.g. the following snippet from my borders, where N indicates non-routed source address and the 195.204.57.* hosts are CPEs with open DNS proxies: 21:55:46.739163 IP 53.48.96.141.51437 195.204.57.162.53: 35936+ A? kvudwxmqder.www.dafa888789.com. (48) 21:55:46.796523 IP 55.163.252.238.27190 195.204.57.164.53: 60924+ A? mtkvc.dafa888567.com. (38) N 21:55:46.850267 IP 102.106.11.104.54844 195.204.57.162.53: 26379+ A? nopqefguvwxyz.dafa888567.com. (46) 21:55:46.863560 IP 64.25.179.88.12684 195.204.57.162.53: 22451+ A? qofrmtalrde.www.dafa888789.com. (48) N 21:55:46.942008 IP 11.217.253.15.4803 195.204.57.164.53: 3837+ A? oxhbpbd.dafa888678.com. (40) 21:55:47.096952 IP 14.75.14.130.10520 195.204.57.162.53: 33038+ A? abpqeftuiwxyz.dafa888678.com. (46) 21:55:47.118716 IP 70.34.188.220.18210 195.204.57.176.53: 56252+ A? oxcfidszqpunuf.dafa888567.com. (47) 21:55:47.161832 IP 41.103.81.228.41650 195.204.57.164.53: 58193+ A? dcyhjqf.www.dafa888789.com. (44) 21:55:47.277108 IP 41.100.100.63.46343 195.204.57.162.53: 15972+ A? abcdrfthiwkyz.dafa888567.com. (46) 21:55:47.343137 IP 53.165.159.154.2278 195.204.57.162.53: 39327+ A? ttvpzgrpncilyhb.dafa888567.com. (48) 21:55:47.369258 IP 68.15.126.166.1222 195.204.57.164.53: 42366+ A? hjghdju.dafa888678.com. (40) N 21:55:47.386443 IP 101.35.159.246.26686 195.204.57.162.53: 62879+ A? j.dafa888567.com. (34) N 21:55:47.388156 IP 21.212.28.24.17856 195.204.57.162.53: 5916+ A? v.dafa888567.com. (34) 21:55:47.415251 IP 18.82.142.16.45236 195.204.57.164.53: 3982+ A? bymbjomwk.dafa888678.com. (42) However - whether the source address is spoofed or not, in the end it still generates the same attack on the authoritative name servers (in this case ns1.haodns.in and ns2.haodns.in, as far as I can see). I wonder the extent to which the end-user equipment is being blamed when it's just routed IPs which are being used. I don't really see a contradiction between CPEs with open DNS proxies and DNS queries from routed IPs. Steinar Haug, Nethelp consulting, sth...@nethelp.no ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
[dns-operations] What's the story on gmail.fr?
According to Google - ns{1,2,3,4}.google.com: gmail.fr. 345600 IN NS ns1.google.com. gmail.fr. 345600 IN NS ns2.google.com. gmail.fr. 345600 IN NS ns3.google.com. gmail.fr. 345600 IN NS ns4.google.com. But according to the name servers for .fr, gmail.fr. 172800 IN NS dns1.emarkmonitor.com. gmail.fr. 172800 IN NS dns2.emarkmonitor.com. From my vantage point in Norway (AS 2116), I can't get any answer at all from dns{1,2}.emarkmonitor.com - thus gmail.fr is for all purposes nonexistent for our customers. I have also tried DNS queries against dns{1,2}.emarkmonitor.com from a few other places outside Norway, and still no replies. Anybody know what the story is here? I came across this while investigating SERVFAILs from our resolvers. Steinar Haug, AS 2116 ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
Re: [dns-operations] What's the story on gmail.fr?
By the way, as far as i know french people use gmail.com in place of gmail.fr. They won't even notice ! ;) Indeed, I've never seen gmail.fr advertised by Google and I'm surprised to learn it is used in Norway. Google Search finds only one link for gmail.fr :-) It may not actually be used. I came across this when investigating SERVFAILs. These days a large number of SERVFAILs are triggered by botnets, and thus it is entirely possible that the queries I found for gmail.fr were also triggered by botnets. Steinar Haug, AS 2116 ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
Re: [dns-operations] PCAP based detector of malicious DNS traffic
In addition to Nick Urbanik's work, which is log file based, we've also provided some tooling to detect the originators and domains in the recent flood of malicious DNS traffic based on PCAP files. From our mailing list post to pdns-users yesterday: Secondly, the botnet mitigation code in Recursor 3.6.0 is holding up well, but we still see A Lot of malicious DNS traffic. To determine exactly which users are attacking your recursor with such traffic, we've enhanced 'dnsscope' (one of our DNS analysis tools) with the --servfail-tree option. This option generates a per-domain suffix list of IP addresses sending servfail-generating traffic. A provisional document for how to benefit from --servfail-tree and use it to configure bulk IP blocking based on ipset can be found on: https://gist.github.com/ahupowerdns/53c9ec191f9b32803392 This also includes links on where to download binary packages of dnsscope. Note by the way that the instructions are not PowerDNS specific, and will also help you protect other nameservers. The output of the tool is, like Nick's work, a list of domain names and additionally the set of IP addresses sending traffic to those domains. Is dnsscope available for other OSes, e.g. FreeBSD? Steinar Haug, AS 2116 ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
Re: [dns-operations] responding to spoofed ANY queries
It would be nice if ANY queries just got thrown away. I can live with the breakage that causes. YMMV. However if there was something that generally blocked or discarded ANY queries, the bad guys would switch to some other QTYPE that can't be blocked without causing significant operational problems. ___ What makes you think they won't? I mean, isn't this a classic mistake of cold war defense modelling, that you assume your enemy will use weapons you can confidently defend against and ignore the ones you suspect you cannot? ANY has good amplification. If its not working, they surely will move to others. Or both. And if it is working they may move to others anyway. The bad guys are *already* using other tools than ANY queries - for instance, I have seen quite a few amplification attacks based on TXT queries. There's nothing new under the sun... Steinar Haug, AS 2116 ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
Re: [dns-operations] DNS ANY requests from Amazon?
I'm seeing a bunch of DNS ANY requests to my authoritative servers with Amazon EC2 source IPs. I guess somebody is now trying to run an amplification attack against Amazon? Highly likely. This is the first time I've seen Amazon targeted this way; are others seeing this (am I just late to the party)? You're just late to the party. This has been going on for months. Steinar Haug, AS 2116 ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
Re: [dns-operations] dotless domains
Surprised no one's brought up http://dk/ as an already existing scenario that doesn't work (try it in various browsers). Bad example. The first *four* browsers I tried (firefox, chrome, safari, and opera on osx) handle this perfectly. http://dk/ doesn't work particularly well on my Win 7 laptop: Opera 12.02 - gets me http://www.dk.no/ Firefox 15.01 - gets me http://www.dk.com/ Chrome 21.0.1180.89 - gets me Oops! Google Chrome could not find dk I think it's safe to say that http://dk/ does not *reliably* work the way it's supposed to work. Nobody on this list should be surprised... Steinar Haug, Nethelp consulting, sth...@nethelp.no ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
Re: [dns-operations] Why would an MTA issue an ANY query instead of an MX query?
Clue appreciated, thanks! One word: qmail. Google qmail dns any query. It would actually be *good* to stop answering the ANY queries if that would force qmail installations do something about this behavior. But I don't have high hopes... Steinar Haug, Nethelp consulting, sth...@nethelp.no ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
Re: [dns-operations] Why would an MTA issue an ANY query instead of an MX query?
One word: qmail. Google qmail dns any query. thinking about or acting against ANY is bad infosec economics. any investment along those lines is wasted, since ANY is merely the low hanging fruit, and an attacker need only switch over to TXT or RRSIG or NSEC to get a similar amplification effect from an authoritative name server, if ANY were widely nonresponsive. Agreed. And along the same lines, limiting EDNS responses to 1460 bytes, as suggested, will block quite a few legitimate replies (not just ANY replies). to that end, vernon schryver and i have been exploring rate limiting in BIND 9. there's a patch available, which i've so far offered only to anyone whose server is currently getting abused. what i'm worried about is that our profile for goodput-vs-badput is wrong headed or too course grained. so far so good. config { // ... rate-limit { responses-per-second 5; window 5; }; }; I'm afraid we may need more control. If my clients are generating a DDoS attack at 20 responses per second, and I limit this to 5 per second - the CC can get the same effect by mobilizing four times as many clients to do the job. On my wishlist, in addition to rate limiting, is also: - Some way of dynamically blackholing clients, based on one or more of -- Rate limit exceeded -- Asking the *same* question (with a large response) repeatedly -- Asking a *specific* question (e.g. ANY isc.org|ripe.net) -- Input from an external system, e.g. via rndc Steinar Haug, Nethelp consulting, sth...@nethelp.no ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
Re: [dns-operations] Why would an MTA issue an ANY query instead of an MX query?
Not supporting ANY queries would also have side effects - simply dropping the query maks the authoritative server appear unresponsive to the recursive server initiating the query. Note that in many cases the server receiving the ANY query is a recursive server, not an authoritative server. For instance, the ISP I work for runs several recursive servers. Those recursive servers are only available to the ISP's customers. Even so, those recursive servers are contributing to DDoS attacks - because so many of the *clients* are either CPEs with a DNS proxy open from the WAN side, or customers' general open recursive servers which use the ISP recursive servers as forwarders. Steinar Haug, Nethelp consulting, sth...@nethelp.no ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs
Re: [dns-operations] Why would an MTA issue an ANY query instead of an MX query?
limiting EDNS responses to 1460 bytes, as suggested [by me], will block quite a few legitimate replies (not just ANY replies). Why? The response will be sent, just with a TC bit, and the client, if it is not lying about its IP address, will retry with TCP. No blocking. Agreed, this should work fine. Muddled thinking on my part. Steinar Haug, Nethelp consulting, sth...@nethelp.no ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations dns-jobs mailing list https://lists.dns-oarc.net/mailman/listinfo/dns-jobs