Re: [dns-operations] K-root in CN leaking outside of CN
On Tue, Nov 9, 2021 at 12:39 AM Magnus Sandberg wrote: > > > As of this complexity (both on general network level and the GFW), I > don't think of local DNS root instances in the way that an instance can > be "country local". > I don't see Internet routing and BGP as a binary thing at network level. > Of cause the routing decision in a single router has to be "binary" to > select next-hop, but on a larger scale you can't predict exact what will > happen with your outgoing packets, as Liman wrote. > Agreed that this is a complex beast :) and given it is complicated to predict exactly what will happen with outgoing packets, being able to verify expectations from external vantage points is at least a good way to confirm that things are operating as expected. I don't mean to have a solution and this is probably a decently difficult problem to solve, but I think it would go a long way. Manu > > Regards, > // mem > > > > Den 2021-11-09 kl. 08:23, skrev Davey Song: > > AFAIK, the root server instances in China are not expected to serve > queries > > outside of China. They are called local Root instances when they are > > introduced. > > > > It is true as Liman said no one wishes to inflict problems on clients > > outside China. > > There are must be a network error I think which allows resolvers out of > > China to reach it. > > > > Network errors always happen, so the old issues will happen again. Sad. > > > > Davey > > > > > > On Mon, 8 Nov 2021 at 16:15, Anand Buddhdev wrote: > > > >> Hi Davey, Manu, > >> > >> The server we operate in Guangzhou was indeed reachable from outside > >> China. This is not the intention, of course. On Saturday, when we got > >> notification about this, we withdrew the prefix from the server, and we > >> are communicating with the host to solve this. > >> > >> Many people have already said this, but I'd like to make it clear that > >> the K-root server was NOT emitting false responses for Facebook and > >> WhatsApp. The responses were being modified by something between the > >> server and its clients. > >> > >> Regards, > >> Anand Buddhdev > >> RIPE NCC > >> > >> On 08/11/2021 08:45, Davey Song wrote: > >> > >>> If it is urgent, I suggest the K root operator withdraw the route of > the > >>> instance in Guangzhou immediately. > > ___ > dns-operations mailing list > dns-operations@lists.dns-oarc.net > https://lists.dns-oarc.net/mailman/listinfo/dns-operations > ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] K-root in CN leaking outside of CN
Hi Davey, On Mon, Nov 8, 2021 at 11:30 PM Davey Song wrote: > AFAIK, the root server instances in China are not expected to serve > queries > outside of China. > agreed > They are called local Root instances when they are introduced. > Yeah, so it seems there is a mixture of Global and Local in China currently, but I would bet the Local/Global terms as defined in https://root-servers.org/faq/ don't apply in term of geopolitics. Currently, it seems there is both Local and Global, the affected server was actually "Local" AFAICT: ``` # https://gist.github.com/chantra/db90d97ebe3936742158fb57b5dd3221 ~/root_servers.py --country CN --site-type Global --site-type Local F: Beijing (Local) Chongqing (Local) Hangzhou (Local) Xining Caojiabu (Local) I: Beijing (Global) Hong Kong (Global) J: Beijing (Global) Hangzhou (Global) K: Beijing (Local) Guangzhou (Local) << this one Guiyang (Global) L: Beijing (Global) Beijing (Global) Beijing (Global) Beijing (Global) Haikou (Global) Shanghai (Global) Wuhan (Global) Wuhan (Global) Xining (Global) Xining (Global) Zhengzhou (Global) Zhengzhou (Global) ``` > > It is true as Liman said no one wishes to inflict problems on clients > outside China. > Understood and assumed so. > There are must be a network error I think which allows resolvers out of > China to reach it. > > Network errors always happen, so the old issues will happen again. Sad. > As you said (and multiple of us have said on this thread), there was and there very likely will be again issues of this type. But that does not mean that we should take that fate for granted and having the right monitoring in place to avoid similar issue (or at least prolonged issues) in the future will be likely go a long way rather than having some anecdotal reports. Manu > Davey > > > On Mon, 8 Nov 2021 at 16:15, Anand Buddhdev wrote: > >> Hi Davey, Manu, >> >> The server we operate in Guangzhou was indeed reachable from outside >> China. This is not the intention, of course. On Saturday, when we got >> notification about this, we withdrew the prefix from the server, and we >> are communicating with the host to solve this. >> >> Many people have already said this, but I'd like to make it clear that >> the K-root server was NOT emitting false responses for Facebook and >> WhatsApp. The responses were being modified by something between the >> server and its clients. >> >> Regards, >> Anand Buddhdev >> RIPE NCC >> >> On 08/11/2021 08:45, Davey Song wrote: >> >> > If it is urgent, I suggest the K root operator withdraw the route of the >> > instance in Guangzhou immediately. >> ___ >> dns-operations mailing list >> dns-operations@lists.dns-oarc.net >> https://lists.dns-oarc.net/mailman/listinfo/dns-operations >> > ___ > dns-operations mailing list > dns-operations@lists.dns-oarc.net > https://lists.dns-oarc.net/mailman/listinfo/dns-operations > ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] K-root in CN leaking outside of CN
Hi all, This comment is my personal views and has nothing to do with Netnod or I-root. My take is that the Great Firewall is a complex thing that acts different depending on situation or protocol in use. For DNS/port 53 I see the GFW more as an IDS (Intrusion Detection System) listening on traffic and acting in ways we only see the result of from time to time. The IDS thing can be anywhere in the network, at the borders (probably not), at core locations at ISP networks or close to the eyeballs. For other protocols, like SSH or your favorite VPN, the GFW probably acts more as a classic "brick" firewall with an inside and outside interface. If the GFW do things with BGP, I don't know. So, I don't see the Great Firewall as firewall as the word let us think. I see it as a complex system made of many different parts and sometimes we notice it when the "impersonation" affects the wrong "audience". As of this complexity (both on general network level and the GFW), I don't think of local DNS root instances in the way that an instance can be "country local". I don't see Internet routing and BGP as a binary thing at network level. Of cause the routing decision in a single router has to be "binary" to select next-hop, but on a larger scale you can't predict exact what will happen with your outgoing packets, as Liman wrote. Regards, // mem Den 2021-11-09 kl. 08:23, skrev Davey Song: AFAIK, the root server instances in China are not expected to serve queries outside of China. They are called local Root instances when they are introduced. It is true as Liman said no one wishes to inflict problems on clients outside China. There are must be a network error I think which allows resolvers out of China to reach it. Network errors always happen, so the old issues will happen again. Sad. Davey On Mon, 8 Nov 2021 at 16:15, Anand Buddhdev wrote: Hi Davey, Manu, The server we operate in Guangzhou was indeed reachable from outside China. This is not the intention, of course. On Saturday, when we got notification about this, we withdrew the prefix from the server, and we are communicating with the host to solve this. Many people have already said this, but I'd like to make it clear that the K-root server was NOT emitting false responses for Facebook and WhatsApp. The responses were being modified by something between the server and its clients. Regards, Anand Buddhdev RIPE NCC On 08/11/2021 08:45, Davey Song wrote: If it is urgent, I suggest the K root operator withdraw the route of the instance in Guangzhou immediately. ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] K-root in CN leaking outside of CN
AFAIK, the root server instances in China are not expected to serve queries outside of China. They are called local Root instances when they are introduced. It is true as Liman said no one wishes to inflict problems on clients outside China. There are must be a network error I think which allows resolvers out of China to reach it. Network errors always happen, so the old issues will happen again. Sad. Davey On Mon, 8 Nov 2021 at 16:15, Anand Buddhdev wrote: > Hi Davey, Manu, > > The server we operate in Guangzhou was indeed reachable from outside > China. This is not the intention, of course. On Saturday, when we got > notification about this, we withdrew the prefix from the server, and we > are communicating with the host to solve this. > > Many people have already said this, but I'd like to make it clear that > the K-root server was NOT emitting false responses for Facebook and > WhatsApp. The responses were being modified by something between the > server and its clients. > > Regards, > Anand Buddhdev > RIPE NCC > > On 08/11/2021 08:45, Davey Song wrote: > > > If it is urgent, I suggest the K root operator withdraw the route of the > > instance in Guangzhou immediately. > ___ > dns-operations mailing list > dns-operations@lists.dns-oarc.net > https://lists.dns-oarc.net/mailman/listinfo/dns-operations > ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] K-root in CN leaking outside of CN
Thanks Liman,
I really appreciate this bit of history/context.
> First, just to re-iterate: I-root servers operated by Netnod responds to
> all DNS queries we can handle, as we receive them, with unaltered
> answers from the true root zone. Period. We are, at the moment, not
> aware of any servers outside of our control operating on I-roots
> IP-addresses.
>
> What happens to the DNS packets beyond the first upstream router is at
> best difficult, and in many cases impossible, for us to control, though.
>
Yes, I am also going to re-iterate that this was not my intent to say that
${letter}.root was modifying those answers. But rather that a route leak
was apparently happening and content changed on the way by saying:
> How do we ensure that those are not advertised outside of China so DNS
answers are not poisoned by the GFW?
>
> As Ray Bellis notes, we had a similar incident with the I-root node in
> Beijing back in 2010. It was fixed blindingly fast and with profuse
> apologies when we reported it to our site host. My experience is that
> Chinese authorities have no wish to inflict problems on clients outside
> China, and that whatever impersonation/leakage happens is indeed due to
> configuration errors on networking equipment.
Also very much agreeing on this, and this was also my impression:
> I don't believe this specific leak I am seeing is malicious, but rather
is just a misconfiguration and I really wonder how this could be
prevented/addressed early on.
>
>
There is no way to guarantee that any one ISP (inside China or not) does
> what you expect and hope with your BGP announcements and the traffic
> going to/from any server of yours (DNS root or other). Specifically, I
> expect that a country with more than a billion citizens has a network
> complexity of certain scale, which, in combination with the intricate
> large scale traffic filters, makes "playing" with NO_EXPORT even
> trickier than normal. Life with anycast is a constant challenge to
> deploy the right number of instances at the right points in topology in
> order to make the right thing happen given an existing budget.
>
Agreed this is a tricky problem and as we see, history repeats itself, old
problems become new again, mistakes will always happen, and it will
probably be very difficult to avoid them moving forward, but if we could
collectively come up with a way to monitor and detect such issues we will
probably be in a better spot to act faster.
>
> If you see any signs of problems with i.root-servers.net, please report
> them without delay to . Every such report is of great
> value to us, as it helps us understand what our service looks like to
> you. These observations are important fixpoints in our continuous
> efforts to improve our service.
>
I would definitely do it. It was very much luck though. The probabilities
of ISP needing to go back to root to the impacted domains and choosing the
affected one would have been pretty low. In this case, a particular ISP
being impacted and having user reports, managing to reproduce it and
eventually this got to us through partner channels which we ended up
looking into and here I am bringing this up here so this does not get fixed
in a silo but hopefully we can learn and be better prepared to detect those
issues in the future. This would be beneficial to all letters and their
users IMO.
>
> And finally to each and every one of you:
>
> Please turn on validation in your resolvers and sign your zones. DNSSEC
> is your friend.
>
Thanks Liman for the reminder. I appreciate you not making this the main
point of your reply and kept on topic.
Manu
>
> Best regards,
> /Liman
>hostmas...@i.root-servers.net
>
> #--
> # Lars-Johan Liman, M.Sc. ! E-mail: li...@netnod.se
> # Senior Systems Specialist ! Tel: +46 8 - 562 860 12
> # Netnod AB, Stockholm ! http://www.netnod.se/
> #--
> ___
> dns-operations mailing list
> dns-operations@lists.dns-oarc.net
> https://lists.dns-oarc.net/mailman/listinfo/dns-operations
>
___
dns-operations mailing list
dns-operations@lists.dns-oarc.net
https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] K-root in CN leaking outside of CN
Thanks Davey for the suggestions and Anand (and team) for withdrawing the route. > As for this specific problem, we have reached out to both the AS that is accepting the leak and RIPE NCC as we identified the issue, provided the ISP possible workaround in the meantime. When we believed we identified what the issue was, this is the first thing we did. On Mon, Nov 8, 2021 at 12:18 AM Anand Buddhdev wrote: > > Many people have already said this, but I'd like to make it clear that > the K-root server was NOT emitting false responses for Facebook and > WhatsApp. The responses were being modified by something between the > server and its clients. > Yes, I would also like to point out that this is not what I was hinting at. What I believed happened was that the route were advertised by mistake (tell me about it given some recent events, this is still very fresh to me :) ) and modified along the way. I tried to convey that in those sentences in my original email, I am sorry if it was not clear enough, or made people believe I was saying that the k-root servers were misbehaving. > How do we ensure that those are not advertised outside of China so DNS answers are not poisoned by the GFW? > I don't believe this specific leak I am seeing is malicious, but rather is just a misconfiguration and I really wonder how this could be prevented/addressed early on. Also, I would like to point out that I did not mean to make this a FB/WA problem. Those names are just examples that highlighted the issue. Thanks, Manu > > Regards, > Anand Buddhdev > RIPE NCC > > On 08/11/2021 08:45, Davey Song wrote: > > > If it is urgent, I suggest the K root operator withdraw the route of the > > instance in Guangzhou immediately. > ___ > dns-operations mailing list > dns-operations@lists.dns-oarc.net > https://lists.dns-oarc.net/mailman/listinfo/dns-operations > ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] K-root in CN leaking outside of CN
All, First, Manu, thanks for noticing the problem and reporting it. Since i.root-servers.net was "dragged up" in this thread, I'd like to comment a couple of things from Netnod's (operators of i.root-servers.net) side. First, just to re-iterate: I-root servers operated by Netnod responds to all DNS queries we can handle, as we receive them, with unaltered answers from the true root zone. Period. We are, at the moment, not aware of any servers outside of our control operating on I-roots IP-addresses. What happens to the DNS packets beyond the first upstream router is at best difficult, and in many cases impossible, for us to control, though. Netnod operates two I-root nodes in China. The one in Beijing has been in operation since 2007 (IIRC) with a longer stop a few years ago, which boiled down to sheer issues of old and malfunctioning hardware. It is now back on-line again on newer hardware. Our second node in China, in Shenyang, is brand new (months). As Ray Bellis notes, we had a similar incident with the I-root node in Beijing back in 2010. It was fixed blindingly fast and with profuse apologies when we reported it to our site host. My experience is that Chinese authorities have no wish to inflict problems on clients outside China, and that whatever impersonation/leakage happens is indeed due to configuration errors on networking equipment. There is no way to guarantee that any one ISP (inside China or not) does what you expect and hope with your BGP announcements and the traffic going to/from any server of yours (DNS root or other). Specifically, I expect that a country with more than a billion citizens has a network complexity of certain scale, which, in combination with the intricate large scale traffic filters, makes "playing" with NO_EXPORT even trickier than normal. Life with anycast is a constant challenge to deploy the right number of instances at the right points in topology in order to make the right thing happen given an existing budget. If you see any signs of problems with i.root-servers.net, please report them without delay to . Every such report is of great value to us, as it helps us understand what our service looks like to you. These observations are important fixpoints in our continuous efforts to improve our service. And finally to each and every one of you: Please turn on validation in your resolvers and sign your zones. DNSSEC is your friend. Best regards, /Liman hostmas...@i.root-servers.net #-- # Lars-Johan Liman, M.Sc. ! E-mail: li...@netnod.se # Senior Systems Specialist ! Tel: +46 8 - 562 860 12 # Netnod AB, Stockholm ! http://www.netnod.se/ #-- ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] K-root in CN leaking outside of CN
Hi Davey, Manu, The server we operate in Guangzhou was indeed reachable from outside China. This is not the intention, of course. On Saturday, when we got notification about this, we withdrew the prefix from the server, and we are communicating with the host to solve this. Many people have already said this, but I'd like to make it clear that the K-root server was NOT emitting false responses for Facebook and WhatsApp. The responses were being modified by something between the server and its clients. Regards, Anand Buddhdev RIPE NCC On 08/11/2021 08:45, Davey Song wrote: If it is urgent, I suggest the K root operator withdraw the route of the instance in Guangzhou immediately. ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] K-root in CN leaking outside of CN
If it is urgent, I suggest the K root operator withdraw the route of the instance in Guangzhou immediately. Davey On Mon, 8 Nov 2021 at 15:23, Davey Song wrote: > Hi Manu, > > Is it still the case? I will try to outreach the people of AS4134 and > AS138457. > > Best regards, > Davey > > On Sat, 6 Nov 2021 at 12:18, Manu Bretelle wrote: > >> Hi all, >> >> Based on https://root-servers.org/, there are a few root servers >> operated from Mainland China. >> >> How do we ensure that those are not advertised outside of China so DNS >> answers are not poisoned by the GFW? >> >> Are there any contracts that root in CN are supposed to follow to prevent >> this? Is the onus put on both the CN ASNs and their respective non-CN ASNs >> peers to not advertise/not accept the root range on those specific peering >> links? If so, how is it ensured that every operator knows about those rules? >> Is there any monitoring performed by root operators to ensure that leaks >> are being detected and possibly addressed? >> >> I don't believe this specific leak I am seeing is malicious, but rather >> is just a misconfiguration and I really wonder how this could be >> prevented/addressed early on. >> I have ran some probes in other regions and do not have proof that this >> is happening more widely than a specific AS, but this was not exhaustive >> and I could have very likely missed something. >> >> As for this specific problem, we have reached out to both the AS that is >> accepting the leak and RIPE NCC as we identified the issue, provided the >> ISP possible workaround in the meantime. >> >> Both DNSSEC and Qname minimization would have helped the resolver >> detecting bogus answers, or just getting to com. in the first place, while >> this would have helped, there is still an ongoing leak. >> >> >> Longer story for the ones that want to dig more into it >> >> I am asking because we (FB/Meta) got reports from an ISP in MX which >> users were not able to access whatsapp.net. For instance, answer would >> be 199.59.149.244 which is not quite the right answer >> >> Some initial debugging from the ISP seemed to point to k-root acting up. >> e.g something alike: >> >> ``` >> # dig +trace d.ns.facebook.com >> >> ; <<>> DiG 9.11.13 <<>> +trace d.ns.facebook.com >> ;; global options: +cmd >> . 518379 IN NS m.root-servers.net. >> . 518379 IN NS e.root-servers.net. >> . 518379 IN NS g.root-servers.net. >> . 518379 IN NS j.root-servers.net. >> . 518379 IN NS k.root-servers.net. >> . 518379 IN NS b.root-servers.net. >> . 518379 IN NS l.root-servers.net. >> . 518379 IN NS a.root-servers.net. >> . 518379 IN NS i.root-servers.net. >> . 518379 IN NS d.root-servers.net. >> . 518379 IN NS f.root-servers.net. >> . 518379 IN NS h.root-servers.net. >> . 518379 IN NS c.root-servers.net. >> . 518379 IN RRSIG NS 8 0 518400 >> 202717 2021110416 14748 . >> inFOlh92Cxaf58/AdV/M4SZ37+MCm6PMOn6RNHDtE1MR6yvD0sfSPui9 >> YR3o9Yix/55zuodOWkCh7A0mMosbC5v2gMeiR9iw5jWko5dU7tPPSMnL >> MZNgsRvIjuR80RWOJnvEVZyz45BXtFWd6UcCIG3BahAUSOXAWhqhkNP4 >> gF6YeDsZHElhjvhWAzBA/44aFCJPT2nySKuzH4cGRulhO0remY6CHD4o >> 59fQooYT8lopP6SWdHOmDYhdb6/UBGDELd35QwGG0MDAMSie6jZGGkeb >> DhAuTFRWzboxlbqQw3nyYlH0Ot8lSatzhx0Cl0rNIBTboFQiWIUMgtVi PeRj0Q== >> ;; Received 1125 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms >> >> ;; expected opt record in response >> d.ns.facebook.com. 65 IN A 31.13.67.19 >> ;; Received 51 bytes from 193.0.14.129#53(k.root-servers.net) in 231 ms >> ``` >> >> Looking a bit more into it: >> >> Querying d.ns.facebook.com/A against k-root directly from MX probes: >> https://atlas.ripe.net/measurements/33184386/ >> ``` >> $ blaeu-resolve -m 33184386 -q A d.ns.facebook.com >> [] : 13 occurrences >> [202.160.128.195] : 1 occurrences >> [199.59.148.97] : 1 occurrences >> [185.89.219.12] : 2 occurrences >> [31.13.96.193] : 1 occurrences >> [208.77.47.172] : 1 occurrences >> Test #33184386 done at 2021-11-05T20:36:59Z >> ``` >> >> Getting an answer in the first place is kind of unexpected but I will not >> focus on the ones returning the correct answer (e.g 185.89.219.12). >> >> Checking the probes that return those results: >> ``` >> ripe-atlas report --renderer dns_compact 33184386 >> ... >> ... >> Probe #27558: 2021-11-05 13:36:59 NOERROR qr ra rd d.ns.facebook.com. 98 >> A 202.160.128.195 >> Probe #31355: 2021-11-05 13:37:01 NOERROR qr ra rd d.ns.facebook.com. >> 146 A 199.59.148.97 >> Probe #52013: 2021-11-05 13:37:01 NOERROR qr ra rd d.ns.facebook.com. >> 179 A
Re: [dns-operations] K-root in CN leaking outside of CN
Hi Manu, Is it still the case? I will try to outreach the people of AS4134 and AS138457. Best regards, Davey On Sat, 6 Nov 2021 at 12:18, Manu Bretelle wrote: > Hi all, > > Based on https://root-servers.org/, there are a few root servers operated > from Mainland China. > > How do we ensure that those are not advertised outside of China so DNS > answers are not poisoned by the GFW? > > Are there any contracts that root in CN are supposed to follow to prevent > this? Is the onus put on both the CN ASNs and their respective non-CN ASNs > peers to not advertise/not accept the root range on those specific peering > links? If so, how is it ensured that every operator knows about those rules? > Is there any monitoring performed by root operators to ensure that leaks > are being detected and possibly addressed? > > I don't believe this specific leak I am seeing is malicious, but rather is > just a misconfiguration and I really wonder how this could be > prevented/addressed early on. > I have ran some probes in other regions and do not have proof that this is > happening more widely than a specific AS, but this was not exhaustive and I > could have very likely missed something. > > As for this specific problem, we have reached out to both the AS that is > accepting the leak and RIPE NCC as we identified the issue, provided the > ISP possible workaround in the meantime. > > Both DNSSEC and Qname minimization would have helped the resolver > detecting bogus answers, or just getting to com. in the first place, while > this would have helped, there is still an ongoing leak. > > > Longer story for the ones that want to dig more into it > > I am asking because we (FB/Meta) got reports from an ISP in MX which users > were not able to access whatsapp.net. For instance, answer would be > 199.59.149.244 which is not quite the right answer > > Some initial debugging from the ISP seemed to point to k-root acting up. > e.g something alike: > > ``` > # dig +trace d.ns.facebook.com > > ; <<>> DiG 9.11.13 <<>> +trace d.ns.facebook.com > ;; global options: +cmd > . 518379 IN NS m.root-servers.net. > . 518379 IN NS e.root-servers.net. > . 518379 IN NS g.root-servers.net. > . 518379 IN NS j.root-servers.net. > . 518379 IN NS k.root-servers.net. > . 518379 IN NS b.root-servers.net. > . 518379 IN NS l.root-servers.net. > . 518379 IN NS a.root-servers.net. > . 518379 IN NS i.root-servers.net. > . 518379 IN NS d.root-servers.net. > . 518379 IN NS f.root-servers.net. > . 518379 IN NS h.root-servers.net. > . 518379 IN NS c.root-servers.net. > . 518379 IN RRSIG NS 8 0 518400 > 202717 2021110416 14748 . > inFOlh92Cxaf58/AdV/M4SZ37+MCm6PMOn6RNHDtE1MR6yvD0sfSPui9 > YR3o9Yix/55zuodOWkCh7A0mMosbC5v2gMeiR9iw5jWko5dU7tPPSMnL > MZNgsRvIjuR80RWOJnvEVZyz45BXtFWd6UcCIG3BahAUSOXAWhqhkNP4 > gF6YeDsZHElhjvhWAzBA/44aFCJPT2nySKuzH4cGRulhO0remY6CHD4o > 59fQooYT8lopP6SWdHOmDYhdb6/UBGDELd35QwGG0MDAMSie6jZGGkeb > DhAuTFRWzboxlbqQw3nyYlH0Ot8lSatzhx0Cl0rNIBTboFQiWIUMgtVi PeRj0Q== > ;; Received 1125 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms > > ;; expected opt record in response > d.ns.facebook.com. 65 IN A 31.13.67.19 > ;; Received 51 bytes from 193.0.14.129#53(k.root-servers.net) in 231 ms > ``` > > Looking a bit more into it: > > Querying d.ns.facebook.com/A against k-root directly from MX probes: > https://atlas.ripe.net/measurements/33184386/ > ``` > $ blaeu-resolve -m 33184386 -q A d.ns.facebook.com > [] : 13 occurrences > [202.160.128.195] : 1 occurrences > [199.59.148.97] : 1 occurrences > [185.89.219.12] : 2 occurrences > [31.13.96.193] : 1 occurrences > [208.77.47.172] : 1 occurrences > Test #33184386 done at 2021-11-05T20:36:59Z > ``` > > Getting an answer in the first place is kind of unexpected but I will not > focus on the ones returning the correct answer (e.g 185.89.219.12). > > Checking the probes that return those results: > ``` > ripe-atlas report --renderer dns_compact 33184386 > ... > ... > Probe #27558: 2021-11-05 13:36:59 NOERROR qr ra rd d.ns.facebook.com. 98 > A 202.160.128.195 > Probe #31355: 2021-11-05 13:37:01 NOERROR qr ra rd d.ns.facebook.com. 146 > A 199.59.148.97 > Probe #52013: 2021-11-05 13:37:01 NOERROR qr ra rd d.ns.facebook.com. 179 > A 31.13.96.193 > Probe #52660: 2021-11-05 13:37:00 NOERROR qr ra rd d.ns.facebook.com. 150 > A 208.77.47.172 > ... > ``` > > Those probes will fail to reach 193.0.14.129 (k-root) over TCP. > > Checking which id.server is returned by the k-roots reached by those > probes: > > ``` > ripe-atlas
Re: [dns-operations] K-root in CN leaking outside of CN
--- Begin Message --- On 6 Nov 2021, at 19:22, Mark Andrews wrote: > Just sign your zones so those that do validate can reject spoofed answers. +1 And if you worry, do validate yourself. Patrik signature.asc Description: OpenPGP digital signature --- End Message --- ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] K-root in CN leaking outside of CN
And if it wasn’t one of the root server instances it could have been one of the COM / NET server instances. BGP results in some strange traffic flows at times. You don’t notice it most of the time. Lots of countries have instructed their ISPs to modify / intercept DNS responses / requests. Spoofed answers reaching unintended clients will always happen. In this case it happened to a query / response directed to a K root server instance. It could have easily been an NET server instance and depending upon the country mucking with the responses a WHATSAPP.NET server instance. Just sign your zones so those that do validate can reject spoofed answers. -- Mark Andrews > On 7 Nov 2021, at 04:58, Manu Bretelle wrote: > > >> On Sat, Nov 6, 2021 at 12:01 AM Mark Andrews wrote: >>> >> >> >> > >> > Both DNSSEC and Qname minimization would have helped the resolver >> > detecting bogus answers, or just getting to com. in the first place, while >> > this would have helped, there is still an ongoing leak. >> >> So why isn’t Facebook/Meta DNSSEC signing their zones? It’s definitely >> possible to do that. DNSSEC validation can’t reject spoofed responses if >> the records being looked up are not signed. You have known that China, >> Turkey, Egypt ... are modifying answers for your zones for years but failed >> to take prudent steps to mitigate the damage. > > Maybe I should rephrase that as "Both DNSSEC validation and Qname > minimization...". I don't want to get into the debate of zone signing the > final zone here and if I am not mistaken, in this specific example, it would > not have changed anything given that the resolvers impacted are not > validating. Had they been validating, they would not accept the invalid > answer from k-root, or at least whatever is responding instead of it, would > have failed to another letter (given that not all letters are impacted from > their vantage point), eventually have found its way to com and proceed as > usual. > >> >> > Longer story for the ones that want to dig more into it >> > >> > I am asking because we (FB/Meta) got reports from an ISP in MX which users >> > were not able to access whatsapp.net. For instance, answer would be >> > 199.59.149.244 which is not quite the right answer >> >> I know lots of ISPs validate responses. If they where and you where signing >> there would not have been an issue. The resolver would have rejected the A >> RRset and tried another server. > > Those ISPs, as much as I can tell in this example, would not have been > impacted. But again, this post is about the behaviour to the root, not the > specific domains used in the example. > > Thanks, > Manu > >> >> > Some initial debugging from the ISP seemed to point to k-root acting up. >> > e.g something alike: >> > >> > ``` >> > # dig +trace d.ns.facebook.com >> > >> > ; <<>> DiG 9.11.13 <<>> +trace d.ns.facebook.com >> > ;; global options: +cmd >> > . 518379 IN NS m.root-servers.net. >> > . 518379 IN NS e.root-servers.net. >> > . 518379 IN NS g.root-servers.net. >> > . 518379 IN NS j.root-servers.net. >> > . 518379 IN NS k.root-servers.net. >> > . 518379 IN NS b.root-servers.net. >> > . 518379 IN NS l.root-servers.net. >> > . 518379 IN NS a.root-servers.net. >> > . 518379 IN NS i.root-servers.net. >> > . 518379 IN NS d.root-servers.net. >> > . 518379 IN NS f.root-servers.net. >> > . 518379 IN NS h.root-servers.net. >> > . 518379 IN NS c.root-servers.net. >> > . 518379 IN RRSIG NS 8 0 518400 >> > 202717 2021110416 14748 . >> > inFOlh92Cxaf58/AdV/M4SZ37+MCm6PMOn6RNHDtE1MR6yvD0sfSPui9 >> > YR3o9Yix/55zuodOWkCh7A0mMosbC5v2gMeiR9iw5jWko5dU7tPPSMnL >> > MZNgsRvIjuR80RWOJnvEVZyz45BXtFWd6UcCIG3BahAUSOXAWhqhkNP4 >> > gF6YeDsZHElhjvhWAzBA/44aFCJPT2nySKuzH4cGRulhO0remY6CHD4o >> > 59fQooYT8lopP6SWdHOmDYhdb6/UBGDELd35QwGG0MDAMSie6jZGGkeb >> > DhAuTFRWzboxlbqQw3nyYlH0Ot8lSatzhx0Cl0rNIBTboFQiWIUMgtVi PeRj0Q== >> > ;; Received 1125 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms >> > >> > ;; expected opt record in response >> > d.ns.facebook.com. 65 IN A 31.13.67.19 >> > ;; Received 51 bytes from 193.0.14.129#53(k.root-servers.net) in 231 ms >> > ``` >> > >> > Looking a bit more into it: >> > >> > Querying d.ns.facebook.com/A against k-root directly from MX probes: >> > https://atlas.ripe.net/measurements/33184386/ >> > ``` >> > $ blaeu-resolve -m 33184386 -q A d.ns.facebook.com >> > [] : 13 occurrences >> > [202.160.128.195] : 1 occurrences >> >
Re: [dns-operations] K-root in CN leaking outside of CN
On Sat, Nov 6, 2021 at 9:35 AM Phillip Hallam-Baker wrote: > On Sat, Nov 6, 2021 at 12:22 AM Manu Bretelle wrote: > >> Hi all, >> >> Based on https://root-servers.org/, there are a few root servers >> operated from Mainland China. >> >> How do we ensure that those are not advertised outside of China so DNS >> answers are not poisoned by the GFW? >> > > You can't. > > All you can do is to authenticate the data and reject invalid responses. > Thanks Philip, I do understand, but there is still a long way to go before this happens globally based on https://stats.labs.apnic.net/dnssec/ it is roughly 30% of resolvers validating. > > I am getting heartily sick of all this fearmongering about China. One of > the chief fearmongers who was largely responsible for coining the phrase > 'yellow peril' was Kaiser Wilhelm II who after telling Europe how China was > going to invade Europe for decades went and invaded Europe himself starting > WWI. > It is not my intent to discuss this on this forum. > > If the DNS protocol were sane the root zone would be published as a > notarized, chained append only log. Every DNS resolver would obtain a list > of updates to that log either directly or indirectly. There would be no > root server to poison or DDoS. > > But the DNS protocol is not sane and is not going to be changed. Not least > because the organizations that run root servers are rather pleased about > the prestige it brings to them. > There is probably a lot that can be done, that could have been done better but as you certainly know, the DNS ecosystem being that widely distributed by totally independent systems is extremely slow to migrate. And while addressing the problem in the long run is definitely something valuable, I wonder how we could prevent the problem in its current state. Manu ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] K-root in CN leaking outside of CN
On Sat, Nov 6, 2021 at 12:01 AM Mark Andrews wrote: > > > > > > >> > Both DNSSEC and Qname minimization would have helped the resolver > detecting bogus answers, or just getting to com. in the first place, while > this would have helped, there is still an ongoing leak. > >> > So why isn’t Facebook/Meta DNSSEC signing their zones? It’s definitely > possible to do that. DNSSEC validation can’t reject spoofed responses if > the records being looked up are not signed. You have known that China, > Turkey, Egypt ... are modifying answers for your zones for years but failed > to take prudent steps to mitigate the damage. > Maybe I should rephrase that as "Both DNSSEC validation and Qname minimization...". I don't want to get into the debate of zone signing the final zone here and if I am not mistaken, in this specific example, it would not have changed anything given that the resolvers impacted are not validating. Had they been validating, they would not accept the invalid answer from k-root, or at least whatever is responding instead of it, would have failed to another letter (given that not all letters are impacted from their vantage point), eventually have found its way to com and proceed as usual. > > > Longer story for the ones that want to dig more into it > >> > > >> > I am asking because we (FB/Meta) got reports from an ISP in MX which > users were not able to access whatsapp.net. For instance, answer would be > 199.59.149.244 which is not quite the right answer > >> > I know lots of ISPs validate responses. If they where and you where > signing there would not have been an issue. The resolver would have > rejected the A RRset and tried another server. > Those ISPs, as much as I can tell in this example, would not have been impacted. But again, this post is about the behaviour to the root, not the specific domains used in the example. Thanks, Manu > > Some initial debugging from the ISP seemed to point to k-root acting up. > e.g something alike: > >> > > >> > ``` > >> > # dig +trace d.ns.facebook.com > >> > > >> > ; <<>> DiG 9.11.13 <<>> +trace d.ns.facebook.com > >> > ;; global options: +cmd > >> > . 518379 IN NS m.root-servers.net. > >> > . 518379 IN NS e.root-servers.net. > >> > . 518379 IN NS g.root-servers.net. > >> > . 518379 IN NS j.root-servers.net. > >> > . 518379 IN NS k.root-servers.net. > >> > . 518379 IN NS b.root-servers.net. > >> > . 518379 IN NS l.root-servers.net. > >> > . 518379 IN NS a.root-servers.net. > >> > . 518379 IN NS i.root-servers.net. > >> > . 518379 IN NS d.root-servers.net. > >> > . 518379 IN NS f.root-servers.net. > >> > . 518379 IN NS h.root-servers.net. > >> > . 518379 IN NS c.root-servers.net. > >> > . 518379 IN RRSIG NS 8 0 518400 > 202717 2021110416 14748 . > inFOlh92Cxaf58/AdV/M4SZ37+MCm6PMOn6RNHDtE1MR6yvD0sfSPui9 > YR3o9Yix/55zuodOWkCh7A0mMosbC5v2gMeiR9iw5jWko5dU7tPPSMnL > MZNgsRvIjuR80RWOJnvEVZyz45BXtFWd6UcCIG3BahAUSOXAWhqhkNP4 > gF6YeDsZHElhjvhWAzBA/44aFCJPT2nySKuzH4cGRulhO0remY6CHD4o > 59fQooYT8lopP6SWdHOmDYhdb6/UBGDELd35QwGG0MDAMSie6jZGGkeb > DhAuTFRWzboxlbqQw3nyYlH0Ot8lSatzhx0Cl0rNIBTboFQiWIUMgtVi PeRj0Q== > >> > ;; Received 1125 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms > >> > > >> > ;; expected opt record in response > >> > d.ns.facebook.com. 65 IN A 31.13.67.19 > >> > ;; Received 51 bytes from 193.0.14.129#53(k.root-servers.net) in 231 ms > >> > ``` > >> > > >> > Looking a bit more into it: > >> > > >> > Querying d.ns.facebook.com/A against k-root directly from MX probes: > >> > https://atlas.ripe.net/measurements/33184386/ > >> > ``` > >> > $ blaeu-resolve -m 33184386 -q A d.ns.facebook.com > >> > [] : 13 occurrences > >> > [202.160.128.195] : 1 occurrences > >> > [199.59.148.97] : 1 occurrences > >> > [185.89.219.12] : 2 occurrences > >> > [31.13.96.193] : 1 occurrences > >> > [208.77.47.172] : 1 occurrences > >> > Test #33184386 done at 2021-11-05T20:36:59Z > >> > ``` > >> > > >> > Getting an answer in the first place is kind of unexpected but I will > not focus on the ones returning the correct answer (e.g 185.89.219.12). > >> > > >> > Checking the probes that return those results: > >> > ``` > >> > ripe-atlas report --renderer dns_compact 33184386 > >> > ... > >> > ... > >> > Probe #27558: 2021-11-05 13:36:59 NOERROR qr ra rd d.ns.facebook.com. > 98 A 202.160.128.195 > >> > Probe #31355: 2021-11-05 13:37:01 NOERROR qr ra rd d.ns.facebook.com. > 146 A 199.59.148.97 > >> > Probe #52013: 2021-11-05 13:37:01 NOERROR qr ra rd d.ns.facebook.com. >
Re: [dns-operations] K-root in CN leaking outside of CN
On Sat, Nov 6, 2021 at 12:52 PM Viktor Dukhovni wrote: > On Sat, Nov 06, 2021 at 12:35:00PM -0400, Phillip Hallam-Baker wrote: > > > If the DNS protocol were sane the root zone would be published as a > > notarized, chained append only log. Every DNS resolver would obtain a > list > > of updates to that log either directly or indirectly. There would be no > > root server to poison or DDoS. > > https://localroot.isi.edu/about/ So near! But the problems are that 1) you need to know you have the up to date copy. so I think you want to go for a chain approach which naturally lends itself to replication 2) you need an infrastructure that allows you to get updates when the source is DDoSed and 3) this has to be turned on by default I know this has been proposed multiple times. But it still isn't common let alone standard operating procedure. It should be of course because local root allows a resolver to kill resolution of non existent TLDs which is 99% of the traffic they send to the root. ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] K-root in CN leaking outside of CN
On Sat, Nov 06, 2021 at 12:35:00PM -0400, Phillip Hallam-Baker wrote: > If the DNS protocol were sane the root zone would be published as a > notarized, chained append only log. Every DNS resolver would obtain a list > of updates to that log either directly or indirectly. There would be no > root server to poison or DDoS. https://localroot.isi.edu/about/ -- Viktor. ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] K-root in CN leaking outside of CN
On Sat, Nov 6, 2021 at 12:22 AM Manu Bretelle wrote: > Hi all, > > Based on https://root-servers.org/, there are a few root servers operated > from Mainland China. > > How do we ensure that those are not advertised outside of China so DNS > answers are not poisoned by the GFW? > You can't. All you can do is to authenticate the data and reject invalid responses. I am getting heartily sick of all this fearmongering about China. One of the chief fearmongers who was largely responsible for coining the phrase 'yellow peril' was Kaiser Wilhelm II who after telling Europe how China was going to invade Europe for decades went and invaded Europe himself starting WWI. If the DNS protocol were sane the root zone would be published as a notarized, chained append only log. Every DNS resolver would obtain a list of updates to that log either directly or indirectly. There would be no root server to poison or DDoS. But the DNS protocol is not sane and is not going to be changed. Not least because the organizations that run root servers are rather pleased about the prestige it brings to them. ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] K-root in CN leaking outside of CN
> On 6 Nov 2021, at 15:13, Manu Bretelle wrote: > > Hi all, > > Based on https://root-servers.org/, there are a few root servers operated > from Mainland China. > > How do we ensure that those are not advertised outside of China so DNS > answers are not poisoned by the GFW? > > Are there any contracts that root in CN are supposed to follow to prevent > this? Is the onus put on both the CN ASNs and their respective non-CN ASNs > peers to not advertise/not accept the root range on those specific peering > links? If so, how is it ensured that every operator knows about those rules? > Is there any monitoring performed by root operators to ensure that leaks are > being detected and possibly addressed? > > I don't believe this specific leak I am seeing is malicious, but rather is > just a misconfiguration and I really wonder how this could be > prevented/addressed early on. > I have ran some probes in other regions and do not have proof that this is > happening more widely than a specific AS, but this was not exhaustive and I > could have very likely missed something. > > As for this specific problem, we have reached out to both the AS that is > accepting the leak and RIPE NCC as we identified the issue, provided the ISP > possible workaround in the meantime. > > Both DNSSEC and Qname minimization would have helped the resolver detecting > bogus answers, or just getting to com. in the first place, while this would > have helped, there is still an ongoing leak. So why isn’t Facebook/Meta DNSSEC signing their zones? It’s definitely possible to do that. DNSSEC validation can’t reject spoofed responses if the records being looked up are not signed. You have known that China, Turkey, Egypt ... are modifying answers for your zones for years but failed to take prudent steps to mitigate the damage. > Longer story for the ones that want to dig more into it > > I am asking because we (FB/Meta) got reports from an ISP in MX which users > were not able to access whatsapp.net. For instance, answer would be > 199.59.149.244 which is not quite the right answer I know lots of ISPs validate responses. If they where and you where signing there would not have been an issue. The resolver would have rejected the A RRset and tried another server. > Some initial debugging from the ISP seemed to point to k-root acting up. e.g > something alike: > > ``` > # dig +trace d.ns.facebook.com > > ; <<>> DiG 9.11.13 <<>> +trace d.ns.facebook.com > ;; global options: +cmd > . 518379 IN NS m.root-servers.net. > . 518379 IN NS e.root-servers.net. > . 518379 IN NS g.root-servers.net. > . 518379 IN NS j.root-servers.net. > . 518379 IN NS k.root-servers.net. > . 518379 IN NS b.root-servers.net. > . 518379 IN NS l.root-servers.net. > . 518379 IN NS a.root-servers.net. > . 518379 IN NS i.root-servers.net. > . 518379 IN NS d.root-servers.net. > . 518379 IN NS f.root-servers.net. > . 518379 IN NS h.root-servers.net. > . 518379 IN NS c.root-servers.net. > . 518379 IN RRSIG NS 8 0 518400 202717 > 2021110416 14748 . > inFOlh92Cxaf58/AdV/M4SZ37+MCm6PMOn6RNHDtE1MR6yvD0sfSPui9 > YR3o9Yix/55zuodOWkCh7A0mMosbC5v2gMeiR9iw5jWko5dU7tPPSMnL > MZNgsRvIjuR80RWOJnvEVZyz45BXtFWd6UcCIG3BahAUSOXAWhqhkNP4 > gF6YeDsZHElhjvhWAzBA/44aFCJPT2nySKuzH4cGRulhO0remY6CHD4o > 59fQooYT8lopP6SWdHOmDYhdb6/UBGDELd35QwGG0MDAMSie6jZGGkeb > DhAuTFRWzboxlbqQw3nyYlH0Ot8lSatzhx0Cl0rNIBTboFQiWIUMgtVi PeRj0Q== > ;; Received 1125 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms > > ;; expected opt record in response > d.ns.facebook.com. 65 IN A 31.13.67.19 > ;; Received 51 bytes from 193.0.14.129#53(k.root-servers.net) in 231 ms > ``` > > Looking a bit more into it: > > Querying d.ns.facebook.com/A against k-root directly from MX probes: > https://atlas.ripe.net/measurements/33184386/ > ``` > $ blaeu-resolve -m 33184386 -q A d.ns.facebook.com > [] : 13 occurrences > [202.160.128.195] : 1 occurrences > [199.59.148.97] : 1 occurrences > [185.89.219.12] : 2 occurrences > [31.13.96.193] : 1 occurrences > [208.77.47.172] : 1 occurrences > Test #33184386 done at 2021-11-05T20:36:59Z > ``` > > Getting an answer in the first place is kind of unexpected but I will not > focus on the ones returning the correct answer (e.g 185.89.219.12). > > Checking the probes that return those results: > ``` > ripe-atlas report --renderer dns_compact 33184386 > ... > ... > Probe #27558: 2021-11-05 13:36:59 NOERROR qr ra rd d.ns.facebook.com. 98 A > 202.160.128.195 > Probe
[dns-operations] K-root in CN leaking outside of CN
Hi all, Based on https://root-servers.org/, there are a few root servers operated from Mainland China. How do we ensure that those are not advertised outside of China so DNS answers are not poisoned by the GFW? Are there any contracts that root in CN are supposed to follow to prevent this? Is the onus put on both the CN ASNs and their respective non-CN ASNs peers to not advertise/not accept the root range on those specific peering links? If so, how is it ensured that every operator knows about those rules? Is there any monitoring performed by root operators to ensure that leaks are being detected and possibly addressed? I don't believe this specific leak I am seeing is malicious, but rather is just a misconfiguration and I really wonder how this could be prevented/addressed early on. I have ran some probes in other regions and do not have proof that this is happening more widely than a specific AS, but this was not exhaustive and I could have very likely missed something. As for this specific problem, we have reached out to both the AS that is accepting the leak and RIPE NCC as we identified the issue, provided the ISP possible workaround in the meantime. Both DNSSEC and Qname minimization would have helped the resolver detecting bogus answers, or just getting to com. in the first place, while this would have helped, there is still an ongoing leak. Longer story for the ones that want to dig more into it I am asking because we (FB/Meta) got reports from an ISP in MX which users were not able to access whatsapp.net. For instance, answer would be 199.59.149.244 which is not quite the right answer Some initial debugging from the ISP seemed to point to k-root acting up. e.g something alike: ``` # dig +trace d.ns.facebook.com ; <<>> DiG 9.11.13 <<>> +trace d.ns.facebook.com ;; global options: +cmd . 518379 IN NS m.root-servers.net. . 518379 IN NS e.root-servers.net. . 518379 IN NS g.root-servers.net. . 518379 IN NS j.root-servers.net. . 518379 IN NS k.root-servers.net. . 518379 IN NS b.root-servers.net. . 518379 IN NS l.root-servers.net. . 518379 IN NS a.root-servers.net. . 518379 IN NS i.root-servers.net. . 518379 IN NS d.root-servers.net. . 518379 IN NS f.root-servers.net. . 518379 IN NS h.root-servers.net. . 518379 IN NS c.root-servers.net. . 518379 IN RRSIG NS 8 0 518400 202717 2021110416 14748 . inFOlh92Cxaf58/AdV/M4SZ37+MCm6PMOn6RNHDtE1MR6yvD0sfSPui9 YR3o9Yix/55zuodOWkCh7A0mMosbC5v2gMeiR9iw5jWko5dU7tPPSMnL MZNgsRvIjuR80RWOJnvEVZyz45BXtFWd6UcCIG3BahAUSOXAWhqhkNP4 gF6YeDsZHElhjvhWAzBA/44aFCJPT2nySKuzH4cGRulhO0remY6CHD4o 59fQooYT8lopP6SWdHOmDYhdb6/UBGDELd35QwGG0MDAMSie6jZGGkeb DhAuTFRWzboxlbqQw3nyYlH0Ot8lSatzhx0Cl0rNIBTboFQiWIUMgtVi PeRj0Q== ;; Received 1125 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms ;; expected opt record in response d.ns.facebook.com. 65 IN A 31.13.67.19 ;; Received 51 bytes from 193.0.14.129#53(k.root-servers.net) in 231 ms ``` Looking a bit more into it: Querying d.ns.facebook.com/A against k-root directly from MX probes: https://atlas.ripe.net/measurements/33184386/ ``` $ blaeu-resolve -m 33184386 -q A d.ns.facebook.com [] : 13 occurrences [202.160.128.195] : 1 occurrences [199.59.148.97] : 1 occurrences [185.89.219.12] : 2 occurrences [31.13.96.193] : 1 occurrences [208.77.47.172] : 1 occurrences Test #33184386 done at 2021-11-05T20:36:59Z ``` Getting an answer in the first place is kind of unexpected but I will not focus on the ones returning the correct answer (e.g 185.89.219.12). Checking the probes that return those results: ``` ripe-atlas report --renderer dns_compact 33184386 ... ... Probe #27558: 2021-11-05 13:36:59 NOERROR qr ra rd d.ns.facebook.com. 98 A 202.160.128.195 Probe #31355: 2021-11-05 13:37:01 NOERROR qr ra rd d.ns.facebook.com. 146 A 199.59.148.97 Probe #52013: 2021-11-05 13:37:01 NOERROR qr ra rd d.ns.facebook.com. 179 A 31.13.96.193 Probe #52660: 2021-11-05 13:37:00 NOERROR qr ra rd d.ns.facebook.com. 150 A 208.77.47.172 ... ``` Those probes will fail to reach 193.0.14.129 (k-root) over TCP. Checking which id.server is returned by the k-roots reached by those probes: ``` ripe-atlas measure dns --query-argument id.server --query-type TXT --query-class CHAOS --from-country MX --target 193.0.14.129 https://atlas.ripe.net/measurements/33184807/ ``` where the interesting snippet is: ``` $ ripe-atlas report --renderer dns_compact 33184807 ... Probe #27558: 2021-11-05 14:08:54 NOERROR qr rd id.server. 0 TXT ns1.cn-ggz.k.ripe.net Probe #31355: 2021-11-05
