Re: [dns-operations] K-root in CN leaking outside of CN
If it is urgent, I suggest the K root operator withdraw the route of the instance in Guangzhou immediately. Davey On Mon, 8 Nov 2021 at 15:23, Davey Song wrote: > Hi Manu, > > Is it still the case? I will try to outreach the people of AS4134 and > AS138457. > > Best regards, > Davey > > On Sat, 6 Nov 2021 at 12:18, Manu Bretelle wrote: > >> Hi all, >> >> Based on https://root-servers.org/, there are a few root servers >> operated from Mainland China. >> >> How do we ensure that those are not advertised outside of China so DNS >> answers are not poisoned by the GFW? >> >> Are there any contracts that root in CN are supposed to follow to prevent >> this? Is the onus put on both the CN ASNs and their respective non-CN ASNs >> peers to not advertise/not accept the root range on those specific peering >> links? If so, how is it ensured that every operator knows about those rules? >> Is there any monitoring performed by root operators to ensure that leaks >> are being detected and possibly addressed? >> >> I don't believe this specific leak I am seeing is malicious, but rather >> is just a misconfiguration and I really wonder how this could be >> prevented/addressed early on. >> I have ran some probes in other regions and do not have proof that this >> is happening more widely than a specific AS, but this was not exhaustive >> and I could have very likely missed something. >> >> As for this specific problem, we have reached out to both the AS that is >> accepting the leak and RIPE NCC as we identified the issue, provided the >> ISP possible workaround in the meantime. >> >> Both DNSSEC and Qname minimization would have helped the resolver >> detecting bogus answers, or just getting to com. in the first place, while >> this would have helped, there is still an ongoing leak. >> >> >> Longer story for the ones that want to dig more into it >> >> I am asking because we (FB/Meta) got reports from an ISP in MX which >> users were not able to access whatsapp.net. For instance, answer would >> be 199.59.149.244 which is not quite the right answer >> >> Some initial debugging from the ISP seemed to point to k-root acting up. >> e.g something alike: >> >> ``` >> # dig +trace d.ns.facebook.com >> >> ; <<>> DiG 9.11.13 <<>> +trace d.ns.facebook.com >> ;; global options: +cmd >> . 518379 IN NS m.root-servers.net. >> . 518379 IN NS e.root-servers.net. >> . 518379 IN NS g.root-servers.net. >> . 518379 IN NS j.root-servers.net. >> . 518379 IN NS k.root-servers.net. >> . 518379 IN NS b.root-servers.net. >> . 518379 IN NS l.root-servers.net. >> . 518379 IN NS a.root-servers.net. >> . 518379 IN NS i.root-servers.net. >> . 518379 IN NS d.root-servers.net. >> . 518379 IN NS f.root-servers.net. >> . 518379 IN NS h.root-servers.net. >> . 518379 IN NS c.root-servers.net. >> . 518379 IN RRSIG NS 8 0 518400 >> 202717 2021110416 14748 . >> inFOlh92Cxaf58/AdV/M4SZ37+MCm6PMOn6RNHDtE1MR6yvD0sfSPui9 >> YR3o9Yix/55zuodOWkCh7A0mMosbC5v2gMeiR9iw5jWko5dU7tPPSMnL >> MZNgsRvIjuR80RWOJnvEVZyz45BXtFWd6UcCIG3BahAUSOXAWhqhkNP4 >> gF6YeDsZHElhjvhWAzBA/44aFCJPT2nySKuzH4cGRulhO0remY6CHD4o >> 59fQooYT8lopP6SWdHOmDYhdb6/UBGDELd35QwGG0MDAMSie6jZGGkeb >> DhAuTFRWzboxlbqQw3nyYlH0Ot8lSatzhx0Cl0rNIBTboFQiWIUMgtVi PeRj0Q== >> ;; Received 1125 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms >> >> ;; expected opt record in response >> d.ns.facebook.com. 65 IN A 31.13.67.19 >> ;; Received 51 bytes from 193.0.14.129#53(k.root-servers.net) in 231 ms >> ``` >> >> Looking a bit more into it: >> >> Querying d.ns.facebook.com/A against k-root directly from MX probes: >> https://atlas.ripe.net/measurements/33184386/ >> ``` >> $ blaeu-resolve -m 33184386 -q A d.ns.facebook.com >> [] : 13 occurrences >> [202.160.128.195] : 1 occurrences >> [199.59.148.97] : 1 occurrences >> [185.89.219.12] : 2 occurrences >> [31.13.96.193] : 1 occurrences >> [208.77.47.172] : 1 occurrences >> Test #33184386 done at 2021-11-05T20:36:59Z >> ``` >> >> Getting an answer in the first place is kind of unexpected but I will not >> focus on the ones returning the correct answer (e.g 185.89.219.12). >> >> Checking the probes that return those results: >> ``` >> ripe-atlas report --renderer dns_compact 33184386 >> ... >> ... >> Probe #27558: 2021-11-05 13:36:59 NOERROR qr ra rd d.ns.facebook.com. 98 >> A 202.160.128.195 >> Probe #31355: 2021-11-05 13:37:01 NOERROR qr ra rd d.ns.facebook.com. >> 146 A 199.59.148.97 >> Probe #52013: 2021-11-05 13:37:01 NOERROR qr ra rd d.ns.facebook.com. >> 179 A
Re: [dns-operations] K-root in CN leaking outside of CN
Hi Manu, Is it still the case? I will try to outreach the people of AS4134 and AS138457. Best regards, Davey On Sat, 6 Nov 2021 at 12:18, Manu Bretelle wrote: > Hi all, > > Based on https://root-servers.org/, there are a few root servers operated > from Mainland China. > > How do we ensure that those are not advertised outside of China so DNS > answers are not poisoned by the GFW? > > Are there any contracts that root in CN are supposed to follow to prevent > this? Is the onus put on both the CN ASNs and their respective non-CN ASNs > peers to not advertise/not accept the root range on those specific peering > links? If so, how is it ensured that every operator knows about those rules? > Is there any monitoring performed by root operators to ensure that leaks > are being detected and possibly addressed? > > I don't believe this specific leak I am seeing is malicious, but rather is > just a misconfiguration and I really wonder how this could be > prevented/addressed early on. > I have ran some probes in other regions and do not have proof that this is > happening more widely than a specific AS, but this was not exhaustive and I > could have very likely missed something. > > As for this specific problem, we have reached out to both the AS that is > accepting the leak and RIPE NCC as we identified the issue, provided the > ISP possible workaround in the meantime. > > Both DNSSEC and Qname minimization would have helped the resolver > detecting bogus answers, or just getting to com. in the first place, while > this would have helped, there is still an ongoing leak. > > > Longer story for the ones that want to dig more into it > > I am asking because we (FB/Meta) got reports from an ISP in MX which users > were not able to access whatsapp.net. For instance, answer would be > 199.59.149.244 which is not quite the right answer > > Some initial debugging from the ISP seemed to point to k-root acting up. > e.g something alike: > > ``` > # dig +trace d.ns.facebook.com > > ; <<>> DiG 9.11.13 <<>> +trace d.ns.facebook.com > ;; global options: +cmd > . 518379 IN NS m.root-servers.net. > . 518379 IN NS e.root-servers.net. > . 518379 IN NS g.root-servers.net. > . 518379 IN NS j.root-servers.net. > . 518379 IN NS k.root-servers.net. > . 518379 IN NS b.root-servers.net. > . 518379 IN NS l.root-servers.net. > . 518379 IN NS a.root-servers.net. > . 518379 IN NS i.root-servers.net. > . 518379 IN NS d.root-servers.net. > . 518379 IN NS f.root-servers.net. > . 518379 IN NS h.root-servers.net. > . 518379 IN NS c.root-servers.net. > . 518379 IN RRSIG NS 8 0 518400 > 202717 2021110416 14748 . > inFOlh92Cxaf58/AdV/M4SZ37+MCm6PMOn6RNHDtE1MR6yvD0sfSPui9 > YR3o9Yix/55zuodOWkCh7A0mMosbC5v2gMeiR9iw5jWko5dU7tPPSMnL > MZNgsRvIjuR80RWOJnvEVZyz45BXtFWd6UcCIG3BahAUSOXAWhqhkNP4 > gF6YeDsZHElhjvhWAzBA/44aFCJPT2nySKuzH4cGRulhO0remY6CHD4o > 59fQooYT8lopP6SWdHOmDYhdb6/UBGDELd35QwGG0MDAMSie6jZGGkeb > DhAuTFRWzboxlbqQw3nyYlH0Ot8lSatzhx0Cl0rNIBTboFQiWIUMgtVi PeRj0Q== > ;; Received 1125 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms > > ;; expected opt record in response > d.ns.facebook.com. 65 IN A 31.13.67.19 > ;; Received 51 bytes from 193.0.14.129#53(k.root-servers.net) in 231 ms > ``` > > Looking a bit more into it: > > Querying d.ns.facebook.com/A against k-root directly from MX probes: > https://atlas.ripe.net/measurements/33184386/ > ``` > $ blaeu-resolve -m 33184386 -q A d.ns.facebook.com > [] : 13 occurrences > [202.160.128.195] : 1 occurrences > [199.59.148.97] : 1 occurrences > [185.89.219.12] : 2 occurrences > [31.13.96.193] : 1 occurrences > [208.77.47.172] : 1 occurrences > Test #33184386 done at 2021-11-05T20:36:59Z > ``` > > Getting an answer in the first place is kind of unexpected but I will not > focus on the ones returning the correct answer (e.g 185.89.219.12). > > Checking the probes that return those results: > ``` > ripe-atlas report --renderer dns_compact 33184386 > ... > ... > Probe #27558: 2021-11-05 13:36:59 NOERROR qr ra rd d.ns.facebook.com. 98 > A 202.160.128.195 > Probe #31355: 2021-11-05 13:37:01 NOERROR qr ra rd d.ns.facebook.com. 146 > A 199.59.148.97 > Probe #52013: 2021-11-05 13:37:01 NOERROR qr ra rd d.ns.facebook.com. 179 > A 31.13.96.193 > Probe #52660: 2021-11-05 13:37:00 NOERROR qr ra rd d.ns.facebook.com. 150 > A 208.77.47.172 > ... > ``` > > Those probes will fail to reach 193.0.14.129 (k-root) over TCP. > > Checking which id.server is returned by the k-roots reached by those > probes: > > ``` > ripe-atlas
Re: [dns-operations] [Ext] K-root in CN leaking outside of CN
I helped deploy the unit. Great war stories about this. I think my favourite was the guys charcoaling lunch in the machineroom next door in the DC, using a wastepaper basket as a cooker, sitting on a salvaged sofa. That, and having rack power die when the room lights were turned out. On Mon, Nov 8, 2021 at 9:53 AM Manu Bretelle wrote: > > > > On Sun, Nov 7, 2021 at 1:48 AM Ray Bellis wrote: >> >> >> >> On 07/11/2021 09:28, Ray Bellis wrote: >> >> > There most certainly were - a similar leak/poison pattern was >> > detected from an I root node hosted in China in March 2010. >> >> and checking our own records, we've had F root servers in China since at >> least 2006. >> >> However we announce our Anycast prefixes "NO_EXPORT" and make it very >> clear that our routes must not propagate beyond the border. > > > Thanks Ray, that’s useful info. > > My original Google searches seemed to indicate that the first root in CN were > quite recent but if did not dig enough. > A colleague pointed me to > https://archive.nanog.org/meetings/nanog53/presentations/Tuesday/Losher.pdf / > https://bgpmon.net/f-root-dns-server-moved-to-beijing/ > > Which was about F hosted in China leaking out back in Oct 2011, but Nanog > slides indicate that answers were not rewritten in this case. > > Manu >> >> >> >> Ray >> ___ >> dns-operations mailing list >> dns-operations@lists.dns-oarc.net >> https://lists.dns-oarc.net/mailman/listinfo/dns-operations > > ___ > dns-operations mailing list > dns-operations@lists.dns-oarc.net > https://lists.dns-oarc.net/mailman/listinfo/dns-operations ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] [Ext] K-root in CN leaking outside of CN
On Sun, Nov 7, 2021 at 1:48 AM Ray Bellis wrote: > > > On 07/11/2021 09:28, Ray Bellis wrote: > > > There most certainly were - a similar leak/poison pattern was > > detected from an I root node hosted in China in March 2010. > > and checking our own records, we've had F root servers in China since at > least 2006. > > However we announce our Anycast prefixes "NO_EXPORT" and make it very > clear that our routes must not propagate beyond the border. Thanks Ray, that’s useful info. My original Google searches seemed to indicate that the first root in CN were quite recent but if did not dig enough. A colleague pointed me to https://archive.nanog.org/meetings/nanog53/presentations/Tuesday/Losher.pdf / https://bgpmon.net/f-root-dns-server-moved-to-beijing/ Which was about F hosted in China leaking out back in Oct 2011, but Nanog slides indicate that answers were not rewritten in this case. Manu > > > Ray > ___ > dns-operations mailing list > dns-operations@lists.dns-oarc.net > https://lists.dns-oarc.net/mailman/listinfo/dns-operations > ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] [Ext] K-root in CN leaking outside of CN
On 07/11/2021 09:28, Ray Bellis wrote: There most certainly were - a similar leak/poison pattern was detected from an I root node hosted in China in March 2010. and checking our own records, we've had F root servers in China since at least 2006. However we announce our Anycast prefixes "NO_EXPORT" and make it very clear that our routes must not propagate beyond the border. Ray ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] [Ext] K-root in CN leaking outside of CN
On 06/11/2021 20:20, Manu Bretelle wrote: I believe back in 2013 there were no root servers in China There most certainly were - a similar leak/poison pattern was detected from an I root node hosted in China in March 2010. I recall the date because Roy Arends and I had previously been studying the spoofed answers returned by the Great Firewall of China, and then reports of incorrect answers being seen outside of China surfaced during a DNS session at IETF 77 at Anaheim. Ray ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
