Re: [dns-operations] [Ext] K-root in CN leaking outside of CN
On Mon, Nov 8, 2021 at 10:13 AM Paul Hoffman wrote: > Did you investigate whether the impersonation persisted after the route > leak was fixed? That is, if someone is impersonating K-root for the vantage > points that you saw, they might be doing it all the time, not just when > there is a known route leak. A route leak makes impersonation easier, but > it is not a requirement. > That's a good point! I just re-ran the measurements: ``` blaeu-resolve -m 33234036 -q A d.ns.facebook.com [] : 16 occurrences [185.89.219.12] : 2 occurrences Test #33234036 done at 2021-11-08T18:14:39Z ``` The 2 occurrences returning `185.89.219.12` are the ones I mentioned earlier which seem to funnel everything to a local server. One of the original probe did not participate. Looking at server ids: ``` blaeu-resolve -m 33234039 -q TXT id.server ["ns1.vn-han.k.ripe.net"] : 1 occurrences ["ns3.us-mia.k.ripe.net"] : 4 occurrences ["ns1.us-mia.k.ripe.net"] : 3 occurrences ["ns1.ru-led.k.ripe.net"] : 2 occurrences ["ns2.us-mia.k.ripe.net"] : 4 occurrences [ERROR: NOTIMP] : 1 occurrences ["ns1.ch-gva.k.ripe.net"] : 1 occurrences [ERROR: SERVFAIL] : 1 occurrences ["ns1.gb-lon.k.ripe.net"] : 1 occurrences Test #33234039 done at 2021-11-08T18:15:35Z ``` The 4 originally impacted probes are going to MIA: ``` blaeu-resolve -m 33234048 -q TXT id.server ["ns3.us-mia.k.ripe.net"] : 1 occurrences ["ns2.us-mia.k.ripe.net"] : 1 occurrences ["ns1.us-mia.k.ripe.net"] : 1 occurrences Test #33234048 done at 2021-11-08T18:22:25Z ``` One of the original probes did not participate. Manu --Paul Hoffman ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] [Ext] K-root in CN leaking outside of CN
Did you investigate whether the impersonation persisted after the route leak was fixed? That is, if someone is impersonating K-root for the vantage points that you saw, they might be doing it all the time, not just when there is a known route leak. A route leak makes impersonation easier, but it is not a requirement. --Paul Hoffman smime.p7s Description: S/MIME cryptographic signature ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] [Ext] K-root in CN leaking outside of CN
>> >> > > Thanks Paul, > > Yeah, agreed, "kind of" is probably not the right term to use. I > essentially did not care in this specific example of any impersonation > which is why I added "but I will not focus on the ones returning the > correct answer (e.g 185.89.219.12)". I believe there could be a bazillion > reasons why a probe would behave like that, possibly someone running their > own pi-hole and redirecting all traffic to it, or something in that vein. > Not to go astray from the initial discussion in this thread, but closing the loop. Those 2 probes returning the "right" answer indeed seem to intercept all DNS traffic within their network to a local DNS server. ``` $ ripe-atlas report --renderer traceroute --traceroute-show-asns 33206215 Probe #51510 Sat Nov 06 13:36:43 PDT 2021 1 193.0.14.129 AS251528.057 ms 1.326 ms 1.258 ms Probe #51975 Sat Nov 06 13:36:42 PDT 2021 1 192.168.50.1 2.208 ms 1.742 ms 6.647 ms 2 192.168.40.1 1.11 ms* * 3 193.0.14.129 AS251521.689 ms 1.913 ms 1.862 ms ``` Manu > > >> >> This does not sound like leaking, it sounds like impersonation. (I say >> this without doing the level of research you clearly have done!) That is, a >> K-root instance inside or outside of $country would reply to a query for " >> d.ns.facebook.com" with a referral, not an answer. Thus, if you are >> sending that query to one of the IP addresses for $x.root-servers.net >> and you get an A record back, the host you are hitting is not run by one of >> the root server operators. >> > > To be more precise, I think it is leaking *and* impersonation. I didn't > mean to say that k-root there would answer incorrectly, but something in > between does. > > Thanks, > Manu > > >> --Paul Hoffman >> > ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] [Ext] K-root in CN leaking outside of CN
I helped deploy the unit. Great war stories about this. I think my favourite was the guys charcoaling lunch in the machineroom next door in the DC, using a wastepaper basket as a cooker, sitting on a salvaged sofa. That, and having rack power die when the room lights were turned out. On Mon, Nov 8, 2021 at 9:53 AM Manu Bretelle wrote: > > > > On Sun, Nov 7, 2021 at 1:48 AM Ray Bellis wrote: >> >> >> >> On 07/11/2021 09:28, Ray Bellis wrote: >> >> > There most certainly were - a similar leak/poison pattern was >> > detected from an I root node hosted in China in March 2010. >> >> and checking our own records, we've had F root servers in China since at >> least 2006. >> >> However we announce our Anycast prefixes "NO_EXPORT" and make it very >> clear that our routes must not propagate beyond the border. > > > Thanks Ray, that’s useful info. > > My original Google searches seemed to indicate that the first root in CN were > quite recent but if did not dig enough. > A colleague pointed me to > https://archive.nanog.org/meetings/nanog53/presentations/Tuesday/Losher.pdf / > https://bgpmon.net/f-root-dns-server-moved-to-beijing/ > > Which was about F hosted in China leaking out back in Oct 2011, but Nanog > slides indicate that answers were not rewritten in this case. > > Manu >> >> >> >> Ray >> ___ >> dns-operations mailing list >> dns-operations@lists.dns-oarc.net >> https://lists.dns-oarc.net/mailman/listinfo/dns-operations > > ___ > dns-operations mailing list > dns-operations@lists.dns-oarc.net > https://lists.dns-oarc.net/mailman/listinfo/dns-operations ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] [Ext] K-root in CN leaking outside of CN
On Sun, Nov 7, 2021 at 1:48 AM Ray Bellis wrote: > > > On 07/11/2021 09:28, Ray Bellis wrote: > > > There most certainly were - a similar leak/poison pattern was > > detected from an I root node hosted in China in March 2010. > > and checking our own records, we've had F root servers in China since at > least 2006. > > However we announce our Anycast prefixes "NO_EXPORT" and make it very > clear that our routes must not propagate beyond the border. Thanks Ray, that’s useful info. My original Google searches seemed to indicate that the first root in CN were quite recent but if did not dig enough. A colleague pointed me to https://archive.nanog.org/meetings/nanog53/presentations/Tuesday/Losher.pdf / https://bgpmon.net/f-root-dns-server-moved-to-beijing/ Which was about F hosted in China leaking out back in Oct 2011, but Nanog slides indicate that answers were not rewritten in this case. Manu > > > Ray > ___ > dns-operations mailing list > dns-operations@lists.dns-oarc.net > https://lists.dns-oarc.net/mailman/listinfo/dns-operations > ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] [Ext] K-root in CN leaking outside of CN
On 07/11/2021 09:28, Ray Bellis wrote: There most certainly were - a similar leak/poison pattern was detected from an I root node hosted in China in March 2010. and checking our own records, we've had F root servers in China since at least 2006. However we announce our Anycast prefixes "NO_EXPORT" and make it very clear that our routes must not propagate beyond the border. Ray ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] [Ext] K-root in CN leaking outside of CN
On 06/11/2021 20:20, Manu Bretelle wrote: I believe back in 2013 there were no root servers in China There most certainly were - a similar leak/poison pattern was detected from an I root node hosted in China in March 2010. I recall the date because Roy Arends and I had previously been studying the spoofed answers returned by the Great Firewall of China, and then reports of incorrect answers being seen outside of China surfaced during a DNS session at IETF 77 at Anaheim. Ray ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] [Ext] K-root in CN leaking outside of CN
On Nov 6, 2021, at 10:59 AM, Manu Bretelle wrote: > > Yeah, agreed, "kind of" is probably not the right term to use. I essentially > did not care in this specific example of any impersonation which is why I > added "but I will not focus on the ones returning the correct answer (e.g > 185.89.219.12)". I usually try to not speak for other people, but *many* of us on this list care very much about the difference between a route leak for a root server instance and a harmful impersonation. Today's widespread use of anycast by root servers makes route leaks almost insignificant; given the low rate of DNSSEC validation, any impersonation can be quite important. --Paul Hoffman smime.p7s Description: S/MIME cryptographic signature ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] [Ext] K-root in CN leaking outside of CN
On Sat, Nov 6, 2021 at 12:57 PM Geoff Huston wrote: > > > > On 7 Nov 2021, at 2:53 am, Paul Hoffman wrote: > > > > On Nov 5, 2021, at 9:13 PM, Manu Bretelle wrote: > >> > >> Looking a bit more into it: > >> > >> Querying d.ns.facebook.com/A against k-root directly from MX probes: > >> https://atlas.ripe.net/measurements/33184386/ > >> ``` > >> $ blaeu-resolve -m 33184386 -q A d.ns.facebook.com > >> [] : 13 occurrences > >> [202.160.128.195] : 1 occurrences > >> [199.59.148.97] : 1 occurrences > >> [185.89.219.12] : 2 occurrences > >> [31.13.96.193] : 1 occurrences > >> [208.77.47.172] : 1 occurrences > >> Test #33184386 done at 2021-11-05T20:36:59Z > >> ``` > >> > >> Getting an answer in the first place is kind of unexpected > > > > Not "kind of": definitely. d.ns.facebook.com is not in the root zone, > so no root server will answer with it. > > > > This does not sound like leaking, it sounds like impersonation. (I say > this without doing the level of research you clearly have done!) That is, a > K-root instance inside or outside of $country would reply to a query for " > d.ns.facebook.com" with a referral, not an answer. Thus, if you are > sending that query to one of the IP addresses for $x.root-servers.net and > you get an A record back, the host you are hitting is not run by one of the > root server operators. > > > I must agree with Paul. This is not a root server, its impersonation. DNS > query interception been observed within China for years - here’s a dig > result I recorded in 2013 when I was in China for an APNIC conference > Thanks Geoff, Yeah, I reply to Paul's message earlier that this was likely leak **and** impersonation. I believe back in 2013 there were no root servers in China, but there is now. What seemed (now fixed) to happen per the traceroutes in ripe-atlas report --renderer traceroute --traceroute-show-asns 33184963 was that traffic from MX transiting through AS22908 would then go through AS4134 (China Telecom Backbone) -> AS58466 (Chinanet Guangdong province) -> AS25152 (RIPE) to get to k-root. So this is what I call the leak, which had a side effect of impersonation probably for the same reasons as your 2013 dig trace. Manu > > $ dig @m.root-servers.net www.facebook.com > ; <<>> DiG 9.9.3-P1 <<>> @m.root-servers.net. www.facebook.com > ; (2 servers found) > ;; global options: +cmd > ;; Got answer: > ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3195 > ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 > > ;; QUESTION SECTION: > ;www.facebook.com IN A > > ;; ANSWER SECTION: www.facebook.com. 300 IN A 255.255.255.255 > ;; Query time: 38 msec > ;; SERVER: 2001:dc3::35#53(2001:dc3::35) > ;; WHEN: Tue Aug 27 19:07:12 EST 2013 > ;; MSG SIZE rcvd: 50 > > > Normally this behaviour (where a query to a root server address received a > response rather than a referral) was only visible within an area that was > covered by the GFW. > > Geoff Huston > > ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] [Ext] K-root in CN leaking outside of CN
> On 7 Nov 2021, at 2:53 am, Paul Hoffman wrote: > > On Nov 5, 2021, at 9:13 PM, Manu Bretelle wrote: >> >> Looking a bit more into it: >> >> Querying d.ns.facebook.com/A against k-root directly from MX probes: >> https://atlas.ripe.net/measurements/33184386/ >> ``` >> $ blaeu-resolve -m 33184386 -q A d.ns.facebook.com >> [] : 13 occurrences >> [202.160.128.195] : 1 occurrences >> [199.59.148.97] : 1 occurrences >> [185.89.219.12] : 2 occurrences >> [31.13.96.193] : 1 occurrences >> [208.77.47.172] : 1 occurrences >> Test #33184386 done at 2021-11-05T20:36:59Z >> ``` >> >> Getting an answer in the first place is kind of unexpected > > Not "kind of": definitely. d.ns.facebook.com is not in the root zone, so no > root server will answer with it. > > This does not sound like leaking, it sounds like impersonation. (I say this > without doing the level of research you clearly have done!) That is, a K-root > instance inside or outside of $country would reply to a query for > "d.ns.facebook.com" with a referral, not an answer. Thus, if you are sending > that query to one of the IP addresses for $x.root-servers.net and you get an > A record back, the host you are hitting is not run by one of the root server > operators. I must agree with Paul. This is not a root server, its impersonation. DNS query interception been observed within China for years - here’s a dig result I recorded in 2013 when I was in China for an APNIC conference $ dig @m.root-servers.net www.facebook.com ; <<>> DiG 9.9.3-P1 <<>> @m.root-servers.net. www.facebook.com ; (2 servers found) ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3195 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;www.facebook.com IN A ;; ANSWER SECTION: www.facebook.com. 300 IN A 255.255.255.255 ;; Query time: 38 msec ;; SERVER: 2001:dc3::35#53(2001:dc3::35) ;; WHEN: Tue Aug 27 19:07:12 EST 2013 ;; MSG SIZE rcvd: 50 Normally this behaviour (where a query to a root server address received a response rather than a referral) was only visible within an area that was covered by the GFW. Geoff Huston ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] [Ext] K-root in CN leaking outside of CN
On Sat, Nov 6, 2021 at 8:53 AM Paul Hoffman wrote: > On Nov 5, 2021, at 9:13 PM, Manu Bretelle wrote: > >> > > >> > Looking a bit more into it: > >> > > >> > Querying d.ns.facebook.com/A against k-root directly from MX probes: > >> > https://atlas.ripe.net/measurements/33184386/ > >> > ``` > >> > $ blaeu-resolve -m 33184386 -q A d.ns.facebook.com > >> > [] : 13 occurrences > >> > [202.160.128.195] : 1 occurrences > >> > [199.59.148.97] : 1 occurrences > >> > [185.89.219.12] : 2 occurrences > >> > [31.13.96.193] : 1 occurrences > >> > [208.77.47.172] : 1 occurrences > >> > Test #33184386 done at 2021-11-05T20:36:59Z > >> > ``` > >> > > >> > Getting an answer in the first place is kind of unexpected > >> > Not "kind of": definitely. d.ns.facebook.com is not in the root zone, so > no root server will answer with it. > Thanks Paul, Yeah, agreed, "kind of" is probably not the right term to use. I essentially did not care in this specific example of any impersonation which is why I added "but I will not focus on the ones returning the correct answer (e.g 185.89.219.12)". I believe there could be a bazillion reasons why a probe would behave like that, possibly someone running their own pi-hole and redirecting all traffic to it, or something in that vein. > > This does not sound like leaking, it sounds like impersonation. (I say > this without doing the level of research you clearly have done!) That is, a > K-root instance inside or outside of $country would reply to a query for " > d.ns.facebook.com" with a referral, not an answer. Thus, if you are > sending that query to one of the IP addresses for $x.root-servers.net and > you get an A record back, the host you are hitting is not run by one of the > root server operators. > To be more precise, I think it is leaking *and* impersonation. I didn't mean to say that k-root there would answer incorrectly, but something in between does. Thanks, Manu > --Paul Hoffman > ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
Re: [dns-operations] [Ext] K-root in CN leaking outside of CN
On Nov 5, 2021, at 9:13 PM, Manu Bretelle wrote: > > Looking a bit more into it: > > Querying d.ns.facebook.com/A against k-root directly from MX probes: > https://atlas.ripe.net/measurements/33184386/ > ``` > $ blaeu-resolve -m 33184386 -q A d.ns.facebook.com > [] : 13 occurrences > [202.160.128.195] : 1 occurrences > [199.59.148.97] : 1 occurrences > [185.89.219.12] : 2 occurrences > [31.13.96.193] : 1 occurrences > [208.77.47.172] : 1 occurrences > Test #33184386 done at 2021-11-05T20:36:59Z > ``` > > Getting an answer in the first place is kind of unexpected Not "kind of": definitely. d.ns.facebook.com is not in the root zone, so no root server will answer with it. This does not sound like leaking, it sounds like impersonation. (I say this without doing the level of research you clearly have done!) That is, a K-root instance inside or outside of $country would reply to a query for "d.ns.facebook.com" with a referral, not an answer. Thus, if you are sending that query to one of the IP addresses for $x.root-servers.net and you get an A record back, the host you are hitting is not run by one of the root server operators. --Paul Hoffman smime.p7s Description: S/MIME cryptographic signature ___ dns-operations mailing list dns-operations@lists.dns-oarc.net https://lists.dns-oarc.net/mailman/listinfo/dns-operations
