And if it wasn’t one of the root server instances it could have been one of the COM / NET server instances. BGP results in some strange traffic flows at times. You don’t notice it most of the time.
Lots of countries have instructed their ISPs to modify / intercept DNS responses / requests. Spoofed answers reaching unintended clients will always happen. In this case it happened to a query / response directed to a K root server instance. It could have easily been an NET server instance and depending upon the country mucking with the responses a WHATSAPP.NET server instance. Just sign your zones so those that do validate can reject spoofed answers. -- Mark Andrews > On 7 Nov 2021, at 04:58, Manu Bretelle <[email protected]> wrote: > > >> On Sat, Nov 6, 2021 at 12:01 AM Mark Andrews <[email protected]> wrote: >>> >> >> >> > >> > Both DNSSEC and Qname minimization would have helped the resolver >> > detecting bogus answers, or just getting to com. in the first place, while >> > this would have helped, there is still an ongoing leak. >> >> So why isn’t Facebook/Meta DNSSEC signing their zones? It’s definitely >> possible to do that. DNSSEC validation can’t reject spoofed responses if >> the records being looked up are not signed. You have known that China, >> Turkey, Egypt ... are modifying answers for your zones for years but failed >> to take prudent steps to mitigate the damage. > > Maybe I should rephrase that as "Both DNSSEC validation and Qname > minimization...". I don't want to get into the debate of zone signing the > final zone here and if I am not mistaken, in this specific example, it would > not have changed anything given that the resolvers impacted are not > validating. Had they been validating, they would not accept the invalid > answer from k-root, or at least whatever is responding instead of it, would > have failed to another letter (given that not all letters are impacted from > their vantage point), eventually have found its way to com and proceed as > usual. > >> >> > Longer story for the ones that want to dig more into it.... >> > >> > I am asking because we (FB/Meta) got reports from an ISP in MX which users >> > were not able to access whatsapp.net. For instance, answer would be >> > 199.59.149.244.... which is not quite the right answer.... >> >> I know lots of ISPs validate responses. If they where and you where signing >> there would not have been an issue. The resolver would have rejected the A >> RRset and tried another server. > > Those ISPs, as much as I can tell in this example, would not have been > impacted. But again, this post is about the behaviour to the root, not the > specific domains used in the example. > > Thanks, > Manu > >> >> > Some initial debugging from the ISP seemed to point to k-root acting up. >> > e.g something alike: >> > >> > ``` >> > # dig +trace d.ns.facebook.com >> > >> > ; <<>> DiG 9.11.13 <<>> +trace d.ns.facebook.com >> > ;; global options: +cmd >> > . 518379 IN NS m.root-servers.net. >> > . 518379 IN NS e.root-servers.net. >> > . 518379 IN NS g.root-servers.net. >> > . 518379 IN NS j.root-servers.net. >> > . 518379 IN NS k.root-servers.net. >> > . 518379 IN NS b.root-servers.net. >> > . 518379 IN NS l.root-servers.net. >> > . 518379 IN NS a.root-servers.net. >> > . 518379 IN NS i.root-servers.net. >> > . 518379 IN NS d.root-servers.net. >> > . 518379 IN NS f.root-servers.net. >> > . 518379 IN NS h.root-servers.net. >> > . 518379 IN NS c.root-servers.net. >> > . 518379 IN RRSIG NS 8 0 518400 >> > 20211117170000 20211104160000 14748 . >> > inFOlh92Cxaf58/AdV/M4SZ37+MCm6PMOn6RNHDtE1MR6yvD0sfSPui9 >> > YR3o9Yix/55zuodOWkCh7A0mMosbC5v2gMeiR9iw5jWko5dU7tPPSMnL >> > MZNgsRvIjuR80RWOJnvEVZyz45BXtFWd6UcCIG3BahAUSOXAWhqhkNP4 >> > gF6YeDsZHElhjvhWAzBA/44aFCJPT2nySKuzH4cGRulhO0remY6CHD4o >> > 59fQooYT8lopP6SWdHOmDYhdb6/UBGDELd35QwGG0MDAMSie6jZGGkeb >> > DhAuTFRWzboxlbqQw3nyYlH0Ot8lSatzhx0Cl0rNIBTboFQiWIUMgtVi PeRj0Q== >> > ;; Received 1125 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms >> > >> > ;; expected opt record in response >> > d.ns.facebook.com. 65 IN A 31.13.67.19 >> > ;; Received 51 bytes from 193.0.14.129#53(k.root-servers.net) in 231 ms >> > ``` >> > >> > Looking a bit more into it: >> > >> > Querying d.ns.facebook.com/A against k-root directly from MX probes: >> > https://atlas.ripe.net/measurements/33184386/ >> > ``` >> > $ blaeu-resolve -m 33184386 -q A d.ns.facebook.com >> > [] : 13 occurrences >> > [202.160.128.195] : 1 occurrences >> > [199.59.148.97] : 1 occurrences >> > [185.89.219.12] : 2 occurrences >> > [31.13.96.193] : 1 occurrences >> > [208.77.47.172] : 1 occurrences >> > Test #33184386 done at 2021-11-05T20:36:59Z >> > ``` >> > >> > Getting an answer in the first place is kind of unexpected but I will not >> > focus on the ones returning the correct answer (e.g 185.89.219.12). >> > >> > Checking the probes that return those results: >> > ``` >> > ripe-atlas report --renderer dns_compact 33184386 >> > ... >> > ... >> > Probe #27558: 2021-11-05 13:36:59 NOERROR qr ra rd d.ns.facebook.com. 98 A >> > 202.160.128.195 >> > Probe #31355: 2021-11-05 13:37:01 NOERROR qr ra rd d.ns.facebook.com. 146 >> > A 199.59.148.97 >> > Probe #52013: 2021-11-05 13:37:01 NOERROR qr ra rd d.ns.facebook.com. 179 >> > A 31.13.96.193 >> > Probe #52660: 2021-11-05 13:37:00 NOERROR qr ra rd d.ns.facebook.com. 150 >> > A 208.77.47.172 >> > ... >> > ``` >> > >> > Those probes will fail to reach 193.0.14.129 (k-root) over TCP. >> > >> > Checking which id.server is returned by the k-roots reached by those >> > probes: >> > >> > ``` >> > ripe-atlas measure dns --query-argument id.server --query-type TXT >> > --query-class CHAOS --from-country MX --target 193.0.14.129 >> > https://atlas.ripe.net/measurements/33184807/ >> > ``` >> > >> > where the interesting snippet is: >> > ``` >> > $ ripe-atlas report --renderer dns_compact 33184807 >> > ... >> > Probe #27558: 2021-11-05 14:08:54 NOERROR qr rd id.server. 0 TXT >> > ns1.cn-ggz.k.ripe.net >> > Probe #31355: 2021-11-05 14:08:55 NOERROR qr rd id.server. 0 TXT >> > ns1.cn-ggz.k.ripe.net >> > Probe #52013: 2021-11-05 14:08:55 NOERROR qr rd id.server. 0 TXT >> > ns1.cn-ggz.k.ripe.net >> > Probe #52660: 2021-11-05 14:08:55 NOERROR qr rd id.server. 0 TXT >> > ns1.cn-ggz.k.ripe.net >> > ... >> > ``` >> > >> > Traceroute from those probes to k-root: >> > https://atlas.ripe.net/measurements/33184963/ >> > >> > >> > Looking at the traceroutes >> > ripe-atlas report --renderer traceroute --traceroute-show-asns 33184963 >> > >> > shows that the last AS before reaching a CN AS and also the first >> > transiting AS from the probe is AS32098 >> > >> > which when checking their looking glass: https://lg.transtelco.net/ uses >> > path: >> > >> > Transtelco Inc. (AS PATH: 32098) >> > -> >> > Asia Pacific Network Information Centre (AS PATH: 4134) >> > -> >> > Not found (AS PATH: 58466) >> > -> >> > China Academy of Information and Communications Technology (AS PATH: >> > 138457) >> > -> >> > Reseaux IP Europeens Network Coordination Centre (RIPE NCC) (AS PATH: >> > 25152) >> > >> > Thanks, >> > >> > Manu >> > _______________________________________________ >> > dns-operations mailing list >> > [email protected] >> > https://lists.dns-oarc.net/mailman/listinfo/dns-operations >> >> -- >> Mark Andrews, ISC >> 1 Seymour St., Dundas Valley, NSW 2117, Australia >> PHONE: +61 2 9871 4742 INTERNET: [email protected] >>>
_______________________________________________ dns-operations mailing list [email protected] https://lists.dns-oarc.net/mailman/listinfo/dns-operations
