Re: verizon fios, northeast, routing issues?
alter.net is just the legacy RDNS for things in AS701 (uunet). Nothing weird there. https://en.wikipedia.org/wiki/UUNET On Sat, Oct 9, 2021 at 1:46 PM Miles Fidelman wrote: > Any Verizon folks here? > > I've been having some rather weird network issues lately - just reading > email via IMAP, from home. Over a 1gig FIOS connection to a machine in > a nearby Tierpoint data center that has LOTS of good connectivity. > > I just tried some traceroutes, and got some interesting results: > > These originate on a machine connected to a 1gig FIOS feed, and end at a > machine, located in a Tierpoint datacenter, about 10 miles from here. > > traceroute to ntcorp.com (207.154.13.58), 64 hops max, 52 byte packets > 1 * fios_quantum_gateway (192.168.1.1) 3.530 ms 2.822 ms > 2 * * * > 3 100.41.27.110 (100.41.27.110) 14.970 ms 5.323 ms 6.306 ms > 4 0.csi1.bstnmafr-mse01-bb-su1.alter.net (140.222.10.32) 11.069 ms > 8.477 ms 17.097 ms > 5 * * * > 6 0.ae1.br1.bos30.alter.net (140.222.236.253) 17.121 ms 19.027 ms > 0.ae2.br1.bos30.alter.net (140.222.236.255) 19.795 ms > 7 * * * > 8 colo4-dalla.bear1.boston1.level3.net (4.53.61.86) 2205.648 ms > 8.331 ms 13.161 ms > 9 static-33-65-203-66.axsne.net (66.203.65.33) 16.951 ms 13.791 ms > static-145-65-203-66.axsne.net (66.203.65.145) 21.503 ms > 10 server1.ntcorp.com (207.154.13.58) 17.872 ms 15.902 ms 14.415 ms > > Several things jump out: > > 1. alter.net is not a common path between here & there - usually a lower > grade connection, when other backbones aren't working right > > 2. origin - alter.net - level.3 - endpoint is just bizarre, one would > think that the regional FIOS network has a direct connection to level.3 > (it also seems kind of odd that the packets are flowing from Acton MA, > to Boston, and back out to Marlboro MA - there's an awful lot of fiber > running along Rt. 495, and the networks are fairly dense around here) > > 3. The intermittent, high delays (factor of 10) jump out (also, when > running ping tests, there seem to be intermittent periods of long > sequences of timeouts) > > All in all it's really mucking with both streaming services, and simply > posting emails (SMTP timeouts). > > All of which leads me to wonder if there's something mucked up with > Verizon's routing tables (or a particular network interface). > > Any insights (or fixes) to be had? > > Thanks, > > Miles Fidelman > > > > -- > In theory, there is no difference between theory and practice. > In practice, there is. Yogi Berra > > Theory is when you know everything but nothing works. > Practice is when everything works but no one knows why. > In our lab, theory and practice are combined: > nothing works and no one knows why. ... unknown > >
Re: verizon fios, northeast, routing issues?
> On Sat, Oct 9, 2021, 13:45 Miles Fidelman > > > 2. origin - alter.net - level.3 - endpoint is just bizarre, one would > think that the regional FIOS network has a direct connection to level.3 No. Former verizon-gni backbone (where FiOS sits) takes transit solely from VZB (now UUNET), this has been the case for many many years, but most people get confused when interpretting traceroutes due to mpls no-ttl-propagate on VZ networks. > (it also seems kind of odd that the packets are flowing from Acton MA, > to Boston, and back out to Marlboro MA - there's an awful lot of fiber > running along Rt. 495, and the networks are fairly dense around here) Level3-VZB-px is at 300 Bent Street, Cambridge, MA, where both parties have peering routers installed (bear1.Boston1.Level3.net and BR1.BOS30.ALTER.NET). It makes perfect sense for interconnection to occur in Cambridge/Boston, rather than maintaining peering routers out in the suburb within the same metro. None of these are contributory factors to the issues you're describing. James
Re: verizon fios, northeast, routing issues?
On Sat, Oct 9, 2021, 13:45 Miles Fidelman wrote: > Any Verizon folks here? > > > > I've been having some rather weird network issues lately - just reading > email via IMAP, from home. Over a 1gig FIOS connection to a machine in > a nearby Tierpoint data center that has LOTS of good connectivity. > > I just tried some traceroutes, and got some interesting results: > > These originate on a machine connected to a 1gig FIOS feed, and end at a > machine, located in a Tierpoint datacenter, about 10 miles from here. > > traceroute to ntcorp.com (207.154.13.58), 64 hops max, 52 byte packets > 1 * fios_quantum_gateway (192.168.1.1) 3.530 ms 2.822 ms > 2 * * * > 3 100.41.27.110 (100.41.27.110) 14.970 ms 5.323 ms 6.306 ms > 4 0.csi1.bstnmafr-mse01-bb-su1.alter.net (140.222.10.32) 11.069 ms > 8.477 ms 17.097 ms > 5 * * * > 6 0.ae1.br1.bos30.alter.net (140.222.236.253) 17.121 ms 19.027 ms > 0.ae2.br1.bos30.alter.net (140.222.236.255) 19.795 ms > 7 * * * > 8 colo4-dalla.bear1.boston1.level3.net (4.53.61.86) 2205.648 ms > 8.331 ms 13.161 ms > 9 static-33-65-203-66.axsne.net (66.203.65.33) 16.951 ms 13.791 ms > static-145-65-203-66.axsne.net (66.203.65.145) 21.503 ms > 10 server1.ntcorp.com (207.154.13.58) 17.872 ms 15.902 ms 14.415 ms > > Several things jump out: > > 1. alter.net is not a common path between here & there - usually a lower > grade connection, when other backbones aren't working right > > > Alternet is the domain used by legacy uunet equipment/ips, or was that > domain "forever". > > > > 2. origin - alter.net - level.3 - endpoint is just bizarre, one would > > Why? "Br" is the role name used for sfp peer interconnect devices on > uunet's network. > > think that the regional FIOS network has a direct connection to level.3 > (it also seems kind of odd that the packets are flowing from Acton MA, > to Boston, and back out to Marlboro MA - there's an awful lot of fiber > running along Rt. 495, and the networks are fairly dense around here) > > 3. The intermittent, high delays (factor of 10) jump out (also, when > running ping tests, there seem to be intermittent periods of long > sequences of timeouts) > > All in all it's really mucking with both streaming services, and simply > posting emails (SMTP timeouts). > > All of which leads me to wonder if there's something mucked up with > Verizon's routing tables (or a particular network interface). > > Any insights (or fixes) to be had? > > Thanks, > > Miles Fidelman > > > > -- > In theory, there is no difference between theory and practice. > In practice, there is. Yogi Berra > > Theory is when you know everything but nothing works. > Practice is when everything works but no one knows why. > In our lab, theory and practice are combined: > nothing works and no one knows why. ... unknown > >
verizon fios, northeast, routing issues?
Any Verizon folks here? I've been having some rather weird network issues lately - just reading email via IMAP, from home. Over a 1gig FIOS connection to a machine in a nearby Tierpoint data center that has LOTS of good connectivity. I just tried some traceroutes, and got some interesting results: These originate on a machine connected to a 1gig FIOS feed, and end at a machine, located in a Tierpoint datacenter, about 10 miles from here. traceroute to ntcorp.com (207.154.13.58), 64 hops max, 52 byte packets 1 * fios_quantum_gateway (192.168.1.1) 3.530 ms 2.822 ms 2 * * * 3 100.41.27.110 (100.41.27.110) 14.970 ms 5.323 ms 6.306 ms 4 0.csi1.bstnmafr-mse01-bb-su1.alter.net (140.222.10.32) 11.069 ms 8.477 ms 17.097 ms 5 * * * 6 0.ae1.br1.bos30.alter.net (140.222.236.253) 17.121 ms 19.027 ms 0.ae2.br1.bos30.alter.net (140.222.236.255) 19.795 ms 7 * * * 8 colo4-dalla.bear1.boston1.level3.net (4.53.61.86) 2205.648 ms 8.331 ms 13.161 ms 9 static-33-65-203-66.axsne.net (66.203.65.33) 16.951 ms 13.791 ms static-145-65-203-66.axsne.net (66.203.65.145) 21.503 ms 10 server1.ntcorp.com (207.154.13.58) 17.872 ms 15.902 ms 14.415 ms Several things jump out: 1. alter.net is not a common path between here & there - usually a lower grade connection, when other backbones aren't working right 2. origin - alter.net - level.3 - endpoint is just bizarre, one would think that the regional FIOS network has a direct connection to level.3 (it also seems kind of odd that the packets are flowing from Acton MA, to Boston, and back out to Marlboro MA - there's an awful lot of fiber running along Rt. 495, and the networks are fairly dense around here) 3. The intermittent, high delays (factor of 10) jump out (also, when running ping tests, there seem to be intermittent periods of long sequences of timeouts) All in all it's really mucking with both streaming services, and simply posting emails (SMTP timeouts). All of which leads me to wonder if there's something mucked up with Verizon's routing tables (or a particular network interface). Any insights (or fixes) to be had? Thanks, Miles Fidelman -- In theory, there is no difference between theory and practice. In practice, there is. Yogi Berra Theory is when you know everything but nothing works. Practice is when everything works but no one knows why. In our lab, theory and practice are combined: nothing works and no one knows why. ... unknown
Re: DNS pulling BGP routes?
Bill Woodcock wrote: It may be that facebook uses all the four name server IP addresses in each edge node. But, it effectively kills essential redundancy of DNS to have two or more name servers (at separate locations) and the natural consequence is, as you can see, mass disaster. Yep. I think we even had a NANOG talk on exactly that specific topic a long time ago. https://www.pch.net/resources/Papers/dns-service-architecture/dns-service-architecture-v10.pdf Yes, having separate sets of anycast addresses by two or more pops should be fine. However, if CDN provider has their own transit backbone, which is, seemingly, not assumed by your slides, and retail ISPs are tightly connected to only one pop of the CDN provider, the CDN provider may be motivated to let users access only one pop killing essential redundancy of DNS, which should be overengineering, which is my concern of the paragraph quoted by you. Masataka Ohta
Re: FYI: NANOG and ICANN
On Fri, Oct 8, 2021 at 4:47 PM Owen DeLong via NANOG wrote: > > > On Oct 8, 2021, at 12:39 PM, Warren Kumari wrote: > > > > On Fri, Oct 8, 2021 at 2:39 PM Owen DeLong via NANOG > wrote: > >> I see this as a way to allow NANOG to help channel some of ICANN’s >> incredible excess of funding >> towards more useful pursuits than those ICANN has endowed so far. >> > > > https://www.icann.org/en/announcements/details/icann-and-first-sign-memorandum-of-understanding-on-dns-threats-mitigation-22-5-2020-en > > > https://www.icann.org/en/announcements/details/icann-signs-memorandum-of-understanding-with-the-georgian-national-communications-commission-7-12-2020-en > > > https://www.icann.org/en/announcements/details/gsma-and-icann-sign-memorandum-of-understanding-at-gsma-mobile-world-congress-28-2-2018-en > > > https://www.icann.org/en/announcements/details/icann-signs-memorandum-of-understanding-with-the-global-cyber-alliance-16-6-2020-en > > I suspect that if you think that this will make it rain, you will be sadly > disappointed... > > > I don’t expect it will do any good at all. I hope that it will be slightly > less damaging than the things ICANN > usually spends money on. > I had some time on my hands early this morning and did a close reading. There'a typo in Article 3 Section 1. Seems like it was more about the education program than anything. I trust NANOG to be less destructive than ICANN and as near as I can tell, > this partnership is mostly ICANN funding > and NANOG doing. > See Article 4. *The Parties agree to use their own funds or financial resources to fulfill their respective responsibilities under this MoU*. This MoU shall not cause any financial obligations on any one of the Parties hereto as a result of enforcing any of its rights or executing any of its obligations hereunder. It's a nice letter from ICANN supporting the education program for the most part. Warm regards, -M<
Re: DNS pulling BGP routes?
> On Oct 9, 2021, at 10:37 AM, Masataka Ohta > wrote: > It may be that facebook uses all the four name server IP addresses > in each edge node. But, it effectively kills essential redundancy > of DNS to have two or more name servers (at separate locations) > and the natural consequence is, as you can see, mass disaster. Yep. I think we even had a NANOG talk on exactly that specific topic a long time ago. https://www.pch.net/resources/Papers/dns-service-architecture/dns-service-architecture-v10.pdf -Bill signature.asc Description: Message signed with OpenPGP
Re: DNS pulling BGP routes?
On Fri, Oct 8, 2021 at 10:04 AM Christopher Morrow wrote: > On Fri, Oct 8, 2021 at 10:22 AM Masataka Ohta > wrote: >> The end result was that our DNS servers became unreachable >> even though they were still operational. >> >> means their DNS servers were serving the zone, even after >> they recognize their zone data were too old, that is, expired. > > that's not what this means. Give it up man. Masataka knows more about how Facebook implemented DNS than people who actually worked there. He will tell them (and us) what their public statements really mean. Regards, Bill Herrin -- William Herrin b...@herrin.us https://bill.herrin.us/
Re: DNS pulling BGP routes?
Christopher Morrow wrote: means their DNS servers were serving the zone, even after they recognize their zone data were too old, that is, expired. that's not what this means. I think Mr. Petach previously described this, He wrote: So, the idea is that if the edge CDN node loses connectivity to the core datacenters, the DNS servers should stop answering queries for A records with the local CDN node's address, and let a different site respond back to the client's DNS request. which may be performed by standard DNS with short expire period, after which name servers will return SERVFAIL and other name servers in other edge node with different IP addresses are tried. It may be that facebook uses all the four name server IP addresses in each edge node. But, it effectively kills essential redundancy of DNS to have two or more name servers (at separate locations) and the natural consequence is, as you can see, mass disaster. but: 1) dns server in pop serves some content (ttls aren't important right now) You MUST distinguish TTL and EXPIRE. They are different. > there's not a lot of magic here... and it's not about the zone data > really at all. Statement of Petach: "the edge CDN node loses connectivity to the core datacenters, the DNS servers should stop answering" means, with DNS terminology, zone data is expired, which has nothing to do with TTL. Masataka Ohta
