Learning about the internet
Hi, I was just reading about transatlantic cabling in some hopes that I would be able to find an answer as to why the latency between here in greece and Los Angeles is roughly ~250ms. This seems to be a really common thing, although I'd like to understand why and the articles on transatlantic cabling as near as I can tell indicate that I am getting screwed if anything (not enough information?) (from Los Angeles to my house) Konsole output Konsole output gw~ #mtr --report-wide xxx.access.hol.gr Start: Mon Nov 3 13:04:02 2014 HOST: gwLoss% Snt Last Avg Best Wrst StDev 1.|-- 208.79.92.65 10.0%101.5 3.6 1.2 15.5 4.6 2.|-- s7.lax.arpnetworks.com 0.0%100.8 10.9 0.8 54.2 20.7 3.|-- vlan953.car2.LosAngeles1.Level3.net 30.0%10 10.5 10.3 10.1 10.8 0.0 4.|-- ae-27-27.edge6.LosAngeles1.Level3.net 30.0%10 21.8 16.2 8.6 47.6 14.7 5.|-- ae-4-90.edge1.LosAngeles6.Level3.net 80.0%109.0 8.9 8.7 9.0 0.0 6.|-- be3036.ccr21.lax04.atlas.cogentco.com 10.0%101.7 2.1 1.4 4.3 0.7 7.|-- be2076.mpd22.lax01.atlas.cogentco.com 10.0%101.6 1.9 1.6 3.2 0.0 8.|-- be2068.ccr22.iah01.atlas.cogentco.com 0.0%10 37.7 37.7 37.3 39.0 0.3 9.|-- be2173.ccr42.atl01.atlas.cogentco.com 0.0%10 51.6 52.4 51.5 57.5 1.7 10.|-- be2171.mpd22.dca01.atlas.cogentco.com 0.0%10 62.6 62.7 62.4 63.3 0.0 11.|-- be2112.ccr41.iad02.atlas.cogentco.com 0.0%10 155.5 155.8 155.5 156.1 0.0 12.|-- be2268.ccr42.par01.atlas.cogentco.com 0.0%10 152.6 152.7 152.5 153.5 0.0 13.|-- be2278.ccr42.fra03.atlas.cogentco.com 0.0%10 155.3 155.4 155.1 155.5 0.0 14.|-- be2229.ccr22.muc01.atlas.cogentco.com 0.0%10 161.2 161.1 160.9 161.3 0.0 15.|-- be2223.ccr21.vie01.atlas.cogentco.com 0.0%10 164.9 165.1 164.9 165.2 0.0 16.|-- be2046.ccr21.sof02.atlas.cogentco.com 0.0%10 189.5 189.4 189.3 189.9 0.0 17.|-- be2118.rcr11.ath01.atlas.cogentco.com 0.0%10 197.5 197.6 197.4 197.7 0.0 18.|-- 149.11.120.38 0.0%10 202.7 202.2 200.3 204.2 1.4 19.|-- 62.38.97.113 80.0%10 208.5 209.8 208.5 211.1 1.7 20.|-- gigaeth04-13.krs00.ar.hol.gr 60.0%10 211.3 213.0 211.2 218.2 3.4 21.|-- ??? 100.0100.0 0.0 0.0 0.0 0.0 22.|-- .access.hol.gr 40.0%10 231.3 231.4 231.2 231.7 0.0 gw~ # And to be more clear: I am hoping to learn about the complex trials that these packets are going through and how time is being lost if the latency across the transatlantic cable is really capable of less the 60ms of latency? Sure over capacity (3.2Tbits/s wow jeez) is one answer, but what are some other possibilities for loss of time? Also it seems with my VPN (OpenVPN) tunnel I get the most reliable connection (fewest drops) with: Konsole output mssfix 576 fragment 576 Although this could be a false positive as it only *seems* to help with reliability since I changed it. Even then but less often than before I still experience drops but I want to believe that's possibly due to my ISP at that point.. but assuming my ISP was absolutely perfect and never a problem what else there to consider? Any and all insight is appreciated. -Paige
Re: Learning about the internet
looks about right in the neighborhood of 9k miles... from lax or therebouts. Upstream Intf Nexthop Sent LossMinAvg MaxDev cogentx x 10 0.0%194.814 210.255240.989 16.518 comcast x x 10 0.0%201.723213.942 239.47 13.205 l3x x 10 0.0% 195.51 208.189226.971 10.455 telia x x 10 0.0%194.552 207.392225.792 10.321 On 11/3/14 1:15 PM, Paige Thompson wrote: Hi, I was just reading about transatlantic cabling in some hopes that I would be able to find an answer as to why the latency between here in greece and Los Angeles is roughly ~250ms. This seems to be a really common thing, although I'd like to understand why and the articles on transatlantic cabling as near as I can tell indicate that I am getting screwed if anything (not enough information?) (from Los Angeles to my house) Konsole output Konsole output gw~ #mtr --report-wide xxx.access.hol.gr Start: Mon Nov 3 13:04:02 2014 HOST: gwLoss% Snt Last Avg Best Wrst StDev 1.|-- 208.79.92.65 10.0%101.5 3.6 1.2 15.5 4.6 2.|-- s7.lax.arpnetworks.com 0.0%100.8 10.9 0.8 54.2 20.7 3.|-- vlan953.car2.LosAngeles1.Level3.net 30.0%10 10.5 10.3 10.1 10.8 0.0 4.|-- ae-27-27.edge6.LosAngeles1.Level3.net 30.0%10 21.8 16.2 8.6 47.6 14.7 5.|-- ae-4-90.edge1.LosAngeles6.Level3.net 80.0%109.0 8.9 8.7 9.0 0.0 6.|-- be3036.ccr21.lax04.atlas.cogentco.com 10.0%101.7 2.1 1.4 4.3 0.7 7.|-- be2076.mpd22.lax01.atlas.cogentco.com 10.0%101.6 1.9 1.6 3.2 0.0 8.|-- be2068.ccr22.iah01.atlas.cogentco.com 0.0%10 37.7 37.7 37.3 39.0 0.3 9.|-- be2173.ccr42.atl01.atlas.cogentco.com 0.0%10 51.6 52.4 51.5 57.5 1.7 10.|-- be2171.mpd22.dca01.atlas.cogentco.com 0.0%10 62.6 62.7 62.4 63.3 0.0 11.|-- be2112.ccr41.iad02.atlas.cogentco.com 0.0%10 155.5 155.8 155.5 156.1 0.0 12.|-- be2268.ccr42.par01.atlas.cogentco.com 0.0%10 152.6 152.7 152.5 153.5 0.0 13.|-- be2278.ccr42.fra03.atlas.cogentco.com 0.0%10 155.3 155.4 155.1 155.5 0.0 14.|-- be2229.ccr22.muc01.atlas.cogentco.com 0.0%10 161.2 161.1 160.9 161.3 0.0 15.|-- be2223.ccr21.vie01.atlas.cogentco.com 0.0%10 164.9 165.1 164.9 165.2 0.0 16.|-- be2046.ccr21.sof02.atlas.cogentco.com 0.0%10 189.5 189.4 189.3 189.9 0.0 17.|-- be2118.rcr11.ath01.atlas.cogentco.com 0.0%10 197.5 197.6 197.4 197.7 0.0 18.|-- 149.11.120.38 0.0%10 202.7 202.2 200.3 204.2 1.4 19.|-- 62.38.97.113 80.0%10 208.5 209.8 208.5 211.1 1.7 20.|-- gigaeth04-13.krs00.ar.hol.gr 60.0%10 211.3 213.0 211.2 218.2 3.4 21.|-- ??? 100.0100.0 0.0 0.0 0.0 0.0 22.|-- .access.hol.gr 40.0%10 231.3 231.4 231.2 231.7 0.0 gw~ # And to be more clear: I am hoping to learn about the complex trials that these packets are going through and how time is being lost if the latency across the transatlantic cable is really capable of less the 60ms of latency? Sure over capacity (3.2Tbits/s wow jeez) is one answer, but what are some other possibilities for loss of time? Also it seems with my VPN (OpenVPN) tunnel I get the most reliable connection (fewest drops) with: Konsole output mssfix 576 fragment 576 Although this could be a false positive as it only *seems* to help with reliability since I changed it. Even then but less often than before I still experience drops but I want to believe that's possibly due to my ISP at that point.. but assuming my ISP was absolutely perfect and never a problem what else there to consider? Any and all insight is appreciated. -Paige signature.asc Description: OpenPGP digital signature
Re: Learning about the internet
On Mon, Nov 3, 2014 at 4:15 PM, Paige Thompson paigead...@gmail.com wrote: I would be able to find an answer as to why the latency between here in greece and Los Angeles is roughly ~250ms. 10.|-- be2171.mpd22.dca01.atlas.cogentco.com 0.0%10 62.6 62.7 62.4 63.3 0.0 11.|-- be2112.ccr41.iad02.atlas.cogentco.com 0.0%10 155.5 155.8 155.5 156.1 0.0 ^^ There's the largest part part of your answer. IAD and DCA are both Northern VA airports; there should be perhaps 4ms of latency between these locations. A more smug answer is, Cogent. Google cogent congestion, Cogent peering, and similar search terms. Other parts of the answer include: 1. The routers are mostly doing store and forward which means each must receive your entire packet before it can forward it onward to the next router. This will take more or less time depending on the link speed. 2. There are layer 2 devices doing the same thing in between some of the routers. Layer 2 is invisible to the layer 3 traceroute. There may also be MPLS systems in there with the same impact. 3. Greece and LA are more than 11,000 km apart as the crow flies. This means round-trip packets travel something like 25,000 km. There is some signal propagation delay involved. U.S. East Coast to Hawaii (about 80% of that distance) is typically 120 ms. Regards, Bill Herrin -- William Herrin her...@dirtside.com b...@herrin.us Owner, Dirtside Systems . Web: http://www.dirtside.com/ May I solve your unusual networking challenges?
Re: Learning about the internet
Hey Paige, That¹s going to be extremely common when traversing transatlantic cable. Hopefully I can help explain. Most of your latency comes from simple speed of light limitations through optical fiber (~35% slower than vacuum) combined with the fact that the fiber must take a somewhat indirect path to your destination. Additionally, a small percentage of latency will be added for all of the switching/routing/regeneration nodes that must convert the fiber signals from optical back to electrical (for processing / regeneration) then back to optical. In your case, your packets are taking the following path: Los Angeles (Arp Networks to Level3 to Cogent) - Houston - Atlanta - Washington DC - Paris - Frankfurt - Munich - Austria - Bulgaria - Athens - Hellas Essentially it¹s taking 60ms round trip to get to DC, then another 95ms to get to Paris, plus 60ms to get to Hellas, then another 20ms to traverse the local ³last mile² access network to your destination. If you had a *direct* fiber from LAX to Hellas Greece, that would be ~22,000km of round trip fiber. Working out some simple math, that would yield approximately 110ms of round trip latency no matter how much bandwidth you have. Now, add in all of the indirect fiber paths through major cities plus all of the routing/switching hops and you can see why you¹re getting 200ms+ latency. Hope this helps! Thanks, Dustin Melancon, Sr. Network Engineer 225-214-3864 Direct ::: 866-978-3698 Toll-Free VENYU : Your Data Made Invincible : www.venyu.com http://www.venyu.com/ Read our latest blog post: blog.venyu.com http://blog.venyu.com/ On 11/3/14, 3:15 PM, Paige Thompson paigead...@gmail.com wrote: Hi, I was just reading about transatlantic cabling in some hopes that I would be able to find an answer as to why the latency between here in greece and Los Angeles is roughly ~250ms. This seems to be a really common thing, although I'd like to understand why and the articles on transatlantic cabling as near as I can tell indicate that I am getting screwed if anything (not enough information?) (from Los Angeles to my house) Konsole output Konsole output gw~ #mtr --report-wide xxx.access.hol.gr Start: Mon Nov 3 13:04:02 2014 HOST: gwLoss% Snt Last Avg Best Wrst StDev 1.|-- 208.79.92.65 10.0%101.5 3.6 1.2 15.5 4.6 2.|-- s7.lax.arpnetworks.com 0.0%100.8 10.9 0.8 54.2 20.7 3.|-- vlan953.car2.LosAngeles1.Level3.net 30.0%10 10.5 10.3 10.1 10.8 0.0 4.|-- ae-27-27.edge6.LosAngeles1.Level3.net 30.0%10 21.8 16.2 8.6 47.6 14.7 5.|-- ae-4-90.edge1.LosAngeles6.Level3.net 80.0%109.0 8.9 8.7 9.0 0.0 6.|-- be3036.ccr21.lax04.atlas.cogentco.com 10.0%101.7 2.1 1.4 4.3 0.7 7.|-- be2076.mpd22.lax01.atlas.cogentco.com 10.0%101.6 1.9 1.6 3.2 0.0 8.|-- be2068.ccr22.iah01.atlas.cogentco.com 0.0%10 37.7 37.7 37.3 39.0 0.3 9.|-- be2173.ccr42.atl01.atlas.cogentco.com 0.0%10 51.6 52.4 51.5 57.5 1.7 10.|-- be2171.mpd22.dca01.atlas.cogentco.com 0.0%10 62.6 62.7 62.4 63.3 0.0 11.|-- be2112.ccr41.iad02.atlas.cogentco.com 0.0%10 155.5 155.8 155.5 156.1 0.0 12.|-- be2268.ccr42.par01.atlas.cogentco.com 0.0%10 152.6 152.7 152.5 153.5 0.0 13.|-- be2278.ccr42.fra03.atlas.cogentco.com 0.0%10 155.3 155.4 155.1 155.5 0.0 14.|-- be2229.ccr22.muc01.atlas.cogentco.com 0.0%10 161.2 161.1 160.9 161.3 0.0 15.|-- be2223.ccr21.vie01.atlas.cogentco.com 0.0%10 164.9 165.1 164.9 165.2 0.0 16.|-- be2046.ccr21.sof02.atlas.cogentco.com 0.0%10 189.5 189.4 189.3 189.9 0.0 17.|-- be2118.rcr11.ath01.atlas.cogentco.com 0.0%10 197.5 197.6 197.4 197.7 0.0 18.|-- 149.11.120.38 0.0%10 202.7 202.2 200.3 204.2 1.4 19.|-- 62.38.97.113 80.0%10 208.5 209.8 208.5 211.1 1.7 20.|-- gigaeth04-13.krs00.ar.hol.gr 60.0%10 211.3 213.0 211.2 218.2 3.4 21.|-- ??? 100.0100.0 0.0 0.0 0.0 0.0 22.|-- .access.hol.gr 40.0%10 231.3 231.4 231.2 231.7 0.0 gw~ # And to be more clear: I am hoping to learn about the complex trials that these packets are going through and how time is being lost if the latency across the transatlantic cable is really capable of less the 60ms of latency? Sure over capacity (3.2Tbits/s wow jeez) is one answer, but what are some other possibilities for loss of time? Also it seems with my VPN (OpenVPN) tunnel I get the most reliable connection (fewest drops) with: Konsole output mssfix 576 fragment 576 Although this could be a false positive as it only *seems* to help with reliability since I changed it. Even then but less often than before I still experience drops but I
Re: Learning about the internet
Isn't this most likely a side effect of MPLS tunneling and the 93ms jump there is actually the trans-atlantic segment? All the european hops following it are very close in latency to hop 11. On Mon, Nov 3, 2014 at 5:14 PM, William Herrin b...@herrin.us wrote: On Mon, Nov 3, 2014 at 4:15 PM, Paige Thompson paigead...@gmail.com wrote: I would be able to find an answer as to why the latency between here in greece and Los Angeles is roughly ~250ms. 10.|-- be2171.mpd22.dca01.atlas.cogentco.com 0.0%10 62.6 62.7 62.4 63.3 0.0 11.|-- be2112.ccr41.iad02.atlas.cogentco.com 0.0%10 155.5 155.8 155.5 156.1 0.0 ^^ There's the largest part part of your answer. IAD and DCA are both Northern VA airports; there should be perhaps 4ms of latency between these locations. -- William Herrin her...@dirtside.com b...@herrin.us Owner, Dirtside Systems . Web: http://www.dirtside.com/ May I solve your unusual networking challenges?
RE: Learning about the internet
Yeap, that's what I was typing up. More than likely what we're seeing is just the far of the link from IAD to Paris. This is evident from the fact that the next hop to Frankfurt is only an additional 3ms beyond what is reported as IAD. -Original Message- From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Brian Loveland Sent: Monday, November 3, 2014 4:33 PM To: William Herrin Cc: nanog@nanog.org Subject: Re: Learning about the internet Isn't this most likely a side effect of MPLS tunneling and the 93ms jump there is actually the trans-atlantic segment? All the european hops following it are very close in latency to hop 11. On Mon, Nov 3, 2014 at 5:14 PM, William Herrin b...@herrin.us wrote: On Mon, Nov 3, 2014 at 4:15 PM, Paige Thompson paigead...@gmail.com wrote: I would be able to find an answer as to why the latency between here in greece and Los Angeles is roughly ~250ms. 10.|-- be2171.mpd22.dca01.atlas.cogentco.com 0.0%10 62.6 62.7 62.4 63.3 0.0 11.|-- be2112.ccr41.iad02.atlas.cogentco.com 0.0%10 155.5 155.8 155.5 156.1 0.0 ^^ There's the largest part part of your answer. IAD and DCA are both Northern VA airports; there should be perhaps 4ms of latency between these locations. -- William Herrin her...@dirtside.com b...@herrin.us Owner, Dirtside Systems . Web: http://www.dirtside.com/ May I solve your unusual networking challenges?
