Re: Long hops on international paths

[Saku Ytti]

1) all (meaning all hitting the zayo.telia) your traceroutes originate
from University in Chicago
2) the zayo.telia device is physically close to the university
3) we should expect physically close-by backbone device to be present
in disproportionate amount of traceroutes
4) almost certainly zayo.telia is imposing the MPLS label of TTL 255,
_NOT_ copying IP TTL, therefore until MPLS label is popped, TTL is not
expiring. I.e. you are seeing ingressPE and egress PE ot Telia, you
are not seeing any P routers.

This is not esoteric knowledge, but a fairly basic Internet concept. I
am worried you are missing too much context to produce actionable
output from your work. It might be interesting to see your curriculum,
why this confusion arose, why it seems logical that the reason must be
that almost all waves are terminated there, because it would not seem
logical for people practising in the field who have even cursory
understanding, this implies problems in the curriculum.

On Tue, 18 Jan 2022 at 07:21, PAUL R BARFORD  wrote:
>
> Please find the examples for the case of Telia below.
>
> FROM jfk-us (jfk-us.team-probing.c008820.20201002.warts.gz)
>
>
>
> traceroute from 216.66.30.102 (Ark probe hosted in New York City, NY, US. No 
> AS info found) to 223.114.235.32 (MAXMIXD: Turpan, CN)
>
> 1  216.66.30.101  0.365 ms
>
> 2  62.115.49.173  3.182 ms
>
> 3  *
>
> 4  62.115.137.59  17.453 ms [x] (chi-b23-link.ip.twelve99.net., CAIDA-GEOLOC 
> -> Chicago, IL, US)
>
> 5  62.115.117.48  59.921 ms [x] (sea-b2-link.ip.twelve99.net., RIPE-IPMAP -> 
> Seattle, WA, US)
>
> 6  62.115.171.221  69.993 ms
>
> 7  223.120.6.53  69.378 ms
>
> 8  223.120.12.34  226.225 ms
>
> 9  221.183.55.110  237.475 ms
>
> 10  221.183.25.201  238.697 ms
>
> 11  221.176.16.213  242.296 ms
>
> 12  221.183.36.62  352.695 ms
>
> 13  221.183.39.2  300.166 ms
>
> 14  117.191.8.118  316.270 ms
>
> 15  *
>
> 16  *
>
> 17  *
>
> 18  *
>
> 19  *
>
>
>
>
>
> FROM ord-us (ord-us.team-probing.c008820.20201002.warts.gz)
>
>
>
> traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at 
> Depaul University-AS20120) to 109.25.215.237 (237.215.25.109.rev.sfr.net., 
> MAXMIXD: La Crau, FR)
>
> 1  140.192.218.129  0.795 ms
>
> 2  140.192.9.124  0.603 ms
>
> 3  64.124.44.158  1.099 ms
>
> 4  64.125.31.172  3.047 ms
>
> 5  *
>
> 6  64.125.15.65  1.895 ms  [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., 
> CAIDA-GEOLOC -> Chicago, IL, US)
>
> 7  62.115.118.59  99.242 ms[x] (prs-b3-link.ip.twelve99.net., 
> CAIDA-GEOLOC -> Paris, FR)
>
> 8  62.115.154.23  105.214 ms
>
> 9  77.136.10.6  119.021 ms
>
> 10  77.136.10.6  118.830 ms
>
> 11  80.118.89.202  118.690 ms
>
> 12  80.118.89.234  118.986 ms
>
> 13  109.24.108.66  119.159 ms
>
> 14  109.25.215.237  126.085 ms
>
>
>
>
>
> traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at 
> Depaul University-AS20120) to 84.249.89.93 
> (dsl-tkubng12-54f959-93.dhcp.inet.fi., MAXMIXD: Turku, FI)
>
> 1  140.192.218.129  0.243 ms
>
> 2  140.192.9.124  0.326 ms
>
> 3  64.124.44.158  0.600 ms
>
> 4  *
>
> 5  *
>
> 6  64.125.15.65  1.792 ms  [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., 
> CAIDA-GEOLOC -> Chicago, IL, US)
>
> 7  62.115.123.27  121.199 ms   [x] (hls-b4-link.ip.twelve99.net., 
> CAIDA-GEOLOC -> Helsinki, FI)
>
> 8  *
>
> 9  141.208.193.190  127.723 ms
>
> 10  84.249.89.93  139.051 ms
>
>
>
>
>
> traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US) to 
> 193.28.231.50 (MAXMIXD: None, HU)
>
> 1  140.192.218.129  0.240 ms
>
> 2  140.192.9.124  0.333 ms
>
> 3  64.124.44.158  0.648 ms
>
> 4  *
>
> 5  64.125.25.75  0.752 ms
>
> 6  64.125.15.65  1.877 ms  [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., 
> CAIDA-GEOLOC -> Chicago, IL, US)
>
> 7  62.115.119.39  123.952 ms   [x] (bpt-b2-link.ip.twelve99.net., **I suspect 
> it is in Budapest, HU**)
>
> 8  62.115.39.122  117.171 ms
>
> 9  88.151.96.148  117.202 ms
>
> 10  88.151.96.213  124.787 ms
>
> 11  *
>
> 12  *
>
> 13  *
>
> 14  *
>
> 15  *
>
>
>
>
>
> traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at 
> Depaul University-AS20120) to 152.195.4.11 (MAXMIXD: Los Angeles, CA, US)
>
> 1  140.192.218.129  0.224 ms
>
> 2  140.192.9.124  0.545 ms
>
> 3  64.124.44.158  0.640 ms
>
> 4  *
>
> 5  *
>
> 6  64.125.15.65  1.786 ms  [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., 
> CAIDA-GEOLOC -> Chicago, IL, US)
>
> 7  62.115.118.247  54.597 ms   [x] (las-b22-link.ip.twelve99.net., 
> CAIDA-GEOLOC -> Los Angeles, CA, US)
>
> 8  62.115.11.129  55.979 ms
>
> 9  *
>
> 10  *
>
> 11  *
>
> 12  *
>
> 13  *
>
>
>
>
>
> traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at 
> Depaul University-AS20120) to 47.31.143.217 (MAXMIXD: Delhi, IN)
>
> 1  140.192.218.129  2.277 ms
>
> 2  140.192.9.124  0.449 ms
>
> 3  64.124.44.158  0.576 ms
>
> 4  *
>
> 5  *
>
> 6  64.125.15.65  1.814 ms  [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., 
> CAIDA-GEOLOC -> Chicago, IL, US)
>
> 7  62.115.114.41  210.056 ms   [x] 

Re: Long hops on international paths

[Christopher Morrow]

Looking at your 1 repeat ORD example:

On Tue, Jan 18, 2022 at 12:17 AM PAUL R BARFORD  wrote:

> 6  64.125.15.65  1.895 ms  [x] (zayo.telia.ter1.ord7.us.zip.zayo.com.,
> CAIDA-GEOLOC -> Chicago, IL, US)
>
> 7  62.115.118.59  99.242 ms[x] (prs-b3-link.ip.twelve99.net.,
> CAIDA-GEOLOC -> Paris, FR)
>
>
>
 65.15.125.64.in-addr.arpa name = zayo.telia.ter1.ord7.us.zip.zayo.com.
64.15.125.64.in-addr.arpa name = ae51.zayo.ter1.ord7.us.zip.zayo.com.

it looks like the probes you selected (at least the depaul univ one(s)) are
finding the 'best path' to whatever destinations via depaul -> zayo ->
telia.
It looks like zayo/telia interconnect at that /31.
Based on:

https://www.teliacarrier.com/dam/jcr:fc260a69-98a2-47d3-8d30-ca7095318413/telia-carrier-map-america-nov-2021.png

i'd guess that:
 1) telia has an mpls core with no-decrement-ttl enabled
 2) the hidden hosp include NYC and possibly cleveland/wdc
 3) judging the path information purely on traceroute hops is error prone.

Re: Long hops on international paths

[PAUL R BARFORD]

Please find the examples for the case of Telia below.


FROM jfk-us (jfk-us.team-probing.c008820.20201002.warts.gz)



traceroute from 216.66.30.102 (Ark probe hosted in New York City, NY, US. No AS 
info found) to 223.114.235.32 (MAXMIXD: Turpan, CN)

1  216.66.30.101  0.365 ms

2  62.115.49.173  3.182 ms

3  *

4  62.115.137.59  17.453 ms [x] (chi-b23-link.ip.twelve99.net., CAIDA-GEOLOC -> 
Chicago, IL, US)

5  62.115.117.48  59.921 ms [x] (sea-b2-link.ip.twelve99.net., RIPE-IPMAP -> 
Seattle, WA, US)

6  62.115.171.221  69.993 ms

7  223.120.6.53  69.378 ms

8  223.120.12.34  226.225 ms

9  221.183.55.110  237.475 ms

10  221.183.25.201  238.697 ms

11  221.176.16.213  242.296 ms

12  221.183.36.62  352.695 ms

13  221.183.39.2  300.166 ms

14  117.191.8.118  316.270 ms

15  *

16  *

17  *

18  *

19  *





FROM ord-us (ord-us.team-probing.c008820.20201002.warts.gz)



traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul 
University-AS20120) to 109.25.215.237 (237.215.25.109.rev.sfr.net., MAXMIXD: La 
Crau, FR)

1  140.192.218.129  0.795 ms

2  140.192.9.124  0.603 ms

3  64.124.44.158  1.099 ms

4  64.125.31.172  3.047 ms

5  *

6  64.125.15.65  1.895 ms  [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., 
CAIDA-GEOLOC -> Chicago, IL, US)

7  62.115.118.59  99.242 ms[x] (prs-b3-link.ip.twelve99.net., CAIDA-GEOLOC 
-> Paris, FR)

8  62.115.154.23  105.214 ms

9  77.136.10.6  119.021 ms

10  77.136.10.6  118.830 ms

11  80.118.89.202  118.690 ms

12  80.118.89.234  118.986 ms

13  109.24.108.66  119.159 ms

14  109.25.215.237  126.085 ms





traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul 
University-AS20120) to 84.249.89.93 (dsl-tkubng12-54f959-93.dhcp.inet.fi., 
MAXMIXD: Turku, FI)

1  140.192.218.129  0.243 ms

2  140.192.9.124  0.326 ms

3  64.124.44.158  0.600 ms

4  *

5  *

6  64.125.15.65  1.792 ms  [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., 
CAIDA-GEOLOC -> Chicago, IL, US)

7  62.115.123.27  121.199 ms   [x] (hls-b4-link.ip.twelve99.net., CAIDA-GEOLOC 
-> Helsinki, FI)

8  *

9  141.208.193.190  127.723 ms

10  84.249.89.93  139.051 ms





traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US) to 
193.28.231.50 (MAXMIXD: None, HU)

1  140.192.218.129  0.240 ms

2  140.192.9.124  0.333 ms

3  64.124.44.158  0.648 ms

4  *

5  64.125.25.75  0.752 ms

6  64.125.15.65  1.877 ms  [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., 
CAIDA-GEOLOC -> Chicago, IL, US)

7  62.115.119.39  123.952 ms   [x] (bpt-b2-link.ip.twelve99.net., **I suspect 
it is in Budapest, HU**)

8  62.115.39.122  117.171 ms

9  88.151.96.148  117.202 ms

10  88.151.96.213  124.787 ms

11  *

12  *

13  *

14  *

15  *





traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul 
University-AS20120) to 152.195.4.11 (MAXMIXD: Los Angeles, CA, US)

1  140.192.218.129  0.224 ms

2  140.192.9.124  0.545 ms

3  64.124.44.158  0.640 ms

4  *

5  *

6  64.125.15.65  1.786 ms  [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., 
CAIDA-GEOLOC -> Chicago, IL, US)

7  62.115.118.247  54.597 ms   [x] (las-b22-link.ip.twelve99.net., CAIDA-GEOLOC 
-> Los Angeles, CA, US)

8  62.115.11.129  55.979 ms

9  *

10  *

11  *

12  *

13  *





traceroute from 140.192.218.138 (Ark probe hosted in Chicago, IL, US at Depaul 
University-AS20120) to 47.31.143.217 (MAXMIXD: Delhi, IN)

1  140.192.218.129  2.277 ms

2  140.192.9.124  0.449 ms

3  64.124.44.158  0.576 ms

4  *

5  *

6  64.125.15.65  1.814 ms  [x] (zayo.telia.ter1.ord7.us.zip.zayo.com., 
CAIDA-GEOLOC -> Chicago, IL, US)

7  62.115.114.41  210.056 ms   [x] (snge-b5-link.ip.twelve99.net.,)

8  62.115.177.11  200.840 ms

 9  103.198.140.16  233.636 ms

10  103.198.140.16  232.871 ms

11  103.198.140.171  232.648 ms

12  *

13  *

14  *

15  *

16  *


From: Lukas Tribus 
Sent: Monday, January 17, 2022 1:52 PM
To: PAUL R BARFORD 
Cc: Nick Hilliard ; nanog@nanog.org ; Esteban 
Carisimo ; Fabian E. Bustamante 

Subject: Re: Long hops on international paths

On Mon, 17 Jan 2022 at 20:00, PAUL R BARFORD  wrote:
> What we're curious about is why we're seeing a concentration of hops at a 
> small number of routers that appear on international paths.

I suggest you share a few actual examples (IP addresses, traceroutes).

I don't think discussing your conclusion based on data we don't have
makes sense.


Lukas

Re: Long hops on international paths

[Dave Cohen]

I guess it depends what you’re considering a “very few” number of routers but 
this seems to be an expected outcome. While there are a large number of wet 
cable landing stations, they are highly concentrated near a small number of 
metro areas, and with the exception of capacity owned by the ILECs, the 
supermajority of routers terminating that capacity in the US are going to live 
in fewer than ten discrete carrier hotel locations. (It’s worth noting that 
terrestrial capacity coming in from Mexico and Canada also terminates in a 
small number of locations, although the overlap between the two lists is fairly 
small). In addition, while the links likely terminate in multiple devices at a 
given location, carriers more likely to undersubscribe transoceanic core 
capacity than other areas of the core, which means that for many carriers it’s 
unlikely you’d see multiple paths show up in a trace unless you catch it during 
an outage situation. That said, seeing transoceanic links terminate in Chicago 
is likely an artifact of hops missing in a trace; although I am familiar with a 
couple of more niche providers that extend transoceanic capacity into 
non-coastal markets on optical gear in order to meet specific performance 
needs, this is unlikely to be seen in the network of a Tier 1 or similarly 
scaled network. 

Dave Cohen
craetd...@gmail.com

> On Jan 17, 2022, at 6:15 PM, Christopher Morrow  
> wrote:
> 
> 
> 
> 
>> On Mon, Jan 17, 2022 at 5:31 PM PAUL R BARFORD  wrote:
>> Dear Pengxiong,
>> 
>> Thanks for your questions:
>> 
>> We are using CAIDA’s Internet Topology Data Kit (ITDK) that uses the MIDAR 
>> alias resolution method to infer IP addresses assigned to the same router.
>> We understand the concerns about IP geolocation.  Interfaces of the router 
>> in question are assigned similar domain names e.g., 
>> “chi-b2-link.ip.twelve99.net” (62.115.50.61). We also used CAIDA’s ITDK, 
>> which provides geolocation information, and indicates that this router is 
>> located in Chicago.  We cross-reference with Maxmind where possible.  In 
>> this particular case, there is the telltale in the use of "chi" in the 
>> domain name. 
>> 
> 
> I think nick's point about ttl expiry and missing some context on topology 
> still stands.
> I'd be that the paths between 2 continents do not actually land in chicago... 
> that you're seeing (or not seeing) missing hops between the coast(s) and 
> chicago inside 1299's network in the US.
>  
>> Hope that helps.
>> 
>> Regards, PB
>> From: Pengxiong Zhu 
>> Sent: Monday, January 17, 2022 3:23 PM
>> To: PAUL R BARFORD 
>> Cc: nanog@nanog.org 
>> Subject: Re: Long hops on international paths
>>  
>> Hi Paul,
>> 
>> Just curious. How do you determine they are the same routers? Is it based on 
>> IP address or MAC addresses? Or using CAIDA’s router alias database?
>> 
>> Also how do you draw the conclusion that the AS1299 router is indeed in 
>> Chicago? IP-geolocation based on rDNS is not always accurate though. 
>> 
>> 
>> Pengxiong 
>> 
>> On Mon, Jan 17, 2022 at 10:03 AM PAUL R BARFORD  wrote:
>> Hello,
>> 
>> I am a researcher at the University of Wisconsin.  My colleagues at 
>> Northwestern University and I are studying international Internet 
>> connectivity and would appreciate your perspective on a recent finding.
>> 
>> We're using traceroute data from CAIDA's Ark project for our work.  We've 
>> observed that many international links (i.e., a single hop on an end-to-end 
>> path that connects two countries where end points on the hop are identified 
>> via rDNS) tend to originate/terminate at the same routers.  Said another 
>> way, we are observing a relatively small set of routers in different 
>> countries tend to have a majority of the international connections - this is 
>> especially the case for hops that terminate in the US.  For example, there 
>> is a router operated by Telia (AS1299) in Chicago that has a high 
>> concentration of such links.  We were a bit surprised by this finding since 
>> even though it makes sense that the set of providers is relatively small 
>> (i.e., those that offer global connectivity), we assumed that the set of 
>> routers that used for international connectivity within any one country 
>> would tend to be more widely distributed (at least with respect to how they 
>> appear in traceroute data - MPLS notwithstanding).
>> 
>> We're interested in whether or not this is indeed standard practice and if 
>> so, the cost/benefit for configuring international connectivity in this way?
>> 
>> Any thoughts or insights you might have would be greatly appreciated - 
>> off-list responses are welcome.
>> 
>> Thank you.
>> 
>> Regards, PB
>> 
>> Paul Barford
>> University of Wisconsin - Madison
>> 
>> -- 
>> 
>> Regards,
>> Pengxiong Zhu
>> Department of Computer Science and Engineering
>> University of California, Riverside

Re: home router battery backup

[Jeff Shultz]

+180v and -180v for a total of 360v. At really low amperage. Still makes a
respectable bang if you short it on the MDF. It gets converted on-site,
either in the DSLAM or in a separate box. I think it's 12v to the ONT.

On Mon, Jan 17, 2022 at 3:30 PM Michael Thomas  wrote:

>
> On 1/17/22 2:39 PM, Jordan wrote:
> > On Thu, Jan 13, 2022 at 02:06:39PM -0800, Michael Thomas wrote:
> >> For my ISP, they maintain backup power for both DSL and POTS.  I
> >> suspect that for a lot of DSL that would hold true because it's
> >> relatively easy for them to power since they already have the
> >> battery backup requirements for POTS.  The setup they have here
> >> is a DSLAM and SIP->POTS termination in a pedestal with fiber
> >> backhaul.  They use the old copper that used to go back to the CO
> >> to power the pedestal.
> > Do you happen to know what voltage is placed across the copper pairs
> > for this purpose?  Maybe 130V like T1 span repeaters?  More?
> >
> > I used to have three POTS lines at home from BellSouth, before the
> > AT&T acquisition, with DSL on one of them, all supposedly served
> > from the same Lucent SLC.  One of these, the one originally used
> > for DSL, would always go down for both voice and data when the SLC
> > lost power-- no DC, no dialtone, no DSL, while the other two
> > remained up.  Despite several claims of a resolution, this was
> > never properly fixed, so eventually I just had them move DSL over
> > to one of the unaffected lines.
> >
> > I could never understand what failure mode would result in losing
> > just a single POTS line like this while the carrier equipment was
> > running from battery, while others remained in service.
> > Speculating, perhaps only the A or B-side was backed up, and an
> > open diode or other defect caused a single ine card to draw only
> > from the "other" source?  But, at this time (circa 2000) the remote
> > DSLAM was definitely a separate piece of equipment, right, joined
> > to a shared subscriber pair with passive splitters?
> >
>
> I have absolutely no idea, but if I had to guess it is the same voltage
> as the local loop but I suppose they could use ring voltage too.
>
> Mike, definitely not a EE
>
>

-- 
Jeff Shultz

-- 
Like us on Social Media for News, Promotions, and other information!!

   
      
      
      














_ This message 
contains confidential information and is intended only for the individual 
named. If you are not the named addressee you should not disseminate, 
distribute or copy this e-mail. Please notify the sender immediately by 
e-mail if you have received this e-mail by mistake and delete this e-mail 
from your system. E-mail transmission cannot be guaranteed to be secure or 
error-free as information could be intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain viruses. The sender therefore does 
not accept liability for any errors or omissions in the contents of this 
message, which arise as a result of e-mail transmission. _

Re: Long hops on international paths

[PAUL R BARFORD]

What we're considering specifically are consecutive (layer 3) hops as 
identified by traceroute.  Thus, TTL is decremented by 1 and no more than 1 
(i.e., we have to get full information (not *) from consecutive hops to 
consider the link).  I have asked my colleague to put together a set of 
examples.  We assume that there are multiple layer 1 and 2 links, and possibly 
layer 3 hops masked from traceroute by MPLS.  But what we're seeing in terms of 
hops exposed by traceroute make it look like a single (TTL decremented by 1) 
hop.

I'll post the examples when I get them.

PB

From: morrowc.li...@gmail.com 
Sent: Monday, January 17, 2022 5:13 PM
To: PAUL R BARFORD 
Cc: Pengxiong Zhu ; nanog@nanog.org 
Subject: Re: Long hops on international paths



On Mon, Jan 17, 2022 at 5:31 PM PAUL R BARFORD 
mailto:p...@cs.wisc.edu>> wrote:
Dear Pengxiong,

Thanks for your questions:


  1.  We are using CAIDA’s Internet Topology Data Kit (ITDK) that uses the 
MIDAR alias resolution method to infer IP addresses assigned to the same router.
  2.  We understand the concerns about IP geolocation.  Interfaces of the 
router in question are assigned similar domain names e.g., 
“chi-b2-link.ip.twelve99.net” 
(62.115.50.61). We also used CAIDA’s ITDK, which provides geolocation 
information, and indicates that this router is located in Chicago.  We 
cross-reference with Maxmind where possible.  In this particular case, there is 
the telltale in the use of "chi" in the domain name.
  3.

I think nick's point about ttl expiry and missing some context on topology 
still stands.
I'd be that the paths between 2 continents do not actually land in chicago... 
that you're seeing (or not seeing) missing hops between the coast(s) and 
chicago inside 1299's network in the US.


  1.

Hope that helps.

Regards, PB

From: Pengxiong Zhu mailto:pzhu...@ucr.edu>>
Sent: Monday, January 17, 2022 3:23 PM
To: PAUL R BARFORD mailto:p...@cs.wisc.edu>>
Cc: nanog@nanog.org 
mailto:nanog@nanog.org>>
Subject: Re: Long hops on international paths

Hi Paul,

Just curious. How do you determine they are the same routers? Is it based on IP 
address or MAC addresses? Or using CAIDA’s router alias database?

Also how do you draw the conclusion that the AS1299 router is indeed in 
Chicago? IP-geolocation based on rDNS is not always accurate though.


Pengxiong

On Mon, Jan 17, 2022 at 10:03 AM PAUL R BARFORD 
mailto:p...@cs.wisc.edu>> wrote:
Hello,

I am a researcher at the University of Wisconsin.  My colleagues at 
Northwestern University and I are studying international Internet connectivity 
and would appreciate your perspective on a recent finding.

We're using traceroute data from CAIDA's Ark project for our work.  We've 
observed that many international links (i.e., a single hop on an end-to-end 
path that connects two countries where end points on the hop are identified via 
rDNS) tend to originate/terminate at the same routers.  Said another way, we 
are observing a relatively small set of routers in different countries tend to 
have a majority of the international connections - this is especially the case 
for hops that terminate in the US.  For example, there is a router operated by 
Telia (AS1299) in Chicago that has a high concentration of such links.  We were 
a bit surprised by this finding since even though it makes sense that the set 
of providers is relatively small (i.e., those that offer global connectivity), 
we assumed that the set of routers that used for international connectivity 
within any one country would tend to be more widely distributed (at least with 
respect to how they appear in traceroute data - MPLS notwithstanding).

We're interested in whether or not this is indeed standard practice and if so, 
the cost/benefit for configuring international connectivity in this way?

Any thoughts or insights you might have would be greatly appreciated - off-list 
responses are welcome.

Thank you.

Regards, PB

Paul Barford
University of Wisconsin - Madison

--

Regards,
Pengxiong Zhu
Department of Computer Science and Engineering
University of California, Riverside

Re: home router battery backup

[Michael Thomas]

On 1/17/22 2:39 PM, Jordan wrote:

On Thu, Jan 13, 2022 at 02:06:39PM -0800, Michael Thomas wrote:

For my ISP, they maintain backup power for both DSL and POTS.  I
suspect that for a lot of DSL that would hold true because it's
relatively easy for them to power since they already have the
battery backup requirements for POTS.  The setup they have here
is a DSLAM and SIP->POTS termination in a pedestal with fiber
backhaul.  They use the old copper that used to go back to the CO
to power the pedestal.

Do you happen to know what voltage is placed across the copper pairs
for this purpose?  Maybe 130V like T1 span repeaters?  More?

I used to have three POTS lines at home from BellSouth, before the
AT&T acquisition, with DSL on one of them, all supposedly served
from the same Lucent SLC.  One of these, the one originally used
for DSL, would always go down for both voice and data when the SLC
lost power-- no DC, no dialtone, no DSL, while the other two
remained up.  Despite several claims of a resolution, this was
never properly fixed, so eventually I just had them move DSL over
to one of the unaffected lines.

I could never understand what failure mode would result in losing
just a single POTS line like this while the carrier equipment was
running from battery, while others remained in service.
Speculating, perhaps only the A or B-side was backed up, and an
open diode or other defect caused a single ine card to draw only
from the "other" source?  But, at this time (circa 2000) the remote
DSLAM was definitely a separate piece of equipment, right, joined
to a shared subscriber pair with passive splitters?



I have absolutely no idea, but if I had to guess it is the same voltage 
as the local loop but I suppose they could use ring voltage too.


Mike, definitely not a EE

Re: Long hops on international paths

[Christopher Morrow]

On Mon, Jan 17, 2022 at 5:31 PM PAUL R BARFORD  wrote:

> Dear Pengxiong,
>
> Thanks for your questions:
>
>
>1. We are using CAIDA’s Internet Topology Data Kit (ITDK) that uses
>the MIDAR alias resolution method to infer IP addresses assigned to the
>same router.
>2. We understand the concerns about IP geolocation.  Interfaces of the
>router in question are assigned similar domain names e.g., “
>chi-b2-link.ip.twelve99.net” (62.115.50.61). We also used CAIDA’s
>ITDK, which provides geolocation information, and indicates that this
>router is located in Chicago.  We cross-reference with Maxmind where
>possible.  In this particular case, there is the telltale in the use of
>"chi" in the domain name.
>3.
>
>
I think nick's point about ttl expiry and missing some context on
topology still stands.
I'd be that the paths between 2 continents do not actually land in
chicago... that you're seeing (or not seeing) missing hops between the
coast(s) and chicago inside 1299's network in the US.


>
>1.
>
> Hope that helps.
>
> Regards, PB
> --
> *From:* Pengxiong Zhu 
> *Sent:* Monday, January 17, 2022 3:23 PM
> *To:* PAUL R BARFORD 
> *Cc:* nanog@nanog.org 
> *Subject:* Re: Long hops on international paths
>
> Hi Paul,
>
> Just curious. How do you determine they are the same routers? Is it based
> on IP address or MAC addresses? Or using CAIDA’s router alias database?
>
> Also how do you draw the conclusion that the AS1299 router is indeed in
> Chicago? IP-geolocation based on rDNS is not always accurate though.
>
>
> Pengxiong
>
> On Mon, Jan 17, 2022 at 10:03 AM PAUL R BARFORD  wrote:
>
> Hello,
>
> I am a researcher at the University of Wisconsin.  My colleagues at
> Northwestern University and I are studying international Internet
> connectivity and would appreciate your perspective on a recent finding.
>
> We're using traceroute data from CAIDA's Ark project for our work.  We've 
> observed
> that many international links (i.e., a single hop on an end-to-end path
> that connects two countries where end points on the hop are identified via
> rDNS) tend to originate/terminate at the same routers.  Said another way,
> we are observing a relatively small set of routers in different countries
> tend to have a majority of the international connections - this is
> especially the case for hops that terminate in the US.  For example,
> there is a router operated by Telia (AS1299) in Chicago that has a high
> concentration of such links.  We were a bit surprised by this finding since
> even though it makes sense that the set of providers is relatively small
> (i.e., those that offer global connectivity), we assumed that the set of
> routers that used for international connectivity within any one country
> would tend to be more widely distributed (at least with respect to how they
> appear in traceroute data - MPLS notwithstanding).
>
> We're interested in whether or not this is indeed standard practice and if
> so, the cost/benefit for configuring international connectivity in this
> way?
>
> Any thoughts or insights you might have would be greatly appreciated -
> off-list responses are welcome.
>
> Thank you.
>
> Regards, PB
>
> Paul Barford
> University of Wisconsin - Madison
>
> --
>
> Regards,
> Pengxiong Zhu
> Department of Computer Science and Engineering
> University of California, Riverside
>

Re: Long hops on international paths

[PAUL R BARFORD]

Dear Pengxiong,

Thanks for your questions:


  1.  We are using CAIDA’s Internet Topology Data Kit (ITDK) that uses the 
MIDAR alias resolution method to infer IP addresses assigned to the same router.
  2.  We understand the concerns about IP geolocation.  Interfaces of the 
router in question are assigned similar domain names e.g., 
“chi-b2-link.ip.twelve99.net” (62.115.50.61). We also used CAIDA’s ITDK, which 
provides geolocation information, and indicates that this router is located in 
Chicago.  We cross-reference with Maxmind where possible.  In this particular 
case, there is the telltale in the use of "chi" in the domain name.
  3.

Hope that helps.

Regards, PB

From: Pengxiong Zhu 
Sent: Monday, January 17, 2022 3:23 PM
To: PAUL R BARFORD 
Cc: nanog@nanog.org 
Subject: Re: Long hops on international paths

Hi Paul,

Just curious. How do you determine they are the same routers? Is it based on IP 
address or MAC addresses? Or using CAIDA’s router alias database?

Also how do you draw the conclusion that the AS1299 router is indeed in 
Chicago? IP-geolocation based on rDNS is not always accurate though.


Pengxiong

On Mon, Jan 17, 2022 at 10:03 AM PAUL R BARFORD 
mailto:p...@cs.wisc.edu>> wrote:
Hello,

I am a researcher at the University of Wisconsin.  My colleagues at 
Northwestern University and I are studying international Internet connectivity 
and would appreciate your perspective on a recent finding.

We're using traceroute data from CAIDA's Ark project for our work.  We've 
observed that many international links (i.e., a single hop on an end-to-end 
path that connects two countries where end points on the hop are identified via 
rDNS) tend to originate/terminate at the same routers.  Said another way, we 
are observing a relatively small set of routers in different countries tend to 
have a majority of the international connections - this is especially the case 
for hops that terminate in the US.  For example, there is a router operated by 
Telia (AS1299) in Chicago that has a high concentration of such links.  We were 
a bit surprised by this finding since even though it makes sense that the set 
of providers is relatively small (i.e., those that offer global connectivity), 
we assumed that the set of routers that used for international connectivity 
within any one country would tend to be more widely distributed (at least with 
respect to how they appear in traceroute data - MPLS notwithstanding).

We're interested in whether or not this is indeed standard practice and if so, 
the cost/benefit for configuring international connectivity in this way?

Any thoughts or insights you might have would be greatly appreciated - off-list 
responses are welcome.

Thank you.

Regards, PB

Paul Barford
University of Wisconsin - Madison

--

Regards,
Pengxiong Zhu
Department of Computer Science and Engineering
University of California, Riverside

Re: Long hops on international paths

[Lady Benjamin Cannon of Glencoe, ASCE]

Carrier class core routers still cost half a million dollars each or (way) 
more, so it’s not uncommon for there to be 2-4 in a metro.

And there are only a few metros that have undersea cable landing stations.

We deploy a minimum of a pair of core routers everywhere, but with our 
BGP/OSPF/iBGP core your path through us generally won’t change even though 
there’s an alternative path with slightly lower route pref. (Absent loss of 
both physical path and physical alternate)

Ms. Lady Benjamin PD Cannon of Glencoe, ASCE
6x7 Networks & 6x7 Telecom, LLC 
CEO 
l...@6by7.net
"The only fully end-to-end encrypted global telecommunications company in the 
world.”

FCC License KJ6FJJ

Sent from my iPhone via RFC1149.

> On Jan 17, 2022, at 1:23 PM, Pengxiong Zhu  wrote:
> 
> 
> Hi Paul,
> 
> Just curious. How do you determine they are the same routers? Is it based on 
> IP address or MAC addresses? Or using CAIDA’s router alias database?
> 
> Also how do you draw the conclusion that the AS1299 router is indeed in 
> Chicago? IP-geolocation based on rDNS is not always accurate though. 
> 
> 
> Pengxiong 
> 
>> On Mon, Jan 17, 2022 at 10:03 AM PAUL R BARFORD  wrote:
>> Hello,
>> 
>> I am a researcher at the University of Wisconsin.  My colleagues at 
>> Northwestern University and I are studying international Internet 
>> connectivity and would appreciate your perspective on a recent finding.
>> 
>> We're using traceroute data from CAIDA's Ark project for our work.  We've 
>> observed that many international links (i.e., a single hop on an end-to-end 
>> path that connects two countries where end points on the hop are identified 
>> via rDNS) tend to originate/terminate at the same routers.  Said another 
>> way, we are observing a relatively small set of routers in different 
>> countries tend to have a majority of the international connections - this is 
>> especially the case for hops that terminate in the US.  For example, there 
>> is a router operated by Telia (AS1299) in Chicago that has a high 
>> concentration of such links.  We were a bit surprised by this finding since 
>> even though it makes sense that the set of providers is relatively small 
>> (i.e., those that offer global connectivity), we assumed that the set of 
>> routers that used for international connectivity within any one country 
>> would tend to be more widely distributed (at least with respect to how they 
>> appear in traceroute data - MPLS notwithstanding).
>> 
>> We're interested in whether or not this is indeed standard practice and if 
>> so, the cost/benefit for configuring international connectivity in this way?
>> 
>> Any thoughts or insights you might have would be greatly appreciated - 
>> off-list responses are welcome.
>> 
>> Thank you.
>> 
>> Regards, PB
>> 
>> Paul Barford
>> University of Wisconsin - Madison
>> 
> -- 
> 
> Regards,
> Pengxiong Zhu
> Department of Computer Science and Engineering
> University of California, Riverside

Re: home router battery backup

[Grant Taylor via NANOG]

On 1/17/22 2:24 PM, Aaron C. de Bruyn via NANOG wrote:
My "small" (< ~5,000 customers) ISP won't uncheck that box for me no 
matter how much I beg, plead, or offer to bring them snacks for their 
office.


Chuckle.

They keep mumbling stuff about FCC requirements which I suspect is just 
handwaving.  Oh well...it's on a generator-protected outlet now.


I've been known to find a crack in such armor when I start asking for 
information on the regulation so that I can read up on it and learn ~> 
understand it.  If it's real, they can usually get it to me in a week or 
so.  If it's fake, I get crickets until I push and escalate for the 
information, at which point I find out "it's company policy".


Policy can be a double edge sword, especially when you find an example 
of them violating their own policy.  }:-)




--
Grant. . . .
unix || die



smime.p7s
Description: S/MIME Cryptographic Signature

Re: home router battery backup

[Aaron C. de Bruyn via NANOG]

On Mon, Jan 17, 2022 at 11:43 AM Jeff Shultz  wrote:

> BTW, Calix ONTs default to "Disable on battery = on" for the GigE ports -
> it's checkbox in the config to turn that off so they stay up when the power
> is out. Which we do uncheck. Particularly since we've going increasingly
> VOIP and our employees can connect remotely. Sadly, I suspect that trying
> to get a major telco to go in and uncheck that box for you would be the
> equivalent to talking to a wall.
>

My "small" (< ~5,000 customers) ISP won't uncheck that box for me no matter
how much I beg, plead, or offer to bring them snacks for their office.
They keep mumbling stuff about FCC requirements which I suspect is just
handwaving.  Oh well...it's on a generator-protected outlet now.

-A

Re: Long hops on international paths

[Pengxiong Zhu]

Hi Paul,

Just curious. How do you determine they are the same routers? Is it based
on IP address or MAC addresses? Or using CAIDA’s router alias database?

Also how do you draw the conclusion that the AS1299 router is indeed in
Chicago? IP-geolocation based on rDNS is not always accurate though.


Pengxiong

On Mon, Jan 17, 2022 at 10:03 AM PAUL R BARFORD  wrote:

> Hello,
>
> I am a researcher at the University of Wisconsin.  My colleagues at
> Northwestern University and I are studying international Internet
> connectivity and would appreciate your perspective on a recent finding.
>
> We're using traceroute data from CAIDA's Ark project for our work.  We've 
> observed
> that many international links (i.e., a single hop on an end-to-end path
> that connects two countries where end points on the hop are identified via
> rDNS) tend to originate/terminate at the same routers.  Said another way,
> we are observing a relatively small set of routers in different countries
> tend to have a majority of the international connections - this is
> especially the case for hops that terminate in the US.  For example,
> there is a router operated by Telia (AS1299) in Chicago that has a high
> concentration of such links.  We were a bit surprised by this finding since
> even though it makes sense that the set of providers is relatively small
> (i.e., those that offer global connectivity), we assumed that the set of
> routers that used for international connectivity within any one country
> would tend to be more widely distributed (at least with respect to how they
> appear in traceroute data - MPLS notwithstanding).
>
> We're interested in whether or not this is indeed standard practice and if
> so, the cost/benefit for configuring international connectivity in this
> way?
>
> Any thoughts or insights you might have would be greatly appreciated -
> off-list responses are welcome.
>
> Thank you.
>
> Regards, PB
>
> Paul Barford
> University of Wisconsin - Madison
>
> --

Regards,
Pengxiong Zhu
Department of Computer Science and Engineering
University of California, Riverside

Re: Long hops on international paths

[Lukas Tribus]

On Mon, 17 Jan 2022 at 20:00, PAUL R BARFORD  wrote:
> What we're curious about is why we're seeing a concentration of hops at a 
> small number of routers that appear on international paths.

I suggest you share a few actual examples (IP addresses, traceroutes).

I don't think discussing your conclusion based on data we don't have
makes sense.


Lukas

Re: home router battery backup

[Jeff Shultz]

As one of those Telco/ISP's, it's growing more and more likely that
DSL/POTS are now on the same card and they are all tied into the 48V
battery and generator protected plant. And Alpha Electronics is probably
selling a lot of those Power over Copper systems for powering remote Calix
E3-12C and E3-48 DSLAMs (as well as their competitor's equivalents), as
well as powering the ONT in any building with more than one dwelling unit
connected to it.

BTW, Calix ONTs default to "Disable on battery = on" for the GigE ports -
it's checkbox in the config to turn that off so they stay up when the power
is out. Which we do uncheck. Particularly since we've going increasingly
VOIP and our employees can connect remotely. Sadly, I suspect that trying
to get a major telco to go in and uncheck that box for you would be the
equivalent to talking to a wall.

As to the original poster's request, after wildfires, ice storms, and wind
storms of the past year and a half, and lumberjacks clearing trees damaged
by those events and dropping them on the power lines since then, more and
more of our customers are investing in backup power solutions, even if it
is just a UPS to level out the brownouts. But it still is probably not a
significant percentage of the total.

On Thu, Jan 13, 2022 at 2:08 PM Michael Thomas  wrote:

>
> On 1/12/22 3:11 PM, Scott T Anderson via NANOG wrote:
>
> Hi everyone,
>
>
>
> Thanks very much for all the responses throughout the day. They are very
> helpful. Your (collective) answers triggered a couple follow-on questions:
>
>
>
> For those individuals with backup battery power for their modem/router, do
> they maintain Internet access throughout a power outage (as long as their
> backup power solution works)? I.e., does the rest of the ISP network
> maintain service throughout a power outage?
>
> For my ISP, they maintain backup power for both DSL and POTS. I suspect
> that for a lot of DSL that would hold true because it's relatively easy for
> them to power since they already have the battery backup requirements for
> POTS. The setup they have here is a DSLAM and SIP->POTS termination in a
> pedestal with fiber backhaul. They use the old copper that used to go back
> to the CO to power the pedestal.
>
> Mike
>
>

-- 
Jeff Shultz

-- 
Like us on Social Media for News, Promotions, and other information!!

   
      
      
      














_ This message 
contains confidential information and is intended only for the individual 
named. If you are not the named addressee you should not disseminate, 
distribute or copy this e-mail. Please notify the sender immediately by 
e-mail if you have received this e-mail by mistake and delete this e-mail 
from your system. E-mail transmission cannot be guaranteed to be secure or 
error-free as information could be intercepted, corrupted, lost, destroyed, 
arrive late or incomplete, or contain viruses. The sender therefore does 
not accept liability for any errors or omissions in the contents of this 
message, which arise as a result of e-mail transmission. _

Re: Long hops on international paths

[PAUL R BARFORD]

Hello Nick,

I've added my collaborators to this reply - Esteban can comment on your 
observation re. Telia.

TTL is​ decremented on the paths we're analyzing.  What we're curious about is 
why we're seeing a concentration of hops at a small number of routers that 
appear on international paths.  I expected that when we looked at paths between 
e.g., US-Asia, US-Europe, US-South America (considering measurements from Ark 
nodes doing traceroutes to/from those locations), we would see the first 
instances of routers located the US along the west coast, east coast and south 
coast respectively. We did not expect to see Chicago as a first hop location in 
the US and are wondering e.g., if large providers do this to simplify their 
operations.  Hopefully that makes sense.  Any further thoughts are appreciated.

Regards, PB

From: Nick Hilliard 
Sent: Monday, January 17, 2022 12:36 PM
To: PAUL R BARFORD 
Cc: nanog@nanog.org 
Subject: Re: Long hops on international paths

PAUL R BARFORD wrote on 17/01/2022 18:02:
> For example, there is a router operated by Telia (AS1299) in Chicago
> that has a high concentration of such links.

this doesn't appear to match 1299's public network topology:

https://www.teliacarrier.com/our-network.html

Is ttl decrement disabled on the test paths you're measuring?

Broadly speaking, if you have a point-to-point link from one location to
another (or parallel set of links with a common failure path, e.g. waves
on a specific fibre path), there's a single router at each end.

Nick

Re: Long hops on international paths

[Nick Hilliard]

PAUL R BARFORD wrote on 17/01/2022 18:02:
For example, there is a router operated by Telia (AS1299) in Chicago 
that has a high concentration of such links.


this doesn't appear to match 1299's public network topology:

https://www.teliacarrier.com/our-network.html

Is ttl decrement disabled on the test paths you're measuring?

Broadly speaking, if you have a point-to-point link from one location to 
another (or parallel set of links with a common failure path, e.g. waves 
on a specific fibre path), there's a single router at each end.


Nick

Long hops on international paths

[PAUL R BARFORD]

Hello,

I am a researcher at the University of Wisconsin.  My colleagues at 
Northwestern University and I are studying international Internet connectivity 
and would appreciate your perspective on a recent finding.

We're using traceroute data from CAIDA's Ark project for our work.  We've 
observed that many international links (i.e., a single hop on an end-to-end 
path that connects two countries where end points on the hop are identified via 
rDNS) tend to originate/terminate at the same routers.  Said another way, we 
are observing a relatively small set of routers in different countries tend to 
have a majority of the international connections - this is especially the case 
for hops that terminate in the US.  For example, there is a router operated by 
Telia (AS1299) in Chicago that has a high concentration of such links.  We were 
a bit surprised by this finding since even though it makes sense that the set 
of providers is relatively small (i.e., those that offer global connectivity), 
we assumed that the set of routers that used for international connectivity 
within any one country would tend to be more widely distributed (at least with 
respect to how they appear in traceroute data - MPLS notwithstanding).

We're interested in whether or not this is indeed standard practice and if so, 
the cost/benefit for configuring international connectivity in this way?

Any thoughts or insights you might have would be greatly appreciated - off-list 
responses are welcome.

Thank you.

Regards, PB

Paul Barford
University of Wisconsin - Madison

Re: SRv6 Capable NOS and Devices

[Mark Tinka]

On 1/17/22 09:57, Brandon Butterworth wrote:


Isn't the argument here that if it's in most chip sets already it might
reasonably be expected to be a standard low end feature by now, along
with IPv6?

That it isn't may be why people are open to SRv6 (I'm assuming some are
based on this discussion) - if they have to pay extra they only want to
do so where they are generating revenue from it, the end points.

Complexity and architectural cleanliness are not a consideration, if a
vendor makes a box that does the job at the right price there is a high
risk people will buy it.


There are other things that are required to support MPLS services than 
just chip and software capability. Control plane, buffer memory, queue 
depth, e.t.c.


That is why you would not find a lack of MPLS support in Ethernet 
switches... you would find "broken" support, because the rest of the 
hardware is not designed to deliver the same level of MPLS service scale 
that a high-end router would; and the vendors make that very clear. An 
Ethernet switch running MPLS can probably be an excellent P router, but 
won't be an awesome PE router.


There are still some IP routing features we consider "basic" in bigger 
routers that attract extra costs in Ethernet switches. Never mind MPLS.


Let's not confuse "MPLS no longer being expensive" with "MPLS being a 
low-end feature".


I find it hard to believe that SRv6 support will be enabled on low-end 
devices like Ethernet switches by the traditional vendors. But hey, I 
have been wrong before.


Mark.

Re: SRv6 Capable NOS and Devices

[Saku Ytti]

> Isn't the argument here that if it's in most chip sets already it might
> reasonably be expected to be a standard low end feature by now, along
> with IPv6?
>
> That it isn't may be why people are open to SRv6 (I'm assuming some are
> based on this discussion) - if they have to pay extra they only want to
> do so where they are generating revenue from it, the end points.

HW doing IPv6 does not imply HW being able to do SRv6. SRv6 is not
IPV6, SRv6 is MPLS embedded in (HW) complex way in EH.

Heck, EH is specified in such a way that no HW device is technically
IPv6 capable. What happens when you stack EH headers is question mark,
some devices revert to crawl (Nokia FP), some devices drop packet in
HW after it sees more than N ER (Juniper Trio). And how does that
imply devices capability to find L4 headers and honour ECMP, ACL, QOS
is a question mark.

-- 
  ++ytti