Re: Has Level3 done away with traceroute??

2017-09-21 Thread Mel Beckman 
I am able to traceroute in and out of Level3, but it seems like some L3 
internal hops are missing, as I appear to go straight from the L3 edge router 
to my L3 customer agg router:

traceroute to 206.83.0.42 (206.83.0.42), 64 hops max, 52 byte packets
 1  47.155.227.1 (47.155.227.1)  4.318 ms  4.661 ms  4.715 ms
 2  172.102.107.166 (172.102.107.166)  10.202 ms  7.521 ms
172.102.102.240 (172.102.102.240)  7.240 ms
 3  ae8---0.scr02.lsan.ca.frontiernet.net (74.40.3.49)  9.609 ms  7.501 ms
ae8---0.scr01.lsan.ca.frontiernet.net (74.40.3.37)  5.298 ms
 4  ae0---0.cbr01.lsan.ca.frontiernet.net (74.40.3.198)  7.169 ms  7.015 ms  
7.277 ms
 5  * * *
 6  ae-4-90.edge1.losangeles9.level3.net (4.69.144.202)  8.384 ms  7.012 ms
ae-2-70.edge1.losangeles9.level3.net (4.69.144.74)  9.865 ms
 7  ae-2-70.edge1.losangeles9.level3.net (4.69.144.74)  7.471 ms
ae-3-80.edge1.losangeles9.level3.net (4.69.144.138)  6.551 ms
ae-2-70.edge1.losangeles9.level3.net (4.69.144.74)  10.520 ms
 8  4.68.111.22 (4.68.111.22)  9.010 ms  7.445 ms  7.314 ms
 9  sba1-ar1-xe-11-0-0-0.us.twtelecom.net (35.248.2.6)  9.788 ms  9.536 ms  
10.266 ms
10  206-190-77-10.static.twtelecom.net (206.190.77.10)  14.739 ms  12.525 ms  
9.979 ms
11  iris1.jet.net (206.83.0.42)  10.285 ms  9.880 ms  10.063 ms

traceroute to 47.155.227.1 (47.155.227.1), 64 hops max, 40 byte packets
 1  router1.sb.becknet.com (206.83.0.1)  1.270 ms  1.023 ms  0.352 ms
 2  206-190-77-9.static.twtelecom.net (206.190.77.9)  0.799 ms  0.741 ms  0.785 
ms
 3  ae1-90g.ar7.lax1.gblx.net (67.17.75.18)  3.052 ms  3.126 ms  2.957 ms
 4  ae10.edge1.losangeles9.level3.net (4.68.111.21)  3.456 ms  3.043 ms  2.981 
ms
 5  * * *
 6  4.16.30.130 (4.16.30.130)  27.564 ms  3.574 ms  3.445 ms
 7  ae1---0.scr02.lsan.ca.frontiernet.net (74.40.3.213)  3.489 ms 
ae1---0.scr01.lsan.ca.frontiernet.net (74.40.3
.197)  3.396 ms ae1---0.scr02.lsan.ca.frontiernet.net (74.40.3.213)  3.428 ms
 8  be10---0.lcr21.lsan.ca.frontiernet.net (74.40.3.38)  11.736 ms  5.026 ms 
be10---0.lcr22.lsan.ca.frontiernet.
net (74.40.3.50)  9.858 ms
 9  47.155.227.1 (47.155.227.1)  6.545 ms  5.398 ms  6.499 ms

 -mel beckman

> On Sep 21, 2017, at 10:45 AM, Filip Hruska  wrote:
> 
> Hi,
> 
> Did a measurement with RIPE Atlas and it seems like a worldwide problem, 
> based on data from 100 probes.
> 
> https://atlas.ripe.net/measurements/9324230/#!tracemon
> 
> 
> Dne 9/21/17 v 19:10 Van Dyk, Donovan via NANOG napsal(a):
>> Hello All,
>> 
>> Recently I was troubleshooting a network event for a client of our who 
>> resides on the Level3 network. While trying to verify the path, I noticed I 
>> am no longer able to traceroute through the Level3 network.
>> The funny thing is this is not just isolated to the /32. It appears to be 
>> that the entire 4.0.0.0/9 network is no longer able to traceroute through. 
>> Everything dies on their edge network.
>> 
>> This appears to be isolated to traceroute. I have check this in NA and EU.
>> 
>> My carrier contacted Level3 who pretty much stated that they can’t provide 
>> anything.
>> 
>> I have checked multiple looking glasses and other online tools and none of 
>> them make it. Even Level3 looking glass drops the packets.
>> 
>> Does anyone know anything about this? I’m pretty sure this is the first time 
>> we are seeing this.
>> 
>> 
>> Random 4.0.0.0/9 address.
>> 
>> NTT looking glass
>> Tracing the route to 4.35.230.7
>> 
>> 1   *
>> ae-2.a00.snjsca04.us.bb.gin.ntt.net (129.250.3.58) 3 msec  1 msec
>>  2   *  *  *
>>  3   *  *  *
>>  4   *  *  *
>>  5   *  *  *
>>  6   *  *  *
>> 
>> 
>> TATA looking glass
>> traceroute to 4.7.6.4 (4.7.6.4), 30 hops max, 52 byte packets
>> 1  if-ae-14-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.89)  2.056 ms 
>> if-ae-6-2.tcore1.FR0-Frankfurt.as6453.net (195.219.50.173)  1.253 ms  1.177 
>> ms
>>  MPLS Label=616998 CoS=0 TTL=1 S=1
>> 2  195.219.50.50 (195.219.50.50)  1.214 ms  1.247 ms  1.535 ms
>> 3  195.219.50.50 (195.219.50.50)  1.144 ms *  2.246 ms
>> 4  * * *
>> 5  * * *
>> 6  * * *
>> 7  * * *
>> 8  * * *
>> 9  * * *
>> 10  * * *
>> 
>> 
>> Telia looking glass
>> traceroute to 4.7.6.4 (4.7.6.4), 30 hops max, 52 byte packets
>> 1  if-ae-14-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.89)  2.056 ms 
>> if-ae-6-2.tcore1.FR0-Frankfurt.as6453.net (195.219.50.173)  1.253 ms  1.177 
>> ms
>>  MPLS Label=616998 CoS=0 TTL=1 S=1
>> 2  195.219.50.50 (195.219.50.50)  1.214 ms  1.247 ms  1.535 ms
>> 3  195.219.50.50 (195.219.50.50)  1.144 ms *  2.246 ms
>> 4  * * *
>> 5  * * *
>> 6  * * *
>> 7  * * *
>> 8  * * *
>> 9  * * *
>> 10  * * *
>> 
>> 
>> Level3 looking glass
>> Traceroute results from Atlanta, GA to 
>> 4.200.65.42(dialup-4.200.65.42.Dial1.LosAngeles1)
>> 
>>   1  0.0.0.0  * * *
>>   2  0.0.0.0  * * *
>>   3  0.0.0.0  * * *
>>   4  0.0.0.0  * * *
>>   5  0.0.0.0  * * *
>>   6  0.0.0.0  * * *
>>   7  0.0.0.0  * * *
>>   8  0.0.0.0  * * *
>>   9  0.0.0.0  * * *
>>  10  0.0.0.0  * * *
>>  11

Re: Has Level3 done away with traceroute??

2017-09-21 Thread ML 

I just performed a few traceroutes.

Comcast to 4.2.2.2

5. hu-0-11-0-0-ar03.ivyland.pa.panjde.comcast.net
 6. xe-4-0-0.edge1.Toronto.Level3.net
 7. ???
 8. b.resolvers.Level3.net

Comcast to Level3 customer

 5. hu-0-11-0-0-ar03.ivyland.pa.panjde.comcast.net
 6. xe-4-0-0.edge1.Toronto.Level3.net
 7. ae-11-11.bar1.Philadelphia1.Level3.net
 8. .bar1.Philadelphia1.Level3.net

I see the problem for sure FROM a Level3 Philadelphia customer to other 
destinations though.






On 9/21/2017 1:45 PM, Filip Hruska wrote:

Hi,

Did a measurement with RIPE Atlas and it seems like a worldwide 
problem, based on data from 100 probes.


https://atlas.ripe.net/measurements/9324230/#!tracemon


Dne 9/21/17 v 19:10 Van Dyk, Donovan via NANOG napsal(a):

Hello All,

Recently I was troubleshooting a network event for a client of our 
who resides on the Level3 network. While trying to verify the path, I 
noticed I am no longer able to traceroute through the Level3 network.
The funny thing is this is not just isolated to the /32. It appears 
to be that the entire 4.0.0.0/9 network is no longer able to 
traceroute through. Everything dies on their edge network.


This appears to be isolated to traceroute. I have check this in NA 
and EU.


My carrier contacted Level3 who pretty much stated that they can’t 
provide anything.


I have checked multiple looking glasses and other online tools and 
none of them make it. Even Level3 looking glass drops the packets.


Does anyone know anything about this? I’m pretty sure this is the 
first time we are seeing this.



Random 4.0.0.0/9 address.

NTT looking glass
Tracing the route to 4.35.230.7

1   *
 ae-2.a00.snjsca04.us.bb.gin.ntt.net (129.250.3.58) 3 msec 1 msec
  2   *  *  *
  3   *  *  *
  4   *  *  *
  5   *  *  *
  6   *  *  *


TATA looking glass
traceroute to 4.7.6.4 (4.7.6.4), 30 hops max, 52 byte packets
1  if-ae-14-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.89) 2.056 
ms if-ae-6-2.tcore1.FR0-Frankfurt.as6453.net (195.219.50.173)  1.253 
ms  1.177 ms

  MPLS Label=616998 CoS=0 TTL=1 S=1
2  195.219.50.50 (195.219.50.50)  1.214 ms  1.247 ms  1.535 ms
3  195.219.50.50 (195.219.50.50)  1.144 ms *  2.246 ms
4  * * *
5  * * *
6  * * *
7  * * *
8  * * *
9  * * *
10  * * *


Telia looking glass
traceroute to 4.7.6.4 (4.7.6.4), 30 hops max, 52 byte packets
1  if-ae-14-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.89) 2.056 
ms if-ae-6-2.tcore1.FR0-Frankfurt.as6453.net (195.219.50.173)  1.253 
ms  1.177 ms

  MPLS Label=616998 CoS=0 TTL=1 S=1
2  195.219.50.50 (195.219.50.50)  1.214 ms  1.247 ms  1.535 ms
3  195.219.50.50 (195.219.50.50)  1.144 ms *  2.246 ms
4  * * *
5  * * *
6  * * *
7  * * *
8  * * *
9  * * *
10  * * *


Level3 looking glass
Traceroute results from Atlanta, GA to 
4.200.65.42(dialup-4.200.65.42.Dial1.LosAngeles1)


   1  0.0.0.0  * * *
   2  0.0.0.0  * * *
   3  0.0.0.0  * * *
   4  0.0.0.0  * * *
   5  0.0.0.0  * * *
   6  0.0.0.0  * * *
   7  0.0.0.0  * * *
   8  0.0.0.0  * * *
   9  0.0.0.0  * * *
  10  0.0.0.0  * * *
  11  0.0.0.0  * * *

--
Donovan Van Dyk
SOC Network Engineer
Fort Lauderdale, FL USA

[cid:image001.png@01D332DA.F4DD00A0]
The information contained in this electronic mail transmission and 
its attachments may be privileged and confidential and protected from 
disclosure. If the reader of this message is not the intended 
recipient (or an individual responsible for delivery of the message 
to such person), you are strictly prohibited from copying, 
disseminating or distributing this communication. If you have 
received this communication in error, please notify the sender 
immediately and destroy all electronic, paper or other versions.

Re: Has Level3 done away with traceroute??

2017-09-21 Thread Filip Hruska 

Hi,

Did a measurement with RIPE Atlas and it seems like a worldwide problem, 
based on data from 100 probes.


https://atlas.ripe.net/measurements/9324230/#!tracemon


Dne 9/21/17 v 19:10 Van Dyk, Donovan via NANOG napsal(a):

Hello All,

Recently I was troubleshooting a network event for a client of our who resides 
on the Level3 network. While trying to verify the path, I noticed I am no 
longer able to traceroute through the Level3 network.
The funny thing is this is not just isolated to the /32. It appears to be that 
the entire 4.0.0.0/9 network is no longer able to traceroute through. 
Everything dies on their edge network.

This appears to be isolated to traceroute. I have check this in NA and EU.

My carrier contacted Level3 who pretty much stated that they can’t provide 
anything.

I have checked multiple looking glasses and other online tools and none of them 
make it. Even Level3 looking glass drops the packets.

Does anyone know anything about this? I’m pretty sure this is the first time we 
are seeing this.


Random 4.0.0.0/9 address.

NTT looking glass
Tracing the route to 4.35.230.7

1   *
 ae-2.a00.snjsca04.us.bb.gin.ntt.net (129.250.3.58) 3 msec  1 msec
  2   *  *  *
  3   *  *  *
  4   *  *  *
  5   *  *  *
  6   *  *  *


TATA looking glass
traceroute to 4.7.6.4 (4.7.6.4), 30 hops max, 52 byte packets
1  if-ae-14-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.89)  2.056 ms 
if-ae-6-2.tcore1.FR0-Frankfurt.as6453.net (195.219.50.173)  1.253 ms  1.177 ms
  MPLS Label=616998 CoS=0 TTL=1 S=1
2  195.219.50.50 (195.219.50.50)  1.214 ms  1.247 ms  1.535 ms
3  195.219.50.50 (195.219.50.50)  1.144 ms *  2.246 ms
4  * * *
5  * * *
6  * * *
7  * * *
8  * * *
9  * * *
10  * * *


Telia looking glass
traceroute to 4.7.6.4 (4.7.6.4), 30 hops max, 52 byte packets
1  if-ae-14-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.89)  2.056 ms 
if-ae-6-2.tcore1.FR0-Frankfurt.as6453.net (195.219.50.173)  1.253 ms  1.177 ms
  MPLS Label=616998 CoS=0 TTL=1 S=1
2  195.219.50.50 (195.219.50.50)  1.214 ms  1.247 ms  1.535 ms
3  195.219.50.50 (195.219.50.50)  1.144 ms *  2.246 ms
4  * * *
5  * * *
6  * * *
7  * * *
8  * * *
9  * * *
10  * * *


Level3 looking glass
Traceroute results from Atlanta, GA to 
4.200.65.42(dialup-4.200.65.42.Dial1.LosAngeles1)

   1  0.0.0.0  * * *
   2  0.0.0.0  * * *
   3  0.0.0.0  * * *
   4  0.0.0.0  * * *
   5  0.0.0.0  * * *
   6  0.0.0.0  * * *
   7  0.0.0.0  * * *
   8  0.0.0.0  * * *
   9  0.0.0.0  * * *
  10  0.0.0.0  * * *
  11  0.0.0.0  * * *

--
Donovan Van Dyk
SOC Network Engineer
Fort Lauderdale, FL USA

[cid:image001.png@01D332DA.F4DD00A0]
The information contained in this electronic mail transmission and its 
attachments may be privileged and confidential and protected from disclosure. 
If the reader of this message is not the intended recipient (or an individual 
responsible for delivery of the message to such person), you are strictly 
prohibited from copying, disseminating or distributing this communication. If 
you have received this communication in error, please notify the sender 
immediately and destroy all electronic, paper or other versions.



--
Best Regards,
Filip Hruska
Linux System Administrator

Re: Has Level3 done away with traceroute??

2017-09-21 Thread David Hubbard 
I’m seeing the same thing across our Level 3 circuits, even if the traffic 
never leaves their network.  This was not the case as recently as two days ago 
when I had a reachability ticket open with them.  My SE says they’re having an 
internal issue that’s being worked on; didn’t provide detail.

On 9/21/17, 1:13 PM, "NANOG on behalf of Van Dyk, Donovan via NANOG" 
 wrote:

Hello All,

Recently I was troubleshooting a network event for a client of our who 
resides on the Level3 network. While trying to verify the path, I noticed I am 
no longer able to traceroute through the Level3 network.
The funny thing is this is not just isolated to the /32. It appears to be 
that the entire 4.0.0.0/9 network is no longer able to traceroute through. 
Everything dies on their edge network.

This appears to be isolated to traceroute. I have check this in NA and EU.

My carrier contacted Level3 who pretty much stated that they can’t provide 
anything.

I have checked multiple looking glasses and other online tools and none of 
them make it. Even Level3 looking glass drops the packets.

Does anyone know anything about this? I’m pretty sure this is the first 
time we are seeing this.


Random 4.0.0.0/9 address.

NTT looking glass
Tracing the route to 4.35.230.7

1   *
ae-2.a00.snjsca04.us.bb.gin.ntt.net (129.250.3.58) 3 msec  1 msec
 2   *  *  *
 3   *  *  *
 4   *  *  *
 5   *  *  *
 6   *  *  *


TATA looking glass
traceroute to 4.7.6.4 (4.7.6.4), 30 hops max, 52 byte packets
1  if-ae-14-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.89)  2.056 ms 
if-ae-6-2.tcore1.FR0-Frankfurt.as6453.net (195.219.50.173)  1.253 ms  1.177 ms
 MPLS Label=616998 CoS=0 TTL=1 S=1
2  195.219.50.50 (195.219.50.50)  1.214 ms  1.247 ms  1.535 ms
3  195.219.50.50 (195.219.50.50)  1.144 ms *  2.246 ms
4  * * *
5  * * *
6  * * *
7  * * *
8  * * *
9  * * *
10  * * *


Telia looking glass
traceroute to 4.7.6.4 (4.7.6.4), 30 hops max, 52 byte packets
1  if-ae-14-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.89)  2.056 ms 
if-ae-6-2.tcore1.FR0-Frankfurt.as6453.net (195.219.50.173)  1.253 ms  1.177 ms
 MPLS Label=616998 CoS=0 TTL=1 S=1
2  195.219.50.50 (195.219.50.50)  1.214 ms  1.247 ms  1.535 ms
3  195.219.50.50 (195.219.50.50)  1.144 ms *  2.246 ms
4  * * *
5  * * *
6  * * *
7  * * *
8  * * *
9  * * *
10  * * *


Level3 looking glass
Traceroute results from Atlanta, GA to 
4.200.65.42(dialup-4.200.65.42.Dial1.LosAngeles1)

  1  0.0.0.0  * * *
  2  0.0.0.0  * * *
  3  0.0.0.0  * * *
  4  0.0.0.0  * * *
  5  0.0.0.0  * * *
  6  0.0.0.0  * * *
  7  0.0.0.0  * * *
  8  0.0.0.0  * * *
  9  0.0.0.0  * * *
 10  0.0.0.0  * * *
 11  0.0.0.0  * * *

--
Donovan Van Dyk
SOC Network Engineer
Fort Lauderdale, FL USA

[cid:image001.png@01D332DA.F4DD00A0]
The information contained in this electronic mail transmission and its 
attachments may be privileged and confidential and protected from disclosure. 
If the reader of this message is not the intended recipient (or an individual 
responsible for delivery of the message to such person), you are strictly 
prohibited from copying, disseminating or distributing this communication. If 
you have received this communication in error, please notify the sender 
immediately and destroy all electronic, paper or other versions.

Has Level3 done away with traceroute??

2017-09-21 Thread Van Dyk, Donovan via NANOG 
Hello All,

Recently I was troubleshooting a network event for a client of our who resides 
on the Level3 network. While trying to verify the path, I noticed I am no 
longer able to traceroute through the Level3 network.
The funny thing is this is not just isolated to the /32. It appears to be that 
the entire 4.0.0.0/9 network is no longer able to traceroute through. 
Everything dies on their edge network.

This appears to be isolated to traceroute. I have check this in NA and EU.

My carrier contacted Level3 who pretty much stated that they can’t provide 
anything.

I have checked multiple looking glasses and other online tools and none of them 
make it. Even Level3 looking glass drops the packets.

Does anyone know anything about this? I’m pretty sure this is the first time we 
are seeing this.


Random 4.0.0.0/9 address.

NTT looking glass
Tracing the route to 4.35.230.7

1   *
ae-2.a00.snjsca04.us.bb.gin.ntt.net (129.250.3.58) 3 msec  1 msec
 2   *  *  *
 3   *  *  *
 4   *  *  *
 5   *  *  *
 6   *  *  *


TATA looking glass
traceroute to 4.7.6.4 (4.7.6.4), 30 hops max, 52 byte packets
1  if-ae-14-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.89)  2.056 ms 
if-ae-6-2.tcore1.FR0-Frankfurt.as6453.net (195.219.50.173)  1.253 ms  1.177 ms
 MPLS Label=616998 CoS=0 TTL=1 S=1
2  195.219.50.50 (195.219.50.50)  1.214 ms  1.247 ms  1.535 ms
3  195.219.50.50 (195.219.50.50)  1.144 ms *  2.246 ms
4  * * *
5  * * *
6  * * *
7  * * *
8  * * *
9  * * *
10  * * *


Telia looking glass
traceroute to 4.7.6.4 (4.7.6.4), 30 hops max, 52 byte packets
1  if-ae-14-3.tcore2.FNM-Frankfurt.as6453.net (195.219.87.89)  2.056 ms 
if-ae-6-2.tcore1.FR0-Frankfurt.as6453.net (195.219.50.173)  1.253 ms  1.177 ms
 MPLS Label=616998 CoS=0 TTL=1 S=1
2  195.219.50.50 (195.219.50.50)  1.214 ms  1.247 ms  1.535 ms
3  195.219.50.50 (195.219.50.50)  1.144 ms *  2.246 ms
4  * * *
5  * * *
6  * * *
7  * * *
8  * * *
9  * * *
10  * * *


Level3 looking glass
Traceroute results from Atlanta, GA to 
4.200.65.42(dialup-4.200.65.42.Dial1.LosAngeles1)

  1  0.0.0.0  * * *
  2  0.0.0.0  * * *
  3  0.0.0.0  * * *
  4  0.0.0.0  * * *
  5  0.0.0.0  * * *
  6  0.0.0.0  * * *
  7  0.0.0.0  * * *
  8  0.0.0.0  * * *
  9  0.0.0.0  * * *
 10  0.0.0.0  * * *
 11  0.0.0.0  * * *

--
Donovan Van Dyk
SOC Network Engineer
Fort Lauderdale, FL USA

[cid:image001.png@01D332DA.F4DD00A0]
The information contained in this electronic mail transmission and its 
attachments may be privileged and confidential and protected from disclosure. 
If the reader of this message is not the intended recipient (or an individual 
responsible for delivery of the message to such person), you are strictly 
prohibited from copying, disseminating or distributing this communication. If 
you have received this communication in error, please notify the sender 
immediately and destroy all electronic, paper or other versions.