Re: is something weird going on with cox, level3, and/or cogent

2017-02-12 Thread Mel Beckman 
It helps to follow some general guidelines to traceroute interpretation:

1. Intermediate hop time is not necessarily a measure of performance, as 
traceroute TTL expiration processing has low priority.

2. Time measurements areround-trip, not latency, and the bulk of the time may 
be incurred on the return trip. The only way to know for sure is bidirectional 
traceroutes.

3. Seeing reported latency in the first few hops indicates a probable issue on 
the local network level.


In your original traceroute, latency jumps from 40ms to 200ms at 72.215.229.22, 
which is a Cox address. Since this is so close to you (third hop), I would 
apply guideline #3. All the hops after that have high latencies too, but if 
most of that is being injected in hop 3 then that means the problem started 
there.

Verify that you don’t have high traffic volume on your individual Cox circuit 
(e.g. a backup running or something). If that’s not the case, then you’re 
likely seeing a Cox problem with their own internal traffic engineering. If Cox 
is doing weekend maintenance, some circuits may be out of service for a while. 
Cox is not good about passing that info down to residential customers, so you 
may never know what happened. A DIA business customer can usually ask about 
global tickets, but Cox is loathe to give that information out even then.

 -mel


On Feb 12, 2017, at 8:27 PM, Miles Fidelman 
> wrote:


Now isn't that interesting.

1. Verizon mobile seems to also route through level3 and cogent.

2. But.. the performance seems to be a lot better.

3. That's really odd.

1  192.168.43.1 (192.168.43.1)  2.063 ms  2.382 ms  1.913 ms
 2  7.sub-66-174-33.myvzw.com (66.174.33.7)  
50.676 ms  36.720 ms  41.142 ms
 3  164.sub-69-83-172.myvzw.com 
(69.83.172.164)  39.831 ms  40.240 ms  40.190 ms
 4  178.sub-69-83-173.myvzw.com 
(69.83.173.178)  39.851 ms  48.650 ms  30.921 ms
 5  194.sub-69-83-173.myvzw.com 
(69.83.173.194)  49.134 ms  31.131 ms  39.946 ms
 6  8.sub-69-83-162.myvzw.com (69.83.162.8)  
42.841 ms  39.532 ms  40.024 ms
 7  73.sub-66-174-29.myvzw.com 
(66.174.29.73)  44.074 ms  31.248 ms  42.199 ms
 8  5-1-1.bear1.phoenix1.level3.net 
(4.16.142.89)  38.474 ms  39.555 ms  57.860 ms
 9  
ae-2-70.edge1.losangeles6.level3.net
 (4.69.144.80)  86.101 ms  78.856 ms

ae-3-80.edge1.losangeles6.level3.net
 (4.69.144.144)  70.001 ms
10  
ae-4-90.edge1.losangeles6.level3.net
 (4.69.144.208)  48.840 ms

ae-2-70.edge1.losangeles6.level3.net
 (4.69.144.80)  71.847 ms

ae-3-80.edge1.losangeles6.level3.net
 (4.69.144.144)  81.841 ms
11  
be3036.ccr41.lax04.atlas.cogentco.com 
(154.54.14.129)  70.538 ms  39.175 ms  45.213 ms
12  
be2964.ccr21.lax01.atlas.cogentco.com 
(154.54.44.77)  89.463 ms  78.656 ms  72.936 ms
13  
be2932.ccr22.phx02.atlas.cogentco.com 
(154.54.45.161)  106.896 ms  82.389 ms  87.310 ms
14  
be2929.ccr21.elp01.atlas.cogentco.com 
(154.54.42.66)  72.782 ms  76.034 ms  114.967 ms
15  
be2928.ccr22.iah01.atlas.cogentco.com 
(154.54.30.161)  87.883 ms  100.641 ms

be2927.ccr41.iah01.atlas.cogentco.com 
(154.54.29.221)  117.904 ms
16  
be2687.ccr41.atl01.atlas.cogentco.com 
(154.54.28.69)  144.881 ms

be2690.ccr42.atl01.atlas.cogentco.com 
(154.54.28.129)  135.465 ms  110.129 ms
17  
be2112.ccr41.dca01.atlas.cogentco.com 
(154.54.7.157)  93.713 ms

be2113.ccr42.dca01.atlas.cogentco.com 
(154.54.24.221)  128.171 ms

be2112.ccr41.dca01.atlas.cogentco.com 
(154.54.7.157)  109.207 ms
18  
be2806.ccr41.jfk02.atlas.cogentco.com 
(154.54.40.105)  136.620 ms

be2807.ccr42.jfk02.atlas.cogentco.com 
(154.54.40.109)  113.346 ms

be2806.ccr41.jfk02.atlas.cogentco.com 
(154.54.40.105)  118.301 ms
19  
be2096.ccr22.bos01.atlas.cogentco.com 
(154.54.30.42)  148.778 ms  166.763 ms

be2094.ccr21.bos01.atlas.cogentco.com 
(154.54.30.14)  153.433 ms
20

Small-mid scale vDPI

2017-02-12 Thread Ramy Hashish 
Good day community,

Any experience based recommendations about cost effective vDPIs in
small-mid scale?

Thanks,

Ramy

Re: is something weird going on with cox, level3, and/or cogent

2017-02-12 Thread Miles Fidelman 

Now isn't that interesting.

1. Verizon mobile seems to also route through level3 and cogent.

2. But.. the performance seems to be a lot better.

3. That's really odd.

1  192.168.43.1 (192.168.43.1)  2.063 ms  2.382 ms  1.913 ms
 2  7.sub-66-174-33.myvzw.com (66.174.33.7)  50.676 ms  36.720 ms 41.142 ms
 3  164.sub-69-83-172.myvzw.com (69.83.172.164)  39.831 ms  40.240 ms  
40.190 ms
 4  178.sub-69-83-173.myvzw.com (69.83.173.178)  39.851 ms  48.650 ms  
30.921 ms
 5  194.sub-69-83-173.myvzw.com (69.83.173.194)  49.134 ms  31.131 ms  
39.946 ms

 6  8.sub-69-83-162.myvzw.com (69.83.162.8)  42.841 ms  39.532 ms 40.024 ms
 7  73.sub-66-174-29.myvzw.com (66.174.29.73)  44.074 ms  31.248 ms  
42.199 ms
 8  5-1-1.bear1.phoenix1.level3.net (4.16.142.89)  38.474 ms 39.555 ms  
57.860 ms

 9  ae-2-70.edge1.losangeles6.level3.net (4.69.144.80)  86.101 ms 78.856 ms
ae-3-80.edge1.losangeles6.level3.net (4.69.144.144)  70.001 ms
10  ae-4-90.edge1.losangeles6.level3.net (4.69.144.208)  48.840 ms
ae-2-70.edge1.losangeles6.level3.net (4.69.144.80)  71.847 ms
ae-3-80.edge1.losangeles6.level3.net (4.69.144.144)  81.841 ms
11  be3036.ccr41.lax04.atlas.cogentco.com (154.54.14.129)  70.538 ms  
39.175 ms  45.213 ms
12  be2964.ccr21.lax01.atlas.cogentco.com (154.54.44.77)  89.463 ms  
78.656 ms  72.936 ms
13  be2932.ccr22.phx02.atlas.cogentco.com (154.54.45.161)  106.896 ms  
82.389 ms  87.310 ms
14  be2929.ccr21.elp01.atlas.cogentco.com (154.54.42.66)  72.782 ms  
76.034 ms  114.967 ms
15  be2928.ccr22.iah01.atlas.cogentco.com (154.54.30.161)  87.883 ms  
100.641 ms

be2927.ccr41.iah01.atlas.cogentco.com (154.54.29.221)  117.904 ms
16  be2687.ccr41.atl01.atlas.cogentco.com (154.54.28.69)  144.881 ms
be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129)  135.465 ms  
110.129 ms

17  be2112.ccr41.dca01.atlas.cogentco.com (154.54.7.157)  93.713 ms
be2113.ccr42.dca01.atlas.cogentco.com (154.54.24.221)  128.171 ms
be2112.ccr41.dca01.atlas.cogentco.com (154.54.7.157)  109.207 ms
18  be2806.ccr41.jfk02.atlas.cogentco.com (154.54.40.105)  136.620 ms
be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109)  113.346 ms
be2806.ccr41.jfk02.atlas.cogentco.com (154.54.40.105)  118.301 ms
19  be2096.ccr22.bos01.atlas.cogentco.com (154.54.30.42)  148.778 ms  
166.763 ms

be2094.ccr21.bos01.atlas.cogentco.com (154.54.30.14)  153.433 ms
20  te0-4-1-7.agr21.bos01.atlas.cogentco.com (154.54.47.254) 143.449 ms
te0-4-1-6.agr21.bos01.atlas.cogentco.com (154.54.47.230) 242.964 ms
te0-4-1-6.agr22.bos01.atlas.cogentco.com (154.54.80.10) 198.678 ms
21  te0-0-2-1.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.78)  
139.454 ms
te0-0-2-0.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.82)  
132.726 ms  107.254 ms

22  38.122.127.18 (38.122.127.18)  157.142 ms  158.882 ms  152.458 ms
23  207.154.0.57 (207.154.0.57)  144.913 ms  112.629 ms  112.519 ms
24  server1.ntcorp.com (207.154.13.58)  116.986 ms  120.276 ms 128.333 ms

Miles

On 2/12/17 8:57 PM, Mel Beckman wrote:

Miles,

Have you tried trace routing through cellular data connections? The results 
you're seeing could be explained by congestion at the point of your modem, 
which I think is with the cox techs are implying.

-mel via cell


On Feb 12, 2017, at 7:46 PM, Mel Beckman  wrote:

It looks like one or more circuits are down, so you're seeing asymmetrical 
routing over congested paths in one direction.

-mel via cell


On Feb 12, 2017, at 7:14 PM, Miles Fidelman  wrote:

Hi Folks,

I'm visiting AZ, and seeing some really really poor performance accessing some of our servers via 
Cox broadband.  The folks at Cox technical support are useless - all they say is "well you're 
on a DOCSIS 2 modem."  Meanwhile, everything I'm seeing is several hops upstream of the local 
segment - and all that Cox level2 tech support will say is "if there was a backbone problem, 
our backbone people would have dealt with it."

I'm having problems reaching both our own server, and sites like google, 
facebook, windows update.

Traceroutes to and from our server are illustrative - note that for most of the 
past week, the average ping time was 85msec. Now we're seeing this:

 From 98.177.135.186 - the public IP address on Cox's local broadband service.
To 107.154.13.58 (ntcorp.server - one of our servers, sitting in a Tierpoint 
data center, near Boston)
traceroute to ntcorp.com (207.154.13.58), 64 hops max, 52 byte packets
1  10.128.128.1 (10.128.128.1)  199.893 ms  75.319 ms  27.295 ms
2  100.127.69.178 (100.127.69.178)  38.710 ms  40.075 ms  43.598 ms
3  72.215.229.22 (72.215.229.22)  39.674 ms  201.368 ms *
4  lag-157.bear2.phoenix1.level3.net (4.28.82.53)  686.499 ms 1837.141 ms  
16.273 ms
5  ae-1-60.edge1.losangeles6.level3.net (4.69.144.16)  35.498 ms 964.377 ms *
6  ae-3-80.edge1.losangeles6.level3.net (4.69.144.144)  551.760 ms  525.014 ms
   ae-1-60.edge1.losangeles6.level3.net (4.69.144.16)  2061.191 ms
7

Re: is something weird going on with cox, level3, and/or cogent

2017-02-12 Thread Mel Beckman 
Miles,

Have you tried trace routing through cellular data connections? The results 
you're seeing could be explained by congestion at the point of your modem, 
which I think is with the cox techs are implying.

-mel via cell

> On Feb 12, 2017, at 7:46 PM, Mel Beckman  wrote:
> 
> It looks like one or more circuits are down, so you're seeing asymmetrical 
> routing over congested paths in one direction. 
> 
> -mel via cell
> 
>> On Feb 12, 2017, at 7:14 PM, Miles Fidelman  
>> wrote:
>> 
>> Hi Folks,
>> 
>> I'm visiting AZ, and seeing some really really poor performance accessing 
>> some of our servers via Cox broadband.  The folks at Cox technical support 
>> are useless - all they say is "well you're on a DOCSIS 2 modem."  Meanwhile, 
>> everything I'm seeing is several hops upstream of the local segment - and 
>> all that Cox level2 tech support will say is "if there was a backbone 
>> problem, our backbone people would have dealt with it."
>> 
>> I'm having problems reaching both our own server, and sites like google, 
>> facebook, windows update.
>> 
>> Traceroutes to and from our server are illustrative - note that for most of 
>> the past week, the average ping time was 85msec. Now we're seeing this:
>> 
>> From 98.177.135.186 - the public IP address on Cox's local broadband service.
>> To 107.154.13.58 (ntcorp.server - one of our servers, sitting in a Tierpoint 
>> data center, near Boston)
>> traceroute to ntcorp.com (207.154.13.58), 64 hops max, 52 byte packets
>> 1  10.128.128.1 (10.128.128.1)  199.893 ms  75.319 ms  27.295 ms
>> 2  100.127.69.178 (100.127.69.178)  38.710 ms  40.075 ms  43.598 ms
>> 3  72.215.229.22 (72.215.229.22)  39.674 ms  201.368 ms *
>> 4  lag-157.bear2.phoenix1.level3.net (4.28.82.53)  686.499 ms 1837.141 ms  
>> 16.273 ms
>> 5  ae-1-60.edge1.losangeles6.level3.net (4.69.144.16)  35.498 ms 964.377 ms *
>> 6  ae-3-80.edge1.losangeles6.level3.net (4.69.144.144)  551.760 ms  525.014 
>> ms
>>   ae-1-60.edge1.losangeles6.level3.net (4.69.144.16)  2061.191 ms
>> 7  be3036.ccr41.lax04.atlas.cogentco.com (154.54.14.129)  847.778 ms  87.601 
>> ms  71.504 ms
>> 8  be2965.ccr22.lax01.atlas.cogentco.com (154.54.45.1)  79.060 ms  225.647 ms
>>   be2964.ccr21.lax01.atlas.cogentco.com (154.54.44.77)  60.306 ms
>> 9  * be2931.ccr21.phx02.atlas.cogentco.com (154.54.44.85) 2264.071 ms  
>> 185.180 ms
>> 10  be2929.ccr21.elp01.atlas.cogentco.com (154.54.42.66)  61.208 ms
>>   be2930.ccr21.elp01.atlas.cogentco.com (154.54.42.78)  386.149 ms  1278.868 
>> ms
>> 11  be2928.ccr22.iah01.atlas.cogentco.com (154.54.30.161)  384.136 ms
>>   be2927.ccr41.iah01.atlas.cogentco.com (154.54.29.221) 2339.833 ms  615.415 
>> ms
>> 12  be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129)  233.061 ms
>>   be2687.ccr41.atl01.atlas.cogentco.com (154.54.28.69)  87.902 ms
>>   be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129)  861.159 ms
>> 13  be2113.ccr42.dca01.atlas.cogentco.com (154.54.24.221)  998.858 ms
>>   be2112.ccr41.dca01.atlas.cogentco.com (154.54.7.157)  249.930 ms *
>> 14  be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109)  768.461 ms
>>   be2806.ccr41.jfk02.atlas.cogentco.com (154.54.40.105)  136.772 ms
>>   be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109)  288.225 ms
>> 15  be2094.ccr21.bos01.atlas.cogentco.com (154.54.30.14)  271.736 ms  
>> 166.224 ms
>>   be2096.ccr22.bos01.atlas.cogentco.com (154.54.30.42)  565.015 ms
>> 16  te0-4-1-7.agr22.bos01.atlas.cogentco.com (154.54.80.34) 1944.479 ms
>>   te0-4-1-6.agr22.bos01.atlas.cogentco.com (154.54.80.10) 149.803 ms
>>   te0-4-1-6.agr21.bos01.atlas.cogentco.com (154.54.47.230) 897.115 ms
>> 17  te0-0-2-0.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.82)  107.207 
>> ms
>>   te0-0-2-1.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.78)  295.881 
>> ms  185.453 ms
>> 18  38.122.127.18 (38.122.127.18)  115.652 ms  461.168 ms  615.526 ms
>> 19  207.154.0.57 (207.154.0.57)  1871.023 ms  1987.832 ms 2165.248 ms
>> 20  server1.ntcorp.com (207.154.13.58)  587.560 ms  263.328 ms 333.542 ms
>> 
>> 
>> Traceroute in the reverse direction:
>> 1  207.154.13.47 (207.154.13.47)  0.000 ms  0.000 ms  0.000 ms
>> 2  * * *
>> 3  h130.207.190.173.static.ip.windstream.net (173.190.207.130) 0.000 ms  
>> 0.000 ms  0.000 ms
>> 4  xe1-2-0-0.cr01.cley01-oh.us.windstream.net (40.128.250.166) 12.000 ms  
>> 12.000 ms  12.000 ms
>> 5  et11-0-0-0.cr01.chcg01-il.us.windstream.net (40.128.248.71) 20.000 ms  
>> 20.000 ms  20.000 ms
>> 6  10gigabitethernet4-1.core1.chi1.he.NET (206.223.119.37)  72.001 ms  
>> 20.000 ms  16.000 ms
>> 7  chgobbrj01pos010100.r2.ch.cox.net (68.105.30.193)  20.000 ms 20.000 ms  
>> 20.000 ms
>> 8  chnddsrj01-ae1.0.rd.ph.cox.net (68.1.5.211)  72.001 ms  72.001 ms  72.001 
>> ms
>> 9  * * *
>> 10  * * *
>> 11  * * *
>> 
>> 30  * * *
>> 
>> 
>> Two things jump out at me:
>> 1. The rather large number of hops from cox to ntcorp - with high delays  
>> from several nodes in both the

Re: is something weird going on with cox, level3, and/or cogent

2017-02-12 Thread Mel Beckman 
It looks like one or more circuits are down, so you're seeing asymmetrical 
routing over congested paths in one direction. 

-mel via cell

> On Feb 12, 2017, at 7:14 PM, Miles Fidelman  
> wrote:
> 
> Hi Folks,
> 
> I'm visiting AZ, and seeing some really really poor performance accessing 
> some of our servers via Cox broadband.  The folks at Cox technical support 
> are useless - all they say is "well you're on a DOCSIS 2 modem."  Meanwhile, 
> everything I'm seeing is several hops upstream of the local segment - and all 
> that Cox level2 tech support will say is "if there was a backbone problem, 
> our backbone people would have dealt with it."
> 
> I'm having problems reaching both our own server, and sites like google, 
> facebook, windows update.
> 
> Traceroutes to and from our server are illustrative - note that for most of 
> the past week, the average ping time was 85msec. Now we're seeing this:
> 
> From 98.177.135.186 - the public IP address on Cox's local broadband service.
> To 107.154.13.58 (ntcorp.server - one of our servers, sitting in a Tierpoint 
> data center, near Boston)
> traceroute to ntcorp.com (207.154.13.58), 64 hops max, 52 byte packets
> 1  10.128.128.1 (10.128.128.1)  199.893 ms  75.319 ms  27.295 ms
> 2  100.127.69.178 (100.127.69.178)  38.710 ms  40.075 ms  43.598 ms
> 3  72.215.229.22 (72.215.229.22)  39.674 ms  201.368 ms *
> 4  lag-157.bear2.phoenix1.level3.net (4.28.82.53)  686.499 ms 1837.141 ms  
> 16.273 ms
> 5  ae-1-60.edge1.losangeles6.level3.net (4.69.144.16)  35.498 ms 964.377 ms *
> 6  ae-3-80.edge1.losangeles6.level3.net (4.69.144.144)  551.760 ms  525.014 ms
>ae-1-60.edge1.losangeles6.level3.net (4.69.144.16)  2061.191 ms
> 7  be3036.ccr41.lax04.atlas.cogentco.com (154.54.14.129)  847.778 ms  87.601 
> ms  71.504 ms
> 8  be2965.ccr22.lax01.atlas.cogentco.com (154.54.45.1)  79.060 ms  225.647 ms
>be2964.ccr21.lax01.atlas.cogentco.com (154.54.44.77)  60.306 ms
> 9  * be2931.ccr21.phx02.atlas.cogentco.com (154.54.44.85) 2264.071 ms  
> 185.180 ms
> 10  be2929.ccr21.elp01.atlas.cogentco.com (154.54.42.66)  61.208 ms
>be2930.ccr21.elp01.atlas.cogentco.com (154.54.42.78)  386.149 ms  1278.868 
> ms
> 11  be2928.ccr22.iah01.atlas.cogentco.com (154.54.30.161)  384.136 ms
>be2927.ccr41.iah01.atlas.cogentco.com (154.54.29.221) 2339.833 ms  615.415 
> ms
> 12  be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129)  233.061 ms
>be2687.ccr41.atl01.atlas.cogentco.com (154.54.28.69)  87.902 ms
>be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129)  861.159 ms
> 13  be2113.ccr42.dca01.atlas.cogentco.com (154.54.24.221)  998.858 ms
>be2112.ccr41.dca01.atlas.cogentco.com (154.54.7.157)  249.930 ms *
> 14  be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109)  768.461 ms
>be2806.ccr41.jfk02.atlas.cogentco.com (154.54.40.105)  136.772 ms
>be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109)  288.225 ms
> 15  be2094.ccr21.bos01.atlas.cogentco.com (154.54.30.14)  271.736 ms  166.224 
> ms
>be2096.ccr22.bos01.atlas.cogentco.com (154.54.30.42)  565.015 ms
> 16  te0-4-1-7.agr22.bos01.atlas.cogentco.com (154.54.80.34) 1944.479 ms
>te0-4-1-6.agr22.bos01.atlas.cogentco.com (154.54.80.10) 149.803 ms
>te0-4-1-6.agr21.bos01.atlas.cogentco.com (154.54.47.230) 897.115 ms
> 17  te0-0-2-0.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.82)  107.207 
> ms
>te0-0-2-1.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.78)  295.881 
> ms  185.453 ms
> 18  38.122.127.18 (38.122.127.18)  115.652 ms  461.168 ms  615.526 ms
> 19  207.154.0.57 (207.154.0.57)  1871.023 ms  1987.832 ms 2165.248 ms
> 20  server1.ntcorp.com (207.154.13.58)  587.560 ms  263.328 ms 333.542 ms
> 
> 
> Traceroute in the reverse direction:
> 1  207.154.13.47 (207.154.13.47)  0.000 ms  0.000 ms  0.000 ms
> 2  * * *
> 3  h130.207.190.173.static.ip.windstream.net (173.190.207.130) 0.000 ms  
> 0.000 ms  0.000 ms
> 4  xe1-2-0-0.cr01.cley01-oh.us.windstream.net (40.128.250.166) 12.000 ms  
> 12.000 ms  12.000 ms
> 5  et11-0-0-0.cr01.chcg01-il.us.windstream.net (40.128.248.71) 20.000 ms  
> 20.000 ms  20.000 ms
> 6  10gigabitethernet4-1.core1.chi1.he.NET (206.223.119.37)  72.001 ms  20.000 
> ms  16.000 ms
> 7  chgobbrj01pos010100.r2.ch.cox.net (68.105.30.193)  20.000 ms 20.000 ms  
> 20.000 ms
> 8  chnddsrj01-ae1.0.rd.ph.cox.net (68.1.5.211)  72.001 ms  72.001 ms  72.001 
> ms
> 9  * * *
> 10  * * *
> 11  * * *
> 
> 30  * * *
> 
> 
> Two things jump out at me:
> 1. The rather large number of hops from cox to ntcorp - with high delays  
> from several nodes in both the level3 and cogent networks.
> 2. That there's a rather more direct path from the datacenter to cox, that 
> shows up in the reverse direction.
> 
> Some kind of routing or peering issue, perhaps?  (And I also note the earlier 
> string of messages regarding youtube streaming problems - that also seemed to 
> involve cox and level3.
> 
> Thanks for any insight (and better, for any fixes!).

is something weird going on with cox, level3, and/or cogent

2017-02-12 Thread Miles Fidelman 

Hi Folks,

I'm visiting AZ, and seeing some really really poor performance 
accessing some of our servers via Cox broadband.  The folks at Cox 
technical support are useless - all they say is "well you're on a DOCSIS 
2 modem."  Meanwhile, everything I'm seeing is several hops upstream of 
the local segment - and all that Cox level2 tech support will say is "if 
there was a backbone problem, our backbone people would have dealt with it."


I'm having problems reaching both our own server, and sites like google, 
facebook, windows update.


Traceroutes to and from our server are illustrative - note that for most 
of the past week, the average ping time was 85msec. Now we're seeing this:


From 98.177.135.186 - the public IP address on Cox's local broadband 
service.
To 107.154.13.58 (ntcorp.server - one of our servers, sitting in a 
Tierpoint data center, near Boston)

traceroute to ntcorp.com (207.154.13.58), 64 hops max, 52 byte packets
 1  10.128.128.1 (10.128.128.1)  199.893 ms  75.319 ms  27.295 ms
 2  100.127.69.178 (100.127.69.178)  38.710 ms  40.075 ms  43.598 ms
 3  72.215.229.22 (72.215.229.22)  39.674 ms  201.368 ms *
 4  lag-157.bear2.phoenix1.level3.net (4.28.82.53)  686.499 ms 1837.141 
ms  16.273 ms
 5  ae-1-60.edge1.losangeles6.level3.net (4.69.144.16)  35.498 ms 
964.377 ms *
 6  ae-3-80.edge1.losangeles6.level3.net (4.69.144.144)  551.760 ms  
525.014 ms

ae-1-60.edge1.losangeles6.level3.net (4.69.144.16)  2061.191 ms
 7  be3036.ccr41.lax04.atlas.cogentco.com (154.54.14.129)  847.778 ms  
87.601 ms  71.504 ms
 8  be2965.ccr22.lax01.atlas.cogentco.com (154.54.45.1)  79.060 ms  
225.647 ms

be2964.ccr21.lax01.atlas.cogentco.com (154.54.44.77)  60.306 ms
 9  * be2931.ccr21.phx02.atlas.cogentco.com (154.54.44.85) 2264.071 ms  
185.180 ms

10  be2929.ccr21.elp01.atlas.cogentco.com (154.54.42.66)  61.208 ms
be2930.ccr21.elp01.atlas.cogentco.com (154.54.42.78)  386.149 ms  
1278.868 ms

11  be2928.ccr22.iah01.atlas.cogentco.com (154.54.30.161)  384.136 ms
be2927.ccr41.iah01.atlas.cogentco.com (154.54.29.221) 2339.833 ms  
615.415 ms

12  be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129)  233.061 ms
be2687.ccr41.atl01.atlas.cogentco.com (154.54.28.69)  87.902 ms
be2690.ccr42.atl01.atlas.cogentco.com (154.54.28.129)  861.159 ms
13  be2113.ccr42.dca01.atlas.cogentco.com (154.54.24.221)  998.858 ms
be2112.ccr41.dca01.atlas.cogentco.com (154.54.7.157)  249.930 ms *
14  be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109)  768.461 ms
be2806.ccr41.jfk02.atlas.cogentco.com (154.54.40.105)  136.772 ms
be2807.ccr42.jfk02.atlas.cogentco.com (154.54.40.109)  288.225 ms
15  be2094.ccr21.bos01.atlas.cogentco.com (154.54.30.14)  271.736 ms  
166.224 ms

be2096.ccr22.bos01.atlas.cogentco.com (154.54.30.42)  565.015 ms
16  te0-4-1-7.agr22.bos01.atlas.cogentco.com (154.54.80.34) 1944.479 ms
te0-4-1-6.agr22.bos01.atlas.cogentco.com (154.54.80.10) 149.803 ms
te0-4-1-6.agr21.bos01.atlas.cogentco.com (154.54.47.230) 897.115 ms
17  te0-0-2-0.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.82)  
107.207 ms
te0-0-2-1.nr11.b000254-0.bos01.atlas.cogentco.com (154.24.9.78)  
295.881 ms  185.453 ms

18  38.122.127.18 (38.122.127.18)  115.652 ms  461.168 ms  615.526 ms
19  207.154.0.57 (207.154.0.57)  1871.023 ms  1987.832 ms 2165.248 ms
20  server1.ntcorp.com (207.154.13.58)  587.560 ms  263.328 ms 333.542 ms


Traceroute in the reverse direction:
 1  207.154.13.47 (207.154.13.47)  0.000 ms  0.000 ms  0.000 ms
 2  * * *
 3  h130.207.190.173.static.ip.windstream.net (173.190.207.130) 0.000 
ms  0.000 ms  0.000 ms
 4  xe1-2-0-0.cr01.cley01-oh.us.windstream.net (40.128.250.166) 12.000 
ms  12.000 ms  12.000 ms
 5  et11-0-0-0.cr01.chcg01-il.us.windstream.net (40.128.248.71) 20.000 
ms  20.000 ms  20.000 ms
 6  10gigabitethernet4-1.core1.chi1.he.NET (206.223.119.37)  72.001 ms  
20.000 ms  16.000 ms
 7  chgobbrj01pos010100.r2.ch.cox.net (68.105.30.193)  20.000 ms 20.000 
ms  20.000 ms
 8  chnddsrj01-ae1.0.rd.ph.cox.net (68.1.5.211)  72.001 ms  72.001 ms  
72.001 ms

 9  * * *
10  * * *
11  * * *

30  * * *


Two things jump out at me:
1. The rather large number of hops from cox to ntcorp - with high 
delays  from several nodes in both the level3 and cogent networks.
2. That there's a rather more direct path from the datacenter to cox, 
that shows up in the reverse direction.


Some kind of routing or peering issue, perhaps?  (And I also note the 
earlier string of messages regarding youtube streaming problems - that 
also seemed to involve cox and level3.


Thanks for any insight (and better, for any fixes!).

Miles Fidelman




--
In theory, there is no difference between theory and practice.
In practice, there is.   Yogi Berra

Re: YouTube streaming failures

2017-02-12 Thread Jay Farrell via NANOG 
Youtube is aware, according to a boilerplate message in their support forum:

Hi there, welcome to the YouTube Help Forum!

"YouTube is aware of the issue. Please stay tuned to YouTube's social media
and the forums for any announcement of a fix.

Thanks for reporting!"

On Sun, Feb 12, 2017 at 9:32 PM, Jay Farrell  wrote:

> Downdetector shows a big spike in reports for youtube in the past several
> hours.
>
> http://downdetector.com/status/youtube
>
> On Sun, Feb 12, 2017 at 9:06 PM, Christopher Morrow <
> morrowc.li...@gmail.com> wrote:
>
>> verizon wired, comcast (on a mobile device) both work in IAD's area.
>>
>> On Sun, Feb 12, 2017 at 8:53 PM, Patrick W. Gilmore 
>> wrote:
>>
>> > I cannot stream on AppleTV or iPhone. Works on my laptop.
>> >
>> > Comcast, Massachusetts.
>> >
>> > --
>> > TTFN,
>> > patrick
>> >
>> > > On Feb 12, 2017, at 8:08 PM, Brett A Mansfield <
>> > li...@silverlakeinternet.com> wrote:
>> > >
>> > > I'm seeing this as well, but only on Apple and Linux products. Seems
>> to
>> > be working fine on Windows.
>> > >
>> > > Thank you,
>> > > Brett A Mansfield
>> > >
>> > >> On Feb 12, 2017, at 5:30 PM, Mel Beckman  wrote:
>> > >>
>> > >> We are getting many customer reports of YouTube streaming failures.
>> The
>> > content directory and search work, but attempts to view videos results
>> in
>> > "something went wrong, click to try again" error messages. We've
>> reproduced
>> > the problem on AT, Level3, Frontier, Cox and Comast networks. We are
>> also
>> > seeing it on cellular data connections, which tends to rule out geo-IP
>> > errors. Is anyone else seeing this?
>> > >>
>> > >> -mel beckman
>> >
>> >
>>
>
>

Re: YouTube streaming failures

2017-02-12 Thread Jay Farrell via NANOG 
Downdetector shows a big spike in reports for youtube in the past several
hours.

http://downdetector.com/status/youtube

On Sun, Feb 12, 2017 at 9:06 PM, Christopher Morrow  wrote:

> verizon wired, comcast (on a mobile device) both work in IAD's area.
>
> On Sun, Feb 12, 2017 at 8:53 PM, Patrick W. Gilmore 
> wrote:
>
> > I cannot stream on AppleTV or iPhone. Works on my laptop.
> >
> > Comcast, Massachusetts.
> >
> > --
> > TTFN,
> > patrick
> >
> > > On Feb 12, 2017, at 8:08 PM, Brett A Mansfield <
> > li...@silverlakeinternet.com> wrote:
> > >
> > > I'm seeing this as well, but only on Apple and Linux products. Seems to
> > be working fine on Windows.
> > >
> > > Thank you,
> > > Brett A Mansfield
> > >
> > >> On Feb 12, 2017, at 5:30 PM, Mel Beckman  wrote:
> > >>
> > >> We are getting many customer reports of YouTube streaming failures.
> The
> > content directory and search work, but attempts to view videos results in
> > "something went wrong, click to try again" error messages. We've
> reproduced
> > the problem on AT, Level3, Frontier, Cox and Comast networks. We are
> also
> > seeing it on cellular data connections, which tends to rule out geo-IP
> > errors. Is anyone else seeing this?
> > >>
> > >> -mel beckman
> >
> >
>

Re: YouTube streaming failures

2017-02-12 Thread Christopher Morrow 
verizon wired, comcast (on a mobile device) both work in IAD's area.

On Sun, Feb 12, 2017 at 8:53 PM, Patrick W. Gilmore 
wrote:

> I cannot stream on AppleTV or iPhone. Works on my laptop.
>
> Comcast, Massachusetts.
>
> --
> TTFN,
> patrick
>
> > On Feb 12, 2017, at 8:08 PM, Brett A Mansfield <
> li...@silverlakeinternet.com> wrote:
> >
> > I'm seeing this as well, but only on Apple and Linux products. Seems to
> be working fine on Windows.
> >
> > Thank you,
> > Brett A Mansfield
> >
> >> On Feb 12, 2017, at 5:30 PM, Mel Beckman  wrote:
> >>
> >> We are getting many customer reports of YouTube streaming failures. The
> content directory and search work, but attempts to view videos results in
> "something went wrong, click to try again" error messages. We've reproduced
> the problem on AT, Level3, Frontier, Cox and Comast networks. We are also
> seeing it on cellular data connections, which tends to rule out geo-IP
> errors. Is anyone else seeing this?
> >>
> >> -mel beckman
>
>

Re: YouTube streaming failures

2017-02-12 Thread Patrick W. Gilmore 
I cannot stream on AppleTV or iPhone. Works on my laptop.

Comcast, Massachusetts.

-- 
TTFN,
patrick

> On Feb 12, 2017, at 8:08 PM, Brett A Mansfield  
> wrote:
> 
> I'm seeing this as well, but only on Apple and Linux products. Seems to be 
> working fine on Windows.
> 
> Thank you,
> Brett A Mansfield
> 
>> On Feb 12, 2017, at 5:30 PM, Mel Beckman  wrote:
>> 
>> We are getting many customer reports of YouTube streaming failures. The 
>> content directory and search work, but attempts to view videos results in 
>> "something went wrong, click to try again" error messages. We've reproduced 
>> the problem on AT, Level3, Frontier, Cox and Comast networks. We are also 
>> seeing it on cellular data connections, which tends to rule out geo-IP 
>> errors. Is anyone else seeing this?
>> 
>> -mel beckman

Re: YouTube streaming failures

2017-02-12 Thread Brett A Mansfield 
I'm seeing this as well, but only on Apple and Linux products. Seems to be 
working fine on Windows.

Thank you,
Brett A Mansfield

> On Feb 12, 2017, at 5:30 PM, Mel Beckman  wrote:
> 
> We are getting many customer reports of YouTube streaming failures. The 
> content directory and search work, but attempts to view videos results in 
> "something went wrong, click to try again" error messages. We've reproduced 
> the problem on AT, Level3, Frontier, Cox and Comast networks. We are also 
> seeing it on cellular data connections, which tends to rule out geo-IP 
> errors. Is anyone else seeing this?
> 
> -mel beckman

YouTube streaming failures

2017-02-12 Thread Mel Beckman 
We are getting many customer reports of YouTube streaming failures. The content 
directory and search work, but attempts to view videos results in "something 
went wrong, click to try again" error messages. We've reproduced the problem on 
AT, Level3, Frontier, Cox and Comast networks. We are also seeing it on 
cellular data connections, which tends to rule out geo-IP errors. Is anyone 
else seeing this?

 -mel beckman