Re: Sad IPv4 story?
On Dec 9, 2011, at 1:37 PM, Franck Martin wrote: I just had a personal email from a brand new ISP in the Asia-Pacific area desperately looking for enough IPv4 to be able to run their business the way they would like… This is just a data point. Franck - Thanks for the data point - I'm certain there are others folks out there with similar experiences that we're not hearing about. Of course, the theory was that at this point they'd be able to use IPv6 to connect customers up to the Internet. Such theory was predicated on a presumed strong motivation for everyone already connected via IPv4 to deploy IPv6 in parallel (i.e. dual-stack) and some elusive TBD transition mechanisms which were to make IPv6 customers interoperate with those that hadn't yet deployed IPv6 in parallel. Reality looks very different, in that existing organizations find it difficult to understand why to add IPv6 connectivity to their existing public-facing servers, and the state of the art in achieving transparent operation for IPv6 connected systems to the rest of the Internet running only IPv4 is still effectively a work-in-progress... While your data point may be from the Asia-Pacific region, that same story is going to repeated in every region (RIPE NCC will be running out shortly, and ARIN has 1 to 2 years depending on the actual request rate that materializes) Service providers in the ARIN region need to carefully consider their answer to that same situation, because it will be occurring here soon enough. There is one thing that everyone can do to reduce the impact of this transition, and this is getting in front of their business customers (and small business/power users who have public-facing content) to explain that the Internet is going be running IPv4 and IPv6 for quite some time in parallel and that getting their public-facing servers connected up also via IPv6 is a very good idea (if anyone wants help doing this sort of customer education ARIN's https://www.arin.net/knowledge, NRO's http://www.nro.net/ipv6, and APNIC's IPv6 Act Now http://www.ipv6actnow.org web sites are all good sources on materials for this sort of effort.) The sooner we get the content on IPv6 in addition to IPv4, the sooner that connecting new customers up via IPv6 without additional unique IPv4 address space becomes viable (and obviously if we had the vast majority of content already on IPv6, then connecting new customers via IPv6 would be simple indeed.) FYI, /John John Curran President and CEO ARIN
Re: Sad IPv4 story?
On 11/12/2011, at 2:37 PM, valdis.kletni...@vt.edu wrote: On Sat, 10 Dec 2011 20:48:45 EST, Barry Shein said: I just had a personal email from a brand new ISP in the Asia-Pacific area desperately looking for enough IPv4 to be able to run their business the way they would like? This sniping elicited by the above seems inappropriate and unprofessional, the request/anecdote seemed reasonable and could elicit solutions such as partnerships, etc. No Barry, I respectfully disagree. It's almost 2012. The first predictions of IPv4 exhaustion were made *last century*. We've been predicting it to the month level for like 5 years now. Any business that is making business plans and models that doesn't take we may not get IPv4 space into account and have a contingency plan for that *deserves* to be soundly mocked and ridiculed in public. You could take this one step further and say any industry that has had this much warning and hasn’t taken it into account *deserves* to be soundly mocked and ridiculed in public.
Re: Inaccessible network from Verizon, accessible elsewhere.
On 12/10/11, NetSecGuy netsec...@gmail.com wrote: I have a Linode VPS in Japan that I can't access from Verizon FIOS, but can access from other locations. I'm not sure who to blame. I can't get to 106.187.34.33 or 106.187.34.1 using Verizon FIOS C:\tracert 106.187.34.33 Tracing route to li377-33.members.linode.com [106.187.34.33] over a maximum of 30 hops: [.. snip ..] 523 ms 4 ms 4 ms so-14-0-0-0.RES-BB-RTR2.verizon-gni.net [130.81.22.56] 673 ms 6 ms 7 ms 0.ae2.BR2.IAD8.ALTER.NET [152.63.34.73] 7 8 ms 6 ms 7 ms dcp-brdr-03.inet.qwest.net [63.146.26.105] 8 8 ms 9 ms 9 ms sl-crs1-dc-0-1-0-0.sprintlink.net [144.232.19.229] 928 ms26 ms44 ms sl-crs1-dc-0-5-3-0.sprintlink.net [144.232.24.37] 10 177 ms 176 ms 177 ms lajbb001.kddnet.ad.jp [59.128.2.173] 1143 ms41 ms42 ms sl-crs1-oma-0-9-2-0.sprintlink.net [144.232.2.177] 12 291 ms * 301 ms cm-fcu203.kddnet.ad.jp [124.215.194.164] 13 286 ms 279 ms 282 ms 124.215.199.122 1481 ms81 ms82 ms sl-crs1-sj-0-5-3-0.sprintlink.net [144.232.20.99] 1588 ms86 ms87 ms sl-st20-pa-9-0-0.sprintlink.net [144.232.8.108] 16 405 ms 406 ms 399 ms 144.223.243.126 17 364 ms 386 ms 406 ms pajbb001.kddnet.ad.jp [111.87.3.41] 18 *** Request timed out. 19 *** Request timed out. 20 *** Request timed out. 21 *** Request timed out. 22 *** Request timed out. 23 ** ^C C:\tracert 106.187.34.1 Tracing route to gw-li377.linode.com [106.187.34.1] over a maximum of 30 hops: [.. snip ..] 5 5 ms24 ms24 ms so-3-1-0-0.RES-BB-RTR2.verizon-gni.net [130.81.151.232] 6 7 ms 7 ms 7 ms 0.ae2.BR2.IAD8.ALTER.NET [152.63.34.73] 7 8 ms 7 ms 7 ms dcp-brdr-03.inet.qwest.net [63.146.26.105] 884 ms84 ms84 ms lap-brdr-03.inet.qwest.net [67.14.22.78] 9 171 ms 174 ms 176 ms 63.146.26.70 10 178 ms 177 ms 177 ms lajbb001.kddnet.ad.jp [59.128.2.173] 11 283 ms 284 ms 284 ms otejbb203.kddnet.ad.jp [203.181.100.9] 12 289 ms 287 ms 287 ms cm-fcu203.kddnet.ad.jp [124.215.194.164] 13 *** Request timed out. 1483 ms81 ms82 ms sl-crs1-sj-0-12-0-1.sprintlink.net [144.232.9.224] 15 *** Request timed out. 16 403 ms 407 ms 404 ms 144.223.243.126 17 *** Request timed out. 18 501 ms 499 ms 501 ms otejbb203.kddnet.ad.jp.100.181.203.in-addr.arpa [203.181.100.137] 19 *** Request timed out. 20 *** Request timed out. 21 *** Request timed out. 22 *** Request timed out. 23 *** Request timed out. 24 *** Request timed out. 25 * ^C Lee The host, 106.187.34.33, is behind the gateway 106.187.34.1: From FIOS to 106.187.34.1 (this works). traceroute to 106.187.34.1 (106.187.34.1), 64 hops max, 52 byte packets 4 so-6-1-0-0.phil-bb-rtr2.verizon-gni.net (130.81.199.4) 9.960 ms 9.957 ms 6.666 ms 5 so-8-0-0-0.lcc1-res-bb-rtr1-re1.verizon-gni.net (130.81.17.3) 12.298 ms 13.463 ms 13.706 ms 6 0.ae2.br1.iad8.alter.net (152.63.32.158) 14.571 ms 14.372 ms 14.003 ms 7 204.255.169.218 (204.255.169.218) 14.692 ms 14.759 ms 13.670 ms 8 sl-crs1-dc-0-1-0-0.sprintlink.net (144.232.19.229) 13.077 ms 12.577 ms 14.954 ms 9 sl-crs1-nsh-0-5-5-0.sprintlink.net (144.232.18.200) 31.443 ms sl-crs1-dc-0-5-3-0.sprintlink.net (144.232.24.37) 33.005 ms sl-crs1-nsh-0-5-5-0.sprintlink.net (144.232.18.200) 31.507 ms 10 sl-crs1-kc-0-0-0-2.sprintlink.net (144.232.18.112) 57.610 ms 58.322 ms 59.098 ms 11 otejbb204.kddnet.ad.jp (203.181.100.45) 196.063 ms otejbb203.kddnet.ad.jp (203.181.100.13) 188.846 ms otejbb204.kddnet.ad.jp (203.181.100.21) 195.277 ms 12 cm-fcu203.kddnet.ad.jp (124.215.194.180) 214.760 ms cm-fcu203.kddnet.ad.jp (124.215.194.164) 198.925 ms cm-fcu203.kddnet.ad.jp (124.215.194.180) 200.583 ms 13 124.215.199.122 (124.215.199.122) 193.086 ms * 194.967 ms This does not work from FIOS: traceroute to 106.187.34.33 (106.187.34.33), 64 hops max, 52 byte packets 4 so-6-1-0-0.phil-bb-rtr2.verizon-gni.net (130.81.199.4) 34.229 ms 8.743 ms 8.878 ms 5 so-8-0-0-0.lcc1-res-bb-rtr1-re1.verizon-gni.net (130.81.17.3) 15.402 ms 13.008 ms 14.932 ms 6 0.ae2.br1.iad8.alter.net (152.63.32.158) 13.325 ms 13.245 ms 13.802 ms 7 204.255.169.218 (204.255.169.218) 14.820 ms 14.232 ms 13.491 ms 8 lap-brdr-03.inet.qwest.net (67.14.22.78) 90.170 ms 92.273 ms 145.887 ms 9 63.146.26.70 (63.146.26.70) 92.482 ms 92.287 ms 94.000 ms 10 sl-crs1-kc-0-0-0-2.sprintlink.net (144.232.18.112) 58.135 ms 58.520
Re: Inaccessible network from Verizon, accessible elsewhere.
I should have included reverse traces to begin with. No firewall on VPS. Trace from the VPS to a router close to me. traceroute to 130.81.199.4 (130.81.199.4), 64 hops max, 40 byte packets 1 106.187.33.2 (106.187.33.2) 1 ms 0 ms 0 ms 2 124.215.199.121 (124.215.199.121) 6 ms 1 ms 13 ms 3 59.128.4.121 (59.128.4.121) 2 ms otejbb204.kddnet.ad.jp (124.215.194.177) 2 ms 2 ms 4 lajbb001.kddnet.ad.jp (203.181.100.14) 126 ms 100 ms lajbb002.kddnet.ad.jp (203.181.100.22) 162 ms 5 ix-la1.kddnet.ad.jp (59.128.2.70) 108 ms ix-la1.kddnet.ad.jp (59.128.2.178) 102 ms 102 ms 6 lap-brdr-03.inet.qwest.net (63.146.26.69) 99 ms 101 ms 99 ms 7 63.146.26.210 (63.146.26.210) 99 ms 101 ms 99 ms 8 0.ae3.XL3.LAX15.ALTER.NET (152.63.113.186) 102 ms 102 ms 101 ms 9 * * * 10 * * * Tracer from VPS to a router close to my other location, not Verizon. traceroute to 4.59.244.49 (4.59.244.49), 64 hops max, 40 byte packets 1 106.187.33.2 (106.187.33.2) 1 ms 1 ms 1 ms 2 124.215.199.121 (124.215.199.121) 9 ms 1 ms 1 ms 3 59.128.4.121 (59.128.4.121) 2 ms otejbb204.kddnet.ad.jp (124.215.194.177) 9 ms 59.128.4.121 (59.128.4.121) 2 ms 4 lajbb001.kddnet.ad.jp (203.181.100.18) 108 ms lajbb002.kddnet.ad.jp (203.181.100.22) 101 ms 101 ms 5 ix-la2.kddnet.ad.jp (59.128.2.102) 116 ms 116 ms ix-la2.kddnet.ad.jp (59.128.2.186) 125 ms 6 xe-11-3-0.edge2.LosAngeles9.Level3.net (4.53.228.13) 111 ms 101 ms 101 ms 7 vlan70.csw2.LosAngeles1.Level3.net (4.69.144.126) 110 ms vlan90.csw4.LosAngeles1.Level3.net (4.69.144.254) 108 ms vlan60.csw1.LosAngeles1.Level3.net (4.69.144.62) 103 ms 8 ae-63-63.ebr3.LosAngeles1.Level3.net (4.69.137.33) 110 ms 117 ms ae-73-73.ebr3.LosAngeles1.Level3.net (4.69.137.37) 108 ms 9 ae-4-4.ebr4.Washington1.Level3.net (4.69.132.82) 178 ms 180 ms 166 ms 10 ae-64-64.csw1.Washington1.Level3.net (4.69.134.178) 174 ms 166 ms 166 ms 11 ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) 172 ms 165 ms 172 ms 12 ae-8-8.car2.Baltimore1.Level3.net (4.69.134.106) 181 ms 174 ms 174 ms 13 ae-11-11.car1.Baltimore1.Level3.net (4.69.134.109) 181 ms * 174 ms
Re: Inaccessible network from Verizon, accessible elsewhere.
I believe 130.81 is blocked. Traceroute to your gateway address. -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. NetSecGuy netsec...@gmail.com wrote: I should have included reverse traces to begin with. No firewall on VPS. Trace from the VPS to a router close to me. traceroute to 130.81.199.4 (130.81.199.4), 64 hops max, 40 byte packets 1 106.187.33.2 (106.187.33.2) 1 ms 0 ms 0 ms 2 124.215.199.121 (124.215.199.121) 6 ms 1 ms 13 ms 3 59.128.4.121 (59.128.4.121) 2 ms otejbb204.kddnet.ad.jp (124.215.194.177) 2 ms 2 ms 4 lajbb001.kddnet.ad.jp (203.181.100.14) 126 ms 100 ms lajbb002.kddnet.ad.jp (203.181.100.22) 162 ms 5 ix-la1.kddnet.ad.jp (59.128.2.70) 108 ms ix-la1.kddnet.ad.jp (59.128.2.178) 102 ms 102 ms 6 lap-brdr-03.inet.qwest.net (63.146.26.69) 99 ms 101 ms 99 ms 7 63.146.26.210 (63.146.26.210) 99 ms 101 ms 99 ms 8 0.ae3.XL3.LAX15.ALTER.NET (152.63.113.186) 102 ms 102 ms 101 ms 9 * * * 10 * * * Tracer from VPS to a router close to my other location, not Verizon. traceroute to 4.59.244.49 (4.59.244.49), 64 hops max, 40 byte packets 1 106.187.33.2 (106.187.33.2) 1 ms 1 ms 1 ms 2 124.215.199.121 (124.215.199.121) 9 ms 1 ms 1 ms 3 59.128.4.121 (59.128.4.121) 2 ms otejbb204.kddnet.ad.jp (124.215.194.177) 9 ms 59.128.4.121 (59.128.4.121) 2 ms 4 lajbb001.kddnet.ad.jp (203.181.100.18) 108 ms lajbb002.kddnet.ad.jp (203.181.100.22) 101 ms 101 ms 5 ix-la2.kddnet.ad.jp (59.128.2.102) 116 ms 116 ms ix-la2.kddnet.ad.jp (59.128.2.186) 125 ms 6 xe-11-3-0.edge2.LosAngeles9.Level3.net (4.53.228.13) 111 ms 101 ms 101 ms 7 vlan70.csw2.LosAngeles1.Level3.net (4.69.144.126) 110 ms vlan90.csw4.LosAngeles1.Level3.net (4.69.144.254) 108 ms vlan60.csw1.LosAngeles1.Level3.net (4.69.144.62) 103 ms 8 ae-63-63.ebr3.LosAngeles1.Level3.net (4.69.137.33) 110 ms 117 ms ae-73-73.ebr3.LosAngeles1.Level3.net (4.69.137.37) 108 ms 9 ae-4-4.ebr4.Washington1.Level3.net (4.69.132.82) 178 ms 180 ms 166 ms 10 ae-64-64.csw1.Washington1.Level3.net (4.69.134.178) 174 ms 166 ms 166 ms 11 ae-62-62.ebr2.Washington1.Level3.net (4.69.134.145) 172 ms 165 ms 172 ms 12 ae-8-8.car2.Baltimore1.Level3.net (4.69.134.106) 181 ms 174 ms 174 ms 13 ae-11-11.car1.Baltimore1.Level3.net (4.69.134.109) 181 ms * 174 ms
Re: Inaccessible network from Verizon, accessible elsewhere.
On Sat, Dec 10, 2011 at 11:49 AM, NetSecGuy netsec...@gmail.com wrote: I have a Linode VPS in Japan that I can't access from Verizon FIOS, but can access from other locations. I'm not sure who to blame. The host, 106.187.34.33, is behind the gateway 106.187.34.1: From FIOS to 106.187.34.1 (this works). traceroute to 106.187.34.1 (106.187.34.1), 64 hops max, 52 byte packets 4 so-6-1-0-0.phil-bb-rtr2.verizon-gni.net (130.81.199.4) 9.960 ms 9.957 ms 6.666 ms 5 so-8-0-0-0.lcc1-res-bb-rtr1-re1.verizon-gni.net (130.81.17.3) 12.298 ms 13.463 ms 13.706 ms 6 0.ae2.br1.iad8.alter.net (152.63.32.158) 14.571 ms 14.372 ms 14.003 ms 7 204.255.169.218 (204.255.169.218) 14.692 ms 14.759 ms 13.670 ms 8 sl-crs1-dc-0-1-0-0.sprintlink.net (144.232.19.229) 13.077 ms 12.577 ms 14.954 ms 9 sl-crs1-nsh-0-5-5-0.sprintlink.net (144.232.18.200) 31.443 ms sl-crs1-dc-0-5-3-0.sprintlink.net (144.232.24.37) 33.005 ms sl-crs1-nsh-0-5-5-0.sprintlink.net (144.232.18.200) 31.507 ms 10 sl-crs1-kc-0-0-0-2.sprintlink.net (144.232.18.112) 57.610 ms 58.322 ms 59.098 ms 11 otejbb204.kddnet.ad.jp (203.181.100.45) 196.063 ms otejbb203.kddnet.ad.jp (203.181.100.13) 188.846 ms otejbb204.kddnet.ad.jp (203.181.100.21) 195.277 ms 12 cm-fcu203.kddnet.ad.jp (124.215.194.180) 214.760 ms cm-fcu203.kddnet.ad.jp (124.215.194.164) 198.925 ms cm-fcu203.kddnet.ad.jp (124.215.194.180) 200.583 ms 13 124.215.199.122 (124.215.199.122) 193.086 ms * 194.967 ms This does not work from FIOS: traceroute to 106.187.34.33 (106.187.34.33), 64 hops max, 52 byte packets 4 so-6-1-0-0.phil-bb-rtr2.verizon-gni.net (130.81.199.4) 34.229 ms 8.743 ms 8.878 ms 5 so-8-0-0-0.lcc1-res-bb-rtr1-re1.verizon-gni.net (130.81.17.3) 15.402 ms 13.008 ms 14.932 ms 6 0.ae2.br1.iad8.alter.net (152.63.32.158) 13.325 ms 13.245 ms 13.802 ms 7 204.255.169.218 (204.255.169.218) 14.820 ms 14.232 ms 13.491 ms 8 lap-brdr-03.inet.qwest.net (67.14.22.78) 90.170 ms 92.273 ms 145.887 ms 9 63.146.26.70 (63.146.26.70) 92.482 ms 92.287 ms 94.000 ms 10 sl-crs1-kc-0-0-0-2.sprintlink.net (144.232.18.112) 58.135 ms 58.520 ms 58.055 ms 11 otejbb203.kddnet.ad.jp (203.181.100.17) 205.844 ms otejbb204.kddnet.ad.jp (203.181.100.25) 189.929 ms otejbb203.kddnet.ad.jp (203.181.100.17) 204.846 ms 12 sl-crs1-oro-0-1-5-0.sprintlink.net (144.232.25.77) 87.229 ms sl-crs1-oro-0-3-3-0.sprintlink.net (144.232.25.207) 88.796 ms 88.717 ms 13 124.215.199.122 (124.215.199.122) 193.584 ms 202.208 ms 192.989 ms 14 * * * Same IP from different network: traceroute to 106.187.34.33 (106.187.34.33), 30 hops max, 60 byte packets 6 ae-8-8.ebr2.Washington1.Level3.net (4.69.134.105) 2.230 ms 1.847 ms 1.938 ms 7 ae-92-92.csw4.Washington1.Level3.net (4.69.134.158) 2.010 ms 1.985 ms ae-62-62.csw1.Washington1.Level3.net (4.69.134.146) 1.942 ms 8 ae-94-94.ebr4.Washington1.Level3.net (4.69.134.189) 12.515 ms ae-74-74.ebr4.Washington1.Level3.net (4.69.134.181) 12.519 ms 12.507 ms 9 ae-4-4.ebr3.LosAngeles1.Level3.net (4.69.132.81) 65.957 ms 65.958 ms 66.056 ms 10 ae-83-83.csw3.LosAngeles1.Level3.net (4.69.137.42) 66.063 ms ae-93-93.csw4.LosAngeles1.Level3.net (4.69.137.46) 65.985 ms ae-63-63.csw1.LosAngeles1.Level3.net (4.69.137.34) 66.026 ms 11 ae-3-80.edge2.LosAngeles9.Level3.net (4.69.144.143) 66.162 ms 66.160 ms 66.238 ms 12 KDDI-AMERIC.edge2.LosAngeles9.Level3.net (4.53.228.14) 193.317 ms 193.447 ms 193.305 ms 13 lajbb001.kddnet.ad.jp (59.128.2.101) 101.544 ms 101.543 ms lajbb002.kddnet.ad.jp (59.128.2.185) 66.563 ms 14 otejbb203.kddnet.ad.jp (203.181.100.13) 164.217 ms 164.221 ms 164.330 ms 15 cm-fcu203.kddnet.ad.jp (124.215.194.164) 180.350 ms cm-fcu203.kddnet.ad.jp (124.215.194.180) 172.779 ms cm-fcu203.kddnet.ad.jp (124.215.194.164) 185.824 ms 16 124.215.199.122 (124.215.199.122) 175.703 ms 175.700 ms 168.268 ms 17 li377-33.members.linode.com (106.187.34.33) 174.381 ms 174.383 ms 174.368 ms In doing a little probing right now, from various source addresses, I'm unable to reproduce the problem. I've seen failures similar to this one (where the source address matters; some work, some don't) when multi-port LAGs or ECMP paths have a single link in them fail, but are still detected and forwarded over as if it was up. This can happen, for example, if you run a LAG with no channeling protocol (like LACP or PAGP), that hashes source and destination IPs to pick a link (to ensure consistent paths per-path, and with ports, per-flow). If one of those links fails in the underlying media or physical path, but the link is still detected as up, packets to some IPs (but not others) will just drop on the flor. Now, in this particular case, it doesn't seem like the path to both destinations seem like they even take the same path (so that previous hypothesis is pure conjecture). Perhaps routes were actively
RE: Inaccessible network from Verizon, accessible elsewhere.
From 90701 - Artesia, CA. FIOS No Go here too!!! C:\WINDOWS\system32tracert 106.187.34.1 Tracing route to gw-li377.linode.com [106.187.34.1] over a maximum of 30 hops: 122 ms34 ms1 ms Tomato [192.168.100.1] 249 ms 1 ms 1 ms Verizon [192.168.1.1] 336 ms 6 ms 6 ms L100.LSANCA-VFTTP-114.verizon-gni.net [173.58.21 1.1] 424 ms 9 ms 9 ms G0-9-1-4.LSANCA-LCR-21.verizon-gni.net [130.81.1 85.72] 524 ms 9 ms 8 ms so-4-1-0-0.LAX01-BB-RTR1.verizon-gni.net [130.81 .151.246] 624 ms 9 ms 8 ms 0.ae1.BR3.LAX15.ALTER.NET [152.63.2.129] 738 ms 8 ms 8 ms ae6.edge1.LosAngeles9.level3.net [4.68.62.169] 825 ms10 ms10 ms 63.146.26.70 924 ms 9 ms 8 ms lajbb001.kddnet.ad.jp [59.128.2.173] 1024 ms 9 ms 8 ms lajbb001.kddnet.ad.jp [59.128.2.181] 11 140 ms 110 ms 108 ms otejbb203.kddnet.ad.jp [203.181.100.9] 12 140 ms 124 ms 111 ms cm-fcu203.kddnet.ad.jp [124.215.194.164] 13 *** Request timed out. 14 *** Request timed out. 15 *** Request timed out. 16 *** Request timed out. 17 *** Request timed out. 18 *** Request timed out. 19 *** Request timed out. 20 *** Request timed out. 21 *** Request timed out. 22 *** Request timed out. 23 *** Request timed out. 24 * ^C C:\WINDOWS\system32tracert 106.187.34.33 Tracing route to li377-33.members.linode.com [106.187.34.33] over a maximum of 30 hops: 122 ms 1 ms1 ms Tomato [192.168.100.1] 231 ms 1 ms 1 ms Verizon [192.168.1.1] 351 ms10 ms11 ms L100.LSANCA-VFTTP-114.verizon-gni.net [173.58.21 1.1] 442 ms 9 ms33 ms G0-9-1-4.LSANCA-LCR-21.verizon-gni.net [130.81.1 85.72] 540 ms15 ms 9 ms so-4-1-0-0.LAX01-BB-RTR1.verizon-gni.net [130.81 .151.246] 631 ms 8 ms 8 ms 0.ae1.BR3.LAX15.ALTER.NET [152.63.2.129] 761 ms10 ms16 ms lap-brdr-03.inet.qwest.net [63.146.26.209] 831 ms10 ms10 ms 63.146.26.70 931 ms 9 ms 9 ms lajbb001.kddnet.ad.jp [59.128.2.173] 1031 ms 9 ms 8 ms lajbb001.kddnet.ad.jp [59.128.2.181] 11 125 ms 118 ms 109 ms otejbb203.kddnet.ad.jp [203.181.100.9] 12 156 ms 111 ms 143 ms 124.215.199.122 13 126 ms 112 ms 137 ms 124.215.199.122 14 *** Request timed out. 15 *** Request timed out. 16 *** Request timed out. 17 ** ^C C:\WINDOWS\system32 E = 4:32Cheers!!! -Original Message- From: Lee [mailto:ler...@gmail.com] Sent: Sunday, December 11, 2011 6:44 AM To: NetSecGuy Cc: nanog@nanog.org Subject: Re: Inaccessible network from Verizon, accessible elsewhere. On 12/10/11, NetSecGuy netsec...@gmail.com wrote: I have a Linode VPS in Japan that I can't access from Verizon FIOS, but can access from other locations. I'm not sure who to blame. I can't get to 106.187.34.33 or 106.187.34.1 using Verizon FIOS C:\tracert 106.187.34.33 Tracing route to li377-33.members.linode.com [106.187.34.33] over a maximum of 30 hops: [.. snip ..] 523 ms 4 ms 4 ms so-14-0-0-0.RES-BB-RTR2.verizon-gni.net [130.81.22.56] 673 ms 6 ms 7 ms 0.ae2.BR2.IAD8.ALTER.NET [152.63.34.73] 7 8 ms 6 ms 7 ms dcp-brdr-03.inet.qwest.net [63.146.26.105] 8 8 ms 9 ms 9 ms sl-crs1-dc-0-1-0-0.sprintlink.net [144.232.19.229] 928 ms26 ms44 ms sl-crs1-dc-0-5-3-0.sprintlink.net [144.232.24.37] 10 177 ms 176 ms 177 ms lajbb001.kddnet.ad.jp [59.128.2.173] 1143 ms41 ms42 ms sl-crs1-oma-0-9-2-0.sprintlink.net [144.232.2.177] 12 291 ms * 301 ms cm-fcu203.kddnet.ad.jp [124.215.194.164] 13 286 ms 279 ms 282 ms 124.215.199.122 1481 ms81 ms82 ms sl-crs1-sj-0-5-3-0.sprintlink.net [144.232.20.99] 1588 ms86 ms87 ms sl-st20-pa-9-0-0.sprintlink.net [144.232.8.108] 16 405 ms 406 ms 399 ms 144.223.243.126 17 364 ms 386 ms 406 ms pajbb001.kddnet.ad.jp [111.87.3.41] 18 *** Request timed out. 19 *** Request timed out. 20 *** Request timed out. 21 *** Request timed out. 22 *** Request timed out. 23 ** ^C C:\tracert 106.187.34.1 Tracing route to gw-li377.linode.com [106.187.34.1] over a maximum of 30 hops: [.. snip ..] 5 5 ms24 ms24 ms so-3-1-0-0.RES-BB-RTR2.verizon-gni.net [130.81.151.232] 6 7 ms 7 ms 7 ms 0.ae2.BR2.IAD8.ALTER.NET [152.63.34.73] 7 8 ms 7 ms 7 ms dcp-brdr-03.inet.qwest.net
Re: Overall Netflix bandwidth usage numbers on a network?
Feel free to contact peering@netflixdotcom - we're happy to provide you with delivery statistics for traffic terminating on your network. Regards, -Dave Temkin Netflix On 12/7/11 8:57 AM, Blake Hudson wrote: Yeah, that's an interesting one. We currently utilize netflow for this, but you also need to consider that netflix streaming is just port 80 www traffic. Because netflix uses CDNs, its difficult to pin down the traffic to specific hosts in the CDN and say that this traffic was netflix, while this traffic was the latest windows update (remember this is often a shared hosting platform). We've done our own testing and have come to a good solution which uses a combination of nbar, packet marking, and netflow to come to a conclusion. On a ~160Mbps link, netflix peaks out between 30-50Mbps around 8-10PM each evening. The rest of the traffic is predominantly other forms of HTTP traffic (including other video streaming services). Martin Hepworth wrote the following on 12/3/2011 2:36 AM: Also checkout Adrian Cockcroft presentations on their architecture which describes how they use aws and CDns etc Martin
RE: Inaccessible network from Verizon, accessible elsewhere.
I hope it's not an outdated martian problem firewall or route filter. For the Traceroute from linode to FiOS, Traceroute to the FiOS gateway address. -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. Network IP Dog network.ip...@gmail.com wrote: From 90701 - Artesia, CA. FIOS No Go here too!!! C:\WINDOWS\system32tracert 106.187.34.1 Tracing route to gw-li377.linode.com [106.187.34.1] over a maximum of 30 hops: 1 22 ms 34 ms 1 ms Tomato [192.168.100.1] 2 49 ms 1 ms 1 ms Verizon [192.168.1.1] 3 36 ms 6 ms 6 ms L100.LSANCA-VFTTP-114.verizon-gni.net [173.58.21 1.1] 4 24 ms 9 ms 9 ms G0-9-1-4.LSANCA-LCR-21.verizon-gni.net [130.81.1 85.72] 5 24 ms 9 ms 8 ms so-4-1-0-0.LAX01-BB-RTR1.verizon-gni.net [130.81 .151.246] 6 24 ms 9 ms 8 ms 0.ae1.BR3.LAX15.ALTER.NET [152.63.2.129] 7 38 ms 8 ms 8 ms ae6.edge1.LosAngeles9.level3.net [4.68.62.169] 8 25 ms 10 ms 10 ms 63.146.26.70 9 24 ms 9 ms 8 ms lajbb001.kddnet.ad.jp [59.128.2.173] 10 24 ms 9 ms 8 ms lajbb001.kddnet.ad.jp [59.128.2.181] 11 140 ms 110 ms 108 ms otejbb203.kddnet.ad.jp [203.181.100.9] 12 140 ms 124 ms 111 ms cm-fcu203.kddnet.ad.jp [124.215.194.164] 13 * * * Request timed out. 14 * * * Request timed out. 15 * * * Request timed out. 16 * * * Request timed out. 17 * * * Request timed out. 18 * * * Request timed out. 19 * * * Request timed out. 20 * * * Request timed out. 21 * * * Request timed out. 22 * * * Request timed out. 23 * * * Request timed out. 24 * ^C C:\WINDOWS\system32tracert 106.187.34.33 Tracing route to li377-33.members.linode.com [106.187.34.33] over a maximum of 30 hops: 1 22 ms 1 ms 1 ms Tomato [192.168.100.1] 2 31 ms 1 ms 1 ms Verizon [192.168.1.1] 3 51 ms 10 ms 11 ms L100.LSANCA-VFTTP-114.verizon-gni.net [173.58.21 1.1] 4 42 ms 9 ms 33 ms G0-9-1-4.LSANCA-LCR-21.verizon-gni.net [130.81.1 85.72] 5 40 ms 15 ms 9 ms so-4-1-0-0.LAX01-BB-RTR1.verizon-gni.net [130.81 .151.246] 6 31 ms 8 ms 8 ms 0.ae1.BR3.LAX15.ALTER.NET [152.63.2.129] 7 61 ms 10 ms 16 ms lap-brdr-03.inet.qwest.net [63.146.26.209] 8 31 ms 10 ms 10 ms 63.146.26.70 9 31 ms 9 ms 9 ms lajbb001.kddnet.ad.jp [59.128.2.173] 10 31 ms 9 ms 8 ms lajbb001.kddnet.ad.jp [59.128.2.181] 11 125 ms 118 ms 109 ms otejbb203.kddnet.ad.jp [203.181.100.9] 12 156 ms 111 ms 143 ms 124.215.199.122 13 126 ms 112 ms 137 ms 124.215.199.122 14 * * * Request timed out. 15 * * * Request timed out. 16 * * * Request timed out. 17 * * ^C C:\WINDOWS\system32 E = 4:32 Cheers!!! -Original Message- From: Lee [mailto:ler...@gmail.com] Sent: Sunday, December 11, 2011 6:44 AM To: NetSecGuy Cc: nanog@nanog.org Subject: Re: Inaccessible network from Verizon, accessible elsewhere. On 12/10/11, NetSecGuy netsec...@gmail.com wrote: I have a Linode VPS in Japan that I can't access from Verizon FIOS, but can access from other locations. I'm not sure who to blame. I can't get to 106.187.34.33 or 106.187.34.1 using Verizon FIOS C:\tracert 106.187.34.33 Tracing route to li377-33.members.linode.com [106.187.34.33] over a maximum of 30 hops: [.. snip ..] 5 23 ms 4 ms 4 ms so-14-0-0-0.RES-BB-RTR2.verizon-gni.net [130.81.22.56] 6 73 ms 6 ms 7 ms 0.ae2.BR2.IAD8.ALTER.NET [152.63.34.73] 7 8 ms 6 ms 7 ms dcp-brdr-03.inet.qwest.net [63.146.26.105] 8 8 ms 9 ms 9 ms sl-crs1-dc-0-1-0-0.sprintlink.net [144.232.19.229] 9 28 ms 26 ms 44 ms sl-crs1-dc-0-5-3-0.sprintlink.net [144.232.24.37] 10 177 ms 176 ms 177 ms lajbb001.kddnet.ad.jp [59.128.2.173] 11 43 ms 41 ms 42 ms sl-crs1-oma-0-9-2-0.sprintlink.net [144.232.2.177] 12 291 ms * 301 ms cm-fcu203.kddnet.ad.jp [124.215.194.164] 13 286 ms 279 ms 282 ms 124.215.199.122 14 81 ms 81 ms 82 ms sl-crs1-sj-0-5-3-0.sprintlink.net [144.232.20.99] 15 88 ms 86 ms 87 ms sl-st20-pa-9-0-0.sprintlink.net [144.232.8.108] 16 405 ms 406 ms 399 ms 144.223.243.126 17 364 ms 386 ms 406 ms pajbb001.kddnet.ad.jp [111.87.3.41] 18 * * * Request timed out. 19 * * * Request timed out. 20 * * * Request timed out. 21 * * * Request timed out. 22 * * * Request timed out. 23 * * ^C C:\tracert 106.187.34.1 Tracing route to gw-li377.linode.com [106.187.34.1] over a maximum of 30 hops: [.. snip ..] 5 5 ms 24 ms 24 ms so-3-1-0-0.RES-BB-RTR2.verizon-gni.net [130.81.151.232] 6 7 ms 7 ms 7 ms 0.ae2.BR2.IAD8.ALTER.NET [152.63.34.73] 7 8 ms 7 ms 7 ms dcp-brdr-03.inet.qwest.net [63.146.26.105] 8 84 ms 84 ms 84 ms lap-brdr-03.inet.qwest.net [67.14.22.78] 9 171 ms 174 ms 176 ms 63.146.26.70 10 178 ms 177 ms 177 ms lajbb001.kddnet.ad.jp [59.128.2.173] 11 283 ms 284 ms 284 ms otejbb203.kddnet.ad.jp [203.181.100.9] 12 289 ms 287 ms 287 ms cm-fcu203.kddnet.ad.jp [124.215.194.164] 13 * * * Request timed out. 14 83 ms 81 ms 82 ms sl-crs1-sj-0-12-0-1.sprintlink.net [144.232.9.224] 15 * * * Request timed out. 16 403 ms 407 ms 404 ms 144.223.243.126 17 * * * Request timed out. 18 501 ms 499 ms 501 ms otejbb203.kddnet.ad.jp.100.181.203.in-addr.arpa [203.181.100.137] 19 * * * Request timed out. 20 * * * Request timed out. 21 * * * Request timed out. 22 * * * Request timed out. 23 * * * Request
Re: Inaccessible network from Verizon, accessible elsewhere.
from home lan % traceroute gw-li377.linode.com traceroute to gw-li377.linode.com (106.187.34.1), 64 hops max, 52 byte packets 1 192.168.0.1 (192.168.0.1) 1.471 ms 0.725 ms 0.555 ms 2 tokyo10-f03.flets.2iij.net (210.149.34.72) 7.241 ms 6.651 ms 6.939 ms 3 tokyo10-ntteast0.flets.2iij.net (210.149.34.157) 5.573 ms 6.109 ms 5.346 ms 4 tky001lip20.iij.net (210.149.34.97) 6.410 ms 7.471 ms 7.934 ms 5 tky001bb10.iij.net (58.138.100.209) 6.670 ms 9.251 ms 5.866 ms 6 tky009bf00.iij.net (58.138.80.17) 6.730 ms tky008bf02.iij.net (58.138.80.13) 7.021 ms tky009bf00.iij.net (58.138.80.17) 8.593 ms 7 tky001ix05.iij.net (58.138.82.2) 9.767 ms tky001ix05.iij.net (58.138.82.6) 6.101 ms tky001ix01.iij.net (58.138.80.106) 8.420 ms 8 203.181.102.61 (203.181.102.61) 19.514 ms 203.181.102.21 (203.181.102.21) 6.054 ms 203.181.102.61 (203.181.102.61) 11.478 ms 9 otejbb203.kddnet.ad.jp (118.155.197.129) 7.457 ms otejbb203.kddnet.ad.jp (59.128.7.129) 7.835 ms otejbb204.kddnet.ad.jp (59.128.7.130) 7.824 ms 10 cm-fcu203.kddnet.ad.jp (124.215.194.180) 15.860 ms 16.401 ms cm-fcu203.kddnet.ad.jp (124.215.194.164) 17.519 ms 11 124.215.199.122 (124.215.199.122) 7.892 ms * 11.984 ms
RE: Inaccessible network from Verizon, accessible elsewhere.
I too am now experiencing issues. I cannot get to www.cisco.com and various websites. Some websites work lightning quick, some take a long time to load, and some just don't load at all. Date: Mon, 12 Dec 2011 09:55:40 +0900 From: ra...@psg.com To: nanog@nanog.org Subject: Re: Inaccessible network from Verizon, accessible elsewhere. from home lan % traceroute gw-li377.linode.com traceroute to gw-li377.linode.com (106.187.34.1), 64 hops max, 52 byte packets 1 192.168.0.1 (192.168.0.1) 1.471 ms 0.725 ms 0.555 ms 2 tokyo10-f03.flets.2iij.net (210.149.34.72) 7.241 ms 6.651 ms 6.939 ms 3 tokyo10-ntteast0.flets.2iij.net (210.149.34.157) 5.573 ms 6.109 ms 5.346 ms 4 tky001lip20.iij.net (210.149.34.97) 6.410 ms 7.471 ms 7.934 ms 5 tky001bb10.iij.net (58.138.100.209) 6.670 ms 9.251 ms 5.866 ms 6 tky009bf00.iij.net (58.138.80.17) 6.730 ms tky008bf02.iij.net (58.138.80.13) 7.021 ms tky009bf00.iij.net (58.138.80.17) 8.593 ms 7 tky001ix05.iij.net (58.138.82.2) 9.767 ms tky001ix05.iij.net (58.138.82.6) 6.101 ms tky001ix01.iij.net (58.138.80.106) 8.420 ms 8 203.181.102.61 (203.181.102.61) 19.514 ms 203.181.102.21 (203.181.102.21) 6.054 ms 203.181.102.61 (203.181.102.61) 11.478 ms 9 otejbb203.kddnet.ad.jp (118.155.197.129) 7.457 ms otejbb203.kddnet.ad.jp (59.128.7.129) 7.835 ms otejbb204.kddnet.ad.jp (59.128.7.130) 7.824 ms 10 cm-fcu203.kddnet.ad.jp (124.215.194.180) 15.860 ms 16.401 ms cm-fcu203.kddnet.ad.jp (124.215.194.164) 17.519 ms 11 124.215.199.122 (124.215.199.122) 7.892 ms * 11.984 ms
Re: Overall Netflix bandwidth usage numbers on a network?
Which leads to a question to be asked... Is netflix willing to peer directly with ISP / NSP's ? Regards. Faisal Imtiaz Snappy Internet Telecom On 12/11/2011 7:29 PM, Dave Temkin wrote: Feel free to contact peering@netflixdotcom - we're happy to provide you with delivery statistics for traffic terminating on your network. Regards, -Dave Temkin Netflix On 12/7/11 8:57 AM, Blake Hudson wrote: Yeah, that's an interesting one. We currently utilize netflow for this, but you also need to consider that netflix streaming is just port 80 www traffic. Because netflix uses CDNs, its difficult to pin down the traffic to specific hosts in the CDN and say that this traffic was netflix, while this traffic was the latest windows update (remember this is often a shared hosting platform). We've done our own testing and have come to a good solution which uses a combination of nbar, packet marking, and netflow to come to a conclusion. On a ~160Mbps link, netflix peaks out between 30-50Mbps around 8-10PM each evening. The rest of the traffic is predominantly other forms of HTTP traffic (including other video streaming services). Martin Hepworth wrote the following on 12/3/2011 2:36 AM: Also checkout Adrian Cockcroft presentations on their architecture which describes how they use aws and CDns etc Martin
Re: Overall Netflix bandwidth usage numbers on a network?
Netflix uses CDNs for content delivery and the platform runs in EC2. What would peering with them achieve? Sent from my iPhone On Dec 11, 2011, at 18:06, Faisal Imtiaz fai...@snappydsl.net wrote: Which leads to a question to be asked... Is netflix willing to peer directly with ISP / NSP's ? Regards. Faisal Imtiaz Snappy Internet Telecom On 12/11/2011 7:29 PM, Dave Temkin wrote: Feel free to contact peering@netflixdotcom - we're happy to provide you with delivery statistics for traffic terminating on your network. Regards, -Dave Temkin Netflix On 12/7/11 8:57 AM, Blake Hudson wrote: Yeah, that's an interesting one. We currently utilize netflow for this, but you also need to consider that netflix streaming is just port 80 www traffic. Because netflix uses CDNs, its difficult to pin down the traffic to specific hosts in the CDN and say that this traffic was netflix, while this traffic was the latest windows update (remember this is often a shared hosting platform). We've done our own testing and have come to a good solution which uses a combination of nbar, packet marking, and netflow to come to a conclusion. On a ~160Mbps link, netflix peaks out between 30-50Mbps around 8-10PM each evening. The rest of the traffic is predominantly other forms of HTTP traffic (including other video streaming services). Martin Hepworth wrote the following on 12/3/2011 2:36 AM: Also checkout Adrian Cockcroft presentations on their architecture which describes how they use aws and CDns etc Martin
Re: Inaccessible network from Verizon, accessible elsewhere.
I'm seeing the same thing from my home lan via fios. I've run a recursive dns server for years and can't reach the roots. Had to switch to using verizon's dns servers as forwarders. Sent from my iPad On Dec 11, 2011, at 8:07 PM, Brandon Kim brandon@brandontek.com wrote: I too am now experiencing issues. I cannot get to www.cisco.com and various websites. Some websites work lightning quick, some take a long time to load, and some just don't load at all. Date: Mon, 12 Dec 2011 09:55:40 +0900 From: ra...@psg.com To: nanog@nanog.org Subject: Re: Inaccessible network from Verizon, accessible elsewhere. from home lan % traceroute gw-li377.linode.com traceroute to gw-li377.linode.com (106.187.34.1), 64 hops max, 52 byte packets 1 192.168.0.1 (192.168.0.1) 1.471 ms 0.725 ms 0.555 ms 2 tokyo10-f03.flets.2iij.net (210.149.34.72) 7.241 ms 6.651 ms 6.939 ms 3 tokyo10-ntteast0.flets.2iij.net (210.149.34.157) 5.573 ms 6.109 ms 5.346 ms 4 tky001lip20.iij.net (210.149.34.97) 6.410 ms 7.471 ms 7.934 ms 5 tky001bb10.iij.net (58.138.100.209) 6.670 ms 9.251 ms 5.866 ms 6 tky009bf00.iij.net (58.138.80.17) 6.730 ms tky008bf02.iij.net (58.138.80.13) 7.021 ms tky009bf00.iij.net (58.138.80.17) 8.593 ms 7 tky001ix05.iij.net (58.138.82.2) 9.767 ms tky001ix05.iij.net (58.138.82.6) 6.101 ms tky001ix01.iij.net (58.138.80.106) 8.420 ms 8 203.181.102.61 (203.181.102.61) 19.514 ms 203.181.102.21 (203.181.102.21) 6.054 ms 203.181.102.61 (203.181.102.61) 11.478 ms 9 otejbb203.kddnet.ad.jp (118.155.197.129) 7.457 ms otejbb203.kddnet.ad.jp (59.128.7.129) 7.835 ms otejbb204.kddnet.ad.jp (59.128.7.130) 7.824 ms 10 cm-fcu203.kddnet.ad.jp (124.215.194.180) 15.860 ms 16.401 ms cm-fcu203.kddnet.ad.jp (124.215.194.164) 17.519 ms 11 124.215.199.122 (124.215.199.122) 7.892 ms * 11.984 ms
Re: Overall Netflix bandwidth usage numbers on a network?
On Sun, 11 Dec 2011 19:21:49 PST, Joel Jaeggli said: Netflix uses CDNs for content delivery and the platform runs in EC2. What would peering with them achieve? I suspect Faisal's *real* question is Who at Netflix do I talk to in order to discuss mutually beneficial traffic engineering? pgpdlV6fIFIdZ.pgp Description: PGP signature
Re: Overall Netflix bandwidth usage numbers on a network?
Simple, keep traffic off paid ip transit circuits Faisal On Dec 11, 2011, at 10:21 PM, Joel Jaeggli joe...@bogus.com wrote: Netflix uses CDNs for content delivery and the platform runs in EC2. What would peering with them achieve? Sent from my iPhone On Dec 11, 2011, at 18:06, Faisal Imtiaz fai...@snappydsl.net wrote: Which leads to a question to be asked... Is netflix willing to peer directly with ISP / NSP's ? Regards. Faisal Imtiaz Snappy Internet Telecom On 12/11/2011 7:29 PM, Dave Temkin wrote: Feel free to contact peering@netflixdotcom - we're happy to provide you with delivery statistics for traffic terminating on your network. Regards, -Dave Temkin Netflix On 12/7/11 8:57 AM, Blake Hudson wrote: Yeah, that's an interesting one. We currently utilize netflow for this, but you also need to consider that netflix streaming is just port 80 www traffic. Because netflix uses CDNs, its difficult to pin down the traffic to specific hosts in the CDN and say that this traffic was netflix, while this traffic was the latest windows update (remember this is often a shared hosting platform). We've done our own testing and have come to a good solution which uses a combination of nbar, packet marking, and netflow to come to a conclusion. On a ~160Mbps link, netflix peaks out between 30-50Mbps around 8-10PM each evening. The rest of the traffic is predominantly other forms of HTTP traffic (including other video streaming services). Martin Hepworth wrote the following on 12/3/2011 2:36 AM: Also checkout Adrian Cockcroft presentations on their architecture which describes how they use aws and CDns etc Martin
Re: Inaccessible network from Verizon, accessible elsewhere.
On Sun, Dec 11, 2011 at 3:11 PM, Joseph Snyder joseph.sny...@gmail.com wrote: I believe 130.81 is blocked. Traceroute to your gateway address. portions (at least) of that are 19262's loopback/ptp space, they block/rate-limit toward that at their edge.
Re: Inaccessible network from Verizon, accessible elsewhere.
On Sun, Dec 11, 2011 at 10:28 PM, Matthew Huff mh...@ox.com wrote: I'm seeing the same thing from my home lan via fios. I've run a recursive dns server for years and can't reach the roots. Had to switch to using verizon's dns servers as forwarders. business or consumer fios? 3 G0-9-4-7.WASHDC-LCR-22.verizon-gni.net (130.81.104.180) 6.662 ms 6.739 ms 6.788 ms 4 so-14-0-0-0.RES-BB-RTR2.verizon-gni.net (130.81.22.56) 6.852 ms 15.384 ms 8.184 ms 5 0.ae2.BR1.IAD8.ALTER.NET (152.63.32.158) 12.857 ms 12.927 ms 13.004 ms 6 dcp-brdr-03.inet.qwest.net (63.146.26.105) 12.429 ms 7.847 ms 6.464 ms 7 lap-brdr-03.inet.qwest.net (67.14.22.78) 89.140 ms 88.929 ms 89.032 ms 8 63.146.26.70 (63.146.26.70) 94.879 ms 94.580 ms 93.120 ms 9 sl-crs1-kc-0-0-0-2.sprintlink.net (144.232.18.112) 58.520 ms 58.330 ms 58.186 ms 10 144.232.25.193 (144.232.25.193) 49.950 ms sl-crs1-oma-0-9-2-0.sprintlink.net (144.232.2.177) 49.962 ms sl-crs1-oma-0-8-0-0.sprintlink.net (144.232.8.171) 47.687 ms 11 sl-crs1-oro-0-3-3-0.sprintlink.net (144.232.25.207) 84.416 ms 83.266 ms sl-crs1-oro-0-12-3-0.sprintlink.net (144.232.25.73) 84.667 ms 12 124.215.199.122 (124.215.199.122) 195.590 ms * * all of this seems to point at some kddi.net rouer gobbling packets, no? (since pretty much everyone's got the same terminating hop) - also note that while some folks traverse L3, my route is via qwest... it's interesting that 701 isn't picking their other peer (sprint) here directly, no? Sent from my iPad On Dec 11, 2011, at 8:07 PM, Brandon Kim brandon@brandontek.com wrote: I too am now experiencing issues. I cannot get to www.cisco.com and various websites. Some websites work lightning quick, some take a long time to load, and some just don't load at all. Date: Mon, 12 Dec 2011 09:55:40 +0900 From: ra...@psg.com To: nanog@nanog.org Subject: Re: Inaccessible network from Verizon, accessible elsewhere. from home lan % traceroute gw-li377.linode.com traceroute to gw-li377.linode.com (106.187.34.1), 64 hops max, 52 byte packets 1 192.168.0.1 (192.168.0.1) 1.471 ms 0.725 ms 0.555 ms 2 tokyo10-f03.flets.2iij.net (210.149.34.72) 7.241 ms 6.651 ms 6.939 ms 3 tokyo10-ntteast0.flets.2iij.net (210.149.34.157) 5.573 ms 6.109 ms 5.346 ms 4 tky001lip20.iij.net (210.149.34.97) 6.410 ms 7.471 ms 7.934 ms 5 tky001bb10.iij.net (58.138.100.209) 6.670 ms 9.251 ms 5.866 ms 6 tky009bf00.iij.net (58.138.80.17) 6.730 ms tky008bf02.iij.net (58.138.80.13) 7.021 ms tky009bf00.iij.net (58.138.80.17) 8.593 ms 7 tky001ix05.iij.net (58.138.82.2) 9.767 ms tky001ix05.iij.net (58.138.82.6) 6.101 ms tky001ix01.iij.net (58.138.80.106) 8.420 ms 8 203.181.102.61 (203.181.102.61) 19.514 ms 203.181.102.21 (203.181.102.21) 6.054 ms 203.181.102.61 (203.181.102.61) 11.478 ms 9 otejbb203.kddnet.ad.jp (118.155.197.129) 7.457 ms otejbb203.kddnet.ad.jp (59.128.7.129) 7.835 ms otejbb204.kddnet.ad.jp (59.128.7.130) 7.824 ms 10 cm-fcu203.kddnet.ad.jp (124.215.194.180) 15.860 ms 16.401 ms cm-fcu203.kddnet.ad.jp (124.215.194.164) 17.519 ms 11 124.215.199.122 (124.215.199.122) 7.892 ms * 11.984 ms
Re: Overall Netflix bandwidth usage numbers on a network?
On Sun, Dec 11, 2011 at 10:46 PM, Faisal Imtiaz fai...@snappydsl.net wrote: Simple, keep traffic off paid ip transit circuits (I think joel's point was: peer with amazon, done-and-done) Faisal On Dec 11, 2011, at 10:21 PM, Joel Jaeggli joe...@bogus.com wrote: Netflix uses CDNs for content delivery and the platform runs in EC2. What would peering with them achieve? Sent from my iPhone On Dec 11, 2011, at 18:06, Faisal Imtiaz fai...@snappydsl.net wrote: Which leads to a question to be asked... Is netflix willing to peer directly with ISP / NSP's ? Regards. Faisal Imtiaz Snappy Internet Telecom On 12/11/2011 7:29 PM, Dave Temkin wrote: Feel free to contact peering@netflixdotcom - we're happy to provide you with delivery statistics for traffic terminating on your network. Regards, -Dave Temkin Netflix On 12/7/11 8:57 AM, Blake Hudson wrote: Yeah, that's an interesting one. We currently utilize netflow for this, but you also need to consider that netflix streaming is just port 80 www traffic. Because netflix uses CDNs, its difficult to pin down the traffic to specific hosts in the CDN and say that this traffic was netflix, while this traffic was the latest windows update (remember this is often a shared hosting platform). We've done our own testing and have come to a good solution which uses a combination of nbar, packet marking, and netflow to come to a conclusion. On a ~160Mbps link, netflix peaks out between 30-50Mbps around 8-10PM each evening. The rest of the traffic is predominantly other forms of HTTP traffic (including other video streaming services). Martin Hepworth wrote the following on 12/3/2011 2:36 AM: Also checkout Adrian Cockcroft presentations on their architecture which describes how they use aws and CDns etc Martin
Re: Inaccessible network from Verizon, accessible elsewhere.
Consumer fios. Verizon forums are full of posts about it. Too tired this evening to worry about it. Sent from my iPad On Dec 11, 2011, at 10:48 PM, Christopher Morrow morrowc.li...@gmail.com wrote: On Sun, Dec 11, 2011 at 10:28 PM, Matthew Huff mh...@ox.com wrote: I'm seeing the same thing from my home lan via fios. I've run a recursive dns server for years and can't reach the roots. Had to switch to using verizon's dns servers as forwarders. business or consumer fios? 3 G0-9-4-7.WASHDC-LCR-22.verizon-gni.net (130.81.104.180) 6.662 ms 6.739 ms 6.788 ms 4 so-14-0-0-0.RES-BB-RTR2.verizon-gni.net (130.81.22.56) 6.852 ms 15.384 ms 8.184 ms 5 0.ae2.BR1.IAD8.ALTER.NET (152.63.32.158) 12.857 ms 12.927 ms 13.004 ms 6 dcp-brdr-03.inet.qwest.net (63.146.26.105) 12.429 ms 7.847 ms 6.464 ms 7 lap-brdr-03.inet.qwest.net (67.14.22.78) 89.140 ms 88.929 ms 89.032 ms 8 63.146.26.70 (63.146.26.70) 94.879 ms 94.580 ms 93.120 ms 9 sl-crs1-kc-0-0-0-2.sprintlink.net (144.232.18.112) 58.520 ms 58.330 ms 58.186 ms 10 144.232.25.193 (144.232.25.193) 49.950 ms sl-crs1-oma-0-9-2-0.sprintlink.net (144.232.2.177) 49.962 ms sl-crs1-oma-0-8-0-0.sprintlink.net (144.232.8.171) 47.687 ms 11 sl-crs1-oro-0-3-3-0.sprintlink.net (144.232.25.207) 84.416 ms 83.266 ms sl-crs1-oro-0-12-3-0.sprintlink.net (144.232.25.73) 84.667 ms 12 124.215.199.122 (124.215.199.122) 195.590 ms * * all of this seems to point at some kddi.net rouer gobbling packets, no? (since pretty much everyone's got the same terminating hop) - also note that while some folks traverse L3, my route is via qwest... it's interesting that 701 isn't picking their other peer (sprint) here directly, no? Sent from my iPad On Dec 11, 2011, at 8:07 PM, Brandon Kim brandon@brandontek.com wrote: I too am now experiencing issues. I cannot get to www.cisco.com and various websites. Some websites work lightning quick, some take a long time to load, and some just don't load at all. Date: Mon, 12 Dec 2011 09:55:40 +0900 From: ra...@psg.com To: nanog@nanog.org Subject: Re: Inaccessible network from Verizon, accessible elsewhere. from home lan % traceroute gw-li377.linode.com traceroute to gw-li377.linode.com (106.187.34.1), 64 hops max, 52 byte packets 1 192.168.0.1 (192.168.0.1) 1.471 ms 0.725 ms 0.555 ms 2 tokyo10-f03.flets.2iij.net (210.149.34.72) 7.241 ms 6.651 ms 6.939 ms 3 tokyo10-ntteast0.flets.2iij.net (210.149.34.157) 5.573 ms 6.109 ms 5.346 ms 4 tky001lip20.iij.net (210.149.34.97) 6.410 ms 7.471 ms 7.934 ms 5 tky001bb10.iij.net (58.138.100.209) 6.670 ms 9.251 ms 5.866 ms 6 tky009bf00.iij.net (58.138.80.17) 6.730 ms tky008bf02.iij.net (58.138.80.13) 7.021 ms tky009bf00.iij.net (58.138.80.17) 8.593 ms 7 tky001ix05.iij.net (58.138.82.2) 9.767 ms tky001ix05.iij.net (58.138.82.6) 6.101 ms tky001ix01.iij.net (58.138.80.106) 8.420 ms 8 203.181.102.61 (203.181.102.61) 19.514 ms 203.181.102.21 (203.181.102.21) 6.054 ms 203.181.102.61 (203.181.102.61) 11.478 ms 9 otejbb203.kddnet.ad.jp (118.155.197.129) 7.457 ms otejbb203.kddnet.ad.jp (59.128.7.129) 7.835 ms otejbb204.kddnet.ad.jp (59.128.7.130) 7.824 ms 10 cm-fcu203.kddnet.ad.jp (124.215.194.180) 15.860 ms 16.401 ms cm-fcu203.kddnet.ad.jp (124.215.194.164) 17.519 ms 11 124.215.199.122 (124.215.199.122) 7.892 ms * 11.984 ms
Re: Overall Netflix bandwidth usage numbers on a network?
Thanks for the explanation...did not consider that before...will investigate.., any tips that can be shared will be welcome. :) Faisal On Dec 11, 2011, at 10:49 PM, Christopher Morrow morrowc.li...@gmail.com wrote: On Sun, Dec 11, 2011 at 10:46 PM, Faisal Imtiaz fai...@snappydsl.net wrote: Simple, keep traffic off paid ip transit circuits (I think joel's point was: peer with amazon, done-and-done) Faisal On Dec 11, 2011, at 10:21 PM, Joel Jaeggli joe...@bogus.com wrote: Netflix uses CDNs for content delivery and the platform runs in EC2. What would peering with them achieve? Sent from my iPhone On Dec 11, 2011, at 18:06, Faisal Imtiaz fai...@snappydsl.net wrote: Which leads to a question to be asked... Is netflix willing to peer directly with ISP / NSP's ? Regards. Faisal Imtiaz Snappy Internet Telecom On 12/11/2011 7:29 PM, Dave Temkin wrote: Feel free to contact peering@netflixdotcom - we're happy to provide you with delivery statistics for traffic terminating on your network. Regards, -Dave Temkin Netflix On 12/7/11 8:57 AM, Blake Hudson wrote: Yeah, that's an interesting one. We currently utilize netflow for this, but you also need to consider that netflix streaming is just port 80 www traffic. Because netflix uses CDNs, its difficult to pin down the traffic to specific hosts in the CDN and say that this traffic was netflix, while this traffic was the latest windows update (remember this is often a shared hosting platform). We've done our own testing and have come to a good solution which uses a combination of nbar, packet marking, and netflow to come to a conclusion. On a ~160Mbps link, netflix peaks out between 30-50Mbps around 8-10PM each evening. The rest of the traffic is predominantly other forms of HTTP traffic (including other video streaming services). Martin Hepworth wrote the following on 12/3/2011 2:36 AM: Also checkout Adrian Cockcroft presentations on their architecture which describes how they use aws and CDns etc Martin
Re: Overall Netflix bandwidth usage numbers on a network?
On 12/11/11 19:49 , Christopher Morrow wrote: On Sun, Dec 11, 2011 at 10:46 PM, Faisal Imtiaz fai...@snappydsl.net wrote: Simple, keep traffic off paid ip transit circuits (I think joel's point was: peer with amazon, done-and-done) also probably your relationships to akamai and level3 Faisal On Dec 11, 2011, at 10:21 PM, Joel Jaeggli joe...@bogus.com wrote: Netflix uses CDNs for content delivery and the platform runs in EC2. What would peering with them achieve? Sent from my iPhone On Dec 11, 2011, at 18:06, Faisal Imtiaz fai...@snappydsl.net wrote: Which leads to a question to be asked... Is netflix willing to peer directly with ISP / NSP's ? Regards. Faisal Imtiaz Snappy Internet Telecom On 12/11/2011 7:29 PM, Dave Temkin wrote: Feel free to contact peering@netflixdotcom - we're happy to provide you with delivery statistics for traffic terminating on your network. Regards, -Dave Temkin Netflix On 12/7/11 8:57 AM, Blake Hudson wrote: Yeah, that's an interesting one. We currently utilize netflow for this, but you also need to consider that netflix streaming is just port 80 www traffic. Because netflix uses CDNs, its difficult to pin down the traffic to specific hosts in the CDN and say that this traffic was netflix, while this traffic was the latest windows update (remember this is often a shared hosting platform). We've done our own testing and have come to a good solution which uses a combination of nbar, packet marking, and netflow to come to a conclusion. On a ~160Mbps link, netflix peaks out between 30-50Mbps around 8-10PM each evening. The rest of the traffic is predominantly other forms of HTTP traffic (including other video streaming services). Martin Hepworth wrote the following on 12/3/2011 2:36 AM: Also checkout Adrian Cockcroft presentations on their architecture which describes how they use aws and CDns etc Martin
Re: Overall Netflix bandwidth usage numbers on a network?
On 12/12/2011, at 4:18 PM, Joel jaeggli wrote: also probably your relationships to akamai and level3 Probably want to add Limelight to that list as well (do Netflix even use Akamai these days?) -Shaun smime.p7s Description: S/MIME cryptographic signature
Re: Overall Netflix bandwidth usage numbers on a network?
On Sun, 11 Dec 2011, Christopher Morrow wrote: On Sun, Dec 11, 2011 at 10:46 PM, Faisal Imtiaz fai...@snappydsl.net wrote: Simple, keep traffic off paid ip transit circuits (I think joel's point was: peer with amazon, done-and-done) DirectConnect seems to be a good way to get a dedicated 1G or 10G link with AWS: http://aws.amazon.com/directconnect/ It's not settlement-free peering, but it's an option if you can't negotiate something. Maybe it will reduce costs in some use cases. Beckman --- Peter Beckman Internet Guy beck...@angryox.com http://www.angryox.com/ ---
Re: Inaccessible network from Verizon, accessible elsewhere.
On Sun, Dec 11, 2011 at 10:54 PM, Matthew Huff mh...@ox.com wrote: Consumer fios. Verizon forums are full of posts about it. Too tired this evening to worry about it. :( I'll have to do some testing when I get near a consumer fios then... So, they squash all DNS NOT to their complexes, that seems rather dastardly of them... considering they deployed that hateful paxfire/nominum garbage on their recursive servers :( -chris On Dec 11, 2011, at 10:48 PM, Christopher Morrow morrowc.li...@gmail.com wrote: On Sun, Dec 11, 2011 at 10:28 PM, Matthew Huff mh...@ox.com wrote: I'm seeing the same thing from my home lan via fios. I've run a recursive dns server for years and can't reach the roots. Had to switch to using verizon's dns servers as forwarders. business or consumer fios? 3 G0-9-4-7.WASHDC-LCR-22.verizon-gni.net (130.81.104.180) 6.662 ms 6.739 ms 6.788 ms 4 so-14-0-0-0.RES-BB-RTR2.verizon-gni.net (130.81.22.56) 6.852 ms 15.384 ms 8.184 ms 5 0.ae2.BR1.IAD8.ALTER.NET (152.63.32.158) 12.857 ms 12.927 ms 13.004 ms 6 dcp-brdr-03.inet.qwest.net (63.146.26.105) 12.429 ms 7.847 ms 6.464 ms 7 lap-brdr-03.inet.qwest.net (67.14.22.78) 89.140 ms 88.929 ms 89.032 ms 8 63.146.26.70 (63.146.26.70) 94.879 ms 94.580 ms 93.120 ms 9 sl-crs1-kc-0-0-0-2.sprintlink.net (144.232.18.112) 58.520 ms 58.330 ms 58.186 ms 10 144.232.25.193 (144.232.25.193) 49.950 ms sl-crs1-oma-0-9-2-0.sprintlink.net (144.232.2.177) 49.962 ms sl-crs1-oma-0-8-0-0.sprintlink.net (144.232.8.171) 47.687 ms 11 sl-crs1-oro-0-3-3-0.sprintlink.net (144.232.25.207) 84.416 ms 83.266 ms sl-crs1-oro-0-12-3-0.sprintlink.net (144.232.25.73) 84.667 ms 12 124.215.199.122 (124.215.199.122) 195.590 ms * * all of this seems to point at some kddi.net rouer gobbling packets, no? (since pretty much everyone's got the same terminating hop) - also note that while some folks traverse L3, my route is via qwest... it's interesting that 701 isn't picking their other peer (sprint) here directly, no? Sent from my iPad On Dec 11, 2011, at 8:07 PM, Brandon Kim brandon@brandontek.com wrote: I too am now experiencing issues. I cannot get to www.cisco.com and various websites. Some websites work lightning quick, some take a long time to load, and some just don't load at all. Date: Mon, 12 Dec 2011 09:55:40 +0900 From: ra...@psg.com To: nanog@nanog.org Subject: Re: Inaccessible network from Verizon, accessible elsewhere. from home lan % traceroute gw-li377.linode.com traceroute to gw-li377.linode.com (106.187.34.1), 64 hops max, 52 byte packets 1 192.168.0.1 (192.168.0.1) 1.471 ms 0.725 ms 0.555 ms 2 tokyo10-f03.flets.2iij.net (210.149.34.72) 7.241 ms 6.651 ms 6.939 ms 3 tokyo10-ntteast0.flets.2iij.net (210.149.34.157) 5.573 ms 6.109 ms 5.346 ms 4 tky001lip20.iij.net (210.149.34.97) 6.410 ms 7.471 ms 7.934 ms 5 tky001bb10.iij.net (58.138.100.209) 6.670 ms 9.251 ms 5.866 ms 6 tky009bf00.iij.net (58.138.80.17) 6.730 ms tky008bf02.iij.net (58.138.80.13) 7.021 ms tky009bf00.iij.net (58.138.80.17) 8.593 ms 7 tky001ix05.iij.net (58.138.82.2) 9.767 ms tky001ix05.iij.net (58.138.82.6) 6.101 ms tky001ix01.iij.net (58.138.80.106) 8.420 ms 8 203.181.102.61 (203.181.102.61) 19.514 ms 203.181.102.21 (203.181.102.21) 6.054 ms 203.181.102.61 (203.181.102.61) 11.478 ms 9 otejbb203.kddnet.ad.jp (118.155.197.129) 7.457 ms otejbb203.kddnet.ad.jp (59.128.7.129) 7.835 ms otejbb204.kddnet.ad.jp (59.128.7.130) 7.824 ms 10 cm-fcu203.kddnet.ad.jp (124.215.194.180) 15.860 ms 16.401 ms cm-fcu203.kddnet.ad.jp (124.215.194.164) 17.519 ms 11 124.215.199.122 (124.215.199.122) 7.892 ms * 11.984 ms
Re: Inaccessible network from Verizon, accessible elsewhere.
We're having strange issues in NYC metropolitan area. We can trace from Verizon FIOS to some IP addresses of our ASN 11579 block. Others don't work. The IP's that don't work seem to die at 130.81.107.228 on the Verizon network. Something is rotten in Denmark. Or NY. You know what I mean. On 12/12/2011 1:02 AM, Christopher Morrow wrote: On Sun, Dec 11, 2011 at 10:54 PM, Matthew Huffmh...@ox.com wrote: Consumer fios. Verizon forums are full of posts about it. Too tired this evening to worry about it. :( I'll have to do some testing when I get near a consumer fios then... So, they squash all DNS NOT to their complexes, that seems rather dastardly of them... considering they deployed that hateful paxfire/nominum garbage on their recursive servers :( -chris On Dec 11, 2011, at 10:48 PM, Christopher Morrowmorrowc.li...@gmail.com wrote: On Sun, Dec 11, 2011 at 10:28 PM, Matthew Huffmh...@ox.com wrote: I'm seeing the same thing from my home lan via fios. I've run a recursive dns server for years and can't reach the roots. Had to switch to using verizon's dns servers as forwarders. business or consumer fios? 3 G0-9-4-7.WASHDC-LCR-22.verizon-gni.net (130.81.104.180) 6.662 ms 6.739 ms 6.788 ms 4 so-14-0-0-0.RES-BB-RTR2.verizon-gni.net (130.81.22.56) 6.852 ms 15.384 ms 8.184 ms 5 0.ae2.BR1.IAD8.ALTER.NET (152.63.32.158) 12.857 ms 12.927 ms 13.004 ms 6 dcp-brdr-03.inet.qwest.net (63.146.26.105) 12.429 ms 7.847 ms 6.464 ms 7 lap-brdr-03.inet.qwest.net (67.14.22.78) 89.140 ms 88.929 ms 89.032 ms 8 63.146.26.70 (63.146.26.70) 94.879 ms 94.580 ms 93.120 ms 9 sl-crs1-kc-0-0-0-2.sprintlink.net (144.232.18.112) 58.520 ms 58.330 ms 58.186 ms 10 144.232.25.193 (144.232.25.193) 49.950 ms sl-crs1-oma-0-9-2-0.sprintlink.net (144.232.2.177) 49.962 ms sl-crs1-oma-0-8-0-0.sprintlink.net (144.232.8.171) 47.687 ms 11 sl-crs1-oro-0-3-3-0.sprintlink.net (144.232.25.207) 84.416 ms 83.266 ms sl-crs1-oro-0-12-3-0.sprintlink.net (144.232.25.73) 84.667 ms 12 124.215.199.122 (124.215.199.122) 195.590 ms * * all of this seems to point at some kddi.net rouer gobbling packets, no? (since pretty much everyone's got the same terminating hop) - also note that while some folks traverse L3, my route is via qwest... it's interesting that 701 isn't picking their other peer (sprint) here directly, no? Sent from my iPad On Dec 11, 2011, at 8:07 PM, Brandon Kimbrandon@brandontek.com wrote: I too am now experiencing issues. I cannot get to www.cisco.com and various websites. Some websites work lightning quick, some take a long time to load, and some just don't load at all. Date: Mon, 12 Dec 2011 09:55:40 +0900 From: ra...@psg.com To: nanog@nanog.org Subject: Re: Inaccessible network from Verizon, accessible elsewhere. from home lan % traceroute gw-li377.linode.com traceroute to gw-li377.linode.com (106.187.34.1), 64 hops max, 52 byte packets 1 192.168.0.1 (192.168.0.1) 1.471 ms 0.725 ms 0.555 ms 2 tokyo10-f03.flets.2iij.net (210.149.34.72) 7.241 ms 6.651 ms 6.939 ms 3 tokyo10-ntteast0.flets.2iij.net (210.149.34.157) 5.573 ms 6.109 ms 5.346 ms 4 tky001lip20.iij.net (210.149.34.97) 6.410 ms 7.471 ms 7.934 ms 5 tky001bb10.iij.net (58.138.100.209) 6.670 ms 9.251 ms 5.866 ms 6 tky009bf00.iij.net (58.138.80.17) 6.730 ms tky008bf02.iij.net (58.138.80.13) 7.021 ms tky009bf00.iij.net (58.138.80.17) 8.593 ms 7 tky001ix05.iij.net (58.138.82.2) 9.767 ms tky001ix05.iij.net (58.138.82.6) 6.101 ms tky001ix01.iij.net (58.138.80.106) 8.420 ms 8 203.181.102.61 (203.181.102.61) 19.514 ms 203.181.102.21 (203.181.102.21) 6.054 ms 203.181.102.61 (203.181.102.61) 11.478 ms 9 otejbb203.kddnet.ad.jp (118.155.197.129) 7.457 ms otejbb203.kddnet.ad.jp (59.128.7.129) 7.835 ms otejbb204.kddnet.ad.jp (59.128.7.130) 7.824 ms 10 cm-fcu203.kddnet.ad.jp (124.215.194.180) 15.860 ms 16.401 ms cm-fcu203.kddnet.ad.jp (124.215.194.164) 17.519 ms 11 124.215.199.122 (124.215.199.122) 7.892 ms * 11.984 ms
Re: Inaccessible network from Verizon, accessible elsewhere.
On Mon, Dec 12, 2011 at 1:26 AM, Adam Greene maill...@webjogger.net wrote: 130.81.107.228 hrm... LCR == lata-core-router... something fairly close to you, like 2 router-hops from your first L3 hop... sounds like someone ought to call the vz customer service line and ask for a fix :)
Re: Inaccessible network from Verizon, accessible elsewhere.
Yep, tried that. Connected with a lower level tech who would not escalate. Anyone know a Verizon NOC direct contact #? On 12/12/2011 2:17 AM, Christopher Morrow wrote: On Mon, Dec 12, 2011 at 1:26 AM, Adam Greenemaill...@webjogger.net wrote: 130.81.107.228 hrm... LCR == lata-core-router... something fairly close to you, like 2 router-hops from your first L3 hop... sounds like someone ought to call the vz customer service line and ask for a fix :)
