RE: The great Netflix vpn debacle! (geofeeds)

[Matthew Huff]

IPv6 tunnels work great for network geeks, but rather poorly for home users 
with streaming, gaming etc...It's not necessarily the performance, it's either 
the geolocation, latency, or the very issue that started this thread - VPN 
banning.

Remember, the streaming services couldn't care less about geolocation or VPN 
banning, it's the contractual obligations with the content providers. The 
content providers care about vpn banning because it gets around geolocation, 
which interferes with their business models (different release schedules to 
different regions, etc..)

Been there, done that...Stuck on Fios with no IPv6. Ran into rather 
"interesting" problems with various streaming services with IPv6 configured.


Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mh...@ox.com | www.ox.com
...

-Original Message-
From: NANOG  On Behalf Of Michael Thomas
Sent: Wednesday, September 1, 2021 2:26 PM
To: Nimrod Levy ; Owen DeLong 
Cc: nanog@nanog.org
Subject: Re: The great Netflix vpn debacle! (geofeeds)


On 9/1/21 10:59 AM, Nimrod Levy wrote:
> All this chatter about IPv6 support on devices is fun and all, but 
> there are providers still not on board.
> They operate in my neighborhood and they know who they are...
>
This is about inside your premise before any NAT's enter the picture. 
What would be nice is if home routers offered up v6 as the default way 
to number and v6 tunnels past ISP's that don't have v6. Home routers 
could make that all rather seamless where users wouldn't need to know 
that was happening. It's really a pity that home routers are a race to 
the bottom where everything else with networking is expected to evolve 
over time.

Mike

RE: Never push the Big Red Button (New York City subway failure)

[Matthew Huff]

Reminds me of something that happened about 25 years ago when an elementary 
school visited our data center of the insurance company where I worked. One of 
our operators strategically positioned himself between the kids and the 
mainframe, leaned back and hit it's EPO button.

Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mh...@ox.com | www.ox.com
...

-Original Message-
From: NANOG  On Behalf Of Sean Donelan
Sent: Friday, September 10, 2021 12:38 PM
To: nanog@nanog.org
Subject: Never push the Big Red Button (New York City subway failure)

NEW YORK CITY TRANSIT RAIL CONTROL CENTER POWER
OUTAGE ISSUE ON AUGUST 29, 2021
Key Findings
September 8, 2021


https://www.governor.ny.gov/sites/default/files/2021-09/WSP_Key_Findings_Summary-for_release.pdf

Key Findings
[...]

3. Based on the electrical equipment log readings and the manufacturer’s 
official assessment, it was determined that the most likely cause of RCC 
shutdown was the “Emergency Power Off” button being manually activated.

Secondary Findings

1. The “Emergency Power Off” button did not have a protective cover at the 
time of the shutdown or the following WSP investigation.

[...]
Mitigation Steps

1. Set up the electrical equipment Control and Communication systems 
properly to stay active so that personnel can monitor RCC electrical 
system operations.

[...]

RE: Never push the Big Red Button (New York City subway failure)

[Matthew Huff]

Since we are telling power horror stories…


How about the call from the night operator that arrived at 10:00pm asking “Is 
there any reason there is no power in the data center?”

Turns out someone had plugged in a new high end workgroup laser printer to the 
outside wall of the datacenter. The power receptacle was wired into the data 
center’s UPS and completely smoked the UPS. Luckily the static transfer 
switched worked, but the three mainframes weren’t’ happy…


Or

Our building had a major ground fault issue that took years to find and 
resolve. We got hit with lightning that caused the mainframe to fault and 
recycle…and two minutes in, we got hit by lightning again. When the system 
failed to start, we called IBM support. When we explained what happened there 
was a very long pause…then some mumbling off phone, then the manager got on the 
line and said someone would be flying out and be onsite within 12 hours. We 
were down for 3 days, and got fined $250,000 by the insurance regulators since 
we couldn’t pay claims.

Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mh...@ox.com<mailto:mh...@ox.com> | www.ox.com<http://www.ox.com>
...

From: Chris Kane 
Sent: Friday, September 10, 2021 3:16 PM
To: Christopher Morrow 
Cc: Matthew Huff ; nanog@nanog.org
Subject: Re: Never push the Big Red Button (New York City subway failure)

True EPO story; maintenance crew carrying new drywall into the data center 
backed into the EPO that didn't have a cover on it. One of the most eerie 
sounds in networking...a completely silent data center.

-chris

On Fri, Sep 10, 2021 at 2:48 PM Christopher Morrow 
mailto:morrowc.li...@gmail.com>> wrote:


On Fri, Sep 10, 2021 at 1:49 PM Matthew Huff 
mailto:mh...@ox.com>> wrote:
Reminds me of something that happened about 25 years ago when an elementary 
school visited our data center of the insurance company where I worked. One of 
our operators strategically positioned himself between the kids and the 
mainframe, leaned back and hit it's EPO button.

Or when your building engineering team cuts themselves a new key for the 'main 
breaker' for the facility... and tests it at 2pm on a tuesday.
Or when that same team cuts a second key (gotta have 2 keys!) and tests that 
key on the same 'main breaker' ... at 2pm on the following tuesday.



not fakenews, a real story from a large building full of gov't employees and 
computers and all manner of 'critical infrastructure' for the agency occupying 
said building.

Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mh...@ox.com<mailto:mh...@ox.com> | www.ox.com<http://www.ox.com>
...

-Original Message-
From: NANOG mailto:ox@nanog.org>> On 
Behalf Of Sean Donelan
Sent: Friday, September 10, 2021 12:38 PM
To: nanog@nanog.org<mailto:nanog@nanog.org>
Subject: Never push the Big Red Button (New York City subway failure)

NEW YORK CITY TRANSIT RAIL CONTROL CENTER POWER
OUTAGE ISSUE ON AUGUST 29, 2021
Key Findings
September 8, 2021


https://www.governor.ny.gov/sites/default/files/2021-09/WSP_Key_Findings_Summary-for_release.pdf

Key Findings
[...]

3. Based on the electrical equipment log readings and the manufacturer’s
official assessment, it was determined that the most likely cause of RCC
shutdown was the “Emergency Power Off” button being manually activated.

Secondary Findings

1. The “Emergency Power Off” button did not have a protective cover at the
time of the shutdown or the following WSP investigation.

[...]
Mitigation Steps

1. Set up the electrical equipment Control and Communication systems
properly to stay active so that personnel can monitor RCC electrical
system operations.

[...]


--
Chris Kane

identity.cisco.com certificate has expired

[Matthew Huff]

Arghh...

Just an FYI, id.cisco.com is fubar'ed. Hopefully cisco has already fixed it and 
the proxies/caches/cdns just need to timeout, but just in case anyone knows a 
contact at Cisco's ops group...

[cid:image001.png@01D8309A.75491410]


Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mh...@ox.com<mailto:mh...@ox.com> | www.ox.com<http://www.ox.com>
...

RE: "Permanent" DST

[Matthew Huff]

They don't want their names on it when what happened in the 70s happens again. 
The effect of setting everything to DST and staying there is that in the 
winter, especially in the norther latitude it will be pitch dark during most of 
the morning when children get picked up at school bus stops. When the tragedy 
happens again, and it will, they will end up undoing this again...

History repeats itself, first as a tragedy, then as a farce...

Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mh...@ox.com | www.ox.com
...

-Original Message-
From: NANOG  On Behalf Of Jay R. Ashworth
Sent: Tuesday, March 15, 2022 5:30 PM
To: Tom Beecher 
Cc: nanog@nanog.org list 
Subject: Re: "Permanent" DST

Oh.  This was "Unanimous Consent"?  AKA "I want to vote for this, but *I do not 
want to be held responsible for having voted for it when it blows up*?"

I'd missed that; thanks.

- Original Message -
> From: "Tom Beecher" 
> To: "Eric Kuhnke" 
> Cc: "nanog@nanog.org list" 
> Sent: Tuesday, March 15, 2022 5:04:02 PM
> Subject: Re: "Permanent" DST

> I would say if something passes the United States Senate in our 
> current political environment by unanimous consent (which this did) , 
> I kinda feel like there won't be a ton of issues with everybody 
> figuring out how to line themselves up appropriately.
> 
> On Tue, Mar 15, 2022 at 5:01 PM Eric Kuhnke  wrote:
> 
>> That is true but at present everything business related in BC has a 
>> clear expectation of being in the same time zone as WA/OR/CA, and AB 
>> matches US Mountain time.
>>
>> On Tue, 15 Mar 2022 at 13:35, Paul Ebersman 
>> wrote:
>>
>>> eric> If Canada doesn't do the same thing at the same time, it'll be 
>>> eric> a real hassle, dealing with a change from -8 to -7 crossing 
>>> eric> the border between BC and WA, for instance. It has to be done 
>>> eric> consistently throughout North America.
>>>
>>> You must not have ever dealt with Indiana, where it was DST or not 
>>> by choice per county. It wasn't quite the cluster***k you'd think.
>>>

-- 
Jay R. Ashworth  Baylink   j...@baylink.com
Designer The Things I Think   RFC 2100
Ashworth & Associates   http://www.bcp38.info  2000 Land Rover DII
St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274

RE: V6 still not supported

[Matthew Huff]

Did you read his email? He was saying that what a lot of people wanted was IPv4 
+ bigger address space, and not any other changes. Speaking for myself, other 
than the bigger address space, for a corporate/enterprise environment I have 
yet to see any advantages of IPv6 and many, many issues.

Information hiding, aka NAT is very important in the financial world.

For example, we have a directly peered connection with our clearing partner at 
our co-location. Both sides use NAT. This way either side can make a network 
change without having to involve the other partner.

Here is what happens without NAT (circa 2008), and yes, this really happened:


  1.  We needed to separate out process and move to a different server. The 
existing server IP couldn’t change since other partners had it in their access 
list.
  2.  It took 9 weeks and:
 *   3 conference calls including one with 30 participants from the 
clearing company
 *   2 approvals by committees at the clearing company



You can say that this was absurd and should be changed. Yes, but it won’t and 
we have no control over them.



With NAT we can change internal IP addresses at any time as long as we update 
the NAT tables.



This is not something important at ISP or hosting providers, but it’s many 
issues like this in the corporate world that prevent IPv6 adoption. Things like 
static addressing, consistent behavior of IPv6 across different operating 
systems including disabling SLAAC,  privacy addressing and others.



Based on my experience and people on tech mailing list that are oriented toward 
enterprises, I would bet that IPv6 deployment (with global addresses) is 
significantly less than 10% nor is it on their horizon.



Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mh...@ox.com<mailto:mh...@ox.com> | www.ox.com<http://www.ox.com>
...

From: NANOG  On Behalf Of Tom Beecher
Sent: Thursday, March 17, 2022 7:41 AM
To: b...@uu3.net
Cc: nanog@nanog.org
Subject: Re: V6 still not supported

“It seems all the market needed was IPv4 with bigger address space.
Instead of delivering it, some contraption has been created trying to solve
non-existant (or already fixed) problems.”

your argument against IPv6 is that they should have created a new version of 
IPv4, but bigger?

So… IPv6?

On Thu, Mar 17, 2022 at 06:32 mailto:b...@uu3.net>> wrote:
It seems team developing IPv6 had ONE way of doing things,
with is actually recipe for disaster. Why? Because they were building an IP
protocol. Something that will be using globally by ALL networks around.
Not some local IOT (useless) shit used here and there.
Thats why such IP protocol should be follow KISS concept and flexibility.
Some people have different vision how to run network. And because
Inter-net is an AS to AS network they should have right to do so.

In my opinion all that crypto stuff should be put layer upper because
crypto is hard, very hard and can get obsolete quickly.

Its same about other weird things embedded into IPv6 that probably
should go layer up. And now people wonder why IPv6 adoption is crap and
there is high resistance. IPv4 made mistakes too, but hell, it was the first.

It seems all the market needed was IPv4 with bigger address space.
Instead of delivering it, some contraption has been created trying to solve
non-existant (or already fixed) problems.

Just my 2 cents...


-- Original message --

From: William Allen Simpson 
mailto:william.allen.simp...@gmail.com>>
To: NANOG mailto:nanog@nanog.org>>
Subject: Re: V6 still not supported
Date: Wed, 16 Mar 2022 22:24:14 -0400

I'd wanted to clearly distinguish this from the old methods:

   This is intended to replace ARP, ICMP Router Advertisement, ICMP
   Redirect, ICMP Information, ICMP Mask, and OSPF Hello in the [IPv6]
   environment. There are also elements of the OSI ES-IS and IS-IS Hello.

We were forward looking to deployments of thousands of systems per link, rather
than the 30 maximum under then current ethernet standards.  We needed fewer
announcements, less chatty traffic, and more specific traffic designation.

We also prioritized network security.  Moreover requiring addresses be
ephemeral,
such that applications would not be able to tie authentication/authorization to
an
IPv6 address and would be motivated to use cryptographic security.

Unfortunately, later committees decided that rather than a single simpler
secured
address assignment scheme, we needed unsecured SLAAC and duplicate DHCPv6.
Three ways to do the same thing are always a recipe for disaster.

Reminder: I was an operator and one of the original NANOG members.

We spent a lot of time considering human factors.  I'd pioneered the
"Operational Considerations" section of the original draft RFCs.

IPv6 i

RE: V6 still not supported

[Matthew Huff]

Good to know. I’ll keep a look out for future implantations. Currently we are 
using Cisco 3548P-XL switches with low-latency nat to support microsecond 
latency natting. Hopefully someday they will support it.


Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mh...@ox.com<mailto:mh...@ox.com> | www.ox.com<http://www.ox.com>
...

From: Dave Bell 
Sent: Thursday, March 17, 2022 8:39 AM
To: Matthew Huff 
Cc: Tom Beecher ; b...@uu3.net; nanog@nanog.org
Subject: Re: V6 still not supported

You can still do NAT with IPv6 like you ask for. It's been around over a decade 
now:
https://datatracker.ietf.org/doc/html/rfc6296

On Thu, 17 Mar 2022 at 12:02, Matthew Huff mailto:mh...@ox.com>> 
wrote:
Did you read his email? He was saying that what a lot of people wanted was IPv4 
+ bigger address space, and not any other changes. Speaking for myself, other 
than the bigger address space, for a corporate/enterprise environment I have 
yet to see any advantages of IPv6 and many, many issues.

Information hiding, aka NAT is very important in the financial world.

For example, we have a directly peered connection with our clearing partner at 
our co-location. Both sides use NAT. This way either side can make a network 
change without having to involve the other partner.

Here is what happens without NAT (circa 2008), and yes, this really happened:


  1.  We needed to separate out process and move to a different server. The 
existing server IP couldn’t change since other partners had it in their access 
list.
  2.  It took 9 weeks and:

 *   3 conference calls including one with 30 participants from the 
clearing company
 *   2 approvals by committees at the clearing company



You can say that this was absurd and should be changed. Yes, but it won’t and 
we have no control over them.



With NAT we can change internal IP addresses at any time as long as we update 
the NAT tables.



This is not something important at ISP or hosting providers, but it’s many 
issues like this in the corporate world that prevent IPv6 adoption. Things like 
static addressing, consistent behavior of IPv6 across different operating 
systems including disabling SLAAC,  privacy addressing and others.



Based on my experience and people on tech mailing list that are oriented toward 
enterprises, I would bet that IPv6 deployment (with global addresses) is 
significantly less than 10% nor is it on their horizon.



Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mh...@ox.com<mailto:mh...@ox.com> | www.ox.com<http://www.ox.com>
...

From: NANOG mailto:ox@nanog.org>> On 
Behalf Of Tom Beecher
Sent: Thursday, March 17, 2022 7:41 AM
To: b...@uu3.net<mailto:b...@uu3.net>
Cc: nanog@nanog.org<mailto:nanog@nanog.org>
Subject: Re: V6 still not supported

“It seems all the market needed was IPv4 with bigger address space.
Instead of delivering it, some contraption has been created trying to solve
non-existant (or already fixed) problems.”

your argument against IPv6 is that they should have created a new version of 
IPv4, but bigger?

So… IPv6?

On Thu, Mar 17, 2022 at 06:32 mailto:b...@uu3.net>> wrote:
It seems team developing IPv6 had ONE way of doing things,
with is actually recipe for disaster. Why? Because they were building an IP
protocol. Something that will be using globally by ALL networks around.
Not some local IOT (useless) shit used here and there.
Thats why such IP protocol should be follow KISS concept and flexibility.
Some people have different vision how to run network. And because
Inter-net is an AS to AS network they should have right to do so.

In my opinion all that crypto stuff should be put layer upper because
crypto is hard, very hard and can get obsolete quickly.

Its same about other weird things embedded into IPv6 that probably
should go layer up. And now people wonder why IPv6 adoption is crap and
there is high resistance. IPv4 made mistakes too, but hell, it was the first.

It seems all the market needed was IPv4 with bigger address space.
Instead of delivering it, some contraption has been created trying to solve
non-existant (or already fixed) problems.

Just my 2 cents...


-- Original message --

From: William Allen Simpson 
mailto:william.allen.simp...@gmail.com>>
To: NANOG mailto:nanog@nanog.org>>
Subject: Re: V6 still not supported
Date: Wed, 16 Mar 2022 22:24:14 -0400

I'd wanted to clearly distinguish this from the old methods:

   This is intended to replace ARP, ICMP Router Advertisement, ICMP
   Redirect, ICMP Information, ICMP Mask, and OSPF Hello in the [IPv6]
 

RE: FCC proposes higher speed goals (100/20 Mbps) for USF providers

[Matthew Huff]

I grew up in rural Texas where my mother still lives. She has adequate speed 
internet, the biggest issue is reliability. The whole town (there is only 1 
provider) has an outage for about an hour every week. Two weeks ago, there was 
no internet for 3 days. Cellular service is 4G and not even that reliable for 
data even on the best days.

From: NANOG  On Behalf Of Brian Turnbow 
via NANOG
Sent: Tuesday, May 24, 2022 9:35 AM
To: David Bass ; Sean Donelan 
Cc: nanog@nanog.org
Subject: RE: FCC proposes higher speed goals (100/20 Mbps) for USF providers

Here in Italy there have been a lot of investments to get better broadband.
Such as government sponsored bundles for areas with no return on investments, 
for schools etc with a lot of focus on reaching gigabit speeds
The results have been mainly positive even though there are delays.
On the end user side in 2020 one of the largest ISPs started offering 2.5Gbps 
service
Adds all over and users started asking for it, even though they don’t have a 
2.5 nic or router,  so now all of the major providers are rolling it out.
Illiad one uped them a couple of months ago pushing  a 5Gbps service and now I 
get people asking me if we offer 5Gbps fiber lines.. pure marketing…
I have a 1Gbps/100Mbps line and it is plenty enough for the family rarely do we 
even get near the limits.
It’s kind of like when I ask for an Italian espresso in the states and get a 
cup full of coffee, no I just want a very small italian style espresso..
The response is Why? you are paying for it take it all
Bigger is better, even if you don’t need it, reigns supreme.

The real problem most users experience isn’t that they have a gig, or even 
100Mb of available download bandwidth…it’s that they infrequently are able to 
use that full bandwidth due to massive over subscription .

The other issue is the minimal upload speed.  It’s fairly easy to consume the 
10Mb that you’re typically getting as a residential customer.  Even “business 
class” broadband service has a pretty poor upload bandwidth limit.

We are a pretty high usage family, and 100/10 has been adequate, but there’s 
been times when we are pegged at the 10 Mb upload limit, and we start to see 
issues.

I’d say 25/5 is a minimum for a single person.

Would 1 gig be nice…yeah as long as the upload speed is dramatically increased 
as part of that.  We would rarely use it, but that would likely be sufficient 
for a long time.  I wouldn’t pay for the extra at this point though.

On Mon, May 23, 2022 at 8:20 PM Sean Donelan 
mailto:s...@donelan.com>> wrote:

Remember, this rulemaking is for 1.1 million locations with the "worst"
return on investment. The end of the tail of the long tail.  Rural and
tribal locations which aren't profitable to provide higher speed
broadband.

These locations have very low customer density, and difficult to serve.

After the Sandwich Isles Communications scandal, gold-plated proposals
will be viewed with skepticism.  While a proposal may have a lower total
cost of ownership over decades, the business case is the cheapest for
the first 10 years of subsidies.  [massive over-simplification]

Historically, these projects have lack of timely completion (abandoned,
incomplete), and bad (overly optimistic?) budgeting.

RE: Congrats to AS701

[Matthew Huff]

Still no IPv6 in Westchester County, NY ☹

Great sign though, maybe NY will get it eventually

From: NANOG  On Behalf Of Joe Loiacono
Sent: Monday, June 13, 2022 10:55 AM
To: nanog@nanog.org
Subject: Re: Congrats to AS701


FiOS from Maryland (anonymized):

enp3s0: flags=4163  mtu 1500
inet 192.168.1.164  netmask 255.255.255.0  broadcast 192.168.1.255
inet6 fe80::b104:8f4d:e5b2:e13b  prefixlen 64  scopeid 0x20
inet6 2600:4040:b27f:cb00:a9b1:5f59::  prefixlen 64  scopeid 
0x0
inet6 2600:4040:b27f:cb00:24a8:7b31::  prefixlen 64  scopeid 
0x0
inet6 2600:4040:b27f:cb00:e1b6:8b83::  prefixlen 64  scopeid 
0x0
ether d0:67:e5:23:ec:fe  txqueuelen 1000  (Ethernet)
RX packets 2518066  bytes 1448982813 (1.4 GB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 2157395  bytes 260073952 (260.0 MB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

a@b:~$ ping 2607:f8b0:4004:c09::6a
PING 2607:f8b0:4004:c09::6a(2607:f8b0:4004:c09::6a) 56 data bytes
64 bytes from 2607:f8b0:4004:c09::6a: icmp_seq=1 ttl=59 time=24.0 ms
64 bytes from 2607:f8b0:4004:c09::6a: icmp_seq=2 ttl=59 time=17.6 ms
64 bytes from 2607:f8b0:4004:c09::6a: icmp_seq=3 ttl=59 time=20.4 ms
64 bytes from 2607:f8b0:4004:c09::6a: icmp_seq=4 ttl=59 time=23.4 ms
^C
--- 2607:f8b0:4004:c09::6a ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 17.618/21.351/23.983/2.555 ms


On 6/12/2022 1:55 PM, Christopher Morrow wrote:


On Sat, Jun 11, 2022 at 11:03 PM Darrel Lewis (darlewis) 
mailto:darle...@cisco.com>> wrote:
I, for one, am having a hard time finding the proper words to express the joy 
that I am feeling at this momentous moment!


It's quite amazing, I think... that it's taken so long to get to deployment you 
can actually see on the fios plant :)
I'd note I can't see the below on my homestead, but I can at a relative's 
(where the ifconfig data is from).

I also can't tell if the upstream will PD a block to the downstream... and the 
VZ CPE is 'not something I want to fiddle with',
because everytime I have tried at my house I've just taken it out behind the 
woodshed with a maul... and replaced it with
something I CAN configure successfully. (plus.. don't want that TR 069 in my 
home...)

-chris

-Darrel


On Jun 11, 2022, at 7:05 PM, Christopher Morrow 
mailto:morrowc.li...@gmail.com>> wrote:


Looks like FIOS customers may be getting ipv6 deployed toward them, finally:

ifconfig snippet from local machine:
inet6 2600:4040:2001:2200:73d2:6bcc:1e6b:43a1  prefixlen 64  scopeid 
0x0
inet6 2600:4040:2001:2200:e87:bf36:b6cb:6ce1  prefixlen 64  scopeid 
0x0

ping attempt:
  64 bytes from bh-in-f106.1e100.net 
(2607:f8b0:4004:c09::6a): icmp_seq=1 ttl=59 time=8.71 ms

8ms from mclean, va to ashburn, va isn't wondrous, but at least it's ipv6 (and 
marginally faster than ipv4)

Congrats to the 701 folk for deploying more widely!
  (note: I don't know exactly when this started, nor how wide it really is, but 
progress here is welcomed by myself at least :) )
-chris

RE: Verizon no BGP route to some of AS38365 (182.61.200.0/24)

[Matthew Huff]

From my limited vantage point it appears that there is some issue between 
Verizon & Baidu. Baidu has 182.61.0.0/16 registered, but is only advertising 
pieces of it globally (or at least from what I can see). In our tables,we are 
receiving none from Verizon of  the subnets that are advertised directly from 
Baidu (origin AS of 38365). The few within that registered range that have a 
different origin AS are coming to us from Verizon. For example:

*>   182.61.0.0/19144.121.203.1410 46887 3356 4134 
58466 38365 i
*>   182.61.0.0/18144.121.203.1410 46887 6461 4134 
58466 38365 38365 i
*>   182.61.32.0/19   144.121.203.1410 46887 3356 4134 
58466 38365 i
*>   182.61.64.0/18   204.148.121.2210 701 6453 55967 i
*182.61.128.0/23  204.148.121.2210 701 4134 4134 
4134 4134 4134 58540 ?
*>   182.61.130.0/24  144.121.203.1410 46887 6461 4134 
23724 38365 38365 38365 i
*>   182.61.130.0/23  144.121.203.1410 46887 6461 4134 
58466 38365 38365 i
*>   182.61.131.0/24  144.121.203.1410 46887 6461 4134 
23724 38365 38365 38365 i
*>   182.61.132.0/23  144.121.203.1410 46887 3356 4134 
58466 38365 i
*>   182.61.132.0/22  144.121.203.1410 46887 6461 4134 
58466 38365 38365 i
*>   182.61.134.0/23  144.121.203.1410 46887 3356 4134 
58466 38365 i
*>   182.61.136.0/22  144.121.203.1410 46887 3356 4134 
58466 38365 i
*>   182.61.136.0/21  144.121.203.1410 46887 6461 4134 
58466 38365 38365 i
*>   182.61.140.0/22  144.121.203.1410 46887 3356 4134 
58466 38365 i
*>   182.61.144.0/21  144.121.203.1410 46887 3356 4134 
58466 38365 i
*>   182.61.144.0/20  144.121.203.1410 46887 6461 4134 
58466 38365 38365 i
*>   182.61.160.0/19  204.148.121.2210 701 6453 55967 i
*>   182.61.192.0/23  144.121.203.1410 46887 3356 4134 
58540 i
*>   182.61.194.0/23  144.121.203.1410 46887 3356 4134 
58540 i
*>   182.61.200.0/22  144.121.203.1410 46887 6461 4134 
23724 38365 i
*>   182.61.200.0/21  144.121.203.1410 46887 6461 4134 
58466 38365 38365 i
*>   182.61.216.0/21  144.121.203.1410 46887 6461 4134 
58466 38365 38365 i
*>   182.61.223.0/24  144.121.203.1410 46887 6461 4134 
58466 38365 38365 i
*>   182.61.224.0/19  144.121.203.1410 46887 6461 4134 
58466 38365 38365 i

We are getting the 182.61.200.0/21 and 182.61.200.0/22 from all of our other 
peers:

asr-inet2#sh ip bgp 182.61.200.0/21
BGP routing table entry for 182.61.200.0/21, version 15779018
Paths: (2 available, best #2, table default)
  Advertised to update-groups:
 3
  Refresh Epoch 1
  54004 6128 1299 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
148.77.99.201 from 148.77.99.201 (24.157.4.25)
  Origin IGP, localpref 100, valid, external, atomic-aggregate
  rx pathid: 0, tx pathid: 0
  Updated on Apr 29 2022 21:02:05 EDT
  Refresh Epoch 1
  46887 6461 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
129.77.17.254 from 129.77.17.254 (129.77.40.7)
  Origin IGP, metric 0, localpref 100, valid, internal, atomic-aggregate, 
best
  rx pathid: 0, tx pathid: 0x0
  Updated on May 3 2022 04:02:50 EDT


From: NANOG  On Behalf Of holow29
Sent: Thursday, June 23, 2022 5:49 PM
To: nanog@nanog.org
Subject: Verizon no BGP route to some of AS38365 (182.61.200.0/24)

I've been trying (to no avail) for over a month now to get Verizon to 
investigate their lack of BGP routing to 
182.61.200.0/24, which hosts Baidu Wangpan at  
pan.baidu.com (Baidu's cloud services/equivalent of 
Google Drive).

Easily verified through Verizon's Looking Glass.

We all know Verizon's BGP routing is a disaster, but does anyone have any ideas?

RE: IERS ponders reverse leapsecond...

[Matthew Huff]

True, 

But it's hard enough to get developers to understand the need to code for 61 
seconds in a minute, and now they would need to code for 59 seconds as well.

If time systems simply skewed the time so that 60 seconds actually just took 61 
seconds or 59 seconds, there would be other issues, but coders wouldn't be 
involved.



-Original Message-
From: NANOG  On Behalf Of Stephane 
Bortzmeyer
Sent: Wednesday, August 3, 2022 11:19 AM
To: Jay Ashworth 
Cc: nanog@nanog.org
Subject: Re: IERS ponders reverse leapsecond...

On Wed, Aug 03, 2022 at 11:09:25AM -0400,  Jay Ashworth  
wrote  a message of 32 lines which said:

> General press loses its *mind*:

Indeed, they seem not to know what they write about. "atomic time – the 
universal way time is measured on Earth – may have to change" They don't even 
know the difference between TAI and UTC.

RE: 400G forwarding - how does it work?

[Matthew Huff]

Also, for data center traffic, especially real-time market data and other UDP 
multicast traffic, micro-bursting is one of the biggest issues especially as 
you scale out your backbone. We have two 100GB switches, and have to distribute 
the traffic over a LACL link with 4 different 100GB ports on different ASIC 
even though the traffic < 1% just to account for micro-bursts.



-Original Message-
From: NANOG  On Behalf Of 
sro...@ronan-online.com
Sent: Monday, August 8, 2022 8:39 AM
To: Masataka Ohta 
Cc: nanog@nanog.org
Subject: Re: 400G forwarding - how does it work?

You keep using the term “imaginary” when presented with evidence that does not 
match your view of things. 

There are many REAL scenarios where single flow high throughout TCP is a real 
requirements as well as high throughput extremely small packet size. In the 
case of the later, the market is extremely large, but it’s not Internet traffic.

Shane

> On Aug 8, 2022, at 7:34 AM, Masataka Ohta  
> wrote:
> 
> Saku Ytti wrote:
> 
>>> which is, unlike Yttinet, the reality.
>> Yttinet has pesky customers who care about single TCP performance 
>> over long fat links, and observe poor performance with shallow 
>> buffers at the provider end.
> 
> With such an imaginary assumption, according to the end to end 
> principle, the customers (the ends) should use paced TCP instead of 
> paying unnecessarily bloated amount of money to intelligent 
> intermediate entities of ISPs using expensive routers with bloated 
> buffers.
> 
>> Yttinet is cost sensitive and does not want to do work, unless 
>> sufficiently motivated by paying customers.
> 
> I understand that if customers follow the end to end principle, 
> revenue of "intelligent" ISPs will be reduced.
> 
>Masataka Ohta
> 
> 
> 

RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Matthew Huff]

switch-core1# sh forwarding route x.x.x.x

slot  1
===


IPv4 routes for table default/base

--+-+--+-+-
Prefix| Next-hop| Interface 
   | Labels  | Partial Install 
--+-+--+-+-
x.x.x.x/24  x.x.x.250Ethernet1/29


switch-core1# show routing hash x.x.x.x y.y.y.y
Load-share parameters used for software forwarding:
load-share mode: address source-destination port source-destination
Hash for VRF "default"
Hashing to path *y.y.y.y Eth1/29
For route:
y.y.y.0/24, ubest/mbest: 1/0
*via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal




From: NANOG  On Behalf Of Mike Hammett
Sent: Monday, April 3, 2023 1:21 AM
To: NANOG 
Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

We have a Nexus 3064 that is setup with partial BGP tables and is routing based 
on that. 


I've done a show ip bgp for an IP of interest and it has an expected next hop 
IP. I show ip arp on that next hop IP and it has the expected interface. 


However, sFlows show the packets leaving on a different interface, the one that 
would carry the default route for routes not otherwise known. 


If the next hop IP is expected and the ARP of that next hop IP is expected, why 
are packets leaving out an unexpected interface? 


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Matthew Huff]

SFlow misconfiguration or bug on either the nexus or the sflow monitor? On the 
monitor, can you verify that the snmp interfaces are mapped to the correct ones 
on the nexus?





From: Mike Hammett 
Sent: Monday, April 3, 2023 8:47 AM
To: Matthew Huff 
Cc: NANOG 
Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

It shows the desired result.


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com


From: "Matthew Huff" mailto:mh...@ox.com>>
To: "Mike Hammett" mailto:na...@ics-il.net>>, "NANOG" 
mailto:nanog@nanog.org>>
Sent: Monday, April 3, 2023 5:38:23 AM
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

switch-core1# sh forwarding route x.x.x.x

slot  1
===


IPv4 routes for table default/base

--+-+--+-+-
Prefix| Next-hop| Interface 
   | Labels  | Partial Install
--+-+--+-+-
x.x.x.x/24  x.x.x.250Ethernet1/29


switch-core1# show routing hash x.x.x.x y.y.y.y
Load-share parameters used for software forwarding:
load-share mode: address source-destination port source-destination
Hash for VRF "default"
Hashing to path *y.y.y.y Eth1/29
For route:
y.y.y.0/24, ubest/mbest: 1/0
*via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal




From: NANOG 
mailto:nanog-bounces+mhuff=ox@nanog.org>>
 On Behalf Of Mike Hammett
Sent: Monday, April 3, 2023 1:21 AM
To: NANOG mailto:nanog@nanog.org>>
Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

We have a Nexus 3064 that is setup with partial BGP tables and is routing based 
on that.


I've done a show ip bgp for an IP of interest and it has an expected next hop 
IP. I show ip arp on that next hop IP and it has the expected interface.


However, sFlows show the packets leaving on a different interface, the one that 
would carry the default route for routes not otherwise known.


If the next hop IP is expected and the ARP of that next hop IP is expected, why 
are packets leaving out an unexpected interface?


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Matthew Huff]

What about VRFs and/or policy based routing?  

switch-core1# show vrf
VRF-Name   VRF-ID State   Reason
default 1 Up  --
management  2 Up  --

switch-core1# show route-map 
route-map rmap_bgp_to_eigrp_b2b, permit, sequence 10 
  Match clauses:
interface: Ethernet1/33 
route-type: internal 
  Set clauses:
metric 4000 10 255 1 1500 
route-map rmap_bgp_to_eigrp_b2b, permit, sequence 20 
  Match clauses:
interface: Ethernet1/34 
route-type: internal 
  Set clauses:
metric 4000 30 255 1 1500 
route-map rmap_static_to_eigrp, permit, sequence 10 
  Match clauses:
ip address prefix-lists: prefix_static_to_eigrp 
  Set clauses:
route-map rmap_static_to_eigrp_v6, permit, sequence 10 
  Match clauses:
ipv6 address prefix-lists: prefix_ipv6_static_to_eigrp 
  Set clauses:



From: Mike Hammett  
Sent: Monday, April 3, 2023 9:00 AM
To: Matthew Huff 
Cc: NANOG 
Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

It could be an sFlow bug, but I come at this from a reported problem and 
gathering data on that problem as opposed to looking at data for problems.

The snmp if index reported by the Nexus matches the if index in ElastiFlow.




-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com


From: "Matthew Huff" <mailto:mh...@ox.com>
To: "Mike Hammett" <mailto:na...@ics-il.net>
Cc: "NANOG" <mailto:nanog@nanog.org>
Sent: Monday, April 3, 2023 7:50:08 AM
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
SFlow misconfiguration or bug on either the nexus or the sflow monitor? On the 
monitor, can you verify that the snmp interfaces are mapped to the correct ones 
on the nexus?
 
 
 
 
 
From: Mike Hammett <mailto:na...@ics-il.net> 
Sent: Monday, April 3, 2023 8:47 AM
To: Matthew Huff <mailto:mh...@ox.com>
Cc: NANOG <mailto:nanog@nanog.org>
Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
 
It shows the desired result.


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com
 

From: "Matthew Huff" <mailto:mh...@ox.com>
To: "Mike Hammett" <mailto:na...@ics-il.net>, "NANOG" <mailto:nanog@nanog.org>
Sent: Monday, April 3, 2023 5:38:23 AM
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

switch-core1# sh forwarding route x.x.x.x

slot  1
===


IPv4 routes for table default/base

--+-+--+-+-
Prefix            | Next-hop                                | Interface         
   | Labels          | Partial Install 
--+-+--+-+-
x.x.x.x/24      x.x.x.250                            Ethernet1/29        


switch-core1# show routing hash x.x.x.x y.y.y.y
Load-share parameters used for software forwarding:
load-share mode: address source-destination port source-destination
Hash for VRF "default"
Hashing to path *y.y.y.y Eth1/29
For route:
y.y.y.0/24, ubest/mbest: 1/0
    *via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal




From: NANOG <mailto:nanog-bounces+mhuff=ox@nanog.org> On Behalf Of Mike 
Hammett
Sent: Monday, April 3, 2023 1:21 AM
To: NANOG <mailto:nanog@nanog.org>
Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

We have a Nexus 3064 that is setup with partial BGP tables and is routing based 
on that. 


I've done a show ip bgp for an IP of interest and it has an expected next hop 
IP. I show ip arp on that next hop IP and it has the expected interface. 


However, sFlows show the packets leaving on a different interface, the one that 
would carry the default route for routes not otherwise known. 


If the next hop IP is expected and the ARP of that next hop IP is expected, why 
are packets leaving out an unexpected interface? 


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com
 

RE: Intermittent "bad gateway"

[Matthew Huff]

We got reports on that on some cloudflare sites, but it disappeared pretty 
quickly. Looks like a CDN issue.

-Original Message-
From: NANOG  On Behalf Of Stephen Satchell
Sent: Tuesday, July 2, 2019 10:17 AM
To: nanog@nanog.org
Subject: Intermittent "bad gateway"

Are we having another BGP problem this morning?

RE: This DNS over HTTP thing

[Matthew Huff]

>From a corporate standpoint, this is exactly correct. There are also some 
>regulatory issues involved (FINRA, SEC, etc...)

We are required to block access to web based email (gmail, etc...) in our 
corporate network (please don't ask why, ours is not to reason why...), so 
every method to "bypass" normal network operations creates headaches for us.



-Original Message-
From: NANOG  On Behalf Of John R. Levine
Sent: Tuesday, October 1, 2019 4:06 PM
To: Aaron C. de Bruyn 
Cc: NANOG mailing list 
Subject: Re: This DNS over HTTP thing

I assumed my point was obvious but evidently I overestimated my audience.

While it is stupid to assert that the only reason to circumvent DNS filters is 
to look at child abuse material, it is equally stupid to assert that the only 
reason to filter is to lie, or to censor.

There are plenty of good reasons to filter DNS responses, with the most obvious 
being to block malware sites whose links are sent out in spam (a whole lot of 
spam these days.)  There are also reasons that enterprises filter DNS on their 
networks, to block stuff that creates a hostile work environment, or is 
obviously unrelated to what employees are hired to do (i.e., facebook.)

R's,
John


On Tue, 1 Oct 2019, Aaron C. de Bruyn wrote:

> "For the children!"
> "Stop resisting!"
> "I was in fear for my life!"
>
> The age-old cries of the oppressor. ...

> On Tue, Oct 1, 2019 at 11:33 AM John Levine  wrote:
>
>> In article <20191001074011.n4xjouqg6lhsv...@nic.fr> you write:
>>> Note that the UK is probably the country in Europe with the biggest
>>> use of lying DNS resolvers for censorship. No wonder that the people
>>> who censor don't like anti-censorship techniques.
>>
>> Most UK ISPs use the Internet Watch Foundation's advice intended to
>> block child sexual abuse material.
>>
>> Circumventing it enables people to access that material.
>>
>> We can shout CHILD PORNOGRAPHY just as loud as you can shout
>> CENSORSHIP so perhaps we should both stop now.  There are plenty of
>> valid reasons for a DNS resolver to block some results.

Regards,
John Levine, jo...@taugh.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

SFP oraganizers / storage recommendations

[Matthew Huff]

Any recommendations to keep track of different SFP and keep them organized? Any 
storage boxes / trays designed for SFPs?

Re: Disney+ Geolocation issues

[Matthew Huff]

It’s not about optimization, it’s about the contract with the content 
providers. The agreement is to restrict content by geographical regions mainly 
for marketing purposes. They block VPN access to keep people from bypassing 
those restrictions. It’s true of all the streaming providers.

> On Nov 13, 2019, at 9:44 AM, Robert Blayzor  wrote:
> 
> On 11/12/19 5:28 PM, Michael Crapse wrote:
>> Myself and a few other ISPs are having our eyeballs complain about
>> disney+ saying that they're on a VPN. Does anyone have any idea, or who
>> to contact regarding this issue?
>> This is most likely improper geolocation databases. Anyone have an idea
>> who they use?
>> 
> 
> 
> Same boat here. ARIN ISP with all valid SWIP clearly showing stateside
> USA. So who knows what Disney+ is doing to block their viewers. Seems
> rather silly to block viewing based on the connecting IP address.
> Wouldn't you base it on the authorized viewer who is logged in and using
> the service? I mean, that's what they are paying for. I get the whole
> CDN steering thing, but the error message message sent back to the
> viewer should not be to "call your ISP". Now you have support desks
> taking thousands of worthless calls
> 
> ESPN+ is guilty of the same garbage
> 
> -- 
> inoc.net!rblayzor
> XMPP: rblayzor.AT.inoc.net
> PGP:  https://pgp.inoc.net/rblayzor/

Need BGP route check

[Matthew Huff]

One of our upstreams is apparently having problems, although they don't appear 
to know about it. I've seen an alert at BGPmon.net about our prefixes being 
withdrawn, and I can't locate our prefixes through that provider on any 
routeviews. Can someone check to see what ASPATHS you are seeing for our 
prefixes?

129.77.0.0/16
2620:0:2810::/48

We should be advertised via AS6128 and AS46887

----
Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

RE: Need BGP route check

[Matthew Huff]

Thanks, but I had checked a number of public looking glasses and only one had 
the 46887 path (HE.net), wanted a more global check. A number of responses are 
seeing only one or the other paths. The 14607 pre-pend is due to depref'ing 
46887 earlier today when we had the instability.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669


> -Original Message-
> From: Ken Chase [mailto:k...@sizone.org]
> Sent: Friday, May 20, 2016 11:39 AM
> To: Matthew Huff 
> Cc: nanog@nanog.org
> Subject: Re: Need BGP route check
> 
> $ telnet route-views.oregon-ix.net
> Username: rviews
> 
> $ show ip bgp paths 14607
> 
>  might help
> 
> /kc
> 
> 
> On Fri, May 20, 2016 at 03:31:48PM +, Matthew Huff said:
>   >One of our upstreams is apparently having problems, although they
> don't appear to know about it. I've seen an alert at BGPmon.net about
> our prefixes being withdrawn, and I can't locate our prefixes through
> that provider on any routeviews. Can someone check to see what ASPATHS
> you are seeing for our prefixes?
>   >
>   >129.77.0.0/16
>   >2620:0:2810::/48
>   >
>   >We should be advertised via AS6128 and AS46887
>   >
>   >
>   >Matthew Huff | 1 Manhattanville Rd
>   >Director of Operations???| Purchase, NY 10577
>   >OTA Management LLC?? | Phone: 914-460-4039
>   >aim: matthewbhuff??? | Fax:?? 914-694-5669
>   >
>   >
> 
> --
> Ken Chase - k...@sizone.org Guelph Canada

RE: Need BGP route check (UPDATE)

[Matthew Huff]

>From responses I received, I have gotten a number of different answers. Some 
>are seeing our routes from 6128, some from 46887 and a few from both. The 
>response from Eric though was typical. Showing the IPv4 prefix only from 
>AS6128, but the IPv6 from both 6128 & 46887. 

I am guessing that 46887 might be set with a community to not export our IPv4 
prefixes except to direct peers? Anyone directly peered with 46887 that could 
see the community for 129.77.0.0/16 and verify?

----
Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669


> -Original Message-
> From: Eric Tykwinski [mailto:eric-l...@truenet.com]
> Sent: Friday, May 20, 2016 11:48 AM
> To: Matthew Huff 
> Subject: RE: Need BGP route check
> 
> Matt,
> 
> show ip bgp 129.77.0.0/16
> BGP routing table entry for 129.77.0.0/16, version 161696687
> Paths: (2 available, best #2, table Default-IP-Routing-Table)
>   Advertised to update-groups:
>  1
>   3356 6128 14607
> 4.59.140.65 from 4.59.140.65 (4.69.183.7)
>   Origin IGP, metric 0, localpref 100, valid, external
>   174 6128 14607
> 38.104.114.73 from 38.104.114.73 (154.26.5.244)
>   Origin IGP, metric 105030, localpref 100, valid, external, best
>   Community: 174:21000 174:22013
> 
> show bgp ipv6 unicast 2620:0:2810::/48
> BGP routing table entry for 2620:0:2810::/48, version 18004880
> Paths: (2 available, best #1, table Global-IPv6-Table)
>   Advertised to update-groups:
>  2
>   3356 6128 14607
> 2001:1900:2100::A2D (FE80::219:7FF:FEDD:2800) from
> 2001:1900:2100::A2D
> (4.69.183.7)
>   Origin IGP, metric 0, localpref 100, valid, external, best
>   Community: 3356:3 3356:22 3356:100 3356:123 3356:575 3356:2039
> 6128:3000 6128:3091 6128:4000 6128:5003 6128:5046 64600:4000
> 64600:65002
>   174 46887 14607 14607
> 2001:550:2:4::B:1 (FE80::D66D:50FF:FE5E:1D3) from 2001:550:2:4::B:1
> (154.26.5.244)
>   Origin IGP, metric 105030, localpref 100, valid, external
>   Community: 174:21001 174:22013
> 
> Sincerely,
> 
> Eric Tykwinski
> TrueNet, Inc.
> P: 610-429-8300
> 
> 
> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Matthew Huff
> Sent: Friday, May 20, 2016 11:32 AM
> To: nanog@nanog.org
> Subject: Need BGP route check
> 
> One of our upstreams is apparently having problems, although they don't
> appear to know about it. I've seen an alert at BGPmon.net about our
> prefixes
> being withdrawn, and I can't locate our prefixes through that provider
> on
> any routeviews. Can someone check to see what ASPATHS you are seeing
> for our
> prefixes?
> 
> 129.77.0.0/16
> 2620:0:2810::/48
> 
> We should be advertised via AS6128 and AS46887
> 
> 
> Matthew Huff | 1 Manhattanville Rd Director of
> Operations   |
> Purchase, NY 10577 OTA Management LLC   | Phone: 914-460-4039
> aim: matthewbhuff    | Fax:   914-694-5669
> 
> 
> 

RE: Netflix VPN detection - actual engineer needed

[Matthew Huff]

I would imagine it was done on purpose. The purpose of the Netflix VPN 
detection was to block users from outside of different regions due to content 
providers requests. Since HE provides free ipv6 tunnels, it's an easy way to 
get around the blockage, hence the restriction.



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669


> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Blair Trosper
> Sent: Friday, June 3, 2016 3:11 PM
> To: mike.hy...@gmail.com
> Cc: NANOG 
> Subject: Re: Netflix VPN detection - actual engineer needed
> 
> Confirmed that Hurricane Electric's TunnelBroker is now blocked by
> Netflix.  Anyone nice people from Netflix perhaps want to take a crack
> at
> this?
> 
> 
> 
> On Thu, Jun 2, 2016 at 2:15 PM,  wrote:
> 
> > Had the same problem at my house, but it was caused by the IPv6
> connection
> > to HE.  Turned of V6 and the device worked.
> >
> >
> > --
> >
> > Sent with Airmail
> >
> > On June 1, 2016 at 10:29:03 PM, Matthew Kaufman (matt...@matthew.at)
> > wrote:
> >
> > Every device in my house is blocked from Netflix this evening due to
> > their new "VPN blocker". My house is on my own IP space, and the
> outside
> > of the NAT that the family devices are on is 198.202.199.254,
> announced
> > by AS 11994. A simple ping from Netflix HQ in Los Gatos to my house
> > should show that I'm no farther away than Santa Cruz, CA as
> microwaves
> > fly.
> >
> > Unfortunately, when one calls Netflix support to talk about this, the
> > only response is to say "call your ISP and have them turn off the VPN
> > software they've added to your account". And they absolutely refuse
> to
> > escalate. Even if you tell them that you are essentially your own
> ISP.
> >
> > So... where's the Netflix network engineer on the list who all of us
> can
> > send these issues to directly?
> >
> > Matthew Kaufman
> >

RE: Netflix VPN detection - actual engineer needed

[Matthew Huff]

Netflix IS acting in their user's best interest. In order to provide content 
that the user's want, the content providers have mandated that they do their 
due diligence to block out of region users including VPN and open tunnel 
access. As Hulu and Amazon prime become more popular and their contracts with 
the content provides come due, they will have to also.

You can argue about the content provides business model all you want, but 
Netflix has to do what they are doing. They aren't blocking IPv6 users, they 
are blocking users that are using VPNs and/or tunnels since their currently is 
no practical way of providing GEOIP information about that users that the 
content providers require.


----
Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Scott Morizot
> Sent: Monday, June 6, 2016 10:50 AM
> To: Mark Tinka 
> Cc: NANOG list 
> Subject: Re: Netflix VPN detection - actual engineer needed
> 
> I have Hulu Plus and Amazon Prime. The only thing I would miss from
> Netflix
> is their Marvel original series. And I can live with that. I can't live
> without my IPv6 enabled home network and Internet connection since
> that's
> an essential part of my job. (I'm the IPv6 transition technical lead
> for a
> large organization.) While I actually manage my home internet gateway
> through a linux server and have fine-grained control over the firewall
> rules, I'm still debating whether I care enough about a handful of
> series
> to continue paying a company that is deliberately acting against its
> users'
> interests. Right now I'm leaning toward no. But I'll discuss it with my
> wife before making a final decision.
> 
> Scott
> 
> On Mon, Jun 6, 2016 at 8:03 AM, Mark Tinka 
> wrote:
> 
> >
> >
> > On 6/Jun/16 01:45, Damian Menscher wrote:
> >
> > >
> > > Who are these non-technical Netflix users who accidentally stumbled
> into
> > > having a HE tunnel broker connection without their knowledge?  I
> wasn't
> > > aware this sort of thing could happen without user consent, and
> would
> > like
> > > to know if I'm wrong.  Only thing I can imagine is if ISPs are
> using HE
> > as
> > > a form of CGN.
> >
> > There are several networks around the world that rely on 6-in-4
> because
> > their local provider does not offer IPv6.
> >
> > Mark.
> >

RE: Netflix VPN detection - actual engineer needed

[Matthew Huff]

Scott,

You are being absurd. The number of Netflix customers using 6in4 tunnels has to 
be in the 0.0001% territory of their users. They would be committing business 
malpractice to risk their contracts with content providers to provide access to 
that negligent amount of users. It’s not laziness to look at the risk versus 
rewards and decide it isn’t worth it from a business practice.

Yes, they could work with tunnel brokers and VPN provides and come up with some 
way of communicating GEOIP information, but even if the content providers were 
okay with that the cost involved versus the number of users they would impact 
would never make it worth their wile.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff| Fax:   914-694-5669

From: Scott Morizot [mailto:tmori...@gmail.com]
Sent: Monday, June 6, 2016 11:04 AM
To: Matthew Huff 
Cc: Mark Tinka ; NANOG list 
Subject: Re: Netflix VPN detection - actual engineer needed

Nonsense. That is hardly their only option as many others have pointed out. 
It's a deliberate and technically lazy choice to block 6in4 tunnels. Those are 
not even vaguely the same thing as a VPN. They've decided to break normal IPv6 
support and do so in a way that does not even fall back to IPv4. They deserve 
all the bad publicity that comes with such a anti-customer decision and the 
blame for their implementation choices cannot be passed back to the content 
providers.

Scott

On Mon, Jun 6, 2016 at 9:59 AM, Matthew Huff 
mailto:mh...@ox.com>> wrote:
Netflix IS acting in their user's best interest. In order to provide content 
that the user's want, the content providers have mandated that they do their 
due diligence to block out of region users including VPN and open tunnel 
access. As Hulu and Amazon prime become more popular and their contracts with 
the content provides come due, they will have to also.

You can argue about the content provides business model all you want, but 
Netflix has to do what they are doing. They aren't blocking IPv6 users, they 
are blocking users that are using VPNs and/or tunnels since their currently is 
no practical way of providing GEOIP information about that users that the 
content providers require.



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff| Fax:   914-694-5669

> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org<mailto:nanog-boun...@nanog.org>] 
> On Behalf Of Scott Morizot
> Sent: Monday, June 6, 2016 10:50 AM
> To: Mark Tinka mailto:mark.ti...@seacom.mu>>
> Cc: NANOG list mailto:nanog@nanog.org>>
> Subject: Re: Netflix VPN detection - actual engineer needed
>
> I have Hulu Plus and Amazon Prime. The only thing I would miss from
> Netflix
> is their Marvel original series. And I can live with that. I can't live
> without my IPv6 enabled home network and Internet connection since
> that's
> an essential part of my job. (I'm the IPv6 transition technical lead
> for a
> large organization.) While I actually manage my home internet gateway
> through a linux server and have fine-grained control over the firewall
> rules, I'm still debating whether I care enough about a handful of
> series
> to continue paying a company that is deliberately acting against its
> users'
> interests. Right now I'm leaning toward no. But I'll discuss it with my
> wife before making a final decision.
>
> Scott
>
> On Mon, Jun 6, 2016 at 8:03 AM, Mark Tinka 
> mailto:mark.ti...@seacom.mu>>
> wrote:
>
> >
> >
> > On 6/Jun/16 01:45, Damian Menscher wrote:
> >
> > >
> > > Who are these non-technical Netflix users who accidentally stumbled
> into
> > > having a HE tunnel broker connection without their knowledge?  I
> wasn't
> > > aware this sort of thing could happen without user consent, and
> would
> > like
> > > to know if I'm wrong.  Only thing I can imagine is if ISPs are
> using HE
> > as
> > > a form of CGN.
> >
> > There are several networks around the world that rely on 6-in-4
> because
> > their local provider does not offer IPv6.
> >
> > Mark.
> >

Re: Netflix VPN detection - actual engineer needed

[Matthew Huff]

Search this email thread (there was a link to a document dump), or use google. 
Neither Netflix nor the content providers have been very shy about this. Now 
for the speculation part … I think it’s possible that Netflix has gone along 
with this because they want to expand into countries that have restrictive 
policies (china, etc..) and will need to have system to either block or limit 
capabilities based on the geo-ip for other reasons. Just a hunch.


> On Jun 6, 2016, at 7:32 PM, Owen DeLong  wrote:
> 
> While I think this may well be the reason for Netflix’s actions, do you have 
> any evidence to back up this claim?
> 
> Actual evidence vs. just a very good educated guess and speculation could 
> prove very useful in this circumstance.
> 
> Owen
> 
>> On Jun 6, 2016, at 7:59 AM, Matthew Huff  wrote:
>> 
>> Netflix IS acting in their user's best interest. In order to provide content 
>> that the user's want, the content providers have mandated that they do their 
>> due diligence to block out of region users including VPN and open tunnel 
>> access. As Hulu and Amazon prime become more popular and their contracts 
>> with the content provides come due, they will have to also.
>> 
>> You can argue about the content provides business model all you want, but 
>> Netflix has to do what they are doing. They aren't blocking IPv6 users, they 
>> are blocking users that are using VPNs and/or tunnels since their currently 
>> is no practical way of providing GEOIP information about that users that the 
>> content providers require.
>> 
>> 
>> 
>> Matthew Huff | 1 Manhattanville Rd
>> Director of Operations   | Purchase, NY 10577
>> OTA Management LLC   | Phone: 914-460-4039
>> aim: matthewbhuff| Fax:   914-694-5669
>> 
>>> -Original Message-
>>> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Scott Morizot
>>> Sent: Monday, June 6, 2016 10:50 AM
>>> To: Mark Tinka 
>>> Cc: NANOG list 
>>> Subject: Re: Netflix VPN detection - actual engineer needed
>>> 
>>> I have Hulu Plus and Amazon Prime. The only thing I would miss from
>>> Netflix
>>> is their Marvel original series. And I can live with that. I can't live
>>> without my IPv6 enabled home network and Internet connection since
>>> that's
>>> an essential part of my job. (I'm the IPv6 transition technical lead
>>> for a
>>> large organization.) While I actually manage my home internet gateway
>>> through a linux server and have fine-grained control over the firewall
>>> rules, I'm still debating whether I care enough about a handful of
>>> series
>>> to continue paying a company that is deliberately acting against its
>>> users'
>>> interests. Right now I'm leaning toward no. But I'll discuss it with my
>>> wife before making a final decision.
>>> 
>>> Scott
>>> 
>>> On Mon, Jun 6, 2016 at 8:03 AM, Mark Tinka 
>>> wrote:
>>> 
>>>> 
>>>> 
>>>> On 6/Jun/16 01:45, Damian Menscher wrote:
>>>> 
>>>>> 
>>>>> Who are these non-technical Netflix users who accidentally stumbled
>>> into
>>>>> having a HE tunnel broker connection without their knowledge?  I
>>> wasn't
>>>>> aware this sort of thing could happen without user consent, and
>>> would
>>>> like
>>>>> to know if I'm wrong.  Only thing I can imagine is if ISPs are
>>> using HE
>>>> as
>>>>> a form of CGN.
>>>> 
>>>> There are several networks around the world that rely on 6-in-4
>>> because
>>>> their local provider does not offer IPv6.
>>>> 
>>>> Mark.
>>>> 
> 

RE: Netflix banning HE tunnels

[Matthew Huff]

The content providers wouldn't care if it was a very small number of people 
evading their region restrictions, but it isn't a small number. Those avoiding 
it are already not in good faith. While I don't agree with the content 
providers business model, it's their content, their rules. 

If you don't think it's right that Netflix is blocking VPNs and tunnels, then 
switch to Hulu and/or Amazon, however it's just matter of time before they 
start blocking VPNs and tunnels themselves.

I agree that matching Geolocation with source IP addresses is a bad idea, but 
until someone comes up with a better idea and gets it implemented ( one that 
can't be modified by the end user), people with a business model that depends 
on it will continue to block based on IP. "Good faith" will be laughed at, and 
rightly so.




Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669


> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Laszlo
> Hanyecz
> Sent: Wednesday, June 8, 2016 3:34 PM
> To: nanog@nanog.org
> Subject: Re: Netflix banning HE tunnels
> 
> 
> 
> On 2016-06-08 18:57, Javier J wrote:
> > Tony, I agree 100% with you. Unfortunately I need ipv6 on my media
> subnet
> > because it's part of my lab. And now that my teenage daughter is
> > complaining about Netflix not working g on her Chromebook I'm
> starting to
> > think consumers should just start complaining to Netflix. Why should
> I have
> > to change my damn network to fix Netflix?
> >
> > In her eyes it's "daddy fix Netflix" but the heck with that. The man
> hours
> > of the consumers who are affected to work around this issue is less
> than
> > the man hours it would take for Netflix to redirect you with a 301 to
> an
> > ipv4 only endpont.
> >
> > If Netflix needs help with this point me in the right direction. I'll
> be
> > happy to fix it for them and send them a bill.
> >
> 
> They're doing the same thing with IPv4 (banning people based on the
> apparent IP address).  Your IPv4 numbers may not be on their blacklist
> at the moment, and disabling IPv6 might work for you, but the
> underlying
> problem is the practice of GeoIP/VPN blocking, and the HE.net tunnels
> are just one example of the collateral damage.
> 
> I don't know why Netflix and other GeoIP users can't just ask customers
> where they are located, instead of telling them.  It is possible that
> some user might lie, but what about "assume good faith"?  It shows how
> much they value you as a customer if they would rather dump you than
> trust you to tell them where you are located.
> 
> -Laszlo
> 

RE: Netflix banning HE tunnels

[Matthew Huff]

Yes we do.

The is a document dump with the contract information between Netflix and the 
content providers. A link was sent in this email chain, or you can do a search 
for it. Neither side has been shy about what they are doing. They publically 
have stated they are blocking VPN access to NetFlix.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff| Fax:   914-694-5669

From: Spencer Ryan [mailto:sr...@arbor.net]
Sent: Wednesday, June 8, 2016 4:02 PM
To: Tony Hain 
Cc: Matthew Huff ; Laszlo Hanyecz ; North 
American Network Operators' Group 
Subject: Re: Netflix banning HE tunnels

We don't know, and will never know if the content providers went to Netflix and 
said "You need to ban based on IP range" speculation at this point isn't useful.


Spencer Ryan | Senior Systems Administrator | 
sr...@arbor.net<mailto:sr...@arbor.net>
Arbor Networks
+1.734.794.5033 (d) | +1.734.846.2053 (m)
www.arbornetworks.com<http://www.arbornetworks.com/>

On Wed, Jun 8, 2016 at 4:00 PM, Tony Hain 
mailto:alh-i...@tndh.net>> wrote:
Matthew,

I was not complaining about the business model, or the need to comply with 
content provider requirements. The issue is the pathetic implementation choice 
that Netflix made when a trivial alternative was available. I agree that 
setting up rwhois and trusting the 3rd party tunnel providers to provide valid 
information is substantially more effort than the ROI on this would justify, 
but a redirect to IPv4-only requires no additional 3rd party trust for geo-loc 
than an IPv4 connection to begin with, would still catch the bad actors, yet 
works correctly for those trying to move the Internet forward.

Tony


> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org<mailto:nanog-boun...@nanog.org>] 
> On Behalf Of Matthew
> Huff
> Sent: Wednesday, June 08, 2016 12:45 PM
> To: Laszlo Hanyecz; nanog@nanog.org<mailto:nanog@nanog.org>
> Subject: RE: Netflix banning HE tunnels
>
> The content providers wouldn't care if it was a very small number of people
> evading their region restrictions, but it isn't a small number. Those avoiding
> it are already not in good faith. While I don't agree with the content
> providers business model, it's their content, their rules.
>
> If you don't think it's right that Netflix is blocking VPNs and tunnels, then
> switch to Hulu and/or Amazon, however it's just matter of time before they
> start blocking VPNs and tunnels themselves.
>
> I agree that matching Geolocation with source IP addresses is a bad idea, but
> until someone comes up with a better idea and gets it implemented ( one
> that can't be modified by the end user), people with a business model that
> depends on it will continue to block based on IP. "Good faith" will be
> laughed at, and rightly so.
>
>
>
> 
> Matthew Huff | 1 Manhattanville Rd Director of Operations   |
> Purchase, NY 10577 OTA Management LLC   | Phone: 
> 914-460-4039
> aim: matthewbhuff| Fax:   914-694-5669
>
>
> > -Original Message-
> > From: NANOG 
> > [mailto:nanog-boun...@nanog.org<mailto:nanog-boun...@nanog.org>] On Behalf 
> > Of Laszlo
> > Hanyecz
> > Sent: Wednesday, June 8, 2016 3:34 PM
> > To: nanog@nanog.org<mailto:nanog@nanog.org>
> > Subject: Re: Netflix banning HE tunnels
> >
> >
> >
> > On 2016-06-08 18:57, Javier J wrote:
> > > Tony, I agree 100% with you. Unfortunately I need ipv6 on my media
> > subnet
> > > because it's part of my lab. And now that my teenage daughter is
> > > complaining about Netflix not working g on her Chromebook I'm
> > starting to
> > > think consumers should just start complaining to Netflix. Why should
> > I have
> > > to change my damn network to fix Netflix?
> > >
> > > In her eyes it's "daddy fix Netflix" but the heck with that. The man
> > hours
> > > of the consumers who are affected to work around this issue is less
> > than
> > > the man hours it would take for Netflix to redirect you with a 301
> > > to
> > an
> > > ipv4 only endpont.
> > >
> > > If Netflix needs help with this point me in the right direction.
> > > I'll
> > be
> > > happy to fix it for them and send them a bill.
> > >
> >
> > They're doing the same thing with IPv4 (banning people based on the
> > apparent IP address).  Your IPv4 numbers may not be on their blacklist
> > at the moment, and disabling IPv6 might work for you, but the
> > underlying problem is the practice of GeoIP/VPN blocking, and the
> > HE.net tunnels are just one example of the collateral damage.
> >
> > I don't know why Netflix and other GeoIP users can't just ask
> > customers where they are located, instead of telling them.  It is
> > possible that some user might lie, but what about "assume good faith"?
> > It shows how much they value you as a customer if they would rather
> > dump you than trust you to tell them where you are located.
> >
> > -Laszlo
> >

Re: Netflix banning HE tunnels

[Matthew Huff]

What does https://www.maxmind.com/en/geoip-demo show for your IPv6 prefix? If 
it is incorrect, try https://support.maxmind.com/geoip-data-correction-request/

On Jun 8, 2016, at 5:08 PM, Chris Knipe  wrote:
> 
> Exactly.
> 
> So what precisely are the metrics they use to block?  I'm not using a proxy
> at all, its my own ASN...
> 
> On Wed, Jun 8, 2016 at 11:06 PM, Andy Ringsmuth  wrote:
> 
>> 
>>> On Jun 8, 2016, at 3:52 PM, Chris Knipe  wrote:
>>> 
>>> Bwahaha
>>> 
>>> Ok - that's me, never ever will I look at NexFlix again.
>>> 
>>> I have my own /48, registered in my own name, my own company, my own
>>> peering links, and my own transit links.  Signup, no problems.  As soon
>> as
>>> I started watching a stream...
>>> 
>>> Wham, blocked.  Proxy Detected.
>>> 
>>> It's clear NetFlix has something against IPv6, not tunnels.
>> 
>> I disagree.
>> 
>> I’ve got IPv6 at work, nothing elaborate, just a /48 given to us by our
>> ISP.
>> 
>> I ran a test on an IPv6-only connection, no IPv4 addressing whatsoever,
>> and a few random Netflix shows played perfectly fine.
>> 
>> 
>> 
>> Andy Ringsmuth
>> a...@newslink.com
>> News Link – Manager Travel, Technology & Facilities
>> 2201 Winthrop Rd., Lincoln, NE 68502-4158
>> (402) 475-6397(402) 304-0083 cellular
>> 
>> 
> 
> 
> -- 
> 
> Regards,
> Chris Knipe
https://www.maxmind.com/en/geoip-demo

RE: Netflix banning HE tunnels

[Matthew Huff]

I think you are missing the point. The problem is not that the GeoIP info is 
missing, the problem with the HE.tunnel is that the GeoIP is not set by the 
provider by verifiable means. Letting end-users set their GeoIP information is 
a non-starter for the content provider and they would still require a ban.

A solution would be for HE to automatically set the IPv6 geoip based on the 
geoip of the IPv4 source address with no user overrides. Since the whole 
process would be fragile (people change their IPv4 source address frequently 
when travelling, etc...) and the delay it takes to put the GeoIP into maxmind's 
database, etc..., I don't know how well it would work, but it would probably be 
the best bet.

----
Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669


> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Michael Still
> Sent: Thursday, June 9, 2016 10:10 AM
> To: John Lightfoot 
> Cc: North American Network Operators' Group 
> Subject: Re: Netflix banning HE tunnels
> 
> I wonder how hard it would be for HE to implement some button on their
> tunnel portal that when selected will update Maxmind's (or whatever)
> geolocation for their allocated IPv6 prefixes to match the results
> returned
> when querying for their IPv4 tunnel end point address...
> 
> I would suggest to make it optional since some may not want to have
> this
> functionality by default but if you are dying for netflix and are in
> whatever geolocation netflix deems acceptable this may solve the issue.
> 
> 
> 
> On Wed, Jun 8, 2016 at 5:39 PM, John Lightfoot 
> wrote:
> 
> > How about:
> >
> > Dear Netflix network engineer who’s on the NANOG list.  Could you
> please
> > get Netflix to fall back to ipv4 if you block your customer’s ipv6
> because
> > it’s in an HE tunnel?  Lots of people who want to watch Netflix, be
> able to
> > reach the whole internet, and have Verizon FiOS would really
> appreciate it.
> >
> > Thanks,
> > John
> >
> > John Lightoot
> >
> >
> >
> 
> 
> --
> [stillwa...@gmail.com ~]$ cat .signature
> cat: .signature: No such file or directory
> [stillwa...@gmail.com ~]$

RE: Netflix banning HE tunnels

[Matthew Huff]

Your correct. I misread your email. Not enough blood in my caffeine stream yet. 
I think your idea of a button and/or a daily/weekly update to maxmind based on 
the source IPv4 address would be a good idea regardless of Netflix.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff| Fax:   914-694-5669

From: Michael Still [mailto:stillwa...@gmail.com]
Sent: Thursday, June 9, 2016 10:33 AM
To: Matthew Huff 
Cc: John Lightfoot ; North American Network Operators' 
Group 
Subject: Re: Netflix banning HE tunnels

Uhm I think you misunderstood me. What you describe matches what I described. I 
never suggested the user can override it with it another value, I am suggesting 
that a user may want to keep it to whatever default value it is as a matter of 
privacy concerns. Otherwise use the IPv4 tunnel end point IP.

As for the point about people moving around frequently I would consider that a 
pretty far reaching use case and most likely not worth considering any action 
for.


On Thu, Jun 9, 2016 at 10:19 AM, Matthew Huff 
mailto:mh...@ox.com>> wrote:
I think you are missing the point. The problem is not that the GeoIP info is 
missing, the problem with the HE.tunnel is that the GeoIP is not set by the 
provider by verifiable means. Letting end-users set their GeoIP information is 
a non-starter for the content provider and they would still require a ban.

A solution would be for HE to automatically set the IPv6 geoip based on the 
geoip of the IPv4 source address with no user overrides. Since the whole 
process would be fragile (people change their IPv4 source address frequently 
when travelling, etc...) and the delay it takes to put the GeoIP into maxmind's 
database, etc..., I don't know how well it would work, but it would probably be 
the best bet.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff| Fax:   914-694-5669


> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org<mailto:nanog-boun...@nanog.org>] 
> On Behalf Of Michael Still
> Sent: Thursday, June 9, 2016 10:10 AM
> To: John Lightfoot mailto:jlightf...@gmail.com>>
> Cc: North American Network Operators' Group 
> mailto:nanog@nanog.org>>
> Subject: Re: Netflix banning HE tunnels
>
> I wonder how hard it would be for HE to implement some button on their
> tunnel portal that when selected will update Maxmind's (or whatever)
> geolocation for their allocated IPv6 prefixes to match the results
> returned
> when querying for their IPv4 tunnel end point address...
>
> I would suggest to make it optional since some may not want to have
> this
> functionality by default but if you are dying for netflix and are in
> whatever geolocation netflix deems acceptable this may solve the issue.
>
>
>
> On Wed, Jun 8, 2016 at 5:39 PM, John Lightfoot 
> mailto:jlightf...@gmail.com>>
> wrote:
>
> > How about:
> >
> > Dear Netflix network engineer who’s on the NANOG list.  Could you
> please
> > get Netflix to fall back to ipv4 if you block your customer’s ipv6
> because
> > it’s in an HE tunnel?  Lots of people who want to watch Netflix, be
> able to
> > reach the whole internet, and have Verizon FiOS would really
> appreciate it.
> >
> > Thanks,
> > John
> >
> > John Lightoot
> >
> >
> >
>
>
> --
> [stillwa...@gmail.com<mailto:stillwa...@gmail.com> ~]$ cat .signature
> cat: .signature: No such file or directory
> [stillwa...@gmail.com<mailto:stillwa...@gmail.com> ~]$



--
[stillwa...@gmail.com<mailto:stillwa...@gmail.com> ~]$ cat .signature
cat: .signature: No such file or directory
[stillwa...@gmail.com<mailto:stillwa...@gmail.com> ~]$

BGP route instabilities

[Matthew Huff]

We saw a slight uptick in routes today (at least since the last time I looked), 
but a large number of route flaps coming out of the APNIC region. Anyone else 
notice anything?


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577

Someone didn't get the leap second memo...

[Matthew Huff]

[root@hayden ~]# ntpq -p
 remote   refid  st t when poll reach   delay   offset  jitter
==
 LOCAL(0).LOCL.  10 l  20d   6400.0000.000   0.000
-clock.xmission. 132.163.4.1032 u  169  256  377   66.078   -1.302   0.164
xclock.sjc.he.ne 10.200.208.2 2 u   13  256  315   65.689  999.633   2.015
+tock.usshc.com  .GPS.1 u   87  256  377   26.930   -0.550   0.121
*ntp.your.org.CDMA.   1 u   43  256  217   23.3390.544   0.069

Our batch system went belly up, but other than that, no other apparent leap 
second issues.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

RE: Questions on IPv6 deployment

[Matthew Huff]

The reason for allocating a /64 for a point to point link is due to various 
denial of service attack vectors. Just do it. The numbers in IPv6 are 
staggering. The generally accepted best practice is to allocate a /64 and use a 
/128 within that /64 for point to point links.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669


> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of William
> Herrin
> Sent: Tuesday, January 17, 2017 4:02 PM
> To: Michael Still 
> Cc: nanog@nanog.org
> Subject: Re: Questions on IPv6 deployment
> 
> On Tue, Jan 17, 2017 at 12:48 PM, Michael Still 
> wrote:
> > http://nabcop.org/index.php/IPv6_Subnetting
> 
> That's overall good advice. I quibble with a couple of points:
> 
> 1. If you plan to use a /126 on a point to point and can't imagine how
> you would use a /64 on that point to point, don't allocate a /64. Odds
> are that by the time you can imagine some way to use a /64 there, the
> details will require you to assign a new block anyway.
> 
> Why be concerned about resource consumption? Because it's a good
> habit. Don't overdo it, IPv6 is not resource constrained the way IPv4
> is, but shrewd use of available resources is a good habit even when
> resources are plentiful.
> 
> 2. Make all your point to points /124. That will work for all your
> point to points. Serial or ethernet. Even the ethernets which have two
> high-availability routers on both ends along with the failover address
> needing a total of 6 IPs plus 1 for your troubleshooting laptop.
> Configuring /124 every time allows you to standardize your
> configuration, the same way /64 standardizes the netmask on a LAN
> deployment.
> 
> 
> 
> One additional point not brought up:
> 
> Minimum assignment to a customer: /60. Never ever /64 or /128. How
> much more than a /60 you choose as your minimum is up to you. Common
> choices are /56 and /48. But never, ever less than a /60.   Your
> customer will want to deploy a /64 to each LAN. And there are so many
> cases where he'll want to deploy more than one LAN.
> 
> I've noticed a lot of hosting providers getting this wrong. Some of
> your customers do create VPNs on their VPC you know.
> 
> Regards,
> Bill Herrin
> 
> 
> --
> William Herrin  her...@dirtside.com  b...@herrin.us
> Owner, Dirtside Systems . Web: <http://www.dirtside.com/>

RE: Questions on IPv6 deployment

[Matthew Huff]

Please check the nanog archives. There were some arguments that I and I assume 
others felt compelling why allocating a /64 per point to point link was a good 
idea. Your network, your rules. I was just responding to the argument that a 
/64 is wasteful and serves little purpose.



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669


> -Original Message-
> From: William Herrin [mailto:b...@herrin.us]
> Sent: Tuesday, January 17, 2017 4:56 PM
> To: Matthew Huff 
> Cc: Michael Still ; nanog@nanog.org
> Subject: Re: Questions on IPv6 deployment
> 
> On Tue, Jan 17, 2017 at 4:07 PM, Matthew Huff  wrote:
> > The reason for allocating a /64 for a point to point link is due to
> various denial of service attack vectors.
> 
> Hi Matthew,
> 
> I'm always interested in learning something new. Please explain the
> DOS vectors you're referring to and how they're mitigated by
> allocating a /64 to the point to point link.
> 
> 
> > Just do it.
> 
> No. But if you offer a good reason, I'll factor your reason in to my
> considerations.
> 
> Regards,
> Bill Herrin
> 
> --
> William Herrin  her...@dirtside.com  b...@herrin.us
> Owner, Dirtside Systems . Web: <http://www.dirtside.com/>

RE: Console Servers

[Matthew Huff]

If anyone is looking for a product that is reasonably priced and is still being 
produced/update, the ADVA Optical (aka MRV, aka Xyplex) console servers still 
work great

https://www.advaoptical.com/en/products/network-infrastructure-assurance/lx-series

From their specs:
4, 8, 16, 32 and 48 serial ports
V.92 modem option
Single or dual power
120-240VAC, 50/60Hz: 0.5A per system
36-72VDC dual feed: 0.75A per system
2 x Ethernet
NEBS Level 3 certified

RE: Oct. 3, 2018 EAS Presidential Alert test

[Matthew Huff]

I received it on my iPhone XS Max running iOS 12.0 with AT&T, wifi calling 
off...

----
Matthew Huff | 1 Manhattanville Rd 
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039


-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Andy Ringsmuth
Sent: Wednesday, October 3, 2018 2:53 PM
To: nanog@nanog.org
Subject: Oct. 3, 2018 EAS Presidential Alert test

Did anyone on AT&T or an iPhone receive the test today? I believe it was 
supposed to happen at 2:18 EDT, followed by one on broadcast radio at 2:20 EDT.

I’m in CDT, so 1:18 and 1:20 p.m. CDT.

Message was heard on my desk radio at 1:21:35 p.m. CDT but as of the sending of 
this at 1:52 p.m. CDT, nothing on phones. I have an office full of AT&T iPhones 
and not a single one of them alerted.

FEMA says https://www.fema.gov/emergency-alert-test

"Cell towers will broadcast the WEA test for approximately 30 minutes beginning 
at 2:18 p.m. EDT. During this time, WEA compatible cell phones that are 
switched on, within range of an active cell tower, and whose wireless provider 
participates in WEA should be capable of receiving the test message. Some cell 
phones will not receive the test message, and cell phones should only receive 
the message once."

My wife, with a Sprint iPhone, received the test.



Andy Ringsmuth
5609 Harding Drive
Lincoln, NE 68521-5831
(402) 304-0083
a...@andyring.com

RE: Verizon: Extremely Strange CPE Routing in NYC/NJ Area

[Matthew Huff]

Windows traceroute uses ICMP by default, Unix based systems use UDP. For 
example from my Mac on Fios in Westchester, NY (and yes, they are doing 
something  with ICMP traceroutes, possibly within their gateway box):


acpro:Documents mhuff$ sudo traceroute www.yahoo.com
traceroute: Warning: www.yahoo.com has multiple addresses; using 72.30.35.9
traceroute to atsv2-fp-shed.wg1.b.yahoo.com (72.30.35.9), 64 hops max, 52 byte 
packets
 1  firewall (10.1.1.1)  0.916 ms  0.419 ms  0.394 ms
 2  * * *
 3  b3369.nycmny-lcr-22.verizon-gni.net (130.81.16.102)  3.939 ms  10.172 ms
b3369.nycmny-lcr-21.verizon-gni.net (130.81.16.100)  5.192 ms
 4  * * *
 5  0.ae4.xl2.buf4.alter.net (140.222.228.143)  13.500 ms *  12.970 ms
 6  0.xe-5-3-0.gw1.buf4.alter.net (152.63.26.62)  12.390 ms  12.122 ms
0.ae4.xl2.buf4.alter.net (140.222.228.143)  12.704 ms
 7  0.xe-4-2-0.gw1.buf4.alter.net (152.63.26.46)  14.326 ms  13.069 ms
verizon.com.customer.alter.net (204.148.47.162)  14.436 ms
 8  et-19-0-0.pat1.bfz.yahoo.com (216.115.97.103)  13.990 ms
unknown-72-30-223-x.yahoo.com (72.30.223.3)  14.220 ms
et-19-0-0.pat1.bfz.yahoo.com (216.115.97.103)  13.711 ms
 9  et-0-0-0.msr2.bf1.yahoo.com (74.6.227.137)  12.098 ms
et-0-0-0.msr1.bf1.yahoo.com (74.6.227.129)  13.388 ms
et-18-0-1.msr1.bf1.yahoo.com (74.6.227.131)  14.457 ms
10  et-19-0-0.clr2-a-gdc.bf1.yahoo.com (74.6.122.37)  14.554 ms
et-0-0-0.msr1.bf1.yahoo.com (74.6.227.129)  14.810 ms
et-1-1-1.msr1.bf1.yahoo.com (74.6.227.135)  13.502 ms
11  eth-18-3.bas2-1-flk.bf1.yahoo.com (98.139.128.75)  14.607 ms  16.608 ms
eth-18-3-bas1-1-flk.bf1.yahoo.com (98.139.128.73)  15.161 ms
12  media-router-fp1.prod1.media.vip.bf1.yahoo.com (72.30.35.9)  13.470 ms
eth-17-3.bas2-1-flk.bf1.yahoo.com (98.139.128.71)  13.582 ms
media-router-fp1.prod1.media.vip.bf1.yahoo.com (72.30.35.9)  12.952 ms

macpro:Documents mhuff$ sudo traceroute -I www.yahoo.com
traceroute: Warning: www.yahoo.com has multiple addresses; using 72.30.35.10
traceroute to atsv2-fp-shed.wg1.b.yahoo.com (72.30.35.10), 64 hops max, 72 byte 
packets
 1  firewall (10.1.1.1)  0.675 ms  0.347 ms  0.322 ms
 2  media-router-fp2.prod1.media.vip.bf1.yahoo.com (72.30.35.10)  2.456 ms  
21.139 ms  12.834 ms


Matthew Huff | 1 Manhattanville Rd 
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039


-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Lee
Sent: Thursday, November 29, 2018 5:42 PM
To: Nick Zurku 
Cc: Brendan Mannella ; nanog@nanog.org; Todd Kammerer 

Subject: Re: Verizon: Extremely Strange CPE Routing in NYC/NJ Area

On 11/29/18, Nick Zurku  wrote:
> Can anyone from Verizon take a look at this behavior for us?
>
> We’re having multiple Verizon FiOS users in the NYC/NJ area appear to 
> teleport from their FiOS router to our IP in the Pittsburgh region.

Verizon is doing something seriously weird to windows traceroute:
C:\Users\Lee>tracert www.yahoo.com

Tracing route to atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232] over a maximum 
of 30 hops:

  1<1 ms<1 ms<1 ms  fw.home.net
  2 1 ms<1 ms<1 ms  vbz-router.home.net [192.168.1.1]
  3 8 ms 3 ms 6 ms
media-router-fp2.prod1.media.vip.ne1.yahoo.com [98.138.219.232]

Trace complete.

C:\Users\Lee>ping -i 12 www.yahoo.com.

Pinging atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232] with 32 bytes of data:
Reply from 98.138.0.87: TTL expired in transit.
Reply from 98.138.0.87: TTL expired in transit.
Reply from 98.138.0.87: TTL expired in transit.
Reply from 98.138.0.87: TTL expired in transit.

Ping statistics for 98.138.219.232:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

C:\Users\Lee>ping -i 13 www.yahoo.com.

Pinging atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232] with 32 bytes of data:
Reply from 98.138.219.232: bytes=32 time=32ms TTL=54 Reply from 98.138.219.232: 
bytes=32 time=31ms TTL=54 Reply from 98.138.219.232: bytes=32 time=33ms TTL=54 
Reply from 98.138.219.232: bytes=32 time=33ms TTL=54

Ping statistics for 98.138.219.232:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip 
times in milli-seconds:
Minimum = 31ms, Maximum = 33ms, Average = 32ms

C:\Users\Lee>

traceroute from a linux box works as expected..

Lee

> Users
> are seeing extreme slowness with TCP traffic, but ping times seem 
> reasonable.
>
> User 1:
>  1  fios_quantum_gateway (192.168.1.1)  1.575 ms  2.426 ms  3.193 ms
>  2  204.16.244.8 (204.16.244.8)  2.269 ms  3.055 ms  2.727 ms
>
> User 2:
>  1  fios_quantum_gateway (192.168.1.1)  1.565 ms  1.048 ms  0.947 ms
>  2  204.16.244.8 (204.16.244.8)  2.162 ms  3.588 ms  3.048 ms
>
> I can provide end-user NYC/NJ IPs off-list if desirable.
>
>
> Here's a normal looking trace from an FiOS line locally in the 
> Pittsburgh
> region:
>
>
> IP:  10

RE: CenturyLink

[Matthew Huff]

Yep.

We are required by FINRA to verify that our clocks on our trading systems are 
within a certain tolerance of NIST time. We are still seeing issues with 3 of 
the NIST servers this morning. Since NIST is on a skeleton crew anyway due to 
the government shutdown, I don't expect any resolution shortly.


Matthew Huff | 1 Manhattanville Rd 
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039


-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Yang Yu
Sent: Friday, December 28, 2018 6:23 PM
To: Stephane Bortzmeyer 
Cc: nanog@nanog.org
Subject: Re: CenturyLink

On Fri, Dec 28, 2018 at 12:05 AM Stephane Bortzmeyer  wrote:
> Is this problem also responsible for the 911 outage? If so, the 
> post-mortem analysis is not useful only for CenturyLink customers but 
> for everyone on the west coast.

Looks like most time.nist.gov servers (3 x NIST sites on AS49) are single homed 
on CenturyLink, anyone noticed NTP issues yesterday?

https://tf.nist.gov/tf-cgi/servers.cgi

RE: CenturyLink

[Matthew Huff]

We have two stratum-1 servers synced with GPS and a PTP feed from a provider 
that also provides PTP to market data systems, but we still have to monitor 
drift between system time and NIST time. Don't ask for the logic behind it, 
it's a regulation, not a technical requirement.

----
Matthew Huff | 1 Manhattanville Rd 
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039


-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Stephen Satchell
Sent: Saturday, December 29, 2018 9:34 AM
To: nanog@nanog.org
Subject: Re: CenturyLink

On 12/28/18 3:23 PM, Yang Yu wrote:
> On Fri, Dec 28, 2018 at 12:05 AM Stephane Bortzmeyer  
> wrote:
>> Is this problem also responsible for the 911 outage? If so, the 
>> post-mortem analysis is not useful only for CenturyLink customers but 
>> for everyone on the west coast.
> 
> Looks like most time.nist.gov servers (3 x NIST sites on AS49) are 
> single homed on CenturyLink, anyone noticed NTP issues yesterday?
> 
> https://tf.nist.gov/tf-cgi/servers.cgi
> 

I have GPS-based Stratum 1 NTP appliances in my network, so I wouldn't see any 
issues.  I suspect many other operators are in the same situation.

RE: CenturyLink

[Matthew Huff]

Actually, on all our trading systems, our times are synced via PTP instead of 
ntpd for at least 50 microsecond accuracy. The stratum 1 clocks as well as NIST 
time are only used as comparison to verify compliance and reality. We use ntpq 
to determine the offset  from NIST for reporting.


Matthew Huff | 1 Manhattanville Rd 
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039


-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Stephen Satchell
Sent: Saturday, December 29, 2018 10:01 AM
To: nanog@nanog.org
Subject: Re: CenturyLink

On 12/29/18 6:51 AM, Matthew Huff wrote:
> We have two stratum-1 servers synced with GPS and a PTP feed from a provider 
> that also provides PTP to market data systems, but we still have to monitor 
> drift between system time and NIST time. Don't ask for the logic behind it, 
> it's a regulation, not a technical requirement.

Having been a participant on Standards Working Groups, I understand completely. 
 Regulations, like Standards, need to be written to apply to as wide a 
population as possible.  Do those regulations dictate how you monitor drift?  
For example, can you have a system (I would use CentOS) that syncs to the 
appliance as well as NIST and your inside NTP sources, and use ntpq(8) to read 
the drift directly?

RE: CenturyLink

[Matthew Huff]

We use an older model of 
https://www.microsemi.com/product-directory/enterprise-network-time-servers/4117-syncserver-s600
 with rubidium oscillator. Not cheap, but hardened and extremely accurate.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039

From: Shawn L [mailto:sha...@up.net]
Sent: Sunday, December 30, 2018 9:40 AM
To: nanog@nanog.org
Cc: Matthew Huff ; l...@satchell.net
Subject: Re: CenturyLink


Speaking of GPS-enabled NTP appliances, etc. wondering what hardware people are 
using for this.



thanks



-Original Message-
From: "Raymond Burkholder" mailto:r...@oneunified.net>>
Sent: Saturday, December 29, 2018 12:01pm
To: "Matthew Huff" mailto:mh...@ox.com>>, 
"l...@satchell.net<mailto:l...@satchell.net>" 
mailto:l...@satchell.net>>, 
"nanog@nanog.org<mailto:nanog@nanog.org>" 
mailto:nanog@nanog.org>>
Subject: Re: CenturyLink

On 2018-12-29 7:51 a.m., Matthew Huff wrote:
> We have two stratum-1 servers synced with GPS and a PTP feed from a provider 
> that also provides PTP to market data systems, but we still have to monitor 
> drift between system time and NIST time. Don't ask for the logic behind it, 
> it's a regulation, not a technical requirement.
>
On one occasion, due to bad firmware or a configuration issue, I have
seen GPS stratum 1 diverge from NTP.  It was somewhat eye brow raising
to the company.  My NTP monitored servers were shown to be diverging
their GPS/NTP, but after looking at twice or thrice, it was the other
way around.

RE: CenturyLink

[Matthew Huff]

Regulatory.

If we were to lose the GPS signal (antenna failure, etc...) then our stratum 1 
time sources wouldn't drift as much and as quickly. For telco and general 
usage, the cost may not be worthwhile, but when you have auditors looking over 
your shoulder


Matthew Huff | 1 Manhattanville Rd 
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039


-Original Message-
From: Saku Ytti [mailto:s...@ytti.fi] 
Sent: Sunday, December 30, 2018 12:30 PM
To: Matthew Huff 
Cc: Shawn L ; nanog@nanog.org
Subject: Re: CenturyLink

Hey Matthew,

> We use an older model of 
> https://www.microsemi.com/product-directory/enterprise-network-time-servers/4117-syncserver-s600
>  with rubidium oscillator. Not cheap, but hardened and extremely accurate.

Out of interest why rubidium? For short term stability oscillator choice isn't 
much of a matter for free running even rubidium will be off good +-1us per day 
or +-30us per week. I've always wondered what niche rubidium covers that isn't 
handled by much cheaper crystal ovens or actually precise oscillators.

--
  ++ytti

RE: CenturyLink

[Matthew Huff]

There isn't a specific regulation on free-running GPS, just "due diligence". I 
work at a algorithmic program trading company (and have been for 20 years). We 
have a high ROI, the cost differential for the rubidium OC versus having to 
drop everything to conform to regulatory requirements due to a short GPS 
outage, makes this a no-brainer.

----
Matthew Huff | 1 Manhattanville Rd 
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039


-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Saku Ytti
Sent: Monday, December 31, 2018 3:28 AM
To: Gary E. Miller 
Cc: nanog@nanog.org
Subject: Re: CenturyLink

Hey Gary,

On Mon, 31 Dec 2018 at 05:02, Gary E. Miller  wrote:

> The Rb frequency reference will be two or three orders of magnitude 
> more stable than an expensive ovenized crystal.

Perhaps, but not supported by this:
https://www.meinbergglobal.com/english/specs/gpsopt.htm

For the tl;dr folk, crystal drifts +-4.5us per day, Rb +-1.1us (both seem like 
unsatisfactorily high numbers to me, i.e. you don't want to be free-running 24h 
with Rb). Luckily today we have GPS, Glonass, BeiDou, Galileo and couple 
smaller ones, so there should be somewhat reasonable amount of redundancy. 
Unsure which commercially available NTP or PPP master clocks support all four.

But I of course readily accept Rb is objectively more accurate than crystal, 
I'm just curious where it matters and I'm curious which regulation applies, who 
fall under the regulation and what specifically does the regulation require 
about free-running accuracy.

--
  ++ytti

Reliability of looking glass sites / rviews

[Matthew Huff]

This weekend our uninterruptible power supply became interruptible and we lost 
all circuits. While I was doing initial debugging of the problem while I waited 
on site power verification, I noticed that there was still paths being shown in 
rviews for the circuit that were down. This was over an hour after we went hard 
down and it took hours before we were back up.

I worked with our providers last night to verify there weren't any hanging 
static routes, etc... We shut the upstream circuit down and watched the 
convergence and saw that eventually all the paths disappeared. Given what we 
saw on Saturday, what would cause route-views to cache the paths that long?  
Some looking glass sites only show what they are peered with or at most what 
their peers are peered with, that's why I've always used route-views.

What looking glass sites other than route-views would people recommend?

Getting an RADB entry removed that was added by a previous peer

[Matthew Huff]

It appears that Reliance Globalcom  (AS6157) added an RADB entry for our prefix 
(129.77.0.0/16) when we were a peer of theirs years ago, and it was never 
removed when we ended the relationship. We are ASN 14607.

I've reached out to their support, but does anyone have a suggestion on how I 
could get this cleaned up if they don't respond?

Re: Reliability of looking glass sites / rviews

[Matthew Huff]

Both should have been similar.

In the first case we lost power to all of our BGP border routers that are 
peered with the upstream providers
In the second case, I did an explicit “shut” on the interface connected to the 
upstream provider that appeared “stuck” after an hour after the outage.

From:  on behalf of Christopher Morrow 

Date: Wednesday, September 13, 2017 at 10:58 AM
To: Matthew Huff 
Cc: nanog2 
Subject: Re: Reliability of looking glass sites / rviews



On Wed, Sep 13, 2017 at 5:30 AM, Matthew Huff 
mailto:mh...@ox.com>> wrote:
This weekend our uninterruptible power supply became interruptible and we lost 
all circuits. While I was doing initial debugging of the problem while I waited 
on site power verification, I noticed that there was still paths being shown in 
rviews for the circuit that were down. This was over an hour after we went hard 
down and it took hours before we were back up.

explicit vs implicit withdrawals causing different handling of the problem 
routes?

I worked with our providers last night to verify there weren't any hanging 
static routes, etc... We shut the upstream circuit down and watched the 
convergence and saw that eventually all the paths disappeared. Given what we 
saw on Saturday, what would cause route-views to cache the paths that long?  
Some looking glass sites only show what they are peered with or at most what 
their peers are peered with, that's why I've always used route-views.

What looking glass sites other than route-views would people recommend?

ripe ris.

RE: FW: Reliability of looking glass sites / rviews

[Matthew Huff]

ASN 14607, and 129.77.0.0/16

After slightly over an hour after our power event where 100% of our equipment 
was down, this is what I saw at routeviews

BGP routing table entry for 129.77.0.0/16, version 24978989
Paths: (7 available, best #7, table default)
  Not advertised to any peer
  Refresh Epoch 1
  134708 3491 6939 46887 14607
    103.197.104.1 from 103.197.104.1 (123.108.254.70)
      Origin IGP, localpref 100, valid, external
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
   1273 6939 46887 14607
    193.0.0.56 from 193.0.0.56 (193.0.0.56)
      Origin IGP, localpref 100, valid, external
      Community: 1273:23000
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
  8283 57866 6762 6939 46887 14607
    94.142.247.3 from 94.142.247.3 (94.142.247.3)
      Origin IGP, metric 0, localpref 100, valid, external
      Community: 6762:33 6762:16500 8283:15 57866:105
      unknown transitive attribute: flag 0xE0 type 0x20 length 0xC
        value  205B  0006  000F 
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
  24441 3491 3491 6939 46887 14607
    202.93.8.242 from 202.93.8.242 (202.93.8.242)
      Origin IGP, localpref 100, valid, external
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
  20912 1267 1273 6939 46887 14607
    212.66.96.126 from 212.66.96.126 (212.66.96.126)
      Origin IGP, localpref 100, valid, external
      Community: 1273:23000 9035:50 9035:100 20912:65001
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
  1221 4637 6939 46887 14607
    203.62.252.83 from 203.62.252.83 (203.62.252.83)
      Origin IGP, localpref 100, valid, external
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
  2497 6939 46887 14607
    202.232.0.2 from 202.232.0.2 (202.232.0.2)
      Origin IGP, localpref 100, valid, external, best
      rx pathid: 0, tx pathid: 0x0


From: Tim Evens [mailto:t...@snas.io] 
Sent: Friday, September 15, 2017 10:45 AM
To: Matthew Huff 
Cc: morrowc.li...@gmail.com; nanog@nanog.org
Subject: Re: FW: Reliability of looking glass sites / rviews

You didn't mention details about which ASN or prefixes you were checking.  Are 
you referring to ASN 14607 that only advertises two prefixes 129.77.0.0/16 and 
2620:0:2810::/48?

Based what we see over the weekend (using routeviews data), we see:

Event Start Time: 2017-09-09 11:29:23 UTC (2017-09-09 07:29:23 EDT)
Event End Time: 2017-09-09 13:31:30 UTC (2017-09-09 09:31:30 EDT)

Are the above times correct?

We see the routes withdraw and then come back.   For example: 
http://demo-rv.snas.io:3000/dashboard/db/prefix-history?orgId=2&var-prefix=129.77.0.0&var-prefix_len=16&var-asn_num=All&var-router_name=All&var-peer_name=All&from=150490800&to=150520320

When you checked routeviews, which router and peer were you looking at?  When 
you did a "show ip bgp ..." did you include the prefix length? If not, it would 
have then shown you 0/0 or 128/5, depending on which router you were on.


--Tim 





On 9/13/17, 8:43 AM, "NANOG on behalf of Matthew Huff"  wrote:

Both should have been similar.

In the first case we lost power to all of our BGP border routers that are 
peered with the upstream providers
In the second case, I did an explicit “shut” on the interface connected to 
the upstream provider that appeared “stuck” after an hour after the outage.

From:  on behalf of Christopher Morrow 

Date: Wednesday, September 13, 2017 at 10:58 AM
To: Matthew Huff 
Cc: nanog2 
Subject: Re: Reliability of looking glass sites / rviews

    
    
On Wed, Sep 13, 2017 at 5:30 AM, Matthew Huff 
mailto:mh...@ox.com>> wrote:
This weekend our uninterruptible power supply became interruptible and we 
lost all circuits. While I was doing initial debugging of the problem while I 
waited on site power verification, I noticed that there was still paths being 
shown in rviews for the circuit that were down. This was over an hour after we 
went hard down and it took hours before we were back up.

explicit vs implicit withdrawals causing different handling of the problem 
routes?

I worked with our providers last night to verify there weren't any hanging 
static routes, etc... We shut the upstream circuit down and watched the 
convergence and saw that eventually all the paths disappeared. Given what we 
saw on Saturday, what would cause route-views to cache the paths that long?  
Some looking glass sites only show what they are peered with or at most what 
their peers are peered with, that's why I've always used route-views.

What looking glass sites other than route-views would people recommend?

ripe ris.


 
 

Opensource SNMP Trap Receivers ???

[Matthew Huff]

We are retiring a legacy SNMP system and are looking for a simple, opensource 
SNMP trap receiver/alerting system. We aren't looking for a full SNMP system, 
just something that will receive snmp traps and email/alert based on them.

1) Looking for something off the shelf, not a development project
2) Opensource or low cost
3) SNMP MIB compiler

Any suggestions?


Matthew Huff | 1 Manhattanville Rd 
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039

RE: Opensource SNMP Trap Receivers ???

[Matthew Huff]

Oh hell yes, there isn’t anything simple about SNMP. A number of people have 
very quickly suggested SNMPTT, which is the sort of product I was looking for. 
My google foo had failed me. Thanks.


Matthew Huff | 1 Manhattanville Rd 
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039

From: kscott.he...@gmail.com [mailto:kscott.he...@gmail.com] On Behalf Of K. 
Scott Helms
Sent: Tuesday, February 13, 2018 8:09 AM
To: Matthew Huff 
Cc: nanog 
Subject: Re: Opensource SNMP Trap Receivers ???

Matthew,

Sadly, open source + SNMP traps != simple.

The simplest option that I've personally used in the past is SNMPTT with Nagios.

https://paulgporter.net/2013/09/16/nagios-snmp-traps/

http://www.snmptt.org/docs/snmptt.shtml

The main problem is that SNMP traps, like most of SNMP, aren't simple despite 
the name.  Having said that it can be done especially if the gear doesn't 
change too often.

Scott Helms

On Tue, Feb 13, 2018 at 7:44 AM, Matthew Huff  wrote:
We are retiring a legacy SNMP system and are looking for a simple, opensource 
SNMP trap receiver/alerting system. We aren't looking for a full SNMP system, 
just something that will receive snmp traps and email/alert based on them.

1) Looking for something off the shelf, not a development project
2) Opensource or low cost
3) SNMP MIB compiler

Any suggestions?


Matthew Huff             | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC       | Phone: 914-460-4039

LCD KVM console pullout that supports display port ???

[Matthew Huff]

Anyone have any recommendations for a 16-17" LCD keyboard/mouse combo pull-out 
tray that supports DisplayPort/USB as an input?


Matthew Huff | 1 Manhattanville Rd 
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039

Scam telemarketers spoofing our NOC phone number for callerid

[Matthew Huff]

We have recently gotten complaints of harrassing and high pressure sales scams 
orginating from our NOC's phone number. Since the number is a virtual number on 
the PBX, it can't be used for outgoing calls. I assume the scammers choose the 
number from the whois db. Anyone else seen this happening? Any suggestions on 
whom we should contact?



----
Matthew Huff   | One Manhattanville Rd
OTA Management LLC | Purchase, NY 10577
http://www.ox.com  | Phone: 914-460-4039
aim: matthewbhuff  | Fax:   914-460-4139



<>

RE: Scam telemarketers spoofing our NOC phone number for callerid

[Matthew Huff]

Our system is PRI based, not sip.


Matthew Huff   | One Manhattanville Rd
OTA Management LLC | Purchase, NY 10577
http://www.ox.com  | Phone: 914-460-4039
aim: matthewbhuff  | Fax:   914-460-4139



> -Original Message-
> From: wher...@gmail.com [mailto:wher...@gmail.com] On Behalf Of William Herrin
> Sent: Wednesday, October 06, 2010 11:15 AM
> To: Dan White
> Cc: Matthew Huff; (nanog@nanog.org)
> Subject: Re: Scam telemarketers spoofing our NOC phone number for callerid
> 
> On Wed, Oct 6, 2010 at 10:37 AM, Dan White  wrote:
> > If your PBX is SIP based, you might be victim of a SIP registration hijack,
> > which are on the rise, based on traffic we've been seeing in our network.
> 
> I had my unpublished asterisk box up for all of two days before
> getting half a megabit per second worth of false SIP registration
> attempts. Filled /var/log. I had to write a script to dynamically
> filter source IPs with too many failures.
> 
> Regards,
> Bill Herrin
> 
> --
> William D. Herrin  her...@dirtside.com  b...@herrin.us
> 3005 Crane Dr. .. Web: <http://bill.herrin.us/>
> Falls Church, VA 22042-3004
<>

RE: Scam telemarketers spoofing our NOC phone number for callerid

[Matthew Huff]

Digital all the way through. No sip. No outside access to the PBX subnet 
either. Just a mininute ago our telco has verified that the calls are not 
orginating from out phone system. It's a simple caller id spoofing. People 
don't realize that caller id can be spoofed and therefore are 100% sure that we 
are makign the harrasing calls. 

Just wanted nanog to be aware of this since the only two numbers that this has 
happened with are the ones in our ARIN whois records.



----
Matthew Huff   | One Manhattanville Rd
OTA Management LLC | Purchase, NY 10577
http://www.ox.com  | Phone: 914-460-4039
aim: matthewbhuff  | Fax:   914-460-4139



> -Original Message-
> From: Jon Lewis [mailto:jle...@lewis.org]
> Sent: Wednesday, October 06, 2010 11:34 AM
> To: Matthew Huff
> Cc: '(nanog@nanog.org)'
> Subject: RE: Scam telemarketers spoofing our NOC phone number for callerid
> 
> On Wed, 6 Oct 2010, Matthew Huff wrote:
> 
> > Our system is PRI based, not sip.
> 
> PRI for origination and termination...but what are your phones?  Old
> school or VOIP/SIP?  If your phone system supports SIP clients, it really
> ought to be IP restricted to only allow your phones access, or use
> something like fail2ban to stop the SIP scanners from eventually gaining
> access.
> 
> --
>   Jon Lewis, MCP :)   |  I route
>   Senior Network Engineer |  therefore you are
>   Atlantic Net|
> _ http://www.lewis.org/~jlewis/pgp for PGP public key_
<>

RE: Need trusted NTP Sources

[Matthew Huff]

Working in the financial world, the best practices is to have 4 ntp servers (if 
not using PTP).

1) You need 3 to determine the correct time (and detect bad tickers)
2) If you lose 1 of the 3 above, then you no longer can determine the correct 
time
3) Therefore with 4, you have redundancy.

We have two Symmetricom Stratum 1 time servers synced via GPS  with Rubidium 
oscillators,  and two RHEL 6 servers running ntpd for our 4 servers.




Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC       | Phone: 914-460-4039

-Original Message-
From: Roy [mailto:r.engehau...@gmail.com] 
Sent: Friday, February 7, 2014 10:23 AM
To: nanog@nanog.org
Subject: Re: Need trusted NTP Sources

On 2/7/2014 3:35 AM, Saku Ytti wrote:
> On (2014-02-06 21:14 -0500), Jay Ashworth wrote:
>
>> My usual practice is to set up two in house servers, each of which 
>> talks to:
>>
>> And then point everyone in house to both of them, assuming they 
>> accept multiple server names.
> Two is worst possible amount of NTP servers to have. Either one fails 
> and your timing is wrong, because you cannot vote false ticker. And 
> chance of either of two failing is higher than one specific of them.
>

"A man with a watch knows what time it is. A man with two watches is never 
sure."

<>

RE: Requirements for IPv6 Firewalls

[Matthew Huff]

I should have clarified that better. I wouldn't manage a corporate network 
without a centrally managed firewall (stateful; or not). Depending on host 
security alone, especially Windows desktops, isn't something I would care to be 
a part of. Some IPv6 pundits have pushed the idea of re-establishing the norm 
of end-to-end connectivity without regard to anything other than host security. 
This may be fine for educational, residential, and/or public networks, but IMHO 
a really bad idea for corporate networks. Compliance and auditing are only a 
few of the issues.

BTW,

There are a number of reasons for coroporate use of source and destination 
NATing that have nothing to do with the type of security that has been 
discussed.

For example:

1) We have a business partner that requires all access to their data come from 
a small IP range (1-4 addresses). We have a distributed batch system that 
downloads/processes nightly data. We had to implement NAT so that all of our 
machines appear to come from those 4 address. I made it known how easy it was 
to defeat their access security, and how inane it was. I spoke directly to 
their VP of IT, and it didn't matter. If we wanted their data, we had to 
comply. Since no other company provides that data, we had to use NAT. Elegant 
design always loses to practical concerns.

2) We use source and destination nat via our extranet provider (BT-Radianz) 
over private lines. NAT is done for a number of reasons including traffic 
engineering and network information hiding. Most of the partners on the other 
side of the extranet have very tight ACLs. If we were to need to change our 
source IP, it would take a miracle to get it changed on their side short of 3-4 
weeks. That's the world some people live in. 








Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC       | Phone: 914-460-4039

-Original Message-
From: christopher.mor...@gmail.com [mailto:christopher.mor...@gmail.com] On 
Behalf Of Christopher Morrow
Sent: Tuesday, April 22, 2014 3:46 PM
To: Matthew Huff
Cc: Brian Johnson; nanog@nanog.org
Subject: Re: Requirements for IPv6 Firewalls

On Tue, Apr 22, 2014 at 3:41 PM, Matthew Huff  wrote:
> I think some of the disconnect is the difference between a provider network 
> and a corporate one.
>
> For example, www.foo.com if it is highly visible and has a high traffic rate 
> would have  load balancers and line rate routers, but no statefull firewalls.
>
> Corporate foo.com, on the other hand, where end-users, and internal servers 
> reside, almost certainly has a statefull firewall.
>

doesn't this come down to design of the whole system though?

or rather, I bet roland would point out that this comes down to the design of 
the whole system... and tradeoffs folk decide to make/break.

watching a corporate mail server complex melt down because some 'well 
intentioned admin' put a stateful firewall (with a single rule; "permit smtp"!) 
in front of the mail servers ... Having to explain to them (and losing because 
'policy') that 'permit tcp any any eq 25' was more effective and better for 
their systems health was quite painful.

eventually the CIO didn't listen and he works elsewhere.

> Personally, if I were told to use only host based security on a corporate 
> network and no central administrated firewall, I'd be shopping my resume.

why? sure there's a place for things like firewalls, but there's also a fine 
place for just 'drop packets with filters and don't maintain state'.  it really 
comes down to the design goals of the whole system.

-chris

>
>
> 
> Matthew Huff | 1 Manhattanville Rd
> Director of Operations   | Purchase, NY 10577
> OTA Management LLC   | Phone: 914-460-4039
>
> -Original Message-
> From: Christopher Morrow [mailto:morrowc.li...@gmail.com]
> Sent: Tuesday, April 22, 2014 3:18 PM
> To: Brian Johnson
> Cc: nanog@nanog.org
> Subject: Re: Requirements for IPv6 Firewalls
>
> On Tue, Apr 22, 2014 at 2:55 PM, Brian Johnson  wrote:
>> Eric,
>>
>> If you read what he posted and really believe that is what he is saying, you 
>> need to re-think your career decision. It is obvious that he is not saying 
>> that.
>>
>
> Roland's saying basically:
>   1) if you deploy something on 'the internet' you should secure that 
> something
>   2) the securing of that 'thing' should NOT be be placing a stateful device 
> between your users and the 'thing'.
>
> In a simple case of:
>   "Put a web server on the internet"
>
> Roland's advice breaks down to:
>   1) deploy server
>   2) put acl on upstream router like:
>   permit tcp any any eq 80
>  

Cogent / Internap issue ??

[Matthew Huff]

We are having troubles reaching services on the other side of cogent/internap 
peering. Anyone else seeing issues?

Basically 14607 -> 6128 -> 174 -> 14744.

Type escape sequence to abort.
Tracing the route to 72.5.52.199
VRF info: (vrf in name/id, vrf out name/id)
  1 24.157.32.249 [AS 6128] 4 msec 4 msec 4 msec
  2 24.38.117.6 [AS 6128] 4 msec 8 msec 4 msec
  3 167.206.183.29 [AS 6128] 4 msec 4 msec 8 msec
  4  *  *  * 
  5  *  *  * 
  6 154.54.47.17 [AS 174] 4 msec
154.54.47.29 [AS 174] 8 msec
154.54.44.69 [AS 174] 4 msec
  7 66.28.4.202 [AS 174] 28 msec 28 msec
154.54.6.190 [AS 174] 28 msec
  8 154.54.6.85 [AS 174] 40 msec
154.54.24.81 [AS 174] 36 msec
154.54.6.117 [AS 174] 40 msec
  9 154.54.26.113 [AS 174] 52 msec
154.54.26.121 [AS 174] 52 msec
154.54.26.129 [AS 174] 48 msec
 10 154.54.25.70 [AS 174] 64 msec
154.54.25.66 [AS 174] 60 msec
154.54.25.70 [AS 174] 60 msec
 11 154.54.2.197 [AS 174] 92 msec 88 msec 84 msec
 12 154.54.0.249 [AS 174] 80 msec
154.54.0.253 [AS 174] 80 msec
154.54.0.249 [AS 174] 84 msec
 13 154.54.41.142 [AS 174] 152 msec 224 msec 92 msec
 14 38.122.90.42 [AS 174] 84 msec 84 msec
38.104.124.82 [AS 174] 80 msec
 15 63.251.160.18 [AS 14744] 76 msec 76 msec 72 msec
 16  *  *  * 
 17  *  *  * 
 18  *  *  * 
 19  *  *  * 
 20  *  *  * 
 21  *  *  * 
 22  *  *  * 
 23  *  *  * 
 24  *  *  * 
 25  *  *  * 
 26  *  *  * 
 27  *  *  * 
 28  *  *  * 
 29  *  *  * 
 30  *  *  *
----
Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC       | Phone: 914-460-4039

RE: Cogent / Internap issue ??

[Matthew Huff]

Just to clarify, it's not just a traceroute issue. I can't ping nor connect to 
port 80/443 to the ip address 72.5.52.199 from our network, although I can do 
both from an external shell account. It's very possible that it's a routing 
issue on the other side, and I'm reaching out to them as well.



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC       | Phone: 914-460-4039

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Matthew Huff
Sent: Tuesday, May 27, 2014 2:25 PM
To: nanog@nanog.org
Subject: Cogent / Internap issue ??

We are having troubles reaching services on the other side of cogent/internap 
peering. Anyone else seeing issues?

Basically 14607 -> 6128 -> 174 -> 14744.

Type escape sequence to abort.
Tracing the route to 72.5.52.199
VRF info: (vrf in name/id, vrf out name/id)
  1 24.157.32.249 [AS 6128] 4 msec 4 msec 4 msec
  2 24.38.117.6 [AS 6128] 4 msec 8 msec 4 msec
  3 167.206.183.29 [AS 6128] 4 msec 4 msec 8 msec
  4  *  *  * 
  5  *  *  * 
  6 154.54.47.17 [AS 174] 4 msec
154.54.47.29 [AS 174] 8 msec
154.54.44.69 [AS 174] 4 msec
  7 66.28.4.202 [AS 174] 28 msec 28 msec
154.54.6.190 [AS 174] 28 msec
  8 154.54.6.85 [AS 174] 40 msec
154.54.24.81 [AS 174] 36 msec
154.54.6.117 [AS 174] 40 msec
  9 154.54.26.113 [AS 174] 52 msec
154.54.26.121 [AS 174] 52 msec
154.54.26.129 [AS 174] 48 msec
 10 154.54.25.70 [AS 174] 64 msec
154.54.25.66 [AS 174] 60 msec
154.54.25.70 [AS 174] 60 msec
 11 154.54.2.197 [AS 174] 92 msec 88 msec 84 msec
 12 154.54.0.249 [AS 174] 80 msec
154.54.0.253 [AS 174] 80 msec
154.54.0.249 [AS 174] 84 msec
 13 154.54.41.142 [AS 174] 152 msec 224 msec 92 msec
 14 38.122.90.42 [AS 174] 84 msec 84 msec
38.104.124.82 [AS 174] 80 msec
 15 63.251.160.18 [AS 14744] 76 msec 76 msec 72 msec
 16  *  *  * 
 17  *  *  * 
 18  *  *  * 
 19  *  *  * 
 20  *  *  * 
 21  *  *  * 
 22  *  *  * 
 23  *  *  * 
 24  *  *  * 
 25  *  *  * 
 26  *  *  * 
 27  *  *  * 
 28  *  *  * 
 29  *  *  * 
 30  *  *  *

Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC       | Phone: 914-460-4039

RE: Ars Technica on IPv4 exhaustion

[Matthew Huff]

Doesn't surprise me at all. Another thing I've seen lately is number of 
software (especially system management software) after being certified/tested 
with IPv6 no longer function when IPv6 is enabled. At least one vendor that 
broke IPv6 with a recent patch told me they only tested it once for IPv6 
compatibility to get the marketing folks off their neck. After that, they no 
longer test with IPv6 since they don't have IPv6 internally.



-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Lee Howard
Sent: Thursday, June 19, 2014 8:54 AM
To: Frank Bulk; 'Jared Mauch'
Cc: NANOG
Subject: Re: Ars Technica on IPv4 exhaustion



On 6/17/14 11:43 PM, "Frank Bulk"  wrote:

>These sites used to be dual-stacked:
>www.cablelabs.com (over 180 days ago via ipv6.cablelabs.com)
>www.att.net (over 44 days ago)
>www.charter.com (over 151 days)
>www.globalcrossing.com (over 802 days)
>www.timewarnercable.com (over 593 days)

Check that one again.

Surprised you didn't mention www.bing.com.

Lee

RE: Wacky Weekend: NERC to relax power grid frequency strictures

[Matthew Huff]

Another couple of reasons to use a delayed transition ATS:

1) Motor lock. Delays on HVAC equipment never get triggered if the system never 
goes offline. Having a correct "open" period allows the motors to spin down, 
and start back up on the delays that are programmed keeping them from being 
synchronized

2) Allowing transformer fields to collapse. Even in phase, without a delayed 
transition ATS you can end up with a partially collapsed transformer field with 
a new field being created at non-ground state. This can cause a transient back 
wave that can snap circuit breakers. Yep, this one happened to us a few times 
before we switched to a delayed ATS, was a PITA to debug and resolve.


-Original Message-
From: Seth Mattinen [mailto:se...@rollernet.us] 
Sent: Saturday, June 25, 2011 8:49 PM
To: nanog@nanog.org
Subject: Re: Wacky Weekend: NERC to relax power grid frequency strictures

On 6/25/2011 16:43, Paul Graydon wrote:
> On 6/25/2011 12:32 PM, Seth Mattinen wrote:
>>
>> For open and closed transitions you'll most certainly want to sync to
>> utility to transition between the two. For the delayed transition model
>> it'll stop at the intermediate "open" point for a configurable amount of
>> time during which the load is disconnected from everything (i.e. let all
>> the motors spin down first).
>>
>> ~Seth
>>
> Take a guess what the datacenter our equipment is currently hosted in
> uses.  Yet another reason to be glad of a datacenter move that's coming up.
> 

Also depends on the operator, so ask to see their xfer switches and how
they're programmed if that's a concern. All of the non-residential
models in that link for three-phase have motor/load disconnect signaling
capability. If the operator is clued enough to use it then it's all
good: shut off signal to motor/compressor loads, phase sync and switch,
signal reconnect after delay. But if they're not... run away.

Even with the delayed transition models the "hold open" delay can be too
short and end up re-energizing the motors too quickly. There's always
plenty of ways to f*ck things up good.

~Seth

RE: [outages] News item: Blackberry services down worldwide

[Matthew Huff]

It's called Microsoft Exchange ActiveSync :) 

It works with Android, Apple and Microsoft devices. I believe both Lotus and 
Groupwise have licensed and support it as well. We have a few (but now, very 
few) blackberry users remaining. They won't let it go until we rip it out of 
their hands.




> -Original Message-
> From: Jamie Bowden [mailto:ja...@photon.com]
> Sent: Thursday, October 13, 2011 7:36 AM
> To: Joe Abley
> Cc: nanog@nanog.org
> Subject: RE: [outages] News item: Blackberry services down worldwide
> 
> You are correct.  The BES uses PSKs to talk to RIM's servers, which then
> uses them to talk to the devices over the carrier networks.  All of this
> was in complete failure mode until sometime overnight when it appears to
> have all started flowing again.  Someday either Google or Apple will get
> off their rear ends and roll out an end to end encrypted service that
> plugs into corporate email/calendar/workgroup services and we can all
> gladly toss these horrid little devices in the recycle bins where they
> belong.
> 
> Jamie
> 
> > -Original Message-
> > From: Joe Abley [mailto:jab...@hopcount.ca]
> > Sent: Wednesday, October 12, 2011 6:06 PM
> > To: Phil Regnauld
> > Cc: nanog@nanog.org
> > Subject: Re: [outages] News item: Blackberry services down worldwide
> >
> >
> > On 2011-10-12, at 18:02, Phil Regnauld wrote:
> >
> > > Joe Abley (jabley) writes:
> > >>
> > >> On 2011-10-12, at 13:05, Leigh Porter wrote:
> > >>
> > >>> Email on my iPhone is working fine.. ;-)
> > >>
> > >> The blackberry message service is centralised with a lot of
> > processing intelligence in the core. Messaging services that use the
> > core as a simple transport and shift the processing intelligence to
> the
> > edge have different, less-dramatic failure modes.
> > >
> > >   This is not the case for corporate customers with dedicated
> > servers,
> > >   AFAIU.
> >
> > I'm no expert, but my understanding is that at some/most/all traffic
> > between handhelds and a BES, carried from the handheld device through
> a
> > cellular network, still flows through RIM.
> >
> >
> > Joe

RE: Encrypted RPC and firewalling

[Matthew Huff]

Also,

Most enterprises that support Exchange remote access use RPC over HTTPS which 
is encrypted and easy to allow on the firewall.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-460-4139


> -Original Message-
> From: valdis.kletni...@vt.edu [mailto:valdis.kletni...@vt.edu]
> Sent: Thursday, November 10, 2011 7:51 AM
> To: Lasse Birnbaum Jensen
> Cc: nanog@nanog.org
> Subject: Re: Encrypted RPC and firewalling
> 
> On Thu, 10 Nov 2011 09:56:51 +0100, Lasse Birnbaum Jensen said:
> > I would like to know how you guys handle encypted rpc across
> firewalls.
> 
> You can always just set the firewall to ban RPC in general, whether or
> not it's encrypted (while you're there, close off ports 137-139 and
> other chucklehead stuff like that), and just make the user who's
> outside the firewall VPN in.  That's a nice, simple, well-understood
> configuration that almost all software and even most users can handle.
> 
> (We don't actually do a big monolithic firewall box - but pretty much
> everything has an iptables ruleset loaded that says "if your source IP
> isn't inside our 2 /16s, your packets go bye bye".  And there's a nice
> PPTP-based VPN solution in place that even a humanities professor
> emeritus can use ;)

Cogent outage?

[Matthew Huff]

About 10 minutes ago we stopped being able to pass traffic through cogent. I 
de-peered us from Cogent, and everything appears
better. When I call cogent, all I get is a busy signal (must be a major 
outage). Anyone else seeing anything?


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-460-4139




smime.p7s
Description: S/MIME cryptographic signature

RE: Cogent outage?

[Matthew Huff]

We are peered in Westchester Co, NY (north of NYC). Reports from 
status.cogentco.com suggest a problem in NYC. I wonder if it's
related to the 75 Broad Street explosion this morning. According to Cogent 
status, they are running on generator. 


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-460-4139

> -Original Message-
> From: Michael Proto [mailto:m...@jellydonut.org]
> Sent: Thursday, December 06, 2012 12:31 PM
> To: Matthew Huff
> Cc: nanog@nanog.org
> Subject: Re: Cogent outage?
> 
> I'm seeing packet loss between my Atlanta Cogent connection and some
> servers we have in both Dallas and London. According to Cogent's
> status page they're having an outage in the NYC area.
> 
> 
> -Proto
> 
> http://status.cogentco.com/
> 
> On Thu, Dec 6, 2012 at 12:11 PM, Matthew Huff  wrote:
> > About 10 minutes ago we stopped being able to pass traffic through cogent. 
> > I de-peered us from
> Cogent, and everything appears
> > better. When I call cogent, all I get is a busy signal (must be a major 
> > outage). Anyone else seeing
> anything?
> >
> > 
> > Matthew Huff | 1 Manhattanville Rd
> > Director of Operations   | Purchase, NY 10577
> > OTA Management LLC   | Phone: 914-460-4039
> > aim: matthewbhuff| Fax:   914-460-4139
> >
> >


smime.p7s
Description: S/MIME cryptographic signature

Re: Inaccessible network from Verizon, accessible elsewhere.

[Matthew Huff]

I'm seeing the same thing from my home lan via fios. I've run a recursive dns 
server for years and can't reach the roots. Had to switch to using verizon's 
dns servers as forwarders. 

Sent from my iPad

On Dec 11, 2011, at 8:07 PM, "Brandon Kim"  wrote:

> 
> I too am now experiencing issues. I cannot get to www.cisco.com and various 
> websites.
> Some websites work lightning quick, some take a long time to load, and some 
> just don't load at all.
> 
> 
> 
> 
>> Date: Mon, 12 Dec 2011 09:55:40 +0900
>> From: ra...@psg.com
>> To: nanog@nanog.org
>> Subject: Re: Inaccessible network from Verizon, accessible elsewhere.
>> 
>> from home lan
>> 
>> % traceroute gw-li377.linode.com
>> traceroute to gw-li377.linode.com (106.187.34.1), 64 hops max, 52 byte 
>> packets
>> 1  192.168.0.1 (192.168.0.1)  1.471 ms  0.725 ms  0.555 ms
>> 2  tokyo10-f03.flets.2iij.net (210.149.34.72)  7.241 ms  6.651 ms  6.939 ms
>> 3  tokyo10-ntteast0.flets.2iij.net (210.149.34.157)  5.573 ms  6.109 ms  
>> 5.346 ms
>> 4  tky001lip20.iij.net (210.149.34.97)  6.410 ms  7.471 ms  7.934 ms
>> 5  tky001bb10.iij.net (58.138.100.209)  6.670 ms  9.251 ms  5.866 ms
>> 6  tky009bf00.iij.net (58.138.80.17)  6.730 ms
>>tky008bf02.iij.net (58.138.80.13)  7.021 ms
>>tky009bf00.iij.net (58.138.80.17)  8.593 ms
>> 7  tky001ix05.iij.net (58.138.82.2)  9.767 ms
>>tky001ix05.iij.net (58.138.82.6)  6.101 ms
>>tky001ix01.iij.net (58.138.80.106)  8.420 ms
>> 8  203.181.102.61 (203.181.102.61)  19.514 ms
>>203.181.102.21 (203.181.102.21)  6.054 ms
>>203.181.102.61 (203.181.102.61)  11.478 ms
>> 9  otejbb203.kddnet.ad.jp (118.155.197.129)  7.457 ms
>>otejbb203.kddnet.ad.jp (59.128.7.129)  7.835 ms
>>otejbb204.kddnet.ad.jp (59.128.7.130)  7.824 ms
>> 10  cm-fcu203.kddnet.ad.jp (124.215.194.180)  15.860 ms  16.401 ms
>>cm-fcu203.kddnet.ad.jp (124.215.194.164)  17.519 ms
>> 11  124.215.199.122 (124.215.199.122)  7.892 ms *  11.984 ms
>> 
> 

Re: Inaccessible network from Verizon, accessible elsewhere.

[Matthew Huff]

Consumer fios. Verizon forums are full of posts about it. Too tired this 
evening to worry about it.

Sent from my iPad

On Dec 11, 2011, at 10:48 PM, "Christopher Morrow"  
wrote:

> On Sun, Dec 11, 2011 at 10:28 PM, Matthew Huff  wrote:
>> I'm seeing the same thing from my home lan via fios. I've run a recursive 
>> dns server for years and can't reach the roots. Had to switch to using 
>> verizon's dns servers as forwarders.
>> 
> 
> business or consumer fios?
> 3  G0-9-4-7.WASHDC-LCR-22.verizon-gni.net (130.81.104.180)  6.662 ms
> 6.739 ms  6.788 ms
> 4  so-14-0-0-0.RES-BB-RTR2.verizon-gni.net (130.81.22.56)  6.852 ms
> 15.384 ms  8.184 ms
> 5  0.ae2.BR1.IAD8.ALTER.NET (152.63.32.158)  12.857 ms  12.927 ms  13.004 ms
> 6  dcp-brdr-03.inet.qwest.net (63.146.26.105)  12.429 ms  7.847 ms  6.464 ms
> 7  lap-brdr-03.inet.qwest.net (67.14.22.78)  89.140 ms  88.929 ms  89.032 ms
> 8  63.146.26.70 (63.146.26.70)  94.879 ms  94.580 ms  93.120 ms
> 9  sl-crs1-kc-0-0-0-2.sprintlink.net (144.232.18.112)  58.520 ms
> 58.330 ms  58.186 ms
> 10  144.232.25.193 (144.232.25.193)  49.950 ms
> sl-crs1-oma-0-9-2-0.sprintlink.net (144.232.2.177)  49.962 ms
> sl-crs1-oma-0-8-0-0.sprintlink.net (144.232.8.171)  47.687 ms
> 11  sl-crs1-oro-0-3-3-0.sprintlink.net (144.232.25.207)  84.416 ms
> 83.266 ms sl-crs1-oro-0-12-3-0.sprintlink.net (144.232.25.73)  84.667
> ms
> 12  124.215.199.122 (124.215.199.122)  195.590 ms * *
> 
> all of this seems to point at some kddi.net rouer gobbling packets,
> no? (since pretty much everyone's got the same terminating hop) - also
> note that while some folks traverse L3, my route is via qwest...
> 
> it's interesting that 701 isn't picking their other peer (sprint) here
> directly, no?
> 
>> Sent from my iPad
>> 
>> On Dec 11, 2011, at 8:07 PM, "Brandon Kim"  
>> wrote:
>> 
>>> 
>>> I too am now experiencing issues. I cannot get to www.cisco.com and various 
>>> websites.
>>> Some websites work lightning quick, some take a long time to load, and some 
>>> just don't load at all.
>>> 
>>> 
>>> 
>>> 
>>>> Date: Mon, 12 Dec 2011 09:55:40 +0900
>>>> From: ra...@psg.com
>>>> To: nanog@nanog.org
>>>> Subject: Re: Inaccessible network from Verizon, accessible elsewhere.
>>>> 
>>>> from home lan
>>>> 
>>>> % traceroute gw-li377.linode.com
>>>> traceroute to gw-li377.linode.com (106.187.34.1), 64 hops max, 52 byte 
>>>> packets
>>>> 1  192.168.0.1 (192.168.0.1)  1.471 ms  0.725 ms  0.555 ms
>>>> 2  tokyo10-f03.flets.2iij.net (210.149.34.72)  7.241 ms  6.651 ms  6.939 ms
>>>> 3  tokyo10-ntteast0.flets.2iij.net (210.149.34.157)  5.573 ms  6.109 ms  
>>>> 5.346 ms
>>>> 4  tky001lip20.iij.net (210.149.34.97)  6.410 ms  7.471 ms  7.934 ms
>>>> 5  tky001bb10.iij.net (58.138.100.209)  6.670 ms  9.251 ms  5.866 ms
>>>> 6  tky009bf00.iij.net (58.138.80.17)  6.730 ms
>>>>tky008bf02.iij.net (58.138.80.13)  7.021 ms
>>>>tky009bf00.iij.net (58.138.80.17)  8.593 ms
>>>> 7  tky001ix05.iij.net (58.138.82.2)  9.767 ms
>>>>tky001ix05.iij.net (58.138.82.6)  6.101 ms
>>>>tky001ix01.iij.net (58.138.80.106)  8.420 ms
>>>> 8  203.181.102.61 (203.181.102.61)  19.514 ms
>>>>203.181.102.21 (203.181.102.21)  6.054 ms
>>>>203.181.102.61 (203.181.102.61)  11.478 ms
>>>> 9  otejbb203.kddnet.ad.jp (118.155.197.129)  7.457 ms
>>>>otejbb203.kddnet.ad.jp (59.128.7.129)  7.835 ms
>>>>otejbb204.kddnet.ad.jp (59.128.7.130)  7.824 ms
>>>> 10  cm-fcu203.kddnet.ad.jp (124.215.194.180)  15.860 ms  16.401 ms
>>>>cm-fcu203.kddnet.ad.jp (124.215.194.164)  17.519 ms
>>>> 11  124.215.199.122 (124.215.199.122)  7.892 ms *  11.984 ms
>>>> 
>>> 
>> 

RE: Inaccessible network from Verizon, accessible elsewhere.

[Matthew Huff]

DSLReports Verizon forum reports routing issues in Westchester, Rockland and 
Nassau. I tried a few traceroutes this morning. Some went through fine, others 
died at the first hop within Verizon.

People are reporting mixed results calling Verizon. Some techs are saying it's 
a known issues, others are going through the standard script (reboot router, 
reboot ONT, check settings on browser, i.e. clueless, even to the point of 
saying that the person's router is bad and they would send them a new one).



----
Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-460-4139

> -Original Message-
> From: Adam Greene [mailto:maill...@webjogger.net]
> Sent: Monday, December 12, 2011 1:27 AM
> To: nanog@nanog.org
> Subject: Re: Inaccessible network from Verizon, accessible elsewhere.
> 
> We're having strange issues in NYC metropolitan area.
> 
> We can trace from Verizon FIOS to some IP addresses of our ASN 11579
> block. Others don't work. The IP's that don't work seem to die at
> 130.81.107.228 on the Verizon network.
> 
> Something is rotten in Denmark. Or NY. You know what I mean.
> 
> On 12/12/2011 1:02 AM, Christopher Morrow wrote:
> > On Sun, Dec 11, 2011 at 10:54 PM, Matthew Huff  wrote:
> >> Consumer fios. Verizon forums are full of posts about it. Too tired
> this evening to worry about it.
> > :( I'll have to do some testing when I get near a consumer fios
> > then... So, they squash all DNS NOT to their complexes, that seems
> > rather dastardly of them... considering they deployed that hateful
> > paxfire/nominum garbage on their recursive servers :(
> >
> > -chris
> >
> >> On Dec 11, 2011, at 10:48 PM, "Christopher
> Morrow"  wrote:
> >>
> >>> On Sun, Dec 11, 2011 at 10:28 PM, Matthew Huff
> wrote:
> >>>> I'm seeing the same thing from my home lan via fios. I've run a
> recursive dns server for years and can't reach the roots. Had to switch
> to using verizon's dns servers as forwarders.
> >>>>
> >>> business or consumer fios?
> >>> 3  G0-9-4-7.WASHDC-LCR-22.verizon-gni.net (130.81.104.180)  6.662
> ms
> >>> 6.739 ms  6.788 ms
> >>> 4  so-14-0-0-0.RES-BB-RTR2.verizon-gni.net (130.81.22.56)  6.852 ms
> >>> 15.384 ms  8.184 ms
> >>> 5  0.ae2.BR1.IAD8.ALTER.NET (152.63.32.158)  12.857 ms  12.927 ms
> >>> 13.004 ms
> >>> 6  dcp-brdr-03.inet.qwest.net (63.146.26.105)  12.429 ms  7.847 ms
> >>> 6.464 ms
> >>> 7  lap-brdr-03.inet.qwest.net (67.14.22.78)  89.140 ms  88.929 ms
> >>> 89.032 ms
> >>> 8  63.146.26.70 (63.146.26.70)  94.879 ms  94.580 ms  93.120 ms
> >>> 9  sl-crs1-kc-0-0-0-2.sprintlink.net (144.232.18.112)  58.520 ms
> >>> 58.330 ms  58.186 ms
> >>> 10  144.232.25.193 (144.232.25.193)  49.950 ms
> >>> sl-crs1-oma-0-9-2-0.sprintlink.net (144.232.2.177)  49.962 ms
> >>> sl-crs1-oma-0-8-0-0.sprintlink.net (144.232.8.171)  47.687 ms
> >>> 11  sl-crs1-oro-0-3-3-0.sprintlink.net (144.232.25.207)  84.416 ms
> >>> 83.266 ms sl-crs1-oro-0-12-3-0.sprintlink.net (144.232.25.73)
> >>> 84.667 ms
> >>> 12  124.215.199.122 (124.215.199.122)  195.590 ms * *
> >>>
> >>> all of this seems to point at some kddi.net rouer gobbling packets,
> >>> no? (since pretty much everyone's got the same terminating hop) -
> >>> also note that while some folks traverse L3, my route is via
> qwest...
> >>>
> >>> it's interesting that 701 isn't picking their other peer (sprint)
> >>> here directly, no?
> >>>
> >>>> Sent from my iPad
> >>>>
> >>>> On Dec 11, 2011, at 8:07 PM, "Brandon
> Kim"  wrote:
> >>>>
> >>>>> I too am now experiencing issues. I cannot get to www.cisco.com
> and various websites.
> >>>>> Some websites work lightning quick, some take a long time to
> load, and some just don't load at all.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>> Date: Mon, 12 Dec 2011 09:55:40 +0900
> >>>>>> From: ra...@psg.com
> >>>>>> To: nanog@nanog.org
> >>>>>> Subject: Re: Inaccessible network from Verizon, accessible
> elsewhere.
> >>>>>>
> >>>>>> from home lan
> >>>>

RE: SSL Certificates

[Matthew Huff]

I've had good experience with Entrust. One thing to be careful with is some 
mobile devices (especially older Android ones) have limited root certificates. 
Network Solutions and Entrust work, some others, not so much. From my 
experience Android 2.3+ has most of the common root certs, but previous 
versions don't.


I wonder if someone has a list comparing root certificate support across 
platforms?

----
Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-460-4139


> -Original Message-
> From: Michael Carey [mailto:mca...@kinber.org]
> Sent: Friday, January 06, 2012 9:15 AM
> To: nanog@nanog.org
> Subject: SSL Certificates
> 
> Looking for a recommendation on who to buy affordable and reputable SSL
> certificates from?  Symantec, Thawte, and Comodo are the names that
> come to mind, just wondering if there are others folks use.
> 
> Thanks,
> 
> --
> Michael D. Carey
> KINBER Network Engineer
> mca...@kinber.org
> M: 814.777.5027
> GV: (814) 205-6773 <https://www.google.com/voice#phones>
> Skype: KINBER.Mike.Carey
> 
> KINBER - Keystone Initiative for Network Based Education and Research -
> www.kinber.org PennREN - Pennsylvania's Research and Education Network

RE: Choice of address for IPv6 default gateway

[Matthew Huff]

I've had good luck in a corporate environment using fe80::1 on Cisco 6500/7600 
with newer IOS. However, some software routers still won't let you use a 
link-local as a VIP (at least in HSRP). I'm upgrading one of our 7200 tonight 
running 15.1(4)M1 to M3, hopefully that will fix it (we are upgrading it for 
other reasons).

For example:

int vlan110
 standby 110 ipv6 FE80::1
 standby 110 timers msec 250 msec 750
 standby 110 priority 110
 standby 110 preempt delay minimum 180

----
Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-460-4139


> -Original Message-
> From: Daniel STICKNEY [mailto:dstick...@optilian.com]
> Sent: Wednesday, January 25, 2012 9:42 AM
> To: nanog@nanog.org
> Subject: Choice of address for IPv6 default gateway
> 
> I'm having trouble finding authoritative sources on the best common
> practice (if there even is one) for the choice of address for an IPv6
> default gateway in a production server environment (not desktops). For
> example in IPv4 it is common to chose the first or last address in the
> subnet (.1 or .254 for example) as the VIP for VRRP/HSRP. I'm
> interested in input from production environments and or
> ARIN/RIPE/IANA/etc or top vendors.
> 
> I've seen some documentation using ::1 with either a global
> prefix or link-local (fe80::1). Anyone use either of these in
> production and have negative or positive feedback? fe80::1 is seductive
> because it is short and the idea of having the same default gateway
> configured everywhere might be simple. At the same time using the same
> address all around the network seems to invite confusion or problems if
> two interfaces with the address ever ended up in the same broadcast
> domain.
> 
> What about using RAs to install the default route on the servers? The
> 'priority' option (high/medium/low) easy fits with an architecture
> using an active/standby router setup where the active router is
> configured with the 'high' priority and the standby 'medium'. With the
> timeout values tuned for relatively rapid (~3 seconds)  failover this
> might be feasible. Anyone use this in production?
> 
> I note that VRRPv3 (and keepalived) and HSRP both support IPv6. Since
> we use VRRP for IPv4, using it for IPv6 would keep our architecture the
> same, which has merit too.
> 
> Thanks in advance,
> 
> Daniel STICKNEY
> 

RE: XBOX 720: possible digital download mass service.

[Matthew Huff]

>From what I've read, the XBOX 720 is still going to have traditional 
>distribution but also including online purchasing (think Steam). The goal is 
>to go with a key system to play the game. I think the idea you will be able to 
>register the game via phone, or other means as well. However, their idea is to 
>rid the world of the secondary market of used games.



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-460-4139


> -Original Message-
> From: Eric Tykwinski [mailto:e...@truenet.com]
> Sent: Friday, January 27, 2012 9:25 AM
> To: nanog
> Subject: RE: XBOX 720: possible digital download mass service.
> 
> That's the case, but yeah, definitely off-topic...
> http://www.gamestop.com/ps-vita/games/uncharted-golden-abyss-ps-
> vita/91436
> 
> Which would be on-topic, though.  If anyone knows of an OnLive box just
> to check out the bandwidth usage, I would be interested.
> 
> Sincerely,
> 
> Eric Tykwinski
> TrueNet, Inc.
> P: 610-429-8300
> F: 610-429-3222
> 
> 
> -Original Message-
> From: -Hammer- [mailto:bhmc...@gmail.com]
> Sent: Friday, January 27, 2012 9:21 AM
> To: nanog@nanog.org
> Subject: Re: XBOX 720: possible digital download mass service.
> 
> Now we are venturing OT but I thought the format was proprietary but
> you still had to get the content on the memory via the glorious
> Internet?
> Are you saying I can go to Gamestop and buy a stick with whatever game
> I'm looking for? Is that the plan?
> 
> -Hammer-
> 
> "I was a normal American nerd"
> -Jack Herer
> 
> 
> 
> On 1/27/2012 8:13 AM, Eric Tykwinski wrote:
> > The PS Vita still uses a proprietary memory card format, so it's not
> > just download only.
> > The best example of download only would be OnLive, which basically is
> > a game system that only delivers on demand games.
> >
> > IMHO, it's the market that will determine whether this is the right
> > choice in the long run.
> > It's a creative way to eliminate the used market and stop piracy, but
> > if the consumers don't join up like the PSP Go, it will eventually
> fail.
> >
> > Sincerely,
> >
> > Eric Tykwinski
> > TrueNet, Inc.
> > P: 610-429-8300
> > F: 610-429-3222
> >
> > -Original Message-
> > From: -Hammer- [mailto:bhmc...@gmail.com]
> > Sent: Friday, January 27, 2012 9:02 AM
> > To: nanog@nanog.org
> > Subject: Re: XBOX 720: possible digital download mass service.
> >
> > Here's your baseline: Sony Vita. They already tossed the UMD out with
> > the PSP-GO and that failed miserably. Now they are trying again to go
> > to digital only with the Vita. It's not the scale of PS3 or XBOX360
> > but it may be a good way to gauge the potential success of the
> concept.
> >
> > -Hammer-
> >
> > "I was a normal American nerd"
> > -Jack Herer
> >
> >
> >
> > On 1/27/2012 7:34 AM, Jared Mauch wrote:
> >> It's already done on a similar scale when apple releases new
> software
> >> for
> > their mobile devices.
> >> Just don't do it if you are on a low cap plan (eg: mobile, satellite
> etc).
> > Caps will be the new market discriminator IMHO.
> >> Jared Mauch
> >>
> >> On Jan 27, 2012, at 3:35 AM, Tei   wrote:
> >>
> >>> Can internet in USA support that?   Call of Duty 15 releases may
> 2014
> >>> and 30 million gamers start downloading a 20 GB files.  Would the
> >>> internet collapse like a house of cards?.
> >
> >
> >
> 
> 
> 

RE: Console Server Recommendation

[Matthew Huff]

We use MRV, and are very happy with them:

http://www.mrv.com/oobn/console-servers/




Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-460-4139


> -Original Message-
> From: Ray Soucy [mailto:r...@maine.edu]
> Sent: Monday, January 30, 2012 11:09 AM
> To: NANOG
> Subject: Console Server Recommendation
> 
> What are people using for console servers these days?  We've
> historically used retired routers with ASYNC ports, but it's time for
> an upgrade.
> 
> OpenGear seems to have some nice stuff, anyone else?
> 
> --
> Ray Soucy
> 
> Epic Communications Specialist
> 
> Phone: +1 (207) 561-3526
> 
> Networkmaine, a Unit of the University of Maine System
> http://www.networkmaine.net/

Increase of DOS attacks using TCP src and/or dst of 0

[Matthew Huff]

Anyone else see a massive increase of scanning/dos with TCP source and/or
dst port of 0? We started seeing a massive increase today creating some
issue with our firewalls.

 

 

 



Matthew Huff | 1 Manhattanville Rd

Director of Operations   | Purchase, NY 10577

OTA Management LLC   | Phone: 914-460-4039

aim: matthewbhuff| Fax:   914-460-4139

 



smime.p7s
Description: S/MIME cryptographic signature

Request to lease IP space, or things that make you want to go hmmmmm..

[Matthew Huff]

Just got an email today to our account associated with our legacy ARIN address 
space. A firm "Precision Management of Texas" is interested in subleasing some 
of our IP space for "on-demand solutions for brand marketers and website 
promotion chiefly through email marketing". 

The one thing clear within the large amount of marketing-speach is they want 
"As is the nature of this business PM seeks to obtain as much diversity in the 
allocated IP space as possible, however the most important thing is the Subnets 
need to have no abuse history."

Anyone else get solicited?

They seem to be flexible "We can take the IPs via GRE or BGP or other such 
tunneling solution to where you have them announced. Alternatively we can 
advertise them ourselves on our network, saving you the back-haul. As a third 
solution we can take a server on your network with the following specs:..."


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-460-4139

Re: Request to lease IP space, or things that make you want to go hmmmmm..

[Matthew Huff]

Of course, we declined. I just thought it was worth posting so others might be 
alerted that this was going on.

Hadn't known about the google page ranking SEO, but it makes sense

On Mar 8, 2012, at 8:06 PM, "George Michaelson"  wrote:

> 
> no. you misunderstand.
> 
> The value proposition is not spam: that works with unallocated space.
> 
> The value proposition is gaming google page rank, by using widely spread and 
> legitimately routed IPs to force your paying customers page rank high, by 
> hits and references. This is a very high value business: one customer paying 
> you big bucks, to have their web high in google pagerank. Not attacking a 
> million mailboxes.
> 
> In this model, the 'target' is google. The IPS need to come from classic, 
> widespread IPs because google now count the source IP and can tell if you use 
> a virtually hosted single IP to try and do this.
> 
> I have a question: are we actually able to state this consumption of address 
> is 'illegal' ? I personally judge it to be unethical, but that is not the 
> same thing.
> 
> -George
> 
> PS since this goes to address policy, I need to declare that I work for an 
> RIR but I am posting in a personal capacity and nothing I say is a reflection 
> of any RIR address policy. I work in the research department, not in 
> registry/allocations

RE: Automatic IPv6 due to broadcast

[Matthew Huff]

To completely disable ipv6 in Redhat:

1) Modify /etc/modprobe.conf (add)
  
  alias ipv6 off
  alias net-pf-10 off
  options ipv6 disable=1

2) Modify /etc/sysconfig/network (add)

   NETWORKING_IPV6=no

   I usually also add

   NOZEROCONF=yes 


That should completely disable ipv6 in Redhat 5.x



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-460-4139


> -Original Message-
> From: Anurag Bhatia [mailto:m...@anuragbhatia.com]
> Sent: Monday, April 16, 2012 2:10 PM
> To: NANOG Mailing List
> Subject: Automatic IPv6 due to broadcast
> 
> Hello everyone
> 
> 
> 
> Just got a awfully crazy issue. I heard from our support team about
> failure of whois during domain registration. Initially I thought of
> port 43 TCP block or something but found it was all ok. Later when ran
> whois manually on server via terminal it failed. Found problem that
> server was connecting to whois server - whois.verisign-grs.com. I was
> stunned! Server got IPv6 and not just that one - almost all. This was
> scary - partial IPv6 setup and it was breaking things.
> 
> In routing tables, routes were all going to a router which I recently
> setup for testing. That router and other servers are under same switch
> but by no means I ever configured that router as default gateway for
> IPv6. I found option of "broadcast" was enabled on router for local
> fe80... address and I guess router broadcasted IPv6 and somehow (??)
> all servers found that they have a IPv6 router on LAN and started using
> it - automated DHCP IPv6?
> 
> I wonder if anyone else also had similar issues? Also, if my guesses
> are correct then how can we disable Red Hat distro oriented servers
> from taking such automated configuration - simple DHCP in IPv6 disable?
> 
> 
> 
> 
> Thanks
> 
> --
> 
> Anurag Bhatia
> anuragbhatia.com
> or simply - http://[2001:470:26:78f::5] if you are on IPv6 connected
> network!
> 
> Twitter: @anurag_bhatia <https://twitter.com/#!/anurag_bhatia>
> Linkedin: http://linkedin.anuragbhatia.com


smime.p7s
Description: S/MIME cryptographic signature

RE: IPv6 day and tunnels

[Matthew Huff]

>> An L2 device should not be fragmenting L3 packets.

Layer 2 fragmentation used (20+ years ago) to be a common thing with bridged 
topologies like token-ring to Ethernet source-routing. Obviously, no so much 
anymore (at least I hope not), but it can and does happen.

I think part of the problem is that ISPs, CDN, hosting companies, etc. have 
assumed IPv6 is just IPv4 with longer addresses and haven't spent the time 
learning the differences like what was pointed out that ICMPv6 is a required 
protocol for IPv6 to work correctly. MTU issues are an annoyance with IPv4 but 
are a brokenness with IPv6. Knowledge with come, but it may take a bit of 
beating over the head for a while.

RE: LinkedIn password database compromised

[Matthew Huff]

True,

Back in 1998-1999 timeline, there was an ongoing project to have the US
Postal service issue X.509 certificates at a nominal fee. The fact that even
the most rural areas have access to a post office made a lot of sense. After
the 2000 election, the project was cancelled because "private business" can
handle it better.



----
Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-460-4139


> -Original Message-
> From: jeff murphy [mailto:jcmur...@jeffmurphy.org]
> Sent: Thursday, June 07, 2012 10:06 AM
> To: Nanog
> Subject: Re: LinkedIn password database compromised
> 
> 
> On Jun 7, 2012, at 9:58 AM, Leo Bicknell wrote:
> 
> > In a message written on Wed, Jun 06, 2012 at 11:14:58PM -0700, Aaron
> C. de Bruyn wrote:
> >> Heck no to X.509.  We'd run into the same issue we have right now--a
> >> select group of companies charging users to prove their identity.
> >
>   ...
> > For instance, I'm not at all opposed to the idea of the government
> > having a way to issue me a signed certificate that I then use to
> > access government services, like submitting my tax return online,
> > renewing my drivers license, or maybe even e-voting.
> 
> 
> 
> All in favor of paying $119/year to vote, please raise your hands.
> 
> http://www.verisign.com/dod-interoperability/



smime.p7s
Description: S/MIME cryptographic signature

RE: Cisco 6509 SUP32 SNMP Meltdown With CatOS

[Matthew Huff]

By any chance were you querying a Sup32 that had BGP full routes? That and 
other large tables can easily swamp the cpu on the Sup32.

This technote is based on IOS, and I don't know if the same facilities exist in 
CatOS, but as Nick mentioned, run, don't walk and convert to IOS. CatOS is dead.

http://www.cisco.com/en/US/tech/tk648/tk362/technologies_tech_note09186a00800948e6.shtml


-Original Message-
From: david peahi [mailto:davidpe...@gmail.com] 
Sent: Friday, November 02, 2012 2:37 PM
To: nanog@nanog.org
Subject: Cisco 6509 SUP32 SNMP Meltdown With CatOS

Anyone have experience with Cisco 6509E/SUP32 crashing under heavy SNMP
polling load, causing high cpu utilization and 6509 lockup, requiring 6509
reboot? CatOS is deployed. Is the behavior any different with 6509 IOS?

David

NJ Data center equipment movers

[Matthew Huff]

I'm looking to have some equipment (2 x HP C7000 blade chassis ( each with 16 
blades), 2 x Cisco 7600, and some small misc equipment) from a datacenter in 
Mahwah, NJ to Secaucus, NJ. Anyone recommend someone?

Prefix withdrawals in Europe/Russia

[Matthew Huff]

BGPMon has been sending out alerts this morning starting around 15:14 UTC about 
our 129.77.0.0/16 prefix. None of our BGP peers have flapped, and according to 
the alert, it appears limited to:

Netherlands
Sweden
Kuwait
Italy
United Kingdom
Russia
Liechtenstein

I haven't seen anything on nanog or outages list, anyone know what's going on?

RE: Incident notification

[Matthew Huff]

The advantage of SMS is that it is out of band. Any smtp or other IP based 
solution requires a stable and working network environment, which is what the 
alert may be trying to tell you is down.



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Thijs Stuurman
Sent: Friday, November 21, 2014 10:52 AM
To: nanog@nanog.org
Subject: Incident notification

Nanog list members,

I was looking at some statistic and noticed we are sending out a massive amount 
of SMS messages from our monitoring systems.
This left me wondering if there isn't a better (and cheaper) alternative to 
this, something just as reliant but IP based. We all have smartphones these 
days anyway.

Therefore my question, what are you using to notify admins of incidents?

Kind regards / Met vriendelijke groet,

Thijs Stuurman



[IS Logo]




IS Group

Wielingenstraat 8

T

+31 (0)299 476 185

i...@is.nl<mailto:i...@is.nl>

1441 ZR Purmerend

F

+31 (0)299 476 288

www.is.nl<http://www.is.nl>



IS Group is ISO 9001:2008, ISO/IEC 27001:2005, ISO 20.000-1:2005, ISAE 3402 
certified. De datacenters zijn PCI DSS en ISO 14001 compliant.

RE: Cisco AnyConnect speed woes!

[Matthew Huff]

Are you using SSLVpn or IPSEC with anyconnect? I have had more luck with 
performance with IPSEC than SSLVpn.

Also, just because your ISP is saying that they aren't shaping/filtering, 
doesn't mean they aren't.

We had major issues with users using AnyConnect when it was transversing 
Cogent. We were getting 5-10% packet loss (although the Cisco stats didn't show 
it), and it was choking on it.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Zachary McGibbon
Sent: Tuesday, December 9, 2014 2:42 PM
To: NANOG
Subject: Cisco AnyConnect speed woes!

I'm looking for some input on a situation that has been plaguing our new
AnyConnect VPN setup.  Any input would be valuable, we are at a loss for
what the problem is.

We recently upgraded our VPN from our old Cisco 3000 VPN concentrators
running PPTP and we are now running a pair of Cisco 5545x ASAs in an HA
active/standby pair.

The big issue we are having is that many of our users are complaining of
low speed when connected to the VPN.  We have done tons of troubleshooting
with Cisco TAC and we still haven't found the root of our problem.

Some tests we have done:

   - We have tested changing MTU values
   - We have tried all combinations of encryption methods (SSL, TLS, IPSec,
   L2TP) with similar results
   - We have switched our active/standby boxes
   - We have tested on our spare 5545x box
   - We connected our spare box directly to our ISP with another IP address
   - We have whitelisted our VPN IP on our shaper (Cisco SCE8000) and our
   IPS (HP Tipping Point)
   - We have bypassed our Shaper and our IPS
   - We made sure that traffic from the routers talking to our ASAs is
   synchronous, OSPF was configured to load balance but this has been changed
   by changing the costs on the links to the ASAs
   - We have verified with our two ISPs that they are not doing any kind of
   filtering or shaping
   - We have noticed that in some instances that if a user is on a low
   speed connection that their VPN speed gets cut by about 1/3.  This doesn't
   seem normal that the VPN would use this much overhead
   - We do not have the issue when connecting to VPN directly on our own
   network, only connections from the Internet

If you have any ideas on what we could try net, please let me know!

- Zachary

RE: OT - Verizon/ATT Cell/4G Signal Booster/Repeater

[Matthew Huff]

Be careful about the new rules that were put into place in the spring. My 
experience is that resellers are still promoting "consumer" devices for use in 
commercial buildings which is now a no-no. Under the new regulation, consumer 
devices are to be used only for individuals in their home, car, RV, boat, etc..

Industrial signal boosters are the only allowed non-grandfathered devices to be 
used in buildings. They have to be installed by certified installers and 
require a FCC license under the new regulations. The new fines are steep at 
$100,000 an instance, so the wireless providers really have a hold of the FCC.


----
Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of John Schiel
Sent: Tuesday, December 16, 2014 1:28 PM
To: nanog@nanog.org
Subject: Re: OT - Verizon/ATT Cell/4G Signal Booster/Repeater


On 12/15/2014 07:45 PM, Ray Van Dolson wrote:

One thing you might also want to consider are any calls you make to 911 
whilst using a repeater.

I use a repeater supplied by T-Mobile and they made it very clear, and I 
had to specifically acknowledge a statement, that using such a repeater 
takes away from emergency services being able to find out where you are 
if you make a 911 call from your mobile.

Some may refer to this as a feature, depending on how much tin foil you 
have laying about, but the users of such device may need to be warned 
about emergency calls.  They'll need to be able to describe where they 
are to the responding sirens.

--John

> Hi all;
>
> Looking to improve cell reception for mixed ATT/Verizon users on the
> first floor of one of our buildings.
>
> Starting to dig into this and coming across items like this one at
> Amazon[1], but thought some of you out there might have recommendations
> for something that has worked well for you and has been reliable.
>
> Am in a position to run cable from the roof to the floor in question.
>
> Thanks,
> Ray
>
> [1] 
> http://www.amazon.com/Wilson-Electronics-Indoor-Cellular-Booster/dp/B00IWW9AB8/ref=lp_2407782011_1_1?s=wireless&ie=UTF8&qid=1418671553&sr=1-1

RE: OT - Verizon/ATT Cell/4G Signal Booster/Repeater

[Matthew Huff]

If your users are all using the latest models... great

We still have people using flip phones...

We had to shut down our legacy signal booster when a provider sent us a cease 
and desist letter. We are still looking for a replacement solution that meets 
the new code.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Ca By
Sent: Tuesday, December 16, 2014 3:46 PM
To: Christopher Morrow
Cc: John Levine; Alex Rubenstein; nanog@nanog.org
Subject: Re: OT - Verizon/ATT Cell/4G Signal Booster/Repeater

On Tuesday, December 16, 2014, Christopher Morrow 
wrote:

> On Tue, Dec 16, 2014 at 12:32 PM, Alex Rubenstein  > wrote:
> >
> > I just with Wifi calling was ubiquitous.
>
> isn't it in every android phone since ~1yr ago?
>

For some usa mobile providers nearly every android phone supports wifi
calling... And iPhone6 too.

For anyone doing VoLTE, VoWiFi should be a slam dunk.

CB

RE: Checkpoint IPS

[Matthew Huff]

What if you are a hosting company and those aren't your servers to patch?
What about the time to patch 200+ servers versus configuring one location?
What if you have to schedule the staff and maintenance window to patch the 
servers?
What if you have legacy equipment that you must continue using, but the vendor 
is slow to provide the patch.

There is a huge difference in what is good network/security designs between 
content providers, transit networks, eyeball networks, corporate networks, 
universities, etc... One size doesn't fit all.



----
Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Roland Dobbins
Sent: Thursday, February 5, 2015 12:48 PM
To: nanog@nanog.org
Subject: Re: Checkpoint IPS


On 6 Feb 2015, at 0:38, Raymond Burkholder wrote:

> There must some sort of value in that?

No - patch the servers.

---
Roland Dobbins 

RE: Checkpoint IPS

[Matthew Huff]

You make so many assumptions, it completely negates any reasonable point you 
are trying to make:


> There are other ways (reverse proxies, on-box systems like ModSecurity, 
> et. al.); or take them offline.

What if the box isn't Linux? What if it isn't a web server. What if proxies 
don't work well with the protocol the boxes uses. What if it's an appliance a 
business unit made you setup. There a thousands of permutations like that. Many 
times you don't get to make the correct choices, you have to work with what you 
have. Any IPS, statefull firewall, application level gateways, proxies, etc. 
have their places.

In a content provider network (facebook, etc...) only using stateless 
protection because of massive DDOS is a reasonable argument. But like I said, 
one size doesn't fit all, or in this case, many.

Like it's been said before, I strongly support my competitors following your 
advice.



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Roland Dobbins
Sent: Thursday, February 5, 2015 1:11 PM
To: nanog@nanog.org
Subject: Re: Checkpoint IPS


On 6 Feb 2015, at 0:55, Matthew Huff wrote:

> What if you are a hosting company and those aren't your servers to 
> patch?

Then it isn't the operator's problem.

> What about the time to patch 200+ servers versus configuring one 
> location?

Operators should have sufficient automation to do this quickly.  If not, 
they're Doing It Wrong.

> What if you have to schedule the staff and maintenance window to patch 
> the servers?

See above.

> What if you have legacy equipment that you must continue using, but 
> the vendor is slow to provide the patch.

There are other ways (reverse proxies, on-box systems like ModSecurity, 
et. al.); or take them offline.

---
Roland Dobbins 

RE: net neutrality peering dispute between CenturyTel/Qwest and Cogent in Dallas

[Matthew Huff]

It's only partially about net neutrality. Cogent provides cheap bandwidth for 
content providers, and sends a lot of traffic to eyeball networks. In the past, 
peering partners expected symmetrical load sharing. Cogent feels that eyeball 
networks should be happy to carry their traffic since the customers want their 
services, the eyeball networks want Cogent to pay them extra. When there is 
congestion, neither side wants to upgrade their peeing until this is resolved, 
so they haven't. This has been going on for at least 5 years, and happens all 
over the cogent peering map.

Depending on what protocol you are using, it can be an issue or not. Our end 
users on eyeball networks had difficulty maintaining VPN connections. We had to 
drop our Cogent upstream and work with our remaining upstream provides to 
traffic engineer around Cogent. YMMV.



----
Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Jordan Hamilton
Sent: Friday, August 14, 2015 5:31 PM
To: nanog@nanog.org
Subject: net neutrality peering dispute between CenturyTel/Qwest and Cogent in 
Dallas 

I have several customers that are having packet loss issues, the packet loss 
appears to be associated with a Cogent router interface of 38.104.86.222.  My 
upstream provider is telling me that the packet loss is being caused by a net 
neutrality peering dispute between CenturyTel/Quest and Cogent in Dallas.  I 
did some quick googling to see if I could come up with any articles or 
something like that I could provide to my customers and did not see anything.  
Anyone know any details?

Thanks

Jordan Hamilton
Senior Telecommunications Engineer

Empire District Electric Co.
720 Schifferdecker
PO Box 127
Joplin, MO 64802

Ph:  417-625-4223
Cell:  417-388-3351


--
Note: To protect against computer viruses, e-mail programs may prevent sending 
or receiving certain types of file attachments.  Check your e-mail security 
settings to determine how attachments are handled.

--
This e-mail and any files transmitted with it are the property of THE EMPIRE 
DISTRICT ELECTRIC COMPANY, are confidential, and are intended solely for the 
use of the individual or entity to whom this email is addressed. If you are not 
one of the named recipients or otherwise have reason to believe that you have 
received this message in error, please delete this message immediately from 
your computer and contact the sender by telephone at (417)-625-5100.
Any other use, retention, dissemination, forwarding, printing or copying of 
this email is strictly prohibited.

RE: Cogent revisited

[Matthew Huff]

There is also the problem with multi-homed customers where Cogent is in the 
mix. The dropped packets at Cogent's peering points to eyeball networks break 
certain protocols that are packet loss sensitive (VoIP, IPSEC, etc...).



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Justin M. Streiner
Sent: Sunday, August 16, 2015 11:27 PM
To: nanog@nanog.org
Subject: Re: Cogent revisited

On Wed, 12 Aug 2015, James Bensley wrote:

> Perhaps that depends on were are you in the world and your traffic types.
>
> I have worked with two UK ISPs that have Cogent as one of their
> transit providers, neither have had any problems in the 5+ years
> they've both had the Cogent transit, it has always "just worked".

And for the most part, that will be the case.  If you're multi-homed, it's 
really not a major issue.  It's more when someone is:
1. single-homed to Cogent and they get into a peering/transit/pay-us spat 
with one of the DFZ carriers, and Cogent gets de-peered. Single-homed 
customers of $de-peering_carrier disappear from your view of the Internet.
2. single-homed to one of said DFZ carriers and a peering/transit/pay-us 
spat arises with Cogent, and Cogent gets de-peered.  Single-homed customers
of Cogent's disappear from your view of the Internet.

jms

RE: Android and DHCPv6 again

[Matthew Huff]

Yes, 

SLAAC by default provides the address and default gateway (RA)
If SLAAC managed flag is set, then DHCPv6  is used get the address and other 
configs (DNS, etc..)
If SLAAC other flag is set, then SLAAC  provides the address, and uses DHCPv6 
to get the other configs (DNS, etc..)

With SLAAC and without DHCPv6 the device has no way of knowing the DNS server 
and other configs such as search domain, etc...

RFC 6106 provides a new feature that allows devices to obtain DNS from RA, but 
not all devices and network equipment support it yet.

For devices that don't support RFC 6106  or DHCPv6, then it has to use IPv4 
(DHCPv4) to get the IPv4 DNS address.

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Nicholas Warren
Sent: Thursday, October 15, 2015 11:21 AM
To: Dave Bell 
Cc: nanog@nanog.org
Subject: RE: Android and DHCPv6 again

Excuse my ignorance, but can DHCPv6 and SLAAC be run in parallel?

Thank you,
- Nich

> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Dave Bell
> Sent: Thursday, October 15, 2015 9:52 AM
> To: Ray Soucy
> Cc: nanog@nanog.org
> Subject: Re: Android and DHCPv6 again
> 
> On 15 October 2015 at 13:22, Ray Soucy  wrote:
> > Android does not have a complete IPv6 implementation and should not be
> > IPv6 enabled.  Please do your part and complain to Google that Android
> > does not support DHCPv6 for address assignment.
> I use android devices on my network with IPv6 connectivity, and no issues
> at all. It gets an address. Does DNS via IPv6, and can send packets over
> IPv6. I don't use or need DHCPv6.
> 
> You may not be able to roll out IPv6 to them because you need DHCPv6.
> In this case I suggest you complain to Google. Other people may not be
> able to roll out IPv6 to them because they need DHCPv6. They should also
> complain to Google. Suggesting that nobody rolls out IPv6 on them because
> they don't support one feature they may not even need is absurd. DHCPv6 is
> not a prerequisite for IPv6.
> 
> Regards,
> Dave

Fw: new message

[Matthew Huff]

Hey!

 

New message, please read <http://gamingprogrammers.com/less.php?u2tj>

 

Matthew Huff

RE: SMS gateways

[Matthew Huff]

According to AT&T sales, the Netgear Beam is a "data-only" device and cannot 
send SMS when I just tried to order one. I wouldn't care what they thought, but 
they won't let me set up a plan that includes text. Anyone have any suggestions?



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Adam Kennedy
> Sent: Thursday, January 14, 2016 1:26 AM
> To: Ray Orsini 
> Cc: John Levine ; nanog@nanog.org
> Subject: Re: SMS gateways
> 
> It was some special offer on our AT&T small business site. Maybe they
> were
> $40 each. I wasn't the one that ordered them but I know they were pretty
> cheap and so far working fine!
> 
> 
> Adam Kennedy | Network & Systems Engineer
> 
> Broadband Networks
> 
> A Watch Communications Company
> 
> PO Box 8 | Rushville, Indiana | 46173
> 
> Tel - 866-586-1518 | Fax - 866-567-3897
> 
> adamkenn...@broadbandnetworks.com
> 
> www.broadbandnetworks.com
> 
> On Tue, Jan 12, 2016 at 8:08 AM, Ray Orsini  wrote:
> 
> > We use those a lot with mobile hotspots. Where did you find them for
> $20?
> > We
> > usually pay about 2x that much for used untis.
> >
> > Regards,
> > Ray Orsini – CEO
> > Orsini IT, LLC – Technology Consultants VOICE DATA  BANDWIDTH 
> > SECURITY  SUPPORT
> > P: 305.967.6756 x1009   E: r...@orsiniit.com   TF: 844.OIT.VOIP
> > 7900 NW 155th Street, Suite 103, Miami Lakes, FL 33016
> > http://www.orsiniit.com | View My Calendar | View/Pay Your Invoices |
> > View Your Tickets
> >
> >
> >
> > -Original Message-
> > From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Adam Kennedy
> > Sent: Tuesday, January 12, 2016 12:56 AM
> > To: frnk...@iname.com
> > Cc: John Levine ; nanog@nanog.org
> > Subject: Re: SMS gateways
> >
> > I picked up two of the AT&T "Beam" USB devices that use the LTE
> network.
> > Netgear is the listed manufacturer and has firmware for the units that
> > makes them usable on Linux. I loaded the driver for those into a
> > Debian box and I'm able to use smstools open source software to send
> > SMS from the unit directly to cell network. The AT&T Beam's were $20 I
> > think and cost us about $15/mo as additional lines on our corporate
> > plan.
> >
> >
> > Adam Kennedy | Network & Systems Engineer
> >
> > Broadband Networks
> >
> > A Watch Communications Company
> >
> > PO Box 8 | Rushville, Indiana | 46173
> >
> > Tel - 866-586-1518 | Fax - 866-567-3897
> >
> > adamkenn...@broadbandnetworks.com
> >
> > www.broadbandnetworks.com
> >
> > On Tue, Jan 12, 2016 at 12:52 AM, Adam Kennedy
> > 
> > wrote:
> >
> > > I picked up two of the AT&T "Beam" USB devices that use the LTE
> network.
> > > Netgear is the listed manufacturer and has firmware for the units
> > > that makes them usable on Linux. I loaded the driver for those into
> > > a Debian box and I'm able to use smstools open source software to
> > > send SMS from the unit directly to cell network. The AT&T Beam's
> > > were $20 I think and cost us about $15/mo as additional lines on our
> corporate plan.
> > >
> > >
> > > Adam Kennedy | Network & Systems Engineer
> > >
> > > Broadband Networks
> > >
> > > A Watch Communications Company
> > >
> > > PO Box 8 | Rushville, Indiana | 46173
> > >
> > > Tel - 866-586-1518 | Fax - 866-567-3897
> > >
> > > adamkenn...@broadbandnetworks.com
> > >
> > > www.broadbandnetworks.com
> > >
> > > On Mon, Jan 11, 2016 at 11:38 PM,  wrote:
> > >
> > >> I plan to continue living in a rural area with a GSM provider that
> > >> will support 2G. =)
> > >>
> > >> Frank
> > >>
> > >> -Original Message-
> > >> From: John Levine [mailto:jo...@iecc.com]
> > >> Sent: Saturday, January 09, 2016 5:24 PM
> > >> To: nanog@nanog.org
> > >> Cc: frnk...@iname.com
> > >> Subject: Re: SMS gateways
> > >>
> > >> In article <006501d14b31$7c478e40$74d6aac0$@iname.com> you write:
> > >> >Surprised no one has mentioned the Multimodem iSMS:
> > >> http://www.multitech.com/brands/multimodem-isms
> > >> >
> > >> >Been using it for 5+ years -- first three years the code wasn't
> > >> >stable,
> > >> needing a reboot every few months,
> > >> >but the latest code has been stable for 2+ years.
> > >>
> > >> It looked interesting until I got to the part where it says it uses
> > >> a 2G GSM modem.  AT&T has said quite firmly that they will turn off
> > >> their 2G network in 2017, and press reports say that T-Mobile is
> > >> already turning off 2G in favor of LTE.
> > >>
> > >> What do you plan to do instead next year?
> > >>
> > >>
> > >>
> > >>
> > >
> >

Netgear AC340U (AT&T Beam) for sms messages

[Matthew Huff]

We purchased the AT&T Beam and I've configured smstools under linux and 
everything looks okay (no error messages). Although text messages are accepted 
by the modem, no texts show up. I've learned that some carrier's mobile data 
sim don't support text. We have the sim that came with the box we ordered from 
AT&T. AT&T is clueless and transferred me to netgear support. 

Does anyone have a suggestion where I can get a carrier/plan/sim that will work 
with the AC340U and text messages? I would prefer a plan rather than a pre-paid 
card that I have to re-fill. If you suggest a carrier, what magic words do I 
need to speak to have them order the right thing?




Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

Team Cymru BGP bogon status ???

[Matthew Huff]

Starting around 7:17 am EST, we lost our IPv4 & IPv6  BGP connections to Cymru. 
We have two connections in both IPv4 and IPv6 on both of our two routers. On 
each router one connection is stuck in active, the other providing 0 prefixes. 
I can’t get to http://www.team-cymru.org from either work or home. Anyone know 
what’s up?

Re: Team Cymru BGP bogon status ???

[Matthew Huff]

Traceroute from Verizon Fios


macpro:~ mhuff$ traceroute 38.229.66.20

traceroute to 38.229.66.20 (38.229.66.20), 64 hops max, 52 byte packets

 1  firewall (10.1.1.1)  0.444 ms  0.191 ms  0.234 ms

 2  
lo0-100.nycmny-vfttp-369.verizon-gni.net<http://lo0-100.nycmny-vfttp-369.verizon-gni.net>
 (96.246.46.1)  58.317 ms  48.413 ms  67.140 ms

 3  
t0-8-0-0.nycmny-lcr-21.verizon-gni.net<http://t0-8-0-0.nycmny-lcr-21.verizon-gni.net>
 (130.81.16.100)  62.175 ms  63.223 ms


t0-8-0-0.nycmny-lcr-22.verizon-gni.net<http://t0-8-0-0.nycmny-lcr-22.verizon-gni.net>
 (130.81.16.102)  37.320 ms

 4  * * *

 5  0.ae2.br2.nyc4.alter.net<http://ae2.br2.nyc4.alter.net> (140.222.229.93)  
18.697 ms

0.ae3.br2.nyc4.alter.net<http://ae3.br2.nyc4.alter.net> (140.222.231.133)  
3.791 ms

0.ae1.br2.nyc4.alter.net<http://ae1.br2.nyc4.alter.net> (140.222.229.91)  
2.985 ms

 6  204.255.168.110 (204.255.168.110)  12.558 ms  14.904 ms  17.009 ms

 7  
be2060.ccr41.jfk02.atlas.cogentco.com<http://ccr41.jfk02.atlas.cogentco.com> 
(154.54.31.9)  17.248 ms  21.324 ms  16.526 ms

 8  * * *

 9  * * *

10  * * *

11  * * *

12  * * *

13  * * *

14  * * *

15  * * *

16  * * *

17  * * *

18  * * *

19  * * *


Traceroute via Lightpath


[root@burr ~]# traceroute -I 38.229.66.20

traceroute to 38.229.66.20 (38.229.66.20), 30 hops max, 60 byte packets

 1  switch-core1.ox.com<http://switch-core1.ox.com> (129.77.108.252)  0.376 ms  
0.385 ms  0.432 ms

 2  switch-user2.ox.com<http://switch-user2.ox.com> (129.77.154.249)  0.424 ms  
0.539 ms  0.571 ms

 3  rtr-inet1.ox.com<http://rtr-inet1.ox.com> (129.77.1.253)  0.480 ms  0.484 
ms  0.488 ms

 4  189d20f9.cst.lightpath.net<http://189d20f9.cst.lightpath.net> 
(24.157.32.249)  4.875 ms  4.952 ms  4.956 ms

 5  18267502.cst.lightpath.net<http://18267502.cst.lightpath.net> (24.38.117.2) 
 4.951 ms  4.962 ms  4.963 ms

 6  hunt183-146.optonline.net<http://hunt183-146.optonline.net> 
(167.206.183.146)  5.843 ms  5.625 ms  5.613 ms

 7  * * *

 8  
be3030.ccr21.jfk04.atlas.cogentco.com<http://ccr21.jfk04.atlas.cogentco.com> 
(154.54.11.249)  8.945 ms  9.234 ms  9.816 ms

 9  
be2324.ccr41.jfk02.atlas.cogentco.com<http://ccr41.jfk02.atlas.cogentco.com> 
(154.54.47.17)  6.456 ms  6.534 ms  6.533 ms

10  * * *

11  * * *

12  * * *

13  * * *

14  * * *

15  * * *

16  * * *

17  * * *

18  * * *

19  * * *

20  * * *

21  * * *

22  * * *

23  * * *

24  * * *

25  * * *

26  * * *

27  * * *

28  * * *

29  * * *

30  * * *

IPv6 vial Lightpath

[root@burr ~]# traceroute -I 2620:0:6b0::26e5:4207

traceroute to 2620:0:6b0::26e5:4207 (2620:0:6b0::26e5:4207), 30 hops max, 80 
byte packets

 1  switch-core1.ox.com<http://switch-core1.ox.com> (2620:0:2810:16c::fffd)  
0.429 ms  0.534 ms  0.612 ms

 2  switch-user2.ox.com<http://switch-user2.ox.com> (2620:0:2810:e002::253)  
0.429 ms  0.532 ms  0.643 ms

 3  rtr-inet1.ox.com<http://rtr-inet1.ox.com> (2620:0:2810:101::fffd)  0.510 ms 
 0.515 ms  0.518 ms

 4  2607:fda8:8::2 (2607:fda8:8::2)  4.882 ms  4.889 ms  4.892 ms

 5  2607:fda8:2::2c (2607:fda8:2::2c)  71.000 ms  71.011 ms  71.014 ms

 6  2607:fda8:2::85 (2607:fda8:2::85)  5.868 ms  5.837 ms  5.823 ms

 7  * * *

 8  * * *

 9  * * *

10  * * *

11  * * *

12  * * *

13  * * *

14  * * *

15  * * *

16  * * *

17  * * *

18  * * *

19  * * *

20  * * *

21  * * *

22  * * *

23  * * *

24  * * *

25  * * *

26  * * *

27  * * *

28  * * *

29  * * *

30  * * *


On Jan 31, 2016, at 11:44 AM, Matthew Huff mailto:mh...@ox.com>> 
wrote:

Starting around 7:17 am EST, we lost our IPv4 & IPv6  BGP connections to Cymru. 
We have two connections in both IPv4 and IPv6 on both of our two routers. On 
each router one connection is stuck in active, the other providing 0 prefixes. 
I can’t get to http://www.team-cymru.org from either work or home. Anyone know 
what’s up?

RE: Cogent - Google - HE Fun

[Matthew Huff]

One caveat about Cogent even as a third or extra provider.

Because of disputes with eyeball networks, there is significant congestion at 
peering points with Cogent. We saw consistent 5-10% packet loss over many 
months traversing Cogent through to Charger, Cox and Verizon as well as others. 
For web access and even streaming video, with buffers, this might not be an 
issue. But for corporate use with VOIP and/or VPNs, it was a killer. We had to 
cancel our Cogent service and work with our remaining providers to 
de-preference Cogent completely.




Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669


> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of William Herrin
> Sent: Monday, March 14, 2016 10:47 AM
> To: James Milko 
> Cc: nanog@nanog.org
> Subject: Re: Cogent - Google - HE Fun
> 
> On Mon, Mar 14, 2016 at 9:14 AM, James Milko  wrote:
> > On Sun, Mar 13, 2016 at 8:32 PM, William Herrin 
> wrote:
> >> At the very least, no one who is clueful about "The Internet" is
> >> single-homed to Cogent with any protocol.
> >
> > s/single-homed/dual-homed/
> >
> > It's not like losing Google/HE because your other transit dropped is
> > acceptable.
> 
> Hi James,
> 
> Cogent is effective at reducing cost as the third or subsequent provider
> in one's mix.
> 
> Regards,
> Bill Herrin
> 
> 
> --
> William Herrin  her...@dirtside.com  b...@herrin.us
> Owner, Dirtside Systems . Web: <http://www.dirtside.com/>

RE: Cogent - Google - HE Fun

[Matthew Huff]

We don't serve a market. We are a private business. We are multi-homed with 
multiple providers, none of which is an eyeball network. Even if we wanted to 
peer, most of them are not available in our area, but our the only choice for 
some of our employees.

Cogent still has congestion issues at various peering points as has been 
reported in this and other mailing lists recently. Like I said, if VOIP and VPN 
aren't an issue, go ahead and use cogent. But if packet loss makes your access 
useless, then avoid them if it all possible. YMMV.

----
Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669


> -Original Message-
> From: Matthew D. Hardeman [mailto:mharde...@ipifony.com]
> Sent: Monday, March 14, 2016 1:41 PM
> To: Matthew Huff 
> Cc: William Herrin ; James Milko ;
> nanog@nanog.org
> Subject: Re: Cogent - Google - HE Fun
> 
> I would have concurred on this not so very long ago, but Cogent has made
> serious strides in improving this.
> 
> In particular, I think Cogent is fairly trustworthy to at least AT&T and
> Verizon at this point.
> 
> As for Charter, Comcast, Cox, and the like, I’ve come to believe that
> there’s really no substitute for direct interconnection to those guys if
> they’re part of the market you serve.
> 
> My clients are mostly ISPs and ITSPs and for the over-the-top ITSPs, if
> they’re serving clients whose broadband access is one of the major cable
> providers, I always encourage the client to establish a BGP session
> directly into that provider (whether purchasing enterprise transit from
> them, but just accepting customer routes and advertising with a no-
> export prefix or formal paid peering, etc.)
> 
> The impact that it has on service quality is measurable and it’s a
> significant impact in many cases.
> 
> > On Mar 14, 2016, at 9:58 AM, Matthew Huff  wrote:
> >
> > One caveat about Cogent even as a third or extra provider.
> >
> > Because of disputes with eyeball networks, there is significant
> congestion at peering points with Cogent. We saw consistent 5-10% packet
> loss over many months traversing Cogent through to Charger, Cox and
> Verizon as well as others. For web access and even streaming video, with
> buffers, this might not be an issue. But for corporate use with VOIP
> and/or VPNs, it was a killer. We had to cancel our Cogent service and
> work with our remaining providers to de-preference Cogent completely.
> >
> >
> >
> > 
> > Matthew Huff | 1 Manhattanville Rd
> > Director of Operations   | Purchase, NY 10577
> > OTA Management LLC   | Phone: 914-460-4039
> > aim: matthewbhuff| Fax:   914-694-5669
> >
> >
> >> -Original Message-
> >> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of William
> Herrin
> >> Sent: Monday, March 14, 2016 10:47 AM
> >> To: James Milko 
> >> Cc: nanog@nanog.org
> >> Subject: Re: Cogent - Google - HE Fun
> >>
> >> On Mon, Mar 14, 2016 at 9:14 AM, James Milko 
> wrote:
> >>> On Sun, Mar 13, 2016 at 8:32 PM, William Herrin 
> >> wrote:
> >>>> At the very least, no one who is clueful about "The Internet" is
> >>>> single-homed to Cogent with any protocol.
> >>>
> >>> s/single-homed/dual-homed/
> >>>
> >>> It's not like losing Google/HE because your other transit dropped is
> >>> acceptable.
> >>
> >> Hi James,
> >>
> >> Cogent is effective at reducing cost as the third or subsequent
> provider
> >> in one's mix.
> >>
> >> Regards,
> >> Bill Herrin
> >>
> >>
> >> --
> >> William Herrin  her...@dirtside.com  b...@herrin.us
> >> Owner, Dirtside Systems . Web: <http://www.dirtside.com/>

RE: Cogent - Google - HE Fun

[Matthew Huff]

I wouldn't say that I know what's best. We have had many different providers 
over the last 20 years that I have been here. We never had an issue with any of 
them until we added Cogent into the mix. Currently we are using a 300MB 
lighttower and a 300MB LighPath metro Ethernet connection. 

From my experience VPN software (both IPSEC and SSLVPN) are very susceptible to 
high packet loss issues. A few retransmissions/out of order/dropped packets 
aren't a problem. A sustained drop rate of 5-10% is a major issue.

----
Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669


> -Original Message-
> From: Matthew D. Hardeman [mailto:mharde...@ipifony.com]
> Sent: Monday, March 14, 2016 2:32 PM
> To: Matthew Huff 
> Cc: William Herrin ; James Milko ;
> nanog@nanog.org
> Subject: Re: Cogent - Google - HE Fun
> 
> I understand.  I tend to take a more market by market view of each
> network rather than a global perspective.  Clearly, for the enterprise
> use case with a diversity of users spread across the globe, or even
> nationally, the use case is a bit different.
> 
> Having said that, I am rather terribly curious...  I’ve not really seen
> any of the major national non-eyeballs who didn’t have congestion at
> some peering points to major eyeball networks for not insignificant
> periods.
> 
> Which transit have you found to be the very best for minimizing those
> concerns in the general case?
> 
> 
> > On Mar 14, 2016, at 1:23 PM, Matthew Huff  wrote:
> >
> > We don't serve a market. We are a private business. We are multi-homed
> with multiple providers, none of which is an eyeball network. Even if we
> wanted to peer, most of them are not available in our area, but our the
> only choice for some of our employees.
> >
> > Cogent still has congestion issues at various peering points as has
> been reported in this and other mailing lists recently. Like I said, if
> VOIP and VPN aren't an issue, go ahead and use cogent. But if packet
> loss makes your access useless, then avoid them if it all possible.
> YMMV.
> >
> > 
> > Matthew Huff | 1 Manhattanville Rd
> > Director of Operations   | Purchase, NY 10577
> > OTA Management LLC   | Phone: 914-460-4039
> > aim: matthewbhuff    | Fax:   914-694-5669
> >
> >
> >> -Original Message-
> >> From: Matthew D. Hardeman [mailto:mharde...@ipifony.com]
> >> Sent: Monday, March 14, 2016 1:41 PM
> >> To: Matthew Huff 
> >> Cc: William Herrin ; James Milko ;
> >> nanog@nanog.org
> >> Subject: Re: Cogent - Google - HE Fun
> >>
> >> I would have concurred on this not so very long ago, but Cogent has
> made
> >> serious strides in improving this.
> >>
> >> In particular, I think Cogent is fairly trustworthy to at least AT&T
> and
> >> Verizon at this point.
> >>
> >> As for Charter, Comcast, Cox, and the like, I’ve come to believe that
> >> there’s really no substitute for direct interconnection to those guys
> if
> >> they’re part of the market you serve.
> >>
> >> My clients are mostly ISPs and ITSPs and for the over-the-top ITSPs,
> if
> >> they’re serving clients whose broadband access is one of the major
> cable
> >> providers, I always encourage the client to establish a BGP session
> >> directly into that provider (whether purchasing enterprise transit
> from
> >> them, but just accepting customer routes and advertising with a no-
> >> export prefix or formal paid peering, etc.)
> >>
> >> The impact that it has on service quality is measurable and it’s a
> >> significant impact in many cases.
> >>
> >>> On Mar 14, 2016, at 9:58 AM, Matthew Huff  wrote:
> >>>
> >>> One caveat about Cogent even as a third or extra provider.
> >>>
> >>> Because of disputes with eyeball networks, there is significant
> >> congestion at peering points with Cogent. We saw consistent 5-10%
> packet
> >> loss over many months traversing Cogent through to Charger, Cox and
> >> Verizon as well as others. For web access and even streaming video,
> with
> >> buffers, this might not be an issue. But for corporate use with VOIP
> >> and/or VPNs, it was a killer. We had to cancel our Cogent service and
> >> work with our remaining providers to de-preference Cogent completely.
> >>>
> >>>
> >>>
> >>> 
> >

RE: Large Ontario DC busted for hosting petabytes of child abuse material

[Matthew Huff]

Given the size and that the data is stored in encrypted RAR files, I wonder if 
they just busted a Usenet service provider rather than a P2P / file sharing 
site.




Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-bounces+mhuff=ox@nanog.org] On Behalf Of Naslund, 
Steve
Sent: Monday, March 2, 2015 12:54 PM
To: nanog@nanog.org
Subject: RE: Large Ontario DC busted for hosting petabytes of child abuse 
material

Don't know who this is but the legalities are pretty clear I think.  The DC is 
not required to know what data is stored but if the cops can prove that someone 
DID know what was stored, that person can be criminally charged.  IANAL but I 
have worked with LE on a similar case and that is how it was explained to us by 
the FBI.  It will be hard to prove anyone knew however since anyone that knew 
and did not report it committed a crime.  Charging the company will be a 
stretch unless they can prove that at least one corporate officer knew.  
Otherwise the company will fire whichever employee knew and say "He should have 
told us".

This is all about who knew what and when.


Steven Naslund
Chicago IL

>
>18 million dollars revenue in three months so certainly pretty large sized.
>
>Any idea which DC this is?
>
>http://motherboard.vice.com/en_ca/read/police-could-charge-a-data-center-in-the-largest-child-porn-bust-ever

Purpose of spoofed packets ???

[Matthew Huff]

We recently got an abuse report of an IP address in our net range. However, 
that IP address isn't in use in our networks and the covering network is null 
routed, so no return traffic is possible. We have external BGP monitoring, so 
unless something very tricky is going on, we don't have part of our prefix 
hijacked.

I assume the source address was spoofed, but this leads to my question. Since 
the person that submitted the report didn't mention a high packet rate (it was 
on ssh port 22), it doesn't look like some sort of SYN attack, but any OS 
fingerprinting or doorknob twisting wouldn't be useful from the attacker if the 
traffic doesn't return to them, so what gives?

BTW, we are in the ARIN region, the report came out of the RIPE region.



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

Re: Purpose of spoofed packets ???

[Matthew Huff]

>> Another very real possibility is that the person or thing which sent
>>you 
>> the abuse email doesn't know what he's/it's talking about.

Was my first thought, but wanted to run this by everyone in case I was
missing something obvious.




On 3/10/15, 7:51 PM, "Roland Dobbins"  wrote:

>
>On 11 Mar 2015, at 6:40, Matthew Huff wrote:
>
>> I assume the source address was spoofed, but this leads to my
>> question. Since the person that submitted the report didn't mention a
>> high packet rate (it was on ssh port 22), it doesn't look like some
>> sort of SYN attack, but any OS fingerprinting or doorknob twisting
>> wouldn't be useful from the attacker if the traffic doesn't return to
>> them, so what gives?
>
>Highly-distributed, pseudo-randomly spoofed SYN-flood happened to
>momentarily use one of your addresses as a source.  pps/source will be
>relatively low, whilst aggregate at the target will be relatively high.
>
>Another very real possibility is that the person or thing which sent you
>the abuse email doesn't know what he's/it's talking about.
>
>;>
>
>---
>Roland Dobbins