RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Matthew Huff]

What about VRFs and/or policy based routing?  

switch-core1# show vrf
VRF-Name   VRF-ID State   Reason
default 1 Up  --
management  2 Up  --

switch-core1# show route-map 
route-map rmap_bgp_to_eigrp_b2b, permit, sequence 10 
  Match clauses:
interface: Ethernet1/33 
route-type: internal 
  Set clauses:
metric 4000 10 255 1 1500 
route-map rmap_bgp_to_eigrp_b2b, permit, sequence 20 
  Match clauses:
interface: Ethernet1/34 
route-type: internal 
  Set clauses:
metric 4000 30 255 1 1500 
route-map rmap_static_to_eigrp, permit, sequence 10 
  Match clauses:
ip address prefix-lists: prefix_static_to_eigrp 
  Set clauses:
route-map rmap_static_to_eigrp_v6, permit, sequence 10 
  Match clauses:
ipv6 address prefix-lists: prefix_ipv6_static_to_eigrp 
  Set clauses:



From: Mike Hammett  
Sent: Monday, April 3, 2023 9:00 AM
To: Matthew Huff 
Cc: NANOG 
Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

It could be an sFlow bug, but I come at this from a reported problem and 
gathering data on that problem as opposed to looking at data for problems.

The snmp if index reported by the Nexus matches the if index in ElastiFlow.




-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com


From: "Matthew Huff" <mailto:mh...@ox.com>
To: "Mike Hammett" <mailto:na...@ics-il.net>
Cc: "NANOG" <mailto:nanog@nanog.org>
Sent: Monday, April 3, 2023 7:50:08 AM
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
SFlow misconfiguration or bug on either the nexus or the sflow monitor? On the 
monitor, can you verify that the snmp interfaces are mapped to the correct ones 
on the nexus?
 
 
 
 
 
From: Mike Hammett <mailto:na...@ics-il.net> 
Sent: Monday, April 3, 2023 8:47 AM
To: Matthew Huff <mailto:mh...@ox.com>
Cc: NANOG <mailto:nanog@nanog.org>
Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging
 
It shows the desired result.


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com
 

From: "Matthew Huff" <mailto:mh...@ox.com>
To: "Mike Hammett" <mailto:na...@ics-il.net>, "NANOG" <mailto:nanog@nanog.org>
Sent: Monday, April 3, 2023 5:38:23 AM
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

switch-core1# sh forwarding route x.x.x.x

slot  1
===


IPv4 routes for table default/base

--+-+--+-+-
Prefix            | Next-hop                                | Interface         
   | Labels          | Partial Install 
--+-+--+-+-
x.x.x.x/24      x.x.x.250                            Ethernet1/29        


switch-core1# show routing hash x.x.x.x y.y.y.y
Load-share parameters used for software forwarding:
load-share mode: address source-destination port source-destination
Hash for VRF "default"
Hashing to path *y.y.y.y Eth1/29
For route:
y.y.y.0/24, ubest/mbest: 1/0
    *via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal




From: NANOG <mailto:nanog-bounces+mhuff=ox@nanog.org> On Behalf Of Mike 
Hammett
Sent: Monday, April 3, 2023 1:21 AM
To: NANOG <mailto:nanog@nanog.org>
Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

We have a Nexus 3064 that is setup with partial BGP tables and is routing based 
on that. 


I've done a show ip bgp for an IP of interest and it has an expected next hop 
IP. I show ip arp on that next hop IP and it has the expected interface. 


However, sFlows show the packets leaving on a different interface, the one that 
would carry the default route for routes not otherwise known. 


If the next hop IP is expected and the ARP of that next hop IP is expected, why 
are packets leaving out an unexpected interface? 


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com
 

RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Matthew Huff]

SFlow misconfiguration or bug on either the nexus or the sflow monitor? On the 
monitor, can you verify that the snmp interfaces are mapped to the correct ones 
on the nexus?





From: Mike Hammett 
Sent: Monday, April 3, 2023 8:47 AM
To: Matthew Huff 
Cc: NANOG 
Subject: Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

It shows the desired result.


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com


From: "Matthew Huff" mailto:mh...@ox.com>>
To: "Mike Hammett" mailto:na...@ics-il.net>>, "NANOG" 
mailto:nanog@nanog.org>>
Sent: Monday, April 3, 2023 5:38:23 AM
Subject: RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

switch-core1# sh forwarding route x.x.x.x

slot  1
===


IPv4 routes for table default/base

--+-+--+-+-
Prefix| Next-hop| Interface 
   | Labels  | Partial Install
--+-+--+-+-
x.x.x.x/24  x.x.x.250Ethernet1/29


switch-core1# show routing hash x.x.x.x y.y.y.y
Load-share parameters used for software forwarding:
load-share mode: address source-destination port source-destination
Hash for VRF "default"
Hashing to path *y.y.y.y Eth1/29
For route:
y.y.y.0/24, ubest/mbest: 1/0
*via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal




From: NANOG 
mailto:nanog-bounces+mhuff=ox@nanog.org>>
 On Behalf Of Mike Hammett
Sent: Monday, April 3, 2023 1:21 AM
To: NANOG mailto:nanog@nanog.org>>
Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

We have a Nexus 3064 that is setup with partial BGP tables and is routing based 
on that.


I've done a show ip bgp for an IP of interest and it has an expected next hop 
IP. I show ip arp on that next hop IP and it has the expected interface.


However, sFlows show the packets leaving on a different interface, the one that 
would carry the default route for routes not otherwise known.


If the next hop IP is expected and the ARP of that next hop IP is expected, why 
are packets leaving out an unexpected interface?


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

RE: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Matthew Huff]

switch-core1# sh forwarding route x.x.x.x

slot  1
===


IPv4 routes for table default/base

--+-+--+-+-
Prefix| Next-hop| Interface 
   | Labels  | Partial Install 
--+-+--+-+-
x.x.x.x/24  x.x.x.250Ethernet1/29


switch-core1# show routing hash x.x.x.x y.y.y.y
Load-share parameters used for software forwarding:
load-share mode: address source-destination port source-destination
Hash for VRF "default"
Hashing to path *y.y.y.y Eth1/29
For route:
y.y.y.0/24, ubest/mbest: 1/0
*via z.z.z.z, Eth1/29, [90/3072], 1w2d, eigrp-100, internal




From: NANOG  On Behalf Of Mike Hammett
Sent: Monday, April 3, 2023 1:21 AM
To: NANOG 
Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

We have a Nexus 3064 that is setup with partial BGP tables and is routing based 
on that. 


I've done a show ip bgp for an IP of interest and it has an expected next hop 
IP. I show ip arp on that next hop IP and it has the expected interface. 


However, sFlows show the packets leaving on a different interface, the one that 
would carry the default route for routes not otherwise known. 


If the next hop IP is expected and the ARP of that next hop IP is expected, why 
are packets leaving out an unexpected interface? 


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

RE: 400G forwarding - how does it work?

[Matthew Huff]

Also, for data center traffic, especially real-time market data and other UDP 
multicast traffic, micro-bursting is one of the biggest issues especially as 
you scale out your backbone. We have two 100GB switches, and have to distribute 
the traffic over a LACL link with 4 different 100GB ports on different ASIC 
even though the traffic < 1% just to account for micro-bursts.



-Original Message-
From: NANOG  On Behalf Of 
sro...@ronan-online.com
Sent: Monday, August 8, 2022 8:39 AM
To: Masataka Ohta 
Cc: nanog@nanog.org
Subject: Re: 400G forwarding - how does it work?

You keep using the term “imaginary” when presented with evidence that does not 
match your view of things. 

There are many REAL scenarios where single flow high throughout TCP is a real 
requirements as well as high throughput extremely small packet size. In the 
case of the later, the market is extremely large, but it’s not Internet traffic.

Shane

> On Aug 8, 2022, at 7:34 AM, Masataka Ohta  
> wrote:
> 
> Saku Ytti wrote:
> 
>>> which is, unlike Yttinet, the reality.
>> Yttinet has pesky customers who care about single TCP performance 
>> over long fat links, and observe poor performance with shallow 
>> buffers at the provider end.
> 
> With such an imaginary assumption, according to the end to end 
> principle, the customers (the ends) should use paced TCP instead of 
> paying unnecessarily bloated amount of money to intelligent 
> intermediate entities of ISPs using expensive routers with bloated 
> buffers.
> 
>> Yttinet is cost sensitive and does not want to do work, unless 
>> sufficiently motivated by paying customers.
> 
> I understand that if customers follow the end to end principle, 
> revenue of "intelligent" ISPs will be reduced.
> 
>Masataka Ohta
> 
> 
> 

RE: IERS ponders reverse leapsecond...

[Matthew Huff]

True, 

But it's hard enough to get developers to understand the need to code for 61 
seconds in a minute, and now they would need to code for 59 seconds as well.

If time systems simply skewed the time so that 60 seconds actually just took 61 
seconds or 59 seconds, there would be other issues, but coders wouldn't be 
involved.



-Original Message-
From: NANOG  On Behalf Of Stephane 
Bortzmeyer
Sent: Wednesday, August 3, 2022 11:19 AM
To: Jay Ashworth 
Cc: nanog@nanog.org
Subject: Re: IERS ponders reverse leapsecond...

On Wed, Aug 03, 2022 at 11:09:25AM -0400,  Jay Ashworth  
wrote  a message of 32 lines which said:

> General press loses its *mind*:

Indeed, they seem not to know what they write about. "atomic time – the 
universal way time is measured on Earth – may have to change" They don't even 
know the difference between TAI and UTC.

RE: Verizon no BGP route to some of AS38365 (182.61.200.0/24)

[Matthew Huff]

From my limited vantage point it appears that there is some issue between 
Verizon & Baidu. Baidu has 182.61.0.0/16 registered, but is only advertising 
pieces of it globally (or at least from what I can see). In our tables,we are 
receiving none from Verizon of  the subnets that are advertised directly from 
Baidu (origin AS of 38365). The few within that registered range that have a 
different origin AS are coming to us from Verizon. For example:

*>   182.61.0.0/19144.121.203.1410 46887 3356 4134 
58466 38365 i
*>   182.61.0.0/18144.121.203.1410 46887 6461 4134 
58466 38365 38365 i
*>   182.61.32.0/19   144.121.203.1410 46887 3356 4134 
58466 38365 i
*>   182.61.64.0/18   204.148.121.2210 701 6453 55967 i
*182.61.128.0/23  204.148.121.2210 701 4134 4134 
4134 4134 4134 58540 ?
*>   182.61.130.0/24  144.121.203.1410 46887 6461 4134 
23724 38365 38365 38365 i
*>   182.61.130.0/23  144.121.203.1410 46887 6461 4134 
58466 38365 38365 i
*>   182.61.131.0/24  144.121.203.1410 46887 6461 4134 
23724 38365 38365 38365 i
*>   182.61.132.0/23  144.121.203.1410 46887 3356 4134 
58466 38365 i
*>   182.61.132.0/22  144.121.203.1410 46887 6461 4134 
58466 38365 38365 i
*>   182.61.134.0/23  144.121.203.1410 46887 3356 4134 
58466 38365 i
*>   182.61.136.0/22  144.121.203.1410 46887 3356 4134 
58466 38365 i
*>   182.61.136.0/21  144.121.203.1410 46887 6461 4134 
58466 38365 38365 i
*>   182.61.140.0/22  144.121.203.1410 46887 3356 4134 
58466 38365 i
*>   182.61.144.0/21  144.121.203.1410 46887 3356 4134 
58466 38365 i
*>   182.61.144.0/20  144.121.203.1410 46887 6461 4134 
58466 38365 38365 i
*>   182.61.160.0/19  204.148.121.2210 701 6453 55967 i
*>   182.61.192.0/23  144.121.203.1410 46887 3356 4134 
58540 i
*>   182.61.194.0/23  144.121.203.1410 46887 3356 4134 
58540 i
*>   182.61.200.0/22  144.121.203.1410 46887 6461 4134 
23724 38365 i
*>   182.61.200.0/21  144.121.203.1410 46887 6461 4134 
58466 38365 38365 i
*>   182.61.216.0/21  144.121.203.1410 46887 6461 4134 
58466 38365 38365 i
*>   182.61.223.0/24  144.121.203.1410 46887 6461 4134 
58466 38365 38365 i
*>   182.61.224.0/19  144.121.203.1410 46887 6461 4134 
58466 38365 38365 i

We are getting the 182.61.200.0/21 and 182.61.200.0/22 from all of our other 
peers:

asr-inet2#sh ip bgp 182.61.200.0/21
BGP routing table entry for 182.61.200.0/21, version 15779018
Paths: (2 available, best #2, table default)
  Advertised to update-groups:
 3
  Refresh Epoch 1
  54004 6128 1299 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
148.77.99.201 from 148.77.99.201 (24.157.4.25)
  Origin IGP, localpref 100, valid, external, atomic-aggregate
  rx pathid: 0, tx pathid: 0
  Updated on Apr 29 2022 21:02:05 EDT
  Refresh Epoch 1
  46887 6461 4134 58466 38365 38365, (aggregated by 38365 119.75.208.225)
129.77.17.254 from 129.77.17.254 (129.77.40.7)
  Origin IGP, metric 0, localpref 100, valid, internal, atomic-aggregate, 
best
  rx pathid: 0, tx pathid: 0x0
  Updated on May 3 2022 04:02:50 EDT


From: NANOG  On Behalf Of holow29
Sent: Thursday, June 23, 2022 5:49 PM
To: nanog@nanog.org
Subject: Verizon no BGP route to some of AS38365 (182.61.200.0/24)

I've been trying (to no avail) for over a month now to get Verizon to 
investigate their lack of BGP routing to 
182.61.200.0/24, which hosts Baidu Wangpan at  
pan.baidu.com (Baidu's cloud services/equivalent of 
Google Drive).

Easily verified through Verizon's Looking Glass.

We all know Verizon's BGP routing is a disaster, but does anyone have any ideas?

RE: Congrats to AS701

[Matthew Huff]

Still no IPv6 in Westchester County, NY ☹

Great sign though, maybe NY will get it eventually

From: NANOG  On Behalf Of Joe Loiacono
Sent: Monday, June 13, 2022 10:55 AM
To: nanog@nanog.org
Subject: Re: Congrats to AS701


FiOS from Maryland (anonymized):

enp3s0: flags=4163  mtu 1500
inet 192.168.1.164  netmask 255.255.255.0  broadcast 192.168.1.255
inet6 fe80::b104:8f4d:e5b2:e13b  prefixlen 64  scopeid 0x20
inet6 2600:4040:b27f:cb00:a9b1:5f59::  prefixlen 64  scopeid 
0x0
inet6 2600:4040:b27f:cb00:24a8:7b31::  prefixlen 64  scopeid 
0x0
inet6 2600:4040:b27f:cb00:e1b6:8b83::  prefixlen 64  scopeid 
0x0
ether d0:67:e5:23:ec:fe  txqueuelen 1000  (Ethernet)
RX packets 2518066  bytes 1448982813 (1.4 GB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 2157395  bytes 260073952 (260.0 MB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

a@b:~$ ping 2607:f8b0:4004:c09::6a
PING 2607:f8b0:4004:c09::6a(2607:f8b0:4004:c09::6a) 56 data bytes
64 bytes from 2607:f8b0:4004:c09::6a: icmp_seq=1 ttl=59 time=24.0 ms
64 bytes from 2607:f8b0:4004:c09::6a: icmp_seq=2 ttl=59 time=17.6 ms
64 bytes from 2607:f8b0:4004:c09::6a: icmp_seq=3 ttl=59 time=20.4 ms
64 bytes from 2607:f8b0:4004:c09::6a: icmp_seq=4 ttl=59 time=23.4 ms
^C
--- 2607:f8b0:4004:c09::6a ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 17.618/21.351/23.983/2.555 ms


On 6/12/2022 1:55 PM, Christopher Morrow wrote:


On Sat, Jun 11, 2022 at 11:03 PM Darrel Lewis (darlewis) 
mailto:darle...@cisco.com>> wrote:
I, for one, am having a hard time finding the proper words to express the joy 
that I am feeling at this momentous moment!


It's quite amazing, I think... that it's taken so long to get to deployment you 
can actually see on the fios plant :)
I'd note I can't see the below on my homestead, but I can at a relative's 
(where the ifconfig data is from).

I also can't tell if the upstream will PD a block to the downstream... and the 
VZ CPE is 'not something I want to fiddle with',
because everytime I have tried at my house I've just taken it out behind the 
woodshed with a maul... and replaced it with
something I CAN configure successfully. (plus.. don't want that TR 069 in my 
home...)

-chris

-Darrel


On Jun 11, 2022, at 7:05 PM, Christopher Morrow 
mailto:morrowc.li...@gmail.com>> wrote:


Looks like FIOS customers may be getting ipv6 deployed toward them, finally:

ifconfig snippet from local machine:
inet6 2600:4040:2001:2200:73d2:6bcc:1e6b:43a1  prefixlen 64  scopeid 
0x0
inet6 2600:4040:2001:2200:e87:bf36:b6cb:6ce1  prefixlen 64  scopeid 
0x0

ping attempt:
  64 bytes from bh-in-f106.1e100.net 
(2607:f8b0:4004:c09::6a): icmp_seq=1 ttl=59 time=8.71 ms

8ms from mclean, va to ashburn, va isn't wondrous, but at least it's ipv6 (and 
marginally faster than ipv4)

Congrats to the 701 folk for deploying more widely!
  (note: I don't know exactly when this started, nor how wide it really is, but 
progress here is welcomed by myself at least :) )
-chris

RE: FCC proposes higher speed goals (100/20 Mbps) for USF providers

[Matthew Huff]

I grew up in rural Texas where my mother still lives. She has adequate speed 
internet, the biggest issue is reliability. The whole town (there is only 1 
provider) has an outage for about an hour every week. Two weeks ago, there was 
no internet for 3 days. Cellular service is 4G and not even that reliable for 
data even on the best days.

From: NANOG  On Behalf Of Brian Turnbow 
via NANOG
Sent: Tuesday, May 24, 2022 9:35 AM
To: David Bass ; Sean Donelan 
Cc: nanog@nanog.org
Subject: RE: FCC proposes higher speed goals (100/20 Mbps) for USF providers

Here in Italy there have been a lot of investments to get better broadband.
Such as government sponsored bundles for areas with no return on investments, 
for schools etc with a lot of focus on reaching gigabit speeds
The results have been mainly positive even though there are delays.
On the end user side in 2020 one of the largest ISPs started offering 2.5Gbps 
service
Adds all over and users started asking for it, even though they don’t have a 
2.5 nic or router,  so now all of the major providers are rolling it out.
Illiad one uped them a couple of months ago pushing  a 5Gbps service and now I 
get people asking me if we offer 5Gbps fiber lines.. pure marketing…
I have a 1Gbps/100Mbps line and it is plenty enough for the family rarely do we 
even get near the limits.
It’s kind of like when I ask for an Italian espresso in the states and get a 
cup full of coffee, no I just want a very small italian style espresso..
The response is Why? you are paying for it take it all
Bigger is better, even if you don’t need it, reigns supreme.

The real problem most users experience isn’t that they have a gig, or even 
100Mb of available download bandwidth…it’s that they infrequently are able to 
use that full bandwidth due to massive over subscription .

The other issue is the minimal upload speed.  It’s fairly easy to consume the 
10Mb that you’re typically getting as a residential customer.  Even “business 
class” broadband service has a pretty poor upload bandwidth limit.

We are a pretty high usage family, and 100/10 has been adequate, but there’s 
been times when we are pegged at the 10 Mb upload limit, and we start to see 
issues.

I’d say 25/5 is a minimum for a single person.

Would 1 gig be nice…yeah as long as the upload speed is dramatically increased 
as part of that.  We would rarely use it, but that would likely be sufficient 
for a long time.  I wouldn’t pay for the extra at this point though.

On Mon, May 23, 2022 at 8:20 PM Sean Donelan 
mailto:s...@donelan.com>> wrote:

Remember, this rulemaking is for 1.1 million locations with the "worst"
return on investment. The end of the tail of the long tail.  Rural and
tribal locations which aren't profitable to provide higher speed
broadband.

These locations have very low customer density, and difficult to serve.

After the Sandwich Isles Communications scandal, gold-plated proposals
will be viewed with skepticism.  While a proposal may have a lower total
cost of ownership over decades, the business case is the cheapest for
the first 10 years of subsidies.  [massive over-simplification]

Historically, these projects have lack of timely completion (abandoned,
incomplete), and bad (overly optimistic?) budgeting.

RE: V6 still not supported

[Matthew Huff]

Good to know. I’ll keep a look out for future implantations. Currently we are 
using Cisco 3548P-XL switches with low-latency nat to support microsecond 
latency natting. Hopefully someday they will support it.


Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mh...@ox.com<mailto:mh...@ox.com> | www.ox.com<http://www.ox.com>
...

From: Dave Bell 
Sent: Thursday, March 17, 2022 8:39 AM
To: Matthew Huff 
Cc: Tom Beecher ; b...@uu3.net; nanog@nanog.org
Subject: Re: V6 still not supported

You can still do NAT with IPv6 like you ask for. It's been around over a decade 
now:
https://datatracker.ietf.org/doc/html/rfc6296

On Thu, 17 Mar 2022 at 12:02, Matthew Huff mailto:mh...@ox.com>> 
wrote:
Did you read his email? He was saying that what a lot of people wanted was IPv4 
+ bigger address space, and not any other changes. Speaking for myself, other 
than the bigger address space, for a corporate/enterprise environment I have 
yet to see any advantages of IPv6 and many, many issues.

Information hiding, aka NAT is very important in the financial world.

For example, we have a directly peered connection with our clearing partner at 
our co-location. Both sides use NAT. This way either side can make a network 
change without having to involve the other partner.

Here is what happens without NAT (circa 2008), and yes, this really happened:


  1.  We needed to separate out process and move to a different server. The 
existing server IP couldn’t change since other partners had it in their access 
list.
  2.  It took 9 weeks and:

 *   3 conference calls including one with 30 participants from the 
clearing company
 *   2 approvals by committees at the clearing company



You can say that this was absurd and should be changed. Yes, but it won’t and 
we have no control over them.



With NAT we can change internal IP addresses at any time as long as we update 
the NAT tables.



This is not something important at ISP or hosting providers, but it’s many 
issues like this in the corporate world that prevent IPv6 adoption. Things like 
static addressing, consistent behavior of IPv6 across different operating 
systems including disabling SLAAC,  privacy addressing and others.



Based on my experience and people on tech mailing list that are oriented toward 
enterprises, I would bet that IPv6 deployment (with global addresses) is 
significantly less than 10% nor is it on their horizon.



Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mh...@ox.com<mailto:mh...@ox.com> | www.ox.com<http://www.ox.com>
...

From: NANOG mailto:ox@nanog.org>> On 
Behalf Of Tom Beecher
Sent: Thursday, March 17, 2022 7:41 AM
To: b...@uu3.net<mailto:b...@uu3.net>
Cc: nanog@nanog.org<mailto:nanog@nanog.org>
Subject: Re: V6 still not supported

“It seems all the market needed was IPv4 with bigger address space.
Instead of delivering it, some contraption has been created trying to solve
non-existant (or already fixed) problems.”

your argument against IPv6 is that they should have created a new version of 
IPv4, but bigger?

So… IPv6?

On Thu, Mar 17, 2022 at 06:32 mailto:b...@uu3.net>> wrote:
It seems team developing IPv6 had ONE way of doing things,
with is actually recipe for disaster. Why? Because they were building an IP
protocol. Something that will be using globally by ALL networks around.
Not some local IOT (useless) shit used here and there.
Thats why such IP protocol should be follow KISS concept and flexibility.
Some people have different vision how to run network. And because
Inter-net is an AS to AS network they should have right to do so.

In my opinion all that crypto stuff should be put layer upper because
crypto is hard, very hard and can get obsolete quickly.

Its same about other weird things embedded into IPv6 that probably
should go layer up. And now people wonder why IPv6 adoption is crap and
there is high resistance. IPv4 made mistakes too, but hell, it was the first.

It seems all the market needed was IPv4 with bigger address space.
Instead of delivering it, some contraption has been created trying to solve
non-existant (or already fixed) problems.

Just my 2 cents...


-- Original message --

From: William Allen Simpson 
mailto:william.allen.simp...@gmail.com>>
To: NANOG mailto:nanog@nanog.org>>
Subject: Re: V6 still not supported
Date: Wed, 16 Mar 2022 22:24:14 -0400

I'd wanted to clearly distinguish this from the old methods:

   This is intended to replace ARP, ICMP Router Advertisement, ICMP
   Redirect, ICMP Information, ICMP Mask, and OSPF Hello in the [IPv6]
   environm

RE: V6 still not supported

[Matthew Huff]

Did you read his email? He was saying that what a lot of people wanted was IPv4 
+ bigger address space, and not any other changes. Speaking for myself, other 
than the bigger address space, for a corporate/enterprise environment I have 
yet to see any advantages of IPv6 and many, many issues.

Information hiding, aka NAT is very important in the financial world.

For example, we have a directly peered connection with our clearing partner at 
our co-location. Both sides use NAT. This way either side can make a network 
change without having to involve the other partner.

Here is what happens without NAT (circa 2008), and yes, this really happened:


  1.  We needed to separate out process and move to a different server. The 
existing server IP couldn’t change since other partners had it in their access 
list.
  2.  It took 9 weeks and:
 *   3 conference calls including one with 30 participants from the 
clearing company
 *   2 approvals by committees at the clearing company



You can say that this was absurd and should be changed. Yes, but it won’t and 
we have no control over them.



With NAT we can change internal IP addresses at any time as long as we update 
the NAT tables.



This is not something important at ISP or hosting providers, but it’s many 
issues like this in the corporate world that prevent IPv6 adoption. Things like 
static addressing, consistent behavior of IPv6 across different operating 
systems including disabling SLAAC,  privacy addressing and others.



Based on my experience and people on tech mailing list that are oriented toward 
enterprises, I would bet that IPv6 deployment (with global addresses) is 
significantly less than 10% nor is it on their horizon.



Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mh...@ox.com<mailto:mh...@ox.com> | www.ox.com<http://www.ox.com>
...

From: NANOG  On Behalf Of Tom Beecher
Sent: Thursday, March 17, 2022 7:41 AM
To: b...@uu3.net
Cc: nanog@nanog.org
Subject: Re: V6 still not supported

“It seems all the market needed was IPv4 with bigger address space.
Instead of delivering it, some contraption has been created trying to solve
non-existant (or already fixed) problems.”

your argument against IPv6 is that they should have created a new version of 
IPv4, but bigger?

So… IPv6?

On Thu, Mar 17, 2022 at 06:32 mailto:b...@uu3.net>> wrote:
It seems team developing IPv6 had ONE way of doing things,
with is actually recipe for disaster. Why? Because they were building an IP
protocol. Something that will be using globally by ALL networks around.
Not some local IOT (useless) shit used here and there.
Thats why such IP protocol should be follow KISS concept and flexibility.
Some people have different vision how to run network. And because
Inter-net is an AS to AS network they should have right to do so.

In my opinion all that crypto stuff should be put layer upper because
crypto is hard, very hard and can get obsolete quickly.

Its same about other weird things embedded into IPv6 that probably
should go layer up. And now people wonder why IPv6 adoption is crap and
there is high resistance. IPv4 made mistakes too, but hell, it was the first.

It seems all the market needed was IPv4 with bigger address space.
Instead of delivering it, some contraption has been created trying to solve
non-existant (or already fixed) problems.

Just my 2 cents...


-- Original message --

From: William Allen Simpson 
mailto:william.allen.simp...@gmail.com>>
To: NANOG mailto:nanog@nanog.org>>
Subject: Re: V6 still not supported
Date: Wed, 16 Mar 2022 22:24:14 -0400

I'd wanted to clearly distinguish this from the old methods:

   This is intended to replace ARP, ICMP Router Advertisement, ICMP
   Redirect, ICMP Information, ICMP Mask, and OSPF Hello in the [IPv6]
   environment. There are also elements of the OSI ES-IS and IS-IS Hello.

We were forward looking to deployments of thousands of systems per link, rather
than the 30 maximum under then current ethernet standards.  We needed fewer
announcements, less chatty traffic, and more specific traffic designation.

We also prioritized network security.  Moreover requiring addresses be
ephemeral,
such that applications would not be able to tie authentication/authorization to
an
IPv6 address and would be motivated to use cryptographic security.

Unfortunately, later committees decided that rather than a single simpler
secured
address assignment scheme, we needed unsecured SLAAC and duplicate DHCPv6.
Three ways to do the same thing are always a recipe for disaster.

Reminder: I was an operator and one of the original NANOG members.

We spent a lot of time considering human factors.  I'd pioneered the
"Operational Considerations" section of the original draft RFCs.

IPv6 is a case s

RE: "Permanent" DST

[Matthew Huff]

They don't want their names on it when what happened in the 70s happens again. 
The effect of setting everything to DST and staying there is that in the 
winter, especially in the norther latitude it will be pitch dark during most of 
the morning when children get picked up at school bus stops. When the tragedy 
happens again, and it will, they will end up undoing this again...

History repeats itself, first as a tragedy, then as a farce...

Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mh...@ox.com | www.ox.com
...

-Original Message-
From: NANOG  On Behalf Of Jay R. Ashworth
Sent: Tuesday, March 15, 2022 5:30 PM
To: Tom Beecher 
Cc: nanog@nanog.org list 
Subject: Re: "Permanent" DST

Oh.  This was "Unanimous Consent"?  AKA "I want to vote for this, but *I do not 
want to be held responsible for having voted for it when it blows up*?"

I'd missed that; thanks.

- Original Message -
> From: "Tom Beecher" 
> To: "Eric Kuhnke" 
> Cc: "nanog@nanog.org list" 
> Sent: Tuesday, March 15, 2022 5:04:02 PM
> Subject: Re: "Permanent" DST

> I would say if something passes the United States Senate in our 
> current political environment by unanimous consent (which this did) , 
> I kinda feel like there won't be a ton of issues with everybody 
> figuring out how to line themselves up appropriately.
> 
> On Tue, Mar 15, 2022 at 5:01 PM Eric Kuhnke  wrote:
> 
>> That is true but at present everything business related in BC has a 
>> clear expectation of being in the same time zone as WA/OR/CA, and AB 
>> matches US Mountain time.
>>
>> On Tue, 15 Mar 2022 at 13:35, Paul Ebersman 
>> wrote:
>>
>>> eric> If Canada doesn't do the same thing at the same time, it'll be 
>>> eric> a real hassle, dealing with a change from -8 to -7 crossing 
>>> eric> the border between BC and WA, for instance. It has to be done 
>>> eric> consistently throughout North America.
>>>
>>> You must not have ever dealt with Indiana, where it was DST or not 
>>> by choice per county. It wasn't quite the cluster***k you'd think.
>>>

-- 
Jay R. Ashworth  Baylink   j...@baylink.com
Designer The Things I Think   RFC 2100
Ashworth & Associates   http://www.bcp38.info  2000 Land Rover DII
St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274

identity.cisco.com certificate has expired

[Matthew Huff]

Arghh...

Just an FYI, id.cisco.com is fubar'ed. Hopefully cisco has already fixed it and 
the proxies/caches/cdns just need to timeout, but just in case anyone knows a 
contact at Cisco's ops group...

[cid:image001.png@01D8309A.75491410]


Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mh...@ox.com<mailto:mh...@ox.com> | www.ox.com<http://www.ox.com>
...

RE: Never push the Big Red Button (New York City subway failure)

[Matthew Huff]

Since we are telling power horror stories…


How about the call from the night operator that arrived at 10:00pm asking “Is 
there any reason there is no power in the data center?”

Turns out someone had plugged in a new high end workgroup laser printer to the 
outside wall of the datacenter. The power receptacle was wired into the data 
center’s UPS and completely smoked the UPS. Luckily the static transfer 
switched worked, but the three mainframes weren’t’ happy…


Or

Our building had a major ground fault issue that took years to find and 
resolve. We got hit with lightning that caused the mainframe to fault and 
recycle…and two minutes in, we got hit by lightning again. When the system 
failed to start, we called IBM support. When we explained what happened there 
was a very long pause…then some mumbling off phone, then the manager got on the 
line and said someone would be flying out and be onsite within 12 hours. We 
were down for 3 days, and got fined $250,000 by the insurance regulators since 
we couldn’t pay claims.

Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mh...@ox.com<mailto:mh...@ox.com> | www.ox.com<http://www.ox.com>
...

From: Chris Kane 
Sent: Friday, September 10, 2021 3:16 PM
To: Christopher Morrow 
Cc: Matthew Huff ; nanog@nanog.org
Subject: Re: Never push the Big Red Button (New York City subway failure)

True EPO story; maintenance crew carrying new drywall into the data center 
backed into the EPO that didn't have a cover on it. One of the most eerie 
sounds in networking...a completely silent data center.

-chris

On Fri, Sep 10, 2021 at 2:48 PM Christopher Morrow 
mailto:morrowc.li...@gmail.com>> wrote:


On Fri, Sep 10, 2021 at 1:49 PM Matthew Huff 
mailto:mh...@ox.com>> wrote:
Reminds me of something that happened about 25 years ago when an elementary 
school visited our data center of the insurance company where I worked. One of 
our operators strategically positioned himself between the kids and the 
mainframe, leaned back and hit it's EPO button.

Or when your building engineering team cuts themselves a new key for the 'main 
breaker' for the facility... and tests it at 2pm on a tuesday.
Or when that same team cuts a second key (gotta have 2 keys!) and tests that 
key on the same 'main breaker' ... at 2pm on the following tuesday.



not fakenews, a real story from a large building full of gov't employees and 
computers and all manner of 'critical infrastructure' for the agency occupying 
said building.

Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mh...@ox.com<mailto:mh...@ox.com> | www.ox.com<http://www.ox.com>
...

-Original Message-
From: NANOG mailto:ox@nanog.org>> On 
Behalf Of Sean Donelan
Sent: Friday, September 10, 2021 12:38 PM
To: nanog@nanog.org<mailto:nanog@nanog.org>
Subject: Never push the Big Red Button (New York City subway failure)

NEW YORK CITY TRANSIT RAIL CONTROL CENTER POWER
OUTAGE ISSUE ON AUGUST 29, 2021
Key Findings
September 8, 2021


https://www.governor.ny.gov/sites/default/files/2021-09/WSP_Key_Findings_Summary-for_release.pdf

Key Findings
[...]

3. Based on the electrical equipment log readings and the manufacturer’s
official assessment, it was determined that the most likely cause of RCC
shutdown was the “Emergency Power Off” button being manually activated.

Secondary Findings

1. The “Emergency Power Off” button did not have a protective cover at the
time of the shutdown or the following WSP investigation.

[...]
Mitigation Steps

1. Set up the electrical equipment Control and Communication systems
properly to stay active so that personnel can monitor RCC electrical
system operations.

[...]


--
Chris Kane

RE: Never push the Big Red Button (New York City subway failure)

[Matthew Huff]

Reminds me of something that happened about 25 years ago when an elementary 
school visited our data center of the insurance company where I worked. One of 
our operators strategically positioned himself between the kids and the 
mainframe, leaned back and hit it's EPO button.

Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mh...@ox.com | www.ox.com
...

-Original Message-
From: NANOG  On Behalf Of Sean Donelan
Sent: Friday, September 10, 2021 12:38 PM
To: nanog@nanog.org
Subject: Never push the Big Red Button (New York City subway failure)

NEW YORK CITY TRANSIT RAIL CONTROL CENTER POWER
OUTAGE ISSUE ON AUGUST 29, 2021
Key Findings
September 8, 2021


https://www.governor.ny.gov/sites/default/files/2021-09/WSP_Key_Findings_Summary-for_release.pdf

Key Findings
[...]

3. Based on the electrical equipment log readings and the manufacturer’s 
official assessment, it was determined that the most likely cause of RCC 
shutdown was the “Emergency Power Off” button being manually activated.

Secondary Findings

1. The “Emergency Power Off” button did not have a protective cover at the 
time of the shutdown or the following WSP investigation.

[...]
Mitigation Steps

1. Set up the electrical equipment Control and Communication systems 
properly to stay active so that personnel can monitor RCC electrical 
system operations.

[...]

RE: The great Netflix vpn debacle! (geofeeds)

[Matthew Huff]

IPv6 tunnels work great for network geeks, but rather poorly for home users 
with streaming, gaming etc...It's not necessarily the performance, it's either 
the geolocation, latency, or the very issue that started this thread - VPN 
banning.

Remember, the streaming services couldn't care less about geolocation or VPN 
banning, it's the contractual obligations with the content providers. The 
content providers care about vpn banning because it gets around geolocation, 
which interferes with their business models (different release schedules to 
different regions, etc..)

Been there, done that...Stuck on Fios with no IPv6. Ran into rather 
"interesting" problems with various streaming services with IPv6 configured.


Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mh...@ox.com | www.ox.com
...

-Original Message-
From: NANOG  On Behalf Of Michael Thomas
Sent: Wednesday, September 1, 2021 2:26 PM
To: Nimrod Levy ; Owen DeLong 
Cc: nanog@nanog.org
Subject: Re: The great Netflix vpn debacle! (geofeeds)


On 9/1/21 10:59 AM, Nimrod Levy wrote:
> All this chatter about IPv6 support on devices is fun and all, but 
> there are providers still not on board.
> They operate in my neighborhood and they know who they are...
>
This is about inside your premise before any NAT's enter the picture. 
What would be nice is if home routers offered up v6 as the default way 
to number and v6 tunnels past ISP's that don't have v6. Home routers 
could make that all rather seamless where users wouldn't need to know 
that was happening. It's really a pity that home routers are a race to 
the bottom where everything else with networking is expected to evolve 
over time.

Mike

Re: Disney+ Geolocation issues

[Matthew Huff]

It’s not about optimization, it’s about the contract with the content 
providers. The agreement is to restrict content by geographical regions mainly 
for marketing purposes. They block VPN access to keep people from bypassing 
those restrictions. It’s true of all the streaming providers.

> On Nov 13, 2019, at 9:44 AM, Robert Blayzor  wrote:
> 
> On 11/12/19 5:28 PM, Michael Crapse wrote:
>> Myself and a few other ISPs are having our eyeballs complain about
>> disney+ saying that they're on a VPN. Does anyone have any idea, or who
>> to contact regarding this issue?
>> This is most likely improper geolocation databases. Anyone have an idea
>> who they use?
>> 
> 
> 
> Same boat here. ARIN ISP with all valid SWIP clearly showing stateside
> USA. So who knows what Disney+ is doing to block their viewers. Seems
> rather silly to block viewing based on the connecting IP address.
> Wouldn't you base it on the authorized viewer who is logged in and using
> the service? I mean, that's what they are paying for. I get the whole
> CDN steering thing, but the error message message sent back to the
> viewer should not be to "call your ISP". Now you have support desks
> taking thousands of worthless calls
> 
> ESPN+ is guilty of the same garbage
> 
> -- 
> inoc.net!rblayzor
> XMPP: rblayzor.AT.inoc.net
> PGP:  https://pgp.inoc.net/rblayzor/

SFP oraganizers / storage recommendations

[Matthew Huff]

Any recommendations to keep track of different SFP and keep them organized? Any 
storage boxes / trays designed for SFPs?

RE: This DNS over HTTP thing

[Matthew Huff]

>From a corporate standpoint, this is exactly correct. There are also some 
>regulatory issues involved (FINRA, SEC, etc...)

We are required to block access to web based email (gmail, etc...) in our 
corporate network (please don't ask why, ours is not to reason why...), so 
every method to "bypass" normal network operations creates headaches for us.



-Original Message-
From: NANOG  On Behalf Of John R. Levine
Sent: Tuesday, October 1, 2019 4:06 PM
To: Aaron C. de Bruyn 
Cc: NANOG mailing list 
Subject: Re: This DNS over HTTP thing

I assumed my point was obvious but evidently I overestimated my audience.

While it is stupid to assert that the only reason to circumvent DNS filters is 
to look at child abuse material, it is equally stupid to assert that the only 
reason to filter is to lie, or to censor.

There are plenty of good reasons to filter DNS responses, with the most obvious 
being to block malware sites whose links are sent out in spam (a whole lot of 
spam these days.)  There are also reasons that enterprises filter DNS on their 
networks, to block stuff that creates a hostile work environment, or is 
obviously unrelated to what employees are hired to do (i.e., facebook.)

R's,
John


On Tue, 1 Oct 2019, Aaron C. de Bruyn wrote:

> "For the children!"
> "Stop resisting!"
> "I was in fear for my life!"
>
> The age-old cries of the oppressor. ...

> On Tue, Oct 1, 2019 at 11:33 AM John Levine  wrote:
>
>> In article <20191001074011.n4xjouqg6lhsv...@nic.fr> you write:
>>> Note that the UK is probably the country in Europe with the biggest
>>> use of lying DNS resolvers for censorship. No wonder that the people
>>> who censor don't like anti-censorship techniques.
>>
>> Most UK ISPs use the Internet Watch Foundation's advice intended to
>> block child sexual abuse material.
>>
>> Circumventing it enables people to access that material.
>>
>> We can shout CHILD PORNOGRAPHY just as loud as you can shout
>> CENSORSHIP so perhaps we should both stop now.  There are plenty of
>> valid reasons for a DNS resolver to block some results.

Regards,
John Levine, jo...@taugh.com, Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

RE: Intermittent "bad gateway"

[Matthew Huff]

We got reports on that on some cloudflare sites, but it disappeared pretty 
quickly. Looks like a CDN issue.

-Original Message-
From: NANOG  On Behalf Of Stephen Satchell
Sent: Tuesday, July 2, 2019 10:17 AM
To: nanog@nanog.org
Subject: Intermittent "bad gateway"

Are we having another BGP problem this morning?

RE: CenturyLink

[Matthew Huff]

There isn't a specific regulation on free-running GPS, just "due diligence". I 
work at a algorithmic program trading company (and have been for 20 years). We 
have a high ROI, the cost differential for the rubidium OC versus having to 
drop everything to conform to regulatory requirements due to a short GPS 
outage, makes this a no-brainer.

----
Matthew Huff | 1 Manhattanville Rd 
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039


-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Saku Ytti
Sent: Monday, December 31, 2018 3:28 AM
To: Gary E. Miller 
Cc: nanog@nanog.org
Subject: Re: CenturyLink

Hey Gary,

On Mon, 31 Dec 2018 at 05:02, Gary E. Miller  wrote:

> The Rb frequency reference will be two or three orders of magnitude 
> more stable than an expensive ovenized crystal.

Perhaps, but not supported by this:
https://www.meinbergglobal.com/english/specs/gpsopt.htm

For the tl;dr folk, crystal drifts +-4.5us per day, Rb +-1.1us (both seem like 
unsatisfactorily high numbers to me, i.e. you don't want to be free-running 24h 
with Rb). Luckily today we have GPS, Glonass, BeiDou, Galileo and couple 
smaller ones, so there should be somewhat reasonable amount of redundancy. 
Unsure which commercially available NTP or PPP master clocks support all four.

But I of course readily accept Rb is objectively more accurate than crystal, 
I'm just curious where it matters and I'm curious which regulation applies, who 
fall under the regulation and what specifically does the regulation require 
about free-running accuracy.

--
  ++ytti

RE: CenturyLink

[Matthew Huff]

Regulatory.

If we were to lose the GPS signal (antenna failure, etc...) then our stratum 1 
time sources wouldn't drift as much and as quickly. For telco and general 
usage, the cost may not be worthwhile, but when you have auditors looking over 
your shoulder


Matthew Huff | 1 Manhattanville Rd 
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039


-Original Message-
From: Saku Ytti [mailto:s...@ytti.fi] 
Sent: Sunday, December 30, 2018 12:30 PM
To: Matthew Huff 
Cc: Shawn L ; nanog@nanog.org
Subject: Re: CenturyLink

Hey Matthew,

> We use an older model of 
> https://www.microsemi.com/product-directory/enterprise-network-time-servers/4117-syncserver-s600
>  with rubidium oscillator. Not cheap, but hardened and extremely accurate.

Out of interest why rubidium? For short term stability oscillator choice isn't 
much of a matter for free running even rubidium will be off good +-1us per day 
or +-30us per week. I've always wondered what niche rubidium covers that isn't 
handled by much cheaper crystal ovens or actually precise oscillators.

--
  ++ytti

RE: CenturyLink

[Matthew Huff]

We use an older model of 
https://www.microsemi.com/product-directory/enterprise-network-time-servers/4117-syncserver-s600
 with rubidium oscillator. Not cheap, but hardened and extremely accurate.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039

From: Shawn L [mailto:sha...@up.net]
Sent: Sunday, December 30, 2018 9:40 AM
To: nanog@nanog.org
Cc: Matthew Huff ; l...@satchell.net
Subject: Re: CenturyLink


Speaking of GPS-enabled NTP appliances, etc. wondering what hardware people are 
using for this.



thanks



-Original Message-
From: "Raymond Burkholder" mailto:r...@oneunified.net>>
Sent: Saturday, December 29, 2018 12:01pm
To: "Matthew Huff" mailto:mh...@ox.com>>, 
"l...@satchell.net<mailto:l...@satchell.net>" 
mailto:l...@satchell.net>>, 
"nanog@nanog.org<mailto:nanog@nanog.org>" 
mailto:nanog@nanog.org>>
Subject: Re: CenturyLink

On 2018-12-29 7:51 a.m., Matthew Huff wrote:
> We have two stratum-1 servers synced with GPS and a PTP feed from a provider 
> that also provides PTP to market data systems, but we still have to monitor 
> drift between system time and NIST time. Don't ask for the logic behind it, 
> it's a regulation, not a technical requirement.
>
On one occasion, due to bad firmware or a configuration issue, I have
seen GPS stratum 1 diverge from NTP.  It was somewhat eye brow raising
to the company.  My NTP monitored servers were shown to be diverging
their GPS/NTP, but after looking at twice or thrice, it was the other
way around.

RE: CenturyLink

[Matthew Huff]

Actually, on all our trading systems, our times are synced via PTP instead of 
ntpd for at least 50 microsecond accuracy. The stratum 1 clocks as well as NIST 
time are only used as comparison to verify compliance and reality. We use ntpq 
to determine the offset  from NIST for reporting.


Matthew Huff | 1 Manhattanville Rd 
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039


-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Stephen Satchell
Sent: Saturday, December 29, 2018 10:01 AM
To: nanog@nanog.org
Subject: Re: CenturyLink

On 12/29/18 6:51 AM, Matthew Huff wrote:
> We have two stratum-1 servers synced with GPS and a PTP feed from a provider 
> that also provides PTP to market data systems, but we still have to monitor 
> drift between system time and NIST time. Don't ask for the logic behind it, 
> it's a regulation, not a technical requirement.

Having been a participant on Standards Working Groups, I understand completely. 
 Regulations, like Standards, need to be written to apply to as wide a 
population as possible.  Do those regulations dictate how you monitor drift?  
For example, can you have a system (I would use CentOS) that syncs to the 
appliance as well as NIST and your inside NTP sources, and use ntpq(8) to read 
the drift directly?

RE: CenturyLink

[Matthew Huff]

We have two stratum-1 servers synced with GPS and a PTP feed from a provider 
that also provides PTP to market data systems, but we still have to monitor 
drift between system time and NIST time. Don't ask for the logic behind it, 
it's a regulation, not a technical requirement.


Matthew Huff | 1 Manhattanville Rd 
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039


-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Stephen Satchell
Sent: Saturday, December 29, 2018 9:34 AM
To: nanog@nanog.org
Subject: Re: CenturyLink

On 12/28/18 3:23 PM, Yang Yu wrote:
> On Fri, Dec 28, 2018 at 12:05 AM Stephane Bortzmeyer  
> wrote:
>> Is this problem also responsible for the 911 outage? If so, the 
>> post-mortem analysis is not useful only for CenturyLink customers but 
>> for everyone on the west coast.
> 
> Looks like most time.nist.gov servers (3 x NIST sites on AS49) are 
> single homed on CenturyLink, anyone noticed NTP issues yesterday?
> 
> https://tf.nist.gov/tf-cgi/servers.cgi
> 

I have GPS-based Stratum 1 NTP appliances in my network, so I wouldn't see any 
issues.  I suspect many other operators are in the same situation.

RE: CenturyLink

[Matthew Huff]

Yep.

We are required by FINRA to verify that our clocks on our trading systems are 
within a certain tolerance of NIST time. We are still seeing issues with 3 of 
the NIST servers this morning. Since NIST is on a skeleton crew anyway due to 
the government shutdown, I don't expect any resolution shortly.


Matthew Huff | 1 Manhattanville Rd 
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039


-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Yang Yu
Sent: Friday, December 28, 2018 6:23 PM
To: Stephane Bortzmeyer 
Cc: nanog@nanog.org
Subject: Re: CenturyLink

On Fri, Dec 28, 2018 at 12:05 AM Stephane Bortzmeyer  wrote:
> Is this problem also responsible for the 911 outage? If so, the 
> post-mortem analysis is not useful only for CenturyLink customers but 
> for everyone on the west coast.

Looks like most time.nist.gov servers (3 x NIST sites on AS49) are single homed 
on CenturyLink, anyone noticed NTP issues yesterday?

https://tf.nist.gov/tf-cgi/servers.cgi

RE: Verizon: Extremely Strange CPE Routing in NYC/NJ Area

[Matthew Huff]

Windows traceroute uses ICMP by default, Unix based systems use UDP. For 
example from my Mac on Fios in Westchester, NY (and yes, they are doing 
something  with ICMP traceroutes, possibly within their gateway box):


acpro:Documents mhuff$ sudo traceroute www.yahoo.com
traceroute: Warning: www.yahoo.com has multiple addresses; using 72.30.35.9
traceroute to atsv2-fp-shed.wg1.b.yahoo.com (72.30.35.9), 64 hops max, 52 byte 
packets
 1  firewall (10.1.1.1)  0.916 ms  0.419 ms  0.394 ms
 2  * * *
 3  b3369.nycmny-lcr-22.verizon-gni.net (130.81.16.102)  3.939 ms  10.172 ms
b3369.nycmny-lcr-21.verizon-gni.net (130.81.16.100)  5.192 ms
 4  * * *
 5  0.ae4.xl2.buf4.alter.net (140.222.228.143)  13.500 ms *  12.970 ms
 6  0.xe-5-3-0.gw1.buf4.alter.net (152.63.26.62)  12.390 ms  12.122 ms
0.ae4.xl2.buf4.alter.net (140.222.228.143)  12.704 ms
 7  0.xe-4-2-0.gw1.buf4.alter.net (152.63.26.46)  14.326 ms  13.069 ms
verizon.com.customer.alter.net (204.148.47.162)  14.436 ms
 8  et-19-0-0.pat1.bfz.yahoo.com (216.115.97.103)  13.990 ms
unknown-72-30-223-x.yahoo.com (72.30.223.3)  14.220 ms
et-19-0-0.pat1.bfz.yahoo.com (216.115.97.103)  13.711 ms
 9  et-0-0-0.msr2.bf1.yahoo.com (74.6.227.137)  12.098 ms
et-0-0-0.msr1.bf1.yahoo.com (74.6.227.129)  13.388 ms
et-18-0-1.msr1.bf1.yahoo.com (74.6.227.131)  14.457 ms
10  et-19-0-0.clr2-a-gdc.bf1.yahoo.com (74.6.122.37)  14.554 ms
et-0-0-0.msr1.bf1.yahoo.com (74.6.227.129)  14.810 ms
et-1-1-1.msr1.bf1.yahoo.com (74.6.227.135)  13.502 ms
11  eth-18-3.bas2-1-flk.bf1.yahoo.com (98.139.128.75)  14.607 ms  16.608 ms
eth-18-3-bas1-1-flk.bf1.yahoo.com (98.139.128.73)  15.161 ms
12  media-router-fp1.prod1.media.vip.bf1.yahoo.com (72.30.35.9)  13.470 ms
eth-17-3.bas2-1-flk.bf1.yahoo.com (98.139.128.71)  13.582 ms
media-router-fp1.prod1.media.vip.bf1.yahoo.com (72.30.35.9)  12.952 ms

macpro:Documents mhuff$ sudo traceroute -I www.yahoo.com
traceroute: Warning: www.yahoo.com has multiple addresses; using 72.30.35.10
traceroute to atsv2-fp-shed.wg1.b.yahoo.com (72.30.35.10), 64 hops max, 72 byte 
packets
 1  firewall (10.1.1.1)  0.675 ms  0.347 ms  0.322 ms
 2  media-router-fp2.prod1.media.vip.bf1.yahoo.com (72.30.35.10)  2.456 ms  
21.139 ms  12.834 ms


Matthew Huff | 1 Manhattanville Rd 
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039


-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Lee
Sent: Thursday, November 29, 2018 5:42 PM
To: Nick Zurku 
Cc: Brendan Mannella ; nanog@nanog.org; Todd Kammerer 

Subject: Re: Verizon: Extremely Strange CPE Routing in NYC/NJ Area

On 11/29/18, Nick Zurku  wrote:
> Can anyone from Verizon take a look at this behavior for us?
>
> We’re having multiple Verizon FiOS users in the NYC/NJ area appear to 
> teleport from their FiOS router to our IP in the Pittsburgh region.

Verizon is doing something seriously weird to windows traceroute:
C:\Users\Lee>tracert www.yahoo.com

Tracing route to atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232] over a maximum 
of 30 hops:

  1<1 ms<1 ms<1 ms  fw.home.net
  2 1 ms<1 ms<1 ms  vbz-router.home.net [192.168.1.1]
  3 8 ms 3 ms 6 ms
media-router-fp2.prod1.media.vip.ne1.yahoo.com [98.138.219.232]

Trace complete.

C:\Users\Lee>ping -i 12 www.yahoo.com.

Pinging atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232] with 32 bytes of data:
Reply from 98.138.0.87: TTL expired in transit.
Reply from 98.138.0.87: TTL expired in transit.
Reply from 98.138.0.87: TTL expired in transit.
Reply from 98.138.0.87: TTL expired in transit.

Ping statistics for 98.138.219.232:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

C:\Users\Lee>ping -i 13 www.yahoo.com.

Pinging atsv2-fp-shed.wg1.b.yahoo.com [98.138.219.232] with 32 bytes of data:
Reply from 98.138.219.232: bytes=32 time=32ms TTL=54 Reply from 98.138.219.232: 
bytes=32 time=31ms TTL=54 Reply from 98.138.219.232: bytes=32 time=33ms TTL=54 
Reply from 98.138.219.232: bytes=32 time=33ms TTL=54

Ping statistics for 98.138.219.232:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip 
times in milli-seconds:
Minimum = 31ms, Maximum = 33ms, Average = 32ms

C:\Users\Lee>

traceroute from a linux box works as expected..

Lee

> Users
> are seeing extreme slowness with TCP traffic, but ping times seem 
> reasonable.
>
> User 1:
>  1  fios_quantum_gateway (192.168.1.1)  1.575 ms  2.426 ms  3.193 ms
>  2  204.16.244.8 (204.16.244.8)  2.269 ms  3.055 ms  2.727 ms
>
> User 2:
>  1  fios_quantum_gateway (192.168.1.1)  1.565 ms  1.048 ms  0.947 ms
>  2  204.16.244.8 (204.16.244.8)  2.162 ms  3.588 ms  3.048 ms
>
> I can provide end-user NYC/NJ IPs off-list if desirable.
>
>
> Here's a normal looking trace from an FiOS line locally in the 
> Pittsburgh
> region:
>
>
> IP:  108.39.

RE: Oct. 3, 2018 EAS Presidential Alert test

[Matthew Huff]

I received it on my iPhone XS Max running iOS 12.0 with AT, wifi calling 
off...


Matthew Huff | 1 Manhattanville Rd 
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039


-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Andy Ringsmuth
Sent: Wednesday, October 3, 2018 2:53 PM
To: nanog@nanog.org
Subject: Oct. 3, 2018 EAS Presidential Alert test

Did anyone on AT or an iPhone receive the test today? I believe it was 
supposed to happen at 2:18 EDT, followed by one on broadcast radio at 2:20 EDT.

I’m in CDT, so 1:18 and 1:20 p.m. CDT.

Message was heard on my desk radio at 1:21:35 p.m. CDT but as of the sending of 
this at 1:52 p.m. CDT, nothing on phones. I have an office full of AT iPhones 
and not a single one of them alerted.

FEMA says https://www.fema.gov/emergency-alert-test

"Cell towers will broadcast the WEA test for approximately 30 minutes beginning 
at 2:18 p.m. EDT. During this time, WEA compatible cell phones that are 
switched on, within range of an active cell tower, and whose wireless provider 
participates in WEA should be capable of receiving the test message. Some cell 
phones will not receive the test message, and cell phones should only receive 
the message once."

My wife, with a Sprint iPhone, received the test.



Andy Ringsmuth
5609 Harding Drive
Lincoln, NE 68521-5831
(402) 304-0083
a...@andyring.com

RE: Console Servers

[Matthew Huff]

If anyone is looking for a product that is reasonably priced and is still being 
produced/update, the ADVA Optical (aka MRV, aka Xyplex) console servers still 
work great

https://www.advaoptical.com/en/products/network-infrastructure-assurance/lx-series

From their specs:
4, 8, 16, 32 and 48 serial ports
V.92 modem option
Single or dual power
120-240VAC, 50/60Hz: 0.5A per system
36-72VDC dual feed: 0.75A per system
2 x Ethernet
NEBS Level 3 certified

LCD KVM console pullout that supports display port ???

[Matthew Huff]

Anyone have any recommendations for a 16-17" LCD keyboard/mouse combo pull-out 
tray that supports DisplayPort/USB as an input?


Matthew Huff | 1 Manhattanville Rd 
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039

RE: Opensource SNMP Trap Receivers ???

[Matthew Huff]

Oh hell yes, there isn’t anything simple about SNMP. A number of people have 
very quickly suggested SNMPTT, which is the sort of product I was looking for. 
My google foo had failed me. Thanks.


Matthew Huff | 1 Manhattanville Rd 
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039

From: kscott.he...@gmail.com [mailto:kscott.he...@gmail.com] On Behalf Of K. 
Scott Helms
Sent: Tuesday, February 13, 2018 8:09 AM
To: Matthew Huff <mh...@ox.com>
Cc: nanog <nanog@nanog.org>
Subject: Re: Opensource SNMP Trap Receivers ???

Matthew,

Sadly, open source + SNMP traps != simple.

The simplest option that I've personally used in the past is SNMPTT with Nagios.

https://paulgporter.net/2013/09/16/nagios-snmp-traps/

http://www.snmptt.org/docs/snmptt.shtml

The main problem is that SNMP traps, like most of SNMP, aren't simple despite 
the name.  Having said that it can be done especially if the gear doesn't 
change too often.

Scott Helms

On Tue, Feb 13, 2018 at 7:44 AM, Matthew Huff <mh...@ox.com> wrote:
We are retiring a legacy SNMP system and are looking for a simple, opensource 
SNMP trap receiver/alerting system. We aren't looking for a full SNMP system, 
just something that will receive snmp traps and email/alert based on them.

1) Looking for something off the shelf, not a development project
2) Opensource or low cost
3) SNMP MIB compiler

Any suggestions?


Matthew Huff             | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC       | Phone: 914-460-4039

Opensource SNMP Trap Receivers ???

[Matthew Huff]

We are retiring a legacy SNMP system and are looking for a simple, opensource 
SNMP trap receiver/alerting system. We aren't looking for a full SNMP system, 
just something that will receive snmp traps and email/alert based on them.

1) Looking for something off the shelf, not a development project
2) Opensource or low cost
3) SNMP MIB compiler

Any suggestions?


Matthew Huff | 1 Manhattanville Rd 
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039

RE: FW: Reliability of looking glass sites / rviews

[Matthew Huff]

ASN 14607, and 129.77.0.0/16

After slightly over an hour after our power event where 100% of our equipment 
was down, this is what I saw at routeviews

BGP routing table entry for 129.77.0.0/16, version 24978989
Paths: (7 available, best #7, table default)
  Not advertised to any peer
  Refresh Epoch 1
  134708 3491 6939 46887 14607
    103.197.104.1 from 103.197.104.1 (123.108.254.70)
      Origin IGP, localpref 100, valid, external
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
   1273 6939 46887 14607
    193.0.0.56 from 193.0.0.56 (193.0.0.56)
      Origin IGP, localpref 100, valid, external
      Community: 1273:23000
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
  8283 57866 6762 6939 46887 14607
    94.142.247.3 from 94.142.247.3 (94.142.247.3)
      Origin IGP, metric 0, localpref 100, valid, external
      Community: 6762:33 6762:16500 8283:15 57866:105
      unknown transitive attribute: flag 0xE0 type 0x20 length 0xC
        value  205B  0006  000F 
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
  24441 3491 3491 6939 46887 14607
    202.93.8.242 from 202.93.8.242 (202.93.8.242)
      Origin IGP, localpref 100, valid, external
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
  20912 1267 1273 6939 46887 14607
    212.66.96.126 from 212.66.96.126 (212.66.96.126)
      Origin IGP, localpref 100, valid, external
      Community: 1273:23000 9035:50 9035:100 20912:65001
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
  1221 4637 6939 46887 14607
    203.62.252.83 from 203.62.252.83 (203.62.252.83)
      Origin IGP, localpref 100, valid, external
      rx pathid: 0, tx pathid: 0
  Refresh Epoch 1
  2497 6939 46887 14607
    202.232.0.2 from 202.232.0.2 (202.232.0.2)
      Origin IGP, localpref 100, valid, external, best
      rx pathid: 0, tx pathid: 0x0


From: Tim Evens [mailto:t...@snas.io] 
Sent: Friday, September 15, 2017 10:45 AM
To: Matthew Huff <mh...@ox.com>
Cc: morrowc.li...@gmail.com; nanog@nanog.org
Subject: Re: FW: Reliability of looking glass sites / rviews

You didn't mention details about which ASN or prefixes you were checking.  Are 
you referring to ASN 14607 that only advertises two prefixes 129.77.0.0/16 and 
2620:0:2810::/48?

Based what we see over the weekend (using routeviews data), we see:

Event Start Time: 2017-09-09 11:29:23 UTC (2017-09-09 07:29:23 EDT)
Event End Time: 2017-09-09 13:31:30 UTC (2017-09-09 09:31:30 EDT)

Are the above times correct?

We see the routes withdraw and then come back.   For example: 
http://demo-rv.snas.io:3000/dashboard/db/prefix-history?orgId=2=129.77.0.0_len=16_num=All_name=All_name=All=150490800=150520320

When you checked routeviews, which router and peer were you looking at?  When 
you did a "show ip bgp ..." did you include the prefix length? If not, it would 
have then shown you 0/0 or 128/5, depending on which router you were on.


--Tim 





On 9/13/17, 8:43 AM, "NANOG on behalf of Matthew Huff" <nanog-boun...@nanog.org 
on behalf of mh...@ox.com> wrote:

Both should have been similar.

In the first case we lost power to all of our BGP border routers that are 
peered with the upstream providers
In the second case, I did an explicit “shut” on the interface connected to 
the upstream provider that appeared “stuck” after an hour after the outage.

From: <christopher.mor...@gmail.com> on behalf of Christopher Morrow 
<morrowc.li...@gmail.com>
Date: Wednesday, September 13, 2017 at 10:58 AM
To: Matthew Huff <mh...@ox.com>
Cc: nanog2 <nanog@nanog.org>
Subject: Re: Reliability of looking glass sites / rviews
    
    

On Wed, Sep 13, 2017 at 5:30 AM, Matthew Huff 
<mh...@ox.com<mailto:mh...@ox.com>> wrote:
This weekend our uninterruptible power supply became interruptible and we 
lost all circuits. While I was doing initial debugging of the problem while I 
waited on site power verification, I noticed that there was still paths being 
shown in rviews for the circuit that were down. This was over an hour after we 
went hard down and it took hours before we were back up.

explicit vs implicit withdrawals causing different handling of the problem 
routes?

I worked with our providers last night to verify there weren't any hanging 
static routes, etc... We shut the upstream circuit down and watched the 
convergence and saw that eventually all the paths disappeared. Given what we 
saw on Saturday, what would cause route-views to cache the paths that long?  
Some looking glass sites only show what they are peered with or at most what 
their peers are peered with, that's why I've always used route-views.

What looking glass sites other than route-views would people recommend?

ripe ris.


 
 

Re: Reliability of looking glass sites / rviews

[Matthew Huff]

Both should have been similar.

In the first case we lost power to all of our BGP border routers that are 
peered with the upstream providers
In the second case, I did an explicit “shut” on the interface connected to the 
upstream provider that appeared “stuck” after an hour after the outage.

From: <christopher.mor...@gmail.com> on behalf of Christopher Morrow 
<morrowc.li...@gmail.com>
Date: Wednesday, September 13, 2017 at 10:58 AM
To: Matthew Huff <mh...@ox.com>
Cc: nanog2 <nanog@nanog.org>
Subject: Re: Reliability of looking glass sites / rviews



On Wed, Sep 13, 2017 at 5:30 AM, Matthew Huff 
<mh...@ox.com<mailto:mh...@ox.com>> wrote:
This weekend our uninterruptible power supply became interruptible and we lost 
all circuits. While I was doing initial debugging of the problem while I waited 
on site power verification, I noticed that there was still paths being shown in 
rviews for the circuit that were down. This was over an hour after we went hard 
down and it took hours before we were back up.

explicit vs implicit withdrawals causing different handling of the problem 
routes?

I worked with our providers last night to verify there weren't any hanging 
static routes, etc... We shut the upstream circuit down and watched the 
convergence and saw that eventually all the paths disappeared. Given what we 
saw on Saturday, what would cause route-views to cache the paths that long?  
Some looking glass sites only show what they are peered with or at most what 
their peers are peered with, that's why I've always used route-views.

What looking glass sites other than route-views would people recommend?

ripe ris.

Getting an RADB entry removed that was added by a previous peer

[Matthew Huff]

It appears that Reliance Globalcom  (AS6157) added an RADB entry for our prefix 
(129.77.0.0/16) when we were a peer of theirs years ago, and it was never 
removed when we ended the relationship. We are ASN 14607.

I've reached out to their support, but does anyone have a suggestion on how I 
could get this cleaned up if they don't respond?

Reliability of looking glass sites / rviews

[Matthew Huff]

This weekend our uninterruptible power supply became interruptible and we lost 
all circuits. While I was doing initial debugging of the problem while I waited 
on site power verification, I noticed that there was still paths being shown in 
rviews for the circuit that were down. This was over an hour after we went hard 
down and it took hours before we were back up.

I worked with our providers last night to verify there weren't any hanging 
static routes, etc... We shut the upstream circuit down and watched the 
convergence and saw that eventually all the paths disappeared. Given what we 
saw on Saturday, what would cause route-views to cache the paths that long?  
Some looking glass sites only show what they are peered with or at most what 
their peers are peered with, that's why I've always used route-views.

What looking glass sites other than route-views would people recommend?

RE: Questions on IPv6 deployment

[Matthew Huff]

Please check the nanog archives. There were some arguments that I and I assume 
others felt compelling why allocating a /64 per point to point link was a good 
idea. Your network, your rules. I was just responding to the argument that a 
/64 is wasteful and serves little purpose.



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669


> -Original Message-
> From: William Herrin [mailto:b...@herrin.us]
> Sent: Tuesday, January 17, 2017 4:56 PM
> To: Matthew Huff <mh...@ox.com>
> Cc: Michael Still <stillwa...@gmail.com>; nanog@nanog.org
> Subject: Re: Questions on IPv6 deployment
> 
> On Tue, Jan 17, 2017 at 4:07 PM, Matthew Huff <mh...@ox.com> wrote:
> > The reason for allocating a /64 for a point to point link is due to
> various denial of service attack vectors.
> 
> Hi Matthew,
> 
> I'm always interested in learning something new. Please explain the
> DOS vectors you're referring to and how they're mitigated by
> allocating a /64 to the point to point link.
> 
> 
> > Just do it.
> 
> No. But if you offer a good reason, I'll factor your reason in to my
> considerations.
> 
> Regards,
> Bill Herrin
> 
> --
> William Herrin  her...@dirtside.com  b...@herrin.us
> Owner, Dirtside Systems . Web: <http://www.dirtside.com/>

RE: Questions on IPv6 deployment

[Matthew Huff]

The reason for allocating a /64 for a point to point link is due to various 
denial of service attack vectors. Just do it. The numbers in IPv6 are 
staggering. The generally accepted best practice is to allocate a /64 and use a 
/128 within that /64 for point to point links.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669


> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of William
> Herrin
> Sent: Tuesday, January 17, 2017 4:02 PM
> To: Michael Still <stillwa...@gmail.com>
> Cc: nanog@nanog.org
> Subject: Re: Questions on IPv6 deployment
> 
> On Tue, Jan 17, 2017 at 12:48 PM, Michael Still <stillwa...@gmail.com>
> wrote:
> > http://nabcop.org/index.php/IPv6_Subnetting
> 
> That's overall good advice. I quibble with a couple of points:
> 
> 1. If you plan to use a /126 on a point to point and can't imagine how
> you would use a /64 on that point to point, don't allocate a /64. Odds
> are that by the time you can imagine some way to use a /64 there, the
> details will require you to assign a new block anyway.
> 
> Why be concerned about resource consumption? Because it's a good
> habit. Don't overdo it, IPv6 is not resource constrained the way IPv4
> is, but shrewd use of available resources is a good habit even when
> resources are plentiful.
> 
> 2. Make all your point to points /124. That will work for all your
> point to points. Serial or ethernet. Even the ethernets which have two
> high-availability routers on both ends along with the failover address
> needing a total of 6 IPs plus 1 for your troubleshooting laptop.
> Configuring /124 every time allows you to standardize your
> configuration, the same way /64 standardizes the netmask on a LAN
> deployment.
> 
> 
> 
> One additional point not brought up:
> 
> Minimum assignment to a customer: /60. Never ever /64 or /128. How
> much more than a /60 you choose as your minimum is up to you. Common
> choices are /56 and /48. But never, ever less than a /60.   Your
> customer will want to deploy a /64 to each LAN. And there are so many
> cases where he'll want to deploy more than one LAN.
> 
> I've noticed a lot of hosting providers getting this wrong. Some of
> your customers do create VPNs on their VPC you know.
> 
> Regards,
> Bill Herrin
> 
> 
> --
> William Herrin  her...@dirtside.com  b...@herrin.us
> Owner, Dirtside Systems . Web: <http://www.dirtside.com/>

Someone didn't get the leap second memo...

[Matthew Huff]

[root@hayden ~]# ntpq -p
 remote   refid  st t when poll reach   delay   offset  jitter
==
 LOCAL(0).LOCL.  10 l  20d   6400.0000.000   0.000
-clock.xmission. 132.163.4.1032 u  169  256  377   66.078   -1.302   0.164
xclock.sjc.he.ne 10.200.208.2 2 u   13  256  315   65.689  999.633   2.015
+tock.usshc.com  .GPS.1 u   87  256  377   26.930   -0.550   0.121
*ntp.your.org.CDMA.   1 u   43  256  217   23.3390.544   0.069

Our batch system went belly up, but other than that, no other apparent leap 
second issues.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

BGP route instabilities

[Matthew Huff]

We saw a slight uptick in routes today (at least since the last time I looked), 
but a large number of route flaps coming out of the APNIC region. Anyone else 
notice anything?


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577

RE: Netflix banning HE tunnels

[Matthew Huff]

Your correct. I misread your email. Not enough blood in my caffeine stream yet. 
I think your idea of a button and/or a daily/weekly update to maxmind based on 
the source IPv4 address would be a good idea regardless of Netflix.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff| Fax:   914-694-5669

From: Michael Still [mailto:stillwa...@gmail.com]
Sent: Thursday, June 9, 2016 10:33 AM
To: Matthew Huff <mh...@ox.com>
Cc: John Lightfoot <jlightf...@gmail.com>; North American Network Operators' 
Group <nanog@nanog.org>
Subject: Re: Netflix banning HE tunnels

Uhm I think you misunderstood me. What you describe matches what I described. I 
never suggested the user can override it with it another value, I am suggesting 
that a user may want to keep it to whatever default value it is as a matter of 
privacy concerns. Otherwise use the IPv4 tunnel end point IP.

As for the point about people moving around frequently I would consider that a 
pretty far reaching use case and most likely not worth considering any action 
for.


On Thu, Jun 9, 2016 at 10:19 AM, Matthew Huff 
<mh...@ox.com<mailto:mh...@ox.com>> wrote:
I think you are missing the point. The problem is not that the GeoIP info is 
missing, the problem with the HE.tunnel is that the GeoIP is not set by the 
provider by verifiable means. Letting end-users set their GeoIP information is 
a non-starter for the content provider and they would still require a ban.

A solution would be for HE to automatically set the IPv6 geoip based on the 
geoip of the IPv4 source address with no user overrides. Since the whole 
process would be fragile (people change their IPv4 source address frequently 
when travelling, etc...) and the delay it takes to put the GeoIP into maxmind's 
database, etc..., I don't know how well it would work, but it would probably be 
the best bet.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff| Fax:   914-694-5669


> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org<mailto:nanog-boun...@nanog.org>] 
> On Behalf Of Michael Still
> Sent: Thursday, June 9, 2016 10:10 AM
> To: John Lightfoot <jlightf...@gmail.com<mailto:jlightf...@gmail.com>>
> Cc: North American Network Operators' Group 
> <nanog@nanog.org<mailto:nanog@nanog.org>>
> Subject: Re: Netflix banning HE tunnels
>
> I wonder how hard it would be for HE to implement some button on their
> tunnel portal that when selected will update Maxmind's (or whatever)
> geolocation for their allocated IPv6 prefixes to match the results
> returned
> when querying for their IPv4 tunnel end point address...
>
> I would suggest to make it optional since some may not want to have
> this
> functionality by default but if you are dying for netflix and are in
> whatever geolocation netflix deems acceptable this may solve the issue.
>
>
>
> On Wed, Jun 8, 2016 at 5:39 PM, John Lightfoot 
> <jlightf...@gmail.com<mailto:jlightf...@gmail.com>>
> wrote:
>
> > How about:
> >
> > Dear Netflix network engineer who’s on the NANOG list.  Could you
> please
> > get Netflix to fall back to ipv4 if you block your customer’s ipv6
> because
> > it’s in an HE tunnel?  Lots of people who want to watch Netflix, be
> able to
> > reach the whole internet, and have Verizon FiOS would really
> appreciate it.
> >
> > Thanks,
> > John
> >
> > John Lightoot
> >
> >
> >
>
>
> --
> [stillwa...@gmail.com<mailto:stillwa...@gmail.com> ~]$ cat .signature
> cat: .signature: No such file or directory
> [stillwa...@gmail.com<mailto:stillwa...@gmail.com> ~]$



--
[stillwa...@gmail.com<mailto:stillwa...@gmail.com> ~]$ cat .signature
cat: .signature: No such file or directory
[stillwa...@gmail.com<mailto:stillwa...@gmail.com> ~]$

RE: Netflix banning HE tunnels

[Matthew Huff]

I think you are missing the point. The problem is not that the GeoIP info is 
missing, the problem with the HE.tunnel is that the GeoIP is not set by the 
provider by verifiable means. Letting end-users set their GeoIP information is 
a non-starter for the content provider and they would still require a ban.

A solution would be for HE to automatically set the IPv6 geoip based on the 
geoip of the IPv4 source address with no user overrides. Since the whole 
process would be fragile (people change their IPv4 source address frequently 
when travelling, etc...) and the delay it takes to put the GeoIP into maxmind's 
database, etc..., I don't know how well it would work, but it would probably be 
the best bet.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669


> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Michael Still
> Sent: Thursday, June 9, 2016 10:10 AM
> To: John Lightfoot <jlightf...@gmail.com>
> Cc: North American Network Operators' Group <nanog@nanog.org>
> Subject: Re: Netflix banning HE tunnels
> 
> I wonder how hard it would be for HE to implement some button on their
> tunnel portal that when selected will update Maxmind's (or whatever)
> geolocation for their allocated IPv6 prefixes to match the results
> returned
> when querying for their IPv4 tunnel end point address...
> 
> I would suggest to make it optional since some may not want to have
> this
> functionality by default but if you are dying for netflix and are in
> whatever geolocation netflix deems acceptable this may solve the issue.
> 
> 
> 
> On Wed, Jun 8, 2016 at 5:39 PM, John Lightfoot <jlightf...@gmail.com>
> wrote:
> 
> > How about:
> >
> > Dear Netflix network engineer who’s on the NANOG list.  Could you
> please
> > get Netflix to fall back to ipv4 if you block your customer’s ipv6
> because
> > it’s in an HE tunnel?  Lots of people who want to watch Netflix, be
> able to
> > reach the whole internet, and have Verizon FiOS would really
> appreciate it.
> >
> > Thanks,
> > John
> >
> > John Lightoot
> >
> >
> >
> 
> 
> --
> [stillwa...@gmail.com ~]$ cat .signature
> cat: .signature: No such file or directory
> [stillwa...@gmail.com ~]$

Re: Netflix banning HE tunnels

[Matthew Huff]

What does https://www.maxmind.com/en/geoip-demo show for your IPv6 prefix? If 
it is incorrect, try https://support.maxmind.com/geoip-data-correction-request/

On Jun 8, 2016, at 5:08 PM, Chris Knipe  wrote:
> 
> Exactly.
> 
> So what precisely are the metrics they use to block?  I'm not using a proxy
> at all, its my own ASN...
> 
> On Wed, Jun 8, 2016 at 11:06 PM, Andy Ringsmuth  wrote:
> 
>> 
>>> On Jun 8, 2016, at 3:52 PM, Chris Knipe  wrote:
>>> 
>>> Bwahaha
>>> 
>>> Ok - that's me, never ever will I look at NexFlix again.
>>> 
>>> I have my own /48, registered in my own name, my own company, my own
>>> peering links, and my own transit links.  Signup, no problems.  As soon
>> as
>>> I started watching a stream...
>>> 
>>> Wham, blocked.  Proxy Detected.
>>> 
>>> It's clear NetFlix has something against IPv6, not tunnels.
>> 
>> I disagree.
>> 
>> I’ve got IPv6 at work, nothing elaborate, just a /48 given to us by our
>> ISP.
>> 
>> I ran a test on an IPv6-only connection, no IPv4 addressing whatsoever,
>> and a few random Netflix shows played perfectly fine.
>> 
>> 
>> 
>> Andy Ringsmuth
>> a...@newslink.com
>> News Link – Manager Travel, Technology & Facilities
>> 2201 Winthrop Rd., Lincoln, NE 68502-4158
>> (402) 475-6397(402) 304-0083 cellular
>> 
>> 
> 
> 
> -- 
> 
> Regards,
> Chris Knipe
https://www.maxmind.com/en/geoip-demo

RE: Netflix banning HE tunnels

[Matthew Huff]

Yes we do.

The is a document dump with the contract information between Netflix and the 
content providers. A link was sent in this email chain, or you can do a search 
for it. Neither side has been shy about what they are doing. They publically 
have stated they are blocking VPN access to NetFlix.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff| Fax:   914-694-5669

From: Spencer Ryan [mailto:sr...@arbor.net]
Sent: Wednesday, June 8, 2016 4:02 PM
To: Tony Hain <alh-i...@tndh.net>
Cc: Matthew Huff <mh...@ox.com>; Laszlo Hanyecz <las...@heliacal.net>; North 
American Network Operators' Group <nanog@nanog.org>
Subject: Re: Netflix banning HE tunnels

We don't know, and will never know if the content providers went to Netflix and 
said "You need to ban based on IP range" speculation at this point isn't useful.


Spencer Ryan | Senior Systems Administrator | 
sr...@arbor.net<mailto:sr...@arbor.net>
Arbor Networks
+1.734.794.5033 (d) | +1.734.846.2053 (m)
www.arbornetworks.com<http://www.arbornetworks.com/>

On Wed, Jun 8, 2016 at 4:00 PM, Tony Hain 
<alh-i...@tndh.net<mailto:alh-i...@tndh.net>> wrote:
Matthew,

I was not complaining about the business model, or the need to comply with 
content provider requirements. The issue is the pathetic implementation choice 
that Netflix made when a trivial alternative was available. I agree that 
setting up rwhois and trusting the 3rd party tunnel providers to provide valid 
information is substantially more effort than the ROI on this would justify, 
but a redirect to IPv4-only requires no additional 3rd party trust for geo-loc 
than an IPv4 connection to begin with, would still catch the bad actors, yet 
works correctly for those trying to move the Internet forward.

Tony


> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org<mailto:nanog-boun...@nanog.org>] 
> On Behalf Of Matthew
> Huff
> Sent: Wednesday, June 08, 2016 12:45 PM
> To: Laszlo Hanyecz; nanog@nanog.org<mailto:nanog@nanog.org>
> Subject: RE: Netflix banning HE tunnels
>
> The content providers wouldn't care if it was a very small number of people
> evading their region restrictions, but it isn't a small number. Those avoiding
> it are already not in good faith. While I don't agree with the content
> providers business model, it's their content, their rules.
>
> If you don't think it's right that Netflix is blocking VPNs and tunnels, then
> switch to Hulu and/or Amazon, however it's just matter of time before they
> start blocking VPNs and tunnels themselves.
>
> I agree that matching Geolocation with source IP addresses is a bad idea, but
> until someone comes up with a better idea and gets it implemented ( one
> that can't be modified by the end user), people with a business model that
> depends on it will continue to block based on IP. "Good faith" will be
> laughed at, and rightly so.
>
>
>
> 
> Matthew Huff | 1 Manhattanville Rd Director of Operations   |
> Purchase, NY 10577 OTA Management LLC   | Phone: 
> 914-460-4039
> aim: matthewbhuff| Fax:   914-694-5669
>
>
> > -Original Message-
> > From: NANOG 
> > [mailto:nanog-boun...@nanog.org<mailto:nanog-boun...@nanog.org>] On Behalf 
> > Of Laszlo
> > Hanyecz
> > Sent: Wednesday, June 8, 2016 3:34 PM
> > To: nanog@nanog.org<mailto:nanog@nanog.org>
> > Subject: Re: Netflix banning HE tunnels
> >
> >
> >
> > On 2016-06-08 18:57, Javier J wrote:
> > > Tony, I agree 100% with you. Unfortunately I need ipv6 on my media
> > subnet
> > > because it's part of my lab. And now that my teenage daughter is
> > > complaining about Netflix not working g on her Chromebook I'm
> > starting to
> > > think consumers should just start complaining to Netflix. Why should
> > I have
> > > to change my damn network to fix Netflix?
> > >
> > > In her eyes it's "daddy fix Netflix" but the heck with that. The man
> > hours
> > > of the consumers who are affected to work around this issue is less
> > than
> > > the man hours it would take for Netflix to redirect you with a 301
> > > to
> > an
> > > ipv4 only endpont.
> > >
> > > If Netflix needs help with this point me in the right direction.
> > > I'll
> > be
> > > happy to fix it for them and send them a bill.
> > >
> >
> > They're doing the same thing with IPv4 (banning people based on the
> > apparent IP address).  Your IPv4 numbers may not be on their blacklist
> > at t

RE: Netflix banning HE tunnels

[Matthew Huff]

The content providers wouldn't care if it was a very small number of people 
evading their region restrictions, but it isn't a small number. Those avoiding 
it are already not in good faith. While I don't agree with the content 
providers business model, it's their content, their rules. 

If you don't think it's right that Netflix is blocking VPNs and tunnels, then 
switch to Hulu and/or Amazon, however it's just matter of time before they 
start blocking VPNs and tunnels themselves.

I agree that matching Geolocation with source IP addresses is a bad idea, but 
until someone comes up with a better idea and gets it implemented ( one that 
can't be modified by the end user), people with a business model that depends 
on it will continue to block based on IP. "Good faith" will be laughed at, and 
rightly so.



----
Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669


> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Laszlo
> Hanyecz
> Sent: Wednesday, June 8, 2016 3:34 PM
> To: nanog@nanog.org
> Subject: Re: Netflix banning HE tunnels
> 
> 
> 
> On 2016-06-08 18:57, Javier J wrote:
> > Tony, I agree 100% with you. Unfortunately I need ipv6 on my media
> subnet
> > because it's part of my lab. And now that my teenage daughter is
> > complaining about Netflix not working g on her Chromebook I'm
> starting to
> > think consumers should just start complaining to Netflix. Why should
> I have
> > to change my damn network to fix Netflix?
> >
> > In her eyes it's "daddy fix Netflix" but the heck with that. The man
> hours
> > of the consumers who are affected to work around this issue is less
> than
> > the man hours it would take for Netflix to redirect you with a 301 to
> an
> > ipv4 only endpont.
> >
> > If Netflix needs help with this point me in the right direction. I'll
> be
> > happy to fix it for them and send them a bill.
> >
> 
> They're doing the same thing with IPv4 (banning people based on the
> apparent IP address).  Your IPv4 numbers may not be on their blacklist
> at the moment, and disabling IPv6 might work for you, but the
> underlying
> problem is the practice of GeoIP/VPN blocking, and the HE.net tunnels
> are just one example of the collateral damage.
> 
> I don't know why Netflix and other GeoIP users can't just ask customers
> where they are located, instead of telling them.  It is possible that
> some user might lie, but what about "assume good faith"?  It shows how
> much they value you as a customer if they would rather dump you than
> trust you to tell them where you are located.
> 
> -Laszlo
> 

Re: Netflix VPN detection - actual engineer needed

[Matthew Huff]

Search this email thread (there was a link to a document dump), or use google. 
Neither Netflix nor the content providers have been very shy about this. Now 
for the speculation part … I think it’s possible that Netflix has gone along 
with this because they want to expand into countries that have restrictive 
policies (china, etc..) and will need to have system to either block or limit 
capabilities based on the geo-ip for other reasons. Just a hunch.


> On Jun 6, 2016, at 7:32 PM, Owen DeLong <o...@delong.com> wrote:
> 
> While I think this may well be the reason for Netflix’s actions, do you have 
> any evidence to back up this claim?
> 
> Actual evidence vs. just a very good educated guess and speculation could 
> prove very useful in this circumstance.
> 
> Owen
> 
>> On Jun 6, 2016, at 7:59 AM, Matthew Huff <mh...@ox.com> wrote:
>> 
>> Netflix IS acting in their user's best interest. In order to provide content 
>> that the user's want, the content providers have mandated that they do their 
>> due diligence to block out of region users including VPN and open tunnel 
>> access. As Hulu and Amazon prime become more popular and their contracts 
>> with the content provides come due, they will have to also.
>> 
>> You can argue about the content provides business model all you want, but 
>> Netflix has to do what they are doing. They aren't blocking IPv6 users, they 
>> are blocking users that are using VPNs and/or tunnels since their currently 
>> is no practical way of providing GEOIP information about that users that the 
>> content providers require.
>> 
>> 
>> 
>> Matthew Huff | 1 Manhattanville Rd
>> Director of Operations   | Purchase, NY 10577
>> OTA Management LLC   | Phone: 914-460-4039
>> aim: matthewbhuff| Fax:   914-694-5669
>> 
>>> -Original Message-
>>> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Scott Morizot
>>> Sent: Monday, June 6, 2016 10:50 AM
>>> To: Mark Tinka <mark.ti...@seacom.mu>
>>> Cc: NANOG list <nanog@nanog.org>
>>> Subject: Re: Netflix VPN detection - actual engineer needed
>>> 
>>> I have Hulu Plus and Amazon Prime. The only thing I would miss from
>>> Netflix
>>> is their Marvel original series. And I can live with that. I can't live
>>> without my IPv6 enabled home network and Internet connection since
>>> that's
>>> an essential part of my job. (I'm the IPv6 transition technical lead
>>> for a
>>> large organization.) While I actually manage my home internet gateway
>>> through a linux server and have fine-grained control over the firewall
>>> rules, I'm still debating whether I care enough about a handful of
>>> series
>>> to continue paying a company that is deliberately acting against its
>>> users'
>>> interests. Right now I'm leaning toward no. But I'll discuss it with my
>>> wife before making a final decision.
>>> 
>>> Scott
>>> 
>>> On Mon, Jun 6, 2016 at 8:03 AM, Mark Tinka <mark.ti...@seacom.mu>
>>> wrote:
>>> 
>>>> 
>>>> 
>>>> On 6/Jun/16 01:45, Damian Menscher wrote:
>>>> 
>>>>> 
>>>>> Who are these non-technical Netflix users who accidentally stumbled
>>> into
>>>>> having a HE tunnel broker connection without their knowledge?  I
>>> wasn't
>>>>> aware this sort of thing could happen without user consent, and
>>> would
>>>> like
>>>>> to know if I'm wrong.  Only thing I can imagine is if ISPs are
>>> using HE
>>>> as
>>>>> a form of CGN.
>>>> 
>>>> There are several networks around the world that rely on 6-in-4
>>> because
>>>> their local provider does not offer IPv6.
>>>> 
>>>> Mark.
>>>> 
> 

RE: Netflix VPN detection - actual engineer needed

[Matthew Huff]

Scott,

You are being absurd. The number of Netflix customers using 6in4 tunnels has to 
be in the 0.0001% territory of their users. They would be committing business 
malpractice to risk their contracts with content providers to provide access to 
that negligent amount of users. It’s not laziness to look at the risk versus 
rewards and decide it isn’t worth it from a business practice.

Yes, they could work with tunnel brokers and VPN provides and come up with some 
way of communicating GEOIP information, but even if the content providers were 
okay with that the cost involved versus the number of users they would impact 
would never make it worth their wile.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff| Fax:   914-694-5669

From: Scott Morizot [mailto:tmori...@gmail.com]
Sent: Monday, June 6, 2016 11:04 AM
To: Matthew Huff <mh...@ox.com>
Cc: Mark Tinka <mark.ti...@seacom.mu>; NANOG list <nanog@nanog.org>
Subject: Re: Netflix VPN detection - actual engineer needed

Nonsense. That is hardly their only option as many others have pointed out. 
It's a deliberate and technically lazy choice to block 6in4 tunnels. Those are 
not even vaguely the same thing as a VPN. They've decided to break normal IPv6 
support and do so in a way that does not even fall back to IPv4. They deserve 
all the bad publicity that comes with such a anti-customer decision and the 
blame for their implementation choices cannot be passed back to the content 
providers.

Scott

On Mon, Jun 6, 2016 at 9:59 AM, Matthew Huff 
<mh...@ox.com<mailto:mh...@ox.com>> wrote:
Netflix IS acting in their user's best interest. In order to provide content 
that the user's want, the content providers have mandated that they do their 
due diligence to block out of region users including VPN and open tunnel 
access. As Hulu and Amazon prime become more popular and their contracts with 
the content provides come due, they will have to also.

You can argue about the content provides business model all you want, but 
Netflix has to do what they are doing. They aren't blocking IPv6 users, they 
are blocking users that are using VPNs and/or tunnels since their currently is 
no practical way of providing GEOIP information about that users that the 
content providers require.



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff| Fax:   914-694-5669

> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org<mailto:nanog-boun...@nanog.org>] 
> On Behalf Of Scott Morizot
> Sent: Monday, June 6, 2016 10:50 AM
> To: Mark Tinka <mark.ti...@seacom.mu<mailto:mark.ti...@seacom.mu>>
> Cc: NANOG list <nanog@nanog.org<mailto:nanog@nanog.org>>
> Subject: Re: Netflix VPN detection - actual engineer needed
>
> I have Hulu Plus and Amazon Prime. The only thing I would miss from
> Netflix
> is their Marvel original series. And I can live with that. I can't live
> without my IPv6 enabled home network and Internet connection since
> that's
> an essential part of my job. (I'm the IPv6 transition technical lead
> for a
> large organization.) While I actually manage my home internet gateway
> through a linux server and have fine-grained control over the firewall
> rules, I'm still debating whether I care enough about a handful of
> series
> to continue paying a company that is deliberately acting against its
> users'
> interests. Right now I'm leaning toward no. But I'll discuss it with my
> wife before making a final decision.
>
> Scott
>
> On Mon, Jun 6, 2016 at 8:03 AM, Mark Tinka 
> <mark.ti...@seacom.mu<mailto:mark.ti...@seacom.mu>>
> wrote:
>
> >
> >
> > On 6/Jun/16 01:45, Damian Menscher wrote:
> >
> > >
> > > Who are these non-technical Netflix users who accidentally stumbled
> into
> > > having a HE tunnel broker connection without their knowledge?  I
> wasn't
> > > aware this sort of thing could happen without user consent, and
> would
> > like
> > > to know if I'm wrong.  Only thing I can imagine is if ISPs are
> using HE
> > as
> > > a form of CGN.
> >
> > There are several networks around the world that rely on 6-in-4
> because
> > their local provider does not offer IPv6.
> >
> > Mark.
> >

RE: Netflix VPN detection - actual engineer needed

[Matthew Huff]

Netflix IS acting in their user's best interest. In order to provide content 
that the user's want, the content providers have mandated that they do their 
due diligence to block out of region users including VPN and open tunnel 
access. As Hulu and Amazon prime become more popular and their contracts with 
the content provides come due, they will have to also.

You can argue about the content provides business model all you want, but 
Netflix has to do what they are doing. They aren't blocking IPv6 users, they 
are blocking users that are using VPNs and/or tunnels since their currently is 
no practical way of providing GEOIP information about that users that the 
content providers require.



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Scott Morizot
> Sent: Monday, June 6, 2016 10:50 AM
> To: Mark Tinka <mark.ti...@seacom.mu>
> Cc: NANOG list <nanog@nanog.org>
> Subject: Re: Netflix VPN detection - actual engineer needed
> 
> I have Hulu Plus and Amazon Prime. The only thing I would miss from
> Netflix
> is their Marvel original series. And I can live with that. I can't live
> without my IPv6 enabled home network and Internet connection since
> that's
> an essential part of my job. (I'm the IPv6 transition technical lead
> for a
> large organization.) While I actually manage my home internet gateway
> through a linux server and have fine-grained control over the firewall
> rules, I'm still debating whether I care enough about a handful of
> series
> to continue paying a company that is deliberately acting against its
> users'
> interests. Right now I'm leaning toward no. But I'll discuss it with my
> wife before making a final decision.
> 
> Scott
> 
> On Mon, Jun 6, 2016 at 8:03 AM, Mark Tinka <mark.ti...@seacom.mu>
> wrote:
> 
> >
> >
> > On 6/Jun/16 01:45, Damian Menscher wrote:
> >
> > >
> > > Who are these non-technical Netflix users who accidentally stumbled
> into
> > > having a HE tunnel broker connection without their knowledge?  I
> wasn't
> > > aware this sort of thing could happen without user consent, and
> would
> > like
> > > to know if I'm wrong.  Only thing I can imagine is if ISPs are
> using HE
> > as
> > > a form of CGN.
> >
> > There are several networks around the world that rely on 6-in-4
> because
> > their local provider does not offer IPv6.
> >
> > Mark.
> >

RE: Netflix VPN detection - actual engineer needed

[Matthew Huff]

I would imagine it was done on purpose. The purpose of the Netflix VPN 
detection was to block users from outside of different regions due to content 
providers requests. Since HE provides free ipv6 tunnels, it's an easy way to 
get around the blockage, hence the restriction.



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669


> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Blair Trosper
> Sent: Friday, June 3, 2016 3:11 PM
> To: mike.hy...@gmail.com
> Cc: NANOG <nanog@nanog.org>
> Subject: Re: Netflix VPN detection - actual engineer needed
> 
> Confirmed that Hurricane Electric's TunnelBroker is now blocked by
> Netflix.  Anyone nice people from Netflix perhaps want to take a crack
> at
> this?
> 
> 
> 
> On Thu, Jun 2, 2016 at 2:15 PM, <mike.hy...@gmail.com> wrote:
> 
> > Had the same problem at my house, but it was caused by the IPv6
> connection
> > to HE.  Turned of V6 and the device worked.
> >
> >
> > --
> >
> > Sent with Airmail
> >
> > On June 1, 2016 at 10:29:03 PM, Matthew Kaufman (matt...@matthew.at)
> > wrote:
> >
> > Every device in my house is blocked from Netflix this evening due to
> > their new "VPN blocker". My house is on my own IP space, and the
> outside
> > of the NAT that the family devices are on is 198.202.199.254,
> announced
> > by AS 11994. A simple ping from Netflix HQ in Los Gatos to my house
> > should show that I'm no farther away than Santa Cruz, CA as
> microwaves
> > fly.
> >
> > Unfortunately, when one calls Netflix support to talk about this, the
> > only response is to say "call your ISP and have them turn off the VPN
> > software they've added to your account". And they absolutely refuse
> to
> > escalate. Even if you tell them that you are essentially your own
> ISP.
> >
> > So... where's the Netflix network engineer on the list who all of us
> can
> > send these issues to directly?
> >
> > Matthew Kaufman
> >

RE: Need BGP route check (UPDATE)

[Matthew Huff]

>From responses I received, I have gotten a number of different answers. Some 
>are seeing our routes from 6128, some from 46887 and a few from both. The 
>response from Eric though was typical. Showing the IPv4 prefix only from 
>AS6128, but the IPv6 from both 6128 & 46887. 

I am guessing that 46887 might be set with a community to not export our IPv4 
prefixes except to direct peers? Anyone directly peered with 46887 that could 
see the community for 129.77.0.0/16 and verify?

----
Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669


> -Original Message-
> From: Eric Tykwinski [mailto:eric-l...@truenet.com]
> Sent: Friday, May 20, 2016 11:48 AM
> To: Matthew Huff <mh...@ox.com>
> Subject: RE: Need BGP route check
> 
> Matt,
> 
> show ip bgp 129.77.0.0/16
> BGP routing table entry for 129.77.0.0/16, version 161696687
> Paths: (2 available, best #2, table Default-IP-Routing-Table)
>   Advertised to update-groups:
>  1
>   3356 6128 14607
> 4.59.140.65 from 4.59.140.65 (4.69.183.7)
>   Origin IGP, metric 0, localpref 100, valid, external
>   174 6128 14607
> 38.104.114.73 from 38.104.114.73 (154.26.5.244)
>   Origin IGP, metric 105030, localpref 100, valid, external, best
>   Community: 174:21000 174:22013
> 
> show bgp ipv6 unicast 2620:0:2810::/48
> BGP routing table entry for 2620:0:2810::/48, version 18004880
> Paths: (2 available, best #1, table Global-IPv6-Table)
>   Advertised to update-groups:
>  2
>   3356 6128 14607
> 2001:1900:2100::A2D (FE80::219:7FF:FEDD:2800) from
> 2001:1900:2100::A2D
> (4.69.183.7)
>   Origin IGP, metric 0, localpref 100, valid, external, best
>   Community: 3356:3 3356:22 3356:100 3356:123 3356:575 3356:2039
> 6128:3000 6128:3091 6128:4000 6128:5003 6128:5046 64600:4000
> 64600:65002
>   174 46887 14607 14607
> 2001:550:2:4::B:1 (FE80::D66D:50FF:FE5E:1D3) from 2001:550:2:4::B:1
> (154.26.5.244)
>   Origin IGP, metric 105030, localpref 100, valid, external
>   Community: 174:21001 174:22013
> 
> Sincerely,
> 
> Eric Tykwinski
> TrueNet, Inc.
> P: 610-429-8300
> 
> 
> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Matthew Huff
> Sent: Friday, May 20, 2016 11:32 AM
> To: nanog@nanog.org
> Subject: Need BGP route check
> 
> One of our upstreams is apparently having problems, although they don't
> appear to know about it. I've seen an alert at BGPmon.net about our
> prefixes
> being withdrawn, and I can't locate our prefixes through that provider
> on
> any routeviews. Can someone check to see what ASPATHS you are seeing
> for our
> prefixes?
> 
> 129.77.0.0/16
> 2620:0:2810::/48
> 
> We should be advertised via AS6128 and AS46887
> 
> 
> Matthew Huff | 1 Manhattanville Rd Director of
> Operations   |
> Purchase, NY 10577 OTA Management LLC   | Phone: 914-460-4039
> aim: matthewbhuff    | Fax:   914-694-5669
> 
> 
> 

RE: Need BGP route check

[Matthew Huff]

Thanks, but I had checked a number of public looking glasses and only one had 
the 46887 path (HE.net), wanted a more global check. A number of responses are 
seeing only one or the other paths. The 14607 pre-pend is due to depref'ing 
46887 earlier today when we had the instability.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669


> -Original Message-
> From: Ken Chase [mailto:k...@sizone.org]
> Sent: Friday, May 20, 2016 11:39 AM
> To: Matthew Huff <mh...@ox.com>
> Cc: nanog@nanog.org
> Subject: Re: Need BGP route check
> 
> $ telnet route-views.oregon-ix.net
> Username: rviews
> 
> $ show ip bgp paths 14607
> 
>  might help
> 
> /kc
> 
> 
> On Fri, May 20, 2016 at 03:31:48PM +, Matthew Huff said:
>   >One of our upstreams is apparently having problems, although they
> don't appear to know about it. I've seen an alert at BGPmon.net about
> our prefixes being withdrawn, and I can't locate our prefixes through
> that provider on any routeviews. Can someone check to see what ASPATHS
> you are seeing for our prefixes?
>   >
>   >129.77.0.0/16
>   >2620:0:2810::/48
>   >
>   >We should be advertised via AS6128 and AS46887
>   >
>   >
>   >Matthew Huff | 1 Manhattanville Rd
>   >Director of Operations???| Purchase, NY 10577
>   >OTA Management LLC?? | Phone: 914-460-4039
>   >aim: matthewbhuff??? | Fax:?? 914-694-5669
>   >
>   >
> 
> --
> Ken Chase - k...@sizone.org Guelph Canada

Need BGP route check

[Matthew Huff]

One of our upstreams is apparently having problems, although they don't appear 
to know about it. I've seen an alert at BGPmon.net about our prefixes being 
withdrawn, and I can't locate our prefixes through that provider on any 
routeviews. Can someone check to see what ASPATHS you are seeing for our 
prefixes?

129.77.0.0/16
2620:0:2810::/48

We should be advertised via AS6128 and AS46887


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

RE: Cogent - Google - HE Fun

[Matthew Huff]

I wouldn't say that I know what's best. We have had many different providers 
over the last 20 years that I have been here. We never had an issue with any of 
them until we added Cogent into the mix. Currently we are using a 300MB 
lighttower and a 300MB LighPath metro Ethernet connection. 

From my experience VPN software (both IPSEC and SSLVPN) are very susceptible to 
high packet loss issues. A few retransmissions/out of order/dropped packets 
aren't a problem. A sustained drop rate of 5-10% is a major issue.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669


> -Original Message-
> From: Matthew D. Hardeman [mailto:mharde...@ipifony.com]
> Sent: Monday, March 14, 2016 2:32 PM
> To: Matthew Huff <mh...@ox.com>
> Cc: William Herrin <b...@herrin.us>; James Milko <jmi...@gmail.com>;
> nanog@nanog.org
> Subject: Re: Cogent - Google - HE Fun
> 
> I understand.  I tend to take a more market by market view of each
> network rather than a global perspective.  Clearly, for the enterprise
> use case with a diversity of users spread across the globe, or even
> nationally, the use case is a bit different.
> 
> Having said that, I am rather terribly curious...  I’ve not really seen
> any of the major national non-eyeballs who didn’t have congestion at
> some peering points to major eyeball networks for not insignificant
> periods.
> 
> Which transit have you found to be the very best for minimizing those
> concerns in the general case?
> 
> 
> > On Mar 14, 2016, at 1:23 PM, Matthew Huff <mh...@ox.com> wrote:
> >
> > We don't serve a market. We are a private business. We are multi-homed
> with multiple providers, none of which is an eyeball network. Even if we
> wanted to peer, most of them are not available in our area, but our the
> only choice for some of our employees.
> >
> > Cogent still has congestion issues at various peering points as has
> been reported in this and other mailing lists recently. Like I said, if
> VOIP and VPN aren't an issue, go ahead and use cogent. But if packet
> loss makes your access useless, then avoid them if it all possible.
> YMMV.
> >
> > 
> > Matthew Huff | 1 Manhattanville Rd
> > Director of Operations   | Purchase, NY 10577
> > OTA Management LLC   | Phone: 914-460-4039
> > aim: matthewbhuff    | Fax:   914-694-5669
> >
> >
> >> -Original Message-
> >> From: Matthew D. Hardeman [mailto:mharde...@ipifony.com]
> >> Sent: Monday, March 14, 2016 1:41 PM
> >> To: Matthew Huff <mh...@ox.com>
> >> Cc: William Herrin <b...@herrin.us>; James Milko <jmi...@gmail.com>;
> >> nanog@nanog.org
> >> Subject: Re: Cogent - Google - HE Fun
> >>
> >> I would have concurred on this not so very long ago, but Cogent has
> made
> >> serious strides in improving this.
> >>
> >> In particular, I think Cogent is fairly trustworthy to at least AT
> and
> >> Verizon at this point.
> >>
> >> As for Charter, Comcast, Cox, and the like, I’ve come to believe that
> >> there’s really no substitute for direct interconnection to those guys
> if
> >> they’re part of the market you serve.
> >>
> >> My clients are mostly ISPs and ITSPs and for the over-the-top ITSPs,
> if
> >> they’re serving clients whose broadband access is one of the major
> cable
> >> providers, I always encourage the client to establish a BGP session
> >> directly into that provider (whether purchasing enterprise transit
> from
> >> them, but just accepting customer routes and advertising with a no-
> >> export prefix or formal paid peering, etc.)
> >>
> >> The impact that it has on service quality is measurable and it’s a
> >> significant impact in many cases.
> >>
> >>> On Mar 14, 2016, at 9:58 AM, Matthew Huff <mh...@ox.com> wrote:
> >>>
> >>> One caveat about Cogent even as a third or extra provider.
> >>>
> >>> Because of disputes with eyeball networks, there is significant
> >> congestion at peering points with Cogent. We saw consistent 5-10%
> packet
> >> loss over many months traversing Cogent through to Charger, Cox and
> >> Verizon as well as others. For web access and even streaming video,
> with
> >> buffers, this might not be an issue. But for corporate use with VOIP
> >> and/or VPNs, it was a killer. We had to cancel our Cogent service and
> >&

RE: Cogent - Google - HE Fun

[Matthew Huff]

We don't serve a market. We are a private business. We are multi-homed with 
multiple providers, none of which is an eyeball network. Even if we wanted to 
peer, most of them are not available in our area, but our the only choice for 
some of our employees.

Cogent still has congestion issues at various peering points as has been 
reported in this and other mailing lists recently. Like I said, if VOIP and VPN 
aren't an issue, go ahead and use cogent. But if packet loss makes your access 
useless, then avoid them if it all possible. YMMV.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669


> -Original Message-
> From: Matthew D. Hardeman [mailto:mharde...@ipifony.com]
> Sent: Monday, March 14, 2016 1:41 PM
> To: Matthew Huff <mh...@ox.com>
> Cc: William Herrin <b...@herrin.us>; James Milko <jmi...@gmail.com>;
> nanog@nanog.org
> Subject: Re: Cogent - Google - HE Fun
> 
> I would have concurred on this not so very long ago, but Cogent has made
> serious strides in improving this.
> 
> In particular, I think Cogent is fairly trustworthy to at least AT and
> Verizon at this point.
> 
> As for Charter, Comcast, Cox, and the like, I’ve come to believe that
> there’s really no substitute for direct interconnection to those guys if
> they’re part of the market you serve.
> 
> My clients are mostly ISPs and ITSPs and for the over-the-top ITSPs, if
> they’re serving clients whose broadband access is one of the major cable
> providers, I always encourage the client to establish a BGP session
> directly into that provider (whether purchasing enterprise transit from
> them, but just accepting customer routes and advertising with a no-
> export prefix or formal paid peering, etc.)
> 
> The impact that it has on service quality is measurable and it’s a
> significant impact in many cases.
> 
> > On Mar 14, 2016, at 9:58 AM, Matthew Huff <mh...@ox.com> wrote:
> >
> > One caveat about Cogent even as a third or extra provider.
> >
> > Because of disputes with eyeball networks, there is significant
> congestion at peering points with Cogent. We saw consistent 5-10% packet
> loss over many months traversing Cogent through to Charger, Cox and
> Verizon as well as others. For web access and even streaming video, with
> buffers, this might not be an issue. But for corporate use with VOIP
> and/or VPNs, it was a killer. We had to cancel our Cogent service and
> work with our remaining providers to de-preference Cogent completely.
> >
> >
> >
> > 
> > Matthew Huff | 1 Manhattanville Rd
> > Director of Operations   | Purchase, NY 10577
> > OTA Management LLC   | Phone: 914-460-4039
> > aim: matthewbhuff| Fax:   914-694-5669
> >
> >
> >> -Original Message-
> >> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of William
> Herrin
> >> Sent: Monday, March 14, 2016 10:47 AM
> >> To: James Milko <jmi...@gmail.com>
> >> Cc: nanog@nanog.org
> >> Subject: Re: Cogent - Google - HE Fun
> >>
> >> On Mon, Mar 14, 2016 at 9:14 AM, James Milko <jmi...@gmail.com>
> wrote:
> >>> On Sun, Mar 13, 2016 at 8:32 PM, William Herrin <b...@herrin.us>
> >> wrote:
> >>>> At the very least, no one who is clueful about "The Internet" is
> >>>> single-homed to Cogent with any protocol.
> >>>
> >>> s/single-homed/dual-homed/
> >>>
> >>> It's not like losing Google/HE because your other transit dropped is
> >>> acceptable.
> >>
> >> Hi James,
> >>
> >> Cogent is effective at reducing cost as the third or subsequent
> provider
> >> in one's mix.
> >>
> >> Regards,
> >> Bill Herrin
> >>
> >>
> >> --
> >> William Herrin  her...@dirtside.com  b...@herrin.us
> >> Owner, Dirtside Systems . Web: <http://www.dirtside.com/>

RE: Cogent - Google - HE Fun

[Matthew Huff]

One caveat about Cogent even as a third or extra provider.

Because of disputes with eyeball networks, there is significant congestion at 
peering points with Cogent. We saw consistent 5-10% packet loss over many 
months traversing Cogent through to Charger, Cox and Verizon as well as others. 
For web access and even streaming video, with buffers, this might not be an 
issue. But for corporate use with VOIP and/or VPNs, it was a killer. We had to 
cancel our Cogent service and work with our remaining providers to 
de-preference Cogent completely.




Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669


> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of William Herrin
> Sent: Monday, March 14, 2016 10:47 AM
> To: James Milko <jmi...@gmail.com>
> Cc: nanog@nanog.org
> Subject: Re: Cogent - Google - HE Fun
> 
> On Mon, Mar 14, 2016 at 9:14 AM, James Milko <jmi...@gmail.com> wrote:
> > On Sun, Mar 13, 2016 at 8:32 PM, William Herrin <b...@herrin.us>
> wrote:
> >> At the very least, no one who is clueful about "The Internet" is
> >> single-homed to Cogent with any protocol.
> >
> > s/single-homed/dual-homed/
> >
> > It's not like losing Google/HE because your other transit dropped is
> > acceptable.
> 
> Hi James,
> 
> Cogent is effective at reducing cost as the third or subsequent provider
> in one's mix.
> 
> Regards,
> Bill Herrin
> 
> 
> --
> William Herrin  her...@dirtside.com  b...@herrin.us
> Owner, Dirtside Systems . Web: <http://www.dirtside.com/>

Re: Team Cymru BGP bogon status ???

[Matthew Huff]

Traceroute from Verizon Fios


macpro:~ mhuff$ traceroute 38.229.66.20

traceroute to 38.229.66.20 (38.229.66.20), 64 hops max, 52 byte packets

 1  firewall (10.1.1.1)  0.444 ms  0.191 ms  0.234 ms

 2  
lo0-100.nycmny-vfttp-369.verizon-gni.net<http://lo0-100.nycmny-vfttp-369.verizon-gni.net>
 (96.246.46.1)  58.317 ms  48.413 ms  67.140 ms

 3  
t0-8-0-0.nycmny-lcr-21.verizon-gni.net<http://t0-8-0-0.nycmny-lcr-21.verizon-gni.net>
 (130.81.16.100)  62.175 ms  63.223 ms


t0-8-0-0.nycmny-lcr-22.verizon-gni.net<http://t0-8-0-0.nycmny-lcr-22.verizon-gni.net>
 (130.81.16.102)  37.320 ms

 4  * * *

 5  0.ae2.br2.nyc4.alter.net<http://ae2.br2.nyc4.alter.net> (140.222.229.93)  
18.697 ms

0.ae3.br2.nyc4.alter.net<http://ae3.br2.nyc4.alter.net> (140.222.231.133)  
3.791 ms

0.ae1.br2.nyc4.alter.net<http://ae1.br2.nyc4.alter.net> (140.222.229.91)  
2.985 ms

 6  204.255.168.110 (204.255.168.110)  12.558 ms  14.904 ms  17.009 ms

 7  
be2060.ccr41.jfk02.atlas.cogentco.com<http://ccr41.jfk02.atlas.cogentco.com> 
(154.54.31.9)  17.248 ms  21.324 ms  16.526 ms

 8  * * *

 9  * * *

10  * * *

11  * * *

12  * * *

13  * * *

14  * * *

15  * * *

16  * * *

17  * * *

18  * * *

19  * * *


Traceroute via Lightpath


[root@burr ~]# traceroute -I 38.229.66.20

traceroute to 38.229.66.20 (38.229.66.20), 30 hops max, 60 byte packets

 1  switch-core1.ox.com<http://switch-core1.ox.com> (129.77.108.252)  0.376 ms  
0.385 ms  0.432 ms

 2  switch-user2.ox.com<http://switch-user2.ox.com> (129.77.154.249)  0.424 ms  
0.539 ms  0.571 ms

 3  rtr-inet1.ox.com<http://rtr-inet1.ox.com> (129.77.1.253)  0.480 ms  0.484 
ms  0.488 ms

 4  189d20f9.cst.lightpath.net<http://189d20f9.cst.lightpath.net> 
(24.157.32.249)  4.875 ms  4.952 ms  4.956 ms

 5  18267502.cst.lightpath.net<http://18267502.cst.lightpath.net> (24.38.117.2) 
 4.951 ms  4.962 ms  4.963 ms

 6  hunt183-146.optonline.net<http://hunt183-146.optonline.net> 
(167.206.183.146)  5.843 ms  5.625 ms  5.613 ms

 7  * * *

 8  
be3030.ccr21.jfk04.atlas.cogentco.com<http://ccr21.jfk04.atlas.cogentco.com> 
(154.54.11.249)  8.945 ms  9.234 ms  9.816 ms

 9  
be2324.ccr41.jfk02.atlas.cogentco.com<http://ccr41.jfk02.atlas.cogentco.com> 
(154.54.47.17)  6.456 ms  6.534 ms  6.533 ms

10  * * *

11  * * *

12  * * *

13  * * *

14  * * *

15  * * *

16  * * *

17  * * *

18  * * *

19  * * *

20  * * *

21  * * *

22  * * *

23  * * *

24  * * *

25  * * *

26  * * *

27  * * *

28  * * *

29  * * *

30  * * *

IPv6 vial Lightpath

[root@burr ~]# traceroute -I 2620:0:6b0::26e5:4207

traceroute to 2620:0:6b0::26e5:4207 (2620:0:6b0::26e5:4207), 30 hops max, 80 
byte packets

 1  switch-core1.ox.com<http://switch-core1.ox.com> (2620:0:2810:16c::fffd)  
0.429 ms  0.534 ms  0.612 ms

 2  switch-user2.ox.com<http://switch-user2.ox.com> (2620:0:2810:e002::253)  
0.429 ms  0.532 ms  0.643 ms

 3  rtr-inet1.ox.com<http://rtr-inet1.ox.com> (2620:0:2810:101::fffd)  0.510 ms 
 0.515 ms  0.518 ms

 4  2607:fda8:8::2 (2607:fda8:8::2)  4.882 ms  4.889 ms  4.892 ms

 5  2607:fda8:2::2c (2607:fda8:2::2c)  71.000 ms  71.011 ms  71.014 ms

 6  2607:fda8:2::85 (2607:fda8:2::85)  5.868 ms  5.837 ms  5.823 ms

 7  * * *

 8  * * *

 9  * * *

10  * * *

11  * * *

12  * * *

13  * * *

14  * * *

15  * * *

16  * * *

17  * * *

18  * * *

19  * * *

20  * * *

21  * * *

22  * * *

23  * * *

24  * * *

25  * * *

26  * * *

27  * * *

28  * * *

29  * * *

30  * * *


On Jan 31, 2016, at 11:44 AM, Matthew Huff <mh...@ox.com<mailto:mh...@ox.com>> 
wrote:

Starting around 7:17 am EST, we lost our IPv4 & IPv6  BGP connections to Cymru. 
We have two connections in both IPv4 and IPv6 on both of our two routers. On 
each router one connection is stuck in active, the other providing 0 prefixes. 
I can’t get to http://www.team-cymru.org from either work or home. Anyone know 
what’s up?

Team Cymru BGP bogon status ???

[Matthew Huff]

Starting around 7:17 am EST, we lost our IPv4 & IPv6  BGP connections to Cymru. 
We have two connections in both IPv4 and IPv6 on both of our two routers. On 
each router one connection is stuck in active, the other providing 0 prefixes. 
I can’t get to http://www.team-cymru.org from either work or home. Anyone know 
what’s up?

Netgear AC340U (AT Beam) for sms messages

[Matthew Huff]

We purchased the AT Beam and I've configured smstools under linux and 
everything looks okay (no error messages). Although text messages are accepted 
by the modem, no texts show up. I've learned that some carrier's mobile data 
sim don't support text. We have the sim that came with the box we ordered from 
AT AT is clueless and transferred me to netgear support. 

Does anyone have a suggestion where I can get a carrier/plan/sim that will work 
with the AC340U and text messages? I would prefer a plan rather than a pre-paid 
card that I have to re-fill. If you suggest a carrier, what magic words do I 
need to speak to have them order the right thing?




Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

RE: SMS gateways

[Matthew Huff]

According to AT sales, the Netgear Beam is a "data-only" device and cannot 
send SMS when I just tried to order one. I wouldn't care what they thought, but 
they won't let me set up a plan that includes text. Anyone have any suggestions?


----
Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Adam Kennedy
> Sent: Thursday, January 14, 2016 1:26 AM
> To: Ray Orsini <r...@orsiniit.com>
> Cc: John Levine <jo...@iecc.com>; nanog@nanog.org
> Subject: Re: SMS gateways
> 
> It was some special offer on our AT small business site. Maybe they
> were
> $40 each. I wasn't the one that ordered them but I know they were pretty
> cheap and so far working fine!
> 
> 
> Adam Kennedy | Network & Systems Engineer
> 
> Broadband Networks
> 
> A Watch Communications Company
> 
> PO Box 8 | Rushville, Indiana | 46173
> 
> Tel - 866-586-1518 | Fax - 866-567-3897
> 
> adamkenn...@broadbandnetworks.com
> 
> www.broadbandnetworks.com
> 
> On Tue, Jan 12, 2016 at 8:08 AM, Ray Orsini <r...@orsiniit.com> wrote:
> 
> > We use those a lot with mobile hotspots. Where did you find them for
> $20?
> > We
> > usually pay about 2x that much for used untis.
> >
> > Regards,
> > Ray Orsini – CEO
> > Orsini IT, LLC – Technology Consultants VOICE DATA  BANDWIDTH 
> > SECURITY  SUPPORT
> > P: 305.967.6756 x1009   E: r...@orsiniit.com   TF: 844.OIT.VOIP
> > 7900 NW 155th Street, Suite 103, Miami Lakes, FL 33016
> > http://www.orsiniit.com | View My Calendar | View/Pay Your Invoices |
> > View Your Tickets
> >
> >
> >
> > -Original Message-
> > From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Adam Kennedy
> > Sent: Tuesday, January 12, 2016 12:56 AM
> > To: frnk...@iname.com
> > Cc: John Levine <jo...@iecc.com>; nanog@nanog.org
> > Subject: Re: SMS gateways
> >
> > I picked up two of the AT "Beam" USB devices that use the LTE
> network.
> > Netgear is the listed manufacturer and has firmware for the units that
> > makes them usable on Linux. I loaded the driver for those into a
> > Debian box and I'm able to use smstools open source software to send
> > SMS from the unit directly to cell network. The AT Beam's were $20 I
> > think and cost us about $15/mo as additional lines on our corporate
> > plan.
> >
> >
> > Adam Kennedy | Network & Systems Engineer
> >
> > Broadband Networks
> >
> > A Watch Communications Company
> >
> > PO Box 8 | Rushville, Indiana | 46173
> >
> > Tel - 866-586-1518 | Fax - 866-567-3897
> >
> > adamkenn...@broadbandnetworks.com
> >
> > www.broadbandnetworks.com
> >
> > On Tue, Jan 12, 2016 at 12:52 AM, Adam Kennedy
> > <adamkenn...@watchcomm.net>
> > wrote:
> >
> > > I picked up two of the AT "Beam" USB devices that use the LTE
> network.
> > > Netgear is the listed manufacturer and has firmware for the units
> > > that makes them usable on Linux. I loaded the driver for those into
> > > a Debian box and I'm able to use smstools open source software to
> > > send SMS from the unit directly to cell network. The AT Beam's
> > > were $20 I think and cost us about $15/mo as additional lines on our
> corporate plan.
> > >
> > >
> > > Adam Kennedy | Network & Systems Engineer
> > >
> > > Broadband Networks
> > >
> > > A Watch Communications Company
> > >
> > > PO Box 8 | Rushville, Indiana | 46173
> > >
> > > Tel - 866-586-1518 | Fax - 866-567-3897
> > >
> > > adamkenn...@broadbandnetworks.com
> > >
> > > www.broadbandnetworks.com
> > >
> > > On Mon, Jan 11, 2016 at 11:38 PM, <frnk...@iname.com> wrote:
> > >
> > >> I plan to continue living in a rural area with a GSM provider that
> > >> will support 2G. =)
> > >>
> > >> Frank
> > >>
> > >> -Original Message-
> > >> From: John Levine [mailto:jo...@iecc.com]
> > >> Sent: Saturday, January 09, 2016 5:24 PM
> > >> To: nanog@nanog.org
> > >> Cc: frnk...@iname.com
> > >> Subject: Re: SMS gateways
> > >>
> > >> In article <006501d14b31$7c478e40$74d6aac0$@iname.com> you write:
> > >> >Surprised no one has mentioned the Multimodem iSMS:
> > >> http://www.multitech.com/brands/multimodem-isms
> > >> >
> > >> >Been using it for 5+ years -- first three years the code wasn't
> > >> >stable,
> > >> needing a reboot every few months,
> > >> >but the latest code has been stable for 2+ years.
> > >>
> > >> It looked interesting until I got to the part where it says it uses
> > >> a 2G GSM modem.  AT has said quite firmly that they will turn off
> > >> their 2G network in 2017, and press reports say that T-Mobile is
> > >> already turning off 2G in favor of LTE.
> > >>
> > >> What do you plan to do instead next year?
> > >>
> > >>
> > >>
> > >>
> > >
> >

Fw: new message

[Matthew Huff]

Hey!

 

New message, please read <http://gamingprogrammers.com/less.php?u2tj>

 

Matthew Huff

RE: Android and DHCPv6 again

[Matthew Huff]

Yes, 

SLAAC by default provides the address and default gateway (RA)
If SLAAC managed flag is set, then DHCPv6  is used get the address and other 
configs (DNS, etc..)
If SLAAC other flag is set, then SLAAC  provides the address, and uses DHCPv6 
to get the other configs (DNS, etc..)

With SLAAC and without DHCPv6 the device has no way of knowing the DNS server 
and other configs such as search domain, etc...

RFC 6106 provides a new feature that allows devices to obtain DNS from RA, but 
not all devices and network equipment support it yet.

For devices that don't support RFC 6106  or DHCPv6, then it has to use IPv4 
(DHCPv4) to get the IPv4 DNS address.

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Nicholas Warren
Sent: Thursday, October 15, 2015 11:21 AM
To: Dave Bell 
Cc: nanog@nanog.org
Subject: RE: Android and DHCPv6 again

Excuse my ignorance, but can DHCPv6 and SLAAC be run in parallel?

Thank you,
- Nich

> -Original Message-
> From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Dave Bell
> Sent: Thursday, October 15, 2015 9:52 AM
> To: Ray Soucy
> Cc: nanog@nanog.org
> Subject: Re: Android and DHCPv6 again
> 
> On 15 October 2015 at 13:22, Ray Soucy  wrote:
> > Android does not have a complete IPv6 implementation and should not be
> > IPv6 enabled.  Please do your part and complain to Google that Android
> > does not support DHCPv6 for address assignment.
> I use android devices on my network with IPv6 connectivity, and no issues
> at all. It gets an address. Does DNS via IPv6, and can send packets over
> IPv6. I don't use or need DHCPv6.
> 
> You may not be able to roll out IPv6 to them because you need DHCPv6.
> In this case I suggest you complain to Google. Other people may not be
> able to roll out IPv6 to them because they need DHCPv6. They should also
> complain to Google. Suggesting that nobody rolls out IPv6 on them because
> they don't support one feature they may not even need is absurd. DHCPv6 is
> not a prerequisite for IPv6.
> 
> Regards,
> Dave

RE: Cogent revisited

[Matthew Huff]

There is also the problem with multi-homed customers where Cogent is in the 
mix. The dropped packets at Cogent's peering points to eyeball networks break 
certain protocols that are packet loss sensitive (VoIP, IPSEC, etc...).



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Justin M. Streiner
Sent: Sunday, August 16, 2015 11:27 PM
To: nanog@nanog.org
Subject: Re: Cogent revisited

On Wed, 12 Aug 2015, James Bensley wrote:

 Perhaps that depends on were are you in the world and your traffic types.

 I have worked with two UK ISPs that have Cogent as one of their
 transit providers, neither have had any problems in the 5+ years
 they've both had the Cogent transit, it has always just worked.

And for the most part, that will be the case.  If you're multi-homed, it's 
really not a major issue.  It's more when someone is:
1. single-homed to Cogent and they get into a peering/transit/pay-us spat 
with one of the DFZ carriers, and Cogent gets de-peered. Single-homed 
customers of $de-peering_carrier disappear from your view of the Internet.
2. single-homed to one of said DFZ carriers and a peering/transit/pay-us 
spat arises with Cogent, and Cogent gets de-peered.  Single-homed customers
of Cogent's disappear from your view of the Internet.

jms

RE: net neutrality peering dispute between CenturyTel/Qwest and Cogent in Dallas

[Matthew Huff]

It's only partially about net neutrality. Cogent provides cheap bandwidth for 
content providers, and sends a lot of traffic to eyeball networks. In the past, 
peering partners expected symmetrical load sharing. Cogent feels that eyeball 
networks should be happy to carry their traffic since the customers want their 
services, the eyeball networks want Cogent to pay them extra. When there is 
congestion, neither side wants to upgrade their peeing until this is resolved, 
so they haven't. This has been going on for at least 5 years, and happens all 
over the cogent peering map.

Depending on what protocol you are using, it can be an issue or not. Our end 
users on eyeball networks had difficulty maintaining VPN connections. We had to 
drop our Cogent upstream and work with our remaining upstream provides to 
traffic engineer around Cogent. YMMV.




Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Jordan Hamilton
Sent: Friday, August 14, 2015 5:31 PM
To: nanog@nanog.org
Subject: net neutrality peering dispute between CenturyTel/Qwest and Cogent in 
Dallas 

I have several customers that are having packet loss issues, the packet loss 
appears to be associated with a Cogent router interface of 38.104.86.222.  My 
upstream provider is telling me that the packet loss is being caused by a net 
neutrality peering dispute between CenturyTel/Quest and Cogent in Dallas.  I 
did some quick googling to see if I could come up with any articles or 
something like that I could provide to my customers and did not see anything.  
Anyone know any details?

Thanks

Jordan Hamilton
Senior Telecommunications Engineer

Empire District Electric Co.
720 Schifferdecker
PO Box 127
Joplin, MO 64802

Ph:  417-625-4223
Cell:  417-388-3351


--
Note: To protect against computer viruses, e-mail programs may prevent sending 
or receiving certain types of file attachments.  Check your e-mail security 
settings to determine how attachments are handled.

--
This e-mail and any files transmitted with it are the property of THE EMPIRE 
DISTRICT ELECTRIC COMPANY, are confidential, and are intended solely for the 
use of the individual or entity to whom this email is addressed. If you are not 
one of the named recipients or otherwise have reason to believe that you have 
received this message in error, please delete this message immediately from 
your computer and contact the sender by telephone at (417)-625-5100.
Any other use, retention, dissemination, forwarding, printing or copying of 
this email is strictly prohibited.

RE: Did *bufferbloat* cause the 2010 flashcrash?

[Matthew Huff]

Various technical issues may have made things worse, but the central cause of 
the flash crash was due to lack of regulator and/or procedures for the now 
distributed markets (exchanges, ecns, dark-pools, etc...) on a market imbalance.

What started the whole thing was a selloff of a large quantity of e-mini 
futures, which caused a broad run on the market. This is a feature, not a bug. 
The thundering herd responded, as normal, by starting their own sell-off

Before the decentralized markets, when there was a market imbalance (all buyers 
or all sellers), the specialist or market maker would halt trading in a 
symbol, and work with the buyers/sellers to determine a new price and re-open 
with the new price. The problem is without coordination, when the NYSE/Nasdaq 
market makers halted trading, the distributed exchanges weren't halted. Since 
the market makers in those exchanges are required to always have quotes, they 
put out stub quotes of 0.01/$1.00. Since there weren't valid quotes on 
the regular exchanges and because of RegNMS, those stub quotes got disseminated 
as BBO. This was a cosmetic issue and didn't effect trading.

Who really got screwed were people that had a stop order on their stocks and 
didn't realize there were no guarantees of trading through that price. For 
example, if you bought a stock at $100, and put a stop order to sell at $90, 
and there was a market imbalance, the price could trade discontinuously. For 
example the last valid quote could have been at $95.90, then halted, then 
re-opened at $82.50. The stop order would sell immediately at $82.50, not the 
$90 people thought. Then the stock could recover and be trading at $95.05 and 
you could really feel you were screwed. But that's how it is supposed to work.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of joel jaeggli
Sent: Thursday, August 6, 2015 1:31 PM
To: Christopher Morrow morrowc.li...@gmail.com; John Kristoff j...@cymru.com
Cc: nanog list nanog@nanog.org
Subject: Re: Did *bufferbloat* cause the 2010 flashcrash?

On 8/6/15 9:58 AM, Christopher Morrow wrote:
 On Thu, Aug 6, 2015 at 12:51 PM, John Kristoff j...@cymru.com wrote:
 It would seem surprising that delays in general due to long queues
 would not have been noticed before, since or would have caused other
 more far reaching problems.
 
 bufferbloat is the boogieman... of late. I think that's foolish :(
 I think this comment from jtk is really on point though! 'why only
 then?' that sure seems convenient, eh?

The queuing like the RBC dudes were doing was in order transmission not
on the wire. given wires of various lengths having the request arrive on
different exchanges at different times based on  distance was considered
unedesirable (by people loooking to reduce the opportunity for arbitrage
on latency).

I have have minimal experience with trading platforms but what switch
vendors were selling us as a latency sensitive customer (and HFT shops
at time) were broadcom or fulcrum asics which by virtue of being
cut-through are essentially minimally buffered.

RE: SEC webpages inaccessible due to Firefox blocking servers with weak DH ciphers

[Matthew Huff]

After making the about:config changes, no warning is given to the user about 
the bad ciphers. Even if you click the SSL lock icon, no warning is given. Only 
if you know that the connection being made with 
TLS_RSA_WITH_AES_128_CBC_SHA,128 bit keys, TLS 1.0 is a bad thing would you 
have any clue.







Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Robert Drake
Sent: Friday, July 17, 2015 8:42 AM
To: nanog@nanog.org
Subject: Re: SEC webpages inaccessible due to Firefox blocking servers with 
weak DH ciphers



On 7/17/2015 4:26 AM, Alexander Maassen wrote:
 Well, this block also affects people who have old management hardware
 around using such ciphers that are for example no longer supported. In my
 case for example the old Dell DRAC's. And it seems there is no way to
 disable this block.

 Ok, it is good to think about security, but not giving you any chance to
 make exceptions is simply forcing users to use another browser in order to
 manage those devices, or to keep an old machine around that not gets
 updated.

Or just fallback to no SSL in some cases :(  We have some old vendor 
things that were chugging along until everyone upgraded firefox and then 
suddenly they stopped working.  The fix was to use the alternate 
non-SSL web port rather than upgrade because even though the software is 
old, it's too critical to upgrade it in-line.

The long term fix is to get new hardware and run it all in virtual 
machines with new software on top, but that may be in next years 
budget.  I've also got a jetty server (opennms) that broke due to this, 
so I upgraded and fixed the SSL options and it's still broken in some 
way that won't log errors.  I have no time to track that down so the 
workaround is to use the unencrypted version until I can figure it out.

Having said that, it seems that there is a workaround in Firefox if 
people need it.  about:config and re-enabling the weak ciphers. 
Hopefully turning them on leaves you with a even bigger warning than 
normal saying it's a bad cert, but you could get back in.  This doesn't 
help my coworkers.  I'm not going to advise a bunch of people with 
varying levels of technical competency to turn on weak ciphers, but it 
does help with a situation like yours where you absolutely can't update 
old DRAC stuff.

https://support.mozilla.org/en-US/questions/1042061

SEC webpages inaccessible due to Firefox blocking servers with weak DH ciphers

[Matthew Huff]

Just ran into this issue this morning. The SEC requires companies to file EDGAR 
reports on https://edgarfiling.sec.gov. The newer versions of Firefox won't let 
you access the webpages without manually going into about:config and 
re-enabling the weak ciphers. Given the recent issue with the OPM, I would 
think this would be a very bad follow-up if the SEC got hacked.

SSLLabs gives the website an F. IE 11 won't work either (for other reasons).  
https://www.ssllabs.com/ssltest/analyze.html?d=edgarfiling.sec.gov

The website looks like it was designed in the '90s. I've tried to reach out to 
their contacts (webmaster, oig, etc...) but haven't gotten a reply yet. It's 
possible that I might get a reply eventually, but does anyone have any direct 
contacts at the SEC?



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

Re: Speaking of NTP...

[Matthew Huff]

Thanks. We have always had a few outliers, but we have never had a large number 
of external NTP have a consistent offset, and not one as big as 10ms. Something 
changed last Friday, probably at some peering point that caused the issue. 
Maybe a symmetric path got created to route around some outage. Maybe some MPLS 
circuit got introduced into the mix that hides the underlying path/latency. 
Glad to know someone else has seen something like this.

Our 3 NTP servers that sync from external sources have at least 5 upstream 
stratum 1 servers and are peered to each other . They have settled on  a sense 
of time that is good within +/i 1 msec of our strata 1 clocks, so all is good, 
but it was a stage occurrence after we had been good for so long. Each of our 
servers are clients of our 2 x strata 1 servers and 3 x strata 2 NTP servers. 
They all look good now.

 


 On Jul 16, 2015, at 3:24 PM, Tony Hain alh-i...@tndh.net wrote:
 
 I have had a consistent 10ms offset on a set of servers for the last 5 years. 
 After extensive one-way tracing, it turns out there is a 20ms asymmetry 
 within the Seattle Westin colo between HE  Comcast, causing all the IPv6 
 peers appearing over the HE tunnel to be 10ms offset from everything else. 
 There may be other instances of indirect peering causing a static asymmetric 
 path delay, and NTP will report that as an offset of half of the difference. 
 
 Tony
 
 
 -Original Message-
 From: NANOG [mailto:nanog-bounces+alh-ietf=tndh@nanog.org] On
 Behalf Of Rafael Possamai
 Sent: Thursday, July 16, 2015 8:53 AM
 To: Matthew Huff
 Cc: nanog@nanog.org
 Subject: Re: Speaking of NTP...
 
 Depending on how exactly you have these servers configured with relation
 to one another, small variations from one single source can be augmented
 down the line.
 
 https://en.wikipedia.org/wiki/Propagation_of_uncertainty
 
 
 
 On Mon, Jul 13, 2015 at 8:17 AM, Matthew Huff mh...@ox.com wrote:
 
 We have 5 NTP server:  2 x stratum 1 rubidium oscillator time servers
 with GPS sync, and 3 servers running NTP 4.2.6p5-3 synced to external
 internet based NTP stratum 1 servers. We monitor our NTP environment
 closely, and over the last 10+ years, normally all of our NTP servers
 are sync'ed within
 +/- 2 msec. Starting last Friday, we started seeing some remote NTP
 +servers
 with GPS reference consistently offset by 10 msec.
 
 Any one else seeing this?
 
 
 Matthew Huff | 1 Manhattanville Rd
 Director of Operations   | Purchase, NY 10577
 OTA Management LLC   | Phone: 914-460-4039
 aim: matthewbhuff| Fax:   914-694-5669
 
 
 

RE: Dual stack IPv6 for IPv4 depletion

[Matthew Huff]

Exactly.

As a business entity and not a provider, we wouldn't have even contemplated 
deploying IPv6 without PI addresses. The myth of easy renumbering and/or having 
multiple prefixes/address per host for failover still shows up from time to 
time, but mostly gets ignored (at least in the corporate world). Remember SHIM? 

Any reasonable size organization that expects reliable internet connections is 
going to go BGP/PI.




Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of John R. Levine
Sent: Tuesday, July 14, 2015 4:50 PM
To: Chuck Church
Cc: nanog@nanog.org
Subject: RE: Dual stack IPv6 for IPv4 depletion

 What about dual-homed customers?  Or are they all expected to have their own
 PI space?

This is IPv6.  Why shouldn't they have their own PI space?

R's,
John

Speaking of NTP...

[Matthew Huff]

We have 5 NTP server:  2 x stratum 1 rubidium oscillator time servers with GPS 
sync, and 3 servers running NTP 4.2.6p5-3 synced to external internet based NTP 
stratum 1 servers. We monitor our NTP environment closely, and over the last 
10+ years, normally all of our NTP servers are sync'ed within +/- 2 msec. 
Starting last Friday, we started seeing some remote NTP servers with GPS 
reference consistently offset by 10 msec. 

Any one else seeing this? 


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

RE: Dual stack IPv6 for IPv4 depletion

[Matthew Huff]

What am I missing? Is it just the splitting on the sextet boundary that is an 
issue, or do people think people really need 64k subnets per household?

With /56 you are giving each residential customer:

256 subnets x 18,446,744,073,709,551,616 hosts per subnet.

I would expect at least 95.0% of residential customers are using 1 subnet, and 
99.9% are using less than 4. I can understand people complaining when some ISPs 
were deciding to only give out a /64, but even with new ideas, new protocols 
and new applications, do people really think residential customers will need 
more than 256 subnets? When such a magical new system is developed, and people 
start to want it, can't ISPs start new /48 delegations? Since DHCP-PD and their 
infrastructure will already be setup for /56, it may not be easy, but it 
shouldn't be that difficult.

I know the saying build it and they will come, but seriously

I'd rather ISPs stop discussing deploying IPv6, and start doing it...

Verizon: The upgrades will start in 2013 and the first phase will include 
Verizon FiOS customers who have a dynamic IP address.. I'm still waiting...(at 
least I have a 6in4 tunnel with he.net).






Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Marco Teixeira
Sent: Thursday, July 9, 2015 11:09 AM
To: Harald Koch
Cc: NANOG list
Subject: Re: Dual stack IPv6 for IPv4 depletion

Probably because he got good advise from his father :)


On Thu, Jul 9, 2015 at 3:46 PM, Harald Koch c...@pobox.com wrote:

 On 9 July 2015 at 09:11, Mike Hammett na...@ics-il.net wrote:

  I think you're confusing very common for a tech guy and very common for
  the common man. I have a dozen or two v4 subnets in my house. Then
 again, I
  also run my ISP out of my house, so I have a ton of stuff going on. I
 can't
  even think of a handful of other people that would have more than one.
 

 My son (who is not a tech guy but is a gamer) has four subnets in his
 (rented) house already: private LAN, guest network, home control network,
 and a separate LAN for the tenant downstairs who is sharing their broadband
 connection. And he's just getting started.

 The common man is becoming much more sophisticated in their networking
 requirements, and they need this stuff to just work. Please don't place
 artificially small limits just because you can't see a need.

 --
 Harald

RE: Dual stack IPv6 for IPv4 depletion

[Matthew Huff]

When I see a car that needs a /56 subnet then I’ll take your use case 
seriously. Otherwise, it’s just plain laughable. Yes, I could theorize a use 
case for this, but then I could theorize that someday everyone will get to work 
using jetpacks.

We have prefix delegation already via DHCP-PD, but some in the IPv6 world don’t 
even want to support DHCP, how does SLAAC do prefix delegation, or am I missing 
something else? I assume each car is going to be running as  RA? Given quality 
of implementations of IPv6 in embedded devices so far, I found that pretty 
ludicrous.

Seriously, the IPv6 world needs to get a clue. Creating new protocols and 
solutions at this point in the game is only making it more difficult for IPv6 
deployment, not less. IPv6 needs to stabilize and get going.. instead it seems 
everyone is musing about theoretical world where users need 64k networks. I 
understand the idea of not wanting to not think things through, but IPv6 is how 
many years old, and we are still arguing about these things? Don’t let the 
prefect be the enemy of the good.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff| Fax:   914-694-5669

From: Harald Koch [mailto:c...@pobox.com]
Sent: Thursday, July 9, 2015 12:01 PM
To: Matthew Huff
Cc: Marco Teixeira; NANOG list
Subject: Re: Dual stack IPv6 for IPv4 depletion

On 9 July 2015 at 11:42, Matthew Huff mh...@ox.commailto:mh...@ox.com wrote:
What am I missing? Is it just the splitting on the sextet boundary that is an 
issue, or do people think people really need 64k subnets per household?

One thing you're missing is that some of these new-fangled uses for IP 
networking will want to do their own subnetting. It's not here's a subnet for 
the car, it's here's a /56 for the car to break into smaller pieces as 
required.

A /56 isn't 256 subnets, it's 8 levels of subnetting (or 2 levels, if you're 
human and want to subnet at nibble boundaries). A /48 is 16 (or 4) levels. I 
have four vehicles, so I'd want to carve out a /52 for the car network to 
make the routing and security easier to manage, and leave room for expansion 
(or for my guests...)

One more consideration for you: we're currently allocating all IPv6 addresses 
out of 2000::/3. That's 1/8th of the space available. If we discover we've 
messed up with this sparse address allocation idea, we have 7/8ths of the 
remaining space left to do something different.

--
Harald

RE: Dual stack IPv6 for IPv4 depletion

[Matthew Huff]

I've seen VLAN/subnet security used frequently in the financial world, even to 
the point of having full firewalls between vlans/subnets. Mostly for regulator 
purposes (Chinese firewall and all that). It's also common to allow outbound 
requests or redirect to different proxies based on source addresses within a 
corporate network.

In residential networks, it's mostly used for guest networks that can route out 
to the internet, but not to other local devices.




Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Tyler Applebaum
Sent: Thursday, July 9, 2015 3:38 PM
To: Naslund, Steve
Cc: nanog@nanog.org
Subject: RE: Dual stack IPv6 for IPv4 depletion

Do people actually use VLANs for security? It's nice to implement them for 
organizational purposes and to prevent broadcast propagation.

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Naslund, Steve
Sent: Thursday, July 09, 2015 12:24 PM
To: nanog@nanog.org
Subject: RE: Dual stack IPv6 for IPv4 depletion

Seems to me that the problem might be thinking that the allocation toward the 
customer is a static thing.  I think it is limiting to think that was going 
forward.  Our industry created DHCP so we didn't have to deal with statically 
configured users who did not want to deal with IP addressing.  Seems to me that 
a natural progression is to hand a network block to the CPE (DHCP-PD) and let 
it deal with it.  No reason a CPE device cannot be created that will request 
more addresses when it needs them and dynamically receive a larger assignment.

When you think about it long term our network infrastructure is pretty archaic 
in that we have to do paperwork to get an block assignment from the regional 
numbering authority and then manually chop that up.  I would expect that model 
to die over time and become more of a hierarchy whereby addresses are 
dynamically assigned top to bottom.  Seems like the numbering authority could 
be a lot more effective if a network could tell them about its utilization and 
have additional addresses assignments happen automatically.  The converse would 
be true as well, a network could reconfigure to free underutilized blocks on 
its own.  If a customer CPE needs more addresses it will request them.  If you 
add a pop to your network it should automatically get an allocation from an 
upstream device.

The only reason why anyone cares what their address is results from the fact 
that our name to address mapping via DNS is so slow to update.  The end user 
does not care what addresses they get as long as everyone can reach what they 
need to.  Your customers would not care about renumbering pain if there wasn't 
any.  Today they could care less if it is V4 or V6 as long as everyone can see 
each other.  My dad gets V6 on his cell phone and he can't even spell IP.

Another inefficient legacy is the assignment of address space on a service 
provider basis when geographic assignment would allow for better summarization. 
 If that happened you could create a better model where less routers need to 
carry a full table view of the Internet.  As long as I know how to get around 
my area and to regional routers that can reach out globally, that is all we 
need.  Now you would not have the limitation that a wide variety of routers 
need to carry every route and the /64 routing limitation goes away.  Today our 
routing is very much all or nothing.  Either use defaults or get a whole table 
via are probably the two most common options (yeah, I know there are others but 
those are the main two).

The ideas on the reasons for building VLANs is pretty out of date too.  It 
drives me nuts when I see the usual books giving you the usual example that 
accounting and their server are on one VLAN and engineering and their server 
are on another VLAN and that this is for performance and security reasons.  
Some of the biggest vendors in the business use examples like this (yes, Cisco, 
I'm looking at you) and it just does not work that way in the real world.  Who 
gets to what server is most often decided by the server (AD membership or group 
policy of some type).  If the accounting and engineering department are both 
going to a cloud service VLAN separation is pretty moot.  In a world where my 
refrigerator wants to talk to the power company and send a shopping list to my 
car, VLAN based security is not really a solution.  In the Internet of things 
we keep hearing about, everything is talking to everything. Security is highly 
dependent in that world on a device defending itself and not relying on a VLAN 
boundary.  From what I am seeing out there today, there are usually far too 
many VLANs and too much layer three going on in most large networks.

In the future it would seem

RE: Dual stack IPv6 for IPv4 depletion

[Matthew Huff]

Sure, they may be 100,000+  networks like that in non-technical households. 
Maybe. I doubt it, but still that would be like 0.01%. Many consumer systems 
have trouble with L3 hops (mDNS/Bonjour, etc...). First thing tech support will 
suggest it to put them on the same network. People have been taught this. My 
experience is that most people that even have a second network, it's their AP 
that sets up a Guest network, and even then, it doesn't route between the 
networks (that's sort of the whole idea).

If an ISP wants to give out a /48, great for them. If they want to give out 
only a /56, I say that's fine. What's more important to me is that they 
implement IPv6. Arguing about prefix size and SLAAC vs DHCP rather than just go 
ahead and implement things, to me is just silly. When IP was first deployed, we 
didn't have DHCP (bootp was still in it's infancy), no mDNS, etc...Lots of 
things grew up after the fact. I agree that we can't foresee what will happen 
in the future, but that to me just proves my point. Worrying about the ability 
to create complex topologies in home networks that may or may not ever be 
needed or wanted just seems absurd to me.




Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: Owen DeLong [mailto:o...@delong.com] 
Sent: Thursday, July 9, 2015 12:36 PM
To: Matthew Huff
Cc: Marco Teixeira; Harald Koch; NANOG list
Subject: Re: Dual stack IPv6 for IPv4 depletion


 On Jul 9, 2015, at 08:42 , Matthew Huff mh...@ox.com wrote:
 
 What am I missing? Is it just the splitting on the sextet boundary that is an 
 issue, or do people think people really need 64k subnets per household?
 

It's the need for a large enough bitfield to do more flexible things with 
auto-delegation in a dynamic self-organizing topology.

8 is 2x2x2 and there's really no other way you can break it down. (2x4, 4x2, 
2x2x2 is it.)

16 is 2x2x2x2 and allows many more possible topologies (4x4, 2x4x2, 2x2x4, 2x8, 
8x2, etc.)

 With /56 you are giving each residential customer:
 
 256 subnets x 18,446,744,073,709,551,616 hosts per subnet.

The host count is irrelevant to the discussion.

 
 I would expect at least 95.0% of residential customers are using 1 subnet, 
 and 99.9% are using less than 4. I can understand people complaining when 
 some ISPs were deciding to only give out a /64, but even with new ideas, new 
 protocols and new applications, do people really think residential customers 
 will need more than 256 subnets? When such a magical new system is developed, 
 and people start to want it, can't ISPs start new /48 delegations? Since 
 DHCP-PD and their infrastructure will already be setup for /56, it may not be 
 easy, but it shouldn't be that difficult.

I would expect that basing decisions about limits on tomorrows network on the 
inadequacy of today's solutions is unlikely to yield good results.

Further, I'm not so sure you are right in your belief. I suspect that there are 
many more networks in most households that you are not counting. Sure, those 
networks are currently usually disjoint, but do you really think it will always 
be that way in the future?

Every phone is a router. Ever tablet is a router. Cars are becoming routers and 
in some cases, collections of routers. Set top boxes are becoming routers.

Utility meters are becoming routers.

Laptops and desktops are capable of being routers.


 I know the saying build it and they will come, but seriously
 
 I'd rather ISPs stop discussing deploying IPv6, and start doing it.

I'm all for that, but do you have a valid reason not to give out /48s per end 
site? Just because /56 might be enough doesn't cut it. I'm asking if you can 
point to any tangible benefit obtained from handing out /56s instead? Is there 
any problem solved, labor saved, or any other benefit whatsoever to giving out 
/56s instead of /48s?

If not, then let's hand out /48s until we discover one.

Owen

Re: United Airlines is Down (!) due to network connectivity problems

[Matthew Huff]

Hmmm,

Wall Street Journal and NYSE both down….

WSJ has a static page up…

DDOS ???



 On Jul 8, 2015, at 10:51 AM, Patrick W. Gilmore patr...@ianai.net wrote:
 
 
 Lifted as of 0920 EDT.
 
 http://www.foxnews.com/us/2015/07/08/united-airlines-flights-in-us-grounded-due-to-computer-issues/?intcmp=latestnews
 
 -- 
 TTFN,
 patrick
 
 On Jul 08, 2015, at 10:06 , Marshall Eubanks marshall.euba...@gmail.com 
 wrote:
 
 http://www.reuters.com/article/2015/07/08/us-ual-flights-idUSKCN0PI1IX20150708
 
 At least, that's what I just heard on the radio. I know no other details.
 
 Regards
 Marshall Eubanks
 

Re: United Airlines is Down (!) due to network connectivity problems

[Matthew Huff]

Once is happenstance
Twice is coincidence
Three times is enemy action…

Serious, could all be just everyone having a bad day. On the other hand, the 
WSJ has to deal with DOS/DDOS all the time, and usually if the NYSE has issues, 
it’s normally on a Monday.



 On Jul 8, 2015, at 12:36 PM, Paul Ferguson fergdawgs...@mykolab.com wrote:
 
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA256
 
 All completely coincidental networking issues, not related to anything
 malicious.
 
 - - ferg
 
 
 On 7/8/2015 9:26 AM, Matthew Huff wrote:
 
 Hmmm,
 
 Wall Street Journal and NYSE both down….
 
 WSJ has a static page up…
 
 DDOS ???
 
 
 
 On Jul 8, 2015, at 10:51 AM, Patrick W. Gilmore
 patr...@ianai.net wrote:
 
 
 Lifted as of 0920 EDT.
 
 http://www.foxnews.com/us/2015/07/08/united-airlines-flights-in-us-g
 rounded-due-to-computer-issues/?intcmp=latestnews
 
 
 
 - -- 
 TTFN, patrick
 
 On Jul 08, 2015, at 10:06 , Marshall Eubanks
 marshall.euba...@gmail.com wrote:
 
 http://www.reuters.com/article/2015/07/08/us-ual-flights-idUSKCN0PI1
 IX20150708
 
 
 
 At least, that's what I just heard on the radio. I know no other details
 .
 
 Regards Marshall Eubanks
 
 
 
 
 - -- 
 Paul Ferguson
 PGP Public Key ID: 0x54DC85B2
 Key fingerprint: 19EC 2945 FEE8 D6C8 58A1 CE53 2896 AC75 54DC 85B2
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v2
 
 iF4EAREIAAYFAlWdUZoACgkQKJasdVTchbK1vAD/Q9gFlefUn9rIzlaRUMHvU0Ku
 Nmv6PSUUUD9f5LRLxX0BAMvXl4G5YE/ZTiz9sB5i/x5BRgmVG9XxzY5nZd/0zNtj
 =Hpoz
 -END PGP SIGNATURE-

Re: United Airlines is Down (!) due to network connectivity problems

[Matthew Huff]

Given that the technical resources at the NYSE are significant and the lengthy 
duration of the outage, I believe this is more serious than is being reported. 
OTOH, the fact that the market is now mostly decentralized and instruments are 
multiply listed, the impact of the NYSE is much less serious than it used to be.


 On Jul 8, 2015, at 1:18 PM, Paul Ferguson fergdawgs...@mykolab.com wrote:
 
 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA256
 
 Given that the Internet is held together with paper clips, bailing
 twine, and bubblegum, I'd prefer to take theses organizations' initial
 word for the fact that there is nothing obviously malicious in these
 outages.
 
 The mainstream press, on the other hand, seems to want it to be a hack
 or data breach or... something other than a glitch. :-)
 
 - - ferg
 
 
 On 7/8/2015 10:15 AM, Mel Beckman wrote:
 
 It's important to not form an opinion too early, especially anyone 
 involved with forensic analysis of these systems. This is a
 classic fault in amateur investigation: an early opinion will lead
 you into confirmation bias, irrationally accepting data agreeing
 with your opinions and rejecting that disproving it.
 
 -mel beckman
 
 On Jul 8, 2015, at 10:07 AM, Paul Ferguson 
 fergdawgs...@mykolab.com wrote:
 
 NYSE: The issue we are experiencing is an internal technical issue
 and is not the result of a cyber breach.
 
 https://twitter.com/NYSE/status/618818929906085888
 
 United Air statement CNBC: “An issue with a router degraded network
 connectivity for various applications. We fixed the router.
 
 https://twitter.com/barronstechblog/status/618816643821633536
 
 - ferg
 
 
 
 - -- 
 Paul Ferguson
 PGP Public Key ID: 0x54DC85B2
 Key fingerprint: 19EC 2945 FEE8 D6C8 58A1 CE53 2896 AC75 54DC 85B2
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v2
 
 iF4EAREIAAYFAlWdW3cACgkQKJasdVTchbLr/wD/aBNnLFv+MU+QI1ja7dd9LiSN
 Zkum4lSIutxFn1NmaYoBAIgO/Ig7FxD4vRzQK8bUturn4YGw9FXMT+EzVTKhIbVG
 =/yYp
 -END PGP SIGNATURE-

Re: United Airlines is Down (!) due to network connectivity problems

[Matthew Huff]

I did say significant…not brilliant :)

Still, it’s possible that Valdis is correct, something got changed that wasn’t 
easy to undo. Might be a combination of network/software changes that will 
require significant overnight downtime.





On Jul 8, 2015, at 1:46 PM, Shane Ronan 
sh...@ronan-online.commailto:sh...@ronan-online.com wrote:


I think you are over estimating the technical resources at NYSE.

On Jul 8, 2015 1:44 PM, Matthew Huff mh...@ox.commailto:mh...@ox.com 
wrote:
Given that the technical resources at the NYSE are significant and the lengthy 
duration of the outage, I believe this is more serious than is being reported. 
OTOH, the fact that the market is now mostly decentralized and instruments are 
multiply listed, the impact of the NYSE is much less serious than it used to be.


 On Jul 8, 2015, at 1:18 PM, Paul Ferguson 
 fergdawgs...@mykolab.commailto:fergdawgs...@mykolab.com wrote:

 -BEGIN PGP SIGNED MESSAGE-
 Hash: SHA256

 Given that the Internet is held together with paper clips, bailing
 twine, and bubblegum, I'd prefer to take theses organizations' initial
 word for the fact that there is nothing obviously malicious in these
 outages.

 The mainstream press, on the other hand, seems to want it to be a hack
 or data breach or... something other than a glitch. :-)

 - - ferg


 On 7/8/2015 10:15 AM, Mel Beckman wrote:

 It's important to not form an opinion too early, especially anyone
 involved with forensic analysis of these systems. This is a
 classic fault in amateur investigation: an early opinion will lead
 you into confirmation bias, irrationally accepting data agreeing
 with your opinions and rejecting that disproving it.

 -mel beckman

 On Jul 8, 2015, at 10:07 AM, Paul Ferguson
 fergdawgs...@mykolab.commailto:fergdawgs...@mykolab.com wrote:

 NYSE: The issue we are experiencing is an internal technical issue
 and is not the result of a cyber breach.

 https://twitter.com/NYSE/status/618818929906085888

 United Air statement CNBC: “An issue with a router degraded network
 connectivity for various applications. We fixed the router.

 https://twitter.com/barronstechblog/status/618816643821633536

 - ferg



 - --
 Paul Ferguson
 PGP Public Key ID: 0x54DC85B2
 Key fingerprint: 19EC 2945 FEE8 D6C8 58A1 CE53 2896 AC75 54DC 85B2
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v2

 iF4EAREIAAYFAlWdW3cACgkQKJasdVTchbLr/wD/aBNnLFv+MU+QI1ja7dd9LiSN
 Zkum4lSIutxFn1NmaYoBAIgO/Ig7FxD4vRzQK8bUturn4YGw9FXMT+EzVTKhIbVG
 =/yYp
 -END PGP SIGNATURE-

Re: United Airlines is Down (!) due to network connectivity problems

[Matthew Huff]

I've been working at a trading firm for the last 18 years. Most of the Market 
traditionally rolls out changes out over the weekends, making every Monday an 
adventure. It's unusual that they would roll out anything during the week, but 
they could have had something that failed and had to be undone last weekend, 
they rolled it out last night because they thought they had it fixed. They may 
have had a reason why they needed it out in a hurry.

The summer is a big time for changes because it's so less busy.. We usually 
roll out changes on Thursday nights since Friday's are the least busy. Summer 
Friday's are completely dead.

This puts NYSE in a double bad light. First the glitch and second the market 
traded close to normal without the NYSE.

On Jul 8, 2015, at 5:49 PM, Dovid Bender 
do...@telecurve.commailto:do...@telecurve.com wrote:

Well that's a given. I am talking about organizations like the NYSE or MaBell,

On Wed, Jul 8, 2015 at 5:44 PM, Keith Stokes 
kei...@neilltech.commailto:kei...@neilltech.com wrote:
Who roles out software in the middle of the week and not on weekends? People 
who have more business on the weekends than the week, such as retail.

On Jul 8, 2015, at 4:40 PM, Dovid Bender 
do...@telecurve.commailto:do...@telecurve.com wrote:

Other than for an emergency repair who roles out a software update in
middle of the week? We test, test and then test some more and only then
roll out on weekends. Our maintenance window is 00:00 - 01:00 Sunday
mornings for sw updates etc.


On Wed, Jul 8, 2015 at 3:02 PM, Matthew Huff 
mh...@ox.commailto:mh...@ox.com wrote:

Traders on the floor are being told that it's a software glitch from new
software that was rolled out Tuesday night. Nothing official has been
said.  The only thing I know for sure is that if the NYSE was hacked, they
wouldn't tell anyone the details for a long time, if ever.

The impact of the NYSE being down is much less significant than it used to
be since most stocks are multiple-listed on other exchanges.

The lack of information through official channels is unusual though. In
previous situations, there has been at least a little hand-holding. So far,
nada. In fact, other than financial service provider's emails, there has
been no emails so far today from the NYSE, including the announcement of
resumption of service. According the the NYSE web page, trading will resume
at 3:05pm EST today with primary specialist, and 3:10 for everyone.




On Jul 8, 2015, at 2:33 PM, Brett Frankenberger 
rbf+na...@panix.commailto:rbf+na...@panix.com
wrote:

On Wed, Jul 08, 2015 at 01:55:43PM -0400, 
valdis.kletni...@vt.edumailto:valdis.kletni...@vt.edu wrote:
On Wed, 08 Jul 2015 17:42:52 -, Matthew Huff said:

Given that the technical resources at the NYSE are significant and
the lengthy duration of the outage, I believe this is more serious
than is being reported.

My personal, totally zero-info suspicion:

Some chuckleheaded NOC banana-eater made a typo, and discovered an
entirely new class of wondrous BGP-wedgie style We know how we got
here, but how do we get back? network misbehaviors

We don't know how long the underlying problem lasted, and how much of
the continued outage time is dealing with the logistics of restarting
trading mid-day.  Completely stopping and then restarting trading
mid-day is likely not a quick process even if the underlying technical
issue is immediately resolved.

(Such things have happened before - like the med school a few years ago
that
extended their ethernet spanning tree one hop too far, and discovered
that
merely removing the one hop too far wasn't sufficient to let it come
back up...)

No, but picking a bridge in the center, giving it priority sufficient
for it to become root, and then configuring timers[1] that would
support a much larger than default diameter, possibly followed by some
reboots, probably would have.

From what has been publicly stated, they likely took a much longer and
more complicated path to service restoration than was strictly
necessary.  (I have no non-public information on that event.  There may
be good reasons, technical or otherwise, why that wasn't the chosen
solution.)

   -- Brett

[1] You only have to configure them on the root; non-root bridges use
what root sends out, not what they ahve configured.




---

Keith Stokes

Re: United Airlines is Down (!) due to network connectivity problems

[Matthew Huff]

Traders on the floor are being told that it’s a software glitch from new 
software that was rolled out Tuesday night. Nothing official has been said.  
The only thing I know for sure is that if the NYSE was hacked, they wouldn’t 
tell anyone the details for a long time, if ever.

The impact of the NYSE being down is much less significant than it used to be 
since most stocks are multiple-listed on other exchanges.

The lack of information through official channels is unusual though. In 
previous situations, there has been at least a little hand-holding. So far, 
nada. In fact, other than financial service provider’s emails, there has been 
no emails so far today from the NYSE, including the announcement of resumption 
of service. According the the NYSE web page, trading will resume at 3:05pm EST 
today with primary specialist, and 3:10 for everyone.




 On Jul 8, 2015, at 2:33 PM, Brett Frankenberger rbf+na...@panix.com wrote:
 
 On Wed, Jul 08, 2015 at 01:55:43PM -0400, valdis.kletni...@vt.edu wrote:
 On Wed, 08 Jul 2015 17:42:52 -, Matthew Huff said:
 
 Given that the technical resources at the NYSE are significant and
 the lengthy duration of the outage, I believe this is more serious
 than is being reported.
 
 My personal, totally zero-info suspicion:
 
 Some chuckleheaded NOC banana-eater made a typo, and discovered an
 entirely new class of wondrous BGP-wedgie style We know how we got
 here, but how do we get back? network misbehaviors
 
 We don't know how long the underlying problem lasted, and how much of
 the continued outage time is dealing with the logistics of restarting
 trading mid-day.  Completely stopping and then restarting trading
 mid-day is likely not a quick process even if the underlying technical
 issue is immediately resolved.
 
 (Such things have happened before - like the med school a few years ago that
 extended their ethernet spanning tree one hop too far, and discovered that
 merely removing the one hop too far wasn't sufficient to let it come back 
 up...)
 
 No, but picking a bridge in the center, giving it priority sufficient
 for it to become root, and then configuring timers[1] that would
 support a much larger than default diameter, possibly followed by some
 reboots, probably would have.  
 
 From what has been publicly stated, they likely took a much longer and
 more complicated path to service restoration than was strictly
 necessary.  (I have no non-public information on that event.  There may
 be good reasons, technical or otherwise, why that wasn't the chosen
 solution.)
 
 -- Brett
 
 [1] You only have to configure them on the root; non-root bridges use
 what root sends out, not what they ahve configured.

Re: REMINDER: LEAP SECOND

[Matthew Huff]

Yes, the clock has to be bad. Been there, done that, especially early Sun x86 
servers.

Leap years and DST are both things people and developers are aware of outside 
of technology, leap seconds, not so much.

 On Jun 23, 2015, at 11:33 PM, Harlan Stenn st...@ntp.org wrote:
 
 Matthew Huff writes:
 A backward step is a known issue and something that people are more
 comfortable dealing with as it can happen on any machine with a noisy
 clock crystal.
 
 A clock crystal has to be REALLY bad for ntpd to need to step the clock.
 
 Having 61 seconds in a minute or 86401 seconds in a day is a different
 story.
 
 Yeah, leap years suck too.
 
 And those jumps around daylight savings time.
 
 H

RE: REMINDER: LEAP SECOND

[Matthew Huff]

That won't work. Being internally sync'ed isn't good enough for FINRA. All the 
machines must be synced to an external accurate source at least once per 
trading day.

Our plan is to disable our two stratum 1 servers, and our 3 stratum 2 servers 
before the leap second turnover, but to be 100% safe we would need to do that 
24 hours before, but that would be a violation of FINRA regulations. 

It looks like the safest thing for us to do is to keep our NTP servers running 
and deal with any crashes/issues. That's better than having to deal with FINRA.



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: Tore Anderson [mailto:t...@fud.no] 
Sent: Wednesday, June 24, 2015 3:26 PM
To: Matthew Huff
Cc: nanog2
Subject: Re: REMINDER: LEAP SECOND

* Matthew Huff

 I saw that, but it says the bits are set before 23:59 on the day of
 insertion, but I was hoping that I could shut it down later than
 23:59:59 of the previous day (8pm EST). The reason is FINRA
 regulations. We have to have the time synced once per trading day
 before the open according to the regulations.

Again AIUI, and I'm no NTP expert so I hope someone corrects me if I'm
wrong:

If you don't configure the leapfile ntpd option, the Leap
Indicator flag will flow down to your servers from the stratum-1s
servers you're synchronising from (directly or indirectly).

So what I think you could do, is to on the 29th remove all your
upstream servers from your NTP server's config, and set fudge
127.127.1.0 stratum 3 or something like that so that clients will
still want to sync to it. At that point, your NTP server's clock chip
will be the reference clock, which might be drift-prone. To work around
that, you could at 8pm on the 30th stop ntpd, manually sync the system
clock with ntpdate, and start ntpd again. That should keep your NTP
server's clock reasonably synchronised, that provides your clients with
(a Leap Indicator-free) NTP service.

I make no guarantees that the above will work the way I think it will,
though... Try it at your own risk.

Tore

Re: REMINDER: LEAP SECOND

[Matthew Huff]

Does anyone know what the latest that we can run our NTP servers and not 
distribute the LEAP_SECOND flag to the NTP clients?


 On Jun 24, 2015, at 2:33 PM, Tore Anderson t...@fud.no wrote:
 
 * Majdi S. Abbas
 
 On Wed, Jun 24, 2015 at 08:33:14AM +0200, Tore Anderson wrote:
 Leap years and DST ladjustments have never caused us any major
 issues. It seems these code paths are well tested and work fine.
 
  I've seen quite a few people that for whatever reason insist
 on running systems in local time zones struggle with the DST reverse
 step.  It's not nearly as much of a non-issue as you claim.
 
 Read again, and note the word us. I am describing my and my
 employer's experience with past DST changes and leap years, and those
 have indeed been completely uneventful.
 
 YMMV.
 
 The leap second in 2012 however ... total and utter carnage.
 Application servers, databases, etc. falling over like dominoes. All
 hands on deck in the middle of the night to clean up. It took days
 before we stopped finding broken stuff.
 
  Total and utter carnage is a bit of a stretch.
 
 As above, I am speaking only about how the 2012 leap second went down
 in our infrastructure. I stand by how I described the event.
 
 Again, YMMV. If you plan on let your infrastructure deal with the
 upcoming leap second head-on, I wish you the best of luck. Hopefully
 all the bugs from 2012 have been fixed. I, however, certainly have no
 intention of being the one to find out otherwise.
 
 Tore

RE: REMINDER: LEAP SECOND

[Matthew Huff]

I saw that, but it says the bits are set before 23:59 on the day of 
insertion, but I was hoping that I could shut it down later than 23:59:59 of 
the previous day (8pm EST). The reason is FINRA regulations. We have to have 
the time synced once per trading day before the open according to the 
regulations. 

We could manually run ntpdate on 100+ servers including 50+ windows servers, 
but that's not a great solution.



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: Tore Anderson [mailto:t...@fud.no] 
Sent: Wednesday, June 24, 2015 3:07 PM
To: Matthew Huff
Cc: nanog2
Subject: Re: REMINDER: LEAP SECOND

* Matthew Huff

 Does anyone know what the latest that we can run our NTP servers and
 not distribute the LEAP_SECOND flag to the NTP clients?

From http://support.ntp.org/bin/view/Support/NTPRelatedDefinitions:

  Leap Indicator

This is a two-bit code warning of an impending leap second to be
inserted in the NTP timescale. The bits are set before 23:59 on the
day of insertion and reset after 00:00 on the following day. This
causes the number of seconds (rollover interval) in the day of
insertion to be increased or decreased by one. 

So the answer to your question is, AIUI, 2015-06-29 23:59:59.

Tore

Re: REMINDER: LEAP SECOND

[Matthew Huff]

A backward step is a known issue and something that people are more comfortable 
dealing with as it can happen on any machine with a noisy clock crystal.

Having 61 seconds in a minute or 86401 seconds in a day is a different story.

 On Jun 23, 2015, at 8:37 PM, Harlan Stenn st...@ntp.org wrote:
 
 shawn wilson writes:
 On Jun 23, 2015 6:26 AM, Nick Hilliard n...@foobar.org wrote:
 
 
 
 Blocking NTP at the NTP edge will probably work fine for most situations.
 Bear in mind that your NTP edge is not necessarily the same as your
 network
 edge.  E.g. you might have internal GPS / radio sources which could
 unexpectedly inject the leap second.  The larger the network, the more
 likely this is to happen.  Most organisations have network fossils and ntp
 is an excellent source of these.  I.e. systems which work away for years
 without any problems before one day accidentally triggering meltdown
 because some developer didn't understand the subtleties of clock
 monotonicity.
 
 
 NTP causes jumps - not skews, right?
 
 Left to its default condition, ntp will step/jump a change in excess of
 128msec.
 
 If you want to slew the clock instead, a 1 second correction will take a
 little over 33 minutes' time to apply.
 
 I don't understand why people believe that stopping ntpd for a few
 minutes while the leap second is applied will help.  If the system clock
 keeps good time, it will *still* be about 1 second ahead when ntpd is
 restarted, and that will trigger a backward step which is fatal to a
 number of applications.
 
 H

Re: Android (lack of) support for DHCPv6

[Matthew Huff]

+1

One IP per device will almost most likely be the preference and implementation 
in corporate/enterprise deployments. Too much procedure, regulation and other 
roadblocks prevent any other solution.

Authentication, Authorization, Accounting, ACLS, NMS, IDS, IP management, 
custom software, and other roadblocks will certainly stall if not stop IPv6 
deployments in enterprises if there isn’t at least the choice of static, single 
IPv6 addresses per device. SLAAC will probably be a complete non-starter in 
many corporate environments. It is in ours. The more ideologues preach about 
restoring peer-to-peer connectivity, dynamic IPs, privacy addresses, etc… the 
less penetration IPv6 will happen in corporate networks.



 On Jun 10, 2015, at 2:11 PM, Ray Soucy r...@maine.edu wrote:
 
 I've already written systems to do this kind of thing, but the logging
 requirements quickly go through the roof for a non-trivial network;
 especially in the case of temporary addressing now default on many
 systems.  That isn't so much the issue as operational consistency and
 supportability.
 
 The requirement for hosts to be dual stack will exist for some time.
 
 For the sanity of IT workers and users alike (most of which don't really
 know the details of IPv6 and barely understand address scopes let alone
 multiple addresses) there needs to be some level of consistency between how
 hosts are addressed and communicate between the two protocols.  Saying
 we'll develop one system for IPv4 and one for IPv6 ultimate results in
 Let's not deploy IPv6 just yet.
 
 This rings especially true when security concerns come up.  Consistency
 between IPv4 and IPv6 needs to be possible in this transition period, or
 people simply decide to not deploy.  Consistency in addressing and
 maintaining a one host one address model has major implications for policy
 construction, flow analysis and incident response, denial of service
 mitigation, etc.  If the consistency isn't there, you need to re-invent
 the wheel for every aspect, then maintain different systems for IPv4 and
 IPv6 along side one another.
 
 Even worse, if we keep seeing adoption held up by inconsistent client
 implementations I fear we'll start seeing commercial products which NAT or
 proxy IPv4 to IPv6 to avoid using IPv6 internally.  It's very ugly and
 requires some DNS magic to work, but it's not impossible.  We're already
 seeing this in terms of cloud computing and virtualization.  Servers have
 IPv4 addresses and only a load balancer with an external IPv6 address.
 
 We absolutely need to make it possible for people to adopt IPv6 before we
 can reach a point where IPv4 can go away, and for some organizations the
 answer of Just use SLAAC is simply not acceptable.
 
 They just won't do it.
 
 Preventing IPv6 adoption hurts us all.  And Android not supporting a MAJOR
 part of IPv6 like DHCPv6 is preventing adoption.
 
 
 
 
 
 On Wed, Jun 10, 2015 at 1:42 PM, Sander Steffann san...@steffann.nl wrote:
 
 
 It's not the *only* option. There are large networks - O(100k) IPv6
 nodes - that do ND monitoring for accountability, and it does work for
 them. Many devices support this via syslog, even. As you can imagine, my
 Android device gets IPv6 at work, even though it doesn't support DHCPv6.
 Other universities, too. It's obviously  not your chosen or preferred
 mechanism, but it does work.
 
 /me starts to write that whitepaper that educates people on how to do this
 
 Cheers,
 Sander
 
 
 
 
 -- 
 Ray Patrick Soucy
 Network Engineer
 University of Maine System
 
 T: 207-561-3526
 F: 207-561-3531
 
 MaineREN, Maine's Research and Education Network
 www.maineren.net

RE: dns on fios/frontier

[Matthew Huff]

Well,

There are frontier users and there are fios users, and now there are frontier 
fios users (users that were customers of Verizon, but Verizon sold off part 
their infrastructure to frontier).




Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Christopher Morrow
Sent: Monday, April 20, 2015 11:42 AM
To: Dave Pooser
Cc: North American Network Operators' Group
Subject: Re: dns on fios/frontier

On Mon, Apr 20, 2015 at 8:21 AM, Dave Pooser dave-na...@pooserville.com wrote:
 On 4/20/15, 1:54 AM, Randy Bush ra...@psg.com wrote:

[ reposted from subscribed address blush ]

anyone on fios/frontier can please run a quickie and see if you can get
to http://psg.com/?


in the other message you make clear 'a frontier customer on the fios
infrastructure'... you do mean that, not 'a frontier customer OR a
verizon fios customer' right?

(they don't share, necessarily, the same DNS or transit paths/equipment)

RE: Galaxy S6 is IPv6 on all US National Mobile carriers

[Matthew Huff]

The earlier generation of ONT has 100MB Ethernet and MOCA. If you upgrade to 
Quantum and order speeds  100MB you'll need an ONT with gig-E and switch from 
MOCA to wired Ethernet. The MOCA standard specifies up to 175MB, but I don't 
think MOCA vendors have made any adapters  100MB.




Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Joel Esler (jesler)
Sent: Tuesday, April 14, 2015 7:17 AM
To: Joe Klein
Cc: nanog@nanog.org
Subject: Re: Galaxy S6 is IPv6 on all US National Mobile carriers

I don't believe Quantum has any changes relative to the external of the house.  
Fios has been capable of pushing those speeds with the old modem for years.  
The difference between the old modem and the new one is that the wireless is 
802.11n whereas the old one was only capable of g.

--
Joel Esler
Sent from my iPhone

On Apr 13, 2015, at 11:20 PM, Joe Klein 
jskl...@gmail.commailto:jskl...@gmail.com wrote:

Was in a meeting over 4 years ago, where the people from Verizon were
claiming they would be rolling out IPv6 for FIOS in the following years.
Still waiting.

Can anyone confirm or deny that Verizon FIOS requires an upgrade to the ONT
and router for its FiOS Quantum service in order to get IPv6?

Joe Klein
Inveniam viam aut faciam

On Mon, Apr 13, 2015 at 10:25 PM, Matt Palmer 
mpal...@hezmatt.orgmailto:mpal...@hezmatt.org wrote:

On Mon, Apr 13, 2015 at 09:42:07PM -0400, Jared Mauch wrote:
On Apr 13, 2015, at 9:02 PM, Christopher Morrow 
morrowc.li...@gmail.commailto:morrowc.li...@gmail.com wrote:
On Mon, Apr 13, 2015 at 7:30 PM, Will Dean 
w...@willscorner.netmailto:w...@willscorner.net
wrote:
Reddit started using CloudFlare late last year, so they should able to
serve content up over v6.

nice!

Sorry to rain on your parade:

dhcp-7f01:~ jared% host -t  www.reddit.comhttp://www.reddit.com.
www.reddit.comhttp://www.reddit.com has no  record

should be able to serve != are serving.

- Matt

--
If you are a trauma surgeon and someone dies on your table, [...] everyone
would know you did your best.  When someone does something truly stupid
with their system and it dies and you can't resuscitate it, you must be
incompetent or an idiot.  -- Julian Macassey, in the Monastery

RE: Galaxy S6 is IPv6 on all US National Mobile carriers

[Matthew Huff]

It's much smaller J

Other than that, I don't know of anything else. I don't use their router anyway.



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff| Fax:   914-694-5669

From: Joel Esler (jesler) [mailto:jes...@cisco.com]
Sent: Tuesday, April 14, 2015 11:38 AM
To: Matthew Huff
Cc: Joe Klein; nanog@nanog.org
Subject: Re: Galaxy S6 is IPv6 on all US National Mobile carriers

So am I correct in assuming that unless you go 100Mb, and other than the N 
router to replace the G router, there isn't anything beneficial?

--
Joel Esler
Open Source Manager
Threat Intelligence Team Lead
Talos Group


I reserve the right to be wrong


On Apr 14, 2015, at 11:11 AM, Matthew Huff mh...@ox.commailto:mh...@ox.com 
wrote:

The earlier generation of ONT has 100MB Ethernet and MOCA. If you upgrade to 
Quantum and order speeds  100MB you'll need an ONT with gig-E and switch from 
MOCA to wired Ethernet. The MOCA standard specifies up to 175MB, but I don't 
think MOCA vendors have made any adapters  100MB.




Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff| Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Joel Esler (jesler)
Sent: Tuesday, April 14, 2015 7:17 AM
To: Joe Klein
Cc: nanog@nanog.orgmailto:nanog@nanog.org
Subject: Re: Galaxy S6 is IPv6 on all US National Mobile carriers

I don't believe Quantum has any changes relative to the external of the house.  
Fios has been capable of pushing those speeds with the old modem for years.  
The difference between the old modem and the new one is that the wireless is 
802.11n whereas the old one was only capable of g.

--
Joel Esler
Sent from my iPhone

On Apr 13, 2015, at 11:20 PM, Joe Klein 
jskl...@gmail.commailto:jskl...@gmail.commailto:jskl...@gmail.com wrote:

Was in a meeting over 4 years ago, where the people from Verizon were
claiming they would be rolling out IPv6 for FIOS in the following years.
Still waiting.

Can anyone confirm or deny that Verizon FIOS requires an upgrade to the ONT
and router for its FiOS Quantum service in order to get IPv6?

Joe Klein
Inveniam viam aut faciam

On Mon, Apr 13, 2015 at 10:25 PM, Matt Palmer 
mpal...@hezmatt.orgmailto:mpal...@hezmatt.orgmailto:mpal...@hezmatt.org 
wrote:

On Mon, Apr 13, 2015 at 09:42:07PM -0400, Jared Mauch wrote:
On Apr 13, 2015, at 9:02 PM, Christopher Morrow 
morrowc.li...@gmail.commailto:morrowc.li...@gmail.commailto:morrowc.li...@gmail.com
 wrote:
On Mon, Apr 13, 2015 at 7:30 PM, Will Dean 
w...@willscorner.netmailto:w...@willscorner.netmailto:w...@willscorner.net
wrote:
Reddit started using CloudFlare late last year, so they should able to
serve content up over v6.

nice!

Sorry to rain on your parade:

dhcp-7f01:~ jared% host -t  
www.reddit.comhttp://www.reddit.comhttp://www.reddit.com.
www.reddit.comhttp://www.reddit.comhttp://www.reddit.com has no  record

should be able to serve != are serving.

- Matt

--
If you are a trauma surgeon and someone dies on your table, [...] everyone
would know you did your best.  When someone does something truly stupid
with their system and it dies and you can't resuscitate it, you must be
incompetent or an idiot.  -- Julian Macassey, in the Monastery

Purpose of spoofed packets ???

[Matthew Huff]

We recently got an abuse report of an IP address in our net range. However, 
that IP address isn't in use in our networks and the covering network is null 
routed, so no return traffic is possible. We have external BGP monitoring, so 
unless something very tricky is going on, we don't have part of our prefix 
hijacked.

I assume the source address was spoofed, but this leads to my question. Since 
the person that submitted the report didn't mention a high packet rate (it was 
on ssh port 22), it doesn't look like some sort of SYN attack, but any OS 
fingerprinting or doorknob twisting wouldn't be useful from the attacker if the 
traffic doesn't return to them, so what gives?

BTW, we are in the ARIN region, the report came out of the RIPE region.



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

Re: Purpose of spoofed packets ???

[Matthew Huff]

 Another very real possibility is that the person or thing which sent
you 
 the abuse email doesn't know what he's/it's talking about.

Was my first thought, but wanted to run this by everyone in case I was
missing something obvious.




On 3/10/15, 7:51 PM, Roland Dobbins rdobb...@arbor.net wrote:


On 11 Mar 2015, at 6:40, Matthew Huff wrote:

 I assume the source address was spoofed, but this leads to my
 question. Since the person that submitted the report didn't mention a
 high packet rate (it was on ssh port 22), it doesn't look like some
 sort of SYN attack, but any OS fingerprinting or doorknob twisting
 wouldn't be useful from the attacker if the traffic doesn't return to
 them, so what gives?

Highly-distributed, pseudo-randomly spoofed SYN-flood happened to
momentarily use one of your addresses as a source.  pps/source will be
relatively low, whilst aggregate at the target will be relatively high.

Another very real possibility is that the person or thing which sent you
the abuse email doesn't know what he's/it's talking about.

;

---
Roland Dobbins rdobb...@arbor.net

RE: Large Ontario DC busted for hosting petabytes of child abuse material

[Matthew Huff]

Given the size and that the data is stored in encrypted RAR files, I wonder if 
they just busted a Usenet service provider rather than a P2P / file sharing 
site.




Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-bounces+mhuff=ox@nanog.org] On Behalf Of Naslund, 
Steve
Sent: Monday, March 2, 2015 12:54 PM
To: nanog@nanog.org
Subject: RE: Large Ontario DC busted for hosting petabytes of child abuse 
material

Don't know who this is but the legalities are pretty clear I think.  The DC is 
not required to know what data is stored but if the cops can prove that someone 
DID know what was stored, that person can be criminally charged.  IANAL but I 
have worked with LE on a similar case and that is how it was explained to us by 
the FBI.  It will be hard to prove anyone knew however since anyone that knew 
and did not report it committed a crime.  Charging the company will be a 
stretch unless they can prove that at least one corporate officer knew.  
Otherwise the company will fire whichever employee knew and say He should have 
told us.

This is all about who knew what and when.


Steven Naslund
Chicago IL


18 million dollars revenue in three months so certainly pretty large sized.

Any idea which DC this is?

http://motherboard.vice.com/en_ca/read/police-could-charge-a-data-center-in-the-largest-child-porn-bust-ever

RE: Checkpoint IPS

[Matthew Huff]

What if you are a hosting company and those aren't your servers to patch?
What about the time to patch 200+ servers versus configuring one location?
What if you have to schedule the staff and maintenance window to patch the 
servers?
What if you have legacy equipment that you must continue using, but the vendor 
is slow to provide the patch.

There is a huge difference in what is good network/security designs between 
content providers, transit networks, eyeball networks, corporate networks, 
universities, etc... One size doesn't fit all.




Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Roland Dobbins
Sent: Thursday, February 5, 2015 12:48 PM
To: nanog@nanog.org
Subject: Re: Checkpoint IPS


On 6 Feb 2015, at 0:38, Raymond Burkholder wrote:

 There must some sort of value in that?

No - patch the servers.

---
Roland Dobbins rdobb...@arbor.net

RE: Checkpoint IPS

[Matthew Huff]

You make so many assumptions, it completely negates any reasonable point you 
are trying to make:


 There are other ways (reverse proxies, on-box systems like ModSecurity, 
 et. al.); or take them offline.

What if the box isn't Linux? What if it isn't a web server. What if proxies 
don't work well with the protocol the boxes uses. What if it's an appliance a 
business unit made you setup. There a thousands of permutations like that. Many 
times you don't get to make the correct choices, you have to work with what you 
have. Any IPS, statefull firewall, application level gateways, proxies, etc. 
have their places.

In a content provider network (facebook, etc...) only using stateless 
protection because of massive DDOS is a reasonable argument. But like I said, 
one size doesn't fit all, or in this case, many.

Like it's been said before, I strongly support my competitors following your 
advice.



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Roland Dobbins
Sent: Thursday, February 5, 2015 1:11 PM
To: nanog@nanog.org
Subject: Re: Checkpoint IPS


On 6 Feb 2015, at 0:55, Matthew Huff wrote:

 What if you are a hosting company and those aren't your servers to 
 patch?

Then it isn't the operator's problem.

 What about the time to patch 200+ servers versus configuring one 
 location?

Operators should have sufficient automation to do this quickly.  If not, 
they're Doing It Wrong.

 What if you have to schedule the staff and maintenance window to patch 
 the servers?

See above.

 What if you have legacy equipment that you must continue using, but 
 the vendor is slow to provide the patch.

There are other ways (reverse proxies, on-box systems like ModSecurity, 
et. al.); or take them offline.

---
Roland Dobbins rdobb...@arbor.net

RE: OT - Verizon/ATT Cell/4G Signal Booster/Repeater

[Matthew Huff]

Be careful about the new rules that were put into place in the spring. My 
experience is that resellers are still promoting consumer devices for use in 
commercial buildings which is now a no-no. Under the new regulation, consumer 
devices are to be used only for individuals in their home, car, RV, boat, etc..

Industrial signal boosters are the only allowed non-grandfathered devices to be 
used in buildings. They have to be installed by certified installers and 
require a FCC license under the new regulations. The new fines are steep at 
$100,000 an instance, so the wireless providers really have a hold of the FCC.



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of John Schiel
Sent: Tuesday, December 16, 2014 1:28 PM
To: nanog@nanog.org
Subject: Re: OT - Verizon/ATT Cell/4G Signal Booster/Repeater


On 12/15/2014 07:45 PM, Ray Van Dolson wrote:

One thing you might also want to consider are any calls you make to 911 
whilst using a repeater.

I use a repeater supplied by T-Mobile and they made it very clear, and I 
had to specifically acknowledge a statement, that using such a repeater 
takes away from emergency services being able to find out where you are 
if you make a 911 call from your mobile.

Some may refer to this as a feature, depending on how much tin foil you 
have laying about, but the users of such device may need to be warned 
about emergency calls.  They'll need to be able to describe where they 
are to the responding sirens.

--John

 Hi all;

 Looking to improve cell reception for mixed ATT/Verizon users on the
 first floor of one of our buildings.

 Starting to dig into this and coming across items like this one at
 Amazon[1], but thought some of you out there might have recommendations
 for something that has worked well for you and has been reliable.

 Am in a position to run cable from the roof to the floor in question.

 Thanks,
 Ray

 [1] 
 http://www.amazon.com/Wilson-Electronics-Indoor-Cellular-Booster/dp/B00IWW9AB8/ref=lp_2407782011_1_1?s=wirelessie=UTF8qid=1418671553sr=1-1

RE: OT - Verizon/ATT Cell/4G Signal Booster/Repeater

[Matthew Huff]

If your users are all using the latest models... great

We still have people using flip phones...

We had to shut down our legacy signal booster when a provider sent us a cease 
and desist letter. We are still looking for a replacement solution that meets 
the new code.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Ca By
Sent: Tuesday, December 16, 2014 3:46 PM
To: Christopher Morrow
Cc: John Levine; Alex Rubenstein; nanog@nanog.org
Subject: Re: OT - Verizon/ATT Cell/4G Signal Booster/Repeater

On Tuesday, December 16, 2014, Christopher Morrow morrowc.li...@gmail.com
wrote:

 On Tue, Dec 16, 2014 at 12:32 PM, Alex Rubenstein a...@corp.nac.net
 javascript:; wrote:
 
  I just with Wifi calling was ubiquitous.

 isn't it in every android phone since ~1yr ago?


For some usa mobile providers nearly every android phone supports wifi
calling... And iPhone6 too.

For anyone doing VoLTE, VoWiFi should be a slam dunk.

CB

RE: Cisco AnyConnect speed woes!

[Matthew Huff]

Are you using SSLVpn or IPSEC with anyconnect? I have had more luck with 
performance with IPSEC than SSLVpn.

Also, just because your ISP is saying that they aren't shaping/filtering, 
doesn't mean they aren't.

We had major issues with users using AnyConnect when it was transversing 
Cogent. We were getting 5-10% packet loss (although the Cisco stats didn't show 
it), and it was choking on it.


Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Zachary McGibbon
Sent: Tuesday, December 9, 2014 2:42 PM
To: NANOG
Subject: Cisco AnyConnect speed woes!

I'm looking for some input on a situation that has been plaguing our new
AnyConnect VPN setup.  Any input would be valuable, we are at a loss for
what the problem is.

We recently upgraded our VPN from our old Cisco 3000 VPN concentrators
running PPTP and we are now running a pair of Cisco 5545x ASAs in an HA
active/standby pair.

The big issue we are having is that many of our users are complaining of
low speed when connected to the VPN.  We have done tons of troubleshooting
with Cisco TAC and we still haven't found the root of our problem.

Some tests we have done:

   - We have tested changing MTU values
   - We have tried all combinations of encryption methods (SSL, TLS, IPSec,
   L2TP) with similar results
   - We have switched our active/standby boxes
   - We have tested on our spare 5545x box
   - We connected our spare box directly to our ISP with another IP address
   - We have whitelisted our VPN IP on our shaper (Cisco SCE8000) and our
   IPS (HP Tipping Point)
   - We have bypassed our Shaper and our IPS
   - We made sure that traffic from the routers talking to our ASAs is
   synchronous, OSPF was configured to load balance but this has been changed
   by changing the costs on the links to the ASAs
   - We have verified with our two ISPs that they are not doing any kind of
   filtering or shaping
   - We have noticed that in some instances that if a user is on a low
   speed connection that their VPN speed gets cut by about 1/3.  This doesn't
   seem normal that the VPN would use this much overhead
   - We do not have the issue when connecting to VPN directly on our own
   network, only connections from the Internet

If you have any ideas on what we could try net, please let me know!

- Zachary

RE: Incident notification

[Matthew Huff]

The advantage of SMS is that it is out of band. Any smtp or other IP based 
solution requires a stable and working network environment, which is what the 
alert may be trying to tell you is down.



Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC   | Phone: 914-460-4039
aim: matthewbhuff    | Fax:   914-694-5669

-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Thijs Stuurman
Sent: Friday, November 21, 2014 10:52 AM
To: nanog@nanog.org
Subject: Incident notification

Nanog list members,

I was looking at some statistic and noticed we are sending out a massive amount 
of SMS messages from our monitoring systems.
This left me wondering if there isn't a better (and cheaper) alternative to 
this, something just as reliant but IP based. We all have smartphones these 
days anyway.

Therefore my question, what are you using to notify admins of incidents?

Kind regards / Met vriendelijke groet,

Thijs Stuurman



[IS Logo]




IS Group

Wielingenstraat 8

T

+31 (0)299 476 185

i...@is.nlmailto:i...@is.nl

1441 ZR Purmerend

F

+31 (0)299 476 288

www.is.nlhttp://www.is.nl



IS Group is ISO 9001:2008, ISO/IEC 27001:2005, ISO 20.000-1:2005, ISAE 3402 
certified. De datacenters zijn PCI DSS en ISO 14001 compliant.

Prefix withdrawals in Europe/Russia

[Matthew Huff]

BGPMon has been sending out alerts this morning starting around 15:14 UTC about 
our 129.77.0.0/16 prefix. None of our BGP peers have flapped, and according to 
the alert, it appears limited to:

Netherlands
Sweden
Kuwait
Italy
United Kingdom
Russia
Liechtenstein

I haven't seen anything on nanog or outages list, anyone know what's going on?

NJ Data center equipment movers

[Matthew Huff]

I'm looking to have some equipment (2 x HP C7000 blade chassis ( each with 16 
blades), 2 x Cisco 7600, and some small misc equipment) from a datacenter in 
Mahwah, NJ to Secaucus, NJ. Anyone recommend someone?

RE: Ars Technica on IPv4 exhaustion

[Matthew Huff]

Doesn't surprise me at all. Another thing I've seen lately is number of 
software (especially system management software) after being certified/tested 
with IPv6 no longer function when IPv6 is enabled. At least one vendor that 
broke IPv6 with a recent patch told me they only tested it once for IPv6 
compatibility to get the marketing folks off their neck. After that, they no 
longer test with IPv6 since they don't have IPv6 internally.



-Original Message-
From: NANOG [mailto:nanog-boun...@nanog.org] On Behalf Of Lee Howard
Sent: Thursday, June 19, 2014 8:54 AM
To: Frank Bulk; 'Jared Mauch'
Cc: NANOG
Subject: Re: Ars Technica on IPv4 exhaustion



On 6/17/14 11:43 PM, Frank Bulk frnk...@iname.com wrote:

These sites used to be dual-stacked:
www.cablelabs.com (over 180 days ago via ipv6.cablelabs.com)
www.att.net (over 44 days ago)
www.charter.com (over 151 days)
www.globalcrossing.com (over 802 days)
www.timewarnercable.com (over 593 days)

Check that one again.

Surprised you didn't mention www.bing.com.

Lee

Cogent / Internap issue ??

[Matthew Huff]

We are having troubles reaching services on the other side of cogent/internap 
peering. Anyone else seeing issues?

Basically 14607 - 6128 - 174 - 14744.

Type escape sequence to abort.
Tracing the route to 72.5.52.199
VRF info: (vrf in name/id, vrf out name/id)
  1 24.157.32.249 [AS 6128] 4 msec 4 msec 4 msec
  2 24.38.117.6 [AS 6128] 4 msec 8 msec 4 msec
  3 167.206.183.29 [AS 6128] 4 msec 4 msec 8 msec
  4  *  *  * 
  5  *  *  * 
  6 154.54.47.17 [AS 174] 4 msec
154.54.47.29 [AS 174] 8 msec
154.54.44.69 [AS 174] 4 msec
  7 66.28.4.202 [AS 174] 28 msec 28 msec
154.54.6.190 [AS 174] 28 msec
  8 154.54.6.85 [AS 174] 40 msec
154.54.24.81 [AS 174] 36 msec
154.54.6.117 [AS 174] 40 msec
  9 154.54.26.113 [AS 174] 52 msec
154.54.26.121 [AS 174] 52 msec
154.54.26.129 [AS 174] 48 msec
 10 154.54.25.70 [AS 174] 64 msec
154.54.25.66 [AS 174] 60 msec
154.54.25.70 [AS 174] 60 msec
 11 154.54.2.197 [AS 174] 92 msec 88 msec 84 msec
 12 154.54.0.249 [AS 174] 80 msec
154.54.0.253 [AS 174] 80 msec
154.54.0.249 [AS 174] 84 msec
 13 154.54.41.142 [AS 174] 152 msec 224 msec 92 msec
 14 38.122.90.42 [AS 174] 84 msec 84 msec
38.104.124.82 [AS 174] 80 msec
 15 63.251.160.18 [AS 14744] 76 msec 76 msec 72 msec
 16  *  *  * 
 17  *  *  * 
 18  *  *  * 
 19  *  *  * 
 20  *  *  * 
 21  *  *  * 
 22  *  *  * 
 23  *  *  * 
 24  *  *  * 
 25  *  *  * 
 26  *  *  * 
 27  *  *  * 
 28  *  *  * 
 29  *  *  * 
 30  *  *  *

Matthew Huff | 1 Manhattanville Rd
Director of Operations   | Purchase, NY 10577
OTA Management LLC       | Phone: 914-460-4039