Re: IPv6 uptake

[Stephen Satchell]

On 2/17/24 10:22 AM, Justin Streiner wrote:

Getting back to the recently revised topic of this thread - IPv6 uptake -
what have peoples' experiences been related to crafting sane v6 firewall
rulesets in recent products from the major firewall players (Palo Alto,
Cisco, Fortinet, etc)?  On the last major v6 deployment I did, working with
the firewalls was definitely one of the major pain points because the
support / stability was really lacking, or there wasn't full feature parity
between their v4 and v6 capabilities.


Depends on how complex you want to be with firewall rules.

My web server is on Ubuntu 20.04.  During the IPv4-only days, I used UFW 
(uncomplicated firewall) to implement a mostly-closed firewall, punching 
pin-holes for 80 and 443, and disable any interface forwarding.  When I 
upgraded to IPv4 and IPv6, the process of duplicating the policy in IPv6 
was easy.


The UFW package is built on top of IPTABLES and IP6TABLES.

Now, my edge router is going to be a different story.  As the number of 
rules goes up, UFW becomes tedious and finicky. Manually crafting rules 
in NFT is tedious and error-prone.  Getting all the rules right the 
first time is, um, hard.  Automation is absolutely required.  So I'm 
writing the automation in Python, and driving the rules generator from a 
YAML database.


Expect this to be published on Github.  When?  Depends on when I find 
the time.  This is not a priority project -- I'm so mad at my upstream 
that I find playing Mahjongg is necessary to settle my nerves.


I've said this earlier: by the time the NEED for IPv6 arises, I expect 
to be dead.

Re: IPv6 uptake (was: The Reg does 240/4)

[Stephen Satchell]

On 2/15/24 9:40 PM, Justin Streiner wrote:

The Internet edge and core portion of deploying IPv6 - dual-stack or
otherwise - is fairly easy. I led efforts to do this at a large .edu
starting in 2010/11.  The biggest hurdles are/were/might still be:
1. Coming up with a good address plan that will do what you want and scale
as needed.  It should also be flexible enough to accommodate re-writes if
you think of something that needs to be added/changed down the road 


Several of the resources and books I picked up over the past five years 
discuss this.  At the leaf level, coming up with a address plan is easy. 
 For example, I define two subnets:  one for public access, one for LAN 
use.  Each subnet has 64K addresses, far more than I need.  The firewall 
protects the LANnet



2. For providers who run older kit, v6 support might still be a bit dodgy.
You might also run into things like TCAM exhaustion, neighbor table
exhaustion, etc.  The point at which box X tips over is often not well
defined and depends on your use case and configuration.


Above my use level as a leaf node.  It may explain part of the situation 
I have with my upstream ISP...but I think the problem is more related to 
account management and not a technical one.



3. The last time I checked, v6 support in firewalls and other middle-mile
devices was still poor.  Hopefully that has gotten better in the last 6-7
years.  My current day job doesn't have me touching firewalls, so I haven't
kept up on developments here.  I recall coming up with a base firewall
ruleset for Cisco ASAs to balance security with the functionality v6 needs
to work correctly.  Hopefully firewall vendors have gotten better about
building templates to handle some of the heavy lifting.


In Linux, there have been significant advances in firewall support. 
Part of that support was in the kernel, part was in the tools.  The 
advent of NFT (NFTABLES) further improves things.  My replacement 
firewall design is to use YAML to define the rules; a Python driver 
converts the data into rules to implement the policy.


Can't speak for others.  By the way, instead of improving IPTABLES to 
handle IPv6, the community build IP6TABLES to support IPv6.  I was told 
that all I needed to do with my BASH-implemented firewall driver was to 
add IP6TABLE commands to the existing IPTABLES rules.  I would have done 
that if my upstream provider wasn't so IPv6-hostile.  I think that would 
have been a mistake.



4. Getting people to unlearn the "NAT=Security" mindset that we were forced
to accept in the v4 world.


That was EASY for me to unlearn.  With IPv4, I never had the luxury of 
subnetting large swaths of addresses.  With IPv6, that's easy, even in 
home networks.




That said, I'm thinking about giving up completely on IPv6 -- too many 
hurdles put in the way by my 800-pound-gorilla ISP.  I'm too old to 
fight the battle any more; the ROI isn't worth the effort.  I'll be dead 
before the lack of IPv6 connectivity becomes a personal problem.

IPv6 uptake (was: The Reg does 240/4)

[Stephen Satchell]

Several people in NANOG have opined that there are a number of mail 
servers on the Internet operating with IPv6 addresses.  OK.  I have a 
mail server, which has been on the Internet for decades.  On IPv4.


For the last four years, every attempt to get a PTR record in ip6.arpa 
from my ISP has been rejected, usually with a nasty dismissive.


Today, I'm trying again to get that all-important PTR record.  If I'm 
successful, then I expect to have my mail server fully up and running in 
the IPv6 space within 72 hours, or when the DNS changes propagate, 
whichever is longer.

Re: The Reg does 240/4

[Stephen Satchell]

On 2/14/24 4:23 PM, Tom Samplonius wrote:

The best option is what is happening right now:  you can’t get new IPv4
addresses, so you have to either buy them, or use IPv6.  The free market
  is solving the problem right now.  Another solution isn’t needed.


Really?  How many mail servers are up on IPv6?  How many legacy mail 
clients can handle IPv6?  How many MTA software packages can handle IPv6 
today "right out of the box" without specific configuration?


Does any IPv6 enabled ISP provide PTR records for mail servers?

How does Google handle mail from an IPv6 server?

The Internet is not just the Web.

Re: The Reg does 240/4

[Stephen Satchell]

On 2/14/24 9:30 AM, Owen DeLong via NANOG wrote:

That experiment already failed with the original v6 adoption process.
It’s been more than 20 years and all we have proven is that as long as
people can have an excuse to avoid v6 deployment, they will continue to
do so.

Giving them another 20 years of excuses is a step against the collective
  good IMHO.


I agree with you, based on my experience with several Internet 
providers.  One of the biggest issues I have seen is a lack of a case to 
adopt IPv6 widely and completely.  The management of the upper level 
providers ask this question: what is the return on the investment? 
Until that is convincingly answered, the foot-dragging of IPv6 adoption 
will continue.


In my particular case, it's the complete lack of support by my upstream 
provider.  Yes, they offer IPv6 connectivity.  No, they don't offer 
guaranteed public IPv6 address space.  No, they don't provide the same 
support for IPv6 that they do for IPv4.  I had to pull toenails to get 
enough information to bring up a Web server in IPv6.  It took getting a 
business fiber account to even get the bare minimum -- and I had to get 
a little creative to get the rest of the details that my ISP didn't provide.


What is the big thing missing, beside public IPv6 space?


$ dig -x 2600:1700:79b0:ddc0::3

; <<>> DiG 9.16.1-Ubuntu <<>> -x 2600:1700:79b0:ddc0::3
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 44020
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.c.d.d.0.b.9.7.0.0.7.1.0.0.6.2.ip6.arpa. IN 
PTR


Now, this is my web server's address.  My mail server's proposed IPv6 
address, is only one digit away.  Can I get a PTR record for it?  No. 
Can I get a delegation for my IPv6 address range?  No.  "We don't 
support IPv6."  That has been the refrain since 2018.  It's 2024 -- you 
do the math.


We are talking about a fairly large many-customer three-letter company, 
not some hole in the wall back-room operation.


Could I handle a delegation?  Yes.  Putting up a DNS server is child's 
play.  On a box with a public IP address.  That is not the barrier.


Now, I can't speak for all companies.  For example, I have no clue what 
support and services Hurricane Electric provides to its customers with 
regard to IPv6, even though I've seen many mentions of HE over the decades.


When the community wants to get serious about advancing the deployment 
of IPv6, the community itself needs to buy into IPv6.  At least one big 
player isn't interested.

Re: The Reg does 240/4

[Stephen Satchell]

On 2/12/24 11:07 PM, Dave Taht wrote:

if I could use the controversy to talk to why it has been so hard to
deploy ipv6 to the edge and how to fix that problem instead rather
than triggering people, it would be helpful.


1.  My provider, AT, keeps saying "we don't support IPv6."  I've 
written about my years-long effort to get my web server to speak IPv6 
over AT fiber.  I finally broke through when I was forced to upgrade 
to business service, and started receiving a better grade of technical 
support.


2.  I have a DNS  record for my web server.  Looking at yesterday's 
access log for SSL, I've had exactly five (5) accesses from two IPv6 
addresses.  Earlier in the month, I found a couple of search engines 
found the IPv6 side of the web server.


3.  I cannot obtain a PTR record for IPv6, so the mail server is a no-go 
because I won't be able to accomplish the minimum effort required for 
major players to recognize my mail server as valid.  My mail server is, 
except for port 25, LAN only.  Haven't run into any IPv6-only mail 
servers, based on the logs.


4.  My new IPv6-aware edge router firewall is in development.  This 
firewall, using NFT, will still NAT uplink IPv4 connections. It will not 
forward new connections from WAN to LAN over a defined subnet of IPv6; 
equipment on the LAN will be assigned IPv6 addresses from that subnet. 
Frankly, I'm not fast-tracking this work because I don't feel blocked by 
not having IPv6 connectivity.


It feels like IPv6 has Second Product Syndrome, where everything but the 
kitchen sink was thrown into it.

Re: sigs wanted for a response to the fcc's NOI for faster broadband speeds

[Stephen Satchell]

On 12/1/23 5:27 PM, Mike Hammett wrote:

It would be better to keep the government out of it altogether, but that has 
little chance of happening.



I agree.  But I do have a question: is there a Best Practices RFC for 
setting buffer sizes in the existing corpus?  The Internet community has 
been pretty good at setting reasonable standards and recommendations, so 
a pointer to a BCP or RFC would go much farther to solve the bufferbloat 
problem, as I believe administrators would prefer the "suggestion" 
instead of ham-handed regulation.


But that's just me.  I do know there has been academic research on the 
subject, but don't recall seeing the results published as a practical 
operational RFC.


(And this is very much on-topic for NANOG, as it is about encouraging 
our peers to implement effective operation in their networks, and in 
their connections with others.)

Re: IPv6 woes - RFC

[Stephen Satchell]

On 9/18/21 11:20 PM, Masataka Ohta wrote:

Mark Andrews wrote:

 > There is nothing at the protocol level stopping AT offering a
 > similar level of service.

Setting up reverse DNS lookup for 16B address is annoying,
which may stop AT offering it.


How many mail servers are on the Internet today?  I don't know.  How  
many business customers (large and small) does AT have?  I don't know,  
and don't expect ever to know.  I assume by "16B" you mean "16 billion";  
if so, why did you select that value?


Based on the routing tables on my edge router for my existing  
connection-based allocation, I have a IPv6 address block with a /60  
prefix.  The AT SBCIS allocation for IPv6 is 2600:1700::/28  
(4.294.967.296 /60 prefixes), and the parent range is /12.  (In a prior  
message, someone mentioned that their customer was able to obtain a  
static /56 IPv6 address block.  Don't know how they did that, unless  
they are tunneling to a different provider like HE.)


AT does offer PTR records for IPv4 static addresses -- I have a set of  
static IPv4 addresses (which I pay for monthly) and one associated  
IN-ADDR.ARPA PTR record.  (I used to have *two* sets of static IP IPv4  
addresses -- both paid for monthly -- and two associated IN-ADDR.ARPA  
PTR records, but I released one of those sets when I discontinued my  
long-time DSL service with AT)


From the AT community forum, from two years ago, a moderator says  
this:  "IPv6 Its [sic] are assigned to your connection; IPv4 static IP  
blocks are assigned to you. This is why they still don't offer reverse  
PTR delegation."


What's missing?  A static prefix of IPv6, and one IP6.ARPA PTR record.  
I'm willing to pay for IPv6 static addresses, as long as I can get the  
one IP6.ARPA PTR record for my mail server.  Connection-based prefix  
would be fine, but I still need the PTR record to satisfy the Best  
Practices requirements for mail servers.


(Why do I run my own mail server?  When I started out with DSL many  
years ago, I used Pacific Bell's mail servers.  The IP reputation was to  
the point that mail from my systems was blocked by so many mail servers  
around the 'Net.  So, Postfix locally.  Never looked back.)

Re: IPv6 woes - RFC

[Stephen Satchell]

On 9/18/21 8:58 PM, Owen DeLong wrote:

I haven’t tried the PTR thing yet, but I do have a small business client that has 
AT business internet and they were able to get a static /56 (For some reason, 
AT refused to do a /48, but we did push them on it.)


When I checked, there were NO options for ANY static IPv6.  Perhaps the 
devil is in the details of my particular "business Internet" package. 
If "package" is the right term; I use them only for upstream 
connectivity and rental of IP (and IPv6) addresses.


If it turns out they won’t do PTR or more likely NS for our ip6.arpa zone, then we’ll probably start looking for an alternative provider 


That's the problem with a facilities-based ISP, there are no alternative 
providers.  Oh, sure, I could get Spectrum here.  Indeed, I had a 
circuit: when I had their business service I had even more problems with 
them than I do with this one.



or get an HE /48 over a tunnel which will do PTR or NS records appropriately.


Hurricane Electric?  Seriously?  I had them when I was working at a web 
host company quite a while ago.  Have they improved their service desk? 
The downside is that I would have a serial pair of points of failure for 
my connectivity.


IPv6 was supposed to SOLVE the problems, not create more problems.

I look back longingly to that product from the 80s:  Internet-in-a-box.

I also remember the birth of Interop, when I visited Telebit at a 
session to work out the interoperability snags in PPP implementations 
among a handful of vendors.

Re: IPv6 woes - RFC

[Stephen Satchell]

I concur that the problem is not a routing hardware problem.  It's a 
perception problem with the various ISPs.  I have fiber service with AT


My little server farm endpoints all have IPv6 addresses, including the 
edge router.  I also have a plan to allocate IPv6 addresses to my LAN 
devices, and to protect the LAN devices from outside interference by 
rules in the NFTABLES firewall that include connection tracking on 
outbound requests.  (IPv4 will still use NAT to keep nefarious people 
from probing my internals.)


Specifically, when I was doing my mail server refresh (moving from Red 
Hat to Canonical) I decided it was time to offer IPv6 connectivity in 
the mail server to "future proof" my setup.  That included adding  
records in my DNS zone files.  Failure!  The issues:


1.  I learned that there are no "static addresses" in IPv6, as far as 
AT was concerned.  By all appearances, though, the IPv6 /64 is 
relatively static, for now, similar to the way that early cable modem 
deployments kept the same IPv4 addresses.  (Until the cable people 
started forcing changes on DHCP lease renewal, that is.)


2.  My request for PTR records was denied, which means I can't satisfy 
Best Practices for a mail server in the IPv6 space.  No PTR records, no 
redirection of ip6.apra space, nothing.  I include AT's refusal below.


3.  I don't know how to get an IPv6 allocation from ARIN, how to request 
AT to route it, or how to deal with the DNS server issues.  Oh, I know 
how to configure BIND9; I would prefer using a 24/7/365 provider.  For 
example, my master zone files are with Register.com, so if my circuit 
goes down the name resolution still happens.  Register.com appears not 
to provide reverse-DNS PTR zone support (in6.arpa).  A Google search 
turned up NOTHING for in6.arpa hosting.


That tells me that IPv6 is not "Internet Ready" for small users.  Given 
the level of FU responses I get trying to work with it, I will stop 
banging my head against the wall.


So I stick with IPv4, because that will be the "standard" until the day 
I die, as far as I can tell.


(I removed the  record, so as not to confuse mail server that DO 
operate IPv6.)



Subject: RE: Need IPv6 PTR record for my IPv6 mail server
Date: Mon, 19 Jul 2021 12:52:53 +
From: Prov-DNS 
To: Prov-DNS , a...@satchell.net 


Hello 
We don't process DNS request on IPv6 addresses. We only process DNS

request on IPv4 static assigned addresses.  If you would like us to
process a DNS request for you on a IPv4 address please provide the
following information.

IPv4 address you would like the record created for Host name you would > like that IP address pointed to 

>

Thanks
Michael AT Prov-DNS

-----Original Message-
From: Stephen Satchell 
Sent: Friday, July 16, 2021 5:42 PM
To: DNSUpdates cB 
Subject: Need IPv6 PTR record for my IPv6 mail server

Here is the record I need inserted into your ip6.arpa DNS zone:



2.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.c.d.d.0.b.9.7.0.0.7.1.0.0.6.2.ip6.arpa. 0 
IN PTR smtp.satchell.net.


This is the result from the question section of a dig(1) request for the PTR 
record for my IPv6 address 2600:1700:79b0:ddc0::32, and the fully-qualified 
domain name of the server.

You can verify the information using dig smtp.satchell.net  and 
checking the reverse.


This is the only server in my collection that needs the IPv6 pointer.

Re: Never push the Big Red Button

[Stephen Satchell]

In the data centers I've worked in over the decades, those Big Red 
Buttons would activate a normally-closed contactor in a breaker panel. 
When pushed, the contactor would open, and turn off all the circults in 
said breaker panel.  Not affected are lights, convenience outlets, door 
locks, and other non-data loads.  Resetting the contactor to the working 
position was done after throwing all the breakers to the off position, 
and then turn on each breaker, one at a time.


The only noise that I have ever heard when the Big Red Button was pushed 
was the loud BANG as the contactor operated.  You hear a similar bang in 
movies in scenes where lights in a large area are turned on and off.


Nothing like the BANG of a 600-amp 3-phase breaker tripping -- 
experienced that at University of Illinois Center for Advanced 
Computation.  You immediately look for the person holding a gun.

Re: The great Netflix vpn debacle! (geofeeds)

[Stephen Satchell]

On 9/3/21 6:54 AM, Mark Tinka wrote:
Everyone that I know who spends most of their time writing code can't 
get enough screens :-).


Size matters, too.  For example, I have a 54" screen.  My record is 
twelve open (tiled) code windows.  Usually, I have three or four code 
windows and a LibreWriter window with the specifiations and requirements.

Re: A crazy idea

[Stephen Satchell]

On 7/19/21 5:41 AM, Feldman, Mark wrote:

What you propose is not outlandish; some ISPs have been dual stack
and providing some combination of these services for years.  They
already provide IPv6 ip6.arpa delegations should their business
customers want them.  Some even provide at least a /56 so customers
can have multiple /64 subnets.  And we, I mean they, can also provide
RFC2317 in-addr.arpa delegations for those smaller IPv4 blocks.


The part that is missing isn't the "some ISPs", it's "all ISPs".  Also, 
I don't know of any DNS service provider that offers a product to handle 
delegations from the IN-ADDR.ARPA and IP6.ARPA trees.


I'm focusing on the SOHO customer market with my proposal.

The allocation of IPv6 space with prefixes shorter than /64 is indeed a 
consideration for bigger administrative domains like country 
governments, but on the other end, SOHO customers would be happy with 
/96, /104 or even /112 allocations if they could get them.  (Just how 
many light bulbs, fridges, toasters, doorbells, phones,  does SOHOs 
have?)  I would *not* like to see "us" make the same mistake with IPv6 
that was made with IPv4, handing out large blocks of space like so many 
pieces of M or Skittles candy.

A crazy idea

[Stephen Satchell]

First, I know this isn't the right place to propose this; need a pointer 
to where to propose an outlandish idea.


PROBLEM:  IPv6 support is still in its birthing pangs.  I see a problem 
that limits deployment of IPv6 fully:  reverse PTR records in the 
".in6.arpa." zones.


(Now that I think about it, this may very well be a network operator 
issue.  Who maintains the ".in.arpa." zones delegated by IANA now?)


I've been going 'round and 'round with AT about "static" IPv6 
addresses.  In particular, I can't get a PTR record in the ip6.arpa. 
zone to save my life.  Now, the problem is not really ripe yet, because 
the big reason for PTR records is for mail servers -- best practice 
calls for /PTR agreement, just like for IPv4 the best practice is 
for A/PTR agreement.


The existing DNS providers can support delegation domains, so that I 
don't have to have DNS servers of my own if I don't want to.  It could 
be that one would need to "buy" the delegation domain, but that's a 
front-office consideration.  Personally, I use register.com for my 
domain DNS zones.  I believe strongly that other registrars that offer 
customer zone editing, plus DNS service providers, can support reverse 
delegation zones with a minimum of hassle, and without charging an arm 
and a leg for the service.


From the customers' viewpoint, a GUI would make the maintenance 
relatively painless.


(Keying the information below took a long time.  Any rational DNS admin 
and DNS service provider would have automation in place to take out the 
painful work.)


What would the domain names look like?  Let's take my current IP/IPv6 
assignments from AT:


  2600:1700:79b0:ddc0::/64
  99.65.194.96/29

The IPv6 delegation would be easy:


0.c.d.d.0.b.9.7.0.0.7.1.0.0.6.2.ip6.arpa. NS my-DNS-server-1.
0.c.d.d.0.b.9.7.0.0.7.1.0.0.6.2.ip6.arpa. NS my-DNS-server-2.


because the delegation would always be on a /64 boundary. The customer 
assignments would be straightforward.  In my BIND9 zone file, it would 
look something like this:



$ORIGIN 0.c.d.d.0.b.9.7.0.0.7.1.0.0.6.2.ip6.arpa.
@ IN SOA ...
@ NS my-DNS-server-1.
@ NS my-DNS-server-2.
1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 IN PTR server1.example.com. 
2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0 IN PTR server2.example.com. 


Delegations for the IP range, not being on an octet boundary, would be a 
little more problematic:


> 96-103.194.65.99.in-addr.arpa. NS my-DNS-server-1
> 96-103.194.65.99.in-addr.arpa. NS my-DNS-server-2> $GENERATE 96-102 $ 
IN CNAME $.96-103.194.65.99.in-addr.arpa.


In my BIND9 zone file, it would look something like this:


$ORIGIN 96-103.194.65.99.in-addr.arpa.
@ SOA ...
@ NS my-dns-server-1.
@ NS my-dns-server-2.
96 IN PTR server1.example.com.
97 IN PTR server2.example.com.


The advantage to this system to the number providers is they would have 
one administrative record per customer, instead of having to deal with 
each PTR record individually.  The advantage to customers is they don't 
have to beg and snivel to get PTR records, just beg and snivel once to 
get the delegation.  The advantage to DNS server providers is they have 
something else to sell.


Want to encourage IPv6 adoption?  This would help.

Re: BCP38 on public-facing Ubuntu servers

[Stephen Satchell]

On 6/8/21 2:38 PM, Fran via NANOG wrote:

Hey,

to my knowledge there is no IPv6 equivalent for 
net.ipv4.conf.all.rp_filter.


Therefore I use netfilter to do the RP filtering for both address families.

ip(6)tables -t raw -I PREROUTING -m rpfilter --invert -j DROP

Using the raw tables less resources are used, but you could also choose 
other tables.

Details abour rpfilter can be found here [1].

This can also be achieved using nftables [2].


I've been in discussions on how to filter packets with bad source 
addresses on several mailing lists, including this one.  For the last 
few weeks, I've been search for all the information I can find for how 
Linux implements rp_filter...which appears to have some holes.


Looking at /proc/sys/net/ipv6, there is no knob for rp_filter, so if 
your system is IPv6 enabled you have to use the built-in firewall.


For IPv4, I found kernel documentation, but it doesn't tell the whole 
story.  For that, I had to comb the kernel sources to find out all the 
details of rp_filter.  I've prepared a RFC letter of what I think I 
found, to be sent to the kernel developers.  Here is the text of what 
I'll be sending, with any constructive criticism I get from here:


Letter begins:

After looking at the source that appears to implement rp_filter
linux/net/ipv4/fib_frontend.c
I believe that I now understand the tests rp_filter performs to
validate the source address when net.ipv4.conf.*.rp_filter is
set to one or two for a given interface.

Does the new paragraph I have written accurately reflect what
happens?  If so, then I find out how to submit a patch to add the
clarification to the kernel document.

Description of rp_filter from
https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt

rp_filter - INTEGER
0 - No source validation.
1 - Strict mode as defined in RFC3704 Strict Reverse Path
Each incoming packet is tested against the FIB and if the
interface is not the best reverse path the packet check will
fail. By default failed packets are discarded.
2 - Loose mode as defined in RFC3704 Loose Reverse Path
Each incoming packet's source address is also tested against
the FIB and if the source address is not reachable via any
interface the packet check will fail.

[*new text here]

Current recommended practice in RFC3704 is to enable strict mode
to prevent IP spoofing from DDos attacks. If using asymmetric
routing or other complicated routing, then loose mode is
recommended.

The max value from conf/{all,interface}/rp_filter is used
when doing source validation on the {interface}.

Default value is 0. Note that some distributions enable it
in startup scripts.


Recommended addition where marked with "[*new text here]":
rp_filter will examine the source address of an incoming IP
packet by performing an FIB lookup.  In loose mode (value 2),
the packet is rejected if the source address is neither
UNICAST nor LOCAL nor IPSEC.  For strict mode (value 1) the
interface indicated by the FIB entry must also match the
interface on which the packet arrived.

BCP38 on public-facing Ubuntu servers

[Stephen Satchell]

Not every uplink service implements BCP38.  When putting up servers 
connected more-or-less directly to the Internet through these uplinks, 
it would be nice if the servers themselves were able to implement 
ingress and egress filtering according to BCP38.  (Sorry about the typo 
in the subject lines of my previous message -- not everyone can get a 
BGP feed.)


(Or, when using Ubuntu server edition to implement edge routers.)

My earlier query was asking if anyone has encoded the blackhole routes 
in YAML for inserting in netplan(5).  My prior message contains the 
routes to be blackholed.  That takes care of egress routing.


(I think I can write a Python program to take my list and convert it to 
the YAML that netplan(5) wants to see.  That way, the routes are 
inserted when the public interface is up, and removed when the public 
interface is down.)


Ingress routing appears to be one-line addition.  IPTABLES can be told 
to weed out packets with unroutable source addresses.  My experiments 
will add something like this line to the firewall:


# iptables -A INPUT -m addrtype -i enp1s0 --src-type BLACKHOLE -j DROP

THIS HAS NOT BEEN VERIFIED.  I'm building a web server that will 
integrate this idea, and try it out.

BGP38 egress filter on Ubuntu Server

[Stephen Satchell]

Before I re-invent the wheel, has anyone come up with blackhole route 
specifications for netplan in Ubuntu servers?  Such a capability would 
perform the egress blocking for an edge server.


The table of blackhole routes I would set up:
IPv4
Address block   Scope   Description
0.0.0.0/8   SoftwareCurrent network (only valid as
source address).
10.0.0.0/8  Private network Used for local communications
within a private network.
100.64.0.0/10   Private network Shared address space[3] for
communications between a service
provider and its subscribers
when using a carrier-grade NAT.
127.0.0.0/8 HostUsed for loopback addresses to
the local host.
169.254.0.0/16  Subnet  Used for link-local addresses
between two hosts on a single
link when no IP address is
otherwise specified, such as
would have normally been
retrieved from a DHCP server.
172.16.0.0/12   Private network Used for local communications
within a private network.
192.0.0.0/24Private network IETF Protocol Assignments.
192.0.2.0/24Documentation   Assigned as TEST-NET-1,
documentation and examples.
192.88.99.0/24  InternetReserved. Formerly used for
IPv6 to IPv4 relay
192.168.0.0/16  Private network Used for local communications
within a private network.
198.18.0.0/15   Private network Used for benchmark testing of
inter-network communications
between two separate subnets.
198.51.100.0/24 Documentation   Assigned as TEST-NET-2,
documentation and examples.
203.0.113.0/24  Documentation   Assigned as TEST-NET-3,
documentation and examples.
224.0.0.0/4 InternetIn use for IP multicast.
240.0.0.0/4 InternetReserved for future use.
255.255.255.255/32  Subnet  Reserved for the "limited
broadcast" destination address.

IPv6
Address block   Usage   Purpose
::/0Routing Default route.
::/128  SoftwareUnspecified address.
::1/128 HostLoopback address to local host.
:::0:0/96   SoftwareIPv4 mapped addresses.
:::0:0:0/96 SoftwareIPv4 translated addresses.
64:ff9b::/96Global Internet IPv4/IPv6 translation.
100::/64Routing Discard prefix.
2001::/32   Global Internet Teredo tunneling.
2001:20::/28SoftwareORCHIDv2.
2001:db8::/32   Documentation   Addresses used in documentation
and example source code.
2002::/16   Global Internet The 6to4 addressing scheme
fc00::/7Private network Unique local address.
fe80::/10   LinkLink-local address.
ff00::/8Global Internet Multicast address.

Re: Texas internet connectivity declining due to blackouts

[Stephen Satchell]

When I lived in Oklahoma, the mantra of the locals was "if you don't 
like the weather, wait five minutes."  As a member of a Boy Scout troop 
in the northern part of the Sooner State, we were told, repeatedly, to 
expect anything from broiling to deep freeze on our campouts.


One such outing was on fallow farmland.  Because the campsite was in the 
middle of nowhere, we were a small group, and came in three cars.  We 
pitched four tents.  During the night, a gullywasher came through and 
dropped several inches of water in one hour.  Three of the tents were 
inundated with water, and the campers ended up sleeping in the cars.  My 
tent was dry inside, because my tent-mate and I had seen the storm 
clouds, dug a trench around the tent, and loosened the ropes.  It helped 
that we had pitched the tent on a slight mound.


Some disasters are unavoidable, like tornados.  Others allow for 
mitigation by the thoughtful.


On 2/22/21 9:18 AM, Rich Kulawiec wrote:

On Tue, Feb 16, 2021 at 12:23:22PM +, Bret Clark wrote:

Texas doesn't generally experience this type of extreme cold.


That was then; this is now.

As scientist Jeff Masters put it most of a decade ago:

The atmosphere I grew up with no longer exists.  My new motto
with regards to the weather is, "expect the unprecedented."

In the years since he's said that we've seen a number of unprecedented
events: Sandy, Harvey, California wildfires, last year's midwest derecho,
and so on.  This event in Texas is just another one; there will be more;
they'll get worse.

We should probably get ready for that.

---rsk

Re: 60 ms cross-continent

[Stephen Satchell via NANOG]

On 6/22/20 12:59 AM, adamv0...@netconsultings.com wrote:

William Herrin

Howdy,

Why is latency between the east and west coasts so bad? Speed of light
accounts for about 15ms each direction for a 30ms round trip. Where does
the other 30ms come from and why haven't we gotten rid of it?


Wallstreet did :)
https://www.wired.com/2012/08/ff_wallstreet_trading/


“Of course, you’d need a particle accelerator to make it work.”

So THAT'S why CERN wants to build an even bigger accelerator than the LHC!

Re: Mystery CDN

[Stephen Satchell]

On 6/17/20 8:29 AM, Clinton Work wrote:

I'm struggling to determine which CDN owns the servers in CenturyLink prefix 
8.240.0.0/12.   During the Call of Duty Season 4 update on June 11th from 06:00 
UTC until 08:30 UTC, we had 240 Gbps of traffic steaming into our network from 
CenturyLink prefix 8.240.0.0/12.   We originally thought it was Akamai, but 
they swear up and down that the servers don't belong to them.

Here are some of the HTTP/HTTPS servers in 8.240.0.0/12:
8.253.151.248
8.251.135.126
8.240.167.126
8.240.228.126
8.240.168.126
8.240.126.254
8.240.191.254


You might ask Level3.

Re: Abuse Desks

[Stephen Satchell]

On 4/29/20 9:57 AM, Mike Hammett wrote:

My routers have ACLs, but my servers for the most part do not.


I'm not trying to argue, but...what servers do you have that don't have 
sysadmin-definable firewalls and tun-able knobs?  My edge routers are 
Linux boxes (CentOS 8 for the one I'm now building).  Moreover, I can 
have NetworkManager fire off a script that modifies the firewall 
settings as interfaces go up and down.



It's kind of counter productive to put ACLs on SMTP, POP3, IMAP, and
HTTP\S ports, now isn't it? SIP, FTP, and SSH may or may not make
sense, depending on the type and volume of users.
I was taught by my networking betters that you need to block certain 
types of public inbound packets, always, that match any of:


1.  WAN packets with local/LAN source address
2.  WAN packets with local/LAN broadcast/net src-dst address
3.  WAN packets with known broadcast/net src-dst address
4.  WAN packets with local/LAN small services
5.  WAN packets with local/LAN unimplemented services
6.  WAN packets with blackholed source address

On EVERY device with a public IP address.  WITHOUT FAIL.

I have these blocks on every single public-facing mail server I build. 
I have these blocks on every single public-facing Web server I build. 
Indeed, I can't fathom why I would *not* have these in place for every 
single public-facing device.  I don't necessarily log every occurance, 
but I do drop matching packets on the floor, unceremoniously.


This is the foundation upon which I build custom additions, such as 
allowing 22/tcp only from specific IP addresses.


I don't depend on the edge router to catch all the cases, because each 
server has specific services it provides.  So, for example, my DNS 
servers not only implement all six basics, but also incorporates request 
rate limiting, to avoid participating in DDOS events.  Ditto NTP 
servers.  80/tcp and 443/tcp?  Dropped on the floor.


Sorry to preach, but I'm in the process of building a NFTABLE-based 
firewall and this happens to be part of the specs for it.

Re: Abuse Desks

[Stephen Satchell]

On 4/29/20 9:24 AM, Mukund Sivaraman wrote:

If there's a lock on my door, and someone tries to pick it, you can call
me at fault for having a lock on my door facing outside all you
want. But the thief picking it has no business doing so, and will be
guilty of a crime if caught.


This is a good start to an analogy.  Let's build on it, courtesy to 
YouTube's "Lock Picking Lawyer".  In a video, the host shows how to 
improve the security of a common easily-picked home lock: drill holes in 
the lock body, such that if someone picks the lock and tries to turn the 
keyway, the pins will fall into those carefully-placed holes and foil 
The Bad Guy(tm).


In the networking world, we use an Access Control List to limit access 
to the service.  Unlike the simple modification shown in LPL's video, 
the "lock" is still usable by users from authorized IP addresses.  Or, 
we require the use of certificates to validate access within the SSHD 
server itself.


Here's the deal:  just blocking access or requiring certificate-based 
access is intrusion prevention.  Having a log event when there are 
unsuccessful probes is intrusion [attempt] detection.  Sure, the 
ne'er-do-well is kept out in the prevention cycle, but a persistent 
cracker lives by the axiom "if at first you don't succeed, try something 
else."  You really want to stop an attacker from making a large number 
of attempts, such as with a Joe script.


I turn off root SSH access, pinhole 22/tcp to a limited number of IP 
addresses, and monitor failed SUDO attempts.  As I build up my new 
firewall, I'll turn off public SSH access completely, and instead use a 
robust VPN implementation.  (Which has its own issues.)

Re: Abuse Desks

[Stephen Satchell]

On 4/29/20 8:41 AM, Mel Beckman wrote:

Is there any reason to have a root-enabled (or any) ssh server
exposed to the bare Internet? Any at all? Can you name one? I can’t.
That’s basically pilot error.


Remember HeartBleed?  That didn't require a rout-enabled SSH server.  It 
didn't require SSH server.


That said, I use TCPWRAPPER to limit access to SSH to specific IP 
addresses.  I process my LogWatch messages manually.  I pull the fire 
alarm for showshoe probes, and excessive number of probes (over 30 in a 
24-hour period).  No registered abuse@ address in the WHOIS?  The 
offending netblock goes into my edge router ACL, because I have learned 
that ne'er-do-wells without working abuse@ usually have other bad habits.


And I disclose this practice to all who use my network.

(Blackmail emails are another set-and-forget trigger, but that's a 
subject for NANAE newsgroup.)

Re: Chairman Pai Proposes Mandating STIR/SHAKEN To Combat Robocalls

[Stephen Satchell]

On 3/8/20 4:00 PM, b...@theworld.com wrote:

As I've said before what would likely work is if every time one of us
(in the US anyhow) got a junk call we immediately called our
congressional and/or senate office(s) and simply said "just got
another junk call! (optionally add description.)"


Doesn't work.  I've been complaining both House and Senate offices every 
time CMS (Medicare billing arm) overcharges me $800 for my premiums. 
It's to the point that my elected officials will listen, then say "write 
a letter" (which I have done several times) and blow me off.


Nothing ever gets fixed.

BBB has told me they don't take complaints about government entities.

Re: Chairman Pai Proposes Mandating STIR/SHAKEN To Combat Robocalls

[Stephen Satchell]

On 3/8/20 9:59 AM, Damian Menscher via NANOG wrote:

In the robocall case, there*is*  something the end user can do to fight the
abuse: answer every call, and keep them on the line as long as possible.
They are paying for connected calls, for the connection duration, and for
the humans to scam people.  If everyone tarpitted them, the business model
would fail.


+1

When I recognize the name and number on caller ID, I'll answer in the 
usual manner.


I answer calls when I don't recognize the name or number, but say 
nothing.  The caller then drops the connection, usually in 10 seconds -- 
and I hear the disconnect -- and usually my cordless phone's base 
station notices the disconnect as well.  (Yes, I still have a standard 
POTS line.)


What if it's an unknown person but otherwise valid and not robo-call? 
They will notice the ringback tone stopping and will say "Hello, hello?" 
at which point I can have a conversation.  (Some robocallers will notice 
the ringback tone stopping and start their automated spew, at which 
point I press "Off.")


This helps keep my blood pressure low, keeps my answering machine from 
filling up with useless calls, and I feel good that someone just spent a 
nickle for nothing.

Re: ATT Microcell in Austin, TX

[Stephen Satchell]

There is power backup and then there is power backup.

The former is a small power pack (batteries, supercapacitors, whatever) 
that will allow the microcell to weather a short blackout or brownout. 
We are talking seconds, to bridge switching transits.  To be useful in a 
deployment, such a holdover battery needs to be very low maintenance. 
(Think about cars that use supercapacitors as a battery replacement -- 
good for short needs like a single engine start.)


The latter is longer-life power to keep the microcell going for days or 
weeks.  It's debatable whether this is mandatory.

Re: power to the internet

[Stephen Satchell]

On 12/26/19 10:55 AM, Michael Thomas wrote:
Here in California, you're going to need a lot more than 8 hours. We had 
one that lasted 3 days, followed by about 8 hours of power, followed by 
2 days of no power. If this is the new normal, and I'm afraid that it 
is, that's probably going to require some pretty hefty backup. Not to 
mention expensive.


The one "good" thing that PG did is expose all of these 
vulnerabilities. Every neighborhood probably knows whether their carrier 
is naughty or nice now.


Here in Nevada, specifically at Lake Tahoe, power is less reliable 
because of heavy snow and sliding trucks (the power equivalent to a 
backhoe disconnect).  One of the cell sites is on the top level of a 
casino parking garage.  I found out about this when the casino went 
bankrupt, the parking garage was blocked off, and I joined the security 
guard crew to protect the on-site gaming equipment.  Months into the 
project, the cell company in question begged the bankruptcy court for 
access -- to replace the empty propane cylinders in their shack.  That's 
right, no mains tap at all.  When the casino lost power because of bill 
non-payment, the cell site stayed up.


A network operator will need to look at the total cost, including labor, 
of backing up mains power. versus using local genertion exclusively -- 
or using mains power as the backup!  Factor in any upcoming fines for 
service outage, re 911.  (Try to avoid piped natural gas as the fuel for 
onsite generation.)


Longer term, review your backhauls and interconnects.  Dark fiber would 
be preferred here, because you would be controlling backup power at both 
ends, and not depending on intermediate nodes.

Re: power to the internet

[Stephen Satchell]

On 12/25/19 6:29 PM, Michael Thomas wrote:
Yes, this is exactly right. My point here isn't to assign blame, but to 
ask what the hell we're going to do about it. Trying to score political 
points is disgusting.


Do you live in California?  Do you have your business in California? 
Take a look at neighboring states.  I did.  California madness is why I 
now live in Nevada.


Our ecology doesn't have the Austrailian plant Eucalyptus.  We do have 
tumbleweeds which pose their own fire risk.  Place like Lake Tahoe is 
heavily forested -- the difference is that in Nevada there is active 
fuel control and controlled burns, so we have fewer burn-to-the-ground 
fires in populated areas.


I used to make a living as a freelance writer.  A GOOD living.  In 
Nevada I'm outside the reach of CA AB 5 should I want to give up 
$DAYJOB.  When I have to subcontract freelance work, I won't hire a 
California resident, because I don't want to be controlled by any 
"long-reach" laws.  Because the law and Calufornia court decisions are 
currently silent about C, S, and LLC writers, I avoid them until the 
climate becomes clearer.


Do I experience power outages?  Yes.  Longest duration?  Several hours, 
when high winds cause damage (but that damage doesn't start wildfires). 
One very nice thing is that where I am, we have a geothermal power plant 
close by (on the order of 15 miles).  No pre-emptive wide-area 
shutdowns, though.


NV Energy has photoelectric arrays that are part of the utility, not 
only on private roof-tops.

Expect (was: Software Defined Networks)

[Stephen Satchell]

I (and another programmer, now at Amazon) migrated our automation from 
TCL/Expect to Python/pexpect.  I've had to write code for those portions 
of Expect that didn't carry over into pexpect.


I also had to build a framework that allowed me to do rule-based 
programming in the same flavor as Expect's "expect" statement, which 
isn't hard but tedious as all get-out.  Maintenance of the code using 
the framework is head and shoulders better than the same tasks in 
Expect.  Particularly when Cisco makes little changes in their 
operations as you move up the revision chain.


On 12/12/19 6:53 PM, Quan Zhou wrote:
I do still use expect(tcl) whole lot at work, it is truly an 
underappreciated tool ever.


On 12/13/19 10:47, Large Hadron Collider wrote:

Tcl still exists, though I don't think they use it for this anymore.

On 19-12-05 10 h 17, Bryan Holloway wrote:


On 12/5/19 6:16 PM, Patrick W. Gilmore wrote:

I tell everyone we had SDNs in the 90s.

But we called it “expect scripts”.

:-)

--
TTFN,
patrick



I miss TCL ...

Re: Elephant in the room - Akamai

[Stephen Satchell]

On 12/5/19 6:02 PM, Valdis Klētnieks wrote:

(I also admit having no idea what percentage of the intermediate routers in the
ISP's networks have gotten de-bloating code.


For SP-grade routers, there isn't "code" that needs to be added to 
combat buffer bloat.  All an admin has to do is cut back on the number 
of packet buffers on each interface -- an interface setting, you see.


The reason that comsumer-grade devices can contribute to buffer bloat is 
because the vendor doesn't expose a knob to adjust buffering.  At least 
in most instances with Best Buy and Office Depot routers.

Re: Disney+ Streaming

[Stephen Satchell]

CAVAET: I don't have a dog in this hunt.

On 11/13/19 6:46 AM, Mel Beckman wrote:

This is silly off-topic. You don’t have to go home, but you can’t
stay here, according to NANOG guidelines.



https://www.nanog.org/resources/usage-guidelines/ > 
https://www.nanog.org/bylaws/


"The NANOG mailing list was established in 1994 to provide an open forum 
for the exchange of technical information, and lively discussion of 
SPECIFIC IMPLEMENTATION CHALLENGES (emphasis mine) that require 
cooperation among network service providers.


"Posts to NANOG’s mailing list should be focused on operational and 
technical content only, as described by the NANOG Bylaws."


Yes, some of the Disney Plus thread has strayed outside the four corners 
of the rules of the mailing list, but the bulk of the thread has to do 
with two things: geolocation inaccuracies, and traffic capacity shifts. 
 For some network operators on this list, the discussion does not 
describe issues on their networks.  But "some" is not "all".

Anyone from NTT America here?

[Stephen Satchell]

Routing loop

>  11.|-- 129.250.24.196 0.0% 1   28.9  28.9  28.9  28.9   0.0
>  12.|-- 129.250.130.2540.0% 1   29.0  29.0  29.0  29.0   0.0
>  13.|-- 129.250.130.2530.0% 1   29.4  29.4  29.4  29.4   0.0
>  14.|-- 129.250.130.2540.0% 1   29.6  29.6  29.6  29.6   0.0
>  15.|-- 129.250.130.2530.0% 1   28.5  28.5  28.5  28.5   0.0
>  16.|-- 129.250.130.2540.0% 1   29.0  29.0  29.0  29.0   0.0
>  17.|-- 129.250.130.2530.0% 1   28.6  28.6  28.6  28.6   0.0
>  18.|-- 129.250.130.2540.0% 1   27.9  27.9  27.9  27.9   0.0
>  19.|-- 129.250.130.2530.0% 1   28.4  28.4  28.4  28.4   0.0
>  20.|-- 129.250.130.2540.0% 1   27.9  27.9  27.9  27.9   0.0
>  21.|-- 129.250.130.2530.0% 1   28.2  28.2  28.2  28.2   0.0
>  22.|-- 129.250.130.2540.0% 1   29.0  29.0  29.0  29.0   0.0
>  23.|-- 129.250.130.2530.0% 1   27.9  27.9  27.9  27.9   0.0
>  24.|-- 129.250.130.2540.0% 1   28.6  28.6  28.6  28.6   0.0
>  25.|-- 129.250.130.2530.0% 1   28.7  28.7  28.7  28.7   0.0

Re: Request comment: list of IPs to block outbound

[Stephen Satchell]

On 10/23/19 8:18 AM, Grant Taylor via NANOG wrote:
> I suspect things like NetworkManager are somewhat at a disadvantage in
> that they are inherently machine local and don't have visibility beyond
> the directly attached network segments.  As such, they can't /safely/
> filter something that may be on the other side of a router.  Thus they
> play it safe and don't do so.

You are 100 percent correct about NetworkManager.  The facility only
manages interfaces (including VPN and bridges).  What I've done is added
the ability to install and remove null routes when the upstream
interface comes on-line and goes off-line.

So this is only the first stage of filtering.  Using NetFilter (in
CentOS 8 case, NFTABLES), I will be adding rules to implement my
policies on each system I have.  What exactly will be accepted, what
will be forwarded, what will be rejected, and what will be ignored.

What adding the null routes does is let me use the FIB test commands so
that the firewall files don't have to know the exact configuration of
networking, or have monster lists that have to be maintained.  Consider
that one suggestion from this group is to look at using
https://www.team-cymru.com/bogon-reference-http.html and doing periodic
updates of the null routes based on the information there.  (With caution.)

This is specific to Linux.  The idea is to let the computer do all the
bookkeeping work, so I don't have to.  Even if I have automation to "help".

The first application of this work will be to replace my existing
firewall router with up-to-date software and comprehensive rules to
handle NAT and DNAT, on a local network with quite a number of VLANs.

Re: Request comment: list of IPs to block outbound

[Stephen Satchell]

On 10/22/19 10:11 PM, Grant Taylor via NANOG wrote:
> The explicit nature of RFC 6598 is on purpose so that there is no chance
> that it will conflict with RFC 1918.  This is important because it means
> that RFC 6598 can /safely/ be used for Carrier Grade NAT by ISPs without
> any fear of conflicting with any potential RFC 1918 IP space that
> clients may be using.
> 
> RFC 6598 ∉ RFC 1918 and RFC 1918 ∉ RFC 6598
> RFC 6598 and RFC 1918 are mutually exclusive of each other.
> 
> Yes, you can run RFC 6598 in your home network.  But you have nobody to
> complain to if (when) your ISP starts using RFC 6598 Shared Address
> Space to support Carrier Grade NAT and you end up with an IP conflict.
> 
> Aside from that caveat, sure, use RFC 6598.

So, to the reason for the comment request, you are telling me not to
blackhole 100.64/10 in the edge router downstream from an ISP as a
general rule, and to accept source addresses from this netblock.  Do I
understand you correctly?

FWIW, I think I've received this recommendation before.  The current
version of my NetworkManager dispatcher-d-bcp38.sh script has the
creation of the blackhole route already disabled; i.e., the netblock is
not quarantined.

Request comment: list of IPs to block outbound

[Stephen Satchell]

After reviewing the comments from people on NANOG and some other
locations, I have updated my list of routes to blackhole.  The
information at the end of this contribution is taken from the
RHEL/CentOS NetworkManager dispatcher.d source file, which I use to
install and remove the blackhole routes when the WAN interface is
started and stopped.

First, let me expand on what I'm trying to do.  The NetFilter NFTABLES
includes in its tests the ability to determine if the source address of
a packet is routeable, and further classifies the result as LOCAL,
BROADCAST, UNICAST, BLACKHOLE, and PROHIBITED, among others, as well as
the interface that would be selected.

By using the routing table in this way, maintaining the configuration of
the firewall is simplified, particularly when interfaces are brought up
or taken down.  There is no coding change to the firewall.

The fact that I can't send packets upstream with bad destinations is not
the goal here.  The goal is to detect packets inbound with bad source
addresses that would affect my network, as well as ensuring that
outbound packets have good source addresses.

Herewith is the revised information for your constructive criticism:

> # Default IPv6 routing table (sorted by ipv6 address):
> # 
> # $ route -n6
> # Kernel IPv6 routing table
> # DestinationNext Hop Flag Met Ref Use If
> # --   --- --- --- ---
> # ::/0   ::   !n   -1  1 0 lo
> # ::/0   ::   !n   -1  1 0 lo
> # ::1/128::   U256 1 0 lo
> # ::1/128::   Un   0   4 0 lo
> # fe80::/64  ::   U256 1 0 enp37s0
> # fe80::7285:c2ff:fec0:bdff/128  ::   Un   0   2 0 enp37s0
> # ff00::/8   ::   U256 6 0 enp37s0
> 
> # [-] -- not included in blacklist, part of default routes
> # [#] -- not included in blacklist, policy
> 
> # investigate https://www.team-cymru.com/bogon-reference-http.html
> # need to better understand Terendo tunneling
> # rp_filter does the same at nftables source routing check
> 
> 
> nets=" 
> 0.0.0.0/8   SoftwareCurrent network (only valid as \
> source address).
> 10.0.0.0/8  Private network Used for local communications \
> within a private network.
> -100.64.0.0/10  Private network Shared address space for \
> communications between a \
> service provider and its \
> subscribers when using a \
> carrier-grade NAT.
> 127.0.0.0/8 HostUsed for loopback addresses \
> to the local host.
> 169.254.0.0/16  Subnet  Used for link-local addresses \
> between two hosts on a single \
> link when no IP address is \
> otherwise specified, such as \
> would have normally been \
> retrieved from a DHCP server.
> 172.16.0.0/12   Private network Used for local communications \
> within a private network.
> 192.0.0.0/24Private network IETF Protocol Assignments.
> 192.0.2.0/24Documentation   Assigned as TEST-NET-1, \
> documentation and examples.
> 192.88.99.0/24  InternetReserved. Formerly used for \
> IPv6 to IPv4 relay (2002::/16).
> 192.168.0.0/16  Private network Used for local communications \
> within a private network.
> 198.18.0.0/15   Private network Used for benchmark testing of \
> inter-network communications \
> between two separate subnets.
> 198.51.100.0/24 Documentation   Assigned as TEST-NET-2, \
> documentation and examples.
> 203.0.113.0/24  Documentation   Assigned as TEST-NET-3, \
> documentation and examples.
> 224.0.0.0/4 InternetIn use for IP multicast. \
> (Former Class D network).
> 240.0.0.0/4 InternetReserved for future use. \
> (Former Class E network).
> 255.255.255.255/32  Subnet  Reserved for the 'limited \
> broadcast' destination address.
> -::/0   Routing Default route.
> ::/128  SoftwareUnspecified address.  
> -::1/128HostLoopback 

Re: Request comment: list of IPs to block outbound

[Stephen Satchell]

On 10/13/19 9:08 AM, Florian Brandstetter wrote:
> Hi,
> 
> sorry - but why would you want to block Teredo?

I know nothing about Terendo tunneling.

> In computer networking, Teredo is a transition technology that gives
> full IPv6 connectivity for IPv6-capable hosts that are on the IPv4
> Internet but have no native connection to an IPv6 network. Unlike
> similar protocols such as 6to4, it can perform its function even from
> behind network address translation (NAT) devices such as home routers.
> 
> Teredo operates using a platform independent tunneling protocol that provides 
> IPv6 (Internet Protocol version 6) connectivity by encapsulating IPv6 
> datagram packets within IPv4 User Datagram Protocol (UDP) packets. Teredo 
> routes these datagrams on the IPv4 Internet and through NAT devices. Teredo 
> nodes elsewhere on the IPv6 network (called Teredo relays) receive the 
> packets, un-encapsulate them, and pass them on. 

Are you saying that Terendo should come off the list?  Is this useful
between an ISP and an edge firewall fronting an internal network?  Would
I see inbound packets with a source address in the 2001::/32 netblock?

> sorry - but why would you want to block 6to4?
In my research, this is marked as deprecated.  Would I see packets with
a source address in the 2002::/16 netblock?

Request comment: list of IPs to block outbound

[Stephen Satchell]

The following list is what I'm thinking of using for blocking traffic
between an edge router acting as a firewall and an ISP/upstream.  This
table is limited to address blocks only; TCP/UDP port filtering, and IP
protocol filtering, is a separate discussion.  This is for an
implementation of BCP-38 recommendations.

I'm trying to decide whether the firewall should just blackhole these
addresses in the routing table, or use rules in NFTABLES against source
and destination addresses, or some combination.  If NFTABLES, the best
place to put the blocks (inbound and outbound) would be in the FORWARD
chain, both inbound and outbound.  (N.B. for endpoint boxes, they go
into the OUTPUT chain.)

In trying to research what would constitute "best practice", the papers
I found were outdated, potentially incomplete (particularly with
reference to IPv6), or geared toward other applications.  This table
currently does not have exceptions -- some may need to be added as a
specific "allow" route or list.

The Linux rp_filter knob is effective for endpoint servers and
workstations, and I turn it on religiously (easy because it's the
default).  For a firewall router without blackhole routes, it's less
effective because, for incoming packets, a source address matching one
of your inside netblocks will pass.  A subset of the list would be
useful in endpoint boxes to relieve pressure on the upstream edge router
-- particularly if a ne'er-do-well successfully hijacks the endpoint box
to participate in a DDoS flood.

IPv4
Address block   Scope   Description
0.0.0.0/8   SoftwareCurrent network (only valid as
source address).
10.0.0.0/8  Private network Used for local communications
within a private network.
100.64.0.0/10   Private network Shared address space[3] for
communications between a service
provider and its subscribers
when using a carrier-grade NAT.
127.0.0.0/8 HostUsed for loopback addresses to
the local host.
169.254.0.0/16  Subnet  Used for link-local addresses
between two hosts on a single
link when no IP address is
otherwise specified, such as
would have normally been
retrieved from a DHCP server.
172.16.0.0/12   Private network Used for local communications
within a private network.
192.0.0.0/24Private network IETF Protocol Assignments.
192.0.2.0/24Documentation   Assigned as TEST-NET-1,
documentation and examples.
192.88.99.0/24  InternetReserved. Formerly used for
IPv6 to IPv4 relay
192.168.0.0/16  Private network Used for local communications
within a private network.
198.18.0.0/15   Private network Used for benchmark testing of
inter-network communications
between two separate subnets.
198.51.100.0/24 Documentation   Assigned as TEST-NET-2,
documentation and examples.
203.0.113.0/24  Documentation   Assigned as TEST-NET-3,
documentation and examples.
224.0.0.0/4 InternetIn use for IP multicast.
240.0.0.0/4 InternetReserved for future use.
255.255.255.255/32  Subnet  Reserved for the "limited
broadcast" destination address.

IPv6
Address block   Usage   Purpose
::/0Routing Default route.
::/128  SoftwareUnspecified address.
::1/128 HostLoopback address to local host.
:::0:0/96   SoftwareIPv4 mapped addresses.
:::0:0:0/96 SoftwareIPv4 translated addresses.
64:ff9b::/96Global Internet IPv4/IPv6 translation.
100::/64Routing Discard prefix.
2001::/32   Global Internet Teredo tunneling.
2001:20::/28SoftwareORCHIDv2.
2001:db8::/32   Documentation   Addresses used in documentation
and example source code.
2002::/16   Global Internet The 6to4 addressing scheme
fc00::/7Private network Unique local address.
fe80::/10   LinkLink-local address.
ff00::/8Global Internet Multicast address.

Re: California public safety power shutdowns

[Stephen Satchell]

On 10/11/19 8:01 AM, Ethan O'Toole wrote:
>> request went all the way to the Court.  The reason for access?  They ran
>> the electronics on bottled propane (NOT mains power AC) and they needed
>> to swap full tanks for the empties.  This was several months into my
>> stint on that site.
>> Not all generators run on diesel, I learned.
> 
> You can drive a gasoline generator with natural gas and propane, there
> is just less energy so it takes more of those fuels to get the same
> energy output.
> 
> There are also fuel cells that take LPG.
> 
> Was this a really tiny microcell? I wouldn't think they could run for
> months on bottled LPG if there is any kind of real load at the site.
> 
>     - Ethan

Not a tiny microcell.  The casino in question is located in Incline
Village, on the shore of Lake Tahoe and in prime ski country.  The
downside of ski country is that you have fairly frequent power outages
in winter due to weather, plus the long haul through mountains of fair
capacity transmission lines.

Now I could be mistaken, and the propane was for a stand-by generator.

Re: California public safety power shutdowns

[Stephen Satchell]

On 10/10/19 8:46 PM, Javier J wrote:
> I have an alternative view. the more generators are running, the more
> trucks semt to refuel the tanks, the more moving parts, the more likely an
> accident is prone to happen somewhere. It's thr same reason you turn your
> vehicles engine off when you fill up at the gas station.
> 
> Diesel doesn't combust easily without conpression, but I'm pretty sure you
> can find incidents where diesel engines catch fire. maybe the roof of a
> datacenter is not a risk factor, but in thinking remote antennas on the top
> of a mountain anything can happen.

When I was between jobs in IT, I worked as a security guard for a year.
 During that year, the company I worked for supplied on-prem security
for a bankrupt casino at Lake Tahoe.

When one of the cell phone companies requested access to their equipment
located on the roof of the parking garage (space they leased) the
request went all the way to the Court.  The reason for access?  They ran
the electronics on bottled propane (NOT mains power AC) and they needed
to swap full tanks for the empties.  This was several months into my
stint on that site.

Not all generators run on diesel, I learned.

Re: dns cache beyond ttl - viasat / exede

[Stephen Satchell]

On 10/7/19 9:08 AM, Mike wrote:
>    I am wondering if perhaps this is due to some kind of (known?)
> bug in the embedded dns cache/client in the client satellite modem, or
> if there is another plausible explanation I am not seeing. It compounds
> my problem slightly since I have to continue running the web sites at
> both the old and new addresses while these things time out I guess and
> it's just inconvenient.

Back when I was the mail/DNS/network admin at a hosting company, and we
would have to renumber, I saw the same thing.  This was back in the days
when the cable companies had small pipes to the Internet.  Their DNS
servers would impose a minimum 1 week TTL on all DNS requests, so that
the vast majority would be resolved "locally" without having to resort
to the root servers.

Other answers point to satellite companies doing something similar, to
combat the long RTD that DNS resolution would require without aggressive
caching.

Almost all of the Web servers I managed used Linux, so I was able to
play games in the firewall to let both numbers get to the Web servers
without having a convoluted configuration in Apache.  The three
Windows/ISS hosts were not that difficult to do, but was tiresome.

Those games stopped when the hosting company got its own /21 allocation.

Re: IPv6 Pain Experiment

[Stephen Satchell]

On 10/7/19 4:37 AM, Valdis Klētnieks wrote:
> On Mon, 07 Oct 2019 03:03:45 -0400, Rob McEwen said:
>> Likewise for spam filtering - spam filtering would be knocked back to
>> the stone ages if IPv4 disappeared overnight. IPv6 is a spam sender's
>> dream come true, since IPv6 DNSBLs are practically worthless.
> 
> Riddle me this:  Why then have spammers not abandoned IPv4 and moved to
> IPv6 where we're totally powerless to stop their floods of spam?
> 
> I'm tired of hearing the excuse "We can't move to IPv6 because then we 
> couldn't
> stop the spam" - if that were true, then every organization that *has* moved
> to IPv6 would be drowning in spam.

Spammers haven't abandoned IPv4 for IPv6 because some significant
percentage of mail servers are still IPv4 only.  I know mine is, and
will most likely stay that way for the reasons that Rob points out.  Any
link between an IPv6 spammer and an IPv4 mail server would have to
undergo NAT of some form, and the IPv4 address of NAT would then be
subject to blocking action.

The BOFH model from the old NFS days won't work anymore.

Re: Update to BCP-38?

[Stephen Satchell]

On 10/3/19 10:13 PM, Fred Baker wrote:
> There is one thing in 1122/1123 and 1812 that is not in those kinds
> of documents that I miss; that is essentially "why". Going through
> 1122/1123 and 1812, you'll ind several sections that say "we require
> X", and follow that with a "discussion" section that says "we thought
> about X, Y, and Z, there were proponents of each, the arguments were
> X', Y', and Z', and we chose X for this reason". I would presume that
> what you're really looking for in a 1812-for-IPv6 is not "we require
> X" as much as "for this reason". Correct me if I'm wrong.

Ah.  What I'm looking for is a list of check-boxes to include in an
implementation specification for an edge router.  It can be references
to a whole bunch of RFCs and "packaged" as a BCP.  The discussions you
describe are better in the individual papers.

Side note: I'm used to rationales being included in Standards, and
welcome them, as long as they are normative and clearly marked so.

> I can kick the idea around in the IETF if its important to you. I'll
> be looking for a LOT of operational input.

It could well me that the data is there, we just need a document to
index it all.  That's what I thought a BPC was supposed to be.  It would
be like an article in ACM Computing Surveys, which references the
existing literature, as opposed to being created from whole cloth.

I think I steered everyone wrong when I was talking about some of the
exposition in the text, specifically the examples.  That kind of
material really belongs in an RFC.  My apologies.

Re: Update to BCP-38?

[Stephen Satchell]

On 10/3/19 2:07 PM, Mark Andrews wrote:
> Now IPv6 examples are nice but getting several 1000’s people to read draft 
> that
> just add addresses in the range 2001:DB8::/32 instead of 11.0.0.0/8, 
> 12.0.0.0/8
> and 204.69.207.0/24, then to get the RFC editor to publish it is quite frankly
> is a waste of time.

Long ago, I was working on Network Graphics Protocol and the draft RFC
for it.  My boss said that I needed to write a couple of paragraphs
about fixed-point binary fractions, which the protocol used, "because
that's not a common thing in the world".  How bad was this?  The person
who was writing the generator of the NGP stream used *floating point*
because that person didn't understand that 7FFF was interpreted as a
bitty-point less than 1.0, and 8001 was a bitty-point less that -1.0
-- and that trying to shoehorn this into IEEE floating-point format
almost worked.  What my poor boss didn't realize is that exactly 1.0 and
-1.0 were not defined.  The specification didn't make that clear.

When I'm doing technical writing, I find all sorts of corner cases that
were missed by the designers and the QA people.  It makes me very
unpopular.  But it also makes for a better product, in the end.

So making everything crystal clear and obvious is definitely not "a
waste of time."  You have no idea what undiscovered bugs may become
obvious when you go through the exercise and show all your work.

You still need a IPv6 version of RFC 1812.  Make it as clean as
possible.  Use an ax instead of a XACTO knife on the current draft.
What is the minimum necessary things that a generic IPv6 router MUST do?

Re: Update to BCP-38?

[Stephen Satchell]

On 10/3/19 8:22 AM, Fred Baker wrote:
> Speaking as v6ops chair and the editor of record for 1812.
> draft-ietf-v6ops-ipv6rtr-reqs kind of fell apart; it was intended to be
> an 1812-like document and adopted as such, but many of the
> "requirements" that came out of it were specific to the author's
> operation and not common to other operators. So it ultimately didn't happen.

Sympathies.  I've been involved in Standards-setting committees, so I
know how that works.  Is there any further work being done?  And just
how much of the current draft can be generalized?

> There is no demand until further IPv4 deployment no longer suffices.
> I would say that there was no real market demand until after January
> 2011, and probably 2012 or 2013.>
> At this point, there is fairly wide deployment among the ISP and CDN
> operators, and vendor implementations are fairly complete. Google,
> APNIC, and Akamai report on traffic levels; Google says that they see
> at least 5% of the requests they receive from 61 countries use IPv6,
> and from one country a tad more that half of the requests they
> receive use IPv6.>
> So we see IPv6 in broadband, in ISPs, and in telephone networks. To
> give you an anecdote, my kids have teased me about IPv6 for years,
> and each now have IPv6 service from their various ISPs and telephone
> networks despite themselves - and use it for the majority of their
> accesses.>
> What is visibly lacking is enterprise deployment.

Interesting you should mention that.  One reason I wanted to do an
IPv6-aware BGP-38 module for firewalld was to help break that logjam.
Enterprises are risk-adverse, which is why adoption meets such strong
resistance.

> And on lists like this, I am told that there is no deployment - that
> nobody wants it, and anyone that disagrees with that assessment has
> lost his or her mind. That all leaves me wondering which of us
> doesn't quite have their eye on the ball.
For the reasons you provided in your original message, the learning
curve for IPv6 -- EVERYTHING about IPv6, not "just enough to get by" --
is steep and uncertain.

And I think you may be misunderstanding the problem.  It's not that
people don't want it.  They lack the zen of it, they don't see the four
corners of the thing, something that people took years to learn in IPv4.
 (I had a leg up, being involved in the original ARPAnet, so I got to
watch it grow.  Still have the 1984 DDN handbooks, too.)

Re: Update to BCP-38?

[Stephen Satchell]

On 10/3/19 8:42 AM, Fred Baker wrote:
> 
> 
>> On Oct 3, 2019, at 9:51 AM, Stephen Satchell  wrote:
>>
>> Someone else mentioned that "IPv6 has been around for 25 years, and why
>> is it taking so long for everyone to adopt it?"  I present as evidence
>> the lack of a formally-released requirements RFC for IPv6.  It suggests
>> that the "science" of IPv6 is not "settled" yet.  That puts the
>> deployment of IPv6 in the category of "experiment" and not "production".
> 
> And, of course, we now have companies like T-Mobile and others
> turning IPv4 off. If that's an experiment, wow.
The cellular data industry appears to have embraced IPv6 in one form or
another.  I would expect that the network engineers have done some work
to keep IPv4 off their *internal* networks, but provide IPv4 access at
the edge.  (Isn't a netblock within IPv6 intended to enable bridging to
IPv4?)  The applications on the phon could be configured to search DNS
for  addresses first.

My AT cell phone has both IPv4 and IPv6 addresses.  The IPv4 address
is from my access point; the IPv6 address appears to be a public address.

I would like to move to IPv6.  I just don't want to shoot myself in the
foot, or cause trouble for other people, by being sure my edge router
"follows all the rules."

Re: Update to BCP-38?

[Stephen Satchell]

On 10/2/19 9:51 PM, Mark Andrews wrote:
> What part of BCP-38 do you think needs to be updated to support IPv6?
> 
> Changing the examples to use IPv6 documentation prefixes instead of IPv4
> documentation prefixes?

For a start, *add* IPv6 examples in parallel with the IPv4 examples.  As
RFCs are peer reviewed, if the examples expose a hole or problem then
the hole can be filled or the problem resolved.

BCP-38 should be reviewed in whole for "IPv6" completeness, and the
preamble of BCP-38 add that the Best Practices include complete
recommendations for both IPv4 and IPv6.

One specific example:

BCP-38 currently references RFC1812, _Requirements for IP Version 4
Routers_.

It appears that the only parallel paper for IPv6 is
draft-ietf-v6ops-ipv6rtr-reqs-04, _Requirements for IPv6 Routers_, which
currently carries a copyright of 2018.  It's a shame that this document
is still in limbo; witness this quote:  "It is inappropriate to use
Internet-Drafts as reference material or to cite them other than as
'work in progress.'

Someone else mentioned that "IPv6 has been around for 25 years, and why
is it taking so long for everyone to adopt it?"  I present as evidence
the lack of a formally-released requirements RFC for IPv6.  It suggests
that the "science" of IPv6 is not "settled" yet.  That puts the
deployment of IPv6 in the category of "experiment" and not "production".

Is that really true?

There may be more issues.  And, yes, I understand that a new BCP paper
may result -- I don't care how it's labeled, as long as it's done.  Or
has such a BCP document already been released?  If so, why do I not see
any references to it here on NANOG, or anywhere else?

Why do I care, you ask.  I'm working on a BCP-38 module proposal for
firewalld, one that can be peer-reviewed for accuracy and completeness.
Servers running that firewall package can then be easily configured to
conform to the requirements of BCP-38 and can easily become good
net-citizens in their own right.

So I call for draft-ietf-v6ops-ipv6rtr-reqs-04 to be finished and
released as a formal RFC, and that BCP-38 be updated to refer to that
finished RFC.

Until that is done, my BCP-38 module will have to carry a "work in
progress" disclaimer.

Update to BCP-38?

[Stephen Satchell]

Is anyone working on an update to include IPv6?

Re: IPv6 Thought Experiment

[Stephen Satchell]

On 10/2/19 9:33 AM, Antonios Chariton wrote:
> Dear list,
> First of all, let me apologize if this post is not allowed by the
> list. To my best interpretation of the guidelines [1] it is allowed, but
> may be in a gray area due to rule #7.
> 
> I would like to propose the following thought experiment about IPv6,
> and I would like your opinion on what you believe would happen in such a
> case. Feel free to reply on or off list.
> 
> What if, globally, and starting at January 1st, 2020, someone 
> (imagine a government or similar, but with global reach) imposed an 
> IPv4 tax. For every IPv4 address on the Global Internet Routing
> Table, you had to pay a tax. Let’s assume that this can be imposed,
> must be paid, and cannot be avoided using some loophole. Let’s say
> that this tax would be $2, and it would double, every 3 or 6 months.

Who exactly would be paying this tax?  The IPv4 address "owner"?  The
SWIP?  The end user who gets IPv4 via DHCP from his provider?

Tax paid to whom?

> What do you think would happen? Would it be the only way to reach 
> 100% IPv6 deployment, or even that wouldn’t be sufficient?
Well, a lot of money would change hands.  Somebody would be enriched by
the tax revenues.

> And for bonus points, consider the following: what if all
> certification bodies of equipment, for certifications like FCC’s or CE
> in Europe, for applications after Jan 1st 2023 would include a “MUST
> NOT support IPv4”..

So how would that affect users trying to access IPv4 resources?

> What I am trying to understand is whether deploying IPv6 is a pure
> financial problem. If it is, in this scenario, it would very very soon
> become much more pricey to not deploy it.

First, there are equipment issues -- not all gear "plays nice" with
IPv6, especially older gear still in use.  There is a capital cost
associated with upgrading gear, and not all organizations and people can
afford the hit.

There are policies in place, beyond the RFCs, by companies and
governments that would need to be updated, and the tax you suggest
doesn't even begin to attack the problems.

> I know there are a lot of gaps in this, for example who imposes this,
> what is the "Global Internet Routing Table", etc. but let’s try to
> see around them, to the core idea behind them.
Has BCP-38 been updated to include IPv6?
https://tools.ietf.org/html/bcp38

All the examples are IPv4.  Additionally, one of the reference is this:
Baker, F., "Requirements for IP Version 4 Routers", RFC 1812, June 1995.

If people are serious about IPv6, isn't it time to update the Best
Practices documents, particularly BCP-38 et al, to address IPv6 as well
as IPv4?

Elad Cohen, show us!

[Stephen Satchell]

On 9/19/19 2:47 AM, Elad Cohen wrote:
> It is not related to nefarious activity as you wrote, FDCServers
> policy is to stop routing any ranges which is in Spamhaus SBL (no
> matter what), due to the phear from Spamhaus to list all of
> FDCServers ranges in SBL, which was told to us in a documented phone
> call, listing all of the ranges by Spamhaus is a known agrressive and
> bullying tactic by Spamhaus as you can find in many webpages online.
Do you think Spamhaus uses "aggressive and bullying tactic[s]"?  You
never lived under the sword of SPEWS.

> https://en.wikipedia.org/wiki/Spam_Prevention_Early_Warning_System

As a former abuse and mail admin in a web hosting company with (at the
time) 3000 domains serviced, I found Spamhaus to be a firm but fair
organization.  Respond quickly and effectively to abuse complaints,
stoping the spam flow, Spamhaus delisted -- sometimes without being
asked to.

Note: this list is of and for network operators.  Spamhaus and other
DNSBLs are the subject for a mailing list of and for mail admins.

Re: Elad Cohen, show us!

[Stephen Satchell]

On 9/19/19 2:47 AM, Elad Cohen wrote:
> It is not related to nefarious activity as you wrote, FDCServers
> policy is to stop routing any ranges which is in Spamhaus SBL (no
> matter what), due to the phear from Spamhaus to list all of
> FDCServers ranges in SBL, which was told to us in a documented phone
> call, listing all of the ranges by Spamhaus is a known agrressive and
> bullying tactic by Spamhaus as you can find in many webpages online.
Do you think Spamhaus uses "aggressive and bullying tactic[s]"?  You
never lived under the sword of SPEWS.

> https://en.wikipedia.org/wiki/Spam_Prevention_Early_Warning_System

As a former abuse and mail admin in a web hosting company with (at the
time) 3000 domains serviced, I found Spamhaus to be a firm but fair
organization.  Respond quickly and effectively to abuse complaints,
stoping the spam flow, Spamhaus delisted -- sometimes without being
asked to.

Note: this list is of and for network operators.  Spamhaus and other
DNSBLs are the subject for a mailing list of and for mail admins.

Re: Weekly Routing Table Report

[Stephen Satchell]

On 9/2/19 4:40 PM, Seth Mattinen wrote:
> May the world come to an end if someone dares to have an independent 
> thought or shares original information that can't be backed up by at 
> least 50 crosschecked references.
Actually, independent thought or original information is welcome to
anyone with a brain, as long as the author shows his/her work such that
it can be verified and reproduced by others.  You don't need a ton of
references to the work (and conclusions) of others if you do a complete
job yourself.

There is a reason for the joke publication _The Journal of
Irreproducible Results_, started in 1955.  So many "scientific"
publications have this little tiny problem: no one can duplicate the work.

DNS cache hold of SERVFAIL responses

[Stephen Satchell]

This is for any Google admin on this list:

When you receive a SERVFAIL from a name server listed as authoritative
for a given domain, how long is that negative look-up cached?

When you receive a SERVFAIL from the root servers, how long is that
negative lookup cached?

Does Google follow RFC 2308?

Is there a common cache for all resolvers, or do each resolver in your
DNS server corps maintain a local cache?

When a SOA (when you see one) says 14400 for the minimum TTL (or
negative cache TTL) does Google honor that hold time?

Re: User Unknown (WAS: really amazon?)

[Stephen Satchell]

On 8/13/19 3:10 PM, Matthew Petach wrote:
> With a global company, there's no such thing
> as a local natural monopoly in play; how would
> you assign oversight to a global entity?  Which
> "public" would be the ones being protected?
> The city of Seattle, WA, where Amazon is
> headquartered?  The State of Washington?
> The United States, at a federal level?   What
> about the "public" that uses Amazon in all
> the other countries of the world?

Consider how radio, television, and telephony grew and became regulated.
 (For a moment there, it felt like a discussion that I would have on the
CyberTelecomm mailing list.)  Each country would regulate the monopoly
in the manner best suited for that country.  Amazon would need to set up
divisions in each country, or union of countries such as the EU.

> There's no way to make a global entity a
> regulated public utility; we don't have an
> organization that has that level of oversight
> across country boundaries, unless you start
> thinking about entities that can enforce *treaties*
> between countries.

Actually, you'd be surprised to learn we already have infrastructure in
place to do exactly that.  The International Telecommunication Union is
a fine example of how this could be done.  Study up on it.  From my
experience in the telco and modem world, the individual countries have
working parties for each element.  The working parties develop Standards
(the initial cap is intentional) within each country.  The output from
the working parties in each country send their recommendations to a
government bureau -- in the United States, that would be a working party
associated with the State Department.  (For example, my work on in-band
modem control went through TIA/EIA TR-29, which then was passed on to
Study Group D, which went to the ITU.)

> And I'm not sure I'd want our Ambassadors
> being the ones at the table deciding how best
> to regulate Amazon.   :/

That's just the point.  The regulation would *not* be done by
ambassadors.  The treaties, rule, regulations, and procedures are
*already* in place to smooth the process through people that are not
political appointees.

Regulation of Amazon would probably be broken into parts: technical,
policy, managment, auditing, perhaps more.  Policy would originate in
the USA with Congress, with help from the industry.  Other parts would
be parceled out to the people better (not necessarily the best) equipped
to do the job.

And that's my pair-o-pennies on the subject.  Other people may have
differing opinions.

Re: User Unknown (WAS: really amazon?)

[Stephen Satchell]

On 8/9/19 4:03 PM, Matthew Petach wrote:
> ...apparently Amazon has become a public utility
> now?
> 
> I look forward with bemusement to the PUC
> tariff filings for AWS pricing.  ^_^;;

Don't scoff too hard.  How do you think that telephone service became a
utility?  Utilities didn't grow on trees, they became utilities when
some bureaucrats convinced legislators to "promote" successful service
providers to utility status.  Especially when such providers are
providing a service as a monopoly.  Particularly a "natural" monopoly.
AWS has competitors,  but if the number of providers remains small (like
fingers of one hand) the politicians wil step in.

And it wouldn't be the PUC, as Amazon is a company national in scope.
It would be something like the FCC.  Public Utility Commissions are at
the local (usually county) or state level.

Re: User Unknown (WAS: really amazon?)

[Stephen Satchell]

On 8/3/19 9:15 PM, John Curran wrote:
> As I have noted previously, I have zero doubt in the enforceability
> of the ARIN registration services agreements in this regard – so
> please carefully consider proposed policy both from the overall
> community benefit being sought, and from the implications faced as a
> number resource holder having to comply oneself with the new
> obligations.

Actually, I would re-write the last part of the last sentence as "...and
from the implications faced as a number resource holder having to comply
oneself with the long-standing and well-known obligations of all network
operators."

I'm a small network operator that has been around and following "the
rules" for many years.  I do understand why you are constrained by the
legal authority you have.  In some respects, I (and others) pine for the
old NSFNET days, when negligence -- particularly willful negligence --
was rewarded with disconnection.

"The rules" have been around for years, and are codified in the RFCs
that are widely published and available to all at zero cost.  (That
wasn't always true, as it wasn't until the DDN Protocol Handbook volumes
were published in 1985 that the RFCs were available to everyone.  I seem
to recall there was an FTP site that provided the RFC documents before
that, but my memory is hazy on that.)  I had access to all the RFCs at
the University of Illinois Center for Advanced Computation, as I was
working at the place as a worker on ARPAnet.

During my career as a web server admin, mail admin, and network admin, I
followed "the rules" strictly.   As the main abuse contact during my
time at a web hosting company, my postmaster@ and abuse@ contact
addresses were according to Hoyle, and published with the company ASN,
netblock, and domain registration records.  It took a little convincing
for the owner of the shop to buy in, and to back up my responses to
abuse reports.

I would have expected any ARIN contracts to include by reference the
RFCs that constituted "the rules".  I have never seen the contracts, so
I don't know how they are formulated.  That said, I would have expected
legacy space to fall under "the rules", particularly with respect to
role electronic mail addresses.

I don't have a dog in this fight.  Currently, I don't "own" any IPv4
address space, nor am I running BGP.

Re: really amazon?

[Stephen Satchell]

On 7/31/19 1:28 PM, Brian J. Murrell wrote:
> On Wed, 2019-07-31 at 23:13 +0300, Scott Christopher wrote:
>>
>> Because it will get spammed if publicly listed in WHOIS.
> 
> I will take that at *least* as ironic as you meant it.

I don't know about your network, but I have five role mail accounts, and
all five get spam, even with spam filters enabled.  Oh, forgot about
abuse@, which has no filter but LOTS of spam.  What's fun is to let it
sit a couple of days, then sort by subject.  Delete the "conversations".

That gets down to the zero or one piece of ham.  But then again, I've
locked down my network so abuse doesn't get out, even when someone
manages to get by the MAC filters on the wireless router.

Re: really amazon?

[Stephen Satchell]

On 7/31/19 12:04 PM, Valdis Klētnieks wrote:
> On Wed, 31 Jul 2019 16:36:08 -, Richard Williams via NANOG said:
> 
>>  To contact AWS SES about spam or abuse the correct email address is 
>> ab...@amazonaws.com
> 
> You know that, and I know that, but why doesn't the person at AWS whose job it
> is to keep the ARIN info correct and up to date know that?

C'mon, you already know the answer to this:  there is no such person.
Someone gets a mail once a year and MIGHT, JUST MIGHT, pass it on to
someone who knows what to do.

Re: Feasibility of using Class E space for public unicast (was re: 44/8)

[Stephen Satchell]

On 7/27/19 2:18 PM, Randy Bush wrote:
> something is broken on the nanog list.  usually we have this discussion
> twice a year.  this time it may have been a couple of years gap.  what
> broke?


44/8.  Sucked up all the oxygen.

Re: 44/8

[Stephen Satchell]

On 7/22/19 12:15 PM, Naslund, Steve wrote:
> 1.  A lot of existing code base does not know how to handle those
> addresses and may refuse to route them or will otherwise mishandle
> them.

Not to mention all the legacy devices that barely do IPv4 at all, and
know nothing about IPv6.  Legacy devices that are orphaned by their
developing companies going out of busiess or dropping all support for
the products.

I'm looking at YOU, MasterSwitch.

Intermittent "bad gateway"

[Stephen Satchell]

Are we having another BGP problem this morning?

Re: FCC workshop: Security vulnerabilities within our communications networks

[Stephen Satchell]

On 6/26/19 2:17 PM, Scott Weeks wrote:
> 
> --- s...@donelan.com wrote:
> From: Sean Donelan 
> 
> If they come up with a better idea, that's great.  I'll 
> take good ideas  from anywere.

In my experience, "design by committee" is most successful when one or
two people take the bull by the horns and flesh out a solution, and the
rest of the bunch nod their heads and say, "Huzza".

Re: CloudFlare issues?

[Stephen Satchell]

On 6/25/19 2:25 AM, Katie Holly wrote:
> Disclaimer: As much as I dislike Cloudflare (I used to complain about
> them a lot on Twitter), this is something I am absolutely agreeing with
> them. Verizon failed to do the most basic of network security, and it
> will happen again, and again, and again...

I used to be a quality control engineer in my career, so I have a
question to ask from the perspective of a QC guy:  what is the Best
Practice for minimizing, if not totally preventing, this sort of
problem?  Is there a "cookbook" answer to this?

(I only run edge networks now, and don't have BGP to worry about.  If my
current $dayjob goes away -- they all do -- I might have to get back
into the BGP game, so this is not an idle query.)

Somehow "just be careful and clueful" isn't the right answer.

Re: Charter and Cox contacts

[Stephen Satchell]

On 5/13/19 12:11 PM, dan...@pyranah.com wrote:
> Does anyone have contacts at Charter (Spectrum) and Cox? For some reason,
> our IP has been blocked by them and our customers are unable to send email
> via their charter/cox accounts. Thanks


Would you be talking about port 25/tcp outbound?  Lots of ISPs will
block port 25 as a rule; you have to call to have it opened.  At the
same time, you can request the PTR record you need to keep big mail
receivers happy.

Re: NTP question

[Stephen Satchell]

One word of caution when using a low-priced NTP appliance: your network
activity could overwhelm the TCP/IP stack of the poor thing, especially
if you want to sync your entire shop to it.  In the case of the networks
I set up, I set up a VLAN specific to the NTP appliance and to the two
servers that sync up with it.  Everything else in the network is
configured to talk to the two servers, but NOT on the three-device "NTP
Appliance VLAN".

NOTE: Don't depend on the appliance to provide VLAN capability; use a
configuration in a connected switch.  How you wire from the appliance to
a port on your network leaves you with a lot of options to reach a
window with good satellite visibility, as CAT 5 at 10 megabits/s can
extend a long way successfully.  Watch your cable dress, particularly
splices and runs against metal. (Or through rooms with MRI machines --
I'm not joking.)

The two servers in question also sync up with NTP servers in the cloud
using whatever baseband or VLANs (other than the "NTP VLAN") you
configure.  Ditto clients using the two servers as time sources.

The goal here is to minimize the amount of traffic in the "NTP Appliance
VLAN".  What killed one installation I did was the huge amount of ARP
traffic that the appliance had to discard; it wasn't up to the deluge.

Learn from my mistakes.

Re: Comcast storing WiFi passwords in cleartext?

[Stephen Satchell]

On 4/24/19 9:32 PM, Mike Bolitho wrote:
>>
>> "than the relatively low risk of a database compromise leading to a
>> miscreant getting ahold of their wireless password and using their access
>> point as free wifi."
>>
> 
> And this is the thing, not only does someone have to 'hack' the database,
> they also need to drive up to your house and sit in your driveway to get
> free Internet. Of all the things to worry about, this is way down on my
> list.

Sounds like you live in a single-family home, in a low-density
neighborhood.  I live in an apartment complex, and WiFi Analyzer shows
more than 20 beacons visible from my kitchen.  Before I implemented MAC
filtering, my DHCP server logs showed connections by rogue equipment.
(And before you ask, my password is not a simple one.)

When I leave town for an extended period, I pull the plug on the wireless.

In an industrial park, I see the same dense tangle of beacons.  Let
alone those wireless systems that have disabled beacons.

If that database of wireless passwords also has the physical location of
the device, then the risks go up.

So the simple solution is to use your own AP, block SNMP access from the
worlds at your edge router, and the risk falls to zero.

Re: Comcast storing WiFi passwords in cleartext?

[Stephen Satchell]

On 4/24/19 7:24 AM, Tom Beecher wrote:
> This is why, in my opinion, people should avoid modem/router combo units
> whenever possible. Any information/configuration entered into such a device
> could be accessible to the MSO (intentionally or otherwise) , as is
> happening here. I'm sure they would come back and say this is necessary to
> provide support for customers who pay them for WiFi service, but it clearly
> shows they don't turn off that functionality for customers who don't.
> 
> Treat you cable modems as foreign network elements. Cause that's what they
> are.

+1.  Encountered this with an AT install.  AT provided router/wifi
combo.  After the installer was done, first thing I did was to turn the
combo's wifi off, and hook up the access point the customer has been
using for years.  Verified that the MAC filtering was still correct
during the post-install.  Customer is happy.

The next step is to build a Protectli firewall to go between the AT
modem and the access point.  Block any chance of AT using SNMP to
sniff the access point.  (Moved the Access Point's IP address for
management and gateway, too.)

Re: GPS WNRO April 6th at GPS Midnight

[Stephen Satchell]

On 4/3/19 3:32 PM, brutal8z via NANOG wrote:
> I've not seen any mention of this here, so it might be off-topic, if so,
> sorry in advance. If you use GPS for time synchronization, this might be
> important.The Juniper ACX500 series and the Cisco 819 both have an
> embedded GPS receivers, for example.
> 
> At 23:59:42 UTC on 4/6/2019 (Midnight GPS time, which differs by 18
> leap-seconds  from UTC) , the
> 10-bit GPS Week Number broadcast by the constellation will reset to zero
> for the second time since the beginning of GPS on 1/6/1980.

There was a mention of this a couple of weeks ago, as a "heads-up".

When it first appeared up here, I checked with the vendor of my GPS time
appliances.  The vendor told me that the testing they did of their
product showed no issues with the roll-over for specific ranges of
serial numbers.  Mine is within one of the ranges.

The proof of the pudding will be what happens at 5pm PDT on my birthday,
April 6, and what ntpq reports on my edge server.

Re: Contacts wanted: OVH, DigitalOcean, and Microsoft (Deutschland)

[Stephen Satchell]

On 3/18/19 11:17 PM, Ronald F. Guilmette wrote:
> I am not sure that there is any other way that a lone outsider can or
> could engage either OVH or DigitalOcean in a way that would actually
> cause either company to take action on the issues I've reported on.
> Complaints from ordinary Internet end-lusers about this, which both
> companies must surely be drowing in by now, don't seem to be doing the
> job.
> 

I have sent reports to DigitalOcean and Microsoft about the abuse
reported to me that appears on my network.  The only response I'm
interested in is the end of said abuse.

When abuse continues to be seen on a DigitalOcean allocation, that
allocation goes into a file-and-forget ACL.

I investigate further only when a customer reports a problem connecting
with a DigitalOcean netblock.  No such reports yet, by the way.

I respond to abuse reports promptly and completely.  I expect others to
do so, just as promptly and completely.  "Pretty please" is not in my
netadmin vocabulary.

GPS rollover

[Stephen Satchell]

So far as I can tell with NTP, there was no issue with time sources
becoming false-tickers, including my local GPS appliance.  FWIW.

Re: WIndows Updates Fail Via IPv6 - Update!

[Stephen Satchell]

On 3/7/19 8:10 AM, Saku Ytti wrote:
> So why not disable ICMP Echo and UDP traceroute, those kids using
> network diagnostics don't need them.
> 
> For clue constrained audience fear will always be the most compelling 
> argument.

OK, OK, so I will continue to rate-limit both, to reasonably high limits
on the order of 250/second.  Absent a DoS, it allows network operators
to use these tools as they should.

My logs show no harm except to attack traffic.

Everything in moderation.

Re: ICMPv6 "too-big" packets ignored (filtered ?) by Cloudflare farms

[Stephen Satchell]

On 3/5/19 2:54 AM, Thomas Bellman wrote:
> Out of curiosity, which operating systems put anything useful (for use
> in ECMP) into the flow label of IPv6 packets?  At the moment, I only
> have access to CentOS 6 and CentOS 7 machines, and both of them set the
> flow label to zero for all traffic.

Did you submit a bug report?

Re: WIndows Updates Fail Via IPv6 - Update!

[Stephen Satchell]

On 3/3/19 1:04 PM, Mark Andrews wrote:
> There are lots of IDIOTS out there that BLOCK ALL ICMP.  That blocks PTB 
> getting
> back to the TCP servers.

For those of us who are in the dark, "PTB" appears to refer to "Packet
Too Big" responses in ICMPv6.

Yes, some admins don't have fine-enough grain tools to block or throttle
specific types of ICMP, but that's the fault of the vendors, not the admins.

Re: sendmail.cf

[Stephen Satchell]

On 2/22/19 11:27 AM, b...@theworld.com wrote:
> I don't know the high-water mark for the number of IMPs or more
> specifically how many existed on the NCP->TCP flag day but I'm pretty
> sure the theoretical maximum was 256 tho no doubt someone had a way to
> extend that. But, w/o extensive changes, 256, probably 254, not sure 0
> or 255 could be an IMP number, whatever!

There was no node 0 or 255.  So the number of nodes was capped at 254.
For each node, there were subnodes so that multiple computers at each
location could be addressed.  It wasn't a full 8-bit field.

Re: A Zero Spam Mail System [Feedback Request]

[Stephen Satchell]

On 2/18/19 9:37 PM, Scott Weeks wrote:
> Not me.  No way.  Never.  ;)

Then why is Mr. Murphy tapping you on the shoulder?  Didn't your Mom and
Dad ever tell you to never say "never"?

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Stephen Satchell]

On 2/1/19 1:23 PM, Mark Andrews wrote:
> Google has started their rollout.

So has Red Hat (RHEL and Centos).  I woke up to a rather large update
this morning.

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Stephen Satchell]

After reading through the thread, this reminds me of the Y2K flap, that
turned into a non-event.  My checks of authoritative DNS servers for my
domains show no issues now.

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Stephen Satchell]

On 1/24/19 11:46 AM, Mark Andrews wrote:
>On 25 Jan 2019, at 2:14 am, Stephen Satchell  wrote:
>> My edge routers block *all* inbound DNS requests -- I was being hit by a
>> ton of them at one point.  Cavaet: I don't run a DNS server that is a
>> domain zone master -- I use a DNS service for that.  I do have a DNS
>> server inside, but only to handle recursive requests from inside my network.
>>
>> Outbound DNS requests?  Lets them through, and responses too.
>
> Well does your DNS service properly manage the firewall in front of their
> servers?  Does the anti DoS scrubbing service they are using also pass the
> well formed packets to the DNS server they are advertising?

I have domains on several DNS services.  Most of the services work
properly according to the ISC tests.  Two of the services show failures.
 So I called support on the pair.  One service says they are deploying
updates before the 1 Feb 19 deadline to all their DNS servers.  The
other one (starts with an "A") doesn't know when they will be fully
compliant "but your customers should have no difficulty with getting DNS
answers on your domains."

I had downloaded the tool, so I tested my inside DNS servers just for
grins.  Passed with flying colors -- I had used Centos 7 in those
servers, updated on a regularly scheduled basis, so of course it flew
with flying colors.  (Or do you prefer colours?)

Re: DNS Flag Day, Friday, Feb 1st, 2019

[Stephen Satchell]

On 1/23/19 8:44 PM, Mark Andrews wrote:
> and they your firewalls don’t block well formed DNS queries (lots of
> them do by default).

My edge routers block *all* inbound DNS requests -- I was being hit by a
ton of them at one point.  Cavaet: I don't run a DNS server that is a
domain zone master -- I use a DNS service for that.  I do have a DNS
server inside, but only to handle recursive requests from inside my network.

Outbound DNS requests?  Lets them through, and responses too.

Re: Top Posting Was: Re: plaintext email?

[Stephen Satchell]

On 1/15/19 8:03 AM, Tom Beecher wrote:
> No disrespect intended to anyone at all, but the pissing and moaning about
> it is a massive waste of time and energy.

But, but, but...most water-cooler conversation is about sports, the
opposite sex, and pissing and moaning about what you don't like.  Sure,
it's a massive waste of time and energy -- but that's what "being
social" is all about.

(Not that I claim to be THAT house-broken.)

Re: the e-mail of the future is the e-mail oft the past, was Enough port 26 talk...

[Stephen Satchell]

On 1/15/19 12:19 AM, Bjørn Mork wrote:
> And everyone has a gmail account anyway, so why bother with outside
> email?

Two words:  "search warrants."  I'm a US citizen, and I do NOT like the
idea of power-hungry people being able to paw through my mail.  Having
my own mail server, residing in my home, with medium security in place,
gives me peace of mind.

Even innocent people have things they want to hide from casual spying.
THOSE people don't have a need to know.  Period.

Not to mention that I can obey the rule common in Business 101 regarding
mail.  It goes like this:

QUESTION:  You are a medium-sized company.  How do you set up your mail
room to be efficient, and needing only a small staff?

ANSWER: You take out a number of post office boxes, and have each
department use its post office box to receive mail for that department.
 You task one person to stop at the post office to pick up the contents
of the post office box, properly banded.

In short, you let the postal system sort your mail for you.  They are
very good at it, and can even bring automation (that you can't afford)
to speed up the process.

For me, I have a mail server with several dozen inboxes
(Postfix/Dovecot).  Only a couple of those e-mail addresses have been
exposed to the world via mailing lists and USENET.  Thunderbird does a
nice job of presenting this gaggle of inboxes.  And I keep adding
mailboxes as the need arises.  I can see which inboxes has incoming
mail, and selectively look at each one as time permits.  Yes, I see all
the incoming spam that makes it through the DNSBLs, but I can ignore the
spam-catchers when I have better things to do.

Re: plaintext email?

[Stephen Satchell]

On 1/14/19 9:40 PM, valdis.kletni...@vt.edu wrote:
> I'm not away of any languages or writing systems that work from
> bottom to top, so that's pretty much everybody.

Typography for at least one pictograph-based language allows for, um,
interesting stunts one can pull to spice up gray matter.  Starting in
the middle of the paragraph and spiraling around, for example.  Nothing
which applies to e-mail, but now you know.

Top-quoting Was: (Netflix/GlobalConnect a/s) Scheduled Open Connect Appliance upgrade is starting

[Stephen Satchell]

On 1/14/19 7:14 PM, Keith Medcalf wrote:
> Please experience the wonders of the top-quote.  See your local psychedelic 
> distributor if you are somehow not "experiencing" anything ...

I experience a savings in time with non-edited top quoting.  If I don't
see meaningful new content within the first 20 lines, I ignore it as
worthless...unless it's a topic I'm following closely.

And, yes, I use a text-only mail reader.  I don't like HTML mail,
because it's an attack vector for ne'er-do-wells.  As long as the mail
reader allows "self-clicking" URLs, just NO.

Re: (Netflix/GlobalConnect a/s) Scheduled Open Connect Appliance upgrade is starting

[Stephen Satchell]

On 1/13/19 8:01 PM, Brian Kantor wrote:
> Clearly, editing inclusions is a lost art.

No, it isn't a lost art.  As you can see, there are some of us who know
perfectly well how to edit, and have e-mail tools that make this easy.
(Using Thunderbird here.)  Smartphone mail programs make excerpting a
hard task, by the nature of the human interface.

Making matters worse, Joe SixPack and Suzie Latchhook are not taught to
do it, because of a despicable lack of BOFH personnel.

(Time to take my gout meds.)

Re: BGP Experiment

[Stephen Satchell]

On 1/8/19 9:31 AM, Töma Gavrichenkov wrote:
> 8 Jan. 2019 г., 20:19 :
>> In the real world, doing the correct thing
> 
> — such as writing RFC compliant code —
> 
>> is often harder than doing
>> an incorrect thing, yes.
> 
> Evidently, yes.

I "grew up" during the early days of PPP.  As a member of the press I
attended an "inter-op" session at Telebit's campus, and watched as a
collection of engineers and programmers matched up implementations of
PPP and found bugs in both the Proposed Standard and in the
implementations thereof.

Watching these guys with all sorts of data monitors trying to figure out
who goofed was an interesting and fascinating experience.

During my stint with the Telecommunications Industry Associate TR-30
committee hashing out modem standards like V.32 et al and V.25 ter was a
similar exercise -- one that lead to me being in a near fight in a
parking lot in San Jose with a Microsoft enginner over clarity problems
with the proposed Standard for side-channel protocol.  "Can you do
better?"  "Yes."  "Prove it."  And I did.  My proposal was accepted by
all, even the Microsoft guy.

(We continued to collaborate until he cashed out of the company.)

Re: CenturyLink

[Stephen Satchell]

On 12/29/18 6:51 AM, Matthew Huff wrote:
> We have two stratum-1 servers synced with GPS and a PTP feed from a provider 
> that also provides PTP to market data systems, but we still have to monitor 
> drift between system time and NIST time. Don't ask for the logic behind it, 
> it's a regulation, not a technical requirement.

Having been a participant on Standards Working Groups, I understand
completely.  Regulations, like Standards, need to be written to apply to
as wide a population as possible.  Do those regulations dictate how you
monitor drift?  For example, can you have a system (I would use CentOS)
that syncs to the appliance as well as NIST and your inside NTP sources,
and use ntpq(8) to read the drift directly?

Re: CenturyLink...is being investigated by the FCC

[Stephen Satchell]

The telephone companies (I'm looking at YOU Verizon!) are bringing this
situation onto the community.  I can see the FCC NPRM now:

"What percentage of E911 terminations is being serviced over VoIP with
carrier-based network switching, or third-party network switching,
interfaced to the PSTN?

"How many emergency service areas terminate E911 VoIP into an
on-premises device like a Cisco voice router with outward-facing T1/E1
cards, or even outward-facing DS0 ports?"

This is just the flip side of the problem with VoIP on the consumer
side, not being able to easily associated a location with a 911 call
without significant help from the calling device.  Think cell phones on
the one hand, and the ubiquitous Cisco VoIP desk set on the other.

(Personal note: I have two copper-based DS0 lines here at my home
office.  And I'll keep them until Nevada Bell pries them out of my cold,
dead hands.  Now, those lines do terminate at a neighborhood SONET ring
fiber terminal with a battery, but it's not my worry.  Fax works fine --
which you can't say for VoIP connections.)

On 12/28/18 2:21 PM, Patrick Boyle via NANOG wrote:
> Ouch. Feel bad for the guys on the ground at C-link. Not a fun 24 hours.
> 
> 
> Sent with ProtonMail Secure Email.
> 
> ‐‐‐ Original Message ‐‐‐
> On Friday, December 28, 2018 3:17 PM, Anne P. Mitchell, Esq. 
>  wrote:
> 
>> And the other latest news is that the FCC is investigating the CenturyLink 
>> outage:
>>
>> https://www.theinternetpatrol.com/fcc-investigating-centurylink-outage-says-unacceptable/
>>
>>> On Dec 28, 2018, at 3:11 PM, Patrick Boyle via NANOG nanog@nanog.org wrote:
>>> Yes, there were 911 services affected. The latest word from C-link as of 
>>> 1:46PM mountain is that all 911 services are restored where they are the 
>>> provider. I'm not 100% sure if that's system-wide, or just my area in the 
>>> northwest, however.
>>> Sent with ProtonMail Secure Email.
>>> ‐‐‐ Original Message ‐‐‐
>>> On Friday, December 28, 2018 1:03 AM, Stephane Bortzmeyer bortzme...@nic.fr 
>>> wrote:
>>>
 On Fri, Dec 28, 2018 at 07:07:42AM +,
 Erik Sundberg esundb...@nitelusa.com wrote
 a message of 131 lines which said:

> CenturyLink will be conducting an extensive post-incident
> investigation and root cause analysis to provide follow-up
> information to our customers

 Is this problem also responsible for the 911 outage? If so, the
 post-mortem analysis is not useful only for CenturyLink customers but
 for everyone on the west coast.
> 
> 

Re: CenturyLink

[Stephen Satchell]

On 12/28/18 3:23 PM, Yang Yu wrote:
> On Fri, Dec 28, 2018 at 12:05 AM Stephane Bortzmeyer  
> wrote:
>> Is this problem also responsible for the 911 outage? If so, the
>> post-mortem analysis is not useful only for CenturyLink customers but
>> for everyone on the west coast.
> 
> Looks like most time.nist.gov servers (3 x NIST sites on AS49) are
> single homed on CenturyLink, anyone noticed NTP issues yesterday?
> 
> https://tf.nist.gov/tf-cgi/servers.cgi
> 

I have GPS-based Stratum 1 NTP appliances in my network, so I wouldn't
see any issues.  I suspect many other operators are in the same situation.

Re: Pinging a Device Every Second

[Stephen Satchell]

On 12/16/18 12:07 AM, Saku Ytti wrote:
> On Sun, 16 Dec 2018 at 00:48, Stephen Satchell  wrote:
> 
>> The 1500 bits are for each ping.  So 1000 hosts would be 1,500,000 bits
> 
> Why? Why did you choose 1500b(it) ping, instead of minimum size or
> 1500B(ytes) IP packets?
> 
> Minimum: 672kbps
> 1500B: 12.16Mbps

I was going from memory, and it is by no means perfect.  But...

A standard ping packet, with no IP options or additional payload, is 64
bytes or 512 bits.  If an application wants to make an accurate
round-trip-delay measurement, it can insert the output of a microsecond
clock, and compare that value for when the answer packet comes back.
Add at least 32 bits, perhaps 64.

Even with this sensible amount of extra ping payload, there is still
plenty of "bandwidth allocation" available to account for
encapsulations:  IPIP, VPN, MPLS, Ethernet framing, ATM framing, 

I can see a network operator with a complex mesh network wanting to turn
on Record Route (RFC791), which adds 24+(hops*32, max 536) bits to both
ping and ping-response packets.

So my 1500 bits for ping was not bad Tennessee windage for the
application described by the original poster, plus comments added by
others.  In fact, it would overestimate the bandwidth required, but not
by that much.

As for how much the use of ping would affect the CPU loading of the
device, that would depend a great deal on the implementation of the
TCP/IP stack in the CPE.  When I wrote _Linux IP Stacks Commentary_, the
code to implement ping is packet receipt, a very small block of code to
build the reply packet, and packet send the above mentioned 64 bytes of
packet.

Consider another service:  Network Time Protocol.  Unlike ping, there is
quite a bit of CPU load to process the time information through the
smoothing filters.  (Counter argument: a properly implemented version of
NTP will send time requests with separations of 60-1024 seconds.)

Re: Pinging a Device Every Second

[Stephen Satchell]

On 12/15/18 12:03 PM, Saku Ytti wrote:
> On Sat, 15 Dec 2018 at 18:52, Stephen Satchell  wrote:
> 
>> Short answer: about 1500 bits of bandwidth, and the CPU loading on the
> 
> I can't parse this.
> 
> 1000 hosts at 1 pps would be 672kbps on ethernetII encapulation with
> minimum size frames.
> 

The 1500 bits are for each ping.  So 1000 hosts would be 1,500,000 bits
per ping cycle at the monitor server, but not on each leaf router.  The
designer would need to analyze the network topology to see if there are
any possible choke points.  In a cable internet system, 1000 customers
on a single up/down channel pair would require 700 kilobits each way per
ping cycle.  Yes, this is payload bandwidth, it doesn't include packet
overhead.

Re: Pinging a Device Every Second

[Stephen Satchell]

On 12/15/18 7:48 AM, Colton Conor wrote:
> How much compute and network resources does it take for a NMS to:
> 
> 1. ICMP ping a device every second
> 2. Record these results.
> 3. Report an alarm after so many seconds of missed pings.
> 
> We are looking for a system to in near real-time monitor if an end
> customers router is up or down. SNMP I assume would be too resource
> intensive, so ICMP pings seem like the only logical solution.
> 
> The question is once a second pings too polling on an NMS and a consumer
> grade router? Does it take much network bandwidth and CPU resources from
> both the NMS and CPE side?
> 
> Lets say this is for a 1,000 customer ISP.

What problem are you trying to solve, exactly?  That more than anything
will dictate what you do.

Short answer: about 1500 bits of bandwidth, and the CPU loading on the
remote device is almost invisible.  Remember the only real difference
between ping and SNMP monitoring (UDP) is the organization of the bits
in the packet and the protocol number in the IP header.  It's still one
packet pair exchanged, unless you get really ambitious with your SNMP
OID list.

When I was in a medium-sized hosting company, I developed an SNMP-based
monitoring system that would query a number of load parameters (CPU,
disk, network, overall) on a once a minute schedule, and would keep
history for hours on the monitoring server.  The boss fretted about the
load such monitoring would impose.  He never saw any.

For pure link monitoring, which is what I'm hearing you want to do, in
my experience I found that a six-second ping cycle gives lots of early
warning for link failures.  Again, it depends on the specifications and
detection targets.

Some things to consider:

1.  Router restarts take a while.  Consumer-grade routers can take a
minute or more to complete a restart to the point where it will respond
to ping.  Carrier-grade routers are more variable but in general have so
many options built into them that it takes longer to complete a restart
cycle.  Since you are talking consumer-grade gear, you probably don't
want to be sensitive to CP power sags.

2.  Depending on the technology used on the link, you may get some
short-term outages, on the order of seconds, so doing "rapid" pings do
nothing for you.  During my DSL time, ATM would drop out for short
intervals -- so watch out for nuisance trips.

3.  Some routers implement ping limiting, so you have to balance your
monitoring sample rate against DoS susceptibility. Offhand, I don't know
the granularity of consumer router ping limiting, as I've never had that
question pop up.

4.  How large a monitoring server are you willing to devote to such a
system?  My web host monitoring used a 400-MHz Pentium II box, and it
didn't even breathe hard.  (A 1U Cobalt box, repurposed with Red Had
Linux, pulled from a junk pile.)  I was monitoring about 150 web host
servers. Extraolatuing the system load on that Cobalt box, I could have
handled 1500 web host servers and more.

Re: Enterprise GPON / Zhone Questions

[Stephen Satchell]

On 12/12/18 10:51 AM, William Herrin wrote:
> The AV lab gets screwed. You're running the coax they need through the
> noisy electrical riser because you didn't build dedicated comms risers
> and closets. Naturally nobody checked with them so you don't yet
> realize they can't do what they need to do with video over IP
> equipment.

There are double-shield coax solutions for noisy risers.  The outer
shield is grounded to the conduit, while the inner shield is grounded at
the source equipment.  One has to be sure that the voltage differential
between shields is kept as low as practical, which means paying
attention to grounding for the conduit AND equipment.

Re: Internet diameter?

[Stephen Satchell]

On 11/21/2018 07:32 PM, Ross Tajvar wrote:
> I'd argue that's just content (though admittedly a lot of it). You can't
> cache, e.g., a SIP trunk, and offices which need to connect to each other
> can't cache one another in a CDN either.


I would further argue that you can't cache active Web content, like bank
account statements, utility billing, help desk request/responses,
equipment status, and other things that change constantly.

Re: CVV (was: Re: bloomberg on supermicro: sky is falling)

[Stephen Satchell]

On 11/08/2018 07:50 PM, Chris Adams wrote:
> Signatures are no longer required for chip card transactions in the US,
> except I think for transactions where the auth is done on the amount
> before an added tip (restaurants).

Signatures are required for chip card transactions above a certain
dollar amount, with that dollar amount varying from merchant to
merchant.  I ran into this at the Sprint store when I used a chip card
to pay $800+ for my company's overdue wireless bill, and I had to apply
pen to paper by hand.  And I didn't do my usual response to "sign here":
draw a triangle and put "yield" in it.

Re: Rising sea levels are going to mess with the internet

[Stephen Satchell]

On 07/26/2018 10:48 AM, William Herrin wrote:
> Submarine cable is needed for deeper water (higher pressures) with
> more armor against damage since it's just laying on the seafloor
> exposed to everything that happens by.


Let's be specific:  everything with teeth that happens by.

Re: California fires: smart speakers and emergency alerts

[Stephen Satchell]

On 07/26/2018 10:31 AM, Chris Boyd wrote:
> 162.400
> 162.425
> 162.450
> 162.475
> 162.500
> 162.525
> 162.550
> 
> That’s about 1.85 meter wavelength, so a quarter wave antenna would
> be pretty large.  I’m sure the RF engineers can come up with a way to
> listen effectively without a huge antenna.
That's what loaded whips are for.  My shortwave radio has a loaded whip,
with the load switched by band.

Re: Rising sea levels are going to mess with the internet

[Stephen Satchell]

On 07/26/2018 09:48 AM, Rod Beck wrote:
> Unfortunately, the science community disagrees with Rob and you.

You mean the community that lives or dies on whether they get grant
money?  And the way to get grant money is to justify why they could be
fed MORE money.  Can you imagine how the "science community" would
continue to survive?

Now, the medical research community is another story.

Perhaps a better use for that grant money would be to develop Best
Practices for installing fiber cable that can withstand immersion to a
depth of 200 feet without failing -- thereby coming up with something
positive in light of the so-called scientific predictions.

Instead of "the sky is falling", say "here is how to prop up the sky on
a dollar a day."

Re: Proving Gig Speed

[Stephen Satchell]

On 07/20/2018 11:22 PM, Scott Weeks wrote:
> Oops, failure to communicate...  They folks on the
> eyeball end have consumer grade satellite internet 
> with VSATs in their yard.  Thus my CDN in the 
> satellite joke.

That idea would work better with a constellation of LEO satellites, as
opposed to geosync.  Especially if there was a link
satellite-to-satellite for update purposes.

Re: at business ipv6

[Stephen Satchell]

On 06/24/2018 07:52 AM, Lee Howard wrote:
> Randy said "at business 1g fiber going into an Arris"
> As fiber, it'll be PON. If it were a traditional cable company, I'd
> guess DPOE (DOCSIS Provisioning Over Ethernet).


AT fiber goes into a PON, and then into an Arris BGW210.

(Yes, I have business fiber here at satchell.net)

Re: What are people using for IPAM these days?

[Stephen Satchell]

On 06/12/2018 08:26 PM, valdis.kletni...@vt.edu wrote:
>> emacs!
> vim!
 ed!
>>> TECO!
>> cat
> IBM 029.

Youngster.  IBM 026.

Re: ICANN GDPR lawsuit

[Stephen Satchell]

On 06/01/2018 09:37 AM, McBride, Mack wrote:
> For routing whois information there aren't going to be many individuals and 
> it would seem
> that the corporations who employee individuals should be the ones protecting 
> those individuals
> work emails by providing a generic contact email forward.  Which is good 
> practice anyway
> since people leave and go on vacation and problems still happen.
> And the routing whois information is a lot more relevant to most of us here.

+1

Perhaps the Right Thing(SM) to do is to update the best practices
documents regarding role e-mail accounts for network operators.

1.  Add "networkmas...@example.com" to the list of required role accounts.

2.  Require that e-mail sent to role "networkmas...@example.com" be
accessible in some way by all technical people for the network in
question.  This can be done using a ticket system, or a simple mail
exploder.

3.  Require that e-mail sent to role account "ab...@example.com" by
accessible in some way by all members of the abuse desk.  This can be
done using a ticket system, or a simple mail exploder.

4.  Require the WHOIS information specify exactly these role accounts
for TECH and ABUSE, not a person.  This gets around the GDPR
requirements while maintaining the usefulness of the WHOIS without
having to go through an intermediate party or web site.

ICANN may want to consider this idea when adjusting its contracts with
registrars to eliminate GDPR exposure.