Re: Fast backbone to NA from Asia

[Justin Streiner]

What do you mean by "really fast transit"?
Are you referring to round-trip latency?  If so, what sort of latency
target are you looking to hit?
Where in North America are you trying to reach, using which providers?
If the networks in North America and Asia are multihomed, that provides
some level of protection from peering disputes.

Thank you
jms


On Tue, May 21, 2024 at 8:25 AM Scott Q.  wrote:

> I apologize if this is off-topic, but I am looking to purchase some VPS in
> Asia on a network that has really fast transit to NA and not affected by
> the latest peering disputes. The network should have really good
> connectivity to India / Indonesia / Thailand and ideally Australia as well.
>
> Please reply off-list.
>

Re: Best TAC Services from Equipment Vendors

[Justin H.]

Richard Laager wrote:

FWIW, I haven't tried calling after hours.
If I have to call after-hours, I get an answer from someone who is going 
to be handling my case (assuming that it doesn't have an engineer who's 
on-shift already assigned to it).  Yes, after-hours has traditionally 
been when I talk to the folks who are still learning, but when hasn't 
that been true?  Having said that, I always get the impression that 
they're having a side-line conversation with a more senior engineer who 
is helping them with any troubleshooting that they're not familiar with.


I admit, I don't use any particularly advanced features, so my 
experience may not be typical, but I've never had an operational issue 
that they haven't been able to solve fairly quickly.


Justin H.

Re: Verizon Business Contact

[Justin Krejci]

For me it is some AS 6167 destinations.
WHOIS for that ASN says this is Verizon Business.


AS Number:  6167

Org Name:   Verizon Business


I am not sure how I am supposed to accurately or authoritatively discern the 
differences in specific IP prefixes (or ASNs) as to whether they are are used 
in the Verizon Wireless, Verizon Business, Verizon XYZ, etc.
I am also not sure what the value would be understanding the difference as I 
have zero contacts at any Verizon entity: Wireless, Business, or any other.

I imagine at some level, there is a parent Verizon umbrella organization that 
is ultimately responsible for all underling organizations/divisions but I am 
not particularly interested in trying to pick apart the business silos of 
Verizon and then from there trying to chase down specific Verizon entity 
contacts to try and figure out who, might be the right contact to look into 
this. I have made efforts, prior to this NANOG thread even starting, to get 
this issue rectified but I have had zero luck so far getting any appropriate 
person at Verizon to take notice.

It kind of feels like trying to reach out to some company regarding a 
geolocation or IP-reputation type issue... just a lot of "Sorry, I don't know. 
try this other group that you already talked to" or simply "piss off" type 
responses. Both of which I have received in sizable quantities. Now that my 
brain is on that tangent, my favorite geolocation response was when I was told 
"your ISP needs to set the correct bits in the IP packets to designate the 
traffic as coming from the correct geography." I laughed and I cried at that 
one.



-Original Message-
From: Richard Laager 
mailto:richard%20laager%20%3crlaa...@wiktel.com%3e>>
To: Justin Krejci 
mailto:justin%20krejci%20%3cjkre...@usinternet.com%3e>>
Cc: nanog@nanog.org 
mailto:%22na...@nanog.org%22%20%3cna...@nanog.org%3e>>
Subject: Re: Verizon Business Contact
Date: Fri, 16 Feb 2024 20:41:04 -0600

On 2024-02-09 18:10, Justin Krejci wrote:

For a good long while (months) we have had similar issues with various Verizon 
destinations.

Only Verizon Wireless destinations, or other Verizon Business things?

As of today, I'm told (via an upstream provider) that Verizon Business says 
this is a Verizon Wireless issue.

Re: AWS WAF list

[Justin H.]

That matches my experience with these types of problems in the past.  
Especially when the end-users don't have a process for white-listing.  
We actually got a response from one WAF user to "connect to another 
network to log in, then you should be able to use the site, because it's 
just the login page that's protected".


I am working with someone off-list, so I have hope this can be resolved 
without account gymnastics. :)


Justin H.

Owen DeLong wrote:

The whole situation with these WAF as a service setups is a nightmare for the 
affected (afflicted) parties.

I saw this problem from both sides when I was at Akamai. It’s not great from 
the service provider side, but it’s an absolute shit show for anyone on the 
wrong side of a block. There’s no accountability or process for redress of 
errors whatsoever. The impacted party isn’t a customer of the WAF publisher, so 
they cant get any traction there. The WAF subscriber blindly applies the WAF 
and it’s virtually impossible to track down anyone there who even knows that 
they subscribe to such a thing, let alone get them to take useful action.

Best of luck.  The only thing I saw that worked while I was at Akamai was a few 
entities subscribed to the WAF service and then complained about getting 
blocked from their own web sites. Since they were then Akamai WAF customers, 
they could get Akamai to take action.

Crazy.

Owen



On Feb 16, 2024, at 09:19, Justin H.  wrote:

Justin H. wrote:

Hello,

We found out recently that we are on the HostingProviderIPList (found here 
https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-ip-rep.html)
 at AWS and it's affecting our customers' access to various websites.  We are a 
datacenter, and a hosting provider, but we have plenty of enterprise customers 
with eyeballs.

We're finding it difficult to find a technical contact that we can reach since 
we're not an AWS customer.  Does anyone have a contact or advice on a solution?

Sadly we're not getting any traction from standard AWS support, and end users 
of the WAF list like Reddit and Eventbrite are refusing to whitelist anyone.  
Does anyone have any AWS contacts that might be able to assist?  Our enterprise 
customers are becoming more and more impacted.

Justin H.

Re: IPv6 uptake (was: The Reg does 240/4)

[Justin Streiner]

We went pretty deep into the weeds on NAT in this thread - far deeper than
I expected ;)

Getting back to the recently revised topic of this thread - IPv6 uptake -
what have peoples' experiences been related to crafting sane v6 firewall
rulesets in recent products from the major firewall players (Palo Alto,
Cisco, Fortinet, etc)?  On the last major v6 deployment I did, working with
the firewalls was definitely one of the major pain points because the
support / stability was really lacking, or there wasn't full feature parity
between their v4 and v6 capabilities.

Thank you
jms

On Fri, Feb 16, 2024 at 11:04 PM William Herrin  wrote:

> On Fri, Feb 16, 2024 at 7:41 PM John R. Levine  wrote:
> > > That it's possible to implement network security well without using
> > > NAT does not contradict the claim that NAT enhances network security.
> >
> > I think we're each overgeneralizing from our individual expeience.
> >
> > You can configure a V6 firewall to be default closed as easily as you can
> > configure a NAT.
>
> Hi John,
>
> We're probably not speaking the same language. You're talking about
> configuring the function of one layer in a security stack. I'm talking
> about adding or removing a layer in a security stack. Address
> overloaded NAT in conjunction with private internal addresses is an
> additional layer in a security stack. It has security-relevant
> properties that the other layers don't duplicate. Regardless of how
> you configure it.
>
> Also, you can't "configure" a layer to be default closed. That's a
> property of the security layer. It either is or it is not.
>
> You can configure a layer to be "default deny," which I assume is what
> you meant. The issue is that anything that can be configured can be
> accidentally unconfigured. When default-deny is accidentally
> unconfigured, the network becomes wide open. When NAT is accidentally
> unconfigured, the network stops functioning entirely. The gate is
> closed.
>
> Regards,
> Bill Herrin
>
>
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/
>

Re: AWS WAF list

[Justin H.]

Justin H. wrote:

Hello,

We found out recently that we are on the HostingProviderIPList (found 
here 
https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-ip-rep.html) 
at AWS and it's affecting our customers' access to various websites.  
We are a datacenter, and a hosting provider, but we have plenty of 
enterprise customers with eyeballs.


We're finding it difficult to find a technical contact that we can 
reach since we're not an AWS customer.  Does anyone have a contact or 
advice on a solution?
Sadly we're not getting any traction from standard AWS support, and end 
users of the WAF list like Reddit and Eventbrite are refusing to 
whitelist anyone.  Does anyone have any AWS contacts that might be able 
to assist?  Our enterprise customers are becoming more and more impacted.


Justin H.

Re: IPv6 uptake (was: The Reg does 240/4)

[Justin Streiner]

The Internet edge and core portion of deploying IPv6 - dual-stack or
otherwise - is fairly easy. I led efforts to do this at a large .edu
starting in 2010/11.  The biggest hurdles are/were/might still be:
1. Coming up with a good address plan that will do what you want and scale
as needed.  It should also be flexible enough to accommodate re-writes if
you think of something that needs to be added/changed down the road :)
2. For providers who run older kit, v6 support might still be a bit dodgy.
You might also run into things like TCAM exhaustion, neighbor table
exhaustion, etc.  The point at which box X tips over is often not well
defined and depends on your use case and configuration.
3. The last time I checked, v6 support in firewalls and other middle-mile
devices was still poor.  Hopefully that has gotten better in the last 6-7
years.  My current day job doesn't have me touching firewalls, so I haven't
kept up on developments here.  I recall coming up with a base firewall
ruleset for Cisco ASAs to balance security with the functionality v6 needs
to work correctly.  Hopefully firewall vendors have gotten better about
building templates to handle some of the heavy lifting.
4. Getting people to unlearn the "NAT=Security" mindset that we were forced
to accept in the v4 world.

Thank you
jms

On Thu, Feb 15, 2024 at 8:43 PM John Levine  wrote:

> It appears that Stephen Satchell  said:
> >Several people in NANOG have opined that there are a number of mail
> >servers on the Internet operating with IPv6 addresses.  OK.  I have a
> >mail server, which has been on the Internet for decades.  On IPv4.
> >
> >For the last four years, every attempt to get a PTR record in ip6.arpa
> >from my ISP has been rejected, usually with a nasty dismissive.
>
> I don't think you'll get much disagreement that AT is not a great ISP.
>
> One straightforward workaround is to get an IPv6 tunnel from
> Hurricane. It's free, it works, and they will delegate the rDNS
> anywhere you want. My local ISP doesn't do IPv6 at all (they're a
> rural phone company who of course say you are the only person who's
> ever asked) so until they do, HE is a quite adequate option.
>
> R's,
> John
>

Re: Verizon Business Contact

[Justin Krejci]

For a good long while (months) we have had similar issues with various Verizon 
destinations.
I observed it only happens when passing through certain geographic regions of 
the US. Other regions make it through without issue.

This is directly observable and repeatable using Cogent's Looking glass website.
Do an IPv4 Trace to 63.59.67.68 using their US-Minneapollis router. It dies.
Do an IPv4 Trace to 63.59.67.68 using their US-Los Angeles router. It reaches 
the destination.

I went through a handful of Cogent's looking glass locations and found some 
that work and some that don't and concluded there must be one or more Verizon 
routers in a certain set geographic area that are having the problems.

Ultimately the issue is not resolved for me but I was able to BGP TE the 
traffic around the problem areas to facilitate reachability to the impacted 
destinations. This is obviously a tenuous band-aid.


Long story short: please, please, please, someone at Verizon or someone who has 
the ear of someone at Verizon, please, please, please, look into this.




-Original Message-
From: Richard Laager 
mailto:richard%20laager%20%3crlaa...@wiktel.com%3e>>
To: nanog@nanog.org
Subject: Verizon Business Contact
Date: Thu, 08 Feb 2024 13:01:14 -0600

Can someone from Verizon Business please contact me?

It appears that your network is losing traffic from Verizon Wireless
(e.g. 63.59.39.232, 63.56.37.4, or 63.59.67.68) to me (AS33362, e.g. to
69.89.207.16). Note that 63.59.166.100 -> 69.89.207.16 was successfully
(around 2023-11-27).

This breaks email between us and it's been MONTHS of VZW getting nowhere.

Based on some traceroutes (on 2023-11-27 and again just now), the
working ones go through 140.222.234.223 (0.ae10.GW7.CHI13.ALTER.NET)
while the broken ones stop at 140.222.234.221 (0.ae9.GW7.CHI13.ALTER.NET).

Re: SOVC - BGp RPKI

[Justin H.]

I'd be curious to know why it thinks that the S is "Stale".  I don't 
suppose it cites its sources?


Compton, Rich via NANOG wrote:


ChatGPT says:

SOVC in the context of RPKI (Resource Public Key Infrastructure) on a 
Cisco router stands for "Stale Origin Validation Cache". RPKI is a 
security framework designed to secure the Internet's routing 
infrastructure, primarily through route origin validation. It ensures 
that the Internet number resources (like IP addresses and AS numbers) 
are used by the legitimate owners or authorized AS (Autonomous System).


In RPKI, Route Origin Authorizations (ROAs) are used to define which 
AS is authorized to announce a specific IP address block. Network 
devices, like Cisco routers, use these ROAs to validate the 
authenticity of BGP (Border Gateway Protocol) route announcements.


The term "stale" in SOVC refers to a situation where the router's 
RPKI-to-Router protocol client has lost its connection to the RPKI 
server, or when the RPKI cache data is outdated and not refreshed for 
some reason. This can happen due to network issues, configuration 
errors, or problems with the RPKI server itself. When the RPKI cache 
is stale, the router cannot reliably validate BGP route announcements 
against the latest ROA data, potentially affecting routing decisions.


In a network security context, maintaining an up-to-date RPKI cache is 
crucial for ensuring that the network only accepts legitimate routing 
announcements, thereby reducing the risk of routing hijacks or 
misconfigurations. As a network security engineer, managing and 
monitoring the RPKI status on routers is an important aspect of 
ensuring network security and integrity.


I see it mentioned in this doc:

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_bgp/configuration/15-s/irg-15-s-book/irg-origin-as.pdf

*From: *NANOG  on 
behalf of Mohammad Khalil 

*Date: *Wednesday, January 31, 2024 at 10:35 AM
*To: *NANOG list 
*Subject: *SOVC - BGp RPKI

Greetings Am have tried to find out what is the abbreviation for SOVC 
with no luck. #sh bgp ipv4 unicast rpki servers  BGP SOVC neighbor is 
X. X. X. 47/323 connected to port 323 Anyone have encountered this? 
Thanks! ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍ ‍


Greetings

Am have tried to find out what is the abbreviation for SOVC with no luck.

#sh bgp ipv4 unicast rpki servers

BGP SOVC neighbor is X.X.X.47/323 connected to port 323

Anyone have encountered this?

Thanks!

AWS WAF list

[Justin H.]

Hello,

We found out recently that we are on the HostingProviderIPList (found 
here 
https://docs.aws.amazon.com/waf/latest/developerguide/aws-managed-rule-groups-ip-rep.html) 
at AWS and it's affecting our customers' access to various websites.  We 
are a datacenter, and a hosting provider, but we have plenty of 
enterprise customers with eyeballs.


We're finding it difficult to find a technical contact that we can reach 
since we're not an AWS customer.  Does anyone have a contact or advice 
on a solution?


Thank you,
Justin H.

Re: Sling TV Geolocation

[Justin Krejci]

I have Digital Element in my own internal wiki page for managing/documenting IP 
geolocation services headaches.

Searching them up on my page I see noted they have a contact us form that 
specifically lists "IP Address Data Update" as a contact reason. Maybe that 
will give you or others some avenue into the proper eyeballs over there.
https://www.digitalelement.com/contact-us/

I appreciate the follow up and will add a note to my page that Sling TV uses 
Digital Element, at least at the moment.

As always, good luck on your endeavor.



-Original Message-
From: Tim Burke mailto:tim%20burke%20%3c...@mid.net%3e>>
To: nanog@nanog.org 
mailto:%22na...@nanog.org%22%20%3cna...@nanog.org%3e>>
Subject: Re: Sling TV Geolocation
Date: Wed, 24 Jan 2024 20:32:10 +

(long overdue) Follow up on this – after plenty of emails, phone calls, and 
research, and our poor customers having to watch the Packer game, I was able to 
find out that Sling is using Digital Envoy/DigitalElement for geolocation... I 
assume the info on 
https://thebrotherswisp.com/index.php/geo-and-vpn/
 should work for this, but I am waiting for hear back from said geolocation 
vendor with an answer.

Thanks,
Tim



From: Tim Burke
Sent: Thursday, December 7, 2023 11:36 AM
To: nanog@nanog.org 
Subject: Sling TV Geolocation

Yet another geolocation post, because content networks don't pay attention to 
geofeeds... :-)

Anyone know who Sling TV is using for geolocation, or have a contact at Sling 
that can help? We acquired a /19 in July that we just started pushing out to 
customers, it is still geolocating back to the previous owner on Sling TV, 
despite publishing the prefix in our geofeed. Checked the usual lists with no 
luck.

Thanks,
Tim

Re: Outside plant - prewire customer demarc preference

[Justin Streiner]

We just built a new house in 2021. The builder ran 2" schedule 40 from the
side of the house out to the distribution point in front of my neighbor's
house.  I didn't specify 2" - that's what the builder ran.  A portion of
that run must have existed before construction because no one had to tear
up my neighbor's yard to get to the distro box.

Once I convinced Verizon that Fios was indeed available in this
neighborhood (separate matter entirely), it was an easy matter for the tech
to pull the drop cable through the empty conduit, drill a hole a few feet
above the foundation and land the cable in the basement.

I didn't run any surface tube or conduit in the basement, but there was
enough room for the install tech to run the cable without too much of a
fight.

Thank you
jms

On Fri, Dec 8, 2023, 2:06 PM Eric Kuhnke  wrote:

> If anyone assumes that residential real estate general contractors and low
> voltage/wiring subcontractors know or care about wifi signal or not putting
> RF units inside metal boxes - that would be a bad assumption to make.
>
>
> On Thu, Dec 7, 2023 at 10:18 PM Jay Hennigan  wrote:
>
>> On 12/6/23 23:22, Eric Kuhnke wrote:
>> > I think an important point for pre-wire and residential real estate
>> > developers to consider is also the conflicting needs of keeping things
>> > "neat and tidy" and last mile CPE location vs wifi coverage.
>>
>> If you assume that the appropriate place for a wifi access point is
>> colocated with the NID/ONT/CPE, you're doing it wrong.
>>
>> --
>> Jay Hennigan - j...@west.net
>> Network Engineering - CCIE #7880
>> 503 897-8550 - WB6RDV
>>
>>

Re: Fastly Peering Contact

[Justin Wilson (Lists)]

We have sent them some inquiries in markets we are with no reply.  Just figured 
they weren’t interested.




Justin Wilson
j...@mtin.net
jus...@fd-ix.com
Https://www.fdi-ix.com

> On Dec 5, 2023, at 4:14 PM, Peter Potvin via NANOG  wrote:
> 
> Looking for someone on the Fastly peering team to reach out regarding peering 
> on a couple mutual IXPs - sent an email to the peering contact as listed on 
> PeeringDB and never heard back, and also have a few colleagues who have 
> experienced the same issue.
> 
> Regards,
> Peter Potvin | Executive Director
> --
> Accuris Technologies Ltd.
> 

Re: ipv6 address management - documentation

[Justin Krejci]

I give +1 for phpipam



-Original Message-
From: Justin Wilson (Lists) 
mailto:%22justin%20wilson%20%28lists%29%22%20%3cli...@mtin.net%3e>>
To: NANOG mailto:nanog%20%3cna...@nanog.org%3e>>
Subject: Re: ipv6 address management - documentation
Date: Sun, 19 Nov 2023 23:38:28 -0500

Netbox or PHPipam. Phpipam allows you to break down subnets easier IMHo.


Justin Wilson
j...@j2sw.com

—
https://j2sw.com (AS399332)
https://blog.j2sw.com - Podcast and Blog

On Nov 16, 2023, at 1:09 PM, Jason Biel  wrote:

My recommendation:

https://github.com/netbox-community


On Thu, Nov 16, 2023 at 12:04 PM Aaron Gould 
mailto:aar...@gvtc.com>> wrote:
For years I've used an MS Excel spreadsheet to manage my IPv4
addresses.  IPv6 is going to be maddening to manage in a spreadsheet.
What does everyone use for their IPv6 address prefix management and
documentation?  Are there open source tools/apps for this?

Re: ipv6 address management - documentation

[Justin Wilson (Lists)]

Netbox or PHPipam. Phpipam allows you to break down subnets easier IMHo.


Justin Wilson
j...@j2sw.com

—
https://j2sw.com (AS399332)
https://blog.j2sw.com - Podcast and Blog

> On Nov 16, 2023, at 1:09 PM, Jason Biel  wrote:
> 
> My recommendation:
> 
> https://github.com/netbox-community
> 
> 
> On Thu, Nov 16, 2023 at 12:04 PM Aaron Gould  <mailto:aar...@gvtc.com>> wrote:
>> For years I've used an MS Excel spreadsheet to manage my IPv4 
>> addresses.  IPv6 is going to be maddening to manage in a spreadsheet.  
>> What does everyone use for their IPv6 address prefix management and 
>> documentation?  Are there open source tools/apps for this?
>> 
>> -- 
>> -Aaron
>> 
> 
> 
> --
> Jason

Survey for operators on IPv6 Extension Headers

[Justin Iurman]

Hi NANOG,

On behalf of the University of Liege (Belgium), we've been doing 
measurements on IPv6 Extension Headers over the last two years. More 
specifically, we're trying to evaluate how they are processed by routers 
along the path, and so from the edge. Now, we would like to compare our 
observations with the reality of operators. For that, we propose this 
very short survey [1] to operators and would really appreciate if you 
could complete it. Note that we guarantee anonymity of your answers by 
not sharing any data.


Looking forward to your input.

Thanks,
Justin

  [1] 
https://docs.google.com/forms/d/e/1FAIpQLSeHgEWVQMrAbB9-pRf-T-mDHXxRxw-mAqOR-B8YQCZE1GQfyw/viewform

Correcting Netflix ipv6 geolocation

[Justin Kilpatrick]

Our ipv6 subnet 2602::FBAD::/40 is showing up as in Kiev Ukraine on Fast.com 
and Netflix.com which is causing all sorts of problems for our US based 
customers.

Other services like Google and MaxMind don't seem to have any issue and report 
correct locations. Happy to follow up with more information off list.

Thanks! 

-- 
 Justin Kilpatrick | Cofounder and CTO
 jus...@althea.net

Re: Akamai Network Partnership

[Justin Krejci]

Hello Edy,


Log into your peeringdb.com account and go to their network, they have a 
peering contact listed there.


https://www.peeringdb.com/net/2




From: NANOG  on behalf of 
em...@edylie.net 
Sent: Tuesday, October 17, 2023 5:10 PM
To: nanog@nanog.org
Subject: Akamai Network Partnership

Dear All,

May I know if anyone could guide me to the right contact for Akamai
Network Partnership?

We are a network operator in Indonesia and is keen to work with Akamai
to speed up access to Akamai Content.

Many Thanks.

Best Regards,
Edy

Re: maximum ipv4 bgp prefix length of /24 ?

[Justin Wilson (Lists)]

I think it is going to have to happen.  We have several folks on the IX and 
various consulting clients who only need 3-6 Ips but have to burn a full /24 to 
participate in BGP. I wrote a blog post awhile back on this topic 
https://blog.j2sw.com/data-center/unpopular-opinion-bgp-should-accept-smaller-than-a-24/




Justin Wilson
j...@mtin.net

—
https://j2sw.com (AS399332)
https://blog.j2sw.com - Podcast and Blog

> On Sep 30, 2023, at 1:48 PM, Randy Bush  wrote:
> 
>> About 60% of the table is /24 routes.
>> Just going to /25 will probably double the table size.
> 
> or maybe just add 60%, not 100%.  and it would take time.
> 
> agree it would be quite painful.  would rather not go there.  sad to
> say, i suspect some degree of lengthening is inevitable.  we have
> ourselves to blame; but blame does not move packets.
> 
> randy, who was in the danvers cabal for the /19 agreement
> 

Prize Picks - gelocation/vpn/fraud system

[Justin Krejci]

Yes, unfortunate geolocation/vpn troubles strike again.


If any from PrizePicks.com are on here I would appreciate if you would reach 
out to me regarding a mutual customer not able to use your services.


If anyone else on NANOG has a contact there, I would appreciate some help 
getting in contact to resolve an issue that the regular support channel is 
unable to do.


Thanks!

Justin Krejci

Re: JunOS config yacc grammar?

[Justin H.]

Christopher Morrow wrote:


In looking around there are examples of some of this, in a way, the
most common thing
I end up looking at, and getting sad about, is some java monstrosity
(who's name escapes me)
but has shown up in a few nanog presentations over the years... it
makes me sad because it's
not super useful  in my world :( 'hard to use' is probably the best
way to describe it.



You're probably thinking of Batfish.

Justin H.

Re: Picking a RIR/obtaining an AS/ressurrecting a legacy space

[Justin Keller]

On Thu, Jul 6, 2023 at 1:05 PM William Herrin  wrote:
>
> On Thu, Jul 6, 2023 at 8:03 AM Dave Taht  wrote:
> > https://bgpview.io/prefix/198.177.242.0/24
>
> This is registered to Thyrsus Enterprises via ARIN, managed by an Eric
> Raymond of Pennsylvania. Refer to
> https://search.arin.net/rdap/?query=198.177.242.0
>
> If your friend happens to be Eric Raymond, his best bet is to simply
> leave it alone as a legacy address under his control rather than try
> to prove himself the legal successor in interest to Thyrsus
> Enterprises. As long as there is no current Thyrsus Enterprises, and
> as the guy on the whois, he'll be able to submit an LOA to an ISP and
> get them to accept the route.
>
> If your friend isn't Eric Raymond or Thyrsus Enterprises still exists
> and is someone else... you're done. Save yourself some grief and just
> go to an address broker. Let them help you through the process of
> getting addresses.
>
>
> > https://bgpview.io/prefix/198.177.243.0/24
>
> This is registered to Chester County Freenet care of Chester County
> Hospital. Refer to https://search.arin.net/rdap/?query=198.177.243.0
>
> Raymond again controls it, but since he's neither the Freenet nor the
> hospital you're going to run into trouble getting it routed let alone
> getting ARIN to recognize you as the legal successor in interest.
>
And if you want to talk to the hospital, it's under Penn, so you'll
probably have to go through them
>
> > the whole /22 was obtained to support the (long since deceased)
> > chester county freenet, but he has no record of that. Neither does
> > anyone else.
>
> Those would form a /22 with 240 and 241. Both are registered to other
> people. Unclear why you thought otherwise. If you were thinking 244
> and 245 (which do not form a /22 with 242 and 243), I'm sorry to tell
> you that they're also registered to someone else.
>
>
> > I presently have one vote for ARIN and another for RIPE. We are us
> > based, but more of the folk using libreqos are located elsewhere.
>
> The addresses are registered at ARIN. Until ARIN recognizes your
> friend as the registrant organization, they will remain so. At which
> point there's not a lot of benefit to moving them.
>
> Regards,
> Bill Herrin
>
>
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/

Re: Your input sought on PeeringDB's Network Type field

[Justin Streiner]

Leo:

The survey might also want to include response options along the lines of:
"Don't know / N/A".

Thank you
jms


On Wed, Jun 14, 2023 at 12:18 PM Leo Vegoda  wrote:

> Hi,
>
> PeeringDB's Product Committee wants your input on whether the Network
> Type field is useful. Should it go? Should it change?
>
> We have published a very short blog post describing the options and
> linking to the survey.
>
> https://docs.peeringdb.com/blog/network_type_your_input_sought/
>
> Your input will influence our decision.
>
> Thanks,
>
> Leo Vegoda for PeeringDB's Product Committee
>

Re: Do ISP's collect and analyze traffic of users?

[Justin Streiner]

Hank:

No doubt there is a massive amount of information that can be gathered from
in-box telemetry.  This thread appears to be more focused on providers
gathering data from traffic in flight across their infrastructure.

Thank you
jms

On Fri, May 19, 2023 at 8:49 AM Hank Nussbacher 
wrote:

> On 19/05/2023 15:27, Justin Streiner wrote:
>
> It amazes me how people can focus on Netflow metadata and ignore things
> like Microsoft telemetry data from every Windows box, or ignore the
> massive amount of html cookies that are traded by companies or how
> almost every corporate firewall or anti-spam box "reports" back to the
> mother ship and sends tons of information via secret channels like
> hashed DNS lookups just to be avoided.
>
> Regards,
> Hank
>
> > There are already so many different ways that organizations can find
> > out all sorts of information about individual users, as others have
> > noted (social media interactions, mobile location/GPS data, call/text
> > history, interactions with specific sites, etc), that there probably
> > isn't much incentive for many providers to harvest data beyond what is
> > needed for troubleshooting and capacity planning.  Plus, gathering
> > more data - potentially down to the level packet payload - is not an
> > easy problem to solve (read: expensive) and doesn't scale well at all.
> > 100G links are very common today, and 400G is becoming so.  I doubt
> > that many infrastructure providers would be able to justify the major
> > investments in extra infrastructure to support this, for a revenue
> > stream that likely wouldn't match that investment, which would make
> > such an investment a loss-leader.
> >
> > Content providers - particularly social media platforms - have a
> > somewhat different business model, but those providers already have
> > many different ways to harvest and sell large troves of user data.
> >
> > Thank you
> > jms
>
>

Re: Do ISP's collect and analyze traffic of users?

[Justin Streiner]

There are already so many different ways that organizations can find out
all sorts of information about individual users, as others have noted
(social media interactions, mobile location/GPS data, call/text history,
interactions with specific sites, etc), that there probably isn't much
incentive for many providers to harvest data beyond what is needed for
troubleshooting and capacity planning.  Plus, gathering more data -
potentially down to the level packet payload - is not an easy problem to
solve (read: expensive) and doesn't scale well at all. 100G links are very
common today, and 400G is becoming so.  I doubt that many infrastructure
providers would be able to justify the major investments in extra
infrastructure to support this, for a revenue stream that likely wouldn't
match that investment, which would make such an investment a loss-leader.

Content providers - particularly social media platforms - have a somewhat
different business model, but those providers already have many different
ways to harvest and sell large troves of user data.

Thank you
jms

On Tue, May 16, 2023 at 3:44 PM Matthew Petach 
wrote:

>
>
> On Tue, May 16, 2023 at 1:10 AM Jeroen Massar  wrote:
>
>>
>>
>> > On 16 May 2023, at 06:46, Matthew Petach  wrote:
>> > [..]
>> > I admit, I'm perhaps a little behind on the latest netflow whiz-bangs,
>> > but I've never seen a netflow record type that included HTTP cookies
>> > or PCAP data before.
>>
>> Take your pick from the "latest" ~2009 IPFIX Information Elements:
>>
>> https://www.iana.org/assignments/ipfix/ipfix.xhtml
>>
>> One can stuff almost anything in there.
>>
>> Now if one should, and if one is allowed to.
>>
>
> Wow.
>
> Thank you, Jeroen, I was indeed a bit out of date.
> Thank you for the pointer!
>
> (For those in the same boat as I, here's the relevant portion that clearly
> points out that yes, you can export the entire packet if you so desire):
>
> 313 ipHeaderPacketSection octetArray default current
>
> This Information Element carries a series of n octets from the IP header
> of a sampled packet, starting sectionOffset octets into the IP header.
>
> However, if no sectionOffset field corresponding to this Information
> Element is present, then a sectionOffset of zero applies, and the octets
> MUST be from the start of the IP header.
>
> With sufficient length, this element also reports octets from the IP
> payload. However, full packet capture of arbitrary packet streams is
> explicitly out of scope per the Security Considerations sections of [
> RFC5477 ] and [RFC2804
> ].
>
>
>
>  Thanks!
>
> Matt
> (still learning after all these years.   ^_^ )
>
>

Re: Aptum refuses to SWIP

[Justin Streiner]

When I worked for a local/regional ISP in the late 90s/early 00s, we
initially SWIP'd assignments for business customers and did generic
assignments for things like dial-up address pools or NAT front-end ranges
for residential customers, but provided more detailed information for
business customers.  Around 1998/1999 we switched from doing SWIPs to
running our own rwhois server.  Because we had good documentation, our IP
block requests to ARIN were generally pretty painless.

Thank you
jms

On Thu, May 4, 2023 at 5:36 PM Lyndon Nerenberg (VE7TFX/VE6BBM) <
lyn...@orthanc.ca> wrote:

> It seems Aptum has decided they will no longer SWIP any of their
> address space.  I've been trying to get a SWIP for a /48 that we
> were allocated in 2017, but they refuse.  And I also see they have
> pro-actively gone in and un-SWIPed both our /24s.
>
> Since you are ignoring my tickets about this, maybe somebody from
> Aptum would care to speak up in public and defend this "policy?"
>
> --lyndon
>

Re: Standard DC rack rail distance, front to back question

[Justin Wilson (Lists)]

I have not seen a standard on cabinets.  I have gear in a wide variety of 
racks.  Some of are real shallow.  Some are deep.  I use these to generically 
solve the sagging issue.


https://www.amazon.com/dp/B00XXDJASY?ref=nb_sb_ss_w_as-reorder-t1_k1_1_11==EFCM0EZP8BMA==navpoint+ra
NavePoint Universal 1U Rack Mount 4-Post Shelf Rail for Dell Compaq IBM HP APC 
- 33.5 Inches deep
amazon.com





Justin Wilson
j...@mtin.net

—
https://j2sw.com (AS399332)
https://blog.j2sw.com - Podcast and Blog

> On Apr 27, 2023, at 9:51 AM, Chuck Church  wrote:
> 
> Hey all.  Question about standard 4 post racks.  We bought some that are 
> adjustable.  Unfortunately, the posts are very flimsy, as these are some 
> fancy cabinets with spacing on the sides for vertical patch panels, etc.  We 
> found that 2 post mounting of most Cisco devices (namely Cat 9500 1RU 
> switches) are sagging quite bad.   We’re used to the new server type rails 
> that extend to support most reasonable distances front rails to back for 4 
> post mounting.  However, for a Cisco ASA1001, there aren’t rails, but rather 
> front and back ‘ears’ you use to hit both front and back posts.  These would 
> appear to not have any adjustability, the front to back post distance would 
> seem to need to match the ears, I assume they don’t adjust placement on the 
> router much.  Is there a ‘standard’ distance between front and back rails 
> that devices usually adhere to?  Googling didn’t find an answer readily.  
> These are 19” wide cabinets by the way.  
>  
> Thanks,
>  
> Chuck

Re: Windstream/Kinetic OSP assistance/clie sought

[Justin Streiner]

Other people on their street do have Kinetic, and I believe Windstream laid
cable from the street to their house, but they are being told there are no
facilities to connect them to an OLT or some other type of termination
device.

Thank you
jms

On Sat, Apr 22, 2023, 23:58 Alex Ryu  wrote:

> It is new construction home, it may be HoA, so it is controlled by HoA
> which may have some deal with one provider for landline.
>
> So it may be dictated by HoA until it reach to certain level when that
> binding deal is expired.
>
>
>
>
>
> Sent from Mail <https://go.microsoft.com/fwlink/?LinkId=550986> for
> Windows
>
>
>
> *From: *Justin Streiner 
> *Sent: *Saturday, April 22, 2023 10:46 PM
> *To: *NANOG 
> *Subject: *Windstream/Kinetic OSP assistance/clie sought
>
>
>
> Some of my family recently moved to an area of North Carolina where
> high-speed residential Internet connectivity options seem to be very
> limited. Outside of the options below, the only thing they're able to get
> is satellite Internet service, and the performance has been very poor
>
>
>
> They moved into an area where the neighbors have Kinetic (Windstream's
> Internet service) but my relatives have been told that there are no
> facilities available for them. This is new construction, so perhaps it's
> possible that their address hasn't been added into whatever systems
> Windstream/Kinetic use for service pre-qualification?
>
>
>
> Is there anyone I can talk to at Winstream to find out if it is indeed
> possible to get service at their address, or if there is way I can get past
> the gatekeepers?
>
>
>
> Any guidance from someone in the know at Windstream would be greatly
> appreciated.
>
>
>
> Thank you
>
> jms
>
>
>
>
>

Windstream/Kinetic OSP assistance/clie sought

[Justin Streiner]

Some of my family recently moved to an area of North Carolina where
high-speed residential Internet connectivity options seem to be very
limited. Outside of the options below, the only thing they're able to get
is satellite Internet service, and the performance has been very poor

They moved into an area where the neighbors have Kinetic (Windstream's
Internet service) but my relatives have been told that there are no
facilities available for them. This is new construction, so perhaps it's
possible that their address hasn't been added into whatever systems
Windstream/Kinetic use for service pre-qualification?

Is there anyone I can talk to at Winstream to find out if it is indeed
possible to get service at their address, or if there is way I can get past
the gatekeepers?

Any guidance from someone in the know at Windstream would be greatly
appreciated.

Thank you
jms

Re: Suggestions for those attending NANOG 88 in Seattle

[Justin H.]

That's good to know.  I was thinking of parking at the Westin since it's 
a familiar garage for me.


Eric Kuhnke wrote:
One observation on that, for those who find themselves in the area of 
the Westin Building for ISP/telecom related work:


The Amazon HQ underground parking on 6th ave, with entrance literally 
across the street from the Westin Building, is available for the 
public to use. Entrance is on 6th ave between Virginia and Lenora. You 
don't need an Amazon access badge/etc to use it, standard pay upon 
exit system.  I believe it's marked on Google Maps as the "Amazon 
Doppler" garage.


It has slightly better prices than the Westin's own parking garage and 
the vehicle spacing is not bad, I have parked full size SUVs in it 
without too much trouble. Not that it's a good idea to leave bags or 
anything visible in an unattended vehicle in downtown Seattle, but the 
garage also has /slightly/ better security than your average unstaffed 
parking garage elsewhere in the city.

Re: Scheduled outage -- Nationwide no driver license updates this weekend

[Justin Streiner]

Sounds like either the National Driver Register or NHTSA is single-homed to
Verizon, or the state DMVs each have a WAN circuit of some sort through
Verizon to where the National Driver Register system physically lives.  If
it's the latter, it sounds like a job that could be handled much more
effectively using site-to-site VPN tunnels, but I understand the realities
of being stuck in a long-term contract, policy mandates that dictate a
physical connection, or things along those lines.  Government agencies can
and sometimes do get funding for infrastructure upgrades and resiliency -
at least if they budget for it...

I'm sure the backstory here is both mundane and interesting.

Thank you
jms

On Sat, Feb 25, 2023 at 6:12 PM Sean Donelan  wrote:

> Verizon network maintenance will impact access to the “National Driver
> Register,” a system that motor vehicle offices around the country need to
> check before handing out a license.
>
> All 50 states and D.C. participate in the National Driver Register, a
> database maintained by the National Highway Traffic Safety Administration.
> The register contains information about drivers who have had their driving
> privileges revoked, suspended or denied due to serious traffic violations,
> such as driving under the influence of alcohol or drugs, reckless driving
> or excessive speeding.
>
>
> The scheduled maintenance should be finished by Monday, in case you needed
> to update your driver's license or planned to do some reckless driving
> this weekend.
>

Re: Cloudflare contact?

[Justin Paine via NANOG]

Replying directly.

On Sun, Feb 19, 2023 at 5:31 PM John Von Essen  wrote:

> I work with DuckDuckGo, and earlier today our macOS browser (which is
> currently available via the App store now) started getting caught by
> Cloudflare’s bot/fraud system. We did a fair amount of debugging, it
> appears to be some kind of browser/UA fingerprinting. This is happening for
> pretty much anyone using our browser, anywhere in the world, when browsing
> cloudflare powered sites. My hunch is this is accidental, but since we have
> no direct contacts at Cloudflare, we’re having a hard time escalating this.
>
> Thanks
> John

-- 

<https://www.cloudflare.com/>

__
*Justin Paine*
He/Him/His
VP, Global Head of Trust & Safety
101 Townsend St, San Francisco, CA 94107 <https://www.cloudflare.com/>

*PGP:* BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D
<https://keys.openpgp.org/vks/v1/by-fingerprint/BBAA6BCE33057FD66452711557B60114DE0B314D>

Lumen @ Nano. Need to meet

[Justin Wilson (Lists)]

If there anyone at NANOG from Lumen? I need to meet on a client of a client 
matter. ‘'


Justin Wilson
j...@mtin.net

—
https://j2sw.com (AS399332)
https://blog.j2sw.com - Podcast and Blog

Contact for androidpolice.com

[Justin Krejci]

Any contacts available that are responsible for androidpolice.com website 
hosting? Some of our IP space is not able to access their website. Other IP 
addresses of ours are working just fine. This appears to be some kind HTTP 
protocol layer issue but only affecting certain IP addresses. I am guessing it 
is some kind of web application firewall using outdated IP list data.


Yes I know it is hosted at Amazon but every time I have tried to go through 
Amazon for support with websites they are hosting, they have 100% of the time 
told me they can't and/or won't help me with website hosting issues on their 
web platform; I have to go through their customer... which I don't have any 
good contact info for. I've tried reaching them on twitter, I've tried blindly 
emailing people listed on their website guessing their email addresses, etc. I 
have had zero response.



Thanks!

Justin

Smaller than a /24 for BGP?

[Justin Wilson (Lists)]

Have there been talks about the best practices to accept things smaller than a 
/24? I qm seeing more and more scenarios where folks need to participate in BGP 
but they do not need a full /24 of space.  Seems wasteful.  I know this would 
bloat the routing table immensely.  I know of several folks who could split 
their /24 into /25s across a few regions and still have plenty of IP space.



Justin Wilson
j...@j2sw.com

—
https://blog.j2sw.com - Podcast and Blog
https://www.fd-ix.com

Re: FIDO2/Passkey now supported for 2FA for ARIN Online (was: Fwd: [arin-announce] New Features Added to ARIN Online)

[Justin Krejci]

Very interesting news. Improving online security is a win and this sounds 
promising.


Never having used FIDO2 for anything I am left, probably not uniquely, in the 
dark for hardware device support. The only link I found on the ARIN website for 
"hardware keys" was a link to another ARIN page, which as of the time I am 
writing this email, results in a 404.


The page with the link to supported hardware key details near the bottom @ 
https://www.arin.net/reference/materials/security/2fa/2fafaq/

The referenced hardware key details page that is 404 @ 
https://www.arin.net/reference/materials/security/wfa/fido2


I searched generally online for FIDO2 hardware keys and found a lot choices out 
there. Are all hardware keys the same? Will all hardware keys work with ARIN 
Online? I realize this is a brand new offering from ARIN so I am not upset that 
there is little data of the sort I am looking for right now but I would suggest 
ARIN get some better hardware key information on their website for people who 
are curious about but have little or no experience with FIDO2 and hardware 
components. After reading this https://en.wikipedia.org/wiki/FIDO2_Project I am 
wondering, can I simply use a smartphone itself as the hardware token to log 
into ARIN Online? Is there an app needed to do this?

I then discovered this FIDO2 keys page from online searching: 
https://www.yubico.com/store/compare/ which seems like one of many pretty 
popular key makers.
I assume there are possible risks affiliated with buying unknown hardware 
devices and plugging them into our trusted computer systems: key loggers, data 
exfiltration, trojan/malware infections, etc. There are even SFPs with built in 
switches or ones running Linux within the SFP itself able to do packet captures 
and all sorts of fun stuff. All the more reason I would appreciate a 
list/suggestion of well trusted hardware token makers. I did find this on 
Microsoft's website that seems like an easy to digest breakdown of some key 
makers: 
https://learn.microsoft.com/en-us/azure/active-directory/authentication/concept-authentication-passwordless#fido2-security-key-providers

Is FIDO2 just another industry buzzword? Am I the last one on NANOG to get into 
FIDO2 and therefore I am just asking a bunch of moronic questions? I rather 
think not and this time it seems like it may be worth getting buzzword 
compliant.

I realize it is not the job of ARIN to educate its customer base on the ins and 
outs of FIDO2 but I think a little extra working information would be quite 
helpful going forward.

<https://www.yubico.com/store/compare/>Thanks to ARIN for implementing this, 
thanks to those that have pushed for the deployment of this protocol, and 
thanks to those that will respond kindly to me in my ignorance on this topic!!


-Justin



From: NANOG  on behalf of Royce 
Williams 
Sent: Tuesday, January 3, 2023 5:20 PM
To: John Curran
Cc: NANOG
Subject: Re: FIDO2/Passkey now supported for 2FA for ARIN Online (was: Fwd: 
[arin-announce] New Features Added to ARIN Online)

On Tue, Jan 3, 2023 at 11:59 AM John Curran 
mailto:jcur...@arin.net>> wrote:
FYI - ARIN Online now has FIDO2/Passkey as an option for two-factor 
authentication (2FA) - this is a noted priority for some organizations.

John - this is a great step forward! Kudos to the tech team who helped make the 
leap - it can be daunting.

Some feedback, take or leave as you see fit, based on my scars:

First, thanks specifically for the support for unique key names (you might be 
surprised at how many services don't!), and for the FIDO2 support of on-key 
PINs.

Second, I'd like to second ;) - but go beyond - Job's feature request for 
multiple-key support, both in count and additional UX. Support for *more* than 
two keys is recommended, to fit a wider variety of use cases and threat/risk 
models (connector availability, shared/role accounts, offsite key backup, etc 
etc). From my survey of 50 providers of U2F / FIDO / FIDO2, key-count support 
ramps up quickly from one (PayPal - come on, y'all!), two (Bank of America), 
and five (AOL/Yahoo and Coinbase), with the rest supporting *ten or more keys* 
(and yes, higher key counts have use cases, though user experience degrades 
above ten keys). And when multiple key support is added, please consider some 
UX around managing the list of keys (like allowing the user to *modify* key 
names without having to delete and re-add them, showing the timestamp, IP, OS 
family / platform, etc. from where the key was last used). Great key UX 
examples to emulate in this space include Dropbox and Google. (And showing the 
IP's ASN would be a uniquely ARIN twist. :D )

Third, please consider allowing a mix of authenticators (instead of the current 
exclusive choice among TOTP, FIDO2, and SMS). While it will be excellent to 
allow users to *eventually* opt into exclusive use of security keys (as with 
Google's Advanced Protection

Verizon Email to SMS gateway

[Justin H.]

Anyone else seeing massive delays in Verizon's email to SMS gateway 
lately?  I'm seeing delays on emails to @vtext and @vzwpix addresses at 
anywhere form 45 minutes to 12 hours.


Justin H.

Re: AWS Blacklisting?

[Justin H.]

Is it possible this is a geolocation issue?  I'm not sure I've heard of 
that causing a 403 Forbidden, but I'm also not too familiar with AWS.


Justin H.

William Herrin wrote:

Sounds like "AWS Shield" but I couldn't begin to tell you who to contact.

On Tue, Oct 18, 2022 at 4:51 PM Justin H.  wrote:

I have a customer who's suddenly been getting 403's today on AWS hosted
sites.  My google-fu seems to be failing me because I can't seem to find
any information on who manages that on their side or how to fix the issue.

I've sent an email to amzn-noc-cont...@amazon.com based on ARIN
contacts, but it doesn't seem to be a responsive address.

Has anyone had to navigate this particular maze before?

Thank you,
Justin H.

AWS Blacklisting?

[Justin H.]

I have a customer who's suddenly been getting 403's today on AWS hosted 
sites.  My google-fu seems to be failing me because I can't seem to find 
any information on who manages that on their side or how to fix the issue.


I've sent an email to amzn-noc-cont...@amazon.com based on ARIN 
contacts, but it doesn't seem to be a responsive address.


Has anyone had to navigate this particular maze before?

Thank you,
Justin H.

2 Byte ASNs??

[Justin Wilson (Lists)]

Whats the availability of two byte asns look like? Anyone able to obtain one 
recently? I have a network that is all Mikrotik and the route targets are 
messing with them.  They can’t use communities with their 4 bytes asn.  It’s 
one of those it really isn’t a big deal but I thought I would ask.  




Justin Wilson
j...@mtin.net

—
https://j2sw.com (AS399332)
https://blog.j2sw.com - Podcast and Blog

Re: Sigh, friends don't let politicians write tech laws

[Justin Krejci]

Leave the private matter of private email handling in the hands of the private 
participants of the private email system.

If congress wants to create a government mandate on political campaign emails, 
the political campaigns themselves ought to be forced to mark their emails as a 
political campaign emails. This would more easily allow sorting and filtering 
of emails by mail providers and by the users and help ensure easier reception 
or easier rejection by the users. I can say "yes, I want campaign emails" and I 
get less or no filtering or I can say "no, I do not want campaign emails" and 
never have to see them again .

I have contacted my reps and expressed my opinions and some relevant facts on 
this matter.



From: NANOG  on behalf of Anne 
Mitchell 
Sent: Friday, July 29, 2022 4:57 PM
To: nanog@nanog.org
Subject: Re: Sigh, friends don't let politicians write tech laws



> On Jul 29, 2022, at 3:37 PM, John Levine  wrote:
>
> It appears that Michael Thomas  said:
>> -=-=-=-=-=-
>>
>>
>> https://www.congress.gov/bill/117th-congress/senate-bill/4409/text?r=9=1
>>
>> the body of the proposed law:
>
> This bill was filed by a bunch of the usual right wing suspects about
> a month ago.  It was referred to committee, like all filed bills, and
> I very much doubt it will ever emerge.

I'm inclined to agree, except that as we've seen Google has already attempted 
to cave, which means that they (the bills' sponsors) will feel even more 
emboldened, and can point to Google's "pilot program" as evidence that "even 
Google admits there is a problem, so we need the law to make the other big 
providers do it."

I believe we can't rely on it being buried without a little help.  It costs 
nothing to send an email to a representative, so..why not provide that help. ;~)

Anne

--
Anne P. Mitchell, Attorney at Law
CEO Institute for Social Internet Public Policy
Author: Section 6 of the CAN-SPAM Act of 2003 (the Federal anti-spam law)
Author: The Email Deliverability Handbook
Board of Directors, Denver Internet Exchange
Dean Emeritus, Cyberlaw & Cybersecurity, Lincoln Law School
Prof. Emeritus, Lincoln Law School
Chair Emeritus, Asilomar Microcomputer Workshop
Counsel Emeritus, eMail Abuse Prevention System (MAPS)

Re: Why are ad networks so slow with IPv6?

[Justin Streiner]

Is your experience the same if you run a browser-based ad blocker or
something external like Pi-hole?  I have Fios, but Verizon hasn't rolled v6
out here that I can see.  My v6 traffic runs over a tunnel to Hurricane
Electric, and I haven't noticed any unusually slow load times for the ads
that make it through Pi-hole/uBlock/etc.

Thank you
jms

On Mon, Jul 11, 2022 at 11:03 AM Sean Donelan  wrote:

>
> Verizon FIOS has been rolling out IPv6 across Northern Virginia. Hurrah!
>
> Stuff with ads (which is almost everything on the modern Internet) now
> load much more slowly or timeout.
>
>
> Changed IPv4/IPv6 connection preferences to use IPv4 first. Much improved
> user experience.
>

Re: Reporting Comcast outside plant issues?

[Justin Streiner]

Thank you to everyone who responded off-list.  I was able to get a repair
ticket opened with Comcast and they will be dispatching a crew to take a
look.

Thank you
jms

On Sun, Jun 26, 2022 at 10:27 PM Justin Streiner 
wrote:

> Does anyone here have a contact at Comcast for reporting outside plant
> issues that are not (at the moment) service-affecting? I am not a Comcast
> customer, and they make it nearly impossible for non-customers to reach
> them unless you're signing up for service.
>
> There is a long coax span (2-300 feetthat has come off of a pair of
> utility poles and is laying on the ground near my house. I moved it off of
> the road to keep it from getting run over, but reaching anyone at Comcast
> to get the cable re-attached to the poles has been difficult.
>
> Any insight anyone (off-list is fine) could offer would be appreciated.
>
> Thank you
> jms
>
>
>

Reporting Comcast outside plant issues?

[Justin Streiner]

Does anyone here have a contact at Comcast for reporting outside plant
issues that are not (at the moment) service-affecting? I am not a Comcast
customer, and they make it nearly impossible for non-customers to reach
them unless you're signing up for service.

There is a long coax span (2-300 feetthat has come off of a pair of utility
poles and is laying on the ground near my house. I moved it off of the road
to keep it from getting run over, but reaching anyone at Comcast to get the
cable re-attached to the poles has been difficult.

Any insight anyone (off-list is fine) could offer would be appreciated.

Thank you
jms

Re: Congrats to AS701

[Justin Streiner]

I might call Verizon and ask about v6 availability as I periodically do.
I'll check if I see anything different on my gear later today.  I have a
GPON business service with static IPv4 at one location and an older BPON
business service with static IPv4 in another location.

Thank you
jms

On Mon, Jun 13, 2022 at 11:18 AM Nimrod Levy  wrote:

> Also, it doesn't seem to be enabled on ports that have static ipv4
>
> but progress is progress. we'll take it.
>
> Nimrod
>
>
> On Mon, Jun 13, 2022 at 11:17 AM Matthew Huff  wrote:
>
>> Still no IPv6 in Westchester County, NY ☹
>>
>>
>>
>> Great sign though, maybe NY will get it eventually
>>
>>
>>
>> *From:* NANOG  * On Behalf Of *Joe
>> Loiacono
>> *Sent:* Monday, June 13, 2022 10:55 AM
>> *To:* nanog@nanog.org
>> *Subject:* Re: Congrats to AS701
>>
>>
>>
>> FiOS from Maryland (anonymized):
>>
>> enp3s0: flags=4163  mtu 1500
>> inet 192.168.1.164  netmask 255.255.255.0  broadcast 192.168.1.255
>> inet6 fe80::b104:8f4d:e5b2:e13b  prefixlen 64  scopeid 0x20
>> inet6 2600:4040:b27f:cb00:a9b1:5f59::  prefixlen 64
>> scopeid 0x0
>> inet6 2600:4040:b27f:cb00:24a8:7b31::  prefixlen 64
>> scopeid 0x0
>> inet6 2600:4040:b27f:cb00:e1b6:8b83::  prefixlen 64
>> scopeid 0x0
>> ether d0:67:e5:23:ec:fe  txqueuelen 1000  (Ethernet)
>> RX packets 2518066  bytes 1448982813 (1.4 GB)
>> RX errors 0  dropped 0  overruns 0  frame 0
>> TX packets 2157395  bytes 260073952 (260.0 MB)
>> TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>>
>> a@b:~$ ping 2607:f8b0:4004:c09::6a
>> PING 2607:f8b0:4004:c09::6a(2607:f8b0:4004:c09::6a) 56 data bytes
>> 64 bytes from 2607:f8b0:4004:c09::6a: icmp_seq=1 ttl=59 time=24.0 ms
>> 64 bytes from 2607:f8b0:4004:c09::6a: icmp_seq=2 ttl=59 time=17.6 ms
>> 64 bytes from 2607:f8b0:4004:c09::6a: icmp_seq=3 ttl=59 time=20.4 ms
>> 64 bytes from 2607:f8b0:4004:c09::6a: icmp_seq=4 ttl=59 time=23.4 ms
>> ^C
>> --- 2607:f8b0:4004:c09::6a ping statistics ---
>> 4 packets transmitted, 4 received, 0% packet loss, time 3004ms
>> rtt min/avg/max/mdev = 17.618/21.351/23.983/2.555 ms
>>
>>
>>
>> On 6/12/2022 1:55 PM, Christopher Morrow wrote:
>>
>>
>>
>>
>>
>> On Sat, Jun 11, 2022 at 11:03 PM Darrel Lewis (darlewis) <
>> darle...@cisco.com> wrote:
>>
>> I, for one, am having a hard time finding the proper words to express the
>> joy that I am feeling at this momentous moment!
>>
>>
>>
>>
>>
>> It's quite amazing, I think... that it's taken so long to get to
>> deployment you can actually see on the fios plant :)
>>
>> I'd note I can't see the below on my homestead, but I can at a relative's
>> (where the ifconfig data is from).
>>
>> I also can't tell if the upstream will PD a block to the downstream...
>> and the VZ CPE is 'not something I want to fiddle with',
>>
>> because everytime I have tried at my house I've just taken it out behind
>> the woodshed with a maul... and replaced it with
>>
>> something I CAN configure successfully. (plus.. don't want that TR 069 in
>> my home...)
>>
>>
>>
>> -chris
>>
>>
>>
>> -Darrel
>>
>>
>>
>> On Jun 11, 2022, at 7:05 PM, Christopher Morrow 
>> wrote:
>>
>> 
>>
>>
>>
>> Looks like FIOS customers may be getting ipv6 deployed toward them,
>> finally:
>>
>> ifconfig snippet from local machine:
>> inet6 2600:4040:2001:2200:73d2:6bcc:1e6b:43a1  prefixlen 64
>>  scopeid 0x0
>> inet6 2600:4040:2001:2200:e87:bf36:b6cb:6ce1  prefixlen 64
>>  scopeid 0x0
>>
>>
>>
>> ping attempt:
>>
>>   64 bytes from bh-in-f106.1e100.net (2607:f8b0:4004:c09::6a):
>> icmp_seq=1 ttl=59 time=8.71 ms
>>
>>
>>
>> 8ms from mclean, va to ashburn, va isn't wondrous, but at least it's ipv6
>> (and marginally faster than ipv4)
>>
>>
>>
>> Congrats to the 701 folk for deploying more widely!
>>
>>   (note: I don't know exactly when this started, nor how wide it really
>> is, but progress here is welcomed by myself at least :) )
>>
>> -chris
>>
>>

Re: Disney+ Issues

[Justin Krejci]

I'd suggest you reach out to hosting company and have them mark the block(s) in 
question as re-allocated to your organization.  Also Neustar does support 
self-published geofeeds so you could also publish your own + leased IP space 
and them get them to subscribe to your list.



From: NANOG  on behalf of 
Norman Jester 
Sent: Friday, April 29, 2022 12:21 PM
To: nanog@nanog.org
Subject: Re: Disney+ Issues

On Fri, Apr 29, 2022 at 6:07 AM Brian Turnbow  wrote:
>
> Hi Norman
> >Anyone from Disney+ here? If you can reply off-list I'd appreciate it. I 
> >have emailed every place I can think of to solve a geoip problem affecting 
> >hundreds of customers, no reply in weeks.
>
>
> Yeah we just went through the same thing.
> Many other providers in Italy have been impacted as well.
> Only way we found to resolve the issue was single customers opening tickets…
> We tried at the  provider level but were continuously rebuffed.
> The single customers opening TTs had it resolved in minutes and after a bunch 
> did  the others were able to connect...
> If you do find a way to get it done on the provider level I would love to 
> hear about it.
>
> Brian

We're having a heck of a time with this, customers are posting all
over social media about it etc.
The company who does their ip classification is Neustar and we have
been talking to them.
For some reason they do not comprehend the fact that companies in
these days must lease ip space
due to the shortages.  We are delegated ipv4 from a datacenter (in
addition to our own ip space) which
is all used for our eyeball network of home users.  They said "This ip
space is from a hosting company", which
it is not.. it's from a datacenter where some of our core gear
aggregates routes from all the carriers in that hotel.
We backhaul all data out to our pops all over San Diego and it ends up
in customers homes.

Ips are properly delegated, but they tag them as VPN and HOSTING when
they are not. Worse off, they said
they won't change it. I asked them if they monitored NANOG and they
didn't know what it was. Nice to know
the people making those decisions are not paying attention to the
network world and making those decisions that
affect many many people.  With great power comes great responsibility.

Re: Geolocation data management practices?

[Justin Krejci]

For corrections/updates, what I have found to be generally successful is


1. make sure to advertise the IP blocks into the DFZ from your ASN as soon as 
possible

2. make sure ARIN data is accurate (we use ARIN, you may use one of the other 
registries)

3. update my geofeed, as referenced already in this thread

4. directly contact organizations that have geolocation services but don't 
subscribe to my geofeed


If anyone has any additional geolocation organizations I didn't list, I would 
be happy to hear about them.



Geofeed subscriptions are in place with these organizations


IP Info
https://ipinfo.io/
https://ipinfo.io/faq/article/49-how-can-i-submit-a-correction
https://ipinfo.io/corrections

dbip
https://db-ip.com/
https://db-ip.com/contact/
support  db-ip.com

IPGeolocation
https://ipgeolocation.io/
support  ipgeolocation.io

Maxmind
https://www.maxmind.com/en/geoip-demo
https://support.maxmind.com/geoip-data-correction-request/

Neustar
https://www.home.neustar/resources/tools/ip-geolocation-lookup-tool
https://www.home.neustar/resources/tools/submit-to-global-ip-database
ipintel  support.neustar

BigDataCloud
https://www.bigdatacloud.com/ip-geolocation/

Digital Element
https://www.digitalelement.com/geolocation/
https://www.digitalelement.com/contact-us/
ip-data  digitalelement.com

ip2location
https://www.ip2location.com/demo
support  ip2location.com
Only accepts feeds when all entries have a city defined

Google
https://isp.google.com
Set geofeed URL within their ISP portal




No geofeed subscriptions in place for these organizations and require 
individual contact for corrections/updates


ipstack
https://ipstack.com/
https://ipstack.com/contact

Geo IP View
https://www.geoipview.com/
andrew  geoipview.com
email address is not currently receiving mail, as such I assume not many are 
using this service

IPligence
http://www.ipligence.com/geolocation
https://www.ipligence.com/contact

ipdata
https://ipdata.co/?ref=iplocation
https://ipdata.co/corrections.html
corrections  ipdata.co
working on adding geofeed support

IPIP
https://en.ipip.net/ip.html
sarah  ipip.net

IPHub
https://iphub.info/

IPinsight
https://ipinsight.io/
william  ipinsight.io

Info Sniper
https://infosniper.net/
https://infosniper.net/geoip-data-correction.php

GeoGuard
https://www.geocomply.com/products/geoguard/
ipintelligence  geoguard.com
More of a "are they using a VPN/hiding service" and not so much of a "Where are 
they" service.



From: NANOG  on behalf of Josh 
Luthman 
Sent: Thursday, April 21, 2022 9:24 AM
To: Rubens Kuhl
Cc: Nanog
Subject: Re: Geolocation data management practices?

Go through this list:
https://thebrotherswisp.com/index.php/geo-and-vpn/

The RFC only works if they're pulling your feed and they'd only know that if 
you contact them in the first place.

On Thu, Apr 21, 2022 at 9:14 AM Rubens Kuhl < 
rube...@gmail.com> wrote:
Besides geofeed, there are also geoidx records in IRRs but whether
geolocation services actually use geofeed or geoidx remains to be
seen. You can see some geoidx: at this IRR entry in TC:
https://bgp.net.br/whois/?q=-s%20TC%20-i%20mnt-by%20MAINT-AS271761

Regarding LACNIC, what LACNIC, NIC.mx and NIC.br do is to select which
RIR or NIR services requests depending on the organisation's country.


Rubens

On Thu, Apr 21, 2022 at 9:53 AM Shawn < 
mailman.nanog@kleinart.net> wrote:
>
> Aloha NANOG,
>
> What is the best practice (or peoples preferred methods) to
> update/correct/maintain geolocation data?
> Do most people start with description field info in route/route6 objects?
>
>
> Also, thoughts and considerations on using IPv4 space from one RIR in
> countries belonging to another RIR?
>
> With IPv4 exhaustion and inter-RIR IPv4 transfers, and geolocation data, it
> seems less applicable than it had been (a decade ago).  The IP's will be
> used for CDN, not by end-users/subscribers.
> Context: trying to work through an administrative "challenge" with LACNIC
> regarding an IPv4 transfer, considering transferring to ARIN and then using
> in LACNIC (then once resolved, transfer from ARIN to LACNIC).  Or just using
> existing ARIN space in Brazil.
> LACNIC is making things more difficult than they need to be.  I know this is
> NANOG... but seeking advice, working on a global network, US HQ, currently
> no active "registration" in LACNIC (except Brazil), but we operate in 5
> countries in the region (data center/colo).  We would use Brazil, but very
> hesitant to use their NIC ( 

AT& T peering Contact?

[Justin Wilson (Lists)]

Folks, 
I need an ATT Wireless/ATT Mobility peering contact.  The emails on 
their peeringdb entries bounce back as non existent.  Have a problem with a 
prefix that works everywhere except when folks are on AT LTE.


Justin Wilson
j...@mtin.net

—
https://j2sw.com (AS399332)
https://blog.j2sw.com - Podcast and Blog

Re: Let's Focus on Moving Forward Re: V6 still not supported re: 202203261833.AYC

[Justin Streiner]

Abe:

To your first point about denying that anyone is being stopped from working
on IPv4, I'm referring to users being able to communicate via IPv4.  I have
seen no evidence of that.

I'm not familiar with the process of submitting ideas to IETF, so I'll
leave that for others who are more knowledgeable on that to speak up if
they're so inclined.

Thank you
jms

On Sat, Mar 26, 2022 at 6:43 PM Abraham Y. Chen  wrote:

>
> 1)"... no one is stopping anyone from working on IPv4 ... ":
> After all these discussions, are you still denying this basic issue? For
> example, there has not been any straightforward way to introduce IPv4
> enhancement ideas to IETF since at least 2015. If you know the way, please
> make it public. I am sure that many are eager to learn about it. Thanks.
>

Re: are underwater routers a thing?

[Justin Streiner]

High voltage DC from landing stations to the underwater amps and submarine
branching units.

jms

On Thu, Mar 17, 2022, 22:46 Karl Auer  wrote:

> On Thu, 2022-03-17 at 21:26 -0500, Jerry Cloe wrote:
> > First thing that comes to mind is power, how would you power them?
>
> Hydroelectricity (or wave energy), *obviously*. Sheesh.
>
> :-)
>
> Regards, K.
>
> --
> ~~~
> Karl Auer (ka...@biplane.com.au)
> http://www.biplane.com.au/kauer
>
> GPG fingerprint: 61A0 99A9 8823 3A75 871E 5D90 BADB B237 260C 9C58
> Old fingerprint: 2561 E9EC D868 E73C 8AF1 49CF EE50 4B1D CCA1 5170
>
>
>
>

RE: Ukraine request yikes

[justin]

The problem with all of these sorts of things and why respectable entities like 
ICANN should avoid such things is because its inherently subjective and prone 
to a sort of viewers bias that is moulded more or less by the propaganda of the 
state from which you come (in our case, North America/US et al).

For instance, an actually unpopular opinion is that this all started when a 
lawfully elected government was overthrown by a minority of the population 
(<1%) and that the majority of Ukrainians were disenfranchised as a result. 
This was particularly acute in the Donbass region that voted for Yanukovych 
very heavily. This brought about an actual rebellion, one that is flatly denied 
by the government in Kyiv, which in turn brought about the Minsk agreement 
where the breakdown was that the rebels sought to have local elections for 
their own governors/mayors that could not be dismissed by the federal 
legislature. For whatever reason, the Government in Kyiv found this unpalatable 
and never implemented this part of the agreement until finally the ceasefire 
broke down and a formal war ensued. The point of this paragraph being that 
discerning which side is representing "democracy" is a matter of perspective.

Because the shoe could easily fit on the other foot and also be legitimately 
correct and the same argument could be made to remove TLDs for UA or supporting 
countries and because which is correct is almost always a matter of 
perspective-- its best for any such governing entity to avoid allowing itself 
to be drawn into such ordeals. 

As for their request, given that the country has more or less banned all 
periodicals in Russian from the news stand irrelevant of content, routinely 
shutdown independent media outlets and because this email simply acknowledging 
valid grievances in south eastern Ukraine could be cause for a 10 year term in 
prison if written from within Ukraine-- I will only say that I find the request 
by the government there to be "extremely consistent with Ukrainian values".


-Original Message-
From: NANOG  On Behalf Of Matt 
Hoppes
Sent: Wednesday, March 2, 2022 5:54 PM
To: George Herbert ; Nanog 
Subject: Re: Ukraine request yikes

My (unpopular opinion) Russia does not deserve any amenities of the modern 
world.  They have made their bed and now they have to sleep in it.

On 3/1/22 3:16 AM, George Herbert wrote:
> Posted by Bill Woodcock on Twitter…
> https://twitter.com/woodyatpch/status/1498472865301098500?s=21
> 
> https://pastebin.com/DLbmYahS
> 
> Ukraine (I think I read as) want ICANN to turn root nameservers off, 
> revoke address delegations, and turn off TLDs for Russia.
> 
> Seems… instability creating…
> 
> -george
> 
> Sent from my iPhone

Telia is now Arelion

[Justin Krejci]

https://www.arelion.com/




Since all other work is now complete in the world I should have plenty of time 
to update documentation, billing, labels, port names, route-maps, contact email 
addresses, etc.


After watching their marketing video I learned the pronunciation of Arelion is 
not R-Lion but is actually A-Ray-Lee-On but I may continue thinking of it as 
R-Lion because it is shorter and it just sounds cooler in my head.

Re: home router battery backup

[Justin Streiner]

I'm one of the atypical users, when compared to the population at large,
but probably in line for this audience.

Critical gear is on a transfer switch and both inputs to that come from
UPSs that are on separate circuits. Less critical gear is fed from one UPS
or the other to balance the load and allow headroom for a load shift due to
a UPS failure.  My office gear is on a separate UPS on a different circuit.

Thank you
jms

On Wed, Jan 12, 2022, 13:01 Scott T Anderson via NANOG 
wrote:

> Hi NANOG mailing list,
>
>
>
> I am a graduate student, currently conducting research on how power
> outages affect home Internet users. I know that the FCC has a regulation
> since 2015 (47 CFR Section 9.20) requiring ISPs to provide an option to
> voice customers to purchase a battery backup for emergency voice services
> during power outages. As this is only an option and only applies to
> customers who subscribe to voice services, I was wondering if anyone had
> any insights on the prevalence of battery backup for home modem/routers?
> I.e., what percentage of home users actually install a battery backup in
> their home modem/router or use an external UPS?
>
>
>
> Thanks.
>
> Scott
>
>
>
> Reference for 47 CFR Section 9.20:
> https://www.ecfr.gov/current/title-47/chapter-I/subchapter-A/part-9/subpart-H/section-9.20
>
>
>

Re: WKBI #586, Redploying most of 127/8 as unicast public

[Justin Streiner]

The proposals I've seen all seem to deliver minimal benefit for the massive
lift (technical, administrative, political, etc) involved to keep IPv4
alive a little longer.

Makes about as much sense as trying to destabilize US currency by
counterfeiting pennies.

Thank you
jms



On Thu, Nov 18, 2021 at 12:39 PM Joe Maimon  wrote:

>
>
> John R. Levine wrote:
> >> The only effort involved on the IETF's jurisdiction was to stop
> >> squatting on 240/4 and perhaps maybe some other small pieces of IPv4
> >> that could possibly be better used elsewhere by others who may choose
> >> to do so.
> >
> > The IETF is not the Network Police, and all IETF standards are
> > entirely voluntary.
>
> And that is exactly why they said that even though they think it might
> possibly entail similar effort to deployment of IPv6 and that IPv6 is
> supposed to obsolete IPv4 before any such effort can be realized, they
> would be amenable to reclassifying 240/4 as anything other than
> reserved, removing that barrier from those whom may voluntarily decide
> to follow that updated standard, should they find the time to squeeze in
> another project the same size and effort of IPv6 into their spare time.
>
> Seems the IETF does indeed think it is the network police. And that they
> get to decide winners and losers.
> >
> > Nothing is keeping you from persuading people to change their software
> > to treat class E addresses as routable other than the detail that the
> > idea is silly.
> >
> > R's,
> > John
> >
>
> And indeed, they have done so. Now who looks silly?
>
> Joe
>
>

Re: Redploying most of 127/8 as unicast public

[Justin Keller]

I'd be fine if newish devices use it like a 1918 but I don't think
it's worth the headache and difficulty of making it globally routed.
Maybe  Amazon could use it too

On Wed, Nov 17, 2021 at 6:31 PM Jay R. Ashworth  wrote:
>
> This seems like a really bad idea to me; am I really the only one who noticed?
>
> https://www.ietf.org/id/draft-schoen-intarea-unicast-127-00.html
>
> That's over a week old and I don't see 3000 comments on it, so maybe it's just
> me.  So many things are just me.
>
> [ Hat tip to Lauren Weinstein, whom I stole it from ]
>
> Cheers,
> -- jra
>
> --
> Jay R. Ashworth  Baylink   
> j...@baylink.com
> Designer The Things I Think   RFC 2100
> Ashworth & Associates   http://www.bcp38.info
> St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274

What’s up with Comcast in Philadelphia area

[Justin Keller]

Hello.
Anyone know what's up with Comcast in the Philadelphia area? There
seems to be a lot of outages both residential and business

Justin

Re: S.Korea broadband firm sues Netflix after traffic surge

[Justin Streiner]

On Wed, Oct 20, 2021 at 3:41 PM Matthew Walster  wrote:
The user initiates the connection to the CDN. The user is paying for a
level of access to the internet via the BT network, with varying tiers of
speed at particular costs. They are advertised as "Unlimited broadband: With
no data caps or download limits, you can do as much as you like online." on
their website. Many CDNs bring the data closer to the customer, either
embedded within their network, or meeting at various PoPs/IXPs around the
country.

Seems pretty disingenuous to now say the called party has to pay as well,
in stark contrast to decades of precedent with their telephone product,
just because their customers are actually using what they were sold.

All in all, this raises an interesting question. Is British Telecom running
> their networks so hot, that just keeping the lights on requires capacity
> upgrades or are they just looking for freebies?


What happened is pretty clear, and not just for BT or SK.  Those providers,
as a business decision, built their business models around a certain level
of oversubscription that would strike a balance between customers not being
unhappy and squeezing as much headroom out of the network before upgrades
are required (beefier routers/switches, fatter pipes, more peering/transit,
etc).  That business model got upended when that acceptable level of
oversubscription changed.  Video streaming was the puddle of
gasoline/petrol on the floor, and the change in user traffic patterns was
the lit match.

By asking content providers to hand over money to those carriers in
exchange for (better) access to their customers, many of the ISPs could in
fact be triple-dipping because they already get revenue from their
customers and some also get various government subsidies to provide certain
types of service or services in certain areas.

Definitely doesn't pass the sniff test.

Thank you
jms

>

Re: Facebook post-mortems...

[Justin Keller]

Per o comments, the linked Facebook outage was from around 5/15/21

On Mon, Oct 4, 2021 at 9:08 PM Rubens Kuhl  wrote:
>
> The FB one seems to be from a previous event. Downtime doesn't match,
> visible flaw effects don't either.
>
>
> Rubens
>
>
> On Mon, Oct 4, 2021 at 9:59 PM  wrote:
> >
> > Fairly abstract - Facebook Engineering - 
> > https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A10158791436142200%7D=%2Fnotes%2Fnote%2F&_rdr
> >
> > Also, Cloudflare’s take on the outage - 
> > https://blog.cloudflare.com/october-2021-facebook-outage/
> >
> > FYI,
> > /John
> >

Re: IPv6 woes - RFC

[Justin Streiner]

On Sat, Sep 4, 2021, 22:49 John Levine  wrote:

> I have asked my ISP about IPv6 and their answer is that that they're not
> opposed to
> it but since I am the only person who has asked for it, it's quite low on
> the list
> of things to do.
>

Sounds like a consulting opportunity :)

Thank you
jms

>

Re: The great Netflix vpn debacle! (geofeeds)

[Justin Krejci]

Well apparently there are VPN applications that rely on fellow VPN users in a 
P2P fashion to share network connectivity. I guess it is like a commercialized 
version of Tor to some extent. Excluding any potential legal risks for illegal 
behavior tunneled through an unsuspecting fellow user, this has great potential 
to cause a contaminating spread of VPN flagged IP addresses, even with just 
normal usage.


One such VPN application is Hola VPN which also has a premium version using 
their VPN server gateways instead of or perhaps in addition to the community 
method.


Dynamic IP address assignments by an ISP could easily allow for one such user 
to get many IP addresses flagged as a VPN gateway. I have communicated with 
some IP reputation companies and they track VPN users and can even supply the 
specific VPN brand associated with certain IP addresses, with timestamps, they 
have observed and added to their reputation databases as VPN users. How they 
obtain their data I do not know for sure but I can think of a few ways.


So we seem to have a battle between

  *   users
  *   streaming content providers
  *   streaming content owners / copyright holders
  *   ISPs
  *   VPN providers
  *   restrictive/invasive governments or network operators
  *   ??

There is definitely collateral damage from their use that should be considered, 
especially if very prominent streaming content providers take a more 
restrictive posture towards users of these kinds of VPN services.




From: NANOG  on behalf of Haudy 
Kazemi via NANOG 
Sent: Wednesday, September 1, 2021 4:44 PM
To: Owen DeLong; nanog list
Cc: b...@theworld.com
Subject: Re: The great Netflix vpn debacle! (geofeeds)

Some TVs may also try to rescale the inputs, or enhance/process the image in 
ways that can improve perceived video quality. Things like increasing frame 
rates of sources that are lower frame rates (thus the 120 Hz and 240 Hz TVs 
that attempt to make 24, 30, and 60 FPS sources look better), or deinterlacing 
1080i ATSC sources.

Some of this image processing may not work well in specific monitor use cases.

I have had generally good results with using a TV as an HTPC monitor.  Only 
issues I've run into over the years are

1.) a 1080p Sony TV with a VGA input that could not handle 1920x1080 (using 
HDMI worked)
and
2.) a 720p Toshiba that could not show the BIOS screen of the attached computer 
(I think this was either an unsupported resolution issue, or a timing issue 
where the TV couldn't wake up fast enough from the 'signal lost' message to 
display a brand new signal input).

YMMV.


VPNs: there is a race going on between streaming services who want to block 
VPNs, and VPN services who have customers who want to be able to watch streams 
(whether in or out of their regions). Some VPN customers buy VPN services 
because they do not trust their ISP to not do stuff like selling browsing 
histories.

I think ISPs are getting caught in the middle, maybe when they have IP ranges 
near or in the middle of ranges that are suspected by IP reputation companies 
as being used by VPN services. I'd guess the problem is more likely to affect 
smaller ISPs, and not the Comcast/Cox/Charter/Spectrum/CenturyLinks of the 
world. There are also 'distributed VPN' services that let people share their 
connections with others.

We are also seeing fragmentation in the cable/streaming service space, similar 
to what happened in the cable/Dish Network/DirecTV wars. Add it all up, some 
customers may throw up their hands in annoyance at the various platforms and 
then revert to other means of obtaining the content they seek.



On Wed, Sep 1, 2021, 15:13 Owen DeLong via NANOG < 
nanog@nanog.org> wrote:


> On Sep 1, 2021, at 11:25 , b...@theworld.com wrote:
>
>
> Every time I've read a thread about using TVs for monitors several
> people who'd tried would say don't do it. I think the gist was that
> the image processors in the TVs would fuzz text or something like
> that. That it was usable but they were unhappy with their attempts, it
> was tiring on the eyes.

That was definitely true of 480 TVs and older 1080p units, but modern sets
are almost designed to be monitors first and everything else second.

> Maybe that's changed or maybe people happy with this don't do a lot of
> text? Or maybe there are settings involved they weren't aware of, or
> some TVs (other than superficial specs like 4K vs 720p) are better for
> this than others so some will say they're happy and others not so
> much?

There are some tradeoffs… For example, sitting normal computer monitor
distance from a 44” 4K screen, you can damn near see the individual pixels
and that can make text look fuzzy, especially if your GPU or OS are stupid
enough to use a technique called anti-aliasing on text (which is the most
probable source of the fuzziness in your originally quoted complaint).

Older TVs would try to 

Re: The great Netflix vpn debacle!

[Justin Krejci]

+1 on Bryan's message.


TL;DR

It seems lots of ISPs are struggling to figure out the why and the where of 
many IP addresses or blocks that are suddenly being blacklisted or flagged as 
VPNs or as out of service area.




I would really love to find, as Bryan said, if there is one particular IP 
reputation data provider who either got real aggressive recently or some 
(contaminated?) data was shared around. If there is I have no problem wading 
through their support processes to get it sorted but as it stands I just don't 
know who to call. It just has been very difficult to glean any actionable info 
and of course the normal support teams at the respective streaming providers 
mostly just are telling customers to call their ISP as if every random ISP 
has some special backdoor contact to every streaming provider where we can just 
get problems resolved quickly and easily while we all have a good laugh at 
people being able to watch their preferred movies and shows.


At least with email DNSBL filtering you usually get informed which DNSBL you 
are listed on and you can sort that out directly. In this case, the overall 
system of IP reputation based filtering seems still comparatively immature. The 
most I have gotten is after a very long phone call with someone at Hulu, they 
confirmed there is some issue affecting multiple networks and they are working 
on the issue and suggested I go through a whitelisting request process which 
may solve the problems but just for Hulu obviously.


I have published and tried to register our own geofeed data as defined in 
RFC8805 with as many IP geolocation providers as possible. I have checked 
around to as many IP geolocation and IP reputations sites as I can find and 
everything is either clean/accurate or there is no query method open to the 
public for troubleshooting that I can find. This is just yet another example to 
me of immaturity on dealing with geolocation problems: just spinning my wheels 
in the dark with mud spraying everywhere. There does not appear to be any 
consistency on handling issues by the content providers using IP geolocation 
and reputation to filter. If the content providers want to reject client 
connections they ought to provide more actionable information in their errors 
messages for ISPs since they are all just telling the users to call their ISPs. 
It just feels like a vicious circle.


So currently we are left with multiple video streaming providers that all 
started to flag many customers across many of our IP blocks all beginning 
earlier this month affecting customers, many of whom have been using the same 
IP address for years without issue until now. Do we try and decommission 
multiple IP subnets shuffle users over to new subnets and risk contaminating 
more subnets if this is an ongoing and regularly updated blacklist data set. 
This would further exacerbate the problem across yet more subnets that are 
getting scarcer. As a tangent, I am curious to see how IP geolocation and 
reputation systems are handling IPv6, I suppose they are just grouping larger 
and larger networks together into the same listings.


Someone who knows something concrete about this current issue, please throw us 
ISPs a bone.


With this email I feel like Leia recording a video plea for help addressed to 
Obi-Wan Kenobi help me Nanog Community... you're my only hope.





From: NANOG  on behalf of Bryan 
Holloway 
Sent: Friday, August 27, 2021 4:56 PM
To: Mike Hammett; John Alcock
Cc: nanog@nanog.org
Subject: Re: The great Netflix vpn debacle!

Is there some new DB that major CDNs are using?

We've been getting several reports of prefixes of ours being blocked,
claiming to be VPNs, even though we've been using those subnets without
incident for years.

HBO, Netflix, and Hulu appear to be common denominators. I have to
wonder if they're all siphoning misinformation off of some new DB
somewhere ...


On 8/14/21 1:45 AM, Mike Hammett wrote:
> https://thebrotherswisp.com/index.php/geo-and-vpn/
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions 
> 
> Midwest Internet Exchange 
> 
> The Brothers WISP 
> 
> 
> *From: *"John Alcock" 
> *To: *nanog@nanog.org
> *Sent: *Friday, August 13, 2021 2:11:16 PM
> *Subject: *The great Netflix vpn debacle!
>
> Well,
>
> It happened. I have multiple subscribers calling in. They can not access
> Netflix.
>
> Any 

Re: Any CloudFlare Rep?

[Justin Paine via NANOG]

Hi,

Replying off list.

<https://www.cloudflare.com/>

__
*Justin Paine*
He/Him/His
Threat Intel
101 Townsend St, San Francisco, CA 94107 <https://www.cloudflare.com/>

*PGP:* BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D
<https://keys.openpgp.org/vks/v1/by-fingerprint/BBAA6BCE33057FD66452711557B60114DE0B314D>


On Mon, Jul 19, 2021 at 8:39 AM Kushal R. via NANOG  wrote:

> Could someone from CloudFlare please contact me off the list? There is
> some crazy abuse going on one a site proxied through CF. Tried the usual
> twitter and abuse form. In the last 4 hours 2 people I know personally have
> lost $500+ each and hundreds are falling prey each day.
>
>
> —
> Kushal R.
> *Executive Management*
> <https://host4geeks.com/>
> WhatsApp: +1-(954)-737-4335 <+19547374335>
> Skype: kush.raha
>
> Host4Geeks LLC - Premium Managed Hosting <https://host4geeks.com>
> Trusted by over 10,000 Clients Globally
>
> <https://www.trustpilot.com/review/www.host4geeks.com>
> <https://h4g.co/TtM493>
>

Re: FreeBSD's ping Integrates IPv6

[Justin Streiner]

I think he meant that the underlying OS on lots of network gear is either
some variant of Linux or BSD.

Thank you
jms

On Sun, Jul 4, 2021, 11:40 Mark Tinka  wrote:

>
>
> On 7/4/21 17:15, Bjørn Mork wrote:
>
> > I seriously doubt that.  You're just not aware of it.
>
> I think I'd know if I've run "ping" on a box.
>
> Mark.
>

1950 Stemmons Meet me rooms?

[Justin Wilson (Lists)]

Who knows about the meet me rooms at 1950 Stemmons in Dallas? I need to get 
from the cologix meet me room to someone inside Equinix.  Our Equnix rep has 
been less than helpful. I was told

"We really don’t have a building meet me room there anymore since we bought the 
building.  Also, I don’t think we have connectivity to Cologix but I will check 
on this.”

Can anyone shed some light on this? Anyone on list that has some dark fiber 
between Cologix and Equinix? Replies off list are fine so I am not cluttering 
up the list.



Justin Wilson
j...@mtin.net

—
https://j2sw.com - All things jsw (AS209109)
https://blog.j2sw.com - Podcast and Blog

Re: Something that should put a smile on everybody's face today

[Justin Paine via NANOG]

Correction -- another one.
https://blog.cloudflare.com/winning-the-blackbird-battle/   :)

Here's an except from the new blog post:

offering $100,000 to be shared by the winners who are successful in finding
such prior art.

Please help!

<https://www.cloudflare.com/>

__
*Justin Paine*
He/Him/His
Head of Trust & Safety
101 Townsend St, San Francisco, CA 94107 <https://www.cloudflare.com/>

*PGP:* BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D
<https://keys.openpgp.org/vks/v1/by-fingerprint/BBAA6BCE33057FD66452711557B60114DE0B314D>


On Tue, Apr 27, 2021 at 3:26 PM Michael Thomas  wrote:

>
> And we can help! Cloudflare is setting out to destroy a patent troll:
>
>
> https://www.techdirt.com/articles/20210426/09454946684/patent-troll-sable-networks-apparently-needs-to-learn-lesson-cloudflare-wants-to-destroy-another-troll
>
> Mike
>
>

Re: login.authorize.net has A and CNAME records

[Justin Paine via NANOG]

For the thread -- we're aware and looking into this.  n...@cloudflare.com
being the best place to report these kinds of things.

<https://www.cloudflare.com/>

__
*Justin Paine*
He/Him/His
Head of Trust & Safety
101 Townsend St, San Francisco, CA 94107 <https://www.cloudflare.com/>

*PGP:* BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D
<https://keys.openpgp.org/vks/v1/by-fingerprint/BBAA6BCE33057FD66452711557B60114DE0B314D>


On Tue, Apr 6, 2021 at 2:49 PM Mark Andrews  wrote:

>
>
> > On 7 Apr 2021, at 05:59, Arne Jensen  wrote:
> >
> >
> > Den 06-04-2021 kl. 21:47 skrev Seth Mattinen:
> >>
> >>>
> >>> What kind of local problem or network problems could cause a servfail
> >>> response from the authoritative ns?
> >>
> >>
> >>
> >> I'm beginning to think this is a DNSSEC related problem, I'll ask on
> >> the pdns-users list. I see it's asking for a DS record on
> >> login.authorize.net.cdn.cloudflare.net when the nearest one appears to
> >> be at cloudflare.net, so for some reason that's not being applied all
> >> the way down.
> >
> > I do somehow take that "local problem" part back again, which also
> > wasn't intended exactly in the way that it was written:
> >
> > ->
> >
> https://dnssec-analyzer.verisignlabs.com/login.authorize.net.cdn.cloudflare.net
> >
> > Is looking at login.authorize.net.cdn.cloudflare.net/DNSKEY, but failing
> > due to the SERVFAIL.
> >
> > -> https://dnsviz.net/d/login.authorize.net.cdn.cloudflare.net/dnssec/
> >
> > Seems to claim that it works just fine.
> >
> > Asking login.authorize.net.cdn.cloudflare.net/DNSKEY or
> > login.authorize.net.cdn.cloudflare.net/DS returns SERVFAIL here too.
> >
> >
> > But I don't think you should be querying /DNSKEY or /DS, except a the
> > (current) delegation's root, e.g. as you say yourself, at
> > "cloudflare.net" in this case.
>
> It shouldn’t matter if you query for them.  If the records don’t exist then
> you should get back NOERROR/NODATA responses with NSEC/NSEC3 records to
> prove
> those responses.
>
> Note the server claims that TXT records exist at
> login.authorize.net.cdn.cloudflare.net
> but can’t return them.
>
>
> % dig login.authorize.net.cdn.cloudflare.net type65 @198.41.222.31 +dnssec
>
> ; <<>> DiG 9.15.4 <<>> login.authorize.net.cdn.cloudflare.net type65 @
> 198.41.222.31 +dnssec
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1641
> ;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1
> ;; WARNING: recursion requested but not available
>
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags: do; udp: 1232
> ;; QUESTION SECTION:
> ;login.authorize.net.cdn.cloudflare.net.IN TYPE65
>
> ;; AUTHORITY SECTION:
> cloudflare.net. 5   IN  SOA ns1.cloudflare.net.
> dns.cloudflare.com. 1617743605 1 2400 604800 5
> login.authorize.net.cdn.cloudflare.net. 5 IN NSEC \
> 000.login.authorize.net.cdn.cloudflare.net. A HINFO MX TXT  LOC SRV
> NAPTR CERT SSHFP RRSIG NSEC TLSA SMIMEA HIP OPENPGPKEY TYPE64 SPF URI CAA
> cloudflare.net. 5   IN  RRSIG   SOA 13 2 5 20210407221325
> 20210405201325 34505 cloudflare.net.
> BfBNcB9zG3T6d7mu5okde144g0OlxBazynPBD78o/ig5y0JHWo+L2ufu
> mhSfOquAkq6lqa/V+3yySMERlQKcIQ==
> login.authorize.net.cdn.cloudflare.net. 5 IN RRSIG NSEC 13 6 5
> 20210407221325 20210405201325 34505 cloudflare.net.
> +shgKZcdkQZvH9ZFEZvdXyHe7+FkX1mCit9xe4V7A+uEEYi3L7vnf16x
> Wyvzs0o4TlQiOJlYBG4vEkKE3d8NwQ==
>
> ;; Query time: 17 msec
> ;; SERVER: 198.41.222.31#53(198.41.222.31)
> ;; WHEN: Wed Apr 07 07:13:25 AEST 2021
> ;; MSG SIZE  rcvd: 417
>
> %
>
> % dig login.authorize.net.cdn.cloudflare.net txt @198.41.222.31 +dnssec
>
> ; <<>> DiG 9.15.4 <<>> login.authorize.net.cdn.cloudflare.net txt @
> 198.41.222.31 +dnssec
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 46557
> ;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
> ;; WARNING: recursion requested but not available
>
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags: do; udp: 1232
> ;; QUESTION SECTION:
> ;login.authorize.net.cdn.cloudflare.net.IN TXT
>
> ;; Query time: 15 msec
> ;; SERVER: 198.41.222.31#53(198.41.222.31)
> ;; WHEN: Wed Apr 07 07:14:22 AEST 2021
> ;; MSG SIZE  rcvd: 67
>
> %
>
> > Or if "cdn.cloudflare.net" had been a sub-delegation, then at that
> point...
> >
> > --
> > Med venlig hilsen / Kind regards,
> > Arne Jensen
> >
> >
>
> --
> Mark Andrews, ISC
> 1 Seymour St., Dundas Valley, NSW 2117, Australia
> PHONE: +61 2 9871 4742  INTERNET: ma...@isc.org
>
>

Re: Microsoft problems...

[Justin Streiner]

Can you be a bit more specific regarding what you're seeing or not seeing?

Are you reaching MS through IP transit/peer connections, or are you having
issues reaching MS cloud services over ExpressRoute circuits?

Thank you
jms

On Mon, Mar 15, 2021 at 4:04 PM  wrote:

> Anyone else noticing major MAJOR problems with various MS services?
>
> Geoff
>
>

Ip space Dilemma

[Justin Wilson (Lists)]

Folks,
We have an IP block I have asked about help on a few times on here.  
This is a block we received from ARIN in June of 2020.  We have several state 
networks here in Indiana dropping this traffic at their firewalls. I have been 
working with them since we discovered this issue in September.  I am not 
getting anywhere with them and was finally told we were not a priority.

I am at the point I need to give the space back because it is unusable 
to the ISP customers. Does anyone have any creative ideas on how to fix this? 



Justin Wilson
j...@mtin.net

—
https://j2sw.com - All things jsw (AS209109)
https://blog.j2sw.com - Podcast and Blog

Re: Is there an established method for reporting/getting removed a company with 100% false peeringdb entries?

[Justin Wilson (Lists)]

I see from peering db:  2020-07-01T14:22:01Z
According to the bg.he.net link
AS18894 has not been visible in the global routing table since November 28, 2020
The information displayed is from that time.


Are they causing you or someone issues Eric? Maybe they went out of business? 
Many businesses don’t worry about peering db entries. Looks like the website 
has been under constructions since 2020.

Sounds to me like they made a splash, and faltered.  


Justin Wilson
j...@mtin.net

—
https://j2sw.com - All things jsw (AS209109)
https://blog.j2sw.com - Podcast and Blog

> On Mar 4, 2021, at 7:14 PM, Eric Kuhnke  wrote:
> 
> First, take a look at this:
> 
> https://www.peeringdb.com/asn/18894
> 
> 
> Now look at these (or use your own BGP table analysis tools):
> 
> https://bgp.he.net/AS18894
> 
> https://stat.ripe.net/18894
> 
> The claimed prefixes announced, traffic levels and POPs appear to have no 
> correlation with reality in global v4/v6 BGP tables.
> 
> It is also noteworthy that I have inquired with a number of persons I know 
> who are active in network engineering in NYC, and nobody has ever encountered 
> this company.
> 
> 
> 
> 

Re: Famous operational issues

[Justin Streiner]

An interesting sub-thread to this could be:

Have you ever unintentionally crashed a device by running a perfectly
innocuous command?
1. Crashed a 6500/Sup2 by typing "show ip dhcp binding".
2. "clear interface XXX" on a Nexus 7K triggered a cascading/undocument
Sev1 bug that caused two linecards to crash and reload, and take down about
two dozen buildings on campus at the .edu where I used to work.
3. For those that ever had the misfortune of using early versions of the
"bcc" command shell* on Bay Networks routers, which was intended to make
the CLI make look and feel more like a Cisco router, you have my
condolences.  One would reasonably expect "delete ?" to respond with a list
of valid arguments for that command.  Instead, it deleted, well...
everything, and prompted an on-site restore/reboot.

BCC originally stood for "Bay Command Console", but we joked that it really
stood for "Blatant Cisco Clone".

On Tue, Feb 16, 2021 at 2:37 PM John Kristoff  wrote:

> Friends,
>
> I'd like to start a thread about the most famous and widespread Internet
> operational issues, outages or implementation incompatibilities you
> have seen.
>
> Which examples would make up your top three?
>
> To get things started, I'd suggest the AS 7007 event is perhaps  the
> most notorious and likely to top many lists including mine.  So if
> that is one for you I'm asking for just two more.
>
> I'm particularly interested in this as the first step in developing a
> future NANOG session.  I'd be particularly interested in any issues
> that also identify key individuals that might still be around and
> interested in participating in a retrospective.  I already have someone
> that is willing to talk about AS 7007, which shouldn't be hard to guess
> who.
>
> Thanks in advance for your suggestions,
>
> John
>

Re: Famous operational issues

[Justin Streiner]

Beyond the widespread outages, I have so many personal war stories that
it's hard to pick a favorite.

My first job out of college in the mid-late 90s was at an ISP in Pittsburgh
that I joined pretty early in its existence, and everyone did a bit of
everything. I was hired to do sysadmin stuff, networking, pretty much
whatever was needed. About a year after I started, we brought up a new mail
system with an external RAID enclosure for the mail store itself.  One day,
we saw indications that one of the disks in the RAID enclosure was starting
to fail, so I scheduled a maintenance window to replace the disk and let
the controller rebuild the data and integrate it back into the RAID set.
No big worries, right?

It's Tuesday at about 2 AM.

Well, the kernel on the RAID controller itself decided that when I pulled
the failing drive would be a fine time to panic, and more or less turn
itself into a bit-blender, and take all the mailstore down with it.  After
a few hours of watching fsck make no progress on anything, in terms of
trying to un-fsck the mailstore, we made the decision in consultation with
the CEO to pull the plug on trying to bring the old RAID enclosure back to
life, and focus on finding suitable replacement hardware and rebuild from
scratch.  We also discovered that the most recent backups of the mailstore
were over a month old :(

I think our CEO ended up driving several hours to procure a suitable
enclosure.  By the time we got the enclosure installed, filesystems built,
and got whatever tape backups we had restored, and tested the integrity of
the system, it was now Thursday around 8 AM. Coincidentally, that was the
same day the company hosted a big VIP gathering (the mayor was there, along
with lots of investors and other bigwigs), so I had to come back and put on
a suit to hobnob with the VIPs after getting a total of 6 hours of sleep in
about the previous 3 days.  I still don't know how I got home that night
without wrapping my vehicle around a utility pole (due to being over-tired,
not due to alcohol).

Many painful lessons learned over that stretch of days, as often the case
as a company grows from startup mode and builds more robust technology and
business processes as a consequence of growth.

jms

On Tue, Feb 16, 2021 at 2:37 PM John Kristoff  wrote:

> Friends,
>
> I'd like to start a thread about the most famous and widespread Internet
> operational issues, outages or implementation incompatibilities you
> have seen.
>
> Which examples would make up your top three?
>
> To get things started, I'd suggest the AS 7007 event is perhaps  the
> most notorious and likely to top many lists including mine.  So if
> that is one for you I'm asking for just two more.
>
> I'm particularly interested in this as the first step in developing a
> future NANOG session.  I'd be particularly interested in any issues
> that also identify key individuals that might still be around and
> interested in participating in a retrospective.  I already have someone
> that is willing to talk about AS 7007, which shouldn't be hard to guess
> who.
>
> Thanks in advance for your suggestions,
>
> John
>

Re: Famous operational issues

[Justin Streiner]

On Thu, Feb 18, 2021 at 5:38 PM Warren Kumari  wrote:

>
> 2: A somewhat similar thing would happen with the Ascend TNT Max, which
> had side-to-side airflow. These were dial termination boxes, and so people
> would install racks and racks of them. The first one would draw in cool air
> on the left, heat it up and ship it out the right. The next one over would
> draw in warm air on the left, heat it up further, and ship it out the
> right... Somewhere there is a fairly famous photo of a rack of TNT Maxes,
> with the final one literally on fire, and still passing packets.
>

We had several racks of TNTs at the peak of our dial POP phase, and I
believe we ended up designing baffles for the sides of those racks to pull
in cool air from the front of the rack to the left side of the chassis and
exhaust it out the back from the right side.  It wasn't perfect, but it did
the job.

The TNTs with channelized T3 interfaces were a great way to terminate lots
of modems in a reasonable amount of rack space with minimal cabling.

Thank you
jms

Re: Famous operational issues

[Justin Wilson (Lists)]

I remember when the big carriers de-peered with Cogent in the early 2000s.  The 
underestimated the amount of web-sites being hosted by people using cogent 
exclusively. 


Justin Wilson
j...@j2sw.com

—
https://j2sw.com - All things jsw (AS209109)
https://blog.j2sw.com - Podcast and Blog

> On Feb 17, 2021, at 10:29 AM, Miles Fidelman  
> wrote:
> 
> John Kristoff wrote:
>> Friends,
>> 
>> I'd like to start a thread about the most famous and widespread Internet
>> operational issues, outages or implementation incompatibilities you
>> have seen.
>> 
> Well... pre-Internet, but the great Northeast fiber cut comes to mind 
> (backhoe vs. fiber, backhoe won).
> 
> Miles Fidelman
> 
> -- 
> In theory, there is no difference between theory and practice.
> In practice, there is.   Yogi Berra
> 
> Theory is when you know everything but nothing works. 
> Practice is when everything works but no one knows why. 
> In our lab, theory and practice are combined: 
> nothing works and no one knows why.  ... unknown

Re: Famous operational issues

[Justin Streiner]

Would this also extend to intentional actions that may have had unintended
consequences, such as provider A intentionally de-peering provider B, or
the monopoly telco for $country cutting itself off from the rest of the
global Internet for various reasons (technical, political, or otherwise)?

That said, I'd still have to stick with AS7007, the Baltimore tunnel fire,
and 9/11 as the most prominent examples of widespread issues/outages and
how those issues were addressed.

Honorable mention: $vendor BGP bugs, either due to $vendor ignoring the
relevant RFCs, implementing them incorrectly, or an outage exposed a design
flaw that the RFCs didn't catch.  Too many of those to list here :)

jms

On Tue, Feb 16, 2021 at 2:37 PM John Kristoff  wrote:

> Friends,
>
> I'd like to start a thread about the most famous and widespread Internet
> operational issues, outages or implementation incompatibilities you
> have seen.
>
> Which examples would make up your top three?
>
> To get things started, I'd suggest the AS 7007 event is perhaps  the
> most notorious and likely to top many lists including mine.  So if
> that is one for you I'm asking for just two more.
>
> I'm particularly interested in this as the first step in developing a
> future NANOG session.  I'd be particularly interested in any issues
> that also identify key individuals that might still be around and
> interested in participating in a retrospective.  I already have someone
> that is willing to talk about AS 7007, which shouldn't be hard to guess
> who.
>
> Thanks in advance for your suggestions,
>
> John
>

Re: Problems with newish IP block assignment issues from ARIN

[Justin Wilson (Lists)]

I enabled 134.195.47.1 on one of our routers.

Justin Wilson
j...@mtin.net

—
https://j2sw.com - All things jsw (AS209109)
https://blog.j2sw.com - Podcast and Blog

> On Feb 8, 2021, at 3:46 PM, Job Snijders via NANOG  wrote:
> 
> Dear Justin,
> 
> On Mon, Feb 08, 2021 at 03:14:47PM -0500, Justin Wilson (Lists) wrote:
>> It acts like the IP block was blacklisted at some point and got on
>> some bad lists but I don’t want ti limit myself to that theory.
>> I have opened up a ticket with ARIN asking for any guidance. Has
>> anyone ran into this with new space assigned? Any tools, sites, etc. I
>> can use to do further troubleshooting.  
> 
> Here are some useful tools:
> 
>ping.pe
>example: http://ping.pe/www.openbsd.org
> 
>https://ring.nlnog.net/
>good introduction here: 
> https://labs.ripe.net/Members/martin_pels_3/10-years-of-nlnog-ring
> 
>https://atlas.ripe.net/
> 
>> The block in question is 134.195.44.0/22. 
> 
> Is there any specific IP address in the range that should always respond
> to ICMP Echo Requests? This will help others see if they can reach you
> or not.
> 
>> It has been RPKI certified and has IRR entries.
> 
> Indeed, nice :-) http://irrexplorer.nlnog.net/search/134.195.44.0/22
> 
> Kind regards,
> 
> Job
> 

Problems with newish IP block assignment issues from ARIN

[Justin Wilson (Lists)]

Folks,
Have a gremlin we have been chasing around for several months now and it’s 
becoming a major issue as we are getting tighter on IPV4 and needing to give 
some provider assigned space back.

In June we received a /22 from ARIN.  As is my workflow I started announcing it 
but waited a month while I checked out the geolocation databases for correct 
info, did testing ,etc. All this time our test accounts could browse web-sites, 
etc. 

We put one of the pools into production and things ran good for awhile.  Then 
we started getting the occasional web-site was not working.  After several of 
these we started assigning the customer an IP out of one of our other ARIN 
blocks and the web-site would be fine and reachable. The issue seems to reside 
just on this /22.  We have other blocks from ARIN and they are just fine.  We 
can assign an IP out of this new block and can’t reach certain web-sites.  We 
turn around and assign out of another block and web-site works just fine.

We have two upstreams and an IX on this network.  We have tried withdrawing the 
route on this particular /22 and isolating to one upstream alone and the 
problems still persist. 

Many of the web-sites in question are government (both state and local), online 
universities, and the occasional local news station.  They are diverse enough 
to not be traced down to a common point, except the IP block.  

We announce the IP block via BGP the same exact way we announce the other 
blocks. Traceroutes show the path going the same way no matter what IP block 
the customer has.

It acts like the IP block was blacklisted at some point and got on some bad 
lists but I don’t want ti limit myself to that theory.  I have opened up a 
ticket with ARIN asking for any guidance.  Has anyone ran into this with new 
space assigned? Any tools, sites, etc. I can use to do further troubleshooting. 
 The IP block does not appear to have any blacklisted IPs according to MX 
toolbox, and some others.

The block in question is 134.195.44.0/22.  It has been RPKI certified and has 
IRR entries.

Thanks in advance


Justin Wilson
j...@mtin.net

—
https://j2sw.com - All things jsw (AS209109)
https://blog.j2sw.com - Podcast and Blog

AWS contact?

[Justin Wilson (Lists)]

What is the best avenue for contacting support for AWS? I have several 
ISPs experiencing reachability issues with AWS hosted sites.  These are from 
different backbones, different gear, etc.  The common denominator is AWS. 

Been googling around and can’t seem to find a contact.



Justin Wilson
j...@mtin.net

—
https://j2sw.com - All things jsw (AS209109)
https://blog.j2sw.com - Podcast and Blog

Sonicwall GEoIP Database

[Justin Wilson (Lists)]

Does anyone know what GEoIP database sonic wall uses? Their tech 
support has been horrid.  We are not a customer but getting customers who are 
getting blocked by some sonic walls due to “unknot” country for GeoIP.  I have 
checked the ips against the database providers listed at: 
https://thebrotherswisp.com/index.php/geo-and-vpn/ 
<https://thebrotherswisp.com/index.php/geo-and-vpn/>

All checkout okay so looking for what SOnicWall uses.



Justin Wilson
j...@mtin.net

—
https://j2sw.com - All things jsw (AS209109)
https://blog.j2sw.com - Podcast and Blog

Re: Juniper configuration recommendations/BCP

[Justin Oeder]

If you are an OSPF shop, Cisco AD is 110 for internal and external
routes.  Juniper is 10 for internal and 150 for external.  This can be
changed via an export (maybe import) policy on the OSPF protocol.

There is no 'network' statement in the Junos world.  There are a few
different ways to solve this same problem.  Up to you how you do it.

Routing engine protection is much easier.  A firewall filter on the
loopback interface.  Here is a sample.  This is really where your BCP
starts.  
https://github.com/jcoeder/juniper-configurations/blob/master/protect-re.txt

Dynamic prefix-lists are pretty cool.  They allow you to create prefix-
list based on other sections of the configuration.

# In this first statement we use wildcards surrounding a . as this is
the format of an IPv4 address.
set policy-options prefix-list BGP_PEERS_DYNAMIC apply-path "protocols
bgp group <*> neighbor <*.*>"

# In this second statement we use wildcards surrounding a : as this is
the format of an IPv6 address.
set policy-options prefix-list BGP_PEERS_DYNAMIC_V6 apply-path
"protocols bgp group <*> neighbor <*:*>"

Justin

On Thu, 2020-10-08 at 03:37 -0600, Forrest Christian (List Account)
wrote:
> 
> After nearly 30 years of being a cisco shop, I'm working on
> configuring our first pair of Juniper MX204's to replace our current
> provider-edge cisco. 
> 
> I've worked through enough of the Juniper documentation/books to have
> a fairly good handle on how to configure these, but I wanted to check
> with the list to see if there are any Juniper-Specific gotchas I
> might run into that isn't documented well.  
> 
> I've done a bit of googling and am either finding stuff that is
> largely Cisco-specific or which is generic - all of which I'm
> rather familiar with based on my past history.   Is there anything I
> should worry about which is Juniper-specific?
> 
> -- 
> - Forrest

Re: Florida: Voter registration website overwhelmed at deadline

[Justin Paine via NANOG]

no indication of a DoS attack.

<https://www.cloudflare.com/>

__
*Justin Paine*
He/Him/His
Head of Trust & Safety
101 Townsend St, San Francisco, CA 94107 <https://www.cloudflare.com/>

*PGP:* BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D
<https://keys.openpgp.org/vks/v1/by-fingerprint/BBAA6BCE33057FD66452711557B60114DE0B314D>


On Tue, Oct 6, 2020 at 9:51 AM Sean Donelan  wrote:

>
> Every election has problems. Most of the time, those problems aren't
> noticed. Elections rely on a lot of back-end infrastructure, besides the
> actual voting itself.
>
> It could be a DDOS attack, or simply duct-taped systems having trouble
> with the load.
>
> Voting early (mail, drop-off, in-person) means more time to fix glitches.
>
>
>
>
> https://apnews.com/article/virus-outbreak-election-2020-florida-elections-ron-desantis-dc8aaf2213b6c50451019a7c0c07c3f7
>
> The FBI and the Cybersecurity and Infrastructure Security Agency warned
> elections officials nationwide last week that cyberattacks could disrupt
> their systems during the run-up to the election. They particularly noted
> “distributed denial-of-service” attacks, which inundate a computer system
> with requests, potentially clogging up servers until the system becomes
> inaccessible to legitimate users.
>

Re: IPv4 Mismanagement

[Justin Streiner]

It is a thankless task, but something that becomes increasingly important
as $provider starts to run low on IPv4 space to assign to customers.

Thank you
jms

On Mon, Oct 5, 2020, 20:19 Tom Hill  wrote:

> On 04/10/2020 02:17, Wayne Bouchard wrote:
> > Groups that have such things I can only presume do not do a good job
> > of periodically going through and auditing their IP allocations or, if
> > they do, then they don't do a good enough job of cleaning up all the
> > details.
>
> It is a long-winded, laborious, thankless task (well, mostly thankless)
> and we should be writing software to do it for us. Of course, we all
> know how bad everyone is at that, ergo it isn't often done.
>
> On the other hand, perhaps these ISPs are worried that they might be
> audited by an RIR?
>
> --
> Tom
>

Re: IPv4 Mismanagement

[Justin Streiner]

I suspect many providers don't have good business processes for reclaiming
IP space that was assigned to customers who have either disconnected or
voluntarily returned the space.

The provider I started out with in the mid/late 90s bootstrapped itself
with IP space from MCI (now, CenturyLink... I think?) and UUNET (now
Verizon Business), but we handed those blocks back when we started getting
provider-independent space from ARIN.  No idea what became of that space
after we stopped announcing it.

jms

On Fri, Oct 2, 2020 at 3:38 PM Ryan Wilkins  wrote:

> I have the same thing with a service that was disconnected a couple years
> ago.  Four IP blocks of /24 size are still swipped to us and we’re
> announcing them.  I don’t put any customers on them and just use them for
> temporary things for fear that some day someone will want them back.
>
> On Oct 2, 2020, at 2:50 PM, Matt Brennan  wrote:
>
>
> A service I disconnected more than 2 years ago still has a /24 of their
> space SWIPED to me. Their NOC closed the ticket I opened to remove. Unknown
> if it's actually in use for another customer.
>
> I also had a conversation last week with another ISP (we were
> renegotiating our contract) about this. The order form they sent me had
> multiple /28's we had "given back" years ago still listed. Turns out
> they're still being routed to us as well.
>
> I would bet it happens all over the place.
>
> -Matt
>
> On Fri, Oct 2, 2020 at 2:00 PM Matt Hoppes <
> mattli...@rivervalleyinternet.net> wrote:
>
>> I'm sitting here in the office on a Friday performing some IP
>> maintenance and I see that one of our upstreams is still filtering an IP
>> range we haven't used in years.   I dig into it a bit more and it turns
>> out a major carrier still has them SWIPed to us.
>>
>> This got me curious and I dug more into IPs from back in our early days
>> and discovered there are two Tier-1 carriers we no longer do business
>> with that still have large blocks of their own IPs SWIPED and allocated
>> to us.
>>
>> This is really confusing and concerning.   I know it's not the
>> end-all-be-all, but I wonder how much IPv4 exhaustion is being caused by
>> this type of IPv4 mis-management, where IPs are still shown as
>> "allocated" to a customer who hasn't used them in years.
>>
>> I've seen this behavior from Frontier and CenturyLink to name just a few.
>>
>> Any thoughts on this?
>>
>
>

Re: Gaming Consoles and IPv4

[Justin Wilson (Lists)]

It is coming back to that, but you still have so much going on that you need 
the open ports.  I don’t gt why people fight IPV6 so much.  


Justin Wilson
j...@mtin.net

—
https://j2sw.com - All things jsw (AS209109)
https://blog.j2sw.com - Podcast and Blog

> On Sep 28, 2020, at 8:34 AM, Mike Hammett  wrote:
> 
> Why stray away from how PC games were 20 years ago where there was a 
> dedicated server and clients just spoke to servers?
> 
> 
> 
> -
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
>  <https://www.facebook.com/ICSIL> 
> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> 
> <https://www.linkedin.com/company/intelligent-computing-solutions> 
> <https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
>  <https://www.facebook.com/mdwestix> 
> <https://www.linkedin.com/company/midwest-internet-exchange> 
> <https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
>  <https://www.facebook.com/thebrotherswisp> 
> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> From: "Justin Wilson (Lists)" mailto:li...@mtin.net>>
> To: "North American Network Operators' Group"  <mailto:nanog@nanog.org>>
> Sent: Monday, September 28, 2020 7:22:28 AM
> Subject: Re: Gaming Consoles and IPv4
> 
> There are many things going on with gaming that makes natted IPv4 an issue 
> when it comes to consoles and gaming in general.   When you break it down it 
> makes sense.
> 
> -You have voice chat
> -You are receiving data from servers about other people in the game
> -You are sending data to servers about yourself
> -If you are using certain features where you are “the host” then you are 
> serving content from your gaming console.  This is not much different than a 
> customer running a web server.  You can’t have more than one customer running 
> a port 80 web-server behind nat.
> -Streaming to services like Twitch or YouTube
> 
> All of these take up standard, agreed upon ports. It’s really only prevalent 
> on gaming consoles because they are doing many functions.  Look at it another 
> way.  You have a customer doing the following.
> 
> -Making a VOIP call
> -Streaming a movie
> -Running a web server
> -Running bittorrent on a single port
> -Having a camera folks need to access from the outside world
> 
> This is why platforms like Xbox developed things like Teredo.
> 
> Justin Wilson
> j...@mtin.net <mailto:j...@mtin.net>
> 
> —
> https://j2sw.com <https://j2sw.com/> - All things jsw (AS209109)
> https://blog.j2sw.com <https://blog.j2sw.com/> - Podcast and Blog
> 
> On Sep 27, 2020, at 9:33 PM, Daniel Sterling  <mailto:sterling.dan...@gmail.com>> wrote:
> 
> Matt Hoppes raises an interesting question,
> 
> At the risk of this being off-topic, in the latest call of duty games I've 
> played, their UDP-NAT-breaking algorithm seems to work rather well and should 
> function fine even behind CGNAT. Ironically turning on upnp makes this 
> *worse*, because when their algorithm probes to see what ports to use, upnp 
> sends all traffic from the "magical xbox port" to one box instead of letting 
> NAT control the ports. This does cause problems when multiple xboxes are 
> behind one NAT doing upnp. If upnp is on and both xboxes are fully powered 
> off and then turned on one at a time, things do work. But when upnp is off 
> everything works w/o having to do that.
> 
> There are many other games and many CPE NAT boxes that may do horrible 
> things, but CGNAT by itself shouldn't cause problems for any recent device / 
> gaming system.
> 
> It is true that I've yet to see any FPS game use ipv6. I assume that's cuz 
> they can't count on users having v6, so they have to support v4, and it 
> wouldn't be worth their while to have their gaming host support dual-stack. 
> just a guess there
> 
> -- Dan
> 
> 
> 
> On Sun, Sep 27, 2020 at 7:29 PM Mike Hammett  <mailto:na...@ics-il.net>> wrote:
> Actually, uPNP is the only way to get two devices to work behind one public 
> IP, at least with XBox 360s. I haven't kept up in that realm.
> 
> 
> 
> -
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
>  <https://www.facebook.com/ICSIL> 
> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> 
> <https://www.linkedin.com/company/intelligent-computing-solutions> 
> <https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
>  <https://www.facebook.com/mdwestix> 
> <https://ww

Re: Gaming Consoles and IPv4

[Justin Wilson (Lists)]

There are many things going on with gaming that makes natted IPv4 an issue when 
it comes to consoles and gaming in general.   When you break it down it makes 
sense.

-You have voice chat
-You are receiving data from servers about other people in the game
-You are sending data to servers about yourself
-If you are using certain features where you are “the host” then you are 
serving content from your gaming console.  This is not much different than a 
customer running a web server.  You can’t have more than one customer running a 
port 80 web-server behind nat.
-Streaming to services like Twitch or YouTube

All of these take up standard, agreed upon ports. It’s really only prevalent on 
gaming consoles because they are doing many functions.  Look at it another way. 
 You have a customer doing the following.

-Making a VOIP call
-Streaming a movie
-Running a web server
-Running bittorrent on a single port
-Having a camera folks need to access from the outside world

This is why platforms like Xbox developed things like Teredo.

Justin Wilson
j...@mtin.net

—
https://j2sw.com - All things jsw (AS209109)
https://blog.j2sw.com - Podcast and Blog

> On Sep 27, 2020, at 9:33 PM, Daniel Sterling  
> wrote:
> 
> Matt Hoppes raises an interesting question,
> 
> At the risk of this being off-topic, in the latest call of duty games I've 
> played, their UDP-NAT-breaking algorithm seems to work rather well and should 
> function fine even behind CGNAT. Ironically turning on upnp makes this 
> *worse*, because when their algorithm probes to see what ports to use, upnp 
> sends all traffic from the "magical xbox port" to one box instead of letting 
> NAT control the ports. This does cause problems when multiple xboxes are 
> behind one NAT doing upnp. If upnp is on and both xboxes are fully powered 
> off and then turned on one at a time, things do work. But when upnp is off 
> everything works w/o having to do that.
> 
> There are many other games and many CPE NAT boxes that may do horrible 
> things, but CGNAT by itself shouldn't cause problems for any recent device / 
> gaming system.
> 
> It is true that I've yet to see any FPS game use ipv6. I assume that's cuz 
> they can't count on users having v6, so they have to support v4, and it 
> wouldn't be worth their while to have their gaming host support dual-stack. 
> just a guess there
> 
> -- Dan
> 
> 
> 
> On Sun, Sep 27, 2020 at 7:29 PM Mike Hammett  <mailto:na...@ics-il.net>> wrote:
> Actually, uPNP is the only way to get two devices to work behind one public 
> IP, at least with XBox 360s. I haven't kept up in that realm.
> 
> 
> 
> -
> Mike Hammett
> Intelligent Computing Solutions <http://www.ics-il.com/>
>  <https://www.facebook.com/ICSIL> 
> <https://plus.google.com/+IntelligentComputingSolutionsDeKalb> 
> <https://www.linkedin.com/company/intelligent-computing-solutions> 
> <https://twitter.com/ICSIL>
> Midwest Internet Exchange <http://www.midwest-ix.com/>
>  <https://www.facebook.com/mdwestix> 
> <https://www.linkedin.com/company/midwest-internet-exchange> 
> <https://twitter.com/mdwestix>
> The Brothers WISP <http://www.thebrotherswisp.com/>
>  <https://www.facebook.com/thebrotherswisp> 
> <https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
> From: "Matt Hoppes"  <mailto:mattli...@rivervalleyinternet.net>>
> To: "Darin Steffl" mailto:darin.ste...@mnwifi.com>>
> Cc: "North American Network Operators' Group"  <mailto:nanog@nanog.org>>
> Sent: Sunday, September 27, 2020 1:22:51 PM
> Subject: Re: Gaming Consoles and IPv4
> 
> I understand that. But there’s a host of reasons why that night not work - 
> two devices trying to use UPNP behind the same PAT device, an apartment 
> complex or hotel WiFi system, etc. 
> 
> On Sep 27, 2020, at 2:17 PM, Darin Steffl  <mailto:darin.ste...@mnwifi.com>> wrote:
> 
> 
> This isn't rocket science.
> 
> Give each customer their own ipv4 IP address and turn on upnp, then they will 
> have open NAT to play their game and host. 
> 
> On Sun, Sep 27, 2020, 12:50 PM Matt Hoppes  <mailto:mattli...@rivervalleyinternet.net>> wrote:
> I know the solution is always “IPv6”, but I’m curious if anyone here knows 
> why gaming consoles are so stupid when it comes to IPv4?  
> 
> We have VoIP and video systems that work fine through multiple layers of PAT 
> and NAT. Why do we still have gaming consoles, in 2020, that can’t find their 
> way through a PAT system with STUN or other methods?
> 
> It seems like this should be a simple solution, why are we still opening 
> ports or having systems that don’t work?
> 

Re: cloudflare 1.1.1.2 filtered DNS

[Justin Paine via NANOG]

Hi Bill,

Report it via the form you mentioned and the team will review it shortly.
We don't currently publish our data sources for the filtered service.

Thanks,
Justin

<https://www.cloudflare.com/>

_
*Justin Paine*
He/Him/His
Head of Trust & Safety
PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D
<https://keys.openpgp.org/vks/v1/by-fingerprint/BBAA6BCE33057FD66452711557B60114DE0B314D>
101 Townsend St, San Francisco, CA 94107



On Tue, Aug 11, 2020 at 3:25 PM William Herrin  wrote:

> Howdy,
>
> Is there an RBL lookup that provides information on why Cloudflare has
> elected to block a name lookup via the "1.1.1.1 for Families" service
> or is it a black box where you can only complain via
> https://report.teams.cloudflare.com/ and maybe they'll do something
> about it?
>
> Thanks,
> Bill Herrin
>
>
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/
>

Re: Tips on dealing with illicit BGP announcements

[Justin Wilson (Lists) via NANOG]

I second the ease on contacting RADB.  They are very easy to work with in cases 
like this.  Have done it several times over the past few months.


Justin Wilson
j...@mtin.net

—
https://j2sw.com - All things jsw (AS209109)
https://blog.j2sw.com - Podcast and Blog

> On Jul 24, 2020, at 2:05 AM, Randy Carpenter  wrote:
> 
> 
> I am working with a client that has recently purchased and transferred an 
> IPv4 block.
> 
> Sometime in between when the purchase and research was done and when the 
> transfer was actually complete, an entity in Asia started illicitly 
> announcing a larger block that includes the block in question. They even have 
> gotten an RADB entry in place for it.
> 
> Does anyone have some tips on how to deal with this? I have a feeling that 
> dealing directly with the offending entity will not be very fruitful.
> 
> thanks,
> -Randy
> 

Re: CloudFlare Issues?

[Justin Paine via NANOG]

The team is working on it.

_
*Justin Paine*
Head of Trust & Safety
PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D
<https://keys.openpgp.org/vks/v1/by-fingerprint/BBAA6BCE33057FD66452711557B60114DE0B314D>
101 Townsend St., San Francisco, CA 94107



On Fri, Jul 17, 2020 at 2:53 PM  wrote:

> Chris Grundemann wrote on 7/17/2020 2:38 PM:
>
> Looks like there may be something big up (read: down) at CloudFlare, but
> their status page is not reporting anything yet.
>
> Am I crazy? Or just time to give up on the internet for this week?
>
> --
> @ChrisGrundemann
> http://chrisgrundemann.com
>
> Status page just updated: Edge network and resolver issues.
>
> We had noticed something was up on our network as well w/ IPv6 name
> resolution timing out for some sites.
>
>
>

Re: Mystery CDN

[Justin Oeder]

Former Level3 operates a CDN.  Might be worth looking into.

On Wed, Jun 17, 2020, 11:43 AM Stephen Satchell  wrote:

> On 6/17/20 8:29 AM, Clinton Work wrote:
> > I'm struggling to determine which CDN owns the servers in CenturyLink
> prefix 8.240.0.0/12.   During the Call of Duty Season 4 update on June
> 11th from 06:00 UTC until 08:30 UTC, we had 240 Gbps of traffic steaming
> into our network from CenturyLink prefix 8.240.0.0/12.   We originally
> thought it was Akamai, but they swear up and down that the servers don't
> belong to them.
> >
> > Here are some of the HTTP/HTTPS servers in 8.240.0.0/12:
> > 8.253.151.248
> > 8.251.135.126
> > 8.240.167.126
> > 8.240.228.126
> > 8.240.168.126
> > 8.240.126.254
> > 8.240.191.254
>
> You might ask Level3.
>
>

PlayStation Web Technical Contact

[Justin Ouellette]

Does anyone have a good technical contact at PlayStation? Our customers are
having some issues accessing their website from one of our prefixes (/22).
I have not had any luck contacting anyone that wants to help resolve the
issue.

Re: [EXT] AS hijacking (Philosophy, rants, GeoMind)

[Justin Wilson (Lists)]

I will probably just get another link to https://isbgpsafeyet.com/ 
<https://isbgpsafeyet.com/> like I did in the first e-mail. LOL


Justin Wilson
j...@mtin.net

—
https://j2sw.com - All things jsw (AS209109)
https://blog.j2sw.com - Podcast and Blog

> On May 29, 2020, at 11:57 AM, Chuck Anderson  wrote:
> 
> Go back to them and tell them that a hijacked prefix is different from a 
> hijacked AS.
> 
> On Fri, May 29, 2020 at 11:39:46AM -0400, Justin Wilson (Lists) wrote:
>> One of the companies I work for recently had an issue with AS 2 (University 
>> of Delaware) hijacking a prefix.  Due to Origin AS, good upstreams, and the 
>> like this has not really affected the traffic to the legit blocks.  However, 
>> GeoMind picked this up almost immediately it seems.  The IP blocks when you 
>> go to speedtest.net come back to the university of Delaware. This seems to 
>> be the only issue at the moment so we are working through contacting the 
>> peers of AS2 and asking them to look into this.  We had also contacted 
>> University of Delaware.
>> 
>> Here is where the philosophy comes into play.  The very terse e-mail we 
>> received back was basically “As2 gets hijacked a lot and it’s not our 
>> problem”. So my question for the NANOG folks.  At what point do you say 
>> “it’s not your problem” when it involves your ASN?
>> 
>> Rant
>> I almost always have issues with GeoMind and others when it comes to IP 
>> space.  Several of my folks have received allocations from Arin in March.  A 
>> few are still fighting with geolocation stuff with a few of the providers.  
>> So why does GeoMind atomically accept a hijacked prefix as correct? All the 
>> right boxes have been ticked.  Origin Validiation, registry sets, etc.
> 

AS hijacking (Philosophy, rants, GeoMind)

[Justin Wilson (Lists)]

One of the companies I work for recently had an issue with AS 2 (University of 
Delaware) hijacking a prefix.  Due to Origin AS, good upstreams, and the like 
this has not really affected the traffic to the legit blocks.  However, GeoMind 
picked this up almost immediately it seems.  The IP blocks when you go to 
speedtest.net come back to the university of Delaware. This seems to be the 
only issue at the moment so we are working through contacting the peers of AS2 
and asking them to look into this.  We had also contacted University of 
Delaware.

Here is where the philosophy comes into play.  The very terse e-mail we 
received back was basically “As2 gets hijacked a lot and it’s not our problem”. 
So my question for the NANOG folks.  At what point do you say “it’s not your 
problem” when it involves your ASN?

Rant
I almost always have issues with GeoMind and others when it comes to IP space.  
Several of my folks have received allocations from Arin in March.  A few are 
still fighting with geolocation stuff with a few of the providers.  So why does 
GeoMind atomically accept a hijacked prefix as correct? All the right boxes 
have been ticked.  Origin Validiation, registry sets, etc.

Happy Friday! 



Justin Wilson
j...@mtin.net

—
https://j2sw.com - All things jsw (AS209109)
https://blog.j2sw.com - Podcast and Blog

DNS cache Validation

[Justin Wilson (Lists)]

What are you folk doing to validate your DNS cache server configs and 
operation? In other words, what are you doing to make sure they are performing 
well, not just alive.

Justin
—
https://blog.j2sw.com

Venmo - Geolocation Challenges

[Justin Krejci]

Hello,


I am looking for a Venmo network contact that can assist with a geolocation 
error in their systems. We have customers on a particular IP prefix who are 
being flagged by Venmo as outside of the USA but they are not outside of the 
USA. All standard geolocation systems I can find, as well as ARIN, all show the 
IP prefix as within the USA. Normal Venmo support channels are not fruitful to 
resolve the issue, they mostly just indicated users need to use their mobile 
data connection to get a different IP address for Venmo transactions. That is 
fine as a temporary work around but that is not a solution. Venmo support has 
expressed they are not going to do anything more for us in this regard.


So if anyone has a relevant contact I might reach out to at Venmo or knows if 
Venmo uses a particular 3rd party geolocation data set and can share that with 
me that would be appreciated. I don't mind working with any organization to 
straighten out any stale data, I just need some assistance getting to someone 
who has the info or access.


Thanks!!

Justin Krejci

Re: Cloudflare Contacts

[Justin Paine via NANOG]

Hi,

I forwarded this internally -- trying to locate the right contact for you.

_
*Justin Paine*
Head of Trust & Safety
PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D
101 Townsend St., San Francisco, CA 94107



On Tue, Mar 31, 2020 at 8:13 PM John Von Essen  wrote:

> Could someone from Cloudflare contact me off-list?
>
> I work for a major search engine (not google or bing), and we just
> launched some assets in Brazil, seeing some weird behavior to Cloudflare
> CDN assets and thinking maybe we are being caught in some kind of
> filter/block.
>
> Our image search traffic is proxied through a single IP, so its definitely
> high volume. We’ve never had an issue in other regions, but it could due to
> the sudden increase.
>
> Thanks
> John Von Essen

Re: Honeypot type services from cloud flare or other security groups?

[Justin Paine via NANOG]

Hi Brielle,

Happy to chat directly — drop me a direct email please? 

Thanks,

Justin

_
*Justin Paine*
Head of Trust & Safety
PGP: BBAA 6BCE 3305 7FD6 6452 7115 57B6 0114 DE0B 314D
101 Townsend St., San Francisco, CA 94107

On Wed, Mar 11, 2020 at 8:28 AM, Brielle < br...@2mbit.com > wrote:

> 
> 
> 
> Hi all,
> 
> 
> 
> Sorry for formatting errors, on my iPad while I have this thought in my
> mind.
> 
> 
> 
> Does anyone know if any of the security groups or CDNs like Cloudflare
> have honeypots out there that can be used for analysis of unusual attacks?
> As in, change the DNS temp for a host and let the honey pot take the brunt
> of it and hopefully get useful data (even for the benefit of the security
> company).
> 
> 
> 
> Got a situation where I’ve got an abnormally high amount of legit looking
> GET requests to a HTTPS git server, but are too high amount to actually be
> legit end users or people cloning the repos. The sources are worldwide,
> distributed, but with the bulk coming from China, Russia, Brazil, and
> Egypt.
> 
> 
> 
> I have some theories and observations that I’d be open to sharing, but
> preferably not on an open mailing list until I’ve had a change to have
> them reviewed by someone with more experience and background.
> 
> 
> 
> Thx!
> 
> 
> 
> Sent from my iPad
> 
> 
>

Re: Google peering in LAX

[Justin Seabrook-Rocha]

You hit the nail on the head. Google only seems to announce a subset of their 
routes to the route servers, but does announce all routes (for some definition 
of “all”) to direct peers. I notice this every time I turn up a new IX and 
traffic heads off onto my backbone instead of the local IX.

I did a spot check and I get that /24 via my direct peering (along with the 
/16).

Justin Seabrook-Rocha
-- 
Xenith || xen...@xenith.org || http://xenith.org/



> On Mar 2, 2020, at 12:40, Seth Mattinen  wrote:
> 
> Anyone know why Google announces only aggregates via peering and disaggregate 
> prefixes over transit?
> 
> For example, I had a customer complaining about a path that was taking the 
> long way instead of via peering and when I looked I saw:
> 
> Only 172.217.0.0/16 over Any2 LAX
> 
> That plus 172.217.14.0/24 over transit
> 
> Any inquiries to Google just get a generic "we're not setting up any new 
> peering but we're on route servers" response for almost a year now. Or is it 
> because they don't send the /24's to route servers and I'm stuck until they 
> finish their forever improvement project to turn up a direct neighbor?

Re: CISCO 0-day exploits

[Justin Wilson]

> 
> I really thought that more Cisco devices were deployed among NANOG.
> 
> I guess that these devices are not used anymore or maybe that I 
> understood wrong the severity of this CVE.

A proper network design helps to mitigate flaws like this. If you have CDP off, 
which many people do, then this exploit is not that big of a deal to you.  If 
your devices are on a management network then it’s not that big of a deal.  
Just because a certain vendor has vulnerabilities exposed doesn’t it’s an all 
hand on deck scenario.  Many of the folks on NANOG have a good grasp of network 
design.  Sure, some don’t.  But for the most part they do. 

Justin Wilson
li...@mtin.net

—
https://j2sw.com - All things jsw (AS209109)
https://blog.j2sw.com - Podcast and Blog

Re: DiviNetworks

[Justin Wilson]

They don’t lease your IP space is the thing.


Justin Wilson
li...@mtin.net


—
https://j2sw.com - All things jsw (AS209109)
https://blog.j2sw.com - Podcast and Blog

> On Feb 6, 2020, at 2:07 PM, Mike Fuller  wrote:
> 
> I'd be very cautious about engaging with any company whose business model is 
> to get a short-term lease of your IP-space.  Many companies use IP reputation 
> data, and so you are essentially lending that reputation to a 3rd party, who 
> may use it in ways you don't anticipate until the reputation is sufficiently 
> damaged, and then return it to you and move on to another ISP.
> 
> Some organizations' response to unwanted traffic is simply to block large IP 
> ranges or entire ASes, and not everyone is good about following-up and 
> expiring such blocks in the future.  I realize your customers haven't 
> ended-up on any spam/abuse blocklists, but that doesn't mean they won't be, 
> or that their IP reputation hasn't already been affected in less obvious 
> ways.  You should ask yourself if you are being sufficiently compensated for 
> these risks as reputable IPv4 space is at a premium, so replacing the IPv4 
> space you lent out could get quite costly.
> 
> --
> Mike Fuller :: Security Reliability Engineer :: Google :: AS15169
> 
> On Wed, Feb 5, 2020 at 12:15 PM Justin Wilson  <mailto:li...@mtin.net>> wrote:
> Have several networks using them.  This he networks get paid, and no 
> blacklists.  Contact me off list if you want more details
> 
> 
> 
> Justin Wilson
> li...@mtin.net <mailto:li...@mtin.net>
> 
> 
> —
> https://j2sw.com <https://j2sw.com/> - All things jsw (AS209109)
> https://blog.j2sw.com <https://blog.j2sw.com/> - Podcast and Blog
> 
> > On Feb 5, 2020, at 2:14 PM, Steve Saner  > <mailto:ssa...@hubris.net>> wrote:
> > 
> > Has anyone here worked with DiviNetworks (https://divinetworks.com/ 
> > <https://divinetworks.com/>) to "sell" their unused bandwidth?
> > 
> > I'd be curious to hear any thoughts or experiences.
> > 
> > Steve
> > 
> > -- 
> > --
> > Steven Saner mailto:ssa...@hubris.net>> 
> >  Voice:  316-858-3000 
> > Director of Network Operations  Fax:  316-858-3001 
> > 
> > Hubris Communicationshttp://www.hubris.net 
> > <http://www.hubris.net/>
> > 
> 

Re: DiviNetworks

[Justin Wilson]

Have several networks using them.  This he networks get paid, and no 
blacklists.  Contact me off list if you want more details



Justin Wilson
li...@mtin.net


—
https://j2sw.com - All things jsw (AS209109)
https://blog.j2sw.com - Podcast and Blog

> On Feb 5, 2020, at 2:14 PM, Steve Saner  wrote:
> 
> Has anyone here worked with DiviNetworks (https://divinetworks.com/) to 
> "sell" their unused bandwidth?
> 
> I'd be curious to hear any thoughts or experiences.
> 
> Steve
> 
> -- 
> --
> Steven Saner   Voice:  316-858-3000
> Director of Network Operations  Fax:  316-858-3001
> Hubris Communicationshttp://www.hubris.net
> 

Re: FYI - Suspension of Cogent access to ARIN Whois

[Justin Wilson]

This shall be my answer from now on.

> On Jan 27, 2020, at 1:22 PM, Dovid Bender  wrote:
> 
> I find it interesting that they say their clients didn't see it as an issue. 
> Whenever they called and asked if I want transit my answer always was when 
> they had v6 peering to He and Gooogle we could talk.
>