Re: Fastly Peering Contact

[Tom Samplonius]

Me too.  Fastly has been promising me peering every year for the past five 
years.  

They are just a few cabinets over, so it has been pretty frustrating.

Tom

Sent from my iPad

> On Dec 5, 2023, at 1:25 PM, Ian Chilton  wrote:
> 
> 
> Hi Peter,
> 
> Sorry you didn't get a response.
> 
> I only just started at Fastly, but I can look / nudge the right person if you 
> drop me an e-mail off-list with details.
> 
> Ian
>  
> 
>> On Tue, 5 Dec 2023, at 9:14 PM, Peter Potvin via NANOG wrote:
>> Looking for someone on the Fastly peering team to reach out regarding 
>> peering on a couple mutual IXPs - sent an email to the peering contact as 
>> listed on PeeringDB and never heard back, and also have a few colleagues who 
>> have experienced the same issue.
> 

Re: Outside plant - prewire customer demarc preference

[Sean Donelan]

You've misunderstood the goal.

The intent is not to protect the fiber, but to make it easier for the 
field tech installing new service in a neat way through finished 
construction and concealled raceways, without cutting sheetrock or 
stapling exposed cabling across walls.


Trying to prevent the next "bad fiber install" set of pictures.

U.S. NEC does not require any mechanical protection for fiber cables.  You 
can run "bare" fiber cables through most residential spaces (with a few 
exceptions for jacket material, i.e. direct burial cable not allowed 
inside habital spaces).  Building codes may vary in other countries.


On the other hand, do some searches for "bad fiber install" for many 
examples of fiber installers stapling fiber around the outside of houses 
or zip-tied to gas pipes.




On Tue, 5 Dec 2023, Martin Hannigan wrote:

Looks like over priced residential inner duct to me. Sheet rock accomplishes
pretty much the same thing. I want reliable home Internet too, but it’s not
a CO. I’d install a PVC sleeve on the OSP to ISP transition. The risk of
outage isn’t going to materially move one way or the other as far as I can
tell. 

Re: Outside plant - prewire customer demarc preference

[Martin Hannigan]

Thanks Sean!

Looks like over priced residential inner duct to me. Sheet rock
accomplishes pretty much the same thing. I want reliable home Internet too,
but it’s not a CO. I’d install a PVC sleeve on the OSP to ISP transition.
The risk of outage isn’t going to materially move one way or the other as
far as I can tell.

YMMV,

-M<


On Tue, Dec 5, 2023 at 21:28 Sean Donelan  wrote:

>
> I should have known better, network engineers don't work on the physical
> infrastructure very much anymore - memories of sitting on concrete floors
> crimping cable ends in to many IXPs :-)
>
> If you never seen or installed ENT Electrical Nonmetallic Tubing
> Conduit, also known as "smurf tube" -- here is a new YouTube video of
> someone installing a smurf tube between an external DEMARC and internal
> distribution point for his fiber connection.
>
> https://www.youtube.com/watch?v=NUCe9lAWY4U
>
>
> In the U.S. - ENT is UL listed as electrical conduit and can be used in
> most residential (and some commercial) runs.  Commonly used for
> low-voltage and fiber runs in the US.  I'm not an expert on other
> countries wiring codes.
>
> ENT is not the same as in-rack wiring management products (i.e. the
> split-wall plastic wire holders).
>

Re: Fiber/OSP Technician Training and Apprenticeship Programs

[Aaron Axvig via NANOG]

On 2023-11-16 2:51 pm, Rhys Barrie via NANOG wrote:

Hey all,

I've recently been working with our county's broadband task force,
investigating the expansion and equity of broadband networks on a
local and state level. Through that, it's become clear that there's a
painful shortage of fiber / outside plant technicians in the state of
Michigan (if not nation-wide) in order to fulfill the workforce
requirements of maintaining the current broadband fiber infrastructure
in the state, much less to fuel fiber expansion, and especially in
rural areas. There appear to be few options for training the required
workforce, especially outside of the large enterprises that have the
resources to run their own internal programs, and small (or even
mid-sized) ISPs seem to be left with predominantly informal
person-to-person transfer of internal knowledge, assuming that they
have the required internal knowledge in the first place. This need for
a qualified workforce is exacerbated in the face of the multitude of
state and federal programs to encourage broadband internet expansion
and equity, such as the upcoming $42.5 billion in BEAD grant funding
and corresponding construction starting in ~12-18 months state- and
nation-wide.

As a result, our workforce development team over here at Mott
Community College (Genesee County, MI) is working to develop a fiber /
outside plant training and apprenticeship program in order to help
address this shortage of qualified personnel and training options at a
local and state level. We're looking for some industry contacts that
would be interested in collaborating with us to establish high-level
requirements regarding what skills need to be taught to prospective
fiber / outside plant technicians, what qualifications trainees should
have after completion in order to fulfill current workforce demands,
and to otherwise provide input in sketching out a high-level
curriculum. We're looking for feedback from a wide cross-section of
industry stakeholders -- large enterprise backbone transit providers,
rural residential ISPs, fiber co-ops and municipal networks,
operations and outside plant managers, etc. -- in order to determine
what the industry wants and needs, and how the entire community
college system can help meet those needs.

If anyone thinks that they have valuable input to provide regarding
these workforce requirements, or knows the right people to talk to,
please reach out and let me know!

Rhys Barrie (He/Him)
Network Engineer - Mott Community College
Member - Genesee County Broadband Task Force
(810) 762-0030 | rhys.bar...@mcc.edu | https://mcc.edu/


The Fiber Optic Association may be of interest to you. "The FOA is an 
international non-profit educational association that is chartered to 
promote professionalism in fiber optics through education, certification 
and standards."


https://www.thefoa.org/

I read a couple of their e-books to self-teach myself some fiber 
knowledge circa 2016 and it was good material.

Internet Governance opportunity for NANOG community

[Eduardo Diaz]

Dear NANOG Community,

John Curran, President and CEO of ARIN, recently underscored the Internet's
significant transformation at NANOG 89. In this context, I invite you to
consider participating in our upcoming 2024 North American School of
Internet Governance (NASIG 2024 ) event from February
28 to March 1, 2024, at EDP University  in San
Juan, Puerto Rico. NANOG has consistently been a pillar of advancement,
pioneering in technological innovation and excellence in networking
practices. However, in our progressively interconnected world, it's vital
to expand our focus beyond technicalities and navigate the complexities of
Internet governance.

NASIG 2024  offers a program tailored to help you
understand Internet Governance topics like Internet fragmentation,
Inclusion & accessibility, DNS Abuse, and more. The school also provides
several* fellowships*, including a three-night accommodation during the
conference, offering a unique opportunity to interact with industry
pioneers and gain indispensable knowledge and insights. Register at
nasig.school.


Thank you for being an integral part of the Internet community.



Warm regards,



Eduardo Díaz

Chair, NASIG 2024

Re: Outside plant - prewire customer demarc preference

[Sean Donelan]

I should have known better, network engineers don't work on the physical 
infrastructure very much anymore - memories of sitting on concrete floors 
crimping cable ends in to many IXPs :-)


If you never seen or installed ENT Electrical Nonmetallic Tubing 
Conduit, also known as "smurf tube" -- here is a new YouTube video of 
someone installing a smurf tube between an external DEMARC and internal 
distribution point for his fiber connection.


https://www.youtube.com/watch?v=NUCe9lAWY4U


In the U.S. - ENT is UL listed as electrical conduit and can be used in 
most residential (and some commercial) runs.  Commonly used for 
low-voltage and fiber runs in the US.  I'm not an expert on other 
countries wiring codes.


ENT is not the same as in-rack wiring management products (i.e. the 
split-wall plastic wire holders).

Re: Any comprehensive listing of where Google's IPs originate from?

[Tom Beecher]

>
> i would expect that google announces the /16 at least from 'everywhere',
> yes.
>

I see the specific /18s Drew asked about initially. Didn't check for the
covering /16.

On Tue, Dec 5, 2023 at 1:29 PM Christopher Morrow 
wrote:

> On Tue, Dec 5, 2023 at 11:06 AM Tom Beecher  wrote:
> >
> > From my observations, all us-east-5 IPs are announced via transit and
> peering at all of my locations Chicago and east.
> >
>
> i would expect that google announces the /16 at least from 'everywhere',
> yes.
>
> > On Mon, Dec 4, 2023 at 9:11 AM Drew Weaver 
> wrote:
> >>
> >> Hello,
> >>
> >>
> >>
> >> We are trying to reduce latency to a region in Google Cloud which we
> are in the same city of. Latency is currently about 22ms rt for the traffic
> to go 9 miles.
> >>
> >>
> >>
> >> I am having the hardest time finding any comprehensive list of what
> exchanges, transit, etc their IP addresses are being announced over.
> >>
>
> 100% chance the best option is peeringdb (I'm betting closest to
> drew is chicago-land-ix-ville)
>
> >>
> >>
> >> Specifically trying to get closer to addresses in these prefixes:
> >>
> >>
> >>
> >> 34.162.192.0/18
> >>
> >> 34.162.64.0/18
> >>
> >>
> >>
> >> Any info is greatly appreciated.
> >>
> >>
> >>
> >> Thanks,
> >>
> >> -Drew
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
>

Re: Fastly Peering Contact

[Ian Chilton]

Hi Peter,

Sorry you didn't get a response.

I only just started at Fastly, but I can look / nudge the right person if you 
drop me an e-mail off-list with details.

Ian
 

On Tue, 5 Dec 2023, at 9:14 PM, Peter Potvin via NANOG wrote:
> Looking for someone on the Fastly peering team to reach out regarding peering 
> on a couple mutual IXPs - sent an email to the peering contact as listed on 
> PeeringDB and never heard back, and also have a few colleagues who have 
> experienced the same issue.

Fastly Peering Contact

[Peter Potvin via NANOG]

Looking for someone on the Fastly peering team to reach out regarding
peering on a couple mutual IXPs - sent an email to the peering contact as
listed on PeeringDB and never heard back, and also have a few colleagues
who have experienced the same issue.

Regards,
Peter Potvin | Executive Director
--
*Accuris Technologies Ltd.*

Re: What are these Google IPs hammering on my DNS server?

[Ray Bellis]

On 05/12/2023 20:08, Christopher Morrow wrote:

is the test framework documented where others could setup/run the 
test(s)? :) (perhaps for mr hare I mean, or me! :) )


https://github.com/isc-projects/perflab

https://www.isc.org/docs/bellis-oarc-perflab.pdf


Are the tests for authoritative or cache resolvers?


Originally it was just for auth, but there's some recursive support too.

I wrote the framework, but these days I'm too busy running F-root so the 
BIND QA guys maintain it.


Ray

Re: What are these Google IPs hammering on my DNS server?

[Christopher Morrow]

On Tue, Dec 5, 2023 at 10:17 AM Ray Bellis  wrote:
>
>
>
> On 05/12/2023 12:29, Michael Hare via NANOG wrote:
>
> > At quick glance following the ISC link I didn’t see the compute
> > infrastructure [core count] needed to get 1Mpps.  There is an obvious
> > difference between 99% load of ~500rps and 1M, so we can maybe advise to
> > not undersize ADNS if that's an issue.
>
> The system under test in ISC's perflab is a 12-core Dell R430 of 2016
> vintage.

is the test framework documented where others could setup/run the
test(s)? :) (perhaps for mr hare I mean, or me! :) )
Are the tests for authoritative or cache resolvers?

-chris

Re: Any comprehensive listing of where Google's IPs originate from?

[Christopher Morrow]

On Tue, Dec 5, 2023 at 11:06 AM Tom Beecher  wrote:
>
> From my observations, all us-east-5 IPs are announced via transit and peering 
> at all of my locations Chicago and east.
>

i would expect that google announces the /16 at least from 'everywhere', yes.

> On Mon, Dec 4, 2023 at 9:11 AM Drew Weaver  wrote:
>>
>> Hello,
>>
>>
>>
>> We are trying to reduce latency to a region in Google Cloud which we are in 
>> the same city of. Latency is currently about 22ms rt for the traffic to go 9 
>> miles.
>>
>>
>>
>> I am having the hardest time finding any comprehensive list of what 
>> exchanges, transit, etc their IP addresses are being announced over.
>>

100% chance the best option is peeringdb (I'm betting closest to
drew is chicago-land-ix-ville)

>>
>>
>> Specifically trying to get closer to addresses in these prefixes:
>>
>>
>>
>> 34.162.192.0/18
>>
>> 34.162.64.0/18
>>
>>
>>
>> Any info is greatly appreciated.
>>
>>
>>
>> Thanks,
>>
>> -Drew
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>

Re: Any comprehensive listing of where Google's IPs originate from?

[Tom Beecher]

>From my observations, all us-east-5 IPs are announced via transit and
peering at all of my locations Chicago and east.

On Mon, Dec 4, 2023 at 9:11 AM Drew Weaver  wrote:

> Hello,
>
>
>
> We are trying to reduce latency to a region in Google Cloud which we are
> in the same city of. Latency is currently about 22ms rt for the traffic to
> go 9 miles.
>
>
>
> I am having the hardest time finding any comprehensive list of what
> exchanges, transit, etc their IP addresses are being announced over.
>
>
>
> Specifically trying to get closer to addresses in these prefixes:
>
>
>
> 34.162.192.0/18
>
> 34.162.64.0/18
>
>
>
> Any info is greatly appreciated.
>
>
>
> Thanks,
>
> -Drew
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>

Re: What are these Google IPs hammering on my DNS server?

[Ray Bellis]

On 05/12/2023 12:29, Michael Hare via NANOG wrote:

At quick glance following the ISC link I didn’t see the compute 
infrastructure [core count] needed to get 1Mpps.  There is an obvious 
difference between 99% load of ~500rps and 1M, so we can maybe advise to 
not undersize ADNS if that's an issue.


The system under test in ISC's perflab is a 12-core Dell R430 of 2016 
vintage.


Ray

RE: What are these Google IPs hammering on my DNS server?

[Michael Hare via NANOG]

Damian-

Not Google or ISCs fault, our customers have made some decisions that have 
exasperated the issues.  By and away the biggest problem facing my customers is 
that they have chosen a stateful border firewall that collapses due to session 
exhaustion and they put everything, including aDNS, behind said firewall.  “If 
it hurts, don’t do it” comes to mind, but out of my hands.

At quick glance following the ISC link I didn’t see the compute infrastructure 
[core count] needed to get 1Mpps.  There is an obvious difference between 99% 
load of ~500rps and 1M, so we can maybe advise to not undersize ADNS if that's 
an issue.

I'm an ISP engineer and am generally not the directly affected party, so I 
don't get to pick these implementation details for my customers.  I appreciate 
the background and suggestions from you and others on this thread like Mark.  
That's an interesting comment about DNSSEC that I hadn't considered.

-Michael

From: Damian Menscher 
Sent: Monday, December 4, 2023 12:21 PM
To: Michael Hare 
Cc: John R. Levine ; nanog@nanog.org
Subject: Re: What are these Google IPs hammering on my DNS server?

Google Public DNS (8.8.8.8) attempts to identify and filter abuse, and while we 
think we're fairly effective for large attacks (eg, those above 1Mpps), it gets 
more challenging (due to risk of false positives) to adequately filter small 
attacks.  I should note that we generally see the attack traffic coming from 
botnets, or forwarding resolvers that blend the attack traffic with legitimate 
traffic.

Based on ISC BIND load-tests [0], a single DNS server can handle O(1Mpps).  
Also, no domain should be served by a single DNS server, so O(1Mpps) seems like 
a safe lower-bound for small administrative domains (larger ones will have more 
redundancy/capacity).  Based on these estimates, we haven't treated mitigation 
of small attacks as a high priority.  If O(25Kpps) attacks are causing real 
problems for the community, I'd appreciate that feedback and some hints as to 
why your experience differs from the ISC BIND load-tests.  With a better 
understanding of the pain-points, we may be able to improve our filtering a 
bit, though I suspect we're nearing the limits of what is attainable.

Since it was mentioned up-thread, I'd caution against dropping queries from 
likely-legitimate recursives, as that will lead to a retry storm that you won't 
like (based on a few reports of authoritatives who suffered outages, the retry 
storm increased demand by 30x and they initially misdiagnosed the root cause as 
a DDoS).  The technically correct (if not entirely practical) mitigation for a 
DNS cache-busting attack laundered through open recursives is to deploy DNSSEC 
and issue NSEC/NSEC3 responses to allow the recursives to cache the 
non-existence of the randomized labels.

[0] https://www.isc.org/blogs/bind-performance-september-2023/

Damian
--
Damian Menscher :: Security Reliability Engineer :: Google :: AS15169

On Sun, Dec 3, 2023 at 1:22 PM Michael Hare via NANOG 
mailto:nanog@nanog.org>> wrote:
John-

This is little consolation, but at AS3128, I see the same thing to our 
downstream at times, claiming to come from both 13335 and 15169 often 
simultaneously at the tune of 25Kpps , "assuming it's not spoofed", which is 
pragmatically impossible to prove for me given our indirect relationships with 
these companies.  When I see these events, I typically also see a wide variety 
of country codes participating simultaneously.  Again, assuming it's not 
spoofed.  To me it just looks like effective harassment with 13335/15169 
helping out.  I pine for the internet of the 1990s.

Recent events in GMT for us were the following, curious if you see the same
~ Nov 26 05:40
~ Nov 30 00:40
~ Nov 30 05:55

Application agnostic, on the low $ end for "fixes", if it's either do something 
or face an outage, I've found some utility in short term automated DSCP 
coloring on ingress paired with light touch policing as close to the end host 
as possible, which at least keeps things mostly working during times of 
conformance.  Cheap/fast and working ... most of the time.  Definitely not 
great or complete at all, and a role I'd rather not play as an educational 
ISP/enterprise.

So what are most folks doing to survive crap like this?  Nothing/waiting it 
out?  Oursourcing DNS?  Scrubbing appliance?  Poormans stuff like I mention 
above?

-Michael

> -Original Message-
> From: NANOG 
> mailto:wisc@nanog.org>> On
> Behalf Of John R. Levine
> Sent: Sunday, December 3, 2023 1:18 PM
> To: Peter Potvin 
> mailto:peter.pot...@accuristechnologies.ca>>
> Cc: nanog@nanog.org
> Subject: Re: What are these Google IPs hammering on my DNS server?
>
> > Did a bit of digging on Google's developer site and came across this:
> > https://developers.google.com/speed/public-
> dns/faq#locations_of_ip_address_ranges_google_public_dns_uses_to_send_
> queries
> >
> > Looks like the IPs you mentioned