Re: Personal Colo 2024

[Mel Beckman]

There are some specialized ones I know of for MacOS hardware, such as 
https://macminicolo.net/ and https://www.macminivault.com/. I don't know if you 
can BYOD, but I know they rent hardware. It's not really a "1U" generalized 
experimental space, but it at least lets you have a virtualization platform on 
dedicated hardware.

I used to operate a "4U" colo, VautledData.com, out of a former AT&T bunker, in 
Yuma, AZ, but the demand just wasn't there so eventually we repurposed the site 
as a satellite uplink with a two-story motorized dish antenna.

  -mel


From: NANOG  on behalf of Tim Utschig 

Sent: Monday, August 5, 2024 10:02 PM
To: nanog@nanog.org 
Subject: Personal Colo 2024

Are there any providers of 1U personal colos these days?

VMs are neat, but they lack the power to experiment with without
paying an arm and a leg.

I was lucky enough to have my 1U hosted by Dave Rand back in the
day.

Thanks.

--
Tim Utschig 
408-644-3861 (mobile)

Re: Current diameter of the Internet?

[Mel Beckman]

Sean,

Your just trying to choose reasonable timeouts for your TCP packets? Well, why 
didn't you say so? (Or perhaps I missed it).

Fortunately, Einstein's General Theory of Relativity was solved long ago, so 
that's take care of. 🙂

If you're trying to choose reasonable timeouts, a wrong question to ask is 
"What is the current diameter of the Internet?" No matter what the diameter is, 
whether measured in RTT or TTL hops, a more useful question is, "How long am I 
willing to wait before considering a packet lost and retransmitting?" That time 
doesn't really matter so much on modern networks, so just choose a nice value 
you can live with, such as an hour (which happens to be the default). The 
timeout field itself can record values up to one day, so someone apparently 
thought that might be useful sometime (see RFC2549).

Keep in mind that  TCP rarely is configured to acknowledge each packet before 
sending the next. Instead it expects an acknowledgement every so-many packet.

Consider the deminimus thought experiment. Your packet timeout is set to 2000 
ms (two seconds). In the midst of an existing TCP session, you fire a slew of 
TCP packets – enough to fill your Send Window, and then wait for an ACK. Before 
you know it, 2000 ms has elapsed, and under TCP rules, you must consider those 
packets "lost". So, you retransmit them. But then, at 2035 ms, you receive the 
presumed missing ACK, which indicates all your packets were received at the far 
end after all. No harm done, because the far end will just treat the 
retransmissions as duplicates and throw them away.

The amount of delay a packet experiences has a lot less to do with the 
distance, diameter, or latency, and a lot more to do with the processing going 
on along the way. For example, a router becomes congested, and hangs on to 
packets longer than usual. The congestion spreads along the path your packets 
are taking, so the delay adds up.

In the end, unless you're using some really unusual technology (e.g. RFC2549), 
just set the timeout to the default 3600 seconds (1 hour). For a detailed 
explanation, refer to Richard Stevens TCP/IP Illustrated, Chapter 14 .TCP 
Timeout and Retransmission.

 -mel

From: Sean Donelan 
Sent: Monday, July 22, 2024 2:05 PM
To: Mel Beckman 
Cc: nanog@nanog.org 
Subject: Re: Current diameter of the Internet?


OMG, Not trying to solve Einstein's General Theory of Relativity.

Just trying to choose reasonable timeouts for my TCP packets
:-)

Middleware boxes suck -- its IETF week.

On Mon, 22 Jul 2024, Mel Beckman wrote:
> *A. Einstein, Relativity: The Special and General Theory, authorized
> translation by R. W. Lawson (New York: Crown Publishers, 1961), p 23.
>
> Perhaps an RFC should be written to address this :)

Re: Current diameter of the Internet?

[Mel Beckman]

For a more academic treatment:

“The crucial problem of how to synchronize clocks and measure the one-way speed 
of light was originally discussed by Poincaré and Einstein. After being 
neglected for many decades, the Poincaré-Einstein problem of synchronization 
revived in 1977 with the work of Mansouri and Sexl, by which the one-way speed 
remains undetermined, allowing for unequal values of the speed of light in 
opposite directions.”


[10053.png]
On measuring the one-way speed of light - The European Physical Journal 
D
link.springer.com



So it’s completely reasonable to assume the speed of light is 1/2 c in one 
direction and infinite in the other. It’s just an optional convention that we 
consider the speed to be the same in both directions.

Einstein said that light’s one-way speed “is in reality neither a supposition 
nor a hypothesis about the physical nature of light, but a stipulation which I 
can make of my own freewill in order to arrive at a definition of 
simultaneity.”*

*A. Einstein, Relativity: The Special and General Theory, authorized 
translation by R. W. Lawson (New York: Crown Publishers, 1961), p 23.

Perhaps an RFC should be written to address this :)

 -mel

On Jul 21, 2024, at 6:38 PM, Scott Q.  wrote:

 Well...it gets complicated :)

https://www.youtube.com/watch?v=pTn6Ewhb27k

On Sunday, 21/07/2024 at 20:15 Josh Luthman wrote:
Whoops, that should have said radio waves travel faster than fiber (more so in 
a vacuum).

On Sun, Jul 21, 2024 at 8:07 PM Chris Adams 
mailto:c...@cmadams.net>> wrote:
Once upon a time, Josh Luthman 
mailto:j...@imaginenetworksllc.com>> said:
> Voyager is using radio waves, which travel faster than the speed of light
> (in a vacuum, too!).

No...
--
Chris Adams mailto:c...@cmadams.net>>

Re: Current diameter of the Internet?

[Mel Beckman]

Easy. Bridge loop.

-mel via cell

On Jul 21, 2024, at 4:06 PM, Josh Luthman  wrote:


Mel,

Voyager is using radio waves, which travel faster than the speed of light (in a 
vacuum, too!).  But my point is more Earth to outside the solar system is ~24 
hours so where did circumnavigating the globe get three days of latency?

On Sun, Jul 21, 2024 at 2:29 PM Mel Beckman 
mailto:m...@beckman.org>> wrote:
Chris,

Of course I do.

 -mel

> On Jul 21, 2024, at 8:55 AM, Chris Adams 
> mailto:c...@cmadams.net>> wrote:
>
> Once upon a time, Mel Beckman mailto:m...@beckman.org>> 
> said:
>> Because the speed of light is different in different mediums. It depends on 
>> the index of refraction. Most of the Internet is on fiber optics, and the 
>> speed of light in glass fiber is dramatically slower than in a vacuum. Long 
>> distance single-mode communication fiber typically has a core index of 
>> refraction of 1.4682 at 1550nm (mid-C-band). So the speed of light in this 
>> type of fiber is the speed of light in a vacuum 299,792,458 m/s divided by 
>> 1.4682 = 204,190,477 m/s. You have to add to that the latency of any optical 
>> to electrical transformations, which happens in most every router or switch. 
>> Three days is probably an underestimate.
>
> Uh, you do know that Voyager isn't unspooling fiber as it goes, right?
>
> --
> Chris Adams mailto:c...@cmadams.net>>

Re: Current diameter of the Internet?

[Mel Beckman]

Chris,

Of course I do. 

 -mel

> On Jul 21, 2024, at 8:55 AM, Chris Adams  wrote:
> 
> Once upon a time, Mel Beckman  said:
>> Because the speed of light is different in different mediums. It depends on 
>> the index of refraction. Most of the Internet is on fiber optics, and the 
>> speed of light in glass fiber is dramatically slower than in a vacuum. Long 
>> distance single-mode communication fiber typically has a core index of 
>> refraction of 1.4682 at 1550nm (mid-C-band). So the speed of light in this 
>> type of fiber is the speed of light in a vacuum 299,792,458 m/s divided by 
>> 1.4682 = 204,190,477 m/s. You have to add to that the latency of any optical 
>> to electrical transformations, which happens in most every router or switch. 
>> Three days is probably an underestimate.
> 
> Uh, you do know that Voyager isn't unspooling fiber as it goes, right?
> 
> --
> Chris Adams 

Re: Current diameter of the Internet?

[Mel Beckman]

Josh,

Because the speed of light is different in different mediums. It depends on the 
index of refraction. Most of the Internet is on fiber optics, and the speed of 
light in glass fiber is dramatically slower than in a vacuum. Long distance 
single-mode communication fiber typically has a core index of refraction of 
1.4682 at 1550nm (mid-C-band). So the speed of light in this type of fiber is 
the speed of light in a vacuum 299,792,458 m/s divided by 1.4682 = 204,190,477 
m/s. You have to add to that the latency of any optical to electrical 
transformations, which happens in most every router or switch. Three days is 
probably an underestimate.

 -mel beckman

On Jul 21, 2024, at 7:47 AM, Josh Luthman  wrote:


Where do you get 3 days?

Voyager 1 is about 15.2B miles or 22.665707 hours at the speed of light.

On Sat, Jul 20, 2024 at 7:12 PM Nathan Angelacos 
mailto:nan...@tetrasec.net>> wrote:
On Sat, 2024-07-20 at 00:58 -0500, Stas Bilder wrote:
Pity we can’t ping Voyagers.

S.


ROTFL,   you actually had me pull out Star Trek - The Movie... Wow... what a 
blast from 1979.

So yeah ... According to our media outlets, RTT of the internet is ... um 3 
days.

Re: HE.net problem

[Mel Beckman]

Aha. Just as I suspected, bureaucrats at Network Solutions are to blame. I have 
had many run-ins with NS and their inscrutable policies and odd viewpoints. I 
was once suspended for running a web cache that NS incorrectly claimed was 
stealing domain content. No engineer on the NS side seemed to know what a web 
cache does.

-mel via cell

On Jul 4, 2024, at 12:42 PM, Mel Beckman  wrote:

 Ryan,

Right you are.  The dig still fails. hopefully the ICANN issue gets fixed, and 
a pox on any bureaucrat who arranged for this to happen over a holiday weekend!

 -mel

On Jul 4, 2024, at 12:33 PM, Ryan Hamel  wrote:


Mel,

Your local caching resolver knows the IPs for ns[1-5].he.net, which skips over 
the need for querying the root DNS resolvers, and gtld-servers (glue records). 
If the TTL (2 days) expires on your resolver before HE fixes their issue, you 
will not be able to resolve anything for that domain.

At the moment, a simple DNS trace (dig he.net +trace) cannot complete fully.

Ryan Hamel


From: Mel Beckman 
Sent: Thursday, July 4, 2024 12:20 PM
To: Jay Ashworth 
Cc: Ryan Hamel ; nanog@nanog.org 
Subject: Re: HE.net problem

Caution: This is an external email and may be malicious. Please take care when 
clicking links or opening attachments.

Our he.net dns appears to be fine at this time:

$ nslookup
server ns1.he.net
Default server: ns1.he.net
Address: 2001:470:100::2#53
Default server: ns1.he.net
Address: 216.218.130.2#53
> set type=A
> jet.net.
Server: ns1.he.net
Address:216.218.130.2#53

Name:   jet.net
Address: 206.83.0.42

 -mel beckman

On Jul 4, 2024, at 12:11 PM, Jay Ashworth  wrote:


Cool, thanks. We had a couple of other reports of people making support calls 
and being asked to reboot their modems, so I wanted to make sure tier 3 had 
gotten it.

And I figured tier 3 would be here. :-)

Cheers,
-- jra


On July 4, 2024 3:00:12 PM EDT, Ryan Hamel  wrote:
I called their support when that outage thread came in, they're already aware 
and taking a look now.

Ryan Hamel


From: NANOG  on behalf of Jay 
Ashworth 
Sent: Thursday, July 4, 2024 11:55 AM
To: nanog@nanog.org 
Subject: HE.net problem

Caution: This is an external email and may be malicious. Please take care when 
clicking links or opening attachments.

We have a report on outages that he.net has been placed in ICANN client hold, 
and people's DNS service is falling over on this Independence day. If you work 
in DNS for HE, you might want to look into this.

I have double checked the report, and I am seeing the status as well.

Hurricane serves lots of dns, I would classify this as a P1 ticket.

Cheers,
-- jra
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: HE.net problem

[Mel Beckman]

Ryan,

Right you are.  The dig still fails. hopefully the ICANN issue gets fixed, and 
a pox on any bureaucrat who arranged for this to happen over a holiday weekend!

 -mel

On Jul 4, 2024, at 12:33 PM, Ryan Hamel  wrote:


Mel,

Your local caching resolver knows the IPs for ns[1-5].he.net, which skips over 
the need for querying the root DNS resolvers, and gtld-servers (glue records). 
If the TTL (2 days) expires on your resolver before HE fixes their issue, you 
will not be able to resolve anything for that domain.

At the moment, a simple DNS trace (dig he.net +trace) cannot complete fully.

Ryan Hamel


From: Mel Beckman 
Sent: Thursday, July 4, 2024 12:20 PM
To: Jay Ashworth 
Cc: Ryan Hamel ; nanog@nanog.org 
Subject: Re: HE.net problem

Caution: This is an external email and may be malicious. Please take care when 
clicking links or opening attachments.

Our he.net dns appears to be fine at this time:

$ nslookup
server ns1.he.net
Default server: ns1.he.net
Address: 2001:470:100::2#53
Default server: ns1.he.net
Address: 216.218.130.2#53
> set type=A
> jet.net.
Server: ns1.he.net
Address:216.218.130.2#53

Name:   jet.net
Address: 206.83.0.42

 -mel beckman

On Jul 4, 2024, at 12:11 PM, Jay Ashworth  wrote:


Cool, thanks. We had a couple of other reports of people making support calls 
and being asked to reboot their modems, so I wanted to make sure tier 3 had 
gotten it.

And I figured tier 3 would be here. :-)

Cheers,
-- jra


On July 4, 2024 3:00:12 PM EDT, Ryan Hamel  wrote:
I called their support when that outage thread came in, they're already aware 
and taking a look now.

Ryan Hamel


From: NANOG  on behalf of Jay 
Ashworth 
Sent: Thursday, July 4, 2024 11:55 AM
To: nanog@nanog.org 
Subject: HE.net problem

Caution: This is an external email and may be malicious. Please take care when 
clicking links or opening attachments.

We have a report on outages that he.net has been placed in ICANN client hold, 
and people's DNS service is falling over on this Independence day. If you work 
in DNS for HE, you might want to look into this.

I have double checked the report, and I am seeing the status as well.

Hurricane serves lots of dns, I would classify this as a P1 ticket.

Cheers,
-- jra
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: HE.net problem

[Mel Beckman]

Our he.net dns appears to be fine at this time:

$ nslookup
server ns1.he.net
Default server: ns1.he.net
Address: 2001:470:100::2#53
Default server: ns1.he.net
Address: 216.218.130.2#53
> set type=A
> jet.net.
Server: ns1.he.net
Address:216.218.130.2#53

Name:   jet.net
Address: 206.83.0.42

 -mel beckman

On Jul 4, 2024, at 12:11 PM, Jay Ashworth  wrote:


Cool, thanks. We had a couple of other reports of people making support calls 
and being asked to reboot their modems, so I wanted to make sure tier 3 had 
gotten it.

And I figured tier 3 would be here. :-)

Cheers,
-- jra


On July 4, 2024 3:00:12 PM EDT, Ryan Hamel  wrote:
I called their support when that outage thread came in, they're already aware 
and taking a look now.

Ryan Hamel


From: NANOG  on behalf of Jay 
Ashworth 
Sent: Thursday, July 4, 2024 11:55 AM
To: nanog@nanog.org 
Subject: HE.net problem

Caution: This is an external email and may be malicious. Please take care when 
clicking links or opening attachments.

We have a report on outages that he.net has been placed in ICANN client hold, 
and people's DNS service is falling over on this Independence day. If you work 
in DNS for HE, you might want to look into this.

I have double checked the report, and I am seeing the status as well.

Hurricane serves lots of dns, I would classify this as a P1 ticket.

Cheers,
-- jra
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

In Remembrance: ARIN Amassador Stacy Taylor (Hughes)

[Mel Beckman]

For those who haven't seen this yet, I'm passing it along from ARIN 
announcements. Stacy passed away suddenly in the hospital last week.

I first met Stacy when she worked as  "IP Godess" at TW Telecom, by which time 
she was already a force of nature within ARIN, and at many RIR and NANOG 
events. No matter who you were, Stacy always made you feel welcome and valued.  
 Good Memories are gathering on her Facebook page at 
https://www.facebook.com/stacy.taylor3k

  -mel



Posted: Friday, 24 May 2024
ARIN 

The ARIN Board of Trustees, Advisory Council, and staff are shocked and 
saddened by the death of their friend and former Advisory Council colleague, 
Stacy Taylor (Hughes).

Stacy, known to many by her email handle “IP Goddess,” served on the ARIN 
Advisory Council from 2002 to 2014. But beyond her contribution to policy, for 
more than a decade Stacy brought a spirit of joy and welcome to the wider ARIN 
community.

ARIN is a community first and foremost, and for many years, Stacy was one of 
its best ambassadors as she participated in Regional Internet Registry meetings 
around the globe. Her impact is still visible in the community where she 
mentored so many, and her spirit of welcome is still part of the Advisory 
Council.

The Board of Trustees hereby extends its sympathies to Stacy’s family and 
friends. We shall miss her spirit and remember her always with respect, 
admiration and warmth of heart.

Regards,

American Registry for Internet Numbers (ARIN)



https://www.arin.net/announcements/20240524/

[https://www.arin.net/img/logo-social.png]
In Remembrance — Stacy Taylor 
(Hughes)
The ARIN Board of Trustees, Advisory Council, and staff are shocked and 
saddened by the death of their friend and former Advisory Council colleague, 
Stacy Taylor (Hughes). Stacy, known to many by her email handle “IP Goddess,” 
served on the ARIN Advisory Council from 2002 to 2014. But beyond her 
contribution to policy, for more than a decade Stacy brought a spirit of joy 
and welcome to the wider ARIN community.
www.arin.net

Re: Q: is RFC3531 still applicable?

[Mel Beckman]

Bill,

I would just make it /64s all the way down. Subnetting a /64 is like taking 
half-breaths from a snorkel: why bother when the supply is effectively infinite?

 -mel

> On May 16, 2024, at 3:35 AM, William Herrin  wrote:
> 
> On Wed, May 15, 2024 at 10:09 PM Mel Beckman  wrote:
>> The RFC seems to be concerned with aggregation efficiency, and while that 
>> may be a concern someday, so far computer and memory capacity has far 
>> outstripped prefix growth in the default-free zone.
>> 
>> If you can explain why a /64 would ever not be enough for a single subnet, 
>> I’m willing to listen.
> 
> The subnet contains a router that gateways to another /64. For
> example, there's a home automation controller and the controller
> implements its own lan of components on a different media type.
> 
> Instead of assigning a pair of /64's, you assign a /63 and then route
> a /64 from the /63 to the home automation controller.
> 
> Except you don't do that either. IPv6 is plentiful and reverse-DNS
> delegates cleanly on 4-bit boundaries so you think in terms of 4-bit
> boundaries instead: assign a /60, use a /64 on the immediate LAN and
> route a /64 to the home automation controller and retain the balance
> for the next device that wants to implement an internal subnet.
> 
> Regards,
> Bill Herrin
> 
> 
> 
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/

Re: Q: is RFC3531 still applicable?

[Mel Beckman]

Jay,

Each IPv6 /64 should be thought of as the same as an IPv4 /32? That seems a tad 
wasteful. A single /64 has billions of times more addresses than the entire 
IPv4 address space. It is enough for any conceivable subnet. There are also 
billions of /56 prefixes available, so no ISP customer would ever exhaust those 
either. A customer can get as many /56s as they need.

The RFC seems to be concerned with aggregation efficiency, and while that may 
be a concern someday, so far computer and memory capacity has far outstripped 
prefix growth in the default-free zone.

If you can explain why a /64 would ever not be enough for a single subnet, I’m 
willing to listen.

 -mel

On May 15, 2024, at 9:52 PM, Jay Acuna  wrote:

A /64 is not "enough" period.  Each IPv6 /64 should be thought of as
the same as an IPv4 /32.
The RFC is still relevant.  You are able to be allocated IPs
justifying 8-bits per customer
(/56) and customers should expect that /56 be the minimum delegated by
their providers.

The prefix delegation for IPv6 is based on number of separate /64
subnets they might have a reason
to use (which can be for many reasons including security and division
of traffic and use cases),

Not number of individual hosts they may have, since subnet divisions
more granular than
/64 are not possible.

On Wed, May 15, 2024 at 8:17 AM Mel Beckman  wrote:
I never could understand the motivation behind RFC3531. Just assign /64s. A 
single /64 subnet has 18,446,744,073,709,551,616  host addresses.  It is 
enough. Period.
-mel

--
-J

Re: Mailing list SPF Failure

[Mel Beckman]

Let us see…

 -mel beckman

> On May 15, 2024, at 7:47 PM, Scott Q.  wrote:
> 
>  Anyone else getting SPF failures on all messages sent to the list ?
> 
> I see them all originating from 50.31.151.76 but nanog.org's SPF record 
> doesn't list that as allowed.
> 

Re: Q: is RFC3531 still applicable?

[Mel Beckman]

I never could understand the motivation behind RFC3531. Just assign /64s. A 
single /64 subnet has 18,446,744,073,709,551,616  host addresses.  It is 
enough. Period.


 -mel

On May 14, 2024, at 12:54 PM, Adam Thompson  wrote:


Not an IPv6 newbie by any stretch, but we still aren’t doing it “at scale” and 
some of you are, so…

For a very small & dense (on 128-bit scales, anyway) network, is RFC3531 still 
the last word in IPv6 allocation strategies?

Right now, we’re just approaching it as “pick the next /64 in the range”, as it 
all gets aggregated at the BGP border anyway, and internally if I really try 
hard, I might get to 200 subnets someday.

Is there any justification for the labour in doing something more complex like 
center-allocation in my situation?  Worrying about allocation strategies seems 
appropriate to me if you have 100,000 subnets, not 100.

Opinions wanted, please.
-Adam

Adam Thompson
Consultant, Infrastructure Services
MERLIN
100 - 135 Innovation Drive
Winnipeg, MB R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
https://www.merlin.mb.ca
Chat with me on 
Teams

Re: Small Internet border router options?

[Mel Beckman]

You have to turn off netflow to get full performance from the XGs. 

-mel via cell

> On May 13, 2024, at 3:22 PM, Harlan Stenn via NANOG  wrote:
> 
> On 5/13/2024 12:45 PM, Jean Franco wrote:
>> Hi Tom.
>> Ubiquiti EdgeRouter Infinity.
> 
> What version of firmware are you using?
> 
> We have never seen our box (an ER-8-XG) route packets for more than about an 
> hour before it falls over.
> 
> If you have having better luck with this and would be open to helping us, 
> please let me know - we'd greatly appreciate it.
> 
>> Best regards,
>> On Mon, May 13, 2024 at 3:54 PM Tom Samplonius > > wrote:
>>   What are using for small campus border routers?  So four to eight
>>10G ports with a FIB for full scale L3?
>>Tom
> 
> --
> Harlan Stenn 
> https://www.nwtime.org/ - be a member!

Re: NOAA Space Weather Prediction Center issued a Severe (G4) Geomagnetic Storm Watch

[Mel Beckman]

They are identical units running the same firmware versions. The TM 1000A 
doesn’t give you an option for any kind of backup syncing to NTP on the 
network. They are pure GPS servers; theoretically they either serve the correct 
time or no time at all. So technically to give such as our time deviation would 
be a bug of some kind.

Unfortunately, I don’t have the actual packets received, so I can’t decode them 
to see if this was just bad interpretation on the part of PRTG. Needless to say 
we’re watching these closely, plus all of our other GPS NTP servers. I am 
dumping PCAP from the corresponding firewalls to a database, since the volume 
of NTP traffic is low.

 -mel

On May 10, 2024, at 3:09 PM, Raymond Burkholder  wrote:

 We had some Meinberg's which did something similar but different some time 
ago.  NTP was out of sync with GPS.  We had a CheckMk instance which detected 
drift between sources in our network.  Turns out there was one or more configs 
in the Meinberg that enable failover from one source to another, and to ensure 
GPS and NTP are working together rather than independently.

Maybe your TimeMachines have similar config variabilities.

On 2024-05-10 14:29, Mel Beckman wrote:


We just had two TM1000 TimeMachine brand GPS NTP servers lose clock sync at the 
same time, in two different cities (LA and Santa Barbara). The  outage lasted 
about five minutes, during which the NTP servers were responding, but with time 
that was 1900 seconds out of sync. The devices showed satellite lock on 8 birds 
(not all the same ones). I've never seen this behavior before with years of NTP 
clock experience.

It could be that these inexpensive NTP servers aren't very selective about 
bogus inputs, as I would have expected them to lose synch in the event of a GPS 
signal failure. Instead they produced garbage. Our PRTG NTP monitor logged the 
problem this way:

  
Sensor SNTP (SNTP) ***
Device  10.2.10.90-TimeMachine NTP server (10.2.10.90)  
New Status at 5/10/2024 12:49:52 PM (Pacific Standard Time):
Down
Last Message:
The target server did not return a valid time. To resolve this issue, use a 
packet analyzing tool and do a trace of the NTP packets to check if all fields 
are correctly populated. (code: PE085)


From: NANOG 
<mailto:nanog-bounces+mel=beckman@nanog.org>
 on behalf of John Curran <mailto:jcur...@arin.net>
Sent: Friday, May 10, 2024 10:54 AM
To: NANOG <mailto:nanog@nanog.org>
Subject: NOAA Space Weather Prediction Center issued a Severe (G4) Geomagnetic 
Storm Watch


SWPC Issues Its First G4 Watch Since 2005 | NOAA / NWS Space Weather Prediction 
Center<https://www.swpc.noaa.gov/news/swpc-issues-its-first-g4-watch-2005>
swpc.noaa.gov<https://www.swpc.noaa.gov/news/swpc-issues-its-first-g4-watch-2005>
[favicon.ico]<https://www.swpc.noaa.gov/news/swpc-issues-its-first-g4-watch-2005>

"Multiple CMEs erupted associated with flare activity from Region 3664 on 07-09 
May. These CMEs are expected to merge with potential arrival expected by early 
May 11 on the UTC day.”

(Low but distinct possibility of effects to radio and transmission systems)

FYI,
/John

John Curran
President and CEO
American Registry for Internet Numbers

Re: NOAA Space Weather Prediction Center issued a Severe (G4) Geomagnetic Storm Watch

[Mel Beckman]

We just had two TM1000 TimeMachine brand GPS NTP servers lose clock sync at the 
same time, in two different cities (LA and Santa Barbara). The  outage lasted 
about five minutes, during which the NTP servers were responding, but with time 
that was 1900 seconds out of sync. The devices showed satellite lock on 8 birds 
(not all the same ones). I've never seen this behavior before with years of NTP 
clock experience.

It could be that these inexpensive NTP servers aren't very selective about 
bogus inputs, as I would have expected them to lose synch in the event of a GPS 
signal failure. Instead they produced garbage. Our PRTG NTP monitor logged the 
problem this way:

  
Sensor SNTP (SNTP) ***
Device  10.2.10.90-TimeMachine NTP server (10.2.10.90)  
New Status at 5/10/2024 12:49:52 PM (Pacific Standard Time):
Down
Last Message:
The target server did not return a valid time. To resolve this issue, use a 
packet analyzing tool and do a trace of the NTP packets to check if all fields 
are correctly populated. (code: PE085)


From: NANOG  on behalf of John Curran 

Sent: Friday, May 10, 2024 10:54 AM
To: NANOG 
Subject: NOAA Space Weather Prediction Center issued a Severe (G4) Geomagnetic 
Storm Watch



SWPC Issues Its First G4 Watch Since 2005 | NOAA / NWS Space Weather Prediction 
Center
swpc.noaa.gov
[favicon.ico]

"Multiple CMEs erupted associated with flare activity from Region 3664 on 07-09 
May. These CMEs are expected to merge with potential arrival expected by early 
May 11 on the UTC day.”

(Low but distinct possibility of effects to radio and transmission systems)

FYI,
/John

John Curran
President and CEO
American Registry for Internet Numbers

Re: Opengear alternatives that support 5g?

[Mel Beckman]

Mike,

Both the Cradlepoint and Teltonica 5G devices are ~$600, even more than the 
PepLink. I’ll compare features, but at first blush the Teltonica at least seems 
to have no VPN support.


-mel via cell

On Apr 26, 2024, at 11:41 PM, Mike Lyon  wrote:


Mel,

My apologies, i confused one mikrotik with another model. You are correct.

I would also check out CradlePoint and Teltonika as well.

Cheers,
Mike

On Apr 26, 2024, at 23:06, Mel Beckman  wrote:


Mike,

Thanks for that info. Alas, I’m not seeing any Mikrotik 5G devices cheaper than 
a ~$500  Peplink. Am I misunderstanding your suggested solution?

 -mel

On Apr 26, 2024, at 9:50 PM, Mike Lyon  wrote:


Peplink is nice, but there are cheaper options: Mikrotik-dot-com

Then for cellular service, sign up for an IOT with an IOT MVNO that bills usage 
based (and can also offer you a static, public, IP address AND will also allow 
you to build a VPN across all of your devices) such as SimBase:  simbase-dot-com



Cheers,
Mike

On Apr 26, 2024, at 21:37, Mel Beckman  wrote:

 I’ve been loooking at the $600 Peplink MAX BR1-MINI (HW3) industrial 5G 
router. It has a 1x embedded 5G modem (Verizon, AT&T, T-Mobile, and FirstNet). 
three GigE ports, four antenna connectors, and comes with an stick antenna set 
and AC PS.  It uses a nanoSIM. Yes, it’s a pure IP router with no knowledge of 
serial protocols. So I would just put an air console behind it to get to my 
serial ports. I’m still evaluating 5G plans, and Verizon just offered an 
amazing $15 per month unlimited data deal, but it seems to have a 50 gig limit 
before you get to throttling. That might not matter at all with serial traffic 
though.

We've been using the Netgear 4G cellular router, but that’s becoming 
increasingly unreliable. The NG has a nailed up IPsec VPN tunnel, obviating the 
need for a static IP, and the keepalive traffic is low enough that it doesn’t 
cost us much on the 4G network. I’m hoping 5G will be even cheaper and faster.

I’d love to see if anybody found anything better before I spring for a Peplink 
test unit.


 -mel

On Apr 26, 2024, at 9:45 AM, Warren Kumari  wrote:






On Fri, Apr 26, 2024 at 12:43 AM, Saku Ytti mailto:s...@ytti.fi>> 
wrote:

On Fri, 26 Apr 2024 at 03:11, David H 
mailto:ispcoloh...@gmail.com>> wrote:

Curious if anyone has particular hardware they like for OOB / serial 
management, similar to OpenGear, but preferably with 5G support, maybe even 
T-Mobile support? It’s becoming increasingly difficult to get static IP 4g 
machine accounts out of Verizon, and the added speed would be nice too. Or do 
you separate the serial from the access device (cell+firewall, etc.)?

You could get a 5G Catalyst with an async NIM or SM.

But I think you're setting up yourself for unnecessary costs and failures by 
designing your OOB to require static IP. You could design it so that the OOB 
spokes dial-in to the central OOB hub, and the OOB hub doesn't care what IP 
they come from, using certificates or PSK for identity, instead of IP.


Yup, I agree — but that simply rewrites the question to be:
"Curious if anyone has particular hardware they like for OOB / serial 
management, similar to OpenGear, but preferably with 5G support, which can be a 
spoke that dials in to the central OOB hub, and the OOB hub doesn't care what 
IP they come from, using certificates or PSK for identity, instead of IP."

I've been on the same quest, and I have some additional requests / features. 
Ideally it:

1: would be small - my particular use-case is for a "traveling rack", and so 0U 
is preferred.

2: would be fairly cheap.

3: would not be a Raspberry-Pi, a USB hub and USB-to-serial cables. We tried 
that for a while, and it was clunky — the SD card died a few times (and jumped 
out entirely once!), people kept futzing with the OS and fighting over which 
console software to use, installing other packages, etc.

4: support modern SSH clients (it seems like you shouldn't have to say this, 
but… )

5: actually be designed as a termserver - the current thing we are using 
doesn't really understand terminals, and so we need to use 'socat 
-,raw,echo=0,escape=0x1d TCP::' to get things like 
tab-completion and "up-arrow for last command" to work.

6: support logging of serial (e.g crash-messages) to some sort of log / buffer 
/ similar (it's useful to be able to see what a device barfed all over the 
console when it crashes.


The Get Console Airconsole TS series meets many of these requirements, but it 
doesn't do #6. It also doesn't really feel like they have been updating / 
maintaining these.

Yes, I fully acknowledge that #3 falls into the "Doctor, Doctor, it hurts when 
I do this" camp, but, well…

W

Re: Opengear alternatives that support 5g?

[Mel Beckman]

Quite often I’m looking for OOBM at antenna sites or in remote DCs where there 
is no Plan B carrier. Cellular has always been the goto choice for this, but we 
keep getting pushed out of contracts by technology upgrades. 2g, then 3g, and  
next 4g LTE are being deprecated.

The main reason for network shutdowns is that the carriers have limited 
spectrum available for expansion. To deliver faster, more cost effective data 
service to customers, carriers must re-use existing spectrum licenses with 
newer, more efficient cellular technology. Old 2G/3G infrastructure makes way 
for new networks, and older cellular devices must be retired. 4g may have a 
decade left before complete absence, but its footprint is already shrinking 
where 5G is available.

I’ve seen this first hand with 4g cellular alarm circuits: suddenly they get 
less reliable or fail completely, and the reason always turns out to be 
degraded RSSI due to 5G deployment.

So 5G is imperative for cellular OOBM, hence the hunt for COTS drop-in 
replacements that won’t break the bank. Upgrading, for example, 100 antenna 
sites is also a major truck roll cost, so we want to get it right the first 
time.  Physical space and power limitations usually rule out 1U rackmount 
refurb Cisco terminal servers, which is why we need 0U gear. Yes, I can cobble 
together a raspberry pi and some hats and cables and dingles and dangles and 
make a science fair solution. But I need something that is commercially 
supported, won’t have me scratching my head later about what version of the 
Ubuntu is going to work, and won’t randomly fry its electronics during a power 
surge.

It’s looking like that solution is firmly priced at ~$500 today. 

 -mel

> On Apr 27, 2024, at 4:59 AM, Mark Tinka  wrote:
> 
> 
> 
>> On 4/27/24 07:56, Saku Ytti wrote:
>> 
>> 
>>  For me Cisco is great here, because it's something an organisation
>> already knows how to source, turn-up, upgrade, troubleshoot, maintain.
>> And you get a broad set of features you might want, IPSEC, DMVPN, BGP,
>> ISIS, and so forth.
> 
> I tend to agree.
> 
> Cisco do this very well, and if you are really low on cash and okay with 
> acquiring these on the cheap, the open market has tons of deals and options 
> from Cisco that have matured over the decades.
> 
> 
> 
>> I keep wondering why everyone is so focused on OOB hardware cost, when
>> in my experience the ethernet connection is ~200-300USD (150USD can be
>> just xconn) MRC. So in 10 years, you'll pay 24k to 36k just for the
>> OOB WAN, masking the hardware price. And 10years, to me, doesn't sound
>> even particularly long a time for a console setup.
> 
> Is a 10Mbps DIA link going for US$200 - US$300 MRC nowadays, excluding the 
> x-connect? I'd have though it's now in US$100 range at the very worst.
> 
> Or are you looking at an OoB link of more than 10Mbps?
> 
> Mark.

Re: Opengear alternatives that support 5g?

[Mel Beckman]

Mike,

Thanks for that info. Alas, I’m not seeing any Mikrotik 5G devices cheaper than 
a ~$500  Peplink. Am I misunderstanding your suggested solution?

 -mel

On Apr 26, 2024, at 9:50 PM, Mike Lyon  wrote:


Peplink is nice, but there are cheaper options: Mikrotik-dot-com

Then for cellular service, sign up for an IOT with an IOT MVNO that bills usage 
based (and can also offer you a static, public, IP address AND will also allow 
you to build a VPN across all of your devices) such as SimBase:  simbase-dot-com



Cheers,
Mike

On Apr 26, 2024, at 21:37, Mel Beckman  wrote:

 I’ve been loooking at the $600 Peplink MAX BR1-MINI (HW3) industrial 5G 
router. It has a 1x embedded 5G modem (Verizon, AT&T, T-Mobile, and FirstNet). 
three GigE ports, four antenna connectors, and comes with an stick antenna set 
and AC PS.  It uses a nanoSIM. Yes, it’s a pure IP router with no knowledge of 
serial protocols. So I would just put an air console behind it to get to my 
serial ports. I’m still evaluating 5G plans, and Verizon just offered an 
amazing $15 per month unlimited data deal, but it seems to have a 50 gig limit 
before you get to throttling. That might not matter at all with serial traffic 
though.

We've been using the Netgear 4G cellular router, but that’s becoming 
increasingly unreliable. The NG has a nailed up IPsec VPN tunnel, obviating the 
need for a static IP, and the keepalive traffic is low enough that it doesn’t 
cost us much on the 4G network. I’m hoping 5G will be even cheaper and faster.

I’d love to see if anybody found anything better before I spring for a Peplink 
test unit.


 -mel

On Apr 26, 2024, at 9:45 AM, Warren Kumari  wrote:






On Fri, Apr 26, 2024 at 12:43 AM, Saku Ytti mailto:s...@ytti.fi>> 
wrote:

On Fri, 26 Apr 2024 at 03:11, David H 
mailto:ispcoloh...@gmail.com>> wrote:

Curious if anyone has particular hardware they like for OOB / serial 
management, similar to OpenGear, but preferably with 5G support, maybe even 
T-Mobile support? It’s becoming increasingly difficult to get static IP 4g 
machine accounts out of Verizon, and the added speed would be nice too. Or do 
you separate the serial from the access device (cell+firewall, etc.)?

You could get a 5G Catalyst with an async NIM or SM.

But I think you're setting up yourself for unnecessary costs and failures by 
designing your OOB to require static IP. You could design it so that the OOB 
spokes dial-in to the central OOB hub, and the OOB hub doesn't care what IP 
they come from, using certificates or PSK for identity, instead of IP.


Yup, I agree — but that simply rewrites the question to be:
"Curious if anyone has particular hardware they like for OOB / serial 
management, similar to OpenGear, but preferably with 5G support, which can be a 
spoke that dials in to the central OOB hub, and the OOB hub doesn't care what 
IP they come from, using certificates or PSK for identity, instead of IP."

I've been on the same quest, and I have some additional requests / features. 
Ideally it:

1: would be small - my particular use-case is for a "traveling rack", and so 0U 
is preferred.

2: would be fairly cheap.

3: would not be a Raspberry-Pi, a USB hub and USB-to-serial cables. We tried 
that for a while, and it was clunky — the SD card died a few times (and jumped 
out entirely once!), people kept futzing with the OS and fighting over which 
console software to use, installing other packages, etc.

4: support modern SSH clients (it seems like you shouldn't have to say this, 
but… )

5: actually be designed as a termserver - the current thing we are using 
doesn't really understand terminals, and so we need to use 'socat 
-,raw,echo=0,escape=0x1d TCP::' to get things like 
tab-completion and "up-arrow for last command" to work.

6: support logging of serial (e.g crash-messages) to some sort of log / buffer 
/ similar (it's useful to be able to see what a device barfed all over the 
console when it crashes.


The Get Console Airconsole TS series meets many of these requirements, but it 
doesn't do #6. It also doesn't really feel like they have been updating / 
maintaining these.

Yes, I fully acknowledge that #3 falls into the "Doctor, Doctor, it hurts when 
I do this" camp, but, well…

W




--
++ytti

Re: Opengear alternatives that support 5g?

[Mel Beckman]

I’ve been loooking at the $600 Peplink MAX BR1-MINI (HW3) industrial 5G router. 
It has a 1x embedded 5G modem (Verizon, AT&T, T-Mobile, and FirstNet). three 
GigE ports, four antenna connectors, and comes with an stick antenna set and AC 
PS.  It uses a nanoSIM. Yes, it’s a pure IP router with no knowledge of serial 
protocols. So I would just put an air console behind it to get to my serial 
ports. I’m still evaluating 5G plans, and Verizon just offered an amazing $15 
per month unlimited data deal, but it seems to have a 50 gig limit before you 
get to throttling. That might not matter at all with serial traffic though.

We've been using the Netgear 4G cellular router, but that’s becoming 
increasingly unreliable. The NG has a nailed up IPsec VPN tunnel, obviating the 
need for a static IP, and the keepalive traffic is low enough that it doesn’t 
cost us much on the 4G network. I’m hoping 5G will be even cheaper and faster.

I’d love to see if anybody found anything better before I spring for a Peplink 
test unit.


 -mel

On Apr 26, 2024, at 9:45 AM, Warren Kumari  wrote:






On Fri, Apr 26, 2024 at 12:43 AM, Saku Ytti mailto:s...@ytti.fi>> 
wrote:

On Fri, 26 Apr 2024 at 03:11, David H 
mailto:ispcoloh...@gmail.com>> wrote:

Curious if anyone has particular hardware they like for OOB / serial 
management, similar to OpenGear, but preferably with 5G support, maybe even 
T-Mobile support? It’s becoming increasingly difficult to get static IP 4g 
machine accounts out of Verizon, and the added speed would be nice too. Or do 
you separate the serial from the access device (cell+firewall, etc.)?

You could get a 5G Catalyst with an async NIM or SM.

But I think you're setting up yourself for unnecessary costs and failures by 
designing your OOB to require static IP. You could design it so that the OOB 
spokes dial-in to the central OOB hub, and the OOB hub doesn't care what IP 
they come from, using certificates or PSK for identity, instead of IP.


Yup, I agree — but that simply rewrites the question to be:
"Curious if anyone has particular hardware they like for OOB / serial 
management, similar to OpenGear, but preferably with 5G support, which can be a 
spoke that dials in to the central OOB hub, and the OOB hub doesn't care what 
IP they come from, using certificates or PSK for identity, instead of IP."

I've been on the same quest, and I have some additional requests / features. 
Ideally it:

1: would be small - my particular use-case is for a "traveling rack", and so 0U 
is preferred.

2: would be fairly cheap.

3: would not be a Raspberry-Pi, a USB hub and USB-to-serial cables. We tried 
that for a while, and it was clunky — the SD card died a few times (and jumped 
out entirely once!), people kept futzing with the OS and fighting over which 
console software to use, installing other packages, etc.

4: support modern SSH clients (it seems like you shouldn't have to say this, 
but… )

5: actually be designed as a termserver - the current thing we are using 
doesn't really understand terminals, and so we need to use 'socat 
-,raw,echo=0,escape=0x1d TCP::' to get things like 
tab-completion and "up-arrow for last command" to work.

6: support logging of serial (e.g crash-messages) to some sort of log / buffer 
/ similar (it's useful to be able to see what a device barfed all over the 
console when it crashes.


The Get Console Airconsole TS series meets many of these requirements, but it 
doesn't do #6. It also doesn't really feel like they have been updating / 
maintaining these.

Yes, I fully acknowledge that #3 falls into the "Doctor, Doctor, it hurts when 
I do this" camp, but, well…

W




--
++ytti

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[Mel Beckman]

Bill is absolutely correct. The spammers lost their case because they were 
demonstrably spammers. We’ve had accidental black hole cases with *US* 
providers that removed the block once they received a C&D. If they don’t have 
iron clad proof in hand. (More than just a few complaints and no traffic 
analysis), it’s just the least risky response.

That doesn’t work well with overseas providers, though, because they’re 
essentially immune to U.S. litigation unless the plaintiff has deep pockets.

 -mel

On Apr 22, 2024, at 5:21 PM, William Herrin  wrote:

On Mon, Apr 22, 2024 at 5:07 PM John R. Levine  wrote:
a complaint would have to show that the
blocking was malicious rather than merited or accidental.  In this case it
seems probably accidental, but for all I know there might have been bad
traffic to merit a block.

Hi John,

I'll try not to belabor it, but accidental that isn't corrected upon
formal legal notification becomes negligent and negligent has more or
less the same legal status as malicious.

The spammers lost because the networks published a terms of use
document that the spammers unambiguously violated. Even though it
interfered with the spammer's business, the block was merited so the
preponderance of the evidence fell in favor of the service provider.

Regards,
Bill Herrin


--
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

[Mel Beckman]

I notice from MXToolbox.com that your domain’s IP address is on the 
UCEPROTECTL3 blacklist.

This is a notoriously evil blacklist that charges people for removal. This may 
be why Spectrum is blackholing your domain. Most respectable ISPs won’t use it. 
But Spectrum…

There is no delisting procedure without making a “donation” to the UCEPROTECT3 
black sparrow account. They’re famous for blacklisting large swaths of IP 
addresses that catch up innocent parties that have never spammed a flea.

-mel

On Apr 22, 2024, at 4:51 AM, Validin Axon  wrote:


Looking for some help/advice. Spectrum is sinkholing my company's domain, 
validin[.]com, to 127.0.0.54. The sinkhole responses come from their recursive 
DNS servers, 209.18.47.61 and 209.18.47.62, which are defaults for and in use 
by many of their customers and are only reachable from within the Spectrum 
network. I've had 4 people over the last week (think: customers, prospects, 
etc) who use Charter/Spectrum tell me that they have difficulty accessing my 
website as a result of this sinkhole behavior. This behavior is causing 
reputational harm to my company.

I've personally confirmed this behavior from the Spectrum network (I am also a 
customer) using dig to test their DNS servers:
```
$ dig +short @209.18.47.61 validin.com
127.0.0.54
$ dig +short @209.18.47.62 validin.com
127.0.0.54
```
 Using Cloudflare/Google/etc works correctly:
```
$ dig +short @1.1.1.1 validin.com
137.184.54.107
157.245.112.183
$ dig +short @8.8.8.8 validin.com
157.245.112.183
137.184.54.107
```

I suspect my domain was blocklisted last year when a threat researcher included 
my domain name in a blog post about a threat they were investigating and cited 
my company as the source for their data. Someone scraped that post, and my 
company's domain was accidentally added to two Alient Vault OTX pulses and at 
least one collection on Virus Total. I removed the domain via false positive 
reporting from everything I could. However, it appears that being added to 
Spectrum's DNS sinkhole list is effectively permanent and there's no clear path 
for false positive remediation.

I've tried the official Spectrum support lines for months to no avail, and 
recently tried reaching out on Twitter, but have had no success there either. 
I'm clearly not able to find the right people through these routes, as none of 
the people I reach understand the difference between a DNS sinkhole and an IP 
block list and don't appear to be aware that DNS blocklisting is a separate 
behavior from their opt-in content filtering via Security Shield.

So, if someone could please help me find the team or individual responsible for 
Spectrum's DNS sinkhole behavior, I would be exceptionally grateful. :-)

As I mentioned, this is causing reputation harm, so switching my own DNS 
servers is not sufficient. People who need to reach me, can't. So, I would 
appreciate any other help or advice you have,

Kenneth

Re: Unimus as NCM (Network Configuration Management) Tool

[Mel Beckman]

We use both Unumus and ManageEngine. Neither covers all device models, or all 
firmware versions of all devices, so we have to use both products to get 
complete device coverage. Scaling depends on host performance, so for large 
device populations you may want to assign different SCM instances to particular 
subgroups.

-mel via cell

On Apr 3, 2024, at 11:28 PM, Mike Lyon  wrote:


I use it for config backups, diffs, etc. Love it.

Theres others such as Rancid but im not sure if it works on anything other than 
Vendor C.

-Mike

On Apr 3, 2024, at 23:16, Shahid Shafi  wrote:


Hi Network Experts,

Is anyone using Unimus as your main NCM tool in production? I am looking at an 
NCM tool that can scale upto 10,000 to 15,000 Network Devices. Do you recommend 
any other solution? The solution should atleast able to support network config 
backups, diffs, and basic network auditing features.

https://unimus.net/

thanks
Shahid

Re: Without further comment:

[Mel Beckman]

Well, Billie goes both ways :)

Sorry, couldn’t resist.

 -mel

> On Mar 30, 2024, at 8:47 AM, Jay Hennigan  wrote:
> 
> On 3/30/24 08:22, William Herrin wrote:
>>> On Sat, Mar 30, 2024 at 7:38 AM Josh Luthman
>>>  wrote:
>>> How do you know the poster's gender??
>> Howdy,
>> As Josh is an uncommon female name, I'm going to play the odds and say
>> that like Bill and I, you're male. Am I mistaken?
> 
> Mike Godwin is also male. The way things are going I wouldn't be surprised to 
> hear from his fan club soon.
> 
> --
> Jay Hennigan - j...@west.net
> Network Engineering - CCIE #7880
> 503 897-8550 - WB6RDV
> 

Re: Network chatter generator

[Mel Beckman]

Keysight’s Ixea Line of traffic generators with concurrent monitoring are 
industrial grade tools for certification testing. I’ve used them to simulate 
thousands of WiFi users to validate an 400 node access point deployment at a 
major airport. Not cheap but has all the knobs and dials you’re looking for. 
You can also rent them.

 -mel

On Feb 24, 2024, at 8:43 AM, Jesse DuPont  wrote:

 I believe you can do most of what you want using a Mikrotik and its Traffic 
Generator. Packet templates can be crafted mimic any of the popular protocols 
(L2, L3, L4), at least at the header level, with less flexibility on the 
payload legitimacy.

On 2/23/24 10:33 AM, Brandon Martin wrote:
Before I go to the trouble of making one myself, does anybody happen to know of 
a pre-canned program to generate realistic and scalable amounts of 
broadcast/broad-multicast network background "chatter" seen on typical consumer 
and business networks?  This would be things like lots of ARP traffic to/from 
various sources/destinations within a subnet, SSDP, MDNS-SD, SMB browser 
traffic, DHCP requests, etc.?

Ideally, said tool would have knobs to control the amount of traffic and 
whether a given type of traffic is present.

This is mostly for torture testing "IoT" type devices by exposing them to lots 
of diverse, essentially nonsense traffic that they're likely to see in a real 
environment.

--
Brandon Martin

Re: Networks ignoring prepends?

[Mel Beckman]

Prepend contraction is becoming more common. You can’t really stop providers 
from doing it, and it reduces BGP table size, which I’ve heard as a secondary 
rationale. I’d love to see the statistics on that though.

BGP Communities seem to be the only alternative, and that limits your 
engineering reach to mostly immediate peers.

Another problem is providers that hide multiple router hops inside MPLS, which 
appears as a single ip hop in traceroutes, making it impossible to know the 
truth path geographically. 

The Internet is lying to itself, and that’s not a situation that can persist 
forever.

-mel via cell

> On Jan 22, 2024, at 4:52 AM, William Herrin  wrote:
> 
> Howdy,
> 
> Does anyone have suggestions for dealing with networks who ignore my
> BGP route prepends?
> 
> I have a primary ingress with no prepends and then several distant
> backups with multiple prepends of my own AS number. My intention, of
> course, is that folks take the short path to me whenever it's
> reachable.
> 
> A few years ago, Comcast decided it would prefer the 5000 mile,
> five-prepend loop to the short 10 mile path. I was able to cure that
> with a community telling my ISP along that path to not advertise my
> route to Comcast. Today it's Centurylink. Same story; they'd rather
> send the packets 5000 miles to the other coast and back than 10 miles
> across town. I know they have the correct route because when I
> withdraw the distant ones entirely, they see and use it. But this time
> it's not just one path; they prefer any other path except the one I
> want them to use. And Centurylink is not a peer of those ISPs, so
> there doesn't appear to be any community I can use to tell them not to
> use the route.
> 
> I hate to litter the table with a batch of more-specifics that only
> originate from the short, preferred link but I'm at a loss as to what
> else to do.
> 
> Advice would be most welcome.
> 
> Regards,
> Bill Herrin
> 
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/

Re: "Hypothetical" Datacenter Overheating

[Mel Beckman]

My sarcasm generator is clearly set incorrectly :)

 -mel

> On Jan 15, 2024, at 10:33 AM, Jay Hennigan  wrote:
> 
> On 1/15/24 07:21, Mel Beckman wrote:
>> Easy. Climate change. Lol!
> 
> It was -8°F in Chicago yesterday.
> 
>>>> On Jan 15, 2024, at 7:17 AM, sro...@ronan-online.com wrote:
>>> 
>>> 
>>> I’m more interested in how you lose six chillers all at once.
> 
> 
> -- 
> Jay Hennigan - j...@west.net
> Network Engineering - CCIE #7880
> 503 897-8550 - WB6RDV
> 

Re: "Hypothetical" Datacenter Overheating

[Mel Beckman]

Easy. Climate change. Lol!

 -mel

On Jan 15, 2024, at 7:17 AM, sro...@ronan-online.com wrote:


I’m more interested in how you lose six chillers all at once.

Shane

On Jan 15, 2024, at 9:11 AM, Mike Hammett  wrote:


Let's say that hypothetically, a datacenter you're in had a cooling failure and 
escalated to an average of 120 degrees before mitigations started having an 
effect. What are normal QA procedures on your behalf? What is the facility 
likely to be doing? What  should be expected in the aftermath?



-
Mike Hammett
Intelligent Computing Solutions
[http://www.ics-il.com/images/fbicon.png][http://www.ics-il.com/images/googleicon.png][http://www.ics-il.com/images/linkedinicon.png][http://www.ics-il.com/images/twittericon.png]
Midwest Internet Exchange
[http://www.ics-il.com/images/fbicon.png][http://www.ics-il.com/images/linkedinicon.png][http://www.ics-il.com/images/twittericon.png]
The Brothers WISP
[http://www.ics-il.com/images/fbicon.png][http://www.ics-il.com/images/youtubeicon.png]

Re: sigs wanted for a response to the fcc's NOI for faster broadband speeds

[Mel Beckman]

bufferbloat is rarely fatal

LOL! I know one person taht may disagree with that :)

 -mel

On Dec 1, 2023, at 9:41 AM, Tom Mitchell  wrote:


Not sure we need the FCC telling us how to build products or run networks.  
Seat belts are life-or-death, but bufferbloat is rarely fatal ;-)  Let it be a 
point of differentiation.

-- Tom


On Thu, Nov 30, 2023 at 4:56 PM Dave Taht 
mailto:dave.t...@gmail.com>> wrote:
Over here:

https://docs.google.com/document/d/19ADByjakzQXCj9Re_pUvrb5Qe5OK-QmhlYRLMBY4vH4/edit

Us bufferbloat folk have been putting together a response to the FCC's
NOI (notice of inquiry) asking for feedback as to increasing the
broadband speeds beyond 100/20 Mbit.

"Calls for further bandwidth increases are analogous to calling for
cars to have top speeds of 100, 200, or 500 miles per hour. Without
calling also for better airbags, bumpers, brakes, or steering wheels,
(or roads designed to minimize travel delay), these initiatives will
fail (and are failing) to meet the needs of present and future users
of the internet."

Comments (and cites) welcomed also! The text is still somewhat in flux...


--
:( My old R&D campus is up for sale: https://tinyurl.com/yurtlab
Dave Täht CSO, LibreQos

Re: ipv6 address management - documentation

[Mel Beckman]

I second Netbox, which has detailed IPv4/6 IPAM plus many other features:


IP Address Management - NetBox 
Documentation
demo.netbox.dev
[favicon.png]


 -mel

On Nov 16, 2023, at 10:31 AM, Jesse DuPont  
wrote:

 phpIPAM for the win. NIPAP is effective, if basic. I've heard of lots of 
people who like Netbox.

On 11/16/23 11:12 AM, Niels Bakker wrote:
* aar...@gvtc.com (Aaron Gould) [Thu 16 Nov 2023, 19:04 
CET]:
For years I've used an MS Excel spreadsheet to manage my IPv4  addresses.  IPv6 
is going to be maddening to manage in a spreadsheet.  What does everyone use 
for their IPv6 address prefix management and documentation?  Are there open 
source tools/apps for this?

The first three hits for "open source ipam" on a search engine are:

- phpipam.net/
- spritelink.github.io/NIPAP/
- github.com/netbox-community/netbox

I'd pick the last option, or possibly Nautobot.

You may want to scroll through https://github.com/topics/ipam for more options.


-- Niels.

Re: Appropriate venue to find out about the state of art of spear phishing defense?

[Mel Beckman]

We use KnowBe4.com's user training. That's really the only way you can fight 
this, since its a human problem, not a technical one. These guys provide fully 
automated, AI based (well, who knows what that means) simulated phishing 
attacks, largely to give users real-world practical experience detecting and 
fending off attacks. You get a report card on each users to, so you know where 
the weaknesses are in your staff knowledge. Their training regimen includes 
some pretty good self-guided instructional videos.

DMARC, SPF, digitally-signed emails, encryption, none of that matters if a user 
can be tricked into letting the crooks in the front door.

 -mel

From: NANOG  on behalf of Michael 
Thomas 
Sent: Monday, November 13, 2023 11:40 AM
To: nanog@nanog.org 
Subject: Appropriate venue to find out about the state of art of spear phishing 
defense?


I know this is only tangentially relevant to nanog, but I'm curious if
anybody knows where I can ask what orgs do to combat spear phishing?
Spear phishing doesn't require that you deploy DMARC since you can know
your own policy even if you aren't comfortable publishing it to the world.

tia, Mike

Re: Amir Golestan sentenced to 5 years in prison for IP theft scheme

[Mel Beckman via NANOG]

It _is_ interesting that this was an IPv4 black market crime. Think of all the 
smaller black marketeers dealing in IPv4 trafficking. 

-mel via cell

> On Oct 17, 2023, at 10:09 PM, Owen DeLong  wrote:
> 
> Sadly, probably not. AUP violations are a civil, not a criminal matter. 
> Fraud,  OTOH, is a criminal matter.
> 
> In the US, at least, people don’t go to prison for civil matters.
> 
> Owen
> 
> 
>> On Oct 17, 2023, at 19:19, Mel Beckman  wrote:
>> 
>> So ARIN DB spammers should get at least a year or two, right? :-)
>> 
>> -mel via cell
>> 
>>>> On Oct 17, 2023, at 7:05 PM, goemon--- via NANOG  wrote:
>>> 
>>> https://krebsonsecurity.com/2023/10/tech-ceo-sentenced-to-5-years-in-ip-address-scheme/
>>> 
>>> And a statement from ARIN: 
>>> https://www.arin.net/blog/2023/10/16/micfo-golestan-sentencing/
> 

Re: Amir Golestan sentenced to 5 years in prison for IP theft scheme

[Mel Beckman]

So ARIN DB spammers should get at least a year or two, right? :-)

-mel via cell

> On Oct 17, 2023, at 7:05 PM, goemon--- via NANOG  wrote:
> 
> https://krebsonsecurity.com/2023/10/tech-ceo-sentenced-to-5-years-in-ip-address-scheme/
> 
> And a statement from ARIN: 
> https://www.arin.net/blog/2023/10/16/micfo-golestan-sentencing/

Re: jon postel

[Mel Beckman]

Jon was very kind to me when I was a wet-behind-the-ears network engineer. He 
once showed me around ISI and gave me an entire shelf of down-version Cisco 
manuals. I had a Cisco 2500 peering with ISI in a maintenance closet in the ISI 
parking structure. A single T1 let me run a few dozen Livingston modems for my 
nascent Jet.net ISP.

 -mel

On Oct 16, 2023, at 2:44 PM, Collider  wrote:


is it candle time?


Le 16 octobre 2023 21:13:50 UTC, Randy Bush  a écrit :

25 years ago, jon postel died.  we stand on the shoulders of jon and
others, a number of whom died in october.  not a cheering month for
old timers.

randy

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: ARIN whois contact abuse from ipv4depot aka Silicon Desert International Inc

[Mel Beckman]

Laura,

just a couple of weeks ago, I reported and ARIN abuse here on NANOG, and ARIN 
responded immediately, contacting the offender and getting them to stop. The 
system works, and ARIN has the power to deter repeat offenders.

 -mel

> On Oct 12, 2023, at 10:01 AM, Laura Smith via NANOG  wrote:
> 
> Honestly Mike I don't think they care.
> 
> I mean, most (all ?) of the registries still can't be bothered to validate 
> the information the resource holders post to the database.  Last time I 
> asked, e.g. RIPE about it, they basically said "not my problem guv" , pointed 
> me to some policy document that said members should provide correct details 
> and well, that was about it.
> 
> So if they don't do that, then what hope is there for them doing something 
> about the harvesters ?
> 
> 
> --- Original Message ---
>> On Thursday, October 12th, 2023 at 17:08, Mike Hammett  
>> wrote:
>> 
>> 
>> Do we know if the organizations with key Internet resources (ARIN, RIPE, 
>> PeeringDB, etc.) have any honeypots in their arsenal? Obviously, publicly 
>> knowing about it kind of defeats the purpose of it, but that might be a way 
>> to help be proactive - make fake entries with unique contact information to 
>> catch those harvesting.
>> 
>> 
>> 
>> -----
>> Mike Hammett
>> Intelligent Computing Solutions
>> http://www.ics-il.com
>> 
>> Midwest-IX
>> http://www.midwest-ix.com
>> 
>> 
>> From: "Mel Beckman" 
>> To: "Tom Beecher" 
>> Cc: "nanog@nanog.org list" 
>> Sent: Thursday, October 12, 2023 11:01:20 AM
>> Subject: Re: ARIN whois contact abuse from ipv4depot aka Silicon Desert 
>> International Inc
>> 
>> Tom,
>> When an ARIN member violates their agreement and spams from ARIN’s 
>> databases, it’s not just an “Internet is fertile ground” deal. It’s a 
>> betrayal of a legal trust, one that demands accountability. I’m quite happy 
>> that ARIN promptly responds to these abuses, and gets results. That only 
>> works if victims report spam and compare notes. Let the “fertile ground” be 
>> elsewhere!
>> 
>>  -mel beckman
>> 
>> 
>>>> On Oct 12, 2023, at 8:49 AM, Tom Beecher  wrote:
>>> 
>>> 
>>> 
>>>> It's ridiculous that they resort to scraping public lists and DBs to try 
>>>> and achieve what they're attempting to do.
>>> 
>>> 
>>> Everyone is always looking for information they can use to advance some 
>>> agenda or purpose. The internet is fertile ground for that. Always has 
>>> been, always will be. 
>>> 
>>> Not taking shots at anyone here, but I am boggled why this is a common 
>>> public complaint. Block the sender and move on. 
>>> 
>>>> On Wed, Oct 11, 2023 at 7:56 PM Peter Potvin via NANOG  
>>>> wrote:
>>> 
>>>> Definitely have received this same spam multiple times and so have a few 
>>>> others I know. It's ridiculous that they resort to scraping public lists 
>>>> and DBs to try and achieve what they're attempting to do.
>>>> Regards,Peter Potvin | Executive Director
>>>> --
>>>> Accuris Technologies Ltd.
>>>> 
>>>> 
>>>> On Wed, Oct 11, 2023 at 7:52 PM Eric Kuhnke  wrote:
>>>> 
>>>>> Is anyone else receiving spam from this organization? Based on the 
>>>>> contents of the cold solicitations they are sending us, and the addresses 
>>>>> being sent to, they have scraped ARIN WHOIS data for noc and abuse POC 
>>>>> contact info and recent ipv4 block transfers. 
>>>>> It's trivially easy to block their entire domain at the mail server 
>>>>> level, of course...
>>>>> 
>>>>> 

Re: ARIN whois contact abuse from ipv4depot aka Silicon Desert International Inc

[Mel Beckman]

Tom,

When an ARIN member violates their agreement and spams from ARIN’s databases, 
it’s not just an “Internet is fertile ground” deal. It’s a betrayal of a legal 
trust, one that demands accountability. I’m quite happy that ARIN promptly 
responds to these abuses, and gets results. That only works if victims report 
spam and compare notes. Let the “fertile ground” be elsewhere!

 -mel beckman

On Oct 12, 2023, at 8:49 AM, Tom Beecher  wrote:


It's ridiculous that they resort to scraping public lists and DBs to try and 
achieve what they're attempting to do.

Everyone is always looking for information they can use to advance some agenda 
or purpose. The internet is fertile ground for that. Always has been, always 
will be.

Not taking shots at anyone here, but I am boggled why this is a common public 
complaint. Block the sender and move on.

On Wed, Oct 11, 2023 at 7:56 PM Peter Potvin via NANOG 
mailto:nanog@nanog.org>> wrote:
Definitely have received this same spam multiple times and so have a few others 
I know. It's ridiculous that they resort to scraping public lists and DBs to 
try and achieve what they're attempting to do.

Regards,
Peter Potvin | Executive Director
--
Accuris Technologies Ltd.



On Wed, Oct 11, 2023 at 7:52 PM Eric Kuhnke 
mailto:eric.kuh...@gmail.com>> wrote:
Is anyone else receiving spam from this organization? Based on the contents of 
the cold solicitations they are sending us, and the addresses being sent to, 
they have scraped ARIN WHOIS data for noc and abuse POC contact info and recent 
ipv4 block transfers.

It's trivially easy to block their entire domain at the mail server level, of 
course...

Re: AT&T Business Center completely broken for months - is it the norm?

[Mel Beckman]

Have you tried a different browser? Chrome works for me. Edge and Firefox don’t.

 -mel beckman

> On Oct 9, 2023, at 8:41 PM, Mirai Azayaka  wrote:
> 
> Hi NANOG,
> 
> Maybe this topic is better suited for the complaint department of AT&T
> but I just want to confirm if it's just me or it's just AT&T...
> 
> So I'm a new customer of AT&T's DIA network and I haven't been able to
> make a payment since day one. (And it has been several months.) Just
> wondering if a completely broken internal billing system is normal...
> I only have limited experience with Hurricane Electric and Equinix
> before. Wondering if Verizon or Comcast is also broken like AT&T. Here
> are the issues I had with their system:
> - Clicking random links around the portal will give you HTTP 400
> errors, sometimes.
> - I'm unable to add payment methods even after following the payment
> tutorial exactly. The portal consistently gives HTTP 413 errors.
> - Live chat doesn't work at all. Clicking the button returns HTTP 404
> in my debugging console.
> - Extremely slow for some tasks which may result in a HTTP 408.
> 
> The system feels like a collection of HTTP error codes... How can it
> be so broken? Are other ISP's internal billing systems broken like
> this? Looking for anecdotes / experiences.
> 
> Azayaka

Re: Low to Mid Range DWDM Platforms

[Mel Beckman]

One of my clients is Calient Technologies:

https://www.calient.net/

Their S320 optical switch is entirely photonic — no electrical transceivers in 
any optical path at all — using 3D MEMS technology. It’s great for reliable 
multi-lambda provisioning, and sports built-in power monitoring and passive 
network diagnostics. It doesn’t do DMDM itself, but it is a great way to 
automate DWDM traffic engineering with real-time optical path provisioning.  
It’s essentially a zero-latency 400 Mbps matrix switch.

 -mel

On Oct 6, 2023, at 6:44 AM, Mark Tinka  wrote:



On 10/6/23 15:07, Mike Hammett wrote:

I've been using various forms of passive WDM for years. I have a couple 
different projects on my plate that require me to look at the next level of 
platform.

In some projects, I'll be looking for needing to have someone long distances of 
glass without any electronics. Some spans could be over 60 miles.

In some projects, I'll need to transport multiple 100-gig waves.

What is the landscape like between basic passive and something like a 30 
terabit Ciena? I know of multiple vendors in that space, but I like to learn 
more about what features I need and what features I don't need from somewhere 
other than the vendor's mouth. Obviously, the most reliability at the least 
cost as well.

400G-ZR pluggables will get you 400Gbps on a p2p dark fibre over 80km - 100km. 
So your main cost there will be routers that will support.

The smallest DCI solution from the leading DWDM vendors is likely to be your 
cheapest option. Alternatively, if you are willing to look at the open market, 
you can find gear based on older CMOS (40nm, for example), which will now be 
EoL for any large scale optical network, but cost next to nothing for a 
start-up with considerable capacity value.

There is a DWDM vendor that showed up on the scene back in 2008 or thereabouts. 
They were selling a very cheap, 1U box that had a different approach to DWDM 
from other vendors at the time. I, for the life of me, cannot remember their 
name - but I do know that Randy introduced them to me back then. Maybe he can 
remember :-). Not sure if they are still in business.

Mark.


https://www.calient.net/

Their S320 optical switch is entirely photonic — no electrical transceivers 
in any optical path at all — using 3D MEMS technology. It’s great for 
reliable multi-lambda provisioning, and sports built-in power monitoring and 
passive network diagnostics. It doesn’t do DMDM itself, but it is a great way 
to automate DWDM traffic engineering with real-time optical path provisioning.  
It’s essentially a zero-latency 400 Mbps matrix switch.

 -mel

> On Oct 6, 2023, at 6:44 AM, Mark Tinka  wrote:
> 
> 
> 
>> On 10/6/23 15:07, Mike Hammett wrote:
>> 
>> I've been using various forms of passive WDM for years. I have a couple 
>> different projects on my plate that require me to look at the next level of 
>> platform.
>> 
>> In some projects, I'll be looking for needing to have someone long distances 
>> of glass without any electronics. Some spans could be over 60 miles.
>> 
>> In some projects, I'll need to transport multiple 100-gig waves.
>> 
>> What is the landscape like between basic passive and something like a 30 
>> terabit Ciena? I know of multiple vendors in that space, but I like to learn 
>> more about what features I need and what features I don't need from 
>> somewhere other than the vendor's mouth. Obviously, the most reliability at 
>> the least cost as well.
> 
> 400G-ZR pluggables will get you 400Gbps on a p2p dark fibre over 80km - 
> 100km. So your main cost there will be routers that will support.
> 
> The smallest DCI solution from the leading DWDM vendors is likely to be your 
> cheapest option. Alternatively, if you are willing to look at the open 
> market, you can find gear based on older CMOS (40nm, for example), which will 
> now be EoL for any large scale optical network, but cost next to nothing for 
> a start-up with considerable capacity value.
> 
> There is a DWDM vendor that showed up on the scene back in 2008 or 
> thereabouts. They were selling a very cheap, 1U box that had a different 
> approach to DWDM from other vendors at the time. I, for the life of me, 
> cannot remember their name - but I do know that Randy introduced them to me 
> back then. Maybe he can remember :-). Not sure if they are still in business.
> 
> Mark.
> 
> 

Re: cogent spamming directly from ARIN records?

[Mel Beckman]

Patrick,

It’s a sales pitch, and ARIN acknowledges it violates their terms and has taken 
action.

 -mel

> On Oct 2, 2023, at 2:47 PM, Patrick W. Gilmore  wrote:
> 
>  Has anyone replied?
> 
> If this is a peering request, not sure that is a bad use of the AS contact 
> info.
> 
> If it is a sales pitch, then yeah, that’s a problem.
> 
> -- 
> TTFN,
> patrick
> 
>> On Oct 2, 2023, at 14:58, Tim Burke  wrote:
>> 
>> Hurricane has been doing the same thing lately... but their schtick is to 
>> say that "we are seeing a significant amount of hops in your AS path and 
>> wanted to know if you are open to resolve this issue".
>> 
>> complia...@arin.net is about all that can be done, other than public 
>> shaming! 
>> 
>> Other outfits have been spamming using the nanog attendees list, but I guess 
>> that’s not as bad as the continued scraping of ARIN records, so I won't call 
>> them out... yet, at least. 😊
>> 
>> -Original Message-
>> From: NANOG  On Behalf Of Mel Beckman
>> Sent: Monday, October 2, 2023 10:28 AM
>> To: nanog list 
>> Subject: cogent spamming directly from ARIN records?
>> 
>> This morning I received an email from someone at Cogent asking about an ASN 
>> I administer. They didn’t give any details, but I assumed it might be 
>> related to some kind of network transport issue. I replied cordially, asking 
>> them what they needed. The person then replied with a blatant spam, 
>> advertising Cogent IP services, in violation of the U.S. CAN-SPAM Act’s 
>> prohibition against deceptive UCE.
>> 
>> I believe they got the contact information from ARIN, because the ARIN 
>> technical POC is the only place where my name and the ASN are connected. I 
>> believe this is a violation of Cogent’s contract with ARIN. Does anybody 
>> know how I can effectively report this to ARIN? If we can’t even police 
>> infrastructure providers for spamming, LIOAWKI.
>> 
>> -mel beckman
> 

Re: cogent spamming directly from ARIN records?

[Mel Beckman]

Jay,

Apparently my sarcasm was too subtle :)

 -mel

> On Oct 2, 2023, at 10:01 AM, Jay Hennigan  wrote:
> 
> On 10/2/23 09:16, Mel Beckman wrote:
>> Tom,
>> Thanks for that pointer! apparently cogent has a history of abuse.
> 
> Apparently?
> 
> In other news, apparently bears have been using our National Forests as their 
> personal toilets for decades.
> 
> -- 
> Jay Hennigan - j...@west.net
> Network Engineering - CCIE #7880
> 503 897-8550 - WB6RDV
> 

Re: cogent spamming directly from ARIN records?

[Mel Beckman]

 dang auto correct! I should’ve caught that bad change. what I meant to 
say, was:

LAWKIWBO.

 -mel

> On Oct 2, 2023, at 9:15 AM, Mel Beckman  wrote:
> 
> John,
> 
> Thank you for your guidance!
> 
> -mel
> 
>> On Oct 2, 2023, at 8:33 AM, John Sweeting  wrote:
>> 
>> Mel, I will reply to you off list. Thanks.
>> 
>> On 10/2/23, 11:28 AM, "NANOG on behalf of Mel Beckman" 
>> mailto:arin@nanog.org> on 
>> behalf of m...@beckman.org <mailto:m...@beckman.org>> wrote:
>> 
>> 
>> This morning I received an email from someone at Cogent asking about an ASN 
>> I administer. They didn’t give any details, but I assumed it might be 
>> related to some kind of network transport issue. I replied cordially, asking 
>> them what they needed. The person then replied with a blatant spam, 
>> advertising Cogent IP services, in violation of the U.S. CAN-SPAM Act’s 
>> prohibition against deceptive UCE.
>> 
>> 
>> I believe they got the contact information from ARIN, because the ARIN 
>> technical POC is the only place where my name and the ASN are connected. I 
>> believe this is a violation of Cogent’s contract with ARIN. Does anybody 
>> know how I can effectively report this to ARIN? If we can’t even police 
>> infrastructure providers for spamming, LIOAWKI.
>> 
>> 
>> -mel beckman
>> 

Re: cogent spamming directly from ARIN records?

[Mel Beckman]

Tom,

Thanks for that pointer! apparently cogent has a history of abuse.

 -mel

On Oct 2, 2023, at 8:34 AM, Tom Beecher  wrote:


complia...@arin.net<mailto:complia...@arin.net>

Refer back to an email John Curran sent to this list on Jan 6 2020 , 
"Suspension of Cogent access to ARIN Whois"

On Mon, Oct 2, 2023 at 11:29 AM Mel Beckman 
mailto:m...@beckman.org>> wrote:
This morning I received an email from someone at Cogent asking about an ASN I 
administer. They didn’t give any details, but I assumed it might be related to 
some kind of network transport issue. I replied cordially, asking them what 
they needed. The person then replied with a blatant spam, advertising Cogent IP 
services, in violation of the U.S. CAN-SPAM Act’s prohibition against deceptive 
UCE.

I believe they got the contact information from ARIN, because the ARIN 
technical POC is the only place where my name and the ASN are connected. I 
believe this is a violation of Cogent’s contract with ARIN. Does anybody know 
how I can effectively report this to ARIN? If we can’t even police 
infrastructure providers for spamming, LIOAWKI.

 -mel beckman

Re: cogent spamming directly from ARIN records?

[Mel Beckman]

John,

Thank you for your guidance!

 -mel

> On Oct 2, 2023, at 8:33 AM, John Sweeting  wrote:
> 
> Mel, I will reply to you off list. Thanks.
> 
> On 10/2/23, 11:28 AM, "NANOG on behalf of Mel Beckman" 
> mailto:arin@nanog.org> on 
> behalf of m...@beckman.org <mailto:m...@beckman.org>> wrote:
> 
> 
> This morning I received an email from someone at Cogent asking about an ASN I 
> administer. They didn’t give any details, but I assumed it might be related 
> to some kind of network transport issue. I replied cordially, asking them 
> what they needed. The person then replied with a blatant spam, advertising 
> Cogent IP services, in violation of the U.S. CAN-SPAM Act’s prohibition 
> against deceptive UCE.
> 
> 
> I believe they got the contact information from ARIN, because the ARIN 
> technical POC is the only place where my name and the ASN are connected. I 
> believe this is a violation of Cogent’s contract with ARIN. Does anybody know 
> how I can effectively report this to ARIN? If we can’t even police 
> infrastructure providers for spamming, LIOAWKI.
> 
> 
> -mel beckman
> 

cogent spamming directly from ARIN records?

[Mel Beckman]

This morning I received an email from someone at Cogent asking about an ASN I 
administer. They didn’t give any details, but I assumed it might be related to 
some kind of network transport issue. I replied cordially, asking them what 
they needed. The person then replied with a blatant spam, advertising Cogent IP 
services, in violation of the U.S. CAN-SPAM Act’s prohibition against deceptive 
UCE.

I believe they got the contact information from ARIN, because the ARIN 
technical POC is the only place where my name and the ASN are connected. I 
believe this is a violation of Cogent’s contract with ARIN. Does anybody know 
how I can effectively report this to ARIN? If we can’t even police 
infrastructure providers for spamming, LIOAWKI.

 -mel beckman

Re: Legal system as a weapon (was Re: AFRINIC placed in receivership)

[Mel Beckman]

Just like a lawyer, trying to add layers to the model. :)

 -mel

> On Sep 30, 2023, at 8:58 AM, Anne Mitchell  wrote:
> 
> 
> 
>> On Sep 29, 2023, at 11:20 PM, Mel Beckman  wrote:
>> 
>> The seven lawyers of the OSI model
>> 
>> 1: Family lawyer (where it all starts)
>> 2: Admiralty lawyer
>> 3: Intellectual Property lawyer (because, of course)
>> 4. Immigration lawyer
>> 5. Real Estate lawyer
>> 6. Entertainment lawyer
>> 7. Labor lawyer
>> 
> 
> You forgot Estate Planning lawyer and, of course, email law and policy 
> lawyer. :-)
> 
> Anne
> 
> --- 
> Anne P. Mitchell, Esq.
> Email Law & Policy Attorney
> CEO Institute for Social Internet Public Policy (ISIPP)
> Author: Section 6 of the CAN-SPAM Act of 2003 (the Federal email marketing 
> law)
> Creator of the term 'deliverability' and founder of the deliverability 
> industry
> Author: The Email Deliverability Handbook
> Board of Directors, Denver Internet Exchange
> Dean Emeritus, Cyberlaw & Cybersecurity, Lincoln Law School
> Prof. Emeritus, Lincoln Law School
> Chair Emeritus, Asilomar Microcomputer Workshop
> Counsel Emeritus, eMail Abuse Prevention System (MAPS)
> 
> 

Re: Legal system as a weapon (was Re: AFRINIC placed in receivership)

[Mel Beckman]

The seven lawyers of the OSI model


1: Family lawyer (where it all starts)

2: Admiralty lawyer

3: Intellectual Property lawyer (because, of course)

4. Immigration lawyer

5. Real Estate lawyer

6. Entertainment lawyer

7. Labor lawyer

 -mel

On Sep 29, 2023, at 10:03 PM, Jay R. Ashworth  wrote:

Layer 8: People

Layer 9: Money

Layer 10: Lawyers.

Cheers,
-- jra

- Original Message -
From: "David Conrad" 
To: nanog@nanog.org
Sent: Thursday, September 28, 2023 6:46:31 PM
Subject: Legal system as a weapon (was Re: AFRINIC placed in receivership)

Somewhat related (at least one of the principals is the same) and perhaps of
interest to some here. While I have strong opinions on the topic, provided
without comment:

https://www.gofundme.com/f/supporting-and-protecting-internet-governance

Regards,
-drc

On Sep 13, 2023, at 6:27 PM, Bryan Fields  wrote:

I think this qualifies as potentially operational.

Afrinic placed in receivership, board elections to be held in six months:
https://archive.ph/jOFE4
--
Bryan Fields

727-409-1194 - Voice
http://bryanfields.net

--
Jay R. Ashworth  Baylink   j...@baylink.com
Designer The Things I Think   RFC 2100
Ashworth & Associates   http://www.bcp38.info  2000 Land Rover DII
St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647 1274

Re: Test Lab Best Practices

[Mel Beckman]

In any lab,I find concurrent access to serial ports is still an essential 
diagnostic tool. In a pinch you can get a used Cisco 2811 for $100, but there 
are multiport devices from lots of vendors. These let you SSH into the server 
and then connect to any serial port, giving you separate serial port windows 
all on the same screen. I’ve become fond of the WiFi-capable multiport modules 
from get-console.com. The ability to record logs from these serial ports in 
real-time helps a lot for documenting regression tests. 

 -mel beckman

> On Sep 28, 2023, at 7:25 AM, Kenneth Vedder  wrote:
> 
> 
> Hello NANOG,
> 
> We have been struggling with firmware bugs from a specific router vendor. I 
> am looking to set up a test lab of our core network and a few remote site 
> routers.  Protocols would include SR-MPLS, ISIS, EVPN MPLS and L3VPN with a 
> little OSPF sprinkled in. I'd be grateful for any tips or resources anyone 
> has that might cover testing strategies and/or best practices. 
> 
> Thanks,
> Ken

Re: SMTP-friendly VPS provider where I can also get a BGP feed

[Mel Beckman]

Tony,

BGP is helpful for email servers if you own your own clean IP space, because 
much cloud IP space is black listed. 

-mel via cell

> On Sep 26, 2023, at 11:41 AM, Tony Wicks  wrote:
> 
> I can't speak to the bgp feed as this seems like unnecessary complication to 
> me, but I use https://www.racknerd.com/ for personal email/web hosting KVM 
> VM's and have found them to be excellent. They have yearly black Friday 
> specials (last years - https://www.racknerd.com/BlackFriday/ ) that are very 
> attractive. They don't block any ports on their US/Europe VM's. I use a 
> primary pair in one city and rsync everything to a backup pair in another 
> city (as well as home just to make sure). Not all cities can get V6 but most 
> do.
> 
> 
> 
> -Original Message-
> From: NANOG  On Behalf Of Daniel 
> Corbe
> Sent: Tuesday, September 26, 2023 11:09 PM
> To: nanog@nanog.org
> Subject: SMTP-friendly VPS provider where I can also get a BGP feed
> 
> Hey all,
> 
> I apologize if this isn't the right place to post this; however, I thought 
> maybe the NANOG community would be able to point me in the right direction.
> 
> I'm looking for a place that I can host a mailer.  My primary use case is a 
> Mailman-style technical discussion list; much like NANOG but software related 
> instead of network related: READ: non-commercial in nature.
> 
> I'm currently a vultr customer, but they're refusing to unblock port 25 on my 
> account.  I've tried explaining my use case but no matter who I talk to over 
> there they just keep pointing me to their spam policy.
> 
> Thanks!
> -Daniel
> 

Re: SMTP-friendly VPS provider where I can also get a BGP feed

[Mel Beckman]

One thing you should consider about running a "family" mail server (or any 
other IT services for friends and family): that you have a clearly documented 
path of management succession. A dear friend of mine passed away  last year and 
was running just such an email server. Nobody really knew how to get into it 
for maintenance, and a couple weeks after he passed. it crashed, and none of us 
knew precisely where it was physically located (on the end of a VPN tunnel, it 
tuns out). This took down email for 100 of his closest friends and family 
members for several weeks. We couldn't even unlock the domain,

Personally, this has spurred me to create much better documentation of  my own 
client services, and to involve a successor unlikely to be traveling with me 🙂

 -mel

From: NANOG  on behalf of Jim 
Shankland via NANOG 
Sent: Tuesday, September 26, 2023 9:46 AM
To: nanog@nanog.org 
Subject: Re: SMTP-friendly VPS provider where I can also get a BGP feed

I've been using Linode, also; works fine on the Linode end, but I still
see occasional rejections based on my Linode IP address (most recently
from outlook.com). It's nothing my specific IP is doing, but appears to
be blacklisting of an address range. And gmail randomly puts some
outgoing mail into recipients' spam folders. Trying to get an address
unblocked by a major provider works almost as well as howling into the wind.

Maybe I'm being stubborn to insist on continuing to run what's basically
a family mail server, but it does seem like there's a matter of
principle there: it should be possible to have an email account without
having all the emails stored by a third party. If the answer ends up
being, "Oh, just use gmail, everybody else does!" ... well, so be it, I
guess, but we should be clear that something got lost in that transition.

Jim Shankland

On 9/26/23 9:10 AM, Jay R. Ashworth wrote:
> I've run a mail server on Linode for 6 or 7 years now; no technical problems.
>
> End-node, Zimbra, postfix.
>
> Cheers,
> -- jra
>
> - Original Message -
>> From: "Jonathan Leist via NANOG" 
>> To: "Daniel Corbe" 
>> Cc: nanog@nanog.org
>> Sent: Tuesday, September 26, 2023 10:32:51 AM
>> Subject: Re: SMTP-friendly VPS provider where I can also get a BGP feed
>> Pretty much every popular provider blocks port 25 out by default, and
>> they'll instead try to steer customers to use a smart host. However, some,
>> including Linode, will unblock port 25 by request:
>> https://www.linode.com/docs/guides/running-a-mail-server/#sending-email-on-linode
>>
>> On Tue, Sep 26, 2023 at 6:11 AM Daniel Corbe  wrote:
>>
>>> Hey all,
>>>
>>> I apologize if this isn't the right place to post this; however, I
>>> thought maybe the NANOG community would be able to point me in the right
>>> direction.
>>>
>>> I'm looking for a place that I can host a mailer.  My primary use case
>>> is a Mailman-style technical discussion list; much like NANOG but
>>> software related instead of network related: READ: non-commercial in
>>> nature.
>>>
>>> I'm currently a vultr customer, but they're refusing to unblock port 25
>>> on my account.  I've tried explaining my use case but no matter who I
>>> talk to over there they just keep pointing me to their spam policy.
>>>
>>> Thanks!
>>> -Daniel
>>>
>>
>> --
>> Jonathan Leist
>> Staff Engineer

Re: SMTP-friendly VPS provider where I can also get a BGP feed

[Mel Beckman]

DigitalOcean.com also lets you send and receive on port 25, provided your MTA 
isn’t configured as an open relay.

 -mel

> On Sep 26, 2023, at 7:38 AM, Brandon Martin  wrote:
> 
> On 9/26/23 06:09, Daniel Corbe wrote:
>> I'm looking for a place that I can host a mailer.  My primary use case is a 
>> Mailman-style technical discussion list; much like NANOG but software 
>> related instead of network related: READ: non-commercial in nature.
>> 
>> I'm currently a vultr customer, but they're refusing to unblock port 25 on 
>> my account.  I've tried explaining my use case but no matter who I talk to 
>> over there they just keep pointing me to their spam policy.
> 
> I've been a happy customer of prgmr.com now TornadoVPS.  They will definitely 
> allow port 25 outbound.  It was unblocked by default when I signed up years 
> ago and I think still is even on new accounts.
> 
> I don't know if they will do BGP.  In the past, they had said they would by 
> request provided you had your own AS and IP space. I think they also had 
> offered to do one under a private ASN for route collection.
> 
> --
> Brandon Martin
> 

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

So, probably not a failure "caused by GPS", rather one caused by poor
design (only two clock sources) combined with unsupported and buggy
devices.

Interesting software bug, but not really germane to this discussion, other than 
as a cautionary tale about time distribution architectures.


 -mel beckman

On Aug 16, 2023, at 3:51 AM, Matthew Richardson  
wrote:

Mel Beckman wrote:-

Do you have a citation for your Jersey event? I doubt GPS caused the problem, 
but I'd like to see the documentation.

The event took place on the evening of Sunday 12 July 2020, and seems NOT
to have been due to an issue caused directly by GPS, but rather to
misbehaviour of a GPS NTP server relating to week numbers.  Our regulator
subsequently issued the following comprehensive document:-

https://www.jcra.je/media/598397/t-027-jt-july-2020-outage-decision-directions.pdf

By way of summary, JT operated two GPS derived NTP servers, with all of
their routers were pointing to both.  On the evening in question, one of
the two reset its clock back to 27 November 2000.

Their interior routing protocol used amongst their mesh of routers was
IS-IS which was using authentication.  The authentication [section 4.19]
was described having a "password validity start date" of 01 July 2012.
Thus, any routers which had picked up the time from the faulty source no
longer had valid IS-IS authentication and were thus isolated.

Whilst only 15% of their routers were affected, this was enough to cause an
almost total failure in their network, affecting telephony (fixed & mobile)
and Internet.  For foreign readers (this is NANOG!) "999" calls refer to
the emergency services in these parts, where any failures attract the
attention of our regulator.

The details of why the clock "failed" start at section 4.23, and seem to
relate a GPS week number rollover.

So, probably not a failure "caused by GPS", rather one caused by poor
design (only two clock sources) combined with unsupported and buggy
devices.

One curious aspect is that some routers followed the "bad" time, which is
alluded to in section 4.31.

Something not discussed in that report is that JT's email failed during the
incident despite its being hosted on Office365.  The reason was that the
two authoritative DNS servers for jtglobal.com were hosted in Jersey inside
their network.  As that network was wholly disconnected, there was no DNS
and hence no email.  Despite my having raised this since with their senior
management, their DNS remains hosted in this way:-

matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns jtglobal.com 
@ns1.jtibs.net
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20462
;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 4

;; QUESTION SECTION:
;jtglobal.com.INNS

;; ANSWER SECTION:
jtglobal.com.60INNSns2.jtibs.net.
jtglobal.com.60INNSns1.jtibs.net.

;; ADDITIONAL SECTION:
ns1.jtibs.net.60INA212.9.0.135
ns2.jtibs.net.60INA212.9.0.136
ns1.jtibs.net.60IN2a02:c28::d1
ns2.jtibs.net.60IN2a02:c28::d2

Rediculously (and again despite my agitation to their management) our
government domain gov.je has similar DNS fragility:-

matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns gov.je @ns1.gov.je
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4249
;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2

;; QUESTION SECTION:
;gov.je.INNS

;; ANSWER SECTION:
gov.je.3600INNSns2.gov.je.
gov.je.3600INNSns1.gov.je.

;; ADDITIONAL SECTION:
ns2.gov.je.3600INA212.9.21.137
ns1.gov.je.3600INA212.9.21.9

--
Best wishes,
Matthew

--
From: Mel Beckman 
To: Matthew Richardson 
Cc: Nanog 
Date: Tue, 8 Aug 2023 15:12:29 +
Subject: Re: NTP Sync Issue Across Tata (Europe)

Until the Internet NTP network can be made secure, no. Do you have a citation 
for your Jersey event? I doubt GPS caused the problem, but I'd like to see the 
documentation.

Using GPS for time sync is simple risk management: the risk of Internet NTP 
with known, well documented vulnerabilities and many security incidents, versus 
the risk of some theoretical GPS-based vulnerability, for which mitigations 
such as geographic diversity are readily available. Sure, you could use 
Internet NTP as a last resort should GPS fail globally (perhaps due to a 
theoretical - but conceivable - meteor storm). But that would be a fall-back. I 
would not mix the systems.

-mel

On Aug 8, 2023, at 1:36 AM, Matthew Richardson  
wrote:

?Mel Beckman wrote:-

It's a problem that has received a lot of attention in both NTP and
aviation navigation circles. What is hard to defend against is total signal
suppression 

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

So, probably not a failure "caused by GPS", rather one caused by poor
design (only two clock sources) combined with unsupported and buggy
devices.



 -mel beckman

On Aug 16, 2023, at 3:51 AM, Matthew Richardson  
wrote:

Mel Beckman wrote:-

Do you have a citation for your Jersey event? I doubt GPS caused the problem, 
but I'd like to see the documentation.

The event took place on the evening of Sunday 12 July 2020, and seems NOT
to have been due to an issue caused directly by GPS, but rather to
misbehaviour of a GPS NTP server relating to week numbers.  Our regulator
subsequently issued the following comprehensive document:-

https://www.jcra.je/media/598397/t-027-jt-july-2020-outage-decision-directions.pdf

By way of summary, JT operated two GPS derived NTP servers, with all of
their routers were pointing to both.  On the evening in question, one of
the two reset its clock back to 27 November 2000.

Their interior routing protocol used amongst their mesh of routers was
IS-IS which was using authentication.  The authentication [section 4.19]
was described having a "password validity start date" of 01 July 2012.
Thus, any routers which had picked up the time from the faulty source no
longer had valid IS-IS authentication and were thus isolated.

Whilst only 15% of their routers were affected, this was enough to cause an
almost total failure in their network, affecting telephony (fixed & mobile)
and Internet.  For foreign readers (this is NANOG!) "999" calls refer to
the emergency services in these parts, where any failures attract the
attention of our regulator.

The details of why the clock "failed" start at section 4.23, and seem to
relate a GPS week number rollover.

So, probably not a failure "caused by GPS", rather one caused by poor
design (only two clock sources) combined with unsupported and buggy
devices.

One curious aspect is that some routers followed the "bad" time, which is
alluded to in section 4.31.

Something not discussed in that report is that JT's email failed during the
incident despite its being hosted on Office365.  The reason was that the
two authoritative DNS servers for jtglobal.com were hosted in Jersey inside
their network.  As that network was wholly disconnected, there was no DNS
and hence no email.  Despite my having raised this since with their senior
management, their DNS remains hosted in this way:-

matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns jtglobal.com 
@ns1.jtibs.net
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20462
;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 4

;; QUESTION SECTION:
;jtglobal.com.INNS

;; ANSWER SECTION:
jtglobal.com.60INNSns2.jtibs.net.
jtglobal.com.60INNSns1.jtibs.net.

;; ADDITIONAL SECTION:
ns1.jtibs.net.60INA212.9.0.135
ns2.jtibs.net.60INA212.9.0.136
ns1.jtibs.net.60IN2a02:c28::d1
ns2.jtibs.net.60IN2a02:c28::d2

Rediculously (and again despite my agitation to their management) our
government domain gov.je has similar DNS fragility:-

matthew@m88:~$ dig +norec +noedns +nocmd +nostats -t ns gov.je @ns1.gov.je
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4249
;; flags: qr aa; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2

;; QUESTION SECTION:
;gov.je.INNS

;; ANSWER SECTION:
gov.je.3600INNSns2.gov.je.
gov.je.3600INNSns1.gov.je.

;; ADDITIONAL SECTION:
ns2.gov.je.3600INA212.9.21.137
ns1.gov.je.3600INA212.9.21.9

--
Best wishes,
Matthew

--
From: Mel Beckman 
To: Matthew Richardson 
Cc: Nanog 
Date: Tue, 8 Aug 2023 15:12:29 +
Subject: Re: NTP Sync Issue Across Tata (Europe)

Until the Internet NTP network can be made secure, no. Do you have a citation 
for your Jersey event? I doubt GPS caused the problem, but I'd like to see the 
documentation.

Using GPS for time sync is simple risk management: the risk of Internet NTP 
with known, well documented vulnerabilities and many security incidents, versus 
the risk of some theoretical GPS-based vulnerability, for which mitigations 
such as geographic diversity are readily available. Sure, you could use 
Internet NTP as a last resort should GPS fail globally (perhaps due to a 
theoretical - but conceivable - meteor storm). But that would be a fall-back. I 
would not mix the systems.

-mel

On Aug 8, 2023, at 1:36 AM, Matthew Richardson  
wrote:

?Mel Beckman wrote:-

It's a problem that has received a lot of attention in both NTP and
aviation navigation circles. What is hard to defend against is total signal
suppression via high powered jamming. But that you can do with a
geographically diverse GPS NTP network.

Whilst looking forward to being corrected, GPS (ev

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

Forrest,

I think you’re gilding the lilly. My original recommendation was to use GPS as 
primary, for its superior accuracy and resistance to attack, and have anti-GPS 
back up.  If you want automatic fail over, do that in an intermediate server on 
your site that makes a conscious test and decision to fail over to Internet NTP.

You’re mistaken to say that the vulnerability of GPS is remotely comparable  to 
the vulnerability of Internet-based NTP. To interfere with a GPS-derived clock, 
an attacker has to physically be present. That’s a huge expense — and risk — 
that hackers are really not interested in undertaking. They would much rather 
sit in Russia or China and attack NTP servers remotely using any of the several 
attack methodologies I’ve cited previously.

So curate Internet NTP or not (personally, that seems like just another thing 
to monitor and maintain), but make GPS your primary time standard.  You’re much 
better off staying air-gapped from Internet NTP until you detect a GPS failure. 
 All the other machinations are pointless while GPS is working, because GPS 
gives you by far the best accuracy and security for the buck.  Like I said, 
spend $400 on a commercial GPS time server and timing problems are solved. Or 
use facility-provided GPS if you can’t get an antenna up.

 -mel

On Aug 14, 2023, at 12:10 AM, Forrest Christian (List Account) 
 wrote:


I've responded in bits and pieces to this thread and haven't done an excellent 
job expressing my overall opinion.   This is probably because my initial goal 
was to point out that GPS-transmitted time is no less subject to being attacked 
than your garden variety NTP-transmitted time. Since this thread has evolved, 
I'd like to describe my overall position to be a bit clearer.

To start, we need a somewhat simplified version of how UTC is created so I can 
refer to it later:

Across the globe, approximately 85 research and standards institutions run a 
set of freestanding atomic clocks that contribute to UTC.   The number of 
atomic clocks across all these institutions totals around 450.   Each 
institution also produces a version of UTC based on its own set of atomic 
clocks.  In the international timekeeping world, this is designated as 
UTC(Laboratory), where Laboratory is replaced with the abbreviation for the lab 
producing that version of UTC.   So UTC(NIST) is the version that NIST produces 
at Boulder, Colorado, NICT produces UTC(NICT) in Tokyo, and so on.

Because no clock is perfectly accurate, all of these versions of UTC drift in 
relation to each other, and you could have significant differences in time 
between different labs.   As a result, there has to be a way to synchronize 
them.  Each month, the standards organization BIPM collects relative time 
measurements and other statistics from each institution described above.  This 
data is then used to determine the actual value of UTC. BIPM then produces a 
report detailing each organization's difference from the correct representation 
of UTC.   Each institution uses this data to adjust its UTC representation, and 
the cycle repeats the next month. In this way, all of the representations of 
UTC end up being pretty close to each other.   The document BIPM produces is 
titled "Circular T."  The most recent version indicates that most of the 
significant standards institutions maintain a UTC version that differs by less 
than 10ns from the official version of UTC.

Note that 10ns is far more accurate than we need for NTP, so most of the UTC 
representations can be considered identical as far as this discussion goes. 
Still, it is essential to realize that UTC(NIST) is generated separately from 
UTC(USNO) or other UTC implementations.  For example, a UTC(NIST) failure 
should not cause UTC(USNO) to fail as they utilize separate hardware and 
systems.

Each of these versions of UTC is also disseminated in various ways.  UTC(NIST) 
goes out via the "WWV" radio stations, NTP, and other esoteric methods.   GPS 
primarily distributes UTC(USNO), which is also available directly via NTP.  
UTC(SU) is the timescale for GLONASS.  And so on.

So, back to NTP and the accuracy required:

Most end users (people running everyday web applications or streaming video or 
similar) don't need precisely synchronized time.   The most sensitive 
application I'm aware of in this space is likely TOTP, which often needs time 
on the server and time on the client (or hardware key) within 90 seconds of 
each other.   In addition, having NTP time fail usually isn't the end of the 
world for these users.  The best way to synchronize their computers (including 
desktop and server systems) to UTC is to point their computer time 
synchronization service (whatever that is) at 
pool.ntp.org, time.windows.com, 
their ISP's time server, or similar.  Or, with modern OS'es, you can leave the 
time configured to whatever server the OS manufacturer preconfigured.   As an 

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

Seth,

My point exactly. Use GPS as primary, and have anti-PS back up, and if you want 
automatic fail over, do that in an intermediate server on your site that makes 
a conscious test and decision to fail over to regular NTP

-mel via cell

> On Aug 9, 2023, at 5:01 PM, Seth Mattinen via NANOG  wrote:
> 
> On 8/9/23 3:25 PM, Forrest Christian (List Account) wrote:
>> Note that NIST operates a pool of 24 time servers for public use.These 
>> are spread across four different locations in two different states.  My 
>> understanding is that they all get their time directly from the official 
>> NIST clocks without GPS or NTP being involved.
> 
> I used to jump through all the hoops for that but honestly I like the 
> appliances better (they are also PTP grandmaster clocks). I can always 
> disable the GPS inputs if any of the doom and gloom actually comes to pass.
> 
> ~Seth

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

While GPS spoofing is technically possible, all the extant spoofing only 
tampers with the ephemeris (satellite position) data, not the timing stream. 
That's because hackers have been aiming at navigation, and may not have 
expressed interest in GPS tampering when NTP tampering is so easy 🙂

To spoof GPS clocks, a hacker has to know where the antennas are, and get above 
them in order to inject a signal with the right directionality. Commercial GPS 
clock vendors have implemented various anti-spoofing measures that, for 
example, only accept signals from a certain cone of visibility, which faces up. 
They have other measures too, some of which exploit geographic diversity, so if 
 you can have two or more GPS clocks in different hub sites, the clocks will 
reject spoofing signals.

This seems like a much easier defense than deploying secure NTP (NTS), which 
adds a huge amount of complexity. At least one researcher has shown that 
poluting the existing public NTP pool with enough bogus servers to seriously 
impact network time is trivial (I cited their paper in an earlier post on this 
thread).  A well funded state actor could be laying the framework for such an 
attack as we speak, lying in wait until an opportunity to disrupt Internet NTP 
globally.

   -mel

From: NANOG  on behalf of Jay Hennigan 

Sent: Wednesday, August 9, 2023 10:58 AM
To: nanog@nanog.org 
Subject: Re: NTP Sync Issue Across Tata (Europe)

On 8/9/23 09:29, Seth Mattinen via NANOG wrote:

> I liked having a WWVB receiver in my mix, but all the hardware
> appliances (at least those offering OCXO or Rubidium oscillator options)
> seem to have rejected it in favor of GPS only. I can only conclude that
> either vendors think options like WWVB are a dead end or there's no
> demand for GPS alternatives.

Both GPS and WWVB are over-the-air. There has been concern expressed of
a bad actor spoofing or jamming GPS. Comparatively speaking, jamming or
spoofing WWVB is a trivial joke.

--
Jay Hennigan - j...@west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

Go for it. I’m sure NTS’ complexity clocks lots of hours for expensive 
consultants :)

Me, I’m sticking with GPS.)

-mel via cell

On Aug 8, 2023, at 11:34 AM, Rubens Kuhl  wrote:


So little deployment that has 3500 occurrences according to 
shodan.io<http://shodan.io>.  With such few choices, It should be hard to find 
suitable options.

Rubens




Em ter., 8 de ago. de 2023 13:02, Mel Beckman 
mailto:m...@beckman.org>> escreveu:
I’m familiar with NTS, which is pointedly not NTP.  That’s like saying “TCP 
port 80 can be made secure,: just use a VPN!” Perhaps when NTS is widely 
deployed it will be an option. As the RFC was only approved in 2020, that will 
probably take a decade. Or more. (I’m talking about you, IPv6 :) Not to mention 
the complexity or NTS for hardware implementation in network elements, a 
primary application (https://tinyurl.com/ntsishard).

 -mel

> On Aug 8, 2023, at 8:26 AM, Rubens Kuhl 
> mailto:rube...@gmail.com>> wrote:
>
> On Tue, Aug 8, 2023 at 12:12 PM Mel Beckman 
> mailto:m...@beckman.org>> wrote:
>>
>> Until the Internet NTP network can be made secure, no.
>
> Internet NTP can be made secure, it's called NTS.
> https://developers.cloudflare.com/time-services/nts/ describes it with
> links to the RFC, and describes one of the many NTP servers that is
> available with NTS, time.cloudflare.com<http://time.cloudflare.com>. I 
> already mentioned 5 others,
> and there are many more than those 6.
>
>
> Rubens

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

Well, an iLEC CO is hardly a data center. Plus you have to be a CLEC to get in.

I’ve built out CLEC spaces in many COs. They invariably connect via fiber to a 
CLEC core data center, where they invariably run their own GPS clocks. Plus 
they run rubidium clocks in the racks at the CO, so they can freewheel for days.

-mel via cell

On Aug 8, 2023, at 11:21 AM, Mike Hammett  wrote:


Frontier COs is the easiest example.No antennas and no time service. Yes, they 
provide BITS, but BITS isn't time.

I was also speaking specifically about installing GPS antennas in viable 
places, not using a facility-provided GPS or NTP service.



-
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>

From: "Mel Beckman" 
To: "Mike Hammett" 
Cc: nanog@nanog.org, "Mark Tinka" 
Sent: Tuesday, August 8, 2023 10:36:46 AM
Subject: Re: NTP Sync Issue Across Tata (Europe)

I’d be interested in an example of a Colo that does NOT provide GPS-based NTP 
even if they don’t let tenants install their own. I’ve never, ever seen one.

 -mel

On Aug 8, 2023, at 8:20 AM, Mike Hammett  wrote:


My facility experience ranges from prohibited to infeasible. I must not be in 
the right facilities.


Yes, many radio platforms have GPS for timing. Some expose it for external time 
and timing purposes, some do not. Naturally, they do have a pretty good view of 
the sky.



-
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>

From: "Mel Beckman" 
To: "Mike Hammett" 
Cc: nanog@nanog.org, "Mark Tinka" 
Sent: Tuesday, August 8, 2023 10:05:55 AM
Subject: Re: NTP Sync Issue Across Tata (Europe)

It works fine, and is an industry standard. you have to mount the GPS antenna 
near a window with sky visibility, or on the roof. Many point-to-point 
microwave radios have GPS built in to obtain accurate timing for transmission 
multiplexing.

 -mel

On Aug 8, 2023, at 7:16 AM, Mike Hammett  wrote:


"We use these exclusively in data centers"

How well does GPS work inside the datacenter?



-
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

I’m familiar with NTS, which is pointedly not NTP.  That’s like saying “TCP 
port 80 can be made secure,: just use a VPN!” Perhaps when NTS is widely 
deployed it will be an option. As the RFC was only approved in 2020, that will 
probably take a decade. Or more. (I’m talking about you, IPv6 :) Not to mention 
the complexity or NTS for hardware implementation in network elements, a 
primary application (https://tinyurl.com/ntsishard).

 -mel 

> On Aug 8, 2023, at 8:26 AM, Rubens Kuhl  wrote:
> 
> On Tue, Aug 8, 2023 at 12:12 PM Mel Beckman  wrote:
>> 
>> Until the Internet NTP network can be made secure, no.
> 
> Internet NTP can be made secure, it's called NTS.
> https://developers.cloudflare.com/time-services/nts/ describes it with
> links to the RFC, and describes one of the many NTP servers that is
> available with NTS, time.cloudflare.com. I already mentioned 5 others,
> and there are many more than those 6.
> 
> 
> Rubens

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

I’d be interested in an example of a Colo that does NOT provide GPS-based NTP 
even if they don’t let tenants install their own. I’ve never, ever seen one.

 -mel

On Aug 8, 2023, at 8:20 AM, Mike Hammett  wrote:


My facility experience ranges from prohibited to infeasible. I must not be in 
the right facilities.


Yes, many radio platforms have GPS for timing. Some expose it for external time 
and timing purposes, some do not. Naturally, they do have a pretty good view of 
the sky.



-
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>

From: "Mel Beckman" 
To: "Mike Hammett" 
Cc: nanog@nanog.org, "Mark Tinka" 
Sent: Tuesday, August 8, 2023 10:05:55 AM
Subject: Re: NTP Sync Issue Across Tata (Europe)

It works fine, and is an industry standard. you have to mount the GPS antenna 
near a window with sky visibility, or on the roof. Many point-to-point 
microwave radios have GPS built in to obtain accurate timing for transmission 
multiplexing.

 -mel

On Aug 8, 2023, at 7:16 AM, Mike Hammett  wrote:


"We use these exclusively in data centers"

How well does GPS work inside the datacenter?



-
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>

From: "Mel Beckman" 
To: "Mark Tinka" 
Cc: nanog@nanog.org
Sent: Saturday, August 5, 2023 2:26:37 PM
Subject: Re: NTP Sync Issue Across Tata (Europe)

Mark,

You might consider setting up your own GPS-based NTP network. Commercial 
Ethernet GPS-sourced NTP servers, such as the Time Machines, TM1000A, are as 
little as $400. Or you can roll your own using a Raspberry Pi or similar nano 
computer with a GPS module and antenna. We use these exclusively in data 
centers now rather than depending on Internet NTP servers, primarily for 
security, because financial transactions in e-commerce can be sensitive to 
false time information. There are also a variety of NTP-based Internet attacks, 
so if you can block NTP at your border you’ve eliminated another attack surface.

-mel via cell

> On Aug 5, 2023, at 11:22 AM, Mark Tinka  wrote:
>
> 
>
>> On 8/5/23 20:17, Chris Adams wrote:
>>
>> It's the NTP pool people you need to talk to - the .freebsd. bit is just
>> a vendored entry into the pool (more for load tracking and management).
>
> Yes, Andreas clarified in unicast. Will do. Thanks.
>
> Mark.

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

Until the Internet NTP network can be made secure, no. Do you have a citation 
for your Jersey event? I doubt GPS caused the problem, but I’d like to see the 
documentation.

Using GPS for time sync is simple risk management: the risk of Internet NTP 
with known, well documented vulnerabilities and many security incidents, versus 
the risk of some theoretical GPS-based vulnerability, for which mitigations 
such as geographic diversity are readily available. Sure, you could use 
Internet NTP as a last resort should GPS fail globally (perhaps due to a 
theoretical — but conceivable — meteor storm). But that would be a fall-back. I 
would not mix the systems.

 -mel

> On Aug 8, 2023, at 1:36 AM, Matthew Richardson  
> wrote:
> 
> Mel Beckman wrote:-
> 
>> It's a problem that has received a lot of attention in both NTP and
>> aviation navigation circles. What is hard to defend against is total signal
>> suppression via high powered jamming. But that you can do with a
>> geographically diverse GPS NTP network.
> 
> Whilst looking forward to being corrected, GPS (even across multiple
> locations) seems to be a SINGLE source of time.  You seem (have I
> misunderstood?) to be a proponent of using GPS exclusively as the external
> clock source.
> 
> Might it be preferable to have a mixture of GPS (perhaps with another GNSS)
> together with carefully selected Internet-based NTP servers?
> 
> I recall an incident over here in Jersey (the one they named New Jersey
> after!) where our primary telco had a substantial time shift on one of
> their two GPS synced servers.  This managed to adjust the clock on enough
> of their routers that the certificate-based OSPF authentication considered
> the certificates invalid, and caused a failure of almost their whole
> network.
> 
> This is, of course, not to say that GPS is not a very good clock source,
> but rather to wonder whether more diversity would be preferable than using
> it as a single source.
> 
> --
> Best wishes,
> Matthew
> 
> --
>> From: Mel Beckman 
>> To: "Forrest Christian (List Account)" 
>> Cc: Nanog 
>> Date: Mon, 7 Aug 2023 14:03:30 +
>> Subject: Re: NTP Sync Issue Across Tata (Europe)
> 
>> Forrest,
>> 
>> GPS spoofing may work with a primitive Raspberry Pi-based NTP server, but 
>> commercial industrial NTP servers have specific anti-spoofing mitigations. 
>> There are also antenna diversity strategies that vendors support to ensure 
>> the signal being relied upon is coming from the right direction. It's a 
>> problem that has received a lot of attention in both NTP and aviation 
>> navigation circles. What is hard to defend against is total signal 
>> suppression via high powered jamming. But that you can do with a 
>> geographically diverse GPS NTP network.
>> 
>> -mel
>> 
>> On Aug 7, 2023, at 1:39 AM, Forrest Christian (List Account) 
>>  wrote:
>> 
>> ?
>> The problem with relying exclusively on GPS to do time distribution is the 
>> ease with which one can spoof the GPS signals.
>> 
>> With a budget of around $1K, not including a laptop, anyone with decent 
>> technical skills could convince a typical GPS receiver it was at any 
>> position and was at any time in the world.   All it takes is a decent 
>> directional antenna, some SDR hardware, and depending on the location and 
>> directivity of your antenna maybe a smallish amplifier.   There is much 
>> discussion right now in the PNT (Position, Navigation and Timing) community 
>> as to how best to secure the GNSS network, but right now one should consider 
>> the data from GPS to be no more trustworthy than some random NTP server on 
>> the internet.
>> 
>> In order to build a resilient NTP server infrastructure you need multiple 
>> sources of time distributed by multiple methods - typically both via 
>> satellite (GPS) and by terrestrial (NTP) methods.   NTP does a pretty good 
>> job of sorting out multiple time servers and discarding sources that are 
>> lying.  But to do this you need multiple time sources.  A common 
>> recommendation is to run a couple/few NTP servers which only get time from a 
>> GPS receiver and only serve time to a second tier of servers that pull from 
>> both those in-house GPS-timed-NTP servers and other trusted NTP servers.   
>> I'd recommend selecting the time servers to gain geographic diversity, i.e. 
>> poll NIST servers in Maryland and Colorado, and possibly both.
>> 
>> Note that NIST will exchange (via mail) a set of keys with you to talk 
>> encrypted NTP with you.   See 
>> https://www.nist.gov/pml/time-and-freque

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

It works fine, and is an industry standard. you have to mount the GPS antenna 
near a window with sky visibility, or on the roof. Many point-to-point 
microwave radios have GPS built in to obtain accurate timing for transmission 
multiplexing.

 -mel

On Aug 8, 2023, at 7:16 AM, Mike Hammett  wrote:


"We use these exclusively in data centers"

How well does GPS work inside the datacenter?



-
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>

From: "Mel Beckman" 
To: "Mark Tinka" 
Cc: nanog@nanog.org
Sent: Saturday, August 5, 2023 2:26:37 PM
Subject: Re: NTP Sync Issue Across Tata (Europe)

Mark,

You might consider setting up your own GPS-based NTP network. Commercial 
Ethernet GPS-sourced NTP servers, such as the Time Machines, TM1000A, are as 
little as $400. Or you can roll your own using a Raspberry Pi or similar nano 
computer with a GPS module and antenna. We use these exclusively in data 
centers now rather than depending on Internet NTP servers, primarily for 
security, because financial transactions in e-commerce can be sensitive to 
false time information. There are also a variety of NTP-based Internet attacks, 
so if you can block NTP at your border you’ve eliminated another attack surface.

-mel via cell

> On Aug 5, 2023, at 11:22 AM, Mark Tinka  wrote:
>
> 
>
>> On 8/5/23 20:17, Chris Adams wrote:
>>
>> It's the NTP pool people you need to talk to - the .freebsd. bit is just
>> a vendored entry into the pool (more for load tracking and management).
>
> Yes, Andreas clarified in unicast. Will do. Thanks.
>
> Mark.

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

Masataka,

To be useful, any atomic clocks you operate must be synchronized to a Stratum 
Zero time source, such as GPS. Such clocks are useful when you need exceptional 
accuracy, such as for Building Integrated Timing Service (BITS), but unless 
they’re synchronized you can’t coordinate time-sensitive activities such as 
digital certificate validation with anyone else on the Internet.

From 
https://www.gps.gov/applications/timing/


Each GPS satellite contains multiple atomic clocks that contribute very precise 
time data to the GPS signals. GPS receivers decode these signals, effectively 
synchronizing each receiver to the atomic clocks. This enables users to 
determine the time to within 100 billionths of a second, without the cost of 
owning and operating atomic clocks.

Precise time is crucial to a variety of economic activities around the world. 
Communication systems, electrical power grids, and financial networks all rely 
on precision timing for synchronization and operational efficiency. The free 
availability of GPS time has enabled cost savings for companies that depend on 
precise time and has led to significant advances in capability.

-mel via cell

On Aug 7, 2023, at 10:04 PM, Masataka Ohta  
wrote:

Forrest Christian (List Account) wrote:

In the middle tends to be a more moderate solution which involves a mix of
time transmission methods from a variety of geographically and/or network
diverse sources.  Taking time from the public trusted ntp servers and
adding lower cost GPS receivers at diverse points in your network seems
like a good compromise in the middle.  That way,  only coordinated attacks
will be successful.

Instead, just rely on atomic clocks operated by you. They are not
so expensive (several thousand dollars) and should be accurate
enough without adjustment for hundreds of years. There can be no
coordinated attacks. They may be remotely accessed through
secured NTP.

   Masataka Ohta

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

Forrest,

GPS spoofing may work with a primitive Raspberry Pi-based NTP server, but 
commercial industrial NTP servers have specific anti-spoofing mitigations. 
There are also antenna diversity strategies that vendors support to ensure the 
signal being relied upon is coming from the right direction. It’s a problem 
that has received a lot of attention in both NTP and aviation navigation 
circles. What is hard to defend against is total signal suppression via high 
powered jamming. But that you can do with a geographically diverse GPS NTP 
network.

 -mel

On Aug 7, 2023, at 1:39 AM, Forrest Christian (List Account) 
 wrote:


The problem with relying exclusively on GPS to do time distribution is the ease 
with which one can spoof the GPS signals.

With a budget of around $1K, not including a laptop, anyone with decent 
technical skills could convince a typical GPS receiver it was at any position 
and was at any time in the world.   All it takes is a decent directional 
antenna, some SDR hardware, and depending on the location and directivity of 
your antenna maybe a smallish amplifier.   There is much discussion right now 
in the PNT (Position, Navigation and Timing) community as to how best to secure 
the GNSS network, but right now one should consider the data from GPS to be no 
more trustworthy than some random NTP server on the internet.

In order to build a resilient NTP server infrastructure you need multiple 
sources of time distributed by multiple methods - typically both via satellite 
(GPS) and by terrestrial (NTP) methods.   NTP does a pretty good job of sorting 
out multiple time servers and discarding sources that are lying.  But to do 
this you need multiple time sources.  A common recommendation is to run a 
couple/few NTP servers which only get time from a GPS receiver and only serve 
time to a second tier of servers that pull from both those in-house 
GPS-timed-NTP servers and other trusted NTP servers.   I'd recommend selecting 
the time servers to gain geographic diversity, i.e. poll NIST servers in 
Maryland and Colorado, and possibly both.

Note that NIST will exchange (via mail) a set of keys with you to talk 
encrypted NTP with you.   See 
https://www.nist.gov/pml/time-and-frequency-division/time-services/nist-authenticated-ntp-service
 .



On Sun, Aug 6, 2023 at 8:36 PM Mel Beckman 
mailto:m...@beckman.org>> wrote:
GPS Selective Availability did not disrupt the timing chain of GPS, only the 
ephemeris (position information).  But a government-disrupted timebase scenario 
has never occurred, while hackers are a documented threat.

DNS has DNSSec, which while not deployed as broadly as we might like, at least 
lets us know which servers we can trust.

Your own atomic clocks still have to be synced to a common standard to be 
useful. To what are they sync’d? GPS, I’ll wager.

I sense hand-waving :)

-mel via cell

On Aug 6, 2023, at 7:04 PM, Rubens Kuhl 
mailto:rube...@gmail.com>> wrote:




On Sun, Aug 6, 2023 at 8:20 PM Mel Beckman 
mailto:m...@beckman.org>> wrote:
Or one can read recent research papers that thoroughly document the incredible 
fragility of the existing NTP hierarchy and soberly consider their 
recommendations for remediation:

The paper suggests the compromise of critical infrastructure. So, besides not 
using NTP, why not stop using DNS ? Just populate a hosts file with all you 
need.

BTW, the stratum-0 source you suggested is known to have been manipulated in 
the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to 
bet on that specific state actor not returning to old habits.

OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can 
keep using GPS as well. If GPS goes bananas on timing, that source will just be 
disregarded (one of the features of the NTP architecture that has been pointed 
out over and over in this thread and you keep ignoring it).

Rubens


--
- Forrest

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

GPS Selective Availability did not disrupt the timing chain of GPS, only the 
ephemeris (position information).  But a government-disrupted timebase scenario 
has never occurred, while hackers are a documented threat.

DNS has DNSSec, which while not deployed as broadly as we might like, at least 
lets us know which servers we can trust.

Your own atomic clocks still have to be synced to a common standard to be 
useful. To what are they sync’d? GPS, I’ll wager.

I sense hand-waving :)

-mel via cell

On Aug 6, 2023, at 7:04 PM, Rubens Kuhl  wrote:




On Sun, Aug 6, 2023 at 8:20 PM Mel Beckman 
mailto:m...@beckman.org>> wrote:
Or one can read recent research papers that thoroughly document the incredible 
fragility of the existing NTP hierarchy and soberly consider their 
recommendations for remediation:

The paper suggests the compromise of critical infrastructure. So, besides not 
using NTP, why not stop using DNS ? Just populate a hosts file with all you 
need.

BTW, the stratum-0 source you suggested is known to have been manipulated in 
the past (https://www.gps.gov/systems/gps/modernization/sa/), so you need to 
bet on that specific state actor not returning to old habits.

OTOH, 4 of the 5 servers I suggested have their own atomic clock, and you can 
keep using GPS as well. If GPS goes bananas on timing, that source will just be 
disregarded (one of the features of the NTP architecture that has been pointed 
out over and over in this thread and you keep ignoring it).

Rubens

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

Bill,

You’re mistaking targeted NTP attacks with global ones. Yes, to attack your 
specific NTP client, the attacker has to know which NTP servers you’re using. 
But to simply succeed at random attacks, the attacker need only spoof popular 
servers. This is how time-shifting attacks work. Once an attack succeeds with a 
random victim, the hacker goes to work compromising time-sensitive security 
services.

And don’t forget that anyone can join pool.ntp.org. Including hackers. There is 
no credentialed vetting process.

 -mel

> On Aug 6, 2023, at 4:30 PM, William Herrin  wrote:
> 
> On Sun, Aug 6, 2023 at 1:19 PM Royce Williams  .com> wrote:
>> Wouldn't a robust implementation of peering - say, seven peers, with the NTP 
>> algorithm handily selecting a subset to peer with for each cycle - require 
>> an attacker to know and overwhelm not just one, but a majority of the peer 
>> IPs?
> 
> Hi Royce,
> 
> More or less, yes. There are some corner cases where that may not be
> true, but in terms of designing a useful attack, the attacker has to
> be able to figure out which NTP servers you're talking to, flood you
> with enough packets forged from all of them that the forged packet
> arrives ahead of the real one, generally no more than 10s of
> milliseconds behind the sent packet that only happens once every 15
> minutes or so. And then it has to move you off real time gradually
> enough for the NTP daemon not to decide the peer has become a false
> ticker.
> 
> It's like DNS cache poisoning but a few orders of magnitude harder.
> 
> 
>> This is *not* to say that I'm advocating against maintaining local stratum 
>> 0s as part of a proper operator-grade NTP mesh. (My definition of "robust 
>> implementation" would include both local stratum 0 and a healthy serving of 
>> Internet stratum 1s). I'm just suggesting "don't peer with public servers" 
>> seems draconian given the deliberate design choices of the protocol.
>> 
>> For this audience, this would recommend a tiered system where multiple 
>> mixed-stratum internal stratrum 0+1s would peer with each other, and 
>> maintain internal-facing consensus for all other downstream internal devices 
>> - such that the vast majority of internal peers would indeed not be talking 
>> to the public Internet (but the "parent" peers would, and have the mesh and 
>> mix that I describe).
> 
> 
> Well, it's not really one-size-fits-all.
> 
> Some systems consist of a well defined network containing routers and
> servers with clear boundaries between self and Internet. Your approach
> would work well there.
> 
> Some systems consist of equipment scattered at various data centers,
> interconnected by the Internet itself. Implementing a GPS time
> receiver at every one could get very expensive very fast. Standard
> security rules apply: don't spend more protecting yourself than the
> risk-cost. Risk is threat times vulnerability. Given the complexity of
> the attack, you have to consider whether you're enough of a target
> (threat) for anyone to bother.
> 
> Some systems are heavily virtualized, scattered over many vendors and
> locations. You could -try- to track down a "secured" NTP source from
> the vendor at each location, but that way lies configuration insanity.
> 
> Some systems are air-gapped from the Internet. You're going to get
> time from GPS or the cellular phone network. GPS devices like the one
> Mel pointed out are probably cheaper and more accurate.
> 
> Regards,
> Bill Herrin
> 
> 
> -- 
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/

Fwd: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

Or one can read recent research papers that thoroughly document the incredible 
fragility of the existing NTP hierarchy and soberly consider their 
recommendations for remediation:

https://www.ndss-symposium.org/wp-content/uploads/ndss2021_1A-2_24302_paper.pdf

Or simply use non-Internet NTP servers based on a Stratum-0 GPS source for 
mission-critical network timing.

Until then, we may all wake up one morning and discover massive data breaches 
traced to an unfounded reliance on insecure public NTP servers. Then the game 
truly will be over, but not in our favor.

 -mel

On Aug 6, 2023, at 2:35 PM, Rubens Kuhl  wrote:

Or one can select NTS-capable NTP servers, like those 5:
a.st1.ntp.br
b.st1.ntp.br
c.st1.ntp.br
d.st1.ntp.br
gps.ntp.br

Or any other NTP server that has NTS deployed. Game-over for NTP impersonation.


Rubens

On Sun, Aug 6, 2023 at 4:41 PM Mel Beckman  wrote:

In a nutshell, no. Refer to my prior cites for detailed explanations. For a 
list of real-world attack incidents, see

https://en.m.wikipedia.org/wiki/NTP_server_misuse_and_abuse#


-mel

On Aug 6, 2023, at 12:03 PM, Royce Williams  wrote:


Naively, instead of abstaining ;) ... isn't robust diversity of NTP peering a 
reasonable mitigation for this, as designed?

Royce

On Sun, Aug 6, 2023 at 10:21 AM Mel Beckman  wrote:

William,

Due to flaws in the NTP protocol, a simple UDP filter is not enough. These 
flaws make it trivial to spoof NTP packets, and many firewalls have no specific 
protection against this. in one attack the malefactor simply fires a continuous 
stream of NTP packets with invalid time at your firewall. When your NTP client 
queries the spoofed server, the malicious packet is the one you likely receive.

That’s just one attack vector. There are several others, and all have complex 
remediation. Why should people bother being exposed to the risk at all? Simply 
avoid Internet-routed NTP. there are many solutions, as I’ve already described. 
Having suffered through such attacks more than once, I can say from personal 
experience that you don’t want to risk it.

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

In a nutshell, no. Refer to my prior cites for detailed explanations. For a 
list of real-world attack incidents, see

https://en.m.wikipedia.org/wiki/NTP_server_misuse_and_abuse#<https://en.m.wikipedia.org/wiki/NTP_server_misuse_and_abuse#:~:text=NTP%20server%20misuse%20and%20abuse%20covers%20a%20number%20of%20practices,the%20NTP%20rules%20of%20engagement.>


 -mel

On Aug 6, 2023, at 12:03 PM, Royce Williams  wrote:


Naively, instead of abstaining ;) ... isn't robust diversity of NTP peering a 
reasonable mitigation for this, as designed?

Royce

On Sun, Aug 6, 2023 at 10:21 AM Mel Beckman 
mailto:m...@beckman.org>> wrote:
William,

Due to flaws in the NTP protocol, a simple UDP filter is not enough. These 
flaws make it trivial to spoof NTP packets, and many firewalls have no specific 
protection against this. in one attack the malefactor simply fires a continuous 
stream of NTP packets with invalid time at your firewall. When your NTP client 
queries the spoofed server, the malicious packet is the one you likely receive.

That’s just one attack vector. There are several others, and all have complex 
remediation. Why should people bother being exposed to the risk at all? Simply 
avoid Internet-routed NTP. there are many solutions, as I’ve already described. 
Having suffered through such attacks more than once, I can say from personal 
experience that you don’t want to risk it.

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

William,

Due to flaws in the NTP protocol, a simple UDP filter is not enough. These 
flaws make it trivial to spoof NTP packets, and many firewalls have no specific 
protection against this. in one attack the malefactor simply fires a continuous 
stream of NTP packets with invalid time at your firewall. When your NTP client 
queries the spoofed server, the malicious packet is the one you likely receive.

That’s just one attack vector. There are several others, and all have complex 
remediation. Why should people bother being exposed to the risk at all? Simply 
avoid Internet-routed NTP. there are many solutions, as I’ve already described. 
Having suffered through such attacks more than once, I can say from personal 
experience that you don’t want to risk it.

 -mel 

> On Aug 6, 2023, at 10:53 AM, William Herrin  wrote:
> 
> On Sat, Aug 5, 2023 at 7:24 PM Mel Beckman  wrote:
>> That still leaves you open to NTP attacks. The USNO accuracy and monitoring 
>> is worthless if you suffer, for example, an NTP DDoS attack.
> 
> Hi Mel,
> 
> From what I can tell, a fairly simple firewall policy of allow UDP 123
> from known NTP clients and established connections (I sent them a UDP
> packet recently) stops every one of those attacks (that's actually an
> NTP attack and not something else like a DNS attack) except for
> upstream address hijack that happens to coincide with your system
> boot. And it still depends on the attacker executing an additional
> sophisticated attack to do more than cause you a denial of service.
> 
> The links you sent are very interesting, at least in an academic
> sense, but they don't cause me to be unduly concerned about employing
> NTP.
> 
> 
>> if you can eliminate such security problems for $400, I say it’s cheap at 
>> twice the price.
> 
> Except you can't. Redundancy is required for any critical service. At
> the $400 price point, your approach has multiple
> single-points-of-failure. The device itself of course. Your ability to
> receive continuous non-jammed GPS signals at the location where you're
> able to place an antenna. And in your plan you'll need one of these in
> every discontiguous network where you have equipment since you're not
> doing NTP over the Internet.
> 
> Not to mention the operations cost. Keeping track of a six inch brick
> with a wall wart and an antenna installed at a remote site is... not
> entirely abnormal but it's a one-off that consumes manpower.
> 
> And then you're only vulnerable to the litany of Internet attacks
> which don't involve NTP. Yay!
> 
> Don't get me wrong: the Time Machines TM1000A you recommended looks
> like a cool little device well worth checking into. As a supplement to
> Internet NTP, not a replacement.
> 
> Regards,
> Bill Herrin
> 
> 
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

Niels,

You’re the first person to mention neutral collocation facilities as a 
requirement. The OP only talked about servers generally. Obviously, building 
your own GPS-based NTP network requires you have visibility to the sky. 
However, that need not be rooftop access. We routinely locate GPS antennae for 
time synch at CLEC colos in an available window, behind the glass. Nobody has 
yet charged us anything more than a one-time cabling fee for that.

But most carrier-neutral colos already have their own GPS-derived, non-Internet 
time source to which you can subscribe. There is thus no reason to build your 
own airgapped network, as one is already available from the facility. If you 
don’t need PTP 50-μs-precise timing for SONET and whatnot, ordinary NTP is very 
cost effective. For example:

<https://docs.equinix.com/en-us/Content/Edge-Services/EPT/EPT.htm>
About Equinix Precision 
Time<https://docs.equinix.com/en-us/Content/Edge-Services/EPT/EPT.htm>
docs.equinix.com<https://docs.equinix.com/en-us/Content/Edge-Services/EPT/EPT.htm>
[favicon-16x16.png]<https://docs.equinix.com/en-us/Content/Edge-Services/EPT/EPT.htm>

Most colos also offer their own Stratum-1 public NTP servers for free. For 
example, Hurricane Electric (https://www.he.net/adm/ntp.html). Being you’re 
already on-net in the colo, that would still give you NTP that doesn’t transit 
the Internet. Still way better than pool.ntp.org!


 -mel

On Aug 6, 2023, at 4:53 AM, Niels Bakker  wrote:

* m...@beckman.org (Mel Beckman) [Sun 06 Aug 2023, 04:26 CEST]:
if you can eliminate such security problems for $400, I say it’s cheap at twice 
the price.

You must be unfamiliar with the prices neutral colocation facilities charge for 
roof access.


   -- Niels.

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

Bill,

That still leaves you open to NTP attacks. The USNO accuracy and monitoring is 
worthless if you suffer, for example, an NTP DDoS attack.

<https://www.cloudflare.com/learning/ddos/ntp-amplification-ddos-attack/>
[ddos-lc.png]
NTP amplification DDoS 
attack<https://www.cloudflare.com/learning/ddos/ntp-amplification-ddos-attack/>
cloudflare.com<https://www.cloudflare.com/learning/ddos/ntp-amplification-ddos-attack/>


There  are also replay and Man in the middle attacks (MITM) which can corrupt 
local NTP servers’ time basis. Worse, security flaws in NTP make others 
security protocols, such as SSL, vulnerable.

https://www.sidn.nl/en/news-and-blogs/security-flaws-in-network-time-protocol-make-other-security-protocols-vulnerable

if you can eliminate such security problems for $400, I say it’s cheap at twice 
the price.

 -mel

On Aug 5, 2023, at 6:18 PM, William Herrin  wrote:

On Sat, Aug 5, 2023 at 12:26 PM Mel Beckman  wrote:
You might consider setting up your own GPS-based NTP network.

GPS time is monitored (and when necessary, adjusted) from the U.S.
Naval Observatory Master Clock, which is -the- authoritative time
source for the United States. The USNO also provides an NTP time
source from the same master clock:

https://www.cnmoc.usff.navy.mil/Our-Commands/United-States-Naval-Observatory/Precise-Time-Department/Network-Time-Protocol-NTP/

You -should not- just point your servers there, but it's useful to
point a few servers each at one of them in order to serve as your
network stratum 2 sources that keep the rest of your machines in sync
with each other.

That last point is key. You don't want your servers in sync with
random Internet time sources. You want them in sync with each other.

Regards,
Bill Herrin



--
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: NTP Sync Issue Across Tata (Europe)

[Mel Beckman]

Mark,

You might consider setting up your own GPS-based NTP network. Commercial 
Ethernet GPS-sourced NTP servers, such as the Time Machines, TM1000A, are as 
little as $400. Or you can roll your own using a Raspberry Pi or similar nano 
computer with a GPS module and antenna. We use these exclusively in data 
centers now rather than depending on Internet NTP servers, primarily for 
security, because financial transactions in e-commerce can be sensitive to 
false time information. There are also a variety of NTP-based Internet attacks, 
so if you can block NTP at your border you’ve eliminated another attack surface.

-mel via cell

> On Aug 5, 2023, at 11:22 AM, Mark Tinka  wrote:
> 
> 
> 
>> On 8/5/23 20:17, Chris Adams wrote:
>> 
>> It's the NTP pool people you need to talk to - the .freebsd. bit is just
>> a vendored entry into the pool (more for load tracking and management).
> 
> Yes, Andreas clarified in unicast. Will do. Thanks.
> 
> Mark.

Re: Flapping Transport

[Mel Beckman]

 Ask them for an RFO report. They should be able to explain the diagnostics and 
reason for the original flapping. It's quite possible they found a problem 
somewhere on the path with a DWDM multiplexor or something. That wouldn't need 
any card replacement on your prem.

 -mel

From: NANOG  on behalf of Mike Hammett 

Sent: Tuesday, August 1, 2023 11:18 AM
To: NANOG 
Subject: Flapping Transport

I have a wave transport vendor that suffered issues twice about ten days apart, 
causing my link to flap a bunch. I put in a ticket on the second set of 
occurrences. I was told that there was a card issue identified and would be 
notified when the replacement happened. Ticket closed.

Three weeks later, I opened a new ticket asking for the status. The new card 
arrived the next day, but since no more flaps were happening, the card would 
not be replaced. Ticket closed.


A) It doesn't seem like they actually did anything to fix the circuit.
B) They admitted a problem and sent a new card.
C) They later decided to not do anything.


Is that normal?
Is that acceptable?


To avoid issues flapping causes, I disabled that circuit until repaired, but it 
seems like they're not going to do anything and I only know that because I 
asked.



-
Mike Hammett
Intelligent Computing Solutions
[http://www.ics-il.com/images/fbicon.png][http://www.ics-il.com/images/googleicon.png][http://www.ics-il.com/images/linkedinicon.png][http://www.ics-il.com/images/twittericon.png]
Midwest Internet Exchange
[http://www.ics-il.com/images/fbicon.png][http://www.ics-il.com/images/linkedinicon.png][http://www.ics-il.com/images/twittericon.png]
The Brothers WISP
[http://www.ics-il.com/images/fbicon.png][http://www.ics-il.com/images/youtubeicon.png]

Re: Request for assistance with Verizon FIOS connection

[Mel Beckman]

Neil,

Sorry, I must have missed that branch of the thread. When you had a computer 
directly connected, did you happen to run a wire shark capture? That wouldn’t 
require an Ethernet tap.

 -mel

> On Jul 15, 2023, at 7:44 AM, Neil Hanlon  wrote:
> 
> On 15.07.2023 07:43, Mel Beckman wrote:
>> Matt,
>> 
>> I missed where the OP indicated they've tried both a direct laptop 
>> connection as well as another router. I think you may have seen my reply 
>> suggesting that and thought that was the OP stating he'd done it.
> 
> I missed it in my initial post, but had responded to your first reply in the 
> thread saying I tried both my laptop and
> another (off the shelf) router.
> 
> I didn't make any changes on my end here, but the connection has been 
> (seemingly) stable for the past ~17 hours now.
> 
> I will continue to monitor.
> 
> Thanks all!
> 
> -Neil
> 
>> 
>> -mel
>> 
>> From: Matt Corallo 
>> Sent: Friday, July 14, 2023 9:44 PM
>> To: Mel Beckman ; Neil Hanlon ; 
>> nanog@nanog.org 
>> Subject: Re: Request for assistance with Verizon FIOS connection
>> 
>> OP indicated they've tried both a direct laptop connection as well as 
>> another router. That seems to
>> meet the requirement for having ruled out his home-made router, though 
>> obviously I agree one should
>> attempt to rule out any possible errors by doing transparent packet sniffing 
>> analyzing the problem
>> carefully before escalating an issue. Hopefully everyone on this list knows 
>> the value of the tech on
>> the other end of the line's time :)
>> 
>> Matt
>> 
>>> On 7/14/23 9:07 PM, Mel Beckman wrote:
>>> Getting the FCC involved seems premature, since the OP hasn't yet ruled out 
>>> a problem with his home
>>> made router. Not that there's anything wrong with making your own router, 
>>> but it seems there is a
>>> burden of proof on the end user to demonstrate the problem isn't at with 
>>> the CPE. Even a test as
>>> simple as connecting a laptop up for a day and running pings would rule out 
>>> the CPE.
>>> 
>>>   -mel
>>> 
>>> *From:* NANOG  on behalf of Matt 
>>> Corallo 
>>> *Sent:* Friday, July 14, 2023 5:46 PM
>>> *To:* Neil Hanlon ; nanog@nanog.org 
>>> *Subject:* Re: Request for assistance with Verizon FIOS connection
>>> I've always had good luck with https://consumercomplaints.fcc.gov/hc/en-us
>>> <https://consumercomplaints.fcc.gov/hc/en-us>. This tends to result in
>>> a higher-level tech getting assigned to your ticket at least at larger 
>>> providers. Depending on where
>>> you are, your local government may have a similar process (e.g. in NYC the 
>>> city has a similar
>>> process that tends to get very high priority tech attention as city council 
>>> members will rake
>>> providers over the coals on individual complaints come contract-renewal 
>>> time).
>>> 
>>> Matt
>>> 
>>> On 7/14/23 8:01 AM, Neil Hanlon wrote:
>>>> Hi all - I apoligize for the not-necessarily-on-topic post, but I've been 
>>>> struggling with this issue
>>>> for the past two
>>>> weeks and am about out of ideas and options other than ask here.
>>>> 
>>>> The short version is I recently got FIOS at my (new) house, and plugged in 
>>>> my router (SFF PC running
>>>> Vyos). Initially,
>>>> all was fine, however, some time later, connectivity to the gateway given 
>>>> by the DHCP server is
>>>> completely lost. If I
>>>> force a renewal, the gateway (sometimes) comes back--sometimes not. When 
>>>> it doesn't work, the
>>>> DHCPDISCOVER process has
>>>> to start over again and I often recive a lease in a completely different 
>>>> subnet--which isn't really
>>>> the problem, but
>>>> seems to be symptomatic of whatever is happening upstream of me.
>>>> 
>>>> The problem, from my perspective, is that the IPv4 gateway given to me in 
>>>> my DHCP lease goes away
>>>> before my lease
>>>> expires--leading to broken v4 connectivity until either 1. the system goes 
>>>> to renew the lease and
>>>> fails, starting over;
>>>> or 2. A watchdog notices and renews the 

Re: Request for assistance with Verizon FIOS connection

[Mel Beckman]

Matt,

I missed where the OP indicated they've tried both a direct laptop connection 
as well as another router. I think you may have seen my reply suggesting that 
and thought that was the OP stating he'd done it.

-mel

From: Matt Corallo 
Sent: Friday, July 14, 2023 9:44 PM
To: Mel Beckman ; Neil Hanlon ; 
nanog@nanog.org 
Subject: Re: Request for assistance with Verizon FIOS connection

OP indicated they've tried both a direct laptop connection as well as another 
router. That seems to
meet the requirement for having ruled out his home-made router, though 
obviously I agree one should
attempt to rule out any possible errors by doing transparent packet sniffing 
analyzing the problem
carefully before escalating an issue. Hopefully everyone on this list knows the 
value of the tech on
the other end of the line's time :)

Matt

On 7/14/23 9:07 PM, Mel Beckman wrote:
> Getting the FCC involved seems premature, since the OP hasn't yet ruled out a 
> problem with his home
> made router. Not that there's anything wrong with making your own router, but 
> it seems there is a
> burden of proof on the end user to demonstrate the problem isn't at with the 
> CPE. Even a test as
> simple as connecting a laptop up for a day and running pings would rule out 
> the CPE.
>
>-mel
> 
> *From:* NANOG  on behalf of Matt 
> Corallo 
> *Sent:* Friday, July 14, 2023 5:46 PM
> *To:* Neil Hanlon ; nanog@nanog.org 
> *Subject:* Re: Request for assistance with Verizon FIOS connection
> I've always had good luck with https://consumercomplaints.fcc.gov/hc/en-us
> <https://consumercomplaints.fcc.gov/hc/en-us>. This tends to result in
> a higher-level tech getting assigned to your ticket at least at larger 
> providers. Depending on where
> you are, your local government may have a similar process (e.g. in NYC the 
> city has a similar
> process that tends to get very high priority tech attention as city council 
> members will rake
> providers over the coals on individual complaints come contract-renewal time).
>
> Matt
>
> On 7/14/23 8:01 AM, Neil Hanlon wrote:
>> Hi all - I apoligize for the not-necessarily-on-topic post, but I've been 
>> struggling with this issue
>> for the past two
>> weeks and am about out of ideas and options other than ask here.
>>
>> The short version is I recently got FIOS at my (new) house, and plugged in 
>> my router (SFF PC running
>> Vyos). Initially,
>> all was fine, however, some time later, connectivity to the gateway given by 
>> the DHCP server is
>> completely lost. If I
>> force a renewal, the gateway (sometimes) comes back--sometimes not. When it 
>> doesn't work, the
>> DHCPDISCOVER process has
>> to start over again and I often recive a lease in a completely different 
>> subnet--which isn't really
>> the problem, but
>> seems to be symptomatic of whatever is happening upstream of me.
>>
>> The problem, from my perspective, is that the IPv4 gateway given to me in my 
>> DHCP lease goes away
>> before my lease
>> expires--leading to broken v4 connectivity until either 1. the system goes 
>> to renew the lease and
>> fails, starting over;
>> or 2. A watchdog notices and renews the lease (This is what I have attempted 
>> to implement, without
>> much success).
>>
>> As a note, IPv6 connectivity (dhcpv6-pd, receiving a /56) is entirely 
>> unaffected when IPv4
>> connectivity breaks.
>>
>> For the past week, I have been monitoring to various IPv4 and IPv6 endpoints 
>> over ICMP and TCP, and
>> have been able to
>> chart the outages over that period. More or less, every two hours, shortly 
>> after a lease is renewed,
>> the gateway
>> disappears. I'm happy to share more details and graphs/logs with anyone who 
>> might be able to help.
>>
>> I have attempted to contact FIOS support several times and even had a 
>> trouble ticket opened at one
>> point--though this
>> has been closed as they cannot apparently find any issue with the ONT.
>>
>> I'm at my wit's end with this issue and would really appreciate any and all 
>> help. Please contact me
>> off list if you need
>> additional details--I can provide ticket numbers/conversation IDs/etc, as 
>> well as graphs/logs/etc.
>>
>> Best,
>> Neil Hanlon

Re: Request for assistance with Verizon FIOS connection

[Mel Beckman]

Getting the FCC involved seems premature, since the OP hasn't yet ruled out a 
problem with his home made router. Not that there's anything wrong with making 
your own router, but it seems there is a burden of proof on the end user to 
demonstrate the problem isn't at with the CPE. Even a test as simple as 
connecting a laptop up for a day and running pings would rule out the CPE.

  -mel

From: NANOG  on behalf of Matt Corallo 

Sent: Friday, July 14, 2023 5:46 PM
To: Neil Hanlon ; nanog@nanog.org 
Subject: Re: Request for assistance with Verizon FIOS connection

I've always had good luck with https://consumercomplaints.fcc.gov/hc/en-us. 
This tends to result in
a higher-level tech getting assigned to your ticket at least at larger 
providers. Depending on where
you are, your local government may have a similar process (e.g. in NYC the city 
has a similar
process that tends to get very high priority tech attention as city council 
members will rake
providers over the coals on individual complaints come contract-renewal time).

Matt

On 7/14/23 8:01 AM, Neil Hanlon wrote:
> Hi all - I apoligize for the not-necessarily-on-topic post, but I've been 
> struggling with this issue
> for the past two
> weeks and am about out of ideas and options other than ask here.
>
> The short version is I recently got FIOS at my (new) house, and plugged in my 
> router (SFF PC running
> Vyos). Initially,
> all was fine, however, some time later, connectivity to the gateway given by 
> the DHCP server is
> completely lost. If I
> force a renewal, the gateway (sometimes) comes back--sometimes not. When it 
> doesn't work, the
> DHCPDISCOVER process has
> to start over again and I often recive a lease in a completely different 
> subnet--which isn't really
> the problem, but
> seems to be symptomatic of whatever is happening upstream of me.
>
> The problem, from my perspective, is that the IPv4 gateway given to me in my 
> DHCP lease goes away
> before my lease
> expires--leading to broken v4 connectivity until either 1. the system goes to 
> renew the lease and
> fails, starting over;
> or 2. A watchdog notices and renews the lease (This is what I have attempted 
> to implement, without
> much success).
>
> As a note, IPv6 connectivity (dhcpv6-pd, receiving a /56) is entirely 
> unaffected when IPv4
> connectivity breaks.
>
> For the past week, I have been monitoring to various IPv4 and IPv6 endpoints 
> over ICMP and TCP, and
> have been able to
> chart the outages over that period. More or less, every two hours, shortly 
> after a lease is renewed,
> the gateway
> disappears. I'm happy to share more details and graphs/logs with anyone who 
> might be able to help.
>
> I have attempted to contact FIOS support several times and even had a trouble 
> ticket opened at one
> point--though this
> has been closed as they cannot apparently find any issue with the ONT.
>
> I'm at my wit's end with this issue and would really appreciate any and all 
> help. Please contact me
> off list if you need
> additional details--I can provide ticket numbers/conversation IDs/etc, as 
> well as graphs/logs/etc.
>
> Best,
> Neil Hanlon

Re: Request for assistance with Verizon FIOS connection

[Mel Beckman]

The first thing I would do is to try a different RJ45 cable AND router to rule 
out your cable or homemade router being the problem. Yes, you’ll have to pick 
up a cheap router, but any $50 gadget, such as a Mikrotik RB or Ubiquiti ER-X 
will do. Most network techs have a garage full of castoff routers they can pull 
out in a pinch.

 -mel beckman

> On Jul 14, 2023, at 8:05 AM, Neil Hanlon  wrote:
> 
> Hi all - I apoligize for the not-necessarily-on-topic post, but I've been 
> struggling with this issue for the past two
> weeks and am about out of ideas and options other than ask here.
> 
> The short version is I recently got FIOS at my (new) house, and plugged in my 
> router (SFF PC running Vyos). Initially,
> all was fine, however, some time later, connectivity to the gateway given by 
> the DHCP server is completely lost. If I
> force a renewal, the gateway (sometimes) comes back--sometimes not. When it 
> doesn't work, the DHCPDISCOVER process has
> to start over again and I often recive a lease in a completely different 
> subnet--which isn't really the problem, but
> seems to be symptomatic of whatever is happening upstream of me.
> 
> The problem, from my perspective, is that the IPv4 gateway given to me in my 
> DHCP lease goes away before my lease
> expires--leading to broken v4 connectivity until either 1. the system goes to 
> renew the lease and fails, starting over;
> or 2. A watchdog notices and renews the lease (This is what I have attempted 
> to implement, without much success).
> 
> As a note, IPv6 connectivity (dhcpv6-pd, receiving a /56) is entirely 
> unaffected when IPv4 connectivity breaks.
> 
> For the past week, I have been monitoring to various IPv4 and IPv6 endpoints 
> over ICMP and TCP, and have been able to
> chart the outages over that period. More or less, every two hours, shortly 
> after a lease is renewed, the gateway
> disappears. I'm happy to share more details and graphs/logs with anyone who 
> might be able to help.
> 
> I have attempted to contact FIOS support several times and even had a trouble 
> ticket opened at one point--though this
> has been closed as they cannot apparently find any issue with the ONT.
> 
> I'm at my wit's end with this issue and would really appreciate any and all 
> help. Please contact me off list if you need
> additional details--I can provide ticket numbers/conversation IDs/etc, as 
> well as graphs/logs/etc.
> 
> Best,
> Neil Hanlon

Re: Are we back to the 2000's again?

[Mel Beckman]

Sure you would. Water companies have reservoirs, and often people put boats in 
the water, move them from dock to dock, take them out of the water, and 
occasionally they get stolen. It’s not the water companies fault.

-mel via cell

On Jun 3, 2023, at 3:08 PM, Terrence de Kat  
wrote:



You wouldn't download a boat ;)

--
Regards,
   Terrence de Kat, PhD/MTh/BPsy
 Darkness Reigns (Holding) B.V.

Please quote relevant replies.

From: Mel Beckman 
Sent: Sunday, 4 June 2023 00:06
To: William Herrin
Cc: nanog@nanog.org
Subject: Re: Are we back to the 2000's again?


William,

But still, boats :)

-mel

> On Jun 3, 2023, at 2:58 PM, William Herrin  wrote:
>
> On Sat, Jun 3, 2023 at 2:03 PM Michael Thomas  wrote:
>> Am I missing something?
>
> That it's old news from 2019? Cox and RIAA are in the appeals process
> from the 2019 verdict.
>
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/

Re: Are we back to the 2000's again?

[Mel Beckman]

William,

But still, boats :)

 -mel

> On Jun 3, 2023, at 2:58 PM, William Herrin  wrote:
> 
> On Sat, Jun 3, 2023 at 2:03 PM Michael Thomas  wrote:
>> Am I missing something?
> 
> That it's old news from 2019? Cox and RIAA are in the appeals process
> from the 2019 verdict.
> 
> -- 
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/

Re: Are we back to the 2000's again?

[Mel Beckman]

It’s like blaming water companies for people stealing boats :)

 -mel beckman

> On Jun 3, 2023, at 2:06 PM, Michael Thomas  wrote:
> 
> 
> Apparently the RIAA is back suing ISP's (Cox in this case) for users pirating 
> music. It was pretty bogus back then, but with the uptake of TLS for almost 
> everything and DoH to conceal DNS requests what exactly is an ISP supposed to 
> do these days? Throw in a VPN and the pirates completely cut out the ISP.
> 
> Am I missing something?
> 
> 
> https://yro.slashdot.org/story/23/06/02/2043209/music-pirates-are-not-terrorists-record-labels-argue-in-court
> 
> Mike
> 

Re: Reliable neutral global networks for connectivity testing

[Mel Beckman]

Most backbone providers operate networks like this internally for their own SLA 
monitoring, which you can purchase, but what motivation would there be for 
someone to offer such services for free? After all, you can’t put an ad on a 
ping :-)

 -mel beckman

On May 5, 2023, at 9:43 AM, Dmitriy A.  wrote:


Hey all, I have a niche problem and wanted to get some feedback and ideas.

I am trying to find some global networks that accept ICMP requests that I can 
use to ping and based on the packet loss try to understand if the client is 
having connectivity issues or uses an unreliable internet connection. The exact 
testing logic is here 
https://github.com/jsdelivr/globalping-probe/blob/master/src/lib/status-manager.ts#L66
It basically requires 2/3 endpoints to have 0 packet loss when sending 6 
packets to each.

Some requirements:
- Must be a global anycasted network since the client devices can be anywhere
- Must be neutral. Meaning no government or ISP having a reason to block it. 
This eliminates Google, Microsoft, Akamai, Cloudflare... (blocked in China and 
elsewhere)
- Must be reliable to avoid false-positives as much as possible.
- Must be public and open. The network must be ok with random devices sending 
packets to them
- Must be safe and controlled by a trusted entity

I am currently trying to use DNS root servers https://root-servers.org/ but 
turns out many of them are unreliable for this use-case, I have noticed bad 
routing resulting in high latency and lots of packet loss at random.

Same goes for pool.ntp.org<http://pool.ntp.org> where some Chinese users hit 
Amsterdam for some reason. This results in high packet loss marking the test as 
false-positive.

Popular public DNS resolvers are also controlled by companies that are mostly 
likely to be banned in certain countries.

Am I missing any other networks that fit the requirements? Any ideas on how to 
go about this would be appreciated,

Thanks

Re: Standard DC rack rail distance, front to back question

[Mel Beckman]

We use shelves rather than hanging all the weight of racked gear on the ears. 
That rarely works well, but a 4-post shelf for every half-dozen or so devices 
works wonderfully. These shelves are usually quite adjustable.

 -mel beckman

On Apr 27, 2023, at 6:54 AM, Chuck Church  wrote:


Hey all.  Question about standard 4 post racks.  We bought some that are 
adjustable.  Unfortunately, the posts are very flimsy, as these are some fancy 
cabinets with spacing on the sides for vertical patch panels, etc.  We found 
that 2 post mounting of most Cisco devices (namely Cat 9500 1RU switches) are 
sagging quite bad.   We’re used to the new server type rails that extend to 
support most reasonable distances front rails to back for 4 post mounting.  
However, for a Cisco ASA1001, there aren’t rails, but rather front and back 
‘ears’ you use to hit both front and back posts.  These would appear to not 
have any adjustability, the front to back post distance would seem to need to 
match the ears, I assume they don’t adjust placement on the router much.  Is 
there a ‘standard’ distance between front and back rails that devices usually 
adhere to?  Googling didn’t find an answer readily.  These are 19” wide 
cabinets by the way.

Thanks,

Chuck

Re: BKA Wiesbaden - Abteilung Cybercrime (Not sure if this is a phishing E-mail or real...)

[Mel Beckman]

Any security “authority” that sends a warning email that requires opening _any_ 
attachment doesn’t deserve to be taken seriously. This include the MPAA et al. 
Also, if they don’t send it to your registered abuse email, into the trash it 
should go without a glance.

 -mel beckman

On Apr 3, 2023, at 4:37 AM, Suresh Ramasubramanian  wrote:


It appears legit.

BKA.DE is the German Bundeskriminalamt (Federal Police)

And the PTR records, SPF etc check out for the domain.

Might as well check the IP in question for malware if they’ve provided date / 
timestamps and such

--srs

From: NANOG  on behalf of Glen A. 
Pearce 
Date: Monday, 3 April 2023 at 12:29 PM
To: nanog@nanog.org 
Subject: BKA Wiesbaden - Abteilung Cybercrime (Not sure if this is a phishing 
E-mail or real...)
Hi All:

I received an E-mail with an attachment claiming something
on my network is infected and that I should look at the
attachment to find out what.

Normally I think everything with an attachment is phishing
to get me to run malware but:

#1: The sites linked to in it seem to be legit German
 government websites based on Wikipedia entries that
 haven't changed in several years.
 (Looked at archive.org)
#2: The attachment is a .txt file which I've normally
 assumed to be safe.
#3: None of the usual dead giveaways that most phishing
 E-mails have.

If it is a phishing E-mail it has got to be the cleverest
one I've ever seen, though someone would try to be cleaver
considering the target would be holders of IP blocks.

I right clicked and checked properties to make sure the
attached ip_addresses.txt file really is a text file and
not some fancy trickery with reverse direction characters
( As seen on https://www.youtube.com/watch?v=ieQUy8YTbFU )

I tried poking around to see if there was some vulnerability
in notepad (or some versions of it) that I didn't know about
and only found a vulnerability in the text editor on Macs
but nothing with Windows Notepad.

The other thing I felt was a bit off is that the originating
mail server is in Deutsche Telekom AG space and not IP Space
registered to the German government.  I'm thinking someone
could rent some IP space from Deutsche Telekom AG with a
connection to them in a data center and get the DNS delegated
to them so they could set the reverse DNS to whatever they want.
A lot of effort to try to look legit by coming out of Germany
and having a government domain in the reverse DNS to look like
a plausible legit outsourcing but again Network operators are
the target audience so the normal tricks that work on the
general public won't work with this group so I can see someone
going that far.

I'll attach the E-mail below with all headers.  Has anyone
else gotten these?  Is there some security risk opening it
in Windows Notepad that I don't know about or is it actually
safe to open this?


Return-Path: 
Delivered-To: [REDACTED]
Received: from ezp08-pco.easydns.vpn ([10.5.10.148])
 by ezb03-pco.easydns.vpn with LMTP
 id oCfeBO/yEmTokhgAzaFxkQ
 (envelope-from )
 for <[REDACTED]>; Thu, 16 Mar 2023 10:43:59 +
Received: from smtp.easymail.ca ([127.0.0.1])
 by ezp08-pco.easydns.vpn with LMTP
 id WCB5BO/yEmSHdgEABcrfzg
 (envelope-from ); Thu, 16 Mar 2023 10:43:59 +
Received: from localhost (localhost [127.0.0.1])
 by smtp.easymail.ca (Postfix) with ESMTP id 0DC85557DF
 for ; Thu, 16 Mar 2023 10:43:59 + (UTC)
X-Virus-Scanned: Debian amavisd-new at ezp08-pco.easydns.vpn
X-Spam-Flag: NO
X-Spam-Score: 0.075
X-Spam-Level:
X-Spam-Status: No, score=0.075 required=4 tests=[BAYES_00=-1.9,
 DEAR_SOMETHING=1.973, HTML_MESSAGE=0.001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=no autolearn_force=no
Received: from smtp.easymail.ca ([127.0.0.1])
 by localhost (ezp08-pco.easydns.vpn [127.0.0.1]) (amavisd-new, port
10024)
 with ESMTP id d0XbPteZN-Io for ;
 Thu, 16 Mar 2023 10:43:55 + (UTC)
Received: from mail.cyber.bka.de (mail.cyber.bka.de [80.146.190.22])
 by smtp.easymail.ca (Postfix) with ESMTPS id 0BC0C557DC
 for ; Thu, 16 Mar 2023 10:43:54 + (UTC)
Date: Thu, 16 Mar 2023 10:43:53 +
To: a...@ve4.ca
From: BKA Wiesbaden - Abteilung Cybercrime 
Reply-To: BKA Wiesbaden - Abteilung Cybercrime 
Subject: Information regarding possible infection with malware
Message-ID:

MIME-Version: 1.0
Content-Type: multipart/mixed;
  boundary="b1_M47LJRZpjy1zymUJDKsNtrYm3RimkafZfZTqeZpauZA"
Content-Transfer-Encoding: 8bit

Dear Sir or Madam,

As part of criminal proceedings, the German Federal Criminal Police
Office (Bundeskriminalamt) has been
informed about public IP addresses and timestamps which indicate a
potential infection by the malicious
software "Bumblebee" of one or more systems behind the respective public
IP address.

Within this letter, the BKA is providing you with the data of the
respective IP addresses which have been

Re: gtt/level3 is down?

[Mel Beckman]

LOL! Where did he/she say “both”? The reasonable interpretation of his/her 
question as asking if GTT *or* Level3 was down. Capiche/comprends?  :)

-mel via cell

> On Mar 1, 2023, at 6:55 AM, Jon Lewis  wrote:
> 
> On Wed, 1 Mar 2023, Dmitry Sherman via NANOG wrote:
> 
>> gtt/level3 is down?
> 
> Those are two totally different global networks...so asking if they're both 
> "down" is kind of a silly question.  Down where?
> 
> --
> Jon Lewis, MCP :)   |  I route
> StackPath, Sr. Neteng   |  therefore you are
> _ http://www.lewis.org/~jlewis/pgp for PGP public key_

Re: gtt/level3 is down?

[Mel Beckman]

I’m in SoCal Level3 now Lumen, and all has been well all night. Nothing down 
that I can see.

 -mel beckman

On Mar 1, 2023, at 5:29 AM, Dmitry Sherman via NANOG  wrote:


gtt/level3 is down?

Widespread connectivity problems in the west, possibly Lumen

[Mel Beckman]

We are getting reports of widespread connectivity problems: VPN failures, email 
delivery failures, RingCentral VoIP dropped calls, and the inability to reach 
O365. Affected users are as far East as Las Cruces, NM. All failing 
destinations are so far in California-based data centers. 

So far a common thread appears to be Lumen connectivity, although we can’t 
verify this is exclusive to Lumen transit. However, the reachability problems 
don’t seem to be getting resolved by BGP routing, so multi homed customers are 
still using lumen, even if lumen has no apparent reachability. 

 -mel

Re: SDN Internet Router (sir)

[Mel Beckman]

Mike,

Thanks for that useful example. On a side note, Netflix is a thorn in all our 
sides :) You could put a localpref filter route to override the default for 
Netflix prefixes, but this impacts resilience. Since you peer with Netflix, I 
suspect we probably agree that Netflix’s ideas on traffic engineering are 
pretty one sided.

I think it’s safe to say that BGP, which has scaled amazingly well, didn’t 
anticipate some of the big gorilla content systems. I don’t really see, though, 
how injecting FIB entries helps more than other methods. And as others have 
pointed out, the risk of creating routing loops is significant.

Perhaps it is time to migrate to a new version of BGP.  Projects like MBGP and 
FP-7‘s 4WARD are working on new follow-on routing models, but nothing is on the 
immediate horizon. I think we all thought we should finish IPv6 migration first 
:)

-mel via cell

On Jan 5, 2023, at 1:11 PM, Mike Hammett  wrote:


I hesitated to get too specific in examples because someone is going to drag 
the conversation into the weeds.

Let's take the the Dallas - New Orleans - Atlanta example where I have a 
connection from New Orleans to Dallas and a connection from New Orleans to 
Atlanta.

Let's say I peer with Netflix in both markets. Netflix chooses to serve me out 
of Atlanta, for whatever reason. Say my default route sends my traffic to 
Dallas. That's not where Netflix wanted it, so now I have to go from Dallas to 
Atlanta, whether that's my circuit or across the public Internet. Potentially, 
it's on MPLS and it rides back through the New Orleans router to get back to 
Atlanta. That's a long trip when I already had a better path, the 
less-than-full-fib router just didn't know about it. Given that Netflix is a 
sizable amount of traffic in an eyeball ISP, that's a lot of traffic to be 
going the wrong way. If the website for Viktor's Arctic Plunge in Siberia was 
hosted in Atlanta, I wouldn't give two craps that the traffic went the wrong 
way because A), I'll probably never go there and B) when someone does, it won't 
be meaningfully enough traffic to accommodate.

Someone's going to tell me to put a full-table router in New Orleans. Maybe I 
should. Okay, so maybe I have a POP in Ashford, Alabama. It has transport to 
New Orleans and Atlanta. There aren't enough grains of sugar in Ashford, 
Alabama to justify a current-generation, full table router. Now I'm even closer 
to Atlanta, but default may point to New Orleans.



-
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>

From: "Mel Beckman" 
To: "Mike Hammett" 
Cc: "Joe Maimon" , "NANOG" 
Sent: Thursday, January 5, 2023 2:54:27 PM
Subject: Re: SDN Internet Router (sir)

Mike,

I’m not sure I understand what you mean by “suboptimal“ routing. Even though 
the Internet uses AS path length for routing,  many of those path lengths are 
bogus, and don’t really represent any kind of path performance value. For 
example, a single AS might hide many hops in an MPLS network as a single hop, 
obscuring asymmetric routing and other uglies. Prepending also occurs when 
destinations are trying to enforce their own engineering  policies, which often 
conflict with yours or mine.

So what do you mean by “suboptimal“? Are you thinking that the “best” path in 
BGP tables actually meant you were getting a performance benefit? Because 
that’s definitely not the case in today’s Internet. Were were you thinking that 
you would be going along less congested paths? That’s really at the mercy of 
the traffic engineering of backbone providers over which we have no control.

I generally populate local router FIBs to merel choose an exit point for 
purposes of load balancing, and nothing more.

 -mel

On Jan 5, 2023, at 12:38 PM, Mike Hammett  wrote:


I guess I wasn't around for those days.

As far as running out, 

Re: SDN Internet Router (sir)

[Mel Beckman]

Mike,

I’m not sure I understand what you mean by “suboptimal“ routing. Even though 
the Internet uses AS path length for routing,  many of those path lengths are 
bogus, and don’t really represent any kind of path performance value. For 
example, a single AS might hide many hops in an MPLS network as a single hop, 
obscuring asymmetric routing and other uglies. Prepending also occurs when 
destinations are trying to enforce their own engineering  policies, which often 
conflict with yours or mine.

So what do you mean by “suboptimal“? Are you thinking that the “best” path in 
BGP tables actually meant you were getting a performance benefit? Because 
that’s definitely not the case in today’s Internet. Were were you thinking that 
you would be going along less congested paths? That’s really at the mercy of 
the traffic engineering of backbone providers over which we have no control.

I generally populate local router FIBs to merel choose an exit point for 
purposes of load balancing, and nothing more.

 -mel

On Jan 5, 2023, at 12:38 PM, Mike Hammett  wrote:


I guess I wasn't around for those days.

As far as running out, again, assuming the tooling works correctly, I'd think 
to target fewer routes than you could hold. Maybe 1k routes is all one would 
need to get a significant percent of the traffic. A lot of room to mess up if 
you can hold 100k, 500k routes.



-
Mike Hammett
Intelligent Computing Solutions
[http://www.ics-il.com/images/fbicon.png][http://www.ics-il.com/images/googleicon.png][http://www.ics-il.com/images/linkedinicon.png][http://www.ics-il.com/images/twittericon.png]
Midwest Internet Exchange
[http://www.ics-il.com/images/fbicon.png][http://www.ics-il.com/images/linkedinicon.png][http://www.ics-il.com/images/twittericon.png]
The Brothers WISP
[http://www.ics-il.com/images/fbicon.png][http://www.ics-il.com/images/youtubeicon.png]

From: "Joe Maimon" 
To: "Mike Hammett" , "Christopher Morrow" 

Cc: "NANOG" 
Sent: Thursday, January 5, 2023 2:30:40 PM
Subject: Re: SDN Internet Router (sir)



Mike Hammett wrote:
> I'm not concerned with which technology or buzzword gets the job done,
> only that the job is done.
>
>
>
> Looking briefly at the couple of things out there, they're evaluating
> the top X prefixes in terms of traffic reported by s-flow, where X is
> the number I define, and those get pushed into the FIB. One
> recalculates every hour, one does so more quickly. How much is
> appropriate? I'm not sure. I can't imagine it would *NEED* to be done
> all of that often, given the traffic/prefix density an eyeball network
> will have. Default routes carry the rest. Default routes could be
> handled outside of this process, such that if this process fails, you
> just get some sub-optimal routing until repaired. Maybe it doesn't
> filter properly and sends a bunch of routes. Then just have a prefix
> limit set on the box. Maybe it sends the wrong prefixes. No harm, no
> foul. If you're routing sub-optimally internally, when it does hit a
> real router with a full FIB, it gets handled appropriately.

Unless it loops.

The rest sounds nice. But flow caching got a bad rap back in the early
worm days. But thats because the situation was a little worse back then.
Cache the wrong routes or run out of cache, router dies. So long as
thats not the case automating optimization is an extremely valuable goal.

>
>
> I would just be looking for solutions that influence what's in the FIB
> and let the rest of the router work as the rest of the router would.

The problem comes when the router wont work at all without the FIB
routes, like in the olden days.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions 
> 
> Midwest Internet Exchange 
> 
> The Brothers WISP 
> 
> 
> *From: *"Christopher Morrow" 
> *To: *"Mike Hammett" 
> *Cc: *"Tom Beecher" , "NANOG" 
> *Sent: *Thursday, January 5, 2023 12:27:08 PM
> *Sub

Re: SDN Internet Router (sir)

[Mel Beckman]

Mike,

Your original question was:

“Given that the project was abandoned six years ago, are there any other 
efforts with a similar goal (more intelligently placing routes into FIBs of 
low-FIB capacity devices?”

People then, respectfully, tried to clarify your request or explain why placing 
routes in a low-FIB capacity device isn’t seen as being beneficial. Only now 
have you added the desire to simply have “more than a default route” in such a 
router.

You can, of course, have more than a default route today - e.g., through local 
pref and BGP communities for things such as company routes. You haven’t said 
what you define as “more intelligently”, so perhaps you can more clearly 
explain the problem you see with the current BGP capabilities via some examples.

 -mel

On Jan 5, 2023, at 8:02 AM, Mike Hammett  wrote:


Then please bless the world with the right way.

You acknowledge that not every router in a network needs to be fully DFZ 
capable, but then crap on my desire to have more than a default route in one.



-
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>
____________
From: "Tom Beecher" 
To: "Mike Hammett" 
Cc: "Mel Beckman" , "NANOG" 
Sent: Thursday, January 5, 2023 9:55:38 AM
Subject: Re: SDN Internet Router (sir)

"The right tool for the job" gets into a religious argument in assuming that 
one's way to do the job is the only reasonable way to do the job

I disagree that it's religious. I completely agree there are locations in 
networks that having full DFZ capable routers doesn't make technical or 
economic sense. But there have long been different products for those different 
use cases.

To perhaps explain my viewpoint better,(and perhaps I didn't properly 
comprehend the problem you're aiming to solve) :

If you are trying to use SDN stuff to shuffle routes on and off a box because 
you have the wrong sized routers in place, then I would argue you're doing it 
wrong.

If you are trying to use SDN stuff to (as Christopher mentioned) make decisions 
that are not strictly LPM, and the equipment you have cannot do that, then 
that's different and entirely reasonable.

If the second use case is more of what you were asking, then I apologize for 
misunderstanding.



On Thu, Jan 5, 2023 at 9:57 AM Mike Hammett 
mailto:na...@ics-il.net>> wrote:
"The right tool for the job" gets into a religious argument in assuming that 
one's way to do the job is the only reasonable way to do the job.

Large networks historically have a very poor (IMO) model of gigantic iron in a 
few locations, which results in sub-optimal routing for the rest of their 
network between those large POPs. I've heard time and time again that someone 
buying service from a major network in say New Orleans has a first hop of 
Dallas or Atlanta. I agree that full-route capable routers need to be in the 
large, central locations, but it isn't cost effective to have them at every 
POP, especially if you're a last-mile provider.

I'd go into more examples of where it doesn't make sense to have full-route 
routers everywhere, but I'm afraid that the Internet would then focus on the 
examples instead of the core idea of intelligently putting routes into the FIBs 
of low-FIB routers throughout my network.



-
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.p

Re: SDN Internet Router (sir)

[Mel Beckman]

It’s not a problem, due to cheap, plentiful high-speed memory and rapid prefix 
search silicon in backbone routers. The entire Internet routing table consumes 
at most a few gigabytes when fully structured (and only a few hundred Mbytes 
stored flat).  That’s less memory than your average laptop sports.


Even in the worst case scenario, where every network decides to announce only 
its most specific prefixes, the BGP backbone would temporarily enter an 
oscillating state that generates a large number of routing updates into the 
inter-domain routing space. In this case, BGP route damping will quickly 
suppress the crazies while  the backbone stabilizes.


Small routers should not be taking full tables, since there is no point to them 
being in the default free zone. For large routers, neither memory nor CPU speed 
are an issue. High-speed routers operating in the default-free zone have a 
critical path in the forwarding decision for each packet: it needs to take less 
than the inter-packet arrival time for minimum-sized IP packets.


This is easy to achieve with today’s hardware. A router line card with an 
aggregate line rate across all of its point-to-point interfaces of 10Tbps 
(readily available in today’s gear) can process packets with just a handful of 
cycles in the FIB Ternary Content Addressable Memory (TCAM) using ASIC-assisted 
lookups. TCAM is the most expensive component you’re paying for in such a 
router.  It’s not cheap,  but backbone routers don’t need to be cheap. They 
just need to not be memory-constrained.

-mel via cell

On Jan 3, 2023, at 7:47 AM, Mike Hammett  wrote:


https://github.com/dbarrosop/sir

I came across this over the weekend. Given that the project was abandoned six 
years ago, are there any other efforts with a similar goal (more intelligently 
placing routes into FIBs of low-FIB capacity devices?



-
Mike Hammett
Intelligent Computing Solutions
[http://www.ics-il.com/images/fbicon.png][http://www.ics-il.com/images/googleicon.png][http://www.ics-il.com/images/linkedinicon.png][http://www.ics-il.com/images/twittericon.png]
Midwest Internet Exchange
[http://www.ics-il.com/images/fbicon.png][http://www.ics-il.com/images/linkedinicon.png][http://www.ics-il.com/images/twittericon.png]
The Brothers WISP
[http://www.ics-il.com/images/fbicon.png][http://www.ics-il.com/images/youtubeicon.png]

Re: Large RTT or Why doesn't my ping traffic get discarded?

[Mel Beckman]

Keep in mind that ping reports round trip time, so there could be a device 
delaying the ping reply on the return trip. In these cases, it helps to have a 
traceroute from both ends, to detect asymmetrical routing and possibly return 
path congestion invisible in a traceroute from you end.

From: NANOG  on behalf of Mel Beckman 

Sent: Wednesday, December 21, 2022 9:22 AM
To: Jason Iannone ; North American Network Operators' 
Group 
Subject: Re: Large RTT or Why doesn't my ping traffic get discarded?

Sometimes this is usually due to high CPU time on the target device. If the 
device is under heavy load, the ICMP Echo process gets lowest priority. With a 
well-known name server like 4.2.2.2, this seems unlikely. It could be an 
intermediate hop or a routing loop, Do a traceroute to get more detailed 
per-hop statistics.

 -mel

From: NANOG  on behalf of Jason 
Iannone 
Sent: Wednesday, December 21, 2022 9:10 AM
To: North American Network Operators' Group 
Subject: Large RTT or Why doesn't my ping traffic get discarded?

Here's a question I haven't bothered to ask until now. Can someone please help 
me understand why I receive a ping reply after almost 5 seconds? As I 
understand it, buffers in SP gear are generally 100ms. According to my math 
this round trip should have been discarded around the 1 second mark, even in a 
long path. Maybe I should buy a lottery ticket. I don't get it. What is 
happening here?

Jason

64 bytes from 4.2.2.2<http://4.2.2.2>: icmp_seq=392 ttl=54 time=4834.737 ms
64 bytes from 4.2.2.2<http://4.2.2.2>: icmp_seq=393 ttl=54 time=4301.243 ms
64 bytes from 4.2.2.2<http://4.2.2.2>: icmp_seq=394 ttl=54 time=3300.328 ms
64 bytes from 4.2.2.2<http://4.2.2.2>: icmp_seq=396 ttl=54 time=1289.723 ms
Request timeout for icmp_seq 400
Request timeout for icmp_seq 401
64 bytes from 4.2.2.2<http://4.2.2.2>: icmp_seq=398 ttl=54 time=4915.096 ms
64 bytes from 4.2.2.2<http://4.2.2.2>: icmp_seq=399 ttl=54 time=4310.575 ms
64 bytes from 4.2.2.2<http://4.2.2.2>: icmp_seq=400 ttl=54 time=4196.075 ms
64 bytes from 4.2.2.2<http://4.2.2.2>: icmp_seq=401 ttl=54 time=4287.048 ms
64 bytes from 4.2.2.2<http://4.2.2.2>: icmp_seq=403 ttl=54 time=2280.466 ms
64 bytes from 4.2.2.2<http://4.2.2.2>: icmp_seq=404 ttl=54 time=1279.348 ms
64 bytes from 4.2.2.2<http://4.2.2.2>: icmp_seq=405 ttl=54 time=276.669 ms

Re: Large RTT or Why doesn't my ping traffic get discarded?

[Mel Beckman]

Sometimes this is usually due to high CPU time on the target device. If the 
device is under heavy load, the ICMP Echo process gets lowest priority. With a 
well-known name server like 4.2.2.2, this seems unlikely. It could be an 
intermediate hop or a routing loop, Do a traceroute to get more detailed 
per-hop statistics.

 -mel

From: NANOG  on behalf of Jason 
Iannone 
Sent: Wednesday, December 21, 2022 9:10 AM
To: North American Network Operators' Group 
Subject: Large RTT or Why doesn't my ping traffic get discarded?

Here's a question I haven't bothered to ask until now. Can someone please help 
me understand why I receive a ping reply after almost 5 seconds? As I 
understand it, buffers in SP gear are generally 100ms. According to my math 
this round trip should have been discarded around the 1 second mark, even in a 
long path. Maybe I should buy a lottery ticket. I don't get it. What is 
happening here?

Jason

64 bytes from 4.2.2.2: icmp_seq=392 ttl=54 time=4834.737 ms
64 bytes from 4.2.2.2: icmp_seq=393 ttl=54 time=4301.243 ms
64 bytes from 4.2.2.2: icmp_seq=394 ttl=54 time=3300.328 ms
64 bytes from 4.2.2.2: icmp_seq=396 ttl=54 time=1289.723 ms
Request timeout for icmp_seq 400
Request timeout for icmp_seq 401
64 bytes from 4.2.2.2: icmp_seq=398 ttl=54 time=4915.096 ms
64 bytes from 4.2.2.2: icmp_seq=399 ttl=54 time=4310.575 ms
64 bytes from 4.2.2.2: icmp_seq=400 ttl=54 time=4196.075 ms
64 bytes from 4.2.2.2: icmp_seq=401 ttl=54 time=4287.048 ms
64 bytes from 4.2.2.2: icmp_seq=403 ttl=54 time=2280.466 ms
64 bytes from 4.2.2.2: icmp_seq=404 ttl=54 time=1279.348 ms
64 bytes from 4.2.2.2: icmp_seq=405 ttl=54 time=276.669 ms

Re: Power outages Maryland near Washington DC (Plane crash into power lines)

[Mel Beckman]

Here’s a pilot’s-perspective report of the crash and amazing rescue, with more 
pics:


https://us4.campaign-archive.com/?e=582d83d772&u=1a3819bbb44ebf8f1e02946a0&id=59eeed292b

 -mel beckman

On Nov 27, 2022, at 5:40 PM, Sean Donelan  wrote:


A small plane crashed into high-voltage transmission power lines in Montgomery 
County, Maryland; near Washington DC.  Most of the data centers are in Northern 
Virginia, but some are in Southern Maryland (Wheaton, Olney, Gaithersburg and 
as far away as Silver Spring).


https://wtop.com/montgomery-county/2022/11/outages-in-montgomery-co-after-small-plane-crashes-into-power-lines/

Re: Random shower thought: GBIC with LC connector...

[Mel Beckman]

Oh. And it’s not “OCD”. It’s “CDO”, with letters in ascending sequence. :)

-mel via cell

> On Nov 15, 2022, at 8:18 AM, Mel Beckman  wrote:
> 
> No. GBIC stands for Great Big Inserted Cartridge. LC stands for Little 
> Connector. Thus they are not compatible. 
> 
> -mel via cell
> 
>> On Nov 15, 2022, at 7:59 AM, Warren Kumari  wrote:
>> 
>> 
>> Hi there all,
>> 
>> While looking through my big box of random optics I suddenly realized that 
>> I'd never seen a GBIC with an LC connector, and I started wondering if 
>> anyone else had / if such a thing actually exists.
>> 
>> Yes, I realize that this would be a fairly niche device - if you arrived 
>> somewhere with a device that took GBICs and there was existing fiber with LC 
>> connectors you could just replace the patch cable or use an LC-SC convertor, 
>> but that doesn't really satisfy my curiosity.
>> 
>> A quick look through the GBIC MSA / SFF documentation implies that such a 
>> thing *could* probably exist (there is a defined value for the 'LC' 
>> connector), but I wasn't able to actually find any. It might not actually be 
>> compliant with the specs (the document I found only lists SC fiber or copper 
>> (coax with BNC, TNC or DB-9?!)), but that doesn't mean that no-one made them.
>> 
>> So, has anyone seen a regular (30mm/1.2") GBIC with LC connectors? And, if 
>> so, "pics or it didn't happen"... :-)
>> 
>> Obviously I don't have an actual use for this, it's just to satisfy my (OCD) 
>> curiosity...
>> W
>> 
>> 

Re: Random shower thought: GBIC with LC connector...

[Mel Beckman]

No. GBIC stands for Great Big Inserted Cartridge. LC stands for Little 
Connector. Thus they are not compatible. 

-mel via cell

> On Nov 15, 2022, at 7:59 AM, Warren Kumari  wrote:
> 
> 
> Hi there all,
> 
> While looking through my big box of random optics I suddenly realized that 
> I'd never seen a GBIC with an LC connector, and I started wondering if anyone 
> else had / if such a thing actually exists.
> 
> Yes, I realize that this would be a fairly niche device - if you arrived 
> somewhere with a device that took GBICs and there was existing fiber with LC 
> connectors you could just replace the patch cable or use an LC-SC convertor, 
> but that doesn't really satisfy my curiosity.
> 
> A quick look through the GBIC MSA / SFF documentation implies that such a 
> thing *could* probably exist (there is a defined value for the 'LC' 
> connector), but I wasn't able to actually find any. It might not actually be 
> compliant with the specs (the document I found only lists SC fiber or copper 
> (coax with BNC, TNC or DB-9?!)), but that doesn't mean that no-one made them.
> 
> So, has anyone seen a regular (30mm/1.2") GBIC with LC connectors? And, if 
> so, "pics or it didn't happen"... :-)
> 
> Obviously I don't have an actual use for this, it's just to satisfy my (OCD) 
> curiosity...
> W
> 
> 

Re: IPv6 on Lumen/CL

[Mel Beckman]

You should get a status response within 24 hours. I would call in to the 
support hotline, (800) 829-0420, and ask for a status update.

-mel via cell

> On Aug 29, 2022, at 4:03 PM, Nathan Anderson  wrote:
> 
> We have a circuit on AS209 that was originally provisioned v4-only.  I'm now 
> trying to get Lumen to turn v6 up on it.  How long does this typically take?  
> I've had a configuration ticket open for nearly 3 biz days now with no 
> movement (or even acknowledgement).  For anybody who has gone through this 
> with them, is this unusual or nah?
> 
> When they do get around to it, what can I expect in terms of how they will 
> prefer to set this up?  Separate BGP session running over v6 itself, or 
> modify existing session to have it also carry v6 NLRIs?
> 
> Thanks,
> 
> -- Nathan

Re: VZ FIOS and Intel TCP IPv6 Checksum Offload problems

[Mel Beckman]

No. In fact, a lot of low-end Ethernet interfaces are completely implemented in 
interrupt-driven driver software that runs in the host OS (such as Windows). 
The only thing the hardware provides is the magnetically to transduce binary 
bit streams. 

Even MAC-address decode is in software, and as a result, broadcast storms can 
slow these hosts to a crawl as the CPU had to check and discard every broadcast 
packet as “not mine”. 

When these tasks are offloaded from the CPU to the Ethernet hardware, the CPU 
doesn’t need to perform these tasks, reducing CPU workload. These also 
offloading resources provide parallel computing and validation of checksums, 
which is otherwise computationally expensive. 

I don’t know how this particular ONT bug works, but I’m guessing that it 
results in checksum failures under certain conditions, leading to 
retransmissions. 

-mel via cell

> On Aug 27, 2022, at 3:08 PM, Michael Thomas  wrote:
> 
> 
>> On 8/27/22 12:00 PM, Sean Donelan wrote:
>> Hopefully, my pain will help someone else.
>> 
>> I've had sporadic Internet slowdowns and stuck networking since IPv6 was 
>> enabled on my FIOS ONT a few months ago.
>> 
>> After too much troubleshooting, I found out some older Intel GbE ethernet 
>> cards have a IPv6 Checksum Offload incompatibility with certain fiber ONT 
>> terminals.  As Verizon is enabling IPv6 on its FIOS network, you might find 
>> intermittent network problems.
>> 
>> Intermittent are the worst kind of problems.
>> 
>> In some situations where a client machine is connected via some specific 
>> Optical Network Terminals (ONTs), and data is appended after the packet 
>> checksum, the network adapter can drop receive packets when using TCP-IPv6 
>> Checksum Offload for receive traffic.
>> 
>> Intel published an alert in 2017, but I didn't have IPv6 on FIOS then.
>> 
>> https://www.intel.com/content/www/us/en/download/19174/disabling-tcp-ipv6-checksum-offload-capability-with-intel-1-10-gbe-controllers.html
>>  
>> 
>> 
>> TLDR; turn off TCP IPv6 Checksum Offload
>> 
>> Affects all operating systems (Windows, BSD, Linux, etc) using the affected 
>> wired Intel ethernet controllers.  Not a problem with Intel WiFi.
> 
> My reaction is "offload from what"? Isn't this all done in silicon?
> 
> Mike
> 

Re: Google Abuse

[Mel Beckman]

Well put, bro. 

-mel via cell

> On Aug 16, 2022, at 9:08 PM, Peter Beckman  wrote:
> 
> To make this more NANOGy, what is OUR role in all of this?
> 
> Two questions that relate here:
> 
>How does NANOG make inbound network abuse easier to stop and harder or
>costlier for networks and clouds to ignore?
> 
>How do NANOG operators attempt to keep private things private?
> 
> 
> For the latter, IMHO most NANOG members likely also run, manage, or interact 
> with
> businesses that hold data.
> 
> Three of the NANOG Principles apply here:
> 
>Security within our digital platforms
>Sustainability of Internet technology professions
>Innovation within the community
> 
> 
> We all should be doing whatever we can within our own organizations to
> improve end user privacy and security. I'm going to make another go at it
> within my own.
> 
> And anything we can do to make it harder for networks and cloud providers
> to ignore abuse reports and stop it is an Innovation that might move the
> burden of network attacks off of the recipients and onto the sources.
> 
> Beckman
> 
>> On Tue, 16 Aug 2022, richey goldberg wrote:
>> 
>> “thought that google fi was a neutral pipe.”
>> 
>> There is nothing neutral about Google or any of companies that are their 
>> competitors.They all have some sort of agenda which is to do what’s best 
>> for them or what they *think* is best for everyone else.  Even if it’s not.
>> 
>> “are google, like fb, recording and retaining direct messages and sms/mms 
>> contents”
>> 
>> They may tell you they are not but there is no doubt in my mind they are and 
>> if they got caught their response would be “Oopsie, my bad”.
>> 
>> -richey
>> 
>> 
>> From: NANOG  on behalf of 
>> Mark Seiden 
>> Date: Tuesday, August 16, 2022 at 3:48 PM
>> To: Jon Lewis 
>> Cc: nanog@nanog.org 
>> Subject: Re: Google Abuse
>> well, that isn’t exactly true.
>> 
>> ALL of the fraudsters, business email compromisers, spoofing accounts are 
>> now from gmail and as far as i can tell,
>> there is no evidence that they do ANYTHING about them.i recently gave a 
>> talk on fraudulent restaurant reviews
>> in google maps.  easy for humans to spot.  (hundreds of machine learning 
>> engineers at google.  what are they doing?)
>> 
>> but here’s a counterexample… not that it serves anyone particularly well:
>> 
>> a colleague of mine (ex googler, superb engineer, with a brother who is a 
>> current googler) had ALL of his google accounts
>> deactivated recently.  a google fi customer, he used it to send an mms photo 
>> of a rash on his toddler’s crotch to his wife,
>> so she could upload it (using https) to their pediatrician’s portal for 
>> diagnosis.
>> 
>> a few days later the cops were at the door with a search warrant.  the cops 
>> agreed it was a false positive, but despite that,
>> the accounts were deactivated (including gmail), seemingly permanently, 
>> despite multiple attempts to revive it and attempts
>> at escalation.
>> 
>> i was actually surprised.  i thought that google fi was a neutral pipe.
>> 
>> who knew that google mines mms images for pink parts?
>> 
>> do the other cell phone companies do the same?  (not that i particularly 
>> need to test it…)
>> 
>> (is there any transparency here regarding the scanning and retention policy 
>> for sms and mms contents?)
>> 
>> which raises, in the post-boggs world, another question:
>> 
>> are google, like fb, recording and retaining direct messages and sms/mms 
>> contents, so they can turn them over
>> to law enforcement who have become “interested" in who was pregnant and who 
>> stopped being pregnant?
>> 
>> https://www.vice.com/en/article/n7zevd/this-is-the-data-facebook-gave-police-to-prosecute-a-teenager-for-abortion
>> 
>> (once again, there ain’t no sanity clause.)
>> 
>> 
 On Aug 16, 2022, at 10:43 AM, Jon Lewis  wrote:
>>> 
 On Tue, 16 Aug 2022, Cristian Cardoso wrote:
>>> 
 Hi
 I'm receiving thousands of requests from a Google Clou VM on my network, 
 I've already sent reports to Abuse from GCP, but without success, does 
 anyone happen to have a Google abuse
 contact to indicate?
>>> 
>>> There is no Google abuse.  It's just traffic you don't want that they don't 
>>> care about.  Block it at your edge and move on.
>>> 
>>> --
>>> Jon Lewis, MCP :)   |  I route
>>> StackPath, Sr. Neteng   |  therefore you are
>>> _ http://www.lewis.org/~jlewis/pgp for PGP public key_
>> 
> 
> ---
> Peter Beckman  Internet Guy
> beck...@angryox.comhttps://www.angryox.com/
> ---

Re: Facebook down?

[Mel Beckman]

According to Heisenberg, it’s up :)

-mel via cell

> On Aug 11, 2022, at 1:44 PM, Michael Thomas  wrote:
> 
> And of course the act of sending this mail caused the wave function to 
> collapse and it seems to be up again, at least for me.
> 
> Mike
> 
>> On 8/11/22 1:37 PM, Michael Thomas wrote:
>> They haven't been serving up images for like an hour or so and now it's 
>> showing their fail whale. Not sure if it's a (internal) network problem or 
>> not.
>> 
>> I'm in California fwiw.
>> 
>> Mike
>> 

Re: IoT - The end of the internet

[Mel Beckman]

Christopher,

What you’re really observing here is that today's technology does not yet 
enable these your chosen use cases. It may someday, but not today, not for any 
amount of money. 1990s modem technology didn’t enable streaming video either, 
but add 20 years of advancement, and today you can watch Seinfeld on your wrist.

Mankind has been to the moon, but you can’t have lunch on the moon next week, 
no matter how much money you have. But I have no doubt that eventually humans 
will be eating lunch on the moon whenever they like.

The Internet has never been “re-thought” throughout it’s entire history. 
Networking has advanced tremendously with stepwise refinement just fine. A 
“re-think” would simply be too expensive and too disruptive.

 -mel

On Aug 10, 2022, at 3:29 PM, Christopher Wolff 
mailto:ch...@vergeinternet.com>> wrote:

Hi NANOG;

I appreciate all the thoughtful replies and I apologize for vague posting when 
I should be sleeping.

Let me paint a little more context and hopefully this will help inform the 
conversation.

Use Case 1:  Augmented Reality/Virtual Reality.  It is stated that round trip 
latency must be <4ms with 100mbit full duplex at the cell edge to prevent 
nausea and dizziness while wearing goggles for a long term.

Use Case 2:  A little closer to “IoT”. An autonomous vehicle under remote 
control requires 100 feet to stop with LTE vs 20 feet with 5G.

Use Case 3:  A Lidar near-miss sensor at an intersection requires 1ms from the 
traffic operations center.

I hypothesize that there is a ‘breaking point’ between safety, health, and 
latency and traditional IP.

Will tomorrow’s applications require a re-thinking of “The Internet” and 
protocols that are low latency compliant?  Will we be building an infinite 
number of mobile edge compute boxes?

If there’s an academic study describing this potential issue it would help 
kickstart some interesting research.

Best,
Christopher

On Aug 10, 2022, at 1:26 PM, Alexander Lyamin 
mailto:l...@qrator.net>> wrote:

It's not devices. It's software and what's worse protocol specifications that 
are implemented in this software.

And we still didn't get the memo in 2022. Some colleagues think that having 
builtin 5x Amplification in protocols freshly out just this year "is OK".

  Cyberhippies

On Wed, Aug 10, 2022, 05:12 Ca By 
mailto:cb.li...@gmail.com>> wrote:


On Tue, Aug 9, 2022 at 7:23 PM Christopher Wolff 
mailto:ch...@vergeinternet.com>> wrote:
Hi folks,

Has anyone proposed that the adoption of billions of IoT devices will 
ultimately ‘break’ the Internet?

It’s not a rhetorical question I promise, just looking for a journal or other 
scholarly article that implies that the Internet is doomed.

In so much as IoT devices are ipv4 udp amplifiers

https://www.ndss-symposium.org/ndss2014/programme/amplification-hell-revisiting-network-protocols-ddos-abuse/