Re: Free(opensource) Ticketing solutions

[Michael Spears via NANOG]

OSTicket is decent, not the prettiest, but works well.
Sent from my iPhone

> On May 27, 2024, at 1:30 PM, Pascal Masha  wrote:
> 
> 
> Hello,
> 
> Which free and good ticketing systems do you folks(for those who do) use?
> 
> Regards,
> Paschal Masha

Re: Mailing list SPF Failure

[Michael Thomas]

On 5/16/24 7:36 PM, John R. Levine wrote:

I think a lot of us have nanog whitelisted or otherwise special cased.


I don't and gmail is my backend. That's trivial falsification that lack 
of an SPF records alone will cause gmail rejects.


Mike



Also, it's been pumping out list mail for decades and I expect has a 
close to zero complaint rate so even without the SPF ths IPs it sends 
from have a good reputation.


On Thu, 16 May 2024, Scott Q. wrote:


I'm surprised nobody noticed for close to 10 days. I was away
from work and upon coming back I saw the little discussion there was ,
in my Spam folder.

On Thursday, 16/05/2024 at 18:56 John R. Levine wrote:

On Thu, 16 May 2024, William Herrin wrote:

The message content (including the message headers) is theoretically
not used for SPF validation. In practice, some SPF validators don't
have direct access to the SMTP session so they rely on the SMTP
session placing the envelope sender in the Return-path header.


But that wasn't the problem here, the SPF record was just
gone.  Oops.

I see that the SPF record is back and seems have the correct addresses
so we can now return to our previously scheduled flamage.

Re: Should FCC look at SS7 vulnerabilities or BGP vulnerabilities

[Michael Thomas]

On 5/16/24 6:55 PM, John Levine wrote:

It appears that Brandon Martin  said:

I think the issue with their lack of effectiveness on spam calls is due
to the comparatively small number of players in the PSTN (speaking of
both classic TDM and modern IP voice-carrying and signaling networks)
world allowing lots of regulatory capture.

It's the opposite. SS7 was designed for a world with a handful of
large trustworthy telcos. But now that we have VoIP, it's a world of a
zillion sleasy little VoIP carriers stuffing junk into the network.
The real telcos have no desire to deliver spam calls. Everything is
bill and keep so they get no revenue and a lot of complaints.

Mike is right that STIR/SHAKEN is more complex than it needs to be but
even after it was widely deployed, the telcos had to argue with the
FCC to change the rules so they were allowed to drop spam calls which
only changed recently. That's why you see PROBABLE SPAM rather than
just not getting the call.


I was screaming at the top of my lungs that P-Asserted-Identity was 
going to bite them in the ass 20 years ago. And then they eventually 
came up with something that solved the wrong problem in the most 
bellheaded way possible 15 years later. Bellheads should not be trusted 
with internet security. The FCC is most likely not blameless here either 
but the telcos/bellheads most certainly aren't either. Anybody who 
thinks this is an either/or problem is wrong.


Mike

Re: Mailing list SPF Failure

[Michael Thomas]

On 5/16/24 7:22 PM, Scott Q. wrote:
Mike, you do realize Google/Gmail rejects e-mails with invalid/missing 
SPF right ?


I was receiving the mail while NANOG had no SPF record, so no? Any 
receiver would be really stupid take a single signal as disqualifying.


Mike




If you want to tell them they're broken...there's a few guys on the 
list here.


On Thursday, 16/05/2024 at 19:17 Michael Thomas wrote:

On 5/16/24 3:54 PM, William Herrin wrote:
> On Thu, May 16, 2024 at 12:03 PM John Levine mailto:jo...@iecc.com>> wrote:
>> It appears that Michael Thomas mailto:m...@mtcc.com>> said:
>>> Since probably 99% of the mail from NANOG is through this list, it
>>> hardly matters since SPF will always fail.
>> Sorry, but no. A mailing list puts its own envelope return
address on
>> the message so with a reasonable SPF record, SPF will normally
>> succeed.
> Exactly. SPF acts on the -envelope- sender. That means the one
> presented in the SMTP From:<> command. For mail from nanog, that's:
> nanog-bounces+addr...@nanog.org
<mailto:nanog-bounces+addr...@nanog.org>, regardless of what the
sender's
> header From address is.
>
> The message content (including the message headers) is theoretically
> not used for SPF validation. In practice, some SPF validators don't
> have direct access to the SMTP session so they rely on the SMTP
> session placing the envelope sender in the Return-path header.

Yes, and why is that needed? The mailing list resigning has the same
effect and then you only need one mechanism instead of two and
with DKIM
you get the benefit that it's signing the 822 address which can be
used
for user level stuff in way that SPF is a little sus. So it makes SPF
pretty irrelevant. IMO, SPF was always a stopgap since there was no
guarantee that DKIM would be deployed. 20 years on, I guess I
don't feel
like I need to keep my trap shut about that.

If a receiving site is rejecting something solely based on the
lack of a
SPF record but has a valid DKIM signature, the site is broken IMO.

Mike

Re: Should FCC look at SS7 vulnerabilities or BGP vulnerabilities

[Michael Thomas]

On 5/16/24 4:17 PM, Brandon Martin wrote:


I think the issue with their lack of effectiveness on spam calls is 
due to the comparatively small number of players in the PSTN (speaking 
of both classic TDM and modern IP voice-carrying and signaling 
networks) world allowing lots of regulatory capture. That's going to 
keep the FCC from issuing mandatory rules much beyond what much of the 
industry is on the road to implementing already to keep their 
customers placated.


I think it should be pointed out that the STIR/SHAKEN crowd doesn't 
really get it either. The problem is mainly a problem of the border 
between bad guys and the onramps onto the PSTN. SIP has made that dirt 
cheap and something anybody can do it for nothing at all down in their 
basements. It's essentially the same thing as email back in the days of 
open relays and no submission auth. STIR/SHAKEN obfuscated that problem 
by trying to solve the problem of who is allowed to assert what E.164 
address when it's much easier to solve in the "where did this come from 
and who should I blame?" realm. I don't hear anybody moaning about 
deploying DKIM except maybe spammer sites that don't want accountability 
and their onramp sites that turn a blind eye making money off them. They 
care these days because for legit senders, baddies cost them money due 
to deliverability. It would have been trivial to attach a DKIM like 
signature to SIP messages and be done with it instead of trying to boil 
the legacy addressing ocean. I should know, I did that for shits and 
giggles about 20 years ago.


Mike

Re: Mailing list SPF Failure

[Michael Thomas]

On 5/16/24 3:54 PM, William Herrin wrote:

On Thu, May 16, 2024 at 12:03 PM John Levine  wrote:

It appears that Michael Thomas  said:

Since probably 99% of the mail from NANOG is through this list, it
hardly matters since SPF will always fail.

Sorry, but no. A mailing list puts its own envelope return address on
the message so with a reasonable SPF record, SPF will normally
succeed.

Exactly. SPF acts on the -envelope- sender. That means the one
presented in the SMTP From:<> command. For mail from nanog, that's:
nanog-bounces+addr...@nanog.org, regardless of what the sender's
header From address is.

The message content (including the message headers) is theoretically
not used for SPF validation. In practice, some SPF validators don't
have direct access to the SMTP session so they rely on the SMTP
session placing the envelope sender in the Return-path header.


Yes, and why is that needed? The mailing list resigning has the same 
effect and then you only need one mechanism instead of two and with DKIM 
you get the benefit that it's signing the 822 address which can be used 
for user level stuff in way that SPF is a little sus. So it makes SPF 
pretty irrelevant. IMO, SPF was always a stopgap since there was no 
guarantee that DKIM would be deployed. 20 years on, I guess I don't feel 
like I need to keep my trap shut about that.


If a receiving site is rejecting something solely based on the lack of a 
SPF record but has a valid DKIM signature, the site is broken IMO.


Mike

Re: Mailing list SPF Failure

[Michael Thomas]

On 5/16/24 8:59 AM, Scott Q. wrote:
Uhm, not really. An SPF failure is really bad even though DKIM works. 
It might depend what they do with DMARC but even so, there's no reason 
they can't just add that IP to their SPF record.


SPF has from day one been known to be broken with mailing lists. It's 
not "really bad", it's just what it is. There are other modes that SPF 
fails too like forwarding. Frankly I've tried to keep clear of "SPF is 
pointless", but it is actually pointless. It doesn't bring anything to 
the table that DKIM can't do better.


Mike

Re: Mailing list SPF Failure

[Michael Thomas]

On 5/16/24 8:11 AM, Peter Potvin via NANOG wrote:
Appears there’s no SPF record at all now for nanog.org 
, which is not ideal…


Since probably 99% of the mail from NANOG is through this list, it 
hardly matters since SPF will always fail. What is more important is 
that they resign with DKIM so that receivers can use that identity. SPF 
is for the most part belt and suspenders.


Mike




Kind regards,
Peter Potvin


On Thu, May 16, 2024 at 02:59 Bjørn Mork  wrote:

"Scott Q."  writes:

> Anyone else getting SPF failures on all messages sent to the list
> ?
>
> I see them all originating from 50.31.151.76 but nanog.org
's SPF
> record doesn't list that as allowed.

I see the same. nanog.org  mail is originated from
2001:1838:2001:8:0:0:0:20 or 50.31.151.76, and the SPF record is
currently

 "v=spf1 a include:_spf.google.com  ~all"

Neither of those are Google addresses so it's a soft fail.


Bjørn

Re: Microsoft missing public DNS TXT entry for DKIM records (msn.com)

[Michael Thomas]

On 4/4/24 12:43 AM, Jay Acuna wrote:

On Thu, Apr 4, 2024 at 1:23 AM Adam Brenner via NANOG  wrote:
..

It seems to me that if msn.com is going to include DKIM headers in their
outgoing email, they should also publish their DKIM public key. If they
are not going to publish their DKIM public key, then they should not
include DKIM headers in their outgoing email.

Microsoft can still sign the message, Even if the signature cannot be verified
because they have not yet published the Public Key, for whatever reason.
That is a partial/incomplete implementation of DKIM then.


There is one potential reason a site might want to do this which is to 
essentially invalidate signatures from a non-repudiation standpoint. 
Simply unpublishing the key while not 100% foolproof is essentially 
saying "we don't take responsibility for mail signed with this key 
anymore" -- sort of like the expirey tag in the header but with 
attitude. The entire kerfuffle about Her Emails (ie Hillary's email 
server) was in part about the fact that the mail on it could still be 
verified and thus not denied. After, there were calls for providers to 
publish their private keys on a regular basis but that went nowhere that 
I've heard of. That's probably not what's going on here -- maybe they 
just botched a key rollover -- but it still amusing to me that we got 
non-repudiation along for the ride [*].


Mike

[*] while DKIM only speaks at the domain level and not an individual 
account, if providers always require submission auth before signing that 
seems pretty airtight to me

Re: Best TAC Services from Equipment Vendors

[michael brooks - ESC]

>
> It may be a pain in the butt to get Cisco equipment, but their TAC is
> sublime.  If something is critical enough, and you push hard enough, Cisco
> will move heaven and earth to solve your issue.
>

>This was an amazing laugh on a Monday morning. Thanks!

O crap, did I miss the sarcasm?




michael brooks

On Mon, Mar 11, 2024 at 8:55 AM Tom Beecher  wrote:

> It may be a pain in the butt to get Cisco equipment, but their TAC is
>> sublime.  If something is critical enough, and you push hard enough, Cisco
>> will move heaven and earth to solve your issue.
>>
>
> This was an amazing laugh on a Monday morning. Thanks!
>
> On Thu, Mar 7, 2024 at 2:47 PM Joel Esler  wrote:
>
>> It may be a pain in the butt to get Cisco equipment, but their TAC is
>> sublime.  If something is critical enough, and you push hard enough, Cisco
>> will move heaven and earth to solve your issue.
>> —
>> Sent from my iPhone
>>
>> On Mar 6, 2024, at 13:42, Pascal Masha  wrote:
>>
>> 
>> For us this has been the experience to a point where 100s of nodes( from
>> vendor x) had to be swapped out because no one had the patience anymore…
>>
>> On Wed, 6 Mar 2024 at 21:29,  wrote:
>>
>>> Interesting, this has never been my experience even with Cisco or
>>> Juniper, have always been able to escalate quickly to engineering. I wonder
>>> if it was related to the size of my accounts.
>>>
>>> Shane
>>>
>>> > On Mar 6, 2024, at 1:27 PM, Pascal Masha 
>>> wrote:
>>> >
>>> > Thought about it but so far I believe companies from China provide
>>> better and fast TAC responses to their customers than the likes of Cisco
>>> and perhaps that’s why some companies(where there are no
>>> restrictions)prefer them for critical services.
>>> >
>>> > For a short period in TAC call you can have over 10 R engineers and
>>> solutions provided in a matter of hours even if it involves software
>>> changes.. while these other companies even before you get in a call with a
>>> TAC engineer it’s hours and when they join you hear something like “my
>>> shift ended 15 minutes ago, hold let me look for another engineer”. WHY?
>>> Thoughts
>>>
>>

-- 
This is a staff email account managed by Adams 12 Five Star Schools.  This 
email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the sender.

Re: Best TAC Services from Equipment Vendors

[michael brooks - ESC]

You are missing the point, we opened the case 3 months ago




michael brooks


On Mon, Mar 11, 2024 at 8:24 AM  wrote:

> To be honest, if your DR environment has been offline for 3 months and you
> are just now opening a case, I would not consider that critical.
>
> Shane
>
> On Mar 11, 2024, at 10:08 AM, michael brooks - ESC <
> michael.bro...@adams12.org> wrote:
>
> 
>
> >It may be a pain in the butt to get Cisco equipment, but their TAC is
> sublime.  If something is critical enough, and you push hard enough, Cisco
> will move heaven and earth to solve your issue.
>
> But is this not the problem itself?
>
> Strap in for an "I remember when" ...
> Once upon a time, I could call (specifically) Cisco TAC, knowing, without
> any doubt, that my quality problem resolution was inbound, no questions
> asked. That came to a crashing halt about 15 years ago when a TAC engineer
> wanted to bounce our Voice VLAN SVI in the middle of an *airport* production
> day. I about turned over my desk trying to wrest the remote control session
> back from him before he hit enter on the shut. Since then, I have had to go
> through a not insignificant evaluation period of TAC engineers before I let
> them take control of a remote session, and it is now simply pure instinct
> to log SSH sessions.
>
> Fast forward and my user base is now ~45k people, not massive by any
> stretch, but certainly not a small org, and we tend to buy Premium/Platinum
> support on all of our critical applications. I truly do have to "push hard"
> and long to get the attention of our support teams to get a seemingly
> simple problem supported and solved. Our support is still in the dumper.
> Take, for instance, a recent case with our replication/DR vendor. Our DR
> environment has been offline for ~3 months, does that not scream
> "critical?" And yet, we are still having engineers jump on a call to
> collect  the same data that the last engineer jumped on a call to
> collect. One of our engineers, as has been not-so-subtly alluded to,
> resorted to a screamfest to get the attention of TAC, and not surprisingly
> that dressing-down got upper levels involved.
>
> For good measure, major networking vendor possibly mentioned earlier spent
> 9 *months*, at the developer level, troubleshooting a multicast issue
> which ultimately turned out to be PIM not configured on all interfaces in
> the path. Yes, I felt like an idiot for not already checking that, but
> should not have the first engineer on the phone asked this?
>
> Admittedly, we are going through a rough patch in terms of support, but it
> is not out of line with the past decade's experiences.
>
>
> michael brooks
>
> On Thu, Mar 7, 2024 at 12:47 PM Joel Esler  wrote:
>
>> It may be a pain in the butt to get Cisco equipment, but their TAC is
>> sublime.  If something is critical enough, and you push hard enough, Cisco
>> will move heaven and earth to solve your issue.
>> —
>> Sent from my iPhone
>>
>> On Mar 6, 2024, at 13:42, Pascal Masha  wrote:
>>
>> 
>> For us this has been the experience to a point where 100s of nodes( from
>> vendor x) had to be swapped out because no one had the patience anymore…
>>
>> On Wed, 6 Mar 2024 at 21:29,  wrote:
>>
>>> Interesting, this has never been my experience even with Cisco or
>>> Juniper, have always been able to escalate quickly to engineering. I wonder
>>> if it was related to the size of my accounts.
>>>
>>> Shane
>>>
>>> > On Mar 6, 2024, at 1:27 PM, Pascal Masha 
>>> wrote:
>>> >
>>> > Thought about it but so far I believe companies from China provide
>>> better and fast TAC responses to their customers than the likes of Cisco
>>> and perhaps that’s why some companies(where there are no
>>> restrictions)prefer them for critical services.
>>> >
>>> > For a short period in TAC call you can have over 10 R engineers and
>>> solutions provided in a matter of hours even if it involves software
>>> changes.. while these other companies even before you get in a call with a
>>> TAC engineer it’s hours and when they join you hear something like “my
>>> shift ended 15 minutes ago, hold let me look for another engineer”. WHY?
>>> Thoughts
>>>
>>
> This is a staff email account managed by Adams 12 Five Star Schools.  This
> email and any files transmitted with it are confidential and intended
> solely for the use of the individual or entity to whom they are addressed.
> If you have received this email in error please notify the sender.
>
>

-- 
This is a staff email account managed by Adams 12 Five Star Schools.  This 
email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the sender.

Re: Best TAC Services from Equipment Vendors

[michael brooks - ESC]

>It may be a pain in the butt to get Cisco equipment, but their TAC is
sublime.  If something is critical enough, and you push hard enough, Cisco
will move heaven and earth to solve your issue.

But is this not the problem itself?

Strap in for an "I remember when" ...
Once upon a time, I could call (specifically) Cisco TAC, knowing, without
any doubt, that my quality problem resolution was inbound, no questions
asked. That came to a crashing halt about 15 years ago when a TAC engineer
wanted to bounce our Voice VLAN SVI in the middle of an *airport* production
day. I about turned over my desk trying to wrest the remote control session
back from him before he hit enter on the shut. Since then, I have had to go
through a not insignificant evaluation period of TAC engineers before I let
them take control of a remote session, and it is now simply pure instinct
to log SSH sessions.

Fast forward and my user base is now ~45k people, not massive by any
stretch, but certainly not a small org, and we tend to buy Premium/Platinum
support on all of our critical applications. I truly do have to "push hard"
and long to get the attention of our support teams to get a seemingly
simple problem supported and solved. Our support is still in the dumper.
Take, for instance, a recent case with our replication/DR vendor. Our DR
environment has been offline for ~3 months, does that not scream
"critical?" And yet, we are still having engineers jump on a call to
collect  the same data that the last engineer jumped on a call to
collect. One of our engineers, as has been not-so-subtly alluded to,
resorted to a screamfest to get the attention of TAC, and not surprisingly
that dressing-down got upper levels involved.

For good measure, major networking vendor possibly mentioned earlier spent
9 *months*, at the developer level, troubleshooting a multicast issue which
ultimately turned out to be PIM not configured on all interfaces in the
path. Yes, I felt like an idiot for not already checking that, but should
not have the first engineer on the phone asked this?

Admittedly, we are going through a rough patch in terms of support, but it
is not out of line with the past decade's experiences.


michael brooks

On Thu, Mar 7, 2024 at 12:47 PM Joel Esler  wrote:

> It may be a pain in the butt to get Cisco equipment, but their TAC is
> sublime.  If something is critical enough, and you push hard enough, Cisco
> will move heaven and earth to solve your issue.
> —
> Sent from my iPhone
>
> On Mar 6, 2024, at 13:42, Pascal Masha  wrote:
>
> 
> For us this has been the experience to a point where 100s of nodes( from
> vendor x) had to be swapped out because no one had the patience anymore…
>
> On Wed, 6 Mar 2024 at 21:29,  wrote:
>
>> Interesting, this has never been my experience even with Cisco or
>> Juniper, have always been able to escalate quickly to engineering. I wonder
>> if it was related to the size of my accounts.
>>
>> Shane
>>
>> > On Mar 6, 2024, at 1:27 PM, Pascal Masha  wrote:
>> >
>> > Thought about it but so far I believe companies from China provide
>> better and fast TAC responses to their customers than the likes of Cisco
>> and perhaps that’s why some companies(where there are no
>> restrictions)prefer them for critical services.
>> >
>> > For a short period in TAC call you can have over 10 R engineers and
>> solutions provided in a matter of hours even if it involves software
>> changes.. while these other companies even before you get in a call with a
>> TAC engineer it’s hours and when they join you hear something like “my
>> shift ended 15 minutes ago, hold let me look for another engineer”. WHY?
>> Thoughts
>>
>

-- 
This is a staff email account managed by Adams 12 Five Star Schools.  This 
email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the sender.

Re: Best TAC Services from Equipment Vendors

[michael brooks - ESC]

Funny you should mention this now, we were just discussing (more like
lamenting...) if support is a dying industry. It seems as though vendor
budgets are shrinking to the point they only have a Sales/Pre-Sales
department, and from Day Two on you are on your own. Dramatic take of
course, but if we are speaking in trajectories




michael brooks
Sr. Network Engineer
Adams 12 Five Star Schools
michael.bro...@adams12.org

"flying is learning how to throw yourself at the ground and miss"



On Wed, Mar 6, 2024 at 11:25 AM Pascal Masha  wrote:

> Thought about it but so far I believe companies from China provide better
> and fast TAC responses to their customers than the likes of Cisco and
> perhaps that’s why some companies(where there are no restrictions)prefer
> them for critical services.
>
> For a short period in TAC call you can have over 10 R engineers and
> solutions provided in a matter of hours even if it involves software
> changes.. while these other companies even before you get in a call with a
> TAC engineer it’s hours and when they join you hear something like “my
> shift ended 15 minutes ago, hold let me look for another engineer”. WHY?
> Thoughts
>

-- 
This is a staff email account managed by Adams 12 Five Star Schools.  This 
email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the sender.

Re: Meta outage

[Michael Rathbun]

On Tue, 05 Mar 2024 11:06:11 -0500, Jay Ashworth  wrote:

>It's making the general press this hour so of course you already know about it 
>but my question is this: who peers with meta and have you seen BGP sessions 
>drop or the like? Do you operate meta CDN nodes in your network? Are they 
>screaming for help? 
>
>This doesn't sound like it's a network layer problem but I'm curious.

What I found intriguing was that I was logged out by Google Docs at the
same moment FB logged me out.  Downdetector showed a number of other
supposedly unrelated services with large outage report spikes at roughly
the same time.

mdr
-- 
 "There are no laws here, only agreements."  
-- Masahiko

Re: Leap Day

[Michael Still]

Here's a link to the full article for anyone that's not a sub to NYT:
https://www.nytimes.com/2024/02/29/science/leap-day-easter.html?unlocked_article_code=1.ZU0.5Thd.N-PiRNtpSr2c=url-share


On Fri, Mar 1, 2024 at 12:39 AM Jay R. Ashworth  wrote:

> Late, just saw the posting on BlueSky:
>
> In the wake of NTP inventor Dave Mills' death, probably the next ranking
> topchimer is NIST's Judah Levine, and the New York Times interviewed him
> about
> why we have leap years, which makes sense; all the other news outlets had
> to
> make do with lower-ranking (and less well spoken) "time experts" (I'm
> lookin'
> at you, NPR...)
>
> https://www.nytimes.com/2024/02/29/science/leap-day-easter.html
>
> Cheers,
> -- jra
>
> --
> Jay R. Ashworth  Baylink
> j...@baylink.com
> Designer The Things I Think   RFC
> 2100
> Ashworth & Associates   http://www.bcp38.info  2000 Land
> Rover DII
> St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647
> 1274
>


-- 
[stillwa...@gmail.com ~]$ cat .signature
cat: .signature: No such file or directory
[stillwa...@gmail.com ~]$ cat .disclaimer
All opinions are my own and do not represent any of my employer.
[stillwa...@gmail.com ~]$ cat .pronouns
He/Him

Re: Leap Day

[michael brooks - ESC]

Good article, enjoyable read. Most interesting tidbit: that UTC and Atomic
time disagree by 34 seconds.




michael brooks
Sr. Network Engineer
Adams 12 Five Star Schools
michael.bro...@adams12.org

"flying is learning how to throw yourself at the ground and miss"



On Thu, Feb 29, 2024 at 10:39 PM Jay R. Ashworth  wrote:

> Late, just saw the posting on BlueSky:
>
> In the wake of NTP inventor Dave Mills' death, probably the next ranking
> topchimer is NIST's Judah Levine, and the New York Times interviewed him
> about
> why we have leap years, which makes sense; all the other news outlets had
> to
> make do with lower-ranking (and less well spoken) "time experts" (I'm
> lookin'
> at you, NPR...)
>
>
> https://urldefense.com/v3/__https://www.nytimes.com/2024/02/29/science/leap-day-easter.html__;!!IR39LLzvxw!MpFPcQQ8RndifKfj3GJ0IIf46WjMUpfoznc5LheDjfFOwFLzACAjw7zdgRmqSf6SzxG2-PTkAR31pkxpl80$
>
> Cheers,
> -- jra
>
> --
> Jay R. Ashworth  Baylink
> j...@baylink.com
> Designer The Things I Think   RFC
> 2100
> Ashworth & Associates
> https://urldefense.com/v3/__http://www.bcp38.info__;!!IR39LLzvxw!MpFPcQQ8RndifKfj3GJ0IIf46WjMUpfoznc5LheDjfFOwFLzACAjw7zdgRmqSf6SzxG2-PTkAR31bqtwkps$
>  2000 Land Rover DII
> St Petersburg FL USA  BCP38: Ask For It By Name!   +1 727 647
> 1274
>

-- 
This is a staff email account managed by Adams 12 Five Star Schools.  This 
email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the sender.

Re: IPv6 uptake

[Michael Thomas]

On 2/18/24 1:10 PM, Nick Hilliard wrote:

Michael Thomas wrote on 18/02/2024 20:56:
That's really great to hear. Of course there is still the problem 
with CPE that doesn't speak v6, but that's not their fault and gives 
some reason to use their CPE.


Already solved: cable modem ipv6 support is usually also excellent, 
both in terms of subscriber services handoff and management. The 
requirements for ipv6 support are very clearly defined in the 
cablelabs docsis 3.0 specification.


So it has its own wireless? I seem to recall that there were some 
economic reasons to use their CPE as little as possible to avoid rent. 
Has that changed? Or can I run down and just buy a Cablelabs certified 
router/modem these days?


Mike

Re: IPv6 uptake

[Michael Thomas]

On 2/18/24 12:50 PM, Nick Hilliard wrote:

Michael Thomas wrote on 18/02/2024 20:28:
I do know that Cablelabs pretty early on -- around the time I 
mentioned above -- has been pushing for v6. Maybe Jason Livingood can 
clue us in. Getting cable operators onboard too would certainly be a 
good thing,


availability of provider-side ipv6 support is generally excellent on 
docsis networks. This includes end-user device support, management, 
client and server side provisioning, the works. This is one of the 
real ipv6 success stories in the service provider arena.


That's really great to hear. Of course there is still the problem with 
CPE that doesn't speak v6, but that's not their fault and gives some 
reason to use their CPE.


Mike

Re: IPv6 uptake (was: The Reg does 240/4)

[Michael Thomas]

On 2/18/24 8:47 AM, Greg Skinner via NANOG wrote:

On Feb 17, 2024, at 11:27 AM, William Herrin  wrote:

On Sat, Feb 17, 2024 at 10:34?AM Michael Thomas  wrote:


Funny, I don't recall Bellovin and Cheswick's Firewall book discussing
NAT.

And mine too, since I hadn't heard of "Firewalls and Internet
Security: Repelling the Wily Hacker" and have not read it.

For what it's worth, both editions of Bellovin and Cheswick's Firewalls book 
are online. [1]  Also, there are discussions about NAT and how it influenced 
IPng (eventually IPv6) on the big-internet list. [2]


FWIW, while at Cisco I started to get wind of some NAT-like proposal 
being floated by 3COM at Packetcable back in the late 90's, early 2000's 
(sorry, I have no memory of the specifics now). That was pretty 
horrifying to me and others as the implication was that we'd have to 
implement it in our routers, which I'm sure 3COM viewed as a feature, 
not a bug. We pushed back that implementing IPv6 was a far better option 
if it came down to that. That sent me and Steve Deering off on an 
adventure to figure out how we might actually make good on that 
alternative in the various service provider BU's. Unsurprisingly the 
BU's were not very receptive not just because of the problems with v6 vs 
hardware forwarding, but mostly because providers weren't asking for it. 
They weren't asking for CGNAT like things either though so it was mostly 
the status quo. IOS on the other hand was taking IPv6 much more 
seriously so that providers could at least deploy it in the small for 
testing, pilots, etc even if it was a patchwork in the various platforms.


The problem with v6 uptake has always been on the provider side. BU's 
wouldn't have wanted to respin silicon but if providers were asking for 
it and it gave them a competitive advantage, they'd have done it in a 
heartbeat. It's heartening to hear that a lot of big providers and orgs 
are using IPv6 internally to simplify management along with LTE's use of 
v6. I don't know what's happening in MSO land these days, but it would 
be good to hear if they too are pushing a LTE-like solution. I do know 
that Cablelabs pretty early on -- around the time I mentioned above -- 
has been pushing for v6. Maybe Jason Livingood can clue us in. Getting 
cable operators onboard too would certainly be a good thing, though LTE 
doesn't have to deal with things like brain dead v4-only wireless 
routers on their network.


Mike

Re: IPv6 uptake (was: The Reg does 240/4)

[Michael Thomas]

On 2/17/24 11:27 AM, William Herrin wrote:

On Sat, Feb 17, 2024 at 10:34 AM Michael Thomas  wrote:

I didn't hear about NAT until the
late 90's, iirc. I've definitely not heard of Gauntlet.

Then there are gaps in your knowledge.


Funny, I don't recall Bellovin and Cheswick's Firewall book discussing
NAT.

And mine too, since I hadn't heard of "Firewalls and Internet
Security: Repelling the Wily Hacker" and have not read it.

I see that the book was published in 1994. In 1994 Gauntlet was
calling their process "transparent application layer gateways," not
NAT.

What was called NAT in 1994 was stateless 1:1 NAT, where one IP mapped
to exactly one IP in both directions. Stateless 1:1 NAT had no impact
on security. But that's not the technology we're talking about in
2024. Stateless 1:1 NAT is so obsolete that support was dropped from
the Linux kernel a long time ago. That actually caused a problem for
me in 2017. I had a use where I wanted 1:1 NAT and wanted to turn off
connection tracking so that I could do asymmetric routing through the
stateless translators. No go.

So it does not surprise me that a 1994 book on network security would
not have discussed NAT. They'd have referred to the comparable
contemporary technology, which was "transparent application layer
gateways." Those behaved like what we now call NAT but did the job a
different way: instead of modifying packets, they terminated the
connection and proxied it.


I don't recall the book talking about proxies, but it's been a long 
time. It was mostly about (stateful) firewalls, iirc. The rapid 
expansion of the internet caused a huge need for a big band-aid, 
especially with shitty windows boxes emerging on the net shortly after. 
A stateful firewall walled off for incoming on client subnets was 
perfectly sufficient though, and need to provision clients with proxies 
and the necessary software. The book is not very long and honestly 
that's a feature as it seemed to mostly be trying to get the word out 
that we should be protecting ourselves at the borders until better 
security could get deployed. If NAT's supposed belt and suspenders 
security was such a big feature, I don't recall anybody talking about it 
that way back then. That's why it's always seemed like a post-hoc 
rationalization. When I was at Cisco, all of the internal networks were 
numbered in public address space and I never once heard any clamor for 
the client space to be renumbered into RFC 1918 space for security 
reasons. Admittedly anybody doing so would have faced fierce resistance, 
but if there were any debate at all it was that adding state to network 
flows was a Bad Thing.


NAT has always been about overloading public IP addresses first and 
foremost. The supposed security gains are vastly dwarfed by the decrease 
in functionality that NAT brings with it. One only has to look at the 
mental gymnastics that goes into filling out SDP announcements for VoIP 
to know that any supposed security benefits are not worth the trouble 
that it brings. If it were only security, nobody would have done it. It 
was always about address conservation first and foremost.


Mike

Re: IPv6 mail The Reg does 240/4

[Michael Thomas]

On 2/17/24 2:21 PM, John Levine wrote:

But what happens under the hood at

major mailbox providers is maddeningly opaque so who really knows? It
would be nice if MAAWG published a best practices or something like that
to outline what is actually happening in live deployments.

Unfortunately, spammers can read just as well as we can so it's not
going to happen.


They already have the recon so they don't need any help. The rest of us 
could be helped by what the current art is.


Mike

Re: The Reg does 240/4

[Michael Thomas]

On 2/17/24 10:19 AM, Owen DeLong via NANOG wrote:
Mike, it’s true that Google used to be a lot less strict on IPv4 email 
than IPv6, but they want SPF and /or DKIM on everything now, so it’s 
mostly the same. There is less reputation data available for IPv6 and 
server reputation is a harder problem in IPv6, but reputation systems 
are becoming less relevant.


I kind of get the impression that once you get to aggregates at the 
domain level like DKIM or SPF, addresses as a reputation vehicle don't 
much figure into decision making. But what happens under the hood at 
major mailbox providers is maddeningly opaque so who really knows? It 
would be nice if MAAWG published a best practices or something like that 
to outline what is actually happening in live deployments.


Mike

Re: IPv6 uptake

[Michael Thomas]

On 2/17/24 10:26 AM, Owen DeLong via NANOG wrote:



On Feb 16, 2024, at 14:20, Jay R. Ashworth  wrote:

- Original Message -

From: "Justin Streiner" 
4. Getting people to unlearn the "NAT=Security" mindset that we were forced
to accept in the v4 world.

NAT doesn't "equal" security.

But it is certainly a *component* of security, placing control of what internal
nodes are accessible from the outside in the hands of the people inside.

Uh, no… no it is not. Stateful inspection (which the kind of NAT (actually 
NAPT) you are assuming here depends on) is a component of security. You can do 
stateful inspection without mutilating the header and have all the same 
security benefits without losing or complicating the audit trail.


Exactly. As I said elsewhere, the security properties of NAT were a 
post-hoc rationalization. In the mean time, it has taken on its own life 
as if not NAT'ing (but still having stateful firewalls) would end the 
known security universe. We can seriously lose NAT for v6 and not lose 
anything of worth.


Mike

Re: IPv6 uptake (was: The Reg does 240/4)

[Michael Thomas]

On 2/16/24 6:33 PM, William Herrin wrote:

On Fri, Feb 16, 2024 at 6:10 PM Ryan Hamel  wrote:

Depending on where that rule is placed within your ACL, yes that can happen 
with *ANY* address family.

Hi Ryan,

Correct. The examples illustrated a difference between a firewall
implementing address-overloaded NAT and a firewall implementing
everything except the address translation. Either example could be
converted to the other address family and it would work the same way.


All things aside, I agree with Dan that NAT was never
ever designed to be a security tool. It is used because
of the scarcity of public address space, and it provides
a "defense" depending on how it is implemented, with
minimal effort. This video tells the story of NAT and the
Cisco PIX, straight from the creators
https://youtu.be/GLrfqtf4txw

NAT's story, the modern version of NAT when we talk about IPv4
firewalls, started in the early '90s with the Gauntlet firewall. It
was described as a transparent application layer gateway. Gauntlet
focused on solving enterprise security issues. Gauntlet's technology
converged with what was then 1:1 NAT to produce the address-overloaded
NAT like what later appeared in the Cisco PIX (also first and foremost
a security product) and is present in all our DSL and cable modems
today.

Security came first, then someone noticed it'd be useful for address
conservation too. Gauntlet's customers generally had or could readily
get a supply of public IP addresses. Indeed, when Gauntlet was
released, IP addresses were still available from
hostmas...@internic.net at zero cost and without any significant
documentation. And Gauntlet was expensive: folks who couldn't easily
obtain public IP addresses also couldn't afford it.


Funny, I don't recall Bellovin and Cheswick's Firewall book discussing 
NAT. That was sort of the go-to book for hard-on-the-outside 
soft-on-the-inside defense. Maybe they were unaware of this, or maybe 
they didn't agree with the premise. I didn't hear about NAT until the 
late 90's, iirc. I've definitely not heard of Gauntlet.


As I recall, it was very much an afterthought with cable/DOCSIS to use 
NAT to conserve addresses. The headend DHCP server just gave public 
addresses to whoever asked. DOCSIS CPE at that time was just an L2 
modem. NAT traversal absolutely was not on the table with Packetcable 
back in the late 90's, and believe me we were very concerned about the 
security of MGCP since it was UDP based.


Which is to say that NAT came around to preserve address space. Any 
security properties were sort of a post-hoc rationalization and hotly 
debated given all the things NAT broke.


Mike

Re: IPv6 uptake (was: The Reg does 240/4)

[Michael Thomas]

On 2/16/24 5:37 PM, William Herrin wrote:

On Fri, Feb 16, 2024 at 5:33 PM Michael Thomas  wrote:

So you're not going to address that this is a management plain problem.

Hi Mike,

What is there to address? I already said that NAT's security
enhancement comes into play when a -mistake- is made with the network
configuration. You want me to say it again? Okay, I've said it again.


The implication being that we should keep NAT'ing ipv6 for... a thin 
veil of security. That all of the other things that NAT breaks is worth 
the trouble because we can't trust our fat fingers on firewall configs.


Mike

Re: IPv6 uptake (was: The Reg does 240/4)

[Michael Thomas]

On 2/16/24 5:30 PM, William Herrin wrote:

On Fri, Feb 16, 2024 at 5:22 PM Michael Thomas  wrote:

On 2/16/24 5:05 PM, William Herrin wrote:

Now, I make a mistake on my firewall. I insert a rule intended to
allow packets outbound from 2602:815:6001::4 but I fat-finger it and
so it allows them inbound to that address instead. Someone tries to
telnet to 2602:815:6001::4. What happens? Hacked.

Yes, but if the DHCP database has a mistake it's pretty much the same
situation since it could be numbered with a public address.

Um. No. You'd have to make multiple mistakes cross-contaminating your
public and private ethernet segments yet somehow without completely
breaking your network rendering it inoperable.


So you're not going to address that this is a management plain problem. ok.

Mike

Re: IPv6 uptake (was: The Reg does 240/4)

[Michael Thomas]

On 2/16/24 5:05 PM, William Herrin wrote:

On Fri, Feb 16, 2024 at 3:13 PM Michael Thomas  wrote:

If you know which subnets need to be NAT'd don't you also know which
ones shouldn't exposed to incoming connections (or conversely, which
should be permitted)? It seems to me that all you're doing is moving
around where that knowledge is stored? Ie, DHCP so it can give it
private address rather than at the firewall knowing which subnets not to
allow access? Yes, DHCP can be easily configured to make everything
private, but DHCP for static reachable addresses is pretty handy too.

Hi Mike,

Suppose I have a firewall at 2602:815:6000::1 with an internal network
of 2602:815:6001::/64. Inside the network on 2602:815:6001::4 I have a
switch that accepts telnet connections with a user/password of
admin/admin. On the firewall, I program it to disallow all Internet
packets to 2602:815:6001::/64 that are not part of an established
connection.

Someone tries to telnet to 2602:815:6001::4. What happens? Blocked.

Now, I make a mistake on my firewall. I insert a rule intended to
allow packets outbound from 2602:815:6001::4 but I fat-finger it and
so it allows them inbound to that address instead. Someone tries to
telnet to 2602:815:6001::4. What happens? Hacked.


Yes, but if the DHCP database has a mistake it's pretty much the same 
situation since it could be numbered with a public address. For both you 
can have the default as "reject" or "accept" -- that's just a default 
and depends on how you want to manage your network.


NAT is not without its own set of problems, so if this boils down to a 
subnet management issue there are obviously ways to deal with that to 
avoid NAT's problems. Both DHCP and firewall configs don't have to be 
the ultimate source of truth about any of this. And that's likely a Good 
Thing since you want them to be pretty much as fungible and replaceable 
as possible. If you're exposed to fat fingers for either, you're 
probably already in trouble. Something in the management plain is far 
more likely to care about this kind of thing than hardware vendors who 
see that as a cost center with predictably shitty implementations.


Mike

Re: IPv6 uptake (was: The Reg does 240/4)

[Michael Thomas]

On 2/16/24 3:01 PM, William Herrin wrote:

On Fri, Feb 16, 2024 at 2:19 PM Jay R. Ashworth  wrote:

From: "Justin Streiner" 
4. Getting people to unlearn the "NAT=Security" mindset that we were forced
to accept in the v4 world.

NAT doesn't "equal" security.

But it is certainly a *component* of security, placing control of what internal
nodes are accessible from the outside in the hands of the people inside.

Hi Jay,

Every firewall does that. What NAT does above and beyond is place
control of what internal nodes are -addressable- from the outside in
the hands of the people inside -- so that most of the common mistakes
with firewall configuration don't cause the internal hosts to -become-
accessible.


If you know which subnets need to be NAT'd don't you also know which 
ones shouldn't exposed to incoming connections (or conversely, which 
should be permitted)? It seems to me that all you're doing is moving 
around where that knowledge is stored? Ie, DHCP so it can give it 
private address rather than at the firewall knowing which subnets not to 
allow access? Yes, DHCP can be easily configured to make everything 
private, but DHCP for static reachable addresses is pretty handy too.


Mike

Re: IPv6 Traffic Re: IPv6? Re: Where to Use 240/4 Re: 202401100645.AYC Re: IPv4 address block

[Michael Thomas]

On 1/15/24 11:02 PM, Saku Ytti wrote:

On Mon, 15 Jan 2024 at 21:08, Michael Thomas  wrote:


An ipv4 free network would be nice, but is hardly needed. There will
always be a long tail of ipv4 and so what? You deal with it at your

I mean Internet free DFZ, so that everyone is not forced to maintain
two stacks at extra cost, fragility and time. Any protocols at the
inside networks are fine, as long as you're meeting the Internet with
IPv6-only stack. I'm sure there are CLNS, IPX, AppleTalk etc networks
there, but that doesn't impose a cost to everyone wanting to play.

Um, so what? There is lots of cruft the world over that would be better 
if it finally died. Somehow we keep on. It's just a cost of doing 
business. If mobile operators can support it with their millions or even 
billions of customers, I think everybody else can too. It's not like 
ipv4 address depletion is a static problem either -- it's only going to 
get worse as time goes on so it's what's really driving opex where v6 is 
pretty much a one-off investment in comparison i'd think.


Mike

Re: IPv6 Traffic Re: IPv6? Re: Where to Use 240/4 Re: 202401100645.AYC Re: IPv4 address block

[Michael Thomas]

On 1/15/24 12:26 AM, Saku Ytti wrote:

On Mon, 15 Jan 2024 at 10:05, jordi.palet--- via NANOG  wrote:


In actual customer deployments I see the same levels, even up to 85% of IPv6 
traffic. It basically depends on the usage of the caches and the % of 
residential vs corporate customers.

You think you are contributing to the IPv6 cause, by explaining how
positive the situation is. But in reality you are damaging it greatly,
because you're not communicating that we are not on a path to IPv4
free Internet. If we had been on such a path, we would have been IPv4
free for more than a decade. And unless we admit we are not on that
path, we will not work to get on that path.

An ipv4 free network would be nice, but is hardly needed. There will 
always be a long tail of ipv4 and so what? You deal with it at your 
borders as a piece of non-recurring engineering and that is that. The 
mobile operators model seems to be working pretty well for them and 
seems likely that it is an opex cost down for them since they don't have 
to run two networks internally nor deal with the cost of ipv4 subnets 
(or at least not as much? not sure how it exactly works). Worrying about 
whether ipv4 will ever go away misses the point, imo.


Mike

Re: IPv6 Traffic Re: IPv6? Re: Where to Use 240/4 Re: 202401100645.AYC Re: IPv4 address block

[Michael Thomas]

On 1/15/24 12:56 AM, jordi.palet--- via NANOG wrote:
No, I’m not saying that. I’m saying "in actual deployments", which 
doesn’t mean that everyone is deploying, we are missing many ISPs, we 
are missing many enterprises.


I don't think what's going on internally with enterprise needs to change 
much if they just gatewayed to a v6 upstream instead of v4 at the border 
if they were given that option. The problem has always been with ISP's 
and routers. When v6 first started to percolate (early 90's) i looked at 
it for my embedded OS and the projects that used it and didn't figure it 
would take much effort to implement it. So for hosts i really don't 
think that was a roadblock. But if hosts don't have something upstream 
to sink v6 traffic and especially to access the public internet there's 
not much incentive to implement it or deploy it. ISP's used the excuse 
that routers didn't implement it which was definitely a huge problem but 
as it turns out it was still an excuse since a lot has changed in the 
last 20 years and still rollout continues at a glacial pace.


I think one of the more encouraging trends are ISP's and enterprises 
switching over to v6 internally as a cost saving measure to not run a 
dual network. Aren't Comcast and Facebook examples?


It's sort of disturbing that there are still people on this list that 
want to relitigate something that happened 30 years ago. that reeks of 
religion not tech. By all means, set up CGNAT's in a pique.


Mike

Re: IPv6? Re: Where to Use 240/4 Re: 202401100645.AYC Re: IPv4 address block

[Michael Thomas]

On 1/12/24 11:54 AM, Darrel Lewis wrote:

On Jan 12, 2024, at 11:47 AM, Seth David Schoen  wrote:

Michael Thomas writes:


I wonder if the right thing to do is to create a standards track RFC that
makes the experimental space officially an add on to rfc 1918. If it works
for you, great, if not your problem. It would at least stop all of these
recurring arguments that we could salvage it for public use when the
knowability of whether it could work is zero.

In 2008 there were two proposals

https://datatracker.ietf.org/doc/draft-fuller-240space/
https://datatracker.ietf.org/doc/draft-wilson-class-e/

where the former was agnostic about how we would eventually be able to
use 240/4, and the latter designated it as RFC 1918-style private space.
Unfortunately, neither proposal was adopted as an RFC then, so we lost a
lot of time in which more vendors and operators could have made more
significant progress on its usability.

Well, we were supposed to all be using IPv6 (only) by now, and making 240/4 
useable was just going to slow that process down.

IMHO, this is what you get when religion is mixed with engineering.


But it wouldn't be globally routable so it wouldn't change much. I'm not 
even sure it would change much on the ground for CGNAT deployment? You 
still need enough public addresses to service the load. It might make it 
easier than partitioning your internal net into multiple 10/8 but on the 
other hand you need to make certain your internal net still works with 
240/4.


I'm mostly throwing this out there as a way to shut down these kinds of 
discussions.


Mike

Re: IPv6? Re: Where to Use 240/4 Re: 202401100645.AYC Re: IPv4 address block

[Michael Thomas]

On 1/12/24 8:45 AM, Owen DeLong via NANOG wrote:
Frankly, I care less. No matter how you use whatever IPv4 space you 
attempt to cajole into whatever new form of degraded service, the 
simple fact remains. IPv4 is a degraded technology that only continues 
to get worse over time. NAT was bad. CGNAT is even worse (and 
tragically does nothing to eliminate consumer NAT, just layers more 
disaster on top of the existing mess).


The only currently available end to end peer to peer technology, for 
better or worse, is IPv6. Despite its naysayers, it is a proven 
technology that has been shouldering a significant fraction of 
internet traffic for many years now and that fraction continues to grow.


You simply can’t make IPv4 adequate and more hackers to extend its 
life merely expands the amount of pain and suffering we must endure 
before it is finally retired.


I wonder if the right thing to do is to create a standards track RFC 
that makes the experimental space officially an add on to rfc 1918. If 
it works for you, great, if not your problem. It would at least stop all 
of these recurring arguments that we could salvage it for public use 
when the knowability of whether it could work is zero.


Mike

Re: 202401100645.AYC Re: IPv4 address block

[Michael Butler via NANOG]

On 1/10/24 10:12, Tom Beecher wrote:

Karim-

Please be cautious about this advice, and understand the full context.

240/4 is still classified as RESERVED space. While you would certainly 
be able to use it on internal networks if your equipment supports it, 
you cannot use it as publicly routable space. There have been many 
proposals over the years to reclassify 240/4, but that has not happened, 
and is unlikely to at any point in the foreseeable future.


While you may be able to get packets from point A to B in a private 
setting, using them might also be .. a challenge.


There's a whole bunch of software out there that makes certain 
assumptions about allowable ranges. That is, they've been compiled with 
a header that defines ..


#define IN_BADCLASS(i)  (((in_addr_t)(i) & 0xf000) == 0xf000)

Michael

RE: What are these Google IPs hammering on my DNS server?

[Michael Hare via NANOG]

Damian-

Not Google or ISCs fault, our customers have made some decisions that have 
exasperated the issues.  By and away the biggest problem facing my customers is 
that they have chosen a stateful border firewall that collapses due to session 
exhaustion and they put everything, including aDNS, behind said firewall.  “If 
it hurts, don’t do it” comes to mind, but out of my hands.

At quick glance following the ISC link I didn’t see the compute infrastructure 
[core count] needed to get 1Mpps.  There is an obvious difference between 99% 
load of ~500rps and 1M, so we can maybe advise to not undersize ADNS if that's 
an issue.

I'm an ISP engineer and am generally not the directly affected party, so I 
don't get to pick these implementation details for my customers.  I appreciate 
the background and suggestions from you and others on this thread like Mark.  
That's an interesting comment about DNSSEC that I hadn't considered.

-Michael

From: Damian Menscher 
Sent: Monday, December 4, 2023 12:21 PM
To: Michael Hare 
Cc: John R. Levine ; nanog@nanog.org
Subject: Re: What are these Google IPs hammering on my DNS server?

Google Public DNS (8.8.8.8) attempts to identify and filter abuse, and while we 
think we're fairly effective for large attacks (eg, those above 1Mpps), it gets 
more challenging (due to risk of false positives) to adequately filter small 
attacks.  I should note that we generally see the attack traffic coming from 
botnets, or forwarding resolvers that blend the attack traffic with legitimate 
traffic.

Based on ISC BIND load-tests [0], a single DNS server can handle O(1Mpps).  
Also, no domain should be served by a single DNS server, so O(1Mpps) seems like 
a safe lower-bound for small administrative domains (larger ones will have more 
redundancy/capacity).  Based on these estimates, we haven't treated mitigation 
of small attacks as a high priority.  If O(25Kpps) attacks are causing real 
problems for the community, I'd appreciate that feedback and some hints as to 
why your experience differs from the ISC BIND load-tests.  With a better 
understanding of the pain-points, we may be able to improve our filtering a 
bit, though I suspect we're nearing the limits of what is attainable.

Since it was mentioned up-thread, I'd caution against dropping queries from 
likely-legitimate recursives, as that will lead to a retry storm that you won't 
like (based on a few reports of authoritatives who suffered outages, the retry 
storm increased demand by 30x and they initially misdiagnosed the root cause as 
a DDoS).  The technically correct (if not entirely practical) mitigation for a 
DNS cache-busting attack laundered through open recursives is to deploy DNSSEC 
and issue NSEC/NSEC3 responses to allow the recursives to cache the 
non-existence of the randomized labels.

[0] https://www.isc.org/blogs/bind-performance-september-2023/

Damian
--
Damian Menscher :: Security Reliability Engineer :: Google :: AS15169

On Sun, Dec 3, 2023 at 1:22 PM Michael Hare via NANOG 
mailto:nanog@nanog.org>> wrote:
John-

This is little consolation, but at AS3128, I see the same thing to our 
downstream at times, claiming to come from both 13335 and 15169 often 
simultaneously at the tune of 25Kpps , "assuming it's not spoofed", which is 
pragmatically impossible to prove for me given our indirect relationships with 
these companies.  When I see these events, I typically also see a wide variety 
of country codes participating simultaneously.  Again, assuming it's not 
spoofed.  To me it just looks like effective harassment with 13335/15169 
helping out.  I pine for the internet of the 1990s.

Recent events in GMT for us were the following, curious if you see the same
~ Nov 26 05:40
~ Nov 30 00:40
~ Nov 30 05:55

Application agnostic, on the low $ end for "fixes", if it's either do something 
or face an outage, I've found some utility in short term automated DSCP 
coloring on ingress paired with light touch policing as close to the end host 
as possible, which at least keeps things mostly working during times of 
conformance.  Cheap/fast and working ... most of the time.  Definitely not 
great or complete at all, and a role I'd rather not play as an educational 
ISP/enterprise.

So what are most folks doing to survive crap like this?  Nothing/waiting it 
out?  Oursourcing DNS?  Scrubbing appliance?  Poormans stuff like I mention 
above?

-Michael

> -Original Message-
> From: NANOG 
> mailto:wisc@nanog.org>> On
> Behalf Of John R. Levine
> Sent: Sunday, December 3, 2023 1:18 PM
> To: Peter Potvin 
> mailto:peter.pot...@accuristechnologies.ca>>
> Cc: nanog@nanog.org<mailto:nanog@nanog.org>
> Subject: Re: What are these Google IPs hammering on my DNS server?
>
> > Did a bit of digging on Google's developer site and came across this:
> > https://developers.google.com/speed/public-
> dns/faq#locations_of_ip_address_

RE: What are these Google IPs hammering on my DNS server?

[Michael Hare via NANOG]

John-

This is little consolation, but at AS3128, I see the same thing to our 
downstream at times, claiming to come from both 13335 and 15169 often 
simultaneously at the tune of 25Kpps , "assuming it's not spoofed", which is 
pragmatically impossible to prove for me given our indirect relationships with 
these companies.  When I see these events, I typically also see a wide variety 
of country codes participating simultaneously.  Again, assuming it's not 
spoofed.  To me it just looks like effective harassment with 13335/15169 
helping out.  I pine for the internet of the 1990s.

Recent events in GMT for us were the following, curious if you see the same
~ Nov 26 05:40
~ Nov 30 00:40
~ Nov 30 05:55

Application agnostic, on the low $ end for "fixes", if it's either do something 
or face an outage, I've found some utility in short term automated DSCP 
coloring on ingress paired with light touch policing as close to the end host 
as possible, which at least keeps things mostly working during times of 
conformance.  Cheap/fast and working ... most of the time.  Definitely not 
great or complete at all, and a role I'd rather not play as an educational 
ISP/enterprise.

So what are most folks doing to survive crap like this?  Nothing/waiting it 
out?  Oursourcing DNS?  Scrubbing appliance?  Poormans stuff like I mention 
above?

-Michael 

> -Original Message-
> From: NANOG  On
> Behalf Of John R. Levine
> Sent: Sunday, December 3, 2023 1:18 PM
> To: Peter Potvin 
> Cc: nanog@nanog.org
> Subject: Re: What are these Google IPs hammering on my DNS server?
> 
> > Did a bit of digging on Google's developer site and came across this:
> > https://developers.google.com/speed/public-
> dns/faq#locations_of_ip_address_ranges_google_public_dns_uses_to_send_
> queries
> >
> > Looks like the IPs you mentioned belong to Google's public DNS resolver
> > based on that list on their site. They could also be spoofed though from a
> > DNS AMP attack, so keep that in mind.
> 
> Per my recent message, the replies are tiny so if it's an amplification
> attack, it's a very incompetent one.  The queries are case randomized so I
> guess it's really Google.  Sigh.
> 
> If anyone is wondering, I have a passive aggressive countermeasure against
> some overqueriers that returns ten NS referral names, and then 25 random
> IP addresses for each of those names, but I don't do that to Google.
> 
> R's,
> John
> 
> > --
> > *Accuris Technologies Ltd.*
> >
> >
> > On Sun, Dec 3, 2023 at 1:51 PM John Levine  wrote:
> >
> >> At contacts.abuse.net, I have a little stunt DNS server that provides
> >> domain contact info, e.g.:
> >>
> >> $ host -t txt comcast.net.contacts.abuse.net
> >> comcast.net.contacts.abuse.net descriptive text "ab...@comcast.net"
> >>
> >> $ host -t hinfo comcast.net.contacts.abuse.net
> >> comcast.net.contacts.abuse.net host information "lookup" "comcast.net"
> >>
> >> Every once in a while someone decides to look up every domain in the
> >> world and DoS'es it until I update my packet filters. This week it's
> >> been this set of IPs that belong to Google. I don't think they're
> >> 8.8.8.8. Any idea what they are? Random Google Cloud customers? A
> >> secret DNS mapping project?
> >>
> >>  172.253.1.133
> >>  172.253.206.36
> >>  172.253.1.130
> >>  172.253.206.37
> >>  172.253.13.196
> >>  172.253.255.36
> >>  172.253.13.197
> >>  172.253.1.131
> >>  172.253.255.35
> >>  172.253.255.37
> >>  172.253.1.132
> >>  172.253.13.193
> >>  172.253.1.129
> >>  172.253.255.33
> >>  172.253.206.35
> >>  172.253.255.34
> >>  172.253.206.33
> >>  172.253.206.34
> >>  172.253.13.194
> >>  172.253.13.195
> >>  172.71.125.63
> >>  172.71.117.60
> >>  172.71.133.51
> >>
> >> R's,
> >> John
> >>
> >
> 
> Regards,
> John Levine, jo...@taugh.com, Primary Perpetrator of "The Internet for
> Dummies",
> Please consider the environment before reading this e-mail. https://jl.ly

Re: sigs wanted for a response to the fcc's NOI for faster broadband speeds

[michael brooks - ESC]

As one beholden to USAC/FCC I have to agree with Shane...




michael brooks
Sr. Network Engineer
Adams 12 Five Star Schools
michael.bro...@adams12.org

"flying is learning how to throw yourself at the ground and miss"



On Fri, Dec 1, 2023 at 10:39 AM Tom Mitchell 
wrote:

> Not sure we need the FCC telling us how to build products or run
> networks.  Seat belts are life-or-death, but bufferbloat is rarely fatal
> ;-)  Let it be a point of differentiation.
>
> -- Tom
>
>
> On Thu, Nov 30, 2023 at 4:56 PM Dave Taht  wrote:
>
>> Over here:
>>
>>
>> https://docs.google.com/document/d/19ADByjakzQXCj9Re_pUvrb5Qe5OK-QmhlYRLMBY4vH4/edit
>> <https://urldefense.com/v3/__https://docs.google.com/document/d/19ADByjakzQXCj9Re_pUvrb5Qe5OK-QmhlYRLMBY4vH4/edit__;!!IR39LLzvxw!M_bC8YhyxVaDflZyWHjIqI3S_nFkGJi-hoTv9t_2pKc_5c68WnYm3nvy9SxIy0mdCEr2GVgtCcvNuGwxBWz84-A2Y2Ag$>
>>
>> Us bufferbloat folk have been putting together a response to the FCC's
>> NOI (notice of inquiry) asking for feedback as to increasing the
>> broadband speeds beyond 100/20 Mbit.
>>
>> "Calls for further bandwidth increases are analogous to calling for
>> cars to have top speeds of 100, 200, or 500 miles per hour. Without
>> calling also for better airbags, bumpers, brakes, or steering wheels,
>> (or roads designed to minimize travel delay), these initiatives will
>> fail (and are failing) to meet the needs of present and future users
>> of the internet."
>>
>> Comments (and cites) welcomed also! The text is still somewhat in flux...
>>
>>
>> --
>> :( My old R campus is up for sale: https://tinyurl.com/yurtlab
>> <https://urldefense.com/v3/__https://tinyurl.com/yurtlab__;!!IR39LLzvxw!M_bC8YhyxVaDflZyWHjIqI3S_nFkGJi-hoTv9t_2pKc_5c68WnYm3nvy9SxIy0mdCEr2GVgtCcvNuGwxBWz846BrgErs$>
>> Dave Täht CSO, LibreQos
>>
>

-- 
This is a staff email account managed by Adams 12 Five Star Schools.  This 
email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the sender.

Re: Zayo woes

[michael brooks - ESC]

If it makes you feel any better, we have been waiting for a simple BCI
Circuit from Comcast for over a year (not 100% their fault, they had to
permit a canal crossing). Two weeks ago, we received a construction
completion email from our account manager, and I scheduled a turnup for the
following week. Then the construction crew showed up the next day




michael brooks
Sr. Network Engineer
Adams 12 Five Star Schools
michael.bro...@adams12.org

"flying is learning how to throw yourself at the ground and miss"



On Tue, Nov 14, 2023 at 11:04 AM Billy Earley via NANOG 
wrote:

> Navigating the Zayo maze has become my five-month odyssey, attempting to
> breathe life into a new wave circuit. Yet, at every turn, the elusive
> Letter of Authorization (LOA) emerges as a formidable opponent, adorned
> with syntax problems or entangled in the chaos of already occupied ports.
>
> Securing new LOAs has evolved into a drawn-out saga, reminiscent of a
> never-ending loop that defies both reason and resolution. Weeks are spent
> coaxing these documents into existence, only to find myself caught in the
> clutches of the same cycle, ad infinitum.
>
> Communication with Zayo has been challenging. Phone calls always go
> unanswered. Persistent radio-silence despite emails promising regular
> updates. Zayo's website boldly touts "waves on demand" with a 24-hour
> turn-up, a tantalizing promise that seems more mirage than reality in my
> current struggle.
>
> Wish me luck as I continue my quest through the Zayo wilderness. May the
> waves finally align in my favour some day soon. If anyone at Zayo is
> reading this - I would love to hear from you off-list.
>
>
>
> On Sep 21, 2023, at 01:15, Richard Holbo  wrote:
>
> Laughing out Loud, really, good views all...
> Having been through this a few times.. and being one of those who is now
> the one of the hated C level guys..
> Much truth is spoken here.  EBITA and size are the issues IMHO in our
> current system.
> Having been the owner of a few "smallish" retail ISPs in the west.. I can
> say with certainty that size is an issue.  in the world of today a
> small/medium ISP can do OK, but can never access the funds or resources
> necessary to actually be a long term survivor.  To that end we look at
> sources of money that require us to sell a majority interest in our company
> to make it to the next level, that and exhaustion from staying awake nights
> wondering how to make next payroll because we spent a couple million on new
> gear.
> This money seeking then comes with an interview system where we court the
> guys with money and they court us.. The outcome is always iffy.  You can
> partner with money that has a good plan or one that sucks (done both).
> Even if you get good money, you will suffer from culture issues.  Small
> ISP's tend to focus on the people and the customers, the larger you get the
> more important the money becomes (EBITA) which for a lot of the employees
> is, well, hard.
> But staying small in general = extinction so  you do what you do to keep
> employees working (at least that's what I tell myself).
> Good views all, and I totally agree with @Matthew Petach miracles
> statement.. I'm no longer that guy, but I like to think I was in the past,
> and I've got a bunch of them working for me now, so I try really hard to
> appreciate it and to recognize that that is what is happening, I fight for
> the money to do it as right as possible.. but I do depend on  you
> miracle workers.
>
> /thanks
> /rh
>
> On Wed, Sep 20, 2023 at 9:30 AM Matthew Petach 
> wrote:
>
>>
>>
>> On Tue, Sep 19, 2023 at 12:21 PM Mike Hammett  wrote:
>>
>>> Well sure, and I would like to think (probably mistakenly) that just no
>>> one important enough (to the money people) made the money people that these
>>> other things are *REQUIRED* to make the deal work.
>>>
>>> Obviously, people lower on the ladder say it all of the time, but the
>>> important enough money people probably don't consider those people
>>> important enough to listen to.
>>>
>>
>>
>> Not quite.
>>
>> It's more of what Mark said:
>>
>> "  I blame this on the success of how well we have built the Internet
>> with whatever box and tool we have, as network engineers."
>>
>> I have worked time and time again with absolute miracle workers in the
>> networking field.
>> They say over and over again "to make this work, we need $X M to get the
>> right hardware", even directly to the CFO.
>>
>> They get handed a roll of duct tape, some baling 

Re: Appropriate venue to find out about the state of art of spear phishing defense?

[Michael Thomas]

On 11/13/23 12:29 PM, Mel Beckman wrote:
We use KnowBe4.com's user training. That's really the only way you can 
fight this, since its a human problem, not a technical one. These guys 
provide fully automated, AI based (well, who knows what that means) 
simulated phishing attacks, largely to give users real-world practical 
experience detecting and fending off attacks. You get a report card on 
each users to, so you know where the weaknesses are in your staff 
knowledge. Their training regimen includes some pretty good 
self-guided instructional videos.


DMARC, SPF, digitally-signed emails, encryption, none of that matters 
if a user can be tricked into letting the crooks in the front door.


I think that both are needed, to be honest. The signatures can be a tool 
in the user's arsenal but if they are clueless and gullible there isn't 
much you can do about that.



Mike

Appropriate venue to find out about the state of art of spear phishing defense?

[Michael Thomas]

I know this is only tangentially relevant to nanog, but I'm curious if 
anybody knows where I can ask what orgs do to combat spear phishing? 
Spear phishing doesn't require that you deploy DMARC since you can know 
your own policy even if you aren't comfortable publishing it to the world.


tia, Mike

Re: [EXTERNAL] DNS filtering in practice, Re: Charter DNS servers returning malware filtered IP addresses

[Michael Thomas]

On 10/28/23 3:13 AM, John Levine wrote:

It appears that Michael Thomas  said:

If you're one of the small minority of retail users that knows enough
about the technology to pick your own resolver, go ahead.  But it's
a reasonable default to keep malware out of Grandma's iPad.

How does this line up with DoH? Aren't they using hardwired resolver
addresses? I would hope they are not doing anything heroic.

Generally, no.  I believe that Chrome probes whatever resolver is configured
into the system and uses that if it does DoH or DoT.

At one point Firefox was going to send everything to their favorite
DoH resolver but they got a great deal of pushback from people who
pointed out that they had policies on their networks and they'd have
to ban Firefox.  Firefox responded with a lame hack
where you can tell your cache to respond to some name and if so
Firefox will use your resolver.


That's probably what I'm remembering with Firefox. But doesn't probing 
the local resolver sort of defeat the point of DoH? That is, I really 
don't want my ISP to be able to snoop on my DNS history. Sending it off 
to one of the well known resolvers at least gives me a chance to know 
whether they are evil or not because there aren't very many of them vs 
every random ISP out there. Since nobody but people like us know about 
those resolvers it seems to me that without preconfiguration meaningful 
DoH is pretty limited?


Or maybe I just don't understand what problem they were trying to solve?

Mike

Re: OSP Management

[michael brooks - ESC]

We used to have an FTE for ArcGIS. We got by pretty well until we needed to
document circuits down to the NIC level, and then we lost that FTE
altogether. PatchManager was chosen from an RFP for its granularity and
(seeming) user-friendliness.




michael brooks
Sr. Network Engineer
Adams 12 Five Star Schools
michael.bro...@adams12.org

"flying is learning how to throw yourself at the ground and miss"



On Tue, Oct 31, 2023 at 4:29 PM Eric Kuhnke  wrote:

> On that topic, I find it interesting to see how different medium/regional
> scale ISPs have developed their own in-house GIS systems, once they reach
> the size and scale where one FTE staff position to run GIS systems/database
> backend is a necessity.
>
> There is a great deal that can be done with QGIS and entirely GPL/BSD
> licensed software, if your GIS person has a background in this sort of
> thing.
>
> Privately hosting a intranet-based tile-server for openstreetmap data and
> overlaying your own network on top of it is not extremely difficult.
>
>
>
> On Tue, Oct 31, 2023 at 6:27 AM michael brooks - ESC <
> michael.bro...@adams12.org> wrote:
>
>> On that note, what do you all use for managing OSP? We have been
>> attempting to stand up PatchManager for quite some time, and find it a good
>> product, but the billions of options can be overwhelming
>>
>>
>>
>>
>> michael brooks
>> Sr. Network Engineer
>> Adams 12 Five Star Schools
>> michael.bro...@adams12.org
>> 
>> "flying is learning how to throw yourself at the ground and miss"
>>
>>
>>
>> On Fri, Oct 27, 2023 at 5:54 AM Mike Hammett  wrote:
>>
>>>  Always fun managing OSP.
>>>
>>>
>>>
>>> -
>>> Mike Hammett
>>> Intelligent Computing Solutions
>>> http://www.ics-il.com
>>> <https://urldefense.com/v3/__http://www.ics-il.com__;!!IR39LLzvxw!KJhHvAXNK_qlGRpKRzdjwrbUiToB9DdWW4qx_WhglUoygnqlGcLjg10kbR_UYNPg2VzXcAk2sSV0HIqI_JfB$>
>>>
>>> Midwest-IX
>>> http://www.midwest-ix.com
>>> <https://urldefense.com/v3/__http://www.midwest-ix.com__;!!IR39LLzvxw!KJhHvAXNK_qlGRpKRzdjwrbUiToB9DdWW4qx_WhglUoygnqlGcLjg10kbR_UYNPg2VzXcAk2sSV0HDRh8w21$>
>>>
>>>
>> This is a staff email account managed by Adams 12 Five Star Schools.
>> This email and any files transmitted with it are confidential and intended
>> solely for the use of the individual or entity to whom they are addressed.
>> If you have received this email in error please notify the sender.
>
>

-- 
This is a staff email account managed by Adams 12 Five Star Schools.  This 
email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the sender.

Cisco 15454 Multishelf

[Michael Lambert]

Does anyone have any experience with converting a Cisco 15454 multishelf 
configuration back into single shelves? A suggestion from the TAC some years 
ago involved putting a “clean” processor board into each shelf and rebooting it 
into single-shelf mode. The GUI suggests it’s possible to accomplish this by 
pointing and clicking (in the multishelf configuration tab).

Any hints (probably off-list since I doubt that this is a platform still of 
interest to many) would be welcome.

Thanks,

Michael

OSP Management

[michael brooks - ESC]

On that note, what do you all use for managing OSP? We have been attempting
to stand up PatchManager for quite some time, and find it a good product,
but the billions of options can be overwhelming




michael brooks
Sr. Network Engineer
Adams 12 Five Star Schools
michael.bro...@adams12.org

"flying is learning how to throw yourself at the ground and miss"



On Fri, Oct 27, 2023 at 5:54 AM Mike Hammett  wrote:

>  Always fun managing OSP.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
> <https://urldefense.com/v3/__http://www.ics-il.com__;!!IR39LLzvxw!KJhHvAXNK_qlGRpKRzdjwrbUiToB9DdWW4qx_WhglUoygnqlGcLjg10kbR_UYNPg2VzXcAk2sSV0HIqI_JfB$>
>
> Midwest-IX
> http://www.midwest-ix.com
> <https://urldefense.com/v3/__http://www.midwest-ix.com__;!!IR39LLzvxw!KJhHvAXNK_qlGRpKRzdjwrbUiToB9DdWW4qx_WhglUoygnqlGcLjg10kbR_UYNPg2VzXcAk2sSV0HDRh8w21$>
>
>

-- 
This is a staff email account managed by Adams 12 Five Star Schools.  This 
email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the sender.

Re: emily postnews

[Michael Hallgren]

You sure :-)^oo
mh

28 octobre 2023 19:32 "Jay R. Ashworth"  a écrit:

> - Original Message -
> 
>> From: "Randy Bush" 
>> 
>> another old dog doing a search wrote to tell me they really appreciated
>> that i still had some antique advice up. i had long forgotten this one.
>> but found it amusing and still more relevant than i might wish.
>> 
>> https://psg.com/emily.html
> 
> I would bet many dollars green American that the venn diagram of "people who
> need that advice these days" and "people who can tell that it is sarcasm/
> satire" is two disjoint circles...
> 
> Cheers,
> -- jra
> --
> Jay R. Ashworth Baylink j...@baylink.com
> Designer The Things I Think RFC 2100
> Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII
> St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274

Re: [EXTERNAL] Re: Charter DNS servers returning malware filtered IP addresses

[Michael Thomas]

On 10/27/23 2:20 PM, John Levine wrote:

It appears that Bryan Fields  said:

-=-=-=-=-=-
-=-=-=-=-=-
On 10/27/23 7:49 AM, John Levine wrote:

But for obvious good reasons,
the vast majority of their customers don't

I'd argue that as a service provider deliberately messing with DNS is an
obvious bad thing.  They're there to deliver packets.

For a network feeding a data center, sure. For a network like
Charter's which is feeding unsophisticated nontechnical users, they
need all the messing they can get.

If you're one of the small minority of retail users that knows enough
about the technology to pick your own resolver, go ahead.  But it's
a reasonable default to keep malware out of Grandma's iPad.


How does this line up with DoH? Aren't they using hardwired resolver 
addresses? I would hope they are not doing anything heroic.


Mike

Re: transit and peering costs projections

[Michael Thomas]

On 10/15/23 8:33 PM, Matthew Petach wrote:



 I think we often forget just how much of a massive inversion the
communications industry has undergone; back in the 80s, when
I started working in networking, everything was DS0 voice channels,
and data was just a strange side business that nobody in the telcos
really understood or wanted to sell to.  At the time, the volume of money
being raked in from those DS0/VGE channels was mammoth compared
to the data networking side; we weren't even a rounding error.  But as 
the

roles reversed and the pyramid inverted, the data networking costs didn't
rise to meet the voice costs (no matter how hard the telcos tried to push
VGE-mileage-based pricing models!

Haha, when I was at Cisco in the late 90's and was working on VoIP stuff 
we were working with Sprint trying to get them onboard for a residential 
voice project. They were really insistent on using AAL2 to conserve 
bandwidth. I told them at the time that the bandwidth for voice was 
going to be insignificant and it wasn't a big deal that RTP wasn't as 
efficient. They looked at me like i had leprosy with body parts falling 
off. Like the next month it was announced that data had surpassed voice 
for the first time. We didn't get the contract, fwiw. But they never 
launched anything either. Was there ever any significant deployment of 
AAL2?


Mike

Re: xfinity not working

[Michael Thomas]

On 10/11/23 11:34 AM, William Herrin wrote:

On Wed, Oct 11, 2023 at 11:12 AM Delong.com  wrote:

There are still some knobs…

e.g. bridge mode or not (usually)

I'm guessing that's only if there's a built-in wifi router. My grand
experience with cable modems counts to exactly two brands and four
models, but in each case the model without wifi was solely a bridge.
The knobs, as such, were: change the password and reboot the modem.


I thought that bridge was the DOCSIS model. Is there a choice these 
days? Back when I was actually working with this, the expectation is the 
modem was pretty dumb and not accessible to the user. It would tftp its 
config from the MSO and that was that.


Mike

Re: Low to Mid Range DWDM Platforms

[michael brooks - ESC]

>On the same topic, anyone have experience with the stuff from fs.com
<https://urldefense.com/v3/__http://fs.com__;!!IR39LLzvxw!PGJ97ToHx5NZWWby4LKNnMDU_u-xfmUhkusE-0RRoatuF_SK72zGd7KWDJvcRXLZ8JNyKO6wDKCHB9UTDCzWvMwMYSLX$>
?

We use their patch cables, both SM and MM. Solid product, and good to work
with (we have had them build custom-length cables).



michael brooks
Sr. Network Engineer
Adams 12 Five Star Schools
michael.bro...@adams12.org

"flying is learning how to throw yourself at the ground and miss"



On Fri, Oct 6, 2023 at 8:04 AM David Bass  wrote:

> On the same topic, anyone have experience with the stuff from fs.com
> <https://urldefense.com/v3/__http://fs.com__;!!IR39LLzvxw!PGJ97ToHx5NZWWby4LKNnMDU_u-xfmUhkusE-0RRoatuF_SK72zGd7KWDJvcRXLZ8JNyKO6wDKCHB9UTDCzWvMwMYSLX$>
> ?
>
>
>>>
>>>

-- 
This is a staff email account managed by Adams 12 Five Star Schools.  This 
email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the sender.

Re: what is acceptible jitter for voip and videoconferencing?

[Michael Thomas]

On 9/22/23 1:54 PM, Mark Andrews wrote:

Telnet sessions where  often initiated from half duplex terminals.  Pushing 
that flow control across the network helped those users.

I'm still confused. Did it require the telnet users to actually take 
action? Like they'd manually need to enter the GA option? It's very 
possible that I didn't fully implement it if so since I didn't realize that.


Mike

Re: what is acceptible jitter for voip and videoconferencing?

[Michael Thomas]

On 9/22/23 9:42 AM, Jay Hennigan wrote:

On 9/21/23 17:04, Michael Thomas wrote:

When I wrote my first implementation of telnet ages ago, i was both 
amused and annoyed about the go-ahead option. Obviously patterned 
after audio meat-space protocols, but I was never convinced it wasn't 
a solution in search of a problem. I wonder if CDMA was really an 
outgrowth of those protocols?


Typically seen with half-duplex implementations, like "Over" in 
two-way radio. Still used in TTY/TDD as "GA".


DId that ever actually occur over the internet such that telnet would 
need it? Half duplex seems to be pretty clearly an L1/L2 problem. IIRC, 
it was something of a pain to implement.


Mike

Re: what is acceptible jitter for voip and videoconferencing?

[Michael Thomas]

On 9/21/23 3:31 PM, William Herrin wrote:

On Thu, Sep 21, 2023 at 6:28 AM Tom Beecher  wrote:

My understanding has always been that 30ms was set based on human 
perceptibility. 30ms was the average point at which the average person could 
start to detect artifacts in the audio.

Hi Tom,

Jitter doesn't necessarily cause artifacts in the audio. Modern
applications implement what's called a "jitter buffer." As the name
implies, the buffer collects and delays audio for a brief time before
playing it for the user. This allows time for the packets which have
been delayed a little longer (jitter) to catch up with the earlier
ones before they have to be played for the user. Smart implementations
can adjust the size of the jitter buffer to match the observed
variation in delay so that sound quality remains the same regardless
of jitter.

Indeed, on Zoom I barely noticed audio artifacts for a friend who was
experiencing 800ms jitter. Yes, really, 800ms. We had to quit our
gaming session because it caused his character actions to be utterly
spastic, but his audio came through okay.


When I wrote my first implementation of telnet ages ago, i was both 
amused and annoyed about the go-ahead option. Obviously patterned after 
audio meat-space protocols, but I was never convinced it wasn't a 
solution in search of a problem. I wonder if CDMA was really an 
outgrowth of those protocols?


But it's my impression that gaming is by far more affected by latency 
and thus jitter buffers for voice. Don't some ISP's even cater to gamers 
about latency?


Mike

Re: So what do you think about the scuttlebutt of Musk interfering in Ukraine?

[Michael Thomas]

On 9/14/23 6:34 AM, Dave Taht wrote:
This is one of those threads where I do think folk would benefit from 
hearing from the horses' mouths. In a recent bio of musk published 
this past week, the author claimed that starlink withdrew service over 
crimea based on the knowledge it was going to be used for a surprise 
attack. Starlink - and that author - now state that ( 
https://twitter.com/elonmusk/status/1700345943105638636 )


The onus is meaningfully different if I refused to act upon a request 
from Ukraine vs. made a deliberate change to Starlink to thwart 
Ukraine. At no point did I or anyone at SpaceX promise coverage over 
Crimea. Moreover, our terms of service clearly prohibit Starlink for 
offensive military action, as we are a civilian system, so they were 
again asking for something that was expressly prohibited. SpaceX is 
building Starshield for the US government, which is similar to, but 
much smaller than Starlink, as it will not have to handle millions of 
users. That system will be owned and controlled by the US government.

Quote
Walter Isaacson
@WalterIsaacson
·
Sep 8
To clarify on the Starlink issue: the Ukrainians THOUGHT coverage was 
enabled all the way to Crimea, but it was not. They asked Musk to 
enable it for their drone sub attack on the Russian fleet. Musk did 
not enable it, because he thought, probably correctly, that would 
cause a…Show more 



Furthermore, Musk stated yesterday that had the request come from the 
us government, he would have complied.


I will refrain from editorializing.


I guess this is a lesson on diversity which every military should pay 
attention to. I had forgotten about other wireless options that Bill 
pointed out, though I'm not sure if geostationary latency would fit 
their requirements. But is trying to reclaim your territory "offensive" 
after being invaded? How would other providers interpret that? Or maybe 
this is just a unicorn.


Mike

Re: So what do you think about the scuttlebutt of Musk interfering in Ukraine?

[Michael Thomas]

On 9/14/23 9:26 AM, Mike Hammett wrote:

*nods* likely plenty of similar examples by less polarizing people.


Then lets hear them? It certainly seems like an  operational issue if 
this starts to become common. How is it dealt with if at all beyond 
diversity which is hard to come by with LEO systems?



Mike

So what do you think about the scuttlebutt of Musk interfering in Ukraine?

[Michael Thomas]

Doesn't this bump up against common carrier protections? I sure don't 
want my utilities weaponizing their monopoly status to the whims of any 
random narcissist billionaire.


Mike

Re: Destination Preference Attribute for BGP

[michael brooks - ESC]

>With AS-PATH prepend you have no control on the choice of which ASN should
do what action on your advertisements.
Robert- It is somewhat this problem we are trying to resolve.

>I was imagining something sexier, especially given how pretty "useless"
AS_PATH prepending is nowadays.
I, too, am looking for something sexy (explained below). But can you
explain why you think AS_PATH is "useless," Mark?

For background, and the reason I asked about DPA:
Currently, our routing carries user traffic to a single data center where
it egresses to the Internet via three ISP circuits, two carriers. We are
peering on a single switch stack, so we let L2 "load balance" user flows
for us. We have now brought up another ISP circuit in a second data center,
and are attempting to influence traffic to return the same path as it
egressed our network. Simply, we now have two datacenters which user
traffic can egress, and if one is used we want that traffic to return to
the same data center. It is a problem of asymmetry. It appears the only
tools we have are AS_Path and MED, and so I have been searching for another
solution, that is when I came across DPA. In further looking at the
problem, BGP Communities also seems to be a possible solution, but as the
thread has explored, communities may/may not be scrubbed upstream. So,
presently we are looking for a solution which can be used with our direct
peers. Obviously, if someone has a better solution, I am all ears.

A bit more info: we are also looking at an internal solution which passes
IGP metric into MED to influence pathing.

To avoid TL;DR I will stop there in the hopes this is an intriguing enough
problem to generate discussion.




michael brooks
Sr. Network Engineer
Adams 12 Five Star Schools
michael.bro...@adams12.org

"flying is learning how to throw yourself at the ground and miss"



On Fri, Aug 18, 2023 at 1:39 AM Robert Raszuk  wrote:

> Jakob,
>
> With AS-PATH prepend you have no control on the choice of which ASN should
> do what action on your advertisements.
>
> However, the practice of publishing communities by (some) ASNs along with
> their remote actions could be treated as an alternative to the DPA
> attribute. It could result in remote PREPEND action too.
>
> If only those communities would not be deleted by some transit networks
> 
>
> Thx,
> R.
>
> On Thu, Aug 17, 2023 at 9:46 PM Jakob Heitz (jheitz) via NANOG <
> nanog@nanog.org> wrote:
>
>> "prepend as-path" has taken its place.
>>
>>
>>
>> Kind Regards,
>>
>> Jakob
>>
>>
>>
>>
>>
>> Date: Wed, 16 Aug 2023 21:42:22 +0200
>> From: Mark Tinka 
>>
>> On 8/16/23 16:16, michael brooks - ESC wrote:
>>
>> > Perhaps (probably) naively, it seems to me that DPA would have been a
>> > useful BGP attribute. Can anyone shed light on why this RFC never
>> > moved beyond draft status? I cannot find much information on this
>> > other than IETF's data tracker
>> > (https://datatracker.ietf.org/doc/draft-ietf-idr-bgp-dpa/
>> <https://urldefense.com/v3/__https://datatracker.ietf.org/doc/draft-ietf-idr-bgp-dpa/__;!!IR39LLzvxw!LfO7m5JiQpZx6Lp-F2LQMBHi-YefsPMl8GdYkbrFSGWXd0G3HHj6wxEJv7_K0Y-_pIAyvahmJPXvcHAc251UDA$>)
>> and RFC6938
>> > (which implies DPA was in use,?but then was deprecated).
>>
>> I've never heard of this draft until now, but reading it, I can see why
>> it would likely not be adopted today (not sure what the consensus would
>> have been back in the '90's).
>>
>> DPA looks like MED on drugs.
>>
>> Not sure operators want remote downstream ISP's arbitrarily choosing
>> which of their peering interconnects (and backbone links) carry traffic
>> from source to them. BGP is a poor communicator of bandwidth and
>> shilling cost, in general. Those kinds of decisions tend to be locally
>> made, and permitting outside influence could be a rather hard sell.
>>
>> It reminds me of how router vendors implemented GMPLS in the hopes that
>> optical operators would allow their customers to build and control
>> circuits in the optical domain in some fantastic fashion.
>>
>> Or how router vendors built Sync-E and PTP into their routers hoping
>> that they could sell timing as a service to mobile network operators as
>> part of a RAN backhaul service.
>>
>> Some things just tend to be sacred.
>>
>> Mark.
>>
>>

-- 
This is a staff email account managed by Adams 12 Five Star Schools.  This 
email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the sender.

Re: Hawaiian ILEC infrastructure and fire

[Michael Thomas]

On 8/17/23 11:26 AM, scott via NANOG wrote:



I don't want to overwhelm the list, but since there's interest here's 
something interesting I just now got from the electric company.  400 
poles and 300 transformers.  Wow!


Those of us from California and the west have watched this in abject 
horror and I myself was completely clueless this was possible.


Mike, who lives 10 miles from where the Caldor fire started that burned 
all the way to Tahoe and grew up going to Paradise to visit my grandparents

Destination Preference Attribute for BGP

[michael brooks - ESC]

Perhaps (probably) naively, it seems to me that DPA would have been a
useful BGP attribute. Can anyone shed light on why this RFC never moved
beyond draft status? I cannot find much information on this other than
IETF's data tracker (
https://datatracker.ietf.org/doc/draft-ietf-idr-bgp-dpa/) and RFC6938
(which implies DPA was in use, but then was deprecated).




michael brooks
Sr. Network Engineer
Adams 12 Five Star Schools
720.972.4110
michael.bro...@adams12.org

"flying is learning how to throw yourself at the ground and miss"

-- 
This is a staff email account managed by Adams 12 Five Star Schools.  This 
email and any files transmitted with it are confidential and intended 
solely for the use of the individual or entity to whom they are addressed. 
If you have received this email in error please notify the sender.

Re: Copper wire thefts increase 139% in one California county

[Michael Thomas]

On 7/1/23 9:46 AM, Sean Donelan wrote:


Copper wire thefts of all kinds appear to be increasing in 2023. Not 
just telecommunications copper cables, but also electric and transit 
cables.


San Joaquin County reported a 139% increase in copper wire thefts over 
four months, and one theft in the county left the 911 center unable to 
receive calls.


https://www.latimes.com/california/story/2023-07-01/copper-wire-thefts-cause-delays-for-metro-railways 




It's been happening here in Amador County as well mostly from yahoos 
from Stockton. They're taking out fiber too probably inadvertently.


Mike

Re: Northern Virginia has had enough with data centers

[Michael Thomas]

On 6/26/23 6:06 PM, Ron Yokubaitis wrote:

Dalles: government subsidized Hydroelectric Power, that’s why.


Well that maybe, but electric rates are hella cheap in Oregon regardless.

Mike




Sent from the iPad of Ron Yokubaitis


On Jun 26, 2023, at 7:37 PM, Michael Thomas  wrote:



On 6/24/23 5:28 AM, Owen DeLong wrote:


On Jun 23, 2023, at 18:04, Michael Thomas  wrote:



On 6/23/23 4:01 PM, 
https://urldefense.proofpoint.com/v2/url?u=http-3A__Delong.com=DwIDaQ=euGZstcaTDllvimEN8b7jXrwqOf-v5A_CdpgnVfiiMM=cGrDT0liF-gD_o4EJ7o_qg=Z1ElQJ6RtDdrUgH7UwCpBHojWq1Iyp4CM49TsykDfXM=BUfOgzy41EDHyDbK_xBslELDt9Xofk6_YBR4nLTQGmo=
 via NANOG wrote:
The electric grid complaints are about the demand on the grid making the entire 
region less stable and proposed construction of new high-voltage tower 
corridors for data centers.
Yeah, I can kind of understand those, but as long as the grid is properly 
planned, it really shouldn’t have a destabilizing effect. In fact, many 
datacenter in California do CoGen and end up providing additional grid 
stability.


Uh, ::cough:: PGE ::cough::

I so wanted to schadenfreude so bad with Texas and their shitty grid, but then 
remembered where I live.

Mike

What’s not to love about a power company that literally qualifies as a 
recidivist felon?

It is my sincere hope to finish my mortgage and then start putting money 
towards electrical independence. (Wind, more solar, batteries, and 
disconnecting Persistent Graft and Extortion).


I'm waiting on a software upgrade for my inverter to hook up a generator to 
refill the battery when it gets too low. Not off the grid, but not at the mercy 
of PGE's fuckery.

How many datacenters are in Norcal? I imagine that it's a lot, but PGE's rates 
are like 2x the rest of the country. At least for residential. I always got a 
kick of Google putting a datacenter in the Dalles in Oregon -- basically 
mainlining the Columbia river.

Mike

Re: Northern Virginia has had enough with data centers

[Michael Thomas]

On 6/24/23 5:28 AM, Owen DeLong wrote:



On Jun 23, 2023, at 18:04, Michael Thomas  wrote:



On 6/23/23 4:01 PM, Delong.com via NANOG wrote:
The electric grid complaints are about the demand on the grid making the entire 
region less stable and proposed construction of new high-voltage tower 
corridors for data centers.
Yeah, I can kind of understand those, but as long as the grid is properly 
planned, it really shouldn’t have a destabilizing effect. In fact, many 
datacenter in California do CoGen and end up providing additional grid 
stability.


Uh, ::cough:: PGE ::cough::

I so wanted to schadenfreude so bad with Texas and their shitty grid, but then 
remembered where I live.

Mike

What’s not to love about a power company that literally qualifies as a 
recidivist felon?

It is my sincere hope to finish my mortgage and then start putting money 
towards electrical independence. (Wind, more solar, batteries, and 
disconnecting Persistent Graft and Extortion).

I'm waiting on a software upgrade for my inverter to hook up a generator 
to refill the battery when it gets too low. Not off the grid, but not at 
the mercy of PGE's fuckery.


How many datacenters are in Norcal? I imagine that it's a lot, but PGE's 
rates are like 2x the rest of the country. At least for residential. I 
always got a kick of Google putting a datacenter in the Dalles in Oregon 
-- basically mainlining the Columbia river.


Mike

Re: Northern Virginia has had enough with data centers

[Michael Thomas]

On 6/23/23 4:01 PM, Delong.com via NANOG wrote:
The electric grid complaints are about the demand on the grid making 
the entire region less stable and proposed construction of new 
high-voltage tower corridors for data centers.

Yeah, I can kind of understand those, but as long as the grid is properly 
planned, it really shouldn’t have a destabilizing effect. In fact, many 
datacenter in California do CoGen and end up providing additional grid 
stability.


Uh, ::cough:: PGE ::cough::

I so wanted to schadenfreude so bad with Texas and their shitty grid, 
but then remembered where I live.


Mike

Re: FCC Chair Rosenworcel Proposes to Investigate Impact of Data Caps

[Michael Thomas]

On 6/17/23 4:14 PM, Tom Beecher wrote:


Also: they plan to use Starship when it's available which has 10x
more capacity. If it really is fully reusable as advertised, that
is going to really drive down the launch cost.


Starship is years away from being flight ready. The most recent test 
launch from Texas was not a 'successful failure' as widely portrayed 
in the media.  Reputable people who have been working in this field 
for decades have pointed out tons of massive problems that are not 
quick fixes.


Betting that Starship won't be a factor is not a bet I'd make. And I'm 
definitely not a Musk fan boy. A lot of the same NASA types didn't 
believe they'd get where they are now either. Because... you know, 
vested interests. I'd bet that Starship will be a factor way before the 
Senate Launch System.


Mike

Re: FCC Chair Rosenworcel Proposes to Investigate Impact of Data Caps

[Michael Thomas]

Whether or not it makes business sense isn't really what I was talking 
about. I was talking about the home dish costing $1k. That sounds like 
it could easily be reduced significantly unless there is some underlying 
tech reason.


Also: they plan to use Starship when it's available which has 10x more 
capacity. If it really is fully reusable as advertised, that is going to 
really drive down the launch cost.


But your calculations don't take into account that they are not at 
anywhere close to a full constellation: they are only at 4k out of the 
40k they need so they literally can't support higher numbers. Their new 
generation of satellite is also suppose to be doing some in-orbit 
routing or something like that which would I would assume will really 
help on the bandwidth front. How much that affects their maximum 
subscriber base when they are fully deployed I don't know but it's bound 
to be a lot more possible subs than they have now.


I mean, this could be a spectacular flop like Iridium but a lot has 
changed in 20 some years not least of which is the cost of launch.


Mike

On 6/17/23 2:53 PM, Tom Beecher wrote:


As I mentioned elsewhere, I'm not sure that the current economics
are the real economics. I'm pretty sure they've been purposefully
throttling demand because they still don't have the capacity so it
would make sense to overcharge in the mean time. Is there
something inherent in their cpe that makes them much more
expensive than, say, satellite tv dishes? I can see marginally
more because of the LEO aspect, but isn't that mainly just
software? It wouldn't surprise me that the main cost is the truck
roll.

- Starlink currently reports around 1.5M subscribers. At $110 a month, 
that's $165M in revenue,


- A Falcon 9 launch is billed out at $67M. A Falcon 9 can carry up to 
60 Starlink sats. That's ~667 launches to reach the stated goal of 40k 
sats in the constellation. So roughly $45B in just launch costs, if 
you assume the public launch price. (Because if they are launching 
their own stuff, they aren't launching an external paying customer.)

- The reported price per sat is $250k.

Assuming they give themselves a friendly internal discount, the 
orbital buildout cost are in the neighborhood of $30B for launches, 
and $10B for sats.


- The satellite failure rate is stated to be ~ 3% annually. On a 40K 
cluster, that's 1200 a year.


That's about 20 more launches a year, and $300M for replacement sats. 
Let's round off and say that's $1B a year there.


 So far, that's a $40B buildout with a $1B annual run rate. And that's 
just the orbital costs. We haven't even calculated the 
manufacturing costs of the receiver dishes, terrestrial network infra 
cost , opex from staff , R, etc .


Numbers kinda speak for themselves here.

I mean, I get that Musk is sort of a cuckoo bird but say what you
will he does have big ambitions.


Ambition is good. But reality tends to win the day. As does math.




On Sat, Jun 17, 2023 at 4:38 PM Michael Thomas  wrote:


On 6/17/23 1:25 PM, Tom Beecher wrote:


Won't Starlink and other LEO configurations be that backstop
sooner
rather than later?


Unlikely. They will remain niche. The economics don't make sense
for those services to completely replace terrestrial only service.


Why would they put up 4 satellites if their ambition is only
niche? I mean, I get that Musk is sort of a cuckoo bird but say
what you will he does have big ambitions.

From my standpoint, they don't have to completely replace the
incumbents. I'd be perfectly happy just keeping them honest.

As I mentioned elsewhere, I'm not sure that the current economics
are the real economics. I'm pretty sure they've been purposefully
throttling demand because they still don't have the capacity so it
would make sense to overcharge in the mean time. Is there
something inherent in their cpe that makes them much more
expensive than, say, satellite tv dishes? I can see marginally
more because of the LEO aspect, but isn't that mainly just
software? It wouldn't surprise me that the main cost is the truck
roll.

Mike




On Fri, Jun 16, 2023 at 4:17 PM Michael Thomas  wrote:


On 6/16/23 1:09 PM, Mark Tinka wrote:
>
>
> On 6/16/23 21:19, Josh Luthman wrote:
>> Mark,
>>
>> In my world I constantly see people with 0 fixed internet
options.
>> Many of these locations do not even have mobile coverage.
>> Competition is fine in town, but for millions of people in
the US
>> (and I'm going to assume it's worse or comparable in
CA/MX) there is
>> no service.
>>
>> As a company primarily delivering to residents,
competition is not a
>> focus fo

Re: FCC Chair Rosenworcel Proposes to Investigate Impact of Data Caps

[Michael Thomas]

On 6/17/23 1:25 PM, Tom Beecher wrote:


Won't Starlink and other LEO configurations be that backstop sooner
rather than later?


Unlikely. They will remain niche. The economics don't make sense for 
those services to completely replace terrestrial only service.


Why would they put up 4 satellites if their ambition is only niche? 
I mean, I get that Musk is sort of a cuckoo bird but say what you will 
he does have big ambitions.


From my standpoint, they don't have to completely replace the 
incumbents. I'd be perfectly happy just keeping them honest.


As I mentioned elsewhere, I'm not sure that the current economics are 
the real economics. I'm pretty sure they've been purposefully throttling 
demand because they still don't have the capacity so it would make sense 
to overcharge in the mean time. Is there something inherent in their cpe 
that makes them much more expensive than, say, satellite tv dishes? I 
can see marginally more because of the LEO aspect, but isn't that mainly 
just software? It wouldn't surprise me that the main cost is the truck 
roll.


Mike




On Fri, Jun 16, 2023 at 4:17 PM Michael Thomas  wrote:


On 6/16/23 1:09 PM, Mark Tinka wrote:
>
>
> On 6/16/23 21:19, Josh Luthman wrote:
>> Mark,
>>
>> In my world I constantly see people with 0 fixed internet options.
>> Many of these locations do not even have mobile coverage.
>> Competition is fine in town, but for millions of people in the US
>> (and I'm going to assume it's worse or comparable in CA/MX)
there is
>> no service.
>>
>> As a company primarily delivering to residents, competition is
not a
>> focus for us and for the urban market it's tough to survive on
a ~1/3
>> take rate.
>
> I should have been clearer... the lack of competition in many
markets
> is not unique to North America. I'd say all of the world suffers
that,
> since there is only so much money and resources to go around.
>
> What I was trying to say is that should a town or village have the
> opportunity to receive competition, where existing services are
> capped, uncapping that via an alternative provider would be low
> hanging fruit to gain local marketshare. Of course, the alternative
> provider would need to show up first, but that's a whole other
thread.
>
Won't Starlink and other LEO configurations be that backstop sooner
rather than later? I don't know if they have caps as well, but
even if
they do they could compete with their caps.

Mike

Re: FCC Chair Rosenworcel Proposes to Investigate Impact of Data Caps

[Michael Thomas]

On 6/16/23 3:18 PM, Keith Stokes wrote:

Cox also has a 1.2 TB cap.

If I can believe my graphs, the metered Cox connection (video 
streaming primarily for wife) is about 90 GB the month of April and 
the unmetered ATT fiber WFH for me is about 370 GB. Total LAN is about 
450 GB. Napkin math but it's pretty close.


Modulo P2P nodes, what drives high usage? I guess game downloads are 
getting gigantic these days, but that's not an every day event. Back in 
the bad old days it wasn't inconceivable to go over the lower caps but 
once it's big enough to support video what is left? I mean, how much 4k 
pr0n can randy teenagers watch in one month?



Mike

Re: FCC Chair Rosenworcel Proposes to Investigate Impact of Data Caps

[Michael Thomas]

On 6/16/23 1:36 PM, Josh Luthman wrote:
Not everyone can afford $1000 to start up Starlink and then pay $130+ 
per month.  That may be an option for some, but certainly not the 
majority.


If 100% of a town was covered by a single company with data caps, 
those that are crying from hitting 1.2 TB/month will not be enough for 
a competitor to come in and build on top.  A TB/mo now is extremely 
high - In May 2023 we had 4 customers that exceeded that (all 4 of 
these customers mentioned are subscribed to <25 mbps plans; we offer 
gig ftth).


I get the impression that they are still in a beta/early adopter 
situation so unaffordability might be feature not a bug to them to keep 
the system from a success disaster. At least for now. I get the 
impression that some/a lot of this is to bring the internet to the rest 
of the world as one of their goals.


I do wonder how they are numbering them though. Are the they using the 
same scheme that the mobile providers are using with ipv6? hmm.


Mike



On Fri, Jun 16, 2023 at 4:22 PM Mark Tinka  wrote:



On 6/16/23 22:16, Michael Thomas wrote:

> Won't Starlink and other LEO configurations be that backstop sooner
> rather than later? I don't know if they have caps as well, but
even if
> they do they could compete with their caps.

Maybe. I really haven't paid any attention to Starlink, although
there
are credible reports of folk testing it here in South Africa's urban
centres.

I have not heard of any mention of Starlink having caps as part of
their
service. Having said that, for services like this, things change
as the
number of customers using them rises.

Mark.

Re: FCC Chair Rosenworcel Proposes to Investigate Impact of Data Caps

[Michael Thomas]

On 6/16/23 1:22 PM, Mark Tinka wrote:



On 6/16/23 22:16, Michael Thomas wrote:

Won't Starlink and other LEO configurations be that backstop sooner 
rather than later? I don't know if they have caps as well, but even 
if they do they could compete with their caps.


Maybe. I really haven't paid any attention to Starlink, although there 
are credible reports of folk testing it here in South Africa's urban 
centres.


I have not heard of any mention of Starlink having caps as part of 
their service. Having said that, for services like this, things change 
as the number of customers using them rises.


I've toyed with getting it which I think I can do here in rural 
California especially given that my ISP installed fiber and inexplicably 
didn't change the rate plans from DSL (given that it's an older 
population here, I doubt that any new over subscription would be much 
problem). But the fact of the matter is that we don't seem to ever be in 
the situation that our 25Mbs service is an actual problem.


Mike

Re: FCC Chair Rosenworcel Proposes to Investigate Impact of Data Caps

[Michael Thomas]

On 6/16/23 1:09 PM, Mark Tinka wrote:



On 6/16/23 21:19, Josh Luthman wrote:

Mark,

In my world I constantly see people with 0 fixed internet options.  
Many of these locations do not even have mobile coverage.  
Competition is fine in town, but for millions of people in the US 
(and I'm going to assume it's worse or comparable in CA/MX) there is 
no service.


As a company primarily delivering to residents, competition is not a 
focus for us and for the urban market it's tough to survive on a ~1/3 
take rate.


I should have been clearer... the lack of competition in many markets 
is not unique to North America. I'd say all of the world suffers that, 
since there is only so much money and resources to go around.


What I was trying to say is that should a town or village have the 
opportunity to receive competition, where existing services are 
capped, uncapping that via an alternative provider would be low 
hanging fruit to gain local marketshare. Of course, the alternative 
provider would need to show up first, but that's a whole other thread.


Won't Starlink and other LEO configurations be that backstop sooner 
rather than later? I don't know if they have caps as well, but even if 
they do they could compete with their caps.


Mike

Re: FCC Chair Rosenworcel Proposes to Investigate Impact of Data Caps

[Michael Thomas]

On 6/15/23 10:41 PM, Crist Clark wrote:
Comcast still has data caps. My service is 1.2 TB per month. If we get 
close, we get a warning email. If we were to go over (hasn’t happened 
yet), we get billed per additional 500 MB.


However, I just looked at my account usage for the first time for a 
few months, and somehow have had zero usage since March of this year.


Is 1.2 TB enough for a typical cord cutter? I just looked at mine and it 
looks to be about 300GB/month, but we may not be typical for your 
average family with kids, say.


Mike

Re: FCC Chair Rosenworcel Proposes to Investigate Impact of Data Caps

[Michael Thomas]

On 6/15/23 3:19 PM, Sean Donelan wrote:


While a lot of ISPs gave up on data caps, the language is still 
lurking in many Terms Of Service.




https://www.fcc.gov/document/chair-rosenworcel-proposes-investigate-impact-data-caps 



proposed Notice of Inquiry to learn more about how broadband providers 
use data caps on consumer plans. Data caps, or usage limits, are a 
common practice where an internet service provider (ISP) restricts how 
much bandwidth or data a consumer uses, though many broadband ISPs 
temporarily or permanently refrained from enforcing or imposing data 
caps in response to the COVID-19 pandemic. In particular, the agency 
would like to better understand the current state of data caps, their 
impact on consumers, and whether the Commission should consider taking 
action to ensure that data caps do not cause harm to competition or 
consumers’ ability to access

broadband Internet services.


So why did they back off? Cost too much in support calls with pissed 
people? Bad publicity? People can't meaningfully use the offered 
bandwidth these days? Something else?


Mike

Re: 365 Datacenters Tampa AC Failure

[Michael Spears via NANOG]

Issue started a little after 2am, I was hard down till about 11:30am (servers did a high temp shutdown)On Jun 12, 2023 8:01 PM, Michael Spears  wrote:Yep there's issues over there. They had some compressors go down. Should be getting back to normal now... Hasn't been a good month for them in regards to cooling..On Jun 12, 2023 7:15 PM, George Herbert  wrote:Oof.  Get ready to replace all spinning media you may have there.  

-George 

Sent from my iPhone

> On Jun 12, 2023, at 4:06 PM, Nick Olsen  wrote:
> 
> 
> Just a heads up to anyone else colo'd at 365 TPA1/TAMSFLDE. Currently seeing floor temps of ~105F as reported by equipment. Started yesterday at ~5:30PM eastern. 2nd AC failure in the last 30 days. They have not sent any advisory notices as of yet.

Re: 365 Datacenters Tampa AC Failure

[Michael Spears via NANOG]

Yep there's issues over there. They had some compressors go down. Should be getting back to normal now... Hasn't been a good month for them in regards to cooling..On Jun 12, 2023 7:15 PM, George Herbert  wrote:Oof.  Get ready to replace all spinning media you may have there.  

-George 

Sent from my iPhone

> On Jun 12, 2023, at 4:06 PM, Nick Olsen  wrote:
> 
> 
> Just a heads up to anyone else colo'd at 365 TPA1/TAMSFLDE. Currently seeing floor temps of ~105F as reported by equipment. Started yesterday at ~5:30PM eastern. 2nd AC failure in the last 30 days. They have not sent any advisory notices as of yet.

Re: New addresses for b.root-servers.net

[Michael Butler via NANOG]

On 6/7/23 15:13, Izaac wrote:

On Wed, Jun 07, 2023 at 09:30:36AM -0700, William Herrin wrote:

Data embedded in the binary is hard-coded. That's what hard-coded
means. If it makes you happier I'll qualify it as a "hard-coded
default," to differentiate it from settings the operator can't
override with configuration.


No.  I will not indulge your invention of terms.  "Hard-coded" means you
need to recompile to change it.  This is a default value.  A
configuration option takes precedence.


BIND-9.18.14 requires recompilation to update the embedded defaults ..

bin/named/config.c: 2001:500:200::b;# b.root-servers.net\n\
bin/named/config.c: 199.9.14.201;   # b.root-servers.net\n\
lib/dns/rootns.c:   "B.ROOT-SERVERS.NET. 360 IN  A 
199.9.14.201\n"
lib/dns/rootns.c:   "B.ROOT-SERVERS.NET. 360 IN   
2001:500:200::b\n"




It's an instance of https://cwe.mitre.org/data/definitions/344.html


No, it is not in any respect.  The code you grepped out generates a
default configuration hints file when one does not exist.

The CWE you cite specifically refers to default values for things like
cryptographic RNG seeds and salts and TCP sequence number generators and
the like.  Viz something like
https://www.debian.org/security/2008/dsa-1571 from 2008.


A quick search of https://cve.mitre.org/cve/search_cve_list.html shows
between 600 and 3700 CVEs related to default configurations that are
either directly insecure or unexpectedly become insecure when some but
not all of the defaults are changed by the operator. The vast majority
of these CVEs exhibit, as you say, no flaw in the computational logic.


You literally just gave me a link to the CVE search page, waved your
hand, and said, "See?"  Well, I'll admit to not being as good at
conducting CVE research as you.  So, as an expert on the topic: How many
of these "between 600 and 3700 CVEs" are related to a violating the
baseless expectation of confidentially in a protocol which does not
guarantee confidentiality?  Somewhere between 0 and 2000?

But you know what, go ahead.  Submit the CVE.  Be the hero that you
believe yourself to be.

Re: Are we back to the 2000's again?

[Michael Thomas]

On 6/3/23 4:24 PM, William Herrin wrote:

On Sat, Jun 3, 2023 at 4:09 PM Michael Thomas  wrote:

How can the RIAA even know? I mean, are they putting up honey pots or
something?

IIRC, they went after folks sharing the files via bit torrent rather
than folks who only downloaded them.

Oh yeah. This oh-so-two decades ago. I can't believe this sort of thing 
is still going on. What a pointless waste of money.


Mike

Re: Are we back to the 2000's again?

[Michael Thomas]

On 6/3/23 4:01 PM, William Herrin wrote:

On Sat, Jun 3, 2023 at 2:51 PM Mel Beckman  wrote:

It’s like blaming water companies for people stealing boats :)

It's been a while and the article is light on the facts of the case,
but IIRC what happened was: RIAA made some DMCA complaints to Cox. Cox
decided that since they were the network rather than the host, they
couldn't "remove" the offending material, wouldn't cut the user's
access and, because they have such respect for customer privacy,
wouldn't tell the RIAA who the customers were unless RIAA subpoenaed
them under a John Doe suit.

The court decided that Cox's behavior was sufficient to waive the
DMCA's liability shield for Internet providers and off they went to
trial.


How can the RIAA even know? I mean, are they putting up honey pots or 
something?


Mike

Are we back to the 2000's again?

[Michael Thomas]

Apparently the RIAA is back suing ISP's (Cox in this case) for users 
pirating music. It was pretty bogus back then, but with the uptake of 
TLS for almost everything and DoH to conceal DNS requests what exactly 
is an ISP supposed to do these days? Throw in a VPN and the pirates 
completely cut out the ISP.


Am I missing something?


https://yro.slashdot.org/story/23/06/02/2043209/music-pirates-are-not-terrorists-record-labels-argue-in-court

Mike

Re: Do ISP's collect and analyze traffic of users?

[Michael Thomas]

On 5/19/23 6:09 AM, Justin Streiner wrote:

Hank:

No doubt there is a massive amount of information that can be gathered 
from in-box telemetry.  This thread appears to be more focused on 
providers gathering data from traffic in flight across their 
infrastructure.


Yeah, my curiosity was whether ISP were trying to get in the monetizing 
traffic analysis biz which seems to be a small degree but they can't 
really compete with the much finer grained information that other means 
can provide and that they have no particular expertise in it or an 
institutional desire. For things like Google and Facebook, that kind of 
analysis was part of their initial business plan.


Mike



Thank you
jms

On Fri, May 19, 2023 at 8:49 AM Hank Nussbacher  
wrote:


On 19/05/2023 15:27, Justin Streiner wrote:

It amazes me how people can focus on Netflow metadata and ignore
things
like Microsoft telemetry data from every Windows box, or ignore the
massive amount of html cookies that are traded by companies or how
almost every corporate firewall or anti-spam box "reports" back to
the
mother ship and sends tons of information via secret channels like
hashed DNS lookups just to be avoided.

Regards,
Hank

> There are already so many different ways that organizations can
find
> out all sorts of information about individual users, as others have
> noted (social media interactions, mobile location/GPS data,
call/text
> history, interactions with specific sites, etc), that there
probably
> isn't much incentive for many providers to harvest data beyond
what is
> needed for troubleshooting and capacity planning.  Plus, gathering
> more data - potentially down to the level packet payload - is
not an
> easy problem to solve (read: expensive) and doesn't scale well
at all.
> 100G links are very common today, and 400G is becoming so.  I doubt
> that many infrastructure providers would be able to justify the
major
> investments in extra infrastructure to support this, for a revenue
> stream that likely wouldn't match that investment, which would make
> such an investment a loss-leader.
>
> Content providers - particularly social media platforms - have a
> somewhat different business model, but those providers already have
> many different ways to harvest and sell large troves of user data.
>
> Thank you
> jms

Re: Do ISP's collect and analyze traffic of users?

[Michael Thomas]

On 5/16/23 7:55 AM, Saku Ytti wrote:

Of course there are other monetisation opportunities via other
mechanism than data-in-the-wire, like DNS


And with DoH, that doesn't sound like a very long term opportunity.

Mike

Re: Do ISP's collect and analyze traffic of users?

[Michael Thomas]

On 5/16/23 7:35 AM, Livingood, Jason via NANOG wrote:


+1 to what Josh writes below. I would also differentiate between 
mobile networks (service provisioned to individual devices & often 
carrier s/w on the device) and wireline networks (home devices behind 
a router/gateway/NAT).


I just don't think sale of data is a business for wireline ISPs. If it 
were - given most companies are public - you'd see it in SEC 10K 
filings and on earnings calls. Indeed, they'd be required to talk 
about it with investors if it was a material revenue stream. I see 
none of that. Rather, the focus is on subscription revenue. If you 
want to know about data monetization - focus on services you don't pay 
for...




Why would there be a difference between wireless and wired?

Mike

Re: Do ISP's collect and analyze traffic of users?

[Michael Thomas]

On 5/15/23 9:46 PM, Matthew Petach wrote:



On Mon, May 15, 2023 at 6:42 PM Dave Phelps  wrote:

I think it's safe to assume they are selling such data.


https://www.techdirt.com/2021/08/25/isps-give-netflow-data-to-third-parties-who-sell-it-without-user-awareness-consent/


https://www.vice.com/en/article/dy3z9a/fbi-bought-netflow-data-team-cymru-contract


From the second article:

"Team Cymru’s products can also include data such as URLs visited, 
cookies, and PCAP data"


Really?  From Netflow?

I admit, I'm perhaps a little behind on the latest netflow whiz-bangs,
but I've never seen a netflow record type that included HTTP cookies
or PCAP data before.

Certainly, the products listed on the Team Cymru website don't make 
any mention
of including cookies or PCAP data, at least not from what I've been 
able to

ascertain from digging through their product listing.

Is there some secret "off the menu" product that allows one to purchase a
data feed that includes cookies and PCAP data?

Given the pervasiveness of TLS these days, even if they could get it off 
the remaining unencrypted data I'm not sure it would have a lot of value.


Mike

Re: Do ISP's collect and analyze traffic of users?

[michael brooks - ESC]

First NANOG post, the topic compels me to chime in.

For me, the question also implies that user-side we are attempting to scrub
any of the data we volunteer on social media (or other) platforms. I am
careful about what I volunteer up to the Internetz, and have been since my
first AOL floppy experience So, the question of do the ISPs collect
data is particularly important because regardless of how careful I am to
anonymize my own contribution to my "online profile," Tom's assessment is
the bleakest possible picture for anyone attempting to limit the data set
which represents us.




michael brooks
Sr. Network Engineer
Adams 12 Five Star Schools

"flying is learning how to throw yourself at the ground and miss"



On Tue, May 16, 2023 at 7:42 AM Josh Luthman 
wrote:

> Our ISP does not collect (nor obviously sell) customer
> information/traffic.  People volunteer all of their information on
> Facebook/Twitter/etc already, I'm not sure I see a concern.
>
> On Tue, May 16, 2023 at 9:07 AM Tom Beecher  wrote:
>
>> I did see an article about Team Cymru selling netflow data from ISPs to
>>> governments though.
>>>
>>
>> Team Cymru sold the same thing to the FBI Cyber Crimes division that any
>> of us could purchase if we wanted to pay for it.
>>
>> On Tue, May 16, 2023 at 8:52 AM Rishi Panthee 
>> wrote:
>>
>>> I’ve got Akvorado and netflow to identify where traffic comes in/goes to
>>> so we can improve our peering and make less traffic go via transit. I did
>>> see an article about Team Cymru selling netflow data from ISPs to
>>> governments though.
>>> https://www.vice.com/en/article/dy3z9a/fbi-bought-netflow-data-team-cymru-contract
>>> <https://urldefense.com/v3/__https://www.vice.com/en/article/dy3z9a/fbi-bought-netflow-data-team-cymru-contract__;!!IR39LLzvxw!ONxcdPXwj8lyFeci0a3JR8IBTqjjqtzZg8vjTO6rYamj4BvqNnTOOTtr4Nebr851S2GGVT0acYBtKMlEhaVq3egEJzNY$>
>>>
>>>
>>> Rishi Panthee
>>> Ryamer LLC
>>> Https://ryamer.com
>>> <https://urldefense.com/v3/__Https://ryamer.com__;!!IR39LLzvxw!ONxcdPXwj8lyFeci0a3JR8IBTqjjqtzZg8vjTO6rYamj4BvqNnTOOTtr4Nebr851S2GGVT0acYBtKMlEhaVq3VIN0_k6$>
>>> rishipant...@ryamer.com
>>>
>>>
>>> On May 15, 2023, at 5:59 PM, Michael Thomas  wrote:
>>>
>>>
>>> And maybe try to monetize it? I'm pretty sure that they can be compelled
>>> to do that, but do they do it for their own reasons too? Or is this way too
>>> much overhead to be doing en mass? (I vaguely recall that netflow, for
>>> example, can make routers unhappy if there is too much "flow").
>>>
>>> Obviously this is likely to depend on local laws but since this is NANOG
>>> we can limit it to here.
>>>
>>> Mike
>>>
>>>
>>>

-- 
 <https://www.adams12.org/>

Do ISP's collect and analyze traffic of users?

[Michael Thomas]

And maybe try to monetize it? I'm pretty sure that they can be compelled 
to do that, but do they do it for their own reasons too? Or is this way 
too much overhead to be doing en mass? (I vaguely recall that netflow, 
for example, can make routers unhappy if there is too much "flow").


Obviously this is likely to depend on local laws but since this is NANOG 
we can limit it to here.


Mike

Re: Best Linux (or BSD) hosted BGP?

[Michael Spears via NANOG]

I run BIRD on Ubuntu and it works well. Feel free to reach out off list Bryan if you want some examples of a basic configThank you,Michael SpearsOn May 1, 2023 12:01 PM, Bryan Fields  wrote:I know best subjective, but I'm looking at a project to announce some IP space 
that's between uses now and see what's there.  I'm planing to run a flow 
logger and ntop on the VM and see what is coming in if anything.  I'm looking 
at the options for BGP out there, and there's quite a few (other than running 
a VM with a router doing BGP), but most data I've seen is focused on scale and 
filtering use, or RPKI.  My use case is a bit different, and I can't find any 
best practices for this use case from what I've found.

That said, is there a better solution other than linux/ntop/ipt-netflow?
-- 
Bryan Fields

727-409-1194 - Voice
http://bryanfields.net

Re: Namecheap's outbound email flow compromised: valid rdns, spf, dkim and dmarc on phishes

[Michael Thomas]

It makes you wonder why they just don't rekey and put up a different 
selector while deleting the compromised selector?


Yes, this is bad but it has a straightforward solution to the compromise 
-- unlike compromised cert signing keys, natch.


Mike

On 2/12/23 4:01 PM, Eric Kuhnke wrote:

Namecheap has updated their status page item to include

"We have stopped all the emails (that includes Auth codes delivery, 
Trusted Devices’ verification, and Password Reset emails, etc.)"



Yikes.


On Sun, Feb 12, 2023, 3:54 PM Michael Thomas  wrote:

I think that it might be appropriate to name and shame the third
party, since they should know better too. It almost has the whiff
of a scam.

Mike

On 2/12/23 3:49 PM, Eric Kuhnke wrote:

One very possible theory is that whoever runs the outbound
marketing communications and email newsletter demanded the keys
and got them, with execs overriding security experts at Namecheap
who know better.

I would sincerely hope that the people whose job titles at
Namecheap include anything related to network engineering,
network security or cryptography at that company do know better.
Large domain registrars are not supposed to make such a rookie
mistake.


On Sun, Feb 12, 2023, 3:46 PM Michael Thomas  wrote:


On 2/12/23 3:40 PM, Eric Kuhnke wrote:
>

https://www.namepros.com/threads/concerning-e-mail-from-namecheap.1294946/page-2#post-8839257

>
>
> https://lowendtalk.com/discussion/184391/namecheap-hacked
>
> It looks like a third party service they gave their keys to
has been
> compromised. I got several phishes that fully pass as legit
Namecheap
> emails.
>
> https://www.namecheap.com/status-updates/archives/74848
>
>
If they actually gave them their own private keys, they
clearly don't
get how that's supposed to work with DKIM. The right thing to
do is
create a new selector with the third party's signing key.
Private keys
should be kept... private.

Mike

Re: Namecheap's outbound email flow compromised: valid rdns, spf, dkim and dmarc on phishes

[Michael Thomas]

I think that it might be appropriate to name and shame the third party, 
since they should know better too. It almost has the whiff of a scam.


Mike

On 2/12/23 3:49 PM, Eric Kuhnke wrote:
One very possible theory is that whoever runs the outbound marketing 
communications and email newsletter demanded the keys and got them, 
with execs overriding security experts at Namecheap who know better.


I would sincerely hope that the people whose job titles at Namecheap 
include anything related to network engineering, network security or 
cryptography at that company do know better. Large domain registrars 
are not supposed to make such a rookie mistake.



On Sun, Feb 12, 2023, 3:46 PM Michael Thomas  wrote:


On 2/12/23 3:40 PM, Eric Kuhnke wrote:
>

https://www.namepros.com/threads/concerning-e-mail-from-namecheap.1294946/page-2#post-8839257

>
>
> https://lowendtalk.com/discussion/184391/namecheap-hacked
>
> It looks like a third party service they gave their keys to has
been
> compromised. I got several phishes that fully pass as legit
Namecheap
> emails.
>
> https://www.namecheap.com/status-updates/archives/74848
>
>
If they actually gave them their own private keys, they clearly don't
get how that's supposed to work with DKIM. The right thing to do is
create a new selector with the third party's signing key. Private
keys
should be kept... private.

Mike

Re: Namecheap's outbound email flow compromised: valid rdns, spf, dkim and dmarc on phishes

[Michael Thomas]

On 2/12/23 3:40 PM, Eric Kuhnke wrote:
https://www.namepros.com/threads/concerning-e-mail-from-namecheap.1294946/page-2#post-8839257 



https://lowendtalk.com/discussion/184391/namecheap-hacked

It looks like a third party service they gave their keys to has been 
compromised. I got several phishes that fully pass as legit Namecheap 
emails.


https://www.namecheap.com/status-updates/archives/74848


If they actually gave them their own private keys, they clearly don't 
get how that's supposed to work with DKIM. The right thing to do is 
create a new selector with the third party's signing key. Private keys 
should be kept... private.


Mike

Re: About emails impersonating Path Network

[Michael Thomas]

On 2/7/23 11:33 AM, Jay Hennigan wrote:

On 2/7/23 11:18, Michael Thomas wrote:

FWIW, lookalike domains can and do happen with http too. Nothing 
unique about that to email.


Then the bad guys throw in the occasional Cyrillic, etc. character 
that looks like a Roman one and things get even more fun.


At least with spear-phishing attacks you can bound the problem detection 
investigation since you know what your own domain's legit names are. 
Beyond that, I have no idea if any of the mailbox providers are doing 
anything about lookalike attacks. Email at least has the advantage that 
it is in hands of a user's provider who could care. CA's I'm sure 
couldn't care less.


Mike

Re: About emails impersonating Path Network

[Michael Thomas]

On 2/7/23 6:09 AM, Rich Kulawiec wrote:

On Mon, Feb 06, 2023 at 12:41:43PM -0800, Michael Thomas wrote:

This seems like a perfect object lesson on why you should use DKIM and SPF
and make sure the sending domain can set up a p=reject policy for DMARC.

But it's not.  DKIM and SPF are mostly useless against competently executed
email forgery -- and can even be exploited to make it worse.  Thanks to
a combination of increasingly bad user interfaces, user ignorance,
TLD proliferation, and the ill-advised conflation of identification with
authenticity, it's not currently possible to stop email forgery in any
meaningful sense of the word "stop".


There has been a real failing on the UI side, but that not the fault of 
the authentication protocols. Thunderbird which is what I use is 
completely useless and silent on authentication. For others who 
implement some UI indications, some of them aren't very obvious and 
often tentative. There is a Usenix paper which researched email auth and 
part of it was on user visible feedback. The long and short is that it 
was useful, but not a silver bullet. A large part of the problem from my 
read of it is that it wasn't ubiquitous and standardized ala the Lock 
icon on browsers. It's easy to make fun of that, but it is clearly 
better than nothing. MUA vendors are always at liberty to bring their 
requirements for extensions for protocols or even new protocols if it 
would help the user experience by guiding how MUA's make the UI 
decisions on the advice of senders. It's pretty clear that they find 
this niche at best. That is a pity because some uniformity could make 
this a positive feedback loop and up the utility greatly.


FWIW, lookalike domains can and do happen with http too. Nothing unique 
about that to email.


Mike

Re: About emails impersonating Path Network

[Michael Thomas]

This seems like a perfect object lesson on why you should use DKIM and 
SPF and make sure the sending domain can set up a p=reject policy for 
DMARC.


Mike

On 2/6/23 10:25 AM, Konrad Zemek wrote:

Hi Nanog,

It looks like someone with an axe to grind against our company has decided to 
email every AS contact they could find on PeeringDB, impersonating us and 
sometimes spoofing our domains.

We're aware of the emails and are sorry for the inconvenience. We've since 
added SPF records to the domains we own but don't use (the perps have since 
name-squatted some new ones). We're also actively working with law enforcement 
on the matter.

Thanks
Konrad Zemek
CTO Path Network
AS396998

RE: Smaller than a /24 for BGP?

[Michael Bolton via NANOG]

I’m late to the conversation, but I would have to agree with most. Below a /24 
route advertisement shouldn’t happen.

I have a /24 that I would love to advertise as 2 /25’s, but the affects on 
everyone else is just too much. I take full routes from 4 providers, and I 
certainly don’t want to add over 100K. Carriers and enterprises have to pay a 
lot for our edge routers doing bgp and most don’t want to upgrade. We would 
benefit from advertising /25’s but it hurt’s more than it helps.

I’m in the alarm industry and they still haven’t started adopting IPv6. If we 
allow /25 subnets, some industries will never change. In a sense, we have to 
“force” them to change.

Mike



From: NANOG  On 
Behalf Of Mike Hammett
Sent: Thursday, January 26, 2023 8:52 AM
To: Chris 
Cc: nanog@nanog.org
Subject: Re: Smaller than a /24 for BGP?


CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.
Implementing v6 is important, but unrelated to allowing smaller v4 prefixes.

Not taking a position either way if smaller v4 prefixes is good or bad.


-
Mike Hammett
Intelligent Computing Solutions
[http://www.ics-il.com/images/fbicon.png][http://www.ics-il.com/images/googleicon.png][http://www.ics-il.com/images/linkedinicon.png][http://www.ics-il.com/images/twittericon.png]
Midwest Internet Exchange
[http://www.ics-il.com/images/fbicon.png][http://www.ics-il.com/images/linkedinicon.png][http://www.ics-il.com/images/twittericon.png]
The Brothers WISP
[http://www.ics-il.com/images/fbicon.png][http://www.ics-il.com/images/youtubeicon.png]

From: "Chris" mailto:ch...@noskillz.com>>
To: "Justin Wilson (Lists)" mailto:li...@mtin.net>>
Cc: nanog@nanog.org
Sent: Wednesday, January 25, 2023 2:24:29 PM
Subject: Re: Smaller than a /24 for BGP?
I would suggest that this is trying to solve the wrong problem.  To me this is 
pressure to migrate to v6, not alter routing rules.

Kind Regards,
Chris Haun

On Tue, Jan 24, 2023 at 12:21 PM Justin Wilson (Lists) 
mailto:li...@mtin.net>> wrote:
Have there been talks about the best practices to accept things smaller than a 
/24? I qm seeing more and more scenarios where folks need to participate in BGP 
but they do not need a full /24 of space.  Seems wasteful.  I know this would 
bloat the routing table immensely.  I know of several folks who could split 
their /24 into /25s across a few regions and still have plenty of IP space.



Justin Wilson
j...@j2sw.com

—
https://blog.j2sw.com - Podcast and Blog
https://www.fd-ix.com

IMPORTANT NOTICE: This e-mail message is intended to be received only by 
persons entitled to receive the confidential information it may contain. E-mail 
messages to clients of Holmes Security Systems may contain information that is 
confidential and legally privileged. Please do not read, copy, forward, or 
store this message unless you are an intended recipient of it. If you have 
received this message in error, please forward it to the sender and delete it 
completely from your computer system.

Re: Starlink routing

[Michael Thomas]

On 1/23/23 3:14 PM, Eric Kuhnke wrote:
The original and traditional high-cost way of how this is done for 
MEO/LEO is exemplified by an o3b terminal, which has two active 
motorized tracking antennas. The antenna presently in use for the 
satellite that is overhead follows it until it's descending towards 
the horizon, while at the same time the second antenna aims itself at 
where the next 'rising' satellite is predicted to appear at the 
opposite horizon, and forms a link to it. Make-before-break. If anyone 
has seen photographs in their marketing material/videos of the Oneweb 
beta test earth stations in Alaska they are operating using the same 
general concept.


Oneweb has clearly positioned their market focus for telecoms and ISPs 
and large enterprise end users, because their CPE equipment is 
considerably larger, expensive and more power hungry. The beta test 
sites I've seen installed on top of a telecom equipment shelter occupy 
an area approximately 8 feet long x 4 feet wide including radomes and 
mounting.


I'm trying to understand this so sorry if this comes off dumb. So does 
the base station mediate all handoffs where the CPE is told when/what to 
handoff? Or does the CPE have some part in it (other than receiving the 
handoff)? Does the CPE accept control traffic (L2?) from any bird? Are 
there cases where the CPE needs to de-dup packets due to handoffs?


This is pretty fascinating stuff.

Mike

Re: Starlink routing

[Michael Thomas]

On 1/22/23 3:05 PM, Matthew Petach wrote:



On Sun, Jan 22, 2023 at 2:45 PM Michael Thomas  wrote:

I read in the Economist that the gen of starlink satellites will have
the ability to route messages between each satellite. Would
conventional
routing protocols be up to such a challenge? Or would it have to be
custom made for that problem? And since a lot of companies and
countries
are getting on that action, it seems like fertile ground for (bad)
wheel
reinvention?

Mike



Unlike most terrestrial links, the distances between satellites are 
not fixed,

and thus the latency between nodes is variable, making the concept of
"Shortest Path First" calculation a much more dynamic and challenging
one to keep current, as the latency along a path may be constantly 
changing
as the satellite nodes move relative to each other, without any link 
state actually

changing to trigger a new SPF calculation.


One thing that is in their favor is that while they are moving, they are 
moving in a predictable manner. It seems that each router could, 
essentially, locally update routes until they are told otherwise?





I suspect a form of OLSR might be more advantageous in a dynamic partial
mesh between satellites, but I haven't given it as much deep thought 
as would

be necessary to form an informed opinion.

So, yes--it's likely the routing protocol used will not be entirely 
"off-the-shelf"

but will instead incorporate continuous latency information in the LSDB,
and path selection will be time-bound based on the rate of increase in 
latency

along currently-selected edges in the graph.


Has IETF looked at this, do you know? Even if the routers can't 
interoperate with other systems, it would be good to have some routing 
clue with a lot of eyeballs on it to not make rookie mistakes.


Mike

Starlink routing

[Michael Thomas]

I read in the Economist that the gen of starlink satellites will have 
the ability to route messages between each satellite. Would conventional 
routing protocols be up to such a challenge? Or would it have to be 
custom made for that problem? And since a lot of companies and countries 
are getting on that action, it seems like fertile ground for (bad) wheel 
reinvention?


Mike

Re: Nov 16th (Overnight Traffic Pattern)

[Michael Zarglis via NANOG]

sounds likely, I think the new Warzone tips the scales at 180GB for PC.

--
M Zarglis

J Crowe wrote on 2022-12-06 2:48 PM:

That was most likely due to Call of Duty Warzone 2.0 going live that day.

On Tue, Dec 6, 2022 at 2:34 PM Nicholas Warren 
mailto:nwar...@barryelectric.com>> wrote:


I noticed something strange looking at our traffic pattern
(eyeball network). Our usage normally drops very low between 4am
and 6pm (central). But, on the 16^th of November it never dropped.

Was there a larger pattern or did a full moon only shine above our
network?

Nich Warren

Fred Brooks has died

[Michael Thomas]

His Mythical Man Month is a must read for anybody even remotely adjacent 
to coding, and frankly it should be read out of that context too.


RIP Fred and thank you, that was one of the most important books I've 
ever read.


Mike

Re: juniper.net down?

[michael brooks - ESC]

Site loading from AS30136


On Tue, Oct 18, 2022, 12:13 PM  wrote:

> juniper.net down?
>
>
>
>
>
>
>
> Aaron
>
> aar...@gvtc.com
>
>
>

-- 
 

Re: jon postel

[michael brooks - ESC]

Fixed:

Where Wizards Stay Up Late should be
required reading for anyone wanting to learn about the history /
development of the ARPAnet and the Internet.




michael brooks
Sr. Network Engineer
Adams 12 Five Star Schools
720.972.4110
michael.bro...@adams12.org

"flying is learning how to throw yourself at the ground and miss"



On Mon, Oct 17, 2022 at 8:59 AM Grant Taylor via NANOG 
wrote:

> On 10/16/22 8:28 PM, Joseph wrote:
> > A good book on the topic of the early internet is "Where Wizards Stay Up
> > Late" by Katie Hafner and Matthew Lyon. A large part of the book covers
> > happenings at Bolt Beranek and Newman, and there are plenty of mentions
> > of Jon Postel.
>
> +1 (with an extremely large value of one)
>
> I have (re)read Where Wizards Stay Up Late /and/ recommended it to many
> people many different times.
>
> In my not so humble opinion, Where Wizards Stay Up Late should be
> required reading for anyone wanting to learn about the history /
> development of the ARPAnet and the Internet.
>
>
>
> --
> Grant. . . .
> unix || die
>
>

-- 
 <https://www.adams12.org/>

Re: FCC chairwoman: Fines alone aren't enough (Robocalls)

[Michael Thomas]

On 10/7/22 12:45 AM, Brian Turnbow via NANOG wrote:

The federal law in 47 USC 227(e) says:

(1)In general

  It shall be unlawful for any person within the United  States, or any person
outside the United States if the recipient is  within the United States, in
connection with any voice service or text  messaging service, to cause any
caller identification service to  knowingly transmit misleading or inaccurate
caller identification  information with the intent to defraud, cause harm, or
wrongfully  obtain anything of value, unless such transmission is exempted
pursuant to paragraph (3)(B).

In (3)(B) is a narrow carve-out for law enforcement and court orders.

The important point is that spoofing is illegal with fraudulent intent, OK with
benign intent.

This is a very interesting conversation as there is a ongoing discussion on how 
to ban spoofed calls here in Italy..
Here operators must identify each customer and ensure that they are screening 
incoming numbers.
I assume by "incoming" you mean incoming from the customer? That is, 
you're making sure your customers are not spoofing?

Most do, but some do not and become sources of spoofed traffic.
The biggest problem however comes from out of country originators that allow 
foreign call centers to use Italian numbers.
Are these spoofed or are they actually allocated to them? If they are 
actually allocated, their upstream provider should be enforcing too, right?

Thus the calls come in from an international carrier.
We are moving twords blocking incoming calls from international trunks 
containing Italian from numbers, something we see already in place for carriers 
in other EU countries such as France.
On the other hand, this makes your "incoming" above sound like incoming 
from the phone network. So I'm probably confused.

Most operators here have been against stir/shaken as a means to resolve the 
problems.


What reasons?

Mike

Re: FCC chairwoman: Fines alone aren't enough (Robocalls)

[Michael Thomas]

On 10/4/22 5:23 PM, Peter Beckman wrote:

On Tue, 4 Oct 2022, Michael Thomas wrote:

Exactly. And that doesn't require an elaborate PKI. Who is allowed to 
use what telephone numbers is an administrative issue for the ingress 
provider to police. It's the equivalent to gmail not allowing me to 
spoof whatever email address I want. The FCC could have required that 
ages ago.


 How does one carrier that gets DIDs from multiple other carriers
 communicate to the termination carrier selected during LCR that the DID
 set as CallerID is indeed serviced by that carrier and authorized to use
 said DID as CallerID?

 If a call is asynchronous, e.g. the DID carrier is not the terminating
 carrier, how can the termination carrier trust/know definitively that
 someone is allowed to use that CallerID?

 Don't forget the resellers!!!

My point is not that the termination carrier believe that it's 
legitimate (although that would be nice), but to get the originating 
carrier to police things before it ever gets forwarded. The FCC could 
have forced that ages ago in most cases. Requiring the receiving end to 
police things is fraught with false positives where the originating 
carrier has a lot more knowledge of who their customer is.


Mike