from:"Mike A"

"a limited set of providers willing to sell it, if at all."

I know of one (Windstream) that offers it on a portion of their footprint. I 
swore others did, but I couldn't find them. Does anyone know who else in the 
NANOG area who does this?

-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

- Original Message -
From: Mark Tinka 
To: Dave Cohen 
Cc: nanog@nanog.org, Mike Hammett 
Sent: Sun, 12 May 2024 17:34:19 -0500 (CDT)
Subject: Re: Alien Waves

On 5/13/24 00:11, Dave Cohen wrote:

> Mark,
>
> Many/all of these points are fair. My experience is purely terrestrial and 
> obviously both the capacity and economic calculations are vastly different in 
> those situations, which I should have called out.

Actually, terrestrial economics are easier to consider because you have 
the one thing the subsea applications don't have in abundance... power.

Fair point, terrestrial revenues are significantly lower than subsea 
revenues on a per-bit basis, but so are the deployment costs. That evens 
out, somewhat.

> However, I don’t think that the optical vendor is really the challenge - I 
> would agree that, generally, spectrum is going to be available through larger 
> providers that are using “traditional carrier grade” platforms - but rather 
> at the service provider level. When something invariably breaks at 3 AM and 
> the third shift Tier I NOC tech who hasn’t read the service playbook says “I 
> don’t see any errors on your transponder, sorry, it’s not on our end” because 
> they’re not aware that they actually don’t have access to the transponder and 
> need to start looking elsewhere, that’s the sort of thing that creates 
> systemic challenges for users regardless of whether the light is being shot 
> across a Ciena 6500 or a Dave’s Box-o’-Lasers 1000.

I think you are contradicting yourself a bit, unless I misunderstand 
your point.

Legacy vendors who have spectrum controllers have made this concern less 
of an issue. But then again, to be fair, adopting spectrum controllers 
along with bandwidth expansions via things like gridless line systems 
and C+L backbone architectures that make spectrum sales a lot more 
viable at scale do come at a hefty $$ premium. So I can understand that 
offering spectrum independent of spectrum controllers is going to be 
more trouble than it is worth.

Ultimately, what I'm saying is that technologically, this is now a 
solved problem, for the most part. That said, I don't think it will be 
the majority of DWDM operators offering spectrum services en masse, for 
at least a few more years. So even if you want to procure managed 
spectrum or spectrum sharing, you are likely to come up against a 
limited set of providers willing to sell it, if at all.

Mark.

Re: VDSL >2 Pair Bonding Modems

I have found a modem with Positron that'll do up to 8 pair of bonding. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Mike Hammett"  
To: "nanog"  
Sent: Friday, April 26, 2024 3:09:07 PM 
Subject: VDSL >2 Pair Bonding Modems 


I recently figured out that my Calix E7s can bond more than 2 pair of VDSL 
lines. However, none of my modem vendors seem to support more than 2 pair. What 
modem platforms are people using in this scenario? 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com

Re: Whitebox Routers Beyond the Datasheet

Some of you have pointed out (onlist and offlist) the importance of the OS to 
these concerns. Yes, that makes sense. THe Venn Diagram of hardware that 
can\can't and OSes that can\can't. 

I'd appreciate some feedback as well on the OS side of things. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Mike Hammett"  
To: nanog@nanog.org 
Sent: Friday, April 12, 2024 8:03:49 AM 
Subject: Whitebox Routers Beyond the Datasheet 


I'm looking at the suitability of whitebox routers for high through, low port 
count, fast BGP performance applications. Power efficiency is important as 
well. 


What I've kind of come down to (based on little more than spec sheets) are the 
EdgeCore AGR400 and the UfiSpace S9600-30DX. They can both accommodate at least 
three directions of 400G for linking to other parts of my network and then have 
enough 100G or slower ports to connect to transit, peers, and customers as 
appropriate. Any other suggestions for platforms similar to those would be 
appreciated. 


They both appear to carry buffers large enough to accommodate buffering 
differences in port capacities, which is an issue I've seen with boxes more 
targeted to cloud\datacenter switching. 


What isn't in the spec sheets is BGP-related information. They don't mention 
how many routes they can hold, just that they have additional TCAM to handle 
more routes and features. That's wonderful and all, but does it take it from 
64k routes to 512k routes, or does it take it from 256k routes up to the 
millions of routes? Also, BGP convergence isn't listed (nor do I rarely ever 
see it talked about in such sheets). I know that software-based routers can now 
load a full table in 30 seconds or less. I know that getting the FIB updated 
takes a little bit longer. I know that withdrawing a route takes a little bit 
longer. However, often, that performance is CPU-based. An underpowered CPU may 
take a minute or more to load that table and may take minutes to handle route 
churn. Can anyone speak to these routers (or routers like these) ability to 
handle modern route table activity? 


My deployment locations and philosophies simply won't have me in an environment 
where I need the density of dozens of 400G\100G ports. That the routers that 
seem to be more marketed to the use case are designed for. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com

Alien Waves

What are your experiences with alien waves, managed spectrum, spectrum as a 
service, etc?



-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

Re: Opengear alternatives that support 5g?

2024-04-27 Thread Mike Lyon

Mel,My apologies, i confused one mikrotik with another model. You are correct.I would also check out CradlePoint and Teltonika as well. Teltonika Networksteltonika-networks.comCheers,MikeOn Apr 26, 2024, at 23:06, Mel Beckman wrote:

Mike,

Thanks for that info. Alas, I’m not seeing any Mikrotik 5G devices cheaper than a ~$500 Peplink. Am I misunderstanding your suggested solution?

-mel

On Apr 26, 2024, at 9:50 PM, Mike Lyon wrote:

Peplink is nice, but there are cheaper options: Mikrotik-dot-com

Then for cellular service, sign up for an IOT with an IOT MVNO that bills usage based (and can also offer you a static, public, IP address AND will also allow you to build a VPN across all of your devices) such as SimBase:
simbase-dot-com

Cheers,
Mike

On Apr 26, 2024, at 21:37, Mel Beckman wrote:

I’ve been loooking at the $600 Peplink MAX BR1-MINI (HW3) industrial 5G router. It has a 1x embedded 5G modem (Verizon, AT, T-Mobile, and FirstNet). three GigE ports, four antenna connectors, and comes with an stick antenna set and AC PS.
It uses a nanoSIM. Yes, it’s a pure IP router with no knowledge of serial protocols. So I would just put an air console behind it to get to my serial ports. I’m still evaluating 5G plans, and Verizon just offered an amazing $15 per month unlimited data deal,
but it seems to have a 50 gig limit before you get to throttling. That might not matter at all with serial traffic though.

We've been using the Netgear 4G cellular router, but that’s becoming increasingly unreliable. The NG has a nailed up IPsec VPN tunnel, obviating the need for a static IP, and the keepalive traffic is low enough that it doesn’t cost us much on the 4G network.
I’m hoping 5G will be even cheaper and faster.

I’d love to see if anybody found anything better before I spring for a Peplink test unit.

-mel

On Apr 26, 2024, at 9:45 AM, Warren Kumari wrote:

On Fri, Apr 26, 2024 at 12:43 AM, Saku Ytti
<s...@ytti.fi> wrote:

On Fri, 26 Apr 2024 at 03:11, David H <ispcolohost@gmail.com> wrote:

Curious if anyone has particular hardware they like for OOB / serial management, similar to OpenGear, but preferably with 5G support, maybe even T-Mobile support? It’s becoming increasingly difficult to get static IP 4g machine accounts out of Verizon,
and the added speed would be nice too. Or do you separate the serial from the access device (cell+firewall, etc.)?

You could get a 5G Catalyst with an async NIM or SM.

But I think you're setting up yourself for unnecessary costs and failures by designing your OOB to require static IP. You could design it so that the OOB spokes dial-in to the central OOB hub, and the OOB hub doesn't care what IP they come from,
using certificates or PSK for identity, instead of IP.

Yup, I agree — but that simply rewrites the question to be:

"Curious if anyone has particular hardware they like for OOB / serial management, similar to OpenGear, but preferably with 5G support, which can be a spoke that dials in to the central OOB hub, and the OOB hub doesn't care what IP they come from, using
certificates or PSK for identity, instead of IP."

I've been on the same quest, and I have some additional requests / features. Ideally it:

1: would be small - my particular use-case is for a "traveling rack", and so 0U is preferred.

2: would be fairly cheap.

3: would not be a Raspberry-Pi, a USB hub and USB-to-serial cables. We tried that for a while, and it was clunky — the SD card died a few times (and jumped out entirely once!), people kept futzing with the OS and fighting over which console software to
use, installing other packages, etc.

4: support modern SSH clients (it seems like you shouldn't have to say this, but… )

5: actually be designed as a termserver - the current thing we are using doesn't really understand terminals, and so we need to use 'socat -,raw,echo=0,escape=0x1d TCP::' to get things like tab-completion and "up-arrow for last command"
to work.

6: support logging of serial (e.g crash-messages) to some sort of log / buffer / similar (it's useful to be able to see what a device barfed all over the console when it crashes.

The Get Console Airconsole TS series meets many of these requirements, but it doesn't do #6. It also doesn't really feel like they have been updating / maintaining these.

Yes, I fully acknowledge that #3 falls into the "Doctor, Doctor, it hurts when I do this" camp, but, well…

--
++ytti

Re: Opengear alternatives that support 5g?

2024-04-27 Thread Mike Lyon

Peplink is nice, but there are cheaper options:MikroTikmikrotik.comThen for cellular service, sign up for an IOT with an IOT MVNO that bills usage based (and can also offer you a static, public, IP address AND will also allow you to build a VPN across all of your devices) such as SimBase: Seamless IoT SIM Card Solutionssimbase.comCheers,MikeOn Apr 26, 2024, at 21:37, Mel Beckman wrote:

I’ve been loooking at the $600 Peplink MAX BR1-MINI (HW3) industrial 5G router. It has a 1x embedded 5G modem (Verizon, AT, T-Mobile, and FirstNet). three GigE ports, four antenna connectors, and comes with an stick antenna set and AC PS. It uses a nanoSIM.
Yes, it’s a pure IP router with no knowledge of serial protocols. So I would just put an air console behind it to get to my serial ports. I’m still evaluating 5G plans, and Verizon just offered an amazing $15 per month unlimited data deal, but it seems to
have a 50 gig limit before you get to throttling. That might not matter at all with serial traffic though.

I’d love to see if anybody found anything better before I spring for a Peplink test unit.

-mel

On Apr 26, 2024, at 9:45 AM, Warren Kumari wrote:

On Fri, Apr 26, 2024 at 12:43 AM, Saku Ytti
wrote:

On Fri, 26 Apr 2024 at 03:11, David H wrote:

You could get a 5G Catalyst with an async NIM or SM.

Yup, I agree — but that simply rewrites the question to be:

I've been on the same quest, and I have some additional requests / features. Ideally it:

1: would be small - my particular use-case is for a "traveling rack", and so 0U is preferred.

2: would be fairly cheap.

4: support modern SSH clients (it seems like you shouldn't have to say this, but… )

6: support logging of serial (e.g crash-messages) to some sort of log / buffer / similar (it's useful to be able to see what a device barfed all over the console when it crashes.

The Get Console Airconsole TS series meets many of these requirements, but it doesn't do #6. It also doesn't really feel like they have been updating / maintaining these.

Yes, I fully acknowledge that #3 falls into the "Doctor, Doctor, it hurts when I do this" camp, but, well…

--
++ytti

VDSL >2 Pair Bonding Modems

2024-04-26 Thread Mike Hammett

I recently figured out that my Calix E7s can bond more than 2 pair of VDSL 
lines. However, none of my modem vendors seem to support more than 2 pair. What 
modem platforms are people using in this scenario? 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com

Re: Whitebox Routers Beyond the Datasheet

2024-04-12 Thread Mike Hammett

That makes sense, but also why I'm going beyond the datasheet here to solicit 
people's feedback. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Tom Beecher"  
To: "Mike Hammett"  
Cc: nanog@nanog.org 
Sent: Friday, April 12, 2024 1:30:04 PM 
Subject: Re: Whitebox Routers Beyond the Datasheet 




Also, BGP convergence isn't listed (nor do I rarely ever see it talked about in 
such sheets). 




I feel like this shouldn't be listed on a data sheet for just the whitebox 
hardware. The software running on it would be the gating factor. 


On Fri, Apr 12, 2024 at 9:05 AM Mike Hammett < na...@ics-il.net > wrote: 





I'm looking at the suitability of whitebox routers for high through, low port 
count, fast BGP performance applications. Power efficiency is important as 
well. 


What I've kind of come down to (based on little more than spec sheets) are the 
EdgeCore AGR400 and the UfiSpace S9600-30DX. They can both accommodate at least 
three directions of 400G for linking to other parts of my network and then have 
enough 100G or slower ports to connect to transit, peers, and customers as 
appropriate. Any other suggestions for platforms similar to those would be 
appreciated. 


They both appear to carry buffers large enough to accommodate buffering 
differences in port capacities, which is an issue I've seen with boxes more 
targeted to cloud\datacenter switching. 


What isn't in the spec sheets is BGP-related information. They don't mention 
how many routes they can hold, just that they have additional TCAM to handle 
more routes and features. That's wonderful and all, but does it take it from 
64k routes to 512k routes, or does it take it from 256k routes up to the 
millions of routes? Also, BGP convergence isn't listed (nor do I rarely ever 
see it talked about in such sheets). I know that software-based routers can now 
load a full table in 30 seconds or less. I know that getting the FIB updated 
takes a little bit longer. I know that withdrawing a route takes a little bit 
longer. However, often, that performance is CPU-based. An underpowered CPU may 
take a minute or more to load that table and may take minutes to handle route 
churn. Can anyone speak to these routers (or routers like these) ability to 
handle modern route table activity? 


My deployment locations and philosophies simply won't have me in an environment 
where I need the density of dozens of 400G\100G ports. That the routers that 
seem to be more marketed to the use case are designed for. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com

Whitebox Routers Beyond the Datasheet

2024-04-12 Thread Mike Hammett

I'm looking at the suitability of whitebox routers for high through, low port 
count, fast BGP performance applications. Power efficiency is important as 
well. 


What I've kind of come down to (based on little more than spec sheets) are the 
EdgeCore AGR400 and the UfiSpace S9600-30DX. They can both accommodate at least 
three directions of 400G for linking to other parts of my network and then have 
enough 100G or slower ports to connect to transit, peers, and customers as 
appropriate. Any other suggestions for platforms similar to those would be 
appreciated. 


They both appear to carry buffers large enough to accommodate buffering 
differences in port capacities, which is an issue I've seen with boxes more 
targeted to cloud\datacenter switching. 


What isn't in the spec sheets is BGP-related information. They don't mention 
how many routes they can hold, just that they have additional TCAM to handle 
more routes and features. That's wonderful and all, but does it take it from 
64k routes to 512k routes, or does it take it from 256k routes up to the 
millions of routes? Also, BGP convergence isn't listed (nor do I rarely ever 
see it talked about in such sheets). I know that software-based routers can now 
load a full table in 30 seconds or less. I know that getting the FIB updated 
takes a little bit longer. I know that withdrawing a route takes a little bit 
longer. However, often, that performance is CPU-based. An underpowered CPU may 
take a minute or more to load that table and may take minutes to handle route 
churn. Can anyone speak to these routers (or routers like these) ability to 
handle modern route table activity? 


My deployment locations and philosophies simply won't have me in an environment 
where I need the density of dozens of 400G\100G ports. That the routers that 
seem to be more marketed to the use case are designed for. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com

Re: Netskrt - ISP-colo CDN

2024-04-07 Thread Mike Hammett

I suppose that depends on the size (bits and miles) of the network and the cost 
of transport within it. In many areas, space + power + port is cheaper than 
transport. 

- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Tim Burke"  
To: "Aaron Gould"  
Cc: nanog@nanog.org 
Sent: Saturday, April 6, 2024 10:00:05 PM 
Subject: Re: Netskrt - ISP-colo CDN 

I have been trying to get _away_ from caching appliances on our network — other 
than Google, we are able to pick up most of the stuff that otherwise would be 
cacheable via private peering; so it doesn’t make a whole lot of sense for us 
to have appliances in the datacenter taking up space, power, and 100G ports, 
and increasing potential attack surface by having devices that we cannot 
control directly connected to edge routers. 

> On Apr 4, 2024, at 2:57 PM, Aaron Gould  wrote: 
> 
> Anyone out there using Netskrt CDN? I mean, installed in your network for 
> content delivery to your customers. I understand Netskrt provides caching for 
> some well known online video streaming services... just wondering if there 
> are any network operators that have worked with Netskrt and deployed their 
> caching servers in your networks and what have you thought about it? What 
> Internet uplink savings are you seeing? 
> 
> Netskrt - https://www.netskrt.io/ 
> 
> 
> -- 
> -Aaron 
>

Re: Netskrt - ISP-colo CDN

2024-04-04 Thread Mike Hammett

It's free. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Eric Dugas via NANOG"  
To: "Aaron Gould"  
Cc: nanog@nanog.org 
Sent: Thursday, April 4, 2024 4:12:38 PM 
Subject: Re: Netskrt - ISP-colo CDN 


That name rang a bell so I looked up my emails. 


They contacted me last year, they were claiming to be "working with some of the 
major streaming brands, such as Amazon Prime Video, to improve the quality of 
both VOD and live streaming while also reducing the load on ISP networks such 
as your own.". 


Based on my quick research, they have a few registered ASNs (their peeringdb 
page ) with a few netblocks but I get 0 traffic from them (we're a sizable 
eyeball network). Their origin network might still not be ready but digging a 
little bit more, it seems they act as a third-party video caching solution and 
not as an origin CDN so in the end, they're really just trying to sell ISPs and 
other types of customers their caching solutions. 


Eric 


On Thu, Apr 4, 2024 at 4:00 PM Aaron Gould < aar...@gvtc.com > wrote: 


Anyone out there using Netskrt CDN? I mean, installed in your network 
for content delivery to your customers. I understand Netskrt provides 
caching for some well known online video streaming services... just 
wondering if there are any network operators that have worked with 
Netskrt and deployed their caching servers in your networks and what 
have you thought about it? What Internet uplink savings are you seeing? 

Netskrt - https://www.netskrt.io/ 


-- 
-Aaron

Re: Unimus as NCM (Network Configuration Management) Tool

2024-04-04 Thread Mike Hammett

Unimus is very open to adding additional platforms and improving support for 
existing platforms. I'd reach out to them for assistance. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Mel Beckman"  
To: "Mike Lyon"  
Cc: "nanog"  
Sent: Thursday, April 4, 2024 1:49:31 AM 
Subject: Re: Unimus as NCM (Network Configuration Management) Tool 

We use both Unumus and ManageEngine. Neither covers all device models, or all 
firmware versions of all devices, so we have to use both products to get 
complete device coverage. Scaling depends on host performance, so for large 
device populations you may want to assign different SCM instances to particular 
subgroups. 


-mel via cell 



On Apr 3, 2024, at 11:28 PM, Mike Lyon  wrote: 








I use it for config backups, diffs, etc. Love it. 


Theres others such as Rancid but im not sure if it works on anything other than 
Vendor C. 


-Mike 



On Apr 3, 2024, at 23:16, Shahid Shafi  wrote: 







Hi Network Experts, 


Is anyone using Unimus as your main NCM tool in production? I am looking at an 
NCM tool that can scale upto 10,000 to 15,000 Network Devices. Do you recommend 
any other solution? The solution should atleast able to support network config 
backups, diffs, and basic network auditing features. 


https://unimus.net/ 



thanks 
Shahid

Re: Unimus as NCM (Network Configuration Management) Tool

2024-04-04 Thread Mike Lyon

I use it for config backups, diffs, etc. Love it.

Theres others such as Rancid but im not sure if it works on anything other than 
Vendor C.

-Mike

> On Apr 3, 2024, at 23:16, Shahid Shafi  wrote:
> 
> 
> Hi Network Experts,
> 
> Is anyone using Unimus as your main NCM tool in production? I am looking at 
> an NCM tool that can scale upto 10,000 to 15,000 Network Devices. Do you 
> recommend any other solution? The solution should atleast able to support 
> network config backups, diffs, and basic network auditing features.
> 
> https://unimus.net/
> 
> thanks
> Shahid

RE: Meta outage

2024-03-05 Thread Mike Lewinski via NANOG

On Tue, 05 Mar 2024 12:17 -0700, Michael Rathbun  wrote:

> What I found intriguing was that I was logged out by Google Docs at the same 
> moment FB logged me out.  Downdetector showed a number of other supposedly 
> unrelated services with large outage report spikes at roughly the same time.

I was logged out of Honeywell's Total Connect Comfort site (remote thermostat 
control) at the same time FB logged me out. I'm not using OAUTH logins anywhere.

I had recently changed SSID and the thermostat wasn't accepting remote 
commands. I thought maybe the SSID change had broken it and had just deleted 
the device from the TCC site to try adding it back when I was logged out.

It's funny timing because earlier today I was telling an employee how we don't 
like to have our maintenance overlap with vendor's maintenance/repair window 
because when something breaks while we are making changes we can't always tell 
who is at fault.

Re: Dark Fiber in DC Metro

2024-03-04 Thread Mike Hammett

I would assume that's going to be highly dependent on which facilities you want 
it to be in. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Theo Voss"  
To: nanog@nanog.org 
Sent: Monday, March 4, 2024 6:29:38 AM 
Subject: Dark Fiber in DC Metro 

Hi all, 

does anyone has a recommendation for a Dark Fiber provider in the DC Metro 
(Ashburn/Reston) area? Preferably an easy to work with one. :-) 

Appreciate any hints, thanks! 

Best regards, 
Theo Voss

Re: starlink ixp peering progress

2024-02-27 Thread Mike Hammett

The best way I've found (and it is indeed rather incomplete) is to have a BGP 
feed going to something like QRator from that AS (or a downstream AS) that then 
performs analytics on the BGP feed. Starlink is unlikely to have BGP customers, 
so that makes it a bit more difficult. 


https://radar.qrator.net/as/14593/ipv4/neighbors/peerings 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Dave Taht"  
To: "NANOG"  
Sent: Tuesday, February 27, 2024 1:54:44 AM 
Subject: starlink ixp peering progress 

One of the things I learned today was that starlink has published an 
extensive guide as to how existing BGP AS holders can peer with them 
to get better service. 

https://starlink-enterprise-guide.readme.io/docs/peering-with-starlink 

I am curious if there is a way to see how many have peered already, 
how many they could actually peer with?, and progress over time since 
inception what would be the right tools for that? This is pretty 
impressive for peering so far: 

https://www.peeringdb.com/net/18747 

Is there a better email list to discuss ixp stuff? 


-- 
https://blog.cerowrt.org/post/2024_predictions/ 
Dave Täht CSO, LibreQos

Re: Network chatter generator

2024-02-24 Thread Mike Hammett

I came to suggest this. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Jesse DuPont"  
To: "Brandon Martin" , nanog@nanog.org 
Sent: Friday, February 23, 2024 12:17:28 PM 
Subject: Re: Network chatter generator 

I believe you can do most of what you want using a Mikrotik and its Traffic 
Generator. Packet templates can be crafted mimic any of the popular protocols 
(L2, L3, L4), at least at the header level, with less flexibility on the 
payload legitimacy. 


On 2/23/24 10:33 AM, Brandon Martin wrote: 


Before I go to the trouble of making one myself, does anybody happen to know of 
a pre-canned program to generate realistic and scalable amounts of 
broadcast/broad-multicast network background "chatter" seen on typical consumer 
and business networks? This would be things like lots of ARP traffic to/from 
various sources/destinations within a subnet, SSDP, MDNS-SD, SMB browser 
traffic, DHCP requests, etc.? 

Ideally, said tool would have knobs to control the amount of traffic and 
whether a given type of traffic is present. 

This is mostly for torture testing "IoT" type devices by exposing them to lots 
of diverse, essentially nonsense traffic that they're likely to see in a real 
environment. 

-- 
Brandon Martin

Re: Verizon Business Contact

2024-02-19 Thread Mike Hammett

But then MCI is the one running fiber to all of the Verizon Wireless sites, so 
that doesn't help in de-muddying the waters. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: sro...@ronan-online.com 
To: "Justin Krejci"  
Cc: nanog@nanog.org 
Sent: Monday, February 19, 2024 1:54:43 PM 
Subject: Re: Verizon Business Contact 



Verizon Business is the fixed line business focused entity, formerly MCI and 
UUNET. Verizon Wireless is the wireless business entity. 


Shane 



On Feb 19, 2024, at 2:44 PM, Justin Krejci  wrote: 







For me it is some AS 6167 destinations. 
WHOIS for that ASN says this is Verizon Business. 

AS Number:  6167 Org Name:   Verizon Business 


I am not sure how I am supposed to accurately or authoritatively discern the 
differences in specific IP prefixes (or ASNs) as to whether they are are used 
in the Verizon Wireless, Verizon Business, Verizon XYZ, etc. 
I am also not sure what the value would be understanding the difference as I 
have zero contacts at any Verizon entity: Wireless, Business, or any other. 


I imagine at some level, there is a parent Verizon umbrella organization that 
is ultimately responsible for all underling organizations/divisions but I am 
not particularly interested in trying to pick apart the business silos of 
Verizon and then from there trying to chase down specific Verizon entity 
contacts to try and figure out who, might be the right contact to look into 
this. I have made efforts, prior to this NANOG thread even starting, to get 
this issue rectified but I have had zero luck so far getting any appropriate 
person at Verizon to take notice. 


It kind of feels like trying to reach out to some company regarding a 
geolocation or IP-reputation type issue... just a lot of "Sorry, I don't know. 
try this other group that you already talked to" or simply "piss off" type 
responses. Both of which I have received in sizable quantities. Now that my 
brain is on that tangent, my favorite geolocation response was when I was told 
"your ISP needs to set the correct bits in the IP packets to designate the 
traffic as coming from the correct geography." I laughed and I cried at that 
one. 






-Original Message- 
>From : Richard Laager < rlaa...@wiktel.com > 
To : Justin Krejci < jkre...@usinternet.com > 
Cc : nanog@nanog.org < nanog@nanog.org > 
Subject : Re: Verizon Business Contact 
Date : Fri, 16 Feb 2024 20:41:04 -0600 


On 2024-02-09 18:10, Justin Krejci wrote: 




For a good long while (months) we have had similar issues with various Verizon 
destinations. 


Only Verizon Wireless destinations, or other Verizon Business things? 

As of today, I'm told (via an upstream provider) that Verizon Business says 
this is a Verizon Wireless issue.

Re: IPv6 uptake

2024-02-19 Thread Mike Hammett

" In IPv6's default operation, if Joe has two connections then each of 
his computers has two IPv6 addresses and two default routes. If one 
connection goes down, one of the routes and sets of IP addresses goes 
away." 

This sounds like a disaster. 

- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "William Herrin"  
To: "Mike Hammett"  
Cc: nanog@nanog.org 
Sent: Monday, February 19, 2024 9:16:52 AM 
Subject: Re: IPv6 uptake 

On Mon, Feb 19, 2024 at 6:52 AM Mike Hammett  wrote: 
> "We can seriously lose NAT for v6 and not lose 
> anything of worth." 
> 
> I'm not going to participate in the security conversation, but we 
> do absolutely need something to fill the role of NAT in v6. If it's 
> already there or not, I don't know. Use case: Joe's Taco Shop. 
> Joe doesn't want a down Internet connection to prevent 
> transactions from completing, so he purchases two diverse 
> broadband connections, say a cable connection and a DSL connection. 

Hi Mike, 

In IPv6's default operation, if Joe has two connections then each of 
his computers has two IPv6 addresses and two default routes. If one 
connection goes down, one of the routes and sets of IP addresses goes 
away. 

Network security for that scenario is, of course, challenging. There's 
a longer list of security-impacting things that can go wrong than with 
the IPv4 NAT + dual ISP scenario. 

There's also the double-ISP loss scenario that causes Joe to lose all 
global-scope IP addresses. He can overcome that by deploying ULA 
addresses (a third set of IPv6 addresses) on the internal hosts, but 
convincing the internal network protocols to stay on the ULA addresses 
is wonky too. 

There's also 1:1 NAT where Joe can just use ULA addresses internally 
and have the firewall translate into the address block of the active 
ISP. However, because this provides a full map between every internal 
address, protocol and port to external addresses and ports (the entire 
internal network is addressible from outside), it has no positive 
impact on security the way IPv4's address-overloaded NAT does. 

Regards, 
Bill Herrin 

-- 
William Herrin 
b...@herrin.us 
https://bill.herrin.us/

Re: IPv6 uptake

2024-02-19 Thread Mike Hammett

" We can seriously lose NAT for v6 and not lose 
anything of worth." 

I'm not going to participate in the security conversation, but we do absolutely 
need something to fill the role of NAT in v6. If it's already there or not, I 
don't know. Use case: Joe's Taco Shop. Joe doesn't want a down Internet 
connection to prevent transactions from completing, so he purchases two diverse 
broadband connections, say a cable connection and a DSL connection. When ISP 
fails, traffic will have to exit ISP B. He's not getting a /48, LOA, BGP, etc. 
to do it on his own, he's just going to do simple NAT. 

- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Michael Thomas"  
To: nanog@nanog.org 
Sent: Saturday, February 17, 2024 12:50:46 PM 
Subject: Re: IPv6 uptake 

On 2/17/24 10:26 AM, Owen DeLong via NANOG wrote: 
> 
>> On Feb 16, 2024, at 14:20, Jay R. Ashworth  wrote: 
>> 
>> - Original Message - 
>>> From: "Justin Streiner"  
>>> 4. Getting people to unlearn the "NAT=Security" mindset that we were forced 
>>> to accept in the v4 world. 
>> NAT doesn't "equal" security. 
>> 
>> But it is certainly a *component* of security, placing control of what 
>> internal 
>> nodes are accessible from the outside in the hands of the people inside. 
> Uh, no… no it is not. Stateful inspection (which the kind of NAT (actually 
> NAPT) you are assuming here depends on) is a component of security. You can 
> do stateful inspection without mutilating the header and have all the same 
> security benefits without losing or complicating the audit trail. 

Exactly. As I said elsewhere, the security properties of NAT were a 
post-hoc rationalization. In the mean time, it has taken on its own life 
as if not NAT'ing (but still having stateful firewalls) would end the 
known security universe. We can seriously lose NAT for v6 and not lose 
anything of worth. 

Mike

Re: The Reg does 240/4

2024-02-16 Thread Mike Hammett

Evidence to support Tom's statement: 

https://auctions.ipv4.global/prior-sales 

- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Tom Beecher"  
To: "Brian Knight"  
Cc: nanog@nanog.org 
Sent: Thursday, February 15, 2024 5:31:42 PM 
Subject: Re: The Reg does 240/4 

$/IPv4 address peaked in 2021, and has been declining since. 

On Thu, Feb 15, 2024 at 16:05 Brian Knight via NANOG < nanog@nanog.org > wrote: 

On 2024-02-15 13:10, Lyndon Nerenberg (VE7TFX/VE6BBM) wrote: 
> I've said it before, and I'll say it again: 
> 
> The only thing stopping global IPv6 deployment is 
> Netflix continuing to offer services over IPv4. 
> 
> If Netflix dropped IPv4, you would see IPv6 available *everywhere* 
> within a month. 

As others have noted, and to paraphrase a long-ago quote from this 
mailing list, I'm sure all of Netflix's competitors hope Netflix does 
that. 

I remain hopeful that the climbing price of unique, available IPv4 
addresses eventually forces migration to v6. From my armchair, only 
through economics will this situation will be resolved. 

> --lyndon 

-Brian

Re: The Reg does 240/4

2024-02-16 Thread Mike Hammett

" Does any IPv6 enabled ISP provide PTR records for mail servers?" 

I think people will conflate doing so at ISP-scale and doing so at residential 
hobbiyst scale (and everything in between). One would expect differences in 
outcomes of attempting PTR records in DIA vs. broadband. 

"How does Google handle mail from an IPv6 server?" 

A few people have posted that it works for them, but unless it has changed 
recently, per conversations on the mailop mailing list, Google does not treat 
IPv6 and IPv4 mail the same and that causes non-null issues. 

- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Stephen Satchell"  
To: nanog@nanog.org 
Sent: Wednesday, February 14, 2024 8:25:03 PM 
Subject: Re: The Reg does 240/4 

On 2/14/24 4:23 PM, Tom Samplonius wrote: 
> The best option is what is happening right now: you can’t get new IPv4 
> addresses, so you have to either buy them, or use IPv6. The free market 
> is solving the problem right now. Another solution isn’t needed. 

Really? How many mail servers are up on IPv6? How many legacy mail 
clients can handle IPv6? How many MTA software packages can handle IPv6 
today "right out of the box" without specific configuration? 

Does any IPv6 enabled ISP provide PTR records for mail servers? 

How does Google handle mail from an IPv6 server? 

The Internet is not just the Web.

Re: The Reg does 240/4

2024-02-16 Thread Mike Hammett

" Think how many more sites could have IPv6 capability already if this wasted 
effort had been put into that, instead. " 


My assumption is not many because the people talking about this likely either 
already have or will not deploy IPv6. Those that are willing to deploy IPv6, 
but have not are too busy to be engaging in the conversation. Well, mostly. 





- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Owen DeLong via NANOG"  
To: "Christopher Hawker"  
Cc: "North American Operators' Group"  
Sent: Wednesday, February 14, 2024 11:23:35 AM 
Subject: Re: The Reg does 240/4 



This gift from the bad idea fairy just keeps on giving. You’ve presented your 
case numerous times. The IETF has repeatedly found no consensus for it and yet 
you persist. 


Think how many more sites could have IPv6 capability already if this wasted 
effort had been put into that, instead. 


Owen 





On Feb 13, 2024, at 14:16, Christopher Hawker  wrote: 







Hi Tom, 


We aren't trying to have a debate on this. All we can do is present our case, 
explain our reasons and hope that we can gain a consensus from the community. 


I understand that some peers don't like the idea of this happening and yes we 
understand the technical work behind getting this across the line. It's easy 
enough for us to say "this will never happen" or to put it into the "too hard" 
basket, however, the one thing I can guarantee is that will never happen, if 
nothing is done. 


Let's not think about ourselves for a moment, and think about the potential 
positive impact that this could bring. 


Regards, 
Christopher Hawker 


From: Tom Beecher  
Sent: Wednesday, February 14, 2024 1:23 AM 
To: Christopher Hawker  
Cc: North American Operators' Group ; aus...@lists.ausnog.net 
; Christopher Hawker via sanog ; 
apnic-t...@lists.apnic.net  
Subject: Re: The Reg does 240/4 




Now, we know there's definitely going to be some pushback on this. This won't 
be easy to accomplish and it will take some time. 




It won't ever be 'accomplished' by trying to debate this in the media. 


On Tue, Feb 13, 2024 at 5:05 AM Christopher Hawker < ch...@thesysadmin.au > 
wrote: 





Hello all, 


[Note: I have cross-posted this reply to a thread from NANOG on AusNOG, SANOG 
and APNIC-Talk in order to invite more peers to engage in the discussion on 
their respective forums.] 


Just to shed some light on the article and our involvement... 


Since September 1981, 240/4 has been reserved for future use, see 
https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.xhtml . 
This space has always been reserved for future use and given the global 
shortage of available space for new network operators we feel it is appropriate 
for this space to be reclassified as Unicast space available for delegation by 
IANA/PTI to RIRs on behalf of ICANN. 


At present, the IP space currently available for RIRs to delegate to new 
members is minimal, if any at all. The primary goal of our call for change is 
to afford smaller players who are wanting to enter the industry the opportunity 
to do so without having to shell out the big dollars for space. Although I do 
not agree with IP space being treated as a commodity (as this was not what it 
was intended to be), those who can afford to purchase space may do so and those 
who cannot should be able to obtain space from their respective RIR without 
having to wait over a year in some cases just to obtain space. It's not 
intended to flood the market with resources that can be sold off to the highest 
bidder, and this can very well be a way for network operators to plan to 
properly roll out IPv6. At this point in time, the uptake and implementation of 
IPv6 is far too low (only 37% according to https://stats.labs.apnic.net/ipv6 ) 
for new networks to deploy IPv6 single-stack, meaning that we need to continue 
supporting IPv4 deployments. 


The reallocation of IPv4 space marked as Future Use would not restrict or 
inhibit the deployment of IPv6, if anything, in our view it will help the 
deployment through allowing these networks to service a greater number of 
customers than what a single /24 v4 prefix will allow. Entire regions of an 
economy have the potential to be serviced by a single /23 IPv4 prefix when used 
in conjunction with IPv6 space. 


Now, some have argued that we should not do anything with IPv4 and simply let 
it die out. IPv4 will be around for the foreseeable future and while it is, we 
need to allow new operators to continue deploying networks. It is unfair of us 
to say "Let's all move towards IPv6 and just let IPv4 die" however the reality 
of the situation is that while we continue to treat it as a commodity and allow 
v6 uptake to progress as slowly as it is, we need to continue supporting it v4. 
Some have also

Re: NANOG 90 Attendance?

2024-02-14 Thread Mike Hammett

This seems more ideological and not overly appropriate for NANOG.

-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

- Original Message -
From: Glen A. Pearce 
To: nanog@nanog.org
Sent: Tue, 13 Feb 2024 18:54:59 -0600 (CST)
Subject: Re: NANOG 90 Attendance?

On 11/02/2024 7:56 a.m., Tom Beecher wrote:
> Yup. Post pandemic, the unfortunate hotel situation, and a non-zero 
> number of companies still have tight travel budgets.
>
> It's been slowly creeping back though.

Except we aren't really "post-pandemic" despite the claims that we are.

As long as COVID exists and we don't reach eradication there are going to
be a number of people who might have otherwise participated in these
types of events that will opt not to if they have the choice. Unfortunately
I don't see us eradicating COVID as long as governments succeed at
convincing people that it's not a problem anymore because that's easier
than taking on the people that throw a fit if pandemic control measures
inconvenience them.

Info on what a problem COVID still is from some of the few people that
still write about it:
https://www.okdoomer.io/its-that-bad/
https://www.textise.net/showText.aspx?strURL=https%253A//www.normalcyfugitive.com/p/the-pandemic-isnt-over

Now I know I can't tell people what to do but I can share what I know
if it might help some people.  This pandemic has been one failure after
another with the virus being underestimated at every turn.  I have
thoughts on what I think should be done to get us to eradication
but that gets political and probably too off topic for NANOG.

So back to the topic, because of this situation there will always be some
people opting out of gatherings wherever possible as long as this drags
on and we won't ever be fully back to "normal" as much as many
people try to fake normal.  (I think Jessica at okdoomer had it nailed
when she used the term "cosplaying 2019".)  It may be a little less
obvious as most of the people opting out aren't really being vocal
about why, they just aren't showing up.

As for why the sales people are still showing up...
...detachment from reality has long been a big part of "sales culture".
(Whereas it's not an inherent into "techy culture".)

-- 
Glen A. Pearce
g...@ve4.ca
Network Manager, Webmaster, Bookkeeper, Fashion Model and Shipping Clerk.
Very Eager 4 Tees
http://www.ve4.ca
ARIN Handle VET-17

Re: NANOG 90 Attendance?

2024-02-11 Thread Mike Hammett

I guess I'm not aware of the hotel situation. I saw one was booked, the other 
was not. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Tom Beecher"  
To: "Ryan Hamel"  
Cc: "Mike Hammett" , "nanog"  
Sent: Sunday, February 11, 2024 7:56:28 AM 
Subject: Re: NANOG 90 Attendance? 


Yup. Post pandemic, the unfortunate hotel situation, and a non-zero number of 
companies still have tight travel budgets. 


It's been slowly creeping back though. 


On Sun, Feb 11, 2024 at 8:44 AM Ryan Hamel < r...@rkhtech.org > wrote: 




Mike, 


The numbers have not bounced back to pre-pandemic levels, and it doesn't help 
that NANOG 90 has had some hotel issues. 


Ryan 




From: NANOG  on behalf of Mike 
Hammett < na...@ics-il.net > 
Sent: Sunday, February 11, 2024 5:31:02 AM 
To: nanog < nanog@nanog.org > 
Subject: NANOG 90 Attendance? 



Caution: This is an external email and may be malicious. Please take care when 
clicking links or opening attachments. 


I haven't been to a NANOG meeting in a while. While going through the attendee 
list for NANOG 90 to try to book meetings with people, I noticed a lack of (or 
extremely minimal) attendance by several organizations that have traditionally 
had several employees attend. I've also noticed that some organizations I had 
an interest in were only sending sales people, not technical people. 


How long has this been a thing? I remember when I attended years ago that there 
simply wasn't enough time to meet with technical people from all of the 
organizations I wanted to meet with. Now the calendar is looking a bit dry. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com

NANOG 90 Attendance?

2024-02-11 Thread Mike Hammett

I haven't been to a NANOG meeting in a while. While going through the attendee 
list for NANOG 90 to try to book meetings with people, I noticed a lack of (or 
extremely minimal) attendance by several organizations that have traditionally 
had several employees attend. I've also noticed that some organizations I had 
an interest in were only sending sales people, not technical people. 


How long has this been a thing? I remember when I attended years ago that there 
simply wasn't enough time to meet with technical people from all of the 
organizations I wanted to meet with. Now the calendar is looking a bit dry. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com

Zayo Network Map

2024-01-25 Thread Mike Hammett

Is the only public source for Zayo's network map their PDF that's behind a sign 
up form? Tranzact is still running, but doesn't display anything when you turn 
on the fiber layer. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP

Re: Mail to Microsoft being falsely marked as spam/bulk

2024-01-21 Thread Mike Hammett

https://www.mailop.org/ 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Christopher Hawker"  
To: "nanog"  
Sent: Saturday, January 20, 2024 5:07:39 AM 
Subject: Mail to Microsoft being falsely marked as spam/bulk 


Hi folks, 


I'm having an issue with Microsoft email filtering, where it is flagging 
messages from a private mail server as spam, with an SCL of 9 and a BCL of 7, 
resulting in it being sent to the Junk folder. 


We are not seeing this issue with Google's mail environment (being filtered to 
junk), confirmed that SPF, DMARC and DKIM are all correct and valid and that 
the domain and IP addresses are not on any block lists. 


If there is anyone from Microsoft around that can look into mail issues, could 
you please reach out to me off-list? Or if anyone has any ideas/suggestions as 
to how to resolve this, I'd be thankful to hear from you. 


Thanks, 
Christopher Hawker

Re: "Hypothetical" Datacenter Overheating

2024-01-18 Thread Mike Hammett






- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Tom Beecher"  
To: "Mike Hammett"  
Cc: sro...@ronan-online.com, "NANOG"  
Sent: Thursday, January 18, 2024 9:19:09 AM 
Subject: Re: "Hypothetical" Datacenter Overheating 




Well right, which came well after the question was posited here. 




Wasn't poo pooing the question, just sharing the information as I didn't see 
that cited otherwise in this thread. 


On Thu, Jan 18, 2024 at 10:15 AM Mike Hammett < na...@ics-il.net > wrote: 





Well right, which came well after the question was posited here. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 



From: "Tom Beecher" < beec...@beecher.cc > 
To: "Mike Hammett" < na...@ics-il.net > 
Cc: sro...@ronan-online.com , "NANOG" < nanog@nanog.org > 
Sent: Thursday, January 18, 2024 9:00:34 AM 
Subject: Re: "Hypothetical" Datacenter Overheating 




and none in the other two facilities you operate in that same building had any 
failures. 




Quoting directly from their outage ticket updates : 



CH2 does not have chillers, cooling arrangement is DX CRACs manufactured by 
another company. CH3 has Smart chillers but are water cooled not air cooled so 
not susceptible to cold ambient air temps as they are indoor chillers. 







On Mon, Jan 15, 2024 at 10:19 AM Mike Hammett < na...@ics-il.net > wrote: 





and none in the other two facilities you operate in that same building had any 
failures. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 



From: sro...@ronan-online.com 
To: "Mike Hammett" < na...@ics-il.net > 
Cc: "NANOG" < nanog@nanog.org > 
Sent: Monday, January 15, 2024 9:14:49 AM 
Subject: Re: "Hypothetical" Datacenter Overheating 



I’m more interested in how you lose six chillers all at once. 


Shane 



On Jan 15, 2024, at 9:11 AM, Mike Hammett < na...@ics-il.net > wrote: 







Let's say that hypothetically, a datacenter you're in had a cooling failure and 
escalated to an average of 120 degrees before mitigations started having an 
effect. What are normal QA procedures on your behalf? What is the facility 
likely to be doing? What should be expected in the aftermath? 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP

Re: "Hypothetical" Datacenter Overheating

2024-01-18 Thread Mike Hammett

Well right, which came well after the question was posited here. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Tom Beecher"  
To: "Mike Hammett"  
Cc: sro...@ronan-online.com, "NANOG"  
Sent: Thursday, January 18, 2024 9:00:34 AM 
Subject: Re: "Hypothetical" Datacenter Overheating 




and none in the other two facilities you operate in that same building had any 
failures. 




Quoting directly from their outage ticket updates : 



CH2 does not have chillers, cooling arrangement is DX CRACs manufactured by 
another company. CH3 has Smart chillers but are water cooled not air cooled so 
not susceptible to cold ambient air temps as they are indoor chillers. 







On Mon, Jan 15, 2024 at 10:19 AM Mike Hammett < na...@ics-il.net > wrote: 





and none in the other two facilities you operate in that same building had any 
failures. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 



From: sro...@ronan-online.com 
To: "Mike Hammett" < na...@ics-il.net > 
Cc: "NANOG" < nanog@nanog.org > 
Sent: Monday, January 15, 2024 9:14:49 AM 
Subject: Re: "Hypothetical" Datacenter Overheating 



I’m more interested in how you lose six chillers all at once. 


Shane 



On Jan 15, 2024, at 9:11 AM, Mike Hammett < na...@ics-il.net > wrote: 







Let's say that hypothetically, a datacenter you're in had a cooling failure and 
escalated to an average of 120 degrees before mitigations started having an 
effect. What are normal QA procedures on your behalf? What is the facility 
likely to be doing? What should be expected in the aftermath? 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP

Re: Shared cache servers on an island's IXP

2024-01-18 Thread Mike Hammett

Some will work directly on the IX via BGP. Others have to go behind a member of 
the IX. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Jérôme Nicolle"  
To: "Mehmet"  
Cc: nanog@nanog.org 
Sent: Thursday, January 18, 2024 8:38:31 AM 
Subject: Re: Shared cache servers on an island's IXP 

Hello Mehmet, 

Le 18/01/2024 à 12:58, Mehmet a écrit : 
> VMs are no go for big content companies except Microsoft. You can run 
> Microsoft CDN on VM but rest of the content will need to be cached. You 
> can actually install this yourself 

I've already read most docs for caching servers provided by major 
actors. What I'm mostly concerned about is their ability to peer with 
multiple AS on the local IXP, as to not over-replicate them. 

Should I establish a dedicated network peering on the IXP ? Or will they 
come with their own ASNs ? The peering case is quite not documented on 
publicly available specs, if even possible. 

> Depending on how much traffic do you have , you may be able to get 
> facebook, youtube, netflix caches i think minimum bw requirement changes 
> per region 

Those I'm nearly sure I could get, if I can pool caches amongst ISPs. 
The current constraints are issues to any content provider, not just for 
local ISPs. 

Best regards, 

-- 
Jérôme Nicolle 
+33 6 19 31 27 14

Re: "Hypothetical" Datacenter Overheating

Someone I talked to while on scene today said their area got to 130 and cooked 
two core routers. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Mike Hammett"  
To: "NANOG"  
Sent: Monday, January 15, 2024 8:08:25 AM 
Subject: "Hypothetical" Datacenter Overheating 


Let's say that hypothetically, a datacenter you're in had a cooling failure and 
escalated to an average of 120 degrees before mitigations started having an 
effect. What are normal QA procedures on your behalf? What is the facility 
likely to be doing? What should be expected in the aftermath? 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP

Re: "Hypothetical" Datacenter Overheating

and none in the other two facilities you operate in that same building had any 
failures. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: sro...@ronan-online.com 
To: "Mike Hammett"  
Cc: "NANOG"  
Sent: Monday, January 15, 2024 9:14:49 AM 
Subject: Re: "Hypothetical" Datacenter Overheating 



I’m more interested in how you lose six chillers all at once. 


Shane 



On Jan 15, 2024, at 9:11 AM, Mike Hammett  wrote: 







Let's say that hypothetically, a datacenter you're in had a cooling failure and 
escalated to an average of 120 degrees before mitigations started having an 
effect. What are normal QA procedures on your behalf? What is the facility 
likely to be doing? What should be expected in the aftermath? 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP

Re: "Hypothetical" Datacenter Overheating

Coincidence indeed ;-) 

- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Clayton Zekelman"  
To: "Mike Hammett" , "NANOG"  
Sent: Monday, January 15, 2024 8:23:37 AM 
Subject: Re: "Hypothetical" Datacenter Overheating 

At 09:08 AM 2024-01-15, Mike Hammett wrote: 
>Let's say that hypothetically, a datacenter you're in had a cooling 
>failure and escalated to an average of 120 degrees before 
>mitigations started having an effect. What are normal QA procedures 
>on your behalf? What is the facility likely to be doing? 
>What should be expected in the aftermath? 

One would hope they would have had disaster recovery plans to bring 
in outside cold air, and have executed on it quickly, rather than 
hoping the chillers got repaired. 

All our owned facilities have large outside air intakes, automatic 
dampers and air mixing chambers in case of mechanical cooling 
failure, because cooling systems are often not designed to run well 
in extreme cold. All of these can be manually run incase of controls 
failure, but people tell me I'm a little obsessive over backup plans 
for backup plans. 

You will start to see premature failure of equipment over the coming 
weeks/months/years. 

Coincidentally, we have some gear in a data centre in the Chicago 
area that is experiencing that sort of issue right now... :-(

"Hypothetical" Datacenter Overheating

Let's say that hypothetically, a datacenter you're in had a cooling failure and 
escalated to an average of 120 degrees before mitigations started having an 
effect. What are normal QA procedures on your behalf? What is the facility 
likely to be doing? What should be expected in the aftermath? 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP

Re: Vint Cerf Re: Backward Compatibility Re: IPv4 address block

2024-01-13 Thread Mike Lyon

Some of us still use pine…-MikeOn Jan 13, 2024, at 12:57, Abraham Y. Chen  wrote:

  

  
  
Hi, Gary:
0)    My apologies!
1)    I thought that I am one of only a few who
insist on using the most basic tools that get the job done, such
preferring hand tools than power tools if possible. I believed
that the ThunderBird eMail client software was pretty basic.
Your message just reminds me that there are colleagues here
probably still using plain text editors for eMail?
I shall keep this in mind for my future eMails.
Regards,

  
Abe (2024-01-13 15:54)







On 2024-01-13 14:45, Gary E. Miller
  wrote:


  Yo Abraham!

On Sat, 13 Jan 2024 07:35:09 -0500
"Abraham Y. Chen"  wrote:


  
     FYI - Please see the below copy of a partial eMail thread. Bold, 
red colored and Italicized letters are to focus on the topic.

  
  Uh, you realize many of us never see your red or italics?

RGDS
GARY
---
Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
	g...@rellim.com  Tel:+1 541 382 8588

	Veritas liberabit vos. -- Quid est veritas?
"If you can't measure it, you can't improve it." - Lord Kelvin




  Virus-free.www.avast.com

Re: Stealthy Overlay Network Re: 202401100645.AYC Re: IPv4 address block

2024-01-12 Thread Mike Hammett

I'm not talking about global, public use, only private use. 

- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Tom Beecher"  
To: "Mike Hammett"  
Cc: "Ryan Hamel" , "Abraham Y. Chen" , 
nanog@nanog.org 
Sent: Friday, January 12, 2024 2:06:32 PM 
Subject: Re: Stealthy Overlay Network Re: 202401100645.AYC Re: IPv4 address 
block 

You don't need everything in the world to support it, just the things "you" 
use. 

You run an ISP, let me posit something. 

Stipulate your entire network infra, services, and applications support 240/4, 
and that it's approved for global , public use tomorrow. Some company gets a 
block in there, stands up some website. Here are some absolutely plausible 
scenarios that you might have to deal with. 

- Some of your customers are running operating systems / network gear that 
doesn't support 240/4. 
- Some of your customers may be using 3rd party DNS resolvers that don't 
support 240/4. 
- Some network in between you and the dest missed a few bogon ACLs , dropping 
your customer's traffic. 

All of this becomes support issues you have to deal with. 

On Fri, Jan 12, 2024 at 2:21 PM Mike Hammett < na...@ics-il.net > wrote: 

I wouldn't say it's unknowable, just that no one with a sufficient enough 
interest in the cause has been loud enough with the research they've done, 
assuming some research has been done.. 

You don't need everything in the world to support it, just the things "you" 
use. 

- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

From: "Tom Beecher" < beec...@beecher.cc > 
To: "Mike Hammett" < na...@ics-il.net > 
Cc: "Ryan Hamel" < r...@rkhtech.org >, "Abraham Y. Chen" < ayc...@alum.mit.edu 
>, nanog@nanog.org 
Sent: Friday, January 12, 2024 1:16:53 PM 
Subject: Re: Stealthy Overlay Network Re: 202401100645.AYC Re: IPv4 address 
block 

How far are we from that, in reality? I don't have any intention on using the 
space, but I would like to put some definition to this boogey man. 

It's unknowable really. 

Lots of network software works just fine today with it. Some don't. To my 
knowledge some NOS vendors have outright refused to support 240/4 unless it's 
reclassified. Beyond network equipment, there is an unknowable number of 
software packages , drivers, etc out in the world which 240/4 is still 
hardcoded not to work. It's been unfortunate to see this fact handwaved away in 
many discussions on the subject. 

The Mirai worm surfaced in 2016. The software vulnerabilities used in its 
attack vectors are still unpatched and present in massive numbers across the 
internet; there are countless variants that still use the same methods, 8 years 
later. Other vulnerabilities still exist after multiple decades. But we somehow 
think devices will be patched to support 240/4 quickly? 

It's just unrealistic. 

On Fri, Jan 12, 2024 at 1:03 PM Mike Hammett < na...@ics-il.net > wrote: 

" every networking vendor, hardware vendor, and OS vendor" 

How far are we from that, in reality? I don't have any intention on using the 
space, but I would like to put some definition to this boogey man. 

- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

From: "Ryan Hamel" < r...@rkhtech.org > 
To: "Abraham Y. Chen" < ayc...@avinta.com >, "Vasilenko Eduard" < 
vasilenko.edu...@huawei.com > 
Cc: "Abraham Y. Chen" < ayc...@alum.mit.edu >, nanog@nanog.org 
Sent: Thursday, January 11, 2024 11:04:31 PM 
Subject: Re: Stealthy Overlay Network Re: 202401100645.AYC Re: IPv4 address 
block 

Abraham, 

You may not need permission from the IETF, but you effectively need it from 
every networking vendor, hardware vendor, and OS vendor. If you do not have buy 
in from key stakeholders, it's dead-on arrival. 

Ryan 

From: NANOG  on behalf of Abraham 
Y. Chen < ayc...@avinta.com > 
Sent: Thursday, January 11, 2024 6:38:52 PM 
To: Vasilenko Eduard < vasilenko.edu...@huawei.com > 
Cc: Chen, Abraham Y. < ayc...@alum.mit.edu >; nanog@nanog.org < nanog@nanog.org 
> 
Subject: Stealthy Overlay Network Re: 202401100645.AYC Re: IPv4 address block 

Caution: This is an external email and may be malicious. Please take care when 
clicking links or opening attachments. 

Hi, Vasilenko: 

1) ... These “ multi-national conglo ” has enough influence on the IETF to not 
permit it. ": 

As classified by Vint Cerf, 240/4 enabled EzIP is an overlay network that may 
be deployed stealthily (just like the events reported by the RIPE-LAB). So, 
EzIP deployment does not need permission from the IETF. 

Regards, 

Abe

Re: Stealthy Overlay Network Re: 202401100645.AYC Re: IPv4 address block

2024-01-12 Thread Mike Hammett

I wouldn't say it's unknowable, just that no one with a sufficient enough 
interest in the cause has been loud enough with the research they've done, 
assuming some research has been done.. 

You don't need everything in the world to support it, just the things "you" 
use. 

- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Tom Beecher"  
To: "Mike Hammett"  
Cc: "Ryan Hamel" , "Abraham Y. Chen" , 
nanog@nanog.org 
Sent: Friday, January 12, 2024 1:16:53 PM 
Subject: Re: Stealthy Overlay Network Re: 202401100645.AYC Re: IPv4 address 
block 

How far are we from that, in reality? I don't have any intention on using the 
space, but I would like to put some definition to this boogey man. 

It's unknowable really. 

Lots of network software works just fine today with it. Some don't. To my 
knowledge some NOS vendors have outright refused to support 240/4 unless it's 
reclassified. Beyond network equipment, there is an unknowable number of 
software packages , drivers, etc out in the world which 240/4 is still 
hardcoded not to work. It's been unfortunate to see this fact handwaved away in 
many discussions on the subject. 

The Mirai worm surfaced in 2016. The software vulnerabilities used in its 
attack vectors are still unpatched and present in massive numbers across the 
internet; there are countless variants that still use the same methods, 8 years 
later. Other vulnerabilities still exist after multiple decades. But we somehow 
think devices will be patched to support 240/4 quickly? 

It's just unrealistic. 

On Fri, Jan 12, 2024 at 1:03 PM Mike Hammett < na...@ics-il.net > wrote: 

" every networking vendor, hardware vendor, and OS vendor" 

How far are we from that, in reality? I don't have any intention on using the 
space, but I would like to put some definition to this boogey man. 

- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

From: "Ryan Hamel" < r...@rkhtech.org > 
To: "Abraham Y. Chen" < ayc...@avinta.com >, "Vasilenko Eduard" < 
vasilenko.edu...@huawei.com > 
Cc: "Abraham Y. Chen" < ayc...@alum.mit.edu >, nanog@nanog.org 
Sent: Thursday, January 11, 2024 11:04:31 PM 
Subject: Re: Stealthy Overlay Network Re: 202401100645.AYC Re: IPv4 address 
block 

Abraham, 

You may not need permission from the IETF, but you effectively need it from 
every networking vendor, hardware vendor, and OS vendor. If you do not have buy 
in from key stakeholders, it's dead-on arrival. 

Ryan 

From: NANOG  on behalf of Abraham 
Y. Chen < ayc...@avinta.com > 
Sent: Thursday, January 11, 2024 6:38:52 PM 
To: Vasilenko Eduard < vasilenko.edu...@huawei.com > 
Cc: Chen, Abraham Y. < ayc...@alum.mit.edu >; nanog@nanog.org < nanog@nanog.org 
> 
Subject: Stealthy Overlay Network Re: 202401100645.AYC Re: IPv4 address block 

Caution: This is an external email and may be malicious. Please take care when 
clicking links or opening attachments. 

Hi, Vasilenko: 

1) ... These “ multi-national conglo ” has enough influence on the IETF to not 
permit it. ": 

As classified by Vint Cerf, 240/4 enabled EzIP is an overlay network that may 
be deployed stealthily (just like the events reported by the RIPE-LAB). So, 
EzIP deployment does not need permission from the IETF. 

Regards, 

Abe (2024-01-11 21:38 EST) 

On 2024-01-11 01:17, Vasilenko Eduard wrote: 

> It has been known that multi-national conglomerates have been using it 
> without announcement. 
This is an assurance that 240/4 would never be permitted for Public Internet. 
These “ multi-national conglo ” has enough influence on the IETF to not permit 
it. 
Ed/ 

From: NANOG [ mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org ] On 
Behalf Of Abraham Y. Chen 
Sent: Wednesday, January 10, 2024 3:35 PM 
To: KARIM MEKKAOUI  
Cc: nanog@nanog.org ; Chen, Abraham Y.  
Subject: 202401100645.AYC Re: IPv4 address block 
Importance: High 

Hi, Karim: 

1) If you have control of your own equipment (I presume that your business 
includes IAP - Internet Access Provider, since you are asking to buy IPv4 
blocks.), you can get a large block of reserved IPv4 address for free by 
disabling the program codes in your current facility that has been disabling 
the use of 240/4 netblock. Please have a look at the below whitepaper. Utilized 
according to the outlined disciplines, this is a practically unlimited 
resources. It has been known that multi-national conglomerates have been using 
it without announcement. So, you can do so stealthily according to the proposed 
mechanism which establishes uniform practices, just as well. 

https://www.avinta.com/phoenix-1/home/RevampTheInternet.pdf 

2) Being an unortho

Re: Stealthy Overlay Network Re: 202401100645.AYC Re: IPv4 address block

2024-01-12 Thread Mike Hammett

" every networking vendor, hardware vendor, and OS vendor" 

How far are we from that, in reality? I don't have any intention on using the 
space, but I would like to put some definition to this boogey man. 

- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Ryan Hamel"  
To: "Abraham Y. Chen" , "Vasilenko Eduard" 

Cc: "Abraham Y. Chen" , nanog@nanog.org 
Sent: Thursday, January 11, 2024 11:04:31 PM 
Subject: Re: Stealthy Overlay Network Re: 202401100645.AYC Re: IPv4 address 
block 

Abraham, 

You may not need permission from the IETF, but you effectively need it from 
every networking vendor, hardware vendor, and OS vendor. If you do not have buy 
in from key stakeholders, it's dead-on arrival. 

Ryan 

From: NANOG  on behalf of Abraham Y. 
Chen  
Sent: Thursday, January 11, 2024 6:38:52 PM 
To: Vasilenko Eduard  
Cc: Chen, Abraham Y. ; nanog@nanog.org  
Subject: Stealthy Overlay Network Re: 202401100645.AYC Re: IPv4 address block 

Caution: This is an external email and may be malicious. Please take care when 
clicking links or opening attachments. 

Hi, Vasilenko: 

1) ... These “ multi-national conglo ” has enough influence on the IETF to not 
permit it. ": 

As classified by Vint Cerf, 240/4 enabled EzIP is an overlay network that may 
be deployed stealthily (just like the events reported by the RIPE-LAB). So, 
EzIP deployment does not need permission from the IETF. 

Regards, 

Abe (2024-01-11 21:38 EST) 

On 2024-01-11 01:17, Vasilenko Eduard wrote: 

> It has been known that multi-national conglomerates have been using it 
> without announcement. 
This is an assurance that 240/4 would never be permitted for Public Internet. 
These “ multi-national conglo ” has enough influence on the IETF to not permit 
it. 
Ed/ 

From: NANOG [ mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org ] On 
Behalf Of Abraham Y. Chen 
Sent: Wednesday, January 10, 2024 3:35 PM 
To: KARIM MEKKAOUI  
Cc: nanog@nanog.org ; Chen, Abraham Y.  
Subject: 202401100645.AYC Re: IPv4 address block 
Importance: High 

Hi, Karim: 

1) If you have control of your own equipment (I presume that your business 
includes IAP - Internet Access Provider, since you are asking to buy IPv4 
blocks.), you can get a large block of reserved IPv4 address for free by 
disabling the program codes in your current facility that has been disabling 
the use of 240/4 netblock. Please have a look at the below whitepaper. Utilized 
according to the outlined disciplines, this is a practically unlimited 
resources. It has been known that multi-national conglomerates have been using 
it without announcement. So, you can do so stealthily according to the proposed 
mechanism which establishes uniform practices, just as well. 

https://www.avinta.com/phoenix-1/home/RevampTheInternet.pdf 

2) Being an unorthodox solution, if not controversial, please follow up with me 
offline. Unless, other NANOGers express their interests. 

Regards, 

Abe (2024-01-10 07:34 EST) 

On 2024-01-07 22:46, KARIM MEKKAOUI wrote: 

Hi Nanog Community 

Any idea please on the best way to buy IPv4 blocs and what is the price? 

Thank you 

KARIM 

Virus-free. www.avast.com

Re: Sufficient Buffer Sizes

2024-01-02 Thread Mike Hammett

I'm assuming that modern queuing mechanisms aren't going to be viable in 
switches until which time they're baked into the silicon. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Dave Taht"  
To: "William Herrin"  
Cc: "Mike Hammett" , "NANOG"  
Sent: Tuesday, January 2, 2024 6:02:27 PM 
Subject: Re: Sufficient Buffer Sizes 

Hoo, boy. This is now such an old debate that I do not know where to 
start anymore. 

I am of the firm opinion nowadays that if you are buffering more than 
a few ms at these enormous speeds, you are doing it wrong, and 
regardless https://arxiv.org/abs/2109.11693 seems to hold as for 
highly multiplexed traffic. 

My outside number for a FIFO buffer in the modern CDN´d world is a 
mere 30ms at lower speeds which allows for good gaming and 
videoconferencing experiences, and good performance with modern paced 
transports (in general linux now does packet pacing across all 
congestion controls) and with a good head drop AQM and FQ algo, far, 
far less buffering is feasible across the board. 

https://blog.cerowrt.org/post/juniper/ 

But regrettably not available at 100Gbit yet (tho libreqos is coming close) 

On Tue, Jan 2, 2024 at 6:22 PM William Herrin  wrote: 
> 
> On Tue, Jan 2, 2024 at 3:02 PM Mike Hammett  wrote: 
> > While attempting to ascertain how big of switch buffers I needed in a 100G 
> > switch, I rediscovered this article where I first learned about switch 
> > buffers. 
> > 
> > https://fasterdata.es.net/network-tuning/router-switch-buffer-size-issues/#:~:text=Optimum%20Buffer%20Size=The%20general%20rule%20of%20thumb,1G%20host%20across%20the%20WAN.
> >  
> > 
> > It suggests that 60 meg is what you need at 10G. Is that per interface? 
> > Would it be linear in that I would need 600 meg at 100G? 
> > 
> > Some 100G switches I was looking at only had 36 megs, so that's 
> > insufficient either way you look at it. 
> 
> Hi Mike, 
> 
> My thoughts: 
> 
> 1. 50 ms is -way- too much buffer. A couple links like that in the 
> path and the user will suffer jitter in excess of 100ms which is 
> incredibly disruptive for interactive applications. 
> 
> 2. The article discussed how much buffer to apply to the -slower- 
> interfaces, not the faster ones, the idea being that data entering 
> from the faster interfaces could otherwise overwhelm the slower ones 
> resulting in needless retransmission and head-end blocking. Are the 
> 100G interfaces on your switch the -slower- ones? 
> 
> 
> I don't know the best number, but I suspect the speed at which packets 
> clear an interface is probably a factor in the equation, so that the 
> reasonable buffer depth in ms when a packet clears in 1ms is probably 
> different than the reasonable buffer depth when a packet clears in 1 
> us. 
> 
> Regards, 
> Bill Herrin 
> 
> 
> 
> -- 
> William Herrin 
> b...@herrin.us 
> https://bill.herrin.us/ 



-- 
40 years of net history, a couple songs: 
https://www.youtube.com/watch?v=D9RGX6QFm5E 
Dave Täht CSO, LibreQos

Sufficient Buffer Sizes

2024-01-02 Thread Mike Hammett

While attempting to ascertain how big of switch buffers I needed in a 100G 
switch, I rediscovered this article where I first learned about switch buffers.

https://fasterdata.es.net/network-tuning/router-switch-buffer-size-issues/#:~:text=Optimum%20Buffer%20Size=The%20general%20rule%20of%20thumb,1G%20host%20across%20the%20WAN.

It suggests that 60 meg is what you need at 10G. Is that per interface? Would 
it be linear in that I would need 600 meg at 100G?


Some 100G switches I was looking at only had 36 megs, so that's insufficient 
either way you look at it.


-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

Re: PCH Contact?

2023-12-19 Thread Mike Hammett

I responded offlist. 

- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Bill Woodcock"  
To: "Mike Hammett"  
Cc: "NANOG"  
Sent: Tuesday, December 19, 2023 8:52:35 AM 
Subject: Re: PCH Contact? 

> On Dec 19, 2023, at 15:22, Mike Hammett  wrote: 
> Has something happened at PCH to reduce their responsiveness? They used to be 
> really good, but it's been tough hearing back from them this year. 

Offhand, I’m not able to see any tickets with your name on them… Can you give 
me a specific ticket number that you’re referencing? 

As always, we can be reached on n...@pch.net (general) or d...@pch.net (DNS 
services) or peer...@pch.net (interconnection). The ticket queue isn’t longer 
than usual, we’re handling an average of 60 tickets per day right now, not 
particularly different than this time last year, or the year before. Happy to 
help however we can. 

-Bill

PCH Contact?

2023-12-19 Thread Mike Hammett

Has something happened at PCH to reduce their responsiveness? They used to be 
really good, but it's been tough hearing back from them this year. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com

Re: Interesting Ali Express web server behavior...

2023-12-09 Thread Mike Lyon

I notice a weird issue like this with Alibaba when i try to use my Comcast 
connection. Turn my wifi off and now it works flawlessly.

Are you using your comcast connection?

-Mike

> On Dec 9, 2023, at 21:17, Stephane Bortzmeyer  wrote:
> 
> On Sat, Dec 09, 2023 at 09:55:31PM -0800,
> Owen DeLong via NANOG  wrote
> a message of 1136 lines which said:
> 
>> But why would AliExpress be redirecting to DDN space? Is this
>> legitimate? Ali hoping to get away with squatting, or something
>> else?
> 
> No idea. The IP address does not reply to HTTP requests, anyway. A
> practical joke?
> 
> Note that this redirection takes places only when there is no
> User-Agent field. If you say 'User-Agent: Mozilla', you get a proper
> redirection, in my case to https://fr.aliexpress.com/.

"Lit" Buildings

2023-12-07 Thread Mike Hammett

For those of you who list your network (usually wireline, but sometimes 
wireless) with third parties, are you supplying just the KMZ or lit buildings 
as well? If lit buildings, are you including residential? How are you defining 
near-net? 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP

Re: What are these Google IPs hammering on my DNS server?

2023-12-03 Thread Mike Hammett

Before when I had my honeypot firewall off everything that crossed it's 
threshold, I ended up blocking myself from a variety of authoritative servers, 
including Google's. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "John Levine"  
To: nanog@nanog.org 
Sent: Sunday, December 3, 2023 12:48:11 PM 
Subject: What are these Google IPs hammering on my DNS server? 

At contacts.abuse.net, I have a little stunt DNS server that provides domain 
contact info, e.g.: 

$ host -t txt comcast.net.contacts.abuse.net 
comcast.net.contacts.abuse.net descriptive text "ab...@comcast.net" 

$ host -t hinfo comcast.net.contacts.abuse.net 
comcast.net.contacts.abuse.net host information "lookup" "comcast.net" 

Every once in a while someone decides to look up every domain in the 
world and DoS'es it until I update my packet filters. This week it's 
been this set of IPs that belong to Google. I don't think they're 
8.8.8.8. Any idea what they are? Random Google Cloud customers? A 
secret DNS mapping project? 

172.253.1.133 
172.253.206.36 
172.253.1.130 
172.253.206.37 
172.253.13.196 
172.253.255.36 
172.253.13.197 
172.253.1.131 
172.253.255.35 
172.253.255.37 
172.253.1.132 
172.253.13.193 
172.253.1.129 
172.253.255.33 
172.253.206.35 
172.253.255.34 
172.253.206.33 
172.253.206.34 
172.253.13.194 
172.253.13.195 
172.71.125.63 
172.71.117.60 
172.71.133.51 

R's, 
John

Re: sigs wanted for a response to the fcc's NOI for faster broadband speeds

2023-12-01 Thread Mike Hammett

It would be better to keep the government out of it altogether, but that has 
little chance of happening. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Tom Beecher"  
To: "Dave Taht"  
Cc: "NANOG" , "NZNOG" , 
""  
Sent: Friday, December 1, 2023 6:34:49 PM 
Subject: Re: sigs wanted for a response to the fcc's NOI for faster broadband 
speeds 


Trying to put technical requirements like this into law and public policy is an 
extremely terrible idea. This letter should never be sent. 


The regulatory agencies today don't have the manpower or expertise to 
adequately enforce the more generic broadband deployment rules. What fantasy 
world exists where they have the manpower or expertise to monitor for and 
enforce something like this? Hell, there are constant , legitimate technical 
discussions between experts on HOW to *properly* monitor things just like this. 
We want to have someone at the FCC deciding what that should look like? 


4.4 What the hell? The regulatory agencies should be allocating spectrum, and 
making sure it's not used improperly with the rules of allocation. Making it 
work 'better' is OUR job in the technical community. Not an FCC rulemaker. 


4.8 There are zero scenarios there should ever be regulatory rules about device 
software. In our space (non-ISP) , TONS of people run older versions of vendor 
code. Why? The shit DOESN'T WORK RIGHT YET and it causes other problems. You 
suggest that regulatory bodies be involved in dictating anything about this? 


The bufferbloat work belongs in the technical area, full stop. Nowhere near 
regulatory / legal. 


On Thu, Nov 30, 2023 at 7:57 PM Dave Taht < dave.t...@gmail.com > wrote: 


Over here: 

https://docs.google.com/document/d/19ADByjakzQXCj9Re_pUvrb5Qe5OK-QmhlYRLMBY4vH4/edit
 

Us bufferbloat folk have been putting together a response to the FCC's 
NOI (notice of inquiry) asking for feedback as to increasing the 
broadband speeds beyond 100/20 Mbit. 

"Calls for further bandwidth increases are analogous to calling for 
cars to have top speeds of 100, 200, or 500 miles per hour. Without 
calling also for better airbags, bumpers, brakes, or steering wheels, 
(or roads designed to minimize travel delay), these initiatives will 
fail (and are failing) to meet the needs of present and future users 
of the internet." 

Comments (and cites) welcomed also! The text is still somewhat in flux... 


-- 
:( My old R campus is up for sale: https://tinyurl.com/yurtlab 
Dave Täht CSO, LibreQos

Re: sigs wanted for a response to the fcc's NOI for faster broadband speeds

2023-12-01 Thread Mike Hammett

The FCC is currently posturing to feel relevant. While they're in one of these 
modes, you're not going to stop them, but you might be able to redirect them on 
a better path. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Tom Mitchell"  
To: "Dave Taht"  
Cc: "NANOG" , "NZNOG" , 
""  
Sent: Friday, December 1, 2023 11:38:10 AM 
Subject: Re: sigs wanted for a response to the fcc's NOI for faster broadband 
speeds 



Not sure we need the FCC telling us how to build products or run networks. Seat 
belts are life-or-death, but bufferbloat is rarely fatal ;-) Let it be a point 
of differentiation. 




-- Tom 



On Thu, Nov 30, 2023 at 4:56 PM Dave Taht < dave.t...@gmail.com > wrote: 


Over here: 

https://docs.google.com/document/d/19ADByjakzQXCj9Re_pUvrb5Qe5OK-QmhlYRLMBY4vH4/edit
 

Us bufferbloat folk have been putting together a response to the FCC's 
NOI (notice of inquiry) asking for feedback as to increasing the 
broadband speeds beyond 100/20 Mbit. 

"Calls for further bandwidth increases are analogous to calling for 
cars to have top speeds of 100, 200, or 500 miles per hour. Without 
calling also for better airbags, bumpers, brakes, or steering wheels, 
(or roads designed to minimize travel delay), these initiatives will 
fail (and are failing) to meet the needs of present and future users 
of the internet." 

Comments (and cites) welcomed also! The text is still somewhat in flux... 


-- 
:( My old R campus is up for sale: https://tinyurl.com/yurtlab 
Dave Täht CSO, LibreQos

Re: Infrapedia

2023-11-29 Thread Mike Hammett

Ah, it looks like that project has died. 


"This email is not actively being monitored as Infrapedia project is sunsetted 
since 2021." 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Mehmet"  
To: "Mike Hammett"  
Cc: "NANOG"  
Sent: Wednesday, November 29, 2023 7:07:01 AM 
Subject: Re: Infrapedia 



You can email ad...@infrapedia.com 





On Wed, Nov 29, 2023 at 07:54 Mike Hammett < na...@ics-il.net > wrote: 




Does anyone have contact information for Infrapedia? Mehmet is no longer there 
and the contact form on their web site is 404. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP

Re: Infrapedia

2023-11-29 Thread Mike Hammett

Thanks. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Mehmet"  
To: "Mike Hammett"  
Cc: "NANOG"  
Sent: Wednesday, November 29, 2023 7:07:01 AM 
Subject: Re: Infrapedia 



You can email ad...@infrapedia.com 





On Wed, Nov 29, 2023 at 07:54 Mike Hammett < na...@ics-il.net > wrote: 




Does anyone have contact information for Infrapedia? Mehmet is no longer there 
and the contact form on their web site is 404. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP

Infrapedia

2023-11-29 Thread Mike Hammett

Does anyone have contact information for Infrapedia? Mehmet is no longer there 
and the contact form on their web site is 404. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP

Re: Outside plant - prewire customer demarc preference

2023-11-28 Thread Mike Hammett

Why not just use SCH40 PVC sticks? Everywhere stocks that in copious levels. 

- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Brandon Martin"  
To: nanog@nanog.org 
Sent: Tuesday, November 28, 2023 9:27:58 AM 
Subject: Re: Outside plant - prewire customer demarc preference 

On 11/27/23 18:52, owen--- via NANOG wrote: 
> Why would 1” be significantly harder to get than 3/4”? Both in EMT and PVC, 
> it’s readily available to the best of my knowledge. 

Most residential electrical contractors are going to use rated ENT since 
it's what they can easily get at a normal supply house or even home 
center. This is the stuff made popular by Carlon in their trademark 
blue that makes people call it "smurf tube". It's readily available in 
1/2" and 3/4" everywhere from home centers to basically any supply 
house. 1" is less common - most home centers don't have it, and since 
it isn't normally needed in residential nor is ENT basically ever used 
in commercial, neither do a lot of strictly electrical supply houses. 

You can of course get corrugated communication duct in that size, often 
with mule tape pre-installed as a bonus, but a lot of electrical supply 
houses that deal only in "electrical" and not "communications" don't 
have that and will send folks to a "low voltage communication supplier" 
or have to special order it. A lot of electrical contractors, 
especially residential, would rather not bother with a separate supply 
run, and having to wait is a pain if it wasn't planned early on and also 
often means you're paying freight separately. 

Even then, most of the folks going to a communications supplier are 
going to reach straight for 1.25" or larger in commercial since you 
don't have to worry about drilling studs. As I recall, my local 
communication cable house didn't even have 1" in stock, but they did 
have 1.25" in stock, and their price was basically the same as the 1" as 
a result. 

So it's not that it doesn't exist or anything, it's just that, at least 
as far as I've seen, there's not a ton of demand for it, so local stock 
is thin. 

All that said, I just checked Home Depot, and they are showing some 
stock on 1" ENT at my local store, so maybe that's changing. Used to be 
they only stocked 3/4" and 1/2". They still only have 25ft hanks of the 
1" stuff (you can get 1/2" and 3/4" in 25 or 100-200ft), but at least 
they have it now. It is still about 4x the price of 3/4" per unit 
length, though. 

-- 
Brandon Martin

Re: Advantages and disadvantages of legacy assets

2023-11-22 Thread Mike Lyon

I’ve been using AltDB for years. Works great and is indeed, free.

The fine folks at FCIX have taken over the project and manage it now.

Lots of good documentstion out there for it as well.

-Mike

> On Nov 22, 2023, at 12:15, William Herrin  wrote:
> 
> On Wed, Nov 22, 2023 at 11:22 AM o...@delong.com  wrote:
>>>> On Nov 21, 2023, at 01:38, William Herrin  wrote:
>>> Disadvantages: Expensive IRR. No RPKI. No vote in ARIN elections. No
>>> legal clarity regarding the status of your resources.
>> 
>> Expensive IRR? ALTDB is free?
> 
> I don't know anything about ALTDB. RADB is pricey.
> 
> 
>> On Wed, Nov 22, 2023 at 11:16 AM owen--- via NANOG  wrote:
>> Apparently, Tata is rejecting routes that have neither RPKI nor an RIR-based 
>> IRR record created after 1993.
> 
> Are you sure? The way I read it, that policy applies to -customer-
> announced routes, not broad Internet routes received from peers and
> transit.
> 
> It still seems unwise, but not entirely insane.
> 
> Regards,
> Bill Herrin
> 
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/

Re: Generally accepted BGP acceptance criteria?

2023-11-17 Thread Mike Hammett

Everyone thinks they're a unicorn and they're special and it's a secret... 
other than those that don't. ;-) 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Tom Samplonius"  
To: nanog@nanog.org 
Sent: Thursday, November 16, 2023 6:54:17 PM 
Subject: Generally accepted BGP acceptance criteria? 


In the world of IRR and RPKI, BGP route acceptance criteria is important to get 
right. 

DE-CIX has published a detailed flow chart documenting their acceptance 
criteria: https://www.de-cix.net/en/locations/frankfurt/route-server-guide 

But I don’t see a lot of operators publishing their criteria. I imagine there 
is a some sort of coalescing industry standard out there, but so far I can’t 
find it. Of the upstreams I use, just one publishes a flowchart. Another is 
basically refusing to explain anything other than they “use” IRR and RPKI, ad 
that RPKI is “good”. 

I assumed that everyone implementing RPKI validation, would skip IRR route 
object validation if the route is RPKI-valid. I assumed that everyone is doing 
this now, or would do this when they implement RPKI validation. But I spoke to 
an operator today, which still expects all routes to pass IRR as well (and 
while they perform RPKI-validation, they effectively do nothing with the 
result). To me, this seems like a different direction than most operators are 
going. Or is it? 

The most surprising thing in the DE-DIX flow chart, was that they check that 
the origin AS exists in the IRR as-set, before doing RPKI, and if the set 
existence fails, they reject the route. I don’t see a problem with this, as 
maintaining as-sets is easy, but it does prevent an eventual 100% RPKI future 
with no IRR at all. 

I also thought there may be an informational RFC on this, but I couldn’t find 
anything. Has there been anything published or any presentations given, on 
generally accepted BGP route acceptance criteria? 


Tom

Re: Out of ideas - Comcast issue BGP peering with Tata

2023-11-17 Thread Mike Hammett

This passing the buck thing was old a very long time ago. 

CDNs and security services are great at it too. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Jamie Chetta via NANOG"  
To: nanog@nanog.org 
Sent: Friday, November 17, 2023 8:17:42 AM 
Subject: Out of ideas - Comcast issue BGP peering with Tata 



I am out of ideas on how to get this fixed. Long story short I am a customer of 
Comcast and am advertising my own /24 block I own through them. Comcast of 
course BGP peers with multiple ISPs. Other ISPs are accepting my prefix just 
fine, except Tata. This is causing random destinations to drop connectivity if 
Comcast routes it through them. Comcast has confirmed they are advertising my 
block to Tata and that the RPKI is good, however when you check the Tata 
looking glass you can see they’re not accepting it. 

I’ve tried escalating within Comcast who refuses to contact Tata as they’ve 
validated the issue is not on their end but they agree with my assessment that 
Tata is not accepting the prefix for some reason. 

I’ve tried multiple email for Tata support (below), but they all circle around 
to a helpdesk who says I do not have a circuit with them so they cannot help 
me. 

Is there anyone from Tata willing to contact me off list to help sort this out? 
Or anyone with ideas on specifically why other ISPs are accepting my route but 
not Tata? It would be greatly appreciated. 

Emails I’ve tried 
Corporate Helpdesk corp.helpd...@tatacommunications.com 
Tata Communications IP Service Support( AS-6453) 
ipservicesupp...@tatacommunications.com 
IPNOC (Tata Communications - AS6453) ip...@tatacommunications.com 
l...@as6453.net 

Response from Tata: 
“ Acknowledge your email. 

However, since you are not associated with TCL we would not be in a position to 
help you on this. 

Request you to contact comcast for the assistance that you are seeking from 
us.” 

Response from Comcast: 
“ This was sent back to me as not us. Basically, it’s not a RADB or RPKI issue. 
This is being accepted and re-advertised to TATA but not being accepted on 
their end. But another route that we checked off of that same SUR is being 
advertised the same way and accepted by them off pe12.350ecermak.il.ibone as an 
example of the TATA looking glass. I would suggest that you would probably need 
to work with other networks as to why those that are specific ones are not 
accepting the block but as previously mentioned it’s not a RADB or RPKI issue 
and as a result not a Comcast issue.”

Re: The rise and fall of the 90's telecom bubble

I saw a map once showing that they seemed to come into town from the west via 
I-88 (while other westerly routes traveled via the Rock Island). I only know of 
CenturyLink (then Digital Teleport) going west from 88, so maybe that's where 
they got their route from? I wonder where their huts\POPs were unless they 
just were in DTI huts too. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Bryan Holloway"  
To: "NANOG"  
Sent: Tuesday, November 14, 2023 9:20:24 AM 
Subject: Re: The rise and fall of the 90's telecom bubble 


On 11/14/23 15:04, Mike Hammett wrote: 
> "It would be nice if some folks on the list could 
> provide some solid information, even if only for one 
> large carrier." 
> 
> One of the busts that I never found much on was the Enron network. Where 
> did it end up? I'd be interested in local route and POP details, offlist 
> would be fine as well. 
> 

I ran a facility at 717 S Wells which they used to occupy on the 10th 
floor back in the day, but they were leveraging other folks' fiber. 
Level3 and Looking Glass if my addled brain remembers ... 



>  
> *From: *"Andrew Odlyzko via NANOG"  
> *To: *"Dave Taht"  
> *Cc: *"NANOG"  
> *Sent: *Monday, November 13, 2023 1:25:39 PM 
> *Subject: *Re: The rise and fall of the 90's telecom bubble 
> 
> Dave Taht's question about all the redundant fiber that 
> was put down in the telecom bubble is a very interesting 
> one. It would be nice if some folks on the list could 
> provide some solid information, even if only for one 
> large carrier. 
> 
> My impression, from communications with various folks, 
> is that much of that fiber from around 2000 was never 
> lit. The reason is that better fiber came on the market. 
> However, what was used (at least in some cases, again, 
> this is something I would love to get real data on) was 
> that some of the empty conduits that were put down then 
> were used to shoot the new generations of fiber through. 
> (It was quite common for carriers to put down 4 conduits, 
> and only pull fiber through one of them, leaving the 
> other 3 for later use.) 
> 
> Concerning the Doug O'Laughlin post that Dave cites, 
> it is very good. For more on the myth of "Internet 
> doubling every 100 days," my paper "Bubbles, gullibility, 
> and other challenges for economics, psychology, sociology, 
> and information sciences" published in First Monday in 
> Sept. 2010, 
> 
> https://firstmonday.org/ojs/index.php/fm/article/view/3142/2603 
> 
> Participants of this list were very helpful in providing 
> information for that paper. (Some of the correspondence is 
> in the list archives, most was off-line.) 
> 
> But O'Laughlin is too hard on Global Crossing, for example, 
> when he says it "was essentially a fundraising scheme looking 
> for a problem." Global Crossing had a real business plan, 
> it was the first transatlantic cable that was not built by 
> consortia of incumbent telcos, and it planned to take 
> advantage of the rising demand for transmission by offering 
> capacity to new players, who would otherwise be gouged by 
> incumbents. (It did get into accounting shenanigans later 
> on, as competition arose, but that was later.) What is 
> most interesting is that their business plan was based 
> on an assumption of demand about doubling each year (which 
> is what was taking place), not doubling every 100 days. 
> (This I learned when I was consulted on some of the 
> litigation after the telecom crash, but by now the 
> information is publicly available.) What killed them 
> is that their assumption that it would be difficult for 
> others to get the (special undersea) fiber, the cable-laying 
> ships, the permits, ..., turned out to be wrong, and so 
> a slew of competitors, inspired by the myth of astronomical 
> growth rates, came on the scene. (Global Crossing's expansion 
> into terrestrial fiber networks was also a major contributor.) 
> 
> One of the astounding observations is that while Global 
> Crossing was assuming 100% annual growth rate in traffic, 
> the industry as a whole (as well as the press, the FCC, 
> and so on) were talking of 1,000% growth rates. And the 
> only observer that I was able to find who noted this in 
> print was George Gilder, who drew the wrong conclusion 
> from this! (Details are in the paper cited above.) 
> 
> Andrew 
> 
> P.S. Some interesting materials from telecom bubble era 
> are available at 
> 
> https://www-users.cse.umn.edu/~odlyzko/isources/inde

Re: The rise and fall of the 90's telecom bubble

Well, and "better" for what purpose? Some glass may not work out well for 
longhaul 25+ terabit services, but would be fine for metro loops of regular 
services. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Tim Požar via NANOG"  
To: odly...@umn.edu, "Dave Taht"  
Cc: "NANOG"  
Sent: Tuesday, November 14, 2023 10:55:19 AM 
Subject: Re: The rise and fall of the 90's telecom bubble 

One consideration about older fiber, is it may not be maintained. I 
know of one large city owned installation that has older fiber that is 
being abandoned in order to pull higher count and better glass in. The 
old fiber will end up being cut due to construction, etc. so it is 
worthless. 

Tim 

On 11/13/23 11:25 AM, Andrew Odlyzko via NANOG wrote: 
> Dave Taht's question about all the redundant fiber that 
> was put down in the telecom bubble is a very interesting 
> one. It would be nice if some folks on the list could 
> provide some solid information, even if only for one 
> large carrier. 
> 
> My impression, from communications with various folks, 
> is that much of that fiber from around 2000 was never 
> lit. The reason is that better fiber came on the market. 
> However, what was used (at least in some cases, again, 
> this is something I would love to get real data on) was 
> that some of the empty conduits that were put down then 
> were used to shoot the new generations of fiber through. 
> (It was quite common for carriers to put down 4 conduits, 
> and only pull fiber through one of them, leaving the 
> other 3 for later use.) 
> 
> Concerning the Doug O'Laughlin post that Dave cites, 
> it is very good. For more on the myth of "Internet 
> doubling every 100 days," my paper "Bubbles, gullibility, 
> and other challenges for economics, psychology, sociology, 
> and information sciences" published in First Monday in 
> Sept. 2010, 
> 
> https://firstmonday.org/ojs/index.php/fm/article/view/3142/2603 
> 
> Participants of this list were very helpful in providing 
> information for that paper. (Some of the correspondence is 
> in the list archives, most was off-line.) 
> 
> But O'Laughlin is too hard on Global Crossing, for example, 
> when he says it "was essentially a fundraising scheme looking 
> for a problem." Global Crossing had a real business plan, 
> it was the first transatlantic cable that was not built by 
> consortia of incumbent telcos, and it planned to take 
> advantage of the rising demand for transmission by offering 
> capacity to new players, who would otherwise be gouged by 
> incumbents. (It did get into accounting shenanigans later 
> on, as competition arose, but that was later.) What is 
> most interesting is that their business plan was based 
> on an assumption of demand about doubling each year (which 
> is what was taking place), not doubling every 100 days. 
> (This I learned when I was consulted on some of the 
> litigation after the telecom crash, but by now the 
> information is publicly available.) What killed them 
> is that their assumption that it would be difficult for 
> others to get the (special undersea) fiber, the cable-laying 
> ships, the permits, ..., turned out to be wrong, and so 
> a slew of competitors, inspired by the myth of astronomical 
> growth rates, came on the scene. (Global Crossing's expansion 
> into terrestrial fiber networks was also a major contributor.) 
> 
> One of the astounding observations is that while Global 
> Crossing was assuming 100% annual growth rate in traffic, 
> the industry as a whole (as well as the press, the FCC, 
> and so on) were talking of 1,000% growth rates. And the 
> only observer that I was able to find who noted this in 
> print was George Gilder, who drew the wrong conclusion 
> from this! (Details are in the paper cited above.) 
> 
> Andrew 
> 
> P.S. Some interesting materials from telecom bubble era 
> are available at 
> 
> https://www-users.cse.umn.edu/~odlyzko/isources/index.html 
> 
> 
> 
> On Sun, 12 Nov 2023, Dave Taht wrote: 
> 
>> Aside from me pinning the start of the bubble closer to 1992 when 
>> commercial activity was allowed, and M for ISPs at insane valuations 
>> per subscriber by 1995 (I had co-founded an ISP in 93, but try as I 
>> might I cannot remember if it peaked at 50 or 60x1 by 1996 (?) and 
>> crashed by 97 (?)), this was a whacking good read, seems accurate, and 
>> moves to comparing it across that to the present day AI bubble. 
>> 
>> https://www.fabricatedknowledge.com/p/lessons-from-history-the-rise-and 
>> 
>> In the end we sold (my ISP

Re: The rise and fall of the 90's telecom bubble

" It would be nice if some folks on the list could 
provide some solid information, even if only for one 
large carrier." 

One of the busts that I never found much on was the Enron network. Where did it 
end up? I'd be interested in local route and POP details, offlist would be fine 
as well. 

- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Andrew Odlyzko via NANOG"  
To: "Dave Taht"  
Cc: "NANOG"  
Sent: Monday, November 13, 2023 1:25:39 PM 
Subject: Re: The rise and fall of the 90's telecom bubble 

Dave Taht's question about all the redundant fiber that 
was put down in the telecom bubble is a very interesting 
one. It would be nice if some folks on the list could 
provide some solid information, even if only for one 
large carrier. 

My impression, from communications with various folks, 
is that much of that fiber from around 2000 was never 
lit. The reason is that better fiber came on the market. 
However, what was used (at least in some cases, again, 
this is something I would love to get real data on) was 
that some of the empty conduits that were put down then 
were used to shoot the new generations of fiber through. 
(It was quite common for carriers to put down 4 conduits, 
and only pull fiber through one of them, leaving the 
other 3 for later use.) 

Concerning the Doug O'Laughlin post that Dave cites, 
it is very good. For more on the myth of "Internet 
doubling every 100 days," my paper "Bubbles, gullibility, 
and other challenges for economics, psychology, sociology, 
and information sciences" published in First Monday in 
Sept. 2010, 

https://firstmonday.org/ojs/index.php/fm/article/view/3142/2603 

Participants of this list were very helpful in providing 
information for that paper. (Some of the correspondence is 
in the list archives, most was off-line.) 

But O'Laughlin is too hard on Global Crossing, for example, 
when he says it "was essentially a fundraising scheme looking 
for a problem." Global Crossing had a real business plan, 
it was the first transatlantic cable that was not built by 
consortia of incumbent telcos, and it planned to take 
advantage of the rising demand for transmission by offering 
capacity to new players, who would otherwise be gouged by 
incumbents. (It did get into accounting shenanigans later 
on, as competition arose, but that was later.) What is 
most interesting is that their business plan was based 
on an assumption of demand about doubling each year (which 
is what was taking place), not doubling every 100 days. 
(This I learned when I was consulted on some of the 
litigation after the telecom crash, but by now the 
information is publicly available.) What killed them 
is that their assumption that it would be difficult for 
others to get the (special undersea) fiber, the cable-laying 
ships, the permits, ..., turned out to be wrong, and so 
a slew of competitors, inspired by the myth of astronomical 
growth rates, came on the scene. (Global Crossing's expansion 
into terrestrial fiber networks was also a major contributor.) 

One of the astounding observations is that while Global 
Crossing was assuming 100% annual growth rate in traffic, 
the industry as a whole (as well as the press, the FCC, 
and so on) were talking of 1,000% growth rates. And the 
only observer that I was able to find who noted this in 
print was George Gilder, who drew the wrong conclusion 
from this! (Details are in the paper cited above.) 

Andrew 

P.S. Some interesting materials from telecom bubble era 
are available at 

https://www-users.cse.umn.edu/~odlyzko/isources/index.html 

On Sun, 12 Nov 2023, Dave Taht wrote: 

> Aside from me pinning the start of the bubble closer to 1992 when 
> commercial activity was allowed, and M for ISPs at insane valuations 
> per subscriber by 1995 (I had co-founded an ISP in 93, but try as I 
> might I cannot remember if it peaked at 50 or 60x1 by 1996 (?) and 
> crashed by 97 (?)), this was a whacking good read, seems accurate, and 
> moves to comparing it across that to the present day AI bubble. 
> 
> https://www.fabricatedknowledge.com/p/lessons-from-history-the-rise-and 
> 
> In the end we sold (my ISP, founded 93) icanect for 3 cents on the 
> dollar in 99, and I lost my shirt (not for the first time) on it, only 
> to move into embedded Linux (Montavista) after the enormous pop 
> redhat's IPO had had in 99. The company I was part of slightly prior 
> (Mediaplex) went public December 12, 1999 and cracked 100/share, only 
> to crash by march, 2000 to half the IPO price (around $7 as I recall), 
> wiping out everyone that had not vested yet. I lost my shirt again on 
> that and Montavista too and decided I would avoid VCs henceforth. 
> 
> I am always interested in anecdotal reports of perso

Re: The rise and fall of the 90's telecom bubble

There were obviously many facets, but I think one of the turns was due to DWDM. 
You no longer needed a pair for every circuit. That then contributed to the 
glut of strands. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Dave Taht"  
To: "Internet-history" , "Network Neutrality 
is back! Let´s make the technical aspects heard this time!" 
, "NANOG" , "bloat" 
 
Sent: Sunday, November 12, 2023 9:48:46 AM 
Subject: The rise and fall of the 90's telecom bubble 

Aside from me pinning the start of the bubble closer to 1992 when 
commercial activity was allowed, and M for ISPs at insane valuations 
per subscriber by 1995 (I had co-founded an ISP in 93, but try as I 
might I cannot remember if it peaked at 50 or 60x1 by 1996 (?) and 
crashed by 97 (?)), this was a whacking good read, seems accurate, and 
moves to comparing it across that to the present day AI bubble. 

https://www.fabricatedknowledge.com/p/lessons-from-history-the-rise-and 

In the end we sold (my ISP, founded 93) icanect for 3 cents on the 
dollar in 99, and I lost my shirt (not for the first time) on it, only 
to move into embedded Linux (Montavista) after the enormous pop 
redhat's IPO had had in 99. The company I was part of slightly prior 
(Mediaplex) went public December 12, 1999 and cracked 100/share, only 
to crash by march, 2000 to half the IPO price (around $7 as I recall), 
wiping out everyone that had not vested yet. I lost my shirt again on 
that and Montavista too and decided I would avoid VCs henceforth. 

I am always interested in anecdotal reports of personal events in this 
increasingly murky past, and in trying to fact check the above link. 

So much fiber got laid by 2000 that it is often claimed that it was at 
least a decade before it was used up, (the article says only 2.7% was 
in use by 2002) and I have always wondered how much dark, broken, 
inaccessible fiber remains that nobody knows where it even is anymore 
due to many lost databases. I hear horror stories... 

The article also focuses solely on the us sector, and I am wondering 
what it looked like worldwide. 

I believed in the 90s we were seeing major productivity gains. The 
present expansion of the internet in my mind should not be much 
associated with "productivity gains", as, imho, reducing the general 
population to two thumbs and a 4 inch screen strikes me as an enormous 
step backwards. 

(I have a bad habit of cross posting my mails to where older denizens 
of the internet reside, sorry! If you end up posting to one of my 
lists I will add a sender allows filter for you) 
-- 
:( My old R campus is up for sale: https://tinyurl.com/yurtlab 
Dave Täht CSO, LibreQos

RE: Strange IPSEC traffic

2023-11-13 Thread Mike Lewinski via NANOG

I can confirm we started seeing this on Nov 9th at 19:10 UTC across all markets 
from a variety of sources.

If you want to filter it with ingress ACLs they need to include subnet base and 
broadcast addresses in addition to interface address, so a router at 
192.168.1.1/30 with a customer potentially running IPSEC at 192.168.1.2 would 
require all this to silence the log messages:

access-list 100 deny esp any host 192.168.1.0
access-list 100 deny esp any host 192.168.1.1
access-list 100 deny esp any host 192.168.1.3
access-list 100 permit ip any any

I started with an ACL only on the interface address and then noticed I was 
still getting logs on base/broadcast addresses.

Could be recon for the IKEv2 vulnerability in this:
https://sec.cloudapps.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-asaftd-ravpn-auth-8LyfCkeC
https://blogs.cisco.com/security/akira-ransomware-targeting-vpns-without-multi-factor-authentication

Or zero day. Even though the devices they are hitting are not configured for 
IPSEC we are filtering it anyway (and for good measure " no crypto isakmp 
enable").


Mike

Re: OSP Management

2023-10-31 Thread Mike Hammett

For our own plant, we use https://mapitright.com/ For managing my collections 
of others' maps, I use ArcGIS. Map It Right does all of the slack loops, splice 
maps, etc. ArcGIS just loads for me a bunch of layers of network map. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "michael brooks - ESC"  
To: "Mike Hammett"  
Cc: "William Herrin" , "NANOG"  
Sent: Tuesday, October 31, 2023 8:26:04 AM 
Subject: OSP Management 




On that note, what do you all use for managing OSP? We have been attempting to 
stand up PatchManager for quite some time, and find it a good product, but the 
billions of options can be overwhelming 























michael brooks 
Sr. Network Engineer 
Adams 12 Five Star Schools 

michael.bro...@adams12.org 

 

"flying is learning how to throw yourself at the ground and miss" 






On Fri, Oct 27, 2023 at 5:54 AM Mike Hammett < na...@ics-il.net > wrote: 





Always fun managing OSP. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 




This is a staff email account managed by Adams 12 Five Star Schools. This email 
and any files transmitted with it are confidential and intended solely for the 
use of the individual or entity to whom they are addressed. If you have 
received this email in error please notify the sender.

Re: Pulling of Network Maps

2023-10-27 Thread Mike Hammett

and I get how that could be. We had a design. Gave the prints to the 
contractors. Someone internally verified the contractors built what was on the 
prints. A year or two goes by and some laterals ended up costing more because 
handholes on the prints were never built. Our locator goes to a handhole to 
send his signal and the handhole doesn't exist or finds a handhole in a spot 
not on the prints. Always fun managing OSP. 

- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "William Herrin"  
To: "Tom Beecher"  
Cc: "Mike Hammett" , "NANOG"  
Sent: Thursday, October 26, 2023 12:33:25 PM 
Subject: Re: Pulling of Network Maps 

On Thu, Oct 26, 2023 at 10:01 AM Tom Beecher  wrote: 
> My experience with maps over the last decade tells me that even most vendors 
> don't actually know where they are. :) 

So true. And not that young a problem. I leased some dark fiber more 
than a decade ago. They sent an unexpectedly expensive build proposal 
to connect my building. I asked: "Why are you trenching to the manhole 
down the street instead of the one right outside?" They asked, "what 
manhole?" Long story short, they dispatched a guy who popped the 
cover, pumped the water out of the vault and confirmed that they had a 
location they didn't know about. 

Regards, 
Bill Herrin 

-- 
William Herrin 
b...@herrin.us 
https://bill.herrin.us/

Re: Pulling of Network Maps

2023-10-26 Thread Mike Hammett

But it already is publicly available to someone that's interested enough via 
the permits issued by the appropriate jurisdictions or if you put in 811 design 
stage tickets. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Denis Fondras"  
To: nanog@nanog.org 
Sent: Thursday, October 26, 2023 12:22:56 PM 
Subject: Re: Pulling of Network Maps 

Le Thu, Oct 26, 2023 at 11:17:22AM -0500, Mike Hammett a écrit : 
> Has anyone else noticed a trend of some network operators that previously 
> offered street-level detailed maps, not only upon request, but also posted 
> publicly have started to only provide them upon quotes? 
> 

There is no small profit :) 

Also some will fear sabotage if the pathway is publicly available.

Re: Pulling of Network Maps

2023-10-26 Thread Mike Hammett

I had that too. The map showed a facility was online. It wasn't. Lots of build 
to get there. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Tom Beecher"  
To: "Mike Hammett"  
Cc: "NANOG"  
Sent: Thursday, October 26, 2023 12:01:33 PM 
Subject: Re: Pulling of Network Maps 




If it's too hard for me to figure out where you are, you just plain won't get 
the sale. 




My experience with maps over the last decade tells me that even most vendors 
don't actually know where they are. :) 


On Thu, Oct 26, 2023 at 12:18 PM Mike Hammett < na...@ics-il.net > wrote: 





Has anyone else noticed a trend of some network operators that previously 
offered street-level detailed maps, not only upon request, but also posted 
publicly have started to only provide them upon quotes? 


Not even the popular online mapping services have current-enough-to-be-useful 
maps. 


The claim is that it's proprietary. A) It wasn't before and B) No it isn't. 
Everything you've ever done is a FOIA request or 811 design ticket away. 


I'm not sure how this helps the companies. It certainly makes it harder for me 
trying to piece networks together when they won't tell me where they are until 
I give them A and Z locations. If it's too hard for me to figure out where you 
are, you just plain won't get the sale. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com

Pulling of Network Maps

2023-10-26 Thread Mike Hammett

Has anyone else noticed a trend of some network operators that previously 
offered street-level detailed maps, not only upon request, but also posted 
publicly have started to only provide them upon quotes? 


Not even the popular online mapping services have current-enough-to-be-useful 
maps. 


The claim is that it's proprietary. A) It wasn't before and B) No it isn't. 
Everything you've ever done is a FOIA request or 811 design ticket away. 


I'm not sure how this helps the companies. It certainly makes it harder for me 
trying to piece networks together when they won't tell me where they are until 
I give them A and Z locations. If it's too hard for me to figure out where you 
are, you just plain won't get the sale. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com

Re: ESPN streaming issues

2023-10-24 Thread Mike Lyon

Check your blocks on the various databases here and see which ones are reporting as out of the country:IP Geolocation and VPN Resourcesthebrotherswisp.comThen follow-up with those individual DBs to get the locations corrected.You should be all set.-MikeOn Oct 24, 2023, at 20:32, Brad Bendy  wrote:Anyone have a contact for ESPN or can someone contact me off list?Have a good amount of IP space that just started reporting the IP isout of the country.Thanks

Re: transit and peering costs projections

2023-10-15 Thread Mike Hammett

Houston is tricky as due to it's geographic scope, it's quite expensive to 
build an IX that goes into enough facilities to achieve meaningful scale. CDN 1 
is in facility A. CDN 2 in facility B. CDN 3 is in facility C. When I last 
looked, it was about 80 driving miles to have a dark fiber ring that 
encompassed all of the facilities one would need to be in. 

- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Tim Burke"  
To: "Dave Taht"  
Cc: "Network Neutrality is back! Let´s make the technical aspects heard this 
time!" , "libreqos" 
, "NANOG"  
Sent: Saturday, October 14, 2023 10:45:47 PM 
Subject: Re: transit and peering costs projections 

I would say that a 1Gbit IP transit in a carrier neutral DC can be had for a 
good bit less than $900 on the wholesale market. 

Sadly, IXP’s are seemingly turning into a pay to play game, with rates almost 
costing as much as transit in many cases after you factor in loop costs. 

For example, in the Houston market (one of the largest and fastest growing 
regions in the US!), we do not have a major IX, so to get up to Dallas it’s 
several thousand for a 100g wave, plus several thousand for a 100g port on one 
of those major IXes. Or, a better option, we can get a 100g flat internet 
transit for just a little bit more. 

Fortunately, for us as an eyeball network, there are a good number of major 
content networks that are allowing for private peering in markets like Houston 
for just the cost of a cross connect and a QSFP if you’re in the right DC, with 
Google and some others being the outliers. 

So for now, we'll keep paying for transit to get to the others (since it’s 
about as much as transporting IXP from Dallas), and hoping someone at Google 
finally sees Houston as more than a third rate city hanging off of Dallas. Or… 
someone finally brings a worthwhile IX to Houston that gets us more than 
peering to Kansas City. Yeah, I think the former is more likely.  

See y’all in San Diego this week, 
Tim 

On Oct 14, 2023, at 18:04, Dave Taht  wrote: 
> 
> This set of trendlines was very interesting. Unfortunately the data 
> stops in 2015. Does anyone have more recent data? 
> 
> https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php
>  
> 
> I believe a gbit circuit that an ISP can resell still runs at about 
> $900 - $1.4k (?) in the usa? How about elsewhere? 
> 
> ... 
> 
> I am under the impression that many IXPs remain very successful, 
> states without them suffer, and I also find the concept of doing micro 
> IXPs at the city level, appealing, and now achievable with cheap gear. 
> Finer grained cross connects between telco and ISP and IXP would lower 
> latencies across town quite hugely... 
> 
> PS I hear ARIN is planning on dropping the price for, and bundling 3 
> BGP AS numbers at a time, as of the end of this year, also. 
> 
> 
> 
> -- 
> Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html 
> Dave Täht CSO, LibreQos

Re: transit and peering costs projections

2023-10-15 Thread Mike Hammett

I've seen some attempts to put an IX at every corner, but I don't think those 
efforts will be overly successful. 

It's still difficult to gain sufficient scale in NFL-sized cities. Big content 
won't join without big eyeballs (well, not the national-level guys because they 
almost never will). Big eyeballs just can't be bothered. Small guys don't move 
the needle enough. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Dave Taht"  
To: "Network Neutrality is back! Let´s make the technical aspects heard this 
time!" , "libreqos" 
, "NANOG"  
Sent: Saturday, October 14, 2023 6:01:54 PM 
Subject: transit and peering costs projections 

This set of trendlines was very interesting. Unfortunately the data 
stops in 2015. Does anyone have more recent data? 

https://drpeering.net/white-papers/Internet-Transit-Pricing-Historical-And-Projected.php
 

I believe a gbit circuit that an ISP can resell still runs at about 
$900 - $1.4k (?) in the usa? How about elsewhere? 

... 

I am under the impression that many IXPs remain very successful, 
states without them suffer, and I also find the concept of doing micro 
IXPs at the city level, appealing, and now achievable with cheap gear. 
Finer grained cross connects between telco and ISP and IXP would lower 
latencies across town quite hugely... 

PS I hear ARIN is planning on dropping the price for, and bundling 3 
BGP AS numbers at a time, as of the end of this year, also. 



-- 
Oct 30: https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html 
Dave Täht CSO, LibreQos

Re: ARIN whois contact abuse from ipv4depot aka Silicon Desert International Inc

2023-10-12 Thread Mike Hammett

Do we know if the organizations with key Internet resources (ARIN, RIPE, 
PeeringDB, etc.) have any honeypots in their arsenal? Obviously, publicly 
knowing about it kind of defeats the purpose of it, but that might be a way to 
help be proactive - make fake entries with unique contact information to catch 
those harvesting. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Mel Beckman"  
To: "Tom Beecher"  
Cc: "nanog@nanog.org list"  
Sent: Thursday, October 12, 2023 11:01:20 AM 
Subject: Re: ARIN whois contact abuse from ipv4depot aka Silicon Desert 
International Inc 

Tom, 


When an ARIN member violates their agreement and spams from ARIN’s databases, 
it’s not just an “Internet is fertile ground” deal. It’s a betrayal of a legal 
trust, one that demands accountability. I’m quite happy that ARIN promptly 
responds to these abuses, and gets results. That only works if victims report 
spam and compare notes. Let the “fertile ground” be elsewhere! 


-mel beckman 



On Oct 12, 2023, at 8:49 AM, Tom Beecher  wrote: 









It's ridiculous that they resort to scraping public lists and DBs to try and 
achieve what they're attempting to do. 





Everyone is always looking for information they can use to advance some agenda 
or purpose. The internet is fertile ground for that. Always has been, always 
will be. 


Not taking shots at anyone here, but I am boggled why this is a common public 
complaint. Block the sender and move on. 


On Wed, Oct 11, 2023 at 7:56 PM Peter Potvin via NANOG < nanog@nanog.org > 
wrote: 



Definitely have received this same spam multiple times and so have a few others 
I know. It's ridiculous that they resort to scraping public lists and DBs to 
try and achieve what they're attempting to do. 




Regards, 
Peter Potvin | Executive Director 
-- 
Accuris Technologies Ltd. 






On Wed, Oct 11, 2023 at 7:52 PM Eric Kuhnke < eric.kuh...@gmail.com > wrote: 



Is anyone else receiving spam from this organization? Based on the contents of 
the cold solicitations they are sending us, and the addresses being sent to, 
they have scraped ARIN WHOIS data for noc and abuse POC contact info and recent 
ipv4 block transfers. 


It's trivially easy to block their entire domain at the mail server level, of 
course...

Re: Low to Mid Range DWDM Platforms

2023-10-06 Thread Mike Hammett

Well, and that's kinda where I was going. 

I've used FS passive systems for years. FS has an active platform or two (that 
I understand, they just whitebox). Does it really do everyone one would need to 
do? How much of a step is it to get something more? 

- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "David Bass"  
To: "Dave Bell"  
Cc: nanog@nanog.org 
Sent: Friday, October 6, 2023 8:55:21 AM 
Subject: Re: Low to Mid Range DWDM Platforms 

On the same topic, anyone have experience with the stuff from fs.com ? 

On Fri, Oct 6, 2023 at 9:53 AM Dave Bell < m...@geordish.org > wrote: 

Smartoptics? 

https://smartoptics.com/ 

Regards, 
Dave 

On Fri, 6 Oct 2023 at 14:43, Mark Tinka  wrote: 

On 10/6/23 15:07, Mike Hammett wrote: 

> I've been using various forms of passive WDM for years. I have a couple 
> different projects on my plate that require me to look at the next level of 
> platform. 
> 
> In some projects, I'll be looking for needing to have someone long distances 
> of glass without any electronics. Some spans could be over 60 miles. 
> 
> In some projects, I'll need to transport multiple 100-gig waves. 
> 
> What is the landscape like between basic passive and something like a 30 
> terabit Ciena? I know of multiple vendors in that space, but I like to learn 
> more about what features I need and what features I don't need from somewhere 
> other than the vendor's mouth. Obviously, the most reliability at the least 
> cost as well. 

400G-ZR pluggables will get you 400Gbps on a p2p dark fibre over 80km - 
100km. So your main cost there will be routers that will support. 

The smallest DCI solution from the leading DWDM vendors is likely to be 
your cheapest option. Alternatively, if you are willing to look at the 
open market, you can find gear based on older CMOS (40nm, for example), 
which will now be EoL for any large scale optical network, but cost next 
to nothing for a start-up with considerable capacity value. 

There is a DWDM vendor that showed up on the scene back in 2008 or 
thereabouts. They were selling a very cheap, 1U box that had a different 
approach to DWDM from other vendors at the time. I, for the life of me, 
cannot remember their name - but I do know that Randy introduced them to 
me back then. Maybe he can remember :-). Not sure if they are still in 
business. 

Mark.

Low to Mid Range DWDM Platforms

2023-10-06 Thread Mike Hammett

I've been using various forms of passive WDM for years. I have a couple 
different projects on my plate that require me to look at the next level of 
platform.

In some projects, I'll be looking for needing to have someone long distances of 
glass without any electronics. Some spans could be over 60 miles.

In some projects, I'll need to transport multiple 100-gig waves.

What is the landscape like between basic passive and something like a 30 
terabit Ciena? I know of multiple vendors in that space, but I like to learn 
more about what features I need and what features I don't need from somewhere 
other than the vendor's mouth. Obviously, the most reliability at the least 
cost as well.

Thanks.



-
Mike Hammett
Intelligent Computing Solutions
http://www.ics-il.com

Midwest-IX
http://www.midwest-ix.com

Re: ARIN email address (was cogent spamming directly from ARIN records?)

2023-10-03 Thread Mike Lyon

Give it time :)

-Mike

> On Oct 3, 2023, at 18:06, Owen DeLong via NANOG  wrote:
> 
> But I seem to have finally gotten Cogent trained not to spam this one, so I 
> think I’ll leave it as is.
> 
> YMMV
> 
> Owen
> 
> 
>> On Oct 3, 2023, at 08:52, Bryan Fields  wrote:
>> 
>>> On 10/2/23 11:28 AM, Mel Beckman wrote:
>>> I believe they got the contact information from ARIN
>> 
>> I'd suggest everyone use an alias unique to ARIN for your POC and/or public 
>> email.  Makes it super simple to verify where it was sourced from.
>> 
>> (and yes I've got the same spam)
>> -- 
>> Bryan Fields
>> 
>> 727-409-1194 - Voice
>> http://bryanfields.net
>

Re: maximum ipv4 bgp prefix length of /24 ?

2023-09-29 Thread Mike Lyon

Apparently i ruffled some feathers with my reply about Cogent.My apologies to the list for my blunt reply…Y’all have a good weekend now.-MikeOn Sep 29, 2023, at 10:43, Valerie Wittkop  wrote:There is one person that reviews the moderation queue of the NANOG list. My morning was rather hectic, and I didn’t get to the queue until just before 12:30 EDT today. Apologies to all for the delay in the messages of this thread. Please note  I try to check the queue a few times throughout the day, and one last time again before I shut down for the night. 
Valerie WittkopProgram Directorvwitt...@nanog.org | +1 734-730-0225 (mobile) | www.nanog.orgNANOG | 305 E. Eisenhower Pkwy, Suite 100 | Ann Arbor, MI 48108, USAASN 19230

On Sep 29, 2023, at 13:36, Ryan Hamel  wrote:Matt,It's not just you or Google, I just got those emails to my Office 365 at the same time. My guess is that the list admins/moderators got the emails and just responded without approving the moderated emails.RyanFrom: NANOG  on behalf of Matthew Petach Sent: Friday, September 29, 2023 10:31 AMTo: VOLKAN SALİH Cc: nanog list Subject: Re: maximum ipv4 bgp prefix length of /24 ? Caution: This is an external email and may be malicious. Please take care when clicking links or opening attachments.On Fri, Sep 29, 2023 at 9:42 AM VOLKAN SALİH  wrote:[...]I presume there would be another 50 big ASNs that belong to CDNs. And I am pretty sure those top 100 networks can invest in gear to support /25-/27.Volkan,So far, you haven't presented any good financial reason those top 100 networks should spend millions of dollars to upgrade their networks just so your /27 can be multihomed.Sure, they *can* invest in gear to support /25-/27; but they won't, because there's no financial benefit for them to do so.I know from *your* side of the table, it would make your life better if everyone would accept /27 prefixes--multihoming for the masses, yay!Try standing in their shoes for a minute, though. You need to spend tens of millions of dollars on a multi-year refresh cycle to upgrade hundreds of routers in your global backbone, tying up network engineering resources on upgrades that at the end, will bring you exactly $0 in additional revenue.Imagine you're the COO or CTO of a Fortune 500 network, and you're meeting with your CFO to pitch this idea.You know your CFO is going to ask one question right off the bat "what's the timeframe for us to recoup the cost ofthis upgrade?" (hint, he's looking for a number less than 40 months).If your answer is "well, we're never going to recoup the cost.  It won't bring us any additional customers, it won't bring us any additional revenue, and it won't make our existing customers any happier with us.  But it will make it easier for some of our smaller compeitors to sign up new customers." I can pretty much guarantee your meeting with the CFO will end right there.If you want networks to do this, you need to figure out a way for it to make financial sense for them to do it.So far, you haven't presented anything that would make it a win-win scenario for the ISPs and CDNs that would need to upgrade to support this.ON a separate note--NANOG mailing list admins, I'm noting that Vokan's emails just arrived a few minutes ago in my gmail inbox.However,  I saw replies to his messages from others on the list yesterday, a day before they made it to the general list.Is there a backed up queue somewhere in the NANOG list processing that is delaying some messages sent to the list by up to a full day?If not, I'll just blame gmail for selectively delaying portions of NANOG for 18+ hours.   ^_^;Thanks!Matt

Re: maximum ipv4 bgp prefix length of /24 ?

2023-09-29 Thread Mike Lyon

Cogent can go fuck themselves. They deserve no charity from Mr. Leber (or 
anyone else, for that matter).

Cogent was the basis for multiple peering wars over the last 20+ years because 
of their greediness.

Cogent illegally scraped ARIN’s records for contact information for their sales 
teams.

Cogent sales reps are the scum of the earth and use relentless sales tactics. 

I refuse to use their services, even they gave it to me free, and ive told them 
that.

Cogent can eat a dick.

-Mike

> On Sep 29, 2023, at 09:44, VOLKAN SALİH  wrote:
> 
> 
> Can we get single-homed and dual-homed ASN counts worldwide by somebody here?
> 
> Checking, https://bgp.he.net/country/US , more than half of networks are 
> either single-homed or dual-homed.
> 
> single-homed networks do not need full-table, for sure. Dual homed networks 
> need to buy partial transit from the notorious tier-1.5 network that has "NO" 
> (close the doors) peering policy, that you know their name.. Then route other 
> traffic via cheapest second Transit Provider..
> 
> BTW, Thanks Mr. M.Leber for shitting in the world-wide IPv6 internet (causing 
> segmentation in the IPv6 world), I guess he thinks FREE means FEELESS while 
> it means mostly freedom..
> 
> It seems like Mr. M.Leber believes dictatorship instead of freedom. Wake up, 
> Nobody have to do peer with you for free (settlement-free), but you can 
> negotiate the price/mbps.
> 
> 
> 
> 29.09.2023 08:01 tarihinde VOLKAN SALİH yazdı:
>> CGNAT is not worse any more, IMHO. 
>> 
>> with Endpoint-independent-NAT you can accept incoming connections, as soon 
>> as you open the port automatically by sending packet to any host. Then any 
>> host can start connection to your host? thats perfect for gamers, streamers, 
>> webmasters.. etc.. Allows P2P connections..
>> 
>> for server setups, how many common ports you need to forward? five or ten, 
>> maybe. not that bad. if it is scripted, then it is automated. if its 
>> automated then it is not headache for network administrator..
>> 
>> There are just about 50 major NSP networks on the Earth, that needs to use 
>> BGP full-table.
>> 
>> I presume there would be another 50 big ASNs that belong to CDNs. And I am 
>> pretty sure those top 100 networks can invest in gear to support /25-/27.
>> 
>> I would suggest Tier3 eyeballs to mark connection depending on incoming 
>> interface (transit provider). Then route outgoing traffic of connections via 
>> same interface (TP). Thats all they need to do. if they do not optimize BGP 
>> based on packetloss rate and latency (performance).
>> 
>> Please Correct me if i am wrong.
>> 
>> Thanks and regards
>> 
>> 
>> 
>> 29.09.2023 07:48 tarihinde Owen DeLong yazdı:
>>> I presume you mean CGNAT? Otherwise, not sure what EINAT is and couldn’t 
>>> find
>>> a reference with a quick google search.
>>> 
>>> Again agree to disagree. NAT is bad and more NAT is just worse.

Re: TACACS+ server recommendations?

2023-09-22 Thread Mike Lewinski via NANOG

> We are using Okta's RADIUS service for 2fa to network gear currently, 
> but looking to switch to tacacs+ for many reasons. Would prefer to 
> implement tacacs+ with two-factor if possible.

tac_plus-ng from https://www.pro-bono-publico.de/projects/tac_plus-ng.html has 
LDAP and PAM backends, among others, so I believe you can implement 2FA through 
them. I haven't implemented this yet but it's on my to-do list (and I'm also 
warily watching passkey developments and wondering how much effort I should put 
into something that likely won't be best practice in another year or two).

I see Marc Huber is also promoting/supporting tacacs+ extension for SSH public 
key auth

https://github.com/MarcJHuber/event-driven-servers/wiki/TACACS_PLUS---SSH-Public-Key-Authentication

Re: TACACS+ server recommendations?

2023-09-20 Thread Mike Lewinski via NANOG

> https://www.shrubbery.net/tac_plus/ 

That tac_plus has python 2 dependencies and so has been removed from Debian 
packages. That's not surprising given the last update was 2015 and Python 2 was 
EOL in 2020: https://www.python.org/doc/sunset-python-2/

Currently I favor this one which is still being actively developed:

https://www.pro-bono-publico.de/projects/tac_plus.html

Re: Zayo woes

*nods* Differences between long term money and opportunistic money, which I 
admit is starting to trend away from NANOG's purpose, but still kinda related 
as it pertains to integration of networks. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Delong.com"  
To: "Mike Hammett"  
Cc: "Matthew Petach" , "nanog"  
Sent: Tuesday, September 19, 2023 1:22:26 PM 
Subject: Re: Zayo woes 

You’ve got the blame right, but the fact that the cost savings don’t 
materialize quickly seems to get forgiven more easily than a sudden (albeit 
one-time, temporary) increase in costs to accelerate that transition. Result: 
In general, no additional money, limp along and realize the cost savings over 
time. 


Certainly not idea from a technical perspective. Probably not ideal from a 
bottom line perspective over the long haul. Almost certainly ends up looking 
better on the first couple of 10Qs and by the 3rd 10Q, everyone is paying 
attention to something else. 


Owen 








On Sep 19, 2023, at 07:44, Mike Hammett  wrote: 


I blame not the network nerds, but the management that put them in that 
position. 




The problem with C is that unless the networks *ARE* integrated quickly, the 
synergies won't happen. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Matthew Petach"  
To: "Bill Murphy"  
Cc: "Mike Hammett" , "Randy Carpenter" 
, "nanog"  
Sent: Tuesday, September 19, 2023 9:41:35 AM 
Subject: Re: Zayo woes 







On Tue, Sep 19, 2023 at 7:19AM Mike Hammett < na...@ics-il.net > wrote: 




[...] 


I've never understood companies that acquire and don't completely integrate as 
quickly as they can. 







Ah, spoken with the voice of someone who's never been in the position of: 
a) acquiring a company not-much-smaller-than-you that 
b) runs on completely different hardware and software and 
c) your executives have promised there will be cost savings after the merger 
due to "synergies" between the two companies. 
^_^; 



Let's say you're an all J shop; your scripts, your tooling, everything expects 
to be talking to J devices. 


Your executives buy a company that has almost the same size network--but it's 
all C devices running classic IOS. 


You can go to your executives and tell them "hey, to integrate quickly with our 
network and tooling, we need to swap out all their C gear for J gear; it's 
gonna cost an extra $50M" 
The executives respond by pointing at c) above, and denying the request for 
money to convert the acquired network to J. 


You can go to your network and say "hey, we need to revamp our tooling and 
systems to understand how to speak to C and J devices equally, in spite of 
wildly different syntaxes for route-maps and the like-it's going to take 4 more 
developer headcount to rewrite all the systems." 
The executives respond by pointing at c) above, and deny the request for 
developer headcount to rewrite your software systems. 


The general result of acquisitions of similar-sized companies is that the 
infrastructure runs in parallel, slowly getting converted over and unified as 
gear needs to be replaced, or sites are phased out--because any other course of 
action costs more money than the executives had promised the shareholders, the 
board, or the VCs, depending on what stage your company is at. 


Swift integrations cost money, and most acquisitions promise cost savings 
instead of warning of increased costs due to integration. 


That's why most companies don't integrate quickly. :( 


Matt

Re: Zayo woes

Ugh... Just as much NOT ISP as ISP. 

- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Mike Hammett"  
To: "Matt Erculiani"  
Cc: nanog@nanog.org 
Sent: Tuesday, September 19, 2023 11:39:49 AM 
Subject: Re: Zayo woes 

Well sure, but what started this tangent is that to me, anyway, it seemed like 
ENA had just as much ISP as it had ISP, if not more. It would be like buying a 
farm to get farm-fresh eggs in your kitchen. 

- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Matt Erculiani"  
To: "Anne Mitchell"  
Cc: nanog@nanog.org 
Sent: Tuesday, September 19, 2023 11:06:57 AM 
Subject: Re: Zayo woes 

> they are left with having to run something that they never really wanted 
> until they can figure out what to do with it 

Buy enough Dark Fiber providers, you'll eventually acquire an ISP 
Buy enough ISPs, you'll eventually acquire a Colo 
Buy enough Colos, you'll eventually acquire a small Cloud 
Buy any small Clouds and you're guaranteed to acquire managed services for 
small companies. 

Now you've gone from being Layer 0 only catering whales, to dabbling in Layer 7 
at Dr. Morris's Family dentistry. 

Spin off what you don't want, and repeat. 

-Matt 

On Tue, Sep 19, 2023 at 9:40 AM Anne Mitchell < amitch...@isipp.com > wrote: 

> On Sep 19, 2023, at 9:23 AM, Mark Tinka  wrote: 
> 
> Sometimes it does not add any material value to either network. Sometimes it 
> takes too long. 

And sometimes the acquisition is really just about acquiring the assets, such 
as the customer list*, and then they are left with having to run something that 
they never really wanted until they can figure out what to do with it. 

*Yes, even in these industries. Do I sound cynical? Well, cynical != not 
accurate. :-( 

Anne 

-- 
Anne P. Mitchell 
Attorney at Law 
Email Law & Policy Attorney 
CEO Institute for Social Internet Public Policy (ISIPP) 
Author: Section 6 of the CAN-SPAM Act of 2003 (the Federal email marketing law) 
Author: The Email Deliverability Handbook 
Board of Directors, Denver Internet Exchange 
Dean Emeritus, Cyberlaw & Cybersecurity, Lincoln Law School 
Prof. Emeritus, Lincoln Law School 
Chair Emeritus, Asilomar Microcomputer Workshop 
Counsel Emeritus, eMail Abuse Prevention System (MAPS) 

-- 

Matt Erculiani, NREMT 
ERCUL-ARIN

Re: Zayo woes

Well sure, but what started this tangent is that to me, anyway, it seemed like 
ENA had just as much ISP as it had ISP, if not more. It would be like buying a 
farm to get farm-fresh eggs in your kitchen. 

- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Matt Erculiani"  
To: "Anne Mitchell"  
Cc: nanog@nanog.org 
Sent: Tuesday, September 19, 2023 11:06:57 AM 
Subject: Re: Zayo woes 

> they are left with having to run something that they never really wanted 
> until they can figure out what to do with it 

Buy enough Dark Fiber providers, you'll eventually acquire an ISP 
Buy enough ISPs, you'll eventually acquire a Colo 
Buy enough Colos, you'll eventually acquire a small Cloud 
Buy any small Clouds and you're guaranteed to acquire managed services for 
small companies. 

Now you've gone from being Layer 0 only catering whales, to dabbling in Layer 7 
at Dr. Morris's Family dentistry. 

Spin off what you don't want, and repeat. 

-Matt 

On Tue, Sep 19, 2023 at 9:40 AM Anne Mitchell < amitch...@isipp.com > wrote: 

> On Sep 19, 2023, at 9:23 AM, Mark Tinka  wrote: 
> 
> Sometimes it does not add any material value to either network. Sometimes it 
> takes too long. 

And sometimes the acquisition is really just about acquiring the assets, such 
as the customer list*, and then they are left with having to run something that 
they never really wanted until they can figure out what to do with it. 

*Yes, even in these industries. Do I sound cynical? Well, cynical != not 
accurate. :-( 

Anne 

-- 
Anne P. Mitchell 
Attorney at Law 
Email Law & Policy Attorney 
CEO Institute for Social Internet Public Policy (ISIPP) 
Author: Section 6 of the CAN-SPAM Act of 2003 (the Federal email marketing law) 
Author: The Email Deliverability Handbook 
Board of Directors, Denver Internet Exchange 
Dean Emeritus, Cyberlaw & Cybersecurity, Lincoln Law School 
Prof. Emeritus, Lincoln Law School 
Chair Emeritus, Asilomar Microcomputer Workshop 
Counsel Emeritus, eMail Abuse Prevention System (MAPS) 

-- 

Matt Erculiani, NREMT 
ERCUL-ARIN

Re: Zayo woes

Some of it is scale-related. Someone's operating just fine at the size they 
are, but the next order of magnitude larger enjoys many benefits from that 
size, but it takes either A) luck or B) the right skills to be able to move up 
to get those benefits. In terms of network operators, there's a big difference 
between a company of 1 and a company of 2. Again a big difference between a 
company of 2 and a company of 10. Another big difference between a company of 
10 and a company of.. I dunno, 100? 


1G waves and 100G waves aren't *THAT* different in price anymore. However, if 1 
is doing you just fine, the much better cost per bit of 100 won't do you a darn 
bit of good and will probably hurt. The hurt is not only due to the higher MRC, 
but now the higher NRC for equipment with 100G interfaces. If you can put 
enough bits on the line to make it worth it, you've got yourself tremendous 
advantage. The acquisition pays for itself in marginally higher costs for much 
higher revenue. 


The company may have been doing just fine before, but couldn't afford the move 
up to 10 or 100 because their present market couldn't justify it and the larger 
market wasn't obtainable until they had it. Catch 22 that can't really be 
overcome by most. Enter M Someone just can't get to the next level they need 
to compete with those that can. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Mark Tinka"  
To: nanog@nanog.org 
Sent: Tuesday, September 19, 2023 10:51:39 AM 
Subject: Re: Zayo woes 




On 9/19/23 17:40, Anne Mitchell wrote: 



And sometimes the acquisition is really just about acquiring the assets, such 
as the customer list*, and then they are left with having to run something that 
they never really wanted until they can figure out what to do with it. 


Right, buying the revenue to prop up the top and bottom line is also a reason 
to acquire. Usually, this is based on assessed profitability, but what tends to 
go unseen during the due diligence process is what is actually contributing to 
that profitability. 

I mean, typically, if a company is doing very well, it won't try to get itself 
sold. Well, not easily anyway... 

Mark.

Re: Zayo woes

Well sure, and I would like to think (probably mistakenly) that just no one 
important enough (to the money people) made the money people that these other 
things are *REQUIRED* to make the deal work. 

Obviously, people lower on the ladder say it all of the time, but the important 
enough money people probably don't consider those people important enough to 
listen to. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Mark Tinka"  
To: nanog@nanog.org 
Sent: Tuesday, September 19, 2023 10:28:26 AM 
Subject: Re: Zayo woes 




On 9/19/23 16:48, Mike Hammett wrote: 



As someone that has been planning to be in the acquiring seat for a while (but 
yet to do one), I've consistently passed to the money people that there's the 
purchase price and then there's the % on top of that for equipment, 
contractors, etc. to integrate, improve, optimize future cashflow, etc. those 
acquisitions with the rest of what we have. 



I blame this on the success of how well we have built the Internet with 
whatever box and tool we have, as network engineers. 

The money people assume that all routers are the same, all vendors are the 
same, all software is the same, and all features are easily deployable. And 
that all that is possible if you can simply do a better job finding the 
cheapest box compared to your competition. 

In general, I don't fancy nuance when designing for the majority. But with 
acquisition and integration, nuance is critical, and nuance quickly shows that 
the acquisition was either underestimated, or not worth doing at all. 

Mark.

Re: Zayo woes

As someone that has been planning to be in the acquiring seat for a while (but 
yet to do one), I've consistently passed to the money people that there's the 
purchase price and then there's the % on top of that for equipment, 
contractors, etc. to integrate, improve, optimize future cashflow, etc. those 
acquisitions with the rest of what we have. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Matthew Petach"  
To: "Bill Murphy"  
Cc: "Mike Hammett" , "Randy Carpenter" 
, "nanog"  
Sent: Tuesday, September 19, 2023 9:41:35 AM 
Subject: Re: Zayo woes 







On Tue, Sep 19, 2023 at 7:19AM Mike Hammett < na...@ics-il.net > wrote: 




[...] 


I've never understood companies that acquire and don't completely integrate as 
quickly as they can. 







Ah, spoken with the voice of someone who's never been in the position of: 
a) acquiring a company not-much-smaller-than-you that 
b) runs on completely different hardware and software and 
c) your executives have promised there will be cost savings after the merger 
due to "synergies" between the two companies. 
^_^; 



Let's say you're an all J shop; your scripts, your tooling, everything expects 
to be talking to J devices. 


Your executives buy a company that has almost the same size network--but it's 
all C devices running classic IOS. 


You can go to your executives and tell them "hey, to integrate quickly with our 
network and tooling, we need to swap out all their C gear for J gear; it's 
gonna cost an extra $50M" 
The executives respond by pointing at c) above, and denying the request for 
money to convert the acquired network to J. 


You can go to your network and say "hey, we need to revamp our tooling and 
systems to understand how to speak to C and J devices equally, in spite of 
wildly different syntaxes for route-maps and the like-it's going to take 4 more 
developer headcount to rewrite all the systems." 
The executives respond by pointing at c) above, and deny the request for 
developer headcount to rewrite your software systems. 


The general result of acquisitions of similar-sized companies is that the 
infrastructure runs in parallel, slowly getting converted over and unified as 
gear needs to be replaced, or sites are phased out--because any other course of 
action costs more money than the executives had promised the shareholders, the 
board, or the VCs, depending on what stage your company is at. 


Swift integrations cost money, and most acquisitions promise cost savings 
instead of warning of increased costs due to integration. 


That's why most companies don't integrate quickly. :( 


Matt

Re: Zayo woes

I blame not the network nerds, but the management that put them in that 
position. 




The problem with C is that unless the networks *ARE* integrated quickly, the 
synergies won't happen. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Matthew Petach"  
To: "Bill Murphy"  
Cc: "Mike Hammett" , "Randy Carpenter" 
, "nanog"  
Sent: Tuesday, September 19, 2023 9:41:35 AM 
Subject: Re: Zayo woes 







On Tue, Sep 19, 2023 at 7:19AM Mike Hammett < na...@ics-il.net > wrote: 




[...] 


I've never understood companies that acquire and don't completely integrate as 
quickly as they can. 







Ah, spoken with the voice of someone who's never been in the position of: 
a) acquiring a company not-much-smaller-than-you that 
b) runs on completely different hardware and software and 
c) your executives have promised there will be cost savings after the merger 
due to "synergies" between the two companies. 
^_^; 



Let's say you're an all J shop; your scripts, your tooling, everything expects 
to be talking to J devices. 


Your executives buy a company that has almost the same size network--but it's 
all C devices running classic IOS. 


You can go to your executives and tell them "hey, to integrate quickly with our 
network and tooling, we need to swap out all their C gear for J gear; it's 
gonna cost an extra $50M" 
The executives respond by pointing at c) above, and denying the request for 
money to convert the acquired network to J. 


You can go to your network and say "hey, we need to revamp our tooling and 
systems to understand how to speak to C and J devices equally, in spite of 
wildly different syntaxes for route-maps and the like-it's going to take 4 more 
developer headcount to rewrite all the systems." 
The executives respond by pointing at c) above, and deny the request for 
developer headcount to rewrite your software systems. 


The general result of acquisitions of similar-sized companies is that the 
infrastructure runs in parallel, slowly getting converted over and unified as 
gear needs to be replaced, or sites are phased out--because any other course of 
action costs more money than the executives had promised the shareholders, the 
board, or the VCs, depending on what stage your company is at. 


Swift integrations cost money, and most acquisitions promise cost savings 
instead of warning of increased costs due to integration. 


That's why most companies don't integrate quickly. :( 


Matt

Re: Zayo woes

The other way around. Zayo acquired ENA. 

That acquisition seemed odd. ENA did a lot of value add and non-IP services. 
Zayo seems to shed those on every acquisition. 

- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Bill Murphy"  
To: "Mike Hammett" , "Randy Carpenter"  
Cc: "nanog"  
Sent: Tuesday, September 19, 2023 8:53:13 AM 
Subject: Re: Zayo woes 

Recently reached out to Zayo and found out we have a new account manager, and 
also discovered they were acquired by a company called ENA... 

Bill 

From: NANOG  on behalf of 
Mike Hammett  
Sent: Tuesday, September 19, 2023 7:19 AM 
To: Randy Carpenter  
Cc: nanog  
Subject: Re: Zayo woes 

External: Increase caution when handling links and attachments. 

I've never understood companies that acquire and don't completely integrate as 
quickly as they can. 

- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

From: "Randy Carpenter"  
To: "JASON BOTHE"  
Cc: "nanog"  
Sent: Monday, September 18, 2023 7:01:03 PM 
Subject: Re: Zayo woes 

The problem we have run into is that there does not appear to be a "Zayo." 
There are dozens of acquisitions of regional providers with completely 
different infrastructure and teams and they have done a very poor job at gluing 
it all together. I have seen service orders that have gone *years* without 
being complete. There are also currently some 
breaking-the-entire-regional-network sorts of outages going on currently. I am 
guessing what clued employees they still have are quite tied up. 

-Randy 

- On Sep 18, 2023, at 7:06 PM, JASON BOTHE via NANOG nanog@nanog.org wrote: 

> Does anyone know what’s happening at Zayo? Tickets are taking weeks and 
> months 
> to get resolved, much less get a tech assigned to them. 
> 
> The folks answering the noc phone are mere order takers and are only reading 
> from a script, manager on duty/escalation lines go to voicemail. 
> 
> If anyone can help get to a human in the transport group, that would be 
> great. 
> I’ve given up all hope at this point. 
> 
> Appreciated. 
> 
> Jason

Re: Zayo woes