subject:"Re\:"

Re: [External] Opengear alternatives that support 5g?

2024-04-29 Thread Doug McIntyre

On Mon, Apr 29, 2024 at 07:04:25AM -0700, Warren Kumari wrote:
> Michel's Banana Pi BPI-R3 suggestion seems intriguing — yes, it still
> suffers from the "Now I have another "machine" to manage and patch, and
> people will try and install iperf / a Quake server / nmap / ruby / 17
> different flavors of Emacs on it", but:
> 1: Perhaps I can mitigate that by making much of the filesystem read-only
> and
> 2: it's a great excuse to buy another toy!

Another option would be a RaPi Compute Module 4 with eMMC
onboard. (ie. the non-Lite version does not need a SDCard)
They have breakout boards for the Compute Module that have spots for
5G WAN.
https://thepihut.com/products/compute-module-4-industrial-iot-base-board

PiKVM does make the file system read-only, that you have to
temporarily disable to do updates, config changes, etc). Potentially, you could
copy what they do there. 

> I also like Jared and Andrew's freetserv /
> https://lathama.net/Tech/Hardware/USB-32COM-RM option. I might see about
> building a bunch of the freetserv boards and connecting them to a Banana
> Pi…. although, more realistically, I'll likely buy a few Banana Pi's, and
> add them to the ever expanding pile of backlog projects…

One suggestion for freetserv boards I have would be to get a board printer like 
https://jlcpcb.com
to also do the assembly of the surface mount components. Last time I checked, 
they
only did one side, Not both sides.. Have them do the complex side. Pricing was
very reasonable. The back non-complex side wasn't that bad overall.

Re: [External] Opengear alternatives that support 5g?

2024-04-29 Thread Hunter Fuller via NANOG

I certainly don't blame you for your frustrations with abusing MikroTiks as
a serial console. The additional computer (Pi or otherwise) is, imo, a
must. Unless you are just using the MIkroTik as an ssh jump box into the
OOB network, which isn't so bad.

-- 
Hunter Fuller (they)
Lead Router Jockey
VBH M-1C
+1 256 824 5331

Office of Information Technology
The University of Alabama in Huntsville
Network Engineering

Re: [External] Opengear alternatives that support 5g?

2024-04-29 Thread Warren Kumari

On Sun, Apr 28, 2024 at 11:55 AM, Hunter Fuller  wrote:

> We use MikroTik for this. All manner of interfaces including LTE and 5G
> are available. I hear you can connect USB serial to them directly,
>

Yup, that's the solution I mentioned above with #5:
"5: actually be designed as a termserver - the current thing we are using
doesn't really understand terminals, and so we need to use 'socat
-,raw,echo=0,escape=0x1d TCP::' to get things like
tab-completion and "up-arrow for last command" to work."

Most term-server type things allow you to ssh / telnet to a TCP port and
the device will expose the serial port wit some useful emulation.

The Mikrotik seems to only expose the serial interface this way using RFC
2217, which means that you need you need to use software that understands
virtual Com ports (like 'Serial' on OSX, Tactical Software's "COM Port
Redirector", or PuTTY or, on Unixes 'socat'). This is far from convenient….

Michel's Banana Pi BPI-R3 suggestion seems intriguing — yes, it still
suffers from the "Now I have another "machine" to manage and patch, and
people will try and install iperf / a Quake server / nmap / ruby / 17
different flavors of Emacs on it", but:
1: Perhaps I can mitigate that by making much of the filesystem read-only
and
2: it's a great excuse to buy another toy!

I also like Jared and Andrew's freetserv /
https://lathama.net/Tech/Hardware/USB-32COM-RM option. I might see about
building a bunch of the freetserv boards and connecting them to a Banana
Pi…. although, more realistically, I'll likely buy a few Banana Pi's, and
add them to the ever expanding pile of backlog projects…

I'm not really sure what happened to AirConsole — when they initially
launched they were great. They were making new devices with new
capabilities, they were updating the software regularly, they had great
support, etc. At some point it feels like they changed management and
everything sort of stopped…

W

but we also drop a surplus Dell OptiPlex at each location and attach the
> serial ports to that device. Total cost is <200 USD per site since we
> already have the older desktops laying around.
>

> --
> Hunter Fuller (they)
> Lead Router Jockey
> VBH M-1C
> +1 256 824 5331
>
> Office of Information Technology
> The University of Alabama in Huntsville
> Network Engineering
>

Re: Opengear alternatives that support 5g?

2024-04-28 Thread Mark Tinka





On 4/28/24 18:54, Siyuan Miao wrote:

We use GL.inet and set up WireGuard VPNs back to our distributed VPN 
servers. Our console servers support dual uplink, so we just connect 
port 1 to the GL.inet LAN and port 2 to our management switch.


Currently, we're still using their LTE model, and it costs ~100 USD 
per site, but their 5G models are expensive and cost around $500.


At new job, I am looking at using pfSense-based VPN's to create the DCN. 
It does consume 1U and a couple of cabinet watts for the server, but 
it's stable, feature-rich, well-supported, and network media agnostic.


Mark.

Re: Opengear alternatives that support 5g?

2024-04-28 Thread Siyuan Miao via NANOG

We use GL.inet and set up WireGuard VPNs back to our distributed VPN
servers. Our console servers support dual uplink, so we just connect port 1
to the GL.inet LAN and port 2 to our management switch.

Currently, we're still using their LTE model, and it costs ~100 USD per
site, but their 5G models are expensive and cost around $500.

On Sun, Apr 28, 2024 at 6:33 PM Mark Tinka  wrote:

>
>
> On 4/27/24 17:49, Mel Beckman wrote:
> > Quite often I’m looking for OOBM at antenna sites or in remote DCs where
> there is no Plan B carrier. Cellular has always been the goto choice for
> this, but we keep getting pushed out of contracts by technology upgrades.
> 2g, then 3g, and  next 4g LTE are being deprecated.
> >
> > The main reason for network shutdowns is that the carriers have limited
> spectrum available for expansion. To deliver faster, more cost effective
> data service to customers, carriers must re-use existing spectrum licenses
> with newer, more efficient cellular technology. Old 2G/3G infrastructure
> makes way for new networks, and older cellular devices must be retired. 4g
> may have a decade left before complete absence, but its footprint is
> already shrinking where 5G is available.
> >
> > I’ve seen this first hand with 4g cellular alarm circuits: suddenly they
> get less reliable or fail completely, and the reason always turns out to be
> degraded RSSI due to 5G deployment.
> >
> > So 5G is imperative for cellular OOBM, hence the hunt for COTS drop-in
> replacements that won’t break the bank. Upgrading, for example, 100 antenna
> sites is also a major truck roll cost, so we want to get it right the first
> time.  Physical space and power limitations usually rule out 1U rackmount
> refurb Cisco terminal servers, which is why we need 0U gear. Yes, I can
> cobble together a raspberry pi and some hats and cables and dingles and
> dangles and make a science fair solution. But I need something that is
> commercially supported, won’t have me scratching my head later about what
> version of the Ubuntu is going to work, and won’t randomly fry its
> electronics during a power surge.
> >
> > It’s looking like that solution is firmly priced at ~$500 today.
>
> Fair enough - if the bulk of your OoB use-case is remote (cell) sites,
> your typical options won't work or will be limited.
>
> Oddly, in our parts, we find remote, non-city locations, tend to keep
> their 3G/4G status, or don't even get considered for 5G at all. But I
> guess this will vary by market the world over, so I could see a remote
> site in Norway, for example, having 5G vs. a remote site in, say, Egypt,
> doing the same.
>
> I think what you probably want to consider for the long-term is
> decoupling the device from the network media. If you can attach a MiFi
> router via a USB port to a cheap device (like Mikrotik), this would help
> keep costs down as mobile operators deprecate GSM data technologies in
> the future. I like Mikrotik because in addition to being cheap and
> feature-rich for basic network access, the firmware is regularly
> upgradeable unlike typical consumer-style CPE's.
>
> Mark.
>

Re: Opengear alternatives that support 5g?

2024-04-28 Thread Mark Tinka





On 4/27/24 17:49, Mel Beckman wrote:

Quite often I’m looking for OOBM at antenna sites or in remote DCs where there 
is no Plan B carrier. Cellular has always been the goto choice for this, but we 
keep getting pushed out of contracts by technology upgrades. 2g, then 3g, and  
next 4g LTE are being deprecated.

The main reason for network shutdowns is that the carriers have limited 
spectrum available for expansion. To deliver faster, more cost effective data 
service to customers, carriers must re-use existing spectrum licenses with 
newer, more efficient cellular technology. Old 2G/3G infrastructure makes way 
for new networks, and older cellular devices must be retired. 4g may have a 
decade left before complete absence, but its footprint is already shrinking 
where 5G is available.

I’ve seen this first hand with 4g cellular alarm circuits: suddenly they get 
less reliable or fail completely, and the reason always turns out to be 
degraded RSSI due to 5G deployment.

So 5G is imperative for cellular OOBM, hence the hunt for COTS drop-in 
replacements that won’t break the bank. Upgrading, for example, 100 antenna 
sites is also a major truck roll cost, so we want to get it right the first 
time.  Physical space and power limitations usually rule out 1U rackmount 
refurb Cisco terminal servers, which is why we need 0U gear. Yes, I can cobble 
together a raspberry pi and some hats and cables and dingles and dangles and 
make a science fair solution. But I need something that is commercially 
supported, won’t have me scratching my head later about what version of the 
Ubuntu is going to work, and won’t randomly fry its electronics during a power 
surge.

It’s looking like that solution is firmly priced at ~$500 today.


Fair enough - if the bulk of your OoB use-case is remote (cell) sites, 
your typical options won't work or will be limited.


Oddly, in our parts, we find remote, non-city locations, tend to keep 
their 3G/4G status, or don't even get considered for 5G at all. But I 
guess this will vary by market the world over, so I could see a remote 
site in Norway, for example, having 5G vs. a remote site in, say, Egypt, 
doing the same.


I think what you probably want to consider for the long-term is 
decoupling the device from the network media. If you can attach a MiFi 
router via a USB port to a cheap device (like Mikrotik), this would help 
keep costs down as mobile operators deprecate GSM data technologies in 
the future. I like Mikrotik because in addition to being cheap and 
feature-rich for basic network access, the firmware is regularly 
upgradeable unlike typical consumer-style CPE's.


Mark.

Re: [External] Opengear alternatives that support 5g?

2024-04-28 Thread Hunter Fuller via NANOG

We use MikroTik for this. All manner of interfaces including LTE and 5G are
available. I hear you can connect USB serial to them directly, but we also
drop a surplus Dell OptiPlex at each location and attach the serial ports
to that device. Total cost is <200 USD per site since we already have the
older desktops laying around.

-- 
Hunter Fuller (they)
Lead Router Jockey
VBH M-1C
+1 256 824 5331

Office of Information Technology
The University of Alabama in Huntsville
Network Engineering

Re: Opengear alternatives that support 5g?

2024-04-28 Thread ic

hi,

> On 27 Apr 2024, at 10:24, Mel Beckman  wrote:
> 
> Mike,
> 
> Both the Cradlepoint and Teltonica 5G devices are ~$600, even more than the 
> PepLink. I’ll compare features, but at first blush the Teltonica at least 
> seems to have no VPN support. 

Teltonika supports any vpn openwrt does. 

BR, ic

Re: Opengear alternatives that support 5g?

2024-04-27 Thread Mel Beckman

Mike,

Both the Cradlepoint and Teltonica 5G devices are ~$600, even more than the 
PepLink. I’ll compare features, but at first blush the Teltonica at least seems 
to have no VPN support.


-mel via cell

On Apr 26, 2024, at 11:41 PM, Mike Lyon  wrote:


Mel,

My apologies, i confused one mikrotik with another model. You are correct.

I would also check out CradlePoint and Teltonika as well.

Cheers,
Mike

On Apr 26, 2024, at 23:06, Mel Beckman  wrote:


Mike,

Thanks for that info. Alas, I’m not seeing any Mikrotik 5G devices cheaper than 
a ~$500  Peplink. Am I misunderstanding your suggested solution?

 -mel

On Apr 26, 2024, at 9:50 PM, Mike Lyon  wrote:


Peplink is nice, but there are cheaper options: Mikrotik-dot-com

Then for cellular service, sign up for an IOT with an IOT MVNO that bills usage 
based (and can also offer you a static, public, IP address AND will also allow 
you to build a VPN across all of your devices) such as SimBase:  simbase-dot-com



Cheers,
Mike

On Apr 26, 2024, at 21:37, Mel Beckman  wrote:

 I’ve been loooking at the $600 Peplink MAX BR1-MINI (HW3) industrial 5G 
router. It has a 1x embedded 5G modem (Verizon, AT, T-Mobile, and FirstNet). 
three GigE ports, four antenna connectors, and comes with an stick antenna set 
and AC PS.  It uses a nanoSIM. Yes, it’s a pure IP router with no knowledge of 
serial protocols. So I would just put an air console behind it to get to my 
serial ports. I’m still evaluating 5G plans, and Verizon just offered an 
amazing $15 per month unlimited data deal, but it seems to have a 50 gig limit 
before you get to throttling. That might not matter at all with serial traffic 
though.

We've been using the Netgear 4G cellular router, but that’s becoming 
increasingly unreliable. The NG has a nailed up IPsec VPN tunnel, obviating the 
need for a static IP, and the keepalive traffic is low enough that it doesn’t 
cost us much on the 4G network. I’m hoping 5G will be even cheaper and faster.

I’d love to see if anybody found anything better before I spring for a Peplink 
test unit.


 -mel

On Apr 26, 2024, at 9:45 AM, Warren Kumari  wrote:






On Fri, Apr 26, 2024 at 12:43 AM, Saku Ytti mailto:s...@ytti.fi>> 
wrote:

On Fri, 26 Apr 2024 at 03:11, David H 
mailto:ispcoloh...@gmail.com>> wrote:

Curious if anyone has particular hardware they like for OOB / serial 
management, similar to OpenGear, but preferably with 5G support, maybe even 
T-Mobile support? It’s becoming increasingly difficult to get static IP 4g 
machine accounts out of Verizon, and the added speed would be nice too. Or do 
you separate the serial from the access device (cell+firewall, etc.)?

You could get a 5G Catalyst with an async NIM or SM.

But I think you're setting up yourself for unnecessary costs and failures by 
designing your OOB to require static IP. You could design it so that the OOB 
spokes dial-in to the central OOB hub, and the OOB hub doesn't care what IP 
they come from, using certificates or PSK for identity, instead of IP.


Yup, I agree — but that simply rewrites the question to be:
"Curious if anyone has particular hardware they like for OOB / serial 
management, similar to OpenGear, but preferably with 5G support, which can be a 
spoke that dials in to the central OOB hub, and the OOB hub doesn't care what 
IP they come from, using certificates or PSK for identity, instead of IP."

I've been on the same quest, and I have some additional requests / features. 
Ideally it:

1: would be small - my particular use-case is for a "traveling rack", and so 0U 
is preferred.

2: would be fairly cheap.

3: would not be a Raspberry-Pi, a USB hub and USB-to-serial cables. We tried 
that for a while, and it was clunky — the SD card died a few times (and jumped 
out entirely once!), people kept futzing with the OS and fighting over which 
console software to use, installing other packages, etc.

4: support modern SSH clients (it seems like you shouldn't have to say this, 
but… )

5: actually be designed as a termserver - the current thing we are using 
doesn't really understand terminals, and so we need to use 'socat 
-,raw,echo=0,escape=0x1d TCP::' to get things like 
tab-completion and "up-arrow for last command" to work.

6: support logging of serial (e.g crash-messages) to some sort of log / buffer 
/ similar (it's useful to be able to see what a device barfed all over the 
console when it crashes.


The Get Console Airconsole TS series meets many of these requirements, but it 
doesn't do #6. It also doesn't really feel like they have been updating / 
maintaining these.

Yes, I fully acknowledge that #3 falls into the "Doctor, Doctor, it hurts when 
I do this" camp, but, well…

W

Re: Opengear alternatives that support 5g?

2024-04-27 Thread Mike Lyon

Mel,My apologies, i confused one mikrotik with another model. You are correct.I would also check out CradlePoint and Teltonika as well. Teltonika Networksteltonika-networks.comCheers,MikeOn Apr 26, 2024, at 23:06, Mel Beckman wrote:

Mike,

Thanks for that info. Alas, I’m not seeing any Mikrotik 5G devices cheaper than a ~$500 Peplink. Am I misunderstanding your suggested solution?

-mel

On Apr 26, 2024, at 9:50 PM, Mike Lyon wrote:

Peplink is nice, but there are cheaper options: Mikrotik-dot-com

Then for cellular service, sign up for an IOT with an IOT MVNO that bills usage based (and can also offer you a static, public, IP address AND will also allow you to build a VPN across all of your devices) such as SimBase:
simbase-dot-com

Cheers,
Mike

On Apr 26, 2024, at 21:37, Mel Beckman wrote:

I’ve been loooking at the $600 Peplink MAX BR1-MINI (HW3) industrial 5G router. It has a 1x embedded 5G modem (Verizon, AT, T-Mobile, and FirstNet). three GigE ports, four antenna connectors, and comes with an stick antenna set and AC PS.
It uses a nanoSIM. Yes, it’s a pure IP router with no knowledge of serial protocols. So I would just put an air console behind it to get to my serial ports. I’m still evaluating 5G plans, and Verizon just offered an amazing $15 per month unlimited data deal,
but it seems to have a 50 gig limit before you get to throttling. That might not matter at all with serial traffic though.

We've been using the Netgear 4G cellular router, but that’s becoming increasingly unreliable. The NG has a nailed up IPsec VPN tunnel, obviating the need for a static IP, and the keepalive traffic is low enough that it doesn’t cost us much on the 4G network.
I’m hoping 5G will be even cheaper and faster.

I’d love to see if anybody found anything better before I spring for a Peplink test unit.

-mel

On Apr 26, 2024, at 9:45 AM, Warren Kumari wrote:

On Fri, Apr 26, 2024 at 12:43 AM, Saku Ytti
wrote:

On Fri, 26 Apr 2024 at 03:11, David H wrote:

Curious if anyone has particular hardware they like for OOB / serial management, similar to OpenGear, but preferably with 5G support, maybe even T-Mobile support? It’s becoming increasingly difficult to get static IP 4g machine accounts out of Verizon,
and the added speed would be nice too. Or do you separate the serial from the access device (cell+firewall, etc.)?

You could get a 5G Catalyst with an async NIM or SM.

But I think you're setting up yourself for unnecessary costs and failures by designing your OOB to require static IP. You could design it so that the OOB spokes dial-in to the central OOB hub, and the OOB hub doesn't care what IP they come from,
using certificates or PSK for identity, instead of IP.

Yup, I agree — but that simply rewrites the question to be:

"Curious if anyone has particular hardware they like for OOB / serial management, similar to OpenGear, but preferably with 5G support, which can be a spoke that dials in to the central OOB hub, and the OOB hub doesn't care what IP they come from, using
certificates or PSK for identity, instead of IP."

I've been on the same quest, and I have some additional requests / features. Ideally it:

1: would be small - my particular use-case is for a "traveling rack", and so 0U is preferred.

2: would be fairly cheap.

3: would not be a Raspberry-Pi, a USB hub and USB-to-serial cables. We tried that for a while, and it was clunky — the SD card died a few times (and jumped out entirely once!), people kept futzing with the OS and fighting over which console software to
use, installing other packages, etc.

4: support modern SSH clients (it seems like you shouldn't have to say this, but… )

5: actually be designed as a termserver - the current thing we are using doesn't really understand terminals, and so we need to use 'socat -,raw,echo=0,escape=0x1d TCP::' to get things like tab-completion and "up-arrow for last command"
to work.

6: support logging of serial (e.g crash-messages) to some sort of log / buffer / similar (it's useful to be able to see what a device barfed all over the console when it crashes.

The Get Console Airconsole TS series meets many of these requirements, but it doesn't do #6. It also doesn't really feel like they have been updating / maintaining these.

Yes, I fully acknowledge that #3 falls into the "Doctor, Doctor, it hurts when I do this" camp, but, well…

--
++ytti

Re: Opengear alternatives that support 5g?

2024-04-27 Thread Mike Lyon

Peplink is nice, but there are cheaper options:MikroTikmikrotik.comThen for cellular service, sign up for an IOT with an IOT MVNO that bills usage based (and can also offer you a static, public, IP address AND will also allow you to build a VPN across all of your devices) such as SimBase: Seamless IoT SIM Card Solutionssimbase.comCheers,MikeOn Apr 26, 2024, at 21:37, Mel Beckman wrote:

I’ve been loooking at the $600 Peplink MAX BR1-MINI (HW3) industrial 5G router. It has a 1x embedded 5G modem (Verizon, AT, T-Mobile, and FirstNet). three GigE ports, four antenna connectors, and comes with an stick antenna set and AC PS. It uses a nanoSIM.
Yes, it’s a pure IP router with no knowledge of serial protocols. So I would just put an air console behind it to get to my serial ports. I’m still evaluating 5G plans, and Verizon just offered an amazing $15 per month unlimited data deal, but it seems to
have a 50 gig limit before you get to throttling. That might not matter at all with serial traffic though.

I’d love to see if anybody found anything better before I spring for a Peplink test unit.

-mel

On Apr 26, 2024, at 9:45 AM, Warren Kumari wrote:

On Fri, Apr 26, 2024 at 12:43 AM, Saku Ytti
wrote:

On Fri, 26 Apr 2024 at 03:11, David H wrote:

You could get a 5G Catalyst with an async NIM or SM.

Yup, I agree — but that simply rewrites the question to be:

I've been on the same quest, and I have some additional requests / features. Ideally it:

1: would be small - my particular use-case is for a "traveling rack", and so 0U is preferred.

2: would be fairly cheap.

4: support modern SSH clients (it seems like you shouldn't have to say this, but… )

6: support logging of serial (e.g crash-messages) to some sort of log / buffer / similar (it's useful to be able to see what a device barfed all over the console when it crashes.

The Get Console Airconsole TS series meets many of these requirements, but it doesn't do #6. It also doesn't really feel like they have been updating / maintaining these.

Yes, I fully acknowledge that #3 falls into the "Doctor, Doctor, it hurts when I do this" camp, but, well…

--
++ytti

Re: Opengear alternatives that support 5g?

2024-04-27 Thread Mel Beckman

Quite often I’m looking for OOBM at antenna sites or in remote DCs where there 
is no Plan B carrier. Cellular has always been the goto choice for this, but we 
keep getting pushed out of contracts by technology upgrades. 2g, then 3g, and  
next 4g LTE are being deprecated.

The main reason for network shutdowns is that the carriers have limited 
spectrum available for expansion. To deliver faster, more cost effective data 
service to customers, carriers must re-use existing spectrum licenses with 
newer, more efficient cellular technology. Old 2G/3G infrastructure makes way 
for new networks, and older cellular devices must be retired. 4g may have a 
decade left before complete absence, but its footprint is already shrinking 
where 5G is available.

I’ve seen this first hand with 4g cellular alarm circuits: suddenly they get 
less reliable or fail completely, and the reason always turns out to be 
degraded RSSI due to 5G deployment.

So 5G is imperative for cellular OOBM, hence the hunt for COTS drop-in 
replacements that won’t break the bank. Upgrading, for example, 100 antenna 
sites is also a major truck roll cost, so we want to get it right the first 
time.  Physical space and power limitations usually rule out 1U rackmount 
refurb Cisco terminal servers, which is why we need 0U gear. Yes, I can cobble 
together a raspberry pi and some hats and cables and dingles and dangles and 
make a science fair solution. But I need something that is commercially 
supported, won’t have me scratching my head later about what version of the 
Ubuntu is going to work, and won’t randomly fry its electronics during a power 
surge.

It’s looking like that solution is firmly priced at ~$500 today. 

 -mel

> On Apr 27, 2024, at 4:59 AM, Mark Tinka  wrote:
> 
> 
> 
>> On 4/27/24 07:56, Saku Ytti wrote:
>> 
>> 
>>  For me Cisco is great here, because it's something an organisation
>> already knows how to source, turn-up, upgrade, troubleshoot, maintain.
>> And you get a broad set of features you might want, IPSEC, DMVPN, BGP,
>> ISIS, and so forth.
> 
> I tend to agree.
> 
> Cisco do this very well, and if you are really low on cash and okay with 
> acquiring these on the cheap, the open market has tons of deals and options 
> from Cisco that have matured over the decades.
> 
> 
> 
>> I keep wondering why everyone is so focused on OOB hardware cost, when
>> in my experience the ethernet connection is ~200-300USD (150USD can be
>> just xconn) MRC. So in 10 years, you'll pay 24k to 36k just for the
>> OOB WAN, masking the hardware price. And 10years, to me, doesn't sound
>> even particularly long a time for a console setup.
> 
> Is a 10Mbps DIA link going for US$200 - US$300 MRC nowadays, excluding the 
> x-connect? I'd have though it's now in US$100 range at the very worst.
> 
> Or are you looking at an OoB link of more than 10Mbps?
> 
> Mark.

Re: Opengear alternatives that support 5g?

2024-04-27 Thread Mark Tinka




On 4/27/24 07:56, Saku Ytti wrote:


For me Cisco is great here, because it's something an organisation
already knows how to source, turn-up, upgrade, troubleshoot, maintain.
And you get a broad set of features you might want, IPSEC, DMVPN, BGP,
ISIS, and so forth.


I tend to agree.

Cisco do this very well, and if you are really low on cash and okay with 
acquiring these on the cheap, the open market has tons of deals and 
options from Cisco that have matured over the decades.



I keep wondering why everyone is so focused on OOB hardware cost, when
in my experience the ethernet connection is ~200-300USD (150USD can be
just xconn) MRC. So in 10 years, you'll pay 24k to 36k just for the
OOB WAN, masking the hardware price. And 10years, to me, doesn't sound
even particularly long a time for a console setup.


Is a 10Mbps DIA link going for US$200 - US$300 MRC nowadays, excluding 
the x-connect? I'd have though it's now in US$100 range at the very worst.


Or are you looking at an OoB link of more than 10Mbps?

Mark.

Re: Opengear alternatives that support 5g?

2024-04-27 Thread Mel Beckman

Mike,

Thanks for that info. Alas, I’m not seeing any Mikrotik 5G devices cheaper than 
a ~$500  Peplink. Am I misunderstanding your suggested solution?

 -mel

On Apr 26, 2024, at 9:50 PM, Mike Lyon  wrote:


Peplink is nice, but there are cheaper options: Mikrotik-dot-com

Then for cellular service, sign up for an IOT with an IOT MVNO that bills usage 
based (and can also offer you a static, public, IP address AND will also allow 
you to build a VPN across all of your devices) such as SimBase:  simbase-dot-com



Cheers,
Mike

On Apr 26, 2024, at 21:37, Mel Beckman  wrote:

 I’ve been loooking at the $600 Peplink MAX BR1-MINI (HW3) industrial 5G 
router. It has a 1x embedded 5G modem (Verizon, AT, T-Mobile, and FirstNet). 
three GigE ports, four antenna connectors, and comes with an stick antenna set 
and AC PS.  It uses a nanoSIM. Yes, it’s a pure IP router with no knowledge of 
serial protocols. So I would just put an air console behind it to get to my 
serial ports. I’m still evaluating 5G plans, and Verizon just offered an 
amazing $15 per month unlimited data deal, but it seems to have a 50 gig limit 
before you get to throttling. That might not matter at all with serial traffic 
though.

We've been using the Netgear 4G cellular router, but that’s becoming 
increasingly unreliable. The NG has a nailed up IPsec VPN tunnel, obviating the 
need for a static IP, and the keepalive traffic is low enough that it doesn’t 
cost us much on the 4G network. I’m hoping 5G will be even cheaper and faster.

I’d love to see if anybody found anything better before I spring for a Peplink 
test unit.


 -mel

On Apr 26, 2024, at 9:45 AM, Warren Kumari  wrote:






On Fri, Apr 26, 2024 at 12:43 AM, Saku Ytti mailto:s...@ytti.fi>> 
wrote:

On Fri, 26 Apr 2024 at 03:11, David H 
mailto:ispcoloh...@gmail.com>> wrote:

Curious if anyone has particular hardware they like for OOB / serial 
management, similar to OpenGear, but preferably with 5G support, maybe even 
T-Mobile support? It’s becoming increasingly difficult to get static IP 4g 
machine accounts out of Verizon, and the added speed would be nice too. Or do 
you separate the serial from the access device (cell+firewall, etc.)?

You could get a 5G Catalyst with an async NIM or SM.

But I think you're setting up yourself for unnecessary costs and failures by 
designing your OOB to require static IP. You could design it so that the OOB 
spokes dial-in to the central OOB hub, and the OOB hub doesn't care what IP 
they come from, using certificates or PSK for identity, instead of IP.


Yup, I agree — but that simply rewrites the question to be:
"Curious if anyone has particular hardware they like for OOB / serial 
management, similar to OpenGear, but preferably with 5G support, which can be a 
spoke that dials in to the central OOB hub, and the OOB hub doesn't care what 
IP they come from, using certificates or PSK for identity, instead of IP."

I've been on the same quest, and I have some additional requests / features. 
Ideally it:

1: would be small - my particular use-case is for a "traveling rack", and so 0U 
is preferred.

2: would be fairly cheap.

3: would not be a Raspberry-Pi, a USB hub and USB-to-serial cables. We tried 
that for a while, and it was clunky — the SD card died a few times (and jumped 
out entirely once!), people kept futzing with the OS and fighting over which 
console software to use, installing other packages, etc.

4: support modern SSH clients (it seems like you shouldn't have to say this, 
but… )

5: actually be designed as a termserver - the current thing we are using 
doesn't really understand terminals, and so we need to use 'socat 
-,raw,echo=0,escape=0x1d TCP::' to get things like 
tab-completion and "up-arrow for last command" to work.

6: support logging of serial (e.g crash-messages) to some sort of log / buffer 
/ similar (it's useful to be able to see what a device barfed all over the 
console when it crashes.


The Get Console Airconsole TS series meets many of these requirements, but it 
doesn't do #6. It also doesn't really feel like they have been updating / 
maintaining these.

Yes, I fully acknowledge that #3 falls into the "Doctor, Doctor, it hurts when 
I do this" camp, but, well…

W




--
++ytti

Re: Opengear alternatives that support 5g?

2024-04-26 Thread Saku Ytti

On Fri, 26 Apr 2024 at 19:43, Warren Kumari  wrote:

> I've been on the same quest, and I have some additional requests / features. 
> Ideally it:
>
> 1: would be small - my particular use-case is for a "traveling rack", and so 
> 0U is preferred.
> 2: would be fairly cheap.
> 3: would not be a Raspberry-Pi, a USB hub and USB-to-serial cables. We tried 
> that for a while, and it was clunky — the SD card died a few times (and 
> jumped out entirely once!), people kept futzing with the OS and fighting over 
> which console software to use, installing other packages, etc.
> 4: support modern SSH clients (it seems like you shouldn't have to say this, 
> but… )
> 5: actually be designed as a termserver - the current thing we are using 
> doesn't really understand terminals, and so we need to use 'socat 
> -,raw,echo=0,escape=0x1d TCP::' to get things like 
> tab-completion and "up-arrow for last command" to work.
> 6: support logging of serial (e.g crash-messages) to some sort of log / 
> buffer / similar (it's useful to be able to see what a device barfed all over 
> the console when it crashes.

Decouple your needs, use whatever hardware to translate RS232 into
SSH, and then use 'conserver' to maintain 24/7 logging and
multiplexing SSH sessions to each console port. Then you have your
logs in your existing NMS box filesystem, and consistent UX
independent of hardware to reach, monitor and multiplex consoles.
For me Cisco is great here, because it's something an organisation
already knows how to source, turn-up, upgrade, troubleshoot, maintain.
And you get a broad set of features you might want, IPSEC, DMVPN, BGP,
ISIS, and so forth.

I keep wondering why everyone is so focused on OOB hardware cost, when
in my experience the ethernet connection is ~200-300USD (150USD can be
just xconn) MRC. So in 10 years, you'll pay 24k to 36k just for the
OOB WAN, masking the hardware price. And 10years, to me, doesn't sound
even particularly long a time for a console setup.

>
>
> The Get Console Airconsole TS series meets many of these requirements, but it 
> doesn't do #6. It also doesn't really feel like they have been updating / 
> maintaining these.
>
> Yes, I fully acknowledge that #3 falls into the "Doctor, Doctor, it hurts 
> when I do this" camp, but, well…
>
> W
>
>
>>
>> --
>> ++ytti
>
>

--
  ++ytti

Re: Opengear alternatives that support 5g?

2024-04-26 Thread Saku Ytti

On Fri, 26 Apr 2024 at 19:43, Warren Kumari  wrote:

>> Curious if anyone has particular hardware they like for OOB / serial 
>> management, similar to OpenGear, but preferably with 5G support, maybe even 
>> T-Mobile support? It’s becoming increasingly difficult to get static IP 4g 
>> machine accounts out of Verizon, and the added speed would be nice too. Or 
>> do you separate the serial from the access device (cell+firewall, etc.)?

Does it? To me OP implied they need 5G, because they can get static in
5G product, but not on 4G. So if need for static is solved, they can
keep existing investments.

-- 
  ++ytti

Re: Opengear alternatives that support 5g?

2024-04-26 Thread Jared Mauch

If someone wants to assemble some of the freetserv boxes, I have some of the 
PCBs and components here if you want them.

- Jared

> On Apr 26, 2024, at 1:27 PM, Andrew Latham  wrote:
> 
> If anyone is interested in https://freetserv.github.io/ but does not want to 
> build one I have sort of documented an alternative at 
> https://lathama.net/Tech/Hardware/USB-32COM-RM so you can use anything to 
> connect the 5G or dialup to
> 
> On Fri, Apr 26, 2024 at 11:21 AM Michel Blais  
> wrote:
> Doesn't meet #3 but I'm testing Banana Pi BPI-R3 and seems way better than 
> RPI for this purpose.
> 
> You need to add the mini-pci modem of your choice but their 2 SIM card slots 
> on board. There are also 5 RJ45 ports if your devices have OOB ethernet ports.
> 
> There are 2 onboard storage (NOR and NAND) and you can add a M2 SSD so it is 
> possible to have failover disks.
> 
> I also like the fact that there are 2 SFP ports. There are some places in our 
> area where the LTE / 5G network is really awful so we can use a fiber 
> wavelength instead. It depends on the same fiber but at least, doesn't depend 
> on any active devices on site.
> 
> The bad is that you still need a USB to serial ports adapter. Also, you can 
> customise OpenWRT as much as you like.
> 
> For me, it's an advantage but in your case, it seems like an issue. For the 
> OP, having several VPN options like zerotier seems like an advantage. 
> 
> Le ven. 26 avr. 2024, à 12 h 44, Warren Kumari  a écrit :
> 
> 
> 
> 
> On Fri, Apr 26, 2024 at 12:43 AM, Saku Ytti  wrote:
> On Fri, 26 Apr 2024 at 03:11, David H  wrote: 
> Curious if anyone has particular hardware they like for OOB / serial 
> management, similar to OpenGear, but preferably with 5G support, maybe even 
> T-Mobile support? It’s becoming increasingly difficult to get static IP 4g 
> machine accounts out of Verizon, and the added speed would be nice too. Or do 
> you separate the serial from the access device (cell+firewall, etc.)? 
> You could get a 5G Catalyst with an async NIM or SM. 
> But I think you're setting up yourself for unnecessary costs and failures by 
> designing your OOB to require static IP. You could design it so that the OOB 
> spokes dial-in to the central OOB hub, and the OOB hub doesn't care what IP 
> they come from, using certificates or PSK for identity, instead of IP.
> 
> 
> Yup, I agree — but that simply rewrites the question to be:
> "Curious if anyone has particular hardware they like for OOB / serial 
> management, similar to OpenGear, but preferably with 5G support, which can be 
> a spoke that dials in to the central OOB hub, and the OOB hub doesn't care 
> what IP they come from, using certificates or PSK for identity, instead of 
> IP."
> 
> I've been on the same quest, and I have some additional requests / features. 
> Ideally it:
> 
> 1: would be small - my particular use-case is for a "traveling rack", and so 
> 0U is preferred.
> 
> 2: would be fairly cheap.
> 
> 3: would not be a Raspberry-Pi, a USB hub and USB-to-serial cables. We tried 
> that for a while, and it was clunky — the SD card died a few times (and 
> jumped out entirely once!), people kept futzing with the OS and fighting over 
> which console software to use, installing other packages, etc.
> 
> 4: support modern SSH clients (it seems like you shouldn't have to say this, 
> but… )
> 
> 5: actually be designed as a termserver - the current thing we are using 
> doesn't really understand terminals, and so we need to use 'socat 
> -,raw,echo=0,escape=0x1d TCP::' to get things like 
> tab-completion and "up-arrow for last command" to work.
> 
> 6: support logging of serial (e.g crash-messages) to some sort of log / 
> buffer / similar (it's useful to be able to see what a device barfed all over 
> the console when it crashes. 
> 
> 
> The Get Console Airconsole TS series meets many of these requirements, but it 
> doesn't do #6. It also doesn't really feel like they have been updating / 
> maintaining these.
> 
> Yes, I fully acknowledge that #3 falls into the "Doctor, Doctor, it hurts 
> when I do this" camp, but, well…
> 
> W
> 
> 
> 
> -- 
> ++ytti
> 
> 
> 
> -- 
> - Andrew "lathama" Latham -

Re: Opengear alternatives that support 5g?

2024-04-26 Thread Andrew Latham

If anyone is interested in https://freetserv.github.io/ but does not want
to build one I have sort of documented an alternative at
https://lathama.net/Tech/Hardware/USB-32COM-RM so you can use anything to
connect the 5G or dialup to

On Fri, Apr 26, 2024 at 11:21 AM Michel Blais 
wrote:

> Doesn't meet #3 but I'm testing Banana Pi BPI-R3 and seems way better than
> RPI for this purpose.
>
> You need to add the mini-pci modem of your choice but their 2 SIM card
> slots on board. There are also 5 RJ45 ports if your devices have OOB
> ethernet ports.
>
> There are 2 onboard storage (NOR and NAND) and you can add a M2 SSD so it
> is possible to have failover disks.
>
> I also like the fact that there are 2 SFP ports. There are some places in
> our area where the LTE / 5G network is really awful so we can use a fiber
> wavelength instead. It depends on the same fiber but at least, doesn't
> depend on any active devices on site.
>
> The bad is that you still need a USB to serial ports adapter. Also, you
> can customise OpenWRT as much as you like.
>
> For me, it's an advantage but in your case, it seems like an issue. For
> the OP, having several VPN options like zerotier seems like an advantage.
>
> Le ven. 26 avr. 2024, à 12 h 44, Warren Kumari  a
> écrit :
>
>>
>>
>>
>>
>> On Fri, Apr 26, 2024 at 12:43 AM, Saku Ytti  wrote:
>>
>>> On Fri, 26 Apr 2024 at 03:11, David H  wrote:
>>>
>>> Curious if anyone has particular hardware they like for OOB / serial
>>> management, similar to OpenGear, but preferably with 5G support, maybe even
>>> T-Mobile support? It’s becoming increasingly difficult to get static IP 4g
>>> machine accounts out of Verizon, and the added speed would be nice too. Or
>>> do you separate the serial from the access device (cell+firewall, etc.)?
>>>
>>> You could get a 5G Catalyst with an async NIM or SM.
>>>
>>> But I think you're setting up yourself for unnecessary costs and
>>> failures by designing your OOB to require static IP. You could design it so
>>> that the OOB spokes dial-in to the central OOB hub, and the OOB hub doesn't
>>> care what IP they come from, using certificates or PSK for identity,
>>> instead of IP.
>>>
>>
>>
>> Yup, I agree — but that simply rewrites the question to be:
>> "Curious if anyone has particular hardware they like for OOB / serial
>> management, similar to OpenGear, but preferably with 5G support, which can
>> be a spoke that dials in to the central OOB hub, and the OOB hub doesn't
>> care what IP they come from, using certificates or PSK for identity,
>> instead of IP."
>>
>> I've been on the same quest, and I have some additional requests /
>> features. Ideally it:
>>
>> 1: would be small - my particular use-case is for a "traveling rack", and
>> so 0U is preferred.
>>
>> 2: would be fairly cheap.
>>
>> 3: would not be a Raspberry-Pi, a USB hub and USB-to-serial cables. We
>> tried that for a while, and it was clunky — the SD card died a few times
>> (and jumped out entirely once!), people kept futzing with the OS and
>> fighting over which console software to use, installing other packages, etc.
>>
>> 4: support modern SSH clients (it seems like you shouldn't have to say
>> this, but… )
>>
>> 5: actually be designed as a termserver - the current thing we are using
>> doesn't really understand terminals, and so we need to use 'socat
>> -,raw,echo=0,escape=0x1d TCP::' to get things like
>> tab-completion and "up-arrow for last command" to work.
>>
>> 6: support logging of serial (e.g crash-messages) to some sort of log /
>> buffer / similar (it's useful to be able to see what a device barfed all
>> over the console when it crashes.
>>
>>
>> The Get Console Airconsole TS series meets many of these requirements,
>> but it doesn't do #6. It also doesn't really feel like they have been
>> updating / maintaining these.
>>
>> Yes, I fully acknowledge that #3 falls into the "Doctor, Doctor, it hurts
>> when I do this" camp, but, well…
>>
>> W
>>
>>
>>
>>> --
>>> ++ytti
>>>
>>
>>

-- 
- Andrew "lathama" Latham -

Re: Opengear alternatives that support 5g?

2024-04-26 Thread Mel Beckman

I’ve been loooking at the $600 Peplink MAX BR1-MINI (HW3) industrial 5G router. 
It has a 1x embedded 5G modem (Verizon, AT, T-Mobile, and FirstNet). three 
GigE ports, four antenna connectors, and comes with an stick antenna set and AC 
PS.  It uses a nanoSIM. Yes, it’s a pure IP router with no knowledge of serial 
protocols. So I would just put an air console behind it to get to my serial 
ports. I’m still evaluating 5G plans, and Verizon just offered an amazing $15 
per month unlimited data deal, but it seems to have a 50 gig limit before you 
get to throttling. That might not matter at all with serial traffic though.

We've been using the Netgear 4G cellular router, but that’s becoming 
increasingly unreliable. The NG has a nailed up IPsec VPN tunnel, obviating the 
need for a static IP, and the keepalive traffic is low enough that it doesn’t 
cost us much on the 4G network. I’m hoping 5G will be even cheaper and faster.

I’d love to see if anybody found anything better before I spring for a Peplink 
test unit.


 -mel

On Apr 26, 2024, at 9:45 AM, Warren Kumari  wrote:






On Fri, Apr 26, 2024 at 12:43 AM, Saku Ytti mailto:s...@ytti.fi>> 
wrote:

On Fri, 26 Apr 2024 at 03:11, David H 
mailto:ispcoloh...@gmail.com>> wrote:

Curious if anyone has particular hardware they like for OOB / serial 
management, similar to OpenGear, but preferably with 5G support, maybe even 
T-Mobile support? It’s becoming increasingly difficult to get static IP 4g 
machine accounts out of Verizon, and the added speed would be nice too. Or do 
you separate the serial from the access device (cell+firewall, etc.)?

You could get a 5G Catalyst with an async NIM or SM.

But I think you're setting up yourself for unnecessary costs and failures by 
designing your OOB to require static IP. You could design it so that the OOB 
spokes dial-in to the central OOB hub, and the OOB hub doesn't care what IP 
they come from, using certificates or PSK for identity, instead of IP.


Yup, I agree — but that simply rewrites the question to be:
"Curious if anyone has particular hardware they like for OOB / serial 
management, similar to OpenGear, but preferably with 5G support, which can be a 
spoke that dials in to the central OOB hub, and the OOB hub doesn't care what 
IP they come from, using certificates or PSK for identity, instead of IP."

I've been on the same quest, and I have some additional requests / features. 
Ideally it:

1: would be small - my particular use-case is for a "traveling rack", and so 0U 
is preferred.

2: would be fairly cheap.

3: would not be a Raspberry-Pi, a USB hub and USB-to-serial cables. We tried 
that for a while, and it was clunky — the SD card died a few times (and jumped 
out entirely once!), people kept futzing with the OS and fighting over which 
console software to use, installing other packages, etc.

4: support modern SSH clients (it seems like you shouldn't have to say this, 
but… )

5: actually be designed as a termserver - the current thing we are using 
doesn't really understand terminals, and so we need to use 'socat 
-,raw,echo=0,escape=0x1d TCP::' to get things like 
tab-completion and "up-arrow for last command" to work.

6: support logging of serial (e.g crash-messages) to some sort of log / buffer 
/ similar (it's useful to be able to see what a device barfed all over the 
console when it crashes.


The Get Console Airconsole TS series meets many of these requirements, but it 
doesn't do #6. It also doesn't really feel like they have been updating / 
maintaining these.

Yes, I fully acknowledge that #3 falls into the "Doctor, Doctor, it hurts when 
I do this" camp, but, well…

W




--
++ytti

Re: Opengear alternatives that support 5g?

2024-04-26 Thread Michel Blais

Doesn't meet #3 but I'm testing Banana Pi BPI-R3 and seems way better than
RPI for this purpose.

You need to add the mini-pci modem of your choice but their 2 SIM card
slots on board. There are also 5 RJ45 ports if your devices have OOB
ethernet ports.

There are 2 onboard storage (NOR and NAND) and you can add a M2 SSD so it
is possible to have failover disks.

I also like the fact that there are 2 SFP ports. There are some places in
our area where the LTE / 5G network is really awful so we can use a fiber
wavelength instead. It depends on the same fiber but at least, doesn't
depend on any active devices on site.

The bad is that you still need a USB to serial ports adapter. Also, you can
customise OpenWRT as much as you like.

For me, it's an advantage but in your case, it seems like an issue. For the
OP, having several VPN options like zerotier seems like an advantage.

Le ven. 26 avr. 2024, à 12 h 44, Warren Kumari  a écrit :

>
>
>
>
> On Fri, Apr 26, 2024 at 12:43 AM, Saku Ytti  wrote:
>
>> On Fri, 26 Apr 2024 at 03:11, David H  wrote:
>>
>> Curious if anyone has particular hardware they like for OOB / serial
>> management, similar to OpenGear, but preferably with 5G support, maybe even
>> T-Mobile support? It’s becoming increasingly difficult to get static IP 4g
>> machine accounts out of Verizon, and the added speed would be nice too. Or
>> do you separate the serial from the access device (cell+firewall, etc.)?
>>
>> You could get a 5G Catalyst with an async NIM or SM.
>>
>> But I think you're setting up yourself for unnecessary costs and failures
>> by designing your OOB to require static IP. You could design it so that the
>> OOB spokes dial-in to the central OOB hub, and the OOB hub doesn't care
>> what IP they come from, using certificates or PSK for identity, instead of
>> IP.
>>
>
>
> Yup, I agree — but that simply rewrites the question to be:
> "Curious if anyone has particular hardware they like for OOB / serial
> management, similar to OpenGear, but preferably with 5G support, which can
> be a spoke that dials in to the central OOB hub, and the OOB hub doesn't
> care what IP they come from, using certificates or PSK for identity,
> instead of IP."
>
> I've been on the same quest, and I have some additional requests /
> features. Ideally it:
>
> 1: would be small - my particular use-case is for a "traveling rack", and
> so 0U is preferred.
>
> 2: would be fairly cheap.
>
> 3: would not be a Raspberry-Pi, a USB hub and USB-to-serial cables. We
> tried that for a while, and it was clunky — the SD card died a few times
> (and jumped out entirely once!), people kept futzing with the OS and
> fighting over which console software to use, installing other packages, etc.
>
> 4: support modern SSH clients (it seems like you shouldn't have to say
> this, but… )
>
> 5: actually be designed as a termserver - the current thing we are using
> doesn't really understand terminals, and so we need to use 'socat
> -,raw,echo=0,escape=0x1d TCP::' to get things like
> tab-completion and "up-arrow for last command" to work.
>
> 6: support logging of serial (e.g crash-messages) to some sort of log /
> buffer / similar (it's useful to be able to see what a device barfed all
> over the console when it crashes.
>
>
> The Get Console Airconsole TS series meets many of these requirements, but
> it doesn't do #6. It also doesn't really feel like they have been updating
> / maintaining these.
>
> Yes, I fully acknowledge that #3 falls into the "Doctor, Doctor, it hurts
> when I do this" camp, but, well…
>
> W
>
>
>
>> --
>> ++ytti
>>
>
>

Re: Opengear alternatives that support 5g?

2024-04-26 Thread Warren Kumari

On Fri, Apr 26, 2024 at 12:43 AM, Saku Ytti  wrote:

> On Fri, 26 Apr 2024 at 03:11, David H  wrote:
>
> Curious if anyone has particular hardware they like for OOB / serial
> management, similar to OpenGear, but preferably with 5G support, maybe even
> T-Mobile support? It’s becoming increasingly difficult to get static IP 4g
> machine accounts out of Verizon, and the added speed would be nice too. Or
> do you separate the serial from the access device (cell+firewall, etc.)?
>
> You could get a 5G Catalyst with an async NIM or SM.
>
> But I think you're setting up yourself for unnecessary costs and failures
> by designing your OOB to require static IP. You could design it so that the
> OOB spokes dial-in to the central OOB hub, and the OOB hub doesn't care
> what IP they come from, using certificates or PSK for identity, instead of
> IP.
>

Yup, I agree — but that simply rewrites the question to be:
"Curious if anyone has particular hardware they like for OOB / serial
management, similar to OpenGear, but preferably with 5G support, which can
be a spoke that dials in to the central OOB hub, and the OOB hub doesn't
care what IP they come from, using certificates or PSK for identity,
instead of IP."

I've been on the same quest, and I have some additional requests /
features. Ideally it:

1: would be small - my particular use-case is for a "traveling rack", and
so 0U is preferred.

2: would be fairly cheap.

3: would not be a Raspberry-Pi, a USB hub and USB-to-serial cables. We
tried that for a while, and it was clunky — the SD card died a few times
(and jumped out entirely once!), people kept futzing with the OS and
fighting over which console software to use, installing other packages, etc.

4: support modern SSH clients (it seems like you shouldn't have to say
this, but… )

5: actually be designed as a termserver - the current thing we are using
doesn't really understand terminals, and so we need to use 'socat
-,raw,echo=0,escape=0x1d TCP::' to get things like
tab-completion and "up-arrow for last command" to work.

6: support logging of serial (e.g crash-messages) to some sort of log /
buffer / similar (it's useful to be able to see what a device barfed all
over the console when it crashes.

The Get Console Airconsole TS series meets many of these requirements, but
it doesn't do #6. It also doesn't really feel like they have been updating
/ maintaining these.

Yes, I fully acknowledge that #3 falls into the "Doctor, Doctor, it hurts
when I do this" camp, but, well…

W

> --
> ++ytti
>

Re: Opengear alternatives that support 5g?

2024-04-25 Thread Saku Ytti

On Fri, 26 Apr 2024 at 03:11, David H  wrote:

> Curious if anyone has particular hardware they like for OOB / serial 
> management, similar to OpenGear, but preferably with 5G support, maybe even 
> T-Mobile support?  It’s becoming increasingly difficult to get static IP 4g 
> machine accounts out of Verizon, and the added speed would be nice too.  Or 
> do you separate the serial from the access device (cell+firewall, etc.)?

You could get a 5G Catalyst with an async NIM or SM.

But I think you're setting up yourself for unnecessary costs and
failures by designing your OOB to require static IP. You could design
it so that the OOB spokes dial-in to the central OOB hub, and the OOB
hub doesn't care what IP they come from, using certificates or PSK for
identity, instead of IP.

-- 
  ++ytti

Re: [EXTERNAL] Re: Help with removing DNS shinkhole FP from Charter/Spectrum

2024-04-23 Thread Validin Axon

Thank you, Jim. Who is the vendor responsible?

Kenneth

On Tue, Apr 23, 2024 at 4:24 PM Rampley, Jim F 
wrote:

>
>
> Hi Kenneth,
>
>
>
> We have been working internally and with our third-party domain reputation
> source to get your domain removed from their malware list.
>
>
>
> Jim
>
>
>
> *From: *NANOG  on behalf
> of Validin Axon 
> *Date: *Tuesday, April 23, 2024 at 2:15 PM
> *To: *Tom Beecher 
> *Cc: *NANOG 
> *Subject: *[EXTERNAL] Re: Help with removing DNS shinkhole FP from
> Charter/Spectrum
>
>
>
> CAUTION: The e-mail below is from an external source. Please exercise
> caution before opening attachments, clicking links, or following guidance.
>
>
>
> Tom,
>
>
>
> Thank you for this! It is very interesting that the behavior is
> intermittent. A friend of mine who tested it this weekend saw correct
> answers on IPv6 and incorrect answers on IPv4.
>
>
>
> Kenneth
>
>
>
> On Tue, Apr 23, 2024 at 2:56 PM Tom Beecher  wrote:
>
> Validin, made an interesting observation on this. I am also a Spectrum
> residential customer,  none of their equipment, run my own DNS server
> (pihole).
>
>
>
> My DHCP Assigned DNS servers are
>
> 2001:1998:f00:1::1
> 2001:1998:f00:2::1
>
> bash-3.2$ dig -x 2001:1998:f00:1::1 +short
> dns-cac-lb-01.rr.com.
> bash-3.2$ dig -x 2001:1998:f00:2::1 +short
> dns-cac-lb-02.rr.com.
> bash-3.2$
>
>
> bash-3.2$ dig dns-cac-lb-01.rr.com +short
> 209.18.47.61
> bash-3.2$ dig dns-cac-lb-02.rr.com +short
> 209.18.47.62
> bash-3.2$
>
> bash-3.2$ dig @209.18.47.61 validin.com +short
> 157.245.112.183
> 137.184.54.107
> bash-3.2$ dig @209.18.47.62 validin.com +short
> 157.245.112.183
> 137.184.54.107
> bash-3.2$
>
> bash-3.2$ dig @2001:1998:f00:1::1 validin.com +short
> 127.0.0.54
> bash-3.2$
>
> bash-3.2$ dig @2001:1998:f00:2::1 validin.com +short
> 127.0.0.54
> bash-3.2$
>
> Same servers on V4 were returning correct info, but on V6 were not.
>
> However, a few minutes later :
>
> bash-3.2$ dig @2001:1998:f00:1::1 validin.com +short
> 157.245.112.183
> 137.184.54.107
> bash-3.2$ dig @2001:1998:f00:2::1 validin.com +short
> 157.245.112.183
> 137.184.54.107
> bash-3.2$
>
> Deltas :
>
> bash-3.2$ dig @2001:1998:f00:1::1  validin.com
>
> ; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com
> ; (1 server found)
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42329
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
>
> ;; QUESTION SECTION:
> ;validin.com.   IN  A
>
> ;; ANSWER SECTION:
> validin.com.60  IN  A   127.0.0.54
>
> ;; Query time: 37 msec
> ;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1)
> ;; WHEN: Tue Apr 23 13:50:03 EDT 2024
> ;; MSG SIZE  rcvd: 45
>
> bash-3.2$
>
> bash-3.2$ dig @2001:1998:f00:1::1 validin.com
>
> ; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com
> ; (1 server found)
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9667
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
>
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 512
> ;; QUESTION SECTION:
> ;validin.com.   IN  A
>
> ;; ANSWER SECTION:
> validin.com.600 IN  A   157.245.112.183
> validin.com.600 IN  A   137.184.54.107
>
> ;; Query time: 157 msec
> ;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1)
> ;; WHEN: Tue Apr 23 14:19:20 EDT 2024
> ;; MSG SIZE  rcvd: 72
>
> bash-3.2$
>
>
>
> Seems like quite possibly they are intermittently caching bunk data from
> something.
>
>
>
>
>
> On Tue, Apr 23, 2024 at 1:39 PM Validin Axon  wrote:
>
> Hi Jason,
>
>
>
> > I suspect what’s happened is an incorrect assumption that DNS is even
> the issue here. Because you mentioned Spectrum Shield, I suspect it is not.
>
>
>
> I appreciate the response and links. However, I've been told repeatedly by
> Spectrum that they're not blocking with Spectrum Shield. Despite these
> assurances, I've filled out a removal request through their published
> removal process several times, and the response I received stated that
> we're not being blocked. This check agrees with that:
>
> https://www.spectrum.net/support/forms/verify_url_security
>
>
>
> "Security Shield Is Not Blocking This Site
>
> The URL provided is not being blocked by Spectrum Security Shield
&g

Re: [EXTERNAL] Re: Help with removing DNS shinkhole FP from Charter/Spectrum

2024-04-23 Thread Rampley, Jim F

Hi Kenneth,

We have been working internally and with our third-party domain reputation 
source to get your domain removed from their malware list.

Jim

From: NANOG  on behalf of 
Validin Axon 
Date: Tuesday, April 23, 2024 at 2:15 PM
To: Tom Beecher 
Cc: NANOG 
Subject: [EXTERNAL] Re: Help with removing DNS shinkhole FP from 
Charter/Spectrum

CAUTION: The e-mail below is from an external source. Please exercise caution 
before opening attachments, clicking links, or following guidance.

Tom,

Thank you for this! It is very interesting that the behavior is intermittent. A 
friend of mine who tested it this weekend saw correct answers on IPv6 and 
incorrect answers on IPv4.

Kenneth

On Tue, Apr 23, 2024 at 2:56 PM Tom Beecher 
mailto:beec...@beecher.cc>> wrote:
Validin, made an interesting observation on this. I am also a Spectrum 
residential customer,  none of their equipment, run my own DNS server (pihole).

My DHCP Assigned DNS servers are

2001:1998:f00:1::1
2001:1998:f00:2::1

bash-3.2$ dig -x 2001:1998:f00:1::1 +short
dns-cac-lb-01.rr.com<http://dns-cac-lb-01.rr.com>.
bash-3.2$ dig -x 2001:1998:f00:2::1 +short
dns-cac-lb-02.rr.com<http://dns-cac-lb-02.rr.com>.
bash-3.2$

bash-3.2$ dig dns-cac-lb-01.rr.com<http://dns-cac-lb-01.rr.com> +short
209.18.47.61
bash-3.2$ dig dns-cac-lb-02.rr.com<http://dns-cac-lb-02.rr.com> +short
209.18.47.62
bash-3.2$

bash-3.2$ dig @209.18.47.61<http://209.18.47.61> 
validin.com<http://validin.com> +short
157.245.112.183
137.184.54.107
bash-3.2$ dig @209.18.47.62<http://209.18.47.62> 
validin.com<http://validin.com> +short
157.245.112.183
137.184.54.107
bash-3.2$

bash-3.2$ dig @2001:1998:f00:1::1 validin.com<http://validin.com> +short
127.0.0.54
bash-3.2$

bash-3.2$ dig @2001:1998:f00:2::1 validin.com<http://validin.com> +short
127.0.0.54
bash-3.2$

Same servers on V4 were returning correct info, but on V6 were not.

However, a few minutes later :

bash-3.2$ dig @2001:1998:f00:1::1 validin.com<http://validin.com> +short
157.245.112.183
137.184.54.107
bash-3.2$ dig @2001:1998:f00:2::1 validin.com<http://validin.com> +short
157.245.112.183
137.184.54.107
bash-3.2$

Deltas :

bash-3.2$ dig @2001:1998:f00:1::1  validin.com<http://validin.com>

; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com<http://validin.com>
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42329
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;validin.com<http://validin.com>.   IN  A

;; ANSWER SECTION:
validin.com<http://validin.com>.60  IN  A   127.0.0.54

;; Query time: 37 msec
;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1)
;; WHEN: Tue Apr 23 13:50:03 EDT 2024
;; MSG SIZE  rcvd: 45

bash-3.2$

bash-3.2$ dig @2001:1998:f00:1::1 validin.com<http://validin.com>

; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com<http://validin.com>
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9667
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;validin.com<http://validin.com>.   IN  A

;; ANSWER SECTION:
validin.com<http://validin.com>.600 IN  A   
157.245.112.183
validin.com<http://validin.com>.600 IN  A   
137.184.54.107

;; Query time: 157 msec
;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1)
;; WHEN: Tue Apr 23 14:19:20 EDT 2024
;; MSG SIZE  rcvd: 72

bash-3.2$

Seems like quite possibly they are intermittently caching bunk data from 
something.

On Tue, Apr 23, 2024 at 1:39 PM Validin Axon 
mailto:a...@validin.com>> wrote:
Hi Jason,

> I suspect what’s happened is an incorrect assumption that DNS is even the 
> issue here. Because you mentioned Spectrum Shield, I suspect it is not.

I appreciate the response and links. However, I've been told repeatedly by 
Spectrum that they're not blocking with Spectrum Shield. Despite these 
assurances, I've filled out a removal request through their published removal 
process several times, and the response I received stated that we're not being 
blocked. This check agrees with that:
https://www.spectrum.net/support/forms/verify_url_security

"Security Shield Is Not Blocking This Site
The URL provided is not being blocked by Spectrum Security Shield
The URL you entered should be accessible."
Further, checking Spectrum DNS servers on the Spectrum network show that my 
company's main domain and all subdomains resolve to 127.0.0.54. So, if 
CujoAI/Spectrum Shield are not using DNS query responses to control access, 
then it's not CujoAI/Spectrum Shi

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

2024-04-23 Thread Validin Axon

Tom,

Thank you for this! It is very interesting that the behavior is
intermittent. A friend of mine who tested it this weekend saw correct
answers on IPv6 and incorrect answers on IPv4.

Kenneth

On Tue, Apr 23, 2024 at 2:56 PM Tom Beecher  wrote:

> Validin, made an interesting observation on this. I am also a Spectrum
> residential customer,  none of their equipment, run my own DNS server
> (pihole).
>
> My DHCP Assigned DNS servers are
>
> 2001:1998:f00:1::1
> 2001:1998:f00:2::1
>
> bash-3.2$ dig -x 2001:1998:f00:1::1 +short
> dns-cac-lb-01.rr.com.
> bash-3.2$ dig -x 2001:1998:f00:2::1 +short
> dns-cac-lb-02.rr.com.
> bash-3.2$
>
>
> bash-3.2$ dig dns-cac-lb-01.rr.com +short
> 209.18.47.61
> bash-3.2$ dig dns-cac-lb-02.rr.com +short
> 209.18.47.62
> bash-3.2$
>
> bash-3.2$ dig @209.18.47.61 validin.com +short
> 157.245.112.183
> 137.184.54.107
> bash-3.2$ dig @209.18.47.62 validin.com +short
> 157.245.112.183
> 137.184.54.107
> bash-3.2$
>
> bash-3.2$ dig @2001:1998:f00:1::1 validin.com +short
> 127.0.0.54
> bash-3.2$
>
> bash-3.2$ dig @2001:1998:f00:2::1 validin.com +short
> 127.0.0.54
> bash-3.2$
>
> Same servers on V4 were returning correct info, but on V6 were not.
>
> However, a few minutes later :
>
> bash-3.2$ dig @2001:1998:f00:1::1 validin.com +short
> 157.245.112.183
> 137.184.54.107
> bash-3.2$ dig @2001:1998:f00:2::1 validin.com +short
> 157.245.112.183
> 137.184.54.107
> bash-3.2$
>
> Deltas :
>
> bash-3.2$ dig @2001:1998:f00:1::1  validin.com
>
> ; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com
> ; (1 server found)
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42329
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
>
> ;; QUESTION SECTION:
> ;validin.com.   IN  A
>
> ;; ANSWER SECTION:
> validin.com.60  IN  A   127.0.0.54
>
> ;; Query time: 37 msec
> ;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1)
> ;; WHEN: Tue Apr 23 13:50:03 EDT 2024
> ;; MSG SIZE  rcvd: 45
>
> bash-3.2$
>
> bash-3.2$ dig @2001:1998:f00:1::1 validin.com
>
> ; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com
> ; (1 server found)
> ;; global options: +cmd
> ;; Got answer:
> ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9667
> ;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
>
> ;; OPT PSEUDOSECTION:
> ; EDNS: version: 0, flags:; udp: 512
> ;; QUESTION SECTION:
> ;validin.com.   IN  A
>
> ;; ANSWER SECTION:
> validin.com.600 IN  A   157.245.112.183
> validin.com.600 IN  A   137.184.54.107
>
> ;; Query time: 157 msec
> ;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1)
> ;; WHEN: Tue Apr 23 14:19:20 EDT 2024
> ;; MSG SIZE  rcvd: 72
>
> bash-3.2$
>
> Seems like quite possibly they are intermittently caching bunk data from
> something.
>
>
> On Tue, Apr 23, 2024 at 1:39 PM Validin Axon  wrote:
>
>> Hi Jason,
>>
>> > I suspect what’s happened is an incorrect assumption that DNS is even
>> the issue here. Because you mentioned Spectrum Shield, I suspect it is not.
>>
>> I appreciate the response and links. However, I've been told repeatedly
>> by Spectrum that they're not blocking with Spectrum Shield. Despite these
>> assurances, I've filled out a removal request through their published
>> removal process several times, and the response I received stated that
>> we're not being blocked. This check agrees with that:
>> https://www.spectrum.net/support/forms/verify_url_security
>>
>> "Security Shield Is Not Blocking This Site
>> The URL provided is not being blocked by Spectrum Security Shield
>> The URL you entered should be accessible."
>>
>> Further, checking Spectrum DNS servers on the Spectrum network show that
>> my company's main domain and all subdomains resolve to 127.0.0.54. So, if
>> CujoAI/Spectrum Shield are not using DNS query responses to control access,
>> then it's not CujoAI/Spectrum Shield that is responsible for the incorrect
>> DNS response. Using a different recursive resolve, I can resolve our
>> domains just fine. I can also resolve other domains that point to the same
>> IPs as the sinkholed domain just fine. However, many people use the
>> Spectrum default DNS servers and cannot access my website because of this.
>>
>> > You should contact Charter/Spectrum to have them investigate what their
>> system might be blocking this content.
>>
>> I have tried, for months, including spending many hours on chat and phone
>> support, to reach someone within Spectrum support who is capable of both
>> understanding and directing me to someone who can fix the problem, but it
>> hasn't happened yet. I've asked to talk to someone on the DNS team and was
>> given a flat "No." I've posted here hoping that someone in the
>> ISP-connected world knows SOMEONE at Spectrum, Akamai, or whichever company
>> is actually responsible for the Spectrum DNS servers who can provide a
>> remediation path.
>>
>>

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

2024-04-23 Thread Tom Beecher

Validin, made an interesting observation on this. I am also a Spectrum
residential customer,  none of their equipment, run my own DNS server
(pihole).

My DHCP Assigned DNS servers are

2001:1998:f00:1::1
2001:1998:f00:2::1

bash-3.2$ dig -x 2001:1998:f00:1::1 +short
dns-cac-lb-01.rr.com.
bash-3.2$ dig -x 2001:1998:f00:2::1 +short
dns-cac-lb-02.rr.com.
bash-3.2$


bash-3.2$ dig dns-cac-lb-01.rr.com +short
209.18.47.61
bash-3.2$ dig dns-cac-lb-02.rr.com +short
209.18.47.62
bash-3.2$

bash-3.2$ dig @209.18.47.61 validin.com +short
157.245.112.183
137.184.54.107
bash-3.2$ dig @209.18.47.62 validin.com +short
157.245.112.183
137.184.54.107
bash-3.2$

bash-3.2$ dig @2001:1998:f00:1::1 validin.com +short
127.0.0.54
bash-3.2$

bash-3.2$ dig @2001:1998:f00:2::1 validin.com +short
127.0.0.54
bash-3.2$

Same servers on V4 were returning correct info, but on V6 were not.

However, a few minutes later :

bash-3.2$ dig @2001:1998:f00:1::1 validin.com +short
157.245.112.183
137.184.54.107
bash-3.2$ dig @2001:1998:f00:2::1 validin.com +short
157.245.112.183
137.184.54.107
bash-3.2$

Deltas :

bash-3.2$ dig @2001:1998:f00:1::1  validin.com

; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 42329
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;validin.com.   IN  A

;; ANSWER SECTION:
validin.com.60  IN  A   127.0.0.54

;; Query time: 37 msec
;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1)
;; WHEN: Tue Apr 23 13:50:03 EDT 2024
;; MSG SIZE  rcvd: 45

bash-3.2$

bash-3.2$ dig @2001:1998:f00:1::1 validin.com

; <<>> DiG 9.10.6 <<>> @2001:1998:f00:1::1 validin.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9667
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;validin.com.   IN  A

;; ANSWER SECTION:
validin.com.600 IN  A   157.245.112.183
validin.com.600 IN  A   137.184.54.107

;; Query time: 157 msec
;; SERVER: 2001:1998:f00:1::1#53(2001:1998:f00:1::1)
;; WHEN: Tue Apr 23 14:19:20 EDT 2024
;; MSG SIZE  rcvd: 72

bash-3.2$

Seems like quite possibly they are intermittently caching bunk data from
something.


On Tue, Apr 23, 2024 at 1:39 PM Validin Axon  wrote:

> Hi Jason,
>
> > I suspect what’s happened is an incorrect assumption that DNS is even
> the issue here. Because you mentioned Spectrum Shield, I suspect it is not.
>
> I appreciate the response and links. However, I've been told repeatedly by
> Spectrum that they're not blocking with Spectrum Shield. Despite these
> assurances, I've filled out a removal request through their published
> removal process several times, and the response I received stated that
> we're not being blocked. This check agrees with that:
> https://www.spectrum.net/support/forms/verify_url_security
>
> "Security Shield Is Not Blocking This Site
> The URL provided is not being blocked by Spectrum Security Shield
> The URL you entered should be accessible."
>
> Further, checking Spectrum DNS servers on the Spectrum network show that
> my company's main domain and all subdomains resolve to 127.0.0.54. So, if
> CujoAI/Spectrum Shield are not using DNS query responses to control access,
> then it's not CujoAI/Spectrum Shield that is responsible for the incorrect
> DNS response. Using a different recursive resolve, I can resolve our
> domains just fine. I can also resolve other domains that point to the same
> IPs as the sinkholed domain just fine. However, many people use the
> Spectrum default DNS servers and cannot access my website because of this.
>
> > You should contact Charter/Spectrum to have them investigate what their
> system might be blocking this content.
>
> I have tried, for months, including spending many hours on chat and phone
> support, to reach someone within Spectrum support who is capable of both
> understanding and directing me to someone who can fix the problem, but it
> hasn't happened yet. I've asked to talk to someone on the DNS team and was
> given a flat "No." I've posted here hoping that someone in the
> ISP-connected world knows SOMEONE at Spectrum, Akamai, or whichever company
> is actually responsible for the Spectrum DNS servers who can provide a
> remediation path.
>
> Regards,
>
> Kenneth
>
> On Tue, Apr 23, 2024 at 12:59 PM 'Livingood, Jason' via axon <
> a...@validin.com> wrote:
>
>> > However, there's no correction process for Spectrum's DNS sinkhole
>>
>> > But back to the topic: someone mentioned to me that Spectrum may not be
>> the direct providers for the DNS services they provide to their customers.
>> If anyone knows anything about how I might discover and reach out to the
>> people responsible, please let me know.
>>

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

2024-04-23 Thread Validin Axon

Hi Jason,

> I suspect what’s happened is an incorrect assumption that DNS is even the
issue here. Because you mentioned Spectrum Shield, I suspect it is not.

I appreciate the response and links. However, I've been told repeatedly by
Spectrum that they're not blocking with Spectrum Shield. Despite these
assurances, I've filled out a removal request through their published
removal process several times, and the response I received stated that
we're not being blocked. This check agrees with that:
https://www.spectrum.net/support/forms/verify_url_security

"Security Shield Is Not Blocking This Site
The URL provided is not being blocked by Spectrum Security Shield
The URL you entered should be accessible."

Further, checking Spectrum DNS servers on the Spectrum network show that my
company's main domain and all subdomains resolve to 127.0.0.54. So, if
CujoAI/Spectrum Shield are not using DNS query responses to control access,
then it's not CujoAI/Spectrum Shield that is responsible for the incorrect
DNS response. Using a different recursive resolve, I can resolve our
domains just fine. I can also resolve other domains that point to the same
IPs as the sinkholed domain just fine. However, many people use the
Spectrum default DNS servers and cannot access my website because of this.

> You should contact Charter/Spectrum to have them investigate what their
system might be blocking this content.

I have tried, for months, including spending many hours on chat and phone
support, to reach someone within Spectrum support who is capable of both
understanding and directing me to someone who can fix the problem, but it
hasn't happened yet. I've asked to talk to someone on the DNS team and was
given a flat "No." I've posted here hoping that someone in the
ISP-connected world knows SOMEONE at Spectrum, Akamai, or whichever company
is actually responsible for the Spectrum DNS servers who can provide a
remediation path.

Regards,

Kenneth

On Tue, Apr 23, 2024 at 12:59 PM 'Livingood, Jason' via axon <
a...@validin.com> wrote:

> > However, there's no correction process for Spectrum's DNS sinkhole
>
> > But back to the topic: someone mentioned to me that Spectrum may not be
> the direct providers for the DNS services they provide to their customers.
> If anyone knows anything about how I might discover and reach out to the
> people responsible, please let me know.
>
>
>
> I suspect what’s happened is an incorrect assumption that DNS is even the
> issue here. Because you mentioned Spectrum Shield, I suspect it is not.
>
> Spectrum Shield (
> https://www.spectrum.com/resources/internet-wifi/benefits-of-spectrum-security-shield)
> is a customer-managed security protection service built into their gateways
> (I assume you can turn it off). The malware and content detection engine
> behind that is very likely run by CujoAI (https://cujo.com/) and it does
> not use DNS query/response exchanges as the control mechanism (in part to
> counter-act DNS-changing malware or malware using its own DoH channel for
> example).
>
> You should contact Charter/Spectrum to have them investigate what their
> system might be blocking this content.
>
> Comcast (where I work) runs a similar system (
> https://www.xfinity.com/support/articles/using-xfinity-xfi-advanced-security)
> and maintains a site to report these sorts of issues (
> https://www.xfinity.com/support/articles/report-blocked-website).
>
> Jason
>
>
>
>
>
>
>
>
>

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

2024-04-23 Thread Livingood, Jason via NANOG

> However, there's no correction process for Spectrum's DNS sinkhole
> But back to the topic: someone mentioned to me that Spectrum may not be the 
> direct providers for the DNS services they provide to their customers. If 
> anyone knows anything about how I might discover and reach out to the people 
> responsible, please let me know.

I suspect what’s happened is an incorrect assumption that DNS is even the issue 
here. Because you mentioned Spectrum Shield, I suspect it is not.

Spectrum Shield 
(https://www.spectrum.com/resources/internet-wifi/benefits-of-spectrum-security-shield)
 is a customer-managed security protection service built into their gateways (I 
assume you can turn it off). The malware and content detection engine behind 
that is very likely run by CujoAI (https://cujo.com/) and it does not use DNS 
query/response exchanges as the control mechanism (in part to counter-act 
DNS-changing malware or malware using its own DoH channel for example).

You should contact Charter/Spectrum to have them investigate what their system 
might be blocking this content.

Comcast (where I work) runs a similar system 
(https://www.xfinity.com/support/articles/using-xfinity-xfi-advanced-security) 
and maintains a site to report these sorts of issues 
(https://www.xfinity.com/support/articles/report-blocked-website).

Jason

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

2024-04-22 Thread John R. Levine


I'm not sure where you saw that message, but I got this message via email
after I submitted an unblock request with Spectrum Shield:


We have reviewed your request to unblock validin.com. This site was not

found to be blocked by Spectrum Shield and should be accessible from your
browser.


Sigh.


I've cleaned up everything I could from that botched blocklist aggregation.
However, there's no correction process for Spectrum's DNS sinkhole, and I'm
not even sure that's how our domain got mixed up there. The support staff
I've spoken with have denied the existence of DNS sinkholing at Spectrum,
and demonstrated they lack the basic technical sophistication needed to
understand the concept.


Yeah, that's the problem.  And given stuff like this link below, I 
wouldn't expect their legal department to be any better.  Clearly there is 
someone somewhere who is competent because their network mostly works, but 
damned if I know how to find them.


https://www.theverge.com/2022/7/29/23282522/charter-spectrum-customer-murder-forged-terms-of-service

R's,
John

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

2024-04-22 Thread John R. Levine

Bill is absolutely correct. The spammers lost their case because they 
were demonstrably spammers.


No, really they did not.  I read the decisions.  Have you?  Hint: under 
CAN SPAM a great deal of spam is completely legal so it didn't matter.


We’ve had accidental black hole cases with *US* providers that removed 
the block once they received a C If they don’t have iron clad proof 
in hand. (More than just a few complaints and no traffic analysis), it’s 
just the least risky response.


I will believe that there are people that cave in response to threats like 
this, but again, there is no case law to support it.


R's,
John

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

2024-04-22 Thread William Herrin

On Mon, Apr 22, 2024 at 5:54 PM Validin Axon  wrote:
> Hi Bill,
>
> I'm not sure where you saw that message, but I got this
> message via email after I submitted an unblock request with Spectrum Shield:

Howdy,

That was Christopher, not me. But you should check the talos link I
sent you privately. Also https://ipcheck.proofpoint.com/. Whatever
they're detecting, it didn't happen last year.

Regards,
Bill Herrin

-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

2024-04-22 Thread Validin Axon

Hi Bill,

I'm not sure where you saw that message, but I got this message via email
after I submitted an unblock request with Spectrum Shield:

> We have reviewed your request to unblock validin.com. This site was not
found to be blocked by Spectrum Shield and should be accessible from your
browser.
>
> Thank you,
>
> Spectrum

My company's domain got caught up in some lazy copy/pasting from this blog
post last year that cited my company as a source for the data. Someone
copy/pasted the whole page, which included my company's domain name, and
that made it to a few AV OTX pulses and VT collections:
https://gi7w0rm.medium.com/uncovering-ddgroup-a-long-time-threat-actor-d3b3020625a4

I've cleaned up everything I could from that botched blocklist aggregation.
However, there's no correction process for Spectrum's DNS sinkhole, and I'm
not even sure that's how our domain got mixed up there. The support staff
I've spoken with have denied the existence of DNS sinkholing at Spectrum,
and demonstrated they lack the basic technical sophistication needed to
understand the concept. They've each ultimately told me that each affected
customer would need to reach out to the Spectrum customer service, which
would then help that customer change their DNS settings to another DNS
provider. Of course, the last thing I'd want to do with a potential
customer is ask them to go through that painful process. I also have no
idea how many potential users or customers can't reach me and simply give
up without letting me know.

Lastly, I AM a Spectrum customer. My home internet service is Spectrum. If
it weren't for that, I'd be truly SOL because support would just ignore me.
But, they they claim the issue is resolved from their perspective because I
can simply change my DNS settings.

But back to the topic: someone mentioned to me that Spectrum may not be the
direct providers for the DNS services they provide to their customers. If
anyone knows anything about how I might discover and reach out to the
people responsible, please let me know. :-)

Regards,

Kenneth

On Mon, Apr 22, 2024 at 8:07 PM Christopher Morrow 
wrote:

> “We checked the website you are trying to access for malicious and
> spear-phishing content and found it likely to be unsafe.”
>
> perhaps charter thinks there's a reason to not permit folks to access
> a possibly dangerous site?
> (it's also possible it just got cough up amongst some other stuff in
> the hosting provider's space, nothing jumps out in passive-dns
> lokoups.)
>
> On Mon, Apr 22, 2024 at 7:39 PM William Herrin  wrote:
> >
> > On Mon, Apr 22, 2024 at 4:00 PM John Levine  wrote:
> > > It appears that William Herrin  said:
> > > >If you can't reach a technical POC, use the legal one. Your lawyer can
> >
> > > The only response to a letter like that is "we run our network to
> > > serve our customers and manage it the way we think is best" and you
> > > know what, they're right.
> >
> > Hi John,
> >
> > Respectfully, you're mistaken. Look up "tortious interference."
> >
> > Operators have considerable legal leeway to block traffic for cause,
> > or even by mistake if corrected upon notification, but a lawyer who
> > blows off a cease-and-desist letter without investigating it with the
> > tech staff has committed malpractice. The lawyer doesn't want to
> > commit malpractice. You write the lawyer via certified mail, he's
> > going to talk to the tech staff and you're going to get a response. At
> > that point, you have an open communication pathway to get things
> > fixed. Which was the problem to be solved.
> >
> >
> > > Having said that, I suspect the least bad alternative if you can't
> > > find an out of band contact is to get some of the Spectrum customers
> > > who can't reach you to complain. They're customers, you aren't.
> >
> > My results going through the support front-door at large companies for
> > oddball problems have been less than stellar. Has your experience
> > truly been different?
> >
> > Regards,
> > Bill Herrin
> >
> >
> > --
> > William Herrin
> > b...@herrin.us
> > https://bill.herrin.us/
>

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

2024-04-22 Thread Mel Beckman

Bill is absolutely correct. The spammers lost their case because they were 
demonstrably spammers. We’ve had accidental black hole cases with *US* 
providers that removed the block once they received a C If they don’t have 
iron clad proof in hand. (More than just a few complaints and no traffic 
analysis), it’s just the least risky response.

That doesn’t work well with overseas providers, though, because they’re 
essentially immune to U.S. litigation unless the plaintiff has deep pockets.

 -mel

On Apr 22, 2024, at 5:21 PM, William Herrin  wrote:

On Mon, Apr 22, 2024 at 5:07 PM John R. Levine  wrote:
a complaint would have to show that the
blocking was malicious rather than merited or accidental.  In this case it
seems probably accidental, but for all I know there might have been bad
traffic to merit a block.

Hi John,

I'll try not to belabor it, but accidental that isn't corrected upon
formal legal notification becomes negligent and negligent has more or
less the same legal status as malicious.

The spammers lost because the networks published a terms of use
document that the spammers unambiguously violated. Even though it
interfered with the spammer's business, the block was merited so the
preponderance of the evidence fell in favor of the service provider.

Regards,
Bill Herrin


--
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

2024-04-22 Thread William Herrin

On Mon, Apr 22, 2024 at 5:07 PM John R. Levine  wrote:
> a complaint would have to show that the
> blocking was malicious rather than merited or accidental.  In this case it
> seems probably accidental, but for all I know there might have been bad
> traffic to merit a block.

Hi John,

I'll try not to belabor it, but accidental that isn't corrected upon
formal legal notification becomes negligent and negligent has more or
less the same legal status as malicious.

The spammers lost because the networks published a terms of use
document that the spammers unambiguously violated. Even though it
interfered with the spammer's business, the block was merited so the
preponderance of the evidence fell in favor of the service provider.

Regards,
Bill Herrin

-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

2024-04-22 Thread John R. Levine


On Mon, 22 Apr 2024, William Herrin wrote:

Respectfully, you're mistaken. Look up "tortious interference."


I'm familiar with it.

But I am also familar with many cases were spammers have sued network 
operators claiming that they're falsely defamed, so the operator has to 
deliver their mail.  They have without exception lost.  If you can find 
actual cases where a court forced an operator to deliver a third party's 
traffic I would like to hear about it.*


43 USC 230(c)(A) provides extremely broad protection for "good faith" 
blocking, which means that a complaint would have to show that the 
blocking was malicious rather than merited or accidental.  In this case it 
seems probably accidental, but for all I know there might have been bad 
traffic to merit a block.


Here's one of the cases where a spammer lost:

https://jl.ly/Email/holomaxx.html
https://jl.ly/Email/holo4.html

And here's one where the judge rejected tortious interference:

https://jl.ly/Email/spamarrest.html


My results going through the support front-door at large companies for
oddball problems have been less than stellar. Has your experience
truly been different?


No, it's terrible, and Spectrum is particularly bad.  I am now in month 
three of trying to get them to route a /24 to my host that belongs to one 
of my users, and their responses can be summarized as very complex 
exegeses of "duh?"


But bogus lawyer letters will just make things worse.

R's,
John

* - let's stay away for now from the Texas and Florida social network 
common carrier laws which are a whole other can of s*

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

2024-04-22 Thread Christopher Morrow

“We checked the website you are trying to access for malicious and
spear-phishing content and found it likely to be unsafe.”

perhaps charter thinks there's a reason to not permit folks to access
a possibly dangerous site?
(it's also possible it just got cough up amongst some other stuff in
the hosting provider's space, nothing jumps out in passive-dns
lokoups.)

On Mon, Apr 22, 2024 at 7:39 PM William Herrin  wrote:
>
> On Mon, Apr 22, 2024 at 4:00 PM John Levine  wrote:
> > It appears that William Herrin  said:
> > >If you can't reach a technical POC, use the legal one. Your lawyer can
>
> > The only response to a letter like that is "we run our network to
> > serve our customers and manage it the way we think is best" and you
> > know what, they're right.
>
> Hi John,
>
> Respectfully, you're mistaken. Look up "tortious interference."
>
> Operators have considerable legal leeway to block traffic for cause,
> or even by mistake if corrected upon notification, but a lawyer who
> blows off a cease-and-desist letter without investigating it with the
> tech staff has committed malpractice. The lawyer doesn't want to
> commit malpractice. You write the lawyer via certified mail, he's
> going to talk to the tech staff and you're going to get a response. At
> that point, you have an open communication pathway to get things
> fixed. Which was the problem to be solved.
>
>
> > Having said that, I suspect the least bad alternative if you can't
> > find an out of band contact is to get some of the Spectrum customers
> > who can't reach you to complain. They're customers, you aren't.
>
> My results going through the support front-door at large companies for
> oddball problems have been less than stellar. Has your experience
> truly been different?
>
> Regards,
> Bill Herrin
>
>
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

2024-04-22 Thread William Herrin

On Mon, Apr 22, 2024 at 4:00 PM John Levine  wrote:
> It appears that William Herrin  said:
> >If you can't reach a technical POC, use the legal one. Your lawyer can

> The only response to a letter like that is "we run our network to
> serve our customers and manage it the way we think is best" and you
> know what, they're right.

Hi John,

Respectfully, you're mistaken. Look up "tortious interference."

Operators have considerable legal leeway to block traffic for cause,
or even by mistake if corrected upon notification, but a lawyer who
blows off a cease-and-desist letter without investigating it with the
tech staff has committed malpractice. The lawyer doesn't want to
commit malpractice. You write the lawyer via certified mail, he's
going to talk to the tech staff and you're going to get a response. At
that point, you have an open communication pathway to get things
fixed. Which was the problem to be solved.

> Having said that, I suspect the least bad alternative if you can't
> find an out of band contact is to get some of the Spectrum customers
> who can't reach you to complain. They're customers, you aren't.

My results going through the support front-door at large companies for
oddball problems have been less than stellar. Has your experience
truly been different?

Regards,
Bill Herrin

-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

2024-04-22 Thread John Levine

It appears that William Herrin  said:
>On Sun, Apr 21, 2024 at 6:21 PM Validin Axon  wrote:
>> Looking for some help/advice. Spectrum is sinkholing my company's domain, 
>> validin[.]com, to 127.0.0.54.
>
>Howdy,
>
>If you can't reach a technical POC, use the legal one. Your lawyer can
>find the appropriate recipient and write a cease-and-desist letter for
>you. After that, it's -their- lawyer's problem to track down the
>correct technical people.

No, that is terrible advice.  In the immortal acronym of Laura Atkins, TWSD.

The only response to a letter like that is "we run our network to
serve our customers and manage it the way we think is best" and you
know what, they're right. It is absolutely legal to block traffic you
think is malicious, even if you are wrong, and there is case law.

Having said that, I suspect the least bad alternative if you can't
find an out of band contact is to get some of the Spectrum customers
who can't reach you to complain. They're customers, you aren't.

R's,
John

Re: Question about mutual transit and complex BGP peering

2024-04-22 Thread Matthew Petach

On Mon, Apr 22, 2024 at 7:35 AM Sriram, Kotikalapudi (Fed) via NANOG <
nanog@nanog.org> wrote:

> Requesting responses to the following questions. Would be helpful in some
> IETF work in progress.
>
> Q1: Consider an AS peering relationship that is complex (or hybrid)
> meaning, for example, provider-to-customer (P2C) for one set of prefixes
> and lateral peers (i.e., transit-free peer-to-peer (P2P)) for another set
> of prefixes.  Are these diverse relationships usually segregated, i.e., P2C
> on one BGP session and P2P on another?  How often they might co-exist
> within one single BGP session?
>
>
Every time I've been in relationships like this, the fundamental answer is
always "follow the money".

If there's dollars flowing relative to the "provider-to-customer"
relationship, but no dollars flowing along the "peer-to-peer" relationship,
you need a solid way to determine which bits are taking the zero-dollar
pathway, and which bits are taking the non-zero-dollar pathway.

Whatever means are available to positively distinguish the traffic on an
unambiguous basis that both networks agree on is what determines the setup.

In many cases, separate physical ports with separate BGP sessions (and
sometimes even separate VRFs) is the only way that both parties fully trust
all the right bits
are being accounted for in each case.

In other relationships, flow data is considered adequate to determine how
much traffic is zero dollar, and how much traffic is non-zero dollar.  In
that case, it can be a single BGP session, single port.

> Q2: Consider an AS peering relationship that is mutual transit (i.e., P2C
> relationship in each direction for all prefixes).  Is this supported within
> one single BGP session?  How often the ASes might setup two separate BGP
> sessions between them -- one for P2C in one direction (AS A to AS B) and
> the other for P2C in the opposite direction (AS B to AS A)?
>

This is just a variant of a normal peer-to-peer relationship, most likely
with a traffic ratio involved.
In most of these situations, as long as the traffic is within the defined
ratio, accounting for the
bits isn't worth it; sending a bill from A to B for $X, and a different
bill from B to A for $+$Y where $Y is
generally much smaller than $X is more headache than it's worth.
And once the ratio goes outside of the prescribed range, you're not really
mutual transit anymore, you're provider to customer,
and the only wrinkle is which one considers themselves the provider, and
which considers themselves the customer.
Witness Level 3 versus Comcast versus Netflix from years ago:
https://arstechnica.com/tech-policy/2010/12/comcastlevel3/
https://publicknowledge.org/netflix-cdn-v-the-cable-guys-or-comcast-v-level-3-part-deux-peering-payback/

Again--when everything is within ratio, and pipes aren't full, no need for
separate ports or separate BGP
sessions.

Once things start to fill up, though, then things get ugly.  That's when
different sessions come into play,
with some traffic being shunted to congested sessions, while the two sides
battle it out.

It still comes down to the same fundamental rule, though--follow the
money.   ^_^;

Thanks!

Matt

> Thank you.
>
> Sriram
> Kotikalapudi Sriram, US NIST
>

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

2024-04-22 Thread Validin Axon

Hi Mel,

I appreciate the suggestion. During my earlier research, I'd noticed that
as well. However, the DNS block includes all validin.com subdomains, covering
those on completely different ASNs. It also does NOT affect other domains
that resolve to the exact same IP addresses (e.g., validin.net). So, I'm
inclined to think it's not that simple, unfortunately.

I'd considered switching domains, but that doesn't guarantee the problem
wouldn't just reappear again, and it'd impact the search engine ranking
we've built up. We rely 100% on inbound, so that'd be a big set back.

Warm regards,

Kenneth

On Mon, Apr 22, 2024 at 10:29 AM Mel Beckman  wrote:

>
> UCEPROTECTL3 137.184.54.107 was listed
>
> I notice from MXToolbox.com that your domain’s IP address is on the
> UCEPROTECTL3 blacklist.
>
> This is a notoriously evil blacklist that charges people for removal. This
> may be why Spectrum is blackholing your domain. Most respectable ISPs won’t
> use it. But Spectrum…
>
> There is no delisting procedure without making a “donation” to the UCEPROTECT3
> black sparrow account. They’re famous for blacklisting large swaths of IP
> addresses that catch up innocent parties that have never spammed a flea.
>
> -mel
>
>
> On Apr 22, 2024, at 7:24 AM, Mel Beckman  wrote:
>
>  I notice you’re on the UCEPROTECT3 blacklist:
>
> 
> Network Tools: DNS,IP,Email
> 
> mxtoolbox.com
> 
> 
>
> UCEPROTECTL3 137.184.54.107 was listed
>  This is a notoriously evil blacklist that charges people for removal.
> This may be why Spectrum is blackholing your domain. Most respectable ISPs
> won’t use it. But Spectrum…
>
> There is no delisting procedure without making a “donation” to the UCEPROTECT3
> black sparrow account. They’re famous for blacklisting large swaths of IP
> addresses that catch up innocent parties that have never spammed a flea.
>
> -mel
>
> On Apr 22, 2024, at 4:51 AM, Validin Axon  wrote:
>
> 
> Looking for some help/advice. Spectrum is sinkholing my company's domain,
> validin[.]com, to 127.0.0.54. The sinkhole responses come from their
> recursive DNS servers, 209.18.47.61 and 209.18.47.62, which are defaults
> for and in use by many of their customers and are only reachable from
> within the Spectrum network. I've had 4 people over the last week (think:
> customers, prospects, etc) who use Charter/Spectrum tell me that they have
> difficulty accessing my website as a result of this sinkhole behavior. This
> behavior is causing reputational harm to my company.
>
> I've personally confirmed this behavior from the Spectrum network (I am
> also a customer) using dig to test their DNS servers:
> ```
> $ dig +short @209.18.47.61 validin.com
> 127.0.0.54
> $ dig +short @209.18.47.62 validin.com
> 127.0.0.54
> ```
>  Using Cloudflare/Google/etc works correctly:
> ```
> $ dig +short @1.1.1.1 validin.com
> 137.184.54.107
> 157.245.112.183
> $ dig +short @8.8.8.8 validin.com
> 157.245.112.183
> 137.184.54.107
> ```
>
> I suspect my domain was blocklisted last year when a threat researcher
> included my domain name in a blog post about a threat they were
> investigating and cited my company as the source for their data. Someone
> scraped that post, and my company's domain was accidentally added to
> two Alient Vault OTX pulses and at least one collection on Virus Total. I
> removed the domain via false positive reporting from everything I could.
> However, it appears that being added to Spectrum's DNS sinkhole list is
> effectively permanent and there's no clear path for false positive
> remediation.
>
> I've tried the official Spectrum support lines for months to no avail, and
> recently tried reaching out on Twitter, but have had no success there
> either. I'm clearly not able to find the right people through these routes,
> as none of the people I reach understand the difference between a DNS
> sinkhole and an IP block list and don't appear to be aware that DNS
> blocklisting is a separate behavior from their opt-in content filtering via
> Security Shield.
>
> So, if someone could please help me find the team or individual
> responsible for Spectrum's DNS sinkhole behavior, I would be exceptionally
> grateful. :-)
>
> As I mentioned, this is causing reputation harm, so switching my own DNS
> servers is not sufficient. People who need to reach me, can't. So, I would
> appreciate any other help or advice you have,
>
> Kenneth
>
>

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

2024-04-22 Thread Mel Beckman

I notice from MXToolbox.com that your domain’s IP address is on the 
UCEPROTECTL3 blacklist.

This is a notoriously evil blacklist that charges people for removal. This may 
be why Spectrum is blackholing your domain. Most respectable ISPs won’t use it. 
But Spectrum…

There is no delisting procedure without making a “donation” to the UCEPROTECT3 
black sparrow account. They’re famous for blacklisting large swaths of IP 
addresses that catch up innocent parties that have never spammed a flea.

-mel

On Apr 22, 2024, at 4:51 AM, Validin Axon  wrote:


Looking for some help/advice. Spectrum is sinkholing my company's domain, 
validin[.]com, to 127.0.0.54. The sinkhole responses come from their recursive 
DNS servers, 209.18.47.61 and 209.18.47.62, which are defaults for and in use 
by many of their customers and are only reachable from within the Spectrum 
network. I've had 4 people over the last week (think: customers, prospects, 
etc) who use Charter/Spectrum tell me that they have difficulty accessing my 
website as a result of this sinkhole behavior. This behavior is causing 
reputational harm to my company.

I've personally confirmed this behavior from the Spectrum network (I am also a 
customer) using dig to test their DNS servers:
```
$ dig +short @209.18.47.61 validin.com
127.0.0.54
$ dig +short @209.18.47.62 validin.com
127.0.0.54
```
 Using Cloudflare/Google/etc works correctly:
```
$ dig +short @1.1.1.1 validin.com
137.184.54.107
157.245.112.183
$ dig +short @8.8.8.8 validin.com
157.245.112.183
137.184.54.107
```

I suspect my domain was blocklisted last year when a threat researcher included 
my domain name in a blog post about a threat they were investigating and cited 
my company as the source for their data. Someone scraped that post, and my 
company's domain was accidentally added to two Alient Vault OTX pulses and at 
least one collection on Virus Total. I removed the domain via false positive 
reporting from everything I could. However, it appears that being added to 
Spectrum's DNS sinkhole list is effectively permanent and there's no clear path 
for false positive remediation.

I've tried the official Spectrum support lines for months to no avail, and 
recently tried reaching out on Twitter, but have had no success there either. 
I'm clearly not able to find the right people through these routes, as none of 
the people I reach understand the difference between a DNS sinkhole and an IP 
block list and don't appear to be aware that DNS blocklisting is a separate 
behavior from their opt-in content filtering via Security Shield.

So, if someone could please help me find the team or individual responsible for 
Spectrum's DNS sinkhole behavior, I would be exceptionally grateful. :-)

As I mentioned, this is causing reputation harm, so switching my own DNS 
servers is not sufficient. People who need to reach me, can't. So, I would 
appreciate any other help or advice you have,

Kenneth

Re: Help with removing DNS shinkhole FP from Charter/Spectrum

2024-04-22 Thread William Herrin

On Sun, Apr 21, 2024 at 6:21 PM Validin Axon  wrote:
> Looking for some help/advice. Spectrum is sinkholing my company's domain, 
> validin[.]com, to 127.0.0.54.

Howdy,

If you can't reach a technical POC, use the legal one. Your lawyer can
find the appropriate recipient and write a cease-and-desist letter for
you. After that, it's -their- lawyer's problem to track down the
correct technical people.

Incidentally, for folks who choose to interdict DNS: whatever your
reasons, pointing the DNS to a loopback IP is bad practice. Really bad
practice. Minimum good practice points it to a web site you control
which provides enough information to get delisted. And provides you
with a test point where you can collect information about what you've
caused to be interdicted.

Regards,
Bill Herrin

-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: constant FEC errors juniper mpc10e 400g

2024-04-22 Thread Mark Tinka





On 4/22/24 09:47, Vasilenko Eduard via NANOG wrote:


Assume that some carrier has 10k FBB subscribers in a particular municipality 
(without any hope of considerably increasing this number).
2Mbps is the current average per household in the busy hour, pretty uniform 
worldwide.
You could multiply it by 8/7 if you like to add wireless -> not much would 
change.
2*2*10GE (2*10GE on the ring in every direction) is 2 times than needed to 
carry 10k subscribers.
The optical ring may be less than 20 municipalities - it is very common.
Hence, the upgrade of old extremely cheap 10GE DWDM systems (for 40 lambdas) 
makes sense for some carriers.
It depends on the population density and the carrier market share.
10GE for the WAN side would not be dead in the next 5 years because 2Mbps per 
household would not grow very fast in the future - this logistic curve is close 
to a plateau.
PS: It is probably not the case for Africa where new subscribers are connected 
to the Internet at a fast rate.


As a function of how much Internet there is in Africa, there really 
aren't that many optical transport service providers. Some 
countries/cities/towns have more than they need, others have just one. 
But in general, you would say there is massive room for improvement if 
you surveyed the entire continent.


Typically, it will be the incumbents, alongside 2 or 3 competitives. In 
fact, in some African countries, only the incumbent may be large enough 
to run an optical backbone, with all the competitives leasing capacity 
from them.


It is not uncommon to find the closest competitor to an incumbent for 
terrestrial services being the mobile network operator, purely because 
they have some excess capacity left over from having to build the 
backbone for their core business, mobile. And, they are flush with cash, 
so a loss-making terrestrial backhaul business can be covered by the 
month's sales in SIM cards.


Truly independent transport providers are few and far between because 
access to dark fibre is not easy (either its lack of availability, the 
incumbent refusing to sell it, or its high price). For the few 
independent transport providers that do spring up, they will focus on a 
limited set of hot routes, and because competition on those routes may 
be wanting, prices and capacity would not be terribly attractive.


So the bulk of Africa's Internet really relies on a handful of key 
African wholesale IP Transit providers taking great effort into 
extending their network as deep into cities as they can, and using their 
size to negotiate the best prices for terrestrial backhaul from the few 
optical network operators that the market has. Those providers then sell 
to the local and regional ISP's, who don't have to worry about running a 
backbone.


All this means is that for those operators that run an optical backbone, 
especially nationally, 10G carriers are very, very legacy. If they still 
have them, it'd be a spin-off off the main core to support some old SDH 
customers that are too comfortable to move to Ethernet.


Metro backhaul and last mile FNO's (fibre network operators) who have 
invested in extending access into homes and businesses are a different 
story, with most countries having a reasonable handful of options 
customers can choose from. Like national backhaul, there is plenty of 
room for improvement - in some markets more than others - but unlike 
national backhaul, not as constrained for choice or price.


Mark.

RE: constant FEC errors juniper mpc10e 400g

2024-04-22 Thread Vasilenko Eduard via NANOG

Assume that some carrier has 10k FBB subscribers in a particular municipality 
(without any hope of considerably increasing this number).
2Mbps is the current average per household in the busy hour, pretty uniform 
worldwide.
You could multiply it by 8/7 if you like to add wireless -> not much would 
change.
2*2*10GE (2*10GE on the ring in every direction) is 2 times than needed to 
carry 10k subscribers.
The optical ring may be less than 20 municipalities - it is very common.
Hence, the upgrade of old extremely cheap 10GE DWDM systems (for 40 lambdas) 
makes sense for some carriers.
It depends on the population density and the carrier market share.
10GE for the WAN side would not be dead in the next 5 years because 2Mbps per 
household would not grow very fast in the future - this logistic curve is close 
to a plateau.
PS: It is probably not the case for Africa where new subscribers are connected 
to the Internet at a fast rate.
Ed/
-Original Message-
From: NANOG  On Behalf Of 
Tarko Tikan
Sent: Saturday, April 20, 2024 19:19
To: nanog@nanog.org
Subject: Re: constant FEC errors juniper mpc10e 400g

hey,

> That said, I don't expect any subsea cables getting built in the next 
> 3 years and later will have 10G as a product on the SLTE itself... it 
> wouldn't be worth the spectrum.

10G wavelengths for new builds died about 10 years ago when coherent 100G 
became available, submarine or not. Putting 10G into same system is not really 
feasible at all.

--
tarko

Re: constant FEC errors juniper mpc10e 400g

2024-04-21 Thread Mark Tinka




On 4/21/24 08:12, Saku Ytti wrote:


Key difference being, WAN-PHY does not provide synchronous timing, so
it's not SDH/SONET compatible for strict definition for it, but it
does have the frame format. And the optical systems which could
regenerate SONET/SDH framing, didn't care about timing, they just
wanted to be able to parse and generate those frames, which they
could, but they could not do it for ethernet frames.


Correct.

In those days, WAN-PHY was considered "SONET/SDH-Lite".

I think it is pretty clear, the driver was to support long haul
regeneration, so it was always going to be a stop-gap solution. Even
though I know some networks, who specifically wanted WAN-PHY for its
error reporting capabilities, I don't think this was majority driver,
majority driver almost certainly was 'thats only thing we can put on
this circuit'.


SONET/SDH did have similar reach as OTN back then, just less bandwidth 
for the distance. It had FEC support for STM-16, STM-64 and STM-256.


I really think the bigger driver was interface cost, because EoS had 
already been selling for 1GE alongside STM-16 for 2.5G. In those days, 
if you needed more than 1G but less than 10G, it was a toss-up between 
2x 1G EoSDH vs. 1x STM-16. Often times, you took the 2x 1G EoSDH because 
2x 1GE ports were cheaper than 1x STM-16 port, even though you ended up 
losing about 405Mbps of capacity in the process, which was a huge deal.


The backbone providers did not like selling EoSDH services, because it 
was too much admin. for them (VC container management), and they ended 
up paying more for transponders on their side than their customers did 
for Ethernet ports on theirs :-).


But by and large, the majority of networks in our market maintained SDH 
services long after coherent became commercially available. It was a 
perception thing, that SDH was more superior to Ethernet, even if that 
Ethernet was transported over a DWDM network.


In the end, SDH port costs were too hard to ignore due to router vendors 
maintaining their mark-up on them despite dying demand, which then led 
to the migration from SDH to EoDWDM growing significantly from about 
2016. Optical vendors also began de-prioritizing SDH transponder ports, 
which had a massive impact on the SLTE (submarine) side in making the 
decision to encourage customers away from SDH for wet services.


Mark.

Re: constant FEC errors juniper mpc10e 400g

2024-04-21 Thread Saku Ytti

On Sun, 21 Apr 2024 at 09:05, Mark Tinka  wrote:

> Technically, what you are describing is EoS (Ethernet over SONET, Ethernet 
> over SDH), which is not the same as WAN-PHY (although the working groups that 
> developed these nearly confused each other in the process, ANSI/ITU for the 
> former vs. IEEE for the latter).
>
> WAN-PHY was developed to be operated across multiple vendors over different 
> media... SONET/SDH, DWDM, IP/MPLS/Ethernet devices and even dark fibre. The 
> goal of WAN-PHY was to deliver a low-cost Ethernet interface that was 
> SONET/SDH-compatible, as EoS interfaces were too costly for operators and 
> their customers.
>
> As we saw in real life, 10GE ports out-sold STM-64/OC-192 ports, as networks 
> replaced SONET/SDH backbones with DWDM and OTN.

Key difference being, WAN-PHY does not provide synchronous timing, so
it's not SDH/SONET compatible for strict definition for it, but it
does have the frame format. And the optical systems which could
regenerate SONET/SDH framing, didn't care about timing, they just
wanted to be able to parse and generate those frames, which they
could, but they could not do it for ethernet frames.

I think it is pretty clear, the driver was to support long haul
regeneration, so it was always going to be a stop-gap solution. Even
though I know some networks, who specifically wanted WAN-PHY for its
error reporting capabilities, I don't think this was majority driver,
majority driver almost certainly was 'thats only thing we can put on
this circuit'.

-- 
  ++ytti

Re: constant FEC errors juniper mpc10e 400g

2024-04-21 Thread Mark Tinka




On 4/20/24 21:36, b...@uu3.net wrote:


Erm, WAN-PHY did not extend into 40G because there was not much
of those STM-256 deployment? (or customers didnt wanted to pay for those).


With SONET/SDH, as the traffic rate increased, so did the number of 
overhead bytes. With every iteration of the data rate, the overhead 
bytes quadrupled. This was one of the key reasons we did not see field 
deployment of STM-256/OC-768 and STM-1024/OC-3072. For example, if 
SONET/SDH had to transport a 100G service, it would require 160Gbps of 
bandwidth. That clearly wasn't going to work.


At the time when STM-256/OC-768 was being developed, DWDM and OTN had 
come a long way. The granularity SONET/SDH required to stand up a 
service had become too small for the growing data rate (primarily VC-3, 
VC-4 and VC-12). If you look at OTN, the smallest container is 1.25Gbps 
(ODU0), which was attractive for networks looking to migrate from E1's, 
E3's, STM-1's, STM-4's and STM-16's - carried over VC-12, VC-4 and VC-3 
SDH circuits - to 1GE, for example, rather than trying to keep their 
PDH/SDH infrastructure going.


DWDM and OTN permitted a very small control overhead, so as data rates 
increased, there wasn't the same penalty you got with SONET/SDH.

WAN-PHY was designed so people could encapsulate Ethernet frames
right into STM-64. Once world moved out of SDH/SONET stuff, there was
no more need for WAN-PHY anymore.


Technically, what you are describing is EoS (Ethernet over SONET, 
Ethernet over SDH), which is not the same as WAN-PHY (although the 
working groups that developed these nearly confused each other in the 
process, ANSI/ITU for the former vs. IEEE for the latter).


WAN-PHY was developed to be operated across multiple vendors over 
different media... SONET/SDH, DWDM, IP/MPLS/Ethernet devices and even 
dark fibre. The goal of WAN-PHY was to deliver a low-cost Ethernet 
interface that was SONET/SDH-compatible, as EoS interfaces were too 
costly for operators and their customers.


As we saw in real life, 10GE ports out-sold STM-64/OC-192 ports, as 
networks replaced SONET/SDH backbones with DWDM and OTN.


Mark.

Re: constant FEC errors juniper mpc10e 400g

2024-04-20 Thread borg

Erm, WAN-PHY did not extend into 40G because there was not much
of those STM-256 deployment? (or customers didnt wanted to pay for those).

WAN-PHY was designed so people could encapsulate Ethernet frames
right into STM-64. Once world moved out of SDH/SONET stuff, there was
no more need for WAN-PHY anymore.


-- Original message --

From: Mark Tinka 
To: Dave Cohen 
Cc: nanog@nanog.org
Subject: Re: constant FEC errors juniper mpc10e 400g
Date: Sat, 20 Apr 2024 17:50:04 +0200


WAN-PHY did not extend to 40G or 100G, which can explain one of the reasons it
lost favour. For 10G, its availability also depended on the type of device, its
NOS, line card and/or pluggable at the time, which made it hard to find a
standard around this if you built multi-vendor networks or purchased backhaul
services from 3rd party providers that had non-standard support for
WAN-PHY/OTN/G.709. In other words, LAN-PHY (and plain Ethernet) became the
lowest common denominator in the majority of cases for customers.

Mark.

Re: constant FEC errors juniper mpc10e 400g

2024-04-20 Thread Mark Tinka





On 4/20/24 18:19, Tarko Tikan wrote:

10G wavelengths for new builds died about 10 years ago when coherent 
100G became available, submarine or not. Putting 10G into same system 
is not really feasible at all.


I was referring to 10G services (client-side), not 10G wavelengths (line 
side).


Mark.

Re: constant FEC errors juniper mpc10e 400g

2024-04-20 Thread Tarko Tikan


hey,

That said, I don't expect any subsea cables getting built in the next 3 
years and later will have 10G as a product on the SLTE itself... it 
wouldn't be worth the spectrum.


10G wavelengths for new builds died about 10 years ago when coherent 
100G became available, submarine or not. Putting 10G into same system is 
not really feasible at all.


--
tarko

Re: constant FEC errors juniper mpc10e 400g

2024-04-20 Thread Mark Tinka





On 4/20/24 14:41, Dave Cohen wrote:


LAN PHY dominates in the US too. Requests for WAN PHY were almost exclusively 
for terrestrial backhaul extending off of legacy subsea systems that still 
commonly had TDM-framed services. It’s been a couple of years since I’ve been 
in optical transport directly but these requests were essentially non-existent 
after 2018 or so. OTN became somewhat more common from 2014 onward as optical 
system interop improved, but actually was more common in the enterprise space 
as providers would generally go straight to fiber in most use cases, and with 
dark fiber opex costs coming down in many markets, I see OTN requests as 
winnowing here as well.


What really changed the game was coherent detection, which breathed new 
life into legacy subsea cables that were built on dispersion-managed 
fibre. Post-2014 when uncompensated (and highly dispersed) fibre has 
been the standard for subsea builds (even for SDM cables), coherent 
optical systems are the mainstay. In fact, because linear dispersion can 
be accurately calculated for the cable span, uncompensated cables are a 
good thing because the dispersion compensation happens in very advanced 
coherent DSP's in the optical engine, rather than in the fibre itself.


WAN-PHY did not extend to 40G or 100G, which can explain one of the 
reasons it lost favour. For 10G, its availability also depended on the 
type of device, its NOS, line card and/or pluggable at the time, which 
made it hard to find a standard around this if you built multi-vendor 
networks or purchased backhaul services from 3rd party providers that 
had non-standard support for WAN-PHY/OTN/G.709. In other words, LAN-PHY 
(and plain Ethernet) became the lowest common denominator in the 
majority of cases for customers.


In 2024, I find that operators care more about bringing the circuit up 
than using its link properties to trigger monitoring, failover and 
reconvergence. The simplest way to do that is to ask for plain Ethernet 
services, particularly for 100G and 400G, but also for 10G. In practice, 
this has been reasonably reliable in the past 2 - 3 years when procuring 
100G backhaul services. So for the most part, users of these services 
seem to be otherwise happy.


Mark.

Re: constant FEC errors juniper mpc10e 400g

2024-04-20 Thread Dave Cohen

LAN PHY dominates in the US too. Requests for WAN PHY were almost exclusively 
for terrestrial backhaul extending off of legacy subsea systems that still 
commonly had TDM-framed services. It’s been a couple of years since I’ve been 
in optical transport directly but these requests were essentially non-existent 
after 2018 or so. OTN became somewhat more common from 2014 onward as optical 
system interop improved, but actually was more common in the enterprise space 
as providers would generally go straight to fiber in most use cases, and with 
dark fiber opex costs coming down in many markets, I see OTN requests as 
winnowing here as well. 

Dave Cohen
craetd...@gmail.com

> On Apr 20, 2024, at 7:57 AM, Mark Tinka  wrote:
> 
> 
> 
>> On 4/20/24 13:39, Saku Ytti wrote:
>> 
>> 
>>  Oh I don't think OTN or WAN-PHY have any large deployment future, the
>> cheapest option is 'good enough'...
> 
> And what we find with EU providers is that Ethernet and OTN services are 
> priced similarly. It's a software toggle on a transponder, but even then, 
> Ethernet still continues to be preferred over OTN.
> 
> Mark.

Re: constant FEC errors juniper mpc10e 400g

2024-04-20 Thread Mark Tinka




On 4/20/24 13:39, Saku Ytti wrote:


Oh I don't think OTN or WAN-PHY have any large deployment future, the
cheapest option is 'good enough'...


And what we find with EU providers is that Ethernet and OTN services are 
priced similarly. It's a software toggle on a transponder, but even 
then, Ethernet still continues to be preferred over OTN.


Mark.

Re: constant FEC errors juniper mpc10e 400g

2024-04-20 Thread Mark Tinka




On 4/20/24 13:39, Saku Ytti wrote:


Oh I don't think OTN or WAN-PHY have any large deployment future, the
cheapest option is 'good enough' and whatever value you could extract
from OTN or WAN-PHY, will be difficult to capitalise, people usually
don't even capitalise the capabilities they already pay for in the
cheaper technologies.


A handful of OEM's still push OTN like it has just been invented, 
especially those still pushing "IPoDWDM" :-). Fair point, if you have a 
highly-meshed metro network with lots of drops to customers across a 
ring-mesh topology, there might be some value in OTN when delivering 
such services at low speeds (10G, 25G, 2.5G, 1G). But while the topology 
is valid, most networks aren't using high-end optical gear to drop 
low-speed services, nowadays. Even though on a per-bit basis, they might 
be cheaper than 1U IP/MPLS router looking to do the same job if all you 
are considering is traffic, and not additional services that want to eat 
packets.



Of course WAN-PHY is dead post 10GE, a big reason for it to exist was
very old optical systems which simply could not regenerate ethernet
framing, not any features or functional benefits.


In our market, we are trending toward a convergence between 10G and 100G 
orders intersecting for long haul and submarine asks. But pockets of 10G 
demand still exist in many African countries, and none of them have any 
WAN-PHY interest of any statistical significance.


That said, I don't expect any subsea cables getting built in the next 3 
years and later will have 10G as a product on the SLTE itself... it 
wouldn't be worth the spectrum.


Mark.

Re: constant FEC errors juniper mpc10e 400g

2024-04-20 Thread Saku Ytti

On Sat, 20 Apr 2024 at 14:35, Mark Tinka  wrote:

> Even when our market seeks OTN from European backhaul providers to extend 
> submarine access into Europe and Asia-Pac, it is often for structured 
> capacity grooming, and not for OAM benefit.
>
> It would be interesting to learn whether other markets in the world still 
> make a preference for OTN in lieu of Ethernet, for the OAM benefit, en masse. 
> When I worked in Malaysia back in the day (2007 - 2012), WAN-PHY was 
> generally asked for for 10G services, until about 2010; when folk started to 
> choose LAN-PHY. The reason, back then, was to get that extra 1% of pipe 
> bandwidth :-).

Oh I don't think OTN or WAN-PHY have any large deployment future, the
cheapest option is 'good enough' and whatever value you could extract
from OTN or WAN-PHY, will be difficult to capitalise, people usually
don't even capitalise the capabilities they already pay for in the
cheaper technologies.
Of course WAN-PHY is dead post 10GE, a big reason for it to exist was
very old optical systems which simply could not regenerate ethernet
framing, not any features or functional benefits.

-- 
  ++ytti

Re: constant FEC errors juniper mpc10e 400g

2024-04-20 Thread Mark Tinka




On 4/20/24 13:25, Saku Ytti wrote:


In most cases, modern optical long haul has a transponder, which
terminates your FEC, because clients offer gray, and you like
something a bit less depressing, like 1570.42nm.

This is not just FEC terminating, but also to a degree autonego
terminating, like RFI signal would be between you and transponder, so
these connections can be, and regularly are, provided without proper
end-to-end hardware liveliness, and even if they were delivered and
tested to have proper end-to-end HW liveliness, that may change during
operation, so line faults may or may not be propagated to both ends as
RFI assertion, and even if they are, how delayed they are, they may
suffer delay to allow for optical protection to engage, which may be
undesirable, as it eats into your convergence budget.

Of course the higher we go in the abstraction, the less likely you are
to get things like HW livelines detection, like I don't really see
anyone asking for this in their pseudowire services, even though it's
something that actually can be delivered. In Junos it's a single
config stanza in interface, to assert RFI to client port, if
pseudowire goes down in the operator network.


In our market (Africa), for both terrestrial and submarine services, 
OTN-type circuits are not typically ordered. Network operators are not 
really interested in receiving the additional link data that OTN or 
WAN-PHY provides. They truly want to leave the operation of the 
underlying transport backbone to the transport operator.


The few times we have come across the market asking for OTN is if they 
want to groom 10x 10G into 1x 100G, for example, to deliver structured 
services downstream.


Even when our market seeks OTN from European backhaul providers to 
extend submarine access into Europe and Asia-Pac, it is often for 
structured capacity grooming, and not for OAM benefit.


It would be interesting to learn whether other markets in the world 
still make a preference for OTN in lieu of Ethernet, for the OAM 
benefit, en masse. When I worked in Malaysia back in the day (2007 - 
2012), WAN-PHY was generally asked for for 10G services, until about 
2010; when folk started to choose LAN-PHY. The reason, back then, was to 
get that extra 1% of pipe bandwidth :-).


Mark.

Re: constant FEC errors juniper mpc10e 400g

2024-04-20 Thread Saku Ytti

On Sat, 20 Apr 2024 at 10:00, Mark Tinka  wrote:

> This would only matter on ultra long haul optical spans where the signal 
> would need to be regenerated, where - among many other values - FEC would 
> need to be decoded, corrected and re-applied.

In most cases, modern optical long haul has a transponder, which
terminates your FEC, because clients offer gray, and you like
something a bit less depressing, like 1570.42nm.

This is not just FEC terminating, but also to a degree autonego
terminating, like RFI signal would be between you and transponder, so
these connections can be, and regularly are, provided without proper
end-to-end hardware liveliness, and even if they were delivered and
tested to have proper end-to-end HW liveliness, that may change during
operation, so line faults may or may not be propagated to both ends as
RFI assertion, and even if they are, how delayed they are, they may
suffer delay to allow for optical protection to engage, which may be
undesirable, as it eats into your convergence budget.

Of course the higher we go in the abstraction, the less likely you are
to get things like HW livelines detection, like I don't really see
anyone asking for this in their pseudowire services, even though it's
something that actually can be delivered. In Junos it's a single
config stanza in interface, to assert RFI to client port, if
pseudowire goes down in the operator network.

-- 
  ++ytti

Re: constant FEC errors juniper mpc10e 400g

2024-04-20 Thread Mark Tinka




On 4/19/24 10:08, Saku Ytti wrote:


Of course there are limits to this, as FEC is hop-by-hop, so in
long-haul you'll know about circuit quality to the transponder, not
end-to-end. Unlike in wan-phy, OTN where you know both.

Technically optical transport could induce FEC errors, if there are
FEC errors on any hop, so consumers of optical networks need not have
access to optical networks to know if it's end-to-end clean.


This would only matter on ultra long haul optical spans where the signal 
would need to be regenerated, where - among many other values - FEC 
would need to be decoded, corrected and re-applied.


SD-FEC already allows for a significant improvement in optical reach for 
a given modulation. This negates the need for early regeneration, 
assuming other optical penalties and impairments are satisfactorily 
compensated for.


Of course, what a market defines as long haul or ultra long haul may 
vary; add to that the variability of regeneration spacing in such 
scenarios being quite wide, on the order of 600km - 1,000km. Much of 
this will come down to fibre, ROADM and coherent pluggable quality.


Mark.

Re: constant FEC errors juniper mpc10e 400g

2024-04-20 Thread Mark Tinka




On 4/19/24 10:08, Saku Ytti wrote:


Of course there are limits to this, as FEC is hop-by-hop, so in
long-haul you'll know about circuit quality to the transponder, not
end-to-end. Unlike in wan-phy, OTN where you know both.

Technically optical transport could induce FEC errors, if there are
FEC errors on any hop, so consumers of optical networks need not have
access to optical networks to know if it's end-to-end clean.


This would only matter on ultra long haul optical spans where the signal 
would need to be regenerated, where - among many other values - FEC 
would need to be decoded, corrected and re-applied.


SD-FEC already allows for a significant improvement in optical reach for 
a given modulation. This negates the need for early regeneration, 
assuming other optical penalties and impairments are satisfactorily 
compensated for.


Of course, what a market defines as long haul or ultra long haul may 
vary; add to that the variability of regeneration spacing in such 
scenarios being quite wide, on the order of 600km - 1,000km. Much of 
this will come down to fibre, ROADM and coherent pluggable quality.

Re: Whitebox Routers Beyond the Datasheet

2024-04-19 Thread heasley

Fri, Apr 12, 2024 at 08:03:49AM -0500, Mike Hammett:
> I'm looking at the suitability of whitebox routers for high through, low port 
> count, fast BGP performance applications. Power efficiency is important as 
> well. 
> 
> 
> What I've kind of come down to (based on little more than spec sheets) are 
> the EdgeCore AGR400 and the UfiSpace S9600-30DX. They can both accommodate at 
> least three directions of 400G for linking to other parts of my network and 
> then have enough 100G or slower ports to connect to transit, peers, and 
> customers as appropriate. Any other suggestions for platforms similar to 
> those would be appreciated. 

Most of the white boxes are same, in mpov, with small variations.  And
that is the whole idea.

I would choose the NOS you want first.  There are several, but few I would
want in production.  If it is a PoS or unmanageable, it does not matter
what the h/w capabilities are.  Was it created by seasoned engineers in
Internet-scale routing?  And, because each box will require some software
specific to it, though limited in scope, the NOS will dictate which boxes
are available to choose among.

Beyond the hardware capabilities, also consider with whom your NOS mfg
has the best working relationship.  That will dictate their ability to
quickly resolve h/w-specific issues in their s/w or even answer
h/w-specific questions for you.

Also consider what the h/w maintenance program is globally.  Is it
important for you to have 4hr replacements in Hong Kong?  That will
affect your decision greatly.

~1.5yr ago, it seemed like everyone was moving toward UfiSpace h/w,
away from EdgeCore.

Ask others about the reliability of the specific h/w you are considering.

Re: constant FEC errors juniper mpc10e 400g

2024-04-19 Thread Saku Ytti

On Fri, 19 Apr 2024 at 10:55, Mark Tinka  wrote:>
FEC is amazing.

> At higher data rates (100G and 400G) for long and ultra long haul optical 
> networks, SD-FEC (Soft Decision FEC) carries a higher overhead penalty 
> compared to HD-FEC (Hard Decision FEC), but the net OSNR gain more than 
> compensates for that, and makes it worth it to increase transmission distance 
> without compromising throughput.

Of course there are limits to this, as FEC is hop-by-hop, so in
long-haul you'll know about circuit quality to the transponder, not
end-to-end. Unlike in wan-phy, OTN where you know both.

Technically optical transport could induce FEC errors, if there are
FEC errors on any hop, so consumers of optical networks need not have
access to optical networks to know if it's end-to-end clean. Much like
cut-through switching can induce errors via some symbols to
communicate the CRC errors happened earlier, so the receiver doesn't
have to worry about problems on their end.

-- 
  ++ytti

Re: constant FEC errors juniper mpc10e 400g

2024-04-19 Thread Mark Tinka




On 4/19/24 08:01, Saku Ytti wrote:


The frames in FEC are idle frames between actual ethernet frames. So
you recall right, without FEC, you won't see this idle traffic.

It's very very good, because now you actually know before putting the
circuit in production, if the circuit works or not.

Lot of people have processes to ping from router-to-router for N time,
trying to determine circuit correctness before putting traffic on it,
which looks absolutely childish compared to FEC, both in terms of how
reliable the presumed outcome is and how long it takes to get to that
presumed outcome.


FEC is amazing.

At higher data rates (100G and 400G) for long and ultra long haul 
optical networks, SD-FEC (Soft Decision FEC) carries a higher overhead 
penalty compared to HD-FEC (Hard Decision FEC), but the net OSNR gain 
more than compensates for that, and makes it worth it to increase 
transmission distance without compromising throughput.


Mark.

Re: constant FEC errors juniper mpc10e 400g

2024-04-19 Thread Saku Ytti

On Thu, 18 Apr 2024 at 21:49, Aaron Gould  wrote:

> Thanks.  What "all the ethernet control frame juju" might you be referring 
> to?  I don't recall Ethernet, in and of itself, just sending stuff back and 
> forth.  Does anyone know if this FEC stuff I see concurring is actually 
> contained in Ethernet Frames?  If so, please send a link to show the ethernet 
> frame structure as it pertains to this 400g fec stuff.  If so, I'd really 
> like to know the header format, etc.

The frames in FEC are idle frames between actual ethernet frames. So
you recall right, without FEC, you won't see this idle traffic.

It's very very good, because now you actually know before putting the
circuit in production, if the circuit works or not.

Lot of people have processes to ping from router-to-router for N time,
trying to determine circuit correctness before putting traffic on it,
which looks absolutely childish compared to FEC, both in terms of how
reliable the presumed outcome is and how long it takes to get to that
presumed outcome.

-- 
  ++ytti

Re: constant FEC errors juniper mpc10e 400g

2024-04-18 Thread Tom Beecher

I'm being sloppy with my verbiage, it's just been a long time since
I thought about this in detail, sorry.

The MAC layer hands bits to the Media Independent Interface, which connects
the MAC to the PHY. The PHY converts the digital 1/0 into the form required
by the media transmission type; the 'what goes over the wire' L1 stuff. The
method of encoding will always add SOME number of bits as overhead. Ex,
64b/66b means that for every 64 bits of data to transmit, 2 bits are added,
so 66 actual bits are transmitted. This encoding overhead is what I meant
when I said 'ethernet control frame juju'. This starts getting into the
weeds on symbol/baud rates and stuff as well, which I dont want to do now
cause I'm even rustier there.

When FEC is enabled, the number of overhead bits added to the transmission
increases. For 400G-FR4 for example, you start with 256b/257b , which is
doubled to 512b/514b for ($reason I cannot remember), then RS-FEC(544,514)
is applied, adding 30 more bits for FEC. Following the example, this means
544 bits are transmitted for every 512 bits of payload data. So , more
overhead. Those additional bits can correct up to 15 corrupted bits of the
payload.

All of these overhead bits are added in the PHY on the way out, and removed
on the way in. So you'll never see them on a packet capture unless you're
using something that's actually grabbing the bits off the wire.

( Pretty sure this is right, anyone please correct me if I munged any of it
up.)

On Thu, Apr 18, 2024 at 2:45 PM Aaron Gould  wrote:

> Thanks.  What "all the ethernet control frame juju" might you be referring
> to?  I don't recall Ethernet, in and of itself, just sending stuff back and
> forth.  Does anyone know if this FEC stuff I see concurring is actually
> contained in Ethernet Frames?  If so, please send a link to show the
> ethernet frame structure as it pertains to this 400g fec stuff.  If so, I'd
> really like to know the header format, etc.
>
> -Aaron
> On 4/18/2024 1:17 PM, Tom Beecher wrote:
>
> FEC is occurring at the PHY , below the PCS.
>
> Even if you're not sending any traffic, all the ethernet control frame
> juju is still going back and forth, which FEC may have to correct.
>
> I *think* (but not 100% sure) that for anything that by spec requires FEC,
> there is a default RS-FEC type that will be used, which *may* be able to be
> changed by the device. Could be fixed though, I honestly cannot remember.
>
> On Thu, Apr 18, 2024 at 1:35 PM Aaron Gould  wrote:
>
>> Not to belabor this, but so interesting... I need a FEC-for-Dummies or 
>> FEC-for-IP/Ethernet-Engineers...
>>
>> Shown below, my 400g interface with NO config at all... Interface has no 
>> traffic at all, no packets at all  BUT, lots of FEC hits.  Interesting 
>> this FEC-thing.  I'd love to have a fiber splitter and see if wireshark 
>> could read it and show me what FEC looks like...but something tells me i 
>> would need a 400g sniffer to read it, lol
>>
>> It's like FEC (fec119 in this case) is this automatic thing running between 
>> interfaces (hardware i guess), with no protocols and nothing needed at all 
>> in order to function.
>>
>> -Aaron
>>
>>
>> {master}
>> me@mx960> show configuration interfaces et-7/1/4 | display set
>>
>> {master}
>> me@mx960>
>>
>> {master}
>> me@mx960> clear interfaces statistics et-7/1/4
>>
>> {master}
>> me@mx960> show interfaces et-7/1/4 | grep packet
>> Input packets : 0
>> Output packets: 0
>>
>> {master}
>> me@mx960> show interfaces et-7/1/4 | grep "put rate"
>>   Input rate : 0 bps (0 pps)
>>   Output rate: 0 bps (0 pps)
>>
>> {master}
>> me@mx960> show interfaces et-7/1/4 | grep rror
>>   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU 
>> Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
>> filtering: Disabled,
>> Bit errors 0
>> Errored blocks 0
>>   Ethernet FEC statistics  Errors
>> FEC Corrected Errors28209
>> FEC Uncorrected Errors  0
>> FEC Corrected Errors Rate2347
>> FEC Uncorrected Errors Rate 0
>>
>> {master}
>> me@mx960> show interfaces et-7/1/4 | grep packet
>> Input packets : 0
>> Output packets: 0
>>
>> {master}
>> me@mx960> show interfaces et-7/1/4 | grep "put rate"
>>   Input rate : 0 bps (0 pps)
>>   Output rate: 0 bps (0 pps)
>>
>> {master}
>> me@mx960> show interfaces et-7/1/4 | grep rror
>>   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU 
>> Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
>> filtering: Disabled,
>> Bit errors 0
>> Errored blocks 0
>>   Ethernet FEC statistics  Errors
>> FEC Corrected Errors45153
>> FEC Uncorrected Errors  0
>> FEC Corrected Errors Rate  29
>> FEC Uncorrected

Re: constant FEC errors juniper mpc10e 400g

2024-04-18 Thread Charles Polisher





On 4/18/24 11:45, Aaron Gould wrote:


Thanks.  What "all the ethernet control frame juju" might you be 
referring to?  I don't recall Ethernet, in and of itself, just sending 
stuff back and forth.  Does anyone know if this FEC stuff I see 
concurring is actually contained in Ethernet Frames?  If so, please 
send a link to show the ethernet frame structure as it pertains to 
this 400g fec stuff.  If so, I'd really like to know the header 
format, etc.


-Aaron



IEEE Std 802.3™‐2022 Standard for Ethernet
(§65.2.3.2 FEC frame format p.2943)
https://ieeexplore.ieee.org/browse/standards/get-program/page/series?id=68

Also helpful, generally:
ITU-T 2000 Recommendation G975 Forward Error Correction for Submarine 
Systems

https://www.itu.int/rec/dologin_pub.asp?lang=e=T-REC-G.975-200010-I!!PDF-E=items

Re: constant FEC errors juniper mpc10e 400g

2024-04-18 Thread JÁKÓ András

> What "all the ethernet control frame juju" might you be referring
> to?  I don't recall Ethernet, in and of itself, just sending stuff back and
> forth.

I did not read the 100G Ethernet specs, but as far as I remember
FastEthernet (e.g. 100BASE-FX) uses 4B/5B coding on the line, borrowed
from FDDI. Octets of Ethernet frames are encoded to these 5-bit
codewords, and there are valid codewords for other stuff, like idle
symbols transmitted continuously between frames.

Gigabit Ethernet (1000BASE-X) uses 8B/10B code on the line (from Fibre
Channel). In GE there are also special (not frame octet) PCS codewords
used for auto-negotiation, frame bursting, etc.

So I guess these are not frames that you see, but codewords representing
other data, outside Ethernet frames.

András

Re: constant FEC errors juniper mpc10e 400g

2024-04-18 Thread Aaron Gould

Thanks.  What "all the ethernet control frame juju" might you be 
referring to?  I don't recall Ethernet, in and of itself, just sending 
stuff back and forth.  Does anyone know if this FEC stuff I see 
concurring is actually contained in Ethernet Frames?  If so, please send 
a link to show the ethernet frame structure as it pertains to this 400g 
fec stuff.  If so, I'd really like to know the header format, etc.


-Aaron

On 4/18/2024 1:17 PM, Tom Beecher wrote:

FEC is occurring at the PHY , below the PCS.

Even if you're not sending any traffic, all the ethernet control frame 
juju is still going back and forth, which FEC may have to correct.


I *think* (but not 100% sure) that for anything that by spec requires 
FEC, there is a default RS-FEC type that will be used, which *may* be 
able to be changed by the device. Could be fixed though, I honestly 
cannot remember.


On Thu, Apr 18, 2024 at 1:35 PM Aaron Gould  wrote:

Not to belabor this, but so interesting... I need a FEC-for-Dummies or 
FEC-for-IP/Ethernet-Engineers...

Shown below, my 400g interface with NO config at all... Interface has no 
traffic at all, no packets at all  BUT, lots of FEC hits.  Interesting this 
FEC-thing.  I'd love to have a fiber splitter and see if wireshark could read 
it and show me what FEC looks like...but something tells me i would need a 400g 
sniffer to read it, lol

It's like FEC (fec119 in this case) is this automatic thing running between 
interfaces (hardware i guess), with no protocols and nothing needed at all in 
order to function.

-Aaron

{master}
me@mx960> show configuration interfaces et-7/1/4 | display set

{master}
me@mx960>

{master}
me@mx960> clear interfaces statistics et-7/1/4

{master}
me@mx960> show interfaces et-7/1/4 | grep packet
     Input packets : 0
     Output packets: 0

{master}
me@mx960> show interfaces et-7/1/4 | grep "put rate"
   Input rate : 0 bps (0 pps)
   Output rate    : 0 bps (0 pps)

{master}
me@mx960> show interfaces et-7/1/4 | grep rror
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU 
Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
     Bit errors 0
     Errored blocks 0
   Ethernet FEC statistics  Errors
     FEC Corrected Errors    28209
     FEC Uncorrected Errors  0
     FEC Corrected Errors Rate    2347
     FEC Uncorrected Errors Rate 0

{master}
me@mx960> show interfaces et-7/1/4 | grep packet
     Input packets : 0
     Output packets: 0

{master}
me@mx960> show interfaces et-7/1/4 | grep "put rate"
   Input rate : 0 bps (0 pps)
   Output rate    : 0 bps (0 pps)

{master}
me@mx960> show interfaces et-7/1/4 | grep rror
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU 
Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
     Bit errors 0
     Errored blocks 0
   Ethernet FEC statistics  Errors
     FEC Corrected Errors    45153
     FEC Uncorrected Errors  0
     FEC Corrected Errors Rate  29
     FEC Uncorrected Errors Rate 0

{master}
me@mx960> show interfaces et-7/1/4 | grep packet
     Input packets : 0
     Output packets: 0

{master}
me@mx960> show interfaces et-7/1/4 | grep "put rate"
   Input rate : 0 bps (0 pps)
   Output rate    : 0 bps (0 pps)

{master}
me@mx960> show interfaces et-7/1/4 | grep rror
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU 
Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
     Bit errors 0
     Errored blocks 0
   Ethernet FEC statistics  Errors
     FEC Corrected Errors    57339
     FEC Uncorrected Errors  0
     FEC Corrected Errors Rate    2378
     FEC Uncorrected Errors Rate 0

{master}
me@mx960>


On 4/18/2024 7:13 AM, Mark Tinka wrote:



On 4/17/24 23:24, Aaron Gould wrote:


Well JTAC just said that it seems ok, and that 400g is going to
show 4x more than 100g "This is due to having to synchronize
much more to support higher data."



We've seen the same between Juniper and Arista boxes in the same
rack running at 100G, despite cleaning fibres, swapping optics,
moving ports, moving line cards, e.t.c. TAC said it's a
non-issue, and to be expected, and shared the same KB's.

It's a bit disconcerting when you plot the data on your NMS, but
it's not material.

Mark.


--

Re: constant FEC errors juniper mpc10e 400g

2024-04-18 Thread Tom Beecher

FEC is occurring at the PHY , below the PCS.

Even if you're not sending any traffic, all the ethernet control frame juju
is still going back and forth, which FEC may have to correct.

I *think* (but not 100% sure) that for anything that by spec requires FEC,
there is a default RS-FEC type that will be used, which *may* be able to be
changed by the device. Could be fixed though, I honestly cannot remember.

On Thu, Apr 18, 2024 at 1:35 PM Aaron Gould  wrote:

> Not to belabor this, but so interesting... I need a FEC-for-Dummies or 
> FEC-for-IP/Ethernet-Engineers...
>
> Shown below, my 400g interface with NO config at all... Interface has no 
> traffic at all, no packets at all  BUT, lots of FEC hits.  Interesting 
> this FEC-thing.  I'd love to have a fiber splitter and see if wireshark could 
> read it and show me what FEC looks like...but something tells me i would need 
> a 400g sniffer to read it, lol
>
> It's like FEC (fec119 in this case) is this automatic thing running between 
> interfaces (hardware i guess), with no protocols and nothing needed at all in 
> order to function.
>
> -Aaron
>
>
> {master}
> me@mx960> show configuration interfaces et-7/1/4 | display set
>
> {master}
> me@mx960>
>
> {master}
> me@mx960> clear interfaces statistics et-7/1/4
>
> {master}
> me@mx960> show interfaces et-7/1/4 | grep packet
> Input packets : 0
> Output packets: 0
>
> {master}
> me@mx960> show interfaces et-7/1/4 | grep "put rate"
>   Input rate : 0 bps (0 pps)
>   Output rate: 0 bps (0 pps)
>
> {master}
> me@mx960> show interfaces et-7/1/4 | grep rror
>   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU 
> Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
> filtering: Disabled,
> Bit errors 0
> Errored blocks 0
>   Ethernet FEC statistics  Errors
> FEC Corrected Errors28209
> FEC Uncorrected Errors  0
> FEC Corrected Errors Rate2347
> FEC Uncorrected Errors Rate 0
>
> {master}
> me@mx960> show interfaces et-7/1/4 | grep packet
> Input packets : 0
> Output packets: 0
>
> {master}
> me@mx960> show interfaces et-7/1/4 | grep "put rate"
>   Input rate : 0 bps (0 pps)
>   Output rate: 0 bps (0 pps)
>
> {master}
> me@mx960> show interfaces et-7/1/4 | grep rror
>   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU 
> Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
> filtering: Disabled,
> Bit errors 0
> Errored blocks 0
>   Ethernet FEC statistics  Errors
> FEC Corrected Errors45153
> FEC Uncorrected Errors  0
> FEC Corrected Errors Rate  29
> FEC Uncorrected Errors Rate 0
>
> {master}
> me@mx960> show interfaces et-7/1/4 | grep packet
> Input packets : 0
> Output packets: 0
>
> {master}
> me@mx960> show interfaces et-7/1/4 | grep "put rate"
>   Input rate : 0 bps (0 pps)
>   Output rate: 0 bps (0 pps)
>
> {master}
> me@mx960> show interfaces et-7/1/4 | grep rror
>   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU 
> Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
> filtering: Disabled,
> Bit errors 0
> Errored blocks 0
>   Ethernet FEC statistics  Errors
> FEC Corrected Errors57339
> FEC Uncorrected Errors  0
> FEC Corrected Errors Rate2378
> FEC Uncorrected Errors Rate 0
>
> {master}
> me@mx960>
>
>
> On 4/18/2024 7:13 AM, Mark Tinka wrote:
>
>
>
> On 4/17/24 23:24, Aaron Gould wrote:
>
> Well JTAC just said that it seems ok, and that 400g is going to show 4x
> more than 100g "This is due to having to synchronize much more to support
> higher data."
>
>
> We've seen the same between Juniper and Arista boxes in the same rack
> running at 100G, despite cleaning fibres, swapping optics, moving ports,
> moving line cards, e.t.c. TAC said it's a non-issue, and to be expected,
> and shared the same KB's.
>
> It's a bit disconcerting when you plot the data on your NMS, but it's not
> material.
>
> Mark.
>
> --
> -Aaron
>
>

Re: constant FEC errors juniper mpc10e 400g

2024-04-18 Thread Aaron Gould


Not to belabor this, but so interesting... I need a FEC-for-Dummies or 
FEC-for-IP/Ethernet-Engineers...

Shown below, my 400g interface with NO config at all... Interface has no 
traffic at all, no packets at all  BUT, lots of FEC hits.  Interesting this 
FEC-thing.  I'd love to have a fiber splitter and see if wireshark could read 
it and show me what FEC looks like...but something tells me i would need a 400g 
sniffer to read it, lol

It's like FEC (fec119 in this case) is this automatic thing running between 
interfaces (hardware i guess), with no protocols and nothing needed at all in 
order to function.

-Aaron

{master}
me@mx960> show configuration interfaces et-7/1/4 | display set

{master}
me@mx960>

{master}
me@mx960> clear interfaces statistics et-7/1/4

{master}
me@mx960> show interfaces et-7/1/4 | grep packet
    Input packets : 0
    Output packets: 0

{master}
me@mx960> show interfaces et-7/1/4 | grep "put rate"
  Input rate : 0 bps (0 pps)
  Output rate    : 0 bps (0 pps)

{master}
me@mx960> show interfaces et-7/1/4 | grep rror
  Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU Error: 
None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
    Bit errors 0
    Errored blocks 0
  Ethernet FEC statistics  Errors
    FEC Corrected Errors    28209
    FEC Uncorrected Errors  0
    FEC Corrected Errors Rate    2347
    FEC Uncorrected Errors Rate 0

{master}
me@mx960> show interfaces et-7/1/4 | grep packet
    Input packets : 0
    Output packets: 0

{master}
me@mx960> show interfaces et-7/1/4 | grep "put rate"
  Input rate : 0 bps (0 pps)
  Output rate    : 0 bps (0 pps)

{master}
me@mx960> show interfaces et-7/1/4 | grep rror
  Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU Error: 
None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
    Bit errors 0
    Errored blocks 0
  Ethernet FEC statistics  Errors
    FEC Corrected Errors    45153
    FEC Uncorrected Errors  0
    FEC Corrected Errors Rate  29
    FEC Uncorrected Errors Rate 0

{master}
me@mx960> show interfaces et-7/1/4 | grep packet
    Input packets : 0
    Output packets: 0

{master}
me@mx960> show interfaces et-7/1/4 | grep "put rate"
  Input rate : 0 bps (0 pps)
  Output rate    : 0 bps (0 pps)

{master}
me@mx960> show interfaces et-7/1/4 | grep rror
  Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU Error: 
None, Loop Detect PDU Error: None, Loopback: Disabled, Source filtering: 
Disabled,
    Bit errors 0
    Errored blocks 0
  Ethernet FEC statistics  Errors
    FEC Corrected Errors    57339
    FEC Uncorrected Errors  0
    FEC Corrected Errors Rate    2378
    FEC Uncorrected Errors Rate 0

{master}
me@mx960>


On 4/18/2024 7:13 AM, Mark Tinka wrote:



On 4/17/24 23:24, Aaron Gould wrote:

Well JTAC just said that it seems ok, and that 400g is going to show 
4x more than 100g "This is due to having to synchronize much more to 
support higher data."




We've seen the same between Juniper and Arista boxes in the same rack 
running at 100G, despite cleaning fibres, swapping optics, moving 
ports, moving line cards, e.t.c. TAC said it's a non-issue, and to be 
expected, and shared the same KB's.


It's a bit disconcerting when you plot the data on your NMS, but it's 
not material.


Mark.


--
-Aaron

Re: constant FEC errors juniper mpc10e 400g

2024-04-18 Thread Mark Tinka





On 4/18/24 15:45, Tom Beecher wrote:



 Just for extra clarity off those KB, probably has nothing to do with 
vendor interop as implied in at least one of those.


Yes, correct.

Mark.

Re: constant FEC errors juniper mpc10e 400g

2024-04-18 Thread Tom Beecher

>
> We've seen the same between Juniper and Arista boxes in the same rack
> running at 100G, despite cleaning fibres, swapping optics, moving ports,
> moving line cards, e.t.c. TAC said it's a non-issue, and to be expected,
> and shared the same KB's.
>
>
 Just for extra clarity off those KB, probably has nothing to do with
vendor interop as implied in at least one of those.

You will see some volume of FEC corrected on 400G FR4 with the same router
hardware and transceiver vendor on both ends, with a 3m patch. Short of
duct taping the transceivers together, not going to get much more optimal
than that.

As far as I can suss out from my reading and what Smart People have told
me, certain combinations of modulation and lamda are just more susceptible
to transmission noise, so for those FEC is required by the standard. PAM4
modulation does seem to be a common thread, but there are some PAM2/NRZs
that FEC is also required for. ( 100GBASE-CWDM4 for example. )

On Thu, Apr 18, 2024 at 8:15 AM Mark Tinka  wrote:

>
>
> On 4/17/24 23:24, Aaron Gould wrote:
>
> > Well JTAC just said that it seems ok, and that 400g is going to show
> > 4x more than 100g "This is due to having to synchronize much more to
> > support higher data."
> >
>
> We've seen the same between Juniper and Arista boxes in the same rack
> running at 100G, despite cleaning fibres, swapping optics, moving ports,
> moving line cards, e.t.c. TAC said it's a non-issue, and to be expected,
> and shared the same KB's.
>
> It's a bit disconcerting when you plot the data on your NMS, but it's
> not material.
>
> Mark.
>

Re: constant FEC errors juniper mpc10e 400g

2024-04-18 Thread Thomas Scott

Standard deviation is now your friend. Learned to alert on outside of SD
FEC and CRCs. Although the second should already be alerting.

On Thu, Apr 18, 2024 at 8:15 AM Mark Tinka  wrote:

> On 4/17/24 23: 24, Aaron Gould wrote: > Well JTAC just said that it seems
> ok, and that 400g is going to show > 4x more than 100g "This is due to
> having to synchronize much more to > support higher data. " > We've seen
> the same between
> 
>
>
> On 4/17/24 23:24, Aaron Gould wrote:
>
> > Well JTAC just said that it seems ok, and that 400g is going to show
> > 4x more than 100g "This is due to having to synchronize much more to
> > support higher data."
> >
>
> We've seen the same between Juniper and Arista boxes in the same rack
> running at 100G, despite cleaning fibres, swapping optics, moving ports,
> moving line cards, e.t.c. TAC said it's a non-issue, and to be expected,
> and shared the same KB's.
>
> It's a bit disconcerting when you plot the data on your NMS, but it's
> not material.
>
> Mark.
>
>

Re: constant FEC errors juniper mpc10e 400g

2024-04-18 Thread Joel Busch via NANOG

Interface index: 226, SNMP ifIndex: 800
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, 
BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
filtering: Disabled,
   Flow control: Enabled
   Pad to minimum frame size: Disabled
   Device flags   : Present Running
   Interface flags: SNMP-Traps Internal: 0x4000
   Link flags : None
   CoS queues : 8 supported, 8 maximum usable queues
   Schedulers : 0
   Last flapped   : 2024-04-17 13:55:28 CDT (00:36:19 ago)
   Input rate : 0 bps (0 pps)
   Output rate: 0 bps (0 pps)
   Active alarms  : None
   Active defects : None
   PCS statistics  Seconds
 Bit errors 0
 Errored blocks 0
   Ethernet FEC Mode  : FEC119
   Ethernet FEC statistics  Errors
 FEC Corrected Errors   801787
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate2054
 FEC Uncorrected Errors Rate 0
   Link Degrade :
 Link Monitoring   :  Disable
   Interface transmit statistics: Disabled

   Logical interface et-7/1/4.0 (Index 420) (SNMP ifIndex 815)
 Flags: Up SNMP-Traps 0x4004000 Encapsulation: ENET2
 Input packets : 1
 Output packets: 1
 Protocol inet, MTU: 1500
 Max nh cache: 75000, New hold nh limit: 75000, Curr nh cnt: 1, 
Curr new hold cnt: 0, NH drop cnt: 0
   Flags: Sendbcast-pkt-to-re
   Addresses, Flags: Is-Preferred Is-Primary
 Destination:10.10.10.76/30  <http://10.10.10.76/30>, Local: 
10.10.10.77, Broadcast: 10.10.10.79

-- 
-Aaron


-- 
-Aaron




--
Joel Busch, Network

SWITCH
Werdstrasse 2, P.O. Box, 8021 Zurich, Switzerland
phone +41 44 268 15 30, direct +41 44 268 16 58

Re: constant FEC errors juniper mpc10e 400g

2024-04-18 Thread Mark Tinka





On 4/17/24 23:24, Aaron Gould wrote:

Well JTAC just said that it seems ok, and that 400g is going to show 
4x more than 100g "This is due to having to synchronize much more to 
support higher data."




We've seen the same between Juniper and Arista boxes in the same rack 
running at 100G, despite cleaning fibres, swapping optics, moving ports, 
moving line cards, e.t.c. TAC said it's a non-issue, and to be expected, 
and shared the same KB's.


It's a bit disconcerting when you plot the data on your NMS, but it's 
not material.


Mark.

Re: constant FEC errors juniper mpc10e 400g

2024-04-18 Thread Joe Antkowiak

  FEC Corrected Errors Rate2054
> FEC Uncorrected Errors Rate 0
>   Link Degrade :
> Link Monitoring   :  Disable
>   Interface transmit statistics: Disabled
>
>   Logical interface et-7/1/4.0 (Index 420) (SNMP ifIndex 815)
> Flags: Up SNMP-Traps 0x4004000 Encapsulation: ENET2
> Input packets : 1
> Output packets: 1
> Protocol inet, MTU: 1500
> Max nh cache: 75000, New hold nh limit: 75000, Curr nh cnt: 1, Curr new 
> hold cnt: 0, NH drop cnt: 0
>   Flags: Sendbcast-pkt-to-re
>   Addresses, Flags: Is-Preferred Is-Primary
> Destination: 10.10.10.76/30, Local: 10.10.10.77, Broadcast: 
> 10.10.10.79
>
> --
> -Aaron

Re: constant FEC errors juniper mpc10e 400g

2024-04-18 Thread Schylar Utley

 Disabled



  Logical interface et-7/1/4.0 (Index 420) (SNMP ifIndex 815)

Flags: Up SNMP-Traps 0x4004000 Encapsulation: ENET2

Input packets : 1

Output packets: 1

Protocol inet, MTU: 1500

Max nh cache: 75000, New hold nh limit: 75000, Curr nh cnt: 1, Curr new 
hold cnt: 0, NH drop cnt: 0

  Flags: Sendbcast-pkt-to-re

  Addresses, Flags: Is-Preferred Is-Primary

Destination: 10.10.10.76/30, Local: 10.10.10.77, Broadcast: 10.10.10.79



--

-Aaron

Re: constant FEC errors juniper mpc10e 400g

2024-04-17 Thread Aaron Gould

cted Errors Rate 0
---(refreshed at 2024-04-17 14:18:59 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, 
BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
filtering: Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors15582
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate 111
 FEC Uncorrected Errors Rate 0
---(refreshed at 2024-04-17 14:19:01 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, 
BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
filtering: Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors20342
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate 256
 FEC Uncorrected Errors Rate 0

{master}
me@mx960> show interfaces et-7/1/4 | grep "put rate"
   Input rate : 0 bps (0 pps)
   Output rate: 0 bps (0 pps)

{master}
me@mx960> show interfaces et-7/1/4
Physical interface: et-7/1/4, Enabled, Physical link is Up
   Interface index: 226, SNMP ifIndex: 800
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, 
BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
filtering: Disabled,
   Flow control: Enabled
   Pad to minimum frame size: Disabled
   Device flags   : Present Running
   Interface flags: SNMP-Traps Internal: 0x4000
   Link flags : None
   CoS queues : 8 supported, 8 maximum usable queues
   Schedulers : 0
   Last flapped   : 2024-04-17 13:55:28 CDT (00:36:19 ago)
   Input rate : 0 bps (0 pps)
   Output rate: 0 bps (0 pps)
   Active alarms  : None
   Active defects : None
   PCS statistics  Seconds
 Bit errors 0
 Errored blocks 0
   Ethernet FEC Mode  : FEC119
   Ethernet FEC statistics  Errors
 FEC Corrected Errors   801787
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate2054
 FEC Uncorrected Errors Rate 0
   Link Degrade :
 Link Monitoring   :  Disable
   Interface transmit statistics: Disabled

   Logical interface et-7/1/4.0 (Index 420) (SNMP ifIndex 815)
 Flags: Up SNMP-Traps 0x4004000 Encapsulation: ENET2
 Input packets : 1
 Output packets: 1
 Protocol inet, MTU: 1500
 Max nh cache: 75000, New hold nh limit: 75000, Curr nh cnt: 1, 
Curr new hold cnt: 0, NH drop cnt: 0
       Flags: Sendbcast-pkt-to-re
   Addresses, Flags: Is-Preferred Is-Primary
 Destination:10.10.10.76/30  <http://10.10.10.76/30>, 
Local: 10.10.10.77, Broadcast: 10.10.10.79

-- 
-Aaron




-- 
Matt Erculiani


-- 
-Aaron




--
Matt Erculiani

--
-Aaron


--
-Aaron

Re: constant FEC errors juniper mpc10e 400g

2024-04-17 Thread Aaron Gould

Source 
filtering: Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors15582
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate 111
 FEC Uncorrected Errors Rate 0
---(refreshed at 2024-04-17 14:19:01 CDT)---
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, 
BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
filtering: Disabled,
 Bit errors 0
 Errored blocks 0
   Ethernet FEC statistics  Errors
 FEC Corrected Errors20342
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate 256
 FEC Uncorrected Errors Rate 0

{master}
me@mx960> show interfaces et-7/1/4 | grep "put rate"
   Input rate : 0 bps (0 pps)
   Output rate: 0 bps (0 pps)

{master}
me@mx960> show interfaces et-7/1/4
Physical interface: et-7/1/4, Enabled, Physical link is Up
   Interface index: 226, SNMP ifIndex: 800
   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, 
BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
filtering: Disabled,
   Flow control: Enabled
   Pad to minimum frame size: Disabled
   Device flags   : Present Running
   Interface flags: SNMP-Traps Internal: 0x4000
   Link flags : None
   CoS queues : 8 supported, 8 maximum usable queues
   Schedulers : 0
   Last flapped   : 2024-04-17 13:55:28 CDT (00:36:19 ago)
   Input rate : 0 bps (0 pps)
   Output rate: 0 bps (0 pps)
   Active alarms  : None
   Active defects : None
   PCS statistics  Seconds
 Bit errors 0
 Errored blocks 0
   Ethernet FEC Mode  : FEC119
   Ethernet FEC statistics  Errors
 FEC Corrected Errors   801787
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate2054
 FEC Uncorrected Errors Rate 0
   Link Degrade :
 Link Monitoring   :  Disable
   Interface transmit statistics: Disabled

   Logical interface et-7/1/4.0 (Index 420) (SNMP ifIndex 815)
 Flags: Up SNMP-Traps 0x4004000 Encapsulation: ENET2
 Input packets : 1
 Output packets: 1
 Protocol inet, MTU: 1500
 Max nh cache: 75000, New hold nh limit: 75000, Curr nh cnt: 1, 
Curr new hold cnt: 0, NH drop cnt: 0
   Flags: Sendbcast-pkt-to-re
   Addresses, Flags: Is-Preferred Is-Primary
 Destination:10.10.10.76/30  <http://10.10.10.76/30>, 
Local: 10.10.10.77, Broadcast: 10.10.10.79

-- 
-Aaron




-- 
Matt Erculiani


-- 
-Aaron




--
Matt Erculiani


--
-Aaron

Re: constant FEC errors juniper mpc10e 400g

2024-04-17 Thread Matt Erculiani

ected Errors Rate 111
>>> FEC Uncorrected Errors Rate 0
>>> ---(refreshed at 2024-04-17 14:19:01 CDT)---
>>>   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU 
>>> Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
>>> filtering: Disabled,
>>> Bit errors 0
>>> Errored blocks 0
>>>   Ethernet FEC statistics  Errors
>>> FEC Corrected Errors20342
>>> FEC Uncorrected Errors  0
>>> FEC Corrected Errors Rate 256
>>> FEC Uncorrected Errors Rate 0
>>>
>>> {master}
>>> me@mx960> show interfaces et-7/1/4 | grep "put rate"
>>>   Input rate : 0 bps (0 pps)
>>>   Output rate: 0 bps (0 pps)
>>>
>>> {master}
>>> me@mx960> show interfaces et-7/1/4
>>> Physical interface: et-7/1/4, Enabled, Physical link is Up
>>>   Interface index: 226, SNMP ifIndex: 800
>>>   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps, BPDU 
>>> Error: None, Loop Detect PDU Error: None, Loopback: Disabled, Source 
>>> filtering: Disabled,
>>>   Flow control: Enabled
>>>   Pad to minimum frame size: Disabled
>>>   Device flags   : Present Running
>>>   Interface flags: SNMP-Traps Internal: 0x4000
>>>   Link flags : None
>>>   CoS queues : 8 supported, 8 maximum usable queues
>>>   Schedulers : 0
>>>   Last flapped   : 2024-04-17 13:55:28 CDT (00:36:19 ago)
>>>   Input rate : 0 bps (0 pps)
>>>   Output rate: 0 bps (0 pps)
>>>   Active alarms  : None
>>>   Active defects : None
>>>   PCS statistics  Seconds
>>> Bit errors 0
>>> Errored blocks 0
>>>   Ethernet FEC Mode  : FEC119
>>>   Ethernet FEC statistics  Errors
>>> FEC Corrected Errors   801787
>>> FEC Uncorrected Errors  0
>>> FEC Corrected Errors Rate2054
>>> FEC Uncorrected Errors Rate 0
>>>   Link Degrade :
>>> Link Monitoring   :  Disable
>>>   Interface transmit statistics: Disabled
>>>
>>>   Logical interface et-7/1/4.0 (Index 420) (SNMP ifIndex 815)
>>> Flags: Up SNMP-Traps 0x4004000 Encapsulation: ENET2
>>> Input packets : 1
>>> Output packets: 1
>>> Protocol inet, MTU: 1500
>>> Max nh cache: 75000, New hold nh limit: 75000, Curr nh cnt: 1, Curr new 
>>> hold cnt: 0, NH drop cnt: 0
>>>   Flags: Sendbcast-pkt-to-re
>>>   Addresses, Flags: Is-Preferred Is-Primary
>>> Destination: 10.10.10.76/30, Local: 10.10.10.77, Broadcast: 
>>> 10.10.10.79
>>>
>>>
>>> --
>>> -Aaron
>>>
>>>
>
> --
> Matt Erculiani
>
> --
> -Aaron
>
>

-- 
Matt Erculiani

Re: constant FEC errors juniper mpc10e 400g

2024-04-17 Thread Tom Beecher

 et-7/1/4 | grep "put rate"
> >   Input rate : 0 bps (0 pps)
> >   Output rate: 0 bps (0 pps)
> >
> > {master}
> > me@mx960> show interfaces et-7/1/4
> > Physical interface: et-7/1/4, Enabled, Physical link is Up
> >   Interface index: 226, SNMP ifIndex: 800
> >   Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 400Gbps,
> > BPDU Error: None, Loop Detect PDU Error: None, Loopback: Disabled,
> > Source filtering: Disabled,
> >   Flow control: Enabled
> >   Pad to minimum frame size: Disabled
> >   Device flags   : Present Running
> >   Interface flags: SNMP-Traps Internal: 0x4000
> >   Link flags : None
> >   CoS queues : 8 supported, 8 maximum usable queues
> >   Schedulers : 0
> >   Last flapped   : 2024-04-17 13:55:28 CDT (00:36:19 ago)
> >   Input rate : 0 bps (0 pps)
> >   Output rate: 0 bps (0 pps)
> >   Active alarms  : None
> >   Active defects : None
> >   PCS statistics  Seconds
> > Bit errors 0
> > Errored blocks 0
> >   Ethernet FEC Mode  : FEC119
> >   Ethernet FEC statistics  Errors
> > FEC Corrected Errors   801787
> > FEC Uncorrected Errors  0
> > FEC Corrected Errors Rate2054
> > FEC Uncorrected Errors Rate 0
> >   Link Degrade :
> > Link Monitoring   :  Disable
> >   Interface transmit statistics: Disabled
> >
> >   Logical interface et-7/1/4.0 (Index 420) (SNMP ifIndex 815)
> > Flags: Up SNMP-Traps 0x4004000 Encapsulation: ENET2
> > Input packets : 1
> > Output packets: 1
> > Protocol inet, MTU: 1500
> > Max nh cache: 75000, New hold nh limit: 75000, Curr nh cnt: 1,
> > Curr new hold cnt: 0, NH drop cnt: 0
> >   Flags: Sendbcast-pkt-to-re
> >   Addresses, Flags: Is-Preferred Is-Primary
> > Destination: 10.10.10.76/30, Local: 10.10.10.77, Broadcast:
> > 10.10.10.79
> >
> > --
> > -Aaron
>

Re: constant FEC errors juniper mpc10e 400g

2024-04-17 Thread Fredrik Holmqvist / I2B

   0
FEC Corrected Errors Rate2054
FEC Uncorrected Errors Rate 0
  Link Degrade :
Link Monitoring   :  Disable
  Interface transmit statistics: Disabled

  Logical interface et-7/1/4.0 (Index 420) (SNMP ifIndex 815)
Flags: Up SNMP-Traps 0x4004000 Encapsulation: ENET2
Input packets : 1
Output packets: 1
Protocol inet, MTU: 1500
Max nh cache: 75000, New hold nh limit: 75000, Curr nh cnt: 1,
Curr new hold cnt: 0, NH drop cnt: 0
  Flags: Sendbcast-pkt-to-re
  Addresses, Flags: Is-Preferred Is-Primary
Destination: 10.10.10.76/30, Local: 10.10.10.77, Broadcast:
10.10.10.79

--
-Aaron

Re: constant FEC errors juniper mpc10e 400g

2024-04-17 Thread Aaron Gould

t statistics: Disabled

   Logical interface et-7/1/4.0 (Index 420) (SNMP ifIndex 815)
 Flags: Up SNMP-Traps 0x4004000 Encapsulation: ENET2
 Input packets : 1
 Output packets: 1
 Protocol inet, MTU: 1500
 Max nh cache: 75000, New hold nh limit: 75000, Curr nh cnt: 1, Curr new 
hold cnt: 0, NH drop cnt: 0
   Flags: Sendbcast-pkt-to-re
   Addresses, Flags: Is-Preferred Is-Primary
 Destination: 10.10.10.76/30, Local: 10.10.10.77, Broadcast: 10.10.10.79

--
-Aaron



--
-Aaron

Re: constant FEC errors juniper mpc10e 400g

2024-04-17 Thread Aaron Gould

m frame size: Disabled
   Device flags   : Present Running
   Interface flags: SNMP-Traps Internal: 0x4000
   Link flags : None
   CoS queues : 8 supported, 8 maximum usable queues
   Schedulers : 0
   Last flapped   : 2024-04-17 13:55:28 CDT (00:36:19 ago)
   Input rate : 0 bps (0 pps)
   Output rate: 0 bps (0 pps)
   Active alarms  : None
   Active defects : None
   PCS statistics  Seconds
 Bit errors 0
 Errored blocks 0
   Ethernet FEC Mode  : FEC119
   Ethernet FEC statistics  Errors
 FEC Corrected Errors   801787
 FEC Uncorrected Errors  0
 FEC Corrected Errors Rate2054
 FEC Uncorrected Errors Rate 0
   Link Degrade :
 Link Monitoring   :  Disable
   Interface transmit statistics: Disabled

   Logical interface et-7/1/4.0 (Index 420) (SNMP ifIndex 815)
 Flags: Up SNMP-Traps 0x4004000 Encapsulation: ENET2
 Input packets : 1
 Output packets: 1
 Protocol inet, MTU: 1500
 Max nh cache: 75000, New hold nh limit: 75000, Curr nh cnt: 1, 
Curr new hold cnt: 0, NH drop cnt: 0
   Flags: Sendbcast-pkt-to-re
   Addresses, Flags: Is-Preferred Is-Primary
 Destination:10.10.10.76/30  <http://10.10.10.76/30>, Local: 
10.10.10.77, Broadcast: 10.10.10.79

-- 
-Aaron




--
Matt Erculiani


--
-Aaron

Re: constant FEC errors juniper mpc10e 400g

2024-04-17 Thread Tom Beecher

gt;   Link flags : None
>>   CoS queues : 8 supported, 8 maximum usable queues
>>   Schedulers : 0
>>   Last flapped   : 2024-04-17 13:55:28 CDT (00:36:19 ago)
>>   Input rate : 0 bps (0 pps)
>>   Output rate: 0 bps (0 pps)
>>   Active alarms  : None
>>   Active defects : None
>>   PCS statistics  Seconds
>> Bit errors 0
>> Errored blocks 0
>>   Ethernet FEC Mode  : FEC119
>>   Ethernet FEC statistics  Errors
>> FEC Corrected Errors   801787
>> FEC Uncorrected Errors  0
>> FEC Corrected Errors Rate2054
>> FEC Uncorrected Errors Rate 0
>>   Link Degrade :
>> Link Monitoring   :  Disable
>>   Interface transmit statistics: Disabled
>>
>>   Logical interface et-7/1/4.0 (Index 420) (SNMP ifIndex 815)
>> Flags: Up SNMP-Traps 0x4004000 Encapsulation: ENET2
>> Input packets : 1
>> Output packets: 1
>> Protocol inet, MTU: 1500
>> Max nh cache: 75000, New hold nh limit: 75000, Curr nh cnt: 1, Curr new 
>> hold cnt: 0, NH drop cnt: 0
>>   Flags: Sendbcast-pkt-to-re
>>   Addresses, Flags: Is-Preferred Is-Primary
>> Destination: 10.10.10.76/30, Local: 10.10.10.77, Broadcast: 
>> 10.10.10.79
>>
>>
>> --
>> -Aaron
>>
>> --
> -Aaron
>
>

Re: constant FEC errors juniper mpc10e 400g

2024-04-17 Thread Matt Erculiani

sabled, Source 
>> filtering: Disabled,
>>   Flow control: Enabled
>>   Pad to minimum frame size: Disabled
>>   Device flags   : Present Running
>>   Interface flags: SNMP-Traps Internal: 0x4000
>>   Link flags : None
>>   CoS queues : 8 supported, 8 maximum usable queues
>>   Schedulers : 0
>>   Last flapped   : 2024-04-17 13:55:28 CDT (00:36:19 ago)
>>   Input rate : 0 bps (0 pps)
>>   Output rate: 0 bps (0 pps)
>>   Active alarms  : None
>>   Active defects : None
>>   PCS statistics  Seconds
>> Bit errors 0
>> Errored blocks 0
>>   Ethernet FEC Mode  : FEC119
>>   Ethernet FEC statistics  Errors
>> FEC Corrected Errors   801787
>> FEC Uncorrected Errors  0
>> FEC Corrected Errors Rate2054
>> FEC Uncorrected Errors Rate 0
>>   Link Degrade :
>> Link Monitoring   :  Disable
>>   Interface transmit statistics: Disabled
>>
>>   Logical interface et-7/1/4.0 (Index 420) (SNMP ifIndex 815)
>> Flags: Up SNMP-Traps 0x4004000 Encapsulation: ENET2
>> Input packets : 1
>> Output packets: 1
>> Protocol inet, MTU: 1500
>> Max nh cache: 75000, New hold nh limit: 75000, Curr nh cnt: 1, Curr new 
>> hold cnt: 0, NH drop cnt: 0
>>   Flags: Sendbcast-pkt-to-re
>>   Addresses, Flags: Is-Preferred Is-Primary
>> Destination: 10.10.10.76/30, Local: 10.10.10.77, Broadcast: 
>> 10.10.10.79
>>
>>
>> --
>> -Aaron
>>
>>

-- 
Matt Erculiani

Re: constant FEC errors juniper mpc10e 400g

2024-04-17 Thread Aaron Gould

FEC Uncorrected Errors  0
 FEC Corrected Errors Rate2054
 FEC Uncorrected Errors Rate 0
   Link Degrade :
 Link Monitoring   :  Disable
   Interface transmit statistics: Disabled

   Logical interface et-7/1/4.0 (Index 420) (SNMP ifIndex 815)
 Flags: Up SNMP-Traps 0x4004000 Encapsulation: ENET2
 Input packets : 1
 Output packets: 1
 Protocol inet, MTU: 1500
 Max nh cache: 75000, New hold nh limit: 75000, Curr nh cnt: 1, Curr 
new hold cnt: 0, NH drop cnt: 0
   Flags: Sendbcast-pkt-to-re
   Addresses, Flags: Is-Preferred Is-Primary
 Destination:10.10.10.76/30  <http://10.10.10.76/30>, Local: 
10.10.10.77, Broadcast: 10.10.10.79

-- 
-Aaron



--
-Aaron

Re: constant FEC errors juniper mpc10e 400g

2024-04-17 Thread Dominik Dobrowolski

ors  0
> FEC Corrected Errors Rate2054
> FEC Uncorrected Errors Rate 0
>   Link Degrade :
> Link Monitoring   :  Disable
>   Interface transmit statistics: Disabled
>
>   Logical interface et-7/1/4.0 (Index 420) (SNMP ifIndex 815)
> Flags: Up SNMP-Traps 0x4004000 Encapsulation: ENET2
> Input packets : 1
> Output packets: 1
> Protocol inet, MTU: 1500
> Max nh cache: 75000, New hold nh limit: 75000, Curr nh cnt: 1, Curr new 
> hold cnt: 0, NH drop cnt: 0
>   Flags: Sendbcast-pkt-to-re
>   Addresses, Flags: Is-Preferred Is-Primary
> Destination: 10.10.10.76/30, Local: 10.10.10.77, Broadcast: 
> 10.10.10.79
>
>
> --
> -Aaron
>
>

Re: Upcoming LACNIC RPKI Migration

2024-04-16 Thread Alex Band

Hi Carlos,

Congrats to you and the team for the smooth migration. 

I can speak for all of us at NLnet Labs that we’re super proud that LACNIC is 
now running Krill. 

Also, a special thanks to Tim Bruijnzeels (now back at the RIPE NCC) for the 
years of hard work on our open-source RPKI project – and for ironing out a 
small bump yesterday together with NIC.br after the switch-over. 

Cheers,

Alex


> On 15 Apr 2024, at 16:24, Carlos Martinez-Cagnazzo  
> wrote:
> 
> Hi all, it's me again.
> 
> The switch is complete. Thank you all for your patience.
> 
> /Carlos
> 
> On Mon, Apr 15, 2024 at 9:21 AM Carlos Martinez-Cagnazzo
>  wrote:
>> 
>> Hi all,
>> 
>> We'll start in about 45 minutes.
>> 
>> /Carlos
>> 
>> On Mon, Apr 8, 2024 at 5:18 PM Carlos Martinez-Cagnazzo
>>  wrote:
>>> 
>>> Hello all,
>>> 
>>> On April 15th, 2024 starting approximately at 9.30am UTC-3 LACNIC will
>>> be migrating from our current legacy RPKI CA system to a new
>>> Krill-based RPKI core.
>>> 
>>> In most cases no action will be required on your part (see below for
>>> some special cases). What follows is a list of events that will take
>>> place at the mentioned time and that may be of interest to you.
>>> 
>>>* Our TAL file won't change at this time. There is no need to
>>> change anything in your current RP configuration.
>>> 
>>>* Our RTA certificate, while keeping the old key will point to a
>>> new manifest.
>>> 
>>> From the outside, what RPs will see is the following sequence of events:
>>> 
>>>   * At some time T0 all our current servers (both RRDP and rsync)
>>> will be shut down, returning "connection refused '' for both http and
>>> rsync.
>>>   * New values for the DNS records will be published (same names,
>>> different IPs).
>>>   * At approximately T0+30min the servers listening on the new IPs
>>> will be started and will start serving the repository as produced by
>>> the new Krill-based system.
>>>   * When they first connect, RPs will see a new RRDP session and will
>>> take it from there.
>>> 
>>> We have tested this migration flow using a set of docker containers
>>> plus a DNS server container using dnsmasq server that allows us to
>>> modify records on the fly. In all the cases we tested this flow works
>>> just fine.
>>> 
>>> We have tested this migration flow with the following RPs:
>>> 
>>>  * rpki-client from “latest” all the way back to 8.2.
>>>  * routinator from “latest” all the way back to 0.8.
>>>  * fort from “latest” all the way back to 1.5.0.
>>> 
>>> What we have not tested:
>>> 
>>>  * RIPE rpki validator: it’s been deprecated for three years. You
>>> shouldn’t be running this and you know it :-) In any case, it should
>>> work.
>>>  * OctoRPKI: also recently deprecated.
>>>  * Rpki-prover.
>>>  * RIPSTR.
>>> 
>>> All of the above should work. However bear in mind the following: If
>>> you are running any of the above and you notice issues, just clear the
>>> local cache, launch a clean instance of your RP and you should be
>>> fine.
>>> 
>>> We have set up a specific email inbox for this migration work:
>>> rpki-migrac...@lacnic.net. It will be closely monitored during April
>>> 15 and the following days. It will be phased out once we are confident
>>> all issues that may arise have been addressed.
>>> 
>>> For those interested, the new servers are already online and can be
>>> used to validate. These can be reached at:
>>> 
>>>  * lb-us-mia.rrdp.lacnic.net
>>>  * lb-us-southeast.rrdp.lacnic.net
>>>  * lb-br-gru.rrdp.lacnic.net
>>> 
>>> Don’t expect to see the exact same VRPs as you see now on our current
>>> production server as minor differences are expected. Don’t hardcode
>>> this either, as during the migration “rrdp.lacnic.net” will be made to
>>> point to these servers and eventually these names may change and/or
>>> new ones may be added.
>>> 
>>> Thank you all!
>>> 
>>> /Carlos
>> 
>> 
>> 
>> --
>> --
>> =
>> Carlos M. Martinez-Cagnazzo
>> http://cagnazzo.me
>> =
> 
> 
> 
> -- 
> --
> =
> Carlos M. Martinez-Cagnazzo
> http://cagnazzo.me
> =

Re: Upcoming LACNIC RPKI Migration

2024-04-15 Thread Carlos Martinez-Cagnazzo

Hi all, it's me again.

The switch is complete. Thank you all for your patience.

/Carlos

On Mon, Apr 15, 2024 at 9:21 AM Carlos Martinez-Cagnazzo
 wrote:
>
> Hi all,
>
> We'll start in about 45 minutes.
>
> /Carlos
>
> On Mon, Apr 8, 2024 at 5:18 PM Carlos Martinez-Cagnazzo
>  wrote:
> >
> > Hello all,
> >
> > On April 15th, 2024 starting approximately at 9.30am UTC-3 LACNIC will
> > be migrating from our current legacy RPKI CA system to a new
> > Krill-based RPKI core.
> >
> > In most cases no action will be required on your part (see below for
> > some special cases). What follows is a list of events that will take
> > place at the mentioned time and that may be of interest to you.
> >
> > * Our TAL file won't change at this time. There is no need to
> > change anything in your current RP configuration.
> >
> > * Our RTA certificate, while keeping the old key will point to a
> > new manifest.
> >
> > From the outside, what RPs will see is the following sequence of events:
> >
> >* At some time T0 all our current servers (both RRDP and rsync)
> > will be shut down, returning "connection refused '' for both http and
> > rsync.
> >* New values for the DNS records will be published (same names,
> > different IPs).
> >* At approximately T0+30min the servers listening on the new IPs
> > will be started and will start serving the repository as produced by
> > the new Krill-based system.
> >* When they first connect, RPs will see a new RRDP session and will
> > take it from there.
> >
> > We have tested this migration flow using a set of docker containers
> > plus a DNS server container using dnsmasq server that allows us to
> > modify records on the fly. In all the cases we tested this flow works
> > just fine.
> >
> > We have tested this migration flow with the following RPs:
> >
> >   * rpki-client from “latest” all the way back to 8.2.
> >   * routinator from “latest” all the way back to 0.8.
> >   * fort from “latest” all the way back to 1.5.0.
> >
> > What we have not tested:
> >
> >   * RIPE rpki validator: it’s been deprecated for three years. You
> > shouldn’t be running this and you know it :-) In any case, it should
> > work.
> >   * OctoRPKI: also recently deprecated.
> >   * Rpki-prover.
> >   * RIPSTR.
> >
> > All of the above should work. However bear in mind the following: If
> > you are running any of the above and you notice issues, just clear the
> > local cache, launch a clean instance of your RP and you should be
> > fine.
> >
> > We have set up a specific email inbox for this migration work:
> > rpki-migrac...@lacnic.net. It will be closely monitored during April
> > 15 and the following days. It will be phased out once we are confident
> > all issues that may arise have been addressed.
> >
> > For those interested, the new servers are already online and can be
> > used to validate. These can be reached at:
> >
> >   * lb-us-mia.rrdp.lacnic.net
> >   * lb-us-southeast.rrdp.lacnic.net
> >   * lb-br-gru.rrdp.lacnic.net
> >
> > Don’t expect to see the exact same VRPs as you see now on our current
> > production server as minor differences are expected. Don’t hardcode
> > this either, as during the migration “rrdp.lacnic.net” will be made to
> > point to these servers and eventually these names may change and/or
> > new ones may be added.
> >
> > Thank you all!
> >
> > /Carlos
>
>
>
> --
> --
> =
> Carlos M. Martinez-Cagnazzo
> http://cagnazzo.me
> =



-- 
--
=
Carlos M. Martinez-Cagnazzo
http://cagnazzo.me
=

Re: Upcoming LACNIC RPKI Migration

2024-04-15 Thread Carlos Martinez-Cagnazzo

Hi all,

We'll start in about 45 minutes.

/Carlos

On Mon, Apr 8, 2024 at 5:18 PM Carlos Martinez-Cagnazzo
 wrote:
>
> Hello all,
>
> On April 15th, 2024 starting approximately at 9.30am UTC-3 LACNIC will
> be migrating from our current legacy RPKI CA system to a new
> Krill-based RPKI core.
>
> In most cases no action will be required on your part (see below for
> some special cases). What follows is a list of events that will take
> place at the mentioned time and that may be of interest to you.
>
> * Our TAL file won't change at this time. There is no need to
> change anything in your current RP configuration.
>
> * Our RTA certificate, while keeping the old key will point to a
> new manifest.
>
> From the outside, what RPs will see is the following sequence of events:
>
>* At some time T0 all our current servers (both RRDP and rsync)
> will be shut down, returning "connection refused '' for both http and
> rsync.
>* New values for the DNS records will be published (same names,
> different IPs).
>* At approximately T0+30min the servers listening on the new IPs
> will be started and will start serving the repository as produced by
> the new Krill-based system.
>* When they first connect, RPs will see a new RRDP session and will
> take it from there.
>
> We have tested this migration flow using a set of docker containers
> plus a DNS server container using dnsmasq server that allows us to
> modify records on the fly. In all the cases we tested this flow works
> just fine.
>
> We have tested this migration flow with the following RPs:
>
>   * rpki-client from “latest” all the way back to 8.2.
>   * routinator from “latest” all the way back to 0.8.
>   * fort from “latest” all the way back to 1.5.0.
>
> What we have not tested:
>
>   * RIPE rpki validator: it’s been deprecated for three years. You
> shouldn’t be running this and you know it :-) In any case, it should
> work.
>   * OctoRPKI: also recently deprecated.
>   * Rpki-prover.
>   * RIPSTR.
>
> All of the above should work. However bear in mind the following: If
> you are running any of the above and you notice issues, just clear the
> local cache, launch a clean instance of your RP and you should be
> fine.
>
> We have set up a specific email inbox for this migration work:
> rpki-migrac...@lacnic.net. It will be closely monitored during April
> 15 and the following days. It will be phased out once we are confident
> all issues that may arise have been addressed.
>
> For those interested, the new servers are already online and can be
> used to validate. These can be reached at:
>
>   * lb-us-mia.rrdp.lacnic.net
>   * lb-us-southeast.rrdp.lacnic.net
>   * lb-br-gru.rrdp.lacnic.net
>
> Don’t expect to see the exact same VRPs as you see now on our current
> production server as minor differences are expected. Don’t hardcode
> this either, as during the migration “rrdp.lacnic.net” will be made to
> point to these servers and eventually these names may change and/or
> new ones may be added.
>
> Thank you all!
>
> /Carlos



-- 
--
=
Carlos M. Martinez-Cagnazzo
http://cagnazzo.me
=

Re: 2600:: No longer pings

2024-04-14 Thread Randy Bush

> Wonderful news, this has now been fixed :)
> Thank you to Cogent for fixing this

indee.  otoh, i still can not resist https://www.kame.net/

randy

Re: Whitebox Routers Beyond the Datasheet

2024-04-14 Thread Tom Beecher

Agreed.

I think as a practical matter, the large majority of operators probably
only care about time from last update / EoRIB -> FIBs forwarding working.
As long as that time delta is acceptable for their environment and
circumstances, that's 'good enough'. Definitely some edge cases and
circumstances that can factor in.

Those of us in the massive scale part of the world certainly have our own
unique problems with this stuff. :)

On Sat, Apr 13, 2024 at 11:58 AM Jared Mauch  wrote:

>
>
> > On Apr 13, 2024, at 12:15 AM, 7ri...@gmail.com wrote:
> >
> >
> >> I feel like this shouldn't be listed on a data sheet for just the
> whitebox hardware. The software running on it would be the gating factor.
> > There would be two things ... BGP convergence, and then the time
> required to get routes from the RIB into the hardware forwarding tables.
> These are completely separate things. Both are gated on software for the
> most part, and it would be hard to measure them unless you know a lot more
> about the environment. Even then it would be a bit of a guess.
> >
> > Contact me off list if you're interested in prior experience in this
> area.
> >
> > :-) /r
>
>
> Yeah, I think the question is coming from the wrong direction, which is
> what route scale do you need then match it to the hardware.  You can load a
> variety of software on these devices, including putting something like cRPD
> on top of it so you have the Juniper software and policy language, or roll
> your own with FRR, BIRD or something else.
>
> The kernel -> FIB (hardware) download performance will vary as will the
> way the TCAM is carved up into the various routes and profiles.
>
> It also depends on what you download to the FIB vs what you have in your
> RIB, for example a fib-filter in Juniper parlance may give you the ability
> to carry a full routing table but just a default and your local stub routes
> depending on the device role.  (Connected/static + local iBGP+eBGP learned)
>
> - Jared

Re: Open source Netflow analysis for monitoring AS-to-AS traffic

2024-04-14 Thread Vincent Bernat


On 2024-03-27 09:09, Marinos Dimolianis wrote:
My only "concern" was that it did not provide an API for consuming data 
externally.


This is very high on my todo list, notably because I don't want to 
reimplement Grafana. The API already exists (the current web interface 
uses it) but it is not "stable" (it may change in future versions).

Re: PRIX update

2024-04-13 Thread Andy Ringsmuth

Congrats, Mehmet! Your persistence is paying off!


Andy Ringsmuth
5609 Harding Drive
Lincoln, NE 68521-5831
(402) 202-1230
a...@andyring.com

> On Apr 13, 2024, at 10:14 AM, Mehmet  wrote:
> 
> Hello NANOG
> 
> Little bit less than 19 years ago when PRIX was launched I had a dream to 
> connect many ISPs in the island.
> 
> during last 19 years several attempts were made to relaunch an IX but finally 
> last three years a small group of volunteers have really helped PRIX to be 
> come a reality not an IX with PowerPoint and no traffic.
> 
> You can hear the story from those who have significantly helped from project 
> here 
> https://www.youtube.com/live/kCp4kK-mavU?si=N82CGRSatdzrmX2H
> 
> This week there is Global Peering Forum in Puerto Rico, and I as one of the 
> old time nanoger + local(i dont live here anymore but, you know what i mean 
> want to welcome and send this update about PRIX
> 
> https://www.puertoricoix.net has broken traffic record 30Gbps last week and 
> with some more caches in the process of being enabled as well as some key 
> peers (listed on the website), next goal is 100G and this can be achieved 
> with those of you who has an ability to help deploy more caches in Puerto 
> Rico!  
> 
> I want to take a moment thank Jeff, Ivan, and Gino for all the hard work 
> behind. 
> 
> If you are attending GPF this week, let’s talk and see what else we can do to 
> improve internet in Puerto Rico and Caribbean! Any questions related to PRIX 
> feel free to contact me offlist 
> 
> Mehmet
> 
> Note: all links below are broken  
> PRIX - Puerto RIco Internet Exchange
> 
> • From: Mehmet Akcin
> • Date: Tue Sep 27 12:50:12 2005
> Hi,
> 
> I'd like to speak with people who were / are working in IX points and/or have
> knowledge about the infrastructure of IX points.
> 
> We , a small group of researchers , have started a Project called Prix [
> http://prix.uprr.pr ] which has the intentions of creating a large table for
> those who would like to sit down, in other terms an internet exchange point
> where all the participants can peer.
> 
> Our main goal is to improve the existing connectivity between the major 
> backbone
> providers and improve the connectivity between them with peering locally 
> instead
> of peering somewhere in the states, which is couple of thousands miles far 
> from
> Puerto Rico.
> 
> We believe improving the network connectivity of the island will definately 
> led
> those who live in Puerto Rico to think of using VoIP technologies more often.
> 
> We, myself and a friend of mine who is also participating in this project, 
> will
> be participating in NANOG and hopefully can meet with those who has experience
> in order to get recommendations about hardware/software and policies.
> 
> Thank you
> 
> Mehmet Akcin, PRIX
> 
> 
>

Re: Whitebox Routers Beyond the Datasheet

2024-04-13 Thread Jared Mauch

> On Apr 13, 2024, at 12:15 AM, 7ri...@gmail.com wrote:
> 
> 
>> I feel like this shouldn't be listed on a data sheet for just the whitebox 
>> hardware. The software running on it would be the gating factor.
> There would be two things ... BGP convergence, and then the time required to 
> get routes from the RIB into the hardware forwarding tables. These are 
> completely separate things. Both are gated on software for the most part, and 
> it would be hard to measure them unless you know a lot more about the 
> environment. Even then it would be a bit of a guess.
> 
> Contact me off list if you're interested in prior experience in this area.
> 
> :-) /r

Yeah, I think the question is coming from the wrong direction, which is what 
route scale do you need then match it to the hardware.  You can load a variety 
of software on these devices, including putting something like cRPD on top of 
it so you have the Juniper software and policy language, or roll your own with 
FRR, BIRD or something else.

The kernel -> FIB (hardware) download performance will vary as will the way the 
TCAM is carved up into the various routes and profiles.

It also depends on what you download to the FIB vs what you have in your RIB, 
for example a fib-filter in Juniper parlance may give you the ability to carry 
a full routing table but just a default and your local stub routes depending on 
the device role.  (Connected/static + local iBGP+eBGP learned)

- Jared

Re[2]: Whitebox Routers Beyond the Datasheet

2024-04-12 Thread 7ri...@gmail.com




I feel like this shouldn't be listed on a data sheet for just the 
whitebox hardware. The software running on it would be the gating 
factor.
There would be two things ... BGP convergence, and then the time 
required to get routes from the RIB into the hardware forwarding tables. 
These are completely separate things. Both are gated on software for the 
most part, and it would be hard to measure them unless you know a lot 
more about the environment. Even then it would be a bit of a guess.


Contact me off list if you're interested in prior experience in this 
area.


:-) /r

Re: Whitebox Routers Beyond the Datasheet

2024-04-12 Thread William Herrin

On Fri, Apr 12, 2024 at 6:03 AM Mike Hammett  wrote:
> What I've kind of come down to (based on little more than spec sheets)
> are the EdgeCore AGR400 and the UfiSpace S9600-30DX.

> That's wonderful and all, but does it take it from 64k routes
>  to 512k routes, or does it take it from 256k routes up to
>  the millions of routes?

Hi Mike,

You're combining two questions here. Break them apart. How many
distinct routes can its line-speed forwarding engine handle (the FIB)
and what processing/DRAM is available to handle the routing database
(RIB).

To take the EdgeCore AGR400, it has an 8-core Xeon and 32 gigs of ram.
It's going to handle a BGP RIB in the many millions of routes and a
BGP convergence time as fast as anything else you can buy.

For switching (FIB), it uses a Broadcom BCM88823. The data sheets on
the broadcom 88800 series chips are suspiciously light on information
about their internal capacity for forwarding decisions. Just a black
box in the diagram described as an "external lookup engine" and some
notes in the "packet processing" section that the ingress receive
packet processor "handles the main packet processing stage" and
"optionally" uses "expandable databases" from an external lookup
interface.

Large TCAMS are power hungry and the chip is physically small, so I'd
bet against it having an DFZ-size embedded TCAM. Smells like a switch
chip that typically has something in the single-digit thousands of
slots but that's strictly a guess. And with only 8 cores, any
software-switched "expandable databases" are going to be wimpy.

I'm not familiar with the specific hardware you mentioned, so none of
the above is certain. Just based on what I glean from the specs I
could find via google and my experience with white-box routing in
general. As a rule though, if the marketing materials don't call out a
component of the product as impressive, it probably isn't.

In general, white box routing is done with a framework like DPDK on
machines with large numbers of CPU cores. While they can handle
100gbps, they do it by running the cores in single-thread busywait
loops that eliminate the need for interrupts from the network devices.
This generates lots of heat and consumes lots of electricity.

Regards,
Bill Herrin

-- 
William Herrin
b...@herrin.us
https://bill.herrin.us/

Re: Whitebox Routers Beyond the Datasheet

2024-04-12 Thread Mike Hammett

That makes sense, but also why I'm going beyond the datasheet here to solicit 
people's feedback. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "Tom Beecher"  
To: "Mike Hammett"  
Cc: nanog@nanog.org 
Sent: Friday, April 12, 2024 1:30:04 PM 
Subject: Re: Whitebox Routers Beyond the Datasheet 




Also, BGP convergence isn't listed (nor do I rarely ever see it talked about in 
such sheets). 




I feel like this shouldn't be listed on a data sheet for just the whitebox 
hardware. The software running on it would be the gating factor. 


On Fri, Apr 12, 2024 at 9:05 AM Mike Hammett < na...@ics-il.net > wrote: 





I'm looking at the suitability of whitebox routers for high through, low port 
count, fast BGP performance applications. Power efficiency is important as 
well. 


What I've kind of come down to (based on little more than spec sheets) are the 
EdgeCore AGR400 and the UfiSpace S9600-30DX. They can both accommodate at least 
three directions of 400G for linking to other parts of my network and then have 
enough 100G or slower ports to connect to transit, peers, and customers as 
appropriate. Any other suggestions for platforms similar to those would be 
appreciated. 


They both appear to carry buffers large enough to accommodate buffering 
differences in port capacities, which is an issue I've seen with boxes more 
targeted to cloud\datacenter switching. 


What isn't in the spec sheets is BGP-related information. They don't mention 
how many routes they can hold, just that they have additional TCAM to handle 
more routes and features. That's wonderful and all, but does it take it from 
64k routes to 512k routes, or does it take it from 256k routes up to the 
millions of routes? Also, BGP convergence isn't listed (nor do I rarely ever 
see it talked about in such sheets). I know that software-based routers can now 
load a full table in 30 seconds or less. I know that getting the FIB updated 
takes a little bit longer. I know that withdrawing a route takes a little bit 
longer. However, often, that performance is CPU-based. An underpowered CPU may 
take a minute or more to load that table and may take minutes to handle route 
churn. Can anyone speak to these routers (or routers like these) ability to 
handle modern route table activity? 


My deployment locations and philosophies simply won't have me in an environment 
where I need the density of dozens of 400G\100G ports. That the routers that 
seem to be more marketed to the use case are designed for. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com

Re: Whitebox Routers Beyond the Datasheet

2024-04-12 Thread Tom Beecher

>
> Also, BGP convergence isn't listed (nor do I rarely ever see it talked
> about in such sheets).


I feel like this shouldn't be listed on a data sheet for just the whitebox
hardware. The software running on it would be the gating factor.

On Fri, Apr 12, 2024 at 9:05 AM Mike Hammett  wrote:

> I'm looking at the suitability of whitebox routers for high through, low
> port count, fast BGP performance applications. Power efficiency is
> important as well.
>
> What I've kind of come down to (based on little more than spec sheets) are
> the EdgeCore AGR400 and the UfiSpace S9600-30DX. They can both accommodate
> at least three directions of 400G for linking to other parts of my network
> and then have enough 100G or slower ports to connect to transit, peers, and
> customers as appropriate. Any other suggestions for platforms similar to
> those would be appreciated.
>
> They both appear to carry buffers large enough to accommodate buffering
> differences in port capacities, which is an issue I've seen with boxes more
> targeted to cloud\datacenter switching.
>
> What isn't in the spec sheets is BGP-related information. They don't
> mention how many routes they can hold, just that they have additional TCAM
> to handle more routes and features. That's wonderful and all, but does it
> take it from 64k routes to 512k routes, or does it take it from 256k routes
> up to the millions of routes? Also, BGP convergence isn't listed (nor do I
> rarely ever see it talked about in such sheets). I know that software-based
> routers can now load a full table in 30 seconds or less. I know that
> getting the FIB  updated takes a little bit longer. I know that withdrawing
> a route takes a little bit longer. However, often, that performance is
> CPU-based. An underpowered CPU may take a minute or more to load that table
> and may take minutes to handle route churn. Can anyone speak to these
> routers (or routers like these) ability to handle modern route table
> activity?
>
> My deployment locations and philosophies simply won't have me in an
> environment where I need the density of dozens of 400G\100G ports. That the
> routers that seem to be more marketed to the use case are designed for.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions
> http://www.ics-il.com
>
> Midwest-IX
> http://www.midwest-ix.com
>
>

Re: Whitebox Routers Beyond the Datasheet

2024-04-12 Thread Michel Blais

I'm surprised they stopped showing those options in the datasheet. They
used to have those in the spec sheet when I bought Edgecore hardware
some years ago. Seems like you will have to contact sales teams.

If it can help, I use AS5916-XKS and the TCAM supports millions of routes.
Out of my mind, it is something like 6 or 10 millions for IPv4 and 1.5
millions for IPv6.

1 2 3 4 5 6 7 8 9 10 >

1 - 100 of 104589 matches

Mail list logo