Re: MPLS and Carrier Ethernet, Oh My!

[Mark Tinka]

On 12/28/24 00:31, Mike Hammett wrote:

*nods* for PtP, I agree. As a buyer (and a seller) waves, waves, 
waves. As a seller, it's less stuff for me to manage. As a buyer, I 
don't have to trust you on oversubscribing a wave because you can't. 
You can oversubscribe the hell out of an Ethernet circuit, though.


Waves are much harder in a PtMP environment, especially with cross 
connect costs.


That is a problem OpenXR has been designed to fix:

https://openxropticsforum.org

Infinera have already started shipping product:

https://www.infinera.com/innovation/xr-optics/

Mark.

Re: MPLS and Carrier Ethernet, Oh My!

[Mike Hammett]

*nods* for PtP, I agree. As a buyer (and a seller) waves, waves, waves. As a 
seller, it's less stuff for me to manage. As a buyer, I don't have to trust you 
on oversubscribing a wave because you can't. You can oversubscribe the hell out 
of an Ethernet circuit, though. 


Waves are much harder in a PtMP environment, especially with cross connect 
costs. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Tom Beecher"  
To: "Mike Hammett"  
Cc: "NANOG"  
Sent: Friday, December 27, 2024 10:28:02 AM 
Subject: Re: MPLS and Carrier Ethernet, Oh My! 



These days, it's a LOT easier to get dedicated ethernet wave service between A 
and Z than it used to be. The pseudowire options were developed to fill that 
gap that customers wanted. 


Still certainly use cases for it, but generally the dedicated waves are much 
easier to get and probably cheaper. The providers would certainly prefer NOT to 
sell you that service either in most cases, because they don't want the 
overhead of running their parts either, and would like to see it die on the 
vine. 


On Thu, Dec 26, 2024 at 9:37 PM Mike Hammett < na...@ics-il.net > wrote: 





A few years back, every Tom, Dick, and Harry was touting MPLS or Carrier 
Ethernet NNIs with 10G ports everywhere. They still are. However, I rarely have 
seen that graduate to 100G ports and I don't think I've seen anyone talk about 
400G ports. 


Is the hardware not there, or is it more a case of the technology hasn't been 
deployed widely, so it'll only be seen at a limited number of locations? I'm 
assuming it's more of the latter as the bigger hardware I've been looking at 
touts those features and port sizes, but maybe there's some other unadvertised 
limitation. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

Re: MPLS and Carrier Ethernet, Oh My!

[Mark Tinka]

On 12/27/24 18:28, Tom Beecher wrote:

These days, it's a LOT easier to get dedicated ethernet wave service 
between A and Z than it used to be. The pseudowire options were 
developed to fill that gap that customers wanted.


Still certainly use cases for it, but generally the dedicated waves 
are much easier to get and probably cheaper. The providers would 
certainly prefer NOT to sell you that service either in most cases, 
because they don't want the overhead of running their parts either, 
and would like to see it die on the vine.


You begin to run into commercial and technical constraints (let's not 
call them limits) when your EoMPLS orders start to enter the 15G - 20G 
range and higher.


Technically, if a customer wants DWDM-type behaviour where the path is 
fixed, you have to build an RSVP-TE tunnel, which is admin. It's 
possible this can be done more easily with SR-MPLS, but it's still a bit 
of admin. Then you have to make considerations for failure scenarios in 
your core, and how you guarantee your pw customer remains happy.


Commercially, if customers do not demand DWDM-type behavious driven by 
ERO's, you have a conundrum about whether you charge for automatic 
protection or not, because without an ERO, the customer will failover 
when the backbone fails over due to a core outage of some sort.


Mark.

Re: MPLS and Carrier Ethernet, Oh My!

[Tom Beecher]

These days, it's a LOT easier to get dedicated ethernet wave service
between A and Z than it used to be. The pseudowire options were developed
to fill that gap that customers wanted.

Still certainly use cases for it, but generally the dedicated waves are
much easier to get and probably cheaper. The providers would certainly
prefer NOT to sell you that service either in most cases, because they
don't want the overhead of running their parts either, and would like to
see it die on the vine.

On Thu, Dec 26, 2024 at 9:37 PM Mike Hammett  wrote:

> A few years back, every Tom, Dick, and Harry was touting MPLS or Carrier
> Ethernet NNIs with 10G ports everywhere. They still are. However, I rarely
> have seen that graduate to 100G ports and I don't think I've seen anyone
> talk about 400G ports.
>
> Is the hardware not there, or is it more a case of the technology hasn't
> been deployed widely, so it'll only be seen at a limited number of
> locations? I'm assuming it's more of the latter as the bigger hardware I've
> been looking at touts those features and port sizes, but maybe there's some
> other unadvertised limitation.
>
>
>
> -
> Mike Hammett
> Intelligent Computing Solutions 
> 
> 
> 
> 
> Midwest Internet Exchange 
> 
> 
> 
> The Brothers WISP 
> 
> 
>

RE: MPLS and Carrier Ethernet, Oh My!

[Vasilenko Eduard via NANOG]

The average subscriber consumes 1.5Mbps in a busy hour (even for the GE 
access). 100GE is 60k subscribers.
Many access regions just do not have enough subscribers to justify 100GE.
Some regions have.
Ed/
From: NANOG  On Behalf Of 
Mike Hammett
Sent: Friday, December 27, 2024 05:35
To: NANOG 
Subject: MPLS and Carrier Ethernet, Oh My!

A few years back, every Tom, Dick, and Harry was touting MPLS or Carrier 
Ethernet NNIs with 10G ports everywhere. They still are. However, I rarely have 
seen that graduate to 100G ports and I don't think I've seen anyone talk about 
400G ports.

Is the hardware not there, or is it more a case of the technology hasn't been 
deployed widely, so it'll only be seen at a limited number of locations? I'm 
assuming it's more of the latter as the bigger hardware I've been looking at 
touts those features and port sizes, but maybe there's some other unadvertised 
limitation.


-
Mike Hammett
Intelligent Computing Solutions
[http://www.ics-il.com/images/fbicon.png][http://www.ics-il.com/images/googleicon.png][http://www.ics-il.com/images/linkedinicon.png][http://www.ics-il.com/images/twittericon.png]
Midwest Internet Exchange
[http://www.ics-il.com/images/fbicon.png][http://www.ics-il.com/images/linkedinicon.png][http://www.ics-il.com/images/twittericon.png]
The Brothers WISP
[http://www.ics-il.com/images/fbicon.png][http://www.ics-il.com/images/youtubeicon.png]

Re: MPLS and Carrier Ethernet, Oh My!

[Mark Tinka]

On 12/27/24 09:21, Mike Hammett wrote:

It seems like the bulk of the responses are assuming a point-to-point service. 
What about aggregation of smaller services? I don't necessarily mean 
aggregation of hundreds of sub 1 gig circuits, but when you need more than 10 
gig, your next step is needing a hundred, even if it's far away. You can only 
fit so many SLAed multi-gig services on a 10 gig port. This becomes a big deal 
when cross connects are as expensive as they are


That is logical, yes. My experience has been the backhaul provider 
either does not have a 100G NNI product, or will require your kidney for 
them to deploy one. This is, of course, before they look at what their 
network will require to support that kind of transport, if you grow.


I have also found that internal product and business development teams 
are happy to have a 100G NNI with a backhaul provider, but they assume 
the cost of a 100G port is the same as that of a 10G port. When they 
realize it's not, their business case falls apart.


So not unheard of, sure... but a few more bits that need to be ironed 
out, I've found, both internally and with your backhaul provider.


Mark.

Re: MPLS and Carrier Ethernet, Oh My!

[Mark Tinka]

On 12/27/24 05:27, Edwin Mallette wrote:

They’re out there.  I’ve helped stand up carriers recently that sell 
them and have also bought such service from other carriers.  At least 
for 100G.


I haven’t seen 400G Ethernet point to point services but also have not 
been asked to engineer to sell them either so I suspect that demand is 
either soft or the majority that want 400G point to point today are as 
Aaron says electing to serve that need via alternate solutions .


The bulk of 400G services are being ordered by the cloud and content folk.

In the telco space, 400G services are seen mostly on the line-side to 
run the optical backbone. Most telco's are still ordering 100G or N x 
100G DWDM services, especially on the subsea side where the distance 
limits the channel bandwidth.


For short metro links, we are seeing some telco's deploy 400G in IPoDWDM 
scenarios, where coherent QSFP-DD 400G plugs are cheap enough because 
they only need 1 - 8 400G channels. Some of these may even be amplified 
links if they are longer than 60km. But this is still quite rare.


Exchange points are also the other use-cases that will run 400G links 
within their short metro network.


I think there will be some 400G interest from telco's by Q4'25 for 
client-side services, and I suspect we shall start to see some content 
and cloud demand for 800G services by the start of Q2'26, if not 
slightly earlier.


Mark.

Re: MPLS and Carrier Ethernet, Oh My!

[Mike Hammett]

It seems like the bulk of the responses are assuming a point-to-point service. 
What about aggregation of smaller services? I don't necessarily mean 
aggregation of hundreds of sub 1 gig circuits, but when you need more than 10 
gig, your next step is needing a hundred, even if it's far away. You can only 
fit so many SLAed multi-gig services on a 10 gig port. This becomes a big deal 
when cross connects are as expensive as they are




-Mike HammettIntelligent Computing SolutionsMidwest Internet ExchangeThe 
Brothers WISP

- Original Message -
From: Mike Hammett 
To: NANOG 
Sent: Thu, 26 Dec 2024 20:34:50 -0600 (CST)
Subject: MPLS and Carrier Ethernet, Oh My!

A few years back, every Tom, Dick, and Harry was touting MPLS or Carrier 
Ethernet NNIs with 10G ports everywhere. They still are. However, I rarely have 
seen that graduate to 100G ports and I don't think I've seen anyone talk about 
400G ports. 


Is the hardware not there, or is it more a case of the technology hasn't been 
deployed widely, so it'll only be seen at a limited number of locations? I'm 
assuming it's more of the latter as the bigger hardware I've been looking at 
touts those features and port sizes, but maybe there's some other unadvertised 
limitation. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

Re: MPLS and Carrier Ethernet, Oh My!

[Mark Tinka]

On 12/27/24 04:34, Mike Hammett wrote:
A few years back, every Tom, Dick, and Harry was touting MPLS or 
Carrier Ethernet NNIs with 10G ports everywhere. They still are. 
However, I rarely have seen that graduate to 100G ports and I don't 
think I've seen anyone talk about 400G ports.


Is the hardware not there, or is it more a case of the technology 
hasn't been deployed widely, so it'll only be seen at a limited number 
of locations? I'm assuming it's more of the latter as the bigger 
hardware I've been looking at touts those features and port sizes, but 
maybe there's some other unadvertised limitation.


There was enough of a gap between 10G trunks and <1G customer services, 
together with the relatively low cost of 10G hardware, that made it 
sensible to deploy 10G NNI's everywhere. But because most core links are 
10G or N x 10G, selling a 10G, 20G, 30G or 40G EoMPLS service means you 
will struggle with a lot of ECMP chaos in the core, if going to 100G 
core links is nowhere near your budget cycle.


Moreover, if you want to deliver 10G or N x 10G EoMPLS services, 100G 
hardware is not as cheap now as 10G was when EoMPLS was all the rage. In 
such cases, a provider will find it easier to deliver your 10G or N x 
10G as an EoDWDM service.


There isn't much of a gap between 100G and 400G (only 4X vs. 10X for 
1G-to-10G and 10G-to-100G). So trying to deliver a, say, 100G or N x 
100G EoMPLS service means you will need to build multiple 400G links 
(either as N x 100G or N x 400G, depending on your budget), or even 
start to consider 800G hardware.


When you look at the cost of 100G, 400G and 800G routers/switches vs. 
DWDM gear, you'd be mad to try to deliver EoMPLS services of that 
magnitude on your IP/MPLS network.


My suspicion is that as operators start to get closer to building 
100G-and-beyond core links on a regular basis, they will slowly migrate 
away from EoMPLS to EoDWDM for their point-to-point circuits.


Mark.

Re: MPLS and Carrier Ethernet, Oh My!

[Mark Tinka]

On 12/27/24 05:15, Jason Bothe via NANOG wrote:

I’m thinking there is not a a wide market for them, not to mention 
last mile support. Folks that need 100 or 400 ports typically want 
point to point and at that juncture, wavelength or IPoDWDM makes sense 
as more than likely these are going between data centers and not to 
your average enterprise premises in a multi tenant building.  Just MHO.


This is correct.

It is a lot of capacity to put in the hands of a router that is sharing 
its hardware and capacity with other services that have nothing to do 
with plain transport. Also, it is a cheaper way to deliver transport 
capacity for both operator and customer.


Mark.

Re: MPLS and Carrier Ethernet, Oh My!

[Edwin Mallette]

They’re out there.  I’ve helped stand up carriers recently that sell them and 
have also bought such service from other carriers.  At least for 100G.

I haven’t seen 400G Ethernet point to point services but also have not been 
asked to engineer to sell them either so I suspect that demand is either soft 
or the majority that want 400G point to point today are as Aaron says electing 
to serve that need via alternate solutions .

Ed

From: NANOG  on behalf of 
Aaron1 
Sent: Friday, December 27, 2024 10:17:31 AM
To: Mike Hammett 
Cc: NANOG 
Subject: Re: MPLS and Carrier Ethernet, Oh My!

Since 2023, we’ve (GVTC) been migrating our CBH MTSO handoffs and partner 
ENNI’s from 1g/10g to 100g.  …moving off older ME3600’s, ASR9k’s and ACX5048’s, 
to newer MX series (204,304,240,480,960)

Aaron

On Dec 26, 2024, at 8:35 PM, Mike Hammett  wrote:


A few years back, every Tom, Dick, and Harry was touting MPLS or Carrier 
Ethernet NNIs with 10G ports everywhere. They still are. However, I rarely have 
seen that graduate to 100G ports and I don't think I've seen anyone talk about 
400G ports.

Is the hardware not there, or is it more a case of the technology hasn't been 
deployed widely, so it'll only be seen at a limited number of locations? I'm 
assuming it's more of the latter as the bigger hardware I've been looking at 
touts those features and port sizes, but maybe there's some other unadvertised 
limitation.



-
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>

Re: MPLS and Carrier Ethernet, Oh My!

[Aaron1]

Since 2023, we’ve (GVTC) been migrating our CBH MTSO handoffs and partner ENNI’s from 1g/10g to 100g.  …moving off older ME3600’s, ASR9k’s and ACX5048’s, to newer MX series (204,304,240,480,960)AaronOn Dec 26, 2024, at 8:35 PM, Mike Hammett  wrote:A few years back, every Tom, Dick, and Harry was touting MPLS or Carrier Ethernet NNIs with 10G ports everywhere. They still are. However, I rarely have seen that graduate to 100G ports and I don't think I've seen anyone talk about 400G ports.Is the hardware not there, or is it more a case of the technology hasn't been deployed widely, so it'll only be seen at a limited number of locations? I'm assuming it's more of the latter as the bigger hardware I've been looking at touts those features and port sizes, but maybe there's some other unadvertised limitation.-Mike HammettIntelligent Computing SolutionsMidwest Internet ExchangeThe Brothers WISP

Re: MPLS and Carrier Ethernet, Oh My!

[Jason Bothe via NANOG]

I’m thinking there is not a a wide market for them, not to mention last mile support. Folks that need 100 or 400 ports typically want point to point and at that juncture, wavelength or IPoDWDM makes sense as more than likely these are going between data centers and not to your average enterprise premises in a multi tenant building.  Just MHO. J~On Dec 26, 2024, at 18:35, Mike Hammett  wrote:A few years back, every Tom, Dick, and Harry was touting MPLS or Carrier Ethernet NNIs with 10G ports everywhere. They still are. However, I rarely have seen that graduate to 100G ports and I don't think I've seen anyone talk about 400G ports.Is the hardware not there, or is it more a case of the technology hasn't been deployed widely, so it'll only be seen at a limited number of locations? I'm assuming it's more of the latter as the bigger hardware I've been looking at touts those features and port sizes, but maybe there's some other unadvertised limitation.-Mike HammettIntelligent Computing SolutionsMidwest Internet ExchangeThe Brothers WISP