subject:"RE\: Routed optical networks"





On 5/11/23 17:26, Jared Mauch wrote:


And as I’ve seen, they continue to become a bit more divergent.  As someone who 
has an access network I see where the majority of my bits go, which is to the 
content folks.  There’s some to the other part, but mostly people want their 
MTV, but since it’s 2023, it’s not MTV, but people want their TikTok, 
Metaverse, Game downloads, Streaming service, Email and cloud ramps 
(enterprise).


100% - and if the trend continues, Telegeography et al will have less 
raw growth to report on unless the content folk willingly open their 
skirts up to them for a peek - which they won't do.


So yes, predicting stable or negative growth for the public Internet is 
not without merit. But that does not translate to what the content folk 
are recording.


It might make more sense to start reporting on traffic growth in the 
edge, specifically, the peering edge, as content networks continue to 
become major on-ramp/off-ramp paths for telco's. But that prediction can 
be extrapolated from submarine cable builds, submarine cable upgrades as 
well as DWDM vendor sales, with some degree of reliability.


Mark.

Re: Routed optical networks

2023-05-11 Thread Jared Mauch

> On May 11, 2023, at 11:11 AM, Mark Tinka  wrote:
> 
> 
> 
> On 5/11/23 15:50, Vasilenko Eduard via NANOG wrote:
> 
>> Hi Jared,
>> Could I make a conclusion from your comments: "only Carrier itself 
>> understand the traffic - see many examples in the text".
>> I would very agree to this.
> 
> I wouldn't say only carriers understand the traffic as much as I would say 
> carrier's traffic is more transparent, and perhaps, even more readily 
> available.
> 
> I just think that it is not relevant to try and lump network and content 
> traffic into one growth pattern. For all intents & purposes, there are two 
> Internets running side-by-side between network and content. They converge at 
> some point, but really, they are very different.

And as I’ve seen, they continue to become a bit more divergent.  As someone who 
has an access network I see where the majority of my bits go, which is to the 
content folks.  There’s some to the other part, but mostly people want their 
MTV, but since it’s 2023, it’s not MTV, but people want their TikTok, 
Metaverse, Game downloads, Streaming service, Email and cloud ramps 
(enterprise).

- jared

RE: Routed optical networks

Why you are so sure that they have access to traffic data for many access 
networks?
Their BRASes market share is far from 100%. DSLAMs finished 20 years ago.
For the cases where they support BRASes they could collect statistics. But are 
they doing it? Carriers have not given permission (not even requested).
I do not have a clue about VNI arrangement – it is magic.
PS: Everything above and below in this thread is just my personal opinion.
Eduard
From: Etienne-Victor Depasquale [mailto:ed...@ieee.org]
Sent: Thursday, May 11, 2023 6:03 PM
To: Vasilenko Eduard 
Cc: Dave Taht ; Phil Bedard ; NANOG 

Subject: Re: Routed optical networks

Eduard, you know the answer as well as I do, right :) ?

Here's my answer: I think that Cisco can only estimate (let's not say 
speculate, it has pretty bad connotations) what comes out of access networks.

No offence meant, I hope none is taken.

Cheers,

Etienne

On Thu, May 11, 2023 at 3:47 PM Vasilenko Eduard 
mailto:vasilenko.edu...@huawei.com>> wrote:
Hi Etienne,
Look carefully what you have shown to me. It is a only speculation again 
(“predictions”). It is just a table with a collection of all predictions in the 
past. Moreover, averaged between years.
I was asking for real data from the past 5 years. Are you sure that VNI has it?

If you would find real historical data in VNI, then we would be capable to 
check the table that you have shown: was the guessing right?
I strongly suspect an answer.

Eduard
From: Etienne-Victor Depasquale [mailto:ed...@ieee.org<mailto:ed...@ieee.org>]
Sent: Thursday, May 11, 2023 2:46 PM
To: Vasilenko Eduard 
mailto:vasilenko.edu...@huawei.com>>
Cc: Dave Taht mailto:dave.t...@gmail.com>>; Phil Bedard 
mailto:bedard.p...@gmail.com>>; NANOG 
mailto:nanog@nanog.org>>
Subject: Re: Routed optical networks

To clarify the table I linked to in the previous email:

Cisco estimates IP traffic exchanged over the access network by both businesses 
and consumers with:

• endpoints over managed networks and
• endpoints over unmanaged networks (“Internet traffic”).

Both the mobile access network and the fixed access network are considered.

Cisco considers IP traffic over managed networks to be characterized by passage 
through a single service provider.
Without explicitly referring to quality of service (QoS),
the implication is clearly that the traffic is controlled to meet the QoS 
demanded by the service level agreement (SLA).

In contrast, “Internet traffic” crosses provider domains;
typically, this traffic is delivered on the basis of providers’ best effort.
These two kinds of traffic complement one another and collectively are referred 
to as total global IP traffic.

Cheers,

Etienne

On Thu, May 11, 2023 at 1:37 PM Etienne-Victor Depasquale 
mailto:ed...@ieee.org>> wrote:
Historically, this is what VNI has 
claimed<https://drive.google.com/file/d/1JUG70rbZfaVHC3Z2HrECMOXJ2OnmtuxV/view?usp=sharing>.

Cheers,

Etienne

On Thu, May 11, 2023 at 1:25 PM Vasilenko Eduard 
mailto:vasilenko.edu...@huawei.com>> wrote:
I did investigate traffic for every Carrier while dealing with it as a 
consultant (repeated many dozens of times).
I have seen over a decade how traffic growth dropped year-over-year (from 60% 
to 25% in 2019 when I dropped this activity in 2020 – covid blocked travel).
Sometimes I talk to old connections and they confirm that it is even less now.
In rear cases, It is typically possible to find this information on the public 
Internet (I remember the case when Google disclosed traffic for Pakistan at the 
conference with the explanation that 30% is attributed to new subscribers, and 
an additional +30% is to more heavy content per subscriber).
But mostly, it was confidential information from a discussion with Carriers – 
they all know very well their traffic growth.
In general, traffic stat is pretty confidential. I did not have the motivation 
to aggregate it.

Sandvine is not representative of global traffic because DPI is installed 
mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic – 
it is not the biggest source. Moreover, Mobiles would look better growing 
because the limiting factor was on technology (5G proposed more than 4G, 4G 
proposed much more than 3G) – it would probably would less disruptive in the 
future.
Fixed Carriers do not pay DPI premiums. And rarely share their traffic 
publicly. Sandvine could not see it.

VNI is claiming so many things. Please show where exactly they show traffic 
growth (I am not interested in prediction speculations). Is it possible to 
understand CAGR for the 5 last years? Is it declining or growing? (traffic 
itself is for sure still growing)

Of course, the disruption could come at any year and add a new S-curve 
(Metaverse?). But disruption is by definition not predictable.

PS: Everything above and below in this thread is just my personal opinion.

Eduard
From: Etienne-Victor Depasquale [mailto:ed...@ieee.org&l

Re: Routed optical networks





On 5/11/23 15:50, Vasilenko Eduard via NANOG wrote:


Hi Jared,
Could I make a conclusion from your comments: "only Carrier itself 
understand the traffic - see many examples in the text".

I would very agree to this.


I wouldn't say only carriers understand the traffic as much as I would 
say carrier's traffic is more transparent, and perhaps, even more 
readily available.


I just think that it is not relevant to try and lump network and content 
traffic into one growth pattern. For all intents & purposes, there are 
two Internets running side-by-side between network and content. They 
converge at some point, but really, they are very different.


Mark.

Re: Routed optical networks




On 5/11/23 14:15, Jared Mauch wrote


We have seen a continued trend of the privatization of traffic and localization 
of that over time.


CDN's, submarine cables and exchange points are decentralizing the "core 
of the Internet", and relegating IP Transit providers to whom CDN's 
subscribe as sources of origin content, if they are unable to build 
their own backbones.


Only in markets where CDN's are not as rife do you still see huge growth 
and stable pricing for IP Transit traffic.




   I’ve watched all the big carriers retreat from their global network reaches 
to be more of regionalized networks.  A decade ago you would have seen European 
national incumbents peering and with market in Asia, and the complete global 
networks continue to shrink.


Yeah - unless you serve a very tight niche of Enterprise customers that 
require stable (not necessarily fast) connectivity to some part of the 
word, it makes little sense, nowadays, for the large telco's of old to 
go poking in other markets that aren't theirs, something the top global 
exchange points are going to learn the hard way.




Meanwhile you have a mix of the content and cloud providers continue to build 
their business-purpose networks expanding into markets that the uppercase 
Internet may not need to reach.


At the moment, the only motivation I see for large telco's trying to 
enter markets they can't be strong in is their "global brand". Often 
times, they can't do very well in that market because a) there is 
sufficient local content and peering, b) they can't build a network as 
robust as what the local telco's can, and c) all they have to offer is 
access to traffic several milliseconds away that is not any faster than 
what the local telco's can, which forms less than 20% of what the local 
customers chase after anyway.


Such telco's usually, then, turn to targeting global Enterprise 
customers with "worldwide service contracts" accompanied by marked-up 
pricing for "private global branch connectivity". It always sounds and 
looks good on PowerPoint slides, but 12 months after the big launch, a 
broken champagne bottle and some photos, the PoP is gathering quite a 
bit of dust in the data centre.




You can look at the proposals in the EU about fees, and I have dual thoughts on 
this which are MY OWN and don’t represent my employer or otherwise, but if you 
read this post from Petra Arts 
-https://blog.cloudflare.com/eu-network-usage-fees/  - it speaks around major 
interconnection points like Frankfurt, which are important but double as 
problematic.  The number of people that need to go to the near market (eg: 
Chicago, while I’m in Detroit area) for good connectivity is an issue, 
meanwhile there’s a robust need to keep traffic within the state of Michigan 
and a halfway decent ecosystem for that via Detroit IX - (disclaimer, I’m on 
the board).  There need to be some aggregation points, so not everyone needs to 
be in Detroit, but also not everyone needs to be in Frankfurt - and content 
localization needs to continue to happen, but is also very regionalized in 
popularity.


Us telco's didn't learn our lesson 23 years ago when the content folk 
tried to do a deal with us. This will be deja vu, although this time, 
with some help from governments that we rather won't like.


Telco's continue to strong-arm content because we own the customer. But, 
it's 2023... I'm not sure we want to see a battle of the customer 
between content and network. It might not end well. We already see 
glimpses of it with QUIC, so...




How to do this all and not have it all route via Chicago or Frankfurt is a 
challenge, but also not everyone will be in Berlin, Munich or these other 
markets.  This is where having a robust optical network capability (or 
backbone) can come into play, that you can deliver deeper from those hub 
points, but at the same time, I’ve been in meetings where companies have their 
own challenges accepting that content in those downstream locations as their 
network was also built to get to/from the major hub cities, or IP space wasn’t 
allocated in a way that can provide consistent routing results or behaviors.  
(This is where IPv6 can be super helpful, it gives the chance to possibly 
Greenfield, aka not screw it up - at least initially).


As content continues to grow, PoP's built in far-flung locations away 
from a telco's core area of business will only remain as relevant as 
traffic that has not yet migrated to a syndicated cloud. As that tapers 
off- which it will - there will be more pressure to either shut those 
PoP's down, or compete for local IP Transit. The latter is much less likely.




There’s huge volumes of IP traffic exchanged, but the largest volumes are being 
moved over private interconnects or a localized IX to those eyeball networks 
with the historical global backbones playing more of the long-distance carrier 
role, which is critical as you want a path to deliver those bits, without it

Re: Routed optical networks





On 5/11/23 15:50, Vasilenko Eduard via NANOG wrote:


Hi Jared,
Could I make a conclusion from your comments: "only Carrier itself understand the 
traffic - see many examples in the text".
I would very agree to this.


I wouldn't say only carriers understand the traffic as much as I would 
say carrier's traffic is more transparent, and perhaps, even more 
readily available.


I just think that it is not relevant to try and lump network and content 
traffic into one growth pattern. For all intents & purposes, there are 
two Internets running side-by-side between network and content. They 
converge at some point, but really, they are very different.


Mark.

Re: Routed optical networks

2023-05-11 Thread Etienne-Victor Depasquale via NANOG

Eduard, you know the answer as well as I do, right :) ?

Here's my answer: I think that Cisco can only estimate (let's not say
speculate, it has pretty bad connotations) what comes out of access
networks.

No offence meant, I hope none is taken.

Cheers,

Etienne

On Thu, May 11, 2023 at 3:47 PM Vasilenko Eduard <
vasilenko.edu...@huawei.com> wrote:

> Hi Etienne,
>
> Look carefully what you have shown to me. It is a only speculation again
> (“predictions”). It is just a table with a collection of all predictions in
> the past. Moreover, averaged between years.
>
> I was asking for real data from the past 5 years. Are you sure that VNI
> has it?
>
>
>
> If you would find real historical data in VNI, then we would be capable to
> check the table that you have shown: was the guessing right?
>
> I strongly suspect an answer.
>
>
>
> Eduard
>
> *From:* Etienne-Victor Depasquale [mailto:ed...@ieee.org]
> *Sent:* Thursday, May 11, 2023 2:46 PM
> *To:* Vasilenko Eduard 
> *Cc:* Dave Taht ; Phil Bedard ;
> NANOG 
> *Subject:* Re: Routed optical networks
>
>
>
> To clarify the table I linked to in the previous email:
>
>
>
> Cisco estimates IP traffic exchanged over the access network by both
> businesses and consumers with:
>
>
> • endpoints over managed networks and
> • endpoints over unmanaged networks (“Internet traffic”).
>
>
> Both the mobile access network and the fixed access network are
> considered.
>
>
>
> Cisco considers IP traffic over managed networks to be characterized by
> passage through a single service provider.
>
> Without explicitly referring to quality of service (QoS),
>
> the implication is clearly that the traffic is controlled to meet the QoS
> demanded by the service level agreement (SLA).
>
>
>
> In contrast, “Internet traffic” crosses provider domains;
>
> typically, this traffic is delivered on the basis of providers’ best
> effort.
>
> These two kinds of traffic complement one another and collectively are
> referred to as total global IP traffic.
>
>
>
> Cheers,
>
>
>
> Etienne
>
>
>
> On Thu, May 11, 2023 at 1:37 PM Etienne-Victor Depasquale 
> wrote:
>
> Historically, this is what VNI has claimed
> <https://drive.google.com/file/d/1JUG70rbZfaVHC3Z2HrECMOXJ2OnmtuxV/view?usp=sharing>
> .
>
>
>
> Cheers,
>
>
> Etienne
>
>
>
> On Thu, May 11, 2023 at 1:25 PM Vasilenko Eduard <
> vasilenko.edu...@huawei.com> wrote:
>
> I did investigate traffic for every Carrier while dealing with it as a
> consultant (repeated many dozens of times).
>
> I have seen over a decade how traffic growth dropped year-over-year (from
> 60% to 25% in 2019 when I dropped this activity in 2020 – covid blocked
> travel).
>
> Sometimes I talk to old connections and they confirm that it is even less
> now.
>
> In rear cases, It is typically possible to find this information on the
> public Internet (I remember the case when Google disclosed traffic for
> Pakistan at the conference with the explanation that 30% is attributed to
> new subscribers, and an additional +30% is to more heavy content per
> subscriber).
>
> But mostly, it was confidential information from a discussion with
> Carriers – they all know very well their traffic growth.
>
> In general, traffic stat is pretty confidential. I did not have the
> motivation to aggregate it.
>
>
>
> Sandvine is not representative of global traffic because DPI is installed
> mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic
> – it is not the biggest source. Moreover, Mobiles would look better growing
> because the limiting factor was on technology (5G proposed more than 4G, 4G
> proposed much more than 3G) – it would probably would less disruptive in
> the future.
>
> Fixed Carriers do not pay DPI premiums. And rarely share their traffic
> publicly. Sandvine could not see it.
>
>
>
> VNI is claiming so many things. Please show where exactly they show
> traffic growth (I am not interested in prediction speculations). Is it
> possible to understand CAGR for the 5 last years? Is it declining or
> growing? (traffic itself is for sure still growing)
>
>
>
> Of course, the disruption could come at any year and add a new S-curve
> (Metaverse?). But disruption is by definition not predictable.
>
>
>
> PS: Everything above and below in this thread is just my personal opinion.
>
>
>
> Eduard
>
> *From:* Etienne-Victor Depasquale [mailto:ed...@ieee.org]
> *Sent:* Thursday, May 11, 2023 12:48 PM
> *To:* Vasilenko Eduard 
> *Cc:* Dave Taht ; Phil Bedard ;
> NANOG 
> *Subject:* Re: Routed

Re: Routed optical networks




On 5/11/23 13:25, Vasilenko Eduard via NANOG wrote:

Sandvine is not representative of global traffic because DPI is 
installed mostly for Mobiles. But Mobile subscriber is 10x less than 
fixed on traffic – it is not the biggest source. Moreover, Mobiles 
would look better growing because the limiting factor was on 
technology (5G proposed more than 4G, 4G proposed much more than 3G) – 
it would probably would less disruptive in the future.


Fixed Carriers do not pay DPI premiums. And rarely share their traffic 
publicly. Sandvine could not see it.




And QUIC is making the the DPI model so loved by MNO's quickly irrelevant.

Mark.

Re: Routed optical networks




On 5/11/23 09:33, Vasilenko Eduard via NANOG wrote:


But it is speculation, not a trend yet.

I remember 10y ago every presentation started from the claim that 100B 
of IoT would drive XXX traffic. It did not happen.


Now we see presentations that AI would be talking to AI that generates 
 traffic.


Maybe some technology would push traffic next S-curve, maybe not. It 
is still speculation.


The traffic growth was stimulated (despite all VNIs) by 1) new 
subscribers, 2) video quality for subscribers. Nothing else yet.


It is almost finished for both trends. We are close to the plateau of 
these S-curves.


For some years (2013-2020) I was carefully looking at numbers for many 
countries: it was always possible to split CAGR for these 2 
components. The video part was extremely consistent between countries. 
The subscriber part was 100% proportional to subscriber CAGR.


Everything else up to now was “marketing” to say it mildly.

Reminder: nothing in nature could grow indefinitely. The limit always 
exists. It is only a question of when.


PS: Of course, marketing people could draw you any traffic growth – it 
depends just on the marketing budget.




I think the place where we are seeing continued growth is in the content 
provider private network, for that DC-to-DC traffic, even across continents.


For everything else, as you say, it's nice to predict. But what analysts 
and marketing departments say and what end users actually do are often 
vastly different.


As long as the majority of the Internet's traffic is "invisible", we 
can't really ever know. We can only tell by how much kit the DWDM 
vendors are selling to non-telco customers (content folk), as well as 
how many submarine cables are being built by non-telco consortia 
(content folk).


Mark.

Re: Routed optical networks





On 5/11/23 13:45, Etienne-Victor Depasquale via NANOG wrote:


To clarify the table I linked to in the previous email:

Cisco estimates IP traffic exchanged over the access network by both 
businesses and consumers with:


• endpoints over managed networks and
• endpoints over unmanaged networks (“Internet traffic”).

Both the mobile access network and the fixed access network are 
considered.


Cisco considers IP traffic over managed networks to be characterized 
by passage through a single service provider.

Without explicitly referring to quality of service (QoS),
the implication is clearly that the traffic is controlled to meet the 
QoS demanded by the service level agreement (SLA).


So either Cisco think/though that the only operators worth 
surveying/predicting were the large ones (NTT, Telia, Tata, Lumen, 
Cogent, e.t.c.), or that on-net MPLS/VPN traffic was more significant 
than public IP Transit both in terms of revenue and strategic direction 
of the operators they surveyed/predicated.


Either way, I'd imagine any results based on those data points would be 
incomplete, at least from a real-world standpoint.




In contrast, “Internet traffic” crosses provider domains;
typically, this traffic is delivered on the basis of providers’ best 
effort.
These two kinds of traffic complement one another and collectively are 
referred to as total global IP traffic.


In the IP space, this traffic type is quickly exceeding any historically 
significant MPLS/VPN traffic, if it hasn't already.


Mark.

RE: Routed optical networks

Hi Jared,
Could I make a conclusion from your comments: "only Carrier itself understand 
the traffic - see many examples in the text".
I would very agree to this.
Eduard
-Original Message-
From: Jared Mauch [mailto:ja...@puck.nether.net] 
Sent: Thursday, May 11, 2023 3:16 PM
To: Etienne-Victor Depasquale 
Cc: Vasilenko Eduard ; NANOG 
Subject: Re: Routed optical networks

> On May 11, 2023, at 7:45 AM, Etienne-Victor Depasquale via NANOG 
>  wrote:
> 
> To clarify the table I linked to in the previous email:
> 
> Cisco estimates IP traffic exchanged over the access network by both 
> businesses and consumers with:
> 
> • endpoints over managed networks and • endpoints over unmanaged 
> networks (“Internet traffic”).
> 
> Both the mobile access network and the fixed access network are considered. 
> 
> Cisco considers IP traffic over managed networks to be characterized by 
> passage through a single service provider. 
> Without explicitly referring to quality of service (QoS), the 
> implication is clearly that the traffic is controlled to meet the QoS 
> demanded by the service level agreement (SLA).
> 
> In contrast, “Internet traffic” crosses provider domains; typically, 
> this traffic is delivered on the basis of providers’ best effort.
> These two kinds of traffic complement one another and collectively are 
> referred to as total global IP traffic.

I think there’s a lot of problems here.  While places like my employer will 
periodically disclose our traffic numbers, and DDoS providers, mitigation 
platforms and otherwise will disclose the peaks they see, much of this data is 
a bit opaque, and tools like AI that do in-metro or cross-metro 
datacenter-datacenter remote DMA type activities, those all count differently.

We have seen a continued trend of the privatization of traffic and localization 
of that over time.  I’ve watched all the big carriers retreat from their global 
network reaches to be more of regionalized networks.  A decade ago you would 
have seen European national incumbents peering and with market in Asia, and the 
complete global networks continue to shrink.

Meanwhile you have a mix of the content and cloud providers continue to build 
their business-purpose networks expanding into markets that the uppercase 
Internet may not need to reach.

You can look at the proposals in the EU about fees, and I have dual thoughts on 
this which are MY OWN and don’t represent my employer or otherwise, but if you 
read this post from Petra Arts - 
https://blog.cloudflare.com/eu-network-usage-fees/ - it speaks around major 
interconnection points like Frankfurt, which are important but double as 
problematic.  The number of people that need to go to the near market (eg: 
Chicago, while I’m in Detroit area) for good connectivity is an issue, 
meanwhile there’s a robust need to keep traffic within the state of Michigan 
and a halfway decent ecosystem for that via Detroit IX - (disclaimer, I’m on 
the board).  There need to be some aggregation points, so not everyone needs to 
be in Detroit, but also not everyone needs to be in Frankfurt - and content 
localization needs to continue to happen, but is also very regionalized in 
popularity.

How to do this all and not have it all route via Chicago or Frankfurt is a 
challenge, but also not everyone will be in Berlin, Munich or these other 
markets.  This is where having a robust optical network capability (or 
backbone) can come into play, that you can deliver deeper from those hub 
points, but at the same time, I’ve been in meetings where companies have their 
own challenges accepting that content in those downstream locations as their 
network was also built to get to/from the major hub cities, or IP space wasn’t 
allocated in a way that can provide consistent routing results or behaviors.  
(This is where IPv6 can be super helpful, it gives the chance to possibly 
Greenfield, aka not screw it up - at least initially).

There’s huge volumes of IP traffic exchanged, but the largest volumes are being 
moved over private interconnects or a localized IX to those eyeball networks 
with the historical global backbones playing more of the long-distance carrier 
role, which is critical as you want a path to deliver those bits, without it 
following the ITU-style sender pays model, as the majority of IP traffic is 
actually requested by the customer of the end-user network.  (All of it if you 
remove network scans, ddos, web bots/crawlers).

Most networks have no SLA once things cross an unpaid boundary (SFI, or even 
private peering) - and if they are a customer and that path is congested, it’s 
up to the customer to upgrade that path.

- Jared (many hats)

RE: Routed optical networks

Hi Etienne,
Look carefully what you have shown to me. It is a only speculation again 
(“predictions”). It is just a table with a collection of all predictions in the 
past. Moreover, averaged between years.
I was asking for real data from the past 5 years. Are you sure that VNI has it?

If you would find real historical data in VNI, then we would be capable to 
check the table that you have shown: was the guessing right?
I strongly suspect an answer.

Eduard
From: Etienne-Victor Depasquale [mailto:ed...@ieee.org]
Sent: Thursday, May 11, 2023 2:46 PM
To: Vasilenko Eduard 
Cc: Dave Taht ; Phil Bedard ; NANOG 

Subject: Re: Routed optical networks

To clarify the table I linked to in the previous email:

Cisco estimates IP traffic exchanged over the access network by both businesses 
and consumers with:

• endpoints over managed networks and
• endpoints over unmanaged networks (“Internet traffic”).

Both the mobile access network and the fixed access network are considered.

Cisco considers IP traffic over managed networks to be characterized by passage 
through a single service provider.
Without explicitly referring to quality of service (QoS),
the implication is clearly that the traffic is controlled to meet the QoS 
demanded by the service level agreement (SLA).

In contrast, “Internet traffic” crosses provider domains;
typically, this traffic is delivered on the basis of providers’ best effort.
These two kinds of traffic complement one another and collectively are referred 
to as total global IP traffic.

Cheers,

Etienne

On Thu, May 11, 2023 at 1:37 PM Etienne-Victor Depasquale 
mailto:ed...@ieee.org>> wrote:
Historically, this is what VNI has 
claimed<https://drive.google.com/file/d/1JUG70rbZfaVHC3Z2HrECMOXJ2OnmtuxV/view?usp=sharing>.

Cheers,

Etienne

On Thu, May 11, 2023 at 1:25 PM Vasilenko Eduard 
mailto:vasilenko.edu...@huawei.com>> wrote:
I did investigate traffic for every Carrier while dealing with it as a 
consultant (repeated many dozens of times).
I have seen over a decade how traffic growth dropped year-over-year (from 60% 
to 25% in 2019 when I dropped this activity in 2020 – covid blocked travel).
Sometimes I talk to old connections and they confirm that it is even less now.
In rear cases, It is typically possible to find this information on the public 
Internet (I remember the case when Google disclosed traffic for Pakistan at the 
conference with the explanation that 30% is attributed to new subscribers, and 
an additional +30% is to more heavy content per subscriber).
But mostly, it was confidential information from a discussion with Carriers – 
they all know very well their traffic growth.
In general, traffic stat is pretty confidential. I did not have the motivation 
to aggregate it.

Sandvine is not representative of global traffic because DPI is installed 
mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic – 
it is not the biggest source. Moreover, Mobiles would look better growing 
because the limiting factor was on technology (5G proposed more than 4G, 4G 
proposed much more than 3G) – it would probably would less disruptive in the 
future.
Fixed Carriers do not pay DPI premiums. And rarely share their traffic 
publicly. Sandvine could not see it.

VNI is claiming so many things. Please show where exactly they show traffic 
growth (I am not interested in prediction speculations). Is it possible to 
understand CAGR for the 5 last years? Is it declining or growing? (traffic 
itself is for sure still growing)

Of course, the disruption could come at any year and add a new S-curve 
(Metaverse?). But disruption is by definition not predictable.

PS: Everything above and below in this thread is just my personal opinion.

Eduard
From: Etienne-Victor Depasquale [mailto:ed...@ieee.org<mailto:ed...@ieee.org>]
Sent: Thursday, May 11, 2023 12:48 PM
To: Vasilenko Eduard 
mailto:vasilenko.edu...@huawei.com>>
Cc: Dave Taht mailto:dave.t...@gmail.com>>; Phil Bedard 
mailto:bedard.p...@gmail.com>>; NANOG 
mailto:nanog@nanog.org>>
Subject: Re: Routed optical networks

Eduard, academics cite the VNI (and the Sandvine Global reports).

Do you know of alternative sources that show traffic growth data you're more 
comfortable with?

Cheers,

Etienne

On Thu, May 11, 2023 at 9:34 AM Vasilenko Eduard 
mailto:vasilenko.edu...@huawei.com>> wrote:
But it is speculation, not a trend yet.
I remember 10y ago every presentation started from the claim that 100B of IoT 
would drive XXX traffic. It did not happen.
Now we see presentations that AI would be talking to AI that generates  
traffic.
Maybe some technology would push traffic next S-curve, maybe not. It is still 
speculation.

The traffic growth was stimulated (despite all VNIs) by 1) new subscribers, 2) 
video quality for subscribers. Nothing else yet.
It is almost finished for both trends. We are close to the plateau of these 
S-curves.
For some years (2013-2020)

Re: Routed optical networks

2023-05-11 Thread Jared Mauch

> On May 11, 2023, at 7:45 AM, Etienne-Victor Depasquale via NANOG 
>  wrote:
> 
> To clarify the table I linked to in the previous email:
> 
> Cisco estimates IP traffic exchanged over the access network by both 
> businesses and consumers with:
> 
> • endpoints over managed networks and 
> • endpoints over unmanaged networks (“Internet traffic”).
> 
> Both the mobile access network and the fixed access network are considered. 
> 
> Cisco considers IP traffic over managed networks to be characterized by 
> passage through a single service provider. 
> Without explicitly referring to quality of service (QoS), 
> the implication is clearly that the traffic is controlled to meet the QoS 
> demanded by the service level agreement (SLA). 
> 
> In contrast, “Internet traffic” crosses provider domains; 
> typically, this traffic is delivered on the basis of providers’ best effort. 
> These two kinds of traffic complement one another and collectively are 
> referred to as total global IP traffic.

I think there’s a lot of problems here.  While places like my employer will 
periodically disclose our traffic numbers, and DDoS providers, mitigation 
platforms and otherwise will disclose the peaks they see, much of this data is 
a bit opaque, and tools like AI that do in-metro or cross-metro 
datacenter-datacenter remote DMA type activities, those all count differently.

We have seen a continued trend of the privatization of traffic and localization 
of that over time.  I’ve watched all the big carriers retreat from their global 
network reaches to be more of regionalized networks.  A decade ago you would 
have seen European national incumbents peering and with market in Asia, and the 
complete global networks continue to shrink.

Meanwhile you have a mix of the content and cloud providers continue to build 
their business-purpose networks expanding into markets that the uppercase 
Internet may not need to reach.

You can look at the proposals in the EU about fees, and I have dual thoughts on 
this which are MY OWN and don’t represent my employer or otherwise, but if you 
read this post from Petra Arts - 
https://blog.cloudflare.com/eu-network-usage-fees/ - it speaks around major 
interconnection points like Frankfurt, which are important but double as 
problematic.  The number of people that need to go to the near market (eg: 
Chicago, while I’m in Detroit area) for good connectivity is an issue, 
meanwhile there’s a robust need to keep traffic within the state of Michigan 
and a halfway decent ecosystem for that via Detroit IX - (disclaimer, I’m on 
the board).  There need to be some aggregation points, so not everyone needs to 
be in Detroit, but also not everyone needs to be in Frankfurt - and content 
localization needs to continue to happen, but is also very regionalized in 
popularity.

How to do this all and not have it all route via Chicago or Frankfurt is a 
challenge, but also not everyone will be in Berlin, Munich or these other 
markets.  This is where having a robust optical network capability (or 
backbone) can come into play, that you can deliver deeper from those hub 
points, but at the same time, I’ve been in meetings where companies have their 
own challenges accepting that content in those downstream locations as their 
network was also built to get to/from the major hub cities, or IP space wasn’t 
allocated in a way that can provide consistent routing results or behaviors.  
(This is where IPv6 can be super helpful, it gives the chance to possibly 
Greenfield, aka not screw it up - at least initially).

There’s huge volumes of IP traffic exchanged, but the largest volumes are being 
moved over private interconnects or a localized IX to those eyeball networks 
with the historical global backbones playing more of the long-distance carrier 
role, which is critical as you want a path to deliver those bits, without it 
following the ITU-style sender pays model, as the majority of IP traffic is 
actually requested by the customer of the end-user network.  (All of it if you 
remove network scans, ddos, web bots/crawlers).

Most networks have no SLA once things cross an unpaid boundary (SFI, or even 
private peering) - and if they are a customer and that path is congested, it’s 
up to the customer to upgrade that path.

- Jared (many hats)

Re: Routed optical networks

2023-05-11 Thread Etienne-Victor Depasquale via NANOG

To clarify the table I linked to in the previous email:

Cisco estimates IP traffic exchanged over the access network by both
businesses and consumers with:

• endpoints over managed networks and
• endpoints over unmanaged networks (“Internet traffic”).

Both the mobile access network and the fixed access network are considered.

Cisco considers IP traffic over managed networks to be characterized by
passage through a single service provider.
Without explicitly referring to quality of service (QoS),
the implication is clearly that the traffic is controlled to meet the QoS
demanded by the service level agreement (SLA).

In contrast, “Internet traffic” crosses provider domains;
typically, this traffic is delivered on the basis of providers’ best
effort.
These two kinds of traffic complement one another and collectively are
referred to as total global IP traffic.

Cheers,

Etienne

On Thu, May 11, 2023 at 1:37 PM Etienne-Victor Depasquale 
wrote:

> Historically, this is what VNI has claimed
> <https://drive.google.com/file/d/1JUG70rbZfaVHC3Z2HrECMOXJ2OnmtuxV/view?usp=sharing>
> .
>
> Cheers,
>
> Etienne
>
> On Thu, May 11, 2023 at 1:25 PM Vasilenko Eduard <
> vasilenko.edu...@huawei.com> wrote:
>
>> I did investigate traffic for every Carrier while dealing with it as a
>> consultant (repeated many dozens of times).
>>
>> I have seen over a decade how traffic growth dropped year-over-year (from
>> 60% to 25% in 2019 when I dropped this activity in 2020 – covid blocked
>> travel).
>>
>> Sometimes I talk to old connections and they confirm that it is even less
>> now.
>>
>> In rear cases, It is typically possible to find this information on the
>> public Internet (I remember the case when Google disclosed traffic for
>> Pakistan at the conference with the explanation that 30% is attributed to
>> new subscribers, and an additional +30% is to more heavy content per
>> subscriber).
>>
>> But mostly, it was confidential information from a discussion with
>> Carriers – they all know very well their traffic growth.
>>
>> In general, traffic stat is pretty confidential. I did not have the
>> motivation to aggregate it.
>>
>>
>>
>> Sandvine is not representative of global traffic because DPI is installed
>> mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic
>> – it is not the biggest source. Moreover, Mobiles would look better growing
>> because the limiting factor was on technology (5G proposed more than 4G, 4G
>> proposed much more than 3G) – it would probably would less disruptive in
>> the future.
>>
>> Fixed Carriers do not pay DPI premiums. And rarely share their traffic
>> publicly. Sandvine could not see it.
>>
>>
>>
>> VNI is claiming so many things. Please show where exactly they show
>> traffic growth (I am not interested in prediction speculations). Is it
>> possible to understand CAGR for the 5 last years? Is it declining or
>> growing? (traffic itself is for sure still growing)
>>
>>
>>
>> Of course, the disruption could come at any year and add a new S-curve
>> (Metaverse?). But disruption is by definition not predictable.
>>
>>
>>
>> PS: Everything above and below in this thread is just my personal opinion.
>>
>>
>>
>> Eduard
>>
>> *From:* Etienne-Victor Depasquale [mailto:ed...@ieee.org]
>> *Sent:* Thursday, May 11, 2023 12:48 PM
>> *To:* Vasilenko Eduard 
>> *Cc:* Dave Taht ; Phil Bedard ;
>> NANOG 
>> *Subject:* Re: Routed optical networks
>>
>>
>>
>> Eduard, academics cite the VNI (and the Sandvine Global reports).
>>
>>
>>
>> Do you know of alternative sources that show traffic growth data you're
>> more comfortable with?
>>
>>
>>
>> Cheers,
>>
>>
>>
>> Etienne
>>
>>
>>
>> On Thu, May 11, 2023 at 9:34 AM Vasilenko Eduard <
>> vasilenko.edu...@huawei.com> wrote:
>>
>> But it is speculation, not a trend yet.
>>
>> I remember 10y ago every presentation started from the claim that 100B of
>> IoT would drive XXX traffic. It did not happen.
>>
>> Now we see presentations that AI would be talking to AI that generates
>>  traffic.
>>
>> Maybe some technology would push traffic next S-curve, maybe not. It is
>> still speculation.
>>
>>
>>
>> The traffic growth was stimulated (despite all VNIs) by 1) new
>> subscribers, 2) video quality for subscribers. Nothing else yet.
>>
>> It is almost finished for both trends. We are close to the pl

Re: Routed optical networks

2023-05-11 Thread Etienne-Victor Depasquale via NANOG

Historically, this is what VNI has claimed
<https://drive.google.com/file/d/1JUG70rbZfaVHC3Z2HrECMOXJ2OnmtuxV/view?usp=sharing>
.

Cheers,

Etienne

On Thu, May 11, 2023 at 1:25 PM Vasilenko Eduard <
vasilenko.edu...@huawei.com> wrote:

> I did investigate traffic for every Carrier while dealing with it as a
> consultant (repeated many dozens of times).
>
> I have seen over a decade how traffic growth dropped year-over-year (from
> 60% to 25% in 2019 when I dropped this activity in 2020 – covid blocked
> travel).
>
> Sometimes I talk to old connections and they confirm that it is even less
> now.
>
> In rear cases, It is typically possible to find this information on the
> public Internet (I remember the case when Google disclosed traffic for
> Pakistan at the conference with the explanation that 30% is attributed to
> new subscribers, and an additional +30% is to more heavy content per
> subscriber).
>
> But mostly, it was confidential information from a discussion with
> Carriers – they all know very well their traffic growth.
>
> In general, traffic stat is pretty confidential. I did not have the
> motivation to aggregate it.
>
>
>
> Sandvine is not representative of global traffic because DPI is installed
> mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic
> – it is not the biggest source. Moreover, Mobiles would look better growing
> because the limiting factor was on technology (5G proposed more than 4G, 4G
> proposed much more than 3G) – it would probably would less disruptive in
> the future.
>
> Fixed Carriers do not pay DPI premiums. And rarely share their traffic
> publicly. Sandvine could not see it.
>
>
>
> VNI is claiming so many things. Please show where exactly they show
> traffic growth (I am not interested in prediction speculations). Is it
> possible to understand CAGR for the 5 last years? Is it declining or
> growing? (traffic itself is for sure still growing)
>
>
>
> Of course, the disruption could come at any year and add a new S-curve
> (Metaverse?). But disruption is by definition not predictable.
>
>
>
> PS: Everything above and below in this thread is just my personal opinion.
>
>
>
> Eduard
>
> *From:* Etienne-Victor Depasquale [mailto:ed...@ieee.org]
> *Sent:* Thursday, May 11, 2023 12:48 PM
> *To:* Vasilenko Eduard 
> *Cc:* Dave Taht ; Phil Bedard ;
> NANOG 
> *Subject:* Re: Routed optical networks
>
>
>
> Eduard, academics cite the VNI (and the Sandvine Global reports).
>
>
>
> Do you know of alternative sources that show traffic growth data you're
> more comfortable with?
>
>
>
> Cheers,
>
>
>
> Etienne
>
>
>
> On Thu, May 11, 2023 at 9:34 AM Vasilenko Eduard <
> vasilenko.edu...@huawei.com> wrote:
>
> But it is speculation, not a trend yet.
>
> I remember 10y ago every presentation started from the claim that 100B of
> IoT would drive XXX traffic. It did not happen.
>
> Now we see presentations that AI would be talking to AI that generates
>  traffic.
>
> Maybe some technology would push traffic next S-curve, maybe not. It is
> still speculation.
>
>
>
> The traffic growth was stimulated (despite all VNIs) by 1) new
> subscribers, 2) video quality for subscribers. Nothing else yet.
>
> It is almost finished for both trends. We are close to the plateau of
> these S-curves.
>
> For some years (2013-2020) I was carefully looking at numbers for many
> countries: it was always possible to split CAGR for these 2 components. The
> video part was extremely consistent between countries. The subscriber part
> was 100% proportional to subscriber CAGR.
>
> Everything else up to now was “marketing” to say it mildly.
>
>
>
> Reminder: nothing in nature could grow indefinitely. The limit always
> exists. It is only a question of when.
>
>
>
> PS: Of course, marketing people could draw you any traffic growth – it
> depends just on the marketing budget.
>
>
>
> Eduard
>
> *From:* Dave Taht [mailto:dave.t...@gmail.com]
> *Sent:* Tuesday, May 9, 2023 11:41 PM
> *To:* Vasilenko Eduard 
> *Cc:* Phil Bedard ; Etienne-Victor Depasquale <
> ed...@ieee.org>; NANOG 
> *Subject:* Re: Routed optical networks
>
>
>
> Up until this moment I was feeling that my take on the decline of traffic
> growth was somewhat isolated, in that I have long felt that we are nearing
> the top of the S curve of the data we humans can create and consume. About
> the only source of future traffic growth I can think of comes from getting
> more humans online, and that is a mere another doubling.
>
>
>
> On the other hand, predictions such as 640k should

RE: Routed optical networks

I did investigate traffic for every Carrier while dealing with it as a 
consultant (repeated many dozens of times).
I have seen over a decade how traffic growth dropped year-over-year (from 60% 
to 25% in 2019 when I dropped this activity in 2020 – covid blocked travel).
Sometimes I talk to old connections and they confirm that it is even less now.
In rear cases, It is typically possible to find this information on the public 
Internet (I remember the case when Google disclosed traffic for Pakistan at the 
conference with the explanation that 30% is attributed to new subscribers, and 
an additional +30% is to more heavy content per subscriber).
But mostly, it was confidential information from a discussion with Carriers – 
they all know very well their traffic growth.
In general, traffic stat is pretty confidential. I did not have the motivation 
to aggregate it.

Sandvine is not representative of global traffic because DPI is installed 
mostly for Mobiles. But Mobile subscriber is 10x less than fixed on traffic – 
it is not the biggest source. Moreover, Mobiles would look better growing 
because the limiting factor was on technology (5G proposed more than 4G, 4G 
proposed much more than 3G) – it would probably would less disruptive in the 
future.
Fixed Carriers do not pay DPI premiums. And rarely share their traffic 
publicly. Sandvine could not see it.

VNI is claiming so many things. Please show where exactly they show traffic 
growth (I am not interested in prediction speculations). Is it possible to 
understand CAGR for the 5 last years? Is it declining or growing? (traffic 
itself is for sure still growing)

Of course, the disruption could come at any year and add a new S-curve 
(Metaverse?). But disruption is by definition not predictable.

PS: Everything above and below in this thread is just my personal opinion.

Eduard
From: Etienne-Victor Depasquale [mailto:ed...@ieee.org]
Sent: Thursday, May 11, 2023 12:48 PM
To: Vasilenko Eduard 
Cc: Dave Taht ; Phil Bedard ; NANOG 

Subject: Re: Routed optical networks

Eduard, academics cite the VNI (and the Sandvine Global reports).

Do you know of alternative sources that show traffic growth data you're more 
comfortable with?

Cheers,

Etienne

On Thu, May 11, 2023 at 9:34 AM Vasilenko Eduard 
mailto:vasilenko.edu...@huawei.com>> wrote:
But it is speculation, not a trend yet.
I remember 10y ago every presentation started from the claim that 100B of IoT 
would drive XXX traffic. It did not happen.
Now we see presentations that AI would be talking to AI that generates  
traffic.
Maybe some technology would push traffic next S-curve, maybe not. It is still 
speculation.

The traffic growth was stimulated (despite all VNIs) by 1) new subscribers, 2) 
video quality for subscribers. Nothing else yet.
It is almost finished for both trends. We are close to the plateau of these 
S-curves.
For some years (2013-2020) I was carefully looking at numbers for many 
countries: it was always possible to split CAGR for these 2 components. The 
video part was extremely consistent between countries. The subscriber part was 
100% proportional to subscriber CAGR.
Everything else up to now was “marketing” to say it mildly.

Reminder: nothing in nature could grow indefinitely. The limit always exists. 
It is only a question of when.

PS: Of course, marketing people could draw you any traffic growth – it depends 
just on the marketing budget.

Eduard
From: Dave Taht [mailto:dave.t...@gmail.com<mailto:dave.t...@gmail.com>]
Sent: Tuesday, May 9, 2023 11:41 PM
To: Vasilenko Eduard 
mailto:vasilenko.edu...@huawei.com>>
Cc: Phil Bedard mailto:bedard.p...@gmail.com>>; 
Etienne-Victor Depasquale mailto:ed...@ieee.org>>; NANOG 
mailto:nanog@nanog.org>>
Subject: Re: Routed optical networks

Up until this moment I was feeling that my take on the decline of traffic 
growth was somewhat isolated, in that I have long felt that we are nearing the 
top of the S curve of the data we humans can create and consume. About the only 
source of future traffic growth I can think of comes from getting more humans 
online, and that is a mere another doubling.

On the other hand, predictions such as 640k should be enough for everyone did 
not pan out.

On the gripping hand, there has been an explosion of LLM stuff of late, with 
enormous models being widely distributed in just the past month:

https://lwn.net/Articles/930939/

Could the AIs takeoff lead to a resumption of traffic growth? I still don´t 
think so...


On Thu, May 4, 2023 at 10:59 PM Vasilenko Eduard via NANOG 
mailto:nanog@nanog.org>> wrote:
Disclaimer: Metaverse has not changed Metro traffic yet. Then …

I am puzzled when people talk about 400GE and Tbps in the Mero context.
For historical reasons, Metro is still about 2*2*10GE (one “2” for redundancy, 
another “2” for capacity) in the majority of cases worldwide.
How many BRASes serve more than 4/1.5=27k users in the busy hour?
It

Re: Routed optical networks

2023-05-11 Thread Etienne-Victor Depasquale via NANOG

Eduard, academics cite the VNI (and the Sandvine Global reports).

Do you know of alternative sources that show traffic growth data you're
more comfortable with?

Cheers,

Etienne

On Thu, May 11, 2023 at 9:34 AM Vasilenko Eduard <
vasilenko.edu...@huawei.com> wrote:

> But it is speculation, not a trend yet.
>
> I remember 10y ago every presentation started from the claim that 100B of
> IoT would drive XXX traffic. It did not happen.
>
> Now we see presentations that AI would be talking to AI that generates
>  traffic.
>
> Maybe some technology would push traffic next S-curve, maybe not. It is
> still speculation.
>
>
>
> The traffic growth was stimulated (despite all VNIs) by 1) new
> subscribers, 2) video quality for subscribers. Nothing else yet.
>
> It is almost finished for both trends. We are close to the plateau of
> these S-curves.
>
> For some years (2013-2020) I was carefully looking at numbers for many
> countries: it was always possible to split CAGR for these 2 components. The
> video part was extremely consistent between countries. The subscriber part
> was 100% proportional to subscriber CAGR.
>
> Everything else up to now was “marketing” to say it mildly.
>
>
>
> Reminder: nothing in nature could grow indefinitely. The limit always
> exists. It is only a question of when.
>
>
>
> PS: Of course, marketing people could draw you any traffic growth – it
> depends just on the marketing budget.
>
>
>
> Eduard
>
> *From:* Dave Taht [mailto:dave.t...@gmail.com]
> *Sent:* Tuesday, May 9, 2023 11:41 PM
> *To:* Vasilenko Eduard 
> *Cc:* Phil Bedard ; Etienne-Victor Depasquale <
> ed...@ieee.org>; NANOG 
> *Subject:* Re: Routed optical networks
>
>
>
> Up until this moment I was feeling that my take on the decline of traffic
> growth was somewhat isolated, in that I have long felt that we are nearing
> the top of the S curve of the data we humans can create and consume. About
> the only source of future traffic growth I can think of comes from getting
> more humans online, and that is a mere another doubling.
>
>
>
> On the other hand, predictions such as 640k should be enough for everyone
> did not pan out.
>
>
>
> On the gripping hand, there has been an explosion of LLM stuff of late,
> with enormous models being widely distributed in just the past month:
>
>
>
> https://lwn.net/Articles/930939/
>
>
>
> Could the AIs takeoff lead to a resumption of traffic growth? I still
> don´t think so...
>
>
>
>
>
> On Thu, May 4, 2023 at 10:59 PM Vasilenko Eduard via NANOG <
> nanog@nanog.org> wrote:
>
> Disclaimer: Metaverse has not changed Metro traffic yet. Then …
>
>
>
> I am puzzled when people talk about 400GE and Tbps in the Mero context.
>
> For historical reasons, Metro is still about 2*2*10GE (one “2” for
> redundancy, another “2” for capacity) in the majority of cases worldwide.
>
> How many BRASes serve more than 4/1.5=27k users in the busy hour?
>
> It means that 50GE is the best interface now for the majority of cases.
> 2*50GE=100Gbps is good room for growth.
>
> Of course, exceptions could be. I know BRAS that handles 86k subscribers
> (do not recommend anybody to push the limits – it was so painful).
>
>
>
> We have just 2 eyes and look at video content about 22h per week (on
> average). Our eyes do not permit us to see resolution better than
> particular for chosen distance (4k for typical TV, HD for smartphones, and
> so on). Color depth 10bits is enough for the majority, 12bits is sure
> enough for everybody. 120 frames/sec is enough for everybody. It would
> never change – it is our genetics.
>
> Fortunately for Carriers, the traffic has a limit. You have probably seen
> that every year traffic growth % is decreasing. The Internet is stabilizing
> and approaching the plateau.
>
> How much growth is still awaiting us? 1.5? 1.4? It needs separate
> research. The result would be tailored for whom would pay for the research.
>
> IMHO: It is not mandatory that 100GE would become massive in the metro. (I
> know that 100GE is already massive in the DC CLOS)
>
>
>
> Additionally, who would pay for this traffic growth? It also limits
> traffic at some point.
>
> I hope it would happen after we would get our 22h/4k/12bit/120hz.
>
>
>
> Now, you could argue that Metaverse would jump and multiply traffic by an
> additional 2x or 3x. Then 400GE may be needed.
>
> Sorry, but it is speculation yet. It is not a trend like the current
> (declining) traffic growth.
>
>
>
> Ed/
>
> *From:* NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org]
> *On Behalf Of *

RE: Routed optical networks

But it is speculation, not a trend yet.
I remember 10y ago every presentation started from the claim that 100B of IoT 
would drive XXX traffic. It did not happen.
Now we see presentations that AI would be talking to AI that generates  
traffic.
Maybe some technology would push traffic next S-curve, maybe not. It is still 
speculation.

The traffic growth was stimulated (despite all VNIs) by 1) new subscribers, 2) 
video quality for subscribers. Nothing else yet.
It is almost finished for both trends. We are close to the plateau of these 
S-curves.
For some years (2013-2020) I was carefully looking at numbers for many 
countries: it was always possible to split CAGR for these 2 components. The 
video part was extremely consistent between countries. The subscriber part was 
100% proportional to subscriber CAGR.
Everything else up to now was “marketing” to say it mildly.

Reminder: nothing in nature could grow indefinitely. The limit always exists. 
It is only a question of when.

PS: Of course, marketing people could draw you any traffic growth – it depends 
just on the marketing budget.

Eduard
From: Dave Taht [mailto:dave.t...@gmail.com]
Sent: Tuesday, May 9, 2023 11:41 PM
To: Vasilenko Eduard 
Cc: Phil Bedard ; Etienne-Victor Depasquale 
; NANOG 
Subject: Re: Routed optical networks

Up until this moment I was feeling that my take on the decline of traffic 
growth was somewhat isolated, in that I have long felt that we are nearing the 
top of the S curve of the data we humans can create and consume. About the only 
source of future traffic growth I can think of comes from getting more humans 
online, and that is a mere another doubling.

On the other hand, predictions such as 640k should be enough for everyone did 
not pan out.

On the gripping hand, there has been an explosion of LLM stuff of late, with 
enormous models being widely distributed in just the past month:

https://lwn.net/Articles/930939/

Could the AIs takeoff lead to a resumption of traffic growth? I still don´t 
think so...


On Thu, May 4, 2023 at 10:59 PM Vasilenko Eduard via NANOG 
mailto:nanog@nanog.org>> wrote:
Disclaimer: Metaverse has not changed Metro traffic yet. Then …

I am puzzled when people talk about 400GE and Tbps in the Mero context.
For historical reasons, Metro is still about 2*2*10GE (one “2” for redundancy, 
another “2” for capacity) in the majority of cases worldwide.
How many BRASes serve more than 4/1.5=27k users in the busy hour?
It means that 50GE is the best interface now for the majority of cases. 
2*50GE=100Gbps is good room for growth.
Of course, exceptions could be. I know BRAS that handles 86k subscribers (do 
not recommend anybody to push the limits – it was so painful).

We have just 2 eyes and look at video content about 22h per week (on average). 
Our eyes do not permit us to see resolution better than particular for chosen 
distance (4k for typical TV, HD for smartphones, and so on). Color depth 10bits 
is enough for the majority, 12bits is sure enough for everybody. 120 frames/sec 
is enough for everybody. It would never change – it is our genetics.
Fortunately for Carriers, the traffic has a limit. You have probably seen that 
every year traffic growth % is decreasing. The Internet is stabilizing and 
approaching the plateau.
How much growth is still awaiting us? 1.5? 1.4? It needs separate research. The 
result would be tailored for whom would pay for the research.
IMHO: It is not mandatory that 100GE would become massive in the metro. (I know 
that 100GE is already massive in the DC CLOS)

Additionally, who would pay for this traffic growth? It also limits traffic at 
some point.
I hope it would happen after we would get our 22h/4k/12bit/120hz.

Now, you could argue that Metaverse would jump and multiply traffic by an 
additional 2x or 3x. Then 400GE may be needed.
Sorry, but it is speculation yet. It is not a trend like the current 
(declining) traffic growth.

Ed/
From: NANOG 
[mailto:nanog-bounces+vasilenko.eduard<mailto:nanog-bounces%2Bvasilenko.eduard>=huawei@nanog.org<mailto:huawei@nanog.org>]
 On Behalf Of Phil Bedard
Sent: Thursday, May 4, 2023 8:32 PM
To: Etienne-Victor Depasquale mailto:ed...@ieee.org>>; NANOG 
mailto:nanog@nanog.org>>
Subject: Re: Routed optical networks

It’s not necessarily metro specific although the metro networks could lend 
themselves to overall optimizations.

The adoption of ZR/ZR+ IPoWDM currently somewhat corresponds with your adoption 
of 400G since today they require a QDD port.   There are 100G QDD ports but 
that’s not all that popular yet.   Of course there is work to do something 
similar in QSFP28 if the power can be reduced to what is supported by an 
existing QSFP28 port in most devices.   In larger networks with higher speed 
requirements and moving to 400G with QDD, using the DCO optics for connecting 
routers is kind of a no-brainer vs. a traditional muxponder.   Whether that’s 
over a ROADM based opti

Re: Routed optical networks

2023-05-10 Thread Etienne-Victor Depasquale via NANOG

Dave, have you seen any recent output from Cisco's VNI on the matter of
traffic growth? Or Sandvine's? How does it compare with your perception?

Cheers,

Etienne

On Tue, May 9, 2023 at 10:40 PM Dave Taht  wrote:

> Up until this moment I was feeling that my take on the decline of traffic
> growth was somewhat isolated, in that I have long felt that we are nearing
> the top of the S curve of the data we humans can create and consume. About
> the only source of future traffic growth I can think of comes from getting
> more humans online, and that is a mere another doubling.
>
> On the other hand, predictions such as 640k should be enough for everyone
> did not pan out.
>
> On the gripping hand, there has been an explosion of LLM stuff of late,
> with enormous models being widely distributed in just the past month:
>
> https://lwn.net/Articles/930939/
>
> Could the AIs takeoff lead to a resumption of traffic growth? I still
> don´t think so...
>
>
> On Thu, May 4, 2023 at 10:59 PM Vasilenko Eduard via NANOG <
> nanog@nanog.org> wrote:
>
>> Disclaimer: Metaverse has not changed Metro traffic yet. Then …
>>
>>
>>
>> I am puzzled when people talk about 400GE and Tbps in the Mero context.
>>
>> For historical reasons, Metro is still about 2*2*10GE (one “2” for
>> redundancy, another “2” for capacity) in the majority of cases worldwide.
>>
>> How many BRASes serve more than 4/1.5=27k users in the busy hour?
>>
>> It means that 50GE is the best interface now for the majority of cases.
>> 2*50GE=100Gbps is good room for growth.
>>
>> Of course, exceptions could be. I know BRAS that handles 86k subscribers
>> (do not recommend anybody to push the limits – it was so painful).
>>
>>
>>
>> We have just 2 eyes and look at video content about 22h per week (on
>> average). Our eyes do not permit us to see resolution better than
>> particular for chosen distance (4k for typical TV, HD for smartphones, and
>> so on). Color depth 10bits is enough for the majority, 12bits is sure
>> enough for everybody. 120 frames/sec is enough for everybody. It would
>> never change – it is our genetics.
>>
>> Fortunately for Carriers, the traffic has a limit. You have probably seen
>> that every year traffic growth % is decreasing. The Internet is stabilizing
>> and approaching the plateau.
>>
>> How much growth is still awaiting us? 1.5? 1.4? It needs separate
>> research. The result would be tailored for whom would pay for the research.
>>
>> IMHO: It is not mandatory that 100GE would become massive in the metro.
>> (I know that 100GE is already massive in the DC CLOS)
>>
>>
>>
>> Additionally, who would pay for this traffic growth? It also limits
>> traffic at some point.
>>
>> I hope it would happen after we would get our 22h/4k/12bit/120hz.
>>
>>
>>
>> Now, you could argue that Metaverse would jump and multiply traffic by an
>> additional 2x or 3x. Then 400GE may be needed.
>>
>> Sorry, but it is speculation yet. It is not a trend like the current
>> (declining) traffic growth.
>>
>>
>>
>> Ed/
>>
>> *From:* NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org]
>> *On Behalf Of *Phil Bedard
>> *Sent:* Thursday, May 4, 2023 8:32 PM
>> *To:* Etienne-Victor Depasquale ; NANOG 
>> *Subject:* Re: Routed optical networks
>>
>>
>>
>> It’s not necessarily metro specific although the metro networks could
>> lend themselves to overall optimizations.
>>
>>
>>
>> The adoption of ZR/ZR+ IPoWDM currently somewhat corresponds with your
>> adoption of 400G since today they require a QDD port.   There are 100G QDD
>> ports but that’s not all that popular yet.   Of course there is work to do
>> something similar in QSFP28 if the power can be reduced to what is
>> supported by an existing QSFP28 port in most devices.   In larger networks
>> with higher speed requirements and moving to 400G with QDD, using the DCO
>> optics for connecting routers is kind of a no-brainer vs. a traditional
>> muxponder.   Whether that’s over a ROADM based optical network or not,
>> especially at metro/regional distances.
>>
>>
>>
>> There are very large deployments of IPoDWDM over passive DWDM or dark
>> fiber for access and aggregation networks where the aggregate required
>> bandwidth doesn’t exceed the capabilities of those optics.  It’s been done
>> at 10G for many years.  With the advent of pluggable EDFA amplifiers, you
>> can even build links up to 120km* (perfect d

Re: Routed optical networks

2023-05-10 Thread Mark Tinka




On 5/9/23 20:37, Phil Bedard wrote:

[phil] These are already available today and have been for some time 
and in use in production networks for over a year now.  This is with 
400G links running up to 600km in routers with QDD ports.  400G-16QAM 
using 60Gbaud (the OpenZR+ standard) can reach around 1300km.   These 
optics are being used in both routers and xponders on the line side.


Right... I've not been keeping in touch with this development on the 
terrestrial side, as our focus is mainly on the submarine end of things. 
But this is good to know.


In our case, the majority of our customers are still seeking 10Gbps and 
100Gbps services over long distances (so 200km and over).


Neither they (nor us) have a requirement for 400Gbps in the metro. That 
said, we do have the capability to do this on terrestrial spans where we 
are able to carry 600Gbps per channel across 700km or so. We are looking 
forward to the next generation of tech. with our vendor that could get 
us to about 700Gbps per channel for the same distance.


[phil] 800Gbps and 1.2Tbps are not really positioned for long haul use 
cases yet.  It requires lowering the modulation to something like QPSK 
which at those speeds requires baud rates which are not yet 
commercially available.




I meant more in terms of the capabilities of the 800G/1.2T embedded 
option to improve spectral efficiency for more capacity over long haul 
use-cases, and not necessarily that they can actually get to 800Gbps or 
1.2Tbps for that length of span.


The next generation of 5nm CMOS with the potential for close to 150 
Gbaud is quite intriguing, although I'm not expecting anything more than 
15% - 18% improvement in performance compared to the current 7nm 100 
Gbaud systems.


Will keep you posted, especially on our submarine application of this.

Mark.

Re: Routed optical networks

2023-05-09 Thread Dave Taht

Up until this moment I was feeling that my take on the decline of traffic
growth was somewhat isolated, in that I have long felt that we are nearing
the top of the S curve of the data we humans can create and consume. About
the only source of future traffic growth I can think of comes from getting
more humans online, and that is a mere another doubling.

On the other hand, predictions such as 640k should be enough for everyone
did not pan out.

On the gripping hand, there has been an explosion of LLM stuff of late,
with enormous models being widely distributed in just the past month:

https://lwn.net/Articles/930939/

Could the AIs takeoff lead to a resumption of traffic growth? I still don´t
think so...


On Thu, May 4, 2023 at 10:59 PM Vasilenko Eduard via NANOG 
wrote:

> Disclaimer: Metaverse has not changed Metro traffic yet. Then …
>
>
>
> I am puzzled when people talk about 400GE and Tbps in the Mero context.
>
> For historical reasons, Metro is still about 2*2*10GE (one “2” for
> redundancy, another “2” for capacity) in the majority of cases worldwide.
>
> How many BRASes serve more than 4/1.5=27k users in the busy hour?
>
> It means that 50GE is the best interface now for the majority of cases.
> 2*50GE=100Gbps is good room for growth.
>
> Of course, exceptions could be. I know BRAS that handles 86k subscribers
> (do not recommend anybody to push the limits – it was so painful).
>
>
>
> We have just 2 eyes and look at video content about 22h per week (on
> average). Our eyes do not permit us to see resolution better than
> particular for chosen distance (4k for typical TV, HD for smartphones, and
> so on). Color depth 10bits is enough for the majority, 12bits is sure
> enough for everybody. 120 frames/sec is enough for everybody. It would
> never change – it is our genetics.
>
> Fortunately for Carriers, the traffic has a limit. You have probably seen
> that every year traffic growth % is decreasing. The Internet is stabilizing
> and approaching the plateau.
>
> How much growth is still awaiting us? 1.5? 1.4? It needs separate
> research. The result would be tailored for whom would pay for the research.
>
> IMHO: It is not mandatory that 100GE would become massive in the metro. (I
> know that 100GE is already massive in the DC CLOS)
>
>
>
> Additionally, who would pay for this traffic growth? It also limits
> traffic at some point.
>
> I hope it would happen after we would get our 22h/4k/12bit/120hz.
>
>
>
> Now, you could argue that Metaverse would jump and multiply traffic by an
> additional 2x or 3x. Then 400GE may be needed.
>
> Sorry, but it is speculation yet. It is not a trend like the current
> (declining) traffic growth.
>
>
>
> Ed/
>
> *From:* NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org]
> *On Behalf Of *Phil Bedard
> *Sent:* Thursday, May 4, 2023 8:32 PM
> *To:* Etienne-Victor Depasquale ; NANOG 
> *Subject:* Re: Routed optical networks
>
>
>
> It’s not necessarily metro specific although the metro networks could lend
> themselves to overall optimizations.
>
>
>
> The adoption of ZR/ZR+ IPoWDM currently somewhat corresponds with your
> adoption of 400G since today they require a QDD port.   There are 100G QDD
> ports but that’s not all that popular yet.   Of course there is work to do
> something similar in QSFP28 if the power can be reduced to what is
> supported by an existing QSFP28 port in most devices.   In larger networks
> with higher speed requirements and moving to 400G with QDD, using the DCO
> optics for connecting routers is kind of a no-brainer vs. a traditional
> muxponder.   Whether that’s over a ROADM based optical network or not,
> especially at metro/regional distances.
>
>
>
> There are very large deployments of IPoDWDM over passive DWDM or dark
> fiber for access and aggregation networks where the aggregate required
> bandwidth doesn’t exceed the capabilities of those optics.  It’s been done
> at 10G for many years.  With the advent of pluggable EDFA amplifiers, you
> can even build links up to 120km* (perfect dark fiber)  carrying tens of
> terabits of traffic without any additional active optical equipment.
>
>
>
> It’s my personal opinion we aren’t to the days yet of where we can simply
> build an all packet network with no photonic switching that carries all
> services, but eventually (random # of years) it gets there for many
> networks.  There are also always going to be high performance applications
> for transponders where pluggable optics aren’t a good fit.
>
>
>
> Carrying high speed private line/wavelength type services as well is a
> different topic than interconnecting IP devices.
>
>
>
> Thanks,
>
> Phil
>
>
&

Re: Routed optical networks

2023-05-09 Thread Phil Bedard

From: Mark Tinka 
Date: Tuesday, May 9, 2023 at 2:03 AM
To: Phil Bedard , nanog@nanog.org 
Subject: Re: Routed optical networks
On 5/8/23 21:53, Phil Bedard wrote:

  There are quite a few QDD pluggables in production today capable of 
supporting 100G signals over 1000s of km or 400G near 1500km.

I think the more interesting QSPF-DD (or OSFP) development is OpenZR+, which is 
an MSA project to standardize 400ZR+. The plan is to be able to support 100Gbps 
up to 5,800km, and 400Gbps up to 480km (EDFA) or about 1,000km (EDFA + Raman). 
With a 4x 100Gbps mode supported on QSFP28 router ports, you can have one 
muxponder talking to 4x routers at 100Gbps each.

[phil] These are already available today and have been for some time and in use 
in production networks for over a year now.  This is with 400G links running up 
to 600km in routers with QDD ports.  400G-16QAM using 60Gbaud (the OpenZR+ 
standard) can reach around 1300km.   These optics are being used in both 
routers and xponders on the line side.

  Now that’s not what you can get out of some external transponders, so those 
will still have their place in high performance applications.   When you move 
to 800G, 1.2Tbps single channel they also have their own distance limitations.  
So it really depends on the application and the network.

800Gbps and 1.2Tbps applications are really for long haul use-cases, especially 
if you used 400Gbps pluggables before and run out of distance (so less than 
1,000km). They are also preferred for submarine use-cases.

[phil] 800Gbps and 1.2Tbps are not really positioned for long haul use cases 
yet.  It requires lowering the modulation to something like QPSK which at those 
speeds requires baud rates which are not yet commercially available.

Re: Routed optical networks

2023-05-09 Thread Etienne-Victor Depasquale via NANOG

>
> The optical network is made up of the photonic portion and then the
> transponder/muxponder portion.
>
Thank you for that direct definition. I'm serious (not sarcastic).

One thing I've written about in papers is the nomenclature problem, and I'm
in good company.
Bill Norton had written explicitly "the lexicon is important", and dwelt on
that theme, in his book "the internet peering playbook".

This is the source of a lot of grief.

Cheers,

Etienne

On Mon, May 8, 2023 at 9:57 PM Phil Bedard  wrote:

> I guess let’s not confuse two things.  The optical network is made up of
> the photonic portion and then the transponder/muxponder portion.   A single
> term like “DWDM” can be confusing since it can refer to both.   It will
> take a long time (maybe never) to remove the photonic switching part of the
> network.  However, it’s always been cheap to deploy because optical vendors
> tended to subsidize that network using sales of the other portion, the
> transponders, which you buy more and more over time.  Those photonic
> components are expensive.
>
>
>
> On the DWDM signal portion, I’m not talking about 100ZR compared to 100G
> on a transponder or DWDM line system.  100ZR has had to deal with the power
> limitations of QSFP28 ports, which QDD/OSFP do not suffer from.  There are
> quite a few QDD pluggables in production today capable of supporting 100G
> signals over 1000s of km or 400G near 1500km.  Now that’s not what you can
> get out of some external transponders, so those will still have their place
> in high performance applications.   When you move to 800G, 1.2Tbps single
> channel they also have their own distance limitations.  So it really
> depends on the application and the network.
>
>
>
> Phil
>
>
>
> *From: *NANOG  on behalf
> of Mark Tinka 
> *Date: *Friday, May 5, 2023 at 12:55 AM
> *To: *nanog@nanog.org 
> *Subject: *Re: Routed optical networks
>
>
>
> On 5/4/23 19:32, Phil Bedard wrote:
>
>
>
> It’s my personal opinion we aren’t to the days yet of where we can simply
> build an all packet network with no photonic switching that carries all
> services, but eventually (random # of years) it gets there for many
> networks.  There are also always going to be high performance applications
> for transponders where pluggable optics aren’t a good fit.
>
>
> I think every time the IP space gets close to running an all-packet
> network, the Transport folk come out with an easier way to do it, that it's
> too hard to ignore.
>
> Based on that, I think they will always be one step ahead, with the key
> advantage being reliability of capacity over the distance, for the cost.
>
> The farther your fibre has to run, the costlier it gets to do it without
> DWDM.
>
> I mean, it's only now that 100G-ZR is becoming a reality for packet
> networks, and we are talking thousands of US$ for optics to get us 80km -
> 120km distance. Meanwhile, DWDM vendors can get you 800Gbps per wavelength
> in the same distance (or 30X that distance) far less cheaply.
>
> I get the appeal of not needing DWDM gear to underlay your packet
> network... it's neater and offers fewer points of failure. But unless you
> are dealing with very short distances and can ride a reasonable balance
> between service features and switching/forwarding capacity in your
> router/switch, it's going to be hard to ignore the DWDM gear if you are
> trying to be a serious operation, at that scale, over a wide area.
>
> Mark.
>


-- 
Ing. Etienne-Victor Depasquale
Assistant Lecturer
Department of Communications & Computer Engineering
Faculty of Information & Communication Technology
University of Malta
Web. https://www.um.edu.mt/profile/etiennedepasquale

Re: Routed optical networks

2023-05-09 Thread Mark Tinka




On 5/8/23 21:53, Phil Bedard wrote:


I guess let’s not confuse two things.  The optical network is made up 
of the photonic portion and then the transponder/muxponder portion.   
A single term like “DWDM” can be confusing since it can refer to both.




Indeed.

I am short-handing to mean DWDM on the line side and grey on the client 
side.



   It will take a long time (maybe never) to remove the photonic 
switching part of the network.




There was a time when my intention was to do just that. But that was 
prior to expecting to ever run links larger than 10Gbps :-).



  However, it’s always been cheap to deploy because optical vendors 
tended to subsidize that network using sales of the other portion, the 
transponders, which you buy more and more over time.  Those photonic 
components are expensive.




It depends on the application... either you want performance, space and 
power cost optimization or functional integration of previously 
disparate system-level features. This will determine whether you focus 
on pluggables or embeddeds, with pluggables promoting cost reduction, 
while embeddeds push performance.



On the DWDM signal portion, I’m not talking about 100ZR compared to 
100G on a transponder or DWDM line system.  100ZR has had to deal with 
the power limitations of QSFP28 ports, which QDD/OSFP do not suffer from.





Right - 100ZR is short-reach (80km). It's really for the metro edge 
where 400Gbps is not needed.



  There are quite a few QDD pluggables in production today capable of 
supporting 100G signals over 1000s of km or 400G near 1500km.




I think the more interesting QSPF-DD (or OSFP) development is OpenZR+, 
which is an MSA project to standardize 400ZR+. The plan is to be able to 
support 100Gbps up to 5,800km, and 400Gbps up to 480km (EDFA) or about 
1,000km (EDFA + Raman). With a 4x 100Gbps mode supported on QSFP28 
router ports, you can have one muxponder talking to 4x routers at 
100Gbps each.



  Now that’s not what you can get out of some external transponders, 
so those will still have their place in high performance 
applications.   When you move to 800G, 1.2Tbps single channel they 
also have their own distance limitations.  So it really depends on the 
application and the network.




800Gbps and 1.2Tbps applications are really for long haul use-cases, 
especially if you used 400Gbps pluggables before and run out of distance 
(so less than 1,000km). They are also preferred for submarine use-cases.


I can't wait to see what happens when the CMOS gets down to 5nm and 3nm.

Mark.

Re: Routed optical networks

2023-05-08 Thread Phil Bedard

I guess let’s not confuse two things.  The optical network is made up of the 
photonic portion and then the transponder/muxponder portion.   A single term 
like “DWDM” can be confusing since it can refer to both.   It will take a long 
time (maybe never) to remove the photonic switching part of the network.  
However, it’s always been cheap to deploy because optical vendors tended to 
subsidize that network using sales of the other portion, the transponders, 
which you buy more and more over time.  Those photonic components are expensive.

On the DWDM signal portion, I’m not talking about 100ZR compared to 100G on a 
transponder or DWDM line system.  100ZR has had to deal with the power 
limitations of QSFP28 ports, which QDD/OSFP do not suffer from.  There are 
quite a few QDD pluggables in production today capable of supporting 100G 
signals over 1000s of km or 400G near 1500km.  Now that’s not what you can get 
out of some external transponders, so those will still have their place in high 
performance applications.   When you move to 800G, 1.2Tbps single channel they 
also have their own distance limitations.  So it really depends on the 
application and the network.

Phil

From: NANOG  on behalf of Mark 
Tinka 
Date: Friday, May 5, 2023 at 12:55 AM
To: nanog@nanog.org 
Subject: Re: Routed optical networks

On 5/4/23 19:32, Phil Bedard wrote:

It’s my personal opinion we aren’t to the days yet of where we can simply build 
an all packet network with no photonic switching that carries all services, but 
eventually (random # of years) it gets there for many networks.  There are also 
always going to be high performance applications for transponders where 
pluggable optics aren’t a good fit.

I think every time the IP space gets close to running an all-packet network, 
the Transport folk come out with an easier way to do it, that it's too hard to 
ignore.

Based on that, I think they will always be one step ahead, with the key 
advantage being reliability of capacity over the distance, for the cost.

The farther your fibre has to run, the costlier it gets to do it without DWDM.

I mean, it's only now that 100G-ZR is becoming a reality for packet networks, 
and we are talking thousands of US$ for optics to get us 80km - 120km distance. 
Meanwhile, DWDM vendors can get you 800Gbps per wavelength in the same distance 
(or 30X that distance) far less cheaply.

I get the appeal of not needing DWDM gear to underlay your packet network... 
it's neater and offers fewer points of failure. But unless you are dealing with 
very short distances and can ride a reasonable balance between service features 
and switching/forwarding capacity in your router/switch, it's going to be hard 
to ignore the DWDM gear if you are trying to be a serious operation, at that 
scale, over a wide area.

Mark.

Re: Routed optical networks




On 5/5/23 15:50, Mike Hammett wrote:


Anyone publicly traded doesn't plan longer than the current quarter.


A phenomenon not unique to them - private companies suffer the same issue.

Mark.

Re: Routed optical networks





On 5/5/23 19:02, Jared Brown wrote:


You can get 100G optics for less than half those prices.

For reference, here are publicly listed prices for optics from an European 
vendor I have in production:

100G 4WDM & CLR4 QSFP28
* 10 km
* 225€
100GBASE-LR4 & OTU4 & 128GFC QSFP28
* 20km
*305€
100GBASE-ER4 Lite & OTU4 & 128GFC QSFP28
* 40 km
* 1030€
100GBASE-ZR4 QSFP28
* 80 km
* 1650€
100GBASE-ZR4+ QSFP28
* 100 km
* 2100€

These prices are for single units without discounts.


It wasn't so much about the pricing that I was posting, but more about 
the value of 100Gbps over anything slower, if your main incentive is 
optics price.


Mark.

Re: Routed optical networks




On 5/5/23 15:49, Mike Hammett wrote:

Incumbents are great at momentum. They're not great at innovation, 
customer experience, etc. They only reason most incumbents are still 
relevant is due to their prior market size.


Most true, and even then, there is a visible change to the bottom line 
on how much money they print now vs. then, despite the fact that they 
are still printing money.


This means that they all know that the dark day is looming, and even if 
they can see it drawing closer, telco's are stuck with not knowing what 
to do about it.


I mean, if anyone remembers the days of SMS (2000 - 2008) - that was as 
free as money printing gets.


Mark.

Re: Routed optical networks




On 5/5/23 13:04, Vasilenko Eduard wrote:


Hi Mark,

Thanks a lot for many of your valuable comments I almost always agree.

1.I agree that 50GE has not got the same popularity as 100GE. Many 
vendors did ignore it for some time. Looks like not many ignore it now.


2.Even in your example for 40km, 100GE is about twice more expensive 
than 50GE.


3.Hence, I have to google for the cheaper proposition in 10km. For 
obvious reasons, I could not reference my employer (my assumption 
about the cost is based on this comparison).

https://opticswave.com/collections/50g-qsfp28
https://www.compufox.com/50G_QSFP28_Transceivers_s/3036.htm
https://www.genuinemodules.com/033030600050
Looks like I have found twice cheaper in public information.

4.Pay attention that bidirectional is available too.



In the real world, a 100Gbps network is very, very different from a 
50Gbps network.


A 100Gbps optic being twice the cost of a 50Gbps optic at 40km is worth 
it for most operators, because the value from 100Gbps far outweighs the 
cost when compared against the "savings" for a 50Gbps network.


Also, most operators will buy a 40km optic yet they need something to 
cover the 11km - 30km range. With 25km and 30km optics costing as much 
as a 40km 50Gbps optic, this will satisfy many operators.



5.The public price is not what you get in the real tender. We are 
talking about big networks, hence, big tenders.




I have zero sympathy for operators who choose to pay equipment vendors 
for their branded optics :-).


Mark.

Re: Routed optical networks

2023-05-05 Thread Jared Mauch

> On May 2, 2023, at 5:11 PM, Etienne-Victor Depasquale  wrote:
> 
> I’ve seen proposals for an LSR MPLS/ROADAM type solution, where imagine you 
> are at a hop where in a long distance system solution, you would end up with 
> OEO, but instead you get directionality capability with an IP/MPLS capable 
> device.  As mentioned previously, the 400-ZR/ZR+/ZR-Bright/+0 optics are the 
> latest example of that.
> 
> Jared, I understand your point in the above statement to be that 
> directionality is cost-effectively implemented through label-switched paths, 
> rather than (ROADM-enabled) optical path switching. 
> 
> Do I understand right?

It might be based on the distances and bandwidth required, and the need or 
desire to have diverse paths.  Much of the economics of this vary based on the 
vendors and underlying either cost or pricing model.

I do believe a few more engineers would be aided with a TCO for networking once 
you add up all the costs, either internal or external.

We’ve seen the one-time-spend to build out the datacenter space exceed the cost 
of the equipment.  (Based on the number of XC/patch panel positions needed for 
example - which might be a direct expense while the hardware may be capitalized 
and depreciated over a period, also TCO to power on a device - it can be quite 
common for that to exceed the Capex as well).

- Jared

Re: Routed optical networks

2023-05-05 Thread Jared Brown

On 5/5/23, Mark Tinka wrote:

> Juxtapose that against 100Gbps pricing:
>
>  * EUR473 @ 10km.
>  * EUR1,300 @ 25km.
>  * EUR1,500 @ 30km.
>  * EUR2,600 @ 40km.
>  * EUR3,925 @ 80km.

You can get 100G optics for less than half those prices.

For reference, here are publicly listed prices for optics from an European 
vendor I have in production:

100G 4WDM & CLR4 QSFP28
* 10 km
* 225€
100GBASE-LR4 & OTU4 & 128GFC QSFP28
* 20km 
*305€
100GBASE-ER4 Lite & OTU4 & 128GFC QSFP28 
* 40 km 
* 1030€
100GBASE-ZR4 QSFP28
* 80 km
* 1650€
100GBASE-ZR4+ QSFP28
* 100 km 
* 2100€

These prices are for single units without discounts. 

- Jared

RE: Routed optical networks

2023-05-05 Thread Vasilenko Eduard via NANOG

There are places in the world (like Middle East) where telephony system did not 
exist historically.
Then no copper, then no ducts.
Then new fiber is very difficult.

But much more places where the Telephony system did exist.
Ed/
From: Mike Hammett [mailto:na...@ics-il.net]
Sent: Friday, May 5, 2023 4:50 PM
To: Vasilenko Eduard 
Cc: Mark Tinka ; nanog@nanog.org
Subject: Re: Routed optical networks

Incumbents are great at momentum. They're not great at innovation, customer 
experience, etc. They only reason most incumbents are still relevant is due to 
their prior market size.

Around here, the incumbent telcos still have lead-sheathed cables in the 
ground, not removing anything. Often, things are abandoned in place, unless 
there's a good enough reason to remove it.

I'm placing my own ducts into the ground and putting my own fiber in it. I 
still put in DWDM between my facilities to minimize the consumption of 
resources. A couple of hundred bucks for a DWDM optic is cheaper than a strand 
between two locations.

-
Mike Hammett
Intelligent Computing Solutions<http://www.ics-il.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/ICSIL>[http://www.ics-il.com/images/googleicon.png]<https://plus.google.com/+IntelligentComputingSolutionsDeKalb>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/intelligent-computing-solutions>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/ICSIL>
Midwest Internet Exchange<http://www.midwest-ix.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/mdwestix>[http://www.ics-il.com/images/linkedinicon.png]<https://www.linkedin.com/company/midwest-internet-exchange>[http://www.ics-il.com/images/twittericon.png]<https://twitter.com/mdwestix>
The Brothers WISP<http://www.thebrotherswisp.com/>
[http://www.ics-il.com/images/fbicon.png]<https://www.facebook.com/thebrotherswisp>[http://www.ics-il.com/images/youtubeicon.png]<https://www.youtube.com/channel/UCXSdfxQv7SpoRQYNyLwntZg>

From: "Vasilenko Eduard via NANOG" mailto:nanog@nanog.org>>
To: "Mark Tinka" mailto:mark@tinka.africa>>, 
nanog@nanog.org<mailto:nanog@nanog.org>
Sent: Wednesday, May 3, 2023 4:10:19 AM
Subject: RE: Routed optical networks
> Yeah, you sound like an equipment vendor whose main customers are incumbent 
> telco's in a few rich markets :-).
You are right. My message was pretty much geared toward incumbents.
But the majority of the access/aggregation is in their possessions, isn’t it?
They typically have ducts that were huge for copper that is already extracted.
One more fiber cable would be easy.

Agree that for competitive carriers DWDM would be more often needed.
Even for competitive carriers, it makes sense to evaluate the cost to put fiber 
into the duct of incumbents.
Especially because in some countries the price would be regulated.
It would solve the problem forever – no need for the DWDM speed upgrade.

I am calling to just not forget to evaluate this option too. Reminder: dark 
fiber is the best technical solution, for sure.

Ed/
From: Mark Tinka [mailto:mark@tinka.africa]
Sent: Wednesday, May 3, 2023 11:39 AM
To: Vasilenko Eduard 
mailto:vasilenko.edu...@huawei.com>>; 
nanog@nanog.org<mailto:nanog@nanog.org>
Subject: Re: Routed optical networks

On 5/3/23 08:20, Vasilenko Eduard wrote:
I would risk to say a little more on this.
Indeed, maybe the situation (in many countries) when the Carrier sells a lot of 
TDM services.
But in general, packet services are enough these days for many carriers/regions.

There aren't enough TDM services to warrant DWDM, nowadays.

The reason for DWDM is mainly being driven by Ethernet, and IP.

At any reasonable scale, it's actually pretty hard to buy a TDM service, in 
most markets.

Additionally, I am sure that in many countries/Metro it is cheaper to lay down 
a new fiber than to provision DWDM, even if it is a pizza box.

I disagree. Existing fibre may be cheap because it was laid down a decade or 
more ago, en masse, by several operators. So the market would be experiencing a 
glut, not because it is cheap to open up the roads and plant more fibre, but 
because there is so much of it to begin with.

At worst, there is still enough duct space that the operator can blow more 
fibre. But when that duct gets full, and there are no more free ducts 
available, or another route needs to get built for whatever reason, it is a 
rather costly affair to open up the roads and trunk some fibre, in any market.

So no, DWDM is not more expensive, if you are delivering services at scale. It 
is actually cheaper. It is only more expensive if you are small scale, because 
in some markets, the fibre glut means you can buy dark fibre for cheaper than 
you can light it with DWDM. But this is a situation unique to small operators,

Re: Routed optical networks

2023-05-05 Thread Mike Hammett

Anyone publicly traded doesn't plan longer than the current quarter. 

- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Mark Tinka"  
To: nanog@nanog.org 
Sent: Thursday, May 4, 2023 6:10:36 AM 
Subject: Re: Routed optical networks 

On 5/4/23 12:58, Vasilenko Eduard via NANOG wrote: 

> Well, ISP is typically plan something for a year. It is more than enough for 
> both. 

The real world is much less certain, especially in these economic times. 

> Funny, that with the current lead times for electronics, Fiber could be 
> faster. 
> Of course, it is a temporary glitch. 

Even in the same market, no two lays of fibre can be guaranteed to be 
completed in the same time. 

Mark.

Re: Routed optical networks

2023-05-05 Thread Mike Hammett

Incumbents are great at momentum. They're not great at innovation, customer 
experience, etc. They only reason most incumbents are still relevant is due to 
their prior market size. 

Around here, the incumbent telcos still have lead-sheathed cables in the 
ground, not removing anything. Often, things are abandoned in place, unless 
there's a good enough reason to remove it. 

I'm placing my own ducts into the ground and putting my own fiber in it. I 
still put in DWDM between my facilities to minimize the consumption of 
resources. A couple of hundred bucks for a DWDM optic is cheaper than a strand 
between two locations. 

- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Vasilenko Eduard via NANOG"  
To: "Mark Tinka" , nanog@nanog.org 
Sent: Wednesday, May 3, 2023 4:10:19 AM 
Subject: RE: Routed optical networks 

> Yeah, you sound like an equipment vendor whose main customers are incumbent 
> telco's in a few rich markets :-). 
You are right. My message was pretty much geared toward incumbents. 
But the majority of the access/aggregation is in their possessions, isn’t it? 
They typically have ducts that were huge for copper that is already extracted. 
One more fiber cable would be easy. 

Agree that for competitive carriers DWDM would be more often needed. 
Even for competitive carriers, it makes sense to evaluate the cost to put fiber 
into the duct of incumbents. 
Especially because in some countries the price would be regulated. 
It would solve the problem forever – no need for the DWDM speed upgrade. 

I am calling to just not forget to evaluate this option too. Reminder: dark 
fiber is the best technical solution, for sure. 

Ed/ 

From: Mark Tinka [mailto:mark@tinka.africa] 
Sent: Wednesday, May 3, 2023 11:39 AM 
To: Vasilenko Eduard ; nanog@nanog.org 
Subject: Re: Routed optical networks 

On 5/3/23 08:20, Vasilenko Eduard wrote: 

I would risk to say a little more on this. 
Indeed, maybe the situation (in many countries) when the Carrier sells a lot of 
TDM services. 
But in general, packet services are enough these days for many 
carriers/regions. 

There aren't enough TDM services to warrant DWDM, nowadays. 

The reason for DWDM is mainly being driven by Ethernet, and IP. 

At any reasonable scale, it's actually pretty hard to buy a TDM service, in 
most markets. 

Additionally, I am sure that in many countries/Metro it is cheaper to lay down 
a new fiber than to provision DWDM, even if it is a pizza box. 

I disagree. Existing fibre may be cheap because it was laid down a decade or 
more ago, en masse, by several operators. So the market would be experiencing a 
glut, not because it is cheap to open up the roads and plant more fibre, but 
because there is so much of it to begin with. 

At worst, there is still enough duct space that the operator can blow more 
fibre. But when that duct gets full, and there are no more free ducts 
available, or another route needs to get built for whatever reason, it is a 
rather costly affair to open up the roads and trunk some fibre, in any market. 

So no, DWDM is not more expensive, if you are delivering services at scale. It 
is actually cheaper. It is only more expensive if you are small scale, because 
in some markets, the fibre glut means you can buy dark fibre for cheaper than 
you can light it with DWDM. But this is a situation unique to small operators, 
not large ones. 

The colored interface is still very expensive. 

This only matters for the line side. 

For client-facing, it's not a drama. And you typically buy more optics for the 
client side than you do the line side. 

Of course, there are some Cities (not “towns”) where it is very expensive or 
maybe even impossible to lay down a new fiber. 
Yes, in the majority of cases, it is cheaper to lay down fiber. 

I think what you mean to say is that in the majority of cases where there is 
fibre glut, and dark fibre is a market option, buying fibre is cheaper than 
lighting it with DWDM. This is true. But I think that on a global scale, this 
is the exception, not the rule. 

In general, you are not likely to be able to buy dark fibre, cheaply or 
otherwise, if you look at all markets in the world. 

Hence, the importance of DWDM for the Metro is overestimated. 

Again, only if you are small scale. 

If you are a large scale operator with as many IP/Ethernet customers as you 
have Transport, DWDM is essential. 

Use only routers. Provision enough fiber. Have always 1 router hop to the 
aggregation (hub-spoke topology), no routers chaining in the ring. 
If fiber is not enough – then use normal DWDM with an external transponder. 
Routers would be still in hub-spoke topology. 

Yeah, you sound like an equipment vendor whose main customers are incumbent 
telco's in a few rich markets :-). 

The life of the average operator, aro

RE: Routed optical networks

2023-05-05 Thread Vasilenko Eduard via NANOG

Hi Mark,
Thanks a lot for many of your valuable comments I almost always agree.


1.   I agree that 50GE has not got the same popularity as 100GE. Many 
vendors did ignore it for some time. Looks like not many ignore it now.

2.   Even in your example for 40km, 100GE is about twice more expensive 
than 50GE.

3.   Hence, I have to google for the cheaper proposition in 10km. For 
obvious reasons, I could not reference my employer (my assumption about the 
cost is based on this comparison).
https://opticswave.com/collections/50g-qsfp28
https://www.compufox.com/50G_QSFP28_Transceivers_s/3036.htm
https://www.genuinemodules.com/033030600050
Looks like I have found twice cheaper in public information.

4.   Pay attention that bidirectional is available too.

5.   The public price is not what you get in the real tender. We are 
talking about big networks, hence, big tenders.

Eduard
From: Mark Tinka [mailto:mark@tinka.africa]
Sent: Friday, May 5, 2023 1:17 PM
To: Vasilenko Eduard ; nanog@nanog.org
Subject: Re: Routed optical networks


On 5/5/23 10:54, Vasilenko Eduard wrote:

50GE is better just because it is half of the cost of 100GE and it is enough 
now for the great majority of cases. Money is very important these days for 
this industry. 100GE single mode is more expensive than the best router port 
itself. Routers have been deprecated 10x for the decade (almost 100x for 2 
decades). Pluggable optics is not that much deprecated.

Not sure where your pricing is coming from, but if I look at Flexoptix's 50Gbps 
QSFP28 optics pricing, I am getting:

  *   EUR724 @ 10km.
  *   EUR1,246 @ 40km.

They are also selling an SFP56 LR for EUR925.

Juxtapose that against 100Gbps pricing:

  *   EUR473 @ 10km.
  *   EUR1,300 @ 25km.
  *   EUR1,500 @ 30km.
  *   EUR2,600 @ 40km.
  *   EUR3,925 @ 80km.

Doesn't immediately seem to me that 50Gbps is cheaper than 100Gbps. There also 
don't seem to be as many deployments of 50Gbps in the metro (same could be said 
for 25Gbps and 40Gbps), but others on the list can chime in with what they are 
seeing/doing.



I do not think that content provider guys call their DCI “Metro”, not very 
often.

Well, whatever they call it, the concept is the same - move lots of traffic 
across town between data centres.



I agree that 100GE for DCI is the minimum, 400GE is probably already needed in 
some places.
IMHO: it is a different story. Very interested too.

Most content providers have no choice but to run DWDM, for even very short 
spans between data centres. That is because it is just cheaper and simpler to 
pack Tbps of capacity in DWDM for the price than you can in a router. And 
besides, most routers don't need to carry Tbps of traffic in a single line 
card, which would be a waste of a fibre pair over that distance.

In such cases, better to use DWDM and drop capacity on individual routers 
and/or line cards as you see fit.




PS: By the way, even if some ISP has 50% of revenue from Enterprise services 
(it is probably the biggest number, typically 30%-40%), it is still just 5% 
compare to residential traffic. Traffic to enterprises is still sold 4x-10x 
(depending on the country).

That is why residential Access networks tend to be 2nd class citizens :-).



Hence, Enterprise does not make sense to mention in the traffic discussion. It 
is a “rounding error”.
Enterprise business created a huge demand for oversubscribed ports to connect 
Enterprises. And QoS/QoE. Not traffic.

Well, not all operators that offer enterprise services also do consumer 
broadband, or vice versa. To a network doing only one or the other, whatever 
traffic they are carrying means the world to them. It's not ours to decide what 
is high or low traffic... that priviledge always remains with the network 
operator.

Mark.

Re: Routed optical networks




On 5/5/23 10:54, Vasilenko Eduard wrote:

50GE is better just because it is half of the cost of 100GE and it is 
enough now for the great majority of cases. Money is very important 
these days for this industry. 100GE single mode is more expensive than 
the best router port itself. Routers have been deprecated 10x for the 
decade (almost 100x for 2 decades). Pluggable optics is not that much 
deprecated.




Not sure where your pricing is coming from, but if I look at Flexoptix's 
50Gbps QSFP28 optics pricing, I am getting:


 * EUR724 @ 10km.
 * EUR1,246 @ 40km.

They are also selling an SFP56 LR for EUR925.

Juxtapose that against 100Gbps pricing:

 * EUR473 @ 10km.
 * EUR1,300 @ 25km.
 * EUR1,500 @ 30km.
 * EUR2,600 @ 40km.
 * EUR3,925 @ 80km.

Doesn't immediately seem to me that 50Gbps is cheaper than 100Gbps. 
There also don't seem to be as many deployments of 50Gbps in the metro 
(same could be said for 25Gbps and 40Gbps), but others on the list can 
chime in with what they are seeing/doing.



I do not think that content provider guys call their DCI “Metro”, not 
very often.




Well, whatever they call it, the concept is the same - move lots of 
traffic across town between data centres.



I agree that 100GE for DCI is the minimum, 400GE is probably already 
needed in some places.


IMHO: it is a different story. Very interested too.



Most content providers have no choice but to run DWDM, for even very 
short spans between data centres. That is because it is just cheaper and 
simpler to pack Tbps of capacity in DWDM for the price than you can in a 
router. And besides, most routers don't need to carry Tbps of traffic in 
a single line card, which would be a waste of a fibre pair over that 
distance.


In such cases, better to use DWDM and drop capacity on individual 
routers and/or line cards as you see fit.



PS: By the way, even if some ISP has 50% of revenue from Enterprise 
services (it is probably the biggest number, typically 30%-40%), it is 
still just 5% compare to residential traffic. Traffic to enterprises 
is still sold 4x-10x (depending on the country).




That is why residential Access networks tend to be 2nd class citizens :-).


Hence, Enterprise does not make sense to mention in the traffic 
discussion. It is a “rounding error”.


Enterprise business created a huge demand for oversubscribed ports to 
connect Enterprises. And QoS/QoE. Not traffic.




Well, not all operators that offer enterprise services also do consumer 
broadband, or vice versa. To a network doing only one or the other, 
whatever traffic they are carrying means the world to them. It's not 
ours to decide what is high or low traffic... that priviledge always 
remains with the network operator.


Mark.

RE: Routed optical networks

2023-05-05 Thread Vasilenko Eduard via NANOG

You are right, my “Metro” definition is about ISP/Carriers.
Mobile or Fixed, despite pure Mobiles would like to call it MBH – it has much 
less traffic (Mobile subscribers would always have 7x less than fixed).
It is still the place where the majority of port capacity lives. Because all 
content and cache are after this link.

50GE is better just because it is half of the cost of 100GE and it is enough 
now for the great majority of cases. Money is very important these days for 
this industry. 100GE single mode is more expensive than the best router port 
itself. Routers have been deprecated 10x for the decade (almost 100x for 2 
decades). Pluggable optics is not that much deprecated.

I do not think that content provider guys call their DCI “Metro”, not very 
often.
I agree that 100GE for DCI is the minimum, 400GE is probably already needed in 
some places.
IMHO: it is a different story. Very interested too.

PS: By the way, even if some ISP has 50% of revenue from Enterprise services 
(it is probably the biggest number, typically 30%-40%), it is still just 5% 
compare to residential traffic. Traffic to enterprises is still sold 4x-10x 
(depending on the country).
Hence, Enterprise does not make sense to mention in the traffic discussion. It 
is a “rounding error”.
Enterprise business created a huge demand for oversubscribed ports to connect 
Enterprises. And QoS/QoE. Not traffic.

Eduard
From: NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org] On 
Behalf Of Mark Tinka
Sent: Friday, May 5, 2023 11:13 AM
To: nanog@nanog.org
Subject: Re: Routed optical networks


On 5/5/23 07:57, Vasilenko Eduard via NANOG wrote:
Disclaimer: Metaverse has not changed Metro traffic yet. Then …

I am puzzled when people talk about 400GE and Tbps in the Mero context.
For historical reasons, Metro is still about 2*2*10GE (one “2” for redundancy, 
another “2” for capacity) in the majority of cases worldwide.
How many BRASes serve more than 4/1.5=27k users in the busy hour?
It means that 50GE is the best interface now for the majority of cases. 
2*50GE=100Gbps is good room for growth.
Of course, exceptions could be. I know BRAS that handles 86k subscribers (do 
not recommend anybody to push the limits – it was so painful).

We have just 2 eyes and look at video content about 22h per week (on average). 
Our eyes do not permit us to see resolution better than particular for chosen 
distance (4k for typical TV, HD for smartphones, and so on). Color depth 10bits 
is enough for the majority, 12bits is sure enough for everybody. 120 frames/sec 
is enough for everybody. It would never change – it is our genetics.
Fortunately for Carriers, the traffic has a limit. You have probably seen that 
every year traffic growth % is decreasing. The Internet is stabilizing and 
approaching the plateau.
How much growth is still awaiting us? 1.5? 1.4? It needs separate research. The 
result would be tailored for whom would pay for the research.
IMHO: It is not mandatory that 100GE would become massive in the metro. (I know 
that 100GE is already massive in the DC CLOS)

Additionally, who would pay for this traffic growth? It also limits traffic at 
some point.
I hope it would happen after we would get our 22h/4k/12bit/120hz.

Now, you could argue that Metaverse would jump and multiply traffic by an 
additional 2x or 3x. Then 400GE may be needed.
Sorry, but it is speculation yet. It is not a trend like the current 
(declining) traffic growth.

So, it depends on what "metro" means to you.

For an ISP selling connectivity to enterprise customers, it can be a bunch of 
Metro-E routers deployed in various commercial buildings within a city. For a 
content provider, it could be DCI. For a telco, it could interconnecting their 
Active-E/GPON/DSLAM/CMTS network.

Whatever the case, the need for 100Gbps is going to be driven by the cost of 
optics over the distance required. Some operators run 2x 10Gbps for 
resilience/redundancy, while some others run 4x 10Gbps for the same. It all 
depends on the platform you are using. At some point, that capacity runs out, 
especially when you account for fibre outages, and you need something larger on 
one side of the ring mainly to provide sufficient bandwidth during failure 
events on the other side of the ring, and not necessarily because you are 
growing by that much.

Also, if the optics are available and are reasonably priced, why muck around 
with 40Gbps when you can just go straight to 100Gbps? The equipment usually can 
support either.

I'm unaware of any popularity around 50Gbps interfaces, but I also probably 
don't pay too much attention to such nuance :-).

So, it's not that we are seeing organic growth that justifies 100Gbps over 
anything smaller. It's more that the optics are available, they are cheap, they 
can go the distance, and the routers/switches can do the speed. At least, for 
us anyway, that is what is driving the next phase of our Metro-E network... 
going str

Re: Routed optical networks




On 5/5/23 07:57, Vasilenko Eduard via NANOG wrote:


Disclaimer: Metaverse has not changed Metro traffic yet. Then …

I am puzzled when people talk about 400GE and Tbps in the Mero context.

For historical reasons, Metro is still about 2*2*10GE (one “2” for 
redundancy, another “2” for capacity) in the majority of cases worldwide.


How many BRASes serve more than 4/1.5=27k users in the busy hour?

It means that 50GE is the best interface now for the majority of 
cases. 2*50GE=100Gbps is good room for growth.


Of course, exceptions could be. I know BRAS that handles 86k 
subscribers (do not recommend anybody to push the limits – it was so 
painful).


We have just 2 eyes and look at video content about 22h per week (on 
average). Our eyes do not permit us to see resolution better than 
particular for chosen distance (4k for typical TV, HD for smartphones, 
and so on). Color depth 10bits is enough for the majority, 12bits is 
sure enough for everybody. 120 frames/sec is enough for everybody. It 
would never change – it is our genetics.


Fortunately for Carriers, the traffic has a limit. You have probably 
seen that every year traffic growth % is decreasing. The Internet is 
stabilizing and approaching the plateau.


How much growth is still awaiting us? 1.5? 1.4? It needs separate 
research. The result would be tailored for whom would pay for the 
research.


IMHO: It is not mandatory that 100GE would become massive in the 
metro. (I know that 100GE is already massive in the DC CLOS)


Additionally, who would pay for this traffic growth? It also limits 
traffic at some point.


I hope it would happen after we would get our 22h/4k/12bit/120hz.

Now, you could argue that Metaverse would jump and multiply traffic by 
an additional 2x or 3x. Then 400GE may be needed.


Sorry, but it is speculation yet. It is not a trend like the current 
(declining) traffic growth.




So, it depends on what "metro" means to you.

For an ISP selling connectivity to enterprise customers, it can be a 
bunch of Metro-E routers deployed in various commercial buildings within 
a city. For a content provider, it could be DCI. For a telco, it could 
interconnecting their Active-E/GPON/DSLAM/CMTS network.


Whatever the case, the need for 100Gbps is going to be driven by the 
cost of optics over the distance required. Some operators run 2x 10Gbps 
for resilience/redundancy, while some others run 4x 10Gbps for the same. 
It all depends on the platform you are using. At some point, that 
capacity runs out, especially when you account for fibre outages, and 
you need something larger on one side of the ring mainly to provide 
sufficient bandwidth during failure events on the other side of the 
ring, and not necessarily because you are growing by that much.


Also, if the optics are available and are reasonably priced, why muck 
around with 40Gbps when you can just go straight to 100Gbps? The 
equipment usually can support either.


I'm unaware of any popularity around 50Gbps interfaces, but I also 
probably don't pay too much attention to such nuance :-).


So, it's not that we are seeing organic growth that justifies 100Gbps 
over anything smaller. It's more that the optics are available, they are 
cheap, they can go the distance, and the routers/switches can do the 
speed. At least, for us anyway, that is what is driving the next phase 
of our Metro-E network... going straight from 10Gbps to 100Gbps links, 
not because that is the growth we are seeing from an organic traffic 
standpoint, but because the routers can do it, and it offers us peace of 
mind that we can handle any traffic re-route when one half of the ring 
fails, without dropping packets.


The 400Gbps market will be restricted to mainly content folk linking up 
data centres, as well as some large telco's, for the time being. It is 
not likely to be the norm for the majority of operators who run some 
kind of metro network.


Mark.

RE: Routed optical networks

Disclaimer: Metaverse has not changed Metro traffic yet. Then ...

I am puzzled when people talk about 400GE and Tbps in the Mero context.
For historical reasons, Metro is still about 2*2*10GE (one "2" for redundancy, 
another "2" for capacity) in the majority of cases worldwide.
How many BRASes serve more than 4/1.5=27k users in the busy hour?
It means that 50GE is the best interface now for the majority of cases. 
2*50GE=100Gbps is good room for growth.
Of course, exceptions could be. I know BRAS that handles 86k subscribers (do 
not recommend anybody to push the limits - it was so painful).

We have just 2 eyes and look at video content about 22h per week (on average). 
Our eyes do not permit us to see resolution better than particular for chosen 
distance (4k for typical TV, HD for smartphones, and so on). Color depth 10bits 
is enough for the majority, 12bits is sure enough for everybody. 120 frames/sec 
is enough for everybody. It would never change - it is our genetics.
Fortunately for Carriers, the traffic has a limit. You have probably seen that 
every year traffic growth % is decreasing. The Internet is stabilizing and 
approaching the plateau.
How much growth is still awaiting us? 1.5? 1.4? It needs separate research. The 
result would be tailored for whom would pay for the research.
IMHO: It is not mandatory that 100GE would become massive in the metro. (I know 
that 100GE is already massive in the DC CLOS)

Additionally, who would pay for this traffic growth? It also limits traffic at 
some point.
I hope it would happen after we would get our 22h/4k/12bit/120hz.

Now, you could argue that Metaverse would jump and multiply traffic by an 
additional 2x or 3x. Then 400GE may be needed.
Sorry, but it is speculation yet. It is not a trend like the current 
(declining) traffic growth.

Ed/
From: NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org] On 
Behalf Of Phil Bedard
Sent: Thursday, May 4, 2023 8:32 PM
To: Etienne-Victor Depasquale ; NANOG 
Subject: Re: Routed optical networks

It's not necessarily metro specific although the metro networks could lend 
themselves to overall optimizations.

The adoption of ZR/ZR+ IPoWDM currently somewhat corresponds with your adoption 
of 400G since today they require a QDD port.   There are 100G QDD ports but 
that's not all that popular yet.   Of course there is work to do something 
similar in QSFP28 if the power can be reduced to what is supported by an 
existing QSFP28 port in most devices.   In larger networks with higher speed 
requirements and moving to 400G with QDD, using the DCO optics for connecting 
routers is kind of a no-brainer vs. a traditional muxponder.   Whether that's 
over a ROADM based optical network or not, especially at metro/regional 
distances.

There are very large deployments of IPoDWDM over passive DWDM or dark fiber for 
access and aggregation networks where the aggregate required bandwidth doesn't 
exceed the capabilities of those optics.  It's been done at 10G for many years. 
 With the advent of pluggable EDFA amplifiers, you can even build links up to 
120km* (perfect dark fiber)  carrying tens of terabits of traffic without any 
additional active optical equipment.

It's my personal opinion we aren't to the days yet of where we can simply build 
an all packet network with no photonic switching that carries all services, but 
eventually (random # of years) it gets there for many networks.  There are also 
always going to be high performance applications for transponders where 
pluggable optics aren't a good fit.

Carrying high speed private line/wavelength type services as well is a 
different topic than interconnecting IP devices.

Thanks,
Phil


From: NANOG 
mailto:nanog-bounces+bedard.phil=gmail@nanog.org>>
 on behalf of Etienne-Victor Depasquale via NANOG 
mailto:nanog@nanog.org>>
Date: Monday, May 1, 2023 at 2:30 PM
To: NANOG mailto:nanog@nanog.org>>
Subject: Routed optical networks
Hello folks,

Simple question: does "routed optical networks" have a clear meaning in the 
metro area context, or not?

Put differently: does it call to mind a well-defined stack of technologies in 
the control and data planes of metro-area networks?

I'm asking because I'm having some thoughts about the clarity of this term, in 
the process of carrying out a qualitative survey of the results of the 
metro-area networks survey.

Cheers,

Etienne

--
Ing. Etienne-Victor Depasquale
Assistant Lecturer
Department of Communications & Computer Engineering
Faculty of Information & Communication Technology
University of Malta
Web. https://www.um.edu.mt/profile/etiennedepasquale

Re: Routed optical networks




On 5/4/23 19:32, Phil Bedard wrote:

It’s my personal opinion we aren’t to the days yet of where we can 
simply build an all packet network with no photonic switching that 
carries all services, but eventually (random # of years) it gets there 
for many networks.  There are also always going to be high performance 
applications for transponders where pluggable optics aren’t a good fit.




I think every time the IP space gets close to running an all-packet 
network, the Transport folk come out with an easier way to do it, that 
it's too hard to ignore.


Based on that, I think they will always be one step ahead, with the key 
advantage being reliability of capacity over the distance, for the cost.


The farther your fibre has to run, the costlier it gets to do it without 
DWDM.


I mean, it's only now that 100G-ZR is becoming a reality for packet 
networks, and we are talking thousands of US$ for optics to get us 80km 
- 120km distance. Meanwhile, DWDM vendors can get you 800Gbps per 
wavelength in the same distance (or 30X that distance) far less cheaply.


I get the appeal of not needing DWDM gear to underlay your packet 
network... it's neater and offers fewer points of failure. But unless 
you are dealing with very short distances and can ride a reasonable 
balance between service features and switching/forwarding capacity in 
your router/switch, it's going to be hard to ignore the DWDM gear if you 
are trying to be a serious operation, at that scale, over a wide area.


Mark.

Re: Routed optical networks





On 5/4/23 15:03, Jared Mauch wrote:


I’m a bit shocked that I now need a 288F cable on some of my routes to support 
future expansion, but that fiber cost is still small compared to the labor.


Yes, labour is generally the cost.

And then way-leaves add cost in terms of time and lost opportunities.

Mark.

Re: Routed optical networks





On 5/4/23 14:27, Vasilenko Eduard wrote:


I had an experience in one big PTT. Fiber was easy in the majority of Metro 
places.
Even faster than DWDM or router commissioning.
It is just 1 PTT. Hence, an example could not be counted.


Yeah - I usually tend to look at what happens in the majority of cases.

There is always at least one example in an exception, but it still 
remains an exception.


Mark.

Re: Routed optical networks

2023-05-04 Thread Phil Bedard

It’s not necessarily metro specific although the metro networks could lend 
themselves to overall optimizations.

The adoption of ZR/ZR+ IPoWDM currently somewhat corresponds with your adoption 
of 400G since today they require a QDD port.   There are 100G QDD ports but 
that’s not all that popular yet.   Of course there is work to do something 
similar in QSFP28 if the power can be reduced to what is supported by an 
existing QSFP28 port in most devices.   In larger networks with higher speed 
requirements and moving to 400G with QDD, using the DCO optics for connecting 
routers is kind of a no-brainer vs. a traditional muxponder.   Whether that’s 
over a ROADM based optical network or not, especially at metro/regional 
distances.

There are very large deployments of IPoDWDM over passive DWDM or dark fiber for 
access and aggregation networks where the aggregate required bandwidth doesn’t 
exceed the capabilities of those optics.  It’s been done at 10G for many years. 
 With the advent of pluggable EDFA amplifiers, you can even build links up to 
120km* (perfect dark fiber)  carrying tens of terabits of traffic without any 
additional active optical equipment.

It’s my personal opinion we aren’t to the days yet of where we can simply build 
an all packet network with no photonic switching that carries all services, but 
eventually (random # of years) it gets there for many networks.  There are also 
always going to be high performance applications for transponders where 
pluggable optics aren’t a good fit.

Carrying high speed private line/wavelength type services as well is a 
different topic than interconnecting IP devices.

Thanks,
Phil


From: NANOG  on behalf of 
Etienne-Victor Depasquale via NANOG 
Date: Monday, May 1, 2023 at 2:30 PM
To: NANOG 
Subject: Routed optical networks
Hello folks,

Simple question: does "routed optical networks" have a clear meaning in the 
metro area context, or not?

Put differently: does it call to mind a well-defined stack of technologies in 
the control and data planes of metro-area networks?

I'm asking because I'm having some thoughts about the clarity of this term, in 
the process of carrying out a qualitative survey of the results of the 
metro-area networks survey.

Cheers,

Etienne

--
Ing. Etienne-Victor Depasquale
Assistant Lecturer
Department of Communications & Computer Engineering
Faculty of Information & Communication Technology
University of Malta
Web. https://www.um.edu.mt/profile/etiennedepasquale

Re: Routed optical networks

2023-05-04 Thread Jared Mauch

> On May 4, 2023, at 6:21 AM, Mark Tinka  wrote:
> 
> 
> 
> On 5/4/23 11:40, Denis Fondras wrote:
> 
>> 
>> You may also take into account the time to deliver.
>> Laying fiber takes much more time than plugging a colored optic.
> 
> Indeed - part of the expense of running new fibre is the time it takes to 
> start making money from it.
> 

One of the advantages in the US right now is that capital cost is being 
subsidized or reimbursed by state or federal government, leaving room to 
expand.  Some providers have missed out on this, and others have capitalized on 
it.  It’s very much a mix of people who are chasing that $ and those that are 
not.  

I will say that merchant silicon has it’s place, but so does the vendor 
silicon.  At some point if you get the fiber to that location, unlocking the 
capacity becomes much easier with CEx or similar modules to overlay services.

Making the choice to build a quality fiber first network is important, and why 
I have already had to take routes that I had planned lower count fibers on and 
upgrade them.  I’m a bit shocked that I now need a 288F cable on some of my 
routes to support future expansion, but that fiber cost is still small compared 
to the labor.

- Jared

Re: Routed optical networks

2023-05-04 Thread Ge DUPIN

Many Big PTTs have a lot of ducts in many places, it is easy for them to lay 
fibers, especially in cities.

Ge

> Le 4 mai 2023 à 14:27, Vasilenko Eduard via NANOG  a écrit :
> 
> I had an experience in one big PTT. Fiber was easy in the majority of Metro 
> places.
> Even faster than DWDM or router commissioning.
> It is just 1 PTT. Hence, an example could not be counted.
> Eduard
> -Original Message-
> From: NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org] On 
> Behalf Of Mark Tinka
> Sent: Thursday, May 4, 2023 2:11 PM
> To: nanog@nanog.org
> Subject: Re: Routed optical networks
> 
> 
> 
> On 5/4/23 12:58, Vasilenko Eduard via NANOG wrote:
> 
>> Well, ISP is typically plan something for a year. It is more than enough for 
>> both.
> 
> The real world is much less certain, especially in these economic times.
> 
> 
>> Funny, that with the current lead times for electronics, Fiber could be 
>> faster.
>> Of course, it is a temporary glitch.
> 
> Even in the same market, no two lays of fibre can be guaranteed to be 
> completed in the same time.
> 
> Mark.
>

RE: Routed optical networks

> The economics are such these days that in many circumstances, bean counters 
> don't want to hear about payback in years, they want to hear it in quarters.  
> Short term financial thinking is dominant.
True. The industry is on decline. On the way to other utilities.
But then any project is a challenge. Not just fiber that may be cheaper for 
Metro than DWDM.
Eduard
From: Tom Beecher [mailto:beec...@beecher.cc]
Sent: Thursday, May 4, 2023 3:26 PM
To: Vasilenko Eduard 
Cc: Denis Fondras ; nanog@nanog.org
Subject: Re: Routed optical networks

Well, ISP is typically plan something for a year. It is more than enough for 
both.

s/more/should be/

The economics are such these days that in many circumstances, bean counters 
don't want to hear about payback in years, they want to hear it in quarters.  
Short term financial thinking is dominant.

On Thu, May 4, 2023 at 6:59 AM Vasilenko Eduard via NANOG 
mailto:nanog@nanog.org>> wrote:
Well, ISP is typically plan something for a year. It is more than enough for 
both.

Funny, that with the current lead times for electronics, Fiber could be faster.
Of course, it is a temporary glitch.

Ed/
-Original Message-
From: NANOG 
[mailto:nanog-bounces+vasilenko.eduard<mailto:nanog-bounces%2Bvasilenko.eduard>=huawei@nanog.org<mailto:huawei@nanog.org>]
 On Behalf Of Denis Fondras
Sent: Thursday, May 4, 2023 12:41 PM
To: nanog@nanog.org<mailto:nanog@nanog.org>
Subject: Re: Routed optical networks

Le Wed, May 03, 2023 at 06:20:48AM +, Vasilenko Eduard via NANOG a écrit :
>
> Additionally, I am sure that in many countries/Metro it is cheaper to lay 
> down a new fiber than to provision DWDM, even if it is a pizza box. The 
> colored interface is still very expensive.
> Of course, there are some Cities (not “towns”) where it is very expensive or 
> maybe even impossible to lay down a new fiber.
> Yes, in the majority of cases, it is cheaper to lay down fiber.
>

You may also take into account the time to deliver.
Laying fiber takes much more time than plugging a colored optic.

RE: Routed optical networks

I had an experience in one big PTT. Fiber was easy in the majority of Metro 
places.
Even faster than DWDM or router commissioning.
It is just 1 PTT. Hence, an example could not be counted.
Eduard
-Original Message-
From: NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org] On 
Behalf Of Mark Tinka
Sent: Thursday, May 4, 2023 2:11 PM
To: nanog@nanog.org
Subject: Re: Routed optical networks



On 5/4/23 12:58, Vasilenko Eduard via NANOG wrote:

> Well, ISP is typically plan something for a year. It is more than enough for 
> both.

The real world is much less certain, especially in these economic times.


> Funny, that with the current lead times for electronics, Fiber could be 
> faster.
> Of course, it is a temporary glitch.

Even in the same market, no two lays of fibre can be guaranteed to be completed 
in the same time.

Mark.

Re: Routed optical networks

2023-05-04 Thread Tom Beecher

>
> Well, ISP is typically plan something for a year. It is more than enough
> for both.
>

s/more/should be/

The economics are such these days that in many circumstances, bean counters
don't want to hear about payback in years, they want to hear it in
quarters.  Short term financial thinking is dominant.

On Thu, May 4, 2023 at 6:59 AM Vasilenko Eduard via NANOG 
wrote:

> Well, ISP is typically plan something for a year. It is more than enough
> for both.
>
> Funny, that with the current lead times for electronics, Fiber could be
> faster.
> Of course, it is a temporary glitch.
>
> Ed/
> -Original Message-
> From: NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org]
> On Behalf Of Denis Fondras
> Sent: Thursday, May 4, 2023 12:41 PM
> To: nanog@nanog.org
> Subject: Re: Routed optical networks
>
> Le Wed, May 03, 2023 at 06:20:48AM +, Vasilenko Eduard via NANOG a
> écrit :
> >
> > Additionally, I am sure that in many countries/Metro it is cheaper to
> lay down a new fiber than to provision DWDM, even if it is a pizza box. The
> colored interface is still very expensive.
> > Of course, there are some Cities (not “towns”) where it is very
> expensive or maybe even impossible to lay down a new fiber.
> > Yes, in the majority of cases, it is cheaper to lay down fiber.
> >
>
> You may also take into account the time to deliver.
> Laying fiber takes much more time than plugging a colored optic.
>
>

Re: Routed optical networks





On 5/4/23 12:58, Vasilenko Eduard via NANOG wrote:


Well, ISP is typically plan something for a year. It is more than enough for 
both.


The real world is much less certain, especially in these economic times.



Funny, that with the current lead times for electronics, Fiber could be faster.
Of course, it is a temporary glitch.


Even in the same market, no two lays of fibre can be guaranteed to be 
completed in the same time.


Mark.

RE: Routed optical networks

Well, ISP is typically plan something for a year. It is more than enough for 
both.

Funny, that with the current lead times for electronics, Fiber could be faster.
Of course, it is a temporary glitch.

Ed/
-Original Message-
From: NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org] On 
Behalf Of Denis Fondras
Sent: Thursday, May 4, 2023 12:41 PM
To: nanog@nanog.org
Subject: Re: Routed optical networks

Le Wed, May 03, 2023 at 06:20:48AM +, Vasilenko Eduard via NANOG a écrit :
> 
> Additionally, I am sure that in many countries/Metro it is cheaper to lay 
> down a new fiber than to provision DWDM, even if it is a pizza box. The 
> colored interface is still very expensive.
> Of course, there are some Cities (not “towns”) where it is very expensive or 
> maybe even impossible to lay down a new fiber.
> Yes, in the majority of cases, it is cheaper to lay down fiber.
> 

You may also take into account the time to deliver.
Laying fiber takes much more time than plugging a colored optic.

Re: Routed optical networks





On 5/4/23 11:40, Denis Fondras wrote:



You may also take into account the time to deliver.
Laying fiber takes much more time than plugging a colored optic.


Indeed - part of the expense of running new fibre is the time it takes 
to start making money from it.


Mark.

Re: Routed optical networks

2023-05-04 Thread Denis Fondras

Le Wed, May 03, 2023 at 06:20:48AM +, Vasilenko Eduard via NANOG a écrit :
> 
> Additionally, I am sure that in many countries/Metro it is cheaper to lay 
> down a new fiber than to provision DWDM, even if it is a pizza box. The 
> colored interface is still very expensive.
> Of course, there are some Cities (not “towns”) where it is very expensive or 
> maybe even impossible to lay down a new fiber.
> Yes, in the majority of cases, it is cheaper to lay down fiber.
> 

You may also take into account the time to deliver.
Laying fiber takes much more time than plugging a colored optic.

Re: Routed optical networks

2023-05-03 Thread Mark Tinka




On 5/3/23 11:10, Vasilenko Eduard wrote:


You are right. My message was pretty much geared toward incumbents.

But the majority of the access/aggregation is in their possessions, 
isn’t it?




Generally, I'd say yes.

But to the OP's survey, the incumbents also have the majority 
installations of copper, and are least likely to use MPLS all the way 
into the Access. They would more typically use 802.1Q or Q-in-Q.


Smaller operators green-fielding Access networks will rely mostly on 
fibre (particularly GPON) and MPLS all the way into the Access.


It's just that FTTx services are growing at a much faster rate than 
copper-based services are. Depending on the market, it may not always be 
the incumbent witnessing this growth, although when they do finally get 
their act together, they can make up for lost time rather quickly.



They typically have ducts that were huge for copper that is already 
extracted.


One more fiber cable would be easy.



Agreed.



Agree that for competitive carriers DWDM would be more often needed.

Even for competitive carriers, it makes sense to evaluate the cost to 
put fiber into the duct of incumbents.


Especially because in some countries the price would be regulated.

It would solve the problem forever – no need for the DWDM speed upgrade.



Well, the only issue with that is that some markets make it more 
difficult to re-open up the roads for several years. Some worse than 
others. In such a case, DWDM is your best option.



I am calling to just not forget to evaluate this option too. Reminder: 
dark fiber is the best technical solution, for sure.




In general, most operators, regardless of size, will prefer dark fibre 
as a first option, especially for short spans like in the metro. But of 
course, real life is vastly different.


Mark.

RE: Routed optical networks

2023-05-03 Thread Vasilenko Eduard via NANOG

> Yeah, you sound like an equipment vendor whose main customers are incumbent 
> telco's in a few rich markets :-).
You are right. My message was pretty much geared toward incumbents.
But the majority of the access/aggregation is in their possessions, isn’t it?
They typically have ducts that were huge for copper that is already extracted.
One more fiber cable would be easy.

Agree that for competitive carriers DWDM would be more often needed.
Even for competitive carriers, it makes sense to evaluate the cost to put fiber 
into the duct of incumbents.
Especially because in some countries the price would be regulated.
It would solve the problem forever – no need for the DWDM speed upgrade.

I am calling to just not forget to evaluate this option too. Reminder: dark 
fiber is the best technical solution, for sure.

Ed/
From: Mark Tinka [mailto:mark@tinka.africa]
Sent: Wednesday, May 3, 2023 11:39 AM
To: Vasilenko Eduard ; nanog@nanog.org
Subject: Re: Routed optical networks

On 5/3/23 08:20, Vasilenko Eduard wrote:
I would risk to say a little more on this.
Indeed, maybe the situation (in many countries) when the Carrier sells a lot of 
TDM services.
But in general, packet services are enough these days for many carriers/regions.

There aren't enough TDM services to warrant DWDM, nowadays.

The reason for DWDM is mainly being driven by Ethernet, and IP.

At any reasonable scale, it's actually pretty hard to buy a TDM service, in 
most markets.

Additionally, I am sure that in many countries/Metro it is cheaper to lay down 
a new fiber than to provision DWDM, even if it is a pizza box.

I disagree. Existing fibre may be cheap because it was laid down a decade or 
more ago, en masse, by several operators. So the market would be experiencing a 
glut, not because it is cheap to open up the roads and plant more fibre, but 
because there is so much of it to begin with.

At worst, there is still enough duct space that the operator can blow more 
fibre. But when that duct gets full, and there are no more free ducts 
available, or another route needs to get built for whatever reason, it is a 
rather costly affair to open up the roads and trunk some fibre, in any market.

So no, DWDM is not more expensive, if you are delivering services at scale. It 
is actually cheaper. It is only more expensive if you are small scale, because 
in some markets, the fibre glut means you can buy dark fibre for cheaper than 
you can light it with DWDM. But this is a situation unique to small operators, 
not large ones.

The colored interface is still very expensive.

This only matters for the line side.

For client-facing, it's not a drama. And you typically buy more optics for the 
client side than you do the line side.

Of course, there are some Cities (not “towns”) where it is very expensive or 
maybe even impossible to lay down a new fiber.
Yes, in the majority of cases, it is cheaper to lay down fiber.

I think what you mean to say is that in the majority of cases where there is 
fibre glut, and dark fibre is a market option, buying fibre is cheaper than 
lighting it with DWDM. This is true. But I think that on a global scale, this 
is the exception, not the rule.

In general, you are not likely to be able to buy dark fibre, cheaply or 
otherwise, if you look at all markets in the world.

Hence, the importance of DWDM for the Metro is overestimated.

Again, only if you are small scale.

If you are a large scale operator with as many IP/Ethernet customers as you 
have Transport, DWDM is essential.

Use only routers. Provision enough fiber. Have always 1 router hop to the 
aggregation (hub-spoke topology), no routers chaining in the ring.
If fiber is not enough – then use normal DWDM with an external transponder. 
Routers would be still in hub-spoke topology.

Yeah, you sound like an equipment vendor whose main customers are incumbent 
telco's in a few rich markets :-).

The life of the average operator, around the world, is far less glamorous.

Mark.

Re: Routed optical networks

2023-05-03 Thread Mark Tinka




On 5/3/23 08:20, Vasilenko Eduard wrote:


I would risk to say a little more on this.

Indeed, maybe the situation (in many countries) when the Carrier sells 
a lot of TDM services.


But in general, packet services are enough these days for many 
carriers/regions.




There aren't enough TDM services to warrant DWDM, nowadays.

The reason for DWDM is mainly being driven by Ethernet, and IP.

At any reasonable scale, it's actually pretty hard to buy a TDM service, 
in most markets.



Additionally, I am sure that in many countries/Metro it is cheaper to 
lay down a new fiber than to provision DWDM, even if it is a pizza box.




I disagree. Existing fibre may be cheap because it was laid down a 
decade or more ago, en masse, by several operators. So the market would 
be experiencing a glut, not because it is cheap to open up the roads and 
plant more fibre, but because there is so much of it to begin with.


At worst, there is still enough duct space that the operator can blow 
more fibre. But when that duct gets full, and there are no more free 
ducts available, or another route needs to get built for whatever 
reason, it is a rather costly affair to open up the roads and trunk some 
fibre, in any market.


So no, DWDM is not more expensive, if you are delivering services at 
scale. It is actually cheaper. It is only more expensive if you are 
small scale, because in some markets, the fibre glut means you can buy 
dark fibre for cheaper than you can light it with DWDM. But this is a 
situation unique to small operators, not large ones.




The colored interface is still very expensive.



This only matters for the line side.

For client-facing, it's not a drama. And you typically buy more optics 
for the client side than you do the line side.



Of course, there are some Cities (not “towns”) where it is very 
expensive or maybe even impossible to lay down a new fiber.


Yes, in the majority of cases, it is cheaper to lay down fiber.



I think what you mean to say is that in the majority of cases where 
there is fibre glut, and dark fibre is a market option, buying fibre is 
cheaper than lighting it with DWDM. This is true. But I think that on a 
global scale, this is the exception, not the rule.


In general, you are not likely to be able to buy dark fibre, cheaply or 
otherwise, if you look at all markets in the world.



Hence, the importance of DWDM for the Metro is overestimated.



Again, only if you are small scale.

If you are a large scale operator with as many IP/Ethernet customers as 
you have Transport, DWDM is essential.



Use only routers. Provision enough fiber. Have always 1 router hop to 
the aggregation (hub-spoke topology), no routers chaining in the ring.


If fiber is not enough – then use normal DWDM with an external 
transponder. Routers would be still in hub-spoke topology.




Yeah, you sound like an equipment vendor whose main customers are 
incumbent telco's in a few rich markets :-).


The life of the average operator, around the world, is far less glamorous.

Mark.

RE: Routed optical networks

2023-05-03 Thread Vasilenko Eduard via NANOG

> At that scale, DWDM in the metro will make sense
I would risk to say a little more on this.
Indeed, maybe the situation (in many countries) when the Carrier sells a lot of 
TDM services.
But in general, packet services are enough these days for many carriers/regions.

Additionally, I am sure that in many countries/Metro it is cheaper to lay down 
a new fiber than to provision DWDM, even if it is a pizza box. The colored 
interface is still very expensive.
Of course, there are some Cities (not “towns”) where it is very expensive or 
maybe even impossible to lay down a new fiber.
Yes, in the majority of cases, it is cheaper to lay down fiber.

Hence, the importance of DWDM for the Metro is overestimated.

Use only routers. Provision enough fiber. Have always 1 router hop to the 
aggregation (hub-spoke topology), no routers chaining in the ring.
If fiber is not enough – then use normal DWDM with an external transponder. 
Routers would be still in hub-spoke topology.

Ed/
From: NANOG [mailto:nanog-bounces+vasilenko.eduard=huawei@nanog.org] On 
Behalf Of Mark Tinka
Sent: Wednesday, May 3, 2023 7:09 AM
To: nanog@nanog.org
Subject: Re: Routed optical networks


On 5/2/23 07:28, Vasilenko Eduard via NANOG wrote:
The incumbent carrier typically has enough fiber strands to avoid any colored 
interfaces (that are 3x expensive compare to gray) in the Metro.
Metro ring typically has 8-10 nodes (or similar). 16-20 strands of fiber were 
not possible to construct anyway – any cable is bigger.
It is the same cost to lay down fiber on 16 strands or 32.
Hence, PTT just does not need DWDM in Metro, not at all. Hence, the DWDM 
optimization that you are talking about below is not needed too.

This may or may not always be the case. Especially for large carriers, where 
there could be a requirement to sell some of those dark fibre pairs to large 
customers (think the content folk coming into town, e.t.c.), they may no longer 
have the priviledge of having plenty of free fibre in the metro. Or if they 
did, the rate of traffic expansion means they burn through those fibre pairs 
pretty quick.

10Gbps isn't a lot nowadays, and 100Gbps may start to push the limits depending 
on the size of the operator, the scope of the Metro-E ring and the level of 
service that needs to be maintained during a re-route (two available paths in 
the ring could balance 100Gbps of traffic, but if one half of that ring breaks, 
the remaining path may need to carry a lot more than 100Gbps, and then packets 
start to fall flat on the floor).

At that scale, DWDM in the metro will make sense, at least more sense than 
400G-ZR, at the moment.




If you rent a single pair of fiber then you need colored interfaces to 
multiplex 8-10 nodes into 1 pair on the ring.
Then the movement of transponders from DWDM into the router would eliminate 2 
gray interfaces on every node (4 per link): one on the router side, and another 
on the DWDM side.
Overall, it is about a 25% cost cut of the whole “router+DWDM”.

Some operators would also be selling Transport services in or along the metro, 
and customers paying for that may require that they do not cross a router 
device.



It is still 2x more expensive compare to using additional fiber strands on YOUR 
fiber.

There are plenty of DWDM pizza boxes that cost next to nothing. At scale, the 
price of these is not a stumbling block. And certainly, the price of these 
would be far lower than a router line card.




By the way, about “well-defined stack of technologies”:
NMS (polished by SDN our days) should be cross-layer: it should manage at the 
same time: ROADM/OADM in DWDM and colored laser in Router.
It is a vendor lock up to now (no multi-vendor). Hence, 25% cost savings would 
go to the vendor that has such NMS, not to the carrier.
Technology still does not make sense because no multivendor support between the 
NMS of one vendor and the router or DWDM of another.
Looking at the NMS history, it would probably never be multi-vendor. For that 
reason, I am pessimistic about the future of the colored interfaces in routers 
(and alien lambdas in DWDM). Despite a potential 25% cost advantage in 
eliminating gray interfaces.

OpenROADM is a good initiative. But it seems it's to be to Transport equipment 
vendors what IPv6 and DNSSEC is to the IP world :-).

Mark.

Re: Routed optical networks





On 5/2/23 22:28, Eve Griliches wrote:

So right Jaredmagic has been in the NPU capacity increase that's 
driven the cost per 100G down on 1RU routers;


But that has only mainly solved for speed. Features have taken a hit, 
especially if the operator is motivated by the costs of merchant silicon.


There has been a marked improvement of features from merchant silicon, 
both from their vendors as well as the router OEM's that implement them 
in clever ways to work around their restrictions, but there are still 
some things only in-house silicon can do, at a price point most 
operators are not comfortable to pay anymore.


I think that as more of the Internet collapses into the hands of a few 
public cloud and content providers, operators are likely to place less 
and less importance on features, and just focus on speed, since the 
public Internet offers very little guarantees, if not none at all. I'm 
keen to see how this pans out.



and the integration of DSPs and more into QSFP-DD form factors at 
much lower power than expected.


Coherent has certainly changed the game, no doubt. If you grow steadily, 
you can wait for the evolution to make it into the IP/MPLS. If you need 
to move faster, you can't ignore the importance of Transport options in 
your network.


Mark.

Re: Routed optical networks

On 5/2/23 16:01, Eve Griliches wrote:

Hi Etienne,
Below is our (Cisco) definition of the Routed Optical Network. The
goal, metro or long haul or subsea, is to reduce the number of control
planes. By migration TDM traffic using CEM or PLE to the IP layer, you
eliminate the OTN control plane and management. Eventually, when
standards are settled the ultimate goal is to have a single
control plane for the network. I'm not trying to be a commercial here,
but you can read more in the resources section on this page:
https://www.cisco.com/c/en/us/solutions/service-provider/routed-optical-networking/index.html

HTH,
Eve

Routed optical networking, is an architecture that delivers improved
operational efficiencies and simplicity. The solution works by merging
IP and private line services onto a single layer where all the
switching is done at Layer 3. Routers are connected with standardized
400G ZR/ZR+ coherent pluggable optics.

With a single service layer based upon IP, flexible management tools
can leverage telemetry and model-driven programmability to streamline
lifecycle operations. This simplified architecture integrates open
data models and standard APIs, enabling a provider to focus on
automation initiatives for a simpler topology.

To be honest, I've been hearing about this since as long as I can
remember. IPoDWDM was another attempt at trying to make the above a reality.

But for some reason, operators prefer to keep these networks separate,
and many customers, especially very large ones, prefer to bypass routers
for their Transport services.

I think the effort will be appreciated, but if history is anything to go
by, vendors are going to struggle to strip operators and customers away
from some degree of separation.

Mark.

Re: Routed optical networks





On 5/2/23 21:32, Jared Mauch wrote:


I’ve seen proposals for an LSR MPLS/ROADAM type solution, where imagine you are 
at a hop where in a long distance system solution, you would end up with OEO, 
but instead you get directionality capability with an IP/MPLS capable device.


My memory is rather fuzzy, but didn't Juniper attempt something like 
this in their PTX's after they picked up BTI? I think the plan was to 
co-locate the ROADM at the bottom of the PTX chassis, or something along 
those lines.


I know Cisco (and Juniper) tried by integrating GMPLS into their code as 
a starting point, but that didn't go very far with customers. It just 
seemed impossible for the Transport teams to allow the IP/MPLS teams 
that level of access into their line system :-).




   As mentioned previously, the 400-ZR/ZR+/ZR-Bright/+0 optics are the latest 
example of that.


A rather high barrier to entry for most operators, but we have to start 
from somewhere.



I know of a few companies that have looked at solutions like this, and can 
expect there to be some interesting solutions that would appear as a result.  
Optical line systems tend to have pretty low power requirements compared to a 
router, but some of the routers are getting pretty low power as well when it 
comes to the power OPEX/bit, and if you have the ability to deliver services as 
an integrated packet optical you could see reduced costs and simplified 
components/sparing.


The main problem is distance.

If you need to move that kind of capacity more than 50km, it's hard to 
avoid DWDM.




I’ll also say that I’ve not yet seen the price compression that I had expected 
in the space yet, but I figure that’s coming.  We are seeing the bits/watt 
ratio improve though, so for the same or less power consumption you get more 
bits.  Some of this technology stuff is truly magical.


I think for long spans, DWDM will not only be cheaper, but the only 
feasible solution.


For the metro, it will come down to what motivates the business... 
plenty of features, or plenty of speed.


Also, DWDM vendors are adding speed and distance faster and cheaper than 
the IP/MPLS vendors can. So they will always be one step ahead in that 
respect; and we have the submarine cable systems to thank for that.


Mark.

Re: Routed optical networks





On 5/2/23 16:25, Izaac wrote:


This is a very convoluted way of backing into the ole packet-switched
vs. circuit switched decision.


A fight that will never go away.

There has been some compromise in recent years, with Transport-heavy 
customers accepting standard Ethernet services, but only if they are 
carried by a Transport device.


Mark.

Re: Routed optical networks