Re: Routed optical networks

[Mark Tinka]

On 5/2/23 22:28, Eve Griliches wrote:

So right Jaredmagic has been in the NPU capacity increase that's 
driven the cost per 100G down on 1RU routers;


But that has only mainly solved for speed. Features have taken a hit, 
especially if the operator is motivated by the costs of merchant silicon.


There has been a marked improvement of features from merchant silicon, 
both from their vendors as well as the router OEM's that implement them 
in clever ways to work around their restrictions, but there are still 
some things only in-house silicon can do, at a price point most 
operators are not comfortable to pay anymore.


I think that as more of the Internet collapses into the hands of a few 
public cloud and content providers, operators are likely to place less 
and less importance on features, and just focus on speed, since the 
public Internet offers very little guarantees, if not none at all. I'm 
keen to see how this pans out.



and the integration of DSPs and more into QSFP-DD form factors at 
much lower power than expected.


Coherent has certainly changed the game, no doubt. If you grow steadily, 
you can wait for the evolution to make it into the IP/MPLS. If you need 
to move faster, you can't ignore the importance of Transport options in 
your network.


Mark.

Re: Routed optical networks

[Mark Tinka]

On 5/2/23 16:01, Eve Griliches wrote:


Hi Etienne,
Below is our (Cisco) definition of the Routed Optical Network. The 
goal, metro or long haul or subsea, is to reduce the number of control 
planes. By migration TDM traffic using CEM or PLE to the IP layer, you 
eliminate the OTN control plane and management. Eventually, when 
standards are settled the ultimate goal is to have a single 
control plane for the network. I'm not trying to be a commercial here, 
but you can read more in the resources section on this page: 
https://www.cisco.com/c/en/us/solutions/service-provider/routed-optical-networking/index.html

HTH,
Eve

Routed optical networking, is an architecture that delivers improved 
operational efficiencies and simplicity. The solution works by merging 
IP and private line services onto a single layer where all the 
switching is done at Layer 3. Routers are connected with standardized 
400G ZR/ZR+ coherent pluggable optics.


With a single service layer based upon IP, flexible management tools 
can leverage telemetry and model-driven programmability to streamline 
lifecycle operations. This simplified architecture integrates open 
data models and standard APIs, enabling a provider to focus on 
automation initiatives for a simpler topology.





To be honest, I've been hearing about this since as long as I can 
remember. IPoDWDM was another attempt at trying to make the above a reality.


But for some reason, operators prefer to keep these networks separate, 
and many customers, especially very large ones, prefer to bypass routers 
for their Transport services.


I think the effort will be appreciated, but if history is anything to go 
by, vendors are going to struggle to strip operators and customers away 
from some degree of separation.


Mark.

Re: Routed optical networks

[Mark Tinka]

On 5/2/23 21:32, Jared Mauch wrote:


I’ve seen proposals for an LSR MPLS/ROADAM type solution, where imagine you are 
at a hop where in a long distance system solution, you would end up with OEO, 
but instead you get directionality capability with an IP/MPLS capable device.


My memory is rather fuzzy, but didn't Juniper attempt something like 
this in their PTX's after they picked up BTI? I think the plan was to 
co-locate the ROADM at the bottom of the PTX chassis, or something along 
those lines.


I know Cisco (and Juniper) tried by integrating GMPLS into their code as 
a starting point, but that didn't go very far with customers. It just 
seemed impossible for the Transport teams to allow the IP/MPLS teams 
that level of access into their line system :-).




   As mentioned previously, the 400-ZR/ZR+/ZR-Bright/+0 optics are the latest 
example of that.


A rather high barrier to entry for most operators, but we have to start 
from somewhere.



I know of a few companies that have looked at solutions like this, and can 
expect there to be some interesting solutions that would appear as a result.  
Optical line systems tend to have pretty low power requirements compared to a 
router, but some of the routers are getting pretty low power as well when it 
comes to the power OPEX/bit, and if you have the ability to deliver services as 
an integrated packet optical you could see reduced costs and simplified 
components/sparing.


The main problem is distance.

If you need to move that kind of capacity more than 50km, it's hard to 
avoid DWDM.




I’ll also say that I’ve not yet seen the price compression that I had expected 
in the space yet, but I figure that’s coming.  We are seeing the bits/watt 
ratio improve though, so for the same or less power consumption you get more 
bits.  Some of this technology stuff is truly magical.


I think for long spans, DWDM will not only be cheaper, but the only 
feasible solution.


For the metro, it will come down to what motivates the business... 
plenty of features, or plenty of speed.


Also, DWDM vendors are adding speed and distance faster and cheaper than 
the IP/MPLS vendors can. So they will always be one step ahead in that 
respect; and we have the submarine cable systems to thank for that.


Mark.

Re: Routed optical networks

[Mark Tinka]

On 5/2/23 16:25, Izaac wrote:


This is a very convoluted way of backing into the ole packet-switched
vs. circuit switched decision.


A fight that will never go away.

There has been some compromise in recent years, with Transport-heavy 
customers accepting standard Ethernet services, but only if they are 
carried by a Transport device.


Mark.

Re: Routed optical networks

[Mark Tinka]

On 5/2/23 07:28, Vasilenko Eduard via NANOG wrote:

The incumbent carrier typically has enough fiber strands to avoid any 
colored interfaces (that are 3x expensive compare to gray) in the Metro.


Metro ring typically has 8-10 nodes (or similar). 16-20 strands of 
fiber were not possible to construct anyway – any cable is bigger.


It is the same cost to lay down fiber on 16 strands or 32.

Hence, PTT just does not need DWDM in Metro, not at all. Hence, the 
DWDM optimization that you are talking about below is not needed too.




This may or may not always be the case. Especially for large carriers, 
where there could be a requirement to sell some of those dark fibre 
pairs to large customers (think the content folk coming into town, 
e.t.c.), they may no longer have the priviledge of having plenty of free 
fibre in the metro. Or if they did, the rate of traffic expansion means 
they burn through those fibre pairs pretty quick.


10Gbps isn't a lot nowadays, and 100Gbps may start to push the limits 
depending on the size of the operator, the scope of the Metro-E ring and 
the level of service that needs to be maintained during a re-route (two 
available paths in the ring could balance 100Gbps of traffic, but if one 
half of that ring breaks, the remaining path may need to carry a lot 
more than 100Gbps, and then packets start to fall flat on the floor).


At that scale, DWDM in the metro will make sense, at least more sense 
than 400G-ZR, at the moment.



If you rent a single pair of fiber then you need colored interfaces to 
multiplex 8-10 nodes into 1 pair on the ring.


Then the movement of transponders from DWDM into the router would 
eliminate 2 gray interfaces on every node (4 per link): one on the 
router side, and another on the DWDM side.


Overall, it is about a 25% cost cut of the whole “router+DWDM”.



Some operators would also be selling Transport services in or along the 
metro, and customers paying for that may require that they do not cross 
a router device.



It is still 2x more expensive compare to using additional fiber 
strands on YOUR fiber.




There are plenty of DWDM pizza boxes that cost next to nothing. At 
scale, the price of these is not a stumbling block. And certainly, the 
price of these would be far lower than a router line card.




By the way, about “well-defined stack of technologies”:

NMS (polished by SDN our days) should be cross-layer: it should manage 
at the same time: ROADM/OADM in DWDM and colored laser in Router.


It is a vendor lock up to now (no multi-vendor). Hence, 25% cost 
savings would go to the vendor that has such NMS, not to the carrier.


Technology still does not make sense because no multivendor support 
between the NMS of one vendor and the router or DWDM of another.


Looking at the NMS history, it would probably never be multi-vendor. 
For that reason, I am pessimistic about the future of the colored 
interfaces in routers (and alien lambdas in DWDM). Despite a potential 
25% cost advantage in eliminating gray interfaces.




OpenROADM is a good initiative. But it seems it's to be to Transport 
equipment vendors what IPv6 and DNSSEC is to the IP world :-).


Mark.

Re: Spectrum networks IPv6 access issue

[Shawn L via NANOG]

We know the feeling well. Try porting from them…..


> On May 2, 2023, at 4:41 PM, Daniel Marks via NANOG  wrote:
> 
> My issue was just trying to convince Spectrum to look into the problem in 
> the first place, I brought the Atlas probe receipts because it’s such a 
> helpful tool, but wasn’t able to get through to anyone helpful (acct mgr, noc 
> email, even the escalation list) until I started lighting fires filing FCC 
> complaints and using social media (which thankfully worked).
> 
> Not sure how accurate it is (I hope it isn’t), but some of the techs I spoke 
> to said a lot of the internal tooling for troubleshooting is incapable of 
> dealing with IPv6, so they weren’t able to do things like run traceroutes to 
> confirm what I was seeing. My guess is that this issue was caught in a 
> catch-22 where they needed impossible to obtain proof on their end to 
> escalate to a team who can actually deal with the issue.
> 
> Sucks for us folk who went all in on v6 only to find out not even the ISP can 
> help us. 
> 
> -Daniel Marks
> 
>> On May 2, 2023, at 15:36, Jared Mauch  wrote:
>> 
>> 
>> 
 On May 2, 2023, at 2:43 PM, Daniel Marks via NANOG  wrote:
>>> 
>>> This has been “resolved", I finally got through to some awesome engineer at 
>>> Spectrum who has rerouted traffic while they work with their hardware 
>>> vendor (thanks Jake):
>> 
>> 
>> One of the tools that I’ve used in the past is the RIPE Atlas service to 
>> measure these things.  It’s helped me isolate IP space reachability issues 
>> for new announcements, because you can get enough of a random sample of 
>> hosts to isolate things, and enough data about that endpoint to launch 
>> follow-up measurements.
>> 
>> - Jared

Re: Routed optical networks

[Etienne-Victor Depasquale via NANOG]

>
> I’ve seen proposals for an LSR MPLS/ROADAM type solution, where imagine
> you are at a hop where in a long distance system solution, you would end up
> with OEO, but instead you get directionality capability with an IP/MPLS
> capable device.  As mentioned previously, the 400-ZR/ZR+/ZR-Bright/+0
> optics are the latest example of that.
>

Jared, I understand your point in the above statement to be that
directionality is cost-effectively implemented through label-switched
paths,
rather than (ROADM-enabled) optical path switching.

Do I understand right?

Thank you.

Etienne

On Tue, May 2, 2023 at 9:33 PM Jared Mauch  wrote:

>
>
> > On May 2, 2023, at 2:29 PM, Etienne-Victor Depasquale via NANOG <
> nanog@nanog.org> wrote:
> >
> > On Mon, May 01, 2023 at 02:56:47PM -0600, Matt Erculiani wrote:
> > > In short, the idea is that optical networks are wasteful and routers
> do a
> > > better job making more use of a network's capacity than ROADMs. Take
> the
> > > extra router hop (or 3 or 8) versus short-cutting it with an optical
> > > network because the silicon is so low-latency anyway that it hardly
> makes a
> > > difference now. Putting more GBs per second on fewer strands means
> saving a
> > > lot of money on infrastructure costs.
> >
> > This is a very convoluted way of backing into the ole packet-switched
> > vs. circuit switched decision.
> >
> > I don't follow.
> > While ROADMs can be thought of as circuit-switchers,
> > the number of concurrent clients and switching latency put ROADMs on a
> different operational level than packet switchers, right?
>
>
> I’ve seen proposals for an LSR MPLS/ROADAM type solution, where imagine
> you are at a hop where in a long distance system solution, you would end up
> with OEO, but instead you get directionality capability with an IP/MPLS
> capable device.  As mentioned previously, the 400-ZR/ZR+/ZR-Bright/+0
> optics are the latest example of that.
>
> I know of a few companies that have looked at solutions like this, and can
> expect there to be some interesting solutions that would appear as a
> result.  Optical line systems tend to have pretty low power requirements
> compared to a router, but some of the routers are getting pretty low power
> as well when it comes to the power OPEX/bit, and if you have the ability to
> deliver services as an integrated packet optical you could see reduced
> costs and simplified components/sparing.
>
> I’ll also say that I’ve not yet seen the price compression that I had
> expected in the space yet, but I figure that’s coming.  We are seeing the
> bits/watt ratio improve though, so for the same or less power consumption
> you get more bits.  Some of this technology stuff is truly magical.
>
> - Jared



-- 
Ing. Etienne-Victor Depasquale
Assistant Lecturer
Department of Communications & Computer Engineering
Faculty of Information & Communication Technology
University of Malta
Web. https://www.um.edu.mt/profile/etiennedepasquale

Re: Spectrum networks IPv6 access issue

[Daniel Marks via NANOG]

My issue was just trying to convince Spectrum to look into the problem in the 
first place, I brought the Atlas probe receipts because it’s such a helpful 
tool, but wasn’t able to get through to anyone helpful (acct mgr, noc email, 
even the escalation list) until I started lighting fires filing FCC complaints 
and using social media (which thankfully worked).

Not sure how accurate it is (I hope it isn’t), but some of the techs I spoke to 
said a lot of the internal tooling for troubleshooting is incapable of dealing 
with IPv6, so they weren’t able to do things like run traceroutes to confirm 
what I was seeing. My guess is that this issue was caught in a catch-22 where 
they needed impossible to obtain proof on their end to escalate to a team who 
can actually deal with the issue.

Sucks for us folk who went all in on v6 only to find out not even the ISP can 
help us. 

-Daniel Marks

> On May 2, 2023, at 15:36, Jared Mauch  wrote:
> 
> 
> 
>> On May 2, 2023, at 2:43 PM, Daniel Marks via NANOG  wrote:
>> 
>> This has been “resolved", I finally got through to some awesome engineer at 
>> Spectrum who has rerouted traffic while they work with their hardware vendor 
>> (thanks Jake):
> 
> 
> One of the tools that I’ve used in the past is the RIPE Atlas service to 
> measure these things.  It’s helped me isolate IP space reachability issues 
> for new announcements, because you can get enough of a random sample of hosts 
> to isolate things, and enough data about that endpoint to launch follow-up 
> measurements.
> 
> - Jared


smime.p7s
Description: S/MIME cryptographic signature

Re: Routed optical networks

[Eve Griliches]

So right Jaredmagic has been in the NPU capacity increase that's driven
the cost per 100G down on 1RU routers; and the integration of DSPs and more
into QSFP-DD form factors at much lower power than expected. The standards
for optical links are maturing as well, but we still have work to do on the
management side for the electrical interfaces.
Eve

On Tue, May 2, 2023 at 3:33 PM Jared Mauch  wrote:

>
>
> > On May 2, 2023, at 2:29 PM, Etienne-Victor Depasquale via NANOG <
> nanog@nanog.org> wrote:
> >
> > On Mon, May 01, 2023 at 02:56:47PM -0600, Matt Erculiani wrote:
> > > In short, the idea is that optical networks are wasteful and routers
> do a
> > > better job making more use of a network's capacity than ROADMs. Take
> the
> > > extra router hop (or 3 or 8) versus short-cutting it with an optical
> > > network because the silicon is so low-latency anyway that it hardly
> makes a
> > > difference now. Putting more GBs per second on fewer strands means
> saving a
> > > lot of money on infrastructure costs.
> >
> > This is a very convoluted way of backing into the ole packet-switched
> > vs. circuit switched decision.
> >
> > I don't follow.
> > While ROADMs can be thought of as circuit-switchers,
> > the number of concurrent clients and switching latency put ROADMs on a
> different operational level than packet switchers, right?
>
>
> I’ve seen proposals for an LSR MPLS/ROADAM type solution, where imagine
> you are at a hop where in a long distance system solution, you would end up
> with OEO, but instead you get directionality capability with an IP/MPLS
> capable device.  As mentioned previously, the 400-ZR/ZR+/ZR-Bright/+0
> optics are the latest example of that.
>
> I know of a few companies that have looked at solutions like this, and can
> expect there to be some interesting solutions that would appear as a
> result.  Optical line systems tend to have pretty low power requirements
> compared to a router, but some of the routers are getting pretty low power
> as well when it comes to the power OPEX/bit, and if you have the ability to
> deliver services as an integrated packet optical you could see reduced
> costs and simplified components/sparing.
>
> I’ll also say that I’ve not yet seen the price compression that I had
> expected in the space yet, but I figure that’s coming.  We are seeing the
> bits/watt ratio improve though, so for the same or less power consumption
> you get more bits.  Some of this technology stuff is truly magical.
>
> - Jared

Re: Spectrum networks IPv6 access issue

[Tom Rini]

I can confirm things have been resolved here, many thanks all!

On Tue, May 02, 2023 at 02:43:29PM -0400, d...@nielmarks.com wrote:

> This has been “resolved", I finally got through to some awesome engineer at 
> Spectrum who has rerouted traffic while they work with their hardware vendor 
> (thanks Jake):
> 
>  1  2605:6000:0:8::f:7acc (2605:6000:0:8::f:7acc)  0.372 ms  0.323 ms  0.282 
> ms
>  2  ae15.ARTNTXAF02H.chtrse.com (2605:6000:0:4::f:87e9)  4.002 ms  3.977 ms  
> 10.884 ms
>  3  * * *
>  4  lag-23.mcr11dllbtxlb.netops.charter.com (2605:6000:0:4::2:1e)  1.464 ms  
> 1.574 ms  1.560 ms
>  5  * * *
>  6  * *
> lag-26.hstqtx0209w-bcr00.netops.charter.com (2001:1998:0:4::528)  6.351 ms
>  7  * * *
>  8  lag-0.pr2.atl20.netops.charter.com (2001:1998:0:4::51f)  18.577 ms  
> 18.572 ms  18.428 ms
>  9  * * *
> 10  e0-36.core2.bna1.he.net (2001:470:0:429::2)  32.553 ms  32.720 ms  32.563 
> ms
> 11  * * *
> 12  * * *
> 13  equinix-ix.dfw2.packet.net (2001:504:0:5:0:5:4825:1)  24.591 ms  24.553 
> ms  24.604 ms
> 14  * * *
> 15  * * *
> 16  dfw.source.kernel.org (2604:1380:4641:c500::1)  24.423 ms  26.020 ms  
> 24.415 ms
> 
> > On Apr 28, 2023, at 11:35 AM, Sam Thomas  wrote:
> > 
> > Actual data from a Spectrum residential customer in DFW.
> > 
> > First, IPv4:
> > 
> >  trace dfw.source.kernel.org
> > traceroute to dfw.source.kernel.org (139.178.84.217), 64 hops max, 40
> > byte packets
> > 1  my.router  0.389 ms  0.350 ms  0.292 ms
> > 2  142-254-130-077.inf.spectrum.com (142.254.130.77)  8.423 ms  8.408
> > ms  8.080 ms
> > 3  lag-63.artrtx2801h.netops.charter.com (24.28.88.17)  27.167 ms
> > 25.065 ms  21.977 ms
> > 4  lag-22.artntxaf01r.netops.charter.com (24.175.49.233)  10.718 ms
> > 10.083 ms  15.886 ms
> > 5  lag-23.mcr11crtntxjt.netops.charter.com (24.175.36.224)  13.386 ms
> > 11.560 ms  11.297 ms
> > 6  lag-21.rcr01dllatx37.netops.charter.com (24.175.49.0)  11.339 ms
> >lag-28.rcr01dllatx37.netops.charter.com (24.175.33.246)  11.904 ms
> > 128.186 ms
> > 7  lag-414.dllstx976iw-bcr00.netops.charter.com (66.109.6.52)  12.603 ms
> >lag-14.dllstx976iw-bcr00.netops.charter.com (66.109.6.88)  12.172 ms
> >lag-414.dllstx976iw-bcr00.netops.charter.com (66.109.6.52)  12.299 ms
> > 8  lag-302.pr3.dfw10.netops.charter.com (209.18.43.77)  21.570 ms
> >lag-0.pr3.dfw10.netops.charter.com (66.109.5.121)  11.763 ms
> >lag-302.pr3.dfw10.netops.charter.com (209.18.43.77)  12.182 ms
> > 9  dls-b23-link.ip.twelve99.net (62.115.156.208)  11.515 ms *  11.706 ms
> > 10  packethost-ic-369414.ip.twelve99-cust.net (213.248.72.3)  11.870
> > ms  30.246 ms  18.199 ms
> > 11  * * *
> > 12  * * *
> > 13  dfw.source.kernel.org (139.178.84.217)  12.021 ms  12.076 ms  11.922 ms
> > 
> > ping dfw.source.kernel.org
> > PING dfw.source.kernel.org (139.178.84.217): 56 data bytes
> > 64 bytes from 139.178.84.217: icmp_seq=0 ttl=50 time=11.590 ms
> > 64 bytes from 139.178.84.217: icmp_seq=1 ttl=50 time=11.785 ms
> > 
> > IPv6:
> > 
> >  trace6 dfw.source.kernel.org
> > traceroute6 to dfw.source.kernel.org (2604:1380:4641:c500::1) from
> > 2603:8080:REDACTED, 64 hops max, 20 byte packets
> > 1  2603-8080-REDACTED.res6.spectrum.com  0.404 ms  0.340 ms  0.322 ms
> > 2  2603-90c5-0003-000e----0001.inf6.spectrum.com  10.308
> > ms  7.901 ms  9.902 ms
> > 3  lag-63.artrtx2801h.netops.charter.com  17.008 ms  10.523 ms  11.077 ms
> > 4  lag-22.artntxaf01r.netops.charter.com  14.638 ms * *
> > 5  lag-23.mcr11crtntxjt.netops.charter.com  11.090 ms  11.612 ms  12.234 ms
> > 6  * * *
> > 7  lag-414.dllstx976iw-bcr00.netops.charter.com  12.572 ms *
> >lag-24.dllstx976iw-bcr00.netops.charter.com  12.160 ms
> > 8  * * *
> > 9  * * *
> > 10  * * *
> > 11  * * *
> > 12  * * *
> > 13  * * *
> > 14  * * *
> > 15  * * *
> > 16  * * *
> > 17  * * *
> > 18  * * *
> > 19  *^C
> > 
> > ping6 dfw.source.kernel.org
> > PING6(56=40+8+8 bytes) 2603:8080:REDACTED --> 2604:1380:4641:c500::1
> > ^C
> > --- dfw.source.kernel.org ping6 statistics ---
> > 5 packets transmitted, 0 packets received, 100.0% packet loss
> > 
> > I have a Linode VM in Dallas that I also can't get to via IPv6.
> > Traffic appears to take the same path for IPv4.
> > 
> > On Wed, Apr 26, 2023 at 10:50 AM Tom Rini  wrote:
> >> 
> >> Hey all,
> >> 
> >> I'm posting this here in hopes of getting the attention of someone that
> >> can get this issue resolved, or at least an internal ticket filed. I've
> >> tried the customer-facing tech support and not been able to get such a
> >> thing done.
> >> 
> >> In short, from within Spectrum's US IPv6 network (verified in both North
> >> Carolina and Ohio), dfw.source.kernel.org (2604:1380:4641:c500::1) is
> >> unreachable and connections time out. This site is otherwise fine and
> >> globally accessible via IPv6, tested on both Qwest and T-Mobile hosted
> >> systems.  This is a regression from some time in early April this year.
> >> 
> >> --
> >> Tom
> 



-- 
Tom

Re: Spectrum networks IPv6 access issue

[Jared Mauch]

> On May 2, 2023, at 2:43 PM, Daniel Marks via NANOG  wrote:
> 
> This has been “resolved", I finally got through to some awesome engineer at 
> Spectrum who has rerouted traffic while they work with their hardware vendor 
> (thanks Jake):


One of the tools that I’ve used in the past is the RIPE Atlas service to 
measure these things.  It’s helped me isolate IP space reachability issues for 
new announcements, because you can get enough of a random sample of hosts to 
isolate things, and enough data about that endpoint to launch follow-up 
measurements.

- Jared

Re: Routed optical networks

[Jared Mauch]

> On May 2, 2023, at 2:29 PM, Etienne-Victor Depasquale via NANOG 
>  wrote:
> 
> On Mon, May 01, 2023 at 02:56:47PM -0600, Matt Erculiani wrote:
> > In short, the idea is that optical networks are wasteful and routers do a
> > better job making more use of a network's capacity than ROADMs. Take the
> > extra router hop (or 3 or 8) versus short-cutting it with an optical
> > network because the silicon is so low-latency anyway that it hardly makes a
> > difference now. Putting more GBs per second on fewer strands means saving a
> > lot of money on infrastructure costs.
> 
> This is a very convoluted way of backing into the ole packet-switched
> vs. circuit switched decision.
> 
> I don't follow. 
> While ROADMs can be thought of as circuit-switchers,
> the number of concurrent clients and switching latency put ROADMs on a 
> different operational level than packet switchers, right?


I’ve seen proposals for an LSR MPLS/ROADAM type solution, where imagine you are 
at a hop where in a long distance system solution, you would end up with OEO, 
but instead you get directionality capability with an IP/MPLS capable device.  
As mentioned previously, the 400-ZR/ZR+/ZR-Bright/+0 optics are the latest 
example of that.

I know of a few companies that have looked at solutions like this, and can 
expect there to be some interesting solutions that would appear as a result.  
Optical line systems tend to have pretty low power requirements compared to a 
router, but some of the routers are getting pretty low power as well when it 
comes to the power OPEX/bit, and if you have the ability to deliver services as 
an integrated packet optical you could see reduced costs and simplified 
components/sparing.

I’ll also say that I’ve not yet seen the price compression that I had expected 
in the space yet, but I figure that’s coming.  We are seeing the bits/watt 
ratio improve though, so for the same or less power consumption you get more 
bits.  Some of this technology stuff is truly magical.

- Jared

Re: Spectrum networks IPv6 access issue

[Daniel Marks via NANOG]

This has been “resolved", I finally got through to some awesome engineer at 
Spectrum who has rerouted traffic while they work with their hardware vendor 
(thanks Jake):

 1  2605:6000:0:8::f:7acc (2605:6000:0:8::f:7acc)  0.372 ms  0.323 ms  0.282 ms
 2  ae15.ARTNTXAF02H.chtrse.com (2605:6000:0:4::f:87e9)  4.002 ms  3.977 ms  
10.884 ms
 3  * * *
 4  lag-23.mcr11dllbtxlb.netops.charter.com (2605:6000:0:4::2:1e)  1.464 ms  
1.574 ms  1.560 ms
 5  * * *
 6  * *
lag-26.hstqtx0209w-bcr00.netops.charter.com (2001:1998:0:4::528)  6.351 ms
 7  * * *
 8  lag-0.pr2.atl20.netops.charter.com (2001:1998:0:4::51f)  18.577 ms  18.572 
ms  18.428 ms
 9  * * *
10  e0-36.core2.bna1.he.net (2001:470:0:429::2)  32.553 ms  32.720 ms  32.563 ms
11  * * *
12  * * *
13  equinix-ix.dfw2.packet.net (2001:504:0:5:0:5:4825:1)  24.591 ms  24.553 ms  
24.604 ms
14  * * *
15  * * *
16  dfw.source.kernel.org (2604:1380:4641:c500::1)  24.423 ms  26.020 ms  
24.415 ms

> On Apr 28, 2023, at 11:35 AM, Sam Thomas  wrote:
> 
> Actual data from a Spectrum residential customer in DFW.
> 
> First, IPv4:
> 
>  trace dfw.source.kernel.org
> traceroute to dfw.source.kernel.org (139.178.84.217), 64 hops max, 40
> byte packets
> 1  my.router  0.389 ms  0.350 ms  0.292 ms
> 2  142-254-130-077.inf.spectrum.com (142.254.130.77)  8.423 ms  8.408
> ms  8.080 ms
> 3  lag-63.artrtx2801h.netops.charter.com (24.28.88.17)  27.167 ms
> 25.065 ms  21.977 ms
> 4  lag-22.artntxaf01r.netops.charter.com (24.175.49.233)  10.718 ms
> 10.083 ms  15.886 ms
> 5  lag-23.mcr11crtntxjt.netops.charter.com (24.175.36.224)  13.386 ms
> 11.560 ms  11.297 ms
> 6  lag-21.rcr01dllatx37.netops.charter.com (24.175.49.0)  11.339 ms
>lag-28.rcr01dllatx37.netops.charter.com (24.175.33.246)  11.904 ms
> 128.186 ms
> 7  lag-414.dllstx976iw-bcr00.netops.charter.com (66.109.6.52)  12.603 ms
>lag-14.dllstx976iw-bcr00.netops.charter.com (66.109.6.88)  12.172 ms
>lag-414.dllstx976iw-bcr00.netops.charter.com (66.109.6.52)  12.299 ms
> 8  lag-302.pr3.dfw10.netops.charter.com (209.18.43.77)  21.570 ms
>lag-0.pr3.dfw10.netops.charter.com (66.109.5.121)  11.763 ms
>lag-302.pr3.dfw10.netops.charter.com (209.18.43.77)  12.182 ms
> 9  dls-b23-link.ip.twelve99.net (62.115.156.208)  11.515 ms *  11.706 ms
> 10  packethost-ic-369414.ip.twelve99-cust.net (213.248.72.3)  11.870
> ms  30.246 ms  18.199 ms
> 11  * * *
> 12  * * *
> 13  dfw.source.kernel.org (139.178.84.217)  12.021 ms  12.076 ms  11.922 ms
> 
> ping dfw.source.kernel.org
> PING dfw.source.kernel.org (139.178.84.217): 56 data bytes
> 64 bytes from 139.178.84.217: icmp_seq=0 ttl=50 time=11.590 ms
> 64 bytes from 139.178.84.217: icmp_seq=1 ttl=50 time=11.785 ms
> 
> IPv6:
> 
>  trace6 dfw.source.kernel.org
> traceroute6 to dfw.source.kernel.org (2604:1380:4641:c500::1) from
> 2603:8080:REDACTED, 64 hops max, 20 byte packets
> 1  2603-8080-REDACTED.res6.spectrum.com  0.404 ms  0.340 ms  0.322 ms
> 2  2603-90c5-0003-000e----0001.inf6.spectrum.com  10.308
> ms  7.901 ms  9.902 ms
> 3  lag-63.artrtx2801h.netops.charter.com  17.008 ms  10.523 ms  11.077 ms
> 4  lag-22.artntxaf01r.netops.charter.com  14.638 ms * *
> 5  lag-23.mcr11crtntxjt.netops.charter.com  11.090 ms  11.612 ms  12.234 ms
> 6  * * *
> 7  lag-414.dllstx976iw-bcr00.netops.charter.com  12.572 ms *
>lag-24.dllstx976iw-bcr00.netops.charter.com  12.160 ms
> 8  * * *
> 9  * * *
> 10  * * *
> 11  * * *
> 12  * * *
> 13  * * *
> 14  * * *
> 15  * * *
> 16  * * *
> 17  * * *
> 18  * * *
> 19  *^C
> 
> ping6 dfw.source.kernel.org
> PING6(56=40+8+8 bytes) 2603:8080:REDACTED --> 2604:1380:4641:c500::1
> ^C
> --- dfw.source.kernel.org ping6 statistics ---
> 5 packets transmitted, 0 packets received, 100.0% packet loss
> 
> I have a Linode VM in Dallas that I also can't get to via IPv6.
> Traffic appears to take the same path for IPv4.
> 
> On Wed, Apr 26, 2023 at 10:50 AM Tom Rini  wrote:
>> 
>> Hey all,
>> 
>> I'm posting this here in hopes of getting the attention of someone that
>> can get this issue resolved, or at least an internal ticket filed. I've
>> tried the customer-facing tech support and not been able to get such a
>> thing done.
>> 
>> In short, from within Spectrum's US IPv6 network (verified in both North
>> Carolina and Ohio), dfw.source.kernel.org (2604:1380:4641:c500::1) is
>> unreachable and connections time out. This site is otherwise fine and
>> globally accessible via IPv6, tested on both Qwest and T-Mobile hosted
>> systems.  This is a regression from some time in early April this year.
>> 
>> --
>> Tom



smime.p7s
Description: S/MIME cryptographic signature

Re: Routed optical networks

[Etienne-Victor Depasquale via NANOG]

Very helpful observations, Matt, thank you.

How comfortably does the phrase "routed optical networks over Ethernet
without ROADMs" sit with you?
I mean: would you accept a limitation of "optical network" to the case of
a network without optical layer switching (of the type done by add-drop
multiplexers)?

Cheers,

Etienne

On Mon, May 1, 2023 at 10:57 PM Matt Erculiani  wrote:

> Hi Etienne
>
> In short, the idea is that optical networks are wasteful and routers do a
> better job making more use of a network's capacity than ROADMs. Take the
> extra router hop (or 3 or 8) versus short-cutting it with an optical
> network because the silicon is so low-latency anyway that it hardly makes a
> difference now. Putting more GBs per second on fewer strands means saving a
> lot of money on infrastructure costs.
>
> 400G ZR comes to mind as a foundational technology since it basically made
> active optical muxponder equipment obsolete in the metro. The savings here
> means telcos/enterprises can afford more router ports, which we've already
> established can utilize paths more efficiently anyway. Otherwise, this is
> more of a concept and can be executed with a variety of pre-existing
> technologies, or someone's new secret sauce that bakes everything together
> like SD-WAN did to its constituent technologies.
>
> -Matt
>
>
> On Mon, May 1, 2023 at 12:30 PM Etienne-Victor Depasquale via NANOG <
> nanog@nanog.org> wrote:
>
>> Hello folks,
>>
>> Simple question: does "routed optical networks" have a clear meaning in
>> the metro area context, or not?
>>
>> Put differently: does it call to mind a well-defined stack of
>> technologies in the control and data planes of metro-area networks?
>>
>> I'm asking because I'm having some thoughts about the clarity of this
>> term, in the process of carrying out a qualitative survey of the results of
>> the metro-area networks survey.
>>
>> Cheers,
>>
>> Etienne
>>
>> --
>> Ing. Etienne-Victor Depasquale
>> Assistant Lecturer
>> Department of Communications & Computer Engineering
>> Faculty of Information & Communication Technology
>> University of Malta
>> Web. https://www.um.edu.mt/profile/etiennedepasquale
>>
>
>
> --
> Matt Erculiani
> ERCUL-ARIN
>


-- 
Ing. Etienne-Victor Depasquale
Assistant Lecturer
Department of Communications & Computer Engineering
Faculty of Information & Communication Technology
University of Malta
Web. https://www.um.edu.mt/profile/etiennedepasquale

Re: Routed optical networks

[Etienne-Victor Depasquale via NANOG]

Hello Eve,

Thank you for weighing in; I'm eager for feedback from the field.
This eagerness stems from my work, over the past two years,
to form my understanding of where current- and next-gen metro area networks
are heading.
I need this understanding to help academics in my field of specialization
to better understand energy consumption in metro-area networks.

Your observation about elimination of OTN resonates well
with what I've heard from webinars, and what I've read in studies.
It also matches what I've shown in the graphic I linked to in an earlier
post in this thread (this graphic

).
However, the larger operators are less inclined to drop OTN as a server
layer network
(layer network used as defined in G.805).

Indeed, part of the scope of the question leading to the results shown,
actually was to try to understand the prevalence of OTN in operators'
current networks.
As regards greenfield, the *NOG results are a bit more nuanced

.
IP/MPLS over Ethernet over DWDM with ROADMs for node bypass gets 34% of the
vote,
up from about 13% of what is currently in their networks.

Cheers,

Etienne


On Tue, May 2, 2023 at 4:01 PM Eve Griliches  wrote:

> Hi Etienne,
> Below is our (Cisco) definition of the Routed Optical Network. The goal,
> metro or long haul or subsea, is to reduce the number of control planes. By
> migration TDM traffic using CEM or PLE to the IP layer, you eliminate the
> OTN control plane and management. Eventually, when standards are settled
> the ultimate goal is to have a single control plane for the network. I'm
> not trying to be a commercial here, but you can read more in the resources
> section on this page:
> https://www.cisco.com/c/en/us/solutions/service-provider/routed-optical-networking/index.html
> HTH,
> Eve
>
> Routed optical networking, is an architecture that delivers improved
> operational efficiencies and simplicity. The solution works by merging IP
> and private line services onto a single layer where all the switching is
> done at Layer 3. Routers are connected with standardized 400G ZR/ZR+
> coherent pluggable optics.
>
> With a single service layer based upon IP, flexible management tools can
> leverage telemetry and model-driven programmability to streamline lifecycle
> operations. This simplified architecture integrates open data models and
> standard APIs, enabling a provider to focus on automation initiatives for a
> simpler topology.
>
> On Mon, May 1, 2023 at 2:30 PM Etienne-Victor Depasquale via NANOG <
> nanog@nanog.org> wrote:
>
>> Hello folks,
>>
>> Simple question: does "routed optical networks" have a clear meaning in
>> the metro area context, or not?
>>
>> Put differently: does it call to mind a well-defined stack of
>> technologies in the control and data planes of metro-area networks?
>>
>> I'm asking because I'm having some thoughts about the clarity of this
>> term, in the process of carrying out a qualitative survey of the results of
>> the metro-area networks survey.
>>
>> Cheers,
>>
>> Etienne
>>
>> --
>> Ing. Etienne-Victor Depasquale
>> Assistant Lecturer
>> Department of Communications & Computer Engineering
>> Faculty of Information & Communication Technology
>> University of Malta
>> Web. https://www.um.edu.mt/profile/etiennedepasquale
>>
>

-- 
Ing. Etienne-Victor Depasquale
Assistant Lecturer
Department of Communications & Computer Engineering
Faculty of Information & Communication Technology
University of Malta
Web. https://www.um.edu.mt/profile/etiennedepasquale

Re: Best Linux (or BSD) hosted BGP?

[Warren Kumari]

+lots.

I've used a number of Linux routing thingies (BIRD, Quagga, VyOS/Ubiquiti,
OpenBGPd, ExBGP), and FRR is (for me at least) by far the friendliest. It's
trivial to spin this up on a cloud VM and start announcing a prefix.

For doing something like Anycast though (where you are mostly just
announcing a route on demand), ExaBGP is great.

W


On Mon, May 01, 2023 at 2:03 PM, Jean Franco  wrote:

> https://frrouting.org/
>
> On Mon, May 1, 2023 at 2:28 PM Josh Luthman 
> wrote:
>
> Doesn't VyOS simply use Quagga?
>
> On Mon, May 1, 2023 at 12:09 PM Jean Franco  wrote:
>
> Hi,
>
> VyOS
>
> Best regards,
>
> On Mon, May 1, 2023 at 1:03 PM Bryan Fields  wrote:
>
> I know best subjective, but I'm looking at a project to announce some IP
> space
> that's between uses now and see what's there.  I'm planing to run a flow
> logger and ntop on the VM and see what is coming in if anything.  I'm
> looking
> at the options for BGP out there, and there's quite a few (other than
> running
> a VM with a router doing BGP), but most data I've seen is focused on scale
> and
> filtering use, or RPKI.  My use case is a bit different, and I can't find
> any
> best practices for this use case from what I've found.
>
> That said, is there a better solution other than linux/ntop/ipt-netflow?
> --
> Bryan Fields
>
> 727-409-1194 - Voice
> http://bryanfields.net
>
>

Re: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging

[Mike Hammett]

We have upgraded NX-OS to a new major version and have the same results. 



Apr 20 09:36:05 UTC: %UFDM-3-FIB_IPv4_ROUTE_CONSISTENCY_CHECKER_FAIL: FIB IPv4 
consistency checker FAILED on slot 1 
Apr 19 13:55:57 UTC: %IPFIB-2-FIB_TCAM_RESOURCE_EXHAUSTION_LPM_IPV4: FIB TCAM 
exhausted for IPV4 routes in LPM table 

show forwarding inconsistency 
... 
69. slot(1), vrf(default), prefix(198.XXX.XXX.0/YY), Route inconsistent in FIB 
Software. 


Well, those seem like problems. 


EASTGATE401-BGP-02# show ip route summary 
IP Route Table for VRF "default" 
Total number of routes: 292 
Total number of paths: 292 

Unicast paths: 
Best paths per protocol: Backup paths per protocol: 
am : 48 None 
local : 8 
direct : 8 
static : 5 
broadcast : 19 
bgp-X : 204 

Number of routes per mask-length: 
/0 : 1 /8 : 1 /18: 2 /19: 4 /20: 26 
/21: 24 /22: 35 /23: 18 /24: 100 /27: 1 
/28: 1 /29: 1 /30: 4 /32: 74 




It's choking on only 292 routes? 


Error Message FIB_TCAM_RESOURCE_EXHAUSTION_LPM_IPV4: FIB TCAM exhausted for 
IPV4 routes in LPM table 
Explanation The TCAM device in the Layer 3 forwarding ASIC has reached its 
system limits for IPv4 entries in the LPM table. 

Recommended Action No action is required. 



TCAM is exhausted and no action is recommended? 


Error Message UFDM-3-FIB_IPv4_ROUTE_CONSISTENCY_CHECKER_FAIL: FIB IPv4 
consistency checker FAILED on slot [chars] 
Explanation FIB Ipv4 route consistency checker Failed. Route database is 
consistent with hardware 

Recommended Action No action is required. 


The FIB is inconsistent and no action is recommended? 


EASTGATE401-BGP-02# show system internal forwarding route summary 

slot 1 
=== 


IPv4 hosts & routes summary on module 1 
- 

Max host route entries : 8192 
Total number of IPv4 host routes used: 72 
Max LPM table entries : 7167 
Total number of IPv4 LPM routes used : 16 



I seem to be out of my depth here in that 292 is less than 7167, but yet it 
still fails. 



- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 


From: "Mike Hammett"  
To: "NANOG"  
Sent: Monday, April 3, 2023 12:21:29 AM 
Subject: Cisco Nexus 3k Route Selection\Packet Forwarding Debugging 

We have a Nexus 3064 that is setup with partial BGP tables and is routing based 
on that. 


I've done a show ip bgp for an IP of interest and it has an expected next hop 
IP. I show ip arp on that next hop IP and it has the expected interface. 


However, sFlows show the packets leaving on a different interface, the one that 
would carry the default route for routes not otherwise known. 


If the next hop IP is expected and the ARP of that next hop IP is expected, why 
are packets leaving out an unexpected interface? 



- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

Re: Routed optical networks

[Izaac]

On Mon, May 01, 2023 at 02:56:47PM -0600, Matt Erculiani wrote:
> In short, the idea is that optical networks are wasteful and routers do a
> better job making more use of a network's capacity than ROADMs. Take the
> extra router hop (or 3 or 8) versus short-cutting it with an optical
> network because the silicon is so low-latency anyway that it hardly makes a
> difference now. Putting more GBs per second on fewer strands means saving a
> lot of money on infrastructure costs.

This is a very convoluted way of backing into the ole packet-switched
vs. circuit switched decision.

-- 
. ___ ___  .   .  ___
.  \/  |\  |\ \
.  _\_ /__ |-\ |-\ \__

Re: Routed optical networks

[Eve Griliches]

Hi Etienne,
Below is our (Cisco) definition of the Routed Optical Network. The goal,
metro or long haul or subsea, is to reduce the number of control planes. By
migration TDM traffic using CEM or PLE to the IP layer, you eliminate the
OTN control plane and management. Eventually, when standards are settled
the ultimate goal is to have a single control plane for the network. I'm
not trying to be a commercial here, but you can read more in the resources
section on this page:
https://www.cisco.com/c/en/us/solutions/service-provider/routed-optical-networking/index.html
HTH,
Eve

Routed optical networking, is an architecture that delivers improved
operational efficiencies and simplicity. The solution works by merging IP
and private line services onto a single layer where all the switching is
done at Layer 3. Routers are connected with standardized 400G ZR/ZR+
coherent pluggable optics.

With a single service layer based upon IP, flexible management tools can
leverage telemetry and model-driven programmability to streamline lifecycle
operations. This simplified architecture integrates open data models and
standard APIs, enabling a provider to focus on automation initiatives for a
simpler topology.

On Mon, May 1, 2023 at 2:30 PM Etienne-Victor Depasquale via NANOG <
nanog@nanog.org> wrote:

> Hello folks,
>
> Simple question: does "routed optical networks" have a clear meaning in
> the metro area context, or not?
>
> Put differently: does it call to mind a well-defined stack of technologies
> in the control and data planes of metro-area networks?
>
> I'm asking because I'm having some thoughts about the clarity of this
> term, in the process of carrying out a qualitative survey of the results of
> the metro-area networks survey.
>
> Cheers,
>
> Etienne
>
> --
> Ing. Etienne-Victor Depasquale
> Assistant Lecturer
> Department of Communications & Computer Engineering
> Faculty of Information & Communication Technology
> University of Malta
> Web. https://www.um.edu.mt/profile/etiennedepasquale
>

Re: Best Linux (or BSD) hosted BGP?

[Andy Davidson]

Hi, Bryan

You wrote:
> I know best subjective, but I'm looking at a project to announce some IP space
> that's between uses now and see what's there.  I'm planing to run a flow
> logger and ntop on the VM and see what is coming in if anything.  I'm looking
> at the options for BGP out there, and there's quite a few […]

Others have pointed several great open source BGP daemons which can interface
with Linux routing.  If you just want to ignore BGP for forwarding, and point a 
default
at one place, you might like to consider ExaBGP.  This is very lightweight - it 
can do
the BGP announcing of your prefix(es) without syncing routing changes to the 
Linux
routing table.  There are some simple AS112 instances, for example, delivered 
with
this model.  https://github.com/Exa-Networks/exabgp

Andy

Re: Best Linux (or BSD) hosted BGP?

[Uesley Correa]

Hi!

I like VyOS or FRR (both in Debian Linux).

Regards,
Uesley Corrêa - Analista de Telecomunicaciones
CEO Telecom Consultoria, Entrenamiento y Servicios
CEO Telecom Fiber Solutions


On Mon, May 1, 2023 at 3:58 PM Mark Tinka  wrote:

>
>
> On 5/1/23 20:04, Tomas Jonsson wrote:
>
> > VyOS uses FRR, but they used to run quagga.
> >
> > And most bsd(?)/linux package managers has frr in their repository so
> > maybe that could be something to look at?
>
> pfSense running FRR is a pretty solid BGP router.
>
> Mark.
>

Re: Best Linux (or BSD) hosted BGP?

[Charlie]

Agreed with Mark, used pfSense running FRR a few months ago without issues.

- Charlie

From: NANOG  on behalf of Mark Tinka 

Sent: 01 May 2023 08:55 PM
To: nanog@nanog.org 
Subject: Re: Best Linux (or BSD) hosted BGP?



On 5/1/23 20:04, Tomas Jonsson wrote:

> VyOS uses FRR, but they used to run quagga.
>
> And most bsd(?)/linux package managers has frr in their repository so
> maybe that could be something to look at?

pfSense running FRR is a pretty solid BGP router.

Mark.

Re: Best Linux (or BSD) hosted BGP?

[Nickolas Stevermer via NANOG]

Hi Bryan,

Openbsd project has openbgpd built right in. Simple and security-minded
implementation.

Presentations: https://www.openbgpd.org/papers.html
Testimonials: https://www.openbgpd.org/users.html

Regards,

Nick

-- 
Confidentiality Notice: This E-mail message, including any attachments, is 
for the sole use of the intended recipient(s) and may contain confidential 
and privileged information. Any unauthorized review, use, disclosure or 
distribution is prohibited. If you are not the intended recipient, please 
contact the sender by reply E-mail and destroy all copies of the original 
message.

Re: Routed optical networks

[Etienne-Victor Depasquale via NANOG]

Josh, thank you, your remarks (and those of Matt and Eduard) are helping me
to understand better.

For some context, please look at this graphic

that shows the results of the question
"Which of the following best describes your current dominant form of
metro-aggregation?"
The left bar chart shows *NOG reponses;
the right bar chart shows responses obtained through market research among
Tier 1 and regional operators.

Operator group respondents overwhelmingly selected "routed optical networks
over Ethernet without ROADMs".
Now, during an interview I held to assess the answers,
the interviewee (an experienced network engineer) questioned the meaning.

I realized that I may have been conditioned by Cisco marketing (I attend a
few webinars), and
I wanted to understand what respondents understood.

Summarizing an answer to your observation, I was conditioned by Cisco
marketing.

Cheers,

Etienne

On Mon, May 1, 2023 at 10:30 PM Josh Luthman 
wrote:

> Maybe some clarification as to what you're asking for would help.  You're
> mixing fiber, networks, and a MAN.  Fiber is just the medium.  It could be
> for IP switching or projecting a light show.  Are you asking if there are
> diverse paths throughout a metro area?
>
> On Mon, May 1, 2023 at 2:30 PM Etienne-Victor Depasquale via NANOG <
> nanog@nanog.org> wrote:
>
>> Hello folks,
>>
>> Simple question: does "routed optical networks" have a clear meaning in
>> the metro area context, or not?
>>
>> Put differently: does it call to mind a well-defined stack of
>> technologies in the control and data planes of metro-area networks?
>>
>> I'm asking because I'm having some thoughts about the clarity of this
>> term, in the process of carrying out a qualitative survey of the results of
>> the metro-area networks survey.
>>
>> Cheers,
>>
>> Etienne
>>
>> --
>> Ing. Etienne-Victor Depasquale
>> Assistant Lecturer
>> Department of Communications & Computer Engineering
>> Faculty of Information & Communication Technology
>> University of Malta
>> Web. https://www.um.edu.mt/profile/etiennedepasquale
>>
>

-- 
Ing. Etienne-Victor Depasquale
Assistant Lecturer
Department of Communications & Computer Engineering
Faculty of Information & Communication Technology
University of Malta
Web. https://www.um.edu.mt/profile/etiennedepasquale