Re: Opengear alternatives that support 5g?

[Mark Tinka]

On 4/27/24 07:56, Saku Ytti wrote:


For me Cisco is great here, because it's something an organisation
already knows how to source, turn-up, upgrade, troubleshoot, maintain.
And you get a broad set of features you might want, IPSEC, DMVPN, BGP,
ISIS, and so forth.


I tend to agree.

Cisco do this very well, and if you are really low on cash and okay with 
acquiring these on the cheap, the open market has tons of deals and 
options from Cisco that have matured over the decades.



I keep wondering why everyone is so focused on OOB hardware cost, when
in my experience the ethernet connection is ~200-300USD (150USD can be
just xconn) MRC. So in 10 years, you'll pay 24k to 36k just for the
OOB WAN, masking the hardware price. And 10years, to me, doesn't sound
even particularly long a time for a console setup.


Is a 10Mbps DIA link going for US$200 - US$300 MRC nowadays, excluding 
the x-connect? I'd have though it's now in US$100 range at the very worst.


Or are you looking at an OoB link of more than 10Mbps?

Mark.

Re: constant FEC errors juniper mpc10e 400g

[Mark Tinka]

On 4/22/24 09:47, Vasilenko Eduard via NANOG wrote:


Assume that some carrier has 10k FBB subscribers in a particular municipality 
(without any hope of considerably increasing this number).
2Mbps is the current average per household in the busy hour, pretty uniform 
worldwide.
You could multiply it by 8/7 if you like to add wireless -> not much would 
change.
2*2*10GE (2*10GE on the ring in every direction) is 2 times than needed to 
carry 10k subscribers.
The optical ring may be less than 20 municipalities - it is very common.
Hence, the upgrade of old extremely cheap 10GE DWDM systems (for 40 lambdas) 
makes sense for some carriers.
It depends on the population density and the carrier market share.
10GE for the WAN side would not be dead in the next 5 years because 2Mbps per 
household would not grow very fast in the future - this logistic curve is close 
to a plateau.
PS: It is probably not the case for Africa where new subscribers are connected 
to the Internet at a fast rate.


As a function of how much Internet there is in Africa, there really 
aren't that many optical transport service providers. Some 
countries/cities/towns have more than they need, others have just one. 
But in general, you would say there is massive room for improvement if 
you surveyed the entire continent.


Typically, it will be the incumbents, alongside 2 or 3 competitives. In 
fact, in some African countries, only the incumbent may be large enough 
to run an optical backbone, with all the competitives leasing capacity 
from them.


It is not uncommon to find the closest competitor to an incumbent for 
terrestrial services being the mobile network operator, purely because 
they have some excess capacity left over from having to build the 
backbone for their core business, mobile. And, they are flush with cash, 
so a loss-making terrestrial backhaul business can be covered by the 
month's sales in SIM cards.


Truly independent transport providers are few and far between because 
access to dark fibre is not easy (either its lack of availability, the 
incumbent refusing to sell it, or its high price). For the few 
independent transport providers that do spring up, they will focus on a 
limited set of hot routes, and because competition on those routes may 
be wanting, prices and capacity would not be terribly attractive.


So the bulk of Africa's Internet really relies on a handful of key 
African wholesale IP Transit providers taking great effort into 
extending their network as deep into cities as they can, and using their 
size to negotiate the best prices for terrestrial backhaul from the few 
optical network operators that the market has. Those providers then sell 
to the local and regional ISP's, who don't have to worry about running a 
backbone.


All this means is that for those operators that run an optical backbone, 
especially nationally, 10G carriers are very, very legacy. If they still 
have them, it'd be a spin-off off the main core to support some old SDH 
customers that are too comfortable to move to Ethernet.


Metro backhaul and last mile FNO's (fibre network operators) who have 
invested in extending access into homes and businesses are a different 
story, with most countries having a reasonable handful of options 
customers can choose from. Like national backhaul, there is plenty of 
room for improvement - in some markets more than others - but unlike 
national backhaul, not as constrained for choice or price.


Mark.

Re: constant FEC errors juniper mpc10e 400g

[Mark Tinka]

On 4/21/24 08:12, Saku Ytti wrote:


Key difference being, WAN-PHY does not provide synchronous timing, so
it's not SDH/SONET compatible for strict definition for it, but it
does have the frame format. And the optical systems which could
regenerate SONET/SDH framing, didn't care about timing, they just
wanted to be able to parse and generate those frames, which they
could, but they could not do it for ethernet frames.


Correct.

In those days, WAN-PHY was considered "SONET/SDH-Lite".

I think it is pretty clear, the driver was to support long haul
regeneration, so it was always going to be a stop-gap solution. Even
though I know some networks, who specifically wanted WAN-PHY for its
error reporting capabilities, I don't think this was majority driver,
majority driver almost certainly was 'thats only thing we can put on
this circuit'.


SONET/SDH did have similar reach as OTN back then, just less bandwidth 
for the distance. It had FEC support for STM-16, STM-64 and STM-256.


I really think the bigger driver was interface cost, because EoS had 
already been selling for 1GE alongside STM-16 for 2.5G. In those days, 
if you needed more than 1G but less than 10G, it was a toss-up between 
2x 1G EoSDH vs. 1x STM-16. Often times, you took the 2x 1G EoSDH because 
2x 1GE ports were cheaper than 1x STM-16 port, even though you ended up 
losing about 405Mbps of capacity in the process, which was a huge deal.


The backbone providers did not like selling EoSDH services, because it 
was too much admin. for them (VC container management), and they ended 
up paying more for transponders on their side than their customers did 
for Ethernet ports on theirs :-).


But by and large, the majority of networks in our market maintained SDH 
services long after coherent became commercially available. It was a 
perception thing, that SDH was more superior to Ethernet, even if that 
Ethernet was transported over a DWDM network.


In the end, SDH port costs were too hard to ignore due to router vendors 
maintaining their mark-up on them despite dying demand, which then led 
to the migration from SDH to EoDWDM growing significantly from about 
2016. Optical vendors also began de-prioritizing SDH transponder ports, 
which had a massive impact on the SLTE (submarine) side in making the 
decision to encourage customers away from SDH for wet services.


Mark.

Re: constant FEC errors juniper mpc10e 400g

[Mark Tinka]

On 4/20/24 21:36, b...@uu3.net wrote:


Erm, WAN-PHY did not extend into 40G because there was not much
of those STM-256 deployment? (or customers didnt wanted to pay for those).


With SONET/SDH, as the traffic rate increased, so did the number of 
overhead bytes. With every iteration of the data rate, the overhead 
bytes quadrupled. This was one of the key reasons we did not see field 
deployment of STM-256/OC-768 and STM-1024/OC-3072. For example, if 
SONET/SDH had to transport a 100G service, it would require 160Gbps of 
bandwidth. That clearly wasn't going to work.


At the time when STM-256/OC-768 was being developed, DWDM and OTN had 
come a long way. The granularity SONET/SDH required to stand up a 
service had become too small for the growing data rate (primarily VC-3, 
VC-4 and VC-12). If you look at OTN, the smallest container is 1.25Gbps 
(ODU0), which was attractive for networks looking to migrate from E1's, 
E3's, STM-1's, STM-4's and STM-16's - carried over VC-12, VC-4 and VC-3 
SDH circuits - to 1GE, for example, rather than trying to keep their 
PDH/SDH infrastructure going.


DWDM and OTN permitted a very small control overhead, so as data rates 
increased, there wasn't the same penalty you got with SONET/SDH.

WAN-PHY was designed so people could encapsulate Ethernet frames
right into STM-64. Once world moved out of SDH/SONET stuff, there was
no more need for WAN-PHY anymore.


Technically, what you are describing is EoS (Ethernet over SONET, 
Ethernet over SDH), which is not the same as WAN-PHY (although the 
working groups that developed these nearly confused each other in the 
process, ANSI/ITU for the former vs. IEEE for the latter).


WAN-PHY was developed to be operated across multiple vendors over 
different media... SONET/SDH, DWDM, IP/MPLS/Ethernet devices and even 
dark fibre. The goal of WAN-PHY was to deliver a low-cost Ethernet 
interface that was SONET/SDH-compatible, as EoS interfaces were too 
costly for operators and their customers.


As we saw in real life, 10GE ports out-sold STM-64/OC-192 ports, as 
networks replaced SONET/SDH backbones with DWDM and OTN.


Mark.

Re: constant FEC errors juniper mpc10e 400g

[Mark Tinka]

On 4/20/24 18:19, Tarko Tikan wrote:

10G wavelengths for new builds died about 10 years ago when coherent 
100G became available, submarine or not. Putting 10G into same system 
is not really feasible at all.


I was referring to 10G services (client-side), not 10G wavelengths (line 
side).


Mark.

Re: constant FEC errors juniper mpc10e 400g

[Mark Tinka]

On 4/20/24 14:41, Dave Cohen wrote:


LAN PHY dominates in the US too. Requests for WAN PHY were almost exclusively 
for terrestrial backhaul extending off of legacy subsea systems that still 
commonly had TDM-framed services. It’s been a couple of years since I’ve been 
in optical transport directly but these requests were essentially non-existent 
after 2018 or so. OTN became somewhat more common from 2014 onward as optical 
system interop improved, but actually was more common in the enterprise space 
as providers would generally go straight to fiber in most use cases, and with 
dark fiber opex costs coming down in many markets, I see OTN requests as 
winnowing here as well.


What really changed the game was coherent detection, which breathed new 
life into legacy subsea cables that were built on dispersion-managed 
fibre. Post-2014 when uncompensated (and highly dispersed) fibre has 
been the standard for subsea builds (even for SDM cables), coherent 
optical systems are the mainstay. In fact, because linear dispersion can 
be accurately calculated for the cable span, uncompensated cables are a 
good thing because the dispersion compensation happens in very advanced 
coherent DSP's in the optical engine, rather than in the fibre itself.


WAN-PHY did not extend to 40G or 100G, which can explain one of the 
reasons it lost favour. For 10G, its availability also depended on the 
type of device, its NOS, line card and/or pluggable at the time, which 
made it hard to find a standard around this if you built multi-vendor 
networks or purchased backhaul services from 3rd party providers that 
had non-standard support for WAN-PHY/OTN/G.709. In other words, LAN-PHY 
(and plain Ethernet) became the lowest common denominator in the 
majority of cases for customers.


In 2024, I find that operators care more about bringing the circuit up 
than using its link properties to trigger monitoring, failover and 
reconvergence. The simplest way to do that is to ask for plain Ethernet 
services, particularly for 100G and 400G, but also for 10G. In practice, 
this has been reasonably reliable in the past 2 - 3 years when procuring 
100G backhaul services. So for the most part, users of these services 
seem to be otherwise happy.


Mark.

Re: constant FEC errors juniper mpc10e 400g

[Mark Tinka]

On 4/20/24 13:39, Saku Ytti wrote:


Oh I don't think OTN or WAN-PHY have any large deployment future, the
cheapest option is 'good enough'...


And what we find with EU providers is that Ethernet and OTN services are 
priced similarly. It's a software toggle on a transponder, but even 
then, Ethernet still continues to be preferred over OTN.


Mark.

Re: constant FEC errors juniper mpc10e 400g

[Mark Tinka]

On 4/20/24 13:39, Saku Ytti wrote:


Oh I don't think OTN or WAN-PHY have any large deployment future, the
cheapest option is 'good enough' and whatever value you could extract
from OTN or WAN-PHY, will be difficult to capitalise, people usually
don't even capitalise the capabilities they already pay for in the
cheaper technologies.


A handful of OEM's still push OTN like it has just been invented, 
especially those still pushing "IPoDWDM" :-). Fair point, if you have a 
highly-meshed metro network with lots of drops to customers across a 
ring-mesh topology, there might be some value in OTN when delivering 
such services at low speeds (10G, 25G, 2.5G, 1G). But while the topology 
is valid, most networks aren't using high-end optical gear to drop 
low-speed services, nowadays. Even though on a per-bit basis, they might 
be cheaper than 1U IP/MPLS router looking to do the same job if all you 
are considering is traffic, and not additional services that want to eat 
packets.



Of course WAN-PHY is dead post 10GE, a big reason for it to exist was
very old optical systems which simply could not regenerate ethernet
framing, not any features or functional benefits.


In our market, we are trending toward a convergence between 10G and 100G 
orders intersecting for long haul and submarine asks. But pockets of 10G 
demand still exist in many African countries, and none of them have any 
WAN-PHY interest of any statistical significance.


That said, I don't expect any subsea cables getting built in the next 3 
years and later will have 10G as a product on the SLTE itself... it 
wouldn't be worth the spectrum.


Mark.

Re: constant FEC errors juniper mpc10e 400g

[Mark Tinka]

On 4/20/24 13:25, Saku Ytti wrote:


In most cases, modern optical long haul has a transponder, which
terminates your FEC, because clients offer gray, and you like
something a bit less depressing, like 1570.42nm.

This is not just FEC terminating, but also to a degree autonego
terminating, like RFI signal would be between you and transponder, so
these connections can be, and regularly are, provided without proper
end-to-end hardware liveliness, and even if they were delivered and
tested to have proper end-to-end HW liveliness, that may change during
operation, so line faults may or may not be propagated to both ends as
RFI assertion, and even if they are, how delayed they are, they may
suffer delay to allow for optical protection to engage, which may be
undesirable, as it eats into your convergence budget.

Of course the higher we go in the abstraction, the less likely you are
to get things like HW livelines detection, like I don't really see
anyone asking for this in their pseudowire services, even though it's
something that actually can be delivered. In Junos it's a single
config stanza in interface, to assert RFI to client port, if
pseudowire goes down in the operator network.


In our market (Africa), for both terrestrial and submarine services, 
OTN-type circuits are not typically ordered. Network operators are not 
really interested in receiving the additional link data that OTN or 
WAN-PHY provides. They truly want to leave the operation of the 
underlying transport backbone to the transport operator.


The few times we have come across the market asking for OTN is if they 
want to groom 10x 10G into 1x 100G, for example, to deliver structured 
services downstream.


Even when our market seeks OTN from European backhaul providers to 
extend submarine access into Europe and Asia-Pac, it is often for 
structured capacity grooming, and not for OAM benefit.


It would be interesting to learn whether other markets in the world 
still make a preference for OTN in lieu of Ethernet, for the OAM 
benefit, en masse. When I worked in Malaysia back in the day (2007 - 
2012), WAN-PHY was generally asked for for 10G services, until about 
2010; when folk started to choose LAN-PHY. The reason, back then, was to 
get that extra 1% of pipe bandwidth :-).


Mark.

Re: constant FEC errors juniper mpc10e 400g

[Mark Tinka]

On 4/19/24 10:08, Saku Ytti wrote:


Of course there are limits to this, as FEC is hop-by-hop, so in
long-haul you'll know about circuit quality to the transponder, not
end-to-end. Unlike in wan-phy, OTN where you know both.

Technically optical transport could induce FEC errors, if there are
FEC errors on any hop, so consumers of optical networks need not have
access to optical networks to know if it's end-to-end clean.


This would only matter on ultra long haul optical spans where the signal 
would need to be regenerated, where - among many other values - FEC 
would need to be decoded, corrected and re-applied.


SD-FEC already allows for a significant improvement in optical reach for 
a given modulation. This negates the need for early regeneration, 
assuming other optical penalties and impairments are satisfactorily 
compensated for.


Of course, what a market defines as long haul or ultra long haul may 
vary; add to that the variability of regeneration spacing in such 
scenarios being quite wide, on the order of 600km - 1,000km. Much of 
this will come down to fibre, ROADM and coherent pluggable quality.


Mark.

Re: constant FEC errors juniper mpc10e 400g

[Mark Tinka]

On 4/19/24 10:08, Saku Ytti wrote:


Of course there are limits to this, as FEC is hop-by-hop, so in
long-haul you'll know about circuit quality to the transponder, not
end-to-end. Unlike in wan-phy, OTN where you know both.

Technically optical transport could induce FEC errors, if there are
FEC errors on any hop, so consumers of optical networks need not have
access to optical networks to know if it's end-to-end clean.


This would only matter on ultra long haul optical spans where the signal 
would need to be regenerated, where - among many other values - FEC 
would need to be decoded, corrected and re-applied.


SD-FEC already allows for a significant improvement in optical reach for 
a given modulation. This negates the need for early regeneration, 
assuming other optical penalties and impairments are satisfactorily 
compensated for.


Of course, what a market defines as long haul or ultra long haul may 
vary; add to that the variability of regeneration spacing in such 
scenarios being quite wide, on the order of 600km - 1,000km. Much of 
this will come down to fibre, ROADM and coherent pluggable quality.

Re: constant FEC errors juniper mpc10e 400g

[Mark Tinka]

On 4/19/24 08:01, Saku Ytti wrote:


The frames in FEC are idle frames between actual ethernet frames. So
you recall right, without FEC, you won't see this idle traffic.

It's very very good, because now you actually know before putting the
circuit in production, if the circuit works or not.

Lot of people have processes to ping from router-to-router for N time,
trying to determine circuit correctness before putting traffic on it,
which looks absolutely childish compared to FEC, both in terms of how
reliable the presumed outcome is and how long it takes to get to that
presumed outcome.


FEC is amazing.

At higher data rates (100G and 400G) for long and ultra long haul 
optical networks, SD-FEC (Soft Decision FEC) carries a higher overhead 
penalty compared to HD-FEC (Hard Decision FEC), but the net OSNR gain 
more than compensates for that, and makes it worth it to increase 
transmission distance without compromising throughput.


Mark.

Re: constant FEC errors juniper mpc10e 400g

[Mark Tinka]

On 4/18/24 15:45, Tom Beecher wrote:



 Just for extra clarity off those KB, probably has nothing to do with 
vendor interop as implied in at least one of those.


Yes, correct.

Mark.

Re: constant FEC errors juniper mpc10e 400g

[Mark Tinka]

On 4/17/24 23:24, Aaron Gould wrote:

Well JTAC just said that it seems ok, and that 400g is going to show 
4x more than 100g "This is due to having to synchronize much more to 
support higher data."




We've seen the same between Juniper and Arista boxes in the same rack 
running at 100G, despite cleaning fibres, swapping optics, moving ports, 
moving line cards, e.t.c. TAC said it's a non-issue, and to be expected, 
and shared the same KB's.


It's a bit disconcerting when you plot the data on your NMS, but it's 
not material.


Mark.

Serious Bug in Cisco's 6500 & 6800 Platforms

[Mark Tinka]

https://sec.cloudapps.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-ios-dos-Hq4d3tZG

Mark.

Re: Unimus as NCM (Network Configuration Management) Tool

[Mark Tinka]

On 4/4/24 08:25, Mike Lyon wrote:


I use it for config backups, diffs, etc. Love it.

Theres others such as Rancid but im not sure if it works on anything 
other than Vendor C.


RANCID works perfectly for Cisco, Juniper, Arista, Brocade (Foundry) and HP.

They are also known to support other obscure vendors.

Mark.

Re: network simulator for service provider

[Mark Tinka]

On 4/3/24 04:13, aun Joe wrote:


  is there anysi  network simulator for carrier networks ?
   well,   from 2023 to 2024 there happes so many carrier network 
outage caused by network operation.
   to my limited knowledge ,   SP guru  from riverbed could simulate 
carrier network. but  I just checked

 riverbed website,  SP guru  stop updating in 2014.


https://www.boson.com/netsim-cisco-network-simulator
https://www.netacad.com/courses/packet-tracer
https://www.gns3.com/

There is a bunch of others, but that should get your started down the 
rabbit hole.


Mark.

Re: N91 Women mixer on Sunday?

[Mark Tinka]

On 3/29/24 21:48, Matthew Petach wrote:

I'm with Randy Bush on this.  The stakeholders in that event should 
have the say in what happens with it; not the rest of us.


While I agree that women would know best how they want to socialize for 
their own interests, I don't think that men engaging in this discussion 
means they are necessarily prescribing how women should do that. All I 
have seen so far are opinions and suggestions to a concern the OP 
raised, not dictation on what will eventually happen.



Those of us old white males need to check our privilege, and recognize 
that we've *been* having "mixers" for decades.
We don't need to put a stake in the ground and push for our equality; 
we've already been on the beneficiary side of the

inequality for decades.


I also do not think that male guilt is necessary for women to achieve 
good outcomes in our shared industry. A complimentary working 
relationship could be remarkably more productive, than not.


Mark.

Re: N91 Women mixer on Sunday?

[Mark Tinka]

On 3/30/24 03:45, Ryan Hamel wrote:


Paul,

Anne's opinion is just as valid as the others here. I have also 
browsed through the recent attendee lists and do not see you listed 
either, pot calling the kettle black. Your comments about her email 
signature and which voices are valid here, are not productive. We are 
allowed to back up and/or side with whomever we please, even if it 
includes the NANOG board, staff, and committee members. We are also 
allowed to call people out on their behavior towards others.


Agreed. Constructive dialogue makes no requirement for prior or final 
agreement.


More talking is better than less talking, especially when we disagree.




Anyway, no one truly knows how many people could have raised the 
scheduling issue with various committee members, the board, staff, or 
provided feedback via the contact form on the website, and who knows 
it could have come from young women. Those voices do not have to come 
from the mailing list, to be just as valid as ours.


I'd imagine that data would, at the very least, be with the PC and other 
members of NANOG staff; and in a manner that they can use to derive 
reasonable solutions for the future.


Mark.

Re: N91 Women mixer on Sunday?

[Mark Tinka]

On 3/29/24 18:31, Tom Beecher wrote:


I don't think anything is 'easily fixed' on this topic.

I do however hope that everyone accepts at face value that the board, 
staff, and PC are *trying* to do the right things here. Doesn't mean 
that it *will* always be the right thing, or even that a majority of 
people can agree on what the right thing actually is.


As I always say on matters such as these, "More talking is better than 
less talking".


And as uncomfortable as such discussions are to have, we get closer to a 
reasonable solution when we talk, and less so when we don't.


Mark.

Re: N91 Women mixer on Sunday?

[Mark Tinka]

On 3/29/24 18:23, Tom Beecher wrote:



My recollection is that every WiT lunch was always open to all. Happy 
to be corrected if any were that I don't recall.


There were definitely a few meetings during my PC years that someone 
complained that men were not allowed to attend. If my memory serves me 
correctly, we at one point were asking session moderators to remind 
people of this in general session for a while too. For some meetings, 
a few of us were standing at the doors telling people who asked that 
men were allowed to attend that if they would like.


I can't remember which year it was, but my recollection was hearing the 
complaints from the men that were upset during the evening social. 
Admittedly, I did not hear this from what I would term "the majority of 
men" at that specific meeting, so it would be disingenuous to state, 
with any authority, that this is how the majority of men felt/feel about 
the WiT lunch.


Having said all that, for the avoidance of doubt (and as Tina has 
already indicated for NANOG 91), it might be worth mentioning, before 
and throughout the conference, that while it is a WiT lunch, all 
attendees are welcome to join if that is, indeed, the intended format.


Mark.

Re: N91 Women mixer on Sunday?

[Mark Tinka]

On 3/29/24 07:03, Eric Parsonage wrote:

It's easily fixed by having a mixer at the same time for the other 
half of the gathering population thus showing all the population 
gathering matters equally.


My memory is a little fuzzy, but I think I recall one of the early WiT 
lunches hosted at NANOG that was women-only, where some men were upset 
for being "left out". Whether that was good or bad is less important 
than understanding what the outcome of a women-only activity is for 
women, especially for those for whom it may not be immediately obvious.


While equal access to opportunity between the genders is the most 
effective policy, I think there is utility in women having their own 
session, given that they face unique challenges in an industry where 
they are the minority operators.


Mark.

Re: N91 Women mixer on Sunday?

[Mark Tinka]

On 3/29/24 06:20, Ren Provo wrote:

I beg to differ here and second Ilissa’s comments.  I miss WiT.  Lunch 
during the meeting worked.  Giving up more of the weekend to travel 
does not show half the population gathering matters.


It would be interesting to survey the % of attending women who:

    - Would be able to make the mixer.
    - Would not be able to make the mixer due to family commitments.
    - Would not be able to make the mixer due to non-family commitments.
    - Would prefer a different women's meeting format and/or activity.

I think this data would be useful because while some women may voice 
difficulty with attending the mixer (or wanting a different women's 
meeting format/activity), it might be premature to attribute that 
difficulty to all women that would be in attendance.


Understanding where the majority of women lie can help the NANOG PC make 
better decisions about how to support both the majority and minority of 
women as it pertains to a mixer or other activity women may like to 
participate in at or around the conference.


Mark.

Re: N91 Women mixer on Sunday?

[Mark Tinka]

On 3/28/24 21:08, Tom Beecher wrote

There was a Women in Tech Mixer on Sunday in Charlotte as well. As I 
recall there was a pretty decent attendance.


During my time on the PC, we always got a lot of feedback about Sunday 
when the topic came up. Some members were strongly opposed to anything 
on Sunday and didn't even like the Hackathon there. Others wanted 
expansion, and more things slotted in. There certainly wasn't 
anything remotely close to a consensus. Sometimes people can make it 
in early on Sunday. Sometimes they can't. There's no one size fits all 
answer.


I'm not sure people realize how much crap that staff and the PC get 
*every meeting* about the agenda. There's always someone unhappy 
because this event wasn't the same, or why was it in this room over 
here, or OMG Wed afternoon, etc. Having seen how that sausage gets 
made, they don't get enough credit.


In my opinion, they found a spot that they had room for, and if people 
can make it with their schedules, then great. If not, hopefully a 
future slot can work.


Typical constraints such as scheduling and resources notwithstanding, 
100% participation is not often guaranteed in most things. It's about 
planning for as many as can make it. With some luck, it would be the 
majority.


Mark.

Re: N91 Women mixer on Sunday?

[Mark Tinka]

On 3/28/24 19:45, Ilissa Miller wrote:

For those that know me, I rarely provide constructive input 
about NANOG matters due to my past affiliation, however, I just saw 
that NANOG announced the Women mixer on Sunday before NANOG 91 and am 
outraged for all of the young professional women who would like to 
participate in NANOG.  While the times are changing, women continue to 
remain primary caregivers for families and this will require them to 
desert their families a day early.  I find it offensive personally and 
feel like you may have missed the mark.


Minds are hard to read, so asking the question before being offended is 
not an unproductive endeavour. That said, we are here now.



The amount of times I hear people complain about having to leave their 
families is one of the reasons this industry has a problem keeping 
young people - especially women.


Does anyone else feel the same?


If you have a better suggestion on what would work best, I'd be keen to 
hear it.


Mark.

Re: Best TAC Services from Equipment Vendors

[Mark Tinka]

Ultimately, I think every vendor will have customers with good 
experiences, and customers with not-so-good experiences. Without 
sufficient data, it is hard to say which vendor, in general, does a good 
job for their customer base re: TAC.


What I will say, albeit anecdotally also, is that TAC experience with 
the vendor will often be good if you have access to an SE on a regular 
basis (i.e., one or two in your country of abode). It will even be 
better if that SE is naturally diligent. Which means that oftentimes, it 
will come down to one person, to give you a sense of how well a large 
organization like an OEM treats your business. And if that one person 
were to leave and move on, the gorilla that is the OEM could come 
crashing down on you in ways you didn't think was possible.


So yes, while it is possible to institutionalize processes and such, 
like with everything else in life, your experience is very likely to 
come down to one or two people that care deeply enough to (want to) make 
a difference.


For me, if I have ever have to have a relationship with a vendor, it is 
a priority that I establish a close relationship with someone on their 
sales and engineering team, as that will determine how much support I 
get if I ever run into trouble. And OEM's that were once great at this 
can quickly become the worst you've ever seen, if those two people were 
no longer there or stopped caring. In other words, there are no 
forever-guarantees, and it comes and goes in cycles.


Mark.

Re: XGS-PON & "Dedicated" Service

[Mark Tinka]

On 10/25/23 01:56, Neader, Brent wrote:


Hello!

Interested in getting the larger community’s thought on this.

The primary question being does XGS-PON have a place in providing a 
dedicated enterprise level service (at least sold as one) in the 
marketplace?  Delivered via a residential (per the data sheet 
description) CPE, Nokia XS-010X-Q for a 1gb/1gb dedicated symmetrical 
service.


Background, ive dealt with 30+ providers over the last 18 years, 
primarily last mile based.  Typically we seek out an 
Enterprise/Dedicated service, with an SLA, typically delivered via 
DWDM, CWDM, or AE, or equivalent.  We have also had a site or two 
delivered via a PON variant, typically with less of an SLA, typically 
maybe half to quarter of the price of a dedicated service.  Price & 
SLA sets the expectation of the service, CPE provided, underlying 
technology, etc.


Dealing with a large over-builder right now who has an “elite” 
enterprise product (highest of 3 tiers) advertised as the following.


-100% dedicated bandwidth so you never have to compete for speed

-Mission Critical Reliability with 99.999% guaranteed uptime

-Financially backed SLA with the most stringent performance objectives

-Enterprise-level customer service and technical support

Now I understand with XGS, you can have various QOS in place (WRR/SP, 
etc), but inherently there are still shared splits involved, that just 
aren’t a thing in other truly dedicated technologies.  Expectations 
were set with the provider’s sales team around what was to be 
delivered and how it was to be delivered that seemingly haven’t been 
met by the product and service team.


That aside, from an SP perspective, is it capable to wrap enough 
layers around service to be “dedicated” even when delivered via a 
conflicting underlying technology? Or could that be considered 
disingenuous for those that want to know and understand the 
difference?  Im hoping the service itself and support team make up for 
the difference, but obviously a little concerned.




Regular GPON is already being used to deliver Enterprise services, 
purely because it "passes by the office complex" on its way to the 
residential neighborhood. Even when the Sales team are told not to use 
GPON for Enterprise services, they end up doing so... first as a 
"temporary, we have told the customer all the pitfalls" solution, which 
eventually becomes permanent, and then it grows like wildfire.


You can expect that XG-PON will go the same way.

Mark.

Re: RPKI unknown for superprefixes of existing ROA ?

[Mark Tinka]

On 10/21/23 16:03, Amir Herzberg wrote:


Hi Owen, Randy, Job and other NANOGers,

I surely agree with you all that we shouldn't expect discarding of 
ROA-unknown `anytime soon' (or ever?). But I have a question: what 
about discarding ROA-unknowns for very large prefixes (say, /12), or 
for superprefixes of prefixes with announced ROAs? Or at least, for 
superprefixes of prefixes with ROA to AS 0?


For motivation, consider the `superprefix hijack attack'. Operator has 
prefix 1.2.4/22, but announce only 1.2.5/24 and 1.2.6/24, with 
appropriate ROAs. To avoid abuse of 1.2.4/24 and 1.2.7/24, they also 
make a ROA for 1.2.4/22 with AS 0. Attacker now announces 1.2.0/20, 
and uses IPs in 1.2.4/24 and 1.2.7/24 to send spam etc.. We introduced 
this threat and analyzed it in our ROV++ paper, btw (NDSS'21 I think - 
available online too of course).


So: would it be conceivable that operators will block such 1.2.0/20  - 
since it's too large a prefix without ROA and in particular includes 
sub-prefixes with ROA, esp. ROA to AS 0?


The question is - who gets to decide how much space is "too large"?

"Too large" will most certainly be different for different networks.

If we try to get the RPKI to do things other than for which it was 
intended which may be interpreted as "unreasonable control", we make the 
case for those who think that is what it was destined to become.


Mark.

Re: Low to Mid Range DWDM Platforms

[Mark Tinka]

On 10/19/23 22:24, Chad Lamb via NANOG wrote:

XKL is still here ...
Some caveats on 400G ZR/ZR+ devices:
- They are power hungry, cooling is a challenge.  800G even more so.  
The OSFP package for 800G looks like the better choice than QSFP-DD 
for this reason.  We'll see, we need more mileage on this.
- For the ZR devices, not all vendors are equal.  Some support 4x100GE 
as well as 400GE, if that matters to you.  Some have integrated BERT 
and encryption, some do not.
- Do not plan on inter-operating different vendors of ZR+ devices.  
They will interoperate only in ZR mode.
- Plenty of filter options out there if you are putting the ZR/ZR+ in 
your router.  I'll throw a plug in for the Coherent free space DWDM 
filter. Free space technology has the best IL and isolation numbers.  
Coherent hasn't packaged it in a module for easy racking yet (XKL can 
provide that).  So if you aren't adding an EDFA, the lowest IL you can 
get helps your reach.
- If you have at least a few 400G channels, add an EDFA and use the Tx 
= -10dBm devices, rather than the Tx = 0dBm devices.  This is cost 
effective.  The EDFA also extends the reach considerably. You can 
launch each channel up to +10dBm or more, for a modest number of 
channels, without getting in trouble with fiber non-linearities 
(four-wave mixing).


Many thanks, Chad.

Glad to hear XKL are still in business, and doing well :-).

Mark.

Re: MX204 tunnel services BW

[Mark Tinka]

On 10/17/23 03:20, Ryan Kozak wrote:



"The MX204 router supports two inline tunnels - one per PIC. To 
configure the tunnel interfaces, include the tunnel-services statement 
and an optional bandwidth of 1 Gbps through 200 Gbps at the \[edit 
chassis fpc fpc-slot pic number\] hierarchy level. If you do not 
specify the tunnel bandwidth then, the tunnel interface can have a 
maximum bandwidth of up to 200 Gbps."


If JTAC is saying it's no longer optional they need to update their docs.


We can commit "tunnel-services" on an MX204 without caveat.

Mark.

Re: MX204 tunnel services BW

[Mark Tinka]

On 10/16/23 21:49, Jeff Behrns via NANOG wrote:


Also leaving tunnel-services bandwidth unspecified is not possible on the 204.


This is true of other MX platforms as well, unless I misunderstand.

Mark.

APRICOT 2024: Call for Presentations

[Mark Tinka]

APRICOT 2024
21st February - 1st March, Bangkok, Thailand
https://2024.apricot.net

CALL FOR PRESENTATIONS
=

The APRICOT 2024 Programme Committee is now seeking contributions for 
Presentations and Tutorials for the APRICOT 2024 Conference.


We are looking for presenters who would:

- Offer a technical tutorial on an appropriate topic;
- Participate in the technical conference sessions as a speaker;
- Convene and chair panel sessions of relevant topics;
- Lead informal Birds of Feather break out sessions.

Please submit on-line at:

https://papers.apricot.net/user/login.php?event=186

CONFERENCE MILESTONES
--

Call for Papers Opens:       Now
Outline Programme Published:    As Papers Confirmed
Final Deadline for Submissions:  29th January 2024
Final Program Published:            5th February 2024
Final Slides Received:                19th February 2024

*SLOTS ARE FILLED ON A FIRST COME, FIRST SERVED BASIS, REGARDLESS OF 
PUBLISHED DEADLINES*


PROGRAMME CONTENT
-

The APRICOT Conference Programme consists of three parts, these being 
Tutorials, the Peering Forum, and Conference Sessions.


Topics proposed must be relevant to Internet Operations and 
Technologies, for example:


- IPv4 / IPv6 Routing and Operations
- Routing Security, including RPKI and MANRS
- Internet backbone operations
- Peering, Interconnects and IXPs
- Content Distribution Network technology & operations
- Research on Internet Operations and Deployment
- Network Virtualisation
- Network Automation/Programming
- Network Infrastructure security
- IPv6 deployment on fixed and Wireless/Cellular networks
- DNS / DNSSEC and KINDNS
- Access and Transport Technologies
- Technical application of Web 3.0, public blockchains and cryptocurrency
- Content & Service Delivery and "Cloud Computing"
- 400G ZR, ZR+ & Open ROADM
- Submarine cables
- ChatGPT

CfP SUBMISSION
-

Draft slides for both tutorials and conference sessions MUST be provided 
with CfP submissions otherwise the submission will be rejected 
immediately. For work in progress, the most current information 
available at the time of submission is acceptable.


All draft and complete slides must be submitted in PDF format only. 
Slides must be of original work, with all company confidential marks 
removed.


Any marketing, sales and vendor proprietary content in a presentation is 
against the spirit of APRICOT and it is strictly prohibited.


Note that proper credit must be given for all content taken from other 
sources. Evidence of plagiarism is grounds for rejection.


Final slides are to be provided by the specified deadline for 
publication on the APRICOT website.


Prospective presenters should note that the majority of speaking slots 
will be filled well before the final submission deadline. The PC may, at 
their discretion, retain a limited number of slots up to the final 
submission deadline for presentations that are exceptionally timely, 
important, or of critical operational importance. Every year we turn 
away submissions, due to filling up all available programme slots before 
the deadline. Presenters should endeavour to get material to the PC 
sooner rather than later.


Any questions or concerns should be addressed to the Programme Committee 
Chairs by e-mail at:


    pc-chairs at apricot.net

We look forward to receiving your presentation proposals.

Mark Tinka, Achie Atienza & Mark Duffell
APRICOT 2024 Programme Committee Chairs
--

Re: FastNetMon Usage in the wild

[Mark Tinka]

On 10/11/23 04:34, Dobbins, Roland via NANOG wrote:


To clarify, Sightline has supported virtualization for many years, FYI.


It does do, yes. But pricing for the software license is not too far off 
from if you chose to buy Netscout's own hardware.


Not a major drama for me - I appreciate that competence has to be 
compensated. What I am saying is that attempts to make it more palatable 
to more operators are not making too much of a dent.


Mark.

Re: Low to Mid Range DWDM Platforms

[Mark Tinka]

Finally, the name came to me :-):

https://www.xkl.com/

Looks like they are still in business.

Mark.

On 10/6/23 16:02, Mark Tinka wrote:



On 10/6/23 15:52, Dave Bell wrote:

Smartoptics?

https://smartoptics.com/


Not them.

I want to say they had an "X" in their name, but memory is fuzzy.

Mark.

Re: maximum ipv4 bgp prefix length of /24 ?

[Mark Tinka]

On 10/7/23 14:32, Willy Manga wrote:

How about we educate each other to not assume you must deaggregate 
your prefix especially with IPv6?


I see 'some' (it's highly relative) networks on IPv4, they 'believe' 
they have to advertise every single /24 they have. And when they start 
with IPv6, they replicate the same mindset with a tons of /48 . You 
can imagine what will happen of course.


A better alternative IMHO is to take advantage to the large prefix 
range and advertise a sub-aggregate when necessary. But absolutely not 
each end-node or customer prefix.


There are a number of operational reasons folk de-aggregate. I do agree 
that there is some behaviour around de-aggregating by default in IPv4 
that transferred to IPv6. But the main issue is that most people only 
consider the state of their own FIB situation. They hardly consider the 
FIB state of other network operators around the world.


As an operator, you have to consciously decide that you will not 
de-aggregate any of your allocations. Of course, there is a cost to that 
as well, so that cannot be ignored. We, for example, do not de-aggregate 
any of our allocations (AS37100), but we can afford to do so because we 
always ensure all peering and transit exit/entry points have the same 
bandwidth (TE being the main reason networks de-aggregate). Not all 
shops can afford that.


Network operations workshops abound where we teach about de-aggregation, 
when it can be necessary, and why it should generally be avoided unless 
in the most extreme of circumstances. However, in real life, even 
engineers that have been through the workshop ringer tend to prefer to 
de-aggregate without caution to the FIB state of other autonomous 
systems. That said, I do agree that, perhaps, network operations 
workshops could emphasize the reluctance to unnecessarily de-aggregate 
in light of the increasing cost of having to maintain FIB's.


Mark.

Re: Low to Mid Range DWDM Platforms

[Mark Tinka]

On 10/6/23 15:53, Dave Cohen wrote:

My experience is primarily with the traditional carrier-grade folks 
like Ciena, Infinera, etc. but over the last decade all of those 
vendors have focused on improving how they scale down without 
sacrificing (most of the) quality and functionality - to varying 
degrees of success. There are also some more recent entrants that 
built their products to target the DCI market but rather than focusing 
on bandwidth density have focused on cost per bit for a mid-range 
solution. There are almost definitely multiple quality options out 
there without having to buy the full 88 channel n-degree ROADM Ciena 
6500 that takes up a full rack - although given the stated 
requirements, there may not be a one-size-fits-all solution that's 
ideal for all of the OP's projects.


There are two markets driving the major vendors right now - DCI and subsea.

The regular longhaul terrestrial gear is mostly the same, bar for 
upgrades to the latest CMOS. For these vendors, the same gear is also 
re-used for their subsea solutions, although with "submarine" versions 
of their terrestrial line cards.


Adva's TeraFlex is a bit different in that it is optimized both for DCI 
and longhaul terrestrial. We are currently looking at it for a 700km 
deployment in one of our markets. Very impressive!


Another entrant into the market is Acacia, now known as the Cisco 
NCS1000. They also use the same platform for both terrestrial and subsea:


https://www.cisco.com/c/en/us/products/optical-networking/network-convergence-system-1000-series/index.html

Mark.

Re: Low to Mid Range DWDM Platforms

[Mark Tinka]

On 10/6/23 15:52, Dave Bell wrote:

Smartoptics?

https://smartoptics.com/


Not them.

I want to say they had an "X" in their name, but memory is fuzzy.

Mark.

Re: Low to Mid Range DWDM Platforms

[Mark Tinka]

On 10/6/23 15:07, Mike Hammett wrote:


I've been using various forms of passive WDM for years. I have a couple 
different projects on my plate that require me to look at the next level of 
platform.

In some projects, I'll be looking for needing to have someone long distances of 
glass without any electronics. Some spans could be over 60 miles.

In some projects, I'll need to transport multiple 100-gig waves.

What is the landscape like between basic passive and something like a 30 
terabit Ciena? I know of multiple vendors in that space, but I like to learn 
more about what features I need and what features I don't need from somewhere 
other than the vendor's mouth. Obviously, the most reliability at the least 
cost as well.


400G-ZR pluggables will get you 400Gbps on a p2p dark fibre over 80km - 
100km. So your main cost there will be routers that will support.


The smallest DCI solution from the leading DWDM vendors is likely to be 
your cheapest option. Alternatively, if you are willing to look at the 
open market, you can find gear based on older CMOS (40nm, for example), 
which will now be EoL for any large scale optical network, but cost next 
to nothing for a start-up with considerable capacity value.


There is a DWDM vendor that showed up on the scene back in 2008 or 
thereabouts. They were selling a very cheap, 1U box that had a different 
approach to DWDM from other vendors at the time. I, for the life of me, 
cannot remember their name - but I do know that Randy introduced them to 
me back then. Maybe he can remember :-). Not sure if they are still in 
business.


Mark.

APRICOT 2024: Call for Programme Committee Volunteers

[Mark Tinka]

Hi everyone.

The APRICOT 2024 Organising Committee would like to welcome everyone to 
join us in Bangkok, Thailand, from 21st February to 1st March 2024.


The APRICOT 2024 Programme Committee is responsible for the solicitation 
and selection of suitable presentation and tutorial content for the 
APRICOT 2024 conference (https://2024.apricot.net/).


We are now seeking nominations from the community to join the APRICOT 
2024 PC to assist with the development of the programme for APRICOT 2024.


Eligible PC candidates must have attended recent APRICOT events, have 
broad technical knowledge of Internet operations, and have good 
familiarity with the format of APRICOT. Having constructive opinions and 
ideas about how the programme content might be improved is of high value 
too. PC members are expected to work actively to solicit content and 
review submissions for technical merit. The PC meets by conference call, 
weekly in frequency during the three months prior to APRICOT.


If you are interested in joining the PC and meet the above eligibility 
criteria, please send a brief note to "pc-chairs at apricot.net". The 
note must include affiliation (if any) and a brief description about why 
you would make a good addition to the PC.


The PC Chairs will accept nominations received by 17:00 UTC+8 on Friday 
20th October 2023, and will announce the new PC shortly thereafter.


Many thanks!

Mark Tinka & Mark Duffell
For the APRICOT 2024 Organising Committee
--

Re: maximum ipv4 bgp prefix length of /24 ?

[Mark Tinka]

On 10/5/23 08:32, Geoff Huston wrote:


Not really.

The stability of number in IPv4 as compared to the monotonic rise in IPv6 is 
what I find to be curious.


I think the fact that RIR's allocate very large IPv6 address space to 
their members may well be what is driving this.


Historically, network operators do enjoy de-aggregating address space. 
One can get significantly more /48's out of a /32 (or shorter) than they 
would /24's out of any IPv4 allocation they acquired.


One way to control this escalation is to shorten the maximum prefix 
length that all operators are willing to accept into the DFZ. The other 
way would be to get RIR's to lengthen the minimum allocation they can 
make to their members. Neither of these solutions is likely to be 
popular, but nor is having to pay large sums of money for more FIB.


Mark.

Re: maximum ipv4 bgp prefix length of /24 ?

[Mark Tinka]

On 10/5/23 08:24, Geoff Huston wrote:

The IPv6 FIB is under the same pressure from more specifics. Its taken 
20 years to get there, but the IPv6 FIB is now looking stable at 60% 
opf the total FIB size [2]. For me, thats a very surprising outcome in 
an essentially unmanaged system. 


Were you expecting it to be lower than IPv4?

Mark.

Re: maximum ipv4 bgp prefix length of /24 ?

[Mark Tinka]

On 10/5/23 07:49, Crist Clark wrote:

But if the assumption is that networks will always eventually totally 
deaggregate to the maximum, we're screwed. Routing IPv4 /32s would be 
nothing. The current practice of accepting /48s could swell to about 
2^(48 - 3) = 2^45 = 35184372088832.


What will prevent unrestricted growth of the IPv6 table if operators 
push everything out to /48 "to counter hijacks" or other misguided 
reasons?


Expecting that the Internet would ever operate at maximum de-aggregation 
is unrealistic. There are too many checkpoints to make that a feasible 
possibility.


It's not an outcome I would spend any brain cycles on, unless as an 
academic exercise.


Mark.

Re: maximum ipv4 bgp prefix length of /24 ?

[Mark Tinka]

On 10/4/23 09:27, Elmar K. Bins wrote:


Justin,

I'm not sure you're not confusing scope here.

Everybody and their sister accept smaller blocks from their customers; we're
all talking about the DFZ here, not customer routes that you aggregate.


Actually, we don't.

From our customers, the most we are accepting today is a /24 and a /48. 
This is for transit customers with their own AS and address space.


Of course, if it's a DIA customer, we can assign longer prefixes, but 
that is internal address space that belongs to us.


Mark.

Re: maximum ipv4 bgp prefix length of /24 ?

[Mark Tinka]

On 10/4/23 12:11, Musa Stephen Honlue wrote:


Which one is easier,

1. Convincing the tens of thousands of network operators and 
equipment vendors to modify configs and code to accept more specifics 
than /24, or


Equipment vendors can already support 10 million entries in FIB. They 
just ask for a little bit of cash for it.


Mark.

Re: cogent spamming directly from ARIN records?

[Mark Tinka]

On 10/2/23 22:59, Matthew Petach wrote:


Huh?

In all my decades of time in the network industry, I have never seen a 
case where a smaller transit contract had lower per mbit cost than a 
larger volume contract.


I would expect that HE would make *more* money off 10 smaller customer 
transit contracts than one big tier 3 wholesaler transit contract.


That's my point.

Smaller ISP's will get better per-Mbps rates from their direct upstreams 
than they would from HE/Cogent. The rates they'd get from HE/Cogent 
would be to HE's/Cogen's favour, and not to the ISP's favour.


So if the goal is transit-free glory vanity, HE/Cogent would have to 
take a bit of a haircut which they "could" make up in contract length.


Just another way to skin the cat.

Mark.

Re: maximum ipv4 bgp prefix length of /24 ?

[Mark Tinka]

On 10/2/23 20:44, Tim Franklin wrote:

Had NOT considered the looping - that's what you get for writing in 
public without thinking it all the way through *blush*.


Thanks for poking holes appropriately,



Like I said, it's going to be a messy experiment - for probably a 
decade, at least.


As Saku has highlighted as well, vendors are likely to find a lot of 
sludge in this experiment that they will never be able to share with 
us... likely because it will be too complicated for us to understand in 
a coherent way, or likely because the experiment changes so rapidly it 
makes no sense to tell us about issues which will quickly become obsolete.


Many lessons will be learned, but ultimately, one would be naive to 
think this black box will just work.


All I want is a "set routing-options fib-compression disable" present 
for Christmas.


Mark.

Re: cogent spamming directly from ARIN records?

[Mark Tinka]

On 10/2/23 20:58, Tim Burke wrote:


Hurricane has been doing the same thing lately... but their schtick is to say that 
"we are seeing a significant amount of hops in your AS path and wanted to know if 
you are open to resolve this issue".


I get what HE are trying to do here, as I am sure all of us do.

The potential fallout is a declining relationship with their existing 
customers that bring other downstream ISP's behind them. Contacting 
those downstream ISP's to "resolve this issue" puts them at odds with 
their existing customers who bring those customers in already.


There is a chance they dilute their income because, well, smaller ISP's 
will not be keen to pay the higher transit fees their upstreams pay to 
HE. Which means that HE are more willing to be closer to eyeballs than 
they are maximizing margins.


Is the loss of customer trust worth the transit-free glory?

Mark.

Re: maximum ipv4 bgp prefix length of /24 ?

[Mark Tinka]

On 9/30/23 01:36, William Herrin wrote:


If I were designing the product, I'd size the SRAM with that in mind.
I'd also keep two full copies of the FIB in the outer DRAM so that the
PPEs could locklessly access the active one while the standby one gets
updated with changes from the RIB. But I'd design the router to
gracefully fail if the FIB exceeded what the SRAM could hold.

When a TCAM fills, the shortest prefixes are ejected to the router's
main CPU. That fails pretty hard since the shortest prefixes tend to
be among the most commonly used. By comparison, an SRAM cache tends to
retain the most commonly used prefixes as an inherent part of how
caches work, regardless of prefix length. It can operate close to full
speed until the actively used routes no longer fit in the cache.


Well, not sure if you're aware, but starting Junos 21.2, Juniper are 
implementing FIB Compression on the PTX routers running Express 4 and 
Junos EVO.


We have some of these boxes in our network (PTX10001), and I have asked 
Juniper to provide a knob to allow us to turn it off, as it is currently 
going to ship as a default-on feature. I'd rather not be part of the 
potential mess that is going to come with the experimentation of that 
over the next decade.


Mark.

Re: maximum ipv4 bgp prefix length of /24 ?

[Mark Tinka]

On 9/29/23 22:56, William Herrin wrote:


Actually, BGP can swing that. Routing involves two distinct
components: the routing information base (RIB) and the forwarding
information base (FIB). BGP is part of the RIB portion of that
process. It's always implemented in software (no hardware
acceleration). It's not consulted per-packet, so as long as the update
rate is slow enough for the CPU to keep up and there's enough DRAM
(which is cheap and plentiful these days) to hold the entire thing,
there's no particular upper bound to the number of routes.


Not unless you are running BGP Add-Paths in its extreme guise to 
consider all available paths, and then there is the chance you could 
push your RAM and CPU into unknown territory, depending on the platform, 
code and control plane you've got.




The limiting factor is the FIB. The FIB is what is implemented with
hardware acceleration on high-end routers in order to achieve large
packet-per-second (PPS) numbers. It relies on storage hardware which
is both faster and more expensive than DRAM. Consequently it has much
less capacity to store information than DRAM. Currently shipping
equipment intended for BGP backbone use can manage 1M to 2M routes in
the hardware-accelerated FIB regardless of the amount of DRAM on the
machine.


There are line cards out there today that are able to hold 10 million 
routes in FIB.


Mark.

Re: maximum ipv4 bgp prefix length of /24 ?

[Mark Tinka]

On 9/29/23 06:43, VOLKAN SALİH wrote:

But when everybody upgrades, memory and processor unit prices 
decrease.. Vendors gain from demand.




I am yet to see that trend...

Mark.

Re: maximum ipv4 bgp prefix length of /24 ?

[Mark Tinka]

On 9/28/23 23:25, VOLKAN SALİH wrote:


hello,

I believe, ISPs should also allow ipv4 prefixes with length between 
/25-/27 instead of limiting maximum length to /24..


I also believe that RIRs and LIRs should allocate /27s which has 32 
IPv4 address. considering IPv4 world is now mostly NAT'ed, 32 IPv4s 
are sufficient for most of the small and medium sized organizations 
and also home office workers like youtubers, and professional gamers 
and webmasters!


It is because BGP research and experiment networks can not get /24 due 
to high IPv4 prices, but they have to get an IPv4 prefix to learn BGP 
in IPv4 world.


What do you think about this?

What could be done here?

Is it unacceptable; considering most big networks that do 
full-table-routing also use multi-core routers with lots of RAM? those 
would probably handle /27s and while small networks mostly use default 
routing, it should be reasonable to allow /25-/27?




RAM is not the issue... it's FIB.

If you pay me for FIB slots, I'm happy to install /32's to your heart's 
content :-).


Mark.

Re: TACACS+ server recommendations?

[Mark Tinka]

On 9/20/23 17:39, Jeff Moore wrote:

We have also used https://www.shrubbery.net/tac_plus/ for some time as 
well. Great product!


Yes, that's one of the ones in the FreeBSD ports.

Works very well.

Mark.

Re: TACACS+ server recommendations?

[Mark Tinka]

On 9/20/23 17:09, Bryan Holloway wrote:

Ah, the good old days when I could download the latest tac_plus code 
from the Cisco FTP site, compile it, and off I go.


But I digress.

Curious if there are any operators out there that have a good 
recommendation on a lightweight TACACS+ server for ~200 NEs and 
access-control for 20-30 folks. Nothing too special, but some sort of 
simple command-control auth would be nice.


Open-source is fine ... we've been looking at commercial options, but 
they're all pretty pricey and have way more features than we need.


Thank you all in advance!


[root@host /home/tinka]# cd /usr/ports/net/tac
tac_plus4/ tacacs/
[root@host /home/tinka]#

They work just fine.

These are on FreeBSD, but I am sure they will compile on Linux too.

Mark.

Re: Zayo woes

[Mark Tinka]

On 9/19/23 18:05, Mike Hammett wrote:

Some of it is scale-related. Someone's operating just fine at the size 
they are, but the next order of magnitude larger enjoys many benefits 
from that size, but it takes either A) luck or B) the right skills to 
be able to move up to get those benefits. In terms of network 
operators, there's a big difference between a company of 1 and a 
company of 2. Again a big difference between a company of 2 and a 
company of 10. Another big difference between a company of 10 and a 
company of..  I dunno, 100?



1G waves and 100G waves aren't *THAT* different in price anymore. 
However, if 1 is doing you just fine, the much better cost per bit of 
100 won't do you a darn bit of good and will probably hurt. The hurt 
is not only due to the higher MRC, but now the higher NRC for 
equipment with 100G interfaces. If you can put enough bits on the line 
to make it worth it, you've got yourself tremendous advantage. The 
acquisition pays for itself in marginally higher costs for much higher 
revenue.


The company may have been doing just fine before, but couldn't afford 
the move up to 10 or 100 because their present market couldn't justify 
it and the larger market wasn't obtainable until they had it. Catch 22 
that can't really be overcome by most. Enter M Someone just can't 
get to the next level they need to compete with those that can.


I'm talking about companies of relatively equal size and influence.

It's all benefit for smaller companies, even if it may be at the cost of 
the larger one.


Mark.

Re: Zayo woes

[Mark Tinka]

On 9/19/23 17:54, Mike Hammett wrote:

Well sure, and I would like to think (probably mistakenly) that just 
no one important enough (to the money people) made the money people 
that these other things are *REQUIRED* to make the deal work.


Obviously, people lower on the ladder say it all of the time, but the 
important enough money people probably don't consider those people 
important enough to listen to.


The ISP world is not a normal market place :-).

Mark.

Re: Zayo woes

[Mark Tinka]

On 9/19/23 17:40, Anne Mitchell wrote:


And sometimes the acquisition is really just about acquiring the assets, such 
as the customer list*, and then they are left with having to run something that 
they never really wanted until they can figure out what to do with it.


Right, buying the revenue to prop up the top and bottom line is also a 
reason to acquire. Usually, this is based on assessed profitability, but 
what tends to go unseen during the due diligence process is what is 
actually contributing to that profitability.


I mean, typically, if a company is doing very well, it won't try to get 
itself sold. Well, not easily anyway...


Mark.

Re: Zayo woes

[Mark Tinka]

On 9/19/23 16:48, Mike Hammett wrote:
As someone that has been planning to be in the acquiring seat for a 
while (but yet to do one), I've consistently passed to the money 
people that there's the purchase price and then there's the % on top 
of that for equipment, contractors, etc. to integrate, improve, 
optimize future cashflow, etc. those acquisitions with the rest of 
what we have.


I blame this on the success of how well we have built the Internet with 
whatever box and tool we have, as network engineers.


The money people assume that all routers are the same, all vendors are 
the same, all software is the same, and all features are easily 
deployable. And that all that is possible if you can simply do a better 
job finding the cheapest box compared to your competition.


In general, I don't fancy nuance when designing for the majority. But 
with acquisition and integration, nuance is critical, and nuance quickly 
shows that the acquisition was either underestimated, or not worth doing 
at all.


Mark.

Re: Zayo woes

[Mark Tinka]

On 9/19/23 16:41, Matthew Petach wrote:

c) your executives have promised there will be cost savings after the 
merger due to "synergies" between the two companies.


Couldn't have said the exact same words any better myself - "cost 
savings from synergies" :-).


Always interesting when a year later, the projected savings from 
synergies are nowhere near reality, and all consolidation analysis work 
has completed :-).


Mark.

Re: Zayo woes

[Mark Tinka]

On 9/19/23 14:19, Mike Hammett wrote:

I've never understood companies that acquire and don't completely 
integrate as quickly as they can.


Sometimes it does not add any material value to either network. 
Sometimes it takes too long.


If the acquired company is orders of magnitude smaller than the 
acquiring one, it's probably best to keep them separate.


Of course, I am referring to network integration. Staff and business 
systems integration should always be the goal.


Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/9/23 22:29, Dave Cohen wrote:

At a previous $dayjob at a Tier 1, we would only support LAG for a 
customer L2/3 service if the ports were on the same card. The response 
we gave if customers pushed back was "we don't consider LAG a form of 
circuit protection, so we're not going to consider physical resiliency 
in the design", which was true, because we didn't, but it was beside 
the point. The real reason was that getting our switching/routing 
platform to actually run traffic symmetrically across a LAG, which 
most end users considered expected behavior in a LAG, required a 
reconfiguration of the default hash, which effectively meant that 
[switching/routing vendor]'s TAC wouldn't help when something 
invariably went wrong. So it wasn't that it wouldn't work (my 
recollection at least is that everything ran fine in lab environments) 
but we didn't trust the hardware vendor support.


We've had the odd bug here and there with LAG's for things like VRRP, 
BFD, e.t.c. But we have not run into that specific issue before on 
ASR1000's, ASR9000's, CRS-X's and MX. 98% of our network is Juniper 
nowadays, but even when we ran Cisco and had LAG's across multiple line 
cards, we didn't see this problem.


The only hashing issue we had with LAG's is when we tried to carry Layer 
2 traffic across them in the core. But this was just a limitation of the 
CRS-X, and happened also on member links of a LAG that shared the same 
line card.


Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/9/23 20:44, Randy Bush wrote:


i am going to be foolish and comment, as i have not seen this raised

if i am running a lag, i can not resist adding a bit of resilience by
having it spread across line cards.

surprise!  line cards from vendor  do not have uniform hashing
or rotating algorithms.


We spread all our LAG's across multiple line cards wherever possible 
(wherever possible = chassis-based hardware).


I am not intimately aware of any hashing concerns for LAG's that 
traverse multiple line cards in the same chassis.


Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/7/23 09:51, Saku Ytti wrote:


Perhaps if congestion control used latency or FEC instead of loss, we
could tolerate reordering while not underperforming under loss, but
I'm sure in decades following that decision we'd learn new ways how we
don't understand any of this.


Isn't this partly what ECN was meant for? It's so old I barely remember 
what it was meant to solve :-).


Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/7/23 09:31, Benny Lyne Amorsen wrote:


Unfortunately that is not strict round-robin load balancing.


Oh? What is it then, if it's not spraying successive packets across 
member links?




  I do not
know about any equipment that offers actual round-robin
load-balancing.


Cisco had both per-destination and per-packet. Is that not it in the 
networking world?




Juniper's solution will cause way too much packet reordering for TCP to
handle. I am arguing that strict round-robin load balancing will
function better than hash-based in a lot of real-world
scenarios.


Ummh, no, it won't.

If it did, it would have been widespread. But it's not.

Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/6/23 18:52, Tom Beecher wrote:

Well, not exactly the same thing. (But it's my mistake, I was 
referring to L3 balancing, not L2 interface stuff.)


Fair enough.


load-balance per-packet will cause massive reordering, because it's 
random spray , caring about nothing except equal loading of the 
members. It's a last resort option that will cause tons of reordering. 
(And they call that out quite clearly in docs.) If you don't care 
about reordering it's great.


load-balance adaptive generally did a decent enough job last time I 
used it much.


Yep, pretty much my experience too.


stateful was hit or miss ; sometimes it tested amazing, other times 
not so much. But it wasn't a primary requirement so I never dove into why


Never tried stateful.

Moving 802.1Q trunk from N x 10Gbps LAG's to native 100Gbps links 
resolved this load balancing conundrum for us. Of course, it works well 
because we spread these router<=>switch links across several 100Gbps 
ports, so no single trunk is ever that busy, even for customers buying N 
x 10Gbps services.


Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/6/23 12:01, Saku Ytti wrote:


Fun fact about the real world, devices do not internally guarantee
order. That is, even if you have identical latency links, 0
congestion, order is not guaranteed between packet1 coming from
interfaceI1 and packet2 coming from interfaceI2, which packet first
goes to interfaceE1 is unspecified.
This is because packets inside lookup engine can be sprayed to
multiple lookup engines, and order is lost even for packets coming
from interface1 exclusively, however after the lookup the order is
restored for _flow_, it is not restored between flows, so packets
coming from interface1 with random ports won't be same order going out
from interface2.

So order is only restored inside a single lookup complex (interfaces
are not guaranteed to be in the same complex) and only for actual
flows.


Yes, this has been my understanding of, specifically, Juniper's 
forwarding complex.


Packets are chopped into near-same-size cells, sprayed across all 
available fabric links by the PFE logic, given a sequence number, and 
protocol engines ensure oversubscription is managed by a request-grant 
mechanism between PFE's.


I'm not sure what mechanisms other vendors implement, but certainly OoO 
cells in the Juniper forwarding complex is not a concern within the same 
internal system itself.


Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/6/23 11:20, Benny Lyne Amorsen wrote:


TCP looks quite different in 2023 than it did in 1998. It should handle
packet reordering quite gracefully; in the best case the NIC will
reassemble the out-of-order TCP packets into a 64k packet and the OS
will never even know they were reordered. Unfortunately current
equipment does not seem to offer per-packet load balancing, so we cannot
test how well it works.


I ran per-packet load balancing on a Juniper LAG between 2015 - 2016. 
Let's just say I won't be doing that again.


It balanced beautifully, but OoO packets made customers' lives 
impossible. So we went back to adaptive load balancing.




It is possible that per-packet load balancing will work a lot better
today than it did in 1998, especially if the equipment does buffering
before load balancing and the links happen to be fairly short and not
very diverse.

Switching back to per-packet would solve quite a lot of problems,
including elephant flows and bad hashing.

I would love to hear about recent studies.


2016 is not 1998, and certainly not 2023... but I've not heard about any 
improvements in Internet-based applications being better at handling OoO 
packets.


Open to new info.

100Gbps ports has given us some breathing room, as have larger buffers 
on Arista switches to move bandwidth management down to the user-facing 
port and not the upsteam router. Clever Trio + Express chips have also 
enabled reasonably even traffic distribution with per-flow load balancing.


We shall revisit the per-flow vs. per-packet problem when 100Gbps starts 
to become as rampant as 10Gbps did.


Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/6/23 17:27, Tom Beecher wrote:



At least on MX, what Juniper calls 'per-packet' is really 'per-flow'.


Unless you specifically configure true "per-packet" on your LAG:

    set interfaces ae2 aggregated-ether-options load-balance per-packet

I ran per-packet on a Juniper LAG 10 years ago. It produced 100% perfect 
traffic distribution. But the reordering was insane, and the 
applications could not tolerate it.


If you applications can tolerate reordering, per-packet is fine. In the 
public Internet space, it seems we aren't there yet.


Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/6/23 16:14, Saku Ytti wrote:


For example Juniper offers true per-packet, I think mostly used in
high performance computing.


Cisco did it too with CEF supporting "ip load-sharing per-packet" at the 
interface level.


I am not sure this is still supported on modern code/boxes.

Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/6/23 09:12, Masataka Ohta wrote:



you now recognize that per-flow load balancing is not a very
good idea.


You keep moving the goal posts. Stay on-topic.

I was asking you to clarify your post as to whether you were speaking of 
per-flow or per-packet load balancing. You did not do that, but I did 
not return to that question because your subsequent posts inferred that 
you were talking to per-packet load balancing.


And just because I said per-flow load balancing has been the gold 
standard for the last 25 years, does not mean it is the best solution. 
It just means it is the gold standard.


I recognize what happens in the real world, not in the lab or text books.

Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/4/23 13:27, Nick Hilliard wrote:



this is an excellent example of what we're not talking about in this 
thread.


It is amusing how he tried to pivot the discussion. Nobody was talking 
about how lane transport in optical modules works.


Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/4/23 13:04, Masataka Ohta wrote:


Are you saying you thought a 100G Ethernet link actually consisting
of 4 parallel 25G links, which is an example of "equal speed multi
parallel point to point links", were relying on hashing?


No... you are saying that.

Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/3/23 15:01, Masataka Ohta wrote:


Why, do you think, you can rely on existence of flows?


You have not quite answered my question - but I will assume you are in 
favour of per-packet load balancing.


I have deployed per-packet load balancing before, ironically, trying to 
deal with large EoMPLS flows in a LAG more than a decade ago. I won't be 
doing that again... OoO packets is nasty at scale.




And nothing beyond, of course.


No serious operator polices in the core.



ECMP, surely, is a too abstract concept to properly manage/operate
simple situations with equal speed multi parallel point to point links.


I must have been doing something wrong for the last 25 years.

Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/3/23 15:01, Masataka Ohta wrote:


Why, do you think, you can rely on existence of flows?


You have not quite answered my question - but I will assume you are in 
favour of per-packet load balancing.


I have deployed per-packet load balancing before, ironically, trying to 
deal with large EoMPLS flows in a LAG more than a decade ago. I won't be 
doing that again... OoO packets is nasty at scale.




And nothing beyond, of course.


No serious operators polices in the core.



ECMP, surely, is a too abstract concept to properly manage/operate
simple situations with equal speed multi parallel point to point links.


I must have been doing something wrong for the last 25 years.

Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/3/23 09:59, Masataka Ohta wrote:



If you have multiple parallel links over which many slow
TCP connections are running, which should be your assumption,
the proper thing to do is to use the links with round robin
fashion without hashing. Without buffer bloat, packet
reordering probability within each TCP connection is
negligible.


So you mean, what... per-packet load balancing, in lieu of per-flow load 
balancing?





So, if you internally have 10 parallel 1G circuits expecting
perfect hashing over them, it is not "non-rate-limited 10gig".


It is understood in the operator space that "rate limiting" generally 
refers to policing at the edge/access.


The core is always abstracted, and that is just capacity planning and 
management by the operator.


Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/2/23 17:38, Masataka Ohta wrote:


Wrong. It can be performed only at the edges by policing total
incoming traffic without detecting flows.


I am not talking about policing in the core, I am talking about 
detection in the core.


Policing at the edge is pretty standard. You can police a 50Gbps EoMPLS 
flow coming in from a customer port in the edge. If you've got N x 
10Gbps links in the core and the core is unable to detect that flow in 
depth to hash it across all those 10Gbps links, you can end up putting 
all or a good chunk of that 50Gbps of EoMPLS traffic into a single 
10Gbps link in the core, despite all other 10Gbps links having ample 
capacity available.




There is no such algorithms because, as I wrote:

: 100 50Mbps flows are as harmful as 1 5Gbps flow.


Do you operate a large scale IP/MPLS network? Because I do, and I know 
what I see with the equipment we deploy.


You are welcome to deny it all you want, however. Not much I can do 
about that.


Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/2/23 17:04, Masataka Ohta wrote:


Both of you are totally wrong, because the proper thing to do
here is to police, if *ANY*, based on total traffic without
detecting any flow.


I don't think it's as much an issue of flow detection as it is the 
core's ability to balance the Layer 2 payload across multiple links 
effectively.


At our shop, we understand the limitations of trying to carry large 
EoMPLS flows across an IP/MPLS network that is, primarily, built to 
carry IP traffic.


While some vendors have implemented adaptive load balancing algorithms 
on decent (if not custom) silicon that can balance EoMPLS flows as well 
as they can IP flows, it is hit & miss depending on the code, hardware, 
vendor, e.t.c.


In our case, our ability to load balance EoMPLS flows as well as we do 
IP flows has improved since we moved to the PTX1000/10001 for our core 
routers. But even then, we will not sell anything above 40Gbps as an 
EoMPLS service. Once it gets there, time for EoDWDM. At least, until 
800Gbps or 1Tbps Ethernet ports become both technically viable and 
commercially feasible.


For as long as core links are based on 100Gbps and 400Gbps ports, 
optical carriage for 40Gbps and above is more sensible than EoMPLS.


Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/2/23 08:43, Saku Ytti wrote:


What in particular are you missing?
As I explained, PTX/MX both allow for example speculating on transit
pseudowires having CW on them. Which is non-default and requires
'zero-control-word'. You should be looking at 'hash-key' on PTX and
'enhanced-hash-key' on MX.  You don't appear to have a single stanza
configured, but I do wonder what you wanted to configure when you
noticed the missing ability to do so.


Sorry for the confusion - let me provide some background context since 
we deployed the PTX ages ago (and core nodes are typically boring).


The issue we ran into was to do with our deployment tooling, which was 
based on 'enhanced-hash-key' that is required for MPC's on the MX.


The tooling used to deploy the PTX was largely built on what we use to 
deploy the MX, with tweaks of critically different items. At the time, 
we did not know that the PTX required 'hash-key' as opposed to 
'enhanced-hash-key'. So nothing got deployed on the PTX specifically for 
load balancing (we might have assumed it to have been non-existent or 
incomplete feature at the time).


So the "surprise" I speak of is how well it all worked with load 
balancing across LAG's and EoMPLS traffic compared to the CRS-X, despite 
not having any load balancing features explicitly configured, which is 
still the case today.


It works, so we aren't keen to break it.

Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/1/23 21:52, Mike Hammett wrote:

It doesn't help the OP at all, but this is why (thus far, anyway), I 
overwhelmingly prefer wavelength transport to anything switched. Can't 
have over-subscription or congestion issues on a wavelength.


Large IP/MPLS operators insist on optical transport for their own 
backbone, but are more than willing to sell packet for transport. I find 
this amusing :-).


I submit that customers who can't afford large links (1Gbps or below) 
are forced into EoMPLS transport due to cost.


Other customers are also forced into EoMPLS transport because there is 
no other option for long haul transport in their city other than a 
provider who can only offer EoMPLS.


There is a struggling trend from some medium sized operators looking to 
turn an optical network into a packet network, i.e., they will ask for a 
100Gbps EoDWDM port, but only seek to pay for a 25Gbps service. The 
large port is to allow them to scale in the future without too much 
hassle, but they want to pay for the bandwidth they use, which is hard 
to limit anyway if it's a proper EoDWDM channel. I am swatting such 
requests away because you tie up a full 100Gbps channel on the line side 
for the majority of hardware that does pure EoDWDM, which is a 
contradiction to the reason a packet network makes sense for sub-rate 
services.


Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/1/23 15:55, Saku Ytti wrote:


Personally I would recommend turning off LSR payload heuristics,
because there is no accurate way for LSR to tell what the label is
carrying, and wrong guess while rare will be extremely hard to root
cause, because you will never hear it, because the person suffering
from it is too many hops away from problem being in your horizon.
I strongly believe edge imposing entropy or fat is the right way to
give LSR hashing hints.


PTX1000/10001 (Express) offers no real configurable options for load 
balancing the same way MX (Trio) does. This is what took us by surprise.


This is all we have on our PTX:

tinka@router# show forwarding-options
family inet6 {
    route-accounting;
}
load-balance-label-capability;

[edit]
tinka@router#

Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/1/23 15:59, Mike Hammett wrote:


I wouldn't call 50 megabit/s an elephant flow


Fair point.

Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/1/23 15:44, Mike Hammett wrote:
and I would say the OP wasn't even about elephant flows, just about a 
network that can't deliver anything acceptable.


Unless Cogent are not trying to accept (and by extension, may not be 
able to guarantee) large Ethernet flows because they can't balance them 
across their various core links, end-to-end...


Pure conjecture...

Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/1/23 15:29, Saku Ytti wrote:


PTX and MX as LSR look inside pseudowire to see if it's IP (dangerous
guess to make for LSR), CSR/ASR9k does not. So PTX and MX LSR will
balance your pseudowire even without FAT.


Yes, this was our conclusion as well after moving our core to PTX1000/10001.

Mark.

Re: Lossy cogent p2p experiences?

[Mark Tinka]

On 9/1/23 10:50, Saku Ytti wrote:


It is a very plausible theory, and everyone has this problem to a
lesser or greater degree. There was a time when edge interfaces were
much lower capacity than backbone interfaces, but I don't think that
time will ever come back. So this problem is systemic.
Luckily there is quite a reasonable solution to the problem, called
'adaptive load balancing', where software monitors balancing, and
biases the hash_result => egress_interface tables to improve balancing
when dealing with elephant flows.


We didn't have much success with FAT when the PE was an MX480 and the P 
a CRS-X (FP40 + FP140 line cards). This was regardless of whether the 
core links were native IP/MPLS or 802.1AX.


When we switched our P devices to PTX1000 and PTX10001, we've had 
surprisingly good performance of all manner of traffic across native 
IP/MPLS and 802.1AX links, even without explicitly configuring FAT for 
EoMPLS traffic.


Of course, our policy is to never transport EoMPLS servics in excess of 
40Gbps. Once a customer requires 41Gbps of EoMPLS service or more, we 
move them to EoDWDM. Cheaper and more scalable that way. It does help 
that we operate both a Transport and IP/MPLS network, but I understand 
this may not be the case for most networks.


Mark.

Re: Destination Preference Attribute for BGP

[Mark Tinka]

On 8/30/23 18:56, michael brooks - ESC wrote:



I, too, am looking for something sexy (explained below). But can you 
explain why you think AS_PATH is "useless," Mark?


Because most network operators use LOCAL_PREF heavily, and no amount of 
AS_PATH prepending will be able fight that with any reasonable success.


You can still achieve some success with AS_PATH prepending, but with the 
way content is getting distributed around the world, it is becoming as 
useful as running a Squid cache yourself these days.





For background, and the reason I asked about DPA:
Currently, our routing carries user traffic to a single data center 
where it egresses to the Internet via three ISP circuits, two 
carriers. We are peering on a single switch stack, so we let L2 "load 
balance" user flows for us. We have now brought up another ISP circuit 
in a second data center, and are attempting to influence traffic to 
return the same path as it egressed our network. Simply, we now have 
two datacenters which user traffic can egress, and if one is used we 
want that traffic to return to the same data center. It is a problem 
of asymmetry. It appears the only tools we have are AS_Path and MED, 
and so I have been searching for another solution, that is when I came 
across DPA. In further looking at the problem, BGP Communities also 
seems to be a possible solution, but as the thread has explored, 
communities may/may not be scrubbed upstream. So, presently we are 
looking for a solution which can be used with our direct peers. 
Obviously, if someone has a better solution, I am all ears.


When you have multiple transit providers, you are really implicitly 
accepting that forwarding asymmetry is now part of your life. You will 
have full control of your outbound forwarding, but return forwarding 
will be a coin toss.


You can guarantee this, to some degree, if you also have lots of 
peering, as most peers will prefer to return traffic to you via peering 
and not via their transit providers. But even this is not always a sure 
thing, especially in situations where a network you are peering with is 
doing Remote Peering.


Ultimately, the level of influence you have on the remote network's 
forwarding decision, especially if you are not peering, is slim. Our 
solution to this has been to focus on the "typically large" transit 
providers, announce routes consistently to them, and ensure we have the 
same port capacity to each one. Even with those attributes, we can't get 
perfect load sharing, because we provide customers the ability to decide 
which transit providers (and exchange points) they want their routes to 
transit, or not. So the only routing we can consistently guarantee is 
our own routes. We can't guarantee customers will let their routes exit 
all transit or peering points, so we just make sure we have ample 
capacity across all such relationships.


I understand that not all networks may have the ability to do this for 
financial reasons, but if you can only afford to have one or two transit 
providers, your best bet is to leverage the existing BGP tools as best 
you can, even though the results will not be perfect.


Mark.

Re: MX204 Virtual Chassis Setup

[Mark Tinka]

On 8/28/23 07:55, Daniel Marks wrote:

(Enterprise AS for context)

This hasn’t been my experience in the US, however we mostly deal in 
tier 2 markets (I.e. Detroit, Miami, Dallas, etc…) and we have plenty 
of 40G private interconnects. I don’t doubt 40G is going away, I’ve 
just never had trouble using it around here.


This would be expected, since most 40Gbps hardware I have seen in Europe 
and Africa is in the enterprise and exchange point space. If service 
providers have them, I'd imagine that inventory is far lower than 100Gbps.


Mark.

Re: MX204 Virtual Chassis Setup

[Mark Tinka]

On 8/28/23 03:05, Mike Hammett wrote:
Well, or they simply found a potential deal on hardware that came with 
40 gig ports. 40 gigs is still a lot of bits to a lot of people.


For internal use, sure.

But when connecting to another AS, the chances of them supporting 40Gbps 
in one or more places is inconsistent to slim.


Exchange points may be an exception.

Mark.

Re: MX204 Virtual Chassis Setup

[Mark Tinka]

On 8/27/23 04:52, Eric Kuhnke wrote:


I sincerely doubt there is much demand for *new* 40G these days.

Look at the population of 40G members on major IXes.

People have either one 10G, 2 x 10G, or 100G.

40G was a dead-end 9 years ago and much so more now.


We have customers that sometimes ask for 40Gbps interconnects. I always 
tell our Pre-Sales team that those are the ones who "led the way", back 
in the day. Sadly, they were a bit too early :-).


Mark.

Re: MX204 Virtual Chassis Setup

[Mark Tinka]

On 8/26/23 00:54, Tom Beecher wrote:

It would, sure. Instead of storing a single prefix/next-hop with flags 
in memory, you now have to store every prefix/next-hop that you are 
announcing as well.


Indeed.

But it has been worth it. The load balancing from PE-to-PE has been 
fantastic, especially when coupled with BGP Multipath.


No more messing about with LOCAL_PREF for multi-homed customers, and it 
works just as well with different (but equal-length) AS_PATH's.


Mark.

Re: MX204 Virtual Chassis Setup

[Mark Tinka]

On 8/25/23 19:16, Tom Beecher wrote:

In my experience and testing with them, you have a decent bit of 
headroom past the published RIB/FIB limits before they'll fall over.


They are holding up pretty well for us, mainly because we do a lot more 
BGP on MX480's than on MX204's. We use the MX204's mainly for peering 
and CDN gateways. Where we use them for edge customers, it's a handful 
of BGP sessions.


On MX480 16GB RE's running two full BGP feeds but hundreds of customer 
sessions, Add-Paths really eats into RAM. We've had to upgrade some of 
the busier routers from 16GB to 64GB RE's, especially on later versions 
of code where ROV can also bite into memory on boxes carrying lots of 
BGP sessions.


Mark.

Re: MX204 Virtual Chassis Setup

[Mark Tinka]

On 8/23/23 17:14, Matt Erculiani wrote:

Does Fusion not make sense in this case? I've not had a ton of 
experience with it, but it does well to add a crazy port count to an 
otherwise very port limited device.


In small edge PoP's, we attach an Arista 1U switch with tons of 1/10Gbps 
ports to an MX204 via 802.1Q. Works a treat. I've never been convinced 
by vendor-specific satellite systems :-).


On another note, the potential issue we might run into is pressure on 
control plane memory on the MX204 for us that run BGP Add-Paths. You can 
always upgrade the RE on an MX240/480/960, but the MX204 is fixed (and 
last time I checked, fiddling with Juniper RE memory was generally 
frowned upon).


Luckily, the MX10003 ships with 64GB of RAM, since it is now EoL.

The MX304 ships with 128GB of RAM, so anybody running Add-Paths on that 
box won't have an issue there.


Mark.

Re: Deployments of Provider Backbone Bridging (PBB)

[Mark Tinka]

On 8/25/23 09:41, Tarko Tikan wrote:

AFAIK this reflects the reality very well. There are huge PBB 
deployments in very large networks but the overall number of networks, 
using PBB, is very low. Even in those networks PBB is/will be phased 
out so don't expect any new deployments. It is still well supported by 
the vendors who initially invested into PBB.


Most operators, especially of small-to-medium size scope, but even some 
larger ones, will go from 802.1Q to Q-in-Q, and then to MPLS.


MPLS end-to-end is not as common as MPLS combined with 802.1Q or Q-in-Q, 
in my very rough anecdotal experience. This is mostly due to cost 
control, as well as a seemingly common preference to have a so-called 
Internet Gateway (IGW) pinning all those pseudowires.


There have been ramblings of VXLAN as an IP-based underlay in lieu of 
MPLS. I don't know how well it has scaled outside of the data centre, 
i.e., in the Metro-E network. But the rate at which I hear about VXLAN 
for Metro-E deployments is also the same rate at which I don't hear 
about VXLAN for Metro-E deployments. In other words, it appears to be 
neither here nor there.


Mark.

Re: MX204 Virtual Chassis Setup

[Mark Tinka]

On 8/23/23 18:29, t...@pelican.org wrote:


Not Trio, and different PLM :)


Yes, aware... I was just speaking in general for what is likely to be a 
very popular platform :-).




MX304 (well, strictly LMIC16) has the same restriction, and a need for another entry in the 
magic port checker (https://apps.juniper.net/home/port-checker/index.html) for restrictions 
beyond "SUM(port-speeds) <= 1.6T".


Yep.

That trick they did where you can live with one RE and get 3 MIC's in 
the MX304 is... well, I guess everyone will have their own opinion.




They make sense once you've looked at the block diagram for the thing and followed the 
lines, but things like "4x10G breakout can only go in odd-numbered ports, and you 
have to leave the corresponding next-lowest even-numbered port empty" are not 
instantly obvious.


They do take some getting used to. But this is what comes with all the 
flexibility operators often seek.


Mark.

Re: MX204 Virtual Chassis Setup

[Mark Tinka]

On 8/23/23 17:01, Tom Beecher wrote:

I'm not sure they allow oversubscription on anything in the MX line 
anymore honestly. I could be wrong, I've been face down in a specific 
subset of equipment for a while, someone please correct me if I am.


On the new ACX line, yes.

If I look at the MPC7E, MPC10E, MX10003 and MX304, no oversubscription 
is allowed.


Even the LC2103 MPC on the MX10003 which has more ports than Trio 
capacity, won't let you use more than 1.2Tbps (3x Trio 3 chips on it).


We don't mess around with any other MX products, so not sure (although 
we are still yet to deploy the MPC10E's and the MX304).


Mark.

Re: MX204 Virtual Chassis Setup

[Mark Tinka]

On 8/23/23 08:00, Pascal Masha wrote:


Thanks just wanted to know whether it was a supported feature.


What would have been nice is if Juniper oversubscribed the face plate of 
this platform, as most people are more likely to run out of ports than 
they would the 400Gbps forwarding capacity of Trio.


In some cases, we deploy more of these in the same PoP just because we 
need more ports, not because we need more capacity; and a chassis would 
not make sense for the function, yet.


Mark.

Re: Destination Preference Attribute for BGP

[Mark Tinka]

On 8/22/23 16:46, Tom Beecher wrote:

Again, as it was stated, the size of or number of BGP communities 
wasn't the problem anyway; it was hashing / memory storage. And you 
know what? Hashing / memory storage HAS been a problem with multiple 
vendors in many other contexts, not just BGP community stuff. Has 
nothing to do with "2023 hardware". You can bog down top of the line 
DDR5 memory pretty easily if you make certain coding choices.


I suppose that can be said of anything in our industry. There is always 
a way we can hurt ourselves, especially if we ignore or have not learned 
our lessons.


With the information that I have, I don't consider it a massive issue at 
the moment. Of course, that is always a changing landscape, and my views 
may alter when that happens sufficiently. But as of now, I am dealing 
with far more severe BGP bugs than what appears to have been, at least 
for the moment, fixed.



You can choose ( as you apparently have ) to just presume that a 
problem that happened before won't ever happen again. Prob not a great 
idea though.


Ummh, okay...

Mark.

Re: Destination Preference Attribute for BGP

[Mark Tinka]

On 8/21/23 17:44, Tom Beecher wrote:


So, while this all sounds good, without any specifics on vendor,
box, code, code revision number, fix, year it happened, current
status, e.t.c., I can't offer any meaningful engagement.


If you clicked Matt's link to the Google search, you could tell from 
the results what vendor , model, and year it was pretty quickly.


I did.

Those are headlines.

The solider that was on the battlefield won't speak to the exact details.

I won't press, especially because nobody that needed a T1600 back then 
probably still runs one today.



Assertion Made : "Networks can scrub communities for memory or 
convergence reasons."

Others : "That doesn't seem like a concern. "
Matt : "Here was a real situation that happened where it was a 
concern, and the specifics on the reason why."


How is that not 'moving the needle? Because you didn't get full 
transcripts of his conversation with the vendor?. I'm sure a lot of 
people didn't even know that hashing / memory hotspotting was even a 
thing. Now they do.


There are a lot of things that vendors have fixed in BGP that we shall 
never know.


What I am saying is that for those that have been fixed, unless someone 
can offer up any additional evidence in 2023, the size of the number of 
BGP communities attached to a path does not scream "danger" in 2023 
hardware. And the T1600 is a long time ago.


Mark.

Re: MX204 Virtual Chassis Setup

[Mark Tinka]

On 8/21/23 16:51, Ryan Hamel wrote:


Paschal,

It is not supported, nor is it recommended for redundancy in a routed 
setup. Please describe your (desired) topology, that way the community 
can discuss alternatives.


Sounds like the OP wants to build a chassis-based system out of 
non-redundant hardware.


Mark.

Re: Destination Preference Attribute for BGP

[Mark Tinka]

On 8/19/23 00:22, Matthew Petach wrote:


Hi Mark,

I know it's annoying that I won't mention specifics.
Unfortunately, the last time I mentioned $vendor-specific information 
on NANOG, it was picked up by the press, and turned into a 
multimillion dollar kerfuffle with me at the center of the cross-hairs:
https://www.google.com/search?q=petach+kablooie_esv=558180114=petah+kablooie=0=1580=1008=2 



After that, I've learned it's best to not name specific very-big-name 
vendors on NANOG posts.


What I *can* say is that this was one of the primary vendors in the 
Internet backbone space, running mainstream code.
The only reason it didn't affect more networks was a function of the 
particular cluster of signalling communities being applied to all 
inbound prefixes, and how they interacted with the vendor's hash 
algorithm.


Corner cases, while valid, do not speak to the majority. If this
was a major issue, there would have been more noise about it by now.


I prefer to look at it the other way; the reason you didn't hear more 
noise about it, is that we stubbed our toes on it early, and had 
relatively fast, direct access to the development engineers to get it 
fixed within two days.  It's precisely *bcause* people trip over 
corner cases and get them fixed that they don't end up causing more 
widespread pain across the rest of the Internet.


There has been quite some noise about lengthy AS_PATH updates that
bring some routers down, which has usually been fixed with
improved BGP code. But even those are not too common, if one
considers a 365-day period.


Oh, absolutely.  Bugs in implementations that either crash the router 
or reset the BGP session are much more immediately visible than 
"that's odd, it's taking my routers longer to converge than it should".


How many networks actually track their convergence time in a time 
series database, and look at unusual trends, and then diagnose why the 
convergence time is increasing, versus how many networks just note an 
increasing number of "hey, your network seems to be slowing down" and 
throw more hardware at the problem, while grumbling about why their 
big expensive routers seem to be less powerful than a *nix box running 
gated?


I suspect there's more of these type of "corner cases" out there than 
you recognize.
It's just that most networks don't dig into routing performance issues 
unless it actually breaks the router, or kills BGP adjacencies.


If you *are* one of the few networks that tracks your router's 
convergence time over time, and identifies and resolves unexpected 
increases in convergence time, then yes, you absolutely have standing 
to tell me to pipe down and go back into my corner again.  ;D


So, while this all sounds good, without any specifics on vendor, box, 
code, code revision number, fix, year it happened, current status, 
e.t.c., I can't offer any meaningful engagement.


We all run into odd stuff as we operate this Internet, but the point of 
a list like this is to share those details so we can learn, fix and move 
forward.


Your ambiguity does not lend itself to a helpful discussion, 
notwithstanding my understanding of your caution.


I am less concerned about keeping smiles on vendors' faces. I tell them 
in public and private if they are great or not. But since you've been 
burned, I get. It's just not moving the needle on this thread, though.


Mark.

Re: Destination Preference Attribute for BGP

[Mark Tinka]

On 8/18/23 22:40, Matthew Petach wrote:



Hi Robert,

Without naming any names, I will note that at some point in the 
not-too-distant past, I was part of a new-years-eve-holiday-escalation 
to $BACKBONE_ROUTER_PROVIDER when the global network I was involved 
with started seeing excessive convergence times (greater than one hour 
from BGP update message received to FIB being updated).
After tracking down development engineer from $RTR_PROVIDER on the new 
years eve holiday, it was determined that the problem lay in 
assumptions made about how communities were stored in memory.  Think 
hashed buckets, with linked lists within each bucket.  If the 
communities all happened to hash to the same bucket, the linked list 
in that bucket became extremely long; and if every prefix coming in, 
say from multiple sessions with a major transit provider, happened to 
be adding one more community to the very long linked list in that one 
hash bucket, well, it ended up slowing down the processing to the 
point where updates to the FIB were still trickling in an hour after 
the BGP neighbor had finished sending updates across.


A new hash function was developed on New Year's day, and a new version 
of code was built for us to deploy under relatively painful 
circumstances.


It's easy to say "Considering that we are talking about control 
plane memory I think the cost/space associated with storing 
communities is less then negligible these days."
The reality is very different, because it's not just about efficiently 
*storing* communities, it's really about efficiently *parsing and 
updating* communities--and the choices made there absolutely *DO* 
"contribute to longer protocol convergences in any measurable way."


Matt
(the names have been obscured to increase my chances of being hireable 
in the industry again at some future date. ;)


To be fair, you are talking about an arbitrary value of years back, on 
boxes you don't name running code you won't mention.


This really not saying much :-).

Corner cases, while valid, do not speak to the majority. If this was a 
major issue, there would have been more noise about it by now.


There has been quite some noise about lengthy AS_PATH updates that bring 
some routers down, which has usually been fixed with improved BGP code. 
But even those are not too common, if one considers a 365-day period.


Mark.

Re: Destination Preference Attribute for BGP

[Mark Tinka]

On 8/18/23 22:20, Jakob Heitz (jheitz) via NANOG wrote:


We support platforms of various capacities.

While we would all like to sell the large ones, people buy the cheap 
ones too.




Even a bare bones x86 platform of some sort with at least 8GB of RAM 
would make the cheapest routers still, well, cheap.


Mark.