Re: Fast backbone to NA from Asia
On 5/22/24 16:55, Scott Q. wrote: Hi Mark, thank you for the very informative post! In the meantime, our own provider moved some routes from GTT to...Cogent and the times decreased within the normal range. Also a few VPS providers such as EdgeNext are using NTT & Cogent and show no issues either. So it seems some providers experience an extra 80ms delay for some reason, I wonder why. This is our new traceroute Yeah - best thing to do would be to reach out to a problematic provider and ask them for an explanation. Usually, if they have bought directly from a subsea provider, then restoring from a subsea outage may be complex depending on how well they secured themselves both from a diversity and capacity standpoint. If they are a customer of a major transit provider, then their provider's subsea inventory situation is similar to my point above. It is very hard to tell unless you ask someone on the inside, but as these things do, when cables get cut, latency and packet loss increases are not unexpected, especially for small/local/regional ISP's that can't afford to have direct access to multiple subsea systems. And in cases where alternative options are either too costly or non-existent, the latency and packet loss penalty would be sustained longer than necessary. Depending on where you are in the food chain, subsea restoration efforts following a major cut can increase normal pricing by 3X - 5X, particularly if the restoration capacity is taken on a short-term basis, e.g., 3 months. Mark.
Re: Fast backbone to NA from Asia
On 5/22/24 14:44, Paul Rolland wrote: Yep, it hurts :( 1. Gi0-3.rtr-01.PAR.witbe.net0.0% 1790.3 0.3 0.2 10.4 0.7 2. 193.251.248.210.0% 1793.3 1.3 0.8 19.1 2.1 3. bundle-ether305.partr2.saint-den 6.7% 179 87.1 4.5 1.1 156.7 17.4 4. prs-b1-link.ip.twelve99.net 22.3% 1799.9 10.4 9.6 48.3 4.4 5. prs-bb2-link.ip.twelve99.net 2.2% 179 10.4 10.3 9.8 27.3 1.3 6. mei-b5-link.ip.twelve99.net 1.1% 178 17.3 18.1 17.2 115.1 7.4 7. prs-bb1-link.ip.twelve99.net 27.5% 178 370.6 365.9 334.7 381.8 8.3 8. ldn-bb1-link.ip.twelve99.net 68.5% 178 366.8 363.0 340.8 379.2 8.3 9. nyk-bb2-link.ip.twelve99.net 11.8% 178 377.6 362.4 322.2 451.0 12.3 10. palo-b24-link.ip.twelve99.net50.8% 178 359.1 364.1 342.8 397.1 8.6 11. port-b3-link.ip.twelve99.net 0.0% 178 177.4 178.0 177.1 188.6 1.9 12. tky-b3-link.ip.twelve99.net 75.7% 178 355.2 364.0 339.8 377.5 8.0 13. tky-b2-link.ip.twelve99.net 50.0% 178 338.7 350.5 321.8 370.9 11.1 14. sng-b7-link.ip.twelve99.net 87.6% 178 307.8 318.6 306.8 332.0 6.7 15. sng-b5-link.ip.twelve99.net 86.4% 178 314.4 315.3 293.6 330.1 10.2 16. epsilon-ic-382489.ip.twelve99-cu 55.7% 177 364.8 362.9 346.5 391.8 9.1 17. 180.178.74.221 59.9% 177 357.6 366.9 343.4 562.7 25.8 18. swi-01-sin.noc.witbe.net 62.9% 176 374.7 366.4 346.6 381.3 8.3 1299 is now routing Paris to Singapore via US and Pacific... The good news is that the Yemeni government have approved repairs for EIG and SEACOM. The bad news is that those approvals don't yet extend to AAE-1, whose cut is the one causing you that pain. It's unclear when, or if, Yemen will give permission to repair AAE-1. The market is speculating mid-June, but there is no hard data to support that. Not sure if transition 6 to 7 is what was expected, with a 350ms increase... Well, on Arelion's network, PAO-SIN = 260ms: Tracing the route to 180.178.74.221 1 sjo-b23-link.ip.twelve99.net (62.115.115.217) 2 msec 2 msec 2 msec 2 * tky-b2-link.ip.twelve99.net (62.115.123.141) 187 msec 162 msec 3 * * * 4 * * 62.115.115.62 251 msec 5 * hnk-b3-link.ip.twelve99.net (62.115.143.241) 257 msec * 6 sng-b4-link.ip.twelve99.net (62.115.116.146) 280 msec * 222 msec 7 * * * 8 * * * 9 180.178.74.221 265 msec 262 msec * For the moment, it looks like you've switched to Zayo for transit in Paris, so unclear what Arelion's on network would do PAO-CDG: Tracing the route to 81.88.96.250 1 sjo-b23-link.ip.twelve99.net (62.115.115.217) 2 msec 2 msec 2 msec 2 ae71.zayo.ter1.sjc7.us.zip.zayo.com (64.125.15.150) 2 msec * * 3 * * * 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 ae1.mcs1.cdg12.fr.eth.zayo.com (64.125.29.87) 148 msec * 158 msec 10 v3.ae10.ter3.eqx2.par.as8218.eu (64.125.30.183) 150 msec 151 msec 151 msec 11 ae6.ter4.eqx2.par.core.as8218.eu (83.167.55.43) 151 msec 152 msec 152 msec 12 ae0.ter3.itx5.par.core.as8218.eu (83.167.55.10) 148 msec 148 msec 148 msec 13 witbe-gw1.ter1.itx5.par.cust.as8218.eu (158.255.117.19) 151 msec 153 msec 153 msec 14 Gi0-3.rtr-01.PAR.witbe.net (81.88.96.250) 152 msec * 151 msec HE did/does that too, prefering to avoid any direct route from EU to Asia. Well, right now, of the modern cables that had capacity and reasonable pricing, only SMW-5 remains up... and SMW-5 is just about out of capacity as well. SMW-6 is currently under construction, so that is not yet an option (the Red Sea debacle notwithstanding). Subsea systems that need to cross the Middle East and Egypt to connect Europe and Africa to South (East) Asia are generally problematic because of the complexities of having to deal with Egypt, and now, the Red Sea. That translates into capacity availability (or lack thereof in times like these) and cost. This creates an incentive for operators to route South (East) Asia through the U.S. to get to Europe, until the situation resolves itself, or new cables with new/cheaper capacity pop up. Mark.
Re: Fast backbone to NA from Asia
On 5/21/24 21:55, Dovid Bender wrote: Could it be related to the fiber cut in the red sea? Asia-Pac <=> North America is typically faster via the Pacific, not the Indian Ocean. The Red Sea cuts would impact Asia-Pac <=> Europe traffic. SMW-5 had an outage on the 19th of April around the Straits of Malacca between Malaysia and Indonesia. The suspected cause is a shunt fault. A shunt fault occurs when the cable insulation is damaged and exposes the electrical wire in the cable, causing a short. This shifts the virtual ground of the electrical circuit toward the shunt fault location. In many cases, the PFE (Power Feeding Equipment) farthest from the shunt is able to re-balance and pump enough power down the cable from the CLS to maintain the required voltage. However, in some cases - such as the one with this particular SMW-5 fault - the short can become significant enough that there is a total loss of current to drive the segment, leading to an outage. At this time, repairs are delayed until around end of May. But given the location of the fault, I don't see how it would impact traffic toward North America from Singapore. Impact seems to mainly be between Bangladesh <=> Singapore. This is Telstra: 1 * * i-92.sgcn-core01.telstraglobal.net (202.84.219.174) 8 msec 2 i-93.istt04.telstraglobal.net (202.84.224.190) 4 msec i-92.sgcn-core01.telstraglobal.net (202.84.219.174) [MPLS: Label 24210 Exp 0] 2 msec 2 msec 3 ae10.cr4-sin1.ip4.gtt.net (67.199.139.109) 22 msec i-91.istt04.telstraglobal.net (202.84.224.197) 3 msec ae10.cr4-sin1.ip4.gtt.net (67.199.139.109) 3 msec 4 ae16.cr0-mtl1.ip4.gtt.net (89.149.186.134) 239 msec ae10.cr4-sin1.ip4.gtt.net (67.199.139.109) 4 msec ae16.cr0-mtl1.ip4.gtt.net (89.149.186.134) 241 msec 5 ae16.cr0-mtl1.ip4.gtt.net (89.149.186.134) 241 msec ip4.gtt.net (72.29.198.6) 325 msec ae16.cr0-mtl1.ip4.gtt.net (89.149.186.134) 240 msec 6 10ge.mtl-bvh-xe3-1.peer1.net (216.187.113.107) 316 msec ip4.gtt.net (72.29.198.6) 330 msec 312 msec 7 managed5.top-consulting.net (69.90.179.5) 329 msec This is Arelion: Tracing the route to 69.90.179.5 1 sjo-b23-link.ip.twelve99.net (62.115.141.126) 183 msec sng-b4-link.ip.twelve99.net (62.115.137.243) 1 msec 10 msec 2 sjo-b23-link.ip.twelve99.net (62.115.136.166) 164 msec 164 msec 163 msec 3 nyk-bb1-link.ip.twelve99.net (62.115.137.168) 250 msec * motl-b2-link.ip.twelve99.net (62.115.137.143) 256 msec 4 motl-b1-link.ip.twelve99.net (62.115.126.220) 244 msec 245 msec 242 msec 5 aptummanaged-ic-367443.ip.twelve99-cust.net (62.115.174.15) 257 msec 256 msec 263 msec 6 10ge.mtl-bvh-xe3-1.peer1.net (216.187.113.107) 274 msec 268 msec 264 msec 7 managed5.top-consulting.net (69.90.179.5) 258 msec 259 msec 261 msec This is GTT: traceroute to 69.90.179.5 (69.90.179.5), 12 hops max, 52 byte packets 1 ae4.lr4-sin1.ip4.gtt.net (89.149.131.98) 1.015 ms 4.696 ms 0.741 ms MPLS Label=185733 CoS=0 TTL=1 S=1 2 et-0-0-4.lr3-lax2.ip4.gtt.net (89.149.131.233) 162.343 ms 163.306 ms 210.101 ms MPLS Label=983708 CoS=0 TTL=1 S=1 3 ae0.lr4-lax2.ip4.gtt.net (89.149.184.30) 161.869 ms 163.878 ms 163.078 ms MPLS Label=417505 CoS=0 TTL=1 S=1 4 ae14.lr7-chi1.ip4.gtt.net (89.149.143.161) 217.467 ms 202.565 ms 203.520 ms MPLS Label=188676 CoS=0 TTL=1 S=1 5 ae18.lr5-chi1.ip4.gtt.net (89.149.136.81) 206.519 ms 202.270 ms 203.047 ms MPLS Label=913278 CoS=0 TTL=1 S=1 6 ae19.lr6-chi1.ip4.gtt.net (89.149.141.194) 202.559 ms 202.853 ms 202.225 ms MPLS Label=422136 CoS=0 TTL=1 S=1 7 ae7.lr3-tor1.ip4.gtt.net (89.149.143.242) 212.461 ms 212.858 ms 230.237 ms MPLS Label=692593 CoS=0 TTL=1 S=1 8 ae6.cr9-mtl1.ip4.gtt.net (89.149.187.242) 221.465 ms 230.552 ms 218.920 ms MPLS Label=284571 CoS=0 TTL=1 S=1 9 ae16.cr0-mtl1.ip4.gtt.net (89.149.186.134) 217.961 ms 219.497 ms 219.392 ms 10 ip4.gtt.net (72.29.198.6) 217.164 ms 218.776 ms 217.011 ms 11 10ge.mtl-bvh-xe3-1.peer1.net (216.187.113.107) 220.991 ms 218.415 ms 220.009 ms 12 managed5.top-consulting.net (69.90.179.5) 217.273 ms 217.417 ms 217.079 ms Looks like Telstra are handing off to GTT. The latency increase at hop 6 suggests asymmetric routing. Arelion and GTT are within your 250ms spec, although it looks like GTT has more efficient U.S. routing to get to MTL. I'm not aware of any major subsea cut in Asia-Pac bar the SMW-5 one, so my guess is Linode and Digital Ocean might need to look into their routing with their upstreams out of Singapore. Mark.
Re: Cogent-TATA peering dispute?
On 5/18/24 08:56, Saku Ytti wrote: As long as our toolbox only has a capitalist hammer, peering disputes are going to be a thing. Cogent has outlived many of its peering dispute history partners. They are the grandfather of disruptive transit pricing, which many others have struggled to meet profitably, and I believe they are a big reason for current transit pricing being as low as it is in the US and EU. Or to put it another way, if the community thought Cogent was not providing some value to them, they would no longer be in business. Mark.
Re: Cogent-TATA peering dispute?
On 5/18/24 06:04, Jon Lewis wrote: Cogent has been trying to establish themselves as a "tier 1" carrier in markets outside their home turf, and Asia is one of the markets in which the established operators are doing their best to keep Cogent out. Back when I was helping to run a global anycast CDN, Cogent was one of our transits [in US and EU POPs]. I identified a bunch of sub-optimal service to networks in Asia who were silly/cheap enough to buy transit from Cogent. Since none of the established players would peer in-market with Cogent, and since we didn't have Cogent transit in any of our Asian POPs (why would we?), anycast request traffic from those Cogent customers would cross the Pacific and land in California rather than hit a nearby POP with NTT, Tata, Singtel, etc. transits. I suspect Cogent has either reached what they consider to be customer critical mass in Asia, or is tired of their Asian customers complaining about latency (since traffic to any other in-market non-Cogent connected network routes via the US or EU) and has decided it's time to play peering chicken to force Tata to peer with them in Asia. Why shouldn't they? The only reason Tata, NTT, etc. won't peer with Cogent in-market is because they don't want Cogent to be able to compete with them in their home market. They have a similar problem in Africa with the major African IP Transit providers; and they are far less deployed in Africa than they are in Asia. Mark.
Re: Should FCC look at SS7 vulnerabilities or BGP vulnerabilities
On 5/16/24 21:53, Brandon Zhi wrote: Are APNs like a vpn for mobile devices to access the public internet? Based on the experience that I used Mobile roaming outside my country. The provider would connect back to the original country via local providers. When roaming, the home mobile network has two options to deliver data services to their customer: * Breakout to the Internet using the local roaming partner, or * Tunnel to the home network via the local roaming partner, and breakout to the Internet there. Both models are viable particularly if the roaming partner and home network are basing their roaming architecture on IPX rather than GRX. Local breakout improves performance because it is low-latency, while remote breakout is often preferred because it does not complicate billing and other traffic controls imposed by the home network. My anecdotal experience has been that you will have local breakout sometimes, and remote breakout most of the time. This will also vary from provider to provider. I also find that home networks tend to prefer remote breakout, while users, unsurprisingly, will have a better experience with local breakout. I've never been able to find conclusive data on which mobile operators implement local vs. remote breakout. It doesn't appear that the GSMA mandate any one model over another against their membership, so mobile operators are likely making individual choices on what they do. Either way, with an IPX-based roaming architecture, it is really just a glorified l3vpn cloud built on a standard IP/MPLS network. If you have time, the below is an interesting read: https://www.gsma.com/newsroom/wp-content/uploads//IR.34-v17.0.pdf Mark.
Re: Alien Waves
On 5/13/24 05:32, Brandon Martin wrote: I doubt that's changed given my dealings with them since (which have been fine, to be clear), but I can't be 100% sure. I suspect they did at least turn up a few of them given that they went to the trouble of creating a full-fledged product for it. Maybe they found out the hassle wasn't worth it? The upfront cash you get is great to make the quarter's/year's number in one go, but after that, all you are earning is annual O, which is not a lot especially if you still have to spend quite a bit maintaining the outside plant. And once that spectrum is gone, you can't touch it again even if the customer you sold it to may not be using all of it at the time. Electrical bandwidth is slow to shift, but it is stable monthly/quarterly revenue that you can bank a business plan on. Mark.
Re: Alien Waves
On 5/13/24 04:07, Dave Cohen wrote: My point was that the technology has little to do with the operational success of the service. Spectrum controllers better enabling providers to get out of their own way in selling spectrum did not actually enable the providers* to get out of their own way in selling spectrum. It *should* be easier than it used to be, but it isn't, and the problem is not really technical, but a question of 1) not having full-throated commitment to wanting to sell spectrum and 2) not having the talent to support it, which is really just a function of #1. Fundamentally, I agree. This is one area where terrestrial operators will be late bloomers, as subsea shows and leads the way. My prediction is that there will be a slightly improved chance of spectrum services gaining a little more traction (not a lot) on the terrestrial side when the new-age DWDM vendors are able to offer more competitive and standards-based spectrum controllers. The other avenue of interest will be in mitigating the costs associated with upgrading to C+L network topologies, where some spectrum comes up for grabs as a quick way to recover the investment. And lastly, content folk looking to enter markets on the back of existing operators where the appetite for negotiating dark fibre is relatively low, will apply pressure on those reluctant operators to offer up some spectrum. We have already seen, across the world, how "convincing" the content folk have been at this sort of thing. But for the most part, yes, I expect the bulk of DWDM services sold to terrestrial network users will continue to be electrical bandwidth, and not optical spectrum, at least for the next few years. I could potentially see a case for a specialist DWDM operator who focuses on a spectrum-based service network that sells to 3 - 5 high-capacity customers, but those will be very specialist. Mark.
Re: Alien Waves
On 5/13/24 00:11, Dave Cohen wrote: Mark, Many/all of these points are fair. My experience is purely terrestrial and obviously both the capacity and economic calculations are vastly different in those situations, which I should have called out. Actually, terrestrial economics are easier to consider because you have the one thing the subsea applications don't have in abundance... power. Fair point, terrestrial revenues are significantly lower than subsea revenues on a per-bit basis, but so are the deployment costs. That evens out, somewhat. However, I don’t think that the optical vendor is really the challenge - I would agree that, generally, spectrum is going to be available through larger providers that are using “traditional carrier grade” platforms - but rather at the service provider level. When something invariably breaks at 3 AM and the third shift Tier I NOC tech who hasn’t read the service playbook says “I don’t see any errors on your transponder, sorry, it’s not on our end” because they’re not aware that they actually don’t have access to the transponder and need to start looking elsewhere, that’s the sort of thing that creates systemic challenges for users regardless of whether the light is being shot across a Ciena 6500 or a Dave’s Box-o’-Lasers 1000. I think you are contradicting yourself a bit, unless I misunderstand your point. Legacy vendors who have spectrum controllers have made this concern less of an issue. But then again, to be fair, adopting spectrum controllers along with bandwidth expansions via things like gridless line systems and C+L backbone architectures that make spectrum sales a lot more viable at scale do come at a hefty $$ premium. So I can understand that offering spectrum independent of spectrum controllers is going to be more trouble than it is worth. Ultimately, what I'm saying is that technologically, this is now a solved problem, for the most part. That said, I don't think it will be the majority of DWDM operators offering spectrum services en masse, for at least a few more years. So even if you want to procure managed spectrum or spectrum sharing, you are likely to come up against a limited set of providers willing to sell it, if at all. Mark.
Re: Alien Waves
On 5/12/24 20:35, Dave Cohen wrote: It’s one of those things that makes a lot more sense on paper than in practice. Not anymore. The majority of SDM subsea cables built with uncompensated fibre are using managed spectrum and spectrum sharing as viable business models for a not-so-insignificant population of their customer base. I’ve found it to be operationally difficult from the perspective the provider and the user, primarily but not solely because any “co-managed” system is going to lend itself to finger pointing when issues arise. And there is a case to be made for that concern. However, if you are in a position to be able to sell spectrum, it is very likely you are going to be implementing a vendor of note. Since that is most likely to be the case, those vendors have spectrum controllers that make this a viable business model, albeit at a $$ premium. This is not the type of service small DWDM operators using new-age DWDM vendors would typically be looking to sell. Such operators have to deal with keeping the lights on, never mind esoteric services like spectrum. Even if it makes more commercial sense at first blush to prefer a spectrum solution over dark or traditional waves, I suspect that factoring in “labor cost wasted over unproductive troubleshooting” changes the equation a bit. Alien wave and spectrum services attract a very high income, mainly through a capex-based upfront cost (IRU) that can be attractive to the host network. At those levels, providing their vendor has decent support for spectrum services, the revenue gain more-than makes up for all the logistical admin. I also suspect that continued price compression on optical hardware will lead to fewer and fewer situations where it might make commercial sense at first blush too. Well, legacy DWDM vendors will continue to charge a premium even for what is now standard electrical bandwidth services. Why? Because they still have all that legacy stuff to support, all their R to recoup, and because the bulk of their customers are no longer the telco, but content folk. New-age DWDM vendors are focused on coherent optical networks, which are primarily 100G and 400G. Why? Because that is where MSA and OpenROADM are currently at re: commercial availability. The legacy vendors will develop proprietary coherent pluggables that will support funky things such as 800G, 1.2T and 1.6T, but those won't be industry standard for some time (800G is getting there, though). What all this means is that if you are a legacy operator that is careful about spending money on newer DWDM technologies, a spectrum service from a larger carrier is going to be more attractive than ripping out your entire line system just so you can get from 10G or 40G to 400G. Of course, if you are a monopoly and have no alternatives to lean on, this doesn't count. YMMV of course and there may be reasons beyond simple commercial models where spectrum might make sense for you, but I’d urge you to only consider it doing with a provider where you’ve had a track record of operational success working with them. New-age DWDM vendors are not the workhorse of most of the large DWDM operator networks out there. That means that any operator of note you are likely to run into is going to be a Ciena, Infinera, Nokia, Adva, e.t.c., house, or something along those lines. Those vendors have reasonable spectrum-based solutions that smaller DWDM operators or ISP's would be willing to spend money on to avoid having to upgrade or deploy an entire line system. The reason that is feasible is because those larger operators are running vendors who push beyond what the MSA and OpenROADM groups are prescribing. You can already get coherent 100G and 400G channels on new-age DWDM vendors... that is not rocket science anymore. Of course, there is always the IPoDWDM question... but if I'm honest, I am still as unconvinced now in 2024 as I was back in 2008 that optical and IP folk would have a meeting of the minds on this. Mark.
Re: Alien Waves
On 5/12/24 14:08, Mike Hammett wrote: What are your experiences with alien waves, managed spectrum, spectrum as a service, etc? Your outcomes will vary depending on whether this is deployed for terrestrial or subsea networks. Subsea networks don't typically do alien waves, but rather, managed spectrum or spectrum sharing. This is especially the case on the newer SDM-based uncompensated submarine cable systems, where there is a huge volume of fibre pairs that makes this feasible, if not the most economical way to sell the asset to volume customers. For terrestrial, alien waves were the original model, and in my opinion, the preferred one, because all the host network has to do is provide a port on their filter with a wavelength. The filter isolates the adjacent signals from one another, which improves launch OSNR. That said, managed spectrum and spectrum sharing are quickly replacing alien waves as the preferred deployment option for terrestrial networks, which can largely be blamed on advances for the same happening on the submarine side of things, even though the original idea was mostly driven by GEANT and a bunch of European NREN's back in the day. Managed spectrum and spectrum sharing are more problematic because the chance of broadcasting bad noise to all other channels increases. Yes, major DWDM vendors now do have significantly improved optical power management systems (a spectrum controller, let's say) that will interact with the WSS in their ROADM, where the ROADM will set the centre frequency and its width, which helps to restrict any negative impact to launch insertion, and not toward the line side. Different vendors will have different spectrum controller options that make managed spectrum and spectrum sharing services either simple or difficult to deliver on their specific type of gear. If it is something you want to be serious about, this will be the one time where PoC'ing all the vendors you are interested in is worth your time. It would also be useful to understand how each vendor supports things such as T-API (Transport-API) and other OpenROADM open architecture features to improve wavelength and optical power management characteristics between different vendors sharing a single OLS (Optical Line System). You may find that support for T-API and other OpenROADM standards may be spotty to non-existent with many vendors, but a vendor with a solid roadmap is certainly not a waste of your time. Major traditional vendors like Ciena, Infinera, Nokia, Adva, Ribbon, and such, will have very extensive spectrum controllers, but they will come with the requisite $$ premium. Newer vendors whose platforms are based primarily on coherent pluggables approved by the MSA and OpenROADM will support alien waves, but may struggle to offer a comparable spectrum controller solution for managed spectrum and spectrum sharing, even if they may have a rudimentary ability to do so. Due diligence is highly warranted here, as the landscape is changing on a daily basis. In essence, "virtual fibre pair services" (if I can call them that) is a matter of security, by way of total optical power control. What you want the vendors you consider to answer is: * If a spectrum customer erroneously provisions spectrum outside of their allocated bandwidth, how does the host network deal with that so that it does not impact any other spectrum customers on the same fibre pair? * How do you effectively restrict spectrum customers from only being able to access just their allocated spectrum, where a simple broadband splitter would not be sufficient for this? * How do you monitor the optical spectrum between each spectrum customer to ensure optimal optical performance on a per-spectrum-customer basis? * Especially for subsea applications, but nowadays, also for terrestrial ones; how do you monitor and manage optical power requirements for unallocated spectrum, including previously-allocated spectrum to a spectrum customer whose signal has now "disappeared" due to a failure of their own SLTE (Submarine Line Terminating Equipment) or transponder? In other words, ASE (Amplified Spontaneous Emission) noise loading capability. Answering these questions makes it easier for interested parties looking to move away from procuring electrical bandwidth to, rather, procuring optical spectrum. Hope this helps. Mark.
Re: Opengear alternatives that support 5g?
On 4/28/24 18:54, Siyuan Miao wrote: We use GL.inet and set up WireGuard VPNs back to our distributed VPN servers. Our console servers support dual uplink, so we just connect port 1 to the GL.inet LAN and port 2 to our management switch. Currently, we're still using their LTE model, and it costs ~100 USD per site, but their 5G models are expensive and cost around $500. At new job, I am looking at using pfSense-based VPN's to create the DCN. It does consume 1U and a couple of cabinet watts for the server, but it's stable, feature-rich, well-supported, and network media agnostic. Mark.
Re: Opengear alternatives that support 5g?
On 4/27/24 17:49, Mel Beckman wrote: Quite often I’m looking for OOBM at antenna sites or in remote DCs where there is no Plan B carrier. Cellular has always been the goto choice for this, but we keep getting pushed out of contracts by technology upgrades. 2g, then 3g, and next 4g LTE are being deprecated. The main reason for network shutdowns is that the carriers have limited spectrum available for expansion. To deliver faster, more cost effective data service to customers, carriers must re-use existing spectrum licenses with newer, more efficient cellular technology. Old 2G/3G infrastructure makes way for new networks, and older cellular devices must be retired. 4g may have a decade left before complete absence, but its footprint is already shrinking where 5G is available. I’ve seen this first hand with 4g cellular alarm circuits: suddenly they get less reliable or fail completely, and the reason always turns out to be degraded RSSI due to 5G deployment. So 5G is imperative for cellular OOBM, hence the hunt for COTS drop-in replacements that won’t break the bank. Upgrading, for example, 100 antenna sites is also a major truck roll cost, so we want to get it right the first time. Physical space and power limitations usually rule out 1U rackmount refurb Cisco terminal servers, which is why we need 0U gear. Yes, I can cobble together a raspberry pi and some hats and cables and dingles and dangles and make a science fair solution. But I need something that is commercially supported, won’t have me scratching my head later about what version of the Ubuntu is going to work, and won’t randomly fry its electronics during a power surge. It’s looking like that solution is firmly priced at ~$500 today. Fair enough - if the bulk of your OoB use-case is remote (cell) sites, your typical options won't work or will be limited. Oddly, in our parts, we find remote, non-city locations, tend to keep their 3G/4G status, or don't even get considered for 5G at all. But I guess this will vary by market the world over, so I could see a remote site in Norway, for example, having 5G vs. a remote site in, say, Egypt, doing the same. I think what you probably want to consider for the long-term is decoupling the device from the network media. If you can attach a MiFi router via a USB port to a cheap device (like Mikrotik), this would help keep costs down as mobile operators deprecate GSM data technologies in the future. I like Mikrotik because in addition to being cheap and feature-rich for basic network access, the firmware is regularly upgradeable unlike typical consumer-style CPE's. Mark.
Re: Opengear alternatives that support 5g?
On 4/27/24 07:56, Saku Ytti wrote: For me Cisco is great here, because it's something an organisation already knows how to source, turn-up, upgrade, troubleshoot, maintain. And you get a broad set of features you might want, IPSEC, DMVPN, BGP, ISIS, and so forth. I tend to agree. Cisco do this very well, and if you are really low on cash and okay with acquiring these on the cheap, the open market has tons of deals and options from Cisco that have matured over the decades. I keep wondering why everyone is so focused on OOB hardware cost, when in my experience the ethernet connection is ~200-300USD (150USD can be just xconn) MRC. So in 10 years, you'll pay 24k to 36k just for the OOB WAN, masking the hardware price. And 10years, to me, doesn't sound even particularly long a time for a console setup. Is a 10Mbps DIA link going for US$200 - US$300 MRC nowadays, excluding the x-connect? I'd have though it's now in US$100 range at the very worst. Or are you looking at an OoB link of more than 10Mbps? Mark.
Re: constant FEC errors juniper mpc10e 400g
On 4/22/24 09:47, Vasilenko Eduard via NANOG wrote: Assume that some carrier has 10k FBB subscribers in a particular municipality (without any hope of considerably increasing this number). 2Mbps is the current average per household in the busy hour, pretty uniform worldwide. You could multiply it by 8/7 if you like to add wireless -> not much would change. 2*2*10GE (2*10GE on the ring in every direction) is 2 times than needed to carry 10k subscribers. The optical ring may be less than 20 municipalities - it is very common. Hence, the upgrade of old extremely cheap 10GE DWDM systems (for 40 lambdas) makes sense for some carriers. It depends on the population density and the carrier market share. 10GE for the WAN side would not be dead in the next 5 years because 2Mbps per household would not grow very fast in the future - this logistic curve is close to a plateau. PS: It is probably not the case for Africa where new subscribers are connected to the Internet at a fast rate. As a function of how much Internet there is in Africa, there really aren't that many optical transport service providers. Some countries/cities/towns have more than they need, others have just one. But in general, you would say there is massive room for improvement if you surveyed the entire continent. Typically, it will be the incumbents, alongside 2 or 3 competitives. In fact, in some African countries, only the incumbent may be large enough to run an optical backbone, with all the competitives leasing capacity from them. It is not uncommon to find the closest competitor to an incumbent for terrestrial services being the mobile network operator, purely because they have some excess capacity left over from having to build the backbone for their core business, mobile. And, they are flush with cash, so a loss-making terrestrial backhaul business can be covered by the month's sales in SIM cards. Truly independent transport providers are few and far between because access to dark fibre is not easy (either its lack of availability, the incumbent refusing to sell it, or its high price). For the few independent transport providers that do spring up, they will focus on a limited set of hot routes, and because competition on those routes may be wanting, prices and capacity would not be terribly attractive. So the bulk of Africa's Internet really relies on a handful of key African wholesale IP Transit providers taking great effort into extending their network as deep into cities as they can, and using their size to negotiate the best prices for terrestrial backhaul from the few optical network operators that the market has. Those providers then sell to the local and regional ISP's, who don't have to worry about running a backbone. All this means is that for those operators that run an optical backbone, especially nationally, 10G carriers are very, very legacy. If they still have them, it'd be a spin-off off the main core to support some old SDH customers that are too comfortable to move to Ethernet. Metro backhaul and last mile FNO's (fibre network operators) who have invested in extending access into homes and businesses are a different story, with most countries having a reasonable handful of options customers can choose from. Like national backhaul, there is plenty of room for improvement - in some markets more than others - but unlike national backhaul, not as constrained for choice or price. Mark.
Re: constant FEC errors juniper mpc10e 400g
On 4/21/24 08:12, Saku Ytti wrote: Key difference being, WAN-PHY does not provide synchronous timing, so it's not SDH/SONET compatible for strict definition for it, but it does have the frame format. And the optical systems which could regenerate SONET/SDH framing, didn't care about timing, they just wanted to be able to parse and generate those frames, which they could, but they could not do it for ethernet frames. Correct. In those days, WAN-PHY was considered "SONET/SDH-Lite". I think it is pretty clear, the driver was to support long haul regeneration, so it was always going to be a stop-gap solution. Even though I know some networks, who specifically wanted WAN-PHY for its error reporting capabilities, I don't think this was majority driver, majority driver almost certainly was 'thats only thing we can put on this circuit'. SONET/SDH did have similar reach as OTN back then, just less bandwidth for the distance. It had FEC support for STM-16, STM-64 and STM-256. I really think the bigger driver was interface cost, because EoS had already been selling for 1GE alongside STM-16 for 2.5G. In those days, if you needed more than 1G but less than 10G, it was a toss-up between 2x 1G EoSDH vs. 1x STM-16. Often times, you took the 2x 1G EoSDH because 2x 1GE ports were cheaper than 1x STM-16 port, even though you ended up losing about 405Mbps of capacity in the process, which was a huge deal. The backbone providers did not like selling EoSDH services, because it was too much admin. for them (VC container management), and they ended up paying more for transponders on their side than their customers did for Ethernet ports on theirs :-). But by and large, the majority of networks in our market maintained SDH services long after coherent became commercially available. It was a perception thing, that SDH was more superior to Ethernet, even if that Ethernet was transported over a DWDM network. In the end, SDH port costs were too hard to ignore due to router vendors maintaining their mark-up on them despite dying demand, which then led to the migration from SDH to EoDWDM growing significantly from about 2016. Optical vendors also began de-prioritizing SDH transponder ports, which had a massive impact on the SLTE (submarine) side in making the decision to encourage customers away from SDH for wet services. Mark.
Re: constant FEC errors juniper mpc10e 400g
On 4/20/24 21:36, b...@uu3.net wrote: Erm, WAN-PHY did not extend into 40G because there was not much of those STM-256 deployment? (or customers didnt wanted to pay for those). With SONET/SDH, as the traffic rate increased, so did the number of overhead bytes. With every iteration of the data rate, the overhead bytes quadrupled. This was one of the key reasons we did not see field deployment of STM-256/OC-768 and STM-1024/OC-3072. For example, if SONET/SDH had to transport a 100G service, it would require 160Gbps of bandwidth. That clearly wasn't going to work. At the time when STM-256/OC-768 was being developed, DWDM and OTN had come a long way. The granularity SONET/SDH required to stand up a service had become too small for the growing data rate (primarily VC-3, VC-4 and VC-12). If you look at OTN, the smallest container is 1.25Gbps (ODU0), which was attractive for networks looking to migrate from E1's, E3's, STM-1's, STM-4's and STM-16's - carried over VC-12, VC-4 and VC-3 SDH circuits - to 1GE, for example, rather than trying to keep their PDH/SDH infrastructure going. DWDM and OTN permitted a very small control overhead, so as data rates increased, there wasn't the same penalty you got with SONET/SDH. WAN-PHY was designed so people could encapsulate Ethernet frames right into STM-64. Once world moved out of SDH/SONET stuff, there was no more need for WAN-PHY anymore. Technically, what you are describing is EoS (Ethernet over SONET, Ethernet over SDH), which is not the same as WAN-PHY (although the working groups that developed these nearly confused each other in the process, ANSI/ITU for the former vs. IEEE for the latter). WAN-PHY was developed to be operated across multiple vendors over different media... SONET/SDH, DWDM, IP/MPLS/Ethernet devices and even dark fibre. The goal of WAN-PHY was to deliver a low-cost Ethernet interface that was SONET/SDH-compatible, as EoS interfaces were too costly for operators and their customers. As we saw in real life, 10GE ports out-sold STM-64/OC-192 ports, as networks replaced SONET/SDH backbones with DWDM and OTN. Mark.
Re: constant FEC errors juniper mpc10e 400g
On 4/20/24 18:19, Tarko Tikan wrote: 10G wavelengths for new builds died about 10 years ago when coherent 100G became available, submarine or not. Putting 10G into same system is not really feasible at all. I was referring to 10G services (client-side), not 10G wavelengths (line side). Mark.
Re: constant FEC errors juniper mpc10e 400g
On 4/20/24 14:41, Dave Cohen wrote: LAN PHY dominates in the US too. Requests for WAN PHY were almost exclusively for terrestrial backhaul extending off of legacy subsea systems that still commonly had TDM-framed services. It’s been a couple of years since I’ve been in optical transport directly but these requests were essentially non-existent after 2018 or so. OTN became somewhat more common from 2014 onward as optical system interop improved, but actually was more common in the enterprise space as providers would generally go straight to fiber in most use cases, and with dark fiber opex costs coming down in many markets, I see OTN requests as winnowing here as well. What really changed the game was coherent detection, which breathed new life into legacy subsea cables that were built on dispersion-managed fibre. Post-2014 when uncompensated (and highly dispersed) fibre has been the standard for subsea builds (even for SDM cables), coherent optical systems are the mainstay. In fact, because linear dispersion can be accurately calculated for the cable span, uncompensated cables are a good thing because the dispersion compensation happens in very advanced coherent DSP's in the optical engine, rather than in the fibre itself. WAN-PHY did not extend to 40G or 100G, which can explain one of the reasons it lost favour. For 10G, its availability also depended on the type of device, its NOS, line card and/or pluggable at the time, which made it hard to find a standard around this if you built multi-vendor networks or purchased backhaul services from 3rd party providers that had non-standard support for WAN-PHY/OTN/G.709. In other words, LAN-PHY (and plain Ethernet) became the lowest common denominator in the majority of cases for customers. In 2024, I find that operators care more about bringing the circuit up than using its link properties to trigger monitoring, failover and reconvergence. The simplest way to do that is to ask for plain Ethernet services, particularly for 100G and 400G, but also for 10G. In practice, this has been reasonably reliable in the past 2 - 3 years when procuring 100G backhaul services. So for the most part, users of these services seem to be otherwise happy. Mark.
Re: constant FEC errors juniper mpc10e 400g
On 4/20/24 13:39, Saku Ytti wrote: Oh I don't think OTN or WAN-PHY have any large deployment future, the cheapest option is 'good enough'... And what we find with EU providers is that Ethernet and OTN services are priced similarly. It's a software toggle on a transponder, but even then, Ethernet still continues to be preferred over OTN. Mark.
Re: constant FEC errors juniper mpc10e 400g
On 4/20/24 13:39, Saku Ytti wrote: Oh I don't think OTN or WAN-PHY have any large deployment future, the cheapest option is 'good enough' and whatever value you could extract from OTN or WAN-PHY, will be difficult to capitalise, people usually don't even capitalise the capabilities they already pay for in the cheaper technologies. A handful of OEM's still push OTN like it has just been invented, especially those still pushing "IPoDWDM" :-). Fair point, if you have a highly-meshed metro network with lots of drops to customers across a ring-mesh topology, there might be some value in OTN when delivering such services at low speeds (10G, 25G, 2.5G, 1G). But while the topology is valid, most networks aren't using high-end optical gear to drop low-speed services, nowadays. Even though on a per-bit basis, they might be cheaper than 1U IP/MPLS router looking to do the same job if all you are considering is traffic, and not additional services that want to eat packets. Of course WAN-PHY is dead post 10GE, a big reason for it to exist was very old optical systems which simply could not regenerate ethernet framing, not any features or functional benefits. In our market, we are trending toward a convergence between 10G and 100G orders intersecting for long haul and submarine asks. But pockets of 10G demand still exist in many African countries, and none of them have any WAN-PHY interest of any statistical significance. That said, I don't expect any subsea cables getting built in the next 3 years and later will have 10G as a product on the SLTE itself... it wouldn't be worth the spectrum. Mark.
Re: constant FEC errors juniper mpc10e 400g
On 4/20/24 13:25, Saku Ytti wrote: In most cases, modern optical long haul has a transponder, which terminates your FEC, because clients offer gray, and you like something a bit less depressing, like 1570.42nm. This is not just FEC terminating, but also to a degree autonego terminating, like RFI signal would be between you and transponder, so these connections can be, and regularly are, provided without proper end-to-end hardware liveliness, and even if they were delivered and tested to have proper end-to-end HW liveliness, that may change during operation, so line faults may or may not be propagated to both ends as RFI assertion, and even if they are, how delayed they are, they may suffer delay to allow for optical protection to engage, which may be undesirable, as it eats into your convergence budget. Of course the higher we go in the abstraction, the less likely you are to get things like HW livelines detection, like I don't really see anyone asking for this in their pseudowire services, even though it's something that actually can be delivered. In Junos it's a single config stanza in interface, to assert RFI to client port, if pseudowire goes down in the operator network. In our market (Africa), for both terrestrial and submarine services, OTN-type circuits are not typically ordered. Network operators are not really interested in receiving the additional link data that OTN or WAN-PHY provides. They truly want to leave the operation of the underlying transport backbone to the transport operator. The few times we have come across the market asking for OTN is if they want to groom 10x 10G into 1x 100G, for example, to deliver structured services downstream. Even when our market seeks OTN from European backhaul providers to extend submarine access into Europe and Asia-Pac, it is often for structured capacity grooming, and not for OAM benefit. It would be interesting to learn whether other markets in the world still make a preference for OTN in lieu of Ethernet, for the OAM benefit, en masse. When I worked in Malaysia back in the day (2007 - 2012), WAN-PHY was generally asked for for 10G services, until about 2010; when folk started to choose LAN-PHY. The reason, back then, was to get that extra 1% of pipe bandwidth :-). Mark.
Re: constant FEC errors juniper mpc10e 400g
On 4/19/24 10:08, Saku Ytti wrote: Of course there are limits to this, as FEC is hop-by-hop, so in long-haul you'll know about circuit quality to the transponder, not end-to-end. Unlike in wan-phy, OTN where you know both. Technically optical transport could induce FEC errors, if there are FEC errors on any hop, so consumers of optical networks need not have access to optical networks to know if it's end-to-end clean. This would only matter on ultra long haul optical spans where the signal would need to be regenerated, where - among many other values - FEC would need to be decoded, corrected and re-applied. SD-FEC already allows for a significant improvement in optical reach for a given modulation. This negates the need for early regeneration, assuming other optical penalties and impairments are satisfactorily compensated for. Of course, what a market defines as long haul or ultra long haul may vary; add to that the variability of regeneration spacing in such scenarios being quite wide, on the order of 600km - 1,000km. Much of this will come down to fibre, ROADM and coherent pluggable quality. Mark.
Re: constant FEC errors juniper mpc10e 400g
On 4/19/24 10:08, Saku Ytti wrote: Of course there are limits to this, as FEC is hop-by-hop, so in long-haul you'll know about circuit quality to the transponder, not end-to-end. Unlike in wan-phy, OTN where you know both. Technically optical transport could induce FEC errors, if there are FEC errors on any hop, so consumers of optical networks need not have access to optical networks to know if it's end-to-end clean. This would only matter on ultra long haul optical spans where the signal would need to be regenerated, where - among many other values - FEC would need to be decoded, corrected and re-applied. SD-FEC already allows for a significant improvement in optical reach for a given modulation. This negates the need for early regeneration, assuming other optical penalties and impairments are satisfactorily compensated for. Of course, what a market defines as long haul or ultra long haul may vary; add to that the variability of regeneration spacing in such scenarios being quite wide, on the order of 600km - 1,000km. Much of this will come down to fibre, ROADM and coherent pluggable quality.
Re: constant FEC errors juniper mpc10e 400g
On 4/19/24 08:01, Saku Ytti wrote: The frames in FEC are idle frames between actual ethernet frames. So you recall right, without FEC, you won't see this idle traffic. It's very very good, because now you actually know before putting the circuit in production, if the circuit works or not. Lot of people have processes to ping from router-to-router for N time, trying to determine circuit correctness before putting traffic on it, which looks absolutely childish compared to FEC, both in terms of how reliable the presumed outcome is and how long it takes to get to that presumed outcome. FEC is amazing. At higher data rates (100G and 400G) for long and ultra long haul optical networks, SD-FEC (Soft Decision FEC) carries a higher overhead penalty compared to HD-FEC (Hard Decision FEC), but the net OSNR gain more than compensates for that, and makes it worth it to increase transmission distance without compromising throughput. Mark.
Re: constant FEC errors juniper mpc10e 400g
On 4/18/24 15:45, Tom Beecher wrote: Just for extra clarity off those KB, probably has nothing to do with vendor interop as implied in at least one of those. Yes, correct. Mark.
Re: constant FEC errors juniper mpc10e 400g
On 4/17/24 23:24, Aaron Gould wrote: Well JTAC just said that it seems ok, and that 400g is going to show 4x more than 100g "This is due to having to synchronize much more to support higher data." We've seen the same between Juniper and Arista boxes in the same rack running at 100G, despite cleaning fibres, swapping optics, moving ports, moving line cards, e.t.c. TAC said it's a non-issue, and to be expected, and shared the same KB's. It's a bit disconcerting when you plot the data on your NMS, but it's not material. Mark.
Serious Bug in Cisco's 6500 & 6800 Platforms
https://sec.cloudapps.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-ios-dos-Hq4d3tZG Mark.
Re: Unimus as NCM (Network Configuration Management) Tool
On 4/4/24 08:25, Mike Lyon wrote: I use it for config backups, diffs, etc. Love it. Theres others such as Rancid but im not sure if it works on anything other than Vendor C. RANCID works perfectly for Cisco, Juniper, Arista, Brocade (Foundry) and HP. They are also known to support other obscure vendors. Mark.
Re: network simulator for service provider
On 4/3/24 04:13, aun Joe wrote: is there anysi network simulator for carrier networks ? well, from 2023 to 2024 there happes so many carrier network outage caused by network operation. to my limited knowledge , SP guru from riverbed could simulate carrier network. but I just checked riverbed website, SP guru stop updating in 2014. https://www.boson.com/netsim-cisco-network-simulator https://www.netacad.com/courses/packet-tracer https://www.gns3.com/ There is a bunch of others, but that should get your started down the rabbit hole. Mark.
Re: N91 Women mixer on Sunday?
On 3/29/24 21:48, Matthew Petach wrote: I'm with Randy Bush on this. The stakeholders in that event should have the say in what happens with it; not the rest of us. While I agree that women would know best how they want to socialize for their own interests, I don't think that men engaging in this discussion means they are necessarily prescribing how women should do that. All I have seen so far are opinions and suggestions to a concern the OP raised, not dictation on what will eventually happen. Those of us old white males need to check our privilege, and recognize that we've *been* having "mixers" for decades. We don't need to put a stake in the ground and push for our equality; we've already been on the beneficiary side of the inequality for decades. I also do not think that male guilt is necessary for women to achieve good outcomes in our shared industry. A complimentary working relationship could be remarkably more productive, than not. Mark.
Re: N91 Women mixer on Sunday?
On 3/30/24 03:45, Ryan Hamel wrote: Paul, Anne's opinion is just as valid as the others here. I have also browsed through the recent attendee lists and do not see you listed either, pot calling the kettle black. Your comments about her email signature and which voices are valid here, are not productive. We are allowed to back up and/or side with whomever we please, even if it includes the NANOG board, staff, and committee members. We are also allowed to call people out on their behavior towards others. Agreed. Constructive dialogue makes no requirement for prior or final agreement. More talking is better than less talking, especially when we disagree. Anyway, no one truly knows how many people could have raised the scheduling issue with various committee members, the board, staff, or provided feedback via the contact form on the website, and who knows it could have come from young women. Those voices do not have to come from the mailing list, to be just as valid as ours. I'd imagine that data would, at the very least, be with the PC and other members of NANOG staff; and in a manner that they can use to derive reasonable solutions for the future. Mark.
Re: N91 Women mixer on Sunday?
On 3/29/24 18:31, Tom Beecher wrote: I don't think anything is 'easily fixed' on this topic. I do however hope that everyone accepts at face value that the board, staff, and PC are *trying* to do the right things here. Doesn't mean that it *will* always be the right thing, or even that a majority of people can agree on what the right thing actually is. As I always say on matters such as these, "More talking is better than less talking". And as uncomfortable as such discussions are to have, we get closer to a reasonable solution when we talk, and less so when we don't. Mark.
Re: N91 Women mixer on Sunday?
On 3/29/24 18:23, Tom Beecher wrote: My recollection is that every WiT lunch was always open to all. Happy to be corrected if any were that I don't recall. There were definitely a few meetings during my PC years that someone complained that men were not allowed to attend. If my memory serves me correctly, we at one point were asking session moderators to remind people of this in general session for a while too. For some meetings, a few of us were standing at the doors telling people who asked that men were allowed to attend that if they would like. I can't remember which year it was, but my recollection was hearing the complaints from the men that were upset during the evening social. Admittedly, I did not hear this from what I would term "the majority of men" at that specific meeting, so it would be disingenuous to state, with any authority, that this is how the majority of men felt/feel about the WiT lunch. Having said all that, for the avoidance of doubt (and as Tina has already indicated for NANOG 91), it might be worth mentioning, before and throughout the conference, that while it is a WiT lunch, all attendees are welcome to join if that is, indeed, the intended format. Mark.
Re: N91 Women mixer on Sunday?
On 3/29/24 07:03, Eric Parsonage wrote: It's easily fixed by having a mixer at the same time for the other half of the gathering population thus showing all the population gathering matters equally. My memory is a little fuzzy, but I think I recall one of the early WiT lunches hosted at NANOG that was women-only, where some men were upset for being "left out". Whether that was good or bad is less important than understanding what the outcome of a women-only activity is for women, especially for those for whom it may not be immediately obvious. While equal access to opportunity between the genders is the most effective policy, I think there is utility in women having their own session, given that they face unique challenges in an industry where they are the minority operators. Mark.
Re: N91 Women mixer on Sunday?
On 3/29/24 06:20, Ren Provo wrote: I beg to differ here and second Ilissa’s comments. I miss WiT. Lunch during the meeting worked. Giving up more of the weekend to travel does not show half the population gathering matters. It would be interesting to survey the % of attending women who: - Would be able to make the mixer. - Would not be able to make the mixer due to family commitments. - Would not be able to make the mixer due to non-family commitments. - Would prefer a different women's meeting format and/or activity. I think this data would be useful because while some women may voice difficulty with attending the mixer (or wanting a different women's meeting format/activity), it might be premature to attribute that difficulty to all women that would be in attendance. Understanding where the majority of women lie can help the NANOG PC make better decisions about how to support both the majority and minority of women as it pertains to a mixer or other activity women may like to participate in at or around the conference. Mark.
Re: N91 Women mixer on Sunday?
On 3/28/24 21:08, Tom Beecher wrote There was a Women in Tech Mixer on Sunday in Charlotte as well. As I recall there was a pretty decent attendance. During my time on the PC, we always got a lot of feedback about Sunday when the topic came up. Some members were strongly opposed to anything on Sunday and didn't even like the Hackathon there. Others wanted expansion, and more things slotted in. There certainly wasn't anything remotely close to a consensus. Sometimes people can make it in early on Sunday. Sometimes they can't. There's no one size fits all answer. I'm not sure people realize how much crap that staff and the PC get *every meeting* about the agenda. There's always someone unhappy because this event wasn't the same, or why was it in this room over here, or OMG Wed afternoon, etc. Having seen how that sausage gets made, they don't get enough credit. In my opinion, they found a spot that they had room for, and if people can make it with their schedules, then great. If not, hopefully a future slot can work. Typical constraints such as scheduling and resources notwithstanding, 100% participation is not often guaranteed in most things. It's about planning for as many as can make it. With some luck, it would be the majority. Mark.
Re: N91 Women mixer on Sunday?
On 3/28/24 19:45, Ilissa Miller wrote: For those that know me, I rarely provide constructive input about NANOG matters due to my past affiliation, however, I just saw that NANOG announced the Women mixer on Sunday before NANOG 91 and am outraged for all of the young professional women who would like to participate in NANOG. While the times are changing, women continue to remain primary caregivers for families and this will require them to desert their families a day early. I find it offensive personally and feel like you may have missed the mark. Minds are hard to read, so asking the question before being offended is not an unproductive endeavour. That said, we are here now. The amount of times I hear people complain about having to leave their families is one of the reasons this industry has a problem keeping young people - especially women. Does anyone else feel the same? If you have a better suggestion on what would work best, I'd be keen to hear it. Mark.
Re: Best TAC Services from Equipment Vendors
Ultimately, I think every vendor will have customers with good experiences, and customers with not-so-good experiences. Without sufficient data, it is hard to say which vendor, in general, does a good job for their customer base re: TAC. What I will say, albeit anecdotally also, is that TAC experience with the vendor will often be good if you have access to an SE on a regular basis (i.e., one or two in your country of abode). It will even be better if that SE is naturally diligent. Which means that oftentimes, it will come down to one person, to give you a sense of how well a large organization like an OEM treats your business. And if that one person were to leave and move on, the gorilla that is the OEM could come crashing down on you in ways you didn't think was possible. So yes, while it is possible to institutionalize processes and such, like with everything else in life, your experience is very likely to come down to one or two people that care deeply enough to (want to) make a difference. For me, if I have ever have to have a relationship with a vendor, it is a priority that I establish a close relationship with someone on their sales and engineering team, as that will determine how much support I get if I ever run into trouble. And OEM's that were once great at this can quickly become the worst you've ever seen, if those two people were no longer there or stopped caring. In other words, there are no forever-guarantees, and it comes and goes in cycles. Mark.
Re: XGS-PON & "Dedicated" Service
On 10/25/23 01:56, Neader, Brent wrote: Hello! Interested in getting the larger community’s thought on this. The primary question being does XGS-PON have a place in providing a dedicated enterprise level service (at least sold as one) in the marketplace? Delivered via a residential (per the data sheet description) CPE, Nokia XS-010X-Q for a 1gb/1gb dedicated symmetrical service. Background, ive dealt with 30+ providers over the last 18 years, primarily last mile based. Typically we seek out an Enterprise/Dedicated service, with an SLA, typically delivered via DWDM, CWDM, or AE, or equivalent. We have also had a site or two delivered via a PON variant, typically with less of an SLA, typically maybe half to quarter of the price of a dedicated service. Price & SLA sets the expectation of the service, CPE provided, underlying technology, etc. Dealing with a large over-builder right now who has an “elite” enterprise product (highest of 3 tiers) advertised as the following. -100% dedicated bandwidth so you never have to compete for speed -Mission Critical Reliability with 99.999% guaranteed uptime -Financially backed SLA with the most stringent performance objectives -Enterprise-level customer service and technical support Now I understand with XGS, you can have various QOS in place (WRR/SP, etc), but inherently there are still shared splits involved, that just aren’t a thing in other truly dedicated technologies. Expectations were set with the provider’s sales team around what was to be delivered and how it was to be delivered that seemingly haven’t been met by the product and service team. That aside, from an SP perspective, is it capable to wrap enough layers around service to be “dedicated” even when delivered via a conflicting underlying technology? Or could that be considered disingenuous for those that want to know and understand the difference? Im hoping the service itself and support team make up for the difference, but obviously a little concerned. Regular GPON is already being used to deliver Enterprise services, purely because it "passes by the office complex" on its way to the residential neighborhood. Even when the Sales team are told not to use GPON for Enterprise services, they end up doing so... first as a "temporary, we have told the customer all the pitfalls" solution, which eventually becomes permanent, and then it grows like wildfire. You can expect that XG-PON will go the same way. Mark.
Re: RPKI unknown for superprefixes of existing ROA ?
On 10/21/23 16:03, Amir Herzberg wrote: Hi Owen, Randy, Job and other NANOGers, I surely agree with you all that we shouldn't expect discarding of ROA-unknown `anytime soon' (or ever?). But I have a question: what about discarding ROA-unknowns for very large prefixes (say, /12), or for superprefixes of prefixes with announced ROAs? Or at least, for superprefixes of prefixes with ROA to AS 0? For motivation, consider the `superprefix hijack attack'. Operator has prefix 1.2.4/22, but announce only 1.2.5/24 and 1.2.6/24, with appropriate ROAs. To avoid abuse of 1.2.4/24 and 1.2.7/24, they also make a ROA for 1.2.4/22 with AS 0. Attacker now announces 1.2.0/20, and uses IPs in 1.2.4/24 and 1.2.7/24 to send spam etc.. We introduced this threat and analyzed it in our ROV++ paper, btw (NDSS'21 I think - available online too of course). So: would it be conceivable that operators will block such 1.2.0/20 - since it's too large a prefix without ROA and in particular includes sub-prefixes with ROA, esp. ROA to AS 0? The question is - who gets to decide how much space is "too large"? "Too large" will most certainly be different for different networks. If we try to get the RPKI to do things other than for which it was intended which may be interpreted as "unreasonable control", we make the case for those who think that is what it was destined to become. Mark.
Re: Low to Mid Range DWDM Platforms
On 10/19/23 22:24, Chad Lamb via NANOG wrote: XKL is still here ... Some caveats on 400G ZR/ZR+ devices: - They are power hungry, cooling is a challenge. 800G even more so. The OSFP package for 800G looks like the better choice than QSFP-DD for this reason. We'll see, we need more mileage on this. - For the ZR devices, not all vendors are equal. Some support 4x100GE as well as 400GE, if that matters to you. Some have integrated BERT and encryption, some do not. - Do not plan on inter-operating different vendors of ZR+ devices. They will interoperate only in ZR mode. - Plenty of filter options out there if you are putting the ZR/ZR+ in your router. I'll throw a plug in for the Coherent free space DWDM filter. Free space technology has the best IL and isolation numbers. Coherent hasn't packaged it in a module for easy racking yet (XKL can provide that). So if you aren't adding an EDFA, the lowest IL you can get helps your reach. - If you have at least a few 400G channels, add an EDFA and use the Tx = -10dBm devices, rather than the Tx = 0dBm devices. This is cost effective. The EDFA also extends the reach considerably. You can launch each channel up to +10dBm or more, for a modest number of channels, without getting in trouble with fiber non-linearities (four-wave mixing). Many thanks, Chad. Glad to hear XKL are still in business, and doing well :-). Mark.
Re: MX204 tunnel services BW
On 10/17/23 03:20, Ryan Kozak wrote: "The MX204 router supports two inline tunnels - one per PIC. To configure the tunnel interfaces, include the tunnel-services statement and an optional bandwidth of 1 Gbps through 200 Gbps at the \[edit chassis fpc fpc-slot pic number\] hierarchy level. If you do not specify the tunnel bandwidth then, the tunnel interface can have a maximum bandwidth of up to 200 Gbps." If JTAC is saying it's no longer optional they need to update their docs. We can commit "tunnel-services" on an MX204 without caveat. Mark.
Re: MX204 tunnel services BW
On 10/16/23 21:49, Jeff Behrns via NANOG wrote: Also leaving tunnel-services bandwidth unspecified is not possible on the 204. This is true of other MX platforms as well, unless I misunderstand. Mark.
APRICOT 2024: Call for Presentations
APRICOT 2024 21st February - 1st March, Bangkok, Thailand https://2024.apricot.net CALL FOR PRESENTATIONS = The APRICOT 2024 Programme Committee is now seeking contributions for Presentations and Tutorials for the APRICOT 2024 Conference. We are looking for presenters who would: - Offer a technical tutorial on an appropriate topic; - Participate in the technical conference sessions as a speaker; - Convene and chair panel sessions of relevant topics; - Lead informal Birds of Feather break out sessions. Please submit on-line at: https://papers.apricot.net/user/login.php?event=186 CONFERENCE MILESTONES -- Call for Papers Opens: Now Outline Programme Published: As Papers Confirmed Final Deadline for Submissions: 29th January 2024 Final Program Published: 5th February 2024 Final Slides Received: 19th February 2024 *SLOTS ARE FILLED ON A FIRST COME, FIRST SERVED BASIS, REGARDLESS OF PUBLISHED DEADLINES* PROGRAMME CONTENT - The APRICOT Conference Programme consists of three parts, these being Tutorials, the Peering Forum, and Conference Sessions. Topics proposed must be relevant to Internet Operations and Technologies, for example: - IPv4 / IPv6 Routing and Operations - Routing Security, including RPKI and MANRS - Internet backbone operations - Peering, Interconnects and IXPs - Content Distribution Network technology & operations - Research on Internet Operations and Deployment - Network Virtualisation - Network Automation/Programming - Network Infrastructure security - IPv6 deployment on fixed and Wireless/Cellular networks - DNS / DNSSEC and KINDNS - Access and Transport Technologies - Technical application of Web 3.0, public blockchains and cryptocurrency - Content & Service Delivery and "Cloud Computing" - 400G ZR, ZR+ & Open ROADM - Submarine cables - ChatGPT CfP SUBMISSION - Draft slides for both tutorials and conference sessions MUST be provided with CfP submissions otherwise the submission will be rejected immediately. For work in progress, the most current information available at the time of submission is acceptable. All draft and complete slides must be submitted in PDF format only. Slides must be of original work, with all company confidential marks removed. Any marketing, sales and vendor proprietary content in a presentation is against the spirit of APRICOT and it is strictly prohibited. Note that proper credit must be given for all content taken from other sources. Evidence of plagiarism is grounds for rejection. Final slides are to be provided by the specified deadline for publication on the APRICOT website. Prospective presenters should note that the majority of speaking slots will be filled well before the final submission deadline. The PC may, at their discretion, retain a limited number of slots up to the final submission deadline for presentations that are exceptionally timely, important, or of critical operational importance. Every year we turn away submissions, due to filling up all available programme slots before the deadline. Presenters should endeavour to get material to the PC sooner rather than later. Any questions or concerns should be addressed to the Programme Committee Chairs by e-mail at: pc-chairs at apricot.net We look forward to receiving your presentation proposals. Mark Tinka, Achie Atienza & Mark Duffell APRICOT 2024 Programme Committee Chairs --
Re: FastNetMon Usage in the wild
On 10/11/23 04:34, Dobbins, Roland via NANOG wrote: To clarify, Sightline has supported virtualization for many years, FYI. It does do, yes. But pricing for the software license is not too far off from if you chose to buy Netscout's own hardware. Not a major drama for me - I appreciate that competence has to be compensated. What I am saying is that attempts to make it more palatable to more operators are not making too much of a dent. Mark.
Re: Low to Mid Range DWDM Platforms
Finally, the name came to me :-): https://www.xkl.com/ Looks like they are still in business. Mark. On 10/6/23 16:02, Mark Tinka wrote: On 10/6/23 15:52, Dave Bell wrote: Smartoptics? https://smartoptics.com/ Not them. I want to say they had an "X" in their name, but memory is fuzzy. Mark.
Re: maximum ipv4 bgp prefix length of /24 ?
On 10/7/23 14:32, Willy Manga wrote: How about we educate each other to not assume you must deaggregate your prefix especially with IPv6? I see 'some' (it's highly relative) networks on IPv4, they 'believe' they have to advertise every single /24 they have. And when they start with IPv6, they replicate the same mindset with a tons of /48 . You can imagine what will happen of course. A better alternative IMHO is to take advantage to the large prefix range and advertise a sub-aggregate when necessary. But absolutely not each end-node or customer prefix. There are a number of operational reasons folk de-aggregate. I do agree that there is some behaviour around de-aggregating by default in IPv4 that transferred to IPv6. But the main issue is that most people only consider the state of their own FIB situation. They hardly consider the FIB state of other network operators around the world. As an operator, you have to consciously decide that you will not de-aggregate any of your allocations. Of course, there is a cost to that as well, so that cannot be ignored. We, for example, do not de-aggregate any of our allocations (AS37100), but we can afford to do so because we always ensure all peering and transit exit/entry points have the same bandwidth (TE being the main reason networks de-aggregate). Not all shops can afford that. Network operations workshops abound where we teach about de-aggregation, when it can be necessary, and why it should generally be avoided unless in the most extreme of circumstances. However, in real life, even engineers that have been through the workshop ringer tend to prefer to de-aggregate without caution to the FIB state of other autonomous systems. That said, I do agree that, perhaps, network operations workshops could emphasize the reluctance to unnecessarily de-aggregate in light of the increasing cost of having to maintain FIB's. Mark.
Re: Low to Mid Range DWDM Platforms
On 10/6/23 15:53, Dave Cohen wrote: My experience is primarily with the traditional carrier-grade folks like Ciena, Infinera, etc. but over the last decade all of those vendors have focused on improving how they scale down without sacrificing (most of the) quality and functionality - to varying degrees of success. There are also some more recent entrants that built their products to target the DCI market but rather than focusing on bandwidth density have focused on cost per bit for a mid-range solution. There are almost definitely multiple quality options out there without having to buy the full 88 channel n-degree ROADM Ciena 6500 that takes up a full rack - although given the stated requirements, there may not be a one-size-fits-all solution that's ideal for all of the OP's projects. There are two markets driving the major vendors right now - DCI and subsea. The regular longhaul terrestrial gear is mostly the same, bar for upgrades to the latest CMOS. For these vendors, the same gear is also re-used for their subsea solutions, although with "submarine" versions of their terrestrial line cards. Adva's TeraFlex is a bit different in that it is optimized both for DCI and longhaul terrestrial. We are currently looking at it for a 700km deployment in one of our markets. Very impressive! Another entrant into the market is Acacia, now known as the Cisco NCS1000. They also use the same platform for both terrestrial and subsea: https://www.cisco.com/c/en/us/products/optical-networking/network-convergence-system-1000-series/index.html Mark.
Re: Low to Mid Range DWDM Platforms
On 10/6/23 15:52, Dave Bell wrote: Smartoptics? https://smartoptics.com/ Not them. I want to say they had an "X" in their name, but memory is fuzzy. Mark.
Re: Low to Mid Range DWDM Platforms
On 10/6/23 15:07, Mike Hammett wrote: I've been using various forms of passive WDM for years. I have a couple different projects on my plate that require me to look at the next level of platform. In some projects, I'll be looking for needing to have someone long distances of glass without any electronics. Some spans could be over 60 miles. In some projects, I'll need to transport multiple 100-gig waves. What is the landscape like between basic passive and something like a 30 terabit Ciena? I know of multiple vendors in that space, but I like to learn more about what features I need and what features I don't need from somewhere other than the vendor's mouth. Obviously, the most reliability at the least cost as well. 400G-ZR pluggables will get you 400Gbps on a p2p dark fibre over 80km - 100km. So your main cost there will be routers that will support. The smallest DCI solution from the leading DWDM vendors is likely to be your cheapest option. Alternatively, if you are willing to look at the open market, you can find gear based on older CMOS (40nm, for example), which will now be EoL for any large scale optical network, but cost next to nothing for a start-up with considerable capacity value. There is a DWDM vendor that showed up on the scene back in 2008 or thereabouts. They were selling a very cheap, 1U box that had a different approach to DWDM from other vendors at the time. I, for the life of me, cannot remember their name - but I do know that Randy introduced them to me back then. Maybe he can remember :-). Not sure if they are still in business. Mark.
APRICOT 2024: Call for Programme Committee Volunteers
Hi everyone. The APRICOT 2024 Organising Committee would like to welcome everyone to join us in Bangkok, Thailand, from 21st February to 1st March 2024. The APRICOT 2024 Programme Committee is responsible for the solicitation and selection of suitable presentation and tutorial content for the APRICOT 2024 conference (https://2024.apricot.net/). We are now seeking nominations from the community to join the APRICOT 2024 PC to assist with the development of the programme for APRICOT 2024. Eligible PC candidates must have attended recent APRICOT events, have broad technical knowledge of Internet operations, and have good familiarity with the format of APRICOT. Having constructive opinions and ideas about how the programme content might be improved is of high value too. PC members are expected to work actively to solicit content and review submissions for technical merit. The PC meets by conference call, weekly in frequency during the three months prior to APRICOT. If you are interested in joining the PC and meet the above eligibility criteria, please send a brief note to "pc-chairs at apricot.net". The note must include affiliation (if any) and a brief description about why you would make a good addition to the PC. The PC Chairs will accept nominations received by 17:00 UTC+8 on Friday 20th October 2023, and will announce the new PC shortly thereafter. Many thanks! Mark Tinka & Mark Duffell For the APRICOT 2024 Organising Committee --
Re: maximum ipv4 bgp prefix length of /24 ?
On 10/5/23 08:32, Geoff Huston wrote: Not really. The stability of number in IPv4 as compared to the monotonic rise in IPv6 is what I find to be curious. I think the fact that RIR's allocate very large IPv6 address space to their members may well be what is driving this. Historically, network operators do enjoy de-aggregating address space. One can get significantly more /48's out of a /32 (or shorter) than they would /24's out of any IPv4 allocation they acquired. One way to control this escalation is to shorten the maximum prefix length that all operators are willing to accept into the DFZ. The other way would be to get RIR's to lengthen the minimum allocation they can make to their members. Neither of these solutions is likely to be popular, but nor is having to pay large sums of money for more FIB. Mark.
Re: maximum ipv4 bgp prefix length of /24 ?
On 10/5/23 08:24, Geoff Huston wrote: The IPv6 FIB is under the same pressure from more specifics. Its taken 20 years to get there, but the IPv6 FIB is now looking stable at 60% opf the total FIB size [2]. For me, thats a very surprising outcome in an essentially unmanaged system. Were you expecting it to be lower than IPv4? Mark.
Re: maximum ipv4 bgp prefix length of /24 ?
On 10/5/23 07:49, Crist Clark wrote: But if the assumption is that networks will always eventually totally deaggregate to the maximum, we're screwed. Routing IPv4 /32s would be nothing. The current practice of accepting /48s could swell to about 2^(48 - 3) = 2^45 = 35184372088832. What will prevent unrestricted growth of the IPv6 table if operators push everything out to /48 "to counter hijacks" or other misguided reasons? Expecting that the Internet would ever operate at maximum de-aggregation is unrealistic. There are too many checkpoints to make that a feasible possibility. It's not an outcome I would spend any brain cycles on, unless as an academic exercise. Mark.
Re: maximum ipv4 bgp prefix length of /24 ?
On 10/4/23 09:27, Elmar K. Bins wrote: Justin, I'm not sure you're not confusing scope here. Everybody and their sister accept smaller blocks from their customers; we're all talking about the DFZ here, not customer routes that you aggregate. Actually, we don't. From our customers, the most we are accepting today is a /24 and a /48. This is for transit customers with their own AS and address space. Of course, if it's a DIA customer, we can assign longer prefixes, but that is internal address space that belongs to us. Mark.
Re: maximum ipv4 bgp prefix length of /24 ?
On 10/4/23 12:11, Musa Stephen Honlue wrote: Which one is easier, 1. Convincing the tens of thousands of network operators and equipment vendors to modify configs and code to accept more specifics than /24, or Equipment vendors can already support 10 million entries in FIB. They just ask for a little bit of cash for it. Mark.
Re: cogent spamming directly from ARIN records?
On 10/2/23 22:59, Matthew Petach wrote: Huh? In all my decades of time in the network industry, I have never seen a case where a smaller transit contract had lower per mbit cost than a larger volume contract. I would expect that HE would make *more* money off 10 smaller customer transit contracts than one big tier 3 wholesaler transit contract. That's my point. Smaller ISP's will get better per-Mbps rates from their direct upstreams than they would from HE/Cogent. The rates they'd get from HE/Cogent would be to HE's/Cogen's favour, and not to the ISP's favour. So if the goal is transit-free glory vanity, HE/Cogent would have to take a bit of a haircut which they "could" make up in contract length. Just another way to skin the cat. Mark.
Re: maximum ipv4 bgp prefix length of /24 ?
On 10/2/23 20:44, Tim Franklin wrote: Had NOT considered the looping - that's what you get for writing in public without thinking it all the way through *blush*. Thanks for poking holes appropriately, Like I said, it's going to be a messy experiment - for probably a decade, at least. As Saku has highlighted as well, vendors are likely to find a lot of sludge in this experiment that they will never be able to share with us... likely because it will be too complicated for us to understand in a coherent way, or likely because the experiment changes so rapidly it makes no sense to tell us about issues which will quickly become obsolete. Many lessons will be learned, but ultimately, one would be naive to think this black box will just work. All I want is a "set routing-options fib-compression disable" present for Christmas. Mark.
Re: cogent spamming directly from ARIN records?
On 10/2/23 20:58, Tim Burke wrote: Hurricane has been doing the same thing lately... but their schtick is to say that "we are seeing a significant amount of hops in your AS path and wanted to know if you are open to resolve this issue". I get what HE are trying to do here, as I am sure all of us do. The potential fallout is a declining relationship with their existing customers that bring other downstream ISP's behind them. Contacting those downstream ISP's to "resolve this issue" puts them at odds with their existing customers who bring those customers in already. There is a chance they dilute their income because, well, smaller ISP's will not be keen to pay the higher transit fees their upstreams pay to HE. Which means that HE are more willing to be closer to eyeballs than they are maximizing margins. Is the loss of customer trust worth the transit-free glory? Mark.
Re: maximum ipv4 bgp prefix length of /24 ?
On 9/30/23 01:36, William Herrin wrote: If I were designing the product, I'd size the SRAM with that in mind. I'd also keep two full copies of the FIB in the outer DRAM so that the PPEs could locklessly access the active one while the standby one gets updated with changes from the RIB. But I'd design the router to gracefully fail if the FIB exceeded what the SRAM could hold. When a TCAM fills, the shortest prefixes are ejected to the router's main CPU. That fails pretty hard since the shortest prefixes tend to be among the most commonly used. By comparison, an SRAM cache tends to retain the most commonly used prefixes as an inherent part of how caches work, regardless of prefix length. It can operate close to full speed until the actively used routes no longer fit in the cache. Well, not sure if you're aware, but starting Junos 21.2, Juniper are implementing FIB Compression on the PTX routers running Express 4 and Junos EVO. We have some of these boxes in our network (PTX10001), and I have asked Juniper to provide a knob to allow us to turn it off, as it is currently going to ship as a default-on feature. I'd rather not be part of the potential mess that is going to come with the experimentation of that over the next decade. Mark.
Re: maximum ipv4 bgp prefix length of /24 ?
On 9/29/23 22:56, William Herrin wrote: Actually, BGP can swing that. Routing involves two distinct components: the routing information base (RIB) and the forwarding information base (FIB). BGP is part of the RIB portion of that process. It's always implemented in software (no hardware acceleration). It's not consulted per-packet, so as long as the update rate is slow enough for the CPU to keep up and there's enough DRAM (which is cheap and plentiful these days) to hold the entire thing, there's no particular upper bound to the number of routes. Not unless you are running BGP Add-Paths in its extreme guise to consider all available paths, and then there is the chance you could push your RAM and CPU into unknown territory, depending on the platform, code and control plane you've got. The limiting factor is the FIB. The FIB is what is implemented with hardware acceleration on high-end routers in order to achieve large packet-per-second (PPS) numbers. It relies on storage hardware which is both faster and more expensive than DRAM. Consequently it has much less capacity to store information than DRAM. Currently shipping equipment intended for BGP backbone use can manage 1M to 2M routes in the hardware-accelerated FIB regardless of the amount of DRAM on the machine. There are line cards out there today that are able to hold 10 million routes in FIB. Mark.
Re: maximum ipv4 bgp prefix length of /24 ?
On 9/29/23 06:43, VOLKAN SALİH wrote: But when everybody upgrades, memory and processor unit prices decrease.. Vendors gain from demand. I am yet to see that trend... Mark.
Re: maximum ipv4 bgp prefix length of /24 ?
On 9/28/23 23:25, VOLKAN SALİH wrote: hello, I believe, ISPs should also allow ipv4 prefixes with length between /25-/27 instead of limiting maximum length to /24.. I also believe that RIRs and LIRs should allocate /27s which has 32 IPv4 address. considering IPv4 world is now mostly NAT'ed, 32 IPv4s are sufficient for most of the small and medium sized organizations and also home office workers like youtubers, and professional gamers and webmasters! It is because BGP research and experiment networks can not get /24 due to high IPv4 prices, but they have to get an IPv4 prefix to learn BGP in IPv4 world. What do you think about this? What could be done here? Is it unacceptable; considering most big networks that do full-table-routing also use multi-core routers with lots of RAM? those would probably handle /27s and while small networks mostly use default routing, it should be reasonable to allow /25-/27? RAM is not the issue... it's FIB. If you pay me for FIB slots, I'm happy to install /32's to your heart's content :-). Mark.
Re: TACACS+ server recommendations?
On 9/20/23 17:39, Jeff Moore wrote: We have also used https://www.shrubbery.net/tac_plus/ for some time as well. Great product! Yes, that's one of the ones in the FreeBSD ports. Works very well. Mark.
Re: TACACS+ server recommendations?
On 9/20/23 17:09, Bryan Holloway wrote: Ah, the good old days when I could download the latest tac_plus code from the Cisco FTP site, compile it, and off I go. But I digress. Curious if there are any operators out there that have a good recommendation on a lightweight TACACS+ server for ~200 NEs and access-control for 20-30 folks. Nothing too special, but some sort of simple command-control auth would be nice. Open-source is fine ... we've been looking at commercial options, but they're all pretty pricey and have way more features than we need. Thank you all in advance! [root@host /home/tinka]# cd /usr/ports/net/tac tac_plus4/ tacacs/ [root@host /home/tinka]# They work just fine. These are on FreeBSD, but I am sure they will compile on Linux too. Mark.
Re: Zayo woes
On 9/19/23 18:05, Mike Hammett wrote: Some of it is scale-related. Someone's operating just fine at the size they are, but the next order of magnitude larger enjoys many benefits from that size, but it takes either A) luck or B) the right skills to be able to move up to get those benefits. In terms of network operators, there's a big difference between a company of 1 and a company of 2. Again a big difference between a company of 2 and a company of 10. Another big difference between a company of 10 and a company of.. I dunno, 100? 1G waves and 100G waves aren't *THAT* different in price anymore. However, if 1 is doing you just fine, the much better cost per bit of 100 won't do you a darn bit of good and will probably hurt. The hurt is not only due to the higher MRC, but now the higher NRC for equipment with 100G interfaces. If you can put enough bits on the line to make it worth it, you've got yourself tremendous advantage. The acquisition pays for itself in marginally higher costs for much higher revenue. The company may have been doing just fine before, but couldn't afford the move up to 10 or 100 because their present market couldn't justify it and the larger market wasn't obtainable until they had it. Catch 22 that can't really be overcome by most. Enter M Someone just can't get to the next level they need to compete with those that can. I'm talking about companies of relatively equal size and influence. It's all benefit for smaller companies, even if it may be at the cost of the larger one. Mark.
Re: Zayo woes
On 9/19/23 17:54, Mike Hammett wrote: Well sure, and I would like to think (probably mistakenly) that just no one important enough (to the money people) made the money people that these other things are *REQUIRED* to make the deal work. Obviously, people lower on the ladder say it all of the time, but the important enough money people probably don't consider those people important enough to listen to. The ISP world is not a normal market place :-). Mark.
Re: Zayo woes
On 9/19/23 17:40, Anne Mitchell wrote: And sometimes the acquisition is really just about acquiring the assets, such as the customer list*, and then they are left with having to run something that they never really wanted until they can figure out what to do with it. Right, buying the revenue to prop up the top and bottom line is also a reason to acquire. Usually, this is based on assessed profitability, but what tends to go unseen during the due diligence process is what is actually contributing to that profitability. I mean, typically, if a company is doing very well, it won't try to get itself sold. Well, not easily anyway... Mark.
Re: Zayo woes
On 9/19/23 16:48, Mike Hammett wrote: As someone that has been planning to be in the acquiring seat for a while (but yet to do one), I've consistently passed to the money people that there's the purchase price and then there's the % on top of that for equipment, contractors, etc. to integrate, improve, optimize future cashflow, etc. those acquisitions with the rest of what we have. I blame this on the success of how well we have built the Internet with whatever box and tool we have, as network engineers. The money people assume that all routers are the same, all vendors are the same, all software is the same, and all features are easily deployable. And that all that is possible if you can simply do a better job finding the cheapest box compared to your competition. In general, I don't fancy nuance when designing for the majority. But with acquisition and integration, nuance is critical, and nuance quickly shows that the acquisition was either underestimated, or not worth doing at all. Mark.
Re: Zayo woes
On 9/19/23 16:41, Matthew Petach wrote: c) your executives have promised there will be cost savings after the merger due to "synergies" between the two companies. Couldn't have said the exact same words any better myself - "cost savings from synergies" :-). Always interesting when a year later, the projected savings from synergies are nowhere near reality, and all consolidation analysis work has completed :-). Mark.
Re: Zayo woes
On 9/19/23 14:19, Mike Hammett wrote: I've never understood companies that acquire and don't completely integrate as quickly as they can. Sometimes it does not add any material value to either network. Sometimes it takes too long. If the acquired company is orders of magnitude smaller than the acquiring one, it's probably best to keep them separate. Of course, I am referring to network integration. Staff and business systems integration should always be the goal. Mark.
Re: Lossy cogent p2p experiences?
On 9/9/23 22:29, Dave Cohen wrote: At a previous $dayjob at a Tier 1, we would only support LAG for a customer L2/3 service if the ports were on the same card. The response we gave if customers pushed back was "we don't consider LAG a form of circuit protection, so we're not going to consider physical resiliency in the design", which was true, because we didn't, but it was beside the point. The real reason was that getting our switching/routing platform to actually run traffic symmetrically across a LAG, which most end users considered expected behavior in a LAG, required a reconfiguration of the default hash, which effectively meant that [switching/routing vendor]'s TAC wouldn't help when something invariably went wrong. So it wasn't that it wouldn't work (my recollection at least is that everything ran fine in lab environments) but we didn't trust the hardware vendor support. We've had the odd bug here and there with LAG's for things like VRRP, BFD, e.t.c. But we have not run into that specific issue before on ASR1000's, ASR9000's, CRS-X's and MX. 98% of our network is Juniper nowadays, but even when we ran Cisco and had LAG's across multiple line cards, we didn't see this problem. The only hashing issue we had with LAG's is when we tried to carry Layer 2 traffic across them in the core. But this was just a limitation of the CRS-X, and happened also on member links of a LAG that shared the same line card. Mark.
Re: Lossy cogent p2p experiences?
On 9/9/23 20:44, Randy Bush wrote: i am going to be foolish and comment, as i have not seen this raised if i am running a lag, i can not resist adding a bit of resilience by having it spread across line cards. surprise! line cards from vendor do not have uniform hashing or rotating algorithms. We spread all our LAG's across multiple line cards wherever possible (wherever possible = chassis-based hardware). I am not intimately aware of any hashing concerns for LAG's that traverse multiple line cards in the same chassis. Mark.
Re: Lossy cogent p2p experiences?
On 9/7/23 09:51, Saku Ytti wrote: Perhaps if congestion control used latency or FEC instead of loss, we could tolerate reordering while not underperforming under loss, but I'm sure in decades following that decision we'd learn new ways how we don't understand any of this. Isn't this partly what ECN was meant for? It's so old I barely remember what it was meant to solve :-). Mark.
Re: Lossy cogent p2p experiences?
On 9/7/23 09:31, Benny Lyne Amorsen wrote: Unfortunately that is not strict round-robin load balancing. Oh? What is it then, if it's not spraying successive packets across member links? I do not know about any equipment that offers actual round-robin load-balancing. Cisco had both per-destination and per-packet. Is that not it in the networking world? Juniper's solution will cause way too much packet reordering for TCP to handle. I am arguing that strict round-robin load balancing will function better than hash-based in a lot of real-world scenarios. Ummh, no, it won't. If it did, it would have been widespread. But it's not. Mark.
Re: Lossy cogent p2p experiences?
On 9/6/23 18:52, Tom Beecher wrote: Well, not exactly the same thing. (But it's my mistake, I was referring to L3 balancing, not L2 interface stuff.) Fair enough. load-balance per-packet will cause massive reordering, because it's random spray , caring about nothing except equal loading of the members. It's a last resort option that will cause tons of reordering. (And they call that out quite clearly in docs.) If you don't care about reordering it's great. load-balance adaptive generally did a decent enough job last time I used it much. Yep, pretty much my experience too. stateful was hit or miss ; sometimes it tested amazing, other times not so much. But it wasn't a primary requirement so I never dove into why Never tried stateful. Moving 802.1Q trunk from N x 10Gbps LAG's to native 100Gbps links resolved this load balancing conundrum for us. Of course, it works well because we spread these router<=>switch links across several 100Gbps ports, so no single trunk is ever that busy, even for customers buying N x 10Gbps services. Mark.
Re: Lossy cogent p2p experiences?
On 9/6/23 12:01, Saku Ytti wrote: Fun fact about the real world, devices do not internally guarantee order. That is, even if you have identical latency links, 0 congestion, order is not guaranteed between packet1 coming from interfaceI1 and packet2 coming from interfaceI2, which packet first goes to interfaceE1 is unspecified. This is because packets inside lookup engine can be sprayed to multiple lookup engines, and order is lost even for packets coming from interface1 exclusively, however after the lookup the order is restored for _flow_, it is not restored between flows, so packets coming from interface1 with random ports won't be same order going out from interface2. So order is only restored inside a single lookup complex (interfaces are not guaranteed to be in the same complex) and only for actual flows. Yes, this has been my understanding of, specifically, Juniper's forwarding complex. Packets are chopped into near-same-size cells, sprayed across all available fabric links by the PFE logic, given a sequence number, and protocol engines ensure oversubscription is managed by a request-grant mechanism between PFE's. I'm not sure what mechanisms other vendors implement, but certainly OoO cells in the Juniper forwarding complex is not a concern within the same internal system itself. Mark.
Re: Lossy cogent p2p experiences?
On 9/6/23 11:20, Benny Lyne Amorsen wrote: TCP looks quite different in 2023 than it did in 1998. It should handle packet reordering quite gracefully; in the best case the NIC will reassemble the out-of-order TCP packets into a 64k packet and the OS will never even know they were reordered. Unfortunately current equipment does not seem to offer per-packet load balancing, so we cannot test how well it works. I ran per-packet load balancing on a Juniper LAG between 2015 - 2016. Let's just say I won't be doing that again. It balanced beautifully, but OoO packets made customers' lives impossible. So we went back to adaptive load balancing. It is possible that per-packet load balancing will work a lot better today than it did in 1998, especially if the equipment does buffering before load balancing and the links happen to be fairly short and not very diverse. Switching back to per-packet would solve quite a lot of problems, including elephant flows and bad hashing. I would love to hear about recent studies. 2016 is not 1998, and certainly not 2023... but I've not heard about any improvements in Internet-based applications being better at handling OoO packets. Open to new info. 100Gbps ports has given us some breathing room, as have larger buffers on Arista switches to move bandwidth management down to the user-facing port and not the upsteam router. Clever Trio + Express chips have also enabled reasonably even traffic distribution with per-flow load balancing. We shall revisit the per-flow vs. per-packet problem when 100Gbps starts to become as rampant as 10Gbps did. Mark.
Re: Lossy cogent p2p experiences?
On 9/6/23 17:27, Tom Beecher wrote: At least on MX, what Juniper calls 'per-packet' is really 'per-flow'. Unless you specifically configure true "per-packet" on your LAG: set interfaces ae2 aggregated-ether-options load-balance per-packet I ran per-packet on a Juniper LAG 10 years ago. It produced 100% perfect traffic distribution. But the reordering was insane, and the applications could not tolerate it. If you applications can tolerate reordering, per-packet is fine. In the public Internet space, it seems we aren't there yet. Mark.
Re: Lossy cogent p2p experiences?
On 9/6/23 16:14, Saku Ytti wrote: For example Juniper offers true per-packet, I think mostly used in high performance computing. Cisco did it too with CEF supporting "ip load-sharing per-packet" at the interface level. I am not sure this is still supported on modern code/boxes. Mark.
Re: Lossy cogent p2p experiences?
On 9/6/23 09:12, Masataka Ohta wrote: you now recognize that per-flow load balancing is not a very good idea. You keep moving the goal posts. Stay on-topic. I was asking you to clarify your post as to whether you were speaking of per-flow or per-packet load balancing. You did not do that, but I did not return to that question because your subsequent posts inferred that you were talking to per-packet load balancing. And just because I said per-flow load balancing has been the gold standard for the last 25 years, does not mean it is the best solution. It just means it is the gold standard. I recognize what happens in the real world, not in the lab or text books. Mark.
Re: Lossy cogent p2p experiences?
On 9/4/23 13:27, Nick Hilliard wrote: this is an excellent example of what we're not talking about in this thread. It is amusing how he tried to pivot the discussion. Nobody was talking about how lane transport in optical modules works. Mark.
Re: Lossy cogent p2p experiences?
On 9/4/23 13:04, Masataka Ohta wrote: Are you saying you thought a 100G Ethernet link actually consisting of 4 parallel 25G links, which is an example of "equal speed multi parallel point to point links", were relying on hashing? No... you are saying that. Mark.
Re: Lossy cogent p2p experiences?
On 9/3/23 15:01, Masataka Ohta wrote: Why, do you think, you can rely on existence of flows? You have not quite answered my question - but I will assume you are in favour of per-packet load balancing. I have deployed per-packet load balancing before, ironically, trying to deal with large EoMPLS flows in a LAG more than a decade ago. I won't be doing that again... OoO packets is nasty at scale. And nothing beyond, of course. No serious operator polices in the core. ECMP, surely, is a too abstract concept to properly manage/operate simple situations with equal speed multi parallel point to point links. I must have been doing something wrong for the last 25 years. Mark.
Re: Lossy cogent p2p experiences?
On 9/3/23 15:01, Masataka Ohta wrote: Why, do you think, you can rely on existence of flows? You have not quite answered my question - but I will assume you are in favour of per-packet load balancing. I have deployed per-packet load balancing before, ironically, trying to deal with large EoMPLS flows in a LAG more than a decade ago. I won't be doing that again... OoO packets is nasty at scale. And nothing beyond, of course. No serious operators polices in the core. ECMP, surely, is a too abstract concept to properly manage/operate simple situations with equal speed multi parallel point to point links. I must have been doing something wrong for the last 25 years. Mark.
Re: Lossy cogent p2p experiences?
On 9/3/23 09:59, Masataka Ohta wrote: If you have multiple parallel links over which many slow TCP connections are running, which should be your assumption, the proper thing to do is to use the links with round robin fashion without hashing. Without buffer bloat, packet reordering probability within each TCP connection is negligible. So you mean, what... per-packet load balancing, in lieu of per-flow load balancing? So, if you internally have 10 parallel 1G circuits expecting perfect hashing over them, it is not "non-rate-limited 10gig". It is understood in the operator space that "rate limiting" generally refers to policing at the edge/access. The core is always abstracted, and that is just capacity planning and management by the operator. Mark.
Re: Lossy cogent p2p experiences?
On 9/2/23 17:38, Masataka Ohta wrote: Wrong. It can be performed only at the edges by policing total incoming traffic without detecting flows. I am not talking about policing in the core, I am talking about detection in the core. Policing at the edge is pretty standard. You can police a 50Gbps EoMPLS flow coming in from a customer port in the edge. If you've got N x 10Gbps links in the core and the core is unable to detect that flow in depth to hash it across all those 10Gbps links, you can end up putting all or a good chunk of that 50Gbps of EoMPLS traffic into a single 10Gbps link in the core, despite all other 10Gbps links having ample capacity available. There is no such algorithms because, as I wrote: : 100 50Mbps flows are as harmful as 1 5Gbps flow. Do you operate a large scale IP/MPLS network? Because I do, and I know what I see with the equipment we deploy. You are welcome to deny it all you want, however. Not much I can do about that. Mark.
Re: Lossy cogent p2p experiences?
On 9/2/23 17:04, Masataka Ohta wrote: Both of you are totally wrong, because the proper thing to do here is to police, if *ANY*, based on total traffic without detecting any flow. I don't think it's as much an issue of flow detection as it is the core's ability to balance the Layer 2 payload across multiple links effectively. At our shop, we understand the limitations of trying to carry large EoMPLS flows across an IP/MPLS network that is, primarily, built to carry IP traffic. While some vendors have implemented adaptive load balancing algorithms on decent (if not custom) silicon that can balance EoMPLS flows as well as they can IP flows, it is hit & miss depending on the code, hardware, vendor, e.t.c. In our case, our ability to load balance EoMPLS flows as well as we do IP flows has improved since we moved to the PTX1000/10001 for our core routers. But even then, we will not sell anything above 40Gbps as an EoMPLS service. Once it gets there, time for EoDWDM. At least, until 800Gbps or 1Tbps Ethernet ports become both technically viable and commercially feasible. For as long as core links are based on 100Gbps and 400Gbps ports, optical carriage for 40Gbps and above is more sensible than EoMPLS. Mark.
Re: Lossy cogent p2p experiences?
On 9/2/23 08:43, Saku Ytti wrote: What in particular are you missing? As I explained, PTX/MX both allow for example speculating on transit pseudowires having CW on them. Which is non-default and requires 'zero-control-word'. You should be looking at 'hash-key' on PTX and 'enhanced-hash-key' on MX. You don't appear to have a single stanza configured, but I do wonder what you wanted to configure when you noticed the missing ability to do so. Sorry for the confusion - let me provide some background context since we deployed the PTX ages ago (and core nodes are typically boring). The issue we ran into was to do with our deployment tooling, which was based on 'enhanced-hash-key' that is required for MPC's on the MX. The tooling used to deploy the PTX was largely built on what we use to deploy the MX, with tweaks of critically different items. At the time, we did not know that the PTX required 'hash-key' as opposed to 'enhanced-hash-key'. So nothing got deployed on the PTX specifically for load balancing (we might have assumed it to have been non-existent or incomplete feature at the time). So the "surprise" I speak of is how well it all worked with load balancing across LAG's and EoMPLS traffic compared to the CRS-X, despite not having any load balancing features explicitly configured, which is still the case today. It works, so we aren't keen to break it. Mark.
Re: Lossy cogent p2p experiences?
On 9/1/23 21:52, Mike Hammett wrote: It doesn't help the OP at all, but this is why (thus far, anyway), I overwhelmingly prefer wavelength transport to anything switched. Can't have over-subscription or congestion issues on a wavelength. Large IP/MPLS operators insist on optical transport for their own backbone, but are more than willing to sell packet for transport. I find this amusing :-). I submit that customers who can't afford large links (1Gbps or below) are forced into EoMPLS transport due to cost. Other customers are also forced into EoMPLS transport because there is no other option for long haul transport in their city other than a provider who can only offer EoMPLS. There is a struggling trend from some medium sized operators looking to turn an optical network into a packet network, i.e., they will ask for a 100Gbps EoDWDM port, but only seek to pay for a 25Gbps service. The large port is to allow them to scale in the future without too much hassle, but they want to pay for the bandwidth they use, which is hard to limit anyway if it's a proper EoDWDM channel. I am swatting such requests away because you tie up a full 100Gbps channel on the line side for the majority of hardware that does pure EoDWDM, which is a contradiction to the reason a packet network makes sense for sub-rate services. Mark.
Re: Lossy cogent p2p experiences?
On 9/1/23 15:55, Saku Ytti wrote:
Personally I would recommend turning off LSR payload heuristics,
because there is no accurate way for LSR to tell what the label is
carrying, and wrong guess while rare will be extremely hard to root
cause, because you will never hear it, because the person suffering
from it is too many hops away from problem being in your horizon.
I strongly believe edge imposing entropy or fat is the right way to
give LSR hashing hints.
PTX1000/10001 (Express) offers no real configurable options for load
balancing the same way MX (Trio) does. This is what took us by surprise.
This is all we have on our PTX:
tinka@router# show forwarding-options
family inet6 {
route-accounting;
}
load-balance-label-capability;
[edit]
tinka@router#
Mark.
Re: Lossy cogent p2p experiences?
On 9/1/23 15:59, Mike Hammett wrote: I wouldn't call 50 megabit/s an elephant flow Fair point. Mark.
Re: Lossy cogent p2p experiences?
On 9/1/23 15:44, Mike Hammett wrote: and I would say the OP wasn't even about elephant flows, just about a network that can't deliver anything acceptable. Unless Cogent are not trying to accept (and by extension, may not be able to guarantee) large Ethernet flows because they can't balance them across their various core links, end-to-end... Pure conjecture... Mark.
Re: Lossy cogent p2p experiences?
On 9/1/23 15:29, Saku Ytti wrote: PTX and MX as LSR look inside pseudowire to see if it's IP (dangerous guess to make for LSR), CSR/ASR9k does not. So PTX and MX LSR will balance your pseudowire even without FAT. Yes, this was our conclusion as well after moving our core to PTX1000/10001. Mark.
Re: Lossy cogent p2p experiences?
On 9/1/23 10:50, Saku Ytti wrote: It is a very plausible theory, and everyone has this problem to a lesser or greater degree. There was a time when edge interfaces were much lower capacity than backbone interfaces, but I don't think that time will ever come back. So this problem is systemic. Luckily there is quite a reasonable solution to the problem, called 'adaptive load balancing', where software monitors balancing, and biases the hash_result => egress_interface tables to improve balancing when dealing with elephant flows. We didn't have much success with FAT when the PE was an MX480 and the P a CRS-X (FP40 + FP140 line cards). This was regardless of whether the core links were native IP/MPLS or 802.1AX. When we switched our P devices to PTX1000 and PTX10001, we've had surprisingly good performance of all manner of traffic across native IP/MPLS and 802.1AX links, even without explicitly configuring FAT for EoMPLS traffic. Of course, our policy is to never transport EoMPLS servics in excess of 40Gbps. Once a customer requires 41Gbps of EoMPLS service or more, we move them to EoDWDM. Cheaper and more scalable that way. It does help that we operate both a Transport and IP/MPLS network, but I understand this may not be the case for most networks. Mark.
Re: Destination Preference Attribute for BGP
On 8/30/23 18:56, michael brooks - ESC wrote: I, too, am looking for something sexy (explained below). But can you explain why you think AS_PATH is "useless," Mark? Because most network operators use LOCAL_PREF heavily, and no amount of AS_PATH prepending will be able fight that with any reasonable success. You can still achieve some success with AS_PATH prepending, but with the way content is getting distributed around the world, it is becoming as useful as running a Squid cache yourself these days. For background, and the reason I asked about DPA: Currently, our routing carries user traffic to a single data center where it egresses to the Internet via three ISP circuits, two carriers. We are peering on a single switch stack, so we let L2 "load balance" user flows for us. We have now brought up another ISP circuit in a second data center, and are attempting to influence traffic to return the same path as it egressed our network. Simply, we now have two datacenters which user traffic can egress, and if one is used we want that traffic to return to the same data center. It is a problem of asymmetry. It appears the only tools we have are AS_Path and MED, and so I have been searching for another solution, that is when I came across DPA. In further looking at the problem, BGP Communities also seems to be a possible solution, but as the thread has explored, communities may/may not be scrubbed upstream. So, presently we are looking for a solution which can be used with our direct peers. Obviously, if someone has a better solution, I am all ears. When you have multiple transit providers, you are really implicitly accepting that forwarding asymmetry is now part of your life. You will have full control of your outbound forwarding, but return forwarding will be a coin toss. You can guarantee this, to some degree, if you also have lots of peering, as most peers will prefer to return traffic to you via peering and not via their transit providers. But even this is not always a sure thing, especially in situations where a network you are peering with is doing Remote Peering. Ultimately, the level of influence you have on the remote network's forwarding decision, especially if you are not peering, is slim. Our solution to this has been to focus on the "typically large" transit providers, announce routes consistently to them, and ensure we have the same port capacity to each one. Even with those attributes, we can't get perfect load sharing, because we provide customers the ability to decide which transit providers (and exchange points) they want their routes to transit, or not. So the only routing we can consistently guarantee is our own routes. We can't guarantee customers will let their routes exit all transit or peering points, so we just make sure we have ample capacity across all such relationships. I understand that not all networks may have the ability to do this for financial reasons, but if you can only afford to have one or two transit providers, your best bet is to leverage the existing BGP tools as best you can, even though the results will not be perfect. Mark.
Re: MX204 Virtual Chassis Setup
On 8/28/23 07:55, Daniel Marks wrote: (Enterprise AS for context) This hasn’t been my experience in the US, however we mostly deal in tier 2 markets (I.e. Detroit, Miami, Dallas, etc…) and we have plenty of 40G private interconnects. I don’t doubt 40G is going away, I’ve just never had trouble using it around here. This would be expected, since most 40Gbps hardware I have seen in Europe and Africa is in the enterprise and exchange point space. If service providers have them, I'd imagine that inventory is far lower than 100Gbps. Mark.
Re: MX204 Virtual Chassis Setup
On 8/28/23 03:05, Mike Hammett wrote: Well, or they simply found a potential deal on hardware that came with 40 gig ports. 40 gigs is still a lot of bits to a lot of people. For internal use, sure. But when connecting to another AS, the chances of them supporting 40Gbps in one or more places is inconsistent to slim. Exchange points may be an exception. Mark.
Re: MX204 Virtual Chassis Setup
On 8/27/23 04:52, Eric Kuhnke wrote: I sincerely doubt there is much demand for *new* 40G these days. Look at the population of 40G members on major IXes. People have either one 10G, 2 x 10G, or 100G. 40G was a dead-end 9 years ago and much so more now. We have customers that sometimes ask for 40Gbps interconnects. I always tell our Pre-Sales team that those are the ones who "led the way", back in the day. Sadly, they were a bit too early :-). Mark.
