Re: IPv6 and CDN's

[Mark Tinka]

On 11/27/21 02:41, Michael Thomas wrote:

Amazon's in this case. They are monetizing their lack of v6 support 
requiring you go through all kinds of expensive hoops instead of doing 
the obvious and routing v6 packets.




Individual CDN's and content providers have better control over how they 
deploy IPv6, vs. ISP's who have far less capital, warm bodies and 
innovation DNA.


I'm arguing for the latter.

Mark.

Re: IPv6 and CDN's

[Mark Tinka]

On 11/26/21 16:16, Jose Luis Rodriguez wrote:


Well … YMMV. We’ve been running v6 for years, and it didn’t really make a dent 
in spend or boxes or rate of v4 depletion. Big part of the problem in our neck 
of the woods is millions of v4-only terminals … as well as large customer/gov 
bids requiring tons of v4 address space.


I can very easily see why "IPv6 saves you on CG-NAT capex might not be 
entirely true" in cases such as these.


On paper, it all adds up.

Mark.

Re: IPv6 and CDN's

[Mark Tinka]

On 11/26/21 15:47, Frank Habicht wrote:



"want to buy 5 of those shiny new CGNAT boxes or only 2 ?"


To which she will respond, "2 or 5, what do I make :-)?"

Mark.

Re: IPv6 and CDN's

[Mark Tinka]

On 11/26/21 15:00, Jean St-Laurent wrote:


With a kicking ass pitch


Can I take your CFO :-)...

Mark.

Re: IPv6 and CDN's

[Mark Tinka]

On 11/3/21 22:13, Max Tulyev wrote:

Implementing IPv6 reduces costs for CGNAT. You will have (twice?) less 
traffic flow through CGNAT, so cheaper hardware and less IPv4 address 
space. Isn't it?


How to express that in numbers CFO can take to the bank?

Mark.

Re: Validating multi-path in production?

[Mark Tinka]

On 11/12/21 23:47, Jeff Tantsura wrote:


LAG - Micro BFD (RFC7130) provides per constituent livability.


Not sure if this has changed, but the last time I looked into it, Micro 
BFD's for LAG's was only supported and functional on point-to-point 
Ethernet links.


In cases where you are running a LAN, it did not apply.

We gave up running BFD on LAG's on LAN's, because of this issue.

Mark.

RPKI-Based Policy Without Route Refresh

[Mark Tinka]

Randy will be presenting draft-ymbk-sidrops-rov-no-rr during RIPE-83, at 
around 1530hrs UTC:


https://datatracker.ietf.org/doc/html/draft-ymbk-sidrops-rov-no-rr-02

Most grateful if you can join, and provide some initial feedback. Thanks.

Mark.

Re: Redploying most of 127/8 as unicast public

[Mark Tinka]

During the nation-wide lockdown of 2020, around the world, I took up 
live-streaming my DJ sets online, since I couldn't play live. For those 
that haven't seen them, you're welcome to my Youtube channel to catch them:


https://yt.djmt.africa/watch

Anyway, what I wanted to say is that I was streaming using an iPhone app 
that turns the iPhone and iPad into a digital video capture device.


As per the links below, if this lowly, rather obscure app from an 
unknown Canadian iOS developer supports IPv6 as a way to setup a remote 
connection between the laptop (encoder) and iPhone/iPad, in order to 
make camera adjustments so you don't have to physically walk toward the 
device, what is your excuse :-)?


https://ibb.co/C8kbpPc
https://ibb.co/yYPWbvd

Mark.

Re: Redploying most of 127/8 as unicast public

[Mark Tinka]

On 11/18/21 01:29, Jay R. Ashworth wrote:


This seems like a really bad idea to me; am I really the only one who noticed?

https://www.ietf.org/id/draft-schoen-intarea-unicast-127-00.html

That's over a week old and I don't see 3000 comments on it, so maybe it's just
me.  So many things are just me.


There is a better chance of writing a successful RFC to clean up "cisco, 
cisco" for user/pass on Google.


Mark.

Fwd: [APRICOT-PC-Chairs] APRICOT 2022 Call for Presentations

[Mark Tinka]

FYI.

Mark.

 Forwarded Message 
Subject:[APRICOT-PC-Chairs] APRICOT 2022 Call for Presentations
Date:   Thu, 7 Oct 2021 21:16:43 +1000
From:   Philip Smith 
Reply-To:   APRICOT PC Chairs 
Organization:   Asia Pacific Regional Conference on Operational Technologies
To: ap...@apops.net
CC: APRICOT PC Chairs 



Hi everyone,

Please find below the call for presentations for APRICOT 2022! :-)

Best wishes,

philip
--

APRICOT 2022
21st February - 3rd March, Online
https://2022.apricot.net

CALL FOR PRESENTATIONS
==

The APRICOT 2022 Programme Committee is now seeking contributions for
Presentations and Tutorials for the APRICOT 2022 Conference.

We are looking for presenters who would:

- Offer a technical tutorial on an appropriate topic;
- Participate in the technical conference sessions as a speaker;
- Convene and chair panel sessions of relevant topics;

Please submit on-line at:

http://papers.apricot.net/user/login.php?event=142

Tutorials will take place during the week of 21st February.
Conference sessions and the Peering Forum will take place on the week
of 28th February. All APRICOT 2022 session times will be one hour
long, with up to four sessions scheduled per day.

CONFERENCE MILESTONES
-

Call for Papers Opens: Now
Initial Draft Programme: 6th December 2021
Final Deadline for Submissions: 31st January 2022
Final Program Published: 7th February 2022
Final Slides Received: 14th February 2022

*SLOTS ARE FILLED ON A FIRST COME, FIRST SERVED BASIS, REGARDLESS OF
PUBLISHED DEADLINES*

PROGRAMME CONTENT
-

The APRICOT Conference Programme consists of three parts, these being
Tutorials, the Peering Forum, and Conference Sessions.

Topics proposed must be relevant to Internet Operations and Technologies:

- IPv4 / IPv6 Routing and Operations
- Internet backbone operations
- Routing Security, including RPKI and MANRS
- Peering, Interconnects and IXPs
- Network Function Virtualisaton
- Network Automation/Programming
- Content Distribution Network technology & operations
- Research on Internet Operations and Deployment
- Network infrastructure security
- IPv6 deployment on fixed and Wireless/Cellular networks
- DNS / DNSSEC
- Access and Transport Technologies
- Content & Service Delivery and "Cloud Computing"


PEERING FORUM
-

Due to APRICOT 2022 being held online, the PC will only accept Peering
Personals prior to the event starting. Submissions must be of a
single slide listing the operator's PeeringDB entry. Please refer to
https://tinyurl.com/y46954n5 for how to create a successful Peering
Personal presentation.


CfP SUBMISSION
--

Draft slides for both tutorials and conference sessions MUST be
provided with CfP submissions otherwise the submission will be
rejected immediately. For work in progress, the most current
information available at time of submission is acceptable.

All draft and complete slides must be submitted in PDF format only.
Slides must be of original work, preferably not presented before at
other venues, with all company confidential marks removed.

Final slides are to be provided by the specified deadline for
publication on the APRICOT website.

Prospective presenters should note that the majority of speaking slots
will be filled well before the final submission deadline. The PC may,
at their discretion, retain a limited number of slots up to the final
submission deadline for presentations that are exceptionally timely,
important, or of critical operational importance. Every year we turn
away submissions, due to filling up all available programme slots
before the deadline. Presenters should endeavour to get material to
the PC sooner rather than later.

For any questions or concerns, please email the PC Chairs.

We look forward to receiving your presentation proposals.

Mark Tinka, Tugsorshikh Badarch & Marijana Novakovic
Co-Chairs, APRICOT 2022 Programme Committee
--

--
To unsubscribe from this group and stop receiving emails from it, send an email 
topc-chairs+unsubscr...@apricot.net.

Re: FORT monitoring/visibility

[Mark Tinka]

On 10/27/21 01:58, Randy Bush wrote:

i run a FORT RPKI relying party instance.  i am looking for some
visibility into its operation.

   is it up: both ways, fetching and serving routers?

   from what CAs has it pulled, how recently and frequently with
   what success?

   what routers is it serving with rpki-rtr 323?

   blah blah blah

my old DRL RP instances produce MRTG graphs etc of the CA
fetching side, though nothing on the rpki-rtr side.


Randy, I actually have an ongoing discussion with the Fort developers 
about this after a BGPSec bug left me with stale VRP's for several days, 
with no clear indication that Fort had "kind of" crashed and "not fully" 
crashed (fair point, I need to work on better internal monitoring of 
Fort, as well).


Will feedback once I have better info.

For now, if you haven't yet done so, recommend upgrading to 1.5.2 to 
avoid this specific issue.


The good news is this issue made the case for running different 
validation RP code, so your NLRI does not share fate, given it's the 
basis of the Internet.


Mark.

Re: ipv4 on mobile networks

[Mark Tinka]

On 10/25/21 18:12, Masataka Ohta wrote:



So are IP entities behind NAT. So?


I still don't understand your point.

Are you asserting that NAT'ed devices do not have an IP address?

Mark.

Re: ipv4 on mobile networks

[Mark Tinka]

On 10/25/21 08:29, Lady Benjamin Cannon of Glencoe, ASCE wrote:


I’m typing this on an LTE UE on our network with a NAT’d IPv4 IP address.

Feels relevant.


We may be missing each other here...

From the point of view of TCP/IP, a node behind a CGN has a unique IP 
address.


So what I'm trying to understand is, despite whether a connection is 
pure or NAT'ed, how does a device on the Internet expect to communicate 
without an IP address?


Mark.

Re: ipv4 on mobile networks

[Mark Tinka]

On 10/25/21 01:35, Masataka Ohta wrote:



So are IP entities behind NAT. So?


Your point being...?

Mark.

Re: ipv4 on mobile networks

[Mark Tinka]

On 10/24/21 19:19, Ca By wrote:

Nodes in an ip network require ip addresses. 4G and 5G are ip networks 
for both voice and data.


I do not believe it is either technically nor economically feasible to 
run a 4g or 5g network without an ip address on the ue.


*shaking_my_head*

For the avoidance of doubt, Cameron is speaking sense.

Mark.

Re: ipv4 on mobile networks

[Mark Tinka]

On 10/24/21 18:25, Masataka Ohta wrote:



Are you saying mobile terminals must be identified by IP addresses?


Ummh, how else do you expect a player on the Internet to, ummh, play?

Mark.

Re: IPv6 and CDN's

[Mark Tinka]

On 10/22/21 18:08, t...@pelican.org wrote:


I don't think it'll ever make money, but I think it will reduce costs.  CGNAT 
boxes cost money, operating them costs money, dealing with the support fallout 
from them costs money.  Especially in the residential space, where essentially 
if the customer calls you, ever, you just blew years' worth of margin.


The problem is accurately modelling cost reduction using native IPv6 in 
lieu of CG-NAT is hard when the folk that need convincing are the CFO's.


They are more used to "spend 1 to get 2". Convincing them to "save 2 by 
spending 1" - not as easy as one may think.


Mark.

Re: IPv6 and CDN's

[Mark Tinka]

On 10/22/21 17:45, Bryan Fields wrote:



Until IPv6 becomes provides a way to make money for the ISP, I don't see it
being offered outside of the datacenter.


It is being offered, it's just not being adopted.

We deliver an IPv6 /126 p2p and /56 or /48 onward assignment to all our 
DIA customers. No interest.


We deliver an IPv6 /125 p2p and eBGP session to all our IP Transit 
customers. 5/10 are interested.


You can only do what you can only do.

Mark.

Re: Providing IPv4 Services in an IPv6 Backbone

[Mark Tinka]

On 10/22/21 15:19, Jason Iannone wrote:

Thanks for sharing. Maybe I have blinders on, but LDPv6 and the v6 SR 
flavors don't have much use if v4 CE sites aren't supported.


Indeed. If your goal is an IPv6-only network with IPv6-only services, 
then Nokia may have an answer for you.


But if you want IPv4-as-a-Service over an IPv6-only network (4PE or 
4VPE), then I'd recommend beating up the vendors.


I'd start with Nokia, since by all accounts, they seem to be leading the 
charge at this point.


Mark.

Re: IPv6 and CDN's

[Mark Tinka]

On 10/22/21 14:03, Jens Link wrote:


I don't think it was overlooked or forgotten. More along

"We have always done it this way", "We had problems enabling IPv6 (ages
ago)" or something else you can find on https://ipv6excuses.com/.


I think it's a combination of both... they tried back in the day, it 
broke, and they "parked" it for later.


When Marketing and The World were happy to see that 
www.insert-favorite-content-url-here.com had  and IPv6 PTR records, 
who cared whether boring, little-known FQDN's were remembered or not.


And then, all the engineers moved on to some other gig, where they did 
better because, why not?


Mark.

Fort - Now Available as a FreeBSD Port

[Mark Tinka]

For those who run FreeBSD, the Fort RPKI validator is now available in 
the ports tree:


https://www.freshports.org/net/fort/

Many thanks to Toni Kalombo for submitting and maintaining the port, and 
to Philip Paeps to committing it.


I've also sent a note to the Fort developers to update the installation 
information for FreeBSD.


Mark.

Re: Network visibility

[Mark Tinka]

On 10/21/21 22:50, Lady Benjamin Cannon of Glencoe, ASCE wrote:


Outside the datacenter is where DC power really shines in my opinion.  Inside 
the DC, everything is AC now and probably for the best.

We never came up with a modular standard for -48VDC. Perhaps that could have 
changed things.

But it sure is nice having 72hrs of battery run time in the field/edge - 
although those are becoming mini data centers themselves and are in turn also 
slowly going AC.


I suppose it depends what business you are in.

If you are a mobile operator and have towers in all sorts of places 
where utility mains may be unavailable or spotty, I guess having to 
convert DC to AC makes little sense if that's your primary source of 
power (especially since solar PV, wind turbines and batteries all output 
DC power anyway).


We run a fair bit of Metro-E network in the countries we operate, and 
most of that is in standard commercial buildings that are not data 
centres. Even there, we run AC, with a UPS, and rely on the building 
generator for an alternate AC source in case of a mains outage.


But yes, for our terrestrial Transport network, that's all DC anyway, 
regardless of the power source.


Mark.

Re: Providing IPv4 Services in an IPv6 Backbone

[Mark Tinka]

On 10/21/21 21:18, Jason Iannone wrote:



Hi all,

Have there been any gap closures on RFC7439? I am particularly 
interested in 4PE, 4VPE, and other MPLS enabled services like L3VPN, 
NG-MVPN, E-Line, E-LAN, and EVPN. Does Juniper have an 
"ipv4-tunneling" mpls keyword?


I posted this here earlier this month:

https://mailman.nanog.org/pipermail/nanog/2021-October/215609.html

Unaware of any other vendor who claims to have solved this problem.

Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/21/21 20:28, Fred Baker wrote:

I’m not sure I disagree, but let throw in a point of consideration. 
Historically, as you note, the caller pays the toll. However, the 
caller also CHOSE to call, even though the called party might find the 
call irritating. With a CDN, the network is out there hoping to be 
called, and the user makes that choice, “calling” the CDN. Could it 
not be accurately said that the user originated the “call”?


We can't seem to dump our telco ways, can we :-)?

Mark.

Re: Network visibility

[Mark Tinka]

On 10/21/21 14:54, Brian Johnson wrote:


There is still zoning on some platforms, but there are now redundancies for the 
zones.


Sounds complex. But to each their own.

Mark.

Re: Network visibility

[Mark Tinka]

On 10/21/21 03:19, Brian Johnson wrote:


+1 on -48VDC.


Wasn't much fun when half the router would shutdown because power 
supplies failed, due to what was known as "power zoning" those days.


I haven't deployed a larger router on DC in over 13 years. I'm not sure 
if this is still a thing, even.


Mark.

Re: Network visibility

[Mark Tinka]

On 10/20/21 20:37, Lady Benjamin Cannon of Glencoe, ASCE wrote:


-48VDC power is still the best.


I really envy folk that love DC for networking gear :-).

Work in 2007 was an all-DC network. I rebuilt it into AC, considering 
the ISP also owned the data centre (most of whose customers bought AC). 
The space we freed up and the ease of deployment was night & day.


Currently, we obviously need DC for the terrestrial Transport and wet 
plants (because that's just how classic telco rolls), but I also 
switched all IP/MPLS gear to AC soon as I arrived. Heck, even the Arbor 
(now Netscout) gear, as well as the HP server rack, was loaded with DC 
power supplies. Those things just had to go.


There is an avenue of pleasure in not having to spend inordinate amounts 
of time adding major electrical planning to deploying/decommissioning a 
router, switch or server.


But yeah, I know the AC vs. DC discussion can become a rat hole.

I'm aware of data centre operators now providing DC as an option for 
their expansion projects, when they previously had it as the norm, FWIW.


Mark.

Re: Network visibility

[Mark Tinka]

On 10/20/21 19:32, Mel Beckman wrote:


such tinkaing...


Cute...


  is rare. It certainly doesn’t rise to the level of “never works out of the 
box.”


Luck you.

Mark.

Re: Network visibility

[Mark Tinka]

On 10/20/21 18:38, Mel Beckman wrote:

I’ve used many commercial NMS platforms. I’ve yet to find one that doesn’t work 
“out of the box”. Unless by “out of the box” you mean “clairvoyantly 
configured”.

Please identify the ones you think fail your test.


Have you always used an NMS that you've never had to have the vendor (or 
community) tweak in a manner that was mostly unique to your operation?


If not, you're a very lucky man...

Mark.

Re: Network visibility

[Mark Tinka]

On 10/20/21 18:08, Mel Beckman wrote:


Mark,

Before 1983, the ARPANET wasn’t an internet, let alone The Internet. 
Each ARPANET connection required a host-specific interface (the “IMP”) 
and simplex Network Control Protocol (NCP). NCP used users' 
email addresses, and routing had to be specified in advance within 
each NCP message.


I do know all of this, mate... I was just being dramatically facetious 
from my first response to the OP.


My point being that considering how long TCP/IP has been around, the 
best monitoring we have gotten, even today, doesn't work out-of-the-box. 
So a single solution is likely impractical, even with the best of 
intentions, and none of the massaging.



Even so, the Internet as a platform open to anyone didn’t start until 
1992. I know you joined late, in 1999, so you probably missed out on 
this history. :)


1995, actually. But that's not important...

Mark.

Re: Network visibility

[Mark Tinka]

On 10/20/21 17:26, Mel Beckman wrote:


Mark,

As long as we’re being pedantic, January 1, 1983 is considered the 
official birthday of the Internet, when TCP/IP first let different 
kinds of computers on different networks talk to each other.


It’s 2021, hence the Internet is /less/ than, not more than, 40 years 
old.  Given your mathematical skills, I put no stock in your claim 
that we still can’t “buy an NMS that just works.” :)


Hehehe :-)...

I guess we can reliably say that the ARPANET wasn't keen on pretty 
pictures, then, hehe :-)...


Mark.

Re: Network visibility

[Mark Tinka]

On 10/20/21 11:55, Nat Fogarty wrote:


Hi there,

I'm interested in what you good folks do in terms of network visibility.

My interests are around Service Provider space - visibility for IPoE, 
PPPoE, TCP(User Experience).


I use a product called "VoIPmonitor" for all things VoIP - and it is 
one of my favourite tools.  It is a web gui for sip/rtp/etc.


Is there a similar tool in the Ethernet(L2)/IP(L3) space?

Are operators using tcpdump/wireshark for this - or is there a 
voipmonitor-esque tool out there?


It's 2021, and more than 40 years of the Internet, we still can't walk 
into a shop and buy an NMS that just works :-).


Oddly, I was searching for a good system to manage subscriber management 
on our end (Broadband), and we eventually landed on Splynx.


So not sure if you want to see things on the wire (Layer 1 - 4), or if 
you are interested in pretty pictures...


At any rate, you may very well need more than one system to monitor your 
entire network.


Mark.

Re: DNS pulling BGP routes?

[Mark Tinka]

On 10/18/21 14:16, Masataka Ohta wrote:


As copper and optical fiber for access politically belongs to ITU,
DSL and optical fiber standards of ITU are followed by the IETF
world.


Yes, but nobody cares about Layer 1 or Layer 2.

Once the road is built, all anyone remembers is the car I drove across 
it, not whether the tar used to build the road was mixed well :-).





I actually joined an ITU meeting at Geneva, when I was actively
acting for DSL in Japan.


Good for you.

Look, I'm not saying the ITU are bad - I am saying that they are "more 
structured and rigid", than Internet-land. And that is okay. There is a 
reason we TCP/IP became dominant.




FYI, IS-IS is part of OSI, which was jointly developed by ISO and ITU,
not by IETF at all.


You might be forgetting that the IETF adapted IS-IS to IP networks:

    https://datatracker.ietf.org/doc/html/rfc1195

I'm not sure anyone running IS-IS in an ISP environment, today, is 
running it for CLNS.


But we thank the ISO, immensely :-).



Are you agreeing with me that they are earning a lot more than
they should?


I have zero interest in being the profit police. Who am I tell anyone 
that they are earning too much?


If you make something people find value in, the billions will 
automatically flow your way - you can't stop it. Is it a perfect system, 
probably not, but it's what we've got.





Access networks are subject to regional monopoly unless unbundling
is forced by regulatory bodies. Worse, with PON, such unbundling is
hard (not impossible, see https://ieeexplore.ieee.org/document/5616389).


Submarine cables are usually either owned by one party, or a small club. 
It's no different - and trying to be a member of the club can be just as 
demoralizing as local regulation on terrestrial builds.


That said, different markets have different policies on access networks. 
So a single policy for what we think is best is not practical. Moreover, 
if access networks are expensive due to backward regulation and 
monopolistic promotion, then that is an artificial problem that can be 
removed, but the actors choose not to. You can't blame a content 
operator for that market position.





So, you are a neo-liberalist. Good luck.


I also like the one where whole gubbermints shutdown the Internet for 
elections, or to hush voices.  I discriminate equally :-).






Though precise definition of "tier 1" is a rat hole, that
there are entities called tier 1, which are the primary
elements of the Internet backbone, is a common concept
shared by most of us, maybe excluding you.


I know many here that have moved on from the "tier" terminologies. But 
it's unnecessary for them to chime in.


There hasn't been "a core of the Internet" for a long while, and anyone 
still believing that either in reality or words is living in a fantasy 
world long gone, which is partially why infrastructure finds itself 
becoming less and less relevant, and being swallowed up by BigContent.


I mean, if you missed the fact that Facebook went down, and people 
thought the Internet had stopped, then maybe Facebook are a Tier 1...


Mark.

Re: DNS pulling BGP routes?

[Mark Tinka]

On 10/18/21 10:11, Masataka Ohta wrote:



As you are seemingly requesting international legal formality,
let me point out there are "International Telecommunication
Regulations", based on which network neutrality is discussed
by ITU.


And since when does the IETF world follow the ITU standards?

Even though ITU heads don't think much of IETF heads, you can't find an 
SDH or DWDM port in a laptop. On the other hand, GMPLS is based on OSPF, 
IS-IS and RSVP-TE :-).





No, of course. So?


Well, I'll be asking my bank to sell me some IP Transit or DIA, then, 
since they are running an IP network.




That cost is negligible compared to the cost to prepare
access network all over the continent, I'm afraid.


It may be, it may not be. The reason is only one or a small handful of 
folk are investing US$700 million into a submarine cable.


On the other hand, access networks are built by several operators, all 
competing. So no single operator building an access network is spending 
more than the content folk laying pipe in the Atlantic and Indian oceans.





The essential difference is whether they are neutral or not.


Well, the issue is you want to label things. I can see this is what is 
causing your confusion.


Willing buyer, willing seller. That's all that's needed. If the seller 
doesn't like the buyer, they move on. If the buyer doesn't like the 
seller, they move on.





Are you saying that there is no such thing as tier 1 ISPs?


Hehe, let's not go down that rat hole.

But no, I don't believe in "tiers" for service provider networks. 
Haven't done so in nearly 15 years.


Heck, my marketing team are always asking if we can identify ourselves 
as "Tier 1" because we own and operate a submarine cable. I'm sure you 
can guess my answer to them...


Personally, I don't care whether you are "transit-free" or not. You 
cannot provide an Internet service to the entire world from just one 
operator (network or content). So trying to be "bigger" than the other 
guy is a pointless exercise. Jane + Thatho don't care about your 
measuring contest.


Mark.

Re: DNS pulling BGP routes?

[Mark Tinka]

On 10/16/21 15:44, Masataka Ohta wrote:



What?


I will use my network for what I want my network to do for me. There are 
no international rules about why a network must be built. Provided that 
I am clear to those whom I want to connect to my network, I can do what 
I want with it and not be a bad actor.




Unless they directly reach their end users, yes, of course.


So by your logic, a bank's internal network used to drive its ATM 
machines is not neutral because one cannot use that network for global 
IP Transit?





The fundamental problem of networking is the last mile problem
that access costs alot more than backbone.


Well, yes and no.

While it is true that one of the biggest problems of the Internet is the 
last mile, it is vital to not be forced into the mistake of 
classification. For some operators, the "last mile" is the biggest cost. 
For other networks, the "backbone" is the biggest cost.


You can't tell me that US$700 million being spent to build a submarine 
cable around a continent is something to scoff at.


For me, I don't want to hold myself back by classifying "access", 
"backbone", "metro", e.t.c. Your business model will determine what is 
costly to you, and what isn't.





As such, long distance carriers may peer with access providers
only when they are neutral or pay some of there revenue share
to access providers.


Again, you are trying to keep the old Internet (and the classic 
telephone company model) alive in 2021. That is not how operators work 
anymore. There are networks that have neither a "backbone" nor an 
"access network" that do very well, and don't cause anyone else pain, 
because they are clear in what their model is.





With your definition, as CDN providers with their own backbone are
not "transit", they can not request access providers (and, ultimately,
end users) peering without paying some as compensation for access
network cost.

Otherwise, CDN providers with their own backbone are free riders
ignoring access costs.


Okay, so by your logic, "access providers" should pay CDN's for peering, 
because the CDN's have spent millions building submarine cables and data 
centres around the world to bring their service to the access providers. 
After all, why give the access providers a free ride either?


In case it's not clear, that last paragraph was sarcastic. It's 2021 - 
long distance, access, backbone, metro, e.t.c. Those are boxes that 
don't exist anymore. Let's not refuse the advancement of the model 
because we can't find a way to make it fit in our old box.


Mark.

Fwd: [members-discuss] Update on legal case - Freeze of Bank accounts

[Mark Tinka]

FYI.

Step by step, the tyrants shall be stripped.

Mark.

 Forwarded Message 
Subject:[members-discuss] Update on legal case - Freeze of Bank accounts
Date:   Fri, 15 Oct 2021 14:53:03 +0400
From:   Eddy Kayihura 
To: AfriNIC Discuss 



Dear Colleagues,

We are happy to inform our stakeholders that there was a court hearing 
today with regard to our application for removal of the freezing order 
against AFRINIC.


The Court, after considering our application and the case initiated by 
Cloud Innovation Ltd, has declared the order null and void. In short, 
AFRINIC has won this case against Cloud Innovation Ltd.


There is another Appeal to be heard on 11thNovember 2021. We will keep 
you informed of the outcome and we are quietly optimistic that justice 
will prevail.


Kind Regards,

Eddy Kayihura
Chief Executive Officer
African Network Information Centre (AFRINIC)
c...@afrinic.net 

…...

Chers Collègues,

Nous sommes heureux de vous informer qu'une audience a eu lieu 
aujourd'hui au tribunal concernant notre demande de levée de 
l'ordonnance de gel des comptes d'AFRINIC.


Le tribunal, après avoir examiné notre demande et l'affaire initiée par 
Cloud Innovation Ltd, a déclaré l'ordonnance nulle et non avenue. En 
bref, AFRINIC a gagné ce procès contre Cloud Innovation Ltd.


Un autre appel doit être entendu le 11 novembre 2021. Nous vous 
tiendrons informés du dénouement et nous sommes optimistes quant à la 
justice.


Cordialement,

Eddy Kayihura
Directeur général
Le Centre d'information du réseau africain
c...@afrinic.net
……...

زملائي الاعزاء،

يسعدنا إبلاغ أصحاب المصلحة لدينا أنه كانت هناك جلسة استماع اليوم فيما 
يتعلق بطلبنا لإزالة أمر التجميد ضد AFRINIC.


أعلنت المحكمة ، بعد النظر في طلبنا والقضية التي رفعتها شركة Cloud 
Innovation Ltd ، أن الأمر باطل ولاغٍ. باختصار ، فازت AFRINIC بقضيتها ضد 
Cloud Innovation Ltd.


هناك نداء آخر سيتم الاستماع إليه في 11 نوفمبر. سنبقيك على اطلاع بالنتيجة 
ونحن متفائلون بهدوء بأن العدالة ستسود.


AFRINIC للاتصالات

.

Caros colegas,

Temos o prazer de informar as nossas partes interessadas de que houve 
hoje uma audiência judicial relativamente ao nosso pedido de retirada da 
ordem de congelamento contra a AFRINIC.


O Tribunal, após considerar o nosso pedido e o processo iniciado pela 
Cloud Innovation Ltd, declarou a ordem nula e sem efeito. Em suma, a 
AFRINIC ganhou o seu processo contra a Cloud Innovation Ltd.


Há outro recurso a ser ouvido a 11 de Novembro de 2021. Manter-vos-emos 
informados sobre o resultado e estamos tranquilamente optimistas de que 
a justiça prevalecerá.


Cumprimentos,

Eddy Kayihura
Chefe do Executivo
O Centro de Informação da Rede Africana
c...@afrinic.net


Vision: “A secure and accessible Internet for sustainable digital growth 
in Africa”
Mission: “To serve the African Internet community by delivering 
efficient services in a global multi-stakeholder environment”

Values: EPIC (■ Excellence ■ Passion ■ Integrity ■ Community Driven)


***
This email and any attachment (s) transmitted with it are 
confidential. They may also be privileged or otherwise protected by law. 
They are intended solely for the use of the individual or entity to 
whom they are addressed. If you have received this email by error 
please notify the sender immediately by e-mail and delete this e-mail 
from your system. You are also notified that disclosing, 
copying, distributing, or taking any action in relation to its contents 
is strictly prohibited and unlawful. By reading the message and 
opening any attachment, the recipient accepts full responsibility for 
taking protective and remedial action about viruses and other defects.


***

___
Members-Discuss mailing list
members-disc...@afrinic.net
https://lists.afrinic.net/mailman/listinfo/members-discuss

Re: Increase bandwidth usage in partial-mesh network?

[Mark Tinka]

On 10/13/21 19:59, Fletcher Kittredge wrote:



Hey! From the description it must be one of our clients!

Just beware if you go this route, a network that is probably already 
unstable and unreliable will become at least an order of magnitude 
worse. You can't fix ten lbs of stuff into a 4 lb stuff bag. The 
internet protocols do not tolerate congestion well.


It sounds like they need to get back to the basics first.

Simplification, in lieu of added complexity, seems to be the appealing 
approach.


Mark.

Re: DNS pulling BGP routes?

[Mark Tinka]

On 10/13/21 17:24, Masataka Ohta wrote:



The problem is that, unlike neutral transit providers, "the bits"
is biased by the CDN provider.

Then, access/retail ISPs who also want to supply their own contents,
even though they must be neutral to contents provided by neutral
transit providers, naturally refuse peering with the anti-neutral
CDN providers.

Remember that CDN providers are not neutral at all.


Well, the purpose of a network is whatever its proprietor deems it to 
be, and makes no false advertising about it.


A private enterprise network that carries a company's internal traffic - 
which may or may not interface with an external network that is 
interested in some or all of that traffic - would, in your eyes, be 
classified as not neutral, because it chooses not to use its network to 
provide global IP Transit?


In my mind, the word "transit" refers to carriage between two 
non-homogeneous points. So network A (customer) will talk to network C 
(content) via my network B (transit). If the traffic originates either 
from A or C, BUT terminates/ends inside of B, I do not consider that 
transit.


I'm unaware of content operators who run their own network and (promise 
to) provide connectivity between A and C.


Mark.

CSR1000v + ASR1000 Code Upgrade Pleasure...

[Mark Tinka]

Hi all.

I thought I'd share our recent experiences, per subject, just in case 
others run into the same problems.


So... we finally decided to try 17.3(4a)MD for the CSR1000v, after years 
of happy operation. Good Lord, what a drama!


At first, we couldn't figure out why iBGP sessions to all Cisco boxes 
could not stand up. Then we realized it's because IS-IS to them could 
not stand up. Then we realized it's because BFD sessions could not stand up.


But even after removing BFD, IS-IS remained down.

After 3 days of searching, we finally landed on CSCuz58508. In case you 
don't have CCO access, it is the same issue as described here:


https://community.cisco.com/t5/cisco-cloud-service-router-csr/b00ocg4q4e-csr-1000v-16-3-1a-can-t-set-mtu-on-gig-interface/td-p/3054853

This was even more confusing for us, because our interface driver on 
VMware ESXi is vmxnet3.


The bug ID suggests the problem is fixed in 16.3(2) and 16.4(1). So to 
be safe, we tested 16.12(5)MD, which allowed us to enable jumbo frames, 
but that only appeared to be a cosmetic thing. In the background, the 
box was simply dropping packets, silently. We found this out when we 
tried to copy other files to the node, and it would just hang without 
any feedback. Removing the jumbo frame support allowed the files to come 
through.


We noticed that nodes still running 3.17(0)S did not have any issues 
with IS-IS or BFD, or MTU. However, this code was only ever released as 
an ED train (and to be fair, we've been having dodgy issues with it in 
recent years), so we decided to downgrade to 3.16(9)S (which is actually 
an upgrade from 3.17(00)S, since the 3.16 train is an MD release, with 
the latest release being March 2019, vs. July 2017 for 3.17(4)SED).


With that, no more MTU issues, BFD and IS-IS are happy, iBGP is happy.

We definitely won't be wasting any more time trying to make Denali, 
Gibraltar, Fuji, Everest or Amsterdam work on our CSR1000v complement.


Needless to say, moving the ASR1000 platform to 17.3 has also come with 
its own avenue of pleasure, what with all the ROMMON, CPLD and FPGA 
upgrade mess that is. What the documentation says and what happens in 
real life are two very different things. It has taken us a week to come 
up with our own working procedure to upgrade just one box, worse if it's 
a dual-RP system.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/12/21 18:33, Sabri Berisha wrote:


Yes, let's go back to 2003. The ISP I worked for at that time was one of
the first in the country (if not the first) to host Akamai's caching servers.

Ten years later I worked on a project where Akamai caching was embedded in
subscriber management routers. It was announced, but never productized. This
concept would have brought caching as close to the subscriber as possible.

Today, with the widespread use of HTTPS, something like this is just not
feasible.


Yes, the utility of Squid and similar local caching servers became less 
helpful as objects got more dynamic. This exacerbated as traffic shifted 
over to tcp/443.


Then the CDN's started shipping content closer and closer to eyeballs, 
and that has generally become the norm over the past decade.


I'm not sure anyone still using Squid & Friends is seeing net gains with 
that model, particularly with a major CDN likely being close by.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/12/21 17:39, Jared Brown wrote:


   Since we aren't talking about random pirated content, but p2p streaming from a 
major content provider it would obviously be point & click.


Yes, in which case Jane + Thatho don't need to worry about device 
compatibility, especially if the device is a major brand like Apple TV, 
an established gaming console or an established smart TV.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/12/21 17:13, Jared Brown wrote:

That's not how it works. Several streaming BitTorrent clients 
specifically request blocks in order so that you can start watching 
immediately.

   Not that you need a special client, it works pretty well with the standard 
client as well on a well seeded torrent, as blocks are generally requested more 
or less in order.


Thanks. I wasn't aware there were torrent clients that streamed straight 
to video. The last time I used torrents, it was to download files for 
later consumption.


I found Usenet feeds more reliable.



Well, yes. Or you could just stream content that is guaranteed to be compatible 
with the device used.


People on this list would bother to check compatibility.

Jane + Thatho just point & click.



They could, and they might even have, I forget, but there is little demand for 
such a thing as a centralized CDN strategy works better.


Which is why I'm not sure it would work, in today's Internet.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 22:57, Matthew Walster wrote:

Ignoring for the moment that P2P is inherently difficult to stream 
with (you're usually downloading chunks in parallel, and with devices 
like Smart TVs etc you don't really have the storage to do so anyway) 
there's also the problem that things like BitTorrent don't know 
network topology and therefore only really increases the 
cross-sectional bandwidth required.


Not to mention that it has been tried before, and didn't work then either.


Yeah, and people also want to click a title and start watching immediately.

Someone can correct me if I'm wrong, but the way I know BitTorrent to 
work is the file is downloaded to disk, unarchived and then listed as 
ready to watch. It also assumes the device has all the necessary apps 
and codecs needed to render the file.


On the other hand, BitTorrent could just make an Apple 
TV/PS4/PS5/Xbox/whatever-device-you-use app as well. But I doubt that 
will work, unless someone can think up a clever way to modify BitTorrent 
to suit today's network architectures.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/12/21 14:20, Jason Iannone wrote:

Isn't this a problem with legacy peering agreements in today's 
internet? The same thing happened between Netflix, Level3, and Verizon 
a few years ago. The legacy concept of settlement-free peering is 
based on traffic forwarding parity. If what I forward to you roughly 
matches what you forward to me, we can agree that we have no reason to 
charge each other for access. The concept works fine when content and 
eyeballs are evenly distributed between providers. This doesn't work 
in today's divergent content and eyeball networks.


If Netflix agreed to settlement-free peering under the legacy 
definition, then as far as the letter of the law goes, Netflix is in 
the wrong.


Indeed.

Traffic ratios to determine peering partners(hips) is, I think, archaic, 
and a cop out for not having to deal with peering requests. There are 
many networks with whom peering would add value, despite not matching 
traffic ratios. As you say, BigContent are such networks.


Specifically for us, geographic network scope is the most important, as 
I don't want to help you build your backbone on the back of mine, for 
free. There may be some volume requirements, but certainly not made on 
the basis of directional ratios.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 22:05, Matthew Petach wrote:



Let's check back in 2026, and see if someone's become fantastically
successful doing this or not.  ;)


I have to say, your idea is quite fantastical. I'm not sure I have 
enough brain cells to consider how it will work, remembering that vCPE's 
were all the rage in 2011, and ten years later, they seem to have 
fizzled out without a real-world deployment of note :-).


At any rate, with the current state-of-the-art, deploying Metro edge 
caches is within the realms of possibility. However, it's such a rich 
solution, that it will only likely ever work in a select group of cities 
around the world.


For the rest of us, a nearby data centre pumping cached content across 
fibre links all the way into homes is as good as we shall get. However, 
what this network looks like in 2021 vs. 2006, perhaps, allows us to 
reconsider the model.


One example that comes to mind is VoD-only ISP's, whose raison d'être is 
to deliver VoD content from local caches, with no infrastructure to 
support access to the global Internet. I'd be keen to build something 
like that, particularly in a world where the traditional infrastructure 
operator is a dying species.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 17:26, Niels Bakker wrote:




I don't think that's being entirely fair. Netflix in plenty places 
differentiates its subscriptions based partly on video resolution: 
https://help.netflix.com/en/node/24926/us


Some people will definitely care enough to sign up for a more 
expensive tier.


In much the same way those that care will opt for the D+ mode of the 
car, and likely more will be happy with just the D model.


Yes, many subscribers are clued up on the different Netflix plans, but 
I'm sure most of them choose the plans not for the video resolution, but 
for the number of active screens that can stream concurrently.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 09:49, Matthew Petach wrote:


Going back to the fact that it's not the content providers "using"
a lot of bandwidth, it's the eyeball customer *requesting* a lot
of bandwidth, I think the best approach is for the content providers
to help manage traffic levels by lowering bit rates towards eyeball
networks that are feeling strained by their users.

Instead of a 4K stream, drop it to 480 or 240; the eyeball network
should be happy at the reduced strain the resulting stream puts
on their network.

The content network can even point out they're being a good
Network Citizen by putting up a brief banner at the top of the
stream saying "reducing bit rate to relieve stress on your ISPs
network".  That way, the happy customer knows that the
content provider is doing their part to help their ISP stay
profitable...I mean, doing their part to help the Internet
run better.


To be fair, Jane + Thatho don't care about video resolution. All they 
will see is that the picture isn't that great, and this creates an 
opportunity for a VoD provider who is "more favorable" toward the 
network operator, and gets granted full 4K resolution transport 
priviledges. If there are any surcharges levied by the network operator, 
the VoD provider compensates for those by attracting more business 
because, well, they can offer a more superior image quality.


I'm not sure about the term, but the "colluding" version for that would 
be - if my few IGF days do not fail me - "net neutrality".


I'm not sure we want to go down that path, either.



The market *is* determining that at the moment...but not in the
direction people expect.  Instead, it's creating a new market for
intermediaries; imagine you're an eyeball network that happens
to have peering with SKB, and largely inbound traffic flows.
Wouldn't it make sense for you to reach out to a player like
Netflix, and offer to host content cache boxes that happen to
only answer requests coming from SKB IP space, at a price
well below what SKB was going to charge the content provider?
As the eyeball network, you'd see your traffic ratios
balance out as the cache traffic filled your under-utilized outbound
port capacity, and you'd get a bit of additional revenue you otherwise
wouldn't get.  As the content provider, you're serving your customers
for a lower price than SKB wants to charge, and without giving into
SKB's extortion tactics.  It's a win-win-lose situation, in which the
content provider wins, the eyeball network that has a peering
relationship with SKB wins, and the only loser is SKB, which
doesn't get the additional revenue it was looking for, and actually
helps funnel money to a competitor that they otherwise wouldn't
have gotten.

I'm pretty sure this is going to start happening more and more,
as ISPs realize that putting content caches into their IP space
to serve not only their own customers, but also customers of
selected peers can be a source of good leverage in the market.


In reality, which small mom & pop will have peering with BigTelco :-)?

Suffice it to say, Netflix would also need to reconsider whether they 
can afford to give OCA's to mom & pop.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 00:10, Niels Bakker wrote:



Sounds like you think SK should be paying Netflix for bringing their 
content all the way from the US to the Korean peninsula. That's some 
expensive wet cable being used there.


Do we know whether Netflix don't already have OCA's and/or local peering 
in South Korea?


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/10/21 23:57, Sabri Berisha wrote:


I have worked for ISPs. And I remember the late 90s. Bandwidth was $35/mbit
on average, at least for the outfit where I was. Consumers paid roughly $40
for their DSL connections, which at the time went up to 2Mbit depending
on the age of the copper and distance to the DSLAM. Consumer connections
were oversubscribed, on average, 1:35 to 1:50. B2B connections got a better
deal, 1:10 to 1:15.

It was simply not feasible to offer 1:1 bandwidth and still make a profit,
unless you're charging fees the average consumer cannot afford.

Especially considering that the average user doesn't even need or use that
much bandwidth. It's a recurring discussion. People demand more bandwidth
without considering whether or not they need it. End-users, business subs,
and host-owners at large enterprises where I worked. The last ones are the
funniest: entire racks using no more than 100mbit/s and hostowners are
demanding an upgrade from 10G to 25G bEcaUse LaTenCy.

The last consumer ISP I worked at had a very small subset of users that
really needed bandwidth: the "download dudes" who were 24/7 leeching news
servers, and the inevitable gamers that complained about the latency due
to the links being full as a result of said leechers. In that case, a
carefully implemented shaping of tcp/119 did the trick.


It's the conundrum of a shared resource. Like managing 1 lift (elevator, 
for the Americans :-)) that needs to move a building full of people 
between floors.


In the early days of the Internet, circuits were long, expensive and 
slow. Equipment cost a lot for what you could get out of it, and content 
was mostly centralized, making getting to it even more expensive, and 
hence, entrenching the current model.


However, in an era where content is making a push to get as close to the 
eyeballs as possible, kit getting cheaper and faster because of merchant 
silicon, and abundance of aggregated capacity at exchange points, can we 
leverage the shorter, faster links to change the model?


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 02:58, Owen DeLong wrote:


That’s irrelevant to what he is saying.

What he’s saying (and he’s 100% correct) is that any tax a corporation pays is 
collected from their customers one way or another.

A corporation has no other source of income with which to pay its taxes beyond 
those revenues collected from customers.

Of course I should probably expect this from someone who thinks IPv4 shortages 
can be avoided by rationing IPv4 addresses.


There you go again, getting overly excited trying to divert the topic to 
your keen area of interest - IPv4 in Africa.


I'll spell it out here so you are clear and have zero doubt:

I do not respect you - for what you represent on my continent, amongst 
other things. So ignore me, because I am ignoring you. But if you don't 
ignore me, that's your problem too, not mine.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 03:05, Owen DeLong wrote:


Which is the kind of ignorant view of the situation that creates this problem 
in the first place.

It’s not for “no $$”, it’s for all the $$ they got from all those 100Mbps links 
that they are delivering those Tbps of traffic to.

If the aggregate $$ they are collecting is insufficient, then they have priced 
their service incorrectly and should either re-evaluate,
or go bankrupt and sell to someone that knows how to run a business.


Mate, keep your pants on... don't get yourself worked up into excitement 
at my folly.


And in case you haven't noticed, I am ignoring you.

Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/10/21 23:42, Doug Barton wrote:



I didn't say income tax. Corporate taxes are considered an expense by 
the corporation paying them. Like all other expenses, they are 
factored into the cost of goods/services sold.


Ah, yes, agreed. I thought you meant something else...




First, I'm not saying "should." I'm saying that given the market 
economics, having the content providers who use "a lot" of bandwidth 
do something to offset those costs to the ISPs might be the best/least 
bad option. Whether "something" is a local cache box, peering, money, 
or  is something I think that the market should determine.


But all the major content providers do this already. They come to 
exchange points. They provide caches. What more do we want them to do?


Content providers that don't do these things don't generally tend to be 
popular, in which case, we don't have to worry about them flooding 
backbone links.


I am almost sure Netflix have some degree of presence in South Korea. 
What I'm not sure about is what else SK wants them to do beyond that.





And to answer Matthew's question, I don't know what "a lot" is. I 
think the market should determine that as well.


Hehe, free markets determine the fundamental principles through price 
and competition, which is why we are in this mess to begin with. Unless 
you want "government" to make the determination :-).





And for the record, not only have I never worked for an ISP, I was 
saying all the way back in the late '90s that the oversubscription 
business model (which almost always includes punishing users who 
actually use their bandwidth) is inherently unfair to the customers, 
and when the Internet becomes more pervasive in daily life will come 
back to bite them in the ass. I was laughed at for being hopelessly 
naive, not understanding how the bandwidth business works, etc.


Totally agreed. However, we are sort of forced to use this model because 
of the underlying technology. Whenever a finite resource has to be 
shared amongst several people, there has to be some way to manage that 
sharing.


Maybe if Internet services were circuit-switched, we wouldn't have this 
problem. But then again, we wouldn't have an Internet like we do today 
either.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 01:43, Keith Medcalf wrote:


This is blatantly incorrect.  The bits were payed for by the requestor.


You totally missed my dig...

I was being sarcastic.

Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/11/21 00:31, Geoff Huston wrote:


In many environments, the words we use to describe this form of price setting 
are generally prefixed by the adjective “illegal” :-)


Indeed - colluding is generally frowned upon, in which case we are 
doomed to the current model, and may the best man win.


Ultimately, many ISP's and telco's will die. Consolidation will occur, 
but the "big operator" will no longer be as fat as they used to be. 
Focus on hauling bits around with no frills will be a good model, 
especially if you can keep the team lean. The chances of having large 
monopolies that do okay but stifle the market, being chased by 
struggling ISP's that favour passion + frills, is what is likely to 
happen, over the next decade or two.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/10/21 22:13, Michael Thomas wrote:

Isn't that what Erlang numbers are all about? My suspicion is that 
after about 100Mbs most people wouldn't notice the difference in most 
cases. My ISP is about 25Mbs on a good day (DSL) and it serves our 
needs fine and have never run into bandwidth constraints. Maybe if we 
were streaming 4k all of the time it might be different, but frankly 
the difference for 4k isn't all that big. It's sort of like phone 
screen resolution: at some point it just doesn't matter and becomes 
marketing hype.




The ISP looking to charge BigContent for increased link saturation isn't 
looking at the individual 100Mbps links they have sold to their 
downstream customers.


They are looking at the aggregate Gbps or Tbps of traffic that 
BigContent is seeking to deliver across their network, for "no $$".


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/10/21 22:10, Geoff Huston wrote:


I have to agree with Doug Barton's earlier observation is that the base problem 
is that the ISPs are using a flawed business model and they don't want to 
charge their customers what it really costs to provide them with high speed 
access, nor do they want to fund additional back-end capacity in their network 
without some form of offset revenue stream.


I think ISP's do want to charge their customers what it actually costs 
to provide them with a service, but they can't because many ISP's 
business models are based purely on undercutting their nearest competitor.


I might be naive and hopeful to think that operators will have a blood 
handshake to set prices where customers can't wag the tail.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/10/21 21:33, Matthew Petach wrote:


If you sell a service for less than it costs to provide, simply
based on the hopes that people won't actually *use* it, that's
called "gambling", and I have very little sympathy for businesses
that gamble and lose.


You arrived at the crux of the issue, quickly, which was the basis of my 
initial response last week - infrastructure is dying. And we simply 
aren't motivated enough to figure it out.


When you spend 25+ years sitting in a chair waiting for the phone to 
ring or the door to open, for someone to ask, "How much for 5Mbps?", 
your misfortune will never be your own fault.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/10/21 21:08, Doug Barton wrote:



Except that Facebook, Microsoft, and Amazon all caved to SK's demands:

"The popularity of the hit series "Squid Game" and other offerings 
have underscored Netflix's status as the country's second-largest data 
traffic generator after Google's YouTube, but the two are the only 
ones to not pay network usage fees, which other content providers such 
as Amazon, Apple and Facebook are paying, SK said."


Which has emboldened SK to go after the bigger fish.


Prior to the popularity of "House Of Cards", Netflix would have bent 
over and taken it without any lube. Heck, they signed away plenty of 
rights around the world to several networks for "House Of Cards", purely 
because they didn't know how well their own in-house production would 
succeed. Fast-forward, it's 2021 now.


Other players in BigContent that haven't yet found their leverage, will do.




One incentive I haven't seen anyone mention is that ISPs don't want to 
charge customers what it really costs to provide them access. If 
you're the only one in your market that is doing that, no one is going 
to sign up because your pricing would be so far out of line with your 
competition.


Isn't this the curse of a service people consider to be a basic utility 
for life to occur?


Unlike water and power, nearly anyone can start an ISP, and further feed 
the race to the bottom.





Given that issue, I have some sympathy for eyeball networks wanting to 
charge content providers for the increased capacity that is needed to 
bring in their content. The cost would be passed on to the content 
provider's customers...


But eyeballs are already paying you a monthly fee for 100Mbps of service 
(for example). So they should pay a surcharge, over-and-above that, that 
determines how they can use that 100Mbps? Seems overly odd, to me.



(in the same way that corporations don't pay taxes, their customers 
do),...



Many a company pays corporate tax, which is separate from the income tax 
they pay for compensation to their staff.


Of course, YMMV depending on where you live.


so the people on that ISP who are creating the increased demand would 
be (indirectly) paying for the increased capacity. That's actually 
fairer for the other customers who aren't Netflix subscribers.


The reason that Netflix doesn't want to do it is the same reason that 
ISPs don't want to charge their customers what it really costs to 
provide them access.


So what rat hole does this lead us down into? People who want to stream 
Youtube should pay their ISP for that? People who want to spend 
unmentionable hours on Linkedin should be their ISP for that? People who 
want to gawk over Samsung's web site because they love it so much, 
should pay their ISP for that?


Hey, maybe you're right. Maybe that's the model that is needed. After 
all, when we go to a rave, the entry fee is just the entry fee. You 
still need to fork out more cash to actually buy drinks, food or engage 
in some kind of entertainment that may be taking place inside that 
walled garden you paid a cover charge to be a part of.


I don't know...

Mark.

Re: What Eyeballs Did During The Facebook Nap

[Mark Tinka]

On 10/8/21 18:22, Sabri Berisha wrote:


Who says they were ... ahem ... watching? :)


I considered that... and decided that scrolling the Netflix library 
doesn't create that much data :-).


Of course, who pays 100% attention anymore these days. You've watched an 
episode if it was playing. Whether you actually saw it or not, is 
irrelevant :-).


Perhaps folk were watching "How to fix Facebook on your iThingy".




I'd be interested to see global birth rates in June 2022...


Hehehe

Mark.

Re: What Eyeballs Did During The Facebook Nap

[Mark Tinka]

On 10/8/21 17:26, Tom Beecher wrote:

There is already lots of published research on social media addiction 
that does call it out just that strongly.


Oh no, I didn't mean that the research to link social media addiction to 
long-term mental harm does not exist. It's just that such research will 
be ignored or buried under piles of stone to never see the light of 
common day, so that BigCorporate can keep cashing in on our addictions.


As a practical example of life-saving research-stomping, between the 
US$1.5 trillion food industry, the pharmaceutical industry, the medical 
fraternity and the medical insurance companies, there is a reason why 
the globe spends US$1 billion on insulin therapy for Type 2 Diabetes. 
Every. Single. Day. And the research about how T2D can be successfully 
reversed, through diet alone, has been around for over a decade:


https://www.businessdailyafrica.com/bd/lifestyle/health-fitness/how-i-reversed-diabetes-3449528




There is a reason why that company has started going to great lengths 
in recent years to make it harder for outside researchers to do 
similar work.


Exactly my point, above.

And sadly, little young Jane + Thabo won't be looking for social media 
feeds on how to get their social media addiction under control.


Look, I like all this traffic, and it brings in revenue for my business. 
But I'm starting to wonder whether it's all worth it, if we end up 
creating a generation with significantly less brain function, for the 
first time in our evolution, less than 50, 60, 70 years from now.


Mark.

Re: What Eyeballs Did During The Facebook Nap

[Mark Tinka]

On 10/8/21 17:12, Tom Beecher wrote:

Yeah.. I mean people couldn't get stuck in the DoomScroll, so they 
chose to do something else. I'm sure plenty of people did something 
besides Netflix.


For sure.

We just happened to measure Netflix on our side. I'm certain other 
operators measuring other apps likely saw the same thing.


Mark.

Re: What Eyeballs Did During The Facebook Nap

[Mark Tinka]

On 10/8/21 17:07, cosmo wrote:


A psychologist would probably describe this as "self soothing behavior"

An addiction specialist would identify it as illicit drug substitution 
: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7370931/


Which they probably will, but it won't be labeled as dangerous or 
leading to immediate mental harm. And as such, will fly under the radar 
as an addiction that needs to be managed in the same way we manage 
opioid addiction, heroine addiction, sex addiction, alcohol addiction, 
nicotine addiction, e.t.c. All the stuff we have anonymous groups and 
sponsors for.


Mark.

Re: What Eyeballs Did During The Facebook Nap

[Mark Tinka]

On 10/8/21 16:34, Steven Bakker via NANOG wrote:

My inner cynic is interpreting this data a bit differently. Instead of 
going out for a walk or just engage in the gentle art of doing 
nothing, the social media users need something to keep their minds 
distracted from the here and now. Anything to avoid having to do any 
kind of self-reflection or, heaven forbid, live in the moment. And I 
fear that it doesn't apply to adolescents only...


Your reflection is the same as mine - we enjoy sedentary lives, staring 
at some kind of electronic. No doubt about that. It's not unlike why 
most people that give up cigarettes end up gaining weight, because they 
give up nicotine, and replace it with chronic excessive consumption of 
carbohydrates :-).


However, if we are going to Netflix as an alternative to "scrolling 
through time lines like droids on a train", then perhaps something that 
stimulates our brain and prevents it from shrinking due to a lack of 
complex body movements that the brain is mainly designed to manage, is 
what Netflix could look into, for humanity's sake.


Mark.

What Eyeballs Did During The Facebook Nap

[Mark Tinka]

So we are reviewing our flow data, and it's very clear, on our network, 
that during the period Facebook were experiencing their global outage, 
Netflix traffic went up 3X for us.


The psychological assessment of this, for me, is most interesting; 
especially because like carbohydrates, social media addiction does not 
create "medically immediate" mental harm, unlike other addictions such 
as cocaine, heroine, crystal meth, LSD, alcohol, e.t.c., which 
governments and the medical fraternity are quick to label as "dangerous 
to human health due to their addictive properties".


Could Netflix, perhaps, play a part in mitigating the increasing impact 
of social media addiction in teenagers, whose young brains aren't 
developed enough to have sufficient executive control, impulse control 
and good judgement?


In the words of Spock, "Fascinating!"

Mark.

Re: DNS pulling BGP routes?

[Mark Tinka]

On 10/8/21 07:25, Sabri Berisha wrote:


Whenever there is an aviation incident, the keyboard warriors at pprune.org
are always the first to start speculating about root causes, and complain how
the air crew made mistakes. They, the keyboard warriors, of course know how
best to fly an aircraft with 20/20 hindsight from their armchairs.

Why do I see so many posts that are basically throwing Facebook engineers
under the bus? Let's for a moment contemplate about the sheer magnitude of
their operation. With almost 3 billion users worldwide, can you imagine the
amount of DNS queries they have to process? Their scale is unprecedented.

Sure, it's ok to speculate about potential operational or design issues that
may have been contributing factors to the outage. But throwing our colleagues
in front of the lions like this is something I would not recommend.

I'm sure they are aware of these posts, but are unable to reply due to the
amount of NDAs signed.


Folk love to complain and critique. It's human nature, and unproductive 
quality that is part of our DNA.


The good news is that lots of human beings have the DNA to ignore noise 
and carry on helping mankind to grow and live better lives.


In any event, if you aren't willing to put your neck on the line and 
fail spectacularly, I don't care what you have to say. Failure is 
finding out what doesn't work, quickly, and inching closer to what does. 
I'm all for that.


Mark.

Re: Global issues @ Telia - doing a "FB/hold my beer" move?

[Mark Tinka]

On 10/7/21 20:46, Max Tulyev wrote:

We have 2 ports from Telia, one in Kiev (Ukraine) and one in New York 
(USA). I have seen both ports simultaneously dropped traffic volume 
for about one hour today.


Our traffic across Telia dipped at 1600hrs UTC yesterday, and recovered 
2hrs later.


No impact, as I'm certain others with resiliency also saw.

Mark.

Re: DNS pulling BGP routes?

[Mark Tinka]

On 10/7/21 18:21, William Herrin wrote:


It wasn't forgotten. Folks gained a lot of experience with anycast DNS
between 2002 and 2006. Not withdrawing the routes when the servers are
deemed malfunctioning turned out not to be an operationally sound
practice. The theory offered in 3258 was wrong.


Especially terrible when you have a DNS daemon that has crashed, but 
Quagga (or whatever routing suite you use) is still humming.


Mark.

Re: IRR for IX peers

[Mark Tinka]

On 10/7/21 16:33, Nick Hilliard wrote:

there was more to it than that.  The grammar was too complicated to 
easily describe common policies and too limited to describe complex 
policies.  The structure was difficult to extend when the routing 
became more complicated (e.g. mpls, route servers, ipv6, complex ibgp, 
etc). The tooling was too complicated for anyone to understand 
properly how it worked and too early to benefit from later additions, 
e.g. scripting language plugins.  If it had been an easy problem 
domain to fix, it would have been fixed a long time ago, but it wasn't.


All the reasons I tried and gave up, back in 2003.

Mark.

Re: DNS pulling BGP routes?

[Mark Tinka]

On 10/7/21 13:18, Jean St-Laurent wrote:


Something public that we know now, is that it's possible to totally shut down 
facebook and restart it.

Can we shutdown the full internet one day and see if it will restart properly 
without too much hack here and there?


I think one thing that I learned from this Facebook outage is that the 
impact to steady supply of electricity to computing and networking gear 
under spool-up load is not a small problem to scoff at.


We could shutdown the entire Internet, and power companies will probably 
love us. But they will hate a tad more as we reboot it.


Mark.

Re: DNS pulling BGP routes?

[Mark Tinka]

On 10/7/21 08:26, Hank Nussbacher wrote:



Better question is why do we not see any FB netadmins on NANOG? I'm 
not talking about October 2021 but rather over the past 3-5 years how 
many FB techies have posted here like we see people from Google, 
Cloudflare, Akamai, etc.?


They are likely here, but BigContent does not really endorse talking 
about their operations in public fora, typically without PR/Legal OK.


For those who talk about stuff, it's either stuff that is already 
public, publicly-known, or in their own capacity not representing their 
employer.


Mark.

Another Perspective - Kentik's View on the Facebook Outage

[Mark Tinka]

https://www.kentik.com/blog/facebooks-historic-outage-explained/?mkt_tok=ODY5LVBBRC04ODcAAAF_9diyPhC6WFsJNFpS4z2ggF-DWEc6FksyD3aW8B3am5tUvtTOJDYl2MIgMdsmmqOLTL-BpUugbQHAIXCT677LE0OxHM8Dy-mqRorJQXnAjg4

Mark.

Re: DNS pulling BGP routes?

[Mark Tinka]

On 10/7/21 00:22, Michael Thomas wrote:



But it wasn't just their DNS subnets that were pulled, I thought. I'm 
obviously really confused. Anycast to a DNS server makes sense that 
they'd pull out if they couldn't contact the backend. But I thought 
that almost all of their routes to the backend were pulled? That is, 
the DFZ was emptied of FB routes.


During the outage, we kept serving traffic to Facebook in various 
locations. So it would seem that while a large amount of their NLRI left 
the DFZ, it wasn't all of them.


However, what was left was not sufficient to actually keep typical 
services up, including their DNS.


Mark.

Re: DNS pulling BGP routes?

[Mark Tinka]

On 10/7/21 00:37, Michael Thomas wrote:

Maybe the problem here is that two things happened and the article 
conflated the two: the DNS infrastructure pulled its routes from the 
anycast address and something else pulled all of the other routes but 
wasn't mentioned in the article.


The origin problem was some "automation thingy" that went to check 
capacity status around the network ahead of some planned maintenance 
work, and that "automation thingy" decided checking was not enough, 
let's just turn the whole thing off.


Mark.

Re: Facebook post-mortems...

[Mark Tinka]

On 10/6/21 06:51, Hank Nussbacher wrote:



- "During one of these routine maintenance jobs, a command was issued 
with the intention to assess the availability of global backbone 
capacity, which unintentionally took down all the connections in our 
backbone network"


Can anyone guess as to what command FB issued that would cause them to 
withdraw all those prefixes?


Hard to say, as it seems that the command was innocent enough, perhaps 
running a batch of other sub-commands to check port status, bandwidth 
utilization, MPLS-TE values, e.t.c. However, sounds like another 
unforeseen bug in the command ran other things, or the cascade process 
of how the sub-commands were ran caused unforeseen problems.


We shall guess this one forever, as I doubt Facebook will go into that 
much detail.


What I can tell you is that all the major content providers spend a lot 
of time, money and effort in automating both capacity planning, as well 
as capacity auditing. It's a bit more complex for them, because their 
variables aren't just links and utilization, but also locations, fibre 
availability, fibre pricing, capacity lease pricing, the presence of 
carrier-neutral data centres, the presence of exchange points, current 
vendor equipment models and pricing, projection of future fibre and 
capacity pricing, e.t.c.


It's a totally different world from normal ISP-land.




- "it was not possible to access our data centers through our normal 
means because their networks were down, and second, the total loss of 
DNS broke many of the internal tools we’d normally use to investigate 
and resolve outages like this.  Our primary and out-of-band network 
access was down..."


Does this mean that FB acknowledges that the loss of DNS broke their 
OOB access?


I need to put my thinking cap on, but not sure whether running DNS in 
the IGP would have been better in this instance.


We run our Anycast DNS network in our IGP, mainly to always guarantee 
latency-based routing, but also to ensure that the failure of a 
higher-level protocol like BGP does not disconnect internal access that 
is needed for troubleshooting and repair. Given the IGP is a much more 
lower-level routing protocol, it's more likely (not impossible) that it 
would not go down with BGP.


In the past, we have, indeed, had BGP issues that allowed us to maintain 
DNS access internally as the IGP was unaffected.


The final statement from that report is interesting:

    "From here on out, our job is to strengthen our testing,
    drills, and overall resilience to make sure events like this
    happen as rarely as possible."

... which, in my rudimentary translation, means that:

    "There are no guarantees that our automation software will not
    poop cows again, but we hope that when that does happen, we
    shall be able to send our guys out to site much more quickly."

... which, to be fair, is totally understandable. These automation 
tools, especially in large networks such as BigContent, are 
significantly more fragile the more complex they get, and the more batch 
tasks they need to perform on various parts of a network of this size 
and scope. It's a pity these automation tools are all homegrown, and 
can't be bought "pre-packaged and pre-approved to never fail" from IT 
Software Store down the road. But it's the only way for networks of this 
capacity to operate, and the risk they always sit with for being that large.


Mark.

Re: Facebook post-mortems...

[Mark Tinka]

On 10/5/21 16:59, Matthew Kaufman wrote:



Disagree for two reasons:

1. If you have some DNS working, you can point it at a static “we are 
down and we know it” page much sooner.


Isn't that what Twirra is for, nowadays :-)...




2. If you have convinced the entire world to install tracking pixels 
on their web pages that all need your IP address, it is rude to the 
rest of the world’s DNS to not be able to always provide a prompt (and 
cacheable) response.


Agreed, but I know many an exec that signs the capex cheques who may 
find "rude" not a noteworthy discussion point when we submit the budget.


Not saying I think being rude is cool, but there is a reason we are 
here, now, today.


Mark.

Re: Facebook post-mortems...

[Mark Tinka]

On 10/5/21 16:49, Joe Greco wrote:


Unrealistic user expectations are not the point.  Users can demand
whatever unrealistic claptrap they wish to.


The user's expectations, today, are always going to be unrealistic, 
especially when they are able to enjoy a half-decent service free-of-charge.


The bar has moved. Nothing we can do about it but adapt.




The point is that there are a lot of helpdesk staff at a lot of
organizations who are responsible for responding to these issues.
When Facebook or Microsoft or Amazon take a dump, you get a storm
of requests.  This is a storm of requests not just to one helpdesk,
but to MANY helpdesks, across a wide number of organizations, and
this means that you have thousands of people trying to investigate
what has happened.


We are in agreement.

And it's no coincidence that the Facebook's of the world rely almost 
100% on non-human contact to give their users support. So that leaves 
us, infrastructure, in the firing line to pick up the slack for a lack 
of warm-body access to BigContent.




It is very common for large companies to forget (or not care) that
their technical failures impact not just their users, but also
external support organizations.


Not just large companies, but I believe all companies... and worse, not 
at ground level where folk on lists like these tend to keep in touch, 
but higher up where money decisions where caring about your footprint on 
other Internet settlers whom you may never meet matters.


You and I can bash our heads till they come home, but if the folk that 
need to say "Yes" to $$$ needed to help external parties troubleshoot 
better don't get it, then perhaps starting a NOG or some such is our 
best bet.




I totally get your disdain and indifference towards end users in these
instances; for the average end user, yes, it indeed makes no difference
if DNS works or not.


On the contrary, I looove customers. I wasn't into them, say, 12 
years ago, but since I began to understand that users will respond to 
empathy and value, I fell in love with them. They drive my entire 
thought-process and decision-making.


This is why I keep saying, "Users don't care about how we build the 
Internet", and they shouldn't. And I support that.


BigContent get it, and for better or worse, they are the ones who've set 
the bar higher than what most network operators are happy with.


Infrastructure still doesn't get it, and we are seeing the effects of 
that play out around the world, with the recent SK Broadband/Netflix 
debacle being the latest barbershop gossip.




However, some of those end users do have a point of contact up the
chain.  This could be their ISP support, or a company helpdesk, and
most of these are tasked with taking an issue like this to some sort
of resolution.  What I'm talking about here is that it is easier to
debug and make a determination that there is an IP connectivity issue
when DNS works.  If DNS isn't working, then you get into a bunch of
stuff where you need to do things like determine if maybe it is some
sort of DNSSEC issue, or other arcane and obscure issues, which tends
to be beyond what front line helpdesk is capable of.


We are in agreement.



These issues often cost companies real time and money to figure out.
It is unlikely that Facebook is going to compensate them for this, so
this brings me back around to the point that it's preferable to have
DNS working when you have a BGP problem, because this is ultimately
easier for people to test and reach a reasonable determination that
the problem is on Facebook's side quickly and easily.


We are in agreement.

So let's see if Facebook can fix the scope of their DNS architecture, 
and whether others can learn from it. I know I have... even though we 
provide friendly secondary for a bunch of folk we are friends with, we 
haven't done the same for our own networks... all our stuff sits on just 
our network - granted in many different countries, but still, one AS.


It's been nagging at the back of my mind for yonks, but yesterday was 
the nudge I needed to get this organized; so off I go.


Mark.

Re: Facebook post-mortems... - Update!

[Mark Tinka]

On 10/5/21 15:40, Mark Tinka wrote:



I don't disagree with you one bit. It's for that exact reason that we 
built:


    https://as37100.net/

... not for us, but specifically for other random network operators 
around the world whom we may never get to drink a crate of wine with.


I have to say that it has likely cut e-mails to our NOC as well as 
overall pain in half, if not more.


What I forgot to add, however, is that unlike Facebook, we aren't a 
major content provider. So we don't have a need to parallel our DNS 
resiliency with our service resiliency, in terms of 3rd party 
infrastructure. If our network were to melt, we'll already be getting it 
from our eyeballs.


If we had content of note that was useful to, say, a handful-billion 
people around the world, we'd give some thought - however complex - to 
having critical services running on 3rd party infrastructure.


Mark.

Re: Facebook post-mortems...

[Mark Tinka]

On 10/5/21 15:04, Joe Greco wrote:



You don't think at least 10,000 helpdesk requests about Facebook being
down were sent yesterday?


That and Jane + Thando likely re-installing all their apps and 
iOS/Android on their phones, and rebooting them 300 times in the hopes 
that Facebook and WhatsApp would work.


Yes, total nightmare yesterday, but sure that 9,999 of the helpdesk 
tickets had nothing to do with DNS. They likely all were - "Your 
Internet is down, just fix it; we don't wanna know".




There's something to be said for building these things to be resilient
in a manner that isn't just convenient internally, but also externally
to those people that network operators sometimes forget also support
their network issues indirectly.


I don't disagree with you one bit. It's for that exact reason that we built:

    https://as37100.net/

... not for us, but specifically for other random network operators 
around the world whom we may never get to drink a crate of wine with.


I have to say that it has likely cut e-mails to our NOC as well as 
overall pain in half, if not more.


Mark.

Re: Facebook post-mortems...

[Mark Tinka]

On 10/5/21 14:58, Jean St-Laurent wrote:

If your NS are in 2 separate entities, you could still resolve your 
MX/A//NS.

Look how Amazon is doing it.

dig +short amazon.com NS
ns4.p31.dynect.net.
ns3.p31.dynect.net.
ns1.p31.dynect.net.
ns2.p31.dynect.net.
pdns6.ultradns.co.uk.
pdns1.ultradns.net.

They use dyn DNS from Oracle and ultradns. 2 very strong network of anycast DNS 
servers.

Amazon would have not been impacted like Facebook yesterday. Unless ultradns 
and Oracle have their DNS servers hosted in Amazon infra? I doubt that Oracle 
has dns hosted in Amazon, but it's possible.

Probably the management overhead to use 2 different entities for DNS is not 
financially viable?


So I'm not worried about DNS stability when split across multiple 
physical entities.


I'm talking about the actual services being hosted on a single network 
that goes bye-bye like what we saw yesterday.


All the DNS resolution means diddly, even if it tells us that DNS is not 
the issue.


Mark.

Re: Facebook post-mortems...

[Mark Tinka]

On 10/5/21 14:52, Joe Greco wrote:


That's not quite true.  It still gives much better clue as to what is
going on; if a host resolves to an IP but isn't pingable/traceroutable,
that is something that many more techy people will understand than if
the domain is simply unresolvable.  Not everyone has the skill set and
knowledge of DNS to understand how to track down what nameservers
Facebook is supposed to have, and how to debug names not resolving.
There are lots of helpdesk people who are not expert in every topic.

Having DNS doesn't magically get you service back, of course, but it
leaves a better story behind than simply vanishing from the network.


That's great for you and me who believe in and like troubleshooting.

Jane and Thando who just want their Instagram timeline feed couldn't 
care less about DNS working but network access is down. To them, it's 
broken, despite your state-of-the-art global DNS architecture.


I'm also yet to find any DNS operator who makes deploying 3rd party 
resiliency to give other random network operators in the wild 
troubleshooting joy their #1 priority for doing so :-).


On the real though, I'm all for as much useful redundancy as we can get 
away with. But given just how much we rely on the web for basic life 
these days, we need to do better about making actual services as 
resilient as we can (and have) the DNS.


Mark.

Re: Facebook post-mortems...

[Mark Tinka]

On 10/5/21 08:55, a...@nethead.de wrote:



Rumour is that when the FB route prefixes had been withdrawn their 
door authentication system stopped working and they could not get back 
into the building or server room :)


Assuming there is any truth to that, guess we can't cancel the hard 
lines yet :-).


#EverythingoIP

Mark.

Re: Facebook post-mortems...

[Mark Tinka]

On 10/5/21 14:08, Jean St-Laurent via NANOG wrote:


Maybe withdrawing those routes to their NS could have been mitigated by having 
NS in separate entities.


Well, doesn't really matter if you can resolve the A//MX records, 
but you can't connect to the network that is hosting the services.


At any rate, having 3rd party DNS hosting for your domain is always a 
good thing to have. But in reality, it only hits the spot if the service 
is also available on a 3rd party network, otherwise, you keep DNS up, 
but get no service.


Mark.

Re: IRR for IX peers

[Mark Tinka]

On 10/5/21 09:29, Łukasz Bromirski wrote:


…like a, say, „single pane of glass”? ;)


Oh dear Lord :-)...

Mark.

Re: massive facebook outage presently

[Mark Tinka]

On 10/4/21 20:48, Luke Guillory wrote:

I believe the original change was 'automatic' (as in configuration 
done via a web interface). However, now that connection to the outside 
world is down, remote access to those tools don't exist anymore, so 
the emergency procedure is to gain physical access to the peering 
routers and do all the configuration locally.




Q: What is automation?
A: Breaking the network at scale.

I suppose we shall know more in the coming days, but to me, it smells 
like a wide-scale automatic update roll-out that didn't go to plan. 
Isn't the first time a major content provider has suffered this; likely 
won't be the last.


In the end, the best thing for all of us is that it is a teaching 
moment, and we move the needle forward.


Mark.

Re: massive facebook outage presently

[Mark Tinka]

On 10/4/21 22:33, Eric Kuhnke wrote:

I am starting to see reports that in ISPs with very large numbers of 
residential users, customers are starting to press the factory-reset 
buttons on their home routers/modems/whatever, in an attempt to make 
Facebook work. This is resulting in much heavier than normal first 
tier support volumes. The longer it stays down the worse this is going 
to get.


A relative could not understand how she received an e-mail from me 
yesterday about a family matter (via GMail) when the Internet was down :-).


Mark.

Re: massive facebook outage presently

[Mark Tinka]

On 10/4/21 22:27, Łukasz Bromirski wrote:



I bet FB tested the change on smaller scale and everything was fine, 
and only then started to roll this over wider network and at that 
point „something” broke. Or some bug needed a moment to start 
cascading issues around the infra.


This is the annoyance that is our world.

In the lab, it's always good.

Mark.

Re: massive facebook outage presently

[Mark Tinka]

On 10/4/21 22:23, Baldur Norddahl wrote:

Not in such a primitive fashion no. But they could definitely have a 
secondary network that will continue to work even if something goes 
wrong with the primary.


On IPv6, no less :-).

On a serious note, I can't even imagine what it takes to run a network 
of that scale, and plan for situations like this, if, indeed, inline 
access was lost.


Mark.

Re: IRR for IX peers

[Mark Tinka]

On 10/4/21 21:55, Nick Hilliard wrote:


 Nearly 30 years on, this is still the state of the art.


Not an unlike an NMS... still can't walk into a shop and just buy one 
that works out of the box :-).


Mark.

Re: massive facebook outage presently

[Mark Tinka]

On 10/4/21 18:22, Eric Kuhnke wrote:

Considering the massive impact of this it would be interesting to see 
some traffic graphs from ISPs that have PNIs with Facebook, or high 
volume peering sessions across an IX, showing traffic to FB falling 
off a cliff.


Our PNI's and FNA's in Africa have dipped.

But PNI's in Europe have nearly doubled, even though we aren't 
delivering any Facebook services of note to our eyeballs.


Strangest thing...

Mark.

Re: slack.com

[Mark Tinka]

On 10/2/21 08:14, Bill Woodcock wrote:

We did not use an NTA, but we did flush our cache immediately once 
Slack had fixed their problem.  I think that’s the right balance of 
carrot and stick.


Tend to agree with this approach.

But I can see how an issue like this could be potentially religious. 
DNSSEC deployment rate is bad enough, as it is.


Mark.

slack.com

[Mark Tinka]

So, that wasn't fun, yesterday:

https://lists.dns-oarc.net/pipermail/dns-operations/2021-September/021340.html

We were also hit, given we run DNSSEC on our resolvers.

Interesting some large open resolver operators use Negative TA's for 
this sort of thing. Not sure how this helps with the DNSSEC objective, 
but given the kind of pain mistakes like these can cause, I can see why 
they may lean on NTA's.


Mark.

Re: [External] Re: uPRF strict more

[Mark Tinka]

On 10/1/21 19:28, Randy Bush wrote:


in fact, this seems to be the modern conservative style for some years.
i sometimes wonder if it is worth the config pain.


If having routers dedicated to peering, transit or edge functions is 
worth the extra pain, in lieu of doing it all on one box?


Mark.

Re: [External] Re: uPRF strict more

[Mark Tinka]

On 10/1/21 20:17, Adam Thompson wrote:

Do people in other parts of the world have access (both physical and 
logical) to enough bilateral peering (and budgets...) that it makes 
sense to deploy a router per peer?


Certainly not a router per peer, but a peering router per city, where it 
may connect to one or more exchange points. This is what we do.


Granted, it does increase your budgeting complexity, but in our case, 
over time, the delineation has actually simplified operations that the 
architecture has paid itself back many times over.


In the real world, this is not always possible, and I understand that a 
peering router for some networks may also be providing transit as well 
as edge functions. This is quite normal, even though it can create other 
complexities depending on whom the eBGP session is with - which then 
lends itself to running parts or all of the Internet in VRF's and all 
that hocus pocus. When we tried this sort of thing at a previous job 
some 14 years ago, it was just simpler to have separate routers each 
handling transit, peering and customer edge.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/1/21 18:31, Tom Hill wrote:


Many (most?) route servers provide little control over who your routes
are advertised toward. This can be fun where DDoS is concerned.

I've used some that did have deny-list controls for ASNs, fail to
consistently apply those rules. Again, that was a 'fun' surprise.


Nowadays, well-run exchange points would have solved this. The reason we 
don't use route servers anymore is because they sometimes break (either 
due to code, or from people), and the policies you thought would give a 
good 3AM sleep suddenly aren't working anymore.


It was cheaper for us to maintain bi-lateral sessions.

That said, route servers have their place, and I won't knock them down.

Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/1/21 18:05, Laura Smith wrote:


Speaking as one of those smaller ISPs willing to do whatever it takes, perhaps 
you could answer me this riddle.

- PoP in one of your "half-decent data centres" ... tick.
- Connnection to one of your "exchange point" ... tick.
- $certain_large_cdn present on said "exchange point" ... tick.

And yet .

- $certain_large_cdn publishes routes on route server ? Nope.
- $certain_large_cdn willing to establish direct peering session ? Nope.

I am well aware of the "big boys club" that operates at most exchanges where 
the large networks see it beneath them to peer with (or publish routes for the benefit 
of) the unwashed masses.

But I struggle to comprehend why $certain_large_cdn would effectively cut off 
their nose to spite their face ?


Yes, this is a rather painful one, and it has been on the rise in the 
last few years as the major content folk struggle to keep up with 
peering requirements, and everything that goes along with maintaining 
those relationships.


They also seem to have a number of complex operational requirements 
between their own backbones, their own PoP's, partner PoP's, transit 
links, e.t.c.


There is no easy answer for this one, apart from doing the leg-work to 
find out who the best person at the other end to speak to is. I'd 
recommend working with the data centre and exchange point operators to 
make meaningful introductions. You may not end up where you want, but 
you certainly increase your chances of doing so.


Mark.

Re: S.Korea broadband firm sues Netflix after traffic surge

[Mark Tinka]

On 10/1/21 16:19, Blake Hudson wrote:



I'll never understand over how ISPs see content providers as the enemy 
(or a rival). The content is why ISPs have customers. Don't get upset 
when your customer uses the service that you sold them (in a way that 
is precisely in accordance with the expected usage)!


It's because infrastructure (that's us, the network operators), still 
don't get it.


We are no longer front & centre in the eyes of our users. They see us as 
an impediment... providers they must buy costly megabytes of mobile data 
from, providers they must call to fix broken fibre, providers they must 
shout at when a single CGN IPv4 address they sit behind breaks their 
Netflix, and so on and so on.


Users only care about the service they use their mobile phone, tablet, 
console or laptop for. They don't care how many customers their ISP has, 
whether the ISP is a small mom & pop or some global behemoth, or whether 
the ISP's CEO is was on the cover of TIME magazine last week.


As my American friend used to say, "They just want their MTV".

In the late 90's and early 2000's, when content folk wanted to work with 
us, infrastructure folk, to grow their businesses, we just saw easy, 
free money to tax toward our shiny new Lamborghinis and beach side 
holiday villas. Well, guess whom we are now begging for seats on their 
submarine cable build projects, community funding programs, and caches 
to be installed in our not-so-huge data centres, all for free?


The reason Google, Facebook, Microsoft, Amazon, e.t.c., all built their 
own global backbones is because of this nonsense that SK Broadband is 
trying to pull with Netflix. At some point, the content folk will get 
fed up, and go build it themselves. What an opportunity infrastructure 
cost itself!


Akamai have also quietly been building their own backbone. Wonder why.

No doubt Netflix have someone either thinking about the same, or putting 
a plan into motion.


The bad news now, is, there are plenty of many, small, local and 
regional ISP's who are willing to do whatever it takes to work with the 
content providers. All that's required is some network, a half-decent 
data centre and an exchange point. Gone are the days where customers 
clamored to sign up with Big Telco.


If anyone wonders why "infrastructure is dead", well, this is why.

21 years later, and we still don't get it! No wonder the mobile 
companies are watching their slow death, from the rosy days of billions 
from basic SMS, to the perils of 5G investments for diddly return.


Wake me up when all this is over. I'll be in my wine stupor until then.

Mark.

VPNv6 Native Signaling - Nokia (And Solutions to RFC 7439)

[Mark Tinka]

So, I have it on good authority that from release 21.7.R1, Nokia not 
only introduces LDPv6 to their IXR platform, but also provides native 
support for l2vpnv6 and l3vpnv6 signaling.


AFAIK, Cisco and Juniper only support mpls2ip forwarding for LDPv6, and 
all other VPN services can only be signaled via LDPv4 only.


So Nokia are able to natively signal the following VPN services via LDPv6:

    - EoMPLS (p2p + p2mp)
    - VPLS (mp2mp)
    - l3vpn (BGP)
    - EVPN (mp2mp)
    - BGP-LU

It's unclear whether this has been tested against their own platforms 
(IXR and SR), but I'd imagine inter-op should not be an issue since it's 
the same shared code, despite different chip sets.


For me, this is a good solution toward fixing the issues identified in 
RFC 7439, which bodes well for operators who have had to hold back their 
IPv6-only ambitions due to this.


I hope other vendors can jump onboard, whenever they choose to.

Mark.

Re: [External] Re: uPRF strict more

[Mark Tinka]

On 10/1/21 01:51, Valdis Klētnieks wrote:


Am I insufficently caffienated, or is uRPF the least of your problems
if you don't have a full table *and* don't have a default route?


A partial table with no default is perfectly fine for a peering router.

As long as your peering router knows how to get to your prefixes and 
those of your customers, as well as the prefixes of the networks you 
peer with, you're good to go.


Mark.

Re: [External] Re: uPRF strict more

[Mark Tinka]

On 9/30/21 17:56, Hunter Fuller wrote:

On Thu, Sep 30, 2021 at 12:08 AM Mark Tinka  wrote:

If you don't plan to run a full BGP table on a device, don't enable uRPF, even 
loose-mode.

At least in Ciscoland, loose URPF checks will pass if you have a
default route. So I do not think it could result in inadvertent
blackholing of traffic.

What it does allow is for *deliberate* blackholing for traffic; if you
null-route a prefix, you now block incoming traffic from that subnet
as well. This can be useful and it is how we are using URPF.


Agreed.

I should have said "If you don't plan to run a full BGP table on a 
device without a default a route as well, don't enable uRPF, even 
loose-mode".


Principally, we don't run default on any of our service routers. 
Technically, we point default to the bin on all our service routers, as 
that's the fastest way for the router to handle illegal traffic it 
"could" receive.


Mark.

Re: uPRF strict more

[Mark Tinka]

On 9/29/21 20:14, Phil Bedard wrote:

Disclosure I work for Cisco and try to look after some of their 
peering guidelines.


Agree with Adam’s statement, use uRPF on edge DIA customers.  Using it 
elsewhere on the network eventually is going to cause some issue and 
its usefulness today is almost nil.  That being said we still see 
large providers who have it turned on for peering/transit interfaces 
either out of legacy configuration or other reasons.  The vast 
majority do not use it for those interface roles.




If you don't plan to run a full BGP table on a device, don't enable 
uRPF, even loose-mode.


Mark.