Re: China’s Slow Transnational Network

[Pengxiong Zhu]

Thank you for your insights. We are not so familiar with interconnect and
peering, we will ask you some questions for clarification first. Hope you
don't mind. :-)

When there is a tri-opoly, with no opportunity of competition, its easily
> possible to set prices which are very different than market conditions.

I assume the tri-opoly involves a Chinese ISP, outside ISP A, outside ISP
B. Who is competing with whom? Why its easily possible to set prices which
are very different than market conditions?


> additionally, the three don't purchase enough to cover demand for their
> own network.
>
Do you mean that the three don't purchase enough capacity for their traffic
going out of their network(China->Outside)? If this is what you mean,
however, we don't observe low speed in that direction. We assume there is
not so much traffic going out of China, comparing to the traffic coming in.
Also, why would the three purchase outbound traffic if they set their
inbound traffic artificially high? They could charge some peers less for
the outbound traffic to solve the problem.

Best,
Pengxiong Zhu
Department of Computer Science and Engineering
University of California, Riverside


On Mon, Mar 2, 2020 at 2:58 PM Tom Paseka  wrote:

> Most of the performance hit is because of commercial actions, not
> censorship.
>
> When there is a tri-opoly, with no opportunity of competition, its easily
> possible to set prices which are very different than market conditions.
> This is what is happening here.
>
> Prices are set artificially high, so their interconnection partners wont
> purchase enough capacity. additionally, the three don't purchase enough to
> cover demand for their own network. Results in congestion.
>
> On Mon, Mar 2, 2020 at 2:49 PM Pengxiong Zhu  wrote:
>
>> You seem to be implying that you don't believe/can't see the GFW
>>
>>
>> No, that's not what I meant. I thought mandatory content filtering at the
>> border means traffic throttling at the border, deliberately or accidentally
>> rate-limiting the traffic, now
>> I think he was referring to GFW and the side effect of deep packet
>> inspection.
>>
>> In fact, we designed a small experiment to locate the hops with GFW
>> presence, and then try to match them with the bottleneck hops. We found
>> only in 34.45% of the cases, the GFW hops match the bottleneck hops.
>>
>> Best,
>> Pengxiong Zhu
>> Department of Computer Science and Engineering
>> University of California, Riverside
>>
>>
>> On Mon, Mar 2, 2020 at 1:13 PM Matt Corallo  wrote:
>>
>>> > find out direct evidence of mandatory content filtering at the border
>>>
>>> You seem to be implying that you don't believe/can't see the GFW, which
>>> seems surprising. I've personally had issues with traffic crossing it
>>> getting RST'd (luckily I was fortunate enough to cross through a GFW
>>> instance which was easy to avoid with a simple iptables DROP), but its
>>> also one of the most well-studied bits of opaque internet censorship
>>> gear in the world. I'm not sure how you could possibly miss it.
>>>
>>> Matt
>>>
>>> On 3/2/20 2:55 PM, Pengxiong Zhu wrote:
>>> > Yes, we agree. The poor transnational Internet performance effectively
>>> > puts any foreign business that does not have a physical presence (i.e.,
>>> > servers) in China at a disadvantage.
>>> > The challenge is to find out direct evidence to prove mandatory content
>>> > filtering at the border, if the government is actually doing it.
>>> >
>>> > Best,
>>> > Pengxiong Zhu
>>> > Department of Computer Science and Engineering
>>> > University of California, Riverside
>>> >
>>> >
>>> > On Mon, Mar 2, 2020 at 8:38 AM Matt Corallo >> > > wrote:
>>> >
>>> > It also gives local competitors a leg up by helping domestic apps
>>> > perform better simply by being hosted domestically (or making
>>> > foreign players host inside China).
>>> >
>>> >> On Mar 2, 2020, at 11:27, Ben Cannon >> >> > wrote:
>>> >>
>>> >> 
>>> >> It’s the Government doing mandatory content filtering at the
>>> >> border.  Their hardware is either deliberately or accidentally
>>> >> poor-performing.
>>> >>
>>> >> I believe providing limited and throttled external connectivity
>>> >> may be deliberate; think of how that curtails for one thing;
>>> >> streaming video?
>>> >>
>>> >> -Ben.
>>> >>
>>> >> -Ben Cannon
>>> >> CEO 6x7 Networks & 6x7 Telecom, LLC
>>> >> b...@6by7.net 
>>> >>
>>> >>
>>> >>
>>> >>> On Mar 1, 2020, at 9:00 PM, Pengxiong Zhu >> >>> > wrote:
>>> >>>
>>> >>> Hi all,
>>> >>>
>>> >>> We are a group of researchers at University of California,
>>> >>> Riverside who have been working on measuring the transnational
>>> >>> network performance (and have previously asked questions on the
>>> >>> mailing list). Our work has now led to a publication in
>>> >>> Sigmetrics 2020 and we are 

Re: China’s Slow Transnational Network

[Pengxiong Zhu]

I see. Thank you!

Best,
Pengxiong Zhu
Department of Computer Science and Engineering
University of California, Riverside


On Sat, Mar 21, 2020 at 9:13 AM Mark Tinka  wrote:

>
>
> On 21/Mar/20 09:09, Pengxiong Zhu wrote:
>
> How do they deliberately congest peering ports? Do you hear from those
> Chinese operators or you observe this from the traffic?
>
>
> Simple - let them run at 350% of capacity and pipeline upgrades for Lord
> knows how long :-).
>
> On a serious note, let's have a beer.
>
>
> Seems like you also think GFW is part of the cause,
>
>
> I do - each of my trips to China have questioned the role of my VPN for my
> online experience.
>
>
> however, we don't have direct evidence.
>
>
> I won't argue with you there, you did the groundwork. I'm just being
> anecdotal.
>
>
> Just curious, What is your "problems"? I thought it's congestion.
>
>
> Accessibility and penetration rates.
>
> Mark.
>

Re: It's not about the congestion, it's about the profit motive driving the industry

[Mark Tinka]

On 21/Mar/20 17:53, Saku Ytti wrote:

>
> If we (me included) would be half as angry about those who have less
> than we have, as we are about those who have more than we have,
> inequality wouldn't exist. We are the beneficiaries of immense
> suffering of millions of people, things are artificially cheap for us
> to buy at human cost 'somewhere else'. We are not the heroes of this
> story.

I clean my pool every morning. When I am holding the rod closer to the
leaf basket, I use less energy. When I am holding the rod farther from
the leaf basket, not only do I use more energy, but I have less control
of the guidance of the rod and basket, as it resists the water.

The businesses that will succeed in this new digital economy will be
those that empathize with customers, listen to them, engage them, and
provide them with value, regardless of their economic (mis)fortunes.

Those businesses that continue to push product and detach themselves
from how customers want to engage with them will quickly become irrelevant.

In the past, countries were set apart by their physical infrastructure;
those that had better infrastructure had a higher chance of succeeding
relative to those that didn't. Today (and in the future), regardless of
the tangible infrastructure one country has vs. another, the Internet is
the single most important thing that levels the playing field.

Today, a kid in Kigali has about the same opportunities as a kid in San
Francisco, to harness the Internet in order to make each of their lives
better.

The Internet in Australia is exactly the same as the Internet in Sao Paulo.

Mark.

Re: COVID-19 vs. our Networks

[Mark Tinka]

On 21/Mar/20 14:43, Alexandre Petrescu wrote:

>  
>
> I tend to agree - I dont think there is any capacity problem in the
> core network or server platforms, including netflix.  I do not see it
> for my part as of now.  I am an end user, not a Network sysadmin.
>
> I heard about EU measures to attenuate such a problem tha tmight
> arrive - I think it is mis-informed.
>
> I also heard EC (European COmmission) looking to push all efforts in
> robotics, how could robotics help this, several ideas like drones, or
> the open source respiratory device, or why not sending a robot do
> shopping for me.  I think these look far fetched but are promissing.

Business models were always changing. What the Coronavirus has done is
amplify and accelerate those transitions.

The thing is even after the Coronavirus pandemic is solved, businesses
are going to have to adapt to new operating models in this new digital
economy. Unfortunately, many are going to assume that it was only
necessary for the period of the pandemic, and will want to revert to
their traditional ways of doing things prior to the outbreak once all
the dust settles. I empathize with those businesses.

Mark.

Re: CISA: Guidance on the Essential Critical Infrastructure Workforce

[Mark Tinka]

On 21/Mar/20 14:38, Alexandre Petrescu wrote:

>  
>
> Today first time I see in FRance TV news one interviewer puts one
> time-use covers over the microphones when interviewing people. (it's a
> one time cover in addition to the typical windshield which is
> expensive and cant be changed each time).  Only today.  Many days
> lost, many viruses spread.
>
> Still on tables on all TV channels I watch they dont have covers on
> microphones.

There was a time when the Coronavirus issue was evolving monthly. Then
weekly. Then daily.

What I tell all my friends is things are now evolving hourly. What you
knew when you went to bed will be wildly different when you wake up.

Just today, several countries in Africa have shutdown airports and
borders. This wasn't the case 12hrs ago.

The rate of change is exponential.

Mark.

Re: COVID-19 vs. our Networks

[Mark Tinka]

On 21/Mar/20 13:53, Mike Hammett wrote:
> Unless the IX or OCA feed goes to the DSLAM, node, tower...  no.

Not sure what you mean.

Mark.

Re: COVID-19 vs. our Networks

[Mark Tinka]

On 21/Mar/20 23:37, Rich Kulawiec wrote:

>
> My remarks weren't about Netflix or any other particular service.
> (FWIW, I agree with you on both quoted points about the lack of
> evidence.  Maybe it'll arrive.  Maybe it won't.)
>
> I was trying to speak, perhaps unsuccessfully, in broader terms about
> trying to best position ourselves for challenges that we may not
> see coming despite the aggregated millenia of expertise here.  I think
> at this particular point in time we need to be ready for anything,
> for a very nebulous and possibly quite surprising value of "anything".

I would, generally, file that under "Why we get up everyday" :-).

The folk over in IT are probably dealing with plenty of noise about
VPN's and cybersecurity things as folk work more from home than usual.

If backbones begin to flutter and last miles start to screech, I'm
certain this group will be woken from its slumber.

Mark.

Re: COVID-19 vs. our Networks

[Rich Kulawiec]

On Sat, Mar 21, 2020 at 04:42:51AM +0200, Mark Tinka wrote:
> All I'm saying is at the moment, there is no empirical information to
> suggest that Netflix will break what's left of the Internet. Nor is
> there any empirical information suggesting that singling them out will
> help keep it going.

My remarks weren't about Netflix or any other particular service.
(FWIW, I agree with you on both quoted points about the lack of
evidence.  Maybe it'll arrive.  Maybe it won't.)

I was trying to speak, perhaps unsuccessfully, in broader terms about
trying to best position ourselves for challenges that we may not
see coming despite the aggregated millenia of expertise here.  I think
at this particular point in time we need to be ready for anything,
for a very nebulous and possibly quite surprising value of "anything".

---rsk

Re: interesting troubleshooting

[Tassos Chatzithomaoglou]

Saku Ytti wrote on 21/3/20 19:04:
> On Sat, 21 Mar 2020 at 18:55, Tassos Chatzithomaoglou
>  wrote:
>
>> I still don't understand why the vendors cannot make it work in one 
>> direction only (the low-end platform would only need to remove an extra 
>> label, no need to inspect traffic).
>> That would help us a lot, since the majority of our traffic is downstream to 
>> the customer.
> It is signalled separately for TX and RX and some vendors do allow you
> to signal it separately.
>
Yep, the RFC gives this option.
Does Juniper MX/ACX series support it?
I know for sure Cisco doesn't.

--
Tassos

Re: interesting troubleshooting

[Christopher Morrow]

(skipping up the thread some)

On Fri, Mar 20, 2020 at 5:58 PM Jared Mauch  wrote:
> It’s the protocol 50 IPSEC VPNs.  They are very sensitive to path changes and 
> reordering as well.
>
> If you’re tunneling more than 5 or 10Gb/s of IPSEC it’s likely going to be a 
> bad day when you find a low speed link in the middle.  Generally providers 
> with these types of flows have both sides on the same network vs going 
> off-net as they’re not stable on peering links that might change paths.

a bunch of times the advice given to folk in this situation is: "Add
more entropy", which really for ipsec/gre/etc vpns means more
endpoints.
For instance, adding 3 more ips on either side for tunnel
egress/ingress will make the flows (ideally) smaller and more probable
to hash across different links in the intermediary network(s).  This
also moves the loadbalancing back behind the customer prem so ideally
perhaps even the nxM flows are now balanced a little better as well.

sometimes this works, sometimes it's hard to accomplish :(

Re: interesting troubleshooting

[Saku Ytti]

On Sat, 21 Mar 2020 at 18:55, Tassos Chatzithomaoglou
 wrote:

> I still don't understand why the vendors cannot make it work in one direction 
> only (the low-end platform would only need to remove an extra label, no need 
> to inspect traffic).
> That would help us a lot, since the majority of our traffic is downstream to 
> the customer.

It is signalled separately for TX and RX and some vendors do allow you
to signal it separately.

-- 
  ++ytti

Re: interesting troubleshooting

[Tassos Chatzithomaoglou]

Mark Tinka wrote on 21/3/20 18:15:
> So the three or four times we tried to get FAT going (in a multi-vendor
> network), it simply didn't work.
>
> Have you (or anyone else) had any luck with it, in practice?
>
> Mark.
>

Only between Cisco boxes.

I still don't understand why the vendors cannot make it work in one direction 
only (the low-end platform would only need to remove an extra label, no need to 
inspect traffic).
That would help us a lot, since the majority of our traffic is downstream to 
the customer.


--
Tassos

Re: interesting troubleshooting

[Saku Ytti]

On Sat, 21 Mar 2020 at 18:19, Mark Tinka  wrote:

> So the three or four times we tried to get FAT going (in a multi-vendor
> network), it simply didn't work.

Yeah we run it in a multivendor network (JNPR, CSCO, NOK), works.

I would also recommend people exclusively using CW+FAT and disabling
LSR payload heuristics (JNPR default, but by default won't do with CW,
can do with CW too).

-- 
  ++ytti

Re: COVID-19 vs. our Networks

[Mark Tinka]

On 21/Mar/20 13:28, Florian Weimer wrote:

>
> 4K isn't supported by all devices and plans.  I'm not sure what kind
> of savings you can actually realize there.  It could be that 4K
> content isn't worth caching near the edge.  Then ditching 4K could
> still have a significant effect despite relatively low usage by
> subscribers.  Similarly anything that reduces content diversity (like
> serving only one category of 1080p streams).

In South Africa, the majority of the population does not own 4K-capable
TV's.

Also, most people do not have access to FTTH services. And for many that
do, having a 25Mbps slot lying around for 4K Netflix is even less common.

That said, a recent survey in the country indicated that the majority of
Netflix subscribers that were polled subscribed to the 4K package. It
wasn't clear whether what they actually wanted as 4K capability or the
ability to support 4 simultaneous streams. Personally, I suspect the latter.


>   Reportedly, the issue is
> backhaul capacity for some CDN nodes in Europe, and not capacity from
> the local cache to the subscriber, but I do not have any direct
> knowledge of that.

It could go either way, but the reason the cache-fill theory is one I do
not necessarily think will create a bottleneck is because Netflix push
content to OCA's or public clusters during off-peak times.

Pressure is more likely to be placed on the edge and the last mile, if
that; but that comes back to why the customers want to spend their own
money, and not having Command & Control tell them how, or why.

Mark.

Re: interesting troubleshooting

[Mark Tinka]

On 21/Mar/20 09:58, Saku Ytti wrote:


> No.
>
> FAT adds additional MPLS label for entropy, ingressPE calculates flow
> hash, based on traditional flow keys and injects that flow number as
> MPLS label, so transit LSR can use MPLS labels for balancing, without
> being able to parse the frame. Similarly VPN provider could do that,
> and inject that flow hash as SPORT at the time of tunneling, by
> looking at the inside packet. And any defensive VPN provider should do
> this, as it would be a competitive advantage.
> Now for some vendors, like Juniper and Nokia transit LSR can look
> inside pseudowire L3 packet for flow keys, so you don't even need FAT
> for this. Some other like ASR9k cannot, and you'll need FAT for it.
>
> But all of this requires that there is entropy to use, if it's truly
> just single fat flow, then you won't balance it. Then you have to
> create bias to the hashResult=>egressInt table, which by default is
> fair, each egressInt has same amount of hashResults, for elephant
> flows you want the congested egressInt to be mapped to fewer amount of
> hashResults.

So the three or four times we tried to get FAT going (in a multi-vendor
network), it simply didn't work.

Have you (or anyone else) had any luck with it, in practice?

Mark.

Re: China’s Slow Transnational Network

[Mark Tinka]

On 21/Mar/20 09:09, Pengxiong Zhu wrote:

> How do they deliberately congest peering ports? Do you hear from those
> Chinese operators or you observe this from the traffic?

Simple - let them run at 350% of capacity and pipeline upgrades for Lord
knows how long :-).

On a serious note, let's have a beer.


> Seems like you also think GFW is part of the cause,

I do - each of my trips to China have questioned the role of my VPN for
my online experience.


> however, we don't have direct evidence.

I won't argue with you there, you did the groundwork. I'm just being
anecdotal.


> Just curious, What is your "problems"? I thought it's congestion.

Accessibility and penetration rates.

Mark.

Re: It's not about the congestion, it's about the profit motive driving the industry

[Saku Ytti]

On Sat, 21 Mar 2020 at 00:15, Matthew Petach  wrote:

> who finds it appalling that we consider it more important to make
> money than to save lives.  :(

If we (me included) would be half as angry about those who have less
than we have, as we are about those who have more than we have,
inequality wouldn't exist. We are the beneficiaries of immense
suffering of millions of people, things are artificially cheap for us
to buy at human cost 'somewhere else'. We are not the heroes of this
story.


-- 
  ++ytti

Re: It's not about the congestion, it's about the profit motive driving the industry

[Livingood, Jason]

> Internet congestion is a symptom, not the cause of this thread.

[JL] I'm wondering if one of the issues is problems with legacy TCP congestion 
control algorithms. The industry has been poking at that for awhile and 
approaches range from BBR to fq_codel. This is worth exploring a bit more IMO.

Fwd: Re: COVID-19 vs. our Networks

[Alexandre Petrescu]

(photo removed, the admins have it, dont ask me in private)



 Message transféré 
Sujet : Re: COVID-19 vs. our Networks
Date :  Sat, 21 Mar 2020 14:20:56 +0100
De :Alexandre Petrescu 
Pour :  nanog@nanog.org




LF/HF

Le 21/03/2020 à 12:28, Florian Weimer a écrit :

* Mike Hammett:

[...]
Relaxing copyright and patent restrictions might also help, at least
in the medium term.


I agree.

Related to copyright and patent restrictions, is this:

Currently the situation is that I cant get to see the ARN (acid rybo  
nucleic) sequence of this virus; the data is present on gisaid.org, 
situated in Germany, at MAx Planck Institute, but my request to see it 
did not succeed - silence.


FRiends tell me that silence is normal because I am not a specialist, 
and those who are specialist do have access.


I disagree.  I want to know what it looks like.  This virus with a crown 
might hurt me as much as it might hurt the specialists, there is no 
difference, we are all Sapiens.


I want to know what it looks like.

The only thing I could find is during a TV news report, see below.

That sequence of A, T, G, C, and their particular ordering, is what 
makes it bad bad.  That's the enemy.


Alex, LF/HF 2 (it means low stress)

Fwd: Your message to NANOG awaits moderator approval

[Alexandre Petrescu]

 Message transféré 
Sujet : Your message to NANOG awaits moderator approval
Date :  Sat, 21 Mar 2020 13:21:18 +
De :nanog-ow...@nanog.org
Pour :  alexandre.petre...@gmail.com



Your mail to 'NANOG' with the subject

Re: COVID-19 vs. our Networks

Is being held until the list moderator can review it for approval.

The reason it is being held:

Message body is too big: 359842 bytes with a limit of 100 KB

Either the message will get posted to the list, or you will receive
notification of the moderator's decision. If you would like to cancel
this posting, please visit the following URL:

https://mailman.nanog.org/mailman/confirm/nanog/373bcb041f05344153379883b27b33a2107eafc1

Re: CISA: Guidance on the Essential Critical Infrastructure Workforce

[Alexandre Petrescu]

Le 21/03/2020 à 08:37, Bill Woodcock a écrit :

In France I must show a paper (not smartphone) printed permit, each
sortie one different paper.  The receiver of it (police) takes it in
his/her gloved hands then s/he passes it back to me.  I do not have
gloves.  I wished the receiver did not use the same gloves for each
pereson who passes by and delivers that paper to him.

Yep, couldn't believe it when my mate in Lyon told me the same thing
this week.
But I suppose this was to be expected, and is an idea that could
potentially spread, worldwide.

I’ve been in Paris all week, and have gone out, on average, once a day.  I 
pre-printed a stack of already-filled-out forms at the beginning of the week, 
so I’ve just checked the appropriate box each time I’ve gone out, no big deal.  
Seems quite reasonable to me.  Gets people to at least give some conscious 
thought as to whether their reason for going out actually meets one of the 
listed criteria.  And I haven’t actually been stopped any of the times I’ve 
gone out.

It’s early days yet, but Paris is handling this way, way better than I’d have 
expected.



YEs, it's early days.  For the next days, we need forecast, like in 
weather forecast.  There is no public forecast of this pandemy.


But there is data.  Data is more than just numbers of cases in various 
web sites.  There is also correlation of cases and the dates and levels 
of declaration of confinement.  The levels of confinement that arrive 
are relatively similar but with different names: close borders, close 
restaurants, close schools, close City level 1, close City level 2, 
close City level max.  Each of these levels has a date, but the precise 
date is not centralized somewhere, and worse - not public.  One has to 
watch thousands of public announcements to understand them.  Or to ask 
those who one knows in that particular country, based on confidence.


This is my forecast based on yesterday's public statements on TV from 
Authority and other public data sources including China and Italy, by 
revese engineering and local data to understand French:


   The peak of the wave (biggest number of new cases a day) in France
   will arrive at earliest somewhere between MArch 26th and MArch
   29th.  The length of the peak is about 10 to 22 days.  At earliest
   we start going down the wave starting April 5th, and at latest we
   start going down that wave on April 22nd.  This going down might be
   a rough descent (Codogno city in Italy had 0 cases after 2 weeks
   total confinement), or much slower (China Wuhan 0 cases yesterday,
   but China total increase New Cases still). That is the horizon.

That might change for the better or for worse with (1) 'mutations' (a 
word I dont understand),  with (2) the hopeful medication (Chloroquine 
of enterprise Sanofi, France; and Favipiravir, China Authority; and 
other molecules invoked by Doctors) and (3) capacity of people to 
understand restrictions to stay home, understand how propagation and 
transmission works and (4) capacity of law enforcement to enforce 
restrictions.


To evaluate how a group of people understands what's happening, there 
are simple questions: is covid-19 a virus or an illness?  is this an 
epidemy or a pandemy?  One can compare that with how we went through 
understanding of AIDS, HIV (SIDA, VIH in French) and the relevant 
protections (condoms initially, tri-therapy these days if I understand 
it correctly).  One can compare that to how we understand SARS (SRAS in 
France), H1N1, H5N1 terms.


If we communicate these terms meaningfully then we understand one 
another meaningfully.


There is no reason to compare this covid situation to 9/11 - this is 
more like a wave, that was more like shock and then go down slowly.


It comes, slowly, but it comes.

Alex, LF/HF 2


And a giant thumbs up to Free, who are keeping my 10G broadband flying along at 
an actual, measurable, 10G.



YEs, lucky you.  Me I am on Free's ADSL 5mbit/s, I cant dare to upload 
all significant data to youtube, I refrain for later to do that.  It 
saves me energy :-)


LEt me add this to relate to Guidance in the topic of the email. There 
is a "handbook-covid19-prevention-treatment-China" in pdf format that 
someone sent me.  68 pages, 34mbyte.  I did not read it.  I received 
also other guides in pdf format dedicated to hospital person (the one 
that takes care of the ill person), but they are in other languages than 
French and English.  I think they might all be on the Internet, and I 
hope people can access them freely, without intermediaries, and that 
they take time when appropriate.


Alex



 -Bill

Re: COVID-19 vs. our Networks

[Alexandre Petrescu]

Le 21/03/2020 à 03:42, Mark Tinka a écrit :


On 20/Mar/20 19:38, Rich Kulawiec wrote:


+100.

In all the decades that I've been here (on the 'nets), the saddest change
I've seen is the lack of responsibility on the part of people who have,
by virtue of their positions, been given incredible power.  This is the
time for those people to step up and (try to) do the right thing.

None of us know what's going to be needed.  How could we?  We could guess,
and we *are* guessing, but we don't really know because we're sailing
off the edge of the map now.

In those circumstances, the virtue of frugality -- a sensible thing
at any time -- now becomes a necessity.  Every single one of us should
be doing whatever we can to prepare for the unknown, and conserving
resources is one part of that.

"Everything we do before a pandemic will seem alarmist.
Everything we do after will seem inadequate."
--- Michael Leavitt, former HHS Secretary

As I write this, doctors and nurses are working without PPE, risking
their own wellbeing to try to save patients.  We're not being asked
to do anything like that.  Hopefully we still have enough left to rise
to the comparatively minor challenge in front of us.

All I'm saying is at the moment, there is no empirical information to
suggest that Netflix will break what's left of the Internet. Nor is
there any empirical information suggesting that singling them out will
help keep it going.



I tend to agree - I dont think there is any capacity problem in the core 
network or server platforms, including netflix.  I do not see it for my 
part as of now.  I am an end user, not a Network sysadmin.


I heard about EU measures to attenuate such a problem tha tmight arrive 
- I think it is mis-informed.


I also heard EC (European COmmission) looking to push all efforts in 
robotics, how could robotics help this, several ideas like drones, or 
the open source respiratory device, or why not sending a robot do 
shopping for me.  I think these look far fetched but are promissing.


Alex, LF/HF 2 (means low stress)




If we go down this path, who's to say which service provider will or
won't be "targeted" next at the whim of some command & control policy
maker? Is it a rabbit hole whose top-soil we want to uncover?
  
If/when the network starts to take a hit, network operators will

respond. But if there is any operator on this list who is willing to
raise their hands and say, "Netflix is breaking my network",
uncongested, free-flowing beer on me when we all come out from the bunkers.

Mark.

Re: CISA: Guidance on the Essential Critical Infrastructure Workforce

[Alexandre Petrescu]

LF/HF

Le 21/03/2020 à 03:24, Mark Tinka a écrit :


On 20/Mar/20 15:53, Alexandre Petrescu wrote:


In France I must show a paper (not smartphone) printed permit, each
sortie one different paper.  The receiver of it (police) takes it in
his/her gloved hands then s/he passes it back to me.  I do not have
gloves.  I wished the receiver did not use the same gloves for each
pereson who passes by and delivers that paper to him.


Yep, couldn't believe it when my mate in Lyon told me the same thing
this week.

But I suppose this was to be expected, and is an idea that could
potentially spread, worldwide.



Today first time I see in FRance TV news one interviewer puts one 
time-use covers over the microphones when interviewing people. (it's a 
one time cover in addition to the typical windshield which is expensive 
and cant be changed each time).  Only today.  Many days lost, many 
viruses spread.


Still on tables on all TV channels I watch they dont have covers on 
microphones.


Alex



Mark.

Re: COVID-19 vs. our Networks

[Mike Hammett]

Unless the IX or OCA feed goes to the DSLAM, node, tower... no. 




- 
Mike Hammett 
Intelligent Computing Solutions 

Midwest Internet Exchange 

The Brothers WISP 

- Original Message -

From: "Mark Tinka"  
To: nanog@nanog.org 
Sent: Friday, March 20, 2020 9:22:45 PM 
Subject: Re: COVID-19 vs. our Networks 




On 20/Mar/20 15:52, Mike Hammett wrote: 



Some of the pipes Netflix goes through is also used by other services that 
aren't as adaptable. 



I think that's case specific on the type of network you have built, and whether 
your feed your customers Netflix content with on on-site OCA or via an exchange 
point or transit link. 

Mark. 

Re: COVID-19 vs. our Networks

[Florian Weimer]

* Mike Hammett:

> Netflix recommends 25 megs for Ultra HD, while only 5 megs for
> HD. That's a 5x difference in something people likely won't notice
> and would make a big difference on the additional VPN, VoIP, video
> conferencing, etc.

4K isn't supported by all devices and plans.  I'm not sure what kind
of savings you can actually realize there.  It could be that 4K
content isn't worth caching near the edge.  Then ditching 4K could
still have a significant effect despite relatively low usage by
subscribers.  Similarly anything that reduces content diversity (like
serving only one category of 1080p streams).  Reportedly, the issue is
backhaul capacity for some CDN nodes in Europe, and not capacity from
the local cache to the subscriber, but I do not have any direct
knowledge of that.

Relaxing copyright and patent restrictions might also help, at least
in the medium term.

Re: interesting troubleshooting

[Saku Ytti]

On Sat, 21 Mar 2020 at 04:20, Steve Meuse  wrote:

> What that large flow in a single LSP? Is this something that FAT lsp would 
> fix?

No.

FAT adds additional MPLS label for entropy, ingressPE calculates flow
hash, based on traditional flow keys and injects that flow number as
MPLS label, so transit LSR can use MPLS labels for balancing, without
being able to parse the frame. Similarly VPN provider could do that,
and inject that flow hash as SPORT at the time of tunneling, by
looking at the inside packet. And any defensive VPN provider should do
this, as it would be a competitive advantage.
Now for some vendors, like Juniper and Nokia transit LSR can look
inside pseudowire L3 packet for flow keys, so you don't even need FAT
for this. Some other like ASR9k cannot, and you'll need FAT for it.

But all of this requires that there is entropy to use, if it's truly
just single fat flow, then you won't balance it. Then you have to
create bias to the hashResult=>egressInt table, which by default is
fair, each egressInt has same amount of hashResults, for elephant
flows you want the congested egressInt to be mapped to fewer amount of
hashResults.

-- 
  ++ytti

Re: interesting troubleshooting

[Saku Ytti]

Hey Matthew,

> There are *several* caveats to doing dynamic monitoring and remapping of
> flows; one of the biggest challenges is that it puts extra demands on the
> line cards tracking the flows, especially as the number of flows rises to
> large values.  I recommend reading
> https://www.juniper.net/documentation/en_US/junos/topics/topic-map/load-balancing-aggregated-ethernet-interfaces.html#id-understanding-aggregated-ethernet-load-balancing
> before configuring it.

You are confusing two features. Stateful and adaptive. I was proposing
adaptive, which just remaps the table, which is free, it is not flow
aware. Amount of flow results is very small bound number, amount of
states is very large unbound number.

-- 
  ++ytti

Re: CISA: Guidance on the Essential Critical Infrastructure Workforce

[Bill Woodcock]

>> In France I must show a paper (not smartphone) printed permit, each
>> sortie one different paper.  The receiver of it (police) takes it in
>> his/her gloved hands then s/he passes it back to me.  I do not have
>> gloves.  I wished the receiver did not use the same gloves for each
>> pereson who passes by and delivers that paper to him.
> 
> Yep, couldn't believe it when my mate in Lyon told me the same thing
> this week.
> But I suppose this was to be expected, and is an idea that could
> potentially spread, worldwide.

I’ve been in Paris all week, and have gone out, on average, once a day.  I 
pre-printed a stack of already-filled-out forms at the beginning of the week, 
so I’ve just checked the appropriate box each time I’ve gone out, no big deal.  
Seems quite reasonable to me.  Gets people to at least give some conscious 
thought as to whether their reason for going out actually meets one of the 
listed criteria.  And I haven’t actually been stopped any of the times I’ve 
gone out.

It’s early days yet, but Paris is handling this way, way better than I’d have 
expected.

And a giant thumbs up to Free, who are keeping my 10G broadband flying along at 
an actual, measurable, 10G.

-Bill



signature.asc
Description: Message signed with OpenPGP

Re: China’s Slow Transnational Network

[Pengxiong Zhu]

>
> I know about Chinese operators who will deliberately congest peering ports
> to influence 3rd party network behaviour.

How do they deliberately congest peering ports? Do you hear from those
Chinese operators or you observe this from the traffic?

Most countries in Africa do not implement great big firewalls. Our problems
> are quite different :-\...

Not having great big firewalls tends to help :-).
>

Seems like you also think GFW is part of the cause, however, we don't have
direct evidence. Just curious, What is your "problems"? I thought it's
congestion.

Best,
Pengxiong Zhu
Department of Computer Science and Engineering
University of California, Riverside


On Sun, Mar 15, 2020 at 11:13 PM Mark Tinka  wrote:

>
>
> On 15/Mar/20 22:51, Frank Habicht wrote:
>
> >
> > thanks for the "quotes", Mark. I agree.
> >
> >
> https://www.caida.org/publications/presentations/2018/investigating_causes_congestion_african_afrinic/investigating_causes_congestion_african_afrinic.pdf
> >
> > page 23:
> > Results Overview
> > • No evidence of widespread congestion
> >- 2.2% of discovered link showed evidence of congestion at the end of
> >  our measurements campaign
> >
> > page 34:
> > Conclusions
> > • Measured IXPs were congestion-free, which promotes peering in the
> >   region
> >
> > https://conferences.sigcomm.org/imc/2017/papers/imc17-final182.pdf
> >
> > my conclusion: s/congestion/congestion or the lack thereof/g
> >
> > Frank Habicht
> >
> > PS: yes, i could name peers that once had inadequate links into an IXP.
> > but for how long did that happen? (yes..., any minute is too long...)
>
> Indeed.
>
> There was a time when backhaul links between ISP routers at the exchange
> point and their nearest PoP were based on E1's, wireless, e.t.c. But
> that could be said of, pretty much, every exchange point that kicked off
> inside of the last 2.5 decades.
>
> Nowadays, such links, if they exist, are the very deep exception, not
> the rule.
>
> Mark.
>