Re: deploying RPKI based Origin Validation

[Joshua Vijsma / True]

Hi all,

Just wanted to share our (AS15703, True B.V.) experience as a hosting provider 
with enabling RPKI invalid filtering (invalid == reject). We've secured (most 
of) our routes since 2014 with ROAs but last Tuesday we have deployed filters 
which reject RPKI invalid routes. We have had two tickets regarding users in 
one certain RPKI invalid prefix not being able to reach our network, but those 
people quickly understood that this wasn't our problem but a problem with their 
hosting partner. They took it up with their hosting partner and it was fixed 
within a day. Overal, I would certainly recommend filtering RPKI invalids (and 
create ROAs for your prefixes!!) to prevent hijacks.

-- 
Met vriendelijke groet / Best regards,

Joshua Vijsma


> On 13 Jul 2018, at 14:00, nanog-requ...@nanog.org wrote:
> 
> 
> --
> 
> Message: 2
> Date: Thu, 12 Jul 2018 17:50:29 +
> From: Job Snijders 
> To: nanog@nanog.org
> Subject: deploying RPKI based Origin Validation
> Message-ID: <20180712175029.gc3...@vurt.meerval.net>
> Content-Type: text/plain; charset=us-ascii
> 
> Hi all,
> 
> I wanted to share with you that a ton of activity is taking place in the
> Dutch networker community to deploy RPKI based BGP Origin Validation.
> The mantra is "invalid == reject" on all EBGP sessions.
> 
> What's of note here is that we're now seeing the first commercial ISPs
> doing Origin Validation. This is a significant step forward compared to
> what we observed so far (it seemed OV was mostly limited to academic
> institutions & toy networks). But six months ago Amsio 
> (https://www.amsio.com/en/)
> made the jump, and today Fusix deployed (https://fusix.nl/deploying-rpki/).
> 
> We've also seen an uptake of Origin Validation at Internet Exchange
> route servers: AMS-IX and FranceIX have already deployed. I've read that
> RPKI OV is under consideration at a number of other exchanges.
> 
> Other cool news is that Cloudflare launched a Certificate Transparency
> initiative to help keep everyone honest. Announcement at:
> https://twitter.com/grittygrease/status/1017224762542587907
> Certificate Transparency is a fascinating tool, really a necessity to
> build confidence in any PKI systems.
> 
> Anyone here working to deploy RPKI based Origin Validation in their
> network and reject invalid announcements? Anything of note to share?
> 
> Kind regards,
> 
> Job

Re: Quickstart Guide to IRR/RPSL

[Kenneth Finnegan]

On Thu, Jul 19, 2018 at 9:19 AM, Job Snijders  wrote:
> Excellent, have you also considered using ARIN-WHOIS and RPKI as data
> sources for your route servers? An excellent tool to generate route
> server configurations is 'arouteserver' http://arouteserver.readthedocs.io/

After the volume of time and level of frustration it took just to get
a handle on IRR, RPKI has been placed very low on our priority list.
It's still on our list, but we've got plenty of other work to do
before I figure out the implications of turning on RPKI, and I'm not
willing to turn on a prefix authentication that I'm not able to walk
my members through a way to correct.

As for ARIN-WHOIS, I think I had gotten confused whether it was
additive or exclusive of IRR objects for allowing prefixes. We have a
lot of members running prefixes off of letters of authorization, so
rejecting routes based on ARIN-WHOIS wasn't attractive, but that
documentation you linked to reads more like it'll accept prefixes in
addition to what gets accepted based on IRR alone. We will experiment
with it.

And yes, arouteserver is an excellent tool; we're currently using it,
so enabling RPKI and ARIN-WHOIS will be relatively easy once we're
willing to qualify it and roll it out.

> 1/ About the "Selecting an IRR Database" section, the best current
> practise is to use the IRR that your RIR provides you. In other words:
> register ARIN managed prefixes in the ARIN IRR, and RIPE managed
> prefixes in the RIPE IRR. Only the RIRs are in a position to attest that
> it was the owner of a prefix that created the route(6): object.

This is true. I wanted to write a more evergreen guide than walking
through the current implementation of a single region's RIR, since
there's several of them with their own unique work-flows and I get the
sense that several of them are currently in flux. If we could
ultimately replace this whitepaper with links to five equivalent
documents from the RIRs, so be it.

> Databases such as RADB, NTTCOM, ALTDB are so-called "third party"
> databases and at this moment in time have no way to validate anything.
> I've highlighted the differences between the various sources of data in
> this talk at RIPE76 https://ripe76.ripe.net/archives/video/22

I did eventually find that talk useful once I had climbed high enough
on the learning curve to really understand what you were talking
about. Thanks.

> 2/ I'd delete the "Step 2: Document Your Autonomous System’s Routing
> Policy" step, nobody uses this.

Is the expectation that the only source of a network's as-set is PeeringDB then?

I have reason to believe there are IRR consumers who do parse
export/mp-export statements. I think at least documenting an mp-export
to AS-ANY policy is reasonable, but I'll reconsider that.

> 4/ Step 4 can be extended to promote creating RPKI ROAs (in addition to)
> the route objects. Anyone can create a route: object in ALTDB, but only
> the owner of a prefix can create a RPKI ROA. As such the RPKI ROAs are a
> far more valuable source of data to filter generators.

Yes. Someone should write a "zero to RPKI" tutorial.

After this whitepaper literally took sitting down and reading the RFCs
for something that people bemoan isn't used by every network, I'm not
excited to try and get the same handle on RPKI to be able to speak
with authority on how to set it up.

> Can I update http://peering.exposed/ and add FCIX with a 'yes' to both
> secure route servers & BCP 214? :-)

Please do. :-) $0 for 10G, N/A for 100G.


The next IRR puzzle for us is converting a CSV of member ASNs to their
as-sets to generate the requested AS33495:AS-MEMBERS as-set so our
members can also generate filters against the route servers. It seems
like there's probably a tool like bgpq3 that can turn a list of ASNs
into an as-set of their exports, but I'm not seeing it. Anyone have
something at hand, or am I breaking out the python soon?
--
Kenneth Finnegan
http://blog.thelifeofkenneth.com/

Re: Proving Gig Speed

[Scott Weeks]

--- mark.ti...@seacom.mu wrote:
From: Mark Tinka 
On 19/Jul/18 22:43, Scott Weeks wrote:

> I know we're talking about Africa and other less well 
> connected countries, but good luck with that in the 
> Pacific, which covers about 1/3 of the planet.
>
> https://oceanexplorer.noaa.gov/facts/pacific-size.html

Yeah, things and people tend to work better if hosted 
on land...
-


I believe I'm on land here... ;)  And there're a lot of
islands with lots of people out here.  1/3 of the planet 
and all. :)
--


What I meant to say is a lot of folks get connectivity 
through satellite.  500msec plus and jitter to spare.

Further it's expensive and all the 'busy' sites cost a
lot of money to download the stuff folks on this list
don't blink an eye at and you can't turn it off.  I 
believe satellite coverage serves a lot of the planet's 
population.

scott

Re: Proving Gig Speed

[Scott Weeks]

--- mark.ti...@seacom.mu wrote:
From: Mark Tinka 
On 19/Jul/18 22:43, Scott Weeks wrote:

> I know we're talking about Africa and other less well 
> connected countries, but good luck with that in the 
> Pacific, which covers about 1/3 of the planet.
>
> https://oceanexplorer.noaa.gov/facts/pacific-size.html

Yeah, things and people tend to work better if hosted 
on land...
-


I believe I'm on land here... ;)  And there're a lot of
islands with lots of people out here.  1/3 of the planet 
and all. :)

scott

NTT now treats RPKI ROAs as IRR route(6)-objects

[Job Snijders]

Dear NANOG,

[ TL:DR - From now on, NTT / AS 2914 allows customers to either register
their announcements in the IRR, or as RPKI ROAs. This is a convenience
service for relevant regions of the world where IRR is not the norm but
RPKI is commonly available. Previously NTT only accepted IRR and
ARIN-WHOIS. I hope competitors and partners will use this approach too! ]

As most of you know, the Resource Public Key Infrastructure (RPKI) is a
modern reimagination of the good ole' IRR system we have come to love
and hate. The main advantage of the RPKI is that a consumer of the data
can cryptographically verify whether it was the actual owner of the IP
prefix that created a so-called "RPKI ROA".
(more reading: https://en.wikipedia.org/wiki/Resource_Public_Key_Infrastructure)

Given that RPKI ROAs are somewhat equivalent to IRR route-objects, but
more reliable in terms of authoritativeness, NTT now has an automated
nightly process that converts RPKI information into IRR format so that
our toolchain can consume the data as if it were just another IRR
source.

Using RPKI ROAs as if they are IRR route(6)-objects is a transitional
step towards increased security in the routing ecosystem.

We are not the first to explore this method (see post-scriptum), but I
think we are the first to republish elements from RPKI ROAs via a
publicly accessible IRRd instance queryable with the RADB IRRd protocol.
This means that anyone that points their bgpq3 or peval programs at
rr.ntt.net can leverage this method without having to update anything
else in the pipeline.

An example can be inspected here:

job@eng0 ~$ whois -h rr.ntt.net 193.0.6.139
[Querying rr.ntt.net]
[rr.ntt.net]
route:  193.0.0.0/21
descr:  RIPE-NCC
origin: AS
mnt-by: RIPE-NCC-MNT
changed:unr...@ripe.net 2101
source: RIPE
remarks:
remarks:* THIS OBJECT IS MODIFIED
remarks:* Please note that all data that is generally regarded as 
personal
remarks:* data has been removed from this object.
remarks:* To view the original object, please query the RIPE 
Database at:
remarks:* http://www.ripe.net/whois
remarks:

route:  193.0.0.0/21
descr:  RPKI ROA for 193.0.0.0/21
remarks:This route object represents routing data retrieved from the 
RPKI
remarks:The original data can be found here: 
https://rpki.gin.ntt.net/r/AS/193.0.0.0/21
remarks:This route object is the result of an automated RPKI-to-IRR 
conversion process.
remarks:maxLength 21
origin: AS
mnt-by: MAINT-JOB
changed:j...@ntt.net 20180718
source: RPKI  # Trust Anchor: RIPE NCC RPKI Root
job@eng0 ~$

The first object is an actual IRR "route:" object from the RIPE NCC
operated IRR, the second object is a representation of the RPKI ROA in
RPSL format published via rr.ntt.net.

In general we can state that RPKI data is very good quality data,
however please keep in mind that it may not be /correct/ data. In this
context "Good Quality" means that it cannot easily be forged or tampered
with by adversaries (but of course that doesn't protect the legitimate
owner against making misconfigurations). Just like with IRR
route(6)-objects, owners may input the wrong origin ASN in this type of
object or configure the wrong MaxLength.

NTT operates a "RPKI Cache Validator" at https://rpki.gin.ntt.net/
Everyone is free to inspect and click around in this webinterface.
Instead of https://rpki.gin.ntt.net/, there also is a command line
interface available via BGPMon's whois service: 

job@vurt ~$ whois -h whois.bgpmon.com 193.0.0.0/21
% This is the BGPmon.net whois Service
% You can use this whois gateway to retrieve information
% about an IP adress or prefix
% We support both IPv4 and IPv6 address.
%
% For more information visit:
% https://portal.bgpmon.net/bgpmonapi.php

Prefix:  193.0.0.0/21
Prefix description:  RIPE-NCC
Country code:NL
Origin AS:   
Origin AS Name:  Reseaux IP Europeens Network Coordination Centre (RIPE 
NCC)
RPKI status: ROA validation successful
First seen:  2011-10-19
Last seen:   2018-07-19
Seen by #peers:  77

Notice in the above output the "ROA validation successful".

Nota bene: the fact that NTT uses RPKI ROA information in the prefix
filter generation process - does _not_ mean that NTT does "RPKI Origin
Validation" for BGP updates (yet). RPKI Origin Validation is an
additional security layer that we hope to deploy in the not too distant
future. Using RPKI ROAs in this way is an important step forward in this
process.

Kind regards,

Job Snijders

Post scriptum on prior work:

Dragon Research Labs implemented the idea in 2015:

Re: deploying RPKI based Origin Validation

[Mark Tinka]

On 19/Jul/18 21:47, Michel Py wrote:

> I understand that; if there is an easier way to do RPKI, people are going to 
> use it instead of the right way. However, I think that the blacklist targets 
> a different kind of customer : the end user. We want the enterprise to 
> certify their prefixes with RPKI and put pressure on their upstreams to 
> deploy it, the more noise we make the better. What I want is my upstreams to 
> give me a clean routing tables without invalids, but it does not happen so in 
> the meantime I'm trying to do what I can with my limited resources.

The script that Job wrote is neat, but I'm sure neither he nor I would
run it in production in lieu of the actual RPKI infrastructure.

Even though you're my competitor, I'd caution against this. But, your
network, your rules.


> The picture from the enterprise is quite different. There is a lot of stuff 
> out there that does not get upgraded, that is not even under a maintenance 
> contract to get the new software, or that is on EOL/EOS hardware.

So don't re-invent this wheel; that is what Delegated RPKI is for.
Several RPKI tools out there support CA functionality, as much as they
support the RP side as well. Let's not create something totally out of
scope to mimic specs and tools already exist.

If you really want to participate in the RPKI, then you seriously need
to consider supporting software that implements it. If not, use your
ISP's CA tools to sign your IP addresses, and then rely on them to have
clean FIB's when you use them for transit.

RPSL got complicated enough with all its good intentions, and we know
how that turned out. Let's not muddy the RPKI waters.

Mark.

Re: Proving Gig Speed

[Mark Tinka]

On 19/Jul/18 22:43, Scott Weeks wrote:

>
> I know we're talking about Africa and other less well 
> connected countries, but good luck with that in the 
> Pacific, which covers about 1/3 of the planet.
>
> https://oceanexplorer.noaa.gov/facts/pacific-size.html

Yeah, things and people tend to work better if hosted on land...

Mark.

Re: Proving Gig Speed

[Scott Weeks]

--- mark.ti...@seacom.mu wrote:
From: Mark Tinka 
On 18/Jul/18 16:58, K. Scott Helms wrote:

> ... What's really interesting is how gaming is changing 
> and within the next few years I do expect a lot of games 
> to move into the remote rendering world.  You need to 
> have <=30 ms of latency to sustain 1080p gaming and 
> obviously jitter and packet loss are also problematic.  
> The traffic is also pretty impressive with spikes of 
> over 50 mbps down and sustained averages over 21 mbps.

And what we need is for them to ensure all these remote 
rendering farms are evenly distributed around the world 
to ensure that 30ms peak latency is achievable.
---


I know we're talking about Africa and other less well 
connected countries, but good luck with that in the 
Pacific, which covers about 1/3 of the planet.

https://oceanexplorer.noaa.gov/facts/pacific-size.html

scott

RE: deploying RPKI based Origin Validation

[Michel Py]

> Mark Tinka wrote :
> but I want to be cautious about encouraging a parallel stream that slows down 
> the deployment of RPKI.

I understand that; if there is an easier way to do RPKI, people are going to 
use it instead of the right way. However, I think that the blacklist targets a 
different kind of customer : the end user. We want the enterprise to certify 
their prefixes with RPKI and put pressure on their upstreams to deploy it, the 
more noise we make the better. What I want is my upstreams to give me a clean 
routing tables without invalids, but it does not happen so in the meantime I'm 
trying to do what I can with my limited resources.

> We generally use typical service provider routers to deliver services. So I'm 
> not sure whether the 3900's you run support it or not.

The picture from the enterprise is quite different. There is a lot of stuff out 
there that does not get upgraded, that is not even under a maintenance contract 
to get the new software, or that is on EOL/EOS hardware.

Michel.

TSI Disclaimer:  This message and any files or text attached to it are intended 
only for the recipients named above and contain information that may be 
confidential or privileged. If you are not the intended recipient, you must not 
forward, copy, use or otherwise disclose this communication or the information 
contained herein. In the event you have received this message in error, please 
notify the sender immediately by replying to this message, and then delete all 
copies of it from your system. Thank you!...

Re: Centurylink/Level3 Plans

[Mehmet Akcin]

Remember Global Crossing & Level3 merger? Perhaps look for an option that
involves neither

There are few good options out there!

Mehmet

On Thu, Jul 19, 2018 at 8:14 PM James Breeden  wrote:

> Does anyone know what the plans or even watercooler chat is concerning
> Centurylink (AS209) and Level3 (AS3356) integration, direct peering, or
> continuing separation?
>
> We are looking at a IP Transit deal involving one or both networks and
> while I'd love to have transit routes from both, I don't want to design to
> be shot in the foot later on either if they are talking about soon-to-be
> integrated or something.
>
> TIA...
>
> James W. Breeden
> Managing Partner
>
> [logo_transparent_background]
> Arenal Group: Arenal Consulting Group | Acilis Telecom | Pines Media
> PO Box 1063 | Smithville, TX 78957
> Email: ja...@arenalgroup.co | office
> 512.360. | cell 512.304.0745 | www.arenalgroup.co<
> http://www.arenalgroup.co>
>
>

Re: Centurylink/Level3 Plans

[Brielle Bruns]

On 7/19/2018 12:19 PM, Joseph Jenkins wrote:

So the reps/SEs/Techs that I have spoken with are saying they are going to
keep them separate. In my case I am tw customer and was worried about their
network when L3 took them over. I was even more worried when Centurylink
then took over L3 that I was going to lose one of the networks. However I
gotten assurances from everyone there that they are going to maintain all 3
networks as separate for the foreseeable future.



They were also forced to divest some of their L3 assets in various 
places - like here in Boise where they sold the L3 part to Syringa Networks.


We've got fiber and BGP from CL in Boise, and our paths still seem to 
only take CL transit and don't happen to hit L3 unless its a specific 
customer on L3.



--
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org/ http://www.ahbl.org

Re: Quickstart Guide to IRR/RPSL

[Job Snijders]

On Thu, Jul 19, 2018 at 11:19:12AM -0700, Kenneth Finnegan wrote:
> As for ARIN-WHOIS, I think I had gotten confused whether it was
> additive or exclusive of IRR objects for allowing prefixes. 

Indeed, in arouteserver it is 'additive'. Documentation from ARIN is
here: 
https://teamarin.net/2016/07/07/origin-as-an-easier-way-to-validate-letters-of-authority/

> > 2/ I'd delete the "Step 2: Document Your Autonomous System’s Routing
> > Policy" step, nobody uses this.
> 
> Is the expectation that the only source of a network's as-set is
> PeeringDB then?

Yes, or the IX/transit operator can ask what AS-SET to use during the
turn-up of the circuit.

> I have reason to believe there are IRR consumers who do parse
> export/mp-export statements. I think at least documenting an mp-export
> to AS-ANY policy is reasonable, but I'll reconsider that.

Globally I think there are only 2 or 3 organisations left that parse
this information. The vast majority either autodiscovers via peeringdb,
or just explicitly asks for it during provisioning.

> > Can I update http://peering.exposed/ and add FCIX with a 'yes' to
> > both secure route servers & BCP 214? :-)
> 
> Please do. :-) $0 for 10G, N/A for 100G.

Excellent, done.

> The next IRR puzzle for us is converting a CSV of member ASNs to their
> as-sets to generate the requested AS33495:AS-MEMBERS as-set so our
> members can also generate filters against the route servers. It seems
> like there's probably a tool like bgpq3 that can turn a list of ASNs
> into an as-set of their exports, but I'm not seeing it.

bgpq3 can only go from IRR sources (using the RADB IRRd protocol) to
outputs such as Cisco, Juniper, BIRD, JSON - not the other way around.

> Anyone have something at hand, or am I breaking out the python soon?

Go python

Kind regards,

Job

Re: Centurylink/Level3 Plans

[Joseph Jenkins]

So the reps/SEs/Techs that I have spoken with are saying they are going to
keep them separate. In my case I am tw customer and was worried about their
network when L3 took them over. I was even more worried when Centurylink
then took over L3 that I was going to lose one of the networks. However I
gotten assurances from everyone there that they are going to maintain all 3
networks as separate for the foreseeable future.


On July 19, 2018 at 11:15:39 AM, James Breeden (ja...@arenalgroup.co) wrote:

Does anyone know what the plans or even watercooler chat is concerning
Centurylink (AS209) and Level3 (AS3356) integration, direct peering, or
continuing separation?

We are looking at a IP Transit deal involving one or both networks and
while I'd love to have transit routes from both, I don't want to design to
be shot in the foot later on either if they are talking about soon-to-be
integrated or something.

TIA...

James W. Breeden
Managing Partner

[logo_transparent_background]
Arenal Group: Arenal Consulting Group | Acilis Telecom | Pines Media
PO Box 1063 | Smithville, TX 78957
Email: ja...@arenalgroup.co | office
512.360. | cell 512.304.0745 | www.arenalgroup.co<
http://www.arenalgroup.co>

Centurylink/Level3 Plans

[James Breeden]

Does anyone know what the plans or even watercooler chat is concerning 
Centurylink (AS209) and Level3 (AS3356) integration, direct peering, or 
continuing separation?

We are looking at a IP Transit deal involving one or both networks and while 
I'd love to have transit routes from both, I don't want to design to be shot in 
the foot later on either if they are talking about soon-to-be integrated or 
something.

TIA...

James W. Breeden
Managing Partner

[logo_transparent_background]
Arenal Group: Arenal Consulting Group | Acilis Telecom | Pines Media
PO Box 1063 | Smithville, TX 78957
Email: ja...@arenalgroup.co | office 512.360. 
| cell 512.304.0745 | www.arenalgroup.co

Re: Quickstart Guide to IRR/RPSL

[Job Snijders]

Dear Kenneth,

On Wed, Jul 18, 2018 at 09:38:23PM -0700, Kenneth Finnegan wrote:
> As part of setting up a new Internet Exchange in Fremont, California,
> we've been investigating prefix filtering on the route servers based
> on IRR.

Excellent, have you also considered using ARIN-WHOIS and RPKI as data
sources for your route servers? An excellent tool to generate route
server configurations is 'arouteserver' http://arouteserver.readthedocs.io/

> Unfortunately, we were not satisfied with any of the existing
> documentation available online as far as taking a network engineer
> from "zero" to "just enough IRR to post our prefixes for filtering".
> Lacking this documentation, it was rather unreasonable for us to turn
> on filtering and drop most of our peer's prefixes without a relevant
> whitepaper to point them to to get started.
> 
> So, we wrote our own guide:
> http://fcix.net/whitepaper/2018/07/14/intro-to-irr-rpsl.html

This is excellent reasoning, there indeed is a lack of easy to consume
information.

> I thought others on this list would find our whitepaper interesting,
> and/or have valuable feedback based on their experience applying IRR
> themselves.

1/ About the "Selecting an IRR Database" section, the best current
practise is to use the IRR that your RIR provides you. In other words:
register ARIN managed prefixes in the ARIN IRR, and RIPE managed
prefixes in the RIPE IRR. Only the RIRs are in a position to attest that
it was the owner of a prefix that created the route(6): object.

Databases such as RADB, NTTCOM, ALTDB are so-called "third party"
databases and at this moment in time have no way to validate anything.
I've highlighted the differences between the various sources of data in
this talk at RIPE76 https://ripe76.ripe.net/archives/video/22

Also, in most regions people are paying their RIR money, this money pays
for IRR services so I'd recommend to use those.

2/ I'd delete the "Step 2: Document Your Autonomous System’s Routing
Policy" step, nobody uses this.

3/ Step 3 is excellent, everybody who generates filters uses AS-SETs. I
like how detail was given to the "AS15562:AS-SNIJDERS" format to ensure
unique names. Good work.

4/ Step 4 can be extended to promote creating RPKI ROAs (in addition to)
the route objects. Anyone can create a route: object in ALTDB, but only
the owner of a prefix can create a RPKI ROA. As such the RPKI ROAs are a
far more valuable source of data to filter generators.

In summary, great work!

Can I update http://peering.exposed/ and add FCIX with a 'yes' to both
secure route servers & BCP 214? :-)

Kind regards,

Job

Re: Proving Gig Speed

[Mark Tinka]

On 19/Jul/18 17:08, Tei wrote:

> tl:dr:  the web is evolving into a network of applications, instead of
> documents.  Documents can't "break" easily. Programs may break
> completelly even to tiny changes. Maybe getting webmasters on board of
> biasing in favor of documents could do us all a favour.

Yes, that would be great, but I recall there was someone asking about
deploying a WISP in rural America several weeks ago and how to deal with
these "busy" web sites of 2018. And your suggestion was one of the
proposals NANOG suggested to the OP.

While it'd be good for webmasters to optimize the content they create, I
appreciate that as the Internet continues to develop, web sites and
content will only get a lot more dynamic and complicated, to appeal to
the human desire for "pretty nice shiny things". Certainly,
optimizations for mobile devices is better than for laptops/desktops for
obvious reasons, but as those mobile devices keep adding power, it will
slowly release the constraints webmasters have when developing for said
platforms.

So in terms of effort expended, I'd be losing if I tried to get content
creators to simplify their content. Rather, I'll hunt down opportunities
that encourage content to move closer to our markets.

Mark.

Re: Proving Gig Speed

[Mark Tinka]

On 19/Jul/18 17:06, Niels Bakker wrote:

>  
>
> That will happen as soon as it's affordable for them to do so - which
> requires an ecosystem of affordable and reliable independent IP
> transit/transport and colocation to exist.

Agreed, but as experience has shown, those aren't the only
considerations they have to make.

You'd be amazed how often delays in deployment are about a lack of
resources on their side to deploy at all the sites on their agenda.

Mark.

Re: Proving Gig Speed

[Eric Kuhnke]

Mark already knows this, but for the benefit of the North American network
operators on the list, **where** in Africa makes a huge difference. Certain
submarine cables reach certain coastal cities at very different transport
prices, depending on location, what sort of organizational structure of
cable it is, age of cable, etc.

For example Sierra Leone and Liberia are logically network stubs, suburbs
of London, UK. To the best of my knowledge the ISPs and mobile network
operators there greatly prefer buying transport capacity to reach London
rather than the other direction to Accra and Lagos. I do not know of any SL
or LR ISPs which have small POPs with IP edge routers in Accra or Lagos,
and definitely not in Cape Town. Whatever circuits exist for voice traffic
that go to Lagos are much smaller.



On Wed, Jul 18, 2018 at 7:27 AM, Mark Tinka  wrote:

>
>
> On 18/Jul/18 16:22, K. Scott Helms wrote:
>
> > Mark,
> >
> > I am glad I don't have your challenges :)
> >
> > What's the Netflix (or other substantial OTT video provider) situation
> > for direct peers?  It's pretty easy and cheap for North American
> > operators to get settlement free peering to Netflix, Amazon, Youtube
> > and others but I don't know what that looks like in Africa.
>
> Peering isn't the problem. Proximity to content is.
>
> Netflix, Google, Akamai and a few others have presence in Africa
> already. So those aren't the problem (although for those currently in
> Africa, not all of the services they offer globally are available here -
> just a few).
>
> A lot of user traffic is not video streaming, so that's where a lot of
> work is required. In particular, cloud and gaming operators are the ones
> causing real pain.
>
> All the peering in the world doesn't help if the latency is well over
> 100ms+. That's what we need to fix.
>
> Mark.
>

Re: Proving Gig Speed

[Mark Tinka]

On 19/Jul/18 14:57, joel jaeggli wrote:

> There is a point beyond which the network ceases to be a serious
> imposition on what you are trying to do.
>
> When it gets there, it fades into the background as a utility function.

I've seen this to be the case when customers are used to buying large
capacity, i.e., 10Gbps, 20Gbps, 50Gbps, 120Gbps, e.t.c. Admittedly,
these tend to be service providers or super large enterprises, and there
is no way they are going to practically ask you to test their 50Gbps
delivery - mostly because it's physically onerous, and also because they
have some clue about speed tests not being any form of scientific measure.

The problem is with the customers that buy orders of magnitude less
than, say, 1Gbps. They will be interested in speed tests as a matter of
course. We notice that as the purchased capacity goes up, customers tend
to be less interested in speed tests. If anything, concern shifts to
more important metrics such as packet loss and latency.


> The fact that multiple streaming audio / video applications in a
> household doesn't have to routinely cheese people off point to the
> threshold having been reached for the those applications at least in
> fixed networks.

One angle of attack is to educate less savvy customers about bandwidth
being more about supporting multiple users on the network at the same
time all with smiles on their faces, than about it making things go
faster. I had to tell a customer, recently, that more bandwidth will
help increase application speed up to a certain point. After that, it's
about being able to add users without each individual user being
disadvantaged. Y'know, a case of 2 highway lanes running at 80km/hr vs.
25 highway lanes running at 80km/hr.


>  For others it will it still be a while. When that 5GB
> software update or a new purchased 25GB game takes 20 minutes to 
> deliver that's a delay between intent and action that the user or
> service operator might seek to minimize.

That's where the CDN's and content operators need to chip in and do
their part. The physics is the physics, and while I can (could) install
DownThemAll on my Firefox install to accelerate downloads, I don't have
those options when waiting for my PS4 to download that 25GB purchase, or
my iPhone to download that 5GB update.


>  Likewise, Latency or Jitter
> associated with network resource contention impacts real-time
> applications. When the network is sufficiently scaled / well behaved
> that these applications can coexist without imposition that stops being
> a point of contention.

All agreed there.

In our market, it's not a lack of backbone resources. It's that the
majority of online resources customers are trying to reach are
physically too far away.

Mark.

Re: Proving Gig Speed

[Tei]

On 19 July 2018 at 07:06, Mark Tinka  wrote:
>
>
> On 18/Jul/18 17:20, Julien Goodwin wrote:
>
>> Living in Australia this is an every day experience, especially for
>> content served out of Europe (or for that matter, Africa).
>>
>> TCP & below are rarely the biggest problem these days (at least with
>> TCP-BBR & friends), far too often applications, web services etc. are
>> simply never tested in an environment with any significant latency.
>>
>> While some issues may exist for static content loading for which a CDN
>> can be helpful, that's not helpful for application traffic.
>
> Yip.
>
> Mark.

Sorry about that.

I feel bad has a webmaster.  Most of us on the web we are creating
websites that are not documents to be download and viewed, but
applications that require to work many small parts that are executed
togeter.

Most VRML examples from 1997 are unavailable because host moved,
directories changed name,  whole websites where redone with new
technologies. Only a 1% of that exist in a readable format. But the
current web is much more delicate, and will break more and sooner than
that.

Perhaps something can be done about it.  Chrome already include a
option to test websites emulating "Slow 3G" that webmasters may use
and want to use.

I suggest a header or html meta tag where a documents disable external
js scripts, or limit these to a white list of hosts.

 .

So if you are a Vodafone customer.  And you are reading a political
document. Vodafone can inject a javascript script in the page. But it
will not run because of the presence of  .  Vodafone can still
further alter the html of the page to remove this meta and inject
their script.

Get webmasters into the idea of making websites that are documents.
That require no execution of scripts. So they will still work in 2048.
And will work in poor network conditions, where a website that load 47
different js files may break.

tl:dr:  the web is evolving into a network of applications, instead of
documents.  Documents can't "break" easily. Programs may break
completelly even to tiny changes. Maybe getting webmasters on board of
biasing in favor of documents could do us all a favour.

-- 
--
ℱin del ℳensaje.

Re: Proving Gig Speed

[Niels Bakker]

* mark.ti...@seacom.mu (Mark Tinka) [Thu 19 Jul 2018, 07:08 CEST]:
I'm not sure about North America, Asia-Pac or South America, but in 
Europe, the gaming folk actually peer very well.


The problem for us is that is anymore from 112ms - 170ms away, 
depending on which side of the continent you are.


And while we peer extensively with them via our network in Europe, 
that latency will simply not go away. They need to come into market 
and satisfy their demand.


That will happen as soon as it's affordable for them to do so - which 
requires an ecosystem of affordable and reliable independent IP 
transit/transport and colocation to exist.



-- Niels.

Quickstart Guide to IRR/RPSL

[Kenneth Finnegan]

Greetings,

As part of setting up a new Internet Exchange in Fremont, California,
we've been investigating prefix filtering on the route servers based
on IRR.

Unfortunately, we were not satisfied with any of the existing
documentation available online as far as taking a network engineer
from "zero" to "just enough IRR to post our prefixes for filtering".
Lacking this documentation, it was rather unreasonable for us to turn
on filtering and drop most of our peer's prefixes without a relevant
whitepaper to point them to to get started.

So, we wrote our own guide:
http://fcix.net/whitepaper/2018/07/14/intro-to-irr-rpsl.html

I thought others on this list would find our whitepaper interesting,
and/or have valuable feedback based on their experience applying IRR
themselves.

--
Kenneth Finnegan
Technical Director
Fremont Cabal Internet Exchange
http://fcix.net/

Re: Comcast outage Southwest Washington?

[Matt Hohman]

Seeing the same in one of 6 locations in Clark County. Our Comcast DIA fiber 
service and all but our east Clark county location are still up and running.

Thanks,
Matt Hohman
Technical Ministries
New Heights Church


From: 20231464200n behalf of
Sent: Wednesday, July 18, 2018 7:33 PM
To: NANOG mailing list
Subject: Comcast outage Southwest Washington?

There a Comcast outage affecting a few of my locations in SW Washington
state. We initially had an estimate of 3:26 PM for service restoration.
That got bumped to 7 PM. Now the phone system isn't giving an ETR and the
phone system says there are excessive hold times.

I'm guessing it's a fiber cut. Can anyone provide some insight?

Thanks,

-A

Implementation plan and dates for NWI5 - Out-of-region ROUTE(6) objects and removal of ASN authorisation for ROUTE(6) object creation.

[Nathalie Trenaman]

Dear colleagues,

In February 2018, the RIPE Database Working Group reached consensus on
NWI-5 - creating out of region ROUTE(6) objects in the RIPE Database.

We will soon be making changes to the way out-of-region objects are
handled in the RIPE Database. It's important that database users are
aware of these changes and what they mean:

1. The RIPE Routing Registry will no longer support the creation of
out-of-region ROUTE(6) and AUT-NUM objects
2. Existing out-of-region objects will have their "source:” attribute
changed to "RIPE-NONAUTH”
3. The creation of ROUTE(6) objects will no longer need authorisation
from the ASN holder

We will implement these changes in the Release Candidate environment on
Thursday, 2 August and go to full production on Tuesday, 4 September.

Our implementation plan can be found at:
https://www.ripe.net/manage-ips-and-asns/db/impact-analysis-for-nwi-5-implementation
 


The RIPE Database Working Group chairs also wrote a RIPE Labs article
that describes the background of NWI-5:
https://labs.ripe.net/Members/denis/out-of-region-route-6-and-aut-num-objects-in-the-ripe-database
 

>

Please contact ripe-...@ripe.net  if you still have 
questions after
reading the implementation plan.

Best regards,

Nathalie Trenaman
Product Manager
RIPE NCC

Re: deploying RPKI based Origin Validation

[Mark Tinka]

On 16/Jul/18 19:00, Paolo Lucente wrote:

> Hi Job, All,
>
> It is definitely great to see progress on the deployment side! I realize
> that there may be some gaps in the network operator toolchain, and this
> may be something i'd like to contribute to.
>
> For network operators to better understand the impact of BGP hijacks in
> terms of revenue or volumes of traffic that went missing, it makes perfect
> sense if network monitoring tools are aware of which BGP announcements
> are invalid or not.
>
> I will look into adding support for the RTR protocol (RFC 6810, RFC
> 8210) to pmacct ( https://github.com/pmacct/pmacct , http://pmacct.net/ )
> and expose the validation state through an extra field (when collecting
> routing tables) and primitive (when accounting traffic and correlating
> it with BGP data).
>
> Updating the telemetry tools to be fully aware of RPKI validation states
> should come in handy!

Sounds great, Paolo. Many thanks for this.

Mark.

Re: NANOG list errors

[Duane Wessels via NANOG]

> On Jul 17, 2018, at 9:24 PM, Andy Ringsmuth  wrote:
> 
> Fellow list members,
> 
> The last several days, I’ve been receiving mail forwarding loop errors for 
> the list. I’ll receive them several hours after sending a message. I’ll paste 
> the latest two of them below, separated by % symbols.
> 
> Anyone able to sort this out and fix?


Andy (and others), if you forward a few full bounces to me I can take a look 
and see what might be going on.

DW

Re: issues through CGNat (juniper ms-mpc-128g in mx960)

[Matt Erculiani]

One of the biggest deal-breakers about the Miltiservices DPC and MPC is
that it does not support NAT Transversal (UDP hole punching), which is
probably the reason you're having trouble with the PS4 online gaming. It
also messes with VoIP too.

As for the IPsec issue, im not sure if that would be related to NAT-T, but
you'd probably need logs. Might just be easier to give anyone using an
IPsec tunnel a public IP.

-M

On Thu, Jul 19, 2018, 09:35 Aaron Gould  wrote:

> (please forgive cross-posting between jnsp and nanog.looking for anyone who
> could help shed light)
>
>
>
> I moved customers behind MS-MPC-128G (MX960) CGNat boundary a few nights
> ago. for the most part it went well. with these couple issues. please let
> me
> know what you know about this and how to fix. I don't know if it's fixed on
> the endpoints, or in the cgnat config or what.
>
>
>
> 1 - IPSEC VPN
>
> -Customer said the vpn connect light on cisco vpn router blinks
> (not
> connected to vpn)
>
> -I found out the vpn addresses that this cisco vpn router is trying
> to connect to.
>
> -I did a fix in cgnat rule stanza where all UDP 500 and 4500
> traffic
> to that distant vpn endpoint(s) will always be natted to one and only one
> ip
> address (I did this thinking that the changing ip of the public pool
> assigned ip addresses for udp 500 and 4500 was possible breaking it)
>
>
>
> 2 - PS4 gaming
>
> -Customer said playing a few games (call of duty, etc) with
> Internet
> players now doesn't work.
>
> -They said the PS4 nat type is nat type 3 (strict) whereas before
> the moved them to cgnat, it was NAT type 2 moderate and worked.
>
>
>
>
>
> -Aaron
>
>
>
>

issues through CGNat (juniper ms-mpc-128g in mx960)

[Aaron Gould]

(please forgive cross-posting between jnsp and nanog.looking for anyone who
could help shed light)

 

I moved customers behind MS-MPC-128G (MX960) CGNat boundary a few nights
ago. for the most part it went well. with these couple issues. please let me
know what you know about this and how to fix. I don't know if it's fixed on
the endpoints, or in the cgnat config or what.

 

1 - IPSEC VPN

-Customer said the vpn connect light on cisco vpn router blinks (not
connected to vpn)

-I found out the vpn addresses that this cisco vpn router is trying
to connect to.

-I did a fix in cgnat rule stanza where all UDP 500 and 4500 traffic
to that distant vpn endpoint(s) will always be natted to one and only one ip
address (I did this thinking that the changing ip of the public pool
assigned ip addresses for udp 500 and 4500 was possible breaking it)

 

2 - PS4 gaming

-Customer said playing a few games (call of duty, etc) with Internet
players now doesn't work.

-They said the PS4 nat type is nat type 3 (strict) whereas before
the moved them to cgnat, it was NAT type 2 moderate and worked.

 

 

-Aaron

 

Re: Proving Gig Speed

[joel jaeggli]

On 7/19/18 1:30 AM, Mark Tinka wrote:
>
> On 18/Jul/18 23:56, Keith Stokes wrote:
>
>> At least in the US, Jane also doesn’t really have a choice of her
>> electricity provider, so she’s not getting bombarded with advertising
>> from vendors selling “Faster WiFi” than the next guy. I don’t get to
>> choose my method of power generation and therefore cost per kWh. I’d
>> love to buy $.04 from the Pacific NW when I’m in the Southern US. 
> And that's why I suspect that 10Gbps to the home will become a reality
> not out of necessity, but out of a race on who can out-market the other.
>
> The problem for us as operators - which is what I was trying to explain
> - was that even though the home will likely not saturate that 10Gbps
> link, never mind even use 1% of it in any sustained fashion, we shall be
> left the burden of proving the "I want to see my 10Gbps that I bought,
> or I'm moving to your competitor" case over and over again.
>
> When are we going to stop feeding the monster we've created (or more
> accurately, that has been created for us)?
There is a point beyond which the network ceases to be a serious
imposition on what you are trying to do.

When it gets there, it fades into the background as a utility function.

The fact that multiple streaming audio / video applications in a
household doesn't have to routinely cheese people off point to the
threshold having been reached for the those applications at least in
fixed networks. For others it will it still be a while. When that 5GB
software update or a new purchased 25GB game takes 20 minutes to 
deliver that's a delay between intent and action that the user or
service operator might seek to minimize. Likewise, Latency or Jitter
associated with network resource contention impacts real-time
applications. When the network is sufficiently scaled / well behaved
that these applications can coexist without imposition that stops being
a point of contention.
> Mark.
>

Re: Proving Gig Speed

[nanog-isp]

No, but you can connect iPhones with gigabit Ethernet over copper. 

Jared

>-Original Message- 
>From: NANOG [mailto:nanog-bounces at nanog.org] On Behalf Of Mike 
>Hammett
>Cc: NANOG list 
>Subject: Re: Proving Gig Speed
>
> I don't think iPhones have SFP cages. 
>
>
>
>
>- 
>Mike Hammett 
>Intelligent Computing Solutions 
>http://www.ics-il.com 
>
>Midwest-IX 
>http://www.midwest-ix.com