Re: Cloudflare's rpki.json file is missing IPv4 ROAs longer than /24

[Job Snijders via NANOG]

On Wed, Sep 18, 2024 at 07:33:37AM -0400, Steven Wallace wrote:
> Internet2 uses Cloudflare’s https://rpki.cloudflare.com/rpki.json as
> an alternate source for RPKI-ROA information. We recently discovered
> that this file omits IPv4 ROAs longer than /24. It would be helpful if
> it included all ROAs.

Yup, that's a clear bug.

Perhaps a suitable alternative for your application is
https://console.rpki-client.org/rpki.json.gz (or rpki.json)

The above file is produced by rpki-client instances and refreshed every
few minutes. The backend servers are operated by me.

The file format follows roughly the same JSON format as CF's, but has
been extended with per-VRP-specific expiry dates expressed as Unix
timestamps.

Kind regards,

Job

RFC 9234 route leak prevention in the wild!

[Job Snijders via NANOG]

Dear all,

I'd like to share an update on RFC 9234 deployment. RFC 9234 titled
"BGP Open Policy" aka the "Only-To-Customer" (OTC) BGP Path Attribute is
an anti-route-leak mechanism which is *NOT* based on RPKI! (yes ...
routing security is more than just RPKI! :-)

The basic idea of 9234 is that BGP routers (based on their role in the
Gao-Rexford inter-domain routing model) attach a special BGP Path
attribute, or take action based on the presence and contents of this BGP
Path attribute - with the intention to constrain a route's propagation
radius to just the downstream customer cone of the neighboring ASN.

Most operators will intuitively understand that any route propagating
through multiple IX route servers operated by different IXPs is a route
leak:

```
 IXP_1 IXP_2
/ \   / \
   /   \ /   \
  ISP_A ISP_B ISP_C
(receives)(leaker) (originates)
``` (figure 1. propagation from right to left; leak scenario)

In the above example, ISP_A originates a route towards IXP_1's route
servers, IXP_1 propagates the route to ISP_B (so far so good); but for
one reason or another ISP_B subsequently continues propagation of the
route towards IXP_2's route servers, who in turn propagate it to ISP_C.
ISP_B is forwarding IP traffic between ISP_A and ISP_C for zero revenue.
ISP_A and ISP_C are probably not expecting ISP_B to be in the middle.
This situation can happen as a result of a misconfiguration in ISP_B's
equipment, even when all participants use IRR & RPKI ROV to attempt to
mitigate the worst routing incidents.

What does it matter / impact


Calgary-based YYCIX deployed RFC9234 support in late 2022/early 2023
using OpenBGPD; and FranceIX deployed support using BIRD in Q2 2024.
Both IXPs configured their route servers to reject BGP routes that have
an OTC attribute attached, and to attach an OTC attribute when
propagating routes to the Route Server's peers.

As it happens to be, currently (Mon Sep 2 12:26:50 UTC 2024) a small route leak
is happening involving both YYCIX and FranceIX Route Servers via an ISP
connected to both IXP's Route Servers; with the leak being stopped at YYCIX
thanks to RFC 9234! Appendix A contains a table of leaked IPv4 routes.

Let's zoom in on 1 entry:

```
$ bgpctl show rib 157.185.154.0/24 detail

BGP routing table entry for 157.185.154.0/24
6939 38040 54994
Nexthop 206.126.225.20 (via 206.126.225.20) Neighbor 206.126.225.20 
(216.218.252.194)
Origin IGP, metric 1911, localpref 100, weight 0, ovs not-found, avs 
unknown, external, otc leak
Last update: 11:58:08 ago
Communities: 0:2906 0:16265 0:16276 0:18638 0:41690 0:48641 0:49029
Ext. Communities: ovs not-found
Large Communities: 53339:11:1 53339:11:3
Aggregator: 54994 [163.171.131.254]
OTC: 51706
``` (figure 2. inspecting an leaked route using OpenBGPD's CLI)

In figure 2. one can see the route is marked as 'otc leak', this was
made possible because FranceIX's route server's attached the OTC
attribute with the ASN value set to their Route Server's ASN (51706).

```
 YYCIX  FranceIX
. x   \
   .   \  /  \
  ISP_A 6939_3804054994
``` (figure 3. right to left: real world example of blocked leak)

In figure 3 AS 54994 originates 157.185.154.0/24 and propagates a route
towards the FranceIX route servers. FranceIX accepts this route
(probably because an IRR route object exists) and propagates it onward
with the "Only-To-Customer" attribute set to 51706. The route is
received by AS 38040, who appear to propagate the route to their
upstream 6939. << An AS 38040 router is likely misconfigured! >>
Then 6939 sends its customer's routes to the YYCIX route server, but the
YYCIX route server recognizes that the route already passed through a
'valley' and thus considers this a leak; and blocks further propagation
towards ISP_A.

Impact with partial deployment
==

RFC 9234 is an easy mechanism to configure and debug for both small &
large network operators and IXP route server operators. In the above
scenario YYCIX and FranceIX are likely the only 2 entities in the entire
AS path which support RFC 9234; but we're already seeing leaks being
blocked despite partial deployment!

It's not hard to imagine that many IX-to-IX route leaks can be blocked
with only the IXP operators themselves enabling RFC 9234 support.
The world's most populair RS implementations (BIRD and OpenBGPD)
already support RFC 9234! Tens of thousands of IX customers would enjoy
the benefits of a few hundred IXPs taking action. RFC 9234 has no
dependency on the RPKI, this means tha

What can you do?


Just ask your vendors (your hardware routers and IXPs) to implement and
deploy RFC 9234! :-) The more peo

Re: A single place for information relating to the deployment and usage of RPKI

[Job Snijders via NANOG]

On Mon, Jul 29, 2024 at 04:32:40AM +, Christopher Hawker wrote:
> When it comes to RPKI, its deployment and usage, there is a fair bit
> of information available on the Internet. Each RIR has their own
> guides for creating ROAs, each router vendor and developer has their
> own guides for deploying route object validation on their devices and
> software, each Relying Party software developer has their own how-to
> guide on installing/configuring/maintaining the software, among
> others. What I haven’t been able to find thus far is a place where all
> of this information is altogether, in one place, for simplicity and
> ease of comparison to find the best solution for a network operator’s
> needs.

Some of it may be covered here: 
https://bgpfilterguide.nlnog.net/guides/reject_invalids/

Kind regards,

Job

Re: IRR mirrors de-sync from ARIN or irrd4 bug or ARIN streaming is broken?

[Job Snijders via NANOG]

On Wed, Jul 10, 2024 at 07:10:48PM -0400, Aliaksei Sheshka wrote:
> nothing!  I suspect the mirror is out of sync.
> 
> Now NTT mirror:

Seems reloading helped:

$ date
Thu Jul 11 03:50:22 UTC 2024

$ whois -h rr.ntt.net 199.52.73.0/24
route:  199.52.73.0/24
origin: AS132055
descr:  EY India
admin-c:IAM12-ARIN
tech-c: DNSAD85-ARIN
tech-c: IAM12-ARIN
mnt-by: MNT-EYL-Z
created:2022-10-19T08:37:50Z
last-modified:  2023-11-27T15:10:44Z
source: ARIN
rpki-ov-state:  valid


route:  199.52.73.0/24
descr:  RPKI ROA for 199.52.73.0/24 / AS132055
remarks:This AS132055 route object represents routing data retrieved
from the RPKI. This route object is the result of an automated
RPKI-to-IRR conversion process performed by IRRd.
max-length: 24
origin: AS132055
source: RPKI  # Trust Anchor: arin

Re: IRR mirrors de-sync from ARIN or irrd4 bug or ARIN streaming is broken?

[Job Snijders via NANOG]

On Wed, Jul 10, 2024 at 09:37:22PM -0400, Aliaksei Sheshka wrote:
> On Wed, Jul 10, 2024 at 9:26 PM Job Snijders via NANOG 
> wrote:
> 
> > Indeed, it appears both NTT’s and RADB’s mirror instances are
> > desynchronized in relationship to ARIN’s IRR. Both NTT and RADB
> > should do a database reload to rectify the issue.
> >
> > Desynchronisation can happen at either server side or client side,
> > so the root cause isn’t apparent to me.
> 
> Now I'm concerned about how frequent such silent corruptions are.
> Also, hard (impossible?) to tell when that happened.

Work is underway to at least get rid of NRTM v3
https://www.ietf.org/archive/id/draft-ietf-grow-nrtm-v4-04.html

> IRR `route` and friend should be retired, the sooner the better.

I am with you on that one, eventually, because 'just turning it off'
doesn't seem a great solution to me.

Not everyone who has access to IRR services has access to RPKI services.
Not everyone agrees with ARIN's Relying Party Agreement. RPKI doesn't do
all the things that IRR does (but I am not in favor of blindly porting
all features & misfeatures from IRR to RPKI!). There is a fair bit of
study to be done in this space.

Kind regards,

Job

Re: IRR mirrors de-sync from ARIN or irrd4 bug or ARIN streaming is broken?

[Job Snijders via NANOG]

Rubens,

ARIN-NONAUTH was deprecated two years ago:
https://www.arin.net/vault/announcements/20220404-irr/

Aliaksei,

Indeed, it appears both NTT’s and RADB’s mirror instances are
desynchronized in relationship to ARIN’s IRR. Both NTT and RADB should do a
database reload to rectify the issue.

Desynchronisation can happen at either server side or client side, so the
root cause isn’t apparent to me.

Kind regards,

Job

On Thu, 11 Jul 2024 at 10:15, Rubens Kuhl  wrote:

> If you look at TC, you will see that this object is part of ARIN-NONAUTH:
>
> https://bgp.net.br/whois.html?q=199.52.73.0%2F24
>
> route:  199.52.72.0/22
> descr:  Ernst & Young, Gurgaon Cyberpark, India
> origin: AS132055
> mnt-by: MNT-EYL
> changed:zanub-h.kalathin...@sg.ey.com 20210614
> source: ARIN-NONAUTH
> remarks:
> remarks:* THIS OBJECT CONTAINS PLACEHOLDER DATA
> remarks:* Please note that all data that is generally regarded
> as personal
> remarks:* data has been removed from this object.
> remarks:* To view the original object, please query the ARIN
> Database at:
> remarks:* http://www.arin.net/whois
> remarks:
> rpki-ov-state:  not_found # No ROAs found, or RPKI validation not
> enabled for source
>
> It's also mapped to an existing RPKI entry:
>
> route:  199.52.73.0/24
> descr:  RPKI ROA for 199.52.73.0/24 / AS132055
> remarks:This AS132055 route object represents routing data
> retrieved
> from the RPKI. This route object is the result of an
> automated
> RPKI-to-IRR conversion process performed by IRRd.
> max-length: 24
> origin: AS132055
> source: RPKI  # Trust Anchor: arin
>
> Perhaps RADB is preferring not to mirror non-authoritative databases ?
>
>
> Rubens
>
> On Wed, Jul 10, 2024 at 10:05 PM Aliaksei Sheshka 
> wrote:
> >
> > Hi folks!
> >
> > I noticed something unusual today, and perhaps some of you know the
> answer.
> >
> > Consider ARIN rr:
> >
> > $ whois -h rr.arin.net 199.52.73.0/24
> > route:  199.52.73.0/24
> > origin: AS132055
> > descr:  EY India
> > admin-c:IAM12-ARIN
> > tech-c: DNSAD85-ARIN
> > tech-c: IAM12-ARIN
> > mnt-by: MNT-EYL-Z
> > created:2022-10-19T08:37:50Z
> > last-modified:  2023-11-27T15:10:44Z
> > source: ARIN
> >
> > everything looks fine.
> >
> > Now RADB mirror:
> >
> > $ whois -h whois.radb.net 199.52.73.0/24
> > %  No entries found for the selected source(s).
> >
> > nothing!  I suspect the mirror is out of sync.
> >
> > Now NTT mirror:
> >
> > $ whois -h rr1.ntt.net 199.52.73.0/24
> > route:  199.52.73.0/24
> > descr:  RPKI ROA for 199.52.73.0/24 / AS132055
> > remarks:This AS132055 route object represents routing data
> retrieved
> > from the RPKI. This route object is the result of an
> automated
> > RPKI-to-IRR conversion process performed by IRRd.
> > max-length: 24
> > origin: AS132055
> > source: RPKI  # Trust Anchor: arin
> >
> > As you can see it returns only RPKI data, and not ARIN. So ARIN data is
> not in sync there as well?
> >
> > However for 199.52.53.0/24 it returns both
> >
> > $ whois -h rr1.ntt.net 199.52.53.0/24
> > route:  199.52.53.0/24
> > origin: AS132055
> > descr:  EY India
> > admin-c:IAM12-ARIN
> > tech-c: DNSAD85-ARIN
> > tech-c: IAM12-ARIN
> > mnt-by: MNT-EYL-Z
> > created:2022-07-11T07:05:30Z
> > last-modified:  2023-11-27T15:10:44Z
> > source: ARIN
> > rpki-ov-state:  valid
> >
> > route:  199.52.53.0/24
> > descr:  RPKI ROA for 199.52.53.0/24 / AS132055
> > remarks:This AS132055 route object represents routing data
> retrieved
> > from the RPKI. This route object is the result of an
> automated
> > RPKI-to-IRR conversion process performed by IRRd.
> > max-length: 24
> > origin: AS132055
> > source: RPKI  # Trust Anchor: arin
> >
> > My question is how and what happened? I suspect whois stream was
> incostent.
> > Because if one check todays ARIN DB it surely has the data
> >
> > $ wget ftp://ftp.arin.net/pub/rr/arin.db.gz -O - 2>/dev/null | gzip -d
> | grep -A10 199.52.73.0/24
> > route:  199.52.73.0/24
> > origin: AS132055
> > descr:  EY India
> > admin-c:IAM12-ARIN
> > tech-c: DNSAD85-ARIN
> > tech-c: IAM12-ARIN
> > mnt-by: MNT-EYL-Z
> > created:2022-10-19T08:37:50Z
> > last-modified:  2023-11-27T15:10:44Z
> > source: ARIN
> >
> > Thanks!
> >
>

Re: HE.net problem

[Job Snijders via NANOG]

On Fri, 5 Jul 2024 at 06:59, Randy Bush  wrote:

> not to distract from everyone diagnosing someone else's problem, but ...
>
> what foss dns monitoring tools do folk use to alert of
>   - iminent delegation expiry
>   - inconsistent service (lame, soa mismatches, ...)
>   - dnssec signing and timer issues
>   - etc.



https://github.com/berthubert/simplomon

Re: Should FCC look at SS7 vulnerabilities or BGP vulnerabilities

[Job Snijders via NANOG]

On Thu, May 16, 2024 at 07:17:37PM -0400, Brandon Martin wrote:
> I suspect that's why we've had some success with getting BGP security
> not just addressed in guidance but actually practically improved.

Ben Cartwright-Cox's axiom (paraphrased): "The real reason the Internet
works is that we want it to work."

https://ripe88.ripe.net/wp-content/uploads/A-Network-Of-Networks-RIPE88-RACI-Ben-Cartwright-Cox.pdf

Kind regards,

Job

Re: FCC proposes Internet Routing Security Reporting Requirements

[Job Snijders via NANOG]

Dear all,

A fact sheet has now been published, with much more detail and
considerations: https://docs.fcc.gov/public/attachments/DOC-402609A1.pdf

This is a VERY interesting read!

Kind regards,

Job

Re: Should FCC look at SS7 vulnerabilities or BGP vulnerabilities

[Job Snijders via NANOG]

On Thu, May 16, 2024 at 04:05:21PM -0400, Josh Luthman wrote:
> Now do you think they're going to properly understand what an SS7 or
> vulnerability is?

The FCC organised several sessions (private and public) where they
invited knowledgeable people from this community to help edifice them on
what BGP is and what risks exist.

https://www.fcc.gov/news-events/events/2023/07/bgp-security-workshop

Watch https://www.youtube.com/watch?v=VQhoNX2Q0aM to see our very own
Tony Tauber looking sharp in a nice suit! :-)

FCC staff attended NANOG & IETF meetings to further explore and discuss
the problem space in the hallway track. If anything, I think the FCC
made a proper effort to connect with various stakeholders and learn from
them.

Kind regards,

Job

Re: Meet NANOG's New Executive Director! N91 Agenda is LIVE! + More

[Job Snijders via NANOG]

On Thu, May 16, 2024 at 02:23:52PM -0400, Nanog News wrote:
> *Jonathan Black has been appointed NANOG Executive Director*
> 
> In his new role, Jonathan will be responsible for the organization's
> operational management and will collaborate with the NANOG Board to
> refine, articulate, and implement NANOG's mission and strategic plan.

Jonathan, I wish you a warm welcome!

Kind regards,

Job

FCC proposes Internet Routing Security Reporting Requirements

[Job Snijders via NANOG]

Dear all,

FYI:
https://docs.fcc.gov/public/attachments/DOC-402579A1.pdf

Kind regards,

Job

Re: Upcoming LACNIC RPKI Migration

[Job Snijders via NANOG]

Dear Carlos, LACNIC, and wider community,

I very much appreciate how LACNIC worked with various stakeholders
before publicly commiting to the schedule outlined in Carlos' email.

>From what I can see, LACNIC pro-actively and properly tested their
purported post-migration environment with very broad set of old and new
versions of a myriad of RPKI cache implementations. Then they also
reached out to anyone they could think of, in a timely manner - to
accommodate the opportunity for feedback and confirm compliance with
IETF RPKI standards pre/during/post the upcoming migration.

LACNIC - your plan seems solid; thank you for sharing it with us.

Kind regards,

Job

Re: BGP Monitoring

[Job Snijders via NANOG]

On Mon, Feb 26, 2024 at 05:41:12PM +, Ray Orsini via NANOG wrote:
> What tools are you using to monitor BGP announcements and route changes?

The wonderful BGP.tools already has been mentioned a few times.

Another excellent option is https://Packetvis.com, I find their RPKI
monitoring approach to be very insightful.

Catchpoint might be another option, https://www.catchpoint.com/bgp,
AFAIK by the same people that worked on "Isolario" a few years ago.

Kind regards,

Job

Re: IRRD & exceptions to RPKI-filtering

[Job Snijders via NANOG]

On Mon, Feb 12, 2024 at 05:01:35PM -0600, Richard Laager wrote:
> On 2024-02-12 15:18, Job Snijders via NANOG wrote:
> > On Mon, Feb 12, 2024 at 04:07:52PM -0500, Geoff Huston wrote:
> > > I was making an observation that the presentation material was
> > > referring to "RPKI-Invalid" while their implementation was using
> > > "ROA-Invalid" There is a difference between these two terms, as I'm
> > > sure you're aware.
> 
> I'm sure Job is aware, but I'm not. Anyone want to teach me the
> difference?

I'll try, but please bear with me as terminology throughout the years
has shifted and perhaps wasn't entirely consistent from the start, and
maybe I missed some bits. :-)

The word "invalid" in context of RPKI and BGP has a lot of additional
context:

RFC 6811 ("BGP Prefix Origin Validation") introduced the concept of a
given BGP route being "NotFound", "Valid", or "Invalid". In later years
many people referred to "Prefix Origin Validation" as "Route Origin
Validation" or "RPKI-based Origin Validation" (both variants abbreviated
to "ROV"). Other variants also exist.

Before one can conduct the RFC 6811 "Prefix Origin Validation" (née
"Route Origin Validation") process, the operator (in an automated
fashion, using a RPKI validator) will ascertain the validity of the ROAs
(Route Origin Authorizations) by checking the cryptographic signatures,
validity time windows, and other properties (See RFC 6488
and https://datatracker.ietf.org/doc/html/draft-ietf-sidrops-rfc6482bis)

In order for the RFC 6811 validation process to arrive at a "Valid" or
"Invalid" outcome, first of all a *valid* ROA needs to exist (as in
cryptographically valid). So, to designate a BGP route as 'invalid', one
needs at least one 'valid' ROA covering the IP address prefix at hand.

The concept of validating BGP routes (or, as some call it 'verifying BGP
routes'), using RPKI derived information, has been transposed to IRR
data as well. For example, in 2018 RIPE NCC started using RPKI data to
untangle and cleanup the "RIPE-NONAUTH" IRR database, as per policy
https://www.ripe.net/publications/docs/ripe-731/ And the NTT Global IP
Network (GIN/AS2914) used the same methodology on its IRRd server
'rr.ntt.net' (the default host used in bgpq4). Now RADB uses the same
methodology (and software) as NTT does.

A ROA can be invalid (for example, because its X.509 EE certificate
expired); a BGP route can be invalid (because no valid RPKI ROA attest
that the route could originate from the ASN at hand), and an IRR object
can be invalid (because no Valid ROA attest the route object's "origin:"
could originate the prefix at hand).

Kind regards,

Job

Re: IRRD & exceptions to RPKI-filtering

[Job Snijders via NANOG]

On Mon, Feb 12, 2024 at 04:07:52PM -0500, Geoff Huston wrote:
> > On 12 Feb 2024, at 3:14 pm, Job Snijders via NANOG  wrote:
> > At NANOG 90, Merit presented on their IRRd v4 deployment. At the
> > microphone Geoff Huston raised a comment which I interpreted as:
> > 
> > [snip]
> 
> no - I was making an observation that the presentation material was
> referring to "RPKI-Invalid" while their implementation was using
> "ROA-Invalid" There is a difference between these two terms, as I'm
> sure you're aware.

Thanks for the clarification! I guess I somehow got very confused
watching the webstream as to what your comment or ask was.

In any regard, - researchers, please just talk to IRRD operators if your
experiment requires the existence of RPKI/ROA-invalid route/route6
objects! There are ways to make it happen, but the default of course is
to keep things as tidy as possible :-)

Greetings from the Netherlands & sorry to miss N90,

Job

IRRD & exceptions to RPKI-filtering

[Job Snijders via NANOG]

Dear all,

At NANOG 90, Merit presented on their IRRd v4 deployment. At the
microphone Geoff Huston raised a comment which I interpreted as:

"Can an exception be made for my research prefixes?"

There are two sides to this:

INSERTING RPKI-invalid route/route6 objects
===
By default, IRRd v4 rejects submitted route/route6 objects if the system
detects the objects to be RPKI-invalid. This helps guard against typos,
mistakes, and some forms of adversarial actions.

However, some researchers find this protection annoying, because the
'noise filtering' mechanism interfers with their ability to insert noise
to measure noise. ;-)

In order to allow users to insert RPKI-invalid route/route6 objects into
the database, the IRRD operator needs to make use of a so-called SLURM
file. SLURM is an IETF-standardized JSON format to describe 'rules' to
be applied to RPKI-derived information passing through a pipeline.

RADB would need to adjust their configuration to point to a SLURM, and
add the prefixes of researchers to the 'prefixFilter' section.

https://irrd.readthedocs.io/en/stable/admins/rpki/#slurm-support

Of course, RADB (or any IRRd v4 operator) will need to vet whether the
researchers actually have some authority to make requests on behalf of
the Resource Holder! I wouldn't like it if some random person could ask
for ROAs related to my employer to be ignored! :-)

It is up to each IRRD operator as to what their policy is on assisting
researchers and making exceptions to the RPKI-filtering mechanism.

QUERYING for RPKI-invalid route/route6 objects
==
By default, IRRd v4 returns RPKI-filter responses for WHOIS queries
related to routes. This is done to help safe guard the ecosystem.

Users can disable filtering of objects by issuing '!fno-rpki-filter' in
the WHOIS connection. This is intended as a debugging aid. See this page
for more information on the various WHOIS queries that IRRD v4 supports:
https://irrd.readthedocs.io/en/stable/users/queries/whois/

Kind regards,

Job

Re: route: 0.0.0.0/32 in LEVEL3 IRR

[Job Snijders via NANOG]

On Tue, Jan 30, 2024 at 07:28:01PM +0300, Frank Habicht wrote:
> I believe that the entry of
> route:  0.0.0.0/32
> 
> does not serve any good purpose?

I don't think so either, I've created an issue to prevent that in future
releases of IRRd v4: https://github.com/irrdnet/irrd/issues/906

Thanks for noticing that!

It'll be up to Lumen to remove the record at hand.

Kind regards,

Job

RPKI's 2023 Year in Review - growth, governments, and innovation

[Job Snijders via NANOG]

Dear all,

Happy new year everyone! Having just closed chapter 2023 - let's look
back at the previous year.

In this memo I'll share some RPKI statistics, summarize highlights from
the IETF Standards Development process, and reflect on emerging trends.


Year to Year Growth of the distributed RPKI database
===

A straight-forward method to compare to last year is to look at the
RPKI's absolute numbers.

The below table was constructed by comparing two December 31st
RPKIviews.org snapshots [1] of validated RPKI caches, primed with the
ARIN, AFRINIC, APNIC, LACNIC, and RIPE NCC Trust Anchors.

   2022-12-31  2023-12-31
Total cache size (KiB): 1,240,572   1,546,728 (+25%)
Total number of files (objects):  242,969 309,802 (+27%)
Publication servers (FQDNs):   52  63 (+21%)
Certification authorities: 34,901  40,511 (+16%)
Route origin authorizations:  138,323 188,345 (+36%)
Unique VRPs:  390,752 497,341 (+27%)
IPv4 addresses covered: 1,354,270,410   2,502,293,068 (+85%)
IPv6 addresses covered: 9,447 * 10^30  17,263 * 10^30 (+83%)
Unique origin ASNs in ROAs:34,455  40,656 (+18%)

The number of IP addresses covered by ROAs almost doubled compared to
last year. More than half the ASes in the global Internet routing system
are listed as Origin AS in at least one ROA. EOY'23 more unique IPv6
prefix-origin pairs in the DFZ are covered by ROAs than not, with IPv4
likely to follow crossing this threshold in Q1 2024.

Kentik [2] estimates that more than half of IP traffic is destined
towards ROA covered destinations. APNIC Labs [3] estimates the global
"I-Rov Filtering Rate" to be at 22.69% now. These numbers mean it will
be worth your while to create and use ROAs for BGP Origin Validation.

In short: it's all up and to the right. :-)


Increased Government Interest in Internet Routing Security
===

Following the FCC's launch of an inquiry into Internet Routing
Vulnerabilities in 2022, in 2023 the agency followed up with a BGP
Security Workshop hosted by the Public Safety and Homeland Security
Bureau [4], the industry seemed well represented with key players
participating.

While the USG is still sizing up the industry's posture and
contemplating whether regulation is warranted, in the Netherlands the
government itself aspires to lead by example: in 2023 the Dutch
government committed to use RPKI by the end of 2024 on all new and
existing IT systems it operates [5]. Note that this ambition includes
both signing ROAs and validating BGP messages! I hope they succeed.

Will more governments follow the Dutch lead in 2024?


The Rise Of Initiatives Re-evaluating The RPKI Trust Model
===

As the industry saw unprecedented turmoil in the realm of governance of
Regional Internet Registries in 2023, two interesting initiatives arose,
both aiming to reduce the risk surface associated with blindly trusting
'the RPKI Roots'.

In the RPKI hierarchical structure, a Trust Anchor (a root) is an
authority for which trust is assumed and not derived. The phrase
"assumed trust" means that violation of that trust is out-of-scope for
the threat model. On the other hand, the phrase "derived trust" refers
to trust automatically and securely computed with subjective logic. In
the context of the RPKI, trust is derived according to the rules for
validation of RPKI Certificates and Signed Objects.

One initiative is to impose 'locally configured constraints' which limit
the effective signing authority of an RIR. The other initiative is to
instantiate a sixth RPKI trust anchor - which chains up to the existing
RIR-operated infrastructure - and impose inter-RIR consensus-driven
constraints on that arc.

Initiative #1 - operator imposed constraints:
Cover letter & discussion: 
https://mailman.nanog.org/pipermail/nanog/2023-September/223354.html
Running code - rpki-client 8.7 & higher
https://cdn.openbsd.org/pub/OpenBSD/rpki-client/rpki-client-8.7.txt
An internet-draft detailing the implementation:

https://datatracker.ietf.org/doc/html/draft-snijders-constraining-rpki-trust-anchors

Initiative #2 - externally imposed constraints:
Cover letter: 
https://labs.apnic.net/index.php/2023/12/13/models-of-trust-in-the-rpki/
Running code: https://labs.apnic.net/nro-ta/
A proposal for the validation algorithm to be less rigid & more robust:

https://datatracker.ietf.org/doc/html/draft-spaghetti-sidrops-rpki-validation-update

I personally don't care much *who* imposes constraints, but at the
end of the day - someone's gotta do it. Keep watching this space!


SIDROPS - Working Group developments
===

Some quick updates from the IETF working group responsible for
de

Celebration: RADB appears to now filter RPKI-invalid IRR route/route6 objects

[Job Snijders via NANOG]

Dear NANOG,

It appears the WHOIS service at whois.radb.net is now filtering out
RPKI-invalid IRR route/route6 objects for common expansion queries!
This really is exciting and excellent news. I'll elaborate a bit on what
this exactly means.

Example ROA & IRR object


Take for example the 194.32.71.0/24 test prefix:
https://irrexplorer.nlnog.net/prefix/194.32.71.0/24

For experimental purposes, NTT (ages ago) created a cryptographically
verifiable RPKI ROA to indicate that any BGP announcements for
194.32.71.0/24 or-longer are RPKI-invalid:
https://console.rpki-client.org/rpki.ripe.net/repository/DEFAULT/01/ee8b98-7240-4b62-acc5-780a25cd0dd9/1/LeUtRtipWJVA-QrrwolA6aEiFsI.roa.html

RPKI ROA:

   ++
   | Prefix: 194.32.71.0/24 |
   | Origin: AS 0   |
   | Trust Anchor:   RIPE NCC   |
   ++

Additionally, an IRR route object was created in the RIPE NCC managed
'RIPE' IRR database:
https://apps.db.ripe.net/db-web-ui/query?bflag=false&dflag=false&rflag=true&searchtext=-r%20%20-T%20route%20194.32.71.0%2F24&source=RIPE

IRR Route Object:

   ++
   | route:  194.32.71.0/24 |
   | origin: AS15562|
   | pingable:   194.32.71.1|
   | ...|
   | source: RIPE   |
   ++

Obviously, the above IRR route object is in conflict with the sole RPKI
ROA for 194.32.71.0/24, which designated AS 0 exclusively. Following the
reasoning that IRR route objects describe possible BGP states, and in
BGP the combo 194.32.71.0/24AS15562 would be rejected; it makes sense to
go a step further and filter out IRR objects which are RPKI-invalid too.

This concept of filtering out RPKI-invalid IRR objects was described in
https://www.ripe.net/publications/docs/ripe-731 and also implemented in
IRRd v4.

RPKI-Invalids no longer visible via whois.radb.net
==

Since recently, whois.radb.net indeed applies such filtering and no
longer displays the RPKI-conflicting IRR object for 194.32.71.0/24:

$ whois -h whois.radb.net 194.32.71.0/24
%  No entries found for the selected source(s).

The above 'No entries found' output means that the IRRd instance
filtered out the RPKI-invalid IRR objects! This is great :-)

In doing so, RADB actively helps safeguard the interests of resource
holders: it no longer is possible for adversaries to create IRR route
objects which are in conflict with RPKI ROAs via the RADB service, and
RADB also filters out RPKI-invalid IRR route/route6 objects received via
the NRTM mirror streams; additionally, this filter mechanism helps clean
up historical route objects that have become irrelevant due to
transfers. The RPKI-based filtering mechanism cleans up the past and
helps protect going forward.

Of course RADB isn't the only IRR operator who apply RPKI-based
filtering, NTT is another one, as is TC, etc; but this week's RADB
migration it certainly moves the needle in a significant way: now all
the world's largest IRR mirrors apply RPKI-based filtering! :-)

Thanks
==

I'd like to express appreciation to the NTT crew (past and present) for
helping launch the IRRd v4 initiative: Dorian Kim, Shawn Morris, Michael
Wheeler, Dan Lowe, Troy Boudreau, Camilo Cardona; to Sasha Romijn from
Reliably Coded for leading the development of IRRd v4; and to the Merit
crew who diligently worked to migrate RADB away from legacy IRRd - I
believe we collectively benefit from their efforts.

The IRRd cross-organizational open-source project is here: http://irrd.net/

お疲れ様です!

Kind regards,

Job

Re: BGP-iSec: Improved Security of Internet Routing Against Post-ROV Attacks

[Job Snijders via NANOG]

Dear Amir,

On Fri, Nov 10, 2023 at 06:02:48PM -0500, Amir Herzberg wrote:
> We will present our new work, titled: `BGP-iSec: Improved Security of
> Internet Routing Against Post-ROV Attacks', in NDSS'24.
> 
> If you're interested in security of Internet routing (BGP), and want a
> copy, see URL below, drop me a message/email - or see us in the
> conference - or just read the final version.
> 
> Available from:
> https://www.researchgate.net/publication/375553362_BGP-iSec_Improved_Security_of_Internet_Routing_Against_Post-ROV_Attacks

Thanks for freely sharing a copy! This allowed me to jot down some
initial thoughts.

* It appears that BGP-iSec requires a deployment flag day per Autonomous
  System, rather than doing security upgrades "one EBGP session at a
  time" (a deployment model both RPKI-ROV and BGPsec support).
  This lack of "per EBGP session"-granularity doesn't make for a
  feasible or viable deployment story in the global Internet routing
  system - as quite some Autonomous Systems are composed of thousands of
  routers which cannot be made to do the same thing at the same moment
  for logistical and various other reasons. What BGP-iSec promotes as a
  'feature' ("Mandatory Signatures for Announcement Integrity") - may
  turn out to be its biggest impediment towards deployment in the real
  world.

* As noted in RFC 7132 [0], Google's [1], Fastly's [2] response to
  the FCC "inquiry into internet routing vulnerabilities" which your
  paper cites; BGPsec was consciously not designed to address route
  leaks, as leaks are not precluded by the specification for BGP and
  might be the result of a local policy that is not publicly disclosed.
  To me the sentence "Even under full adoption, BGPsec does not prevent
  route leaks" reads contentious, because of an implicit assertion that
  BGPsec was intended to address route leaks.
  To me the above reads as "Even under full adoption of IPv6, world
  hunger will not be achieved". E.g. seems a false equivalence is drawn.

* It appears BGP-iSec took some inspiration from ASPA, but instead of
  signalling the list of providers out-of-band (which ASPA does via the
  RPKI publication system); BGP-iSec proposed an in-band signal in the
  form of 'UP' messages encapsulated inside BGP Path Attributes. Again,
  this suggests that BGP-iSec requires a flag day for all EBGP routers
  in a given Autonomous System; whereas deployment of the ASPA signal
  happens via a singular place (the RIR RPKI web portal), and in order
  to publish an ASPA object, no changes have to be executed on the
  Autonomous System's EBGP routers.

* The bold assertion made by the BGP-iSec authors that BGPSec offers
  "meagre benefits in partial adoption" to me seems a subjective one.
  Consider when two BGPSec-capable networks interconnect, both parties
  immediately benefit from having secured that interconnection point.
  For example, if this happens to be the interconnection between two
  major cloud providers, the advantages are massive and can positively
  benefit billions of indirect constituents. Similarly, in the past I've
  argued that only the biggest networks in the world need to do RPKI-ROV
  for all of the Internet to take benefit from that deployment action by
  a single small group. Comparing RPKI-ROV & BGPSec is apples and
  oranges; but the point stands that in an Internet with increased
  centralization we have to rethink what exactly the consequences are of
  so-called 'partial deployments' of security features.

I enjoyed reading the BGP-iSec paper and certainly view and appreciate
it as an interesting perspective on the routing security problem space;
but I think for very good reasons AS_PATH integrity (BGPsec) and AS_PATH
route leaks (OPEN Policy, OTC, ASPA) are addressed as independent
technology extensions to the routing stack.

Sometimes, trying to kill two or three birds with one stone
inadvertently introduces significant cost in unexpected places,
presenting surprising barriers to deployment.

Kind regards,

Job

[0]: https://datatracker.ietf.org/doc/html/rfc7132
[1]: https://www.fcc.gov/ecfs/document/10411283450/1
[2]: 
https://sobornost.net/~job/fastly-fcc-noi-secure-internet-routing-reply-comments-20220510-201259363-pdf.pdf

Re: RPKI unknown for superprefixes of existing ROA ?

[Job Snijders via NANOG]

On Tue, Oct 24, 2023 at 05:28:31PM -0700, Owen DeLong wrote:
> Yes, but we weren’t talking about an IXP here.
> We’re talking about an ISP.

Sure, perhaps you were

I intended to submit an example where a resource holder constructively
uses a ROA designating AS 0 as purported originator, actually helps the
overall ecosystem.

> Believe it or not, Job, there are parts of the internet that exchange
> traffic and move packets that are not IXPs.

¯\_(ツ)_/¯

Re: RPKI unknown for superprefixes of existing ROA ?

[Job Snijders via NANOG]

On Sun, 22 Oct 2023 at 20:33, Tom Beecher  wrote:

> Basically, I guess, it means that the AS 0 solution shouldn't be used, at
>> least not usually.
>>
>
> It's like everything else. Understand what the tools do and what they
> don't do, and use them appropriately.
>


A primary risk for an IXP is the existence of a more-specific of the IX
peering LAN prefix, a less-specific wouldn’t matter or inflict damage.

So in the above context an AS 0 ROAs can be useful to improve protection of
IXP Peering LANs where the IX operator doesn’t want the fabric to be
globally reachable - and one of the IX participants failed to correctly
EBGP in/out policies.

Kind regards,

Job

>

Re: RPKI unknown for superprefixes of existing ROA ?

[Job Snijders via NANOG]

On Sun, 22 Oct 2023 at 19:35, Owen DeLong  wrote:

> Actually, Job, the 1.2.0/20 would be the longest prefix announced for
> 1.2.4/24 and 1.2.7/24 in this case. It’s a rather clever end-run. The /20
> won’t match the more specific as0 ROAs, so it gets accepted. The /24s
> either aren’t advertised or they get discarded as invalid.
>


You wouldn’t create AS 0 ROAs if you want to announce the IP space pull
traffic into the discard filters on your edge.

Kind regards,

Job

>

Re: RPKI unknown for superprefixes of existing ROA ?

[Job Snijders via NANOG]

On Sun, 22 Oct 2023 at 18:10, William Herrin  wrote:

> Then someone comes along and advertises a portion of the RIR space
> larger than any allocation. Since your subnet is intentionally absent
> from the Internet, that larger route draws the packets allowing a
> hijack of your address space.
>
> In essence, this means that a ROA to AS0 doesn't work as intended.
>


Right, so in order to discard packets towards a network, it’s more robust
to actually advertise the IP space which you don’t intend to publicly use,
and use ACLs on that edge to discard the packets yourself (rather than
relying on all other ISPs having deployed ROV and less-specifics not
existing).

Given the frequency of ISPs accidentally announcing giant blocks, and this
apparently not causing much grief
https://www.ripe.net/ripe/mail/archives/routing-wg/2022-July/004588.html
I’m skeptical there much need for change.

As to Ruben’s point - when an ISP is operating their network with a default
route & an incomplete routing table, indeed chances are packets will end up
on the wrong path … because the ISP is using an incomplete routing table.

Kind regards,

Job

Re: RPKI unknown for superprefixes of existing ROA ?

[Job Snijders via NANOG]

On Sun, 22 Oct 2023 at 17:42, Amir Herzberg  wrote:

> Bill, thanks! You explained the issue much better than me. Yes, the
> problem is that, in my example, the operator was allocated  1.2.4/22 but
> the attacker is announcing  1.2.0/20, which is larger than the allocation,
> so the operator cannot issue ROA for it (or covering it). Of course, the
> RIR _could_ do it (but I don't think they do, right?). So this `superprefix
> hijack' may succeed in spite of all the ROAs that the operator could
> publish.
>
> I'm not saying this is much of a concern, as I never heard of such attacks
> in the wild, but I guess it _could_ happen in the future.
>


How is “success” measured here?

The attacker won’t be drawing traffic towards itself destined for addresses
in the /22, because of LPM
https://en.wikipedia.org/wiki/Longest_prefix_match

Attackers don’t hijack IP traffic by announcing less-specifics. It don’t
work that way.

Kind regards,

Job

>

Re: Acceptance of RPKI unknown in ROV

[Job Snijders via NANOG]

On Thu, 19 Oct 2023 at 12:12, Aftab Siddiqui 
wrote:

> A quick check to my routing table suggests that I have 206700
> preferred routes (v4/v6) to notfound (unknown) destinations. So yeah I
> don't think anyone can afford to do this right now.
>


I don’t think anyone can afford to ever do this, regardless of the number
of unknown destinations!

Imagine not being able to reach North American destinations for 23 hours
because of a cryptographic signing issue at the RIR [0] causing all ROAs to
blip out of existence.

Kind regards,

Job

[0]
https://www.arin.net/announcements/20200826/

Re: Acceptance of RPKI unknown in ROV

[Job Snijders via NANOG]

On Thu, 19 Oct 2023 at 11:56, Owen DeLong  wrote:

>
> On Thu, 19 Oct 2023 at 11:46, Owen DeLong via NANOG 
> wrote:
>
>> A question for network operators out there that implement ROV…
>>
>> Is anyone rejecting RPKI unknown routes at this time?
>>
>> I know that it’s popular to reject RPKI invalid (a ROA exists, but
>> doesn’t match the route), but I’m wondering if anyone  is currently or has
>> any plans to start rejecting routes which don’t have a matching ROA at all?
>
>
>
> This would be a bad idea and cause needless fragility in the network
> without any upsides.
>
>
> I’m not intending to advocate it, I’m asking if anyone is currently doing
> it.
>


I’m not aware of anyone doing this, and have not heard operators express
interest in doing this (probably because it seems such an unpleasant
concept).

Somewhat related:

I do know of operators that require a ROA (if it’s non-legacy space) during
their customer onboarding process, for example, in BOYIP for DIA cases.

But those operators do not expect the ROA to continually exist after the
provisioning has been completed successfully. Making the continued
availability of a route dependent on the continued validity of a ROA is
where friction starts to form.

Kind regards,

Job

>

Re: Acceptance of RPKI unknown in ROV

[Job Snijders via NANOG]

On Thu, 19 Oct 2023 at 11:46, Owen DeLong via NANOG  wrote:

> A question for network operators out there that implement ROV…
>
> Is anyone rejecting RPKI unknown routes at this time?
>
> I know that it’s popular to reject RPKI invalid (a ROA exists, but doesn’t
> match the route), but I’m wondering if anyone  is currently or has any
> plans to start rejecting routes which don’t have a matching ROA at all?



This would be a bad idea and cause needless fragility in the network
without any upsides.

Regards,

Job

Re: constraining RPKI Trust Anchors

[Job Snijders via NANOG]

Dear Martin,

On Wed, Oct 11, 2023 at 10:01:53AM +0200, Martin Pels wrote:
> I think this is important work.

Thanks!

> As you indicated in your mail you have spent quite some time compiling
> the constraints files in the appendix. Keeping them up to date
> requires tracking allocations and policy developments in all RIRs. It
> reminds me of bogon filters for unallocated IP space, and the
> associated problems of networks not updating them [0].

Yes, indeed there is a burden associated with this risk mitigation
approach. I deem tracking of ratified policies in all RIRs feasible, but
yeah... it'll definitely be a recurring quarterly todo item. The current
approach in developing these default constraint listings is to focus on
coarse-grained filters, and not bother to document unallocated space
because the resulting churn would hard to manage & distribute.

> So while each RP should be able to make policy decisions based on its
> own local criteria, managing a default set of constraints is something
> that is best done centralized. Who do you envision should manage these
> lists? RP software maintainers? RIRs? Others?

I guess initially it'll be the RP developers (like me), because who else
is chartered to produce such listings at this moment? I do intend to
keep [1] updated. Would you like to help? :-)

I envision the default constraints can be distributed via packages like
rpki-trust-anchors [2] and integral in operating systems like OpenBSD in
order to reduce the burden on operators.

A potential follow-up exercise here could be to propose to increase the
level of detail in IANA's IPv4 Address Space Registry [0] by - for
example - documenting the longer-than-/8 blocks each RIR transferred to
AFRINIC when AFRINIC was instantiated.

Kind regards,

Job

[0]: 
https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.xhtml
[1]: 
https://www.ietf.org/archive/id/draft-snijders-constraining-rpki-trust-anchors-00.html
[2]: https://packages.debian.org/stable/rpki-trust-anchors

Fwd: RADb will migrate to IRRdv4 on October 30, 2023

[Job Snijders via NANOG]

Dear all,

Please see the below announcement, I think this is really good news!

RPKI-based filtering at large databases and mirror services like RADB
really helps take the sting out of potentially harmful RPKI-invalid IRR
route objects. This will positively impact operators who use bgpq3, irrpt,
or irrtoolset/peval to generate filters, because those tools use
whois.radb.net by default.

Kind regards,

Job

-- Forwarded message -
From: Merit RADb 
Date: Thu, 28 Sep 2023 at 21:00
Subject: RADb will migrate to IRRdv4 on October 30, 2023
To: 


RADB will switch to IRRdv4 on October 30
[image: RADb blue logo-tagline]


Dear RADb Community,



As originally announced in March 2023, RADb is transitioning to
IRRd (Internet Routing Registry daemon) version 4. On October 30th, 2023,
we will migrate RADb over from IRRdv3, which you are currently using, to
IRRDv4. Below are links to demo servers that will allow you to interact
with the new service prior to the migration.



As you start planning for the transition, here are some key things to note:

   -

   We will import the current database into IRRDv4 on October 30th.
   -

   We will work to maintain the current serial numbers for NRTM. If we are
   unable to you will need to update your mirror before NRTM can continue.
   -

   We will enable RPKI.
   -

  The rpki-ov-state RPSL field notes the RPKI status (valid, invalid,
  or not-found)
  -

  whois will hide RPKI invalid objects unless you use !fno-rpki-filters
  -

   IRRDv4 is more strict, but we are starting with most of this strictness
   disabled initially.
   -

   IRRDv4 may output RPSL objects differently.
   -

   The RADb API is the same, but may return different error messages, or
   similar error messages with different wording. For example, it will now
   report RPKI errors.
   -

   Existing mntner objects will be imported even with invalid
   admin-c/tech-c handler. Any updates will require at least one person or
   role object associated with it. We will be updating our registration
   process to create these objects for you.
   -

   New mntner objects must also have at least one person or role object
   associated with it. We will initially handle this for you the the customer
   portal, but you should check the created person or role for correctness.
   -

   New mntner objects will use bcrypt for passwords. This will happen
   automatically for you through the customer portal

The following demo environments are available for your testing (please use
your existing credentials):

   -

   whois2.radb.net <#m_-1278871392211290094_m_-2351897707799796429_> -
   IRRdv4 with current RADb objects and live mirroring of other RIRs
   -

   whois2demo.radb.net <#m_-1278871392211290094_m_-2351897707799796429_> -
   IRRdv4 with sample data (v4 version of whoisdemo.radb.net
   <#m_-1278871392211290094_m_-2351897707799796429_>)
   -

   wwwdemo.radb.net
   

- demo web portal backed by whois2.radb.net
   

   -

   apidemo2.radb.net
   

- demo API (available October 1st)

This move to IRRDv4 will bring many benefits to you, including:

   -

   Improved RADb data quality
   -

   Prevention of RPKI-invalid objects from being created in RADb
   -

   Prevention of object creation for invalid prefixes (such as RFC 1918
   space)
   -

   as-set and route-set object creation will be limited to the account that
   maintains the aut-num o

Re: constraining RPKI Trust Anchors

[Job Snijders via NANOG]

Dear Matthew,

See below

On Tue, 26 Sep 2023 at 20:49, Matthew Petach  wrote:

>
> Job,
>
> This looks fantastic, thank you!
>
> For my edification and clarification, the reason you don't need a
>
> deny 2000::/3
>
> or
>
> deny 0::/0
>
> at the bottom of the ARIN list of allows is that every file comes with an
> implicit "deny all", is that correct?
>
> Is there a drawback to adding the explicit "deny 0::/0" at the bottom of
> the file, to make it clear that everything else will return "invalid"?
> I tend to prefer being explicit in my configurations, rather than
> depending upon implicit behaviours which might change with future versions
> of software releases.
>


Sorry, the lede is a bit buried on how exactly the policy language works.
It’s in the appendix, and the example source code offers hints too
https://marc.info/?l=openbsd-tech&m=169574305230941&w=2

I’ll elaborate a bit here: the order of the entries in each constraints
file is not significant. All “deny” entries take precedence over all “allow
entries”. Individual “deny” entries may not overlap with any other “deny”
entries. Individual “allow” entries may not overlap with other “allow”
entries. Deny entries are available to punch holes in allow entries, as a
shortcut. If overlapping constraints are configured the program errors.

If a constraint is applied to a resource class (for example by specifying
just a single “allow 2000::/3” entry), all EE certificates with IPv6
resources in their respective RFC 3779 extensions under that TA must be
encompassed in that single allow entry. So the “implicit deny” comes into
effect the moment you’d configure at least one allow entry for a resource
class (IPv4, IPv6, or AS numbers). This is why no additional “deny the
rest” line is needed in the case of ARIN.

This approach was the best I could muster on short notice. My objective
wasn’t to invent a policy language everyone should adopt, but rather to
draw attention to the concept of constraining RPKI trust anchors and
provide some running code to advance the dialogue.

Thank you for reading the document and asking questions!

Kind regards,

Job

>

constraining RPKI Trust Anchors

[Job Snijders via NANOG]

Dear all,

Two weeks ago AFRINIC was placed under receivership by the Supreme Court
of Mauritius. This event prompted me to rethink the RPKI trust model and
associated risk surface.

The RPKI technology was designed to be versatile and flexible to
accommodate a myriad of real-world deployment scenarios including
multiple trust anchors existing, inter-registry transfers, multiple
transports, and permissionless innovation for signed objects, for
example. All good and well ... but ofcouse there is a fine print. :-)

Over the years various people have expressed astonishment about each RIR
having issued so-called 'all-resources' (0.0.0.0/0 + ::/0) trust anchor
certificates, but this aspect often is misunderstood: the risk is not
necessarily in the listing of 'all-resources' itself, it is in the RIR
being able to issue an 'all-resources' certificate in the first place.
RPKI trust anchor operators indeed can voluntarily reduce the scope of
subordinate Internet Number Resources, but just as easily increase the
scope of their authority. In other words, a trust anchor cannot truly
constrain itself.

Upon reconsideration on how exactly RPKI hooks into the real world; I
concluded trust anchors do not require unbounded trust in order to
provide constructive services in the realm of BGP routing security.

Some examples: ARIN does not support inter-RIR IPv6 transfers, so it
would not make any sense to see a ROA subordinate to ARIN's trust anchor
covering RIPE-managed IPv6 space. Conversely, it wouldn't make sense
to observe a ROA covering ARIN-managed IPv6 space under APNIC's,
LACNIC's, or RIPE's trust anchor - even if a cryptographically valid
certificate path existed. Along these lines AFRINIC doesn't support
inter-RIR transfers of any kind; and none of the RIRs have authority
over private resources like 10.0.0.0/8 or AS 65535. It seems feasible to
paint constraints around RPKI trust anchors in broad strokes.

Over the last two weeks I've diligently worked with Theo Buehler to
research RIR transfer policies, untangle the history of the IANA->RIR
and RIR->RIR allocation spaghetti, design & implement a maintainable
constraints mechanism for rpki-client(8), and publicly document the
concept of applying operator-defined policy to derived trust arcs.

Please take a moment to read
https://www.ietf.org/archive/id/draft-snijders-constraining-rpki-trust-anchors-00.html

Your feedback is appreciated.

Kind regards,

Job

Re: JunOS/FRR/Nokia et al BGP critical issue

[Job Snijders via NANOG]

On Fri, Sep 01, 2023 at 11:54:57AM +0100, Nick Hilliard wrote:
> it's not really. If the receiving BGP stack understands the attribute,
> then it should be parsed as default, i.e. carefully.  Unfortunately,
> junos slipped up on this and didn't validate the input correctly,
> which is a parsing bug. Param validation bugs happen. They shouldn't
> happen, but they do.
> 
> If an intermediate router doesn't understand a transitive attribute,
> it should be ignored, and life should move on.

+1 to what Nick stated.

Rob Shakir did a great job describing the various 'scopes' in which BGP
errors can appear and what that means for the 'blast radius'.

I recommend everyone to read section 3 of this draft document:
https://datatracker.ietf.org/doc/html/draft-ietf-grow-ops-reqs-for-bgp-error-handling-07#section-3

Kind regards,

Job

Re: Friday Thanks

[Job Snijders via NANOG]

On Fri, 11 Aug 2023 at 17:54, Graham Johnston via NANOG 
wrote:

> I've been busy over the last few days trying to clean up IRR information
> for our subnets and issue ROAs for our address space. Invariably I came
> across stale entries in various IRR databases. They aren't really hurting
> me, but I feel like there shouldn't be competing incorrect information out
> there that I'm not in control of. The database maintainers have been mixed
> in their response so far. This email isn't about name and shame though, I'm
> not at that point. Instead, I want to provide thanks to two organizations
> that have been very responsive and easy to deal with in getting records
> cleaned up.
>
> RADb - Thank you.
>
> Level3/CenturyLink/Lumen - Thank you.
>
> Separately, I know there is some work to do with ARIN's recent RPKI/IRR
> changes, but as someone from a regional ISP, rather than a national or
> multi-national ISP, I can say that the tools are moving in the
> right direction. The experience right now is better than it was several
> years ago. Thank you ARIN for the improvements and the dedication to work
> with us on making further improvements.
>
> Have a good weekend,
>

I think that’s great to read!

Kind regards,

Job

Re: Changes to ARIN Online - Routing Security Dashboard - RPKI & IRR integration (was: Fwd: [arin-announce] New Features Added to ARIN Online)

[Job Snijders via NANOG]

Dear Mark,

Thank you for sharing all the details in your previous email. For
brevity I'm snipping most of your reply.

On Tue, Aug 08, 2023 at 03:59:19PM +, Mark Kosters wrote:
> Job Snijders wrote:
>
> > Would it not be advantageous to create at a minimum the 256 of the
> > 'least-specific' objects?
> 
> MK: That may be a reasonable approach. Do you see any adverse effects
> in simplifying the IRR Route creation logic to just have
> least-specific?

I don't think I see a downside of mapping a single VRP to a single IRR
route/route6 object.

Synthesizing only the least-specific is how RPKI-integration was
implemented in IRRd v4, and that implemntation has seen quite some
flight hours by now. The approach seemed to work well in the last ~ 5
years. No request ever came up to extend IRRd v4 to synthesize (a
limited number of) more-specific route/route6 objects if maxLength was
present in the ROA generation request.

IRRd v4 does expose the 'maxLength' value by inserting a non-standard
RPSL attribute called 'max-length:' in the route/route6 object. While
this is not IETF-standardized, it seemed intuitive enough. RPSL parsers
are expected to ignore what they don't recognize anyway. ARIN could
mimick this practise. See the following example:

$ whois -h rr.ntt.net -- '-s RPKI -T route6 2001:67c:208c::/48' \
  | egrep "route6|max-length|origin" | paste - - -
route6: 2001:67c:208c::/48  max-length: 48  origin: AS0
route6: 2001:67c:208c::/48  max-length: 48  origin: AS15562

I'd suggest to consider simplying the IRR creation logic to just a
singular least-specific route/route object, and not bother with
generating 'potentially up to 256' more-specific route/route6 objects.

I see a few advantages:

G* Every single VRP resulting from a ROA generation request will always
  result in the instantiation of exactly one IRR route/route6 object. As
  I understand the current proposal, sometimes a route/route6 object is
  created, sometimes not, depending on the value of the maxLength. In
  this I see a source for potential confusion.

* By creating just 1 route/route6 object per validated ROA payload, any
  potential load on the NRTM mirror ecosystem is minimized to the
  fullest extend possible: 65,285 ROAs on 'rpki.arin.net' will result in
  72,082 Validated ROA Payloads thus result in 13,933 'route6:' objects
  and 58,149 'route:' objects. A significant smaller number compared to
  expanding up to 256 additional objects if maxLength is set wide enough.

* the use of maxLength isn't really recommended anyway, see BCP 185 /
  RFC 9319 https://datatracker.ietf.org/doc/html/rfc9319. As a rule of
  thumb I recommend: 1 BGP announcement == 1 ROA == 1 IRR object and
  avoid just using maxLength all together.

* Most transit providers and IXP route server operators create so-called
  'upto /24' prefix-list-filters, regardless of maxLength values. So in
  the vast majority of cases I don't think the additional up to 256
  more-specific route objects would change anything.

Finally, if people can articulate why exactly synthesizing up to 256
more-specific route objects really would be a useful feature, it can
always be added at a later point in time. Adding extra objects to the
IRR has always been easier than removing them. :-)

Thanks!

Kind regards,

Job

Re: Changes to ARIN Online - Routing Security Dashboard - RPKI & IRR integration (was: Fwd: [arin-announce] New Features Added to ARIN Online)

[Job Snijders via NANOG]

Dear John, ARIN, NANOG,

On Mon, Aug 07, 2023 at 06:24:09PM +, John Curran wrote:
> We have made some fairly significant changes for those customers using
> ARIN Online for routing security administration – see attached message
> for specifics.

Yes, significant changes! I very much appreciate ARIN's efforts to
streamline IRR & RPKI operations for INR holders. :-)

I too think that having to enter "sorta similar" data in 2 places of
course is more prone to error (in terms of internal discrepancies
between the two inputs of data) compared to entering routing data in
just one place. I imagine the scheduled simplification of the user
interface (intertwining IRR & RPKI) will lead to improved data accuracy
and fewer operator errors. Thank you ARIN for that!

I think the industry's transition from plain-text IRR data towards
cryptographically verifiable RPKI data really starts happening when
there are automated processes coupling the two sets of data, and indeed
also retroactively glueing the two together (within ARIN's auspices the
'ARIN Online' environment).

A few questions arose in my mind while wondering about implementation
aspects on ARIN's side:

- is the IRR state directly derived from the RPKI state?

  An example for context: should some kind of unfortunate failure happen
  in ARIN's HSMs and thusly a new Manifest + CRL pair isn't signed and
  published before the 'nextUpdate' timestamp of the previous pair,
  would the associated IRR objects be deleted via NRTM? Or is the
  creation of ROAs and IRR route:/route6: objects discoupled in the
  sense that an operator creates an abstract object which then is
  transformed into both IRR and RPKI objects?

- What is the expected delay (if any) between creating a RPKI ROA and
  the associated IRR route/route6 objects appearing via NRTM?
  Is there online documentation outlining expectations, and is there
  internal monitoring on the delivery of the RPKI-to-IRR transformation
  service?

- The documentation states "If the creation of a ROA would result in
  more than 256 IRR Route Objects, no managed IRR Route Objects will be
  created." - but, why not? Would it not be advantageous to create at
  a minimum the 256 of the 'least-specific' objects? I wonder if the
  current approach violates the principle of least astonishment in the
  sense that an operator picking an unfortunate 'maxLength' value
  results in no IRR objects at all.

Kind regards,

Job

Fw: [Sidrops] Estimating timeline for ASPA Deployment

[Job Snijders via NANOG]

Heya NANOG,

I thought this email conversation might be of interest to the group:
https://mailarchive.ietf.org/arch/msg/sidrops/RdbccLbXEHUrmmdIS5K9GOdJFXA/

Kind regards,

Job

- Forwarded message from Job Snijders  -

Date: Fri, 19 May 2023 20:54:26 +0200
From: Job Snijders 
To: sidr...@ietf.org
Subject: Re: [Sidrops] Estimating timeline for ASPA Deployment

Dear Tony,

On Fri, May 19, 2023 at 10:18:58AM -0400, Tony Tauber wrote:
> I wonder what people think of likely soonest target dates for adoption
> of ASPA?

... Where do psychics go to dance?
 The crystal ball ;-) ...

Disclaimer: The following is based on unsubstantiated information and/or
my gutfeeling, I can't vouch for accuracy.

I'd like to share some information categorized as 'here work needs to be
done', a timeline, and a conclusion referencing earlier RPKI
experiences.

What needs to happen where?
---

There are a few categories of 'moving parts' in the global Internet
routing system that need upgrades in order for ASPA to see widespread
adoption.

Category  | (Non-exhaustive) list of implementations
--+--
Relaying Party packages:   OpenBSD rpki-client, NLnet Labs Routinator,
   LACNIC/NicMX FORT, Cloudflare octorpki,
   Mikhail Puzanov's rpki-prover, ZDNS RPSTIR2.

Signer implementations:NLnet Labs Krill, Dragon Labs rpki.net, and a
   number of Regional Internet Registries (RIRs)
   have their own in-house Signer solution.

BGP implementations:   OpenBSD OpenBGPD, NIC.CZ BIRD, FRRouting,
   Nokia SR-OS, Cisco IOS XR, Juniper Junos, and
   many others, see the 'BGP speakers' list at:
   http://largebgpcommunities.net/implementations/

Standalone RTR servers:StayRTR, NLnet Labs RTRTR, GoRTR, rpkirtr
Integrated RTR servers:FORT, Routinator

In order for there to be any ASPA deployment, at least one ASPA-capable
implementation in all of the RP, Signer, and BGP categories is needed.

Without a Signer there isn't anything for the RP to validate, without an
RP there isn't any for the BGP speaker to verify BGP UPDATEs against.
This is a great example of why the IETF process is so valuable: this
process has brought together many stakeholders from different corners of
the ecosystem each with different roles to jointly develop a solution
for BGP route leaks.

Timeline


2023 - "the early wave": ASPA support arrives in select BGP, RP, RTR,
   and Signer implementations.

   At the moment of writing OpenBSD rpki-client & OpenBGPD; NLnet
   Labs Routinator, Krill & RTRTR, StayRTR, rpki-prover, and RIPE
   NCC have either already released ASPA-capable software or are in
   an advanced stage to do so.

   These 'early wave' implementations are incredible achievements as
   there weren't any examples for the developers to look at, they
   were building across unchartered territory.

   One Internet Exchange Point managed to deploy fully ASPA-capable
   Route Server stack using bleeding edge software, however this
   effort should be considered an outliner from a timeline perspective:
   https://mailman.nanog.org/pipermail/nanog/2023-February/221471.html

2024 - The IETF ratifies ASPA by publishing a series of RFC documents
   detailing the specification. While SIDROPS (this working group)
   at this moment in time is in the late stages of specifying the
   ASPA standard, the Internet Engineering Steering Group (IESG) and
   RFC Editor still need to review the draft documents; all in all
   this might take another 6-10 months (from now).

   BGP monitoring software such as BGPAlerter and BGP.Tools might
   take an interest and start using ASPA data to enhance their
   alerting capabilities: a growing amount of ASPA data (which can
   be used to identify route leaks) is becoming available through
   the 'delegated' (permissionless) RPKI distribution channels.

2025 - Having access to both published RFCs and multiple battle-tested
   RP implementations (which originated in the 'early wave'), more
   and more RIRs will feel comfortable scheduling and committing
   development time to productize an ASPA-offering for their
   respective RPKI dashboard/web portal/api services.

   Why 2025? RIRs oftentimes are somewhat conservative and like to
   see which way the wind blows before offering new services to
   their constitutes. This is very understandable as for an RIR to
   offer ASPA the work involved is significantly more than just
   developing the technical Signer software. RIRs ne

Re: FIDO2/Passkey now supported for 2FA for ARIN Online (was: Fwd: [arin-announce] New Features Added to ARIN Online)

[Job Snijders via NANOG]

Dear John,

On Tue, Jan 03, 2023 at 08:57:47PM +, John Curran wrote:
> NANOGers -
> 
> FYI - ARIN Online now has FIDO2/Passkey as an option for two-factor
> authentication (2FA) - this is a noted priority for some
> organizations.

Thank you for sharing this wonderful news! I tried the new shiny thing,
it was very easy to switch from TOPT to FIDO2.

A feature request: it would be nice if (at least) two FIDO2 Security
Keys can be associated to a single account. I like having a spare key
with traveling and all. :-)

Kind regards,

Job

RPKI's 2022 Year in Review: growth & innovation

[Job Snijders via NANOG]

Dear all,

With 2023 at our doorstep, I'd like to share some perspective on how
RPKI evolved in the year 2022.

Impact on the Global Internet Routing System


Decision makers might wonder: is investing time and resources worth it?
What is the effectiveness of RPKI Route Origin Validation (RPKI-ROV)?
In the last year a number of interesting reports were published.

Even though less than half of BGP routes is covered by RPKI ROAs [6],
based on flow data, Kentik estimates [2] nowadays the majority of IP
traffic is destined towards RPKI-valid BGP routes. Their follow-up
report [3] (analysing BGP control-plane data) suggests that evaluation
of a BGP route as RPKI-invalid reduces its propagation by anywhere
between one half to two thirds. Cloudflare [4] published a report
analysing data-plane connectivity between a select number of ASes and
RPKI-invalid destinations: they estimate 6.5% (lower-bound) of
residential Internet users enjoy the benefits their ISP doing RPKI-ROV.
Another experiment report [5] (focussed on data-plane connectivity
between validators and RPKI-valid/RPKI-invalid destinations), concluded
the existence of RPKI ROAs helped move 75% of test traffic towards the
correct destination.

The above metrics might appear all over the place (6.5% up to 75%), but
keep in mind these analyses are not mutually exclusive. Observations of
the Internet's topology are a function of the observer's vantage point.

All the referenced reports agree on key points:

  * ROAs have a measurable & significant impact on global IP traffic delivery
  * RPKI-ROV helps reduce the "blast radius" of BGP routing incidents
  * They recommend to continue the global deployment of RPKI-ROV
(rejecting RPKI-invalid BGP routes), and create ROAs for all IP
address space.

Year to Year Growth of the distributed RPKI database


In comparison to "effectiveness", the bare existence, size, contents,
and number of Signed Objects in the globally distributed RPKI repository
system is much easier to quantify.

The below table was constructed by comparing two December 31st
RPKIviews.org snapshots [1] of validated RPKI caches, primed with the
ARIN, AFRINIC, APNIC, LACNIC, and RIPE Trust Anchors.

   2021-12-31 2022-12-31
Total cache size (KiB):   996,216  1,240,572  (+24%)
Total number of files (objects):  192,503242,969  (+26%)
Publication servers (FQDNs):   36 52  (+44%)
Certification authorities: 28,328 34,901  (+23%)
Route origin authorizations:  101,645138,323  (+36%)
Unique VRPs:  302,025390,752  (+29%)
IPv4 addresses covered: 1,139,561,719  1,354,270,410  (+19%)
IPv6 addresses covered: 7,499,405,083  9,446,853,925  (+26%) *10^24
Unique origin ASNs in ROAs:27,174 34,455  (+27%)

A healthy growth rate across the board!

With the ubiquitous availability of "Publication as a Service" hosted by
RIRs, I expect (and hope!) the growth of the number of distinct
publication servers to stall, or even drop in 2023.

The number of Certification Authorities (CAs) closely corresponds to the
number of RIR members (RIR customers) who opted to enable RPKI services
for their Internet Number Resources, making it a useful proxy metric to
understand how many organisations are creating RPKI ROAs.

A single Route origin authorizations (ROA) can contain one or more
Validated ROA Payloads (VRPs), and one or multiple ROAs can contain the
exact same VRP information. "Unique" in the above table indicates the
metric's underlaying data was deduplicated.

Each ROA can only contain a single Origin ASN. Multiple ROAs can refer
to the same Origin ASN value.

Innovation through Standardisation
==

The IETF SIDROPS [7] working group (the designated forum in which
volunteers collaborate to define and specify open standards for RPKI and
RPKI-based technologies) was fairly productive in 2022 and managed to
publish 5 RFCs:

RFC 9286 - Manifests for the RPKI   (revision)
RFC 9255 - The 'I' in RPKI Does Not Stand for Identity (clarification)
RFC 9319 - The Use of maxLength in the RPKI(clarification)
RFC 9323 - A Profile for RPKI Signed Checklists (RSCs)(innovation)
RFC 9324 - Policy Based on the RPKI without Route Refresh (innovation)

The above body of work consists mostly of revisions of older work or
clarifications on how to use the RPKI, to me this demonstrates a
somewhat conservative approach (rather than innovation at breakneck
speed), which I consider a good thing.

Outlook & Conclusion


Now that globally Route Origin Validation has advanced as far as it has,
the next obvious target is BGP path validation, to mitigate two distinct
problems: BGP route leaks and BGP AS_PATH spoofing. Both painful to
network operato

Re: Geoip database update

[Job Snijders via NANOG]

On Sat, Dec 17, 2022 at 04:58:18PM -0800, Randy Bush wrote:
> https://www.rfc-archive.org/getrfc?rfc=9092
> 
> and note that massimo has a collio toolset
> 
> https://github.com/massimocandela/geofeed-finder

Rpki-client (version 8.2 and higher) supports authenticating signed
Geofeed data against the RPKI:

First figure out the location of the Geofeed data (the above mentioned
'geofeed-finder' utility will do a better job searching at scale!):

$ whois -h whois.ripe.net 2001:67c:208c::/48 | egrep 'inet6num|Geofeed '
inet6num:   2001:67c:208c::/48
remarks:Geofeed https://sobornost.net/geofeed.csv

Then validate the embedded signature:

$ sudo apt install rpki-client && sudo systemctl start rpki-client
$ wget https://sobornost.net/geofeed.csv
$ rpki-client -j -f geofeed.csv
{
"file": "geofeed.csv",
"hash_id": "VOXBRdQpiyALlLRdo3OkLbLIY4PexRlci/0EM9Fc21U=",
"type": "geofeed",
"ski": "D4:05:34:DB:56:A6:4D:A2:ED:4D:EF:AD:A9:C1:31:DA:19:56:DC:A7",
"cert_issuer": "/CN=caa805dbac364749b9b115590ab6ef0f970cdbd8",
"cert_serial": "06",
"aki": "CA:A8:05:DB:AC:36:47:49:B9:B1:15:59:0A:B6:EF:0F:97:0C:DB:D8",
"aia": 
"rsync://rpki.ripe.net/repository/DEFAULT/yqgF26w2R0m5sRVZCrbvD5cM29g.cer",
"valid_until": 1700930092,
"records": [
{ "prefix": "2001:67c:208c::/48", "location": 
"NL,NL-NH,Amsterdam,"}
],
"validation": "OK"
}

Kind regards,

Job

Re: AS3356 Announcing 2000::/12

[Job Snijders via NANOG]

The Internet delivers when we need it the most! :-)

https://is2000slash12announcedagain.com/

Props to Ben Cartwright-Cox

Re: AS3356 Announcing 2000::/12

[Job Snijders via NANOG]

Hi all,

On Wed, Dec 07, 2022 at 08:24:54PM -0800, Ryan Hamel wrote:
> AS3356 has been announcing 2000::/12 for about 3 hours now, an aggregate
> covering over 23K prefixes (just over 25%) of the IPv6 DFZ.

A few months ago I wrote: "Frequently Asked Questions about 2000::/12
and related routing errors":

https://www.ripe.net/ripe/mail/archives/routing-wg/2022-July/004588.html

Kind regards,

Job

Re: afrinic rpki issue

[Job Snijders via NANOG]

Hi all,

It appears PacketVis correctly identified an issue.

AFRINIC's self-signed root AfriNIC.cer [1] points via its SIA to
'afrinic-ca.cer' [2] which in turn references a RPKI Manifest named
'K1eJenypZMPIt_e92qek2jSpj4A.mft'.

The K1eJenypZMPIt_e92qek2jSpj4A Manifest lists 499 Certificate
Authorities. This Manifest represents the demarcation point between
"Afrinic as root CA operator" and "Afrinic hosting rpki on behalf of its
members". In other words; this is an important top-level Manifest in the
critical path towards the ROAs of the Afrinic members.

There was a ~ 7 hour gap in the validity window of this Manifest and its
companion CRL (from 20221120T000311Z until 20221120T071514Z). The
serials 1E19 and 1E1A (respectively 12B2 and 12B3) are successive.

rpki.afrinic.net/repository/afrinic/K1eJenypZMPIt_e92qek2jSpj4A.crl
CRL Serial Number:1E19
CRL valid since:  Nov 18 00:03:11 2022 GMT
CRL valid until:  Nov 20 00:03:11 2022 GMT

CRL Serial Number:1E1A
CRL valid since:  Nov 20 07:15:12 2022 GMT
CRL valid until:  Nov 22 07:15:12 2022 GMT

rpki.afrinic.net/repository/afrinic/K1eJenypZMPIt_e92qek2jSpj4A.mft
Manifest Number:  12B2
Manifest valid since: Nov 18 00:03:13 2022 GMT
Manifest valid until: Nov 20 00:03:13 2022 GMT

Manifest Number:  12B3
Manifest valid since: Nov 20 07:15:14 2022 GMT
Manifest valid until: Nov 22 07:15:14 2022 GMT

(The above can be reconstructed using archives from http://www.rpkiviews.org)

The rcynic validator hosted at Afrinic also noticed a gap in objects:
https://validator.afrinic.net/rpki/rcynic/rpki.afrinic.net_week_svg.html

A possible recommendation might be to increase the validity window of
these two objects from a sliding 48-hour window to a 1 or 2 week window.
This way any stalling in the issuance process wouldn't case operational
issues on the weekend.

Kind regards,

Job

[1]: SKI EB:68:0F:38:F5:D6:C7:1B:B4:B1:06:B8:BD:06:58:50:12:DA:31:B6
[2]: SKI 2B:57:89:7A:7C:A9:64:C3:C8:B7:F7:BD:DA:A7:A4:DA:34:A9:8F:80



On Sat, Nov 19, 2022 at 08:36:23PM -0800, Randy Bush wrote:
> From: PacketVis 
> Date: Sun, 20 Nov 2022 04:30:44 +
> 
> Possible TA malfunction or incomplete VRP file: 73.95% of the ROAs 
> disappeared from afrinic
> 
> See more details about the event:
> https://packetvis.com/#/bgp/event/905ec8b7d37e89a2d7b547bca99fd57e-372b0bf3-9056-407e-9e8d-e986567155fc/4f309cb51ba9314fafa64da53d007e342faca613

Re: Why do ROV-ASes announce some invalid route?

[Job Snijders via NANOG]

Dear 孙乐童,

On Mon, Nov 07, 2022 at 08:40:57PM +0800, 孙乐童 wrote:
> We learned from Cloudflare's https://isbgpsafeyet.com/ that some ASes
> have deployed RPKI Origin Validation (ROV). However, we downloaded BGP
> collection data from RouteViews and RipeRis platforms and found that
> some ROV-ASes can announce some invalid routes. For example, from RIB
> data at 2022-10-31 00:00:00, 13 out of 17 ASes which declared to
> deploy ROV announced invalid routes, and we list the number of related
> prefixes for each AS below.
>
> [snip]
> 
> As a comparison, we count the invalid routes the non-ROV ASes (also
> declared in https://isbgpsafeyet.com/) announces, as below:
> 
> We can see that ROV ASes announced apparently fewer invalid routes
> compared to the non-ROV ASes, though they did not filter all the
> invalids. 
>
> [snip]
> 
> Can anyone help us to correctly interpret this case? Thank you very much.

You ask great questions! I hope an answer to your questions can be found
in a message I sent a year ago:

https://mailman.nanog.org/pipermail/nanog/2021-April/213346.html

The summary: in any sufficiently large network, chances are not 100% of
all equipment supports RPKI-based BGP Route Origin Validation; in such
cases a handful of invalid routes may still percolate through the
system. Another contributing factor might be certain types of software
upgrades; where ROV temporarily is disabled on one or more devices. Or
perhaps an ISP made a handful of exceptions for test/beacon invalid
routes to propagate.

Kind regards,

Job 

Re: Fastly Peering Contact

[Job Snijders via NANOG]

Hi Dustin, others,

Sure thing! Someone from the Fastly peering team will follow up with you
off-list.

Information about peering with Fastly: https://www.peeringdb.com/asn/54113
and  https://www.fastly.com/peering/

Kind regards,

Job

On Fri, 30 Sep 2022 at 14:39, Dustin Brooks  wrote:

> Can someone from Fastly reach out to me about peering.
>
>
>
> --
>
> *Dustin Brooks*
>
> *Sr. Network Engineer*
>
>
>
> [image: signature_4080889297]
>
>
>
> 2028 Highway 115,
> 
>  Mansura, LA 71350
> 
>
> 318.597.0303 Office | 337.781.4506 Mobile |  www.Conterra.com
> 
>
>
> This e-mail may contain information that is confidential or privileged. If
> you are not the intended recipient, do not read, copy or distribute the
> e-mail or any attachments. Instead, please notify the sender and delete the
> e-mail and any attachments. Thank you.
>

Request for BGP Community-to-text mappings for BGP Looking Glass

[Job Snijders via NANOG]

Dear all,

I'd like to ask help from the EBGP hivemind: the shiny new BGP looking
glass at https://lg.ring.nlnog.net/ supports displaying text strings
mapped from BGP community values (both simple and large communities).

Mapping BGP Community values to simple English human-readable text
phrases can help network operators better understand what the heck is
going on.

An example can be seen here, if you hover your mouse pointer over
2914:3200 you'll see "Europe":
https://lg.ring.nlnog.net/prefix?q=2001%3A67c%3A208c%3A%3A&match=exact&peer=all

This looking glass does not cache results, it is real time.  What's
displayed on the website comes straight from the route collector's BGP
daemon.

Contributing mappings is very easy: just send pull-requests for files
in this directory: 
https://github.com/NLNOG/lg.ring.nlnog.net/tree/main/communities
The data format supports wildcards and ranges in case blocks of BGP
communities basically mean the same thing.

Please consider submitting what you know! The more the merrier. 

[ disclaimer & welcome sign: Anyone can contribute! You don't need to be
  the holder of a given ASN to submit comm2text mappings for that ASN.
  All data is user-submitted. No attempt is made to validate the
  mappings for accuracy. ]

Shout out to Teun Vink for authoring this very pretty webinterface for
OpenBGPD's JSON web API!

Kind regards,

Job

Re: Looking for contact at Fastly

[Job Snijders via NANOG]

Dear Mark,

I’ll follow up off-list.

Kind regards,

Job

On Fri, 16 Sep 2022 at 20:06, Mark Spring  wrote:

> In short, I am having issues with a couple of our subnets not being able
> to traverse a fastly peer which I don't manage, it is upstream from me. I
> need to get this resolved as it is causing issues with a good amount of
> traffic. I emailed their noc several hours ago with no response yet.
>
> Mark Spring
> Information Systems Manager
>

Re: Providing geofeed info to Google

[Job Snijders via NANOG]

On Tue, Aug 30, 2022 at 01:28:18PM -0700, Hugo Slabbert wrote:
> @Job:
> 
> Thanks! I was aware of the RIPE whois option, but the relevant resources
> for us are in ARIN.  I wasn't aware of the RPSL *remark* option for
> providing that.  We should be able to give that a bash.

Hmmm, there might be an obstacle due to lack of inetnum support in ARIN:
https://www.arin.net/participate/community/acsp/suggestions/2021/2021-27/

However there is good news: the last paragraph of RFC 9092 section 3
suggests a workaround specific to ARIN:

"Currently, the registry data published by ARIN are not the same
RPSL as that of the other registries (see [RFC7485] for a survey
of the WHOIS Tower of Babel); therefore, when fetching from ARIN
via FTP [RFC0959], WHOIS [RFC3912], the Registration Data Access
Protocol (RDAP) [RFC9082], etc., the "NetRange" attribute/key
MUST be treated as "inetnum", and the "Comment" attribute MUST
be treated as "remarks".

Perhaps you insert a "Comment: Geofeed https://xxx/geofeed.csv"; in the
place where NetRange blobs come from?

Kind regards,

Job

Re: Providing geofeed info to Google

[Job Snijders via NANOG]

Dear Hugo,

On Tue, Aug 30, 2022 at 12:34:41PM -0700, Hugo Slabbert wrote:
> Google folks:
> 
> I see historical reference to needing to use the Google Peering Portal (
> http://peering.google.com) if you need to provide Google with geofeed info
> for GeoIP info on network blocks, ref
> https://mailman.nanog.org/pipermail/nanog/2015-May/075229.html.
> 
> Is that still the case?  Are there any avenues to provide Google with
> geofeed info if you're *not* currently peering with 15169? Or to get access
> to just the geofeed update portion of the Peering Portal?

(I don't work for Google), but ...

There is a RFC detailing how to find Geofeed data (and make Geofeed data
findable): https://datatracker.ietf.org/doc/html/rfc9092

The idea is that in inetnum/inet6num objects (which are maintained by
the IP prefix holder), the holder can point to the location where
Geofeed data can be found.

There are a few methods:

1) Use the 'geofeed:' RPSL attribute (the RIPE Whois server supports
   this), example:

   $ whois -h whois.ripe.net 146.75.0.0/16 | grep geofeed
   geofeed:https://ip-geolocation.fastly.com/

2) A slightly uglier hack: stick a reference to the Geofeed location in
   a RPSL remark (should work in databases which don't (yet) support the
   'geofeed:' attribute), example:

   $ whois -h whois.ripe.net 2001:67c:208c::/48 | grep Geofeed
   remarks:Geofeed https://sobornost.net/geofeed.csv

Kind regards,

Job

Re: [EXTERNAL] Re: Yet another BGP hijacking towards AS16509

[Job Snijders via NANOG]

Heya,

On Wed, Aug 24, 2022 at 09:17:03AM +0200, Claudio Jeker wrote:
> On Tue, Aug 23, 2022 at 08:07:29PM +0200, Job Snijders via NANOG wrote:
> > In this sense, ASPA (just by itself) suffers the same challenge as
> > RPKI ROA-based Origin Validation: the input (the BGP AS_PATH) is
> > unsigned and unsecured; thus spoofable.
> 
> ASPA enforces that the neighbor AS appears as first element in the
> ASPATH. It also disallows empty ASPATHs from eBGP sessions. 

Yup, this is a helpful property of ASPA. ASPA also nukes routes which
have an AS_SET segment anywhere in the AS_PATH (which helps the
community to get a move on with 
https://datatracker.ietf.org/doc/html/draft-ietf-idr-deprecate-as-set-confed-set)
The addition of type of constraints helps keep the global Internet
routing tables clean.

> Because of this spoofing becomes harder. The problem is that this only
> works for paths that are validated by ASPA (all AS hops have been
> verified). An ASPA-unknown path can still be spoofed.

We might be talking about different types of 'spoofing'. ASPA doesn't
help verify the *authenticity* of the neighbor (or the ASes behind the
neighbor). Does the AS number transmitted in the BGP OPEN message really
belong to the entity that controls the router on the other side of the
link? Is the neighbor on the other side of the IX Route Server really
who they claim they are? ASPA doesn't solve that type of question.

Publication of ASPA records & verification of BGP UPDATES against the
published ASPA records will impose additional constraints on the global
routing table "so and so ASN should only appear behind AS X". This is
helpful, and I'm sure it'll knock down some fake paths generated by BGP
optimizers. :-)

> Spoofing will become much harder once a critical mass of infrastructure
> deployed ASPA.

I'd phrase it as "fat fingering will become even harder". :-)

Route Origin Validation based on RPKI ROAs reduced the number of BGP
routing incidents; but cynical critics could argue "silly you, you
published the exact list of Origin ASNs we need to spoof to bypass
ROV!". Similarly, publication of ASPA records tells the world what
exactly the fabricated AS_PATH should look like to bypass ASPA validation.
This is OK, it just means that ROV + ASPA is not a complete solution.

I think in-band signatures (BGPsec) are also needed to complete the
puzzle.

Kind regards,

Job

Re: Yet another BGP hijacking towards AS16509

[Job Snijders via NANOG]

Hi Douglas, group,

On Tue, Aug 23, 2022 at 03:03:31PM -0300, Douglas Fischer wrote:
> I was thinking a little about this case...
> 
> I'm almost certain that this case cited by Siyuan would have been
> avoided if there was a cross-check between the items contained in the
> AS-SET objects (and others such as the Route-Set), and the "member-of"
> attributes of the referred objects.

You are right that stronger 'arcs' ("connections") between the various
IRR objects at first glance potentially offer a solution; unfortunately
the objects exist in separate databases ("namespaces"), one has to be
cautious for object name collisions!

Cross-references (through "member-of:" <> "members:" links) for RPSL
objects only work within a single IRR source, in other words: if objects
exist in the same database. An object in one database can't reference
(through 'member-of:') an object in a different database.

> I participated in a small exchange of ideas about this, and there were
> questions about whether this crosscheck should be done by the consumer
> of the IRR data, or if it should be validated at the time of insertion
> in the base through NRTM.

As far as I understand the data model, only the ultimate consumer of IRR
data would be in a position to enforce some kind of policy (such as
requiring cross-references both ways 'members:' <> 'member-of:'); the
middle layer (software packages like IRRD) lack context.

I know of examples where fairly robust RTBH filters were generated using
members:/member-of pairing as a prerequisite; but I'm not aware of a
"cross-RIR" solution.

Kind regards,

Job

Re: [EXTERNAL] Re: Yet another BGP hijacking towards AS16509

[Job Snijders via NANOG]

On Tue, Aug 23, 2022 at 05:18:42PM +, Compton, Rich A wrote:
> I was under the impression that ASPA could prevent route leaks as well
> as path spoofing.  This "BGP Route Security Cycling to the Future!"
> presentation from NANOG seems to indicate this is the case:
> https://youtu.be/0Fi2ghCnXi0?t=1093 

I'm not sure how ASPA can prevent AS Path spoofing. Perhaps something
got lost in translation?

ASPA records are published in the RPKI, from there a RPKI RP transforms
the ASN.1/X.509/crypto stuff into something 'plain text'. This 'plain
text' data is loaded into EBGP routers via RTR, which then compare the
*plain text* AS_PATH attribute to the table of plain-text ASPA records,
to determine if it came via an authorized upstream provider or not.

In this sense, ASPA (just by itself) suffers the same challenge as RPKI
ROA-based Origin Validation: the input (the BGP AS_PATH) is unsigned and
unsecured; thus spoofable.

> Also, can't the path spoofing protection that BGPsec provides be
> defeated by an attacker advertising a hijacked prefix with a forged
> AS_PATH without BGPsec?

An attacker certainly can try to announce hijacked prefixes with a
forged AS_PATH (knowing that BGPsec-protected paths also exist). But the
attacker has no influence on the local routing policy of other networks.
I imagine that in a 'partial BGPsec' deployment future some operators
might opt to assign a higher LOCAL_PREF to BGPSec secured paths; or
apply a form of 'peerlock' ("I only accept routes with so-and-so ASN if
seen via BGPsec secured paths); this is an area of future study.

> In order to get around this vulnerability, all of the Internet would
> have to only perform BGPsec which doesn't seem realistic.

The vulnerability is "if the operator configures their routers to use
non-signed paths, the operator choose to use a non-signed path" (I hope
this makes sense).

> Security solutions where everyone has to implement the control before
> it works effectively are rarely adopted.

Fully agreed. Fortunately, I see a number of scenarios in which a partial
deployment of BGPsec benefits those who deployed it. Ultimately BGPsec
deployment is a 'selfish' act, as in that the benefits are immediate to
the two adjacent networks using BGPsec to protect the paths between
then.

Perhaps I should prepare a NANOG presentation to outline why BGPsec does
not require 100% deployment to be beneficial?

Kind regards,

Job

Re: Yet another BGP hijacking towards AS16509

[Job Snijders via NANOG]

Dear Siyuan, others,

Thank you for the elaborate write-up and the log snippets. You
contributed a comprehensive overview of what transpired from a
publicly-visible perspective, what steps led up to the strike.

I want to jump in on one small point which I often see as a point of
confusion in our industry:

On Tue, Aug 23, 2022 at 01:54:50AM +0200, Siyuan Miao wrote:
> Nowadays hijacking a service by forging AS path is pretty easy and
> RPKI won't be able to solve this (as it validates origin AS and
> prefixes only) :-(

I do think RPKI can help solve this! The "RPKI" is a cryptographically
secured distributed database of authorizations for Internet Resource
Numbers (IP addresses & AS numbers). (((yikes, that's a mouthful :-)))

Another way of looking at the RPKI is as a "general purpose framework",
a framework on top of which the Internet community can build multiple
"applications". These applications include:

A) Route Origin Validation (aka "BGP Prefix Origin Validation", RFC 6811)
B) BGPSec (AS Path validation, RFC 8205)
C) ASPA (draft-ietf-sidrops-aspa-{profile,verification}, combating
 routeleaks by publishing what ASNs are your upstreams)
D) .. and others: https://datatracker.ietf.org/wg/sidrops/documents/

Nowadays Item A ("BGP Origin Validation") is widely deployed: all major
IP Transit carriers & major IX Route Server operators use RPKI ROAs to
filter out BGP announcements which have the wrong BGP Origin AS in the
AS_PATH. This is fantastic (and relatively recent) news!

Item B ("BGPsec") and C ("ASPA") are "work in progress": people are
building software, running experiments, studying what it would take to
get those technologies deployed in the wild (the 'production Internet'). 

BGPSec and ASPA are complementary solutions, each has its challenges and
opportunities. BGPsec prevents path spoofing, while ASPA can prevent
route leaks. These are similar but not identical threats that are often
conflated. ASPA and BGPsec should not be thought of as mutually
exclusive or incompatible; both of these technologies will support
routing security in the long term.

I recently co-authored an elaboration to the FCC on where the industry
stands and how some technologies relate to each other, this might be of
interest to some folks:
https://sobornost.net/~job/fastly-fcc-noi-secure-internet-routing-reply-comments-20220510-201259363-pdf.pdf

Kind regards,

Job

Re: 2 Byte ASNs??

[Job Snijders via NANOG]

On Fri, Aug 05, 2022 at 11:16:03AM -0400, Justin Wilson (Lists) wrote:
> Whats the availability of two byte asns look like? Anyone able to
> obtain one recently? 

Yes, at $work we obtained one recently (without hassle, thank you ARIN
hostmasters!).

So, I recommend to follow normal procedure and just ask ARIN. :-)

Kind regards,

Job

Re: irrd or ...?

[Job Snijders via NANOG]

Hi Randy,

On Sun, 19 Jun 2022 at 23:07, Randy Bush  wrote:

> >> It will also take much less RAM if you turn RPKI validation off.
> >
> > oh dear ghod.  do i need to turn the dancing donkeys off too?
> >
> > "Make each program do one thing well. To do a new job, build afresh
> > rather than complicate old programs by adding new "features"."
> > -- ken thompson - unix philosophy
> >
> > a good side to a bit of economic contraction might be a side effect of
> > code bloat and featuritis contraction.
>
> to be clearer, i now run a 4GB VM with irrd2, rancid, nfsen, and a wiki.
> so i will stick with irrd2.



Are you looking for to set up just an “authoritative IRR source” (RGNET?),
or to set up an instance which mirrors all the world’s IRRs? The latter
option is quite memory heavy.

If mirroring other databases is not the goal; an “auth only” IRRd v4
deployment will easily fit your VM alongside those other apps.

Or perhaps you are interested to fund development of a modern lightweight
version of the IRRd software? :-)

Kind regards,

Job

>

Re: Bgpmon alternative

[Job Snijders via NANOG]

Hi,

I recommend taking a look at
https://github.com/nttgin/BGPalerter

https://www.lacnic.net/innovaportal/file/4489/1/bgpalerter_lacnic33.pdf

It offers a great blend of BGP and RPKI ROA monitoring

Kind regards,

Job

On Wed, 15 Jun 2022 at 16:45, Mehmet Akcin  wrote:

> Hi there
>
> What are the best alternatives to BGPmon these days?
>
> Goal is to try to monitor bgp routing changes for specific prefixes.
>
> Mehmet
> --
> Mehmet
> +1-424-298-1903
>

Re: Newbie x Cisco IOS-XR x ROV: BCP to not harassing peer(s) and upstream(s)

[Job Snijders via NANOG]

On Wed, May 11, 2022 at 01:22:32PM -0600, Grant Taylor via NANOG wrote:
> On 5/11/22 10:53 AM, Job Snijders via NANOG wrote:
> > This knob slightly increase your own memory consumption, but makes your
> > router more “neighbourly”! :-)
> 
> I question how accurate "slightly" is.
> 
> My understanding is that soft reconfiguration inbound (whatever the syntax
> for a given IOS is) causes a full copy of the received prefix list to be
> retained in memory for each of the peers with soft reconfiguration enabled.
> 
> So, to me, the amount of impact to memory will be based on both the number
> of prefixes advertised and the number of peers that soft reconfiguration is
> enabled on.
> 
> Please enlighten me if I'm wrong / misunderstanding something.

How much memory exactly is consumed, will depend on the architecture of
the application (whether duplicity of information such as path
attributes is avoided as much as possible). Indeed, YMMV.

>From experience at a previous employer I recall that
'soft-reconfiguration inbound' on routers (with multiple full routing
tables) was problematic on 32-bit versions of the operating system; but
not an issue on 64-bit.

If unsure, test on a few peers and monitor memory usage! Its also a
valid question to the Technical Assistance Center "hey, will enabling
this soft-reconfiguration feature land me in hot water?"

Kind regards,

Job

Re: Newbie x Cisco IOS-XR x ROV: BCP to not harassing peer(s) and upstream(s)

[Job Snijders via NANOG]

Hi!

In current versions I think enabling “soft-reconfiguration-inbound always”
(also described at
https://bgpfilterguide.nlnog.net/guides/reject_invalids/#cisco-ios-xr )
should be enough.

Make sure to enable it on every EBGP peer you apply ROV to, or just all
EBGP peers.

This knob slightly increase your own memory consumption, but makes your
router more “neighbourly”! :-)

Kind regards,

Job

On Wed, 11 May 2022 at 18:44, Pirawat WATANAPONGSE via NANOG <
nanog@nanog.org> wrote:

> Dear Guru(s),
>
>
> We used to run our ‘Gateway Router’ with ROV turned on.
> Then, we “upgraded” it to a Cisco NCS-55A1 (5500 Series) running IOS-XR
> just a few weeks ago.
>
> Consequently, during my rummage through Google for a (the?) best (ROV)
> configuration template for the new router,
> I found a tutorial by Philip Smith
> [Reference: https://www.bgp4all.com/pfs/_media/workshops/02-rpki.pdf,
> Slide #55]
> which cautioned me of Cisco IOS-XR essentially “harassing” all peers and
> upstreams with ‘Route Refresh’ whenever there is a VRP change.
> The tutorial advised turning on ‘Soft Reconfiguration’ to help with the
> problem.
>
> On the one hand, we have a very special relationship with our upstream
> [they’re kind of community transit provider; we have an in-kind stake in
> them as well], so we obviously don’t want to cause them grievances [their
> grievance is our grievance].
> On the other hand, we can't afford to just throw away a newly bought
> gateway and buy a new one.
>
> So, here goes the question:
> Is setting 'Soft Reconfiguration' enough for me to keep ROV running?
> If not, is there any other solution?
> Or am I screwed anyway?
>
> I would very much appreciate clarification and pointer(s) to the
> solution(s).
>
>
> Thank you in advance for the help,
>
> Pirawat.
>
>

Re: Geolocation data management practices?

[Job Snijders via NANOG]

Hi Shawn,

On Wed, Apr 20, 2022 at 01:14:29PM -1000, Shawn wrote:
> What is the best practice (or peoples preferred methods) to
> update/correct/maintain geolocation data?
> Do most people start with description field info in route/route6 objects?
>
> [snip]
> 
> Maybe I am not using the magic word?

The magic word is "geofeed"! :-)

The format is specified in RFC 8805. Finding and using Geofeed data is
described in RFC 9092.

Example in the wild:

$ whois -h whois.ripe.net 146.75.0.0 | fgrep geofeed:
geofeed:https://ip-geolocation.fastly.com/

Another example below, in this instance the geofeed information is
stored in a 'remarks:' attribute. Unfortunately this particular RIR does
not (yet?) properly support the native RPSL geofeed attribute for IPv6
/48 PI blocks.

$ whois -h whois.ripe.net 2001:67c:208c::/48 | grep geofeed
remarks:Geofeed https://sobornost.net/geofeed.csv

Both approaches work.

Kind regards,

Job

Re: Something observed while doing IRR cleanup (generic name collisions)

[Job Snijders via NANOG]

Hi Dan!

You highlight a common pitfall in IRR-based prefix filter generation.

On Mon, Apr 11, 2022 at 09:56:59AM -0700, Dan Mahoney (Gushi) wrote:
> [snip]
> as-set: AS-PEERS
> descr:  Peer AS Numbers
> members:AS132251,AS132561,AS132516
> source: APNIC
> 
> as-set: AS-PEERS
> descr:  swell.network Peers
> members:AS-HE,AS-NTT
> source: ARIN
> 
> ..four separate organizations felt it would be clever to create a
> vaguely-named AS-PEERS object, each in a different IRR, because the various
> IRR's all allow this, and don't check for the existence of objects in
> another.  No IRR's require any special names, nor do they block on any
> generic names. No IRR sends a member warnings when their objects exist in
> more than one registry with different data.

The RPSL RFCs permit a syntax which helps increase the potential for
'uniqueness' of object names across multiple databases: the trick is to
prepend the object with one's ASN. An example: "AS15562:AS-SNIJDERS"

This approach is also known as "hierarchical AS-SET naming". IRRd v4.2
instances require this naming approach for newly created objects.
Because of this feature the LACNIC IRR data source contains only
hierarchically named AS-SETs! Progress might seem slow, but all journeys
start with a few small steps. :-)

Hierarchical naming does not prevent the existence of duplicate names
across IRR databases, but it sure does help!

> I haven't tried to query the peeringdb API to see if any of these are
> used as advertised AS-Sets for public use, or if people just created
> public objects for their own internal tools.  I'm sure this is not the
> only case of this.
> 
> This might be why we can't have nice things.

Patience is the path towards nice things :-)

Kind regards,

Job

Re: 2749 routes AT RISK - Re: TIMELY/IMPORTANT - Approximately 40 hours until potentially significant routing changes (re: Retirement of ARIN Non-Authenticated IRR scheduled for 4 April 2022)

[Job Snijders via NANOG]

On Mon, Apr 04, 2022 at 06:35:31PM -0400, Jon Lewis wrote:
> On Tue, 5 Apr 2022, Job Snijders wrote:
> > > Are others jumping ship or planning to from ALTDB (no offense intended, 
> > > and
> > > grateful for the service you've provided) and other non-auth IRRs like 
> > > RADB
> > > due to networks like Tata announcing that they won't honor route objects
> > > created in non-authoratative IRR DBs after late last year and plan to 
> > > ignore
> > > them entirely by late next year?  i.e.
> > > 
> > > From: https://lg.as6453.net/doc/cust-routing-policy.html
> > > 
> > >   Special note, deprecation of non-authoritative registries
> > > 
> > >   Please note that 'route' and 'route6' objects created after 2021-Aug-15
> > >   in non-authoritative registries like RADB, NTTCOM, ALTDB and others
> > >   will not work. Objects created before that date will continue to work 
> > > till
> > >   2023-Aug-15. It is recommended to create RPKI ROA objects instead. In
> > >   rare cases if that's not possible, 'route' and 'route6' must be created
> > >   in the authoritative registry - AfriNIC, APNIC, ARIN, LACNIC, RIPE, 
> > > RIPE,
> > >   NIC.br or IDNIC.
> > 
> > I very much appreciate Tata's efforts to strive to only use authoritive
> > data when making BGP routing decisions; however the scope of their
> > charter is of course confined to just Tata's own operations. Tata's
> > routing policies affect only Tata's customer cone.
> 
> I'm (well, work is) a Tata customer.  So their policy wrt which IRR's
> they'll honor objects in matters to me, and going forward, it makes no sense
> for us to create new objects in ALTDB or RADB...and those proxy
> registrations Kenneth created in ALTDB, if any of those networks are
> originated by Tata customers, I presume the new ALTDB objects won't cause
> Tata prefix-list filters to include those routes.

Right.

> I just wonder if Tata is alone leading the charge to deprecate non-auth
> IRRs, or if there are other notable networks with similar policies?

I think there clearly is an industry-wide trend to move away from
'unsigned plain-text non-authoritative' datasets, towards better sources
of truth such as the VRP data available through the RIR RPKI Trust
Anchors.

There are variances in how stakeholders implement this paradigm shift:
some operators move towards wholesale ignorance of non-auth databases
(like Tata); some operators use softer transition mechanisms (examples:
what RIPE NCC did in lieu of RIPE-731, or how IRRd v4 in its default
configuration magically makes RPKI-invalid IRR objects disappear).

I think all of us recognize a need to declaw "third party" IRR databases
like RADB and ALTDB ("declawing" meaning that it is not desirable that
anyone can just register *anything*); on the other hand our community
also has to be cognizant about there being parts of the Internet which
are not squatting on anyone's numbers *and* also are not contracted to a
specific RIR.

Kind regards,

Job

Re: 2749 routes AT RISK - Re: TIMELY/IMPORTANT - Approximately 40 hours until potentially significant routing changes (re: Retirement of ARIN Non-Authenticated IRR scheduled for 4 April 2022)

[Job Snijders via NANOG]

Dear Jon, others,

On Mon, Apr 04, 2022 at 05:48:42PM -0400, Jon Lewis wrote:
> On Mon, 4 Apr 2022, Kenneth Finnegan wrote:
> > While I agree that it might be politically entertaining to let this
> > one blow up as a demonstration of how ARIN conducts business, this
> > list of networks includes too many small networks who likely don't
> > have a savy networking engineering team.
> > 
> > In my opinion, they are not acceptable collateral damage to
> > demonstrate ARIN's lack of regard for the community in shutting this
> > down without a transition plan for the RPSL objects, so as one of the
> > admins for the ALTDB IRR database, I've taken it upon myself to create
> > proxy registrations for all of these prefixes in ALTDB.
> > 
> > Like any proxy registration, asset owners are welcome to contact the
> > maint POC, and if no response from them, db-ad...@altdb.net,
> > requesting that stale records be deleted, but please also note that
> > ALTDB automatically deletes any route objects which conflict with a
> > publishes RPKI ROA, so the most effective way to clean up stale IRR
> > records is to publish RPKI ROAs for your address space.
> 
> Does any other IRR do that?

An event not too dissimilar to the current situation at hand was when
the server systems on which the "SAVVIS" IRR database existed
experienced a catastrophic failure. At that time, Savvis had already
been acquired by another entity, but they were unable to
integrate/migrate dataset in time, and no backups appeared to be
available.

After some delibration and attempts to untangle the spaghetti of
potential consequences for the BGP customer cone of my then-employer, I
took the liberty to copy (proxy register) a significant number of
route+route6 objects into NTTCOM (similar to what Kenneth did) because
the alternative seemed worse. Deprecation of IRR databases is something
very few people in this industry really have hands-on experience with.

> What does ALTDB do if a route object exists (or multiple route objects exist
> for the same route with different origins) and multiple ROAs exist allowing
> the route to be originated by multiple ASNs?  Technically, some of those
> ROAs would conflict with some route objects.

I can't speak with any authority on ALTDB operational matters, but I
think they use a tool based on https://github.com/job/irr-nonauth-cleanup/

The 'irr-nonauth-cleanup' utility uses an algorithm similar to what is
described in RFC 6811 in context of classifying BGP routes, and also is
similar to what RIPE NCC uses to periodically clean up the
"RIPE-NONAUTH" IRR database. RIPE NCC's cleanup effort was codified
through community consensus and published as 
https://www.ripe.net/publications/docs/ripe-731

In the same spirit as RFC 6811 stipulates - RPKI ROAs never are
considered "in conflict" with each other. As long as at least one ROA
permits the specific (Prefix, Origin AS) tuple at hand to exist, the IRR
object may exist. It's fine for multiple ROAs to exist which cover the
same address space, this is what makes migrations/reconfigurations
possible.

> Are others jumping ship or planning to from ALTDB (no offense intended, and
> grateful for the service you've provided) and other non-auth IRRs like RADB
> due to networks like Tata announcing that they won't honor route objects
> created in non-authoratative IRR DBs after late last year and plan to ignore
> them entirely by late next year?  i.e.
> 
> From: https://lg.as6453.net/doc/cust-routing-policy.html
> 
>   Special note, deprecation of non-authoritative registries
> 
>   Please note that 'route' and 'route6' objects created after 2021-Aug-15
>   in non-authoritative registries like RADB, NTTCOM, ALTDB and others
>   will not work. Objects created before that date will continue to work till
>   2023-Aug-15. It is recommended to create RPKI ROA objects instead. In
>   rare cases if that's not possible, 'route' and 'route6' must be created
>   in the authoritative registry - AfriNIC, APNIC, ARIN, LACNIC, RIPE, RIPE,
>   NIC.br or IDNIC.

I very much appreciate Tata's efforts to strive to only use authoritive
data when making BGP routing decisions; however the scope of their
charter is of course confined to just Tata's own operations. Tata's
routing policies affect only Tata's customer cone.

Depending on the exact characteristics of one's customer cone it may be
feasible to only use RIR-provided IRR data sources, but it's not hard to
imagine that for some providers this is a bridge too far at this point
in time.

Kind regards,

Job

2749 routes AT RISK - Re: TIMELY/IMPORTANT - Approximately 40 hours until potentially significant routing changes (re: Retirement of ARIN Non-Authenticated IRR scheduled for 4 April 2022)

[Job Snijders via NANOG]

Dear all,

On Sat, Apr 02, 2022 at 09:09:58PM +, John Curran wrote:
> As previously reported here, ARIN will be shutting down the
> ARIN-NONAUTH IRR database on Monday, 4 April 2022 at 12:00 PM ET.
> 
> It is quite likely that some network operators will see different
> route processing as a result of this change, as validation against the
> ARIN-NONAUTH IRR database will not longer be possible.
> 
> Please be aware of this upcoming event and make alternative
> arrangements if you are presently relying on upon routing objects in
> the ARIN-NONAUTH IRR database.

I ran an analysis just now in which I created an intersection between
prefixes observed in the BGP default-free zone and exactly matching
route:/route6: objects in ARIN-NONAUTH. I then substracted exact
matching objects found in the RADB, ALTDB, TC, NTTCOM, LEVEL3, RIPE, and
APNIC IRR sources. The result is a list of routes which might
experience service disruptions due to missing IRR objects.

The below 2,749 Prefix + Origin AS pairings are at risk as a result of
ARIN shutting down the ARIN-NONAUTH IRR database. Any potential effects
are likely to manifest themselves in the coming 24 - 32 hours. Prior to
this announcement, ARIN consulted with its community on the future of
its IRR service.

I voiced objection and raised concerns about (what appeared to be)
limited visibility into what exactly the effects of such a database
shutdown would mean for the Internet: 
https://lists.arin.net/pipermail/arin-consult/2021-March/001237.html
Other community members also shared concerns: 
https://lists.arin.net/pipermail/arin-consult/2021-February/001195.html
A number of graceful alternative mechanisms were proposed, but not
acted upon: https://lists.arin.net/pipermail/arin-consult/2021-March/001240.html

One might argue "well, folks had more than a year to move their
objects!", but on the other hand, it is entirely possible not all the
right people were reached, or in cases where affected parties did
receive a communication from ARIN, they perhaps were unable to
understand the message.

Please check if any of your prefixes are on the below list, and if so,
double check whether your IRR administration is able to overcome the
disappearance of ARIN-NONAUTH. Godspeed!

This tool might be useful: https://irrexplorer.nlnog.net/

Kind regards,

Job

Prefix OriginAS
100.42.100.0/24 33353
100.42.101.0/24 33353
100.42.102.0/24 33353
100.42.104.0/24 33353
100.42.105.0/24 33353
100.42.106.0/23 33353
100.42.108.0/24 33353
100.42.109.0/24 33353
100.42.96.0/23 33353
100.42.98.0/24 33353
103.11.202.0/24 33517
103.13.12.0/24 38057
103.13.135.0/24 51830
103.15.168.0/23 55532
103.196.22.0/23 7489
103.219.78.0/24 55256
103.219.79.0/24 55256
103.232.224.0/24 125
103.250.176.0/24 134795
103.250.177.0/24 134795
103.250.178.0/24 134795
103.250.179.0/24 134795
103.35.217.0/24 125
103.47.244.0/24 55256
103.88.172.0/24 136271
103.88.173.0/24 136271
103.88.174.0/24 136271
104.128.96.0/20 19233
104.142.128.0/23 33353
104.142.130.0/23 33353
104.142.136.0/22 33353
104.142.140.0/23 33353
104.142.144.0/24 33353
104.142.145.0/24 33353
104.142.146.0/24 33353
104.142.147.0/24 33353
104.142.148.0/24 33353
104.142.149.0/24 33353
104.142.152.0/24 33353
104.142.153.0/24 33353
104.142.156.0/24 33353
104.142.160.0/24 33353
104.142.164.0/24 33353
104.142.165.0/24 33353
104.142.175.0/24 33353
104.142.176.0/24 33353
104.142.177.0/24 33353
104.142.180.0/24 33353
104.142.181.0/24 33353
104.142.184.0/24 33353
104.142.185.0/24 33353
104.142.186.0/24 33353
104.142.187.0/24 33353
104.142.188.0/24 33353
104.142.189.0/24 33353
104.142.190.0/24 33353
104.142.191.0/24 33353
104.142.192.0/24 33353
104.142.224.0/24 33353
104.142.248.0/21 33353
104.142.249.0/24 33353
104.142.251.0/24 33353
104.142.252.0/24 33353
104.142.253.0/24 33353
104.142.254.0/24 33353
104.142.255.0/24 33353
104.143.64.0/20 13739
104.152.174.0/24 393586
104.152.175.0/24 393586
104.152.40.0/22 10302
104.153.160.0/22 36203
104.153.180.0/23 26021
104.153.182.0/23 26414
104.156.200.0/22 21743
104.167.96.0/19 394991
104.192.254.0/24 14244
104.193.40.0/22 396317
104.219.166.0/23 54804
104.225.100.0/24 36236
104.225.102.0/24 36236
104.225.103.0/24 36236
104.225.107.0/24 36236
104.225.11.0/24 36236
104.225.2.0/24 36236
104.225.21.0/24 36236
104.225.22.0/24 36236
104.225.99.0/24 36236
104.243.160.0/20 63353
104.244.144.0/21 14869
104.244.208.0/24 13737
104.244.210.0/24 13737
104.255.40.0/21 23465
104.37.242.0/24 7311
107.182.192.0/21 22534
107.182.200.0/21 22534
107.191.224.0/21 23317
107.191.232.0/22 23317
108.161.224.0/20 14244
108.169.208.0/23 62943
119.252.189.0/24 56106
12.10.107.0/24 40230
12.109.82.0/23 20034
12.111.223.0/24 23094
12.144.72.0/22 33247
12.148.232.0/22 33247
12.149.246.0/23 33247
12.151.71.0/24 33358
12.175.119.0/24 23094
12.182.253.0/24 40110
12.188.18.0/24 36136
12.191.163.0/24 27632
12.199.50.0/24 40230
12.203.197.0/24 22034
12.205.14.0/24 40230
12.229.190.0/23 1641
12.229.60.8/29 46826
12.27.186.0/23 33247
12.27.188.0/22 332

RFC 9225 - Software Defects Considered Harmful

[Job Snijders via NANOG]

Hi all,

It's super official now: no more software bugs in networking gear.
Sorry it took so long to document what the best current practise is!

Kind regards,

Job / Chris / Remco

- Forwarded message from rfc-edi...@rfc-editor.org -
Date: Fri,  1 Apr 2022 10:17:37 -0700 (PDT)
From: rfc-edi...@rfc-editor.org
To: ietf-annou...@ietf.org, rfc-d...@rfc-editor.org
Cc: drafts-update-...@iana.org, rfc-edi...@rfc-editor.org
Subject: RFC 9225 on Software Defects Considered Harmful

A new Request for Comments is now available in online RFC libraries.

RFC 9225

Title:  Software Defects Considered Harmful 
Author: J. Snijders,
C. Morrow,
R. van Mook
Status: Informational
Stream: Independent
Date:   1 April 2022
Mailbox:j...@fastly.com,
morr...@ops-netman.net,
re...@asteroidhq.com
Pages:  6
Updates/Obsoletes/SeeAlso:   None

I-D Tag:draft-dont-write-bugs-00.txt

URL:https://www.rfc-editor.org/info/rfc9225
DOI:10.17487/RFC9225

This document discourages the practice of introducing software
defects in general and in network protocol implementations
specifically.  Software defects are one of the largest cost drivers
for the networking industry.  This document is intended to clarify
the best current practice in this regard.


INFORMATIONAL: This memo provides information for the Internet community.
It does not specify an Internet standard of any kind. Distribution of
this memo is unlimited.

This announcement is sent to the IETF-Announce and rfc-dist lists.
To subscribe or unsubscribe, see
  https://www.ietf.org/mailman/listinfo/ietf-announce
  https://mailman.rfc-editor.org/mailman/listinfo/rfc-dist

For searching the RFC series, see https://www.rfc-editor.org/search
For downloading RFCs, see https://www.rfc-editor.org/retrieve/bulk

Requests for special distribution should be addressed to either the
author of the RFC in question, or to rfc-edi...@rfc-editor.org.  Unless
specifically noted otherwise on the RFC itself, all RFCs are for
unlimited distribution.

The RFC Editor Team
Association Management Solutions, LLC

___
IETF-Announce mailing list
ietf-annou...@ietf.org
https://www.ietf.org/mailman/listinfo/ietf-announce

- End forwarded message -

Re: A few questions regarding about RPKI/invalids

[Job Snijders via NANOG]

On Wed, Mar 30, 2022 at 01:29:25PM +, Drew Weaver wrote:
> Ex 45.176.191.0/24   3356 3549 11172 270150
> 
> RPKI ROA entry for 45.176.191.0/24-24
>   Origin-AS: 265621
> 
> Two questions:
> 
> First, are you also seeing this on this specific route?

It is visible in a few places, but the 61% score in for example RIPE
stat is very low, which is a strong hint some kind of issue exists:
https://stat.ripe.net/ui2013/45.176.191.0%2F24#tabId=routing

> Second, is there a certain number of "expected" invalid routes? (not
> including unknowns)

Through large transit providers that do RPKI ROV with 'invalid ==
reject' you'll generally see less than a 100 invalids at any given time
(1299, 174, 3257, 3303, 6830, etc).

Then there are large transit providers who (as far as the public record
is concerned) have not yet deployed RPKI ROV on their EBGP edges. Via AS
6762 I see ~ 2,300 invalids, and via AS 6461 about 3,000 invalids.

For historical perspective: this 3,000 upperbound number used to be ~
6,000 back in the 'pre RPKI era' in 2018/2019.

> Third, how are you handling specifically the large number of routes
> from 3356 3549 which invalid origin AS? Are you just "letting the
> bodies hit the floor"? or are you carving those out somehow?

I'd reject them. Why carve out an exception merely because the
number is 'large'? :-)

Kind regards,

Job

Re: Routes to twitter via 8359 8359 8342

[Job Snijders via NANOG]

On Mon, Mar 28, 2022 at 12:33:05PM +, Drew Weaver wrote:
> Is anyone else seeing this route destined for Twitter in the US being
> directed through 8359 announced by 8342?
> 
> 104.244.42.0/24
> 
> Just curious, replies off list welcome.

Seems visible in a handful of places:

$ w3m -dump 
'https://lg.ring.nlnog.net/prefix_detail/lg01/ipv4?q=104.244.42.0/24' | fgrep 
8342
BGP.as_path: 198644 21283 8447 8359 8342
BGP.as_path: 8607 8359 8342
BGP.as_path: 16086 8359 8342
BGP.as_path: 31027 8359 8342

There is a RPKI ROA which only authorizes AS 13414 to originate that
/24, chances are that RPKI ROV is helping limit the blast radius.

https://console.rpki-client.org/rpki.arin.net/repository/arin-rpki-ta/5e4a23ea-e80a-403e-b08c-2171da2157d3/521eb33f-9672-4cd9-acce-137227e971ac/69f7b69a-b519-4b2d-a7be-03aa505b2576/672a3d03-abc2-3f66-8052-26ed409619a3.roa.html

Kind regards,

Job

Re: Can it really be this quiet?

[Job Snijders via NANOG]

Hi Allen,

Yes, it can be this quiet. It’s good news, it means the thing is mostly
working :-)

I wish everyone a happy and calm 2022!

Kind regards,

Job

On Mon, 3 Jan 2022 at 20:47, Allen McKinley Kitchen (gmail) <
allenmckinleykitc...@gmail.com> wrote:

> Or has NANOG also succumbed to a signed-integer date problem?
>
> Waiting to see what I get back..
>
> ..Allen
>

Re: Incrementally deployable secure Internet routing: operator survey

[Job Snijders via NANOG]

Hi all,

On Fri, 17 Dec 2021 at 19:50, Adrian Perrig  wrote:

> other proposed approaches such as RPKI that only protects a route’s origin
> first AS, or BGPsec that requires widespread adoption and significant
> infrastructure upgrades.
>


For both RPKI-based BGP Route Origin Validation and RPKI-based BGPsec -
that meme “widespread adoption is a prerequisite to benefit” is somewhat
annoying in getting widespread adoption going. Plz Stop It! :-)

In my opinion, global scale BGP routing security does *not* depend on
concepts like “herd immunity”. Rather, I would frame “BGP routing security”
as a problem requiring selfish acts, not collective action. The benefits
become immediately available to you and your EBGP peer (who agreed to
participate in the effort). Commercial incentives align with upgrading
(both transport capacity and security) one peer at a time.

All of RPKI ROV, BGPsec, ASPA/peerlock, and even older plain-text stuff
like “IRR” are incrementally deployable technologies; because how else
would one ever get anything deployed in fast-and-wide growing
multiple-operator networks such as the Internet? Nothing happens at the
same time. But when it happens, it progresses at the pace of decades, at
times so slow one might think the paint isn’t drying on the wall.

BGP sessions “worth protecting” usually are the revenue generating/cost
reduction sessions, and as such usually are assigned highest LOCAL_PREF. I
think this property has interesting implications on how routing security
features become available and are demanded from others throughout the
ecosystem. For most networks at the edge, the private peering sessions also
are the BGP sessions with the least BGP state on either side, compared to
say upstream.

The “significant upgrades” aspect is just part of the job and happen no
matter what. Every network replaces all their kit at some point in time;
but sometimes it takes as long as ten to fifteen years! The good news is
that every replacement also comes with improved cryptographic op
accelerators in the CPU and more memory; and it all seems to be converging
towards commonly available general purpose computing systems on which
people can run any BGP stack they want.

I’m bullish on BGP routing security tech already specified and published
through the IETF process :-)

Kind regards,

Job

>

Re: Theorical question about cyclic dependency in IRR filtering

[Job Snijders via NANOG]

Hi Anurag,

Circular dependencies definitely are a thing to keep in mind when designing
IRR and RPKI pipelines!

In the case of IRR: It is quite rare to query the RIR IRR services
directly. Instead, the common practise is that utilities such as bgpq3,
peval, and bgpq4 query “IRRd” (https://IRRd.net) instances at for example
whois.radb.net and rr.ntt.net. You can verify this with tcpdump. These IRRd
instances serve as intermediate caches, and will continue to serve old
cached data in case the origin is down. This phenomenon in the global IRR
deployment avoids a lot of potential for circular dependencies.

Also, some organisations use threshold checks before deploying new
IRR-based filters to reduce risk of “misfiring”.


The RPKI case is slightly different: the timers are far more aggressive
compared to IRR, and until “Publish in Parent” (RFC 8181) becomes common
place, there are more publication points, thus more potential for operators
to paint themselves into a corner.

Certainly, in the case of RPKI, all Publication Point (PP) operators need
to take special care to not host CAs which have the PP’s INRs listed as
subordinate resources inside the PP.

See RFC 7115 Section 5 for more information: “Operators should be aware
that there is a trade-off in placement of an RPKI repository in address
space for which the repository’s content is authoritative. On one hand, an
operator will wish to maximize control over the repository. On the other
hand, if there are reachability problems to the address space, changes in
the repository to correct them may not be easily access by others”

Ryan Sleevi once told me: "yes, it strikes me that you should prevent
self-compromise from being able to perpetually own yourself, by limiting an
attacker’s ability to persist beyond remediation."

A possible duct tape approach is outlined at
https://bgpfilterguide.nlnog.net/guides/slurm_ta/
However, I can’t really recommend the SLURM file approach. Instead, RPKI
repository operators are probably best off hosting their repository
*outside* their own address space.

Just like with Authoritative DNS servers, make sure you also can serve your
records via a competitor! :-)

For example, if ARIN moved one of their three publication point clusters
into address space managed by any of the other four RIRs, some risk would
be reduced.

Kind regards,

Job

On Mon, 29 Nov 2021 at 13:37, Anurag Bhatia  wrote:

> Hello everyone,
>
> While discussing IRR on some groups recently, I was thinking if there can
> be (and if there is) cycling dependency in filtering where IRR (run by
> whoever APNIC, RIPE, RADB etc) uses some upstream and accepts only routes
> with existing & valid route object.
>
>
>
> So hypothetical case (can apply to any IRR):
>
>
>1. APNIC registry source is whois.apnic.net and points
>to 202.12.28.136 / 2001:dc0:1:0:4777::136. The aggregate of both these has
>a valid route object at the APNIC registry itself.
>
>2. Their upstreams say AS X, Y and Z have tooling in place to
>generate and push filters by checking all popular IRRs. All is well till
>this point.
>
>3. Say APNIC has some server/service issue for a few mins and X Y and
>Z are updating their filters at the same time. They cannot contact
>whois.apnic.net and hence miss generating filters for all APNIC IRR
>hosted prefixes.
>
>4. X, Y and  Z drop APNIC prefixes including those of IRR & the loop
>goes on from this point onwards.
>
>
> So my question is: Can that actually happen?
> If not, do X, Y and Z and possible all upstreams till default-free zone
> treat these prefixes in a special manner to avoid such loop in resolution?
>
>
>
>
> Thanks!
>
> --
> Anurag Bhatia
> anuragbhatia.com
>

Re: What are best practices for RPKI ROV in transit networks....

[Job Snijders via NANOG]

On Fri, Oct 29, 2021 at 01:20:33AM +0400, Musa Stephen Honlue wrote:
> Personally I recommend dropping them invalids.

100%

> However, you could set local preferences as follows:
> - Valids routes get the highest local pref
> - unknown routes get a medium local pref 
> - Invalids routes get the lowest local pref
> 
> In this way, if you have competing routes, the one with the higher
> local pref gets preferred. By so doing, you are sure that an invalid
> route will never get preferred over an unknown one or a valid one.

There are two core issues with the above approach:

1/ invalid 'more-specific' routes, regardless of local pref, will 'win',
   so the approach is 'useless'.
2/ modifying BGP path attributes based on the validation state introduces
   needless churn and BGP UPDATE flooding.

Consider the following scenario: someone new to the RPKI ecosystem
decides to create RPKI ROAs. They expect nothing to happen, but under
the hood new BGP UPDATEs are flooded in all directions because the
LOCAL_PREF attributes needs to be updated. Same problem when associating
BGP communities to validation states.

To quote myself from https://bgpfilterguide.nlnog.net/guides/reject_invalids/

"""
It is considered harmful to manipulate BGP Path Attributes (for example
LOCAL_PREF or COMMUNITY) based on the RPKI Origin Validation state.
Making BGP Path Attributes dependent on RPKI Validation states
introduces needless brittleness in the global routing system as
explained here. Additionally, the use of RFC 8097 is STRONGLY ABSOLUTELY
NOT RECOMMENDED. RFC 8097 has caused issues for multi-vendor network
operators.
"""

Nick Hilliard shared similar sentiment:
https://mailarchive.ietf.org/arch/msg/sidrops/dwQi9lgYKRVctdlMAHhtgYkzhSM/

Kind regards,

Job

Re: FORT monitoring/visibility

[Job Snijders via NANOG]

On Tue, Oct 26, 2021 at 04:58:20PM -0700, Randy Bush wrote:
> i run a FORT RPKI relying party instance.  i am looking for some
> visibility into its operation.
> 
>   is it up: both ways, fetching and serving routers?
> 
>   from what CAs has it pulled, how recently and frequently with
>   what success?
> 
>   what routers is it serving with rpki-rtr 323?
> 
>   blah blah blah
> 
> my old DRL RP instances produce MRTG graphs etc of the CA
> fetching side, though nothing on the rpki-rtr side.

Have you taken a look at 'rtrmon' (RPKI-To-Router Monitor)

https://github.com/bgp/stayrtr#monitoring-rtr-and-json-endpoints

Kind regards,

Job

Re: question about enabling RPKI using Hosted mode

[Job Snijders via NANOG]

Dear Edvinas,

On Mon, Oct 25, 2021 at 11:49:09PM +0300, Edvinas Kairys wrote:
> We're thinking of enabling BGP ROA, because more and more ISPs are using
> strict RPKI mode.
> 
> Does enabling Hosted Mode (where it doesn't requires any additional
> configuration on client end) on RPKI could for some reason could cause a
> traffic loss ?
> 
> The only disasterious scenario i could think of, is if we would enable ROA
> with incorrect sub prefixes, maximum prefix length. Am i Right ?

I think you correctly identified most of the potential pitfalls. Another
pitfall might be when a typo in the Origin AS value slips into the RPKI ROA.

For example, I originate 2001:67c:208c::/48 in the DFZ from AS 15562.
Should I'd accidentally modify the covering ROA to only permit AS 15563,
the planet's connectivity towards 2001:67c:208c::/48 would become
spotty.

So... - BEFORE - creating RPKI ROAs, I recommend setting up a BGP/RPKI
monitoring tool. NTT's excellent BGPAlerter might be useful in this
context: https://github.com/nttgin/BGPalerter

Don't deploy things without monitoring! :-)

Kind regards,

Job

Re: IPv6 and CDN's

[Job Snijders via NANOG]

On Mon, Oct 25, 2021 at 04:20:28PM -0400, Jared Mauch wrote:
>   Some of the other CDNs do have IPv6 on the authorities and
> should work without issues.
> 
> eg:
> 
> dig -6 +trace www.akamai.com.

Yes of course :-)

dig -6 +trace www.fastly.com.

Kind regards,

Job

Re: IPv6 and CDN's

[Job Snijders via NANOG]

Hi everyone, goedenmiddag Marco!

On Fri, Oct 22, 2021 at 01:40:42PM +0200, Marco Davids via NANOG wrote:
> We currently live in times where is actually fun to go IPv6-only. In my
> case, as in: running a FreeBSD kernel compiled without the IPv4-stack.

Indeed, this is fun experimentation. Shaking the (source code) trees
through excercises like these is a valuable way to identify gaps.

> It turns out that there underlying CDN's with domain names such as
> ‘l-msedge.net’ and ‘trafficmanager.net’ (Microsoft) or 'fastly.net', that
> reside on authoritative name servers that *only* have an IPv4 address.

As some observant readers noticed (hint: https://ip6.nl/#!deb.debian.org),
Fastly is working hard with select customers and friends to support IPv6
for everyone.

> I guess my question is simple: Why?
>
> Are there good architectural reason for this? Or is it just something
> that is overlooked and forgotten about?

The universal deployment of IPv6 appears to be a multi-decennial
multigenerational project. Allow me to shed some light on various
aspects.

One of the challenges faced by those wishing to deploy IPv6 (compared to
IPv4) is how from a BGP Default-Free Zone perspective, IPv4 and IPv6 are
not alike at all! The Internet's IPv6 routing topology is vastly
different from the IPv4 topology.

The above phenomenon is perfectly understandable following from the fact
that IPv4 predates IPv6 - and IP networks grow as they grow. In a
perfect world the IPv6 network would grow perfectly congruent alongside
the global IPv4 network. In this perfect world indeed IPv6 can "just be
enabled", and used whenever available!

Unfortunately the reality of the situation is far more chaotic! For
example if you look at PeeringDB's 'netixlan' table, large discrepancies
between the number of absent IPv4 entries and absent IPv6 entries are
visible:

$ curl -s https://peeringdb.com/api/netixlan | jq '.' | fgrep -c 
'"ipaddr4": null'
1286

$ curl -s https://peeringdb.com/api/netixlan | jq '.' | fgrep -c 
'"ipaddr6": null'
8160

>From the above it's implied that the density of the 'IPv4 mesh' is much
higher than the density and diversity of the 'IPv6 mesh', simply because
more operators present more IPv4 traffic-exchange opportunities to other
operators - compared to IPv6. This has performance implications.

Another aspect that flabbergasts me anno 2021 is how there *still* are
BGP peering disputes between (more than two) major global internet service
providers in which IPv6 is 'held hostage' as part of slow commercial
negotiations. Surely end-to-end IPv6 connectivity should be a priority?

Anyway, back to your question: content delivery networks who leverage
all possible technical knobs and buttons to increase performance (such
as BGP traffic engineering) might be reluctant to offer IPv6 services
"as if they are the same as IPv4". More study is required.

Tl;DR - work in progress! :-)

Kind regards,

Job

ps. Have you tried running an IPv6-only RPKI validator? About 1.4% of
RPKI VRPs appears to be 'missing' in IPv6-only environments :-/

Re: Questions about IRR best practices

[Job Snijders via NANOG]

Dear Lee,

*ring ring* - "IRR/RPKI helpdesk how may I help you today?" :-)

On Fri, Oct 22, 2021 at 08:25:10AM -0500, Lee Fawkes wrote:
> I have a couple of questions about best practices for Internet Routing
> Registries. I'm able to find lots of documentation about *how* to do
> things, but not a lot of documentation about when I *should* do things. I
> work for a medium-sized ISP in the US, and we are currently using both RADb
> and the ARIN IRR. We peer all over the place, but my main concern is how
> Cogent and Hurricane Electric build prefix filters from our IRRs.
> 
> 1. Netflix is asking us to add the AS of a downstream customer of one of
> our customers to our customer AS-SET. We have a direct relationship with
> this organization's provider, but not with this organization itself. Is
> this appropriate?

Another way to satisfy this request is to ask the organization's
provider to create an AS-SET (preferably RIR-operatored IRR such as
ARIN, RIPE, etc), and then reference their AS-SET on your own AS-SET.
IRR AS-SETs permit both referencing AS Numbers and AS-SETs as 'members:'.

> 2. On the ARIN side, when ARIN-NONAUTH goes away next year, does that
> do away with our ability to do proxy route objects? Do we need to
> require all of our BGP customers to set up their own IRRs?

The industry trend (very noticable the last 3 years) is that the ability
to create proxy route object registrations is slowly fading away.

At at first glance proxy registrations seem better than 'no
registration', the downside is that anyone can create proxy
registrations for any prefix: proxies are not very safe!

The recommendation is that each and every IP resource holder creates IRR
and/or RPKI objects themselves, or delegates the authority to do so to
their service provider.

These days everyone wants to see firm cryptographic proof!

> 3. On the RADb side, if we're turning up a new customer that doesn't have
> an IRR, and another ISP already has a proxy registration for that customer,
> is it sufficient for us to add that customer's AS to our customer AS-SET?

Technically this is likely to work, but the downside is that you end up
with a hard dependency on another ISP's proxy registration. If for
whatever reason that registration lapses (failure to pay bills, M&A, who
knows) ... you might end up with a hard to troubleshoot situation where
it is not immediately clear "it was working yesterday, but not today?!".

The best course of action is to ensure that objects are either managed
by yourself, or by the customer, so the responsibilities and object
ownership are clear to everyone involved.

> I've been getting around the fact that RADb doesn't allow multiple
> proxy registrations by registering proxy route objects in
> ARIN-NONAUTH, but that won't be an option much longer, and I can't
> really experiment with our customers' route objects to see what works.

A great tool to gain some insight into various IRR/BGP/RPKI data sources
and what the registration status of various objecst might mean can be
found at this awesome tool: https://irrexplorer.nlnog.net/

Follow up questions welcome!

Kind regards,

Job

Re: Fastly Peering Contact?

[Job Snijders via NANOG]

Hi Bryan,

On Thu, 16 Sep 2021 at 19:53, Bryan Holloway  wrote:

> Hey all ... looking for a Fastly (54113) peering contact that might be
> able to get me in touch with the right folks to do stuff.


I’ll follow up with you off-list.

Kind regards,

Job

Re: Cogent x RPKI

[Job Snijders via NANOG]

Dear Rubens,

On Mon, Aug 09, 2021 at 08:41:48AM -0300, Rubens Kuhl wrote:
> From a Cogent support ticket:
>> Please see the attached LOA.
>> 
>> Regarding the RPKI ROA, for now, we don't create ROA for our prefixes
>> nor for prefixes that we assign to our customers and we don't plan to
>> do it. Unfortunately, this is not an option."
>> 
>> Someone that poses as a Tier-1 and doesn't even plan to sign their
>> announcements ? How much more depeering will make them reconsider ?

Please keep in mind that sound technical, administrative, or financial
reasons can exist that hamper one's ability to create RPKI ROAs.

For example, if the IP space is LEGACY, and not covered by a 'LRSA'
(ARIN) or 'Legacy Agreement' (RIPE), then RPKI certificate issuance
services simply are not available for that IP space. Without an
agreement with a RIR to arrange RPKI services, logically, one cannot
create ROAs.

Perhaps in your case, if RPKI is a requirement, you are better off
bringing your own (RPKI capable) IP space?

Kind regards,

Job

Re: Tier1 BGP filter generation data sources & frequency

[Job Snijders via NANOG]

On Mon, May 24, 2021 at 02:04:32PM -0400, Luca Salvatore wrote:
> Curious if anyone is aware of other Tier1s deprecating support for RADB?

Rather than deprecating RADB, I think the industry would be better off
if either RADB or the Tier1s (in their local caching layer) deploy IRR
database software capable of RPKI Origin Validation ala RIPE-731.

https://www.ripe.net/publications/docs/ripe-731

http://irrd.net/

Kind regards,

Job

Re: [nanog] TC x IRRd 4.2

[Job Snijders via NANOG]

Dear Ruben, all,

On Tue, Apr 27, 2021 at 10:18:32PM -0300, Rubens Kuhl wrote:
> TC IRR, an IRR operator focused on Brazilian networks, just changed to
> IRRd 4.2.  The new version allowed TC to deploy RPKI validation
> (thanks NTT for sponsoring that development) and expose HTTPS
> endpoints for WHOIS and submission that we hope will foster innovation
> around the database.
> 
> Every precaution was taken for this migration to be seamless for other IRR
> operators, including matching of serial numbers. Every IRR server that
> mirrored TC and supported -j status query was verified that it followed and
> still correctly follows database journals.
> 
> But if anything appears broken, please let me know or e-mail
> db-ad...@bgp.net.br.

Congratulations to you and the TC team for reaching this milestone!

TC's use of RPKI-based IRR Object filtering combined with the efforts of
NIC.BR, IX.br, and LACNIC to promote RPKI in Brazil, make the Brazilian
community a positive example of a seamless integration between IRR and
RPKI.

Thank you for your efforts to increase the data quality of the TC
registry.

Kind regards,

Job

Re: Cogent RPKI invalid filtering

[Job Snijders via NANOG]

Hi Robert, NANOG,

On Mon, Apr 26, 2021 at 09:29:27AM -0400, Robert Blayzor via NANOG wrote:
> According to Cloudflares isbgpsafeyet.com, Cogent has been considered "safe"
> and is filtering invalids.
> 
> But I have found that to be untrue (mostly). It appears that some days they
> filter IPv4, sometimes not, and IPv6 invalids are always coming through. I
> know it's Cogent, but curious as to what others are seeing.

   [ Disclaimer: I'm not affiliated with the companies referenced in the
 above message. But as I love talking about RPKI, I'd like to share
 some perspective based on my own experience with both small and
 large scale RPKI deployments. ]

TL;DR - RPKI Route Origin Validation (ROV) is incrementally deployed
inside networks, and incrementally across the Default-Free Zone. This
means right now (and for years to come), operators will see RPKI invalid
routes spill through the cracks of the global routing system.
This is expected and unavoidable.

Details ---

There are a few caveats to consider when using the isbgpsafeyet.com
testing utility to determine whether a network is doing RPKI ROV with
'invalid == reject' EBGP policies. The isbgpsafeyet.com beacon prefixes
are anycasted from many vantage points, this 'skews' the testing results
in some ways.  Imagine the prefixes being anycasted from (hypothetical)
a 100 POPs, this essentially is a 100 attempts to propagate RPKI invalid
routes into the default-free zone. Only a single route (out of the 100)
needs to slip past any potential 'invalid == reject' barriers between
the testsite and the visitor. The Cloudflare test essentially goes out
of its way to circumvent RPKI filters, but at the same time is easily
fooled in the presence of default routes (0.0.0.0/0 + ::/0).

To get a broader sense of how one's local internet connection is impacted
by RPKI, is to compare traceroutes to 103.21.244.15 versus traceroutes
to 1.1.1.1 - if the first trace takes a bit of a detour compared to the
latter IP, it might be indicative of only one (or a few) routers in a
global IP backbone are not RPKI-capable.

In addition to the CF test, I recommend also testing similar but
alternative tools, such as https://sg-pub.ripe.net/jasper/rpki-web-test/
The ripe.net test is *not* anycasted and single-homed behind a
transit-free carrier, this too skews the results in some way.

Another test can be done by pinging the RIPE RIS "Resource Certification
(RPKI) Routing Beacons" at the bottom of this page:
https://www.ripe.net/analyse/internet-measurements/routing-information-service-ris/current-ris-routing-beacons

And yet another way of measuring to what degree RPKI ROV has been
deployed in an individual AS or the DFZ as a whole, is by looking at BGP
data. The NLNOG RING LG (AS 199036, http://lg.ring.nlnog.net/summary/lg01/ipv4)
receives tens of full table feeds from various BGP speakers around the
planet. Every few hours a script takes a snapshot of the LG's Local RIB
and applies the RFC 6811 Origin Validation procedure to all paths, and
for a select few ASNs stores the list of prefixes.

Cecilia Testart et al. did a thorough study using similar methodology:
https://www.caida.org/publications/papers/2020/filter_not_filter/filter_not_filter.pdf
This paper is a fun friday afternoon read!

Below is the current top ten "RPKI invalid distributor" ASNs as seen
from AS 199036:

   RPKI invalid routes | Transiting Autonomous System
   +-
 2,224 | AS6461 - Zayo
 2,094 | AS3320 - Deutsche Telekom
 1,989 | AS8220 - Colt
 1,976 | AS5511 - Orange
 1,924 | AS6762 - Telecom Italia
 1,613 | AS1273 - Vodafone
   573 | AS6453 - Tata
   436 | AS6939 - Hurricane Electric
   425 | AS6830 - Liberty Global
   355 | AS3491 - PCCW
 (rough estimates as of April 26th, 2021)

Cogent (AS 174) isn't even in the global top ten RPKI Invalids
distributors! :-) Banana for scale: in 2018-2019 the top ten was
distributing between 5,000 and 6,000 unique RPKI invalid routes.

Many in the community deploying RPKI consider a RPKI deployment
'functionally complete' when a transit network dives below propagating
~ 30% of the total of DFZ invalids (and manages to stay there).

The gap of ~ 1,600 prefixes between Zayo/Deutsche Telekom - and the group
of ASNs propagating less than 600 - is the difference between not
rejecting invalids on any EBGP session, and rejecting invalids on most
EBGP sessions.

How does one end up deploying RPKI ROV on most, but not all EBGP sessions?

In the last few years HUNDREDS of RPKI-related software defects have
been uncovered in BGP implementations. Some bugs are cosmetic in nature,
other bugs are of the "if you enable RPKI, the entire router crashes"
severity level. When bugs are identified and fixed, it'll take
additional time for the QA process to complete and deploym

Re: BGP and The zero window edge

[Job Snijders via NANOG]

On Thu, Apr 22, 2021 at 02:29:31PM +0300, Alexandre Snarskii wrote:
> 9002. Hit by Juniper PR1562090, route stuck in DeletePending..
> Workaround applied, sessions with 6939 restarted, route is gone.

Thank you for the details and clearing the issue.

Kind regards,

Job

Re: BGP and The zero window edge

[Job Snijders via NANOG]

On Wed, Apr 21, 2021 at 09:22:57PM +, Jakob Heitz (jheitz) wrote:
> I'd like to get some data on what actually happened in the real cases
> and analyze it.
>
> [snip]
> 
> TCP zero window is possible, but many other things could
> cause it too.

Indeed. There could be a number of reasons that caused it.

Switchings away from TCP win=0 towards "Zombie Routes":

*RIGHT NOW* (at the moment of writing), there are a number of zombie
route visible in the IPv6 Default-Free Zone:

One example is 
http://lg.ring.nlnog.net/prefix_detail/lg01/ipv6?q=2a0b:6b86:d15::/48

2a0b:6b86:d15::/48 via:
BGP.as_path: 204092 57199 35280 6939 42615 42615 212232
BGP.as_path: 208627 207910 57199 35280 6939 42615 42615 212232
BGP.as_path: 208627 207910 57199 35280 6939 42615 42615 212232
(first announced April 15th, last withdrawn April 15th, 2021)

Another one is 
http://lg.ring.nlnog.net/prefix_detail/lg01/ipv6?q=2a0b:6b86:d24::/48

2a0b:6b86:d24::/48 via:
BGP.as_path: 201701 9002 6939 42615 212232
BGP.as_path: 34927 9002 6939 42615 212232
BGP.as_path: 207960 34927 9002 6939 42615 212232
BGP.as_path: 44103 50673 9002 6939 42615 212232
BGP.as_path: 208627 207910 34927 9002 6939 42615 212232
BGP.as_path: 3280 34927 9002 6939 42615 212232
BGP.as_path: 206628 34927 9002 6939 42615 212232
BGP.as_path: 208627 207910 34927 9002 6939 42615 212232
(first announced March 24th, last withdrawn March 24th, 2021)

Just now, I literally rebooted the BGP speaker behind lg.ring.nlnog.net
to make ensure that those routes are not stuck in the BGP looking glass
itself. 

2a0b:6b86:d24::/48 was first announced on March 24th, 2021, and
withdrawn at the end of March 24th, 2021 by the originator, and now
almost a month later, this prefix still is visible in the default-free
zone despite WITHDRAW messages having been sent and the AS 212232
operator confirming they are not announcing that IP prefix anywhere.

I checked the AS 6939 Looking glass, but the d24::/48 route is not
visible in the http://lg.he.net/ web interface. This leads me to believe
the the route got stuck somewhere along way in either of 201701, 204092,
206628, 207910, 207960, 208627, 3280, 34927, 35280, 44103, 50673, 57199,
and/or 9002.

This implies indeed might be multiple reasons a BGP route gets stuck
('stuck' as in - a WITHDRAW was not generated, or ignored). Perhaps on
any one of these edges there is a very high Out Queue for one reason or
another:

34927 9002
206628 34927
44103 50673
207960 34927
3280 34927
9002 6939
201701 9002
208627 207910

I'm not sure all the these sightings of stuck routes can be pinpointed
to one specific BGP vendor (or one bug).

Kind regards,

Job

Re: BGP and The zero window edge

[Job Snijders via NANOG]

Dear Jakob, group,

On Wed, Apr 21, 2021 at 08:59:06PM +, Jakob Heitz (jheitz) via NANOG wrote:
> Ben's blog details an experiment in which he advertises routes and then
> withdraws them, but some of them remain stuck for days.
> 
> I'd like to get to the bottom of this problem.

I think there are *two* problems:

1) some BGP implementations (or multi-node BGP configurations) sometimes
   end up getting stuck in one way or another.

2) other BGP nodes are not able to disconnect/reconnect to systems
   suffering from instantiations of problem #1.

While on the one hand it is important to follow-up on each and every
instantiation of problem #1, I personally think it also is worthwhile
exploring whether the BGP FSM itself can be redefined in a way that
encourages BGP protocol implementations to be more robust and rely less
on the remote peer behaving correctly.

Once Problem #2 is addressed, finding and isolating instances of Problem
#1 will become much easier.

> Has anyone else seen this before or can provide data to analyze?
> On or off list.

>From the BGP Default-Free Zone perspective it is hard to differentiate
between an entire (multi-vendor) Autonomous System being stuck, or just
one router.

To test individual router implementations this tool is useful
https://github.com/benjojo/bgp-zerowindow-test - but please keep in mind
that "TCP Recv Wind == 0" trick is just one way to easily get a BGP peer
to manifest the problematic behavior.

>From a BGP protocol perspective BGP nodes shouldn't inspect the TCP
receive window, but rather focus on whether all locally available
signals indicate that the remote peer is still progressing data.

Kind regards,

Job

Re: ARIN-NONAUTH IRR final retirement set for 31 March 2022 (was: ARIN-NONAUTH data ARIN-NONAUTH dataFwd: [ARIN-consult] ACSP Consultation 2021.1: Future of ARIN’s Unauthenticated IRR is now Closed)

[Job Snijders via NANOG]

Dear John,

Thank you for extending the deadline with another 6 months. Obviously 6
months amidst a global pandamic would never be enough time. :-)

Both John Sweeting [1] and myself [2] assert there are tens of thousands
of objects for which the relationship between the object's existence and
the state of related routes in the BGP default-free zone routing are not
fully understood.

It appears ARIN has choosen to ignore a specific community suggestion:
please first prove ARIN can manage cleanup of a select few objects,
before proceeding to deprecate the entire ARIN-NONAUTH database.

I'm not convinced mere 'community outreach' is an appropriate and
substantive response to the anteriority of 'Legacy Holders' who might
depend on both ARIN's reverse DNS service and ARIN's (Non Authoritative)
IRR service.

I personally would like to see ARIN postpone any decision on whether to
sunset the ARIN-NONAUTH database in favor of first applying
community-recognised object clean-up mechanisms, and then based on the
'lessons learned' from such a cleanup process, inform future go/no-go
decisions.

Can you share with me what the process would be to appeal the decision
made here and how course can changed?

Kind regards,

Job

[1]: https://lists.arin.net/pipermail/arin-consult/2021-March/001241.html
[2]: https://lists.arin.net/pipermail/arin-consult/2021-March/001237.html

On Tue, Mar 16, 2021 at 11:03:56AM +, John Curran wrote:
> NANOGers -
> FYI - Outcome of the community consultation on the Future of ARIN’s 
> Unauthenticated IRR.
> /John
> 
> John Curran
> President and CEO
> American Registry for Internet Numbers
> 
> Begin forwarded message:
> 
> From: ARIN mailto:i...@arin.net>>
> Subject: [ARIN-consult] ACSP Consultation 2021.1: Future of ARIN’s 
> Unauthenticated IRR is now Closed
> Date: 15 March 2021 at 4:27:04 PM EDT
> To: mailto:arin-cons...@arin.net>>
> 
> I would like to thank everyone who provided valuable feedback during this 
> consultation on the future of ARIN’s unauthenticated IRR. Input provided by 
> the community is a vital part of our planning processes at ARIN, and after 
> reviewing responses to the consultation, we are making a few significant 
> adjustments in our path forward.
> 
> One concern raised during the consultation was that six months was not enough 
> time for organizations that currently depend on data in ARIN’s 
> unauthenticated IRR to make changes to their systems. As a result, we will be 
> extending the availability of ARIN’s non-authenticated IRR for an additional 
> six months with final retirement set for 31 March 2022.
> 
> The second major suggestion was that ARIN conduct proactive engagement to 
> notifying customers who currently use our unauthenticated IRR. We have 
> already initiated this outreach, and for the next twelve months, we will 
> continue this direct outreach to customers who have records in ARIN’s 
> non-authenticated IRR to inform them of their options and provide necessary 
> assistance.
> 
> We will also provide regular reminders and updates to our community and 
> organizations making use of the outdated and non-authenticated IRR data 
> stream so they can be ready for when we cease publishing the ARIN-NONAUTH 
> data stream. These reminders leading up to the retirement of ARIN’s 
> non-authenticated IRR will include statistics on the number of records 
> remaining in that system and related routing coverage.
> 
> Thank you again to those who provided valuable feedback on this consultation.
> 
> Regards,
> 
> John Curran
> President and CEO
> American Registry for Internet Numbers
> ___
> ARIN-Consult
> You are receiving this message because you are subscribed to the ARIN Consult 
> Mailing
> List (arin-cons...@arin.net).
> Unsubscribe or manage your mailing list subscription at:
> https://lists.arin.net/mailman/listinfo/arin-consult Please contact the ARIN 
> Member Services
> Help Desk at i...@arin.net if you experience any issues.

Re: RPKI invalid logs?

[Job Snijders via NANOG]

Dear Hank,

On Sat, Feb 20, 2021 at 07:37:08PM +0200, Hank Nussbacher wrote:
> Is there a place where one can examine RPKI invalid logs for a specific date
> & time 

I have set up a publicly accessible archiver instance in Dallas, and one
in Amsterdam which capture and archive data every 20 minutes.

Please visit for access to downloadable archives http://www.rpkiviews.org/

> or even better logs showing those that dropped RPKI invalid
> announcements?

You can extract the rpki-client.json file from the archive from the
timestamp you are interested in, and pass it as cache file to
https://github.com/job/rpki-ov-checker, and via STDIN feed it a list of
Prefix + OriginAS combos (sourced from MRT data or your internal
administration / expectations).

If you like this service, please consider making a server in Israel
available to rpkiviews.org. All that is required is a
POSIX.1-ish-compliant server (BSD, Linux, or UNIX), and about 6
terabytes of storage (should be good for next 3 years), and a globally
unique publicly reachable IP address. You pick the hostname.

Kind regards,

Job

Re: Famous operational issues

[Job Snijders via NANOG]

On Tue, Feb 16, 2021 at 01:37:35PM -0600, John Kristoff wrote:
> I'd like to start a thread about the most famous and widespread Internet
> operational issues, outages or implementation incompatibilities you
> have seen.
> 
> Which examples would make up your top three?

This was a fantastic outage, one could really feel the tremors into the
far corners of the BGP default-free zone:

https://labs.ripe.net/Members/erik/ripe-ncc-and-duke-university-bgp-experiment/

The experiment triggered a bug in some Cisco router models: affected
Ciscos would corrupt this specific BGP announcement ** ON OUTBOUND **.
Any peers of such Ciscos receiving this BGP update, would (according to
then current RFCs) consider the BGP UPDATE corrupted, and would
subsequently tear down the BGP sessions with the Ciscos. Because the
corruption was not detected by the Ciscos themselves, whenever the
sessions would come back online again they'd reannounce the corrupted
update, causing a session tear down. Bounce ... Bounce ... Bounce ... at
global scale in both IBGP and EBGP! :-)

Luckily the industry took these, and many other lessons to heart: in
2015 the IETF published RFC 7606 ("Revised Error Handling for BGP UPDATE
Messages") which specifices far more robust behaviour for BGP speakers.

Kind regards,

Job

Re: Problems with newish IP block assignment issues from ARIN

[Job Snijders via NANOG]

On Mon, Feb 08, 2021 at 04:02:14PM -0500, Justin Wilson (Lists) wrote:
> I enabled 134.195.47.1 on one of our routers.

Cool! I noticed the following: from many NLNOG RING nodes I can reach
that IP address, but not from 195.66.134.42:

deepmedia01.ring.nlnog.net:~$ mtr -z -w -r 134.195.47.1
Start: 2021-02-08T21:19:32+
HOST: deepmedia01.ring.nlnog.netLoss%   Snt   Last   Avg  Best  
Wrst StDev
  1. AS39022  vlan100.ccr-1.gs.as39022.net   0.0%100.5   0.5   0.4  
 0.5   0.1
  2. AS???speed-ix.he.net0.0%100.8   1.0   0.7  
 2.5   0.5
  3. AS6939   100ge16-1.core1.lon2.he.net0.0%106.8   7.0   6.7  
 8.1   0.5
  4. AS6939   100ge4-1.core1.nyc4.he.net 0.0%10   83.7  77.7  72.5  
93.8   8.4
  5. AS6939   ve951.core2.nyc4.he.net0.0%10   73.0  73.0  72.6  
74.9   0.7
  6. AS6939   100ge0-31.core2.cmh1.he.net0.0%10   85.7  86.4  85.6  
88.7   1.1
  7. AS6939   100ge9-2.core1.ind1.he.net 0.0%10   93.4  93.4  93.2  
94.6   0.4
  8. AS6939   184.105.30.134 0.0%10   93.0  93.1  92.9  
93.3   0.1
  9. AS??????   100.0100.0   0.0   0.0  
 0.0   0.0

Do you have a BGP route for 195.66.134.0/23 on the router with
134.195.47.1 ?

Do you have a traceroute towards 195.66.134.42?

Kind regards,

Job

Re: Problems with newish IP block assignment issues from ARIN

[Job Snijders via NANOG]

Dear Justin,

On Mon, Feb 08, 2021 at 03:14:47PM -0500, Justin Wilson (Lists) wrote:
> It acts like the IP block was blacklisted at some point and got on
> some bad lists but I don’t want ti limit myself to that theory.
> I have opened up a ticket with ARIN asking for any guidance. Has
> anyone ran into this with new space assigned? Any tools, sites, etc. I
> can use to do further troubleshooting.  

Here are some useful tools:

ping.pe
example: http://ping.pe/www.openbsd.org

https://ring.nlnog.net/
good introduction here: 
https://labs.ripe.net/Members/martin_pels_3/10-years-of-nlnog-ring

https://atlas.ripe.net/

> The block in question is 134.195.44.0/22. 

Is there any specific IP address in the range that should always respond
to ICMP Echo Requests? This will help others see if they can reach you
or not.

> It has been RPKI certified and has IRR entries.

Indeed, nice :-) http://irrexplorer.nlnog.net/search/134.195.44.0/22

Kind regards,

Job

Re: Issues with NANOG mailing list operations and subscriptions

[Job Snijders]

Hi Sean, Will, group,

On Sun, Jan 17, 2021 at 03:01:22PM -0800, William Herrin wrote:
> On Sun, Jan 17, 2021 at 1:37 PM Sean Donelan  wrote:
> > Some people think its funny to ghost subscribe email addresses, and
> > the NANOG mailing list auomation doesn't catch them in the verification
> > process.
> 
> Hi Sean,
> 
> How is that possible? This is exactly what a correctly implemented
> confirmed opt-in procedure is designed to prevent.

Someone was kind enough to share a plausible theory outlining a
potentially fatal flaw in the current verification process. Some
auto-responders are (by accident) able to respond in such a way they end
up positively passing the automated verification process. 

I've shared some suggestions with the volunteers & nanog staff (who run
this mailing list) to improve the subscription verification process.
I am optimistic in the near future we'll be able to prevent this type of
situation from happening again.

Kind regards,

Job

ps. I'm one of the volunteers who support the NANOG staff in a technical
capacity to operate this mailman instance.

Re: what is the policy about sharing email offlist?

[Job Snijders]

Dear all,

On Mon, Jan 18, 2021 at 11:17:06AM -0700, Anne P. Mitchell, Esq. wrote:
> Either Alexandria Ocasio-Cortez' office is on the NANOG list or
> someone is forwarding NANOG email to AOC's press office (in which case
> either spoofed as the original sender or AOC's office sends an ack to
> every email address it can find)..as I received this auto-ack in
> response to my email to to the list.
> 
> Anyone have any insight into this?

The NANOG Mailing-list usage guidelines prohibit auto-reponse postings
to the list or to other users: https://www.nanog.org/resources/usage-guidelines/

The mailing list admin team (adm...@nanog.org) can take action to
prevent auto-responders decrease the signal to noise ratio. As multiple
users have now made a report about an auto-responder, I'm sure the issue
will be resolved soon.

> And either way, what is the policy about forwarding list email to
> someone who is not on the list?

The NANOG mailing list is a so-called 'public' mailing list. Emails send
to the list are distributed to all subscribed email addresses, and in
addition to such this distribution, list postings archived at a publicly
accessible location for anyone prefering to use a web browser:
https://mailman.nanog.org/pipermail/nanog/

It is possible for users to distribute NANOG mailing list postings
through other means, such as additional forwarding and archiving
(example: https://marc.info/?l=nanog)

The NANOG mailing list AUP does not prohibit forwarding of emails which
were sent to NANOG the list to people not on the list. 

Kind regards,

Job

Fw: [lacnog] Update on LACNIC's IRR: Near-Real-Time Mirroring Now Available

[Job Snijders]

FYI

- Forwarded message from "Carlos M. Martinez"  -

Date: Mon, 23 Nov 2020 16:18:44 -0300
From: "Carlos M. Martinez" 
To: Latin America and Caribbean Region Network Operators Group

Subject: [lacnog] Update on LACNIC's IRR: Near-Real-Time Mirroring Now
Available

Colleagues,

We are pleased to inform that LACNIC's Internet Routing Registry (IRR) which
came into production earlier this year now supports Near-Real-Time Mirroring
(NRTM) using IRRd Version 4.

The mirroring policy is an open policy and those organizations that wish to
mirror the information contained in LACNIC's IRR can do so.

To set up a mirror, you will need the following information:

NOC: n...@lacnic.net
Dump: ftp://irr.lacnic.net/lacnic.db.gz / http://irr.lacnic.net/lacnic.db.gz
CURRENT SERIAL: ftp://irr.lacnic.net/LACNIC.CURRENTSERIAL
http://irr.lacnic.net/LACNIC.CURRENTSERIAL
NRTM Host: irr.lacnic.net
NRTM Port:43

When LACNIC enables NRTM in the coming days, other IRRs such as RADB and NTT
will begin mirroring the LACNIC source.

We would also like to thank the DashCare team (https://dashcare.nl/), Job
Snijders (NTT) and the RADB team for their support.

Should you have any questions or doubts, please don't hesitate to contact us
at tecnolo...@lacnic.net/

The LACNIC Team

___
LACNOG mailing list
lac...@lacnic.net
https://mail.lacnic.net/mailman/listinfo/lacnog
Cancelar suscripcion: https://mail.lacnic.net/mailman/options/lacnog


- End forwarded message -

Re: inspecting RPKI data: console.rpki-client.org

[Job Snijders]

On Fri, Nov 20, 2020 at 12:02:04PM -0500, Tom Beecher wrote:
> In before snark of "OMG "http" links to RPKI info HURF BLURF!"

But Tom, that is exactly the whole point of the RPKI :-)

It's funny, but true! You really can safely use the RPKI data from the
console website in your own production environment, even after it has
been transported via mere HTTP - provided you have the TAL files to
build the chain of trust.

This applies also applies to the console's HTML itself: if you have the
TAL files + rpki-client + rsync + the openssl cli utility + ksh + perl;
you can generate any of the pages yourself and thus confirm their
authenticity and integrity.

Of course I don't expect anyone to jump through those hoops, but the
source code is here: https://github.com/job/console.rpki-client.org

I'll concede HTTPS does provide some privacy while looking at these
gorgeous ASN.1 data structures ;-)

Kind regards,

Job

inspecting RPKI data: console.rpki-client.org

[Job Snijders]

Dear all,

I'd like to introduce another tool to inspect RPKI data... the
rpki-client console! Comes with an authentic 90s look & feel :-)

The Frontpage - http://console.rpki-client.org/
---
On the front page you can see stdout + stderr of the most recent
rpki-client run. The log shows which publication points were contacted
and prints any issues encountered with specific RPKI files.

Those of us publishing RPKI data should keep an eye out not to show up
in this type of log with warnings or errors. For example:

rpki-client: cc.rg.net/rpki/RGnet-cc/1opByAd8x8R2F-SzstgaYzVXK8Q.mft: mft 
expired on Oct 12 17:58:45 2020 GMT

However, the above line might be the result of some kind of experiment someone 
is conducting :-)

The RPKI distributed database currently is more than 120,000 (!)
certificate/roa/manifest files, and only a handful of files have some
kind of completeness or expiration date issue. Good job everyone! :-)

The ASN specific pages - http://console.rpki-client.org/AS2914.html
---
You can substitute the 'AS2914' portion in the URL for any ASN to see
which .roa files reference the given ASN. Another example, here one can
see all ROAs which authorize AS 8283 as origin: 
https://console.rpki-client.org/AS8283.html
If you encounter a HTTP 404 error, no ROAs reference the ASN. 

On the 'per ASN page' you can search click the .roa files on the left
side to inspect the ROA. Each object in the RPKI has a unique Subject
Key Identifier (SKI). An example of a SKI is this hexadecimal identifier
'06:96:B3:F7:CC:AD:55:45:A5:3A:64:32:31:2B:7F:E1:2B:7A:15:22' which
maps to a filename like 
'rpki.apnic.net/member_repository/A91A4C60/B526FF74D84111E9A4521413C4F9AE02/12F0D72E7BC111EA8503D815C4F9AE02.roa'

Yeah... compared to DNS names mapping to IPv6 addresses, in the RPKI
neither the path name nor the SKI are easy to remember :-)

The console can show that .roa file in human readable format, just
append .html: 
http://console.rpki-client.org/rpki.apnic.net/member_repository/A91A4C60/B526FF74D84111E9A4521413C4F9AE02/12F0D72E7BC111EA8503D815C4F9AE02.roa.html

Every object in the RPKI is subordinate to another object (all objects
are signed by a parent certificate, except the Trust Anchors). The
parent is identified by the Authority Key Identifier (AKI). So one
object's AKI is another object's SKI! If you click the AKI, the console
brings you to the parent object, from where you can continue to explore
other objects related to parent.

Certificates point to Manifests, and .mft files contain the 'directory
indexes' of the RPKI: 
http://console.rpki-client.org/rpki.apnic.net/member_repository/A91A4C60/B526FF74D84111E9A4521413C4F9AE02/nvnkN242ZTJ1x5Y1mNa0W3CvgJk.mft.html
>From the manifest overview you can jump to the parent, click the
referenced .roa, .cer or .crl files.

All directories on the webserver are 'open', except the root. This
allows you to explore this RPKI cache by browsing through the filesystem
directly, example: 
http://console.rpki-client.org/rpki.apnic.net/member_repository/

Final notes
---
The rpki-client console provides a view on *validated* RPKI data. First
rpki-client runs and prunes bad files, then all HTML is generated. The
console provides a view on the data as used in production Internet
routers. Please note: the console's rendering is delayed by a bit over
an hour compared to the real thing.

Another entry point, you can use your browser's 'find on page' function
to search for anything in all of it on this humongous page:
http://console.rpki-client.org/roas.html

The RPKI is very intricate collection of references, I hope this console
offers another useful perspective on the tree-like structures. Enjoy!

Kind regards,

Job

Re: Newbie Questions: How-to remove spurious IRR records (and keep them out for good)?

[Job Snijders]

Dear Pirawat,

On Mon, Oct 26, 2020 at 08:13:19PM +0700, Pirawat WATANAPONGSE wrote:
> I am seeking advice concerning someone else announcing IRR records on
> resources belonging to me.

Change is underway in the IRR ecosystem! The situation we are all used
to is that it is rather cumbersome to get IRR databases to remove IRR
objects. The IRR database operator may not trust your request for object
removal, or is busy doing other things. There was no industry-wide
automated process for IRR object removal.

With the introduction of "RIPE-731" 
(https://www.ripe.net/publications/docs/ripe-731)
in the RIPE region, the "RIPE-NONAUTH" database has slowly been
shrinking. The RIPE-NONAUTH database exclusively contains IRR objects
covering non-RIPE space. As more and more people create RPKI ROAs, which
in turn provide automated evidence whether objects in RIPE-NONAUTH are
valid or not valid. If an object is found to be invalid, it is deleted.

While RIPE-731 addressed the issue of stale objects in the RIPE-NONAUTH
database, of course it did not change anything for non-RIPE databases.
Most non-RIPE databases use software called "IRRd" (the likes of NTTCOM,
RADB, TC, etc). The IRRd software is the main entrypoint into the IRR
system, and recently IRRd v4.1.0 was released which can automatically
delete RPKI invalid IRR route objects.

A youtube video from last week with some information on IRRdv4 can be
seen here: https://www.youtube.com/watch?v=V9fsU0mNcA4

NTT has not yet upgraded from 4.0.0 -> 4.1.0, we are working on that.
RADB is also investigating a migration path. LACNIC & ARIN already are
on the v4 train.

The moment NTT and RADB have deployed 4.1.0 at rr.ntt.net and
whois.radb.net there will be an industry-wide fully automated IRR
cleanup process running which accomplishes two things:

- stale/rogue/erroneous objects (conflicting with RPKI) are deleted
- new objects which are in conflict with RPKI ROAs cannot be created

Using RPKI to clean up the IRR is a continuous process: this mechanism
helps clean up the past, but also going forward ensures that IRR does
not contain new information which is in conflict with published
cryptographically signed RPKI ROAs.

This 2018 video outlines the strategy how to migrate to an improved
state of internet routing security: https://www.youtube.com/watch?v=3BAwBClazWc
https://nlnog.net/static/nlnogday2018/9_routing_security_roadmap_nlnog_2018_snijders.pdf
Reality is now nearly synced up to all slides of the deck :-)

Kind regards,

Job

Re: plea for comcast/sprint handoff debug help

[Job Snijders]

On Mon, Nov 02, 2020 at 09:13:16AM +0100, Tim Bruijnzeels wrote:
> On the other hand, the fallback exposes a Malicious-in-the-Middle
> replay attack surface for 100% of the prefixes published using RRDP,
> 100% of the time. This allows attackers to prevent changes in ROAs to
> be seen.

This is a mischaracterization of what is going on. The implication of
what you say here is that RPKI cannot work reliably over RSYNC, which is
factually incorrect and an injustice to all existing RSYNC based
deployment. Your view on the security model seems to ignore the
existence of RPKI manifests and the use of CRLs, which exist exactly to
mitigate replays.

Up until 2 weeks ago Routintar indeed was not correctly validating RPKI
data, fortunately this has now been fixed:
https://mailman.nanog.org/pipermail/nanog/2020-October/210318.html

Also via the RRDP protocol old data be replayed, because because just
like RSYNC, the RRDP protocol does not have authentication. When RPKI
data is transported from Publication Point (RP) to Relying Party, the RP
cannot assume there was an unbroken 'chain of custody' and therefor has
to validate all the RPKI signatures.

For example, if a CDN is used to distribute RRDP data, the CDN is the
MITM (that is literally what CDNs are: reverse proxies, in the middle).
The CDN could accidentally serve up old (cached) content or misserve
current content (swap 2 filenames with each other).

> This is a tradeoff. I think that protecting against replay should be
> considered more important here, given the numbers and time to fix
> HTTPS issue.

The 'replay' issue you perceive is also present in RRDP. The RPKI is a
*deployed* system on the Internet and it is important for Routinator to
remain interopable with other non-nlnetlabs implementations.

Routinator not falling back to rsync does *not* offer a security
advantage, but does negatively impact our industry's ability to migrate
to RRDP. We are in 'phase 0' as described in Section 3 of
https://tools.ietf.org/html/draft-sidrops-bruijnzeels-deprecate-rsync

Regards,

Job

RPKI over RSYNC vs RRDP (Was: plea for comcast/sprint handoff debug help)

[Job Snijders]

On Fri, Oct 30, 2020 at 12:47:44PM +0100, Alex Band wrote:
> > On 30 Oct 2020, at 01:10, Randy Bush  wrote:
> > i'll see your blog post and raise you a peer reviewed academic paper
> > and two rfcs :)
> 
> For the readers wondering what is going on here: there is a reason
> there is only a vague mention to two RFCs instead of the specific
> paragraph where it says that Relying Party software must fall back to
> rsync immediately if RRDP is temporarily unavailable. That is because
> this section doesn’t exist.

*skeptical face* Alex, you got it backwards: the section that does not
exist, is to *not* fall back to rsync. But on the other hand, there are
ample RFC sections which outline rsync is the mandatory-to-implement
protocol. Starts at RFC 6481 Section 3: "The publication repository
MUST be available using rsync".

Even the RRDP RFC itself (RFC 8182) describes that RSYNC and RRDP
*co-exist*. I think this co-existence was factored into both the design
of RPKIoverRSYNC and subsequently RPKIoverRRDP. An rsync publication
point does not become invalid because of the demise of an
once-upon-a-time valid RRDP publication point.

Only a few weeks ago a large NIR (IDNIC) disabled their RRDP service
because somehow the RSYNC and RRDP repositories were out-of-sync with
each other. The RRDP service remained disabled for a number of days
until they repaired their RPKI Certificate Authority service.

I suppose that during this time, Routinator was unable to receive any
updates related to the IDNIC CA (pinned to RRDP -> because of a prior
successful fetch prior to the partial IDNIC RPKI outage). This in turn
deprived the IDNIC subordinate Resource Holders the ability to update
their Route Origin Authorization attestations (from Routinator's
perspective).

Given that RRDP is an *optional* protocol in the RPKI stack, it doesn't
make sense to me to strictly pin fetching operations to RRDP: Over time
(months, years), a CA could enable / disable / enable / disable RRDP
service, while listing the RRDP URI as a valid SIA, amongst other valid
SIAs.

An analogy to DNS: A website operator may add  records to indicate
IPv6 reachability, but over time may also remove the  record if
there (temporarily) is some kind of issue with the IPv6 service. The
Internet operations community of course encourages everyone to add 
records, and IPv6 Happy Eyeballs were a concept to for a long time even
*favor* IPv6 over IPv4 to help improve IPv6 adoption, but a dual-stack
browser will always try to make benefit of the redundancy that exists
through the two address families.

RSYNC and RRDP should be viewed in a similar context as v4 vs v6, but
unlike with IPv4 and IPv6, I am convinced that RSYNC can be deprecated
in the span of 3 or 4 years, the draft-sidrops-bruijnzeels-deprecate-rsync
document is helping towards that goal! 

> Be that as it may, operators can rest assured that if consensus goes
> against our logic, we will change our design.

Please change the implementation a little bit (0.8.1). I think it is too
soon for the internet wide 'rsync to RRDP' migration project to be
declared complete and successfull, and this actually hampers the
transition to RRDP.

Pinning to RRDP *forever* violates the principle-of-least-astonishment
in a world where draft-sidrops-bruijnzeels-deprecate-rsync-00 was
published only as recent as November 2019. That draft now is a working
group document, and it will probably take another 1 or 2 years before it
is published as RFC.

Section 5 of 'draft-deprecate-rsync' says RRDP *SHOULD* be used when it
is available. Thus it logically follows, when it is not available, the
lowest common denominator is to be used: rsync. After all, the Issuing
CA put an RSYNC URI in the 'Subject Information Access' (SIA). Who knows
better than the CA?

The ability to publish routing intentions, and for others to honor the
intentions of the CA is what RPKI is all about. When the CA says
delegated RPKI data is available at both an RSYNC URI and an RRDP URI,
both are valid network entrypoints to the publication point. The
resource holder's X.509 signature even is on those 'reference to there'
directions (URIs)! :-)

If I can make a small suggestion: make 0.8.1 fall back to rsync after
waiting an hour or so, (meanwhile polling to see if the the RRDP service
restores). This way the network operator takes advantage of both
transport protocols, whichever is available, with a clear preference to
try RRDP first, then eventually rsync.

RPKI was designed in such a way that it can be transported even over
printed paper, usb stick, bluetooth, vinyl, rsync, and also https (as
rrdp). Because RPKI data is signed using the X.509 framework, the
transportation method really is irrelevant. IP holders can publish RPKI
data via horse + cart, and still make productive use of it!

Routinator's behavior is not RFC compliant, and has tangible effects in
the default-free zone.

Regards,

Job

Re: plea for comcast/sprint handoff debug help

[Job Snijders]

On Thu, Oct 29, 2020 at 09:14:16PM +0100, Alex Band wrote:
> In fact, we argue that it's actually a bad idea to do so:
> 
> https://blog.nlnetlabs.nl/why-routinator-doesnt-fall-back-to-rsync/
>
> We're interested to hear views on this from both an operational and
> security perspective.

I don't see a compelling reason to not use rsync when RRDP is
unavailable.

Quoting from the blog post:

"While this isn’t threatening the integrity of the RPKI – all data
is cryptographically signed making it really difficult to forge data
– it is possible to withhold information or replay old data."

RRDP does not solve the issue of withholding data or replaying old data.
The RRDP protocol /also/ is unauthenticated, just like rsync. The RRDP
protocol basically is rsync wrapped in XML over HTTPS.

Withholding of information is detected through verification of RPKI
manifests (something Routinator didn't verify up until last week!),
and replaying of old data is addressed by checking validity dates and
CRLs (something Routinator also didn't do until last week!).

Of course I see advantages to this industry mainly using RRDP, but those
are not security advantages. The big migration towards RRDP can happen
somewhere in the next few years.

The arguments brought forward in the blog post don't make sense to me.
The '150,000' number in the blog post seems a number pulled from thin
air.

Regards,

Job

Recommendation to update RPKI validators

[Job Snijders]

Hi all,

About eight months ago I discovered a number of issues in the validation
procedure of most RPKI validator softwares (including the RIPE NCC
Validator, Routinator, and OctoRPKI). The impact of improper
verification of Manifests (and associated aspects of the X.509 system)
in the RPKI can have rather dramatic effects in today's Internet routing
landscape. When handling a manifest, make sure everything is accounted
for!

The mitigation guidance is at present is very simple: just make sure all
deployed RPKI validators are updated to the latest version.

Going forward I hope our industry as a whole will be able to respond
faster to issues of this type. A write-up with examples and details is
available here: http://sobornost.net/~job/manifest_handling_issue.txt

Thank you to all involved who helped fix & progress this issue.

Kind regards,

Job

Re: IRR Explorer Error/Issue

[Job Snijders]

Dear Kevin,

I am the maintainer of NLNOG's IRRexplorer and can help.

On Wed, Oct 07, 2020 at 08:37:00PM +, Kevin McCormick wrote:
> There seems to an issue with IRR Explorer.
>
> I check the following prefix and I get the message, “The server
> encountered an internal error and was unable to complete your request.
> Either the server is overloaded or there is an error in the
> application.”
> 
> http://irrexplorer.nlnog.net/search/216.71.119.0/24
> 
> I am also seeing ARIN records are not updating on IRR Explorer.
> 
> That prefix should show the ASN under ARIN for the ARIN IRR route that
> is registered.
> 
> We are also advertising the prefix and which being advertised by
> Hurricane, but IRR Explorer does not see the route advertised by BGP.
> 
> The website has been like this since yesterday when I first checked.

Thanks for the report! I took a look and restarted the routing sources
database replication process. The server appears to be able to serve
requests again.

A new version of irrexplorer is in development (by programmers more
skilled than me!) - hopefully later this year we can all enjoy it.

Kind regards,

Job