Re: [routing-wg] request for feedback: a RPKI Certificate Transparency project?

[Ben Maddison via routing-wg]

Hi Tim,

On 09/10, Tim Bruijnzeels wrote:
> 
> 
> > On 10 Sep 2021, at 11:57, Job Snijders  wrote:
> > 
> > On Fri, Sep 10, 2021 at 11:39:39AM +0200, Tim Bruijnzeels wrote:
> >> I think all would agree that transparency is good.
> >> 
> >> A key difference between RPKI and most other PKIs is that in the RPKI
> >> all objects are published in the open for all the see. 
> > 
> > Small nitpick: all objects are SUPPOSED to be published, in the open,
> > for all to see. However it is important to keep in mind we cannot assume
> > all objects were published in a way for all to see.
> > 
> >> As you mentioned your RPKI validator may miss intermediate state
> >> changes if it retrieves objects using rsync, but the RRDP protocol
> >> supports deltas, see [1].
> >> 
> >> I believe that transparency can most easily be achieved by ensuring
> >> that these deltas are preserved, and that they cannot be modified.
> > 
> > RRDP is an unauthenticated and unsigned protocol. It is possible for a
> > Publication Point to present different RRDP deltas to one RP compared to
> > what they present to another RP. Archiving RRDP deltas is interesting,
> > but IMHO happens too late in the pipeline for TA/CA audit purposes.
> > 
> > RRDP is not a replacement for Certificate Transparency, both
> > technologies solve different problems.
> 
> I did not say that it was.
> 
> I just suggested that *in the context of RPKI* RRDP can be used as a basis
> to keep track of all historic public changes.
> 
Archiving the RRDP deltas can certainly provide information as to what
was observed at the publication points, but the security of the RPKI
system lives at the object-signing layer, and so an audit log needs to
capture activity at that layer: issuance actions by the CA.

Comparing a CT log to RRDP delta archive could certainly be useful in
many cases, but that's exactly because they say things about different
parts of the infrastructure.

Cheers,

Ben


signature.asc
Description: PGP signature

Re: [routing-wg] Issue affecting rsync RPKI repository fetching

[Ben Maddison via routing-wg]

Hi Nathalie,

On 04/12, Nathalie Trenaman wrote:
> Dear colleagues,
> 
> 
>
> We are planning on changing our publication infrastructure and using the same 
> "revisions" RRDP uses for the content of the rsync repository. Rsync is an 
> officially supported distribution protocol for RPKI repository data, and it 
> is one of our highest priorities that the data published is atomic and 
> consistent. We plan to release the new publication infrastructure in Q2/Q3 
> 2021. Part of this work will mitigate these non-repeatable-reads for clients 
> using rsync.
> 
> We will update you on our progress during RIPE 82, taking place online from 
> 17-21 May 2021.
> 
The above description seems to suggest that repository access via rsync
is an optional extra that the RIPE NCC provides as a value add.

However, as of course you know, rsync is the only mandatory to implement
access method *today* and we are not yet on an agreed path towards an
RRDP-only future.

It seems to me that this issue deserves the same urgency as "the
publication server periodically reboots when used correctly".

If that requires a workaround (of the form suggested by others) now, and
then a redesign in 6 months, fine. But it is a dis-service to the
Internet community as a whole to skip the "workaround now" step.

Cheers,

Ben


signature.asc
Description: PGP signature

Re: [routing-wg] RPKI Route Origin Validation and AS3333

[Ben Maddison via routing-wg]

Hi Nathalie,

On 03/18, Nathalie Trenaman wrote:
> Dear Colleagues, Working Group,
> 
> As discussed previously in this mailing list, some community members
> expressed that they would like to see the RIPE NCC perform Route
> Origin Validation on AS. We decided to ask the community for
> advice and guidance on how we should proceed. 
> 
> What is Route Origin Validation?  Route Origin Validation is a
> mechanism by which route advertisements can be authenticated as
> originating from an expected autonomous system (AS).  The best current
> practice is to drop RPKI invalid BGP announcements. These are
> announcements that conflict with the statement as described in a Route
> Origin Authorization (ROA).
> 
I believe that you have hit the nail on the head here: dropping ROV
Invalids has (IMO) now become the best practice for operators of all
sizes. It is no longer some experimental technique for academics and
people that live at the bleeding edge.

We wouldn't have the same debate about dropping martians, right?

> What is AS?  This is the AS Number for the RIPE NCC’s main service
> network. It includes most of our *.ripe.net 
> websites, including the LIR Portal (my.ripe.net )
> and the RIPE Database. 
> 
> What is the Problem?  Currently, some of our upstream providers
> already perform ROV. This means that some of our members that
> potentially misconfigured their ROA or members who have lost control
> of creation and modification of their ROAs cannot reach our services
> via those peers. 
> 
> On the other hand, some of our upstream providers do not perform ROV,
> and if a member’s prefix is being announced by a hijacker, they cannot
> access our services. We already received a report about this.This is
> also not an ideal situation. 
> 
> From the network operations perspective, there are no obstacles to
> enable  ROV on AS, however, we have to consider that members or
> End Users who announce something different in BGP than their ROA
> claims, will be dropped and lose access to our services from their
> network. This includes the RPKI Dashboard where they can make changes
> to their ROAs. This is specially relevant when members operate
> certificate generation in hosted mode which is the current operation
> mode for almost all for our members. 
> 
Your explanation seems to suggest that the above scenarios are similar.
I would suggest that they are actually opposites of one-another:

If a network cannot access RIPE's online services because they broke
their routing, including through creating a conflicting ROA, the blame
lies with the operator of that network.

On the other hand, if a network cannot access those services because
RIPE's routers are selecting a route towards it that is identifiably
bogus using best current practices, including RPKI ROV, then the fault
lies with the RIPE NCC.

Again, no one would be wringing their hands because someone was unable
to get to the LIR portal from 10/8, right?

> From an analysis we made on 10 February, there were 511 of such
> announcements from our members and End Users.
> 
ROV is well deployed today. Either these routes are covered by Valid
aggregates, or these members are already experiencing widespread routing
issues.

In the later case, whilst that is unfortunate for those operators and
their customers in the short term, experiencing some inconvenience is
likely the only possible incentive for the problematic objects to get
cleaned up.

I don't think that RIPE or it's members should be expected to sit around
and wait for that to happen by magic.

> Our current RPKI Terms and Conditions do not mention that a Member or
> End User ROA should match their routing intentions, or any
> implications it may have if the ROA does not match their BGP
> announcement. If the community decides it is important that AS
> performs ROV, our legal team needs to update the RPKI Terms and
> Conditions to reflect the potential impact. 
> 
As others have already indicated, I don't think that this has any place
in those Ts

I'm sure that if RIPE were ever to be subject to a claim arising from a
member kicking themselves out of the LIR portal, you would find no
shortage of expert witnesses on this list queuing up to help have it
laughed out of court.

> I welcome a respectful discussion and look forward to your advice and
> guidance.
> 
I think that the feedback you have received in the last 24 hours is
quite unambiguous.

I hope we'll see some swift action in response.

Cheers,

Ben


signature.asc
Description: PGP signature

Re: [routing-wg] 2019-08 New Policy Proposal (RPKI ROAs for Unallocated and Unassigned RIPE NCC Address Space)

[Ben Maddison via routing-wg]

On Thu, 2019-10-31 at 19:34 +, Nick Hilliard wrote:
> Petrit Hasani wrote on 31/10/2019 14:28:
> > A new RIPE Policy proposal, 2019-08, "RPKI ROAs for Unallocated and
> > Unassigned RIPE NCC Address Space" is now available for discussion.
> 
>  From a political point of view, I'm deeply uncomfortable with the
> idea 
> of the RIPE NCC setting out to make preemptive declarations of 
> routability for anything other than holders of resource allocations
> / 
> assignments.  This is new and precedents like this could weaken the
> RIPE 
> NCC's case if it were to argue in court that it was inappropriate for
> it 
> to create false ROAs for address blocks.
> 
The declaration that is envisioned is more akin to "there is no party
authorised to make routing attestations regarding this space". If by
"false ROAs" you mean ones that contradict the intentions of the
assignee, then this is hardly the same thing.

Perhaps an alternative would be an resource x.509 attribute that says
"this CA cert does not issue EE certs", so that any resources that it
contains could be implicitly considered bogons unless delegated to a
subordinate CA. This is of course a (large) protocol change.

Cheers,

Ben

Re: [routing-wg] 2019-08 New Policy Proposal (RPKI ROAs for Unallocated and Unassigned RIPE NCC Address Space)

[Ben Maddison via routing-wg]

On Fri, 2019-11-01 at 07:09 +0100, Job Snijders wrote:
> On Fri, Nov 01, 2019 at 01:23:44AM +, Job Snijders wrote:
> > On Thu, Oct 31, 2019 at 09:42:21PM +0100, Gert Doering wrote:
> > > On Thu, Oct 31, 2019 at 07:10:02PM +, Job Snijders wrote:
> > > > 1/ What is the ROI? I think there is only a few prefixes in the
> > > > default-free zone that are managed by RIPE NCC, but not
> > > > assigned or
> > > > allocated by RIPE NCC. So putting this machinery in place might
> > > > not
> > > > have that much benefit, while the cost of 'getting it wrong'
> > > > could
> > > > be substantial.
> > > 
> > > This was my first thought as well, but then I discovered this
> > > IPv6
> > > thing :)
> > 
> > Other than that there is lots of unassigned space in IPv6, and no
> > shortage, what is the relevance? Did you take a look at how many
> > unassigned/unallocated IPv6 prefixes (managed by RIPE NCC) are
> > actually
> > in the DFZ?
> 
> I ran the numbers, as far as I can tell we only have a handful of
> IPv6
> prefixes in the default-free zone that are RIPE NCC managed space,
> and
> in the 'reserved' category (looked at today's delegated-latest)
> 
The issue for me is not that there are many such prefixes sitting in
the DFZ in steady state, but that as part of attack-prep they can be
stood up to defeat loose-urpf-based anti-spoofing mechanisms.

We currently mitigate this using the team cymru bgp service, which has
two primary drawbacks:
1. the dependency on the underlying transport of the multihop bgp
session
2. the implicit trust that we place in TC (however merited) to make
assertions about the status of the address space.

The current proposal solves #2 outright.

#1 would only be partially solved, given that we would still rely on
connectivity between our validation caches and the ripe publication
endpoints. However, this should be far less fragile in the face of
short-term reachability issues than a bgp session.

I still support.

Cheers,

Ben

Re: [routing-wg] 2019-08 New Policy Proposal (RPKI ROAs for Unallocated and Unassigned RIPE NCC Address Space)

[Ben Maddison via routing-wg]

Hi all,

On Thu, 2019-10-31 at 15:28 +0100, Petrit Hasani wrote:
> Dear colleagues,
> 
> A new RIPE Policy proposal, 2019-08, "RPKI ROAs for Unallocated and
> Unassigned RIPE NCC Address Space"
> is now available for discussion.
> 
> This proposal aims to instructs the RIPE NCC to create ROAs with
> origin AS0 for all unallocated and unassigned address space under its
> control.
> 
I support this proposal.

We are one of the "not many operators" that try to maintain bogon
filters at our eBGP edge, and we have also been rejecting rpki-ov
invalids since earlier this year.

This policy would therefore simplify our operations considerably.

Cheers,

Ben Maddison

Re: [routing-wg] Fw: [db-wg] Suggestion further validity-checking

[Ben Maddison via routing-wg]

None at all that I have ever come across. Support depreciation.

Cheers,

Ben

Get Outlook for Android


From: routing-wg  on behalf of ripedenis--- via 
routing-wg 
Sent: Tuesday, May 28, 2019 3:13:03 PM
To: RIPE Routing Working Group
Subject: [routing-wg] Fw: [db-wg] Suggestion further validity-checking

Colleagues

We have had a suggestion (with some support) on the DB-WG mailing list about 
deprecating the "holes:" attribute in ROUTE(6) objects. Perhaps the Routing WG 
could consider if this attribute has any value in the RIPE Database.

cheers
denis

co-chair DB-WG

- Forwarded message -
From: Nick Hilliard via db-wg 
To: Edward Shryane 
Cc: "db...@ripe.net" 
Sent: Tuesday, 28 May 2019, 13:30:30 CEST
Subject: Re: [db-wg] Suggestion further validity-checking

Edward Shryane via db-wg wrote on 28/05/2019 12:12:

> Unfortunately, no cleanup was done when this rule was implemented,
> but in recent times we try to do this. I will also contact the
> maintainers of these route objects and ask them to fix the holes
> attribute(s).

I wonder if this key should be formally deprecated.  It's used for 643
out of 302354 route: objects and 40 out of 28803 route6: objects, i.e.
~0.2% and 0.1% respectively. The complexity associated with handling it
is substantial and most tools simply ignore it.

Nick

Re: [routing-wg] 2018-06 Discussion Phase (RIPE NCC IRR Database Non-Authoritative Route Object Clean-up)

[Ben Maddison via routing-wg]

Hi WG,


On 2019-05-22 13:29:10+02:00 routing-wg wrote:

We encourage you to review this proposal and send your comments to
routing-wg@ripe.net before 20 June 2019.


I strongly support this proposal. Although we don't have any, many of our 
customers have out-of-region objects in the db, which are often 
stale/duplicated/wrong. I welcome any steps that can be taken to clean this 
mess out without waiting around for operators to "do-the-right-thing".

Cheers,

Ben

Re: [routing-wg] RPKI Route Origin Validation - Africa

[Ben Maddison via routing-wg]

Hi Job,


On 2019-05-15 10:29:04+02:00 Job Snijders wrote:

Dear Ben and Mark,

You are now almost 5 weeks into the deployment - can you share any
insights on issues (or lack of issues) you've faced in the last 5 weeks?

Did you have to create exceptions for "important destinations covered by
misconfigured ROAs"? Are you aware of incidents that were prevented
because of the actions you took? Any feedback from customers or
partners?


For us it's remained mostly smooth sailing throughout. We're still seeing close 
to 0 support traffic as a result of the change, and we haven't had to create 
any local exceptions.

That's not to say that we're not seeing things that need fixing. These are 
mostly the result of vendor implementation issues. As a recent example, we've 
seen ios-xe occasionally accepting invalid routes on certain ebgp sessions 
contrary to configured policy, and despite the device correctly displaying the 
status==invalid at the cli. This has been an irritation rather than actually 
causing customer issues, but is another thing that occupies valuable 
engineering time.

Our observation has mostly been that features which get implemented and sit 
un-deployed for a considerable time are a recipe for hard-to-resolve trouble 
operator issues down the line.

The other time-sink that we need to address is our use of the ripe validator. 
It's been clear that this was going to be something that we would need to leave 
behind even before we began dropping invalids last month. The code requires an 
utterly un-reasonable amount of babysitting - we're finding that either the 
daemon or the OS needs some intervention (mostly restarts) on an almost daily 
basis. Our way forward almost certainly involves routinator and one other (tbd).

I would suggest that this community gives some serious thought to whether RIR 
resources should continue to be invested into a code base that appears to have 
outlived it's operational usefulness?

I'm around the meeting this week if anyone would like to ask questions or swap 
stories!

Cheers,

Ben



Kind regards,

Job

On Tue, Apr 09, 2019 at 11:51:49AM +, Ben Maddison via routing-wg wrote:
gt; Hello all.
gt; In November 2018 during the ZAPF (South African Peering Forum) meeting 
in Cape Town, 3 major African ISP's announced that they would enable RPKI-based 
ROV (Route Origin Validation), including dropping Invalid routes as part of 
efforts to improve Internet routing security, on the 1st April, 2019.
gt; On the 1st of April, Workonline Communications (AS37271) enabled ROV 
and began dropping Invalid routes. This applies to all eBGP sessions, both IPv4 
and IPv6.
gt; On the 5th of April, SEACOM (AS37100) enabled ROV and began dropping 
Invalid routes. This applies to eBGP sessions with public peers, private peers 
and transit providers, both for IPv4 and IPv6. eBGP sessions toward downstream 
customers will follow in 3 months time.
gt; Implementation at the third ISP has yet to be completed. We are sure 
they will communicate with the community when they are ready to do so.
gt; Please note that for the legal reasons previously discussed in various 
fora, neither Workonline nor SEACOM are utilising the ARIN TAL. As a result, 
any routes covered only by a ROA issued under the ARIN TAL will fall back to a 
status of Not Found. Unfortunately, this means that ARIN members will not see 
any improved routing security for their prefixes on our networks until this is 
resolved.
gt; We will each re-evaluate this decision if and when ARIN's policy 
changes. We are hopeful that this will happen sooner rather than later.
gt; If you interconnect with either of us and believe that you are 
experiencing any routing issues potentially related to this new policy, please 
feel free to reach out to either:
gt; - noc@workonline.africa
gt; - peer...@seacom.mu
gt; Workonline Communications and SEACOM hope that this move encourages 
the rest of the ISP community around the world to ramp up their deployment of 
RPKI ROV and begin dropping Invalid routes. We appreciate the work that 
ATamp;T and others have carried out in the same vein.
gt; In the mean time, we are happy to answer any questions you may have 
about our deployments.
gt; Thanks,
gt; Mark Tinka (SEACOM) amp; Ben Maddison (Workonline Communications).

[routing-wg] RPKI Route Origin Validation - Africa

[Ben Maddison via routing-wg]

Hello all.
In November 2018 during the ZAPF (South African Peering Forum) meeting in Cape 
Town, 3 major African ISP's announced that they would enable RPKI-based ROV 
(Route Origin Validation), including dropping Invalid routes as part of efforts 
to improve Internet routing security, on the 1st April, 2019.
On the 1st of April, Workonline Communications (AS37271) enabled ROV and began 
dropping Invalid routes. This applies to all eBGP sessions, both IPv4 and IPv6.
On the 5th of April, SEACOM (AS37100) enabled ROV and began dropping Invalid 
routes. This applies to eBGP sessions with public peers, private peers and 
transit providers, both for IPv4 and IPv6. eBGP sessions toward downstream 
customers will follow in 3 months time.
Implementation at the third ISP has yet to be completed. We are sure they will 
communicate with the community when they are ready to do so.
Please note that for the legal reasons previously discussed in various fora, 
neither Workonline nor SEACOM are utilising the ARIN TAL. As a result, any 
routes covered only by a ROA issued under the ARIN TAL will fall back to a 
status of Not Found. Unfortunately, this means that ARIN members will not see 
any improved routing security for their prefixes on our networks until this is 
resolved.
We will each re-evaluate this decision if and when ARIN's policy changes. We 
are hopeful that this will happen sooner rather than later.
If you interconnect with either of us and believe that you are experiencing any 
routing issues potentially related to this new policy, please feel free to 
reach out to either:
- noc@workonline.africa
- peer...@seacom.mu
Workonline Communications and SEACOM hope that this move encourages the rest of 
the ISP community around the world to ramp up their deployment of RPKI ROV and 
begin dropping Invalid routes. We appreciate the work that AT and others have 
carried out in the same vein.
In the mean time, we are happy to answer any questions you may have about our 
deployments.
Thanks,
Mark Tinka (SEACOM) & Ben Maddison (Workonline Communications).