Dear colleagues, Summary: An issue with our RPKI software caused an invalid certificate to be published from 9.40-10:43 (UTC+1) today. This has resulted in outages. We strongly recommend network operators update their Relying Party software to the latest version.
At 9:32 (UTC+1) this morning, we processed an outgoing transfer of IP resources to another RIR service region. This caused our system to update the corresponding RPKI certificates in our Certificate Authority (CA). Unfortunately, a bug in our software caused the publication of the updated child certificate ahead of its parent. As a result, our CA published incorrect information from 9:40-10:43 (UTC+1). Some Relying Parties had applied a strict interpretation in their validator software. This meant that they were configured to reject all certificates in the manifest if a single entry was invalid. As a consequence, all RPKI certificates covering RIPE resources were rejected by these validators during this period. While RPKI is designed to "fail-open", an unrelated issue with some routers seems to have prevented this from happening, which resulted in outages. Some Relying Parties have since updated their validator software to apply a less-strict approach in light of this issue[1]. We have published a release candidate of our own RIPE NCC RPKI Validator and we plan to release this to production tomorrow. While our CA is now publishing correct information, the underlying cause remains unsolved. We are continuing to investigate and will share more details when we have them, followed by a post mortem report once everything is resolved. In the meantime, we recommend that network operators update their RPKI validator software to the latest version. Regards, Ties de Kock Software Engineer RIPE NCC [1] Note: Routinator 0.8.2, Fort, rpki-client and octorpki 1.2.2 are either unaffected or contain the updated interpretation.
