thanks mark! :) On Mon, Nov 23, 2020 at 4:34 PM Mark Kosters <[email protected]> wrote: > > Summary > > On Nov 19 at 2:30PM EST (UTC-5), ARIN updated the software that generates the > RPKI repository. On Nov 20 at 9:48PM EST (UTC-5), we were notified by a 3rd > party that validators no longer were fetching ROAs from organizations that > had selected the delegated option. Upon review, ARIN Engineering discovered > that a certificate was not included in the manifest for each delegated > organization. The fix was to include that certificate in the manifest for > each delegated organization was deployed at 1:20AM EST (UTC-5) on Nov 21. At > that time, ROAs from the affected delegated repositories could then again be > fetched and validated. > > ARIN's hosted RPKI customers were not affected by this outage in any way. > > Root Cause > > The root cause of this failure was a software bug that was introduced by the > RPKI repository generator. > > Scope of Issue > > This bug meant that validators would not fetch information from the delegated > repositories during the affected period. ARIN has nine delegated > organizations and affected approximately 180 ROAs that may have disappeared > from the global RPKI system for approximately 35 hours and 40 minutes > starting on Nov 19 at 2:30PM EST (UTC-5). Depending on how validation is > setup by the ISPs who use RPKI, the route origins associated with these 180 > ROA’s may have remained in the secure state or became unsecure during this > period. >
I think 'unknown' and maybe (if there were ROA for supernets possibly invalid :( oops. > After Action Items > > ARIN will add additional delegated repository tests to prevent this type of > operational issue to happen again. Additionally, as planned, ARIN will be > adding additional improvements to its external monitoring that uses various > validators to ensure that the repository is working as intended. > Do you all document what is tested? so either RP software folk can integrate similar tests (or suggest others), and/or so actual ARIN users of the RPKI can test their (or instrument their) repository collections? > Regards, > Mark Kosters > ARIN CTO > > _______________________________________________ > arin-tech-discuss mailing list > [email protected] > https://lists.arin.net/mailman/listinfo/arin-tech-discuss _______________________________________________ arin-tech-discuss mailing list [email protected] https://lists.arin.net/mailman/listinfo/arin-tech-discuss
