Dear colleagues, This morning, 12 September 2022, during scheduled maintenance, the state of rrdp.ripe.net was inconsistent between 09:35 and 09:50 UTC. After becoming aware of a potential issue, we switched to secondary RRDP infrastructure before starting our investigation, mitigating the problem.
During the incident, the RRDP repository https://rrdp.ripe.net/notification.xml referred to files that were not available, and 3480 requests resulted in a 404 response out of more than 10.97k requests (lower bound, excludes traffic after failover). According to our logs and external monitoring, the rsync service at rsync://rpki.ripe.net was fully available during this period. The alert we received indicated that notification.xml was missing while the delta/snapshot files were unavailable, which delayed our response to the incident. As an immediate action, we have already started making necessary changes to improve our alerting. We will continue investigating the root cause and focus on possible race conditions during the shutdown of the publication server (which could have happened during patching). Kind regards, Ties de Kock -- To unsubscribe from this mailing list, get a password reminder, or change your subscription options, please visit: https://lists.ripe.net/mailman/listinfo/routing-wg
