Dear IETF Community - As you know, about 30 hours ago, we moved the IETF to a new server, containing upgraded OS, software, and a new Python version. As a part of that process, the Tools Team moved the Datatracker and their other software to Python 3.
14 hours ago, that new server suffered a significant data loss. Henrik was online shortly after the data loss occurred, and called me immediately. Investigation determined that the loss was caused by a command in the daily Datatracker cron script. One rsync command in that script is designed to make iana yang- parameters available to the Datatracker. After the upgrade to Python 3, that script generated a bad command-line argument, resulting in the rsync command running with an incorrect target and incorrectly deleting server data. The missing data then caused the Datatracker, the Mail Archive, and other tools to malfunction or fail. We of course operate a number of hot backup servers, all of which dutifully picked up the data changes immediately. Fortunately, just prior to that script's execution, one of AMS' offsite backup systems had grabbed a complete copy of the data on the new server. So, with the exception of approximately 2 hours of traffic during which the operating servers were impaired (roughly 0815-1015 GMT Tuesday morning) , no other data was lost. However, given the estimated time to restore that data (3-4 hours over the Internet), and given that there could be other unknowns in the software we hadn't yet identified, the optimal course was clearly to bring the old server group back online, which I did, restoring service using the old server approximately 3 hours after the problem started. At this time, AMS programmers are working on merging yesterday's Mail archive data into the live archive, while the Tools Team members are working on merging drafts and Datatracker data and other information they manage into the live system. We will send an update when this process is completed, and we will then consult with IETF leadership and schedule a new cutover event in the near future. Thank you for your patience. Glen -- Glen Barney IT Director AMS (IETF Secretariat) _______________________________________________ IETF-Announce mailing list [email protected] https://www.ietf.org/mailman/listinfo/ietf-announce
