All - It is with deep regret that I announce that we encountered a system failure on the new server overnight.
Three hours ago, at 00:15 Pacific time, several key data directories on the new server were wiped out, almost as if a synchronization job had failed or was overwriting data. The damage was not in an identifiable pattern. An initial investigation was performed without an immediate result. Missing directories seemed to be random. After consulting with Henrik, we felt we had no choice but to bring the old server back online, and we have done so. Although this doesn't appear to be a hacker/attack, I have not dismissed that possibility, and have taken appropriate precautions on the old server in the event foul play was involved. Instead I suspect that something pertaining to our software upgrade work was the culprit here, but it may take a while before we identify the cause. What this means, for now, unfortunately, is that actions taken yesterday, within the past 14 hours from the time of this message, including emails sent and files uploaded, are lost. If you uploaded a draft yesterday, or took any other Datatracker action, you will need to repeat those actions today. Unfortunately whatever occurred was immediately propagated to the backup machines, since the servers didn't crash. But in addition to those things, I have a secondary backup system that runs daily. We have files and email archive messages up to about 4 hours ago, and we'll be working to restore those as soon as we can. In the meantime, the old servers are back up and running, and we will investigate and determine a way forward as soon as possible. Thank you for your patience. Glen -- Glen Barney IT Director AMS (IETF Secretariat) _______________________________________________ IETF-Announce mailing list IETF-Announce@ietf.org https://www.ietf.org/mailman/listinfo/ietf-announce
