FYI ---------- Forwarded message ---------- From: Garrett Rooney <[EMAIL PROTECTED]> Date: May 11, 2006 7:08 AM Subject: Outage on www.apache.org, svn.apache.org, and other related infrastructure To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED]
We've been having a heck of a day in infrastructure land, as you may have noticed if you tried to access the Subversion repository or any of the web sites... Early this morning (PDT) minotaur, the machine that hosts svn.apache.org, the ASF web sites, people.apache.org, and various other things, kernel paniced. After it was brought back up there was some odd behavior observed, random programs aborting, stuff like that. As a result, we decided to take some action before bringing all services back online. The data stored on the machine has been backed up, both to ajax (the european backup server) and to helios, another machine in the same datacenter as minotaur. We've also run 'svnadmin verify' on the repositories, both on the backup copies and on minotaur itself, to confirm that whatever is wrong with minotaur has had no ill effects. We also ran memtest86 tests on minotaur, and have found no sign of memory failure, which was our primary fear. During this time, DNS for the websites served off of minotaur was failed over to ajax, but svn.apache.org, perl.apache.org, and tcl.apache.org remained down. Subversion stayed down because we needed to verify that the repository was ok before doing anything with it, and TCL and Perl remained down because they require special setups that are not mirrored on ajax. At this point we have brought minotaur back to multiuser mode, and the mail that was backed up on it (@apache.org mailing lists and addresses) is flowing again. The Subversion repositories have been verified to be ok, there was no corruption found. Soon the Subversion repository will be brought back up in a read-only mode. We are also planning on upgrading minotaur to a newer version of FreeBSD, because our best guess for what caused the initial problem is now a kernel bug. Once that is done we will likely turn write access to the Subversion repository back on. Sometime in the near future the Subversion repository will be moved to an entirely new machine, but the details of when and how that will happen are still being determined. Thank you for your patience during this outage, The Apache Infrastructure Team -- -----------------------------------------< [EMAIL PROTECTED] >--- Tobias Bocanegra, Day Management AG, Barfuesserplatz 6, CH - 4001 Basel T +41 61 226 98 98, F +41 61 226 98 97 -----------------------------------------------< http://www.day.com >---
