On behalf of the UnaMesa Association I would like to apologize to the members of the TiddlyWiki community for the recent downtime of TiddlyWiki.org. Due to a miscommunication error on my part, the site was taken off-line on November 23. As of Dec 3, the site and all data have been restored. For those who are interested, I have detailed below why the outage occurred, why it took so long to restore the data, and what we are doing to avoid any future downtime and contiue improving the overally experience for TiddlyWiki users and developers. Caution this is a long post. If you want just want the short recap, here is the chronology of events:
Oct: transfer trac.tiddlywiki.org, svn.tiddlywiki.org, www.tiddlywiki.org, wiki.tiddlywiki.org services to EC2 from old hosting provider - all services seem to be operating properly after a few glitches Oct 30: miscommunication - files and DNS were not updated for www.tiddlywiki.org / wiki.tiddlywiki.org Nov 24: old hosts shutdown Nov 24: www.tiddlywiki.org problem detected and diagnosed Nov 25-Dec 1: access to old host images delayed due to travel, holidays, and incorrect formats Dec 2: old hosts virtual machines received Dec 3: data for www.tiddlywiki.org restored Today: all services operating normally with regular online and offline backup, new support system in place Now the full story with a bit of background. For several years, the non-profit UnaMesa Association has been supporting the TiddlyWiki community. As a tool that allows anyone to create and distribute Web applications, TiddlyWiki is a great match for the healthcare, educational, and other communities served by UnaMesa. Operating the wiki site (http://wiki.tiddlywiki.org/), developer site (http://trac.tiddlywiki.org), and subversion repository (http:// svn.tiddlywiki.org/) is one of the main ways that we support the TiddlyWiki community. In the fall of 2008 we decided to move those servers from our existing hosting provider after receiving ongoing complaints of slow response times and bandwidth bottlenecks. In looking at hosting options, we chose Amazon's EC2 service because we believed it offered the best combination of scalable performance, reliability and manageability for a nonprofit staffed with volunteer effort. The transfer to EC2 was largely completed in October thanks to the hard work of Andreas, FND, Saq, and others. Although there were a few glitches, the developer site (trac and svn) have been up and running on the new site for over a month. The wiki site was also set up at that time. However copying over the data was delayed due to a few incompatibilities between MediaWiki versions. For the new host, we are using the new version of MediaWiki which has full support for an API that allows reading and writing MediaWiki content using TiddlyWiki (thanks Martin for creating the appropriate server side adaptors for TiddlyWiki.) Due to a micommunication, I thought that at the end of October the data had been copied over to the new site and that the DNS entry for http://wiki.tiddlywiki.org/ had been modified so that we were operating off of the new site. Since the actual work of copying the data and administering the host is split up amongst Andreas, FND, and others, we use a combination of an admin wiki and emails to update each other on the status. I misinterpreted an entry on the admin wiki to mean that the transfer was complet. I made the mistake of not confirming this by looking at the actual IP addresses reported by dig and confirming that we were indeed live with the new host. Based on my mistaken thought that the transfer was completed, I authorized shutting down the old hosts in late Nov. So on Nov 24, our old hosting provider took those hosts (virtual images) offline. At that point it became obvious that we were still live with the old host for http://wiki.tiddlywiki.org/ and that the data transfer had not been completed. (Additional mistakes on my part, I did not get an exact date for the shutdown or ask that they send a warning message before actually shutting down the hosts, nor did I ask for copies of the virtual machine images to be sent to us before the shutdown. At the time, I thought we had all the data we needed and that it was reliably stored on Amazon S3.) Further complicating matters, I happened to be traveling that week in a very rural area with no internet connectivity and that Thur was the Thanksgiving holiday in America which is where our old hosting provider is located. Fortunately Saq & Andreas were able to figure out what had happened and contact the old hosting provider. The hosting provider was eventually able to provide us with a copy of the virtual machines for all the hosts (so we now have full backup copies of all the data from the wiki, svn, trac repositories and others, just in case). We got the first copy on Friday (due to the nature of the problem, the hosting provider was only able to generate these copies on regular work days) but it was in the wrong format. On Monday we got an updated copy in an accessible format, but it turned out to be the wrong virtual machine. On Tue we finally got all the virtual machines (these are BIG files... Andreas spent lots of time downloading over the past few days) and was able to restore the data to our new MediaWiki instance. As of Dec 3, all the data should be restored and all of the *.tiddlywiki.org hosts should be working properly. We now have all data stored on Amazon's storage service and performance should be as good or better than before. We are also putting in place procedures to make sure that we have regular off-line backups as an additional precaution against future outages. We especially hope that network issues should no longer be an issue as Amazon's service seems to be quite good on a global scale. We have also taken steps to streamline the support process and make it more transperent. http://support.unamesa.org/ is the public wiki where we have begun to handle all support requests and provide updates to keep the community informed. We've also added another adminstrator who has come up to speed on the hosting infrastructure and can step in if Andreas, FND, or myself are unavailable. We're also consolidating responsibilities for DNS and host administration to avoid future miscommunications. Saq will be posting more details on the new process in the next few days. I would greatly appreciate hearing from you if the service quality is not up to your expectations or if you have additional suggestions on how we could do better. Over the next few weeks, the UnaMesa Association will be completing our plans for 2009. As part of that planning process, we will be reaching out more to the community through this group and asking where you would like to see TiddlyWiki go and what we can do to help. As you know, TiddlyWiki itself is a community effort powered by your actions. We're here to support you, so once we're sure this hosting issue is covered, we'll be coming back to ask what else we could be doing better. Lastly, I would like to acknowledge the outstanding financial support from BT Osmosoft. Their support in 2008 has made it possible to continue our support for the TiddlyWiki community both through hosting the sites and helping Eric, Saq, Martin, and others answer questions and continue improving the core code. UnaMesa is pleased to facilitate the unique partnership between BT and the TiddlyWiki community. Part of the mission of UnaMesa is to bring together corporations, NGO's, communities, and individuals to collaborate on innovative ideas for mutual benefit in a fair way that protects and serves everyone's interest. The relationship between BT, UnaMesa, TiddlyWiki develoopers, and TiddlyWiki users is a pretty good example of how that collaboration can work in practice and we look forward to hearing from you how to make it even better. -Greg --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "TiddlyWiki" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/TiddlyWiki?hl=en -~----------~----~----~----~------~----~------~--~---
