Hi everyone, As most of you know we just had a site outage that lasted about 90 minutes. A particular query from one site was getting repeated in massive volume... we had just made some changes to our own cache servers at the exact same time, so we lost a little debugging time figuring out that it wasn't something we changed but rather something new. When we found the particular query in question, we tried clearing out the backlog of query requests to the database server and restarting our apaches and our databases... but the queries would flood right back in and fill everything up.
So I've asked that the domain for that particular site be turned off and redirected to www.wikia.com ... it's a drastic measure... and one that I wouldn't take without extremely good cause. As soon as we did this, everything went back to normal on all wikia sites. We'll be working with the admins from that wiki to see what happened and will turn them on again as soon as we have this figured out. Stepping back a bit, Wikia has been growing a lot over the last few months and we're seeing the need for architectural changes as well as more equipment. A few weeks ago, I ordered about $50k of equipment to beef up both our main site and our back-up colo. Those machines (cache servers, apache servers, and more databases) will be coming on-line shortly. We started rotating in a fast new cache server and apache server yesterday and this morning and will bring those in permanently within a few days. Same with the new database servers. That will help to address short-term speed issues. Architectural changes are needed to address this longer term and we're working on those, too. As always, thanks for your patience, John Q. _______________________________________________ Wikia-l mailing list [email protected] http://lists.wikia.com/mailman/listinfo/wikia-l
