Hehe. Yeah, I'm with Carcharoth. Not sure what any of that meant Andrew, but it sounds important. Glad it's back up. Thanks for keeping us informed.
Just like there's a certain amount of (anti)prestige associated with being one of the admins who've managed to delete the mainpage http://en.wikipedia.org/w/index.php?title=Special:Log&page=Main_Page is there also a barnstar for being a techie who has unintentionally taken the whole site down? :-) As Brion says "The internet is burning!" -Liam [[witty lama]] wittylama.com/blog Peace, love & metadata On Tue, Nov 17, 2009 at 2:28 AM, Carcharoth <[email protected]>wrote: > nagios? > ganglia? > 4-CPU apache? > scap? > swap? > memcached node? > > <eyes glazing over> > > Is it fixed now? Oh, good. :-) > > Carcharoth > > On Mon, Nov 16, 2009 at 3:04 PM, David Gerard <[email protected]> wrote: > > ---------- Forwarded message ---------- > > From: Andrew Garrett <[email protected]> > > Date: 2009/11/16 > > Subject: [Wikitech-l] Downtime this morning > > To: Wikimedia developers <[email protected]> > > > > > > Hi all, > > > > There has been some downtime this morning (about 15 minutes) due to a > > software update. > > > > I pushed a software update, and immediately servers started crashing > > according to nagios. Looking at ganglia, it looks like the issue was > > the familiar issue where scap pushes a few 4-CPU apaches into swap, > > which then crash and come back a few minutes later. This time, > > however, obviously a key memcached node fell over, causing a database > > overload, resulting in the site being mostly inaccessible for about > > ten minutes. > > > > I prepared to revert the software update, but determined that the > > problem was not the software update, and a scap would exacerbate the > > issue. The problem resolved itself spontaneously. > > > > We need to fix things up so the scap script is less liable to push > > machines into swap :) > > > > -- > > Andrew Garrett > > [email protected] > > http://werdn.us/ > > > > > > _______________________________________________ > > Wikitech-l mailing list > > [email protected] > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l > > > > _______________________________________________ > > WikiEN-l mailing list > > [email protected] > > To unsubscribe from this mailing list, visit: > > https://lists.wikimedia.org/mailman/listinfo/wikien-l > > > > _______________________________________________ > WikiEN-l mailing list > [email protected] > To unsubscribe from this mailing list, visit: > https://lists.wikimedia.org/mailman/listinfo/wikien-l > _______________________________________________ WikiEN-l mailing list [email protected] To unsubscribe from this mailing list, visit: https://lists.wikimedia.org/mailman/listinfo/wikien-l
