nerd rage is hilarious On Feb 25, 12:32 pm, Eli Jones <[email protected]> wrote: > I demand you post videos to You Tube of the GAE team frantically cursing, > angrily casting blame, and furiously banging on their keyboards while > investigating yesterdays outage. Maybe grappling on the floor or throwing > things. > > "I TOLD you that relying on mulitmastermode hotspot failover for > transcontinental datacenter backup sites WOULDN'T WORK! Who unplugged all > 200 superredundant fiber routers just to plug in their hot plate?" > > And then possibly, personal essays from everyone involved describing their > feelings during the event.. How did it make you feel? Were you > professionally detached and getting the job done (Determined.. like steel)? > Were you worried that this outage implied you've become a total failure, > and it brought up all of these issues with your father and how he was going > to call you up and say, "You're just like your mother! Good for nothing... > (grumble..).. not highly available.."? Were you angry at someone else who > you just knew was the reason this outage occurred? > > What did you eat that morning? (Was it morning, late night... evening?) > Were you eating right then? Were you thinking about maybe eating? Did you > get an e-mail alert indicating System A was down? And you're like dammit.. > and then you got an e-mail alert saying Systems A and B failed in the > failover process for System A.. and you're like stopping eating and going to > a computer.. and then you get the phone call that all Backup Systems Ai, Bi > for Primary Failover Systems Ai, Bi have failed and there is a wild > hobogoblin loose in one of the offices and he's eaten the head of the Team > App Engine Disaster Recovery (TEADR)? And you're like.. "Why didn't we > purchase that Anti-Hobogoblin Sytem? That's what I said.. I said, 'You > never know... you never know when the hobogoblins strike... ..until its too > late.." > > And yes.. I want real time updates next time.. possibly a Disaster Recover > Webcast.. maybe you could hire someone to follow team members around and > live blog the whole thing. > > I think, if you devote more of your resources to these things... real live > customer service (instead of wasting time fixing things and improving the > backend.. you need to shift a lot of these scarce resources to white glove > customer appreciation b.s.) We need more of that facade of direct eye > contact, "Yes I understand your concerns"-reassurances instead of real work. > So.. when the backend doesn't get better because all of the resources > slowly move towards this kabuki play business model.. and things start > breaking.. We can feel better about it because the GAE team will be right > there, telling us that.. they understand our frustrations.. and the issue is > being taken very seriously indeed and everything is being done to fix the > problem and don't worry it'll be okay. > > Yes, yes.. I know that the reasonable, serious and well-considered response > from the GAE Team is and will be: "We can do better. And we will be even > more transparent next time. And we will keep working just as hard on the > back end while now devoting even more energy to responding to people's > e-mails about 'how am i supposed to sign up if I don't have a mobile phone' > or 'how do I do a join in GQL.. can I use a different database with > appengine?' or maybe 'Why doesn't GAE support PHP? GAE must support PHP!' " > > Anyway.. I look forward to the Post Mortem.. it's always instructive to read > them.. and if you read between the lines, there's always a little dramaturgy > that you can fill in. > > On Thu, Feb 25, 2010 at 11:09 AM, Nick Johnson (Google) < > > > > [email protected]> wrote: > > Hi, > > > We've already committed to publishing a full postmortem on the most recent > > outage, as we did for our other major outage in July last year ( > >https://groups.google.com/group/google-appengine/msg/ba95ded980c8c179...). > > Such reports take a while to prepare, however, and in the middle of a > > crisis, it's obviously necessary to prioritize fixing the issue over > > documenting it. > > > -Nick Johnson > > > On Wed, Feb 24, 2010 at 8:34 PM, Flips <[email protected]> wrote: > > >> I want to know *more*. It is not very satisfying if you just confirm > >> the outage, wait an hour and tell us that there are new problems. Half > >> an hour later you announced that all applications are in read-only > >> mode and the next status update is that the problem is solved. > > >> Specifically I want you to notify me if you know what the cause of the > >> problem is and tell me how much time you approximately need to fix it. > > >> Problem Switch to the seccond datacenter > >> On Feb 24, 7:50 pm, "Ikai L (Google)" <[email protected]> wrote: > >> > We have the downtime notify list for this specific purpose: > > >> >http://groups.google.com/group/google-appengine-downtime-notify > > >> > On Wed, Feb 24, 2010 at 9:28 AM, Flips <[email protected]> wrote: > >> > > If you want to know more about outages please star this feature > >> > > request: > > >> > >http://code.google.com/p/googleappengine/issues/detail?id=2867 > > >> > > -- > >> > > You received this message because you are subscribed to the Google > >> Groups > >> > > "Google App Engine" group. > >> > > To post to this group, send email to > >> [email protected]. > >> > > To unsubscribe from this group, send email to > >> > > [email protected]<google-appengine%2Bunsubscrib > >> > > [email protected]><google-appengine%2Bunsubscrib > >> [email protected]> > >> > > . > >> > > For more options, visit this group at > >> > >http://groups.google.com/group/google-appengine?hl=en. > > >> > -- > >> > Ikai Lan > >> > Developer Programs Engineer, Google App Enginehttp:// > >> googleappengine.blogspot.com|http://twitter.com/app_engine > > >> -- > >> You received this message because you are subscribed to the Google Groups > >> "Google App Engine" group. > >> To post to this group, send email to [email protected]. > >> To unsubscribe from this group, send email to > >> [email protected]<google-appengine%2Bunsubscrib > >> [email protected]> > >> . > >> For more options, visit this group at > >>http://groups.google.com/group/google-appengine?hl=en. > > > -- > > Nick Johnson, Developer Programs Engineer, App Engine > > Google Ireland Ltd. :: Registered in Dublin, Ireland, Registration Number: > > 368047 > > > -- > > You received this message because you are subscribed to the Google Groups > > "Google App Engine" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]<google-appengine%2Bunsubscrib > > [email protected]> > > . > > For more options, visit this group at > >http://groups.google.com/group/google-appengine?hl=en.
-- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
