On Nov 9, 2012, at 12:08 PM, Bruce Johnson <[email protected]> wrote:
>> >> Yes, it's not comforting to learn that they had no beta testing phase at >> all, and rolled it out like that. Arrogance or ignorance? > > This is what you get for hiring consultants to do your core business > functions…oh, wait, that was Bain's entire business plan…either that or > liberal programmers are just plain better than conservatives :-P This came out today: <http://www.theatlantic.com/technology/archive/2012/11/when-the-nerds-go-marching-in/265325/> This, in a nutshell, is what it looks like when you 'eat your own dog food' versus hiring consultants. "Josh Thayer, the lead engineer of Narwhal, had just been informed that they'd lost another one of the services powering their software. That was bad: Narwhal was the code name for the data platform that underpinned the campaign and let it track voters and volunteers. If it broke, so would everything else. They were talking with people at Amazon Web Services, but all they knew was that they had packet loss. Earlier that day, they lost their databases, their East Coast servers, and their memcache clusters. Thayer was ready to kill Nick Hatch, a DevOps engineer who was the official bearer of bad news. Another of their vendors, StallionDB, was fixing databases, but needed to rebuild the replicas. It was going to take time, Hatch said. They didn't have time. They'd been working 14-hour days, six or seven days a week, trying to reelect the president, and now everything had been broken at just the wrong time. It was like someone had written a Murphy's Law algorithm and deployed it at scale. And that was the point. "Game day" was October 21. The election was still 17 days away, and this was a live action role playing (LARPing!) exercise that the campaign's chief technology officer, Harper Reed, was inflicting on his team." This is how you deploy mission critical software that absolutely, positively has to work. You break it, and break it, and break it some more. You don't refuse to let your users see the thing before The Day, because you're afraid someone will see it. The interesting stuff (at least from my 'sysadmin' point of view) is the first half of the article; most of the rest is a paean to just how cool Harper Reed is… :-/ -- Bruce Johnson University of Arizona College of Pharmacy Information Technology Group Institutions do not have opinions, merely customs -- You received this message because you are subscribed to the Google Groups "StrataList-OT" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/stratalist-ot?hl=en.
