This is what I sent to the list first, but because of the length, it got held back for approval by the list owner. So I split it in two parts and resent. That's what showed up a week ago. This showed after the approval came through.
Sent from my iPad On Oct 20, 2011, at 8:21 AM, [email protected] wrote: > Posted this already a week ago yes? > > Sent via BlackBerry by AT&T > > -----Original Message----- > From: "Anthony Q. Martin" <[email protected]> > Sender: [email protected] > Date: Fri, 14 Oct 2011 13:21:53 > To: <[email protected]> > Reply-To: [email protected] > Subject: [H] Stevey's Google Platforms Rant > > I think this is interesting reading. > ---------------------------------------------------- > > Stevey's Google Platforms Rant > > I was at Amazon for about six and a half years, and now I've been at > Google for that long. One thing that struck me immediately about the two > companies -- an impression that has been reinforced almost daily -- is > that Amazon does everything wrong, and Google does everything right. > Sure, it's a sweeping generalization, but a surprisingly accurate one. > It's pretty crazy. There are probably a hundred or even two hundred > different ways you can compare the two companies, and Google is superior > in all but three of them, if I recall correctly. I actually did a > spreadsheet at one point but Legal wouldn't let me show it to anyone, > even though recruiting loved it. > > I mean, just to give you a very brief taste: Amazon's recruiting process > is fundamentally flawed by having teams hire for themselves, so their > hiring bar is incredibly inconsistent across teams, despite various > efforts they've made to level it out. And their operations are a mess; > they don't really have SREs and they make engineers pretty much do > everything, which leaves almost no time for coding - though again this > varies by group, so it's luck of the draw. They don't give a single shit > about charity or helping the needy or community contributions or > anything like that. Never comes up there, except maybe to laugh about > it. Their facilities are dirt-smeared cube farms without a dime spent on > decor or common meeting areas. Their pay and benefits suck, although > much less so lately due to local competition from Google and Facebook. > But they don't have any of our perks or extras -- they just try to match > the offer-letter numbers, and that's the end of it. Their code base is a > disaster, with no engineering standards whatsoever except what > individual teams choose to put in place. > > To be fair, they do have a nice versioned-library system that we really > ought to emulate, and a nice publish-subscribe system that we also have > no equivalent for. But for the most part they just have a bunch of > crappy tools that read and write state machine information into > relational databases. We wouldn't take most of it even if it were free. > > I think the pubsub system and their library-shelf system were two out of > the grand total of three things Amazon does better than google. > > I guess you could make an argument that their bias for launching early > and iterating like mad is also something they do well, but you can argue > it either way. They prioritize launching early over everything else, > including retention and engineering discipline and a bunch of other > stuff that turns out to matter in the long run. So even though it's > given them some competitive advantages in the marketplace, it's created > enough other problems to make it something less than a slam-dunk. > > But there's one thing they do really really well that pretty much makes > up for ALL of their political, philosophical and technical screw-ups. > > Jeff Bezos is an infamous micro-manager. He micro-manages every single > pixel of Amazon's retail site. He hired Larry Tesler, Apple's Chief > Scientist and probably the very most famous and respected human-computer > interaction expert in the entire world, and then ignored every goddamn > thing Larry said for three years until Larry finally -- wisely -- left > the company. Larry would do these big usability studies and demonstrate > beyond any shred of doubt that nobody can understand that frigging > website, but Bezos just couldn't let go of those pixels, all those > millions of semantics-packed pixels on the landing page. They were like > millions of his own precious children. So they're all still there, and > Larry is not. > > Micro-managing isn't that third thing that Amazon does better than us, > by the way. I mean, yeah, they micro-manage really well, but I wouldn't > list it as a strength or anything. I'm just trying to set the context > here, to help you understand what happened. We're talking about a guy > who in all seriousness has said on many public occasions that people > should be paying him to work at Amazon. He hands out little yellow > stickies with his name on them, reminding people "who runs the company" > when they disagree with him. The guy is a regular... well, Steve Jobs, I > guess. Except without the fashion or design sense. Bezos is super smart; > don't get me wrong. He just makes ordinary control freaks look like > stoned hippies. > > So one day Jeff Bezos issued a mandate. He's doing that all the time, of > course, and people scramble like ants being pounded with a rubber mallet > whenever it happens. But on one occasion -- back around 2002 I think, > plus or minus a year -- he issued a mandate that was so out there, so > huge and eye-bulgingly ponderous, that it made all of his other mandates > look like unsolicited peer bonuses. > > His Big Mandate went something along these lines: > > 1) All teams will henceforth expose their data and functionality through > service interfaces. > > 2) Teams must communicate with each other through these interfaces. > > 3) There will be no other form of interprocess communication allowed: no > direct linking, no direct reads of another team's data store, no > shared-memory model, no back-doors whatsoever. The only communication > allowed is via service interface calls over the network. > > 4) It doesn't matter what technology they use. HTTP, Corba, Pubsub, > custom protocols -- doesn't matter. Bezos doesn't care. > > 5) All service interfaces, without exception, must be designed from the > ground up to be externalizable. That is to say, the team must plan and > design to be able to expose the interface to developers in the outside > world. No exceptions. > > 6) Anyone who doesn't do this will be fired. > > 7) Thank you; have a nice day! > > Ha, ha! You 150-odd ex-Amazon folks here will of course realize > immediately that #7 was a little joke I threw in, because Bezos most > definitely does not give a shit about your day. > > #6, however, was quite real, so people went to work. Bezos assigned a > couple of Chief Bulldogs to oversee the effort and ensure forward > progress, headed up by Uber-Chief Bear Bulldog Rick Dalzell. Rick is an > ex-Armgy Ranger, West Point Academy graduate, ex-boxer, ex-Chief > Torturer slash CIO at Wal*Mart, and is a big genial scary man who used > the word "hardened interface" a lot. Rick was a walking, talking > hardened interface himself, so needless to say, everyone made LOTS of > forward progress and made sure Rick knew about it. > > Over the next couple of years, Amazon transformed internally into a > service-oriented architecture. They learned a tremendous amount while > effecting this transformation. There was lots of existing documentation > and lore about SOAs, but at Amazon's vast scale it was about as useful > as telling Indiana Jones to look both ways before crossing the street. > Amazon's dev staff made a lot of discoveries along the way. A teeny tiny > sampling of these discoveries included: > > - pager escalation gets way harder, because a ticket might bounce > through 20 service calls before the real owner is identified. If each > bounce goes through a team with a 15-minute response time, it can be > hours before the right team finally finds out, unless you build a lot of > scaffolding and metrics and reporting. > > - every single one of your peer teams suddenly becomes a potential DOS > attacker. Nobody can make any real forward progress until very serious > quotas and throttling are put in place in every single service. > > - monitoring and QA are the same thing. You'd never think so until you > try doing a big SOA. But when your service says "oh yes, I'm fine", it > may well be the case that the only thing still functioning in the server > is the little component that knows how to say "I'm fine, roger roger, > over and out" in a cheery droid voice. In order to tell whether the > service is actually responding, you have to make individual calls. The > problem continues recursively until your monitoring is doing > comprehensive semantics checking of your entire range of services and > data, at which point it's indistinguishable from automated QA. So > they're a continuum. > > - if you have hundreds of services, and your code MUST communicate with > other groups' code via these services, then you won't be able to find > any of them without a service-discovery mechanism. And you can't have > that without a service registration mechanism, which itself is another > service. So Amazon has a universal service registry where you can find > out reflectively (programmatically) about every service, what its APIs > are, and also whether it is currently up, and where. > > - debugging problems with someone else's code gets a LOT harder, and is > basically impossible unless there is a universal standard way to run > every service in a debuggable sandbox. > > That's just a very small sample. There are dozens, maybe hundreds of > individual learnings like these that Amazon had to discover organically. > There were a lot of wacky ones around externalizing services, but not as > many as you might think. Organizing into services taught teams not to > trust each other in most of the same ways they're not supposed to trust > external developers. > > This effort was still underway when I left to join Google in mid-2005, > but it was pretty far advanced. From the time Bezos issued his edict > through the time I left, Amazon had transformed culturally into a > company that thinks about everything in a services-first fashion. It is > now fundamental to how they approach all designs, including internal > designs for stuff that might never see the light of day externally. > > At this point they don't even do it out of fear of being fired. I mean, > they're still afraid of that; it's pretty much part of daily life there, > working for the Dread Pirate Bezos and all. But they do services because > they've come to understand that it's the Right Thing. There are without > question pros and cons to the SOA approach, and some of the cons are > pretty long. But overall it's the right thing because SOA-driven design > enables Platforms. > > That's what Bezos was up to with his edict, of course. He didn't (and > doesn't) care even a tiny bit about the well-being of the teams, nor > about what technologies they use, nor in fact any detail whatsoever > about how they go about their business unless they happen to be screwing > up. But Bezos realized long before the vast majority of Amazonians that > Amazon needs to be a platform. > > You wouldn't really think that an online bookstore needs to be an > extensible, programmable platform. Would you? > > Well, the first big thing Bezos realized is that the infrastructure > they'd built for selling and shipping books and sundry could be > transformed an excellent repurposable computing platform. So now they > have the Amazon Elastic Compute Cloud, and the Amazon Elastic MapReduce, > and the Amazon Relational Database Service, and a whole passel' o' other > services browsable at aws.amazon.com. These services host the backends > for some pretty successful companies, reddit being my personal favorite > of the bunch. > > The other big realization he had was that he can't always build the > right thing. I think Larry Tesler might have struck some kind of chord > in Bezos when he said his mom couldn't use the goddamn website. It's not > even super clear whose mom he was talking about, and doesn't really > matter, because nobody's mom can use the goddamn website. In fact I > myself find the website disturbingly daunting, and I worked there for > over half a decade. I've just learned to kinda defocus my eyes and > concentrate on the million or so pixels near the center of the page > above the fold. > > I'm not really sure how Bezos came to this realization -- the insight > that he can't build one product and have it be right for everyone. But > it doesn't matter, because he gets it. There's actually a formal name > for this phenomenon. It's called Accessibility, and it's the most > important thing in the computing world. > > The. Most. Important. Thing. > > If you're sorta thinking, "huh? You mean like, blind and deaf people > Accessibility?" then you're not alone, because I've come to understand > that there are lots and LOTS of people just like you: people for whom > this idea does not have the right Accessibility, so it hasn't been able > to get through to you yet. It's not your fault for not understanding, > any more than it would be your fault for being blind or deaf or > motion-restricted or living with any other disability. When software -- > or idea-ware for that matter -- fails to be accessible to anyone for any > reason, it is the fault of the software or of the messaging of the idea. > It is an Accessibility failure. > > Like anything else big and important in life, Accessibility has an evil > twin who, jilted by the unbalanced affection displayed by their parents > in their youth, has grown into an equally powerful Arch-Nemesis (yes, > there's more than one nemesis to accessibility) named Security. And boy > howdy are the two ever at odds. > > But I'll argue that Accessibility is actually more important than > Security because dialing Accessibility to zero means you have no product > at all, whereas dialing Security to zero can still get you a reasonably > successful product such as the Playstation Network. > > So yeah. In case you hadn't noticed, I could actually write a book on > this topic. A fat one, filled with amusing anecdotes about ants and > rubber mallets at companies I've worked at. But I will never get this > little rant published, and you'll never get it read, unless I start to > wrap up. > > That one last thing that Google doesn't do well is Platforms. We don't > understand platforms. We don't "get" platforms. Some of you do, but you > are the minority. This has become painfully clear to me over the past > six years. I was kind of hoping that competitive pressure from Microsoft > and Amazon and more recently Facebook would make us wake up collectively > and start doing universal services. Not in some sort of ad-hoc, > half-assed way, but in more or less the same way Amazon did it: all at > once, for real, no cheating, and treating it as our top priority from > now on. > > But no. No, it's like our tenth or eleventh priority. Or fifteenth, I > don't know. It's pretty low. There are a few teams who treat the idea > very seriously, but most teams either don't think about it all, ever, or > only a small percentage of them think about it in a very small way. > > It's a big stretch even to get most teams to offer a stubby service to > get programmatic access to their data and computations. Most of them > think they're building products. And a stubby service is a pretty > pathetic service. Go back and look at that partial list of learnings > from Amazon, and tell me which ones Stubby gives you out of the box. As > far as I'm concerned, it's none of them. Stubby's great, but it's like > parts when you need a car. > > A product is useless without a platform, or more precisely and > accurately, a platform-less product will always be replaced by an > equivalent platform-ized product. > > Google+ is a prime example of our complete failure to understand > platforms from the very highest levels of executive leadership (hi > Larry, Sergey, Eric, Vic, howdy howdy) down to the very lowest leaf > workers (hey yo). We all don't get it. The Golden Rule of platforms is > that you Eat Your Own Dogfood. The Google+ platform is a pathetic > afterthought. We had no API at all at launch, and last I checked, we had > one measly API call. One of the team members marched in and told me > about it when they launched, and I asked: "So is it the Stalker API?" > She got all glum and said "Yeah." I mean, I was joking, but no... the > only API call we offer is to get someone's stream. So I guess the joke > was on me. > > Microsoft has known about the Dogfood rule for at least twenty years. > It's been part of their culture for a whole generation now. You don't > eat People Food and give your developers Dog Food. Doing that is simply > robbing your long-term platform value for short-term successes. > Platforms are all about long-term thinking. > > Google+ is a knee-jerk reaction, a study in short-term thinking, > predicated on the incorrect notion that Facebook is successful because > they built a great product. But that's not why they are successful. > Facebook is successful because they built an entire constellation of > products by allowing other people to do the work. So Facebook is > different for everyone. Some people spend all their time on Mafia Wars. > Some spend all their time on Farmville. There are hundreds or maybe > thousands of different high-quality time sinks available, so there's > something there for everyone. > > Our Google+ team took a look at the aftermarket and said: "Gosh, it > looks like we need some games. Let's go contract someone to, um, write > some games for us." Do you begin to see how incredibly wrong that > thinking is now? The problem is that we are trying to predict what > people want and deliver it for them. > > You can't do that. Not really. Not reliably. There have been precious > few people in the world, over the entire history of computing, who have > been able to do it reliably. Steve Jobs was one of them. We don't have a > Steve Jobs here. I'm sorry, but we don't. > > Larry Tesler may have convinced Bezos that he was no Steve Jobs, but > Bezos realized that he didn't need to be a Steve Jobs in order to > provide everyone with the right products: interfaces and workflows that > they liked and felt at ease with. He just needed to enable third-party > developers to do it, and it would happen automatically. > > I apologize to those (many) of you for whom all this stuff I'm saying is > incredibly obvious, because yeah. It's incredibly frigging obvious. > Except we're not doing it. We don't get Platforms, and we don't get > Accessibility. The two are basically the same thing, because platforms > solve accessibility. A platform is accessibility. > > So yeah, Microsoft gets it. And you know as well as I do how surprising > that is, because they don't "get" much of anything, really. But they > understand platforms as a purely accidental outgrowth of having started > life in the business of providing platforms. So they have thirty-plus > years of learning in this space. And if you go to msdn.com, and spend > some time browsing, and you've never seen it before, prepare to be > amazed. Because it's staggeringly huge. They have thousands, and > thousands, and THOUSANDS of API calls. They have a HUGE platform. Too > big in fact, because they can't design for squat, but at least they're > doing it. > > Amazon gets it. Amazon's AWS (aws.amazon.com) is incredible. Just go > look at it. Click around. It's embarrassing. We don't have any of that > stuff. > > Apple gets it, obviously. They've made some fundamentally non-open > choices, particularly around their mobile platform. But they understand > accessibility and they understand the power of third-party development > and they eat their dogfood. And you know what? They make pretty good > dogfood. Their APIs are a hell of a lot cleaner than Microsoft's, and > have been since time immemorial. > > Facebook gets it. That's what really worries me. That's what got me off > my lazy butt to write this thing. I hate blogging. I hate... plussing, > or whatever it's called when you do a massive rant in Google+ even > though it's a terrible venue for it but you do it anyway because in the > end you really do want Google to be successful. And I do! I mean, > Facebook wants me there, and it'd be pretty easy to just go. But Google > is home, so I'm insisting that we have this little family intervention, > uncomfortable as it might be. > > After you've marveled at the platform offerings of Microsoft and Amazon, > and Facebook I guess (I didn't look because I didn't want to get too > depressed), head over to developers.google.com and browse a little. > Pretty big difference, eh? It's like what your fifth-grade nephew might > mock up if he were doing an assignment to demonstrate what a big > powerful platform company might be building if all they had, > resource-wise, was one fifth grader. > > Please don't get me wrong here -- I know for a fact that the dev-rel > team has had to FIGHT to get even this much available externally. > They're kicking ass as far as I'm concerned, because they DO get > platforms, and they are struggling heroically to try to create one in an > environment that is at best platform-apathetic, and at worst often > openly hostile to the idea. > > I'm just frankly describing what developers.google.com looks like to an > outsider. It looks childish. Where's the Maps APIs in there for Christ's > sake? Some of the things in there are labs projects. And the APIs for > everything I clicked were... they were paltry. They were obviously dog > food. Not even good organic stuff. Compared to our internal APIs it's > all snouts and horse hooves. > > And also don't get me wrong about Google+. They're far from the only > offenders. This is a cultural thing. What we have going on internally is > basically a war, with the underdog minority Platformers fighting a more > or less losing battle against the Mighty Funded Confident Producters. > > Any teams that have successfully internalized the notion that they > should be externally programmable platforms from the ground up are > underdogs -- Maps and Docs come to mind, and I know GMail is making > overtures in that direction. But it's hard for them to get funding for > it because it's not part of our culture. Maestro's funding is a feeble > thing compared to the gargantuan Microsoft Office programming platform: > it's a fluffy rabbit versus a T-Rex. The Docs team knows they'll never > be competitive with Office until they can match its scripting > facilities, but they're not getting any resource love. I mean, I assume > they're not, given that Apps Script only works in Spreadsheet right now, > and it doesn't even have keyboard shortcuts as part of its API. That > team looks pretty unloved to me. > > Ironically enough, Wave was a great platform, may they rest in peace. > But making something a platform is not going to make you an instant > success. A platform needs a killer app. Facebook -- that is, the stock > service they offer with walls and friends and such -- is the killer app > for the Facebook Platform. And it is a very serious mistake to conclude > that the Facebook App could have been anywhere near as successful > without the Facebook Platform. > > You know how people are always saying Google is arrogant? I'm a Googler, > so I get as irritated as you do when people say that. We're not > arrogant, by and large. We're, like, 99% Arrogance-Free. I did start > this post -- if you'll reach back into distant memory -- by describing > Google as "doing everything right". We do mean well, and for the most > part when people say we're arrogant it's because we didn't hire them, or > they're unhappy with our policies, or something along those lines. > They're inferring arrogance because it makes them feel better. > > But when we take the stance that we know how to design the perfect > product for everyone, and believe you me, I hear that a lot, then we're > being fools. You can attribute it to arrogance, or naivete, or whatever > -- it doesn't matter in the end, because it's foolishness. There IS no > perfect product for everyone. > > And so we wind up with a browser that doesn't let you set the default > font size. Talk about an affront to Accessibility. I mean, as I get > older I'm actually going blind. For real. I've been nearsighted all my > life, and once you hit 40 years old you stop being able to see things up > close. So font selection becomes this life-or-death thing: it can lock > you out of the product completely. But the Chrome team is flat-out > arrogant here: they want to build a zero-configuration product, and > they're quite brazen about it, and Fuck You if you're blind or deaf or > whatever. Hit Ctrl-+ on every single page visit for the rest of your life. > > It's not just them. It's everyone. The problem is that we're a Product > Company through and through. We built a successful product with broad > appeal -- our search, that is -- and that wild success has biased us. > > Amazon was a product company too, so it took an out-of-band force to > make Bezos understand the need for a platform. That force was their > evaporating margins; he was cornered and had to think of a way out. But > all he had was a bunch of engineers and all these computers... if only > they could be monetized somehow... you can see how he arrived at AWS, in > hindsight. > > Microsoft started out as a platform, so they've just had lots of > practice at it. > > Facebook, though: they worry me. I'm no expert, but I'm pretty sure they > started off as a Product and they rode that success pretty far. So I'm > not sure exactly how they made the transition to a platform. It was a > relatively long time ago, since they had to be a platform before (now > very old) things like Mafia Wars could come along. > > Maybe they just looked at us and asked: "How can we beat Google? What > are they missing?" > > The problem we face is pretty huge, because it will take a dramatic > cultural change in order for us to start catching up. We don't do > internal service-oriented platforms, and we just as equally don't do > external ones. This means that the "not getting it" is endemic across > the company: the PMs don't get it, the engineers don't get it, the > product teams don't get it, nobody gets it. Even if individuals do, even > if YOU do, it doesn't matter one bit unless we're treating it as an > all-hands-on-deck emergency. We can't keep launching products and > pretending we'll turn them into magical beautiful extensible platforms > later. We've tried that and it's not working. > > The Golden Rule of Platforms, "Eat Your Own Dogfood", can be rephrased > as "Start with a Platform, and Then Use it for Everything." You can't > just bolt it on later. Certainly not easily at any rate -- ask anyone > who worked on platformizing MS Office. Or anyone who worked on > platformizing Amazon. If you delay it, it'll be ten times as much work > as just doing it correctly up front. You can't cheat. You can't have > secret back doors for internal apps to get special priority access, not > for ANY reason. You need to solve the hard problems up front. > > I'm not saying it's too late for us, but the longer we wait, the closer > we get to being Too Late. > > I honestly don't know how to wrap this up. I've said pretty much > everything I came here to say today. This post has been six years in the > making. I'm sorry if I wasn't gentle enough, or if I misrepresented some > product or team or person, or if we're actually doing LOTS of platform > stuff and it just so happens that I and everyone I ever talk to has just > never heard about it. I'm sorry. > > But we've gotta start doing this right. > >
