Eric and Jim, you're both right I think. :-) I'm working on our sharding project at Criagslist now and we get to decide what our sweet spot is going to be. Do we end up with 4 clusters that can keep 30% of their InnoDB data in RAM today? Or do we go with 16 clusters that have room for everything in RAM and a lot more headroom for the future?
If so, how many disks do we bother with? What's the overhead of dealing with all these boxes/blades? And at what point does the number of square feet used in the datacenter become more important than power consumption (and switch ports, etc)? I drool at the possibility of putting more than 15-20% of our InnoDB data into RAM at a given time. RAM/Network is definitely going to blow away disk performance for the forseeable future, so we might as well aim for that. But we need to understand what other factors we have to opitmize around. Jeremy On Mon, Mar 2, 2009 at 12:47 PM, Jim Starkey <[email protected]> wrote: > Eric Bergen wrote: > >> I think most of the focus on cloud computing is based around raw >> processing power. They forget that all that data needs to be stored >> somewhere at some point. I think the challenge for drizzle workloads >> will be finding the correct balance of cpu, memory, i/o latency and >> power consumption. Most of these cloud computing discussions forget >> all about i/o latency. We can't. >> >> >> > May I suggest that it's time to re-examine your assumptions. From the > beginning of time (i.e. ~1970), database systems were things that ran on > single computers, sitting between the disk and the database client. Since > then processors have increased over a thousandfold in power, memory has gone > from a megabyte or two to 8 gigabytes, LANs to a gigabit, and disks almost > almost four times faster than they used to be. > > Think network centric. Sun didn't get much right, but "the network is the > computer" was dead-on. The cost to fetch something from another system is a > small fraction of the cost to get something from disk. If you're concerned > about latency -- and you should be -- you want to pick something off the > network, not off a disk. > > The design center should be an elastic set of expendable computers, not a > single computer. It is also cheaper to add another computer than to make an > existing one faster (as if that were even possible without taking it down > and replacing it). This requires a change in thinking and a corresponding > change in architectures. > > I know the inherited code base needs vast amounts of work to be > maintainable, but wouldn't it make a lot of sense to think about alternative > architectures while that process continues? For example, would Oracle RAC > is a suitable model? If so, what might drizzle be doing differently now to > accommodate that later? What are the requirements, goals, trade-offs, and > non-goals? > > Justing think about the number of network transfers you can while waiting > for the silly disk to turn and the arm to move should stimulate the > imagination... > > (No, SSDs are the answer. They raise, but don't eliminate, the bar.) > > > > -- > Jim Starkey > President, NimbusDB, Inc. > 978 526-1376 > > > _______________________________________________ > Mailing list: > https://launchpad.net/~drizzle-discuss<https://launchpad.net/%7Edrizzle-discuss> > Post to : [email protected] > Unsubscribe : > https://launchpad.net/~drizzle-discuss<https://launchpad.net/%7Edrizzle-discuss> > More help : https://help.launchpad.net/ListHelp >
_______________________________________________ Mailing list: https://launchpad.net/~drizzle-discuss Post to : [email protected] Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
