Jeremy Zawodny wrote:
Eric and Jim, you're both right I think. :-)
I'm working on our sharding project at Criagslist now and we get to
decide what our sweet spot is going to be. Do we end up with 4
clusters that can keep 30% of their InnoDB data in RAM today? Or do
we go with 16 clusters that have room for everything in RAM and a lot
more headroom for the future?
If so, how many disks do we bother with? What's the overhead of
dealing with all these boxes/blades? And at what point does the
number of square feet used in the datacenter become more important
than power consumption (and switch ports, etc)?
You might dig up what Hamilton has to say about blades. He argues that
increase density at a relatively high cost, but floor space is cheapest
part of a data center (after servers, cooling infrastructure, power
infrastructure).
I drool at the possibility of putting more than 15-20% of our InnoDB
data into RAM at a given time. RAM/Network is definitely going to
blow away disk performance for the forseeable future, so we might as
well aim for that. But we need to understand what other factors we
have to opitmize around.
Jeremy
On Mon, Mar 2, 2009 at 12:47 PM, Jim Starkey <[email protected]
<mailto:[email protected]>> wrote:
Eric Bergen wrote:
I think most of the focus on cloud computing is based around raw
processing power. They forget that all that data needs to be
stored
somewhere at some point. I think the challenge for drizzle
workloads
will be finding the correct balance of cpu, memory, i/o
latency and
power consumption. Most of these cloud computing discussions
forget
all about i/o latency. We can't.
May I suggest that it's time to re-examine your assumptions. From
the beginning of time (i.e. ~1970), database systems were things
that ran on single computers, sitting between the disk and the
database client. Since then processors have increased over a
thousandfold in power, memory has gone from a megabyte or two to 8
gigabytes, LANs to a gigabit, and disks almost almost four times
faster than they used to be.
Think network centric. Sun didn't get much right, but "the
network is the computer" was dead-on. The cost to fetch something
from another system is a small fraction of the cost to get
something from disk. If you're concerned about latency -- and you
should be -- you want to pick something off the network, not off a
disk.
The design center should be an elastic set of expendable
computers, not a single computer. It is also cheaper to add
another computer than to make an existing one faster (as if that
were even possible without taking it down and replacing it).
This requires a change in thinking and a corresponding change in
architectures.
I know the inherited code base needs vast amounts of work to be
maintainable, but wouldn't it make a lot of sense to think about
alternative architectures while that process continues? For
example, would Oracle RAC is a suitable model? If so, what might
drizzle be doing differently now to accommodate that later? What
are the requirements, goals, trade-offs, and non-goals?
Justing think about the number of network transfers you can while
waiting for the silly disk to turn and the arm to move should
stimulate the imagination...
(No, SSDs are the answer. They raise, but don't eliminate, the bar.)
--
Jim Starkey
President, NimbusDB, Inc.
978 526-1376
_______________________________________________
Mailing list: https://launchpad.net/~drizzle-discuss
<https://launchpad.net/%7Edrizzle-discuss>
Post to : [email protected]
<mailto:[email protected]>
Unsubscribe : https://launchpad.net/~drizzle-discuss
<https://launchpad.net/%7Edrizzle-discuss>
More help : https://help.launchpad.net/ListHelp
--
Jim Starkey
President, NimbusDB, Inc.
978 526-1376
_______________________________________________
Mailing list: https://launchpad.net/~drizzle-discuss
Post to : [email protected]
Unsubscribe : https://launchpad.net/~drizzle-discuss
More help : https://help.launchpad.net/ListHelp
