That WaffleGrid stuff looks pretty interesting too. Not a real
cluster, but it would allow the InnoDB tablespace to live in RAM (even
if it's on another server). I haven't had a chance to test it, but it
looks really promising.
Tim S.
On Mar 2, 2009, at 3:20 PM, Jeremy Zawodny wrote:
Eric and Jim, you're both right I think. :-)
I'm working on our sharding project at Criagslist now and we get to
decide
what our sweet spot is going to be. Do we end up with 4 clusters
that can
keep 30% of their InnoDB data in RAM today? Or do we go with 16
clusters
that have room for everything in RAM and a lot more headroom for the
future?
If so, how many disks do we bother with? What's the overhead of
dealing
with all these boxes/blades? And at what point does the number of
square
feet used in the datacenter become more important than power
consumption
(and switch ports, etc)?
I drool at the possibility of putting more than 15-20% of our InnoDB
data
into RAM at a given time. RAM/Network is definitely going to blow
away disk
performance for the forseeable future, so we might as well aim for
that.
But we need to understand what other factors we have to opitmize
around.
Jeremy
On Mon, Mar 2, 2009 at 12:47 PM, Jim Starkey <[email protected]>
wrote:
Eric Bergen wrote:
I think most of the focus on cloud computing is based around raw
processing power. They forget that all that data needs to be stored
somewhere at some point. I think the challenge for drizzle workloads
will be finding the correct balance of cpu, memory, i/o latency and
power consumption. Most of these cloud computing discussions forget
all about i/o latency. We can't.
May I suggest that it's time to re-examine your assumptions. From
the
beginning of time (i.e. ~1970), database systems were things that
ran on
single computers, sitting between the disk and the database
client. Since
then processors have increased over a thousandfold in power, memory
has gone
from a megabyte or two to 8 gigabytes, LANs to a gigabit, and disks
almost
almost four times faster than they used to be.
Think network centric. Sun didn't get much right, but "the network
is the
computer" was dead-on. The cost to fetch something from another
system is a
small fraction of the cost to get something from disk. If you're
concerned
about latency -- and you should be -- you want to pick something
off the
network, not off a disk.
The design center should be an elastic set of expendable computers,
not a
single computer. It is also cheaper to add another computer than
to make an
existing one faster (as if that were even possible without taking
it down
and replacing it). This requires a change in thinking and a
corresponding
change in architectures.
I know the inherited code base needs vast amounts of work to be
maintainable, but wouldn't it make a lot of sense to think about
alternative
architectures while that process continues? For example, would
Oracle RAC
is a suitable model? If so, what might drizzle be doing
differently now to
accommodate that later? What are the requirements, goals, trade-
offs, and
non-goals?
Justing think about the number of network transfers you can while
waiting
for the silly disk to turn and the arm to move should stimulate the
imagination...
(No, SSDs are the answer. They raise, but don't eliminate, the bar.)
--
Jim Starkey
President, NimbusDB, Inc.
978 526-1376
_______________________________________________
Mailing list: https://launchpad.net/~drizzle-discuss<https://launchpad.net/%7Edrizzle-discuss
>
Post to : [email protected]
Unsubscribe : https://launchpad.net/~drizzle-discuss<https://launchpad.net/%7Edrizzle-discuss
>
More help : https://help.launchpad.net/ListHelp
_______________________________________________
Mailing list: https://launchpad.net/~drizzle-discuss
Post to : [email protected]
Unsubscribe : https://launchpad.net/~drizzle-discuss
More help : https://help.launchpad.net/ListHelp
_______________________________________________
Mailing list: https://launchpad.net/~drizzle-discuss
Post to : [email protected]
Unsubscribe : https://launchpad.net/~drizzle-discuss
More help : https://help.launchpad.net/ListHelp
