So each row = 24 column-families(?) * 300,000,000 entries/family * ~40
bytes/entry = about 270GB/row ?
And that * 100,000 rows = about 27 petabytes of data?
Is my math right here? :)
With a big enough cluster, you might be able to get that amount of data in
hadoop. I'm not sure anyone has had an HBase installation that big.
One thing that is definitely not going to work with HBase is having single
rows that are many GBs.
A row can never be split across regions, and the default region size is
256MB (though configurable), so you'd be 3 orders of magnitude greater than
the recommended maximum. So to directly answer your questions, one
limitation is the size of a single row. The other limitation is the number
of regions that can be handled on each node. The upper limits are in the
400-500 region / region-server range though this can vary depending on your
hardware and usage patterns. That's about 100GB on an HBase node, so if you
were to get this much data into HBase you'd need several hundreds of
One thing you'd definitely need to do is rework your schema a bit, spreading
things across more rows so you can have reasonably sized regions.
My short answer would be that this is not currently possible in HBase unless
you had a very very large cluster and a bit of time to work out some bugs
that I'm sure will pop up with an installation of this size. My question to
you is, do you really need random access of this granularity to 27 petabytes
From: Alex Newman [mailto:[EMAIL PROTECTED]
Sent: Wednesday, September 24, 2008 6:21 AM
Subject: Scalability of HBase
Where are the scalability limitations with hbase. Number of tablets? The
size of the columns? I am thinking about
But with on the order 300M entries per column with something like
(timestamp,<32 byte string>)
would something like this scale.