Thanks for the information, it should help.
Regarding using a lot of families... They are currently partitioned in a
manner that reflects the various data groups that are likely to be read
together... We're doing a lot of big scans on the regions of only one of
those families, with scans of the full table being much shorter/rarer.
By having separate store files I was hoping this separation would result
in less overhead from not reading data that we simply don't need(stuff
from the other families). Is the overhead from splitting the store files
up large enough to make any savings on file access times not worth it?
Or am I missing something else?
Thanks,
Juhani
On 12/09/2010 03:04 AM, Jean-Daniel Cryans wrote:
Hey Juhani,
The current state of client retries/sleep is something that needs to
be reviewed/redone. It's currently on the roadmap for 0.92, see
https://issues.apache.org/jira/browse/HBASE-2445
Regarding what you can do right now, the sleeps are done using an
exponential backoff meaning that the more it sleeps, the longer it
does. Enabling DEBUG in your client should give you more details, but
according to what I see in the logs setting the retries lower should
definitely make it fail faster.
Now, I see that in your log the split took 55 seconds to complete.
I've recently been working on a major deficiency around that part of
the code. I posted a patch that's ready for commit here
https://issues.apache.org/jira/browse/HBASE-3308
I believe that in your case the problem is amplified by the fact that
you are using a ton of families. HBase has some performance issues
managing them, but also in general that many families is probably a
bad design. 99.99% of the time, I see no reason to use more than 1
family. Try it for yourself and you should see a big improvement
across the board.
Regarding region size, if you have decent hardware then it's safe to
set MAX_FILESIZE to 1GB on the tables. Also you could just create the
table pre-split and be done with it, for example see
http://hbase.apache.org/docs/r0.89.20100924/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#createTable(org.apache.hadoop.hbase.HTableDescriptor,
byte[], byte[], int)
J-D
On Wed, Dec 8, 2010 at 12:56 AM, Juhani Connolly<[email protected]> wrote:
Hi there,
We're currently running a cluster under expected load, and testing various
hardware failure cases. Among them is a lost regionServer/dataNode, which
results in our writer process(in our case a servlet under tomcat) just
waiting indefinitely on put flushes until the region becomes available
again(in the process the threads stack up until the server limit). I've
included logs of the relevant time period from one of my regionservers at
http://pastie.org/1358217 .
During the 15minutes from around 16:12->16:27 all writes failed.
Incidentally, during this time I am still able to read data fine with
another process which is only reading from hbase.
Is this period of not being available to write to for 15 working as
intended, or is something wrong with the way I'm trying to access hbase? The
main access code I'm using can be seen at http://pastie.org/1358224 . tPool
is an initialised HTablePool, and the general idea is to store puts without
flushing until they have been held onto for a while(to batch the flushes a
little bit)
If it is working as intended, what would be the correct steps to reduce
it(perhaps reducing configuration for region sizes)?
Is there anything I can do to just make the writes fail when the region
isn't available for writing? As is, threads keep getting generated till the
container max is reached, waiting for something(presumably the region to
become available again?). I expected that hbase.client.retries.number would
be appropriate, but based on the lack of any logs for failed writes, the
current writes simply aren't aborting.
Everything is running off the latest CDH3(hbase-0.89.20100924+28,
hadoop-0.20.2+737-core) and works well under normal conditions
Any advice/information would be appreciated.
Thanks,
Juhani
