Hey Juhani, The current state of client retries/sleep is something that needs to be reviewed/redone. It's currently on the roadmap for 0.92, see https://issues.apache.org/jira/browse/HBASE-2445
Regarding what you can do right now, the sleeps are done using an exponential backoff meaning that the more it sleeps, the longer it does. Enabling DEBUG in your client should give you more details, but according to what I see in the logs setting the retries lower should definitely make it fail faster. Now, I see that in your log the split took 55 seconds to complete. I've recently been working on a major deficiency around that part of the code. I posted a patch that's ready for commit here https://issues.apache.org/jira/browse/HBASE-3308 I believe that in your case the problem is amplified by the fact that you are using a ton of families. HBase has some performance issues managing them, but also in general that many families is probably a bad design. 99.99% of the time, I see no reason to use more than 1 family. Try it for yourself and you should see a big improvement across the board. Regarding region size, if you have decent hardware then it's safe to set MAX_FILESIZE to 1GB on the tables. Also you could just create the table pre-split and be done with it, for example see http://hbase.apache.org/docs/r0.89.20100924/apidocs/org/apache/hadoop/hbase/client/HBaseAdmin.html#createTable(org.apache.hadoop.hbase.HTableDescriptor, byte[], byte[], int) J-D On Wed, Dec 8, 2010 at 12:56 AM, Juhani Connolly <[email protected]> wrote: > Hi there, > > We're currently running a cluster under expected load, and testing various > hardware failure cases. Among them is a lost regionServer/dataNode, which > results in our writer process(in our case a servlet under tomcat) just > waiting indefinitely on put flushes until the region becomes available > again(in the process the threads stack up until the server limit). I've > included logs of the relevant time period from one of my regionservers at > http://pastie.org/1358217 . > > During the 15minutes from around 16:12->16:27 all writes failed. > Incidentally, during this time I am still able to read data fine with > another process which is only reading from hbase. > > Is this period of not being available to write to for 15 working as > intended, or is something wrong with the way I'm trying to access hbase? The > main access code I'm using can be seen at http://pastie.org/1358224 . tPool > is an initialised HTablePool, and the general idea is to store puts without > flushing until they have been held onto for a while(to batch the flushes a > little bit) > > If it is working as intended, what would be the correct steps to reduce > it(perhaps reducing configuration for region sizes)? > > Is there anything I can do to just make the writes fail when the region > isn't available for writing? As is, threads keep getting generated till the > container max is reached, waiting for something(presumably the region to > become available again?). I expected that hbase.client.retries.number would > be appropriate, but based on the lack of any logs for failed writes, the > current writes simply aren't aborting. > > Everything is running off the latest CDH3(hbase-0.89.20100924+28, > hadoop-0.20.2+737-core) and works well under normal conditions > > Any advice/information would be appreciated. > Thanks, > Juhani > > > >
