Jonathan Gray wrote:
Can you provide more information about the hardware on your nodes?
Sure. The cluster is 5 nodes, each with 4GB of ram, 2 cores (opteron 250) and 2 x 1TB drives.
I think I saw you had 13 mappers running? And only 512MB of heap for the regionserver? That is a very small amount of heap for HBase to run.
The HBASE_HEAPSIZE is set to 3000. Checking the HADOOP_HEAPSIZE, I see it's set to 2000 - so before I run any M/R jobs I guess I have some problems here. I inherited the config so I didn't check this before now (mea culpa) - I guess this illustrates why the data nodes might have been dropping out anyways.
The 512MB heap was allocated to the mapred tasks - and I was experimenting with different numbers of mappers and reducers in an effort to get the Maholo backup job running. Initial efforts with 2+1 didn't get anywhere, while bumping it to 13+5 seemed to allow it to progress a lot further. But obviously caused other problems.
In terms if proper heap sizing for hadoop, hbase, mapreduce - should I avoid swapping at all costs? Whats a good proportion of memory between hbase heap and hadoop heap (I haven't seen this documented anywhere, if it is - feel free to hit me with the rtfm stick :)
How many cores do you have and what are your disks like? 13 mappers per node is WAY too high. If you have 4 core nodes and you are running the regionserver with the datanode and also trying to get MR tasks on the same nodes, you should probably not go over 2 concurrent mappers per node.
Thanks for this - as I said, I was playing around with the number of mappers in an effort to see if the backup job would proceed any better - it sounds like this is a bad idea. Thanks and noted.
As far as robustness of HBase w.r.t. missing blocks, there's not much HBase can do. HBase uses HDFS as it's persistent storage. If the blocks are not available, then your data is simply not available. This would be the same for any database on any filesystem, if the filesystem says the file doesn't exist, the database can't do much.
Sure. This makes sense. I guess I'm wondering, in the event that I have lost some blocks, is my entire HBase hosed or can it tolerate the loss and remove the corrupt rows from the table(s) or do I need to repopulate my data again (I plan to do this in the longer term anyway, but wondering do I need to accelerate that plan).
It does seem that your issues are from too much load on insufficient resources. And I would not expect 0.20 to behave better in that respect, as it now uses the CMS garbage collector and is a bit more resource hungry than it's predecessors (more resources, but far better performance).
Thanks also for this clarification - it sounds like 0.20 is still worth pursuing for the other reason, but as you note, it's not going to expand my capacity.
Many thanks for the detailed reply, lots to reflect on. -stephen -- Stephen Mulcahy, DI2, Digital Enterprise Research Institute, NUI Galway, IDA Business Park, Lower Dangan, Galway, Ireland http://di2.deri.ie http://webstar.deri.ie http://sindice.com
