we've been trying for a couple of days (without success) to import our data into
hbase.
initially we ran into quite a few OOME errors, but we've seem to overcome that
by adjusting our jvm memory heap sizes.
however, we're still running into many other roadblocks, and in my opinion we
just don't have the right configuration options in our conf files (or maybe not
enough resources to get the job done)
in the end, this is a one time task. if we can be successful, we believe this
will be a good introduction for us using hbase, and we can continue by
integrating it further into our project.
i'ld appreciate if someone would offer us some advice for the following
task/setup that we are trying to accomplish. here are the details:
---------------------------------------------------------------
1) everything (hdfs/hbase) is running on one machine currently (short term)
2) we are importing 60k files, each ranging between 100k and 64MB along with
necessary meta-data in other column-families
3) our machine has 2GB, amd64 dual core, dedicated to the import task, hbase
heapsize is set to 1000
4) our import program is single threaded java program, iterating through our
files and doing batchoperations for each file into hbase
5) after about 15 minutes of successful importing, we see
'INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for 'IPC
Server handler 8 on 60020' on region dmls,,1216768730386: Memcache size 64.0m is
>= than blocking 64.0m size '
after another 15 minutes of inactivity, we see the threads slowly get unblocked,
and importing continues (albeit much slower)
6) we've then hit only one 'FileNotFoundErrors', and for the rest of the import,
it continually runs into 'org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException:'
occasionally a file or two will import, but generally we hit the
NotServingRegionException, and the majority of files just don't get imported
------------------------------------------------------------------
thanks for the support, we appreciate it!
/leith
