I seen the blocking problem also in some of my imports to a fresh install
but they seam to go away after the region servers have more regions hosted.
In the past what I had to do was import one file at a time and sleep between
jobs. But there was some patch added that should make hbase much more stable
in version 0.2.0.
"leith" <[EMAIL PROTECTED]> wrote in message
news:[EMAIL PROTECTED]
we've been trying for a couple of days (without success) to import our
data into hbase.
initially we ran into quite a few OOME errors, but we've seem to overcome
that by adjusting our jvm memory heap sizes.
however, we're still running into many other roadblocks, and in my opinion
we just don't have the right configuration options in our conf files (or
maybe not enough resources to get the job done)
in the end, this is a one time task. if we can be successful, we believe
this will be a good introduction for us using hbase, and we can continue
by integrating it further into our project.
i'ld appreciate if someone would offer us some advice for the following
task/setup that we are trying to accomplish. here are the details:
---------------------------------------------------------------
1) everything (hdfs/hbase) is running on one machine currently (short
term)
2) we are importing 60k files, each ranging between 100k and 64MB along
with necessary meta-data in other column-families
3) our machine has 2GB, amd64 dual core, dedicated to the import task,
hbase heapsize is set to 1000
4) our import program is single threaded java program, iterating through
our files and doing batchoperations for each file into hbase
5) after about 15 minutes of successful importing, we see
'INFO org.apache.hadoop.hbase.regionserver.HRegion: Blocking updates for
'IPC Server handler 8 on 60020' on region dmls,,1216768730386: Memcache
size 64.0m is
>= than blocking 64.0m size '
after another 15 minutes of inactivity, we see the threads slowly get
unblocked, and importing continues (albeit much slower)
6) we've then hit only one 'FileNotFoundErrors', and for the rest of the
import, it continually runs into
'org.apache.hadoop.hbase.NotServingRegionException:
org.apache.hadoop.hbase.NotServingRegionException:'
occasionally a file or two will import, but generally we hit the
NotServingRegionException, and the majority of files just don't get
imported
------------------------------------------------------------------
thanks for the support, we appreciate it!
/leith
