Can you check region server log to see if there was concurrent compaction going on for table50,\x05\x80\x00\x00\x00\x00\x1E\x84\x80\ xDF\x00\x00\x01D\xFD\xFD\x06\x7F^\xFF\xFF\xFF\xFD,1395709055737. 5a772656accd69dd0354a09fc0c4415f. ?
If so, how long did the compaction take ? Cheers On Tue, Mar 25, 2014 at 2:52 PM, Mahesh Garg <[email protected]> wrote: > Hi, > > We are trying to bulk load into HBase. But, it it failing. We are using a > very standard process: > > 1. Create HFiles using an MR job. > 2. Call "completebulkload" to move HFiles into regions. > > The table is pre-split. We are using HBase version "0.94.15-cdh4.6.0". > > The load process runs in a loop and, about 10 million rows are loaded in > each attempt. However, the process fails in > "completebulkload" with RegionTooBusyException. This error happen at > different times after loading about 120 M rows. > > Any suggestion on how to avoid/debug this problem? Any parameters to tune? > The stack track is listed below. > > Thanks for your help! > Garg > > > 14/03/25 03:10:39 INFO mapreduce.LoadIncrementalHFiles: Trying to load > hfile=hdfs://XXX.XXX.XXX.XXX:8020/user/userName/outputdir/cf2/e2680aee969a47ae97c6980bbd76ea66 > first=\x06\x00\x00\x00\x00\x00\x00\x00\x07_0_\x00\x00\x01D\xF73\xB7\x1F_\x00\x00\x00\x01 > last=\x06\x00\x00\x00\x00\x00w[B_25_\x00\x00\x01D\xF74\xD7\xAC_\x00\x00\x00\x01 > 14/03/25 03:21:50 ERROR mapreduce.LoadIncrementalHFiles: Encountered > unrecoverable error from region server > org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after > attempts=10, exceptions: > Tue Mar 25 03:11:39 UTC 2014, > org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles$3@29324479, > org.apache.hadoop.hbase.RegionTooBusyException: > org.apache.hadoop.hbase.RegionTooBusyException: failed to get a lock in > 60000 ms. > regionName=table50,\x05\x80\x00\x00\x00\x00\x1E\x84\x80\xDF\x00\x00\x01D\xFD\xFD\x06\x7F^\xFF\xFF\xFF\xFD,1395709055737.5a772656accd69dd0354a09fc0c4415f., > server=XXX.XXX.XXX.XXX,60020,1395216882104 > at > org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:5898) > at > org.apache.hadoop.hbase.regionserver.HRegion.lock(HRegion.java:5884) > at > org.apache.hadoop.hbase.regionserver.HRegion.startBulkRegionOperation(HRegion.java:5843) > at > org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3640) > at > org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:3623) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.bulkLoadHFiles(HRegionServer.java:2978) > at sun.reflect.GeneratedMethodAccessor61.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRp > >
