I'm are running HBase 1.1.8 on MapR 5.2.1 and I have a table with a little
over 1000 regions. All the regions load fine, except for one, which is
stuck in the opening state. I've tried restarting the masters and all the
region servers, but I'm getting the same result. The logs don't seem very
helpful. They show that the request to open the region, but it never
succeeds.
2017-09-20 05:26:02,095 INFO
[PriorityRpcServer.handler=3,queue=1,port=16020]
regionserver.RSRpcServices: Open whois-versioned,bobandamy.net
,1505420182154.489a652d37429afe19955512f644f64f.
2017-09-20 05:26:02,103 INFO
[StoreOpener-489a652d37429afe19955512f644f64f-1] hfile.CacheConfig:
blockCache=LruBlockCache{blockCount=17212, currentSize=1451765224,
freeSize=4952805912, maxSize=6404571136, heapSize=1451765224,
minSize=6084342272, minFactor=0.95, multiSize=3042171136, multiFactor=0.5,
singleSize=1521085568, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false, cacheEvictOnClose=false,
cacheDataCompressed=false, prefetchOnOpen=false
2017-09-20 05:26:02,103 INFO
[StoreOpener-489a652d37429afe19955512f644f64f-1]
compactions.CompactionConfiguration: size [134217728, 9223372036854775807);
files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point
2684354560; major period 604800000, major jitter 0.500000, min locality to
compact 0.000000
2017-09-20 05:26:02,106 INFO
[StoreOpener-489a652d37429afe19955512f644f64f-1] hfile.CacheConfig:
blockCache=LruBlockCache{blockCount=17212, currentSize=1451765224,
freeSize=4952805912, maxSize=6404571136, heapSize=1451765224,
minSize=6084342272, minFactor=0.95, multiSize=3042171136, multiFactor=0.5,
singleSize=1521085568, singleFactor=0.25}, cacheDataOnRead=true,
cacheDataOnWrite=false, cacheIndexesOnWrite=false,
cacheBloomsOnWrite=false, cacheEvictOnClose=false,
cacheDataCompressed=false, prefetchOnOpen=false
hbck reports this error:
ERROR: Region { meta =>
whois-versioned,bobandamy.net,1505420182154.489a652d37429afe19955512f644f64f.,
hdfs =>
maprfs:///hbase/data/default/whois-versioned/489a652d37429afe19955512f644f64f,
deployed => , replicaId => 0 } not deployed on any region server.
2017-09-20 05:29:10,602 INFO [main] util.HBaseFsck: Handling overlap
merges in parallel. set hbasefsck.overlap.merge.parallel to false to run
serially.
ERROR: There is a hole in the region chain between bobandamy.net and
bodyinbalanceshop.com. You need to create a new .regioninfo and region dir
in hdfs to plug the hole.
ERROR: Found inconsistency in table whois-versioned
Repairing the table created more of a mess, and didn't fix this issue. I've
tried the assigning it from the shell, but the operation just times out.
Any ideas on how to debug this issue in order to recover the data would be
greatly appreciated. Thank you.
