version: HBase 1.4.8
We have a stuck RIT that wont clear. The parent was offlined, and
daughter region were created and moved out of the parents .split dir.
But the parent region is never removed and the RIT doesn't clear. We
have tried to run `move '<parent_region_name>'`, `unassign
'<parent_region_name>'`, and |sudo stop hbase-regionserver && sudo start
hbase-regionserver |on the parent and daughter regionservers. My only
other thought is to disable the table, delete the parent region and
enable the table. Any advice on how to clear the RIT or insight on what
is causing it would be appreciated. I have included some logs bellow
that might be helpful.
Thanks,
Zach
Region server logs:
2020-12-04 14:00:35,430 INFO
[RpcServer.priority.FPBQ.Fifo.handler=17,queue=7,port=16020]
regionserver.RSRpcServices: Close d0b06e343bc8ae49ef7e5b66089fcb2d,
moving to null
2020-12-04 14:00:35,431 INFO
[StoreCloserThread-synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.-1]
regionserver.HStore: Closed d
2020-12-04 14:00:35,436 INFO [RS_CLOSE_REGION-ip-10-0-1-4:16020-0]
regionserver.HRegion: Closed
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
2020-12-04 14:00:35,438 INFO
[RpcServer.priority.FPBQ.Fifo.handler=17,queue=7,port=16020]
regionserver.RSRpcServices: Open
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
2020-12-04 14:00:35,703 INFO
[StoreOpener-d0b06e343bc8ae49ef7e5b66089fcb2d-1] hfile.CacheConfig:
Created cacheConfig for d:
blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@58ff4bf2,
cacheDataOnRead=true, cacheDataOnWrite=false,
cacheIndexesOnWrite=false, cacheBloomsOnWrite=false,
cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2020-12-04 14:00:35,703 INFO
[StoreOpener-d0b06e343bc8ae49ef7e5b66089fcb2d-1]
compactions.CompactionConfiguration: size [1409286144, 5000000000,
5000000000); files [6, 10); ratio 1.000000; off-peak ratio 5.000000;
throttle point 28185722880; major period 0, major jitter 0.500000,
min locality to compact 0.000000; tiered compaction: max_age
9223372036854775807, incoming window min 6, compaction policy for
tiered window
org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy,
single output for minor true, compaction window factory
org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
2020-12-04 14:00:45,076 INFO [RS_OPEN_REGION-ip-10-0-1-4:16020-2]
regionserver.HRegion: Onlined d0b06e343bc8ae49ef7e5b66089fcb2d; next
sequenceid=2009960
2020-12-04 14:00:45,077 INFO
[PostOpenDeployTasks:d0b06e343bc8ae49ef7e5b66089fcb2d]
regionserver.HRegionServer: Post open deploy tasks for
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
2020-12-04 14:16:13,156 INFO
[RpcServer.priority.FPBQ.Fifo.handler=17,queue=7,port=16020]
regionserver.RSRpcServices: Splitting
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
2020-12-04 14:16:20,003 INFO
[regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
regionserver.SplitTransaction: Starting split of region
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
2020-12-04 14:16:20,457 INFO
[StoreCloserThread-synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.-1]
regionserver.HStore: Closed d
2020-12-04 14:16:20,462 INFO
[regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
regionserver.HRegion: Closed
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
2020-12-04 14:16:20,462 INFO
[regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
regionserver.SplitTransaction: Preparing to split 14 storefiles for
region
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
using 8 threads
2020-12-04 14:16:22,228 INFO
[regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
s3n2.S3NativeFileSystem2: rename
s3://bucket/root/data/default/synthetic_PRM_attr_dtg_v8_00609/d0b06e343bc8ae49ef7e5b66089fcb2d/.splits/453d941ff97a10bcf28e6ffe22d1d0e2
s3://bucket/root/data/default/synthetic_PRM_attr_dtg_v8_00609/453d941ff97a10bcf28e6ffe22d1d0e2
2020-12-04 14:16:23,988 INFO
[regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
s3n2.S3NativeFileSystem2: rename
s3://bucket/root/data/default/synthetic_PRM_attr_dtg_v8_00609/d0b06e343bc8ae49ef7e5b66089fcb2d/.splits/9d210ddbe593c83c62ac2d02413b7d97
s3://bucket/root/data/default/synthetic_PRM_attr_dtg_v8_00609/9d210ddbe593c83c62ac2d02413b7d97
2020-12-04 14:16:26,391 INFO
[regionserver/ip-10-0-1-4.ec2.internal/10.0.1.4:16020-splits-1606786421444]
regionserver.SplitRequest: Running rollback/cleanup of failed split
of
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.;
Failed
ip-10-0-1-4.ec2.internal,16020,1606786380554-daughterOpener=9d210ddbe593c83c62ac2d02413b7d97
echo "scan 'hbase:meta'" grep file for region
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
column=info:regioninfo, timestamp=1607091385153, value={ENCODED =>
d0b06e343bc8ae49ef7e5b66089fcb2d, NAME =>
'synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.',
STARTKEY => '\x018000017536e25cf0\x00e13', ENDKEY => '\x02', OFFLINE => true, SPLIT
=> true}
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
column=info:seqnumDuringOpen, timestamp=1607090445077,
value=\x00\x00\x00\x00\x00\x1E\xABh
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
column=info:server, timestamp=1607090445077,
value=ip-10-0-1-4.ec2.internal:16020
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
column=info:serverstartcode, timestamp=1607090445077, value=1606786380554
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
column=info:sn, timestamp=1606786396087,
value=ip-10-0-1-4.ec2.internal,16020,1606786380554
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
column=info:splitA, timestamp=1607091385153, value={ENCODED =>
453d941ff97a10bcf28e6ffe22d1d0e2, NAME =>
'synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1607091380003.453d941ff97a10bcf28e6ffe22d1d0e2.',
STARTKEY => '\x018000017536e25cf0\x00e13', ENDKEY => '\x01800001753f422970\x0063'}
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
column=info:splitB, timestamp=1607091385153, value={ENCODED =>
9d210ddbe593c83c62ac2d02413b7d97, NAME =>
'synthetic_PRM_attr_dtg_v8_00609,\x01800001753f422970\x0063,1607091380003.9d210ddbe593c83c62ac2d02413b7d97.',
STARTKEY => '\x01800001753f422970\x0063', ENDKEY => '\x02'}
synthetic_PRM_attr_dtg_v8_00609,\x018000017536e25cf0\x00e13,1603312639292.d0b06e343bc8ae49ef7e5b66089fcb2d.
column=info:state, timestamp=1607091380003, value=SPLITTING
