Re: Merged daughter region

Fri, 05 Jul 2019 10:28:27 -0700

Here's what our meta for this example looks like. I don't have the entries for the merged regions, aside from what is provided by the old parent. To try to prevent some confusion, in this example, it's region A that got compacted and merged (different from my initial example). Also this is 1.4.8 running on EMR and S3. 

Parent:
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f. column=info:regioninfo, timestamp=1561992899379, value={ENCODED => 7b0d3c6836cf417b007771ab48d0450f, NAME => 'banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f.', STARTKEY => '\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5', ENDKEY => '\x02\x09\xFC\x17\x9C\xC1\x01\xF0 n\xC3', OFFLINE => true, SPLIT => true} 
 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f.
 column=info:seqnumDuringOpen, timestamp=1561992267396, 
value=\x00\x00\x00\x00\x00\x12\xD59
 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f.
 column=info:server, timestamp=1561992267396, 
value=ip-10-0-61-195.ec2.internal:16020
 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f.
 column=info:serverstartcode, timestamp=1561992267396, value=1560961976157
 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f.
 column=info:sn, timestamp=1560968683601, 
value=ip-10-0-61-195.ec2.internal,16020,1560961976157
 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f.
 column=info:splitA, timestamp=1561992899379, value={ENCODED => 
a1d452e3efa13c9a8162a85b6ada9564, NAME => 
'banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564.',
 STARTKEY => '\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5', ENDKEY => 
'\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8'}
 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f.
 column=info:splitB, timestamp=1561992899379, value={ENCODED => 
4046237c869f876fc8f9a5f090dc8060, NAME => 
'banana-2556_banana_z3_v2,\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8,1561992895078.4046237c869f876fc8f9a5f090dc8060.',
 STARTKEY => '\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8', ENDKEY => 
'\x02\x09\xFC\x17\x9C\xC1\x01\xF0 n\xC3'}
 
banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f.
 column=info:state, timestamp=1561992906651, value=SPLIT

Split A + previous region:

 
banana-2556_banana_z3_v2,\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80,1561995018192.8152f86473b92265f7704d3ff9291f78.
 column=info:regioninfo, timestamp=1561995022370, value={ENCODED => 
8152f86473b92265f7704d3ff9291f78, NAME => 
'banana-2556_banana_z3_v2,\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80,15619
95018192.8152f86473b92265f7704d3ff9291f78.', STARTKEY => '\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80', ENDKEY => '\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8'} 
 
banana-2556_banana_z3_v2,\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80,1561995018192.8152f86473b92265f7704d3ff9291f78.
 column=info:seqnumDuringOpen, timestamp=1561995028121, 
value=\x00\x00\x00\x00\x00\x00\x00\x01
 
banana-2556_banana_z3_v2,\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80,1561995018192.8152f86473b92265f7704d3ff9291f78.
 column=info:server, timestamp=1561995028121, 
value=ip-10-0-62-125.ec2.internal:16020
 
banana-2556_banana_z3_v2,\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80,1561995018192.8152f86473b92265f7704d3ff9291f78.
 column=info:serverstartcode, timestamp=1561995028121, value=1561126540919
 
banana-2556_banana_z3_v2,\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80,1561995018192.8152f86473b92265f7704d3ff9291f78.
 column=info:state, timestamp=1561995028121, value=OPEN

Split B:
banana-2556_banana_z3_v2,\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8,1561992895078.4046237c869f876fc8f9a5f090dc8060. column=info:regioninfo, timestamp=1561992899379, value={ENCODED => 4046237c869f876fc8f9a5f090dc8060, NAME => 'banana-2556_banana_z3_v2,\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8,1561992895078.4046237c869f876fc8f9a5f090dc8060.', STARTKEY => '\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8', ENDKEY => '\x02\x09\xFC\x17\x9C\xC1\x01\xF0 n\xC3'} 
 
banana-2556_banana_z3_v2,\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8,1561992895078.4046237c869f876fc8f9a5f090dc8060.
 column=info:seqnumDuringOpen, timestamp=1561992906652, 
value=\x00\x00\x00\x00\x00\x00\x00\x01
 
banana-2556_banana_z3_v2,\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8,1561992895078.4046237c869f876fc8f9a5f090dc8060.
 column=info:server, timestamp=1561992906652, 
value=ip-10-0-61-195.ec2.internal:16020
 
banana-2556_banana_z3_v2,\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8,1561992895078.4046237c869f876fc8f9a5f090dc8060.
 column=info:serverstartcode, timestamp=1561992906652, value=1560961976157
 
banana-2556_banana_z3_v2,\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8,1561992895078.4046237c869f876fc8f9a5f090dc8060.
 column=info:state, timestamp=1561992906652, value=OPEN
You can see that the end key of "Split A + previous region" is the start key of Split B.
Looking at the logs from the operating regionserver we see the parent split (7b0d3c6836cf417b007771ab48d0450f):
2019-07-01 14:54:54,950 INFO [RpcServer.priority.FPBQ.Fifo.handler=19,queue=9,port=16020] regionserver.RSRpcServices: Splitting banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f. 2019-07-01 14:54:55,078 INFO [regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] regionserver.SplitTransaction: Starting split of region banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f. 2019-07-01 14:54:55,298 INFO [StoreCloserThread-banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f.-1] regionserver.HStore: Closed d 2019-07-01 14:54:55,303 INFO [regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] regionserver.HRegion: Closed banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f. 2019-07-01 14:54:55,303 INFO [regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] regionserver.SplitTransaction: Preparing to split 13 storefiles for region banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f. using 8 threads 2019-07-01 14:54:57,168 INFO [regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/.splits/a1d452e3efa13c9a8162a85b6ada9564 s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564 2019-07-01 14:54:57,814 INFO [regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] regionserver.RegionCoprocessorHost: Loaded coprocessor org.locationtech.geomesa.hbase.coprocessor.GeoMesaCoprocessor from HTD of banana-2556_banana_z3_v2 successfully. 2019-07-01 14:54:58,406 INFO [regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/.splits/4046237c869f876fc8f9a5f090dc8060 s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/4046237c869f876fc8f9a5f090dc8060 2019-07-01 14:54:59,166 INFO [regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] regionserver.RegionCoprocessorHost: Loaded coprocessor org.locationtech.geomesa.hbase.coprocessor.GeoMesaCoprocessor from HTD of banana-2556_banana_z3_v2 successfully. 
...
2019-07-01 14:55:06,654 INFO [regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] regionserver.SplitRequest: Region split, hbase:meta updated, and report to master. Parent=banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f., new regions: banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564., banana-2556_banana_z3_v2,\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8,1561992895078.4046237c869f876fc8f9a5f090dc8060.. Split took 11sec 2019-07-01 14:55:06,623 INFO [ip-10-0-61-195.ec2.internal,16020,1560961976157-daughterOpener=a1d452e3efa13c9a8162a85b6ada9564] regionserver.HRegion: Onlined a1d452e3efa13c9a8162a85b6ada9564; next sequenceid=1234226 2019-07-01 14:55:06,651 INFO [ip-10-0-61-195.ec2.internal,16020,1560961976157-daughterOpener=4046237c869f876fc8f9a5f090dc8060] regionserver.HRegion: Onlined 4046237c869f876fc8f9a5f090dc8060; next sequenceid=1234227 2019-07-01 14:55:06,654 INFO [regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] regionserver.SplitRequest: Region split, hbase:meta updated, and report to master. Parent=banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f., new regions: banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564., banana-2556_banana_z3_v2,\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8,1561992895078.4046237c869f876fc8f9a5f090dc8060.. Split took 11sec 2019-07-01 14:55:06,654 INFO [regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] regionserver.SplitRequest: Split transaction journal: 
    STARTED at 1561992895077
    PREPARED at 1561992895078
    BEFORE_PRE_SPLIT_HOOK at 1561992895078
    AFTER_PRE_SPLIT_HOOK at 1561992895078
    SET_SPLITTING at 1561992895079
    CREATE_SPLIT_DIR at 1561992895297
    CLOSED_PARENT_REGION at 1561992895303
    OFFLINED_PARENT at 1561992895303
    STARTED_REGION_A_CREATION at 1561992896718
    STARTED_REGION_B_CREATION at 1561992897974
    PONR at 1561992899379
    OPENED_REGION_A at 1561992906624
    OPENED_REGION_B at 1561992906651
    BEFORE_POST_SPLIT_HOOK at 1561992906653
    AFTER_POST_SPLIT_HOOK at 1561992906653
    COMPLETED at 1561992906653

Now for SplitA (a1d452e3efa13c9a8162a85b6ada9564):
2019-07-01 14:54:57,168 INFO [regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/.splits/a1d452e3efa13c9a8162a85b6ada9564 s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564 2019-07-01 14:54:59,438 INFO [StoreOpener-a1d452e3efa13c9a8162a85b6ada9564-1] hfile.CacheConfig: Created cacheConfig for d: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@93fa68a, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2019-07-01 14:54:59,438 INFO [StoreOpener-a1d452e3efa13c9a8162a85b6ada9564-1] compactions.CompactionConfiguration: size [1409286144, 5000000000, 5000000000); files [6, 10); ratio 1.000000; off-peak ratio 5.000000; throttle point 28185722880; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2019-07-01 15:15:18,542 INFO [RpcServer.priority.FPBQ.Fifo.handler=19,queue=9,port=16020] regionserver.RSRpcServices: Close a1d452e3efa13c9a8162a85b6ada9564, moving to ip-10-0-39-221.ec2.internal,16020,1560961973442 2019-07-01 15:15:18,544 INFO [StoreCloserThread-banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564.-1] regionserver.HStore: Closed d 2019-07-01 15:15:18,556 INFO [RS_CLOSE_REGION-ip-10-0-61-195:16020-1] regionserver.HRegion: Closed banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564. 2019-07-01 15:15:18,556 INFO [RS_CLOSE_REGION-ip-10-0-61-195:16020-1] regionserver.HRegionServer: Adding moved region record: a1d452e3efa13c9a8162a85b6ada9564 to ip-10-0-39-221.ec2.internal,16020,1560961973442 as of 1234226 

<moved servers>
2019-07-01 15:15:18,566 INFO [RpcServer.priority.FPBQ.Fifo.handler=19,queue=9,port=16020] regionserver.RSRpcServices: Open banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564. 2019-07-01 15:15:18,675 INFO [RS_OPEN_REGION-ip-10-0-39-221:16020-0] regionserver.RegionCoprocessorHost: Loaded coprocessor org.locationtech.geomesa.hbase.coprocessor.GeoMesaCoprocessor from HTD of banana-2556_banana_z3_v2 successfully. 2019-07-01 15:15:18,781 INFO [StoreOpener-a1d452e3efa13c9a8162a85b6ada9564-1] hfile.CacheConfig: Created cacheConfig for d: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@35b813db, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2019-07-01 15:15:18,782 INFO [StoreOpener-a1d452e3efa13c9a8162a85b6ada9564-1] compactions.CompactionConfiguration: size [1409286144, 5000000000, 5000000000); files [6, 10); ratio 1.000000; off-peak ratio 5.000000; throttle point 28185722880; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2019-07-01 15:15:21,266 INFO  [StoreFileOpenerThread-d-1] regionserver.StoreFile: Bloom filter turned off by CF config for 2743f276b09d4a7e9bfcbef5f210c444_SeqId_4_.7b0d3c6836cf417b007771ab48d0450f 2019-07-01 15:15:28,018 INFO [RS_OPEN_REGION-ip-10-0-39-221:16020-0] regionserver.HRegion: Onlined a1d452e3efa13c9a8162a85b6ada9564; next sequenceid=1234230 2019-07-01 15:15:28,025 INFO [PostOpenDeployTasks:a1d452e3efa13c9a8162a85b6ada9564] regionserver.HRegionServer: Post open deploy tasks for banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564. 2019-07-01 15:15:28,025 INFO [regionserver/ip-10-0-39-221.ec2.internal/10.0.39.221:16020-longCompactions-1560968456884] regionserver.HRegion: Starting compaction on d in region banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564. 2019-07-01 15:15:28,025 INFO [regionserver/ip-10-0-39-221.ec2.internal/10.0.39.221:16020-longCompactions-1560968456884] regionserver.HStore: Starting compaction of 10 file(s) in d of banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564. into tmpdir=s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/.tmp, totalSize=30.8 G 2019-07-01 15:18:40,376 INFO [regionserver/ip-10-0-39-221.ec2.internal/10.0.39.221:16020-longCompactions-1560968456884] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/.tmp/f36978da3bd0413a9b3a57f51d00bb9b s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/f36978da3bd0413a9b3a57f51d00bb9b 2019-07-01 15:18:52,125 INFO [regionserver/ip-10-0-39-221.ec2.internal/10.0.39.221:16020-longCompactions-1560968456884] regionserver.HStore: Completed compaction of 10 file(s) in d of banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564. into f36978da3bd0413a9b3a57f51d00bb9b(size=7.9 G), total size for store is 9.5 G. This selection was in queue for 0sec, and took 3mins, 24sec to execute. 2019-07-01 15:18:52,125 INFO [regionserver/ip-10-0-39-221.ec2.internal/10.0.39.221:16020-longCompactions-1560968456884] regionserver.CompactSplitThread: Completed compaction: Request = regionName=banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564., storeName=d, fileCount=10, fileSize=30.8 G (9.3 G, 3.2 G, 3.2 G, 3.4 G, 4.1 G, 677.2 M, 674.4 M, 4.7 G, 9.0 M, 16.0 M), priority=35, time=1818008526138670; duration=3mins, 24sec 2019-07-01 15:18:59,295 INFO [RS_COMPACTED_FILES_DISCHARGER-ip-10-0-39-221:16020-4] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/2743f276b09d4a7e9bfcbef5f210c444_SeqId_4_.7b0d3c6836cf417b007771ab48d0450f s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/2743f276b09d4a7e9bfcbef5f210c444_SeqId_4_.7b0d3c6836cf417b007771ab48d0450f 2019-07-01 15:18:59,648 INFO [RS_COMPACTED_FILES_DISCHARGER-ip-10-0-39-221:16020-4] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/d63e5131cdf5475e904a39f771c459e9.7b0d3c6836cf417b007771ab48d0450f s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/d63e5131cdf5475e904a39f771c459e9.7b0d3c6836cf417b007771ab48d0450f 2019-07-01 15:19:00,042 INFO [RS_COMPACTED_FILES_DISCHARGER-ip-10-0-39-221:16020-4] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/864b1dbe8fcd43d2aaaf573944dea7e7.7b0d3c6836cf417b007771ab48d0450f s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/864b1dbe8fcd43d2aaaf573944dea7e7.7b0d3c6836cf417b007771ab48d0450f 2019-07-01 15:19:00,405 INFO [RS_COMPACTED_FILES_DISCHARGER-ip-10-0-39-221:16020-4] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/06cfc7719e644cbf9a3a0ffb844aef7f.7b0d3c6836cf417b007771ab48d0450f s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/06cfc7719e644cbf9a3a0ffb844aef7f.7b0d3c6836cf417b007771ab48d0450f 2019-07-01 15:19:00,748 INFO [RS_COMPACTED_FILES_DISCHARGER-ip-10-0-39-221:16020-4] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/be1a9bee31c44afc9ae0e8a6c8af2053.7b0d3c6836cf417b007771ab48d0450f s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/be1a9bee31c44afc9ae0e8a6c8af2053.7b0d3c6836cf417b007771ab48d0450f 2019-07-01 15:19:01,194 INFO [RS_COMPACTED_FILES_DISCHARGER-ip-10-0-39-221:16020-4] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/322040d1fb4343f0abb91c9592433288.7b0d3c6836cf417b007771ab48d0450f s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/322040d1fb4343f0abb91c9592433288.7b0d3c6836cf417b007771ab48d0450f 2019-07-01 15:19:01,517 INFO [RS_COMPACTED_FILES_DISCHARGER-ip-10-0-39-221:16020-4] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/7bf98e832ff743edae58bbcab60bad75.7b0d3c6836cf417b007771ab48d0450f s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/7bf98e832ff743edae58bbcab60bad75.7b0d3c6836cf417b007771ab48d0450f 2019-07-01 15:19:02,008 INFO [RS_COMPACTED_FILES_DISCHARGER-ip-10-0-39-221:16020-4] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/be4c74dd039245f4ad1d57b768126501.7b0d3c6836cf417b007771ab48d0450f s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/be4c74dd039245f4ad1d57b768126501.7b0d3c6836cf417b007771ab48d0450f 2019-07-01 15:19:02,422 INFO [RS_COMPACTED_FILES_DISCHARGER-ip-10-0-39-221:16020-4] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/356197dbd10f4bb684af39152c637b5e.7b0d3c6836cf417b007771ab48d0450f s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/356197dbd10f4bb684af39152c637b5e.7b0d3c6836cf417b007771ab48d0450f 2019-07-01 15:19:02,893 INFO [RS_COMPACTED_FILES_DISCHARGER-ip-10-0-39-221:16020-4] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/be22002763d14728b151524afd376fd7.7b0d3c6836cf417b007771ab48d0450f s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/be22002763d14728b151524afd376fd7.7b0d3c6836cf417b007771ab48d0450f 2019-07-01 15:22:43,326 INFO [RpcServer.priority.FPBQ.Fifo.handler=19,queue=9,port=16020] regionserver.RSRpcServices: Close a1d452e3efa13c9a8162a85b6ada9564, moving to ip-10-0-62-125.ec2.internal,16020,1561126540919 2019-07-01 15:22:43,327 INFO [StoreCloserThread-banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564.-1] regionserver.HStore: Closed d 2019-07-01 15:22:43,338 INFO [RS_CLOSE_REGION-ip-10-0-39-221:16020-1] regionserver.HRegion: Closed banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564. 2019-07-01 15:22:43,338 INFO [RS_CLOSE_REGION-ip-10-0-39-221:16020-1] regionserver.HRegionServer: Adding moved region record: a1d452e3efa13c9a8162a85b6ada9564 to ip-10-0-62-125.ec2.internal,16020,1561126540919 as of 1234230 

<moved servers>
2019-07-01 15:22:43,340 INFO [RpcServer.priority.FPBQ.Fifo.handler=18,queue=8,port=16020] regionserver.RSRpcServices: Open banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564. 2019-07-01 15:22:43,535 INFO [StoreOpener-a1d452e3efa13c9a8162a85b6ada9564-1] hfile.CacheConfig: Created cacheConfig for d: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@2581002f, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2019-07-01 15:22:43,536 INFO [StoreOpener-a1d452e3efa13c9a8162a85b6ada9564-1] compactions.CompactionConfiguration: size [1409286144, 5000000000, 5000000000); files [6, 10); ratio 1.000000; off-peak ratio 5.000000; throttle point 28185722880; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2019-07-01 15:22:46,574 INFO [RS_OPEN_REGION-ip-10-0-62-125:16020-7] regionserver.HRegion: Onlined a1d452e3efa13c9a8162a85b6ada9564; next sequenceid=1234234 2019-07-01 15:22:46,575 INFO [PostOpenDeployTasks:a1d452e3efa13c9a8162a85b6ada9564] regionserver.HRegionServer: Post open deploy tasks for banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564. 2019-07-01 15:22:46,576 INFO [regionserver/ip-10-0-62-125.ec2.internal/10.0.62.125:16020-shortCompactions-1561136546586] regionserver.HRegion: Starting compaction on d in region banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564. 2019-07-01 15:22:46,576 INFO [regionserver/ip-10-0-62-125.ec2.internal/10.0.62.125:16020-shortCompactions-1561136546586] regionserver.HStore: Starting compaction of 3 file(s) in d of banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564. into tmpdir=s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/.tmp, totalSize=1.6 G 2019-07-01 15:22:46,587 INFO [RpcServer.priority.FPBQ.Fifo.handler=18,queue=8,port=16020] regionserver.RSRpcServices: Receiving merging request for banana-2556_banana_z3_v2,\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80,1550014268044.8a99df0ca2c097ee707bc1eeb7402745., banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564.,forcible=true 2019-07-01 15:22:52,511 INFO [regionserver/ip-10-0-62-125.ec2.internal/10.0.62.125:16020-shortCompactions-1561136546586] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/.tmp/57d4d0aa4e1e437ca5ff8dbcc953a070 s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/57d4d0aa4e1e437ca5ff8dbcc953a070 2019-07-01 15:22:54,458 INFO [regionserver/ip-10-0-62-125.ec2.internal/10.0.62.125:16020-shortCompactions-1561136546586] regionserver.HStore: Completed compaction of 3 file(s) in d of banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564. into 57d4d0aa4e1e437ca5ff8dbcc953a070(size=78.1 M), total size for store is 8.0 G. This selection was in queue for 0sec, and took 7sec to execute. 2019-07-01 15:22:54,458 INFO [regionserver/ip-10-0-62-125.ec2.internal/10.0.62.125:16020-shortCompactions-1561136546586] regionserver.CompactSplitThread: Completed compaction: Request = regionName=banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564., storeName=d, fileCount=3, fileSize=1.6 G (676.8 M, 677.1 M, 291.4 M), priority=44, time=868204396698301; duration=7sec 2019-07-01 15:23:45,642 INFO [RS_COMPACTED_FILES_DISCHARGER-ip-10-0-62-125:16020-5] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/25053cc8175047ed94d9acae9bb43a32.7b0d3c6836cf417b007771ab48d0450f s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/25053cc8175047ed94d9acae9bb43a32.7b0d3c6836cf417b007771ab48d0450f 2019-07-01 15:23:46,529 INFO [RS_COMPACTED_FILES_DISCHARGER-ip-10-0-62-125:16020-5] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/2834cd5210dd41e0ae7a523b276bf040.7b0d3c6836cf417b007771ab48d0450f s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/2834cd5210dd41e0ae7a523b276bf040.7b0d3c6836cf417b007771ab48d0450f 2019-07-01 15:23:46,989 INFO [RS_COMPACTED_FILES_DISCHARGER-ip-10-0-62-125:16020-5] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/cd61f91a73574edeb1c56f0fd50908db.7b0d3c6836cf417b007771ab48d0450f s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/a1d452e3efa13c9a8162a85b6ada9564/d/cd61f91a73574edeb1c56f0fd50908db.7b0d3c6836cf417b007771ab48d0450f 2019-07-01 15:30:18,099 INFO [RpcServer.priority.FPBQ.Fifo.handler=18,queue=8,port=16020] regionserver.RSRpcServices: Receiving merging request for banana-2556_banana_z3_v2,\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80,1550014268044.8a99df0ca2c097ee707bc1eeb7402745., banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564.,forcible=true 2019-07-01 15:30:18,192 INFO [regionserver/ip-10-0-62-125.ec2.internal/10.0.62.125:16020-merges-1561472619525] regionserver.RegionMergeTransactionImpl: Starting merge of banana-2556_banana_z3_v2,\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80,1550014268044.8a99df0ca2c097ee707bc1eeb7402745. and banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564., forcible=true 2019-07-01 15:30:18,472 INFO [StoreCloserThread-banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564.-1] regionserver.HStore: Closed d 2019-07-01 15:30:18,476 INFO [regionserver/ip-10-0-62-125.ec2.internal/10.0.62.125:16020-merges-1561472619525] regionserver.HRegion: Closed banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564. 2019-07-01 15:30:28,124 INFO [regionserver/ip-10-0-62-125.ec2.internal/10.0.62.125:16020-merges-1561472619525] regionserver.RegionMergeRequest: Regions merged, hbase:meta updated, and report to master. region_a=banana-2556_banana_z3_v2,\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80,1550014268044.8a99df0ca2c097ee707bc1eeb7402745., region_b=banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564.,merged region=banana-2556_banana_z3_v2,\x01\x09\xFC{\x9D\x8C\x87\xB3eX\x80,1561995018192.8152f86473b92265f7704d3ff9291f78.. Region merge took 10sec 

And SplitB (4046237c869f876fc8f9a5f090dc8060):
2019-07-01 14:54:58,406 INFO [regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/.splits/4046237c869f876fc8f9a5f090dc8060 s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/4046237c869f876fc8f9a5f090dc8060 2019-07-01 14:54:59,431 INFO [StoreOpener-4046237c869f876fc8f9a5f090dc8060-1] hfile.CacheConfig: Created cacheConfig for d: blockCache=org.apache.hadoop.hbase.io.hfile.CombinedBlockCache@93fa68a, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2019-07-01 14:54:59,431 INFO [StoreOpener-4046237c869f876fc8f9a5f090dc8060-1] compactions.CompactionConfiguration: size [1409286144, 5000000000, 5000000000); files [6, 10); ratio 1.000000; off-peak ratio 5.000000; throttle point 28185722880; major period 0, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory 2019-07-01 14:55:06,651 INFO [ip-10-0-61-195.ec2.internal,16020,1560961976157-daughterOpener=4046237c869f876fc8f9a5f090dc8060] regionserver.HRegion: Onlined 4046237c869f876fc8f9a5f090dc8060; next sequenceid=1234227 2019-07-01 14:55:06,654 INFO [regionserver/ip-10-0-61-195.ec2.internal/10.0.61.195:16020-splits-1561414889426] regionserver.SplitRequest: Region split, hbase:meta updated, and report to master. Parent=banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f., new regions: banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1561992895078.a1d452e3efa13c9a8162a85b6ada9564., banana-2556_banana_z3_v2,\x02\x09\xFC\x06\x1F\xAB{\xDD\xE2$\xDE8,1561992895078.4046237c869f876fc8f9a5f090dc8060.. Split took 11sec
There is then no activity around SplitB or the parent until I kicked off a major compaction a few days later. After which we then saw the parent being removed:
hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 09:52:21,960 INFO [ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/06cfc7719e644cbf9a3a0ffb844aef7f s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/06cfc7719e644cbf9a3a0ffb844aef7f hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 09:52:32,028 INFO [ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/25053cc8175047ed94d9acae9bb43a32 s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/25053cc8175047ed94d9acae9bb43a32 hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 09:52:35,875 INFO [ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/2743f276b09d4a7e9bfcbef5f210c444_SeqId_4_ s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/2743f276b09d4a7e9bfcbef5f210c444_SeqId_4_ hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 09:52:55,075 INFO [ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/2834cd5210dd41e0ae7a523b276bf040 s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/2834cd5210dd41e0ae7a523b276bf040 hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 09:52:58,668 INFO [ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/322040d1fb4343f0abb91c9592433288 s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/322040d1fb4343f0abb91c9592433288 hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 09:53:01,337 INFO [ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/356197dbd10f4bb684af39152c637b5e s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/356197dbd10f4bb684af39152c637b5e hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 09:53:02,125 INFO [ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/7bf98e832ff743edae58bbcab60bad75 s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/7bf98e832ff743edae58bbcab60bad75 hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 09:53:15,254 INFO [ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/864b1dbe8fcd43d2aaaf573944dea7e7 s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/864b1dbe8fcd43d2aaaf573944dea7e7 hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 09:53:21,139 INFO [ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/be1a9bee31c44afc9ae0e8a6c8af2053 s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/be1a9bee31c44afc9ae0e8a6c8af2053 hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 09:53:30,645 INFO [ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/be22002763d14728b151524afd376fd7 s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/be22002763d14728b151524afd376fd7 hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 09:53:31,515 INFO [ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/be4c74dd039245f4ad1d57b768126501 s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/be4c74dd039245f4ad1d57b768126501 hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 09:53:40,128 INFO [ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/cd61f91a73574edeb1c56f0fd50908db s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/cd61f91a73574edeb1c56f0fd50908db hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 09:53:44,638 INFO [ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] s3n2.S3NativeFileSystem2: rename s3://orange.banana16/root/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/d63e5131cdf5475e904a39f771c459e9 s3://orange.banana16/root/archive/data/default/banana-2556_banana_z3_v2/7b0d3c6836cf417b007771ab48d0450f/d/d63e5131cdf5475e904a39f771c459e9 hbase-hbase-master-ip-10-0-53-128.log.2019-07-05-09.gz:2019-07-05 09:53:51,974 INFO [ip-10-0-53-128.ec2.internal,16000,1560968452253_ChoreService_5] hbase.MetaTableAccessor: Deleted banana-2556_banana_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f. 

After this compaction queries to this key space began work again.

Thanks for the help,
Austin

On 7/5/19 10:44 AM, Wellington Chevreuil wrote:

Hi Austin, which hbase version is in use? Is this
'7b0d3c6836cf417b007771ab48d0450f' the parent region, or one of the
daughter regions that got merged? What does meta scan show for the parent
region and original two daughters, plus the resulting region from the merge?

Em sex, 5 de jul de 2019 às 15:25, Austin Heyne <ahe...@ccri.com> escreveu:


Hey,

We're continuing to get some heavy use out of our normalizers (split
point pr still to come), and we're seeing an issue with daughters
getting merged. The situation is a region is splitting and becoming the
parent to regions A and B, one of these regions, say B is then getting
compacted out but region A is still waiting to be compacted. Region B is
then getting merged by our normalization plan with it's neighbor region,
C, producing region D. This leaves us with regions A and D and the
metadata for the parent.

Now when a query comes in that spans A through D, the query planner sees
the reference to B in the parent's metadata and tries to find that
region. After the timeout period we get an exception that says the child
region (B) of the parent isn't online yet but should be soon.

19/07/03 18:33:06 ERROR Executor: Exception in task 3.0 in stage 9.0
(TID 43)
org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the
location
      at

org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:316)
      at

org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
      at

org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
      at

org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:212)
      at
org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:314)
      at

org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:289)
      at

org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:164)
      at
org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:159)
      at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:796)

...
Caused by: org.apache.hadoop.hbase.client.RegionOfflineException: the
only available region for the required row is a split parent, the
daughters should be online soon:

table_z3_v2,\x01\x09\xFC}\x18\xFA\xD2o\x99\x8D\xE5,1550014268044.7b0d3c6836cf417b007771ab48d0450f.
      at

org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1307)
      at

org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1183)
      at

org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)

  From what I gather we should be removing the reference to B from the
parent's metadata when B gets merged with C.

Is this something people deal with or is there a good way to prevent this?

Thanks,
Austin

Reply via email to