[
https://issues.apache.org/jira/browse/HBASE-23931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17052256#comment-17052256
]
Michael Stack edited comment on HBASE-23931 at 3/5/20, 3:27 PM:
----------------------------------------------------------------
Let me reopen. This came of an upgrade from 2.1 to 2.3 with more to follow.
Will see this again. Need to take a snapshot of meta pre-upgrade Here is
storyline.
* On restart, region is assigned.
* CJ runs and reports overlap (Region an 'overlap' pre-upgrade?
* Run hbck2 fixMeta but it fails w/ below:
{{2020-03-04 23:40:43,874 ERROR
org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure: Error
trying to merge [8d4e9d9dbae0a90d47b9f72c89973693,
62dbb839b3a70d5b670b9a30b0a05648] in scooter_analytics_segment (in
state=MERGE_TABLE_REGIONS_PREPARE)}}
{{org.apache.hadoop.hbase.exceptions.MergeRegionException: Skip merging
[8d4e9d9dbae0a90d47b9f72c89973693, 62dbb839b3a70d5b670b9a30b0a05648], because
parent [8d4e9d9dbae0a90d47b9f72c89973693] has a merge qualifier}}
{{ at
org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.prepareMergeRegion(MergeTableRegionsProcedure.java:483)}}
{{ at
org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:197)}}
{{ at
org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:77)}}
{{ at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:194)}}
{{ at
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:962)}}
{{ at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1669)}}
{{ at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1416)}}
{{ at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:79)}}
{{ at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1986)}}
was (Author: stack):
Let me reopen. This came of an upgrade from 2.1 to 2.3 with more to follow.
Will see this again. Need to take a snapshot of meta pre-upgrade Here is
storyline.
* On restart, region is assigned.
* CJ runs and reports overlap (Region an 'overlap' pre-upgrade?
* Run hbck2 fixMeta
* {{It schedules a merge but merge fails with MergeTableRegionsProcedure: Skip
merging [8d4e9d9dbae0a90d47b9f72c89973693, 62dbb839b3a70d5b670b9a30b0a05648],
because parent [8d4e9d9dbae0a90d47b9f72c89973693] has a merge qualifier}}
> CatalogJanitor consistency check adds merging regions to orphan list (tooo)
> ---------------------------------------------------------------------------
>
> Key: HBASE-23931
> URL: https://issues.apache.org/jira/browse/HBASE-23931
> Project: HBase
> Issue Type: Bug
> Components: hbck2
> Reporter: Michael Stack
> Priority: Major
>
> When the CJ runs, it does its consistency checking first before figuring if
> the current row is that of a splitting or merging region. Down in the guts of
> consistency we will check if its a splitting region and then cater
> appropriately accommodating daughters, etc. For merge we don't do the same.
> We'll actually add merge children parents as orphans. Its transitory but
> confusing.
>
> Here is an (ugly) example. Here is how the merge shows in orphans section of
> HBCK Report
> {code:java}
> {ENCODED => 8d4e9d9dbae0a90d47b9f72c89973693, NAME =>
> 'scooter_analytics_segment,\x01\x03\x04\x01\x02\x03,1583362144713.8d4e9d9dbae0a90d47b9f72c89973693.',
> STARTKEY => '\x01\x03\x04\x01\x02\x03', ENDKEY =>
> '\x01\x03\x04\x02\x02\x03'}{ENCODED => 62dbb839b3a70d5b670b9a30b0a05648, NAME
> =>
> 'scooter_analytics_segment,\x01\x03\x04\x01\x02\x03\x02\x01\x01\x02201904090000\x00\x00qwest\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0DW:S:P:D\x03\x00\x00\x03,1554833869031.62dbb839b3a70d5b670b9a30b0a05648.',
> STARTKEY =>
> '\x01\x03\x04\x01\x02\x03\x02\x01\x01\x02201904090000\x00\x00qwest\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x0DW:S:P:D\x03\x00\x00\x03',
> ENDKEY => '\x01\x03\x04\x02\x01\x01'}{ENCODED =>
> 8d4e9d9dbae0a90d47b9f72c89973693, NAME =>
> 'scooter_analytics_segment,\x01\x03\x04\x01\x02\x03,1583362144713.8d4e9d9dbae0a90d47b9f72c89973693.',
> STARTKEY => '\x01\x03\x04\x01\x02\x03', ENDKEY =>
> '\x01\x03\x04\x02\x02\x03'}{ENCODED => 2666b87ae4b58b05e815a3f14cc33215, NAME
> =>
> 'scooter_analytics_segment,\x01\x03\x04\x02\x01\x01,1583365243857.2666b87ae4b58b05e815a3f14cc33215.',
> STARTKEY => '\x01\x03\x04\x02\x01\x01', ENDKEY =>
> '\x01\x03\x04\x02\x02\x03'}{code}
> Then... here is the row in meta...
>
> {code:java}
>
> scooter_analytics_segment,\x01\x03\x04\x01\x02\x03,1583362144713.8d4e9d9dbae0a90d47b9f72c89973693.
> column=info:merge0000, timestamp=1583364483794,
> value=PBUF\x08\xE6\xE8\xFC\x9C\xA0-\x12$\x0A\x07default\x12\x19scooter_analytics_segment\x1A\x06\x01\x03\x04\x01\x02\x03"\x06\x01\x03\x04\x02\x02\x03(\x000\x008\x00B
> d097cacbd74c36e84a599926547e3249
>
> scooter_analytics_segment,\x01\x03\x04\x01\x02\x03,1583362144713.8d4e9d9dbae0a90d47b9f72c89973693.
> column=info:merge0001, timestamp=1583364483794,
> value=PBUF\x08\xC8\xF3\xF3\xBD\x8A.\x12$\x0A\x07default\x12\x19scooter_analytics_segment\x1A\x06\x01\x03\x04\x02\x01\x01"\x06\x01\x03\x04\x02\x02\x03(\x000\x008\x00B
> 6b2ac2d1878e688da50483014fc5ab58
>
> scooter_analytics_segment,\x01\x03\x04\x01\x02\x03,1583362144713.8d4e9d9dbae0a90d47b9f72c89973693.
> column=info:regioninfo, timestamp=1583364484283, value={ENCODED =>
> 8d4e9d9dbae0a90d47b9f72c89973693, NAME =>
> 'scooter_analytics_segment,\x01\x03\x04\x01\x02\x03,1583362144713.8d4e9d9dbae0a90d47b9f72c89973693.',
> STARTKEY => '\x01\x03\x04\x01\x02\x03', ENDKEY =>
> '\x01\x03\x04\x02\x02\x03'}
>
> scooter_analytics_segment,\x01\x03\x04\x01\x02\x03,1583362144713.8d4e9d9dbae0a90d47b9f72c89973693.
> column=info:seqnumDuringOpen, timestamp=1583364484283,
> value=\x00\x00\x00\x00\x00\x00\x010
>
> scooter_analytics_segment,\x01\x03\x04\x01\x02\x03,1583362144713.8d4e9d9dbae0a90d47b9f72c89973693.
> column=info:server, timestamp=1583364484283, value=a.example.com:16020
>
> scooter_analytics_segment,\x01\x03\x04\x01\x02\x03,1583362144713.8d4e9d9dbae0a90d47b9f72c89973693.
> column=info:serverstartcode, timestamp=1583364484283, value=1583349446638
>
> scooter_analytics_segment,\x01\x03\x04\x01\x02\x03,1583362144713.8d4e9d9dbae0a90d47b9f72c89973693.
> column=info:sn, timestamp=1583364483948,
> value=a.example.com,16020,1583349446638
>
> scooter_analytics_segment,\x01\x03\x04\x01\x02\x03,1583362144713.8d4e9d9dbae0a90d47b9f72c89973693.
> column=info:state, timestamp=1583364484283, value=OPEN
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
