[
https://issues.apache.org/jira/browse/HBASE-24376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Huaxiang Sun updated HBASE-24376:
---------------------------------
Description:
Currently, we found normalizer was merging regions which are non-adjacent, it
will cause inconsistencies in the cluster.
{code:java}
439055 2020-05-08 17:47:09,814 INFO
org.apache.hadoop.hbase.master.normalizer.MergeNormalizationPlan: Executing
merging normalization plan: MergeNormalizationPlan{firstRegion={ENCODED =>
47fe236a5e3649ded95cb64ad0c08492, NAME =>
'TABLE,\x03\x01\x05\x01\x04\x02,1554838974870.47fe236a5e3649ded95cb64ad
0c08492.', STARTKEY => '\x03\x01\x05\x01\x04\x02', ENDKEY =>
'\x03\x01\x05\x01\x04\x02\x01\x02\x02201904082200\x00\x00\x03Mac\x00\x00\x00\x00\x00\x00\x00\x00\x00iMac13,1\x00\x00\x00\x00\x00\x049.3-14E260\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x05'},
secondRegion={ENCODED => 0c0f2aa67f4329d5c4 8ba0320f173d31, NAME =>
'TABLE,\x03\x01\x05\x02\x01\x01,1554830735526.0c0f2aa67f4329d5c48ba0320f173d31.',
STARTKEY => '\x03\x01\x05\x02\x01\x01', ENDKEY => '\x03\x01\x05\x02\x01\x02'}}
439056 2020-05-08 17:47:11,438 INFO org.apache.hadoop.hbase.ScheduledChore:
CatalogJanitor-*****:16000 average execution time: 1676219193 ns.
439057 2020-05-08 17:47:11,730 INFO org.apache.hadoop.hbase.master.HMaster:
Client=null/null merge regions [47fe236a5e3649ded95cb64ad0c08492],
[0c0f2aa67f4329d5c48ba0320f173d31]
{code}
The root cause is that getMergeNormalizationPlan() uses a list of regionInfo
which is ordered by regionName. regionName does not necessary guarantee the
order of STARTKEY (let's say 'aa1', 'aa1!', in order of regionName, it will be
'aa1!' followed by 'aa1'. This will result in normalizer merging non-adjacent
regions into one and creates overlaps. This is not an issue in branch-1 as the
list is already ordered by RegionInfo.COMPARATOR in normalizer.
was:
Currently, we found normalizer was merging regions which are non-adjacent, it
will cause inconsistencies in the cluster.
{code:java}
439055 2020-05-08 17:47:09,814 INFO
org.apache.hadoop.hbase.master.normalizer.MergeNormalizationPlan: Executing
merging normalization plan: MergeNormalizationPlan{firstRegion={ENCODED =>
47fe236a5e3649ded95cb64ad0c08492, NAME =>
'TABLE,\x03\x01\x05\x01\x04\x02,1554838974870.47fe236a5e3649ded95cb64ad
0c08492.', STARTKEY => '\x03\x01\x05\x01\x04\x02', ENDKEY =>
'\x03\x01\x05\x01\x04\x02\x01\x02\x02201904082200\x00\x00\x03Mac\x00\x00\x00\x00\x00\x00\x00\x00\x00iMac13,1\x00\x00\x00\x00\x00\x049.3-14E260\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x05'},
secondRegion={ENCODED => 0c0f2aa67f4329d5c4 8ba0320f173d31, NAME =>
'TABLE,\x03\x01\x05\x02\x01\x01,1554830735526.0c0f2aa67f4329d5c48ba0320f173d31.',
STARTKEY => '\x03\x01\x05\x02\x01\x01', ENDKEY => '\x03\x01\x05\x02\x01\x02'}}
439056 2020-05-08 17:47:11,438 INFO org.apache.hadoop.hbase.ScheduledChore:
CatalogJanitor-*****:16000 average execution time: 1676219193 ns.
439057 2020-05-08 17:47:11,730 INFO org.apache.hadoop.hbase.master.HMaster:
Client=null/null merge regions [47fe236a5e3649ded95cb64ad0c08492],
[0c0f2aa67f4329d5c48ba0320f173d31]
{code}
> MergeNormalizer is merging non-adjacent regions and causing region
> overlaps/holes.
> ----------------------------------------------------------------------------------
>
> Key: HBASE-24376
> URL: https://issues.apache.org/jira/browse/HBASE-24376
> Project: HBase
> Issue Type: Bug
> Components: master
> Affects Versions: 2.3.0
> Reporter: Huaxiang Sun
> Assignee: Huaxiang Sun
> Priority: Critical
>
> Currently, we found normalizer was merging regions which are non-adjacent, it
> will cause inconsistencies in the cluster.
> {code:java}
> 439055 2020-05-08 17:47:09,814 INFO
> org.apache.hadoop.hbase.master.normalizer.MergeNormalizationPlan: Executing
> merging normalization plan: MergeNormalizationPlan{firstRegion={ENCODED =>
> 47fe236a5e3649ded95cb64ad0c08492, NAME =>
> 'TABLE,\x03\x01\x05\x01\x04\x02,1554838974870.47fe236a5e3649ded95cb64ad
> 0c08492.', STARTKEY => '\x03\x01\x05\x01\x04\x02', ENDKEY =>
> '\x03\x01\x05\x01\x04\x02\x01\x02\x02201904082200\x00\x00\x03Mac\x00\x00\x00\x00\x00\x00\x00\x00\x00iMac13,1\x00\x00\x00\x00\x00\x049.3-14E260\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x05'},
> secondRegion={ENCODED => 0c0f2aa67f4329d5c4 8ba0320f173d31, NAME =>
> 'TABLE,\x03\x01\x05\x02\x01\x01,1554830735526.0c0f2aa67f4329d5c48ba0320f173d31.',
> STARTKEY => '\x03\x01\x05\x02\x01\x01', ENDKEY =>
> '\x03\x01\x05\x02\x01\x02'}}
> 439056 2020-05-08 17:47:11,438 INFO org.apache.hadoop.hbase.ScheduledChore:
> CatalogJanitor-*****:16000 average execution time: 1676219193 ns.
> 439057 2020-05-08 17:47:11,730 INFO org.apache.hadoop.hbase.master.HMaster:
> Client=null/null merge regions [47fe236a5e3649ded95cb64ad0c08492],
> [0c0f2aa67f4329d5c48ba0320f173d31]
> {code}
>
> The root cause is that getMergeNormalizationPlan() uses a list of regionInfo
> which is ordered by regionName. regionName does not necessary guarantee the
> order of STARTKEY (let's say 'aa1', 'aa1!', in order of regionName, it will
> be 'aa1!' followed by 'aa1'. This will result in normalizer merging
> non-adjacent regions into one and creates overlaps. This is not an issue in
> branch-1 as the list is already ordered by RegionInfo.COMPARATOR in
> normalizer.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
