[ https://issues.apache.org/jira/browse/HBASE-24376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17158914#comment-17158914 ]
Ruslan Sabitov commented on HBASE-24376: ---------------------------------------- [~huaxiangsun] thank you for your reply. Here is another example: 2020-06-04 17:31:59,611 INFO org.apache.hadoop.hbase.master.normalizer.MergeNormalizationPlan: Executing merging normalization plan: MergeNormalizationPlan\{firstRegion={ENCODED => 5d2d90c3697145c15bc2dc1841bbd140, NAME => 'tableName,-7-7d--TPxHgzWi3x6Nw,1591257734317.5d2d90c3697145c15bc2dc1841bbd140.', STARTKEY => '-7-7d--TPxHgzWi3x6Nw', ENDKEY => '-F--'}, secondRegion=\{ENCODED => 5efe3811574f9f390d4cc8fc6099aa18, NAME => 'tableName,-Myro7PP4ncDkwLSQqVv,1591257751881.5efe3811574f9f390d4cc8fc6099aa18.', STARTKEY => '-Myro7PP4ncDkwLSQqVv', ENDKEY => '-V–'}} I parsed HBase Master log and collected all lines with text MergeNormalizationPlan and where r1 ENDKEY is not equal r2 STARTKEY: [https://pastebin.com/5wqr28br] I'd like to add this table becomes inconsistent each time I enable normalization for the table. Do you have any ideas how to check that the problem in the normalization? > MergeNormalizer is merging non-adjacent regions and causing region > overlaps/holes. > ---------------------------------------------------------------------------------- > > Key: HBASE-24376 > URL: https://issues.apache.org/jira/browse/HBASE-24376 > Project: HBase > Issue Type: Bug > Components: master > Affects Versions: 2.3.0 > Reporter: Huaxiang Sun > Assignee: Huaxiang Sun > Priority: Critical > Fix For: 3.0.0-alpha-1, 2.3.0 > > > Currently, we found normalizer was merging regions which are non-adjacent, it > will cause inconsistencies in the cluster. > {code:java} > 439055 2020-05-08 17:47:09,814 INFO > org.apache.hadoop.hbase.master.normalizer.MergeNormalizationPlan: Executing > merging normalization plan: MergeNormalizationPlan{firstRegion={ENCODED => > 47fe236a5e3649ded95cb64ad0c08492, NAME => > 'TABLE,\x03\x01\x05\x01\x04\x02,1554838974870.47fe236a5e3649ded95cb64ad > 0c08492.', STARTKEY => '\x03\x01\x05\x01\x04\x02', ENDKEY => > '\x03\x01\x05\x01\x04\x02\x01\x02\x02201904082200\x00\x00\x03Mac\x00\x00\x00\x00\x00\x00\x00\x00\x00iMac13,1\x00\x00\x00\x00\x00\x049.3-14E260\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x05'}, > secondRegion={ENCODED => 0c0f2aa67f4329d5c4 8ba0320f173d31, NAME => > 'TABLE,\x03\x01\x05\x02\x01\x01,1554830735526.0c0f2aa67f4329d5c48ba0320f173d31.', > STARTKEY => '\x03\x01\x05\x02\x01\x01', ENDKEY => > '\x03\x01\x05\x02\x01\x02'}} > 439056 2020-05-08 17:47:11,438 INFO org.apache.hadoop.hbase.ScheduledChore: > CatalogJanitor-*****:16000 average execution time: 1676219193 ns. > 439057 2020-05-08 17:47:11,730 INFO org.apache.hadoop.hbase.master.HMaster: > Client=null/null merge regions [47fe236a5e3649ded95cb64ad0c08492], > [0c0f2aa67f4329d5c48ba0320f173d31] > {code} > > The root cause is that getMergeNormalizationPlan() uses a list of regionInfo > which is ordered by regionName. regionName does not necessary guarantee the > order of STARTKEY (let's say 'aa1', 'aa1!', in order of regionName, it will > be 'aa1!' followed by 'aa1'. This will result in normalizer merging > non-adjacent regions into one and creates overlaps. This is not an issue in > branch-1 as the list is already ordered by RegionInfo.COMPARATOR in > normalizer. > -- This message was sent by Atlassian Jira (v8.3.4#803005)