Michael Stack created HBASE-24246:
-------------------------------------
Summary: Miscellaneous hbck2 fixMeta bulk merge fixes: better
logging around merges/overlap-fixing, 'HBCK Report' overlap listing, and
configuration
Key: HBASE-24246
URL: https://issues.apache.org/jira/browse/HBASE-24246
Project: HBase
Issue Type: Bug
Reporter: Michael Stack
Bit of a grab bag of small items.
Improve log messages when bulk merge runs (particularly out of 'hbck2 fixMeta').
Change config so we up the merge maximum since current low max is hit easily.
When its hit, its merge of merge of merges w/ confusing output while happening.
LOG when we hit max!
{code}
private static final String MAX_MERGE_COUNT_KEY =
"hbase.master.metafixer.max.merge.count";
private static final int MAX_MERGE_COUNT_DEFAULT = 10;
{code}
Fix the 'HBCK Report' page so it only logs encoded region names because if
region name is large, the UI becomes unreadable.
Fixup some logging messages. As is, can confuse operator.
E.g. explain why having a merge qualifier blocks a region being merged up
{code}
org.apache.hadoop.hbase.exceptions.MergeRegionException: Skip merging
[e1ded5983cb00c1a1a45e17414c5914c, 2634c71a10cf4a216b7c1608b7d24e6a,
e7d40bb7ab50cdeb20118043aedf2cdc, a9ea228464bd8a40b8908cd263e2f6be,
544d32880131443eebb2a01691994b0d, a82d462a6edca634e4f8ecff4e8bc55c,
03e30d277a1ab977f93c103cb917c7d7, d0d5fb364838b00942448b89b3d802ff,
00ba78a9be4b27cf076622f49c45a43e, 338efcfc48928815423d7a6601b9e477], because
parent [e1ded5983cb00c1a1a45e17414c5914c] has a merge qualifier
{code}
Add note to orphans page that you shouldn't try to address orphans until all
meta issues are fixed (overlaps, holes).
Add example on how to fix orphans with bulkload to the orphans section in hbck
report. See HBASE-23928.
Why the angle brackets in here:
{code}
Why angle brackets here? Usually for lists.
2020-04-23 15:21:13,471 INFO org.apache.hadoop.hbase.master.HMaster:
Client=hbase//10.103.230.91 merge regions [e1ded5983cb00c1a1a45e17414c5914c],
[2634c71a10cf4a216b7c1608b7d24e6a], [e7d40bb7ab50cdeb20118043aedf2cdc],
[a9ea228464bd8a40b8908cd263e2f6be], [544d32880131443eebb2a01691994b0d],
[a82d462a6edca634e4f8ecff4e8bc55c], [03e30d277a1ab977f93c103cb917c7d7],
[d0d5fb364838b00942448b89b3d802ff], [50cbee16764bb09a9887a07d3a01304f],
[338efcfc48928815423d7a6601b9e477]
{code}
These logs make no sense... need region name at least
{code}
What is this? Missing region
Whats it doing?
2020-04-23 15:36:19,896 WARN
org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress:
totalCompactingKVs=500851692 less than currentCompactedKVs=562814154
2020-04-23 15:36:22,953 WARN
org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress:
totalCompactingKVs=500851692 less than currentCompactedKVs=564904980
2020-04-23 15:36:26,011 WARN
org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress:
totalCompactingKVs=500851692 less than currentCompactedKVs=566948397
2020-04-23 15:36:29,071 WARN
org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress:
totalCompactingKVs=500851692 less than currentCompactedKVs=568943946
2020-04-23 15:36:32,153 WARN
org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress:
totalCompactingKVs=500851692 less than currentCompactedKVs=570961155
2020-04-23 15:36:35,219 WARN
org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress:
totalCompactingKVs=500851692 less than currentCompactedKVs=573055384
2020-04-23 15:36:38,274 WARN
org.apache.hadoop.hbase.regionserver.compactions.CompactionProgress:
totalCompactingKVs=500851692 less than currentCompactedKVs=575099844
{code}
This 'no state' is confusing... should be unknown region.
{code}
Why this message
2020-04-22 23:34:14,617 ERROR
org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure: Error
trying to merge [5aee12dd3e126b1fc4f5884be9712a5e,
1b4540d46c8ae2221759c72e328e3b28, 16f90525635f972bc984a498a17ceca4,
f22469447941caf3d885b860d2f9ac8b, 971e35bff0c4edfd5416db246f499ac9,
441010c4577ee79c07a857e507567dc2, 94eaaee33dd2d22eebf40dec418e1f1c,
d9a4fce4cc742a4d79cffb67a1b77510, b136369ec08d84d3761dcdde511c32e1,
a6f6f29ce45d368fe49651723cacce10] in bobby_analytics (in
state=MERGE_TABLE_REGIONS_PREPARE)
org.apache.hadoop.hbase.UnknownRegionException: No state for
[5aee12dd3e126b1fc4f5884be9712a5e]
at
org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.prepareMergeRegion(MergeTableRegionsProcedure.java:487)
at
org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:197)
at
org.apache.hadoop.hbase.master.assignment.MergeTableRegionsProcedure.executeFromState(MergeTableRegionsProcedure.java:77)
at
org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:194)
at
org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:962)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1669)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1416)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$1100(ProcedureExecutor.java:79)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1986)
{code}
Fix this log so can just copy paste it to shell... fix quotes:
org.apache.hadoop.hbase.exceptions.MergeRegionException: Unable to merge
non-adjacent or non-overlapping regions 9090c9927244d39d08dcf24335d2aa48,
04dc9c18f74937d2359e4ac2aaee9937 when force=false
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
