[
https://issues.apache.org/jira/browse/HBASE-15249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15180623#comment-15180623
]
Aravindan Vijayan commented on HBASE-15249:
-------------------------------------------
On further investigation, we find that the normalizer is not causing an issue
for the AMS use case.
We ran the normalizer in the following scenarios and verified that it respects
the initial splits recommended by AMS.
1. Small cluster (~10 Nodes)
{code}
2016-02-29 23:22:29,372 DEBUG
[avijayan-hbase-3.novalocal,56004,1456787532496_ChoreService_1]
normalizer.SimpleRegionNormalizer: Computing normalization plan for table:
METRIC_RECORD, number of regions: 10
2016-02-29 23:22:29,372 DEBUG
[avijayan-hbase-3.novalocal,56004,1456787532496_ChoreService_1]
normalizer.SimpleRegionNormalizer: Table METRIC_RECORD, total aggregated
regions size: 0
2016-02-29 23:22:29,372 DEBUG
[avijayan-hbase-3.novalocal,56004,1456787532496_ChoreService_1]
normalizer.SimpleRegionNormalizer: Table METRIC_RECORD, average region size: 0.0
2016-02-29 23:22:29,372 DEBUG
[avijayan-hbase-3.novalocal,56004,1456787532496_ChoreService_1]
normalizer.SimpleRegionNormalizer: No normalization needed, regions look good
for table: METRIC_RECORD
{code}
2. Large cluster (~900 Nodes) with hbase.normalizer.period = 10 minutes
{code}
2016-03-04 20:55:14,168 DEBUG
[perf-a-2.c.pramod-thangali.internal,61300,1457124301027_ChoreService_1]
normalizer.SimpleRegionNormalizer: Computing normalization plan for table:
METRIC_RECORD, number of r
egions: 10
2016-03-04 20:55:14,168 DEBUG
[perf-a-2.c.pramod-thangali.internal,61300,1457124301027_ChoreService_1]
normalizer.SimpleRegionNormalizer: Table METRIC_RECORD, total aggregated
regions size: 157
2016-03-04 20:55:14,168 DEBUG
[perf-a-2.c.pramod-thangali.internal,61300,1457124301027_ChoreService_1]
normalizer.SimpleRegionNormalizer: Table METRIC_RECORD, average region size:
15.7
2016-03-04 20:55:14,169 DEBUG
[perf-a-2.c.pramod-thangali.internal,61300,1457124301027_ChoreService_1]
normalizer.SimpleRegionNormalizer: Table METRIC_RECORD, largest region
[B@6901ddd has size 37, more t
han 2 times than avg size, splitting
2016-03-04 20:55:14,170 INFO
[perf-a-2.c.pramod-thangali.internal,61300,1457124301027_ChoreService_1]
normalizer.SplitNormalizationPlan: Executing splitting normalization plan:
SplitNormalizationPlan{reg
ionInfo={ENCODED => 318e17da72832e72005e202f09a7ee55, NAME =>
'METRIC_RECORD,regionserver.Server.Increment_num_ops\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1457124324424.318e17da72832e72005e202f09a
7ee55.', STARTKEY =>
'regionserver.Server.Increment_num_ops\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',
ENDKEY =>
'rpc.rpc.CallQueueLength\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00'},
splitPoi
nt=null}
{code}
3. Large cluster with hbase.normalizer.period = 1 minute, to test normalizer
behavior for close to empty regions
{code}
2016-03-04 21:50:50,465 DEBUG
[perf-a-2.c.pramod-thangali.internal,61300,1457127998192_ChoreService_1]
normalizer.SimpleRegionNormalizer: Computing normalization plan for table:
METRIC_RECORD, number of r
egions: 10
2016-03-04 21:50:50,467 DEBUG
[perf-a-2.c.pramod-thangali.internal,61300,1457127998192_ChoreService_1]
normalizer.SimpleRegionNormalizer: Table METRIC_RECORD, total aggregated
regions size: 0
2016-03-04 21:50:50,467 DEBUG
[perf-a-2.c.pramod-thangali.internal,61300,1457127998192_ChoreService_1]
normalizer.SimpleRegionNormalizer: Table METRIC_RECORD, average region size: 0.0
2016-03-04 21:50:50,467 DEBUG
[perf-a-2.c.pramod-thangali.internal,61300,1457127998192_ChoreService_1]
normalizer.SimpleRegionNormalizer: No normalization needed, regions look good
for table: METRIC_RECOR
D
{code}
> Provide lower bound on number of regions in region normalizer for pre-split
> tables
> ----------------------------------------------------------------------------------
>
> Key: HBASE-15249
> URL: https://issues.apache.org/jira/browse/HBASE-15249
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Labels: normalization
> Attachments: HBASE-15249.v1.txt, HBASE-15249.v2.txt
>
>
> AMS (Ambari Metrics System) developer found the following scenario:
> Metrics table was pre-split with many regions on large cluster (1600 nodes).
> After some time, AMS stopped working because region normalizer merged the
> regions into few big regions which were not able to serve high read / write
> load.
> This is a big problem since the write requests flood the regions faster than
> the splits can happen resulting in poor performance.
> We should consider setting reasonable lower bound on region count.
> If the table is pre-split, we can use initial region count as the lower bound.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
