[
https://issues.apache.org/jira/browse/HBASE-17515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15839598#comment-15839598
]
Ted Yu commented on HBASE-17515:
--------------------------------
bq. justification than just 18 fields vs 4?
The 4 fields in BalancerRegionLoad would take 24 bytes (2 long's and 2 int's).
For 1M Regions, that would be ~24MB. If heap is 256G, this amount is pretty
small.
Among the fields in ClusterStatusProtos.RegionLoad, there is
storeCompleteSequenceId which is of type
List<org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterStatusProtos.StoreSequenceId>.
The savings given by BalancerRegionLoad would better than 1/4.
bq. Is 4G prohibitive on a master node w/ 256G of RAM?
In hbase 2.0, more functionality is shifted to master (split / merge / some
replication). Preserving heap for those components would be good practice -
considering potentially high number of regions in deployment.
bq. Should we be looking to a stats package
Which stats package(s) is recommended ?
bq. should we be working on shaving the size of RL which would have benefit all
around?
Let me dig into other classes which reference any of the 4 fields mentioned in
the description to see if they keep sliding window around.
> Reduce memory footprint of RegionLoads kept by StochasticLoadBalancer
> ---------------------------------------------------------------------
>
> Key: HBASE-17515
> URL: https://issues.apache.org/jira/browse/HBASE-17515
> Project: HBase
> Issue Type: Improvement
> Reporter: Ted Yu
> Assignee: Tim Brown
> Fix For: 2.0.0, 1.4.0
>
> Attachments: 17515.branch-1.v2.txt, 17515.v1.txt, 17515.v2.txt
>
>
> Currently StochasticLoadBalancer uses the following fields of RegionLoad :
> {code}
> return rl.getReadRequestsCount();
> return rl.getWriteRequestsCount();
> return rl.getMemStoreSizeMB();
> return rl.getStorefileSizeMB();
> {code}
> However, RegionLoad refers to ClusterStatusProtos.RegionLoad which has 18
> fields.
> This means we keep hbase.master.balancer.stochastic.numRegionLoadsToRemember
> (default value 15) RegionLoad's in memory but only use a small portion of
> them.
> This JIRA is to consider creating POJO which wraps the above 4 fields so that
> the memory footprint can be lowered.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
